| *** baojg has quit IRC | 00:20 | |
| *** portdirect has joined #openstack-swift | 00:25 | |
| *** SkyRocknRoll has quit IRC | 00:28 | |
| *** kiennt has joined #openstack-swift | 00:47 | |
| *** mat128 has joined #openstack-swift | 01:27 | |
| *** bkopilov has quit IRC | 01:28 | |
| *** StevenK_ is now known as StevenK | 01:30 | |
| *** gkadam has joined #openstack-swift | 02:15 | |
| *** gkadam has quit IRC | 02:17 | |
| *** gkadam has joined #openstack-swift | 02:18 | |
| *** vint_bra has joined #openstack-swift | 02:33 | |
| mattoliverau | Wow its quiet in here when the US folk are on holidays. | 02:38 |
|---|---|---|
| *** mat128 has quit IRC | 02:39 | |
| *** bkopilov has joined #openstack-swift | 02:46 | |
| *** kiennt_ has joined #openstack-swift | 03:33 | |
| *** kiennt has quit IRC | 03:36 | |
| *** cshastri has joined #openstack-swift | 03:37 | |
| openstackgerrit | Pete Zaitcev proposed openstack/swift master: PUT+POST and its development test https://review.openstack.org/427911 | 03:37 |
| *** kiennt_ is now known as kiennt | 03:38 | |
| *** kiennt_ has joined #openstack-swift | 03:44 | |
| *** kiennt has quit IRC | 03:47 | |
| *** links has joined #openstack-swift | 03:51 | |
| *** vint_bra has quit IRC | 03:56 | |
| *** kei_yama has quit IRC | 03:59 | |
| *** kei_yama has joined #openstack-swift | 04:01 | |
| mahatic | good morning | 04:13 |
| mahatic | mattoliverau: o/ | 04:13 |
| *** kiennt_ is now known as kiennt_AWAY | 04:16 | |
| *** psachin has joined #openstack-swift | 04:21 | |
| *** klrmn has quit IRC | 04:21 | |
| *** pcaruana has joined #openstack-swift | 04:52 | |
| kota_ | mahatic, mattoliverau: o/ | 05:07 |
| mattoliverau | mahatic, kota_ o/ | 05:07 |
| mahatic | kota_: hello | 05:07 |
| kota_ | it's too quiet since yesterday so that i can find my voice on yesterday easily in the log | 05:08 |
| *** silor has joined #openstack-swift | 05:20 | |
| mattoliverau | kota_: seems we know what country makes the most noise :P | 05:28 |
| *** silor has quit IRC | 05:30 | |
| *** silor has joined #openstack-swift | 05:31 | |
| mahatic | :D | 05:36 |
| *** hoonetorg has quit IRC | 06:05 | |
| *** hseipp has joined #openstack-swift | 06:17 | |
| *** hoonetorg has joined #openstack-swift | 06:22 | |
| *** rcernin has joined #openstack-swift | 06:46 | |
| *** tesseract has joined #openstack-swift | 07:13 | |
| *** cschwede_ has joined #openstack-swift | 07:23 | |
| openstackgerrit | Kazuhiro MIYAHARA proposed openstack/swift master: Update Container-Sync Docs for internal_client.conf https://review.openstack.org/500703 | 07:24 |
| *** geaaru has joined #openstack-swift | 07:29 | |
| *** m_kazuhiro has joined #openstack-swift | 07:40 | |
| kota_ | mattoliverau: lol | 07:43 |
| *** m_kazuhiro has quit IRC | 08:12 | |
| *** cbartz has joined #openstack-swift | 08:23 | |
| *** ChubYann has quit IRC | 08:26 | |
| *** ChubYann has joined #openstack-swift | 08:40 | |
| *** kallenp has joined #openstack-swift | 08:49 | |
| kallenp | Hi all. Please, can I ask for a help howto copy swift storage node from production server (prod.domain.com) to devel server (devel.domain.com) ? What's the easyest way howto do it ? I google for it, but i don't find any usefuly howto. Thanks for a help. Petr | 08:51 |
| kallenp | Any idea howto do it ? Please ? | 09:08 |
| *** bkopilov has quit IRC | 10:31 | |
| *** kallenp has quit IRC | 10:39 | |
| *** kallenp has joined #openstack-swift | 10:40 | |
| *** gkadam has quit IRC | 10:57 | |
| *** kallenp has left #openstack-swift | 11:01 | |
| openstackgerrit | junbo.li proposed openstack/swift master: doc migration: update the doc link address https://review.openstack.org/500775 | 11:19 |
| openstackgerrit | junbo.li proposed openstack/swift master: doc migration: update the doc link address[2/3] https://review.openstack.org/500776 | 11:20 |
| openstackgerrit | junbo.li proposed openstack/swift master: doc migration: update the doc link address[1/3] https://review.openstack.org/500775 | 11:22 |
| openstackgerrit | junbo.li proposed openstack/swift master: doc migration: update the doc link address[3/3] https://review.openstack.org/500777 | 11:25 |
| *** skudlik has joined #openstack-swift | 11:26 | |
| *** skudlik has left #openstack-swift | 11:27 | |
| *** rcernin has quit IRC | 11:36 | |
| *** rcernin has joined #openstack-swift | 11:36 | |
| *** kei_yama has quit IRC | 11:44 | |
| *** kiennt_AWAY has quit IRC | 11:46 | |
| *** mat128 has joined #openstack-swift | 12:06 | |
| *** SkyRocknRoll has joined #openstack-swift | 12:17 | |
| *** mat128 has quit IRC | 12:33 | |
| *** mat128 has joined #openstack-swift | 12:34 | |
| *** cshastri has quit IRC | 12:45 | |
| *** skudlik has joined #openstack-swift | 12:50 | |
| *** juzuluag has joined #openstack-swift | 12:53 | |
| *** catintheroof has joined #openstack-swift | 12:55 | |
| *** catintheroof has quit IRC | 12:55 | |
| *** catintheroof has joined #openstack-swift | 12:55 | |
| *** zhurong has joined #openstack-swift | 12:56 | |
| *** bkopilov has joined #openstack-swift | 13:07 | |
| *** zhurong has quit IRC | 13:10 | |
| *** juzuluag has quit IRC | 13:34 | |
| *** SkyRocknRoll has quit IRC | 14:07 | |
| *** SkyRocknRoll has joined #openstack-swift | 14:09 | |
| *** amarao has joined #openstack-swift | 14:10 | |
| amarao | How to disable tempest tests which are marked as '@utils.requires_ext(extension='container_quotas', service='object')'. I've tried to use "[object-storage-feature-enabled] container_quotas=False", but it's ignored. | 14:12 |
| *** therve has joined #openstack-swift | 14:37 | |
| *** links has quit IRC | 14:37 | |
| *** vint_bra has joined #openstack-swift | 14:40 | |
| *** geaaru has quit IRC | 14:42 | |
| therve | Hi all. I opened https://bugs.launchpad.net/swift/+bug/1715177 , if anyone would like to chat about it | 14:43 |
| openstack | Launchpad bug 1715177 in OpenStack Object Storage (swift) "Container cache management is racy" [Undecided,New] | 14:43 |
| therve | cschwede_, ^^ | 14:43 |
| *** cbartz has quit IRC | 15:04 | |
| openstackgerrit | Daniel Čech proposed openstack/python-swiftclient master: Fixed capability discovery endpoint hardcode https://review.openstack.org/500875 | 15:08 |
| openstackgerrit | Daniel Čech proposed openstack/python-swiftclient master: Fixed capability discovery endpoint hardcode https://review.openstack.org/500875 | 15:09 |
| openstackgerrit | Daniel Čech proposed openstack/python-swiftclient master: Fixed capability discovery endpoint hardcode https://review.openstack.org/500875 | 15:15 |
| *** lucasxu has joined #openstack-swift | 15:17 | |
| openstackgerrit | Daniel Čech proposed openstack/python-swiftclient master: Fixed capability discovery endpoint hardcode https://review.openstack.org/500883 | 15:17 |
| *** gyee has joined #openstack-swift | 15:32 | |
| *** b4rti has quit IRC | 15:33 | |
| notmyname | good morning | 15:37 |
| notmyname | amarao: that is really interesting | 15:38 |
| notmyname | not in a good way ;-) | 15:38 |
| *** amarao has quit IRC | 15:48 | |
| *** skudlik has quit IRC | 15:53 | |
| *** psachin has quit IRC | 15:55 | |
| notmyname | acoles: I think the X-Backend-Recheck-Container-Existence is a red herring. yes, it's set, but yes, it's filtered by gateway | 16:21 |
| *** itlinux has joined #openstack-swift | 16:29 | |
| timburke | good morning | 16:37 |
| notmyname | so far I'm not able to repro the bug | 16:37 |
| timburke | eh? my bouncer seems to have lost some messages... | 16:39 |
| notmyname | https://bugs.launchpad.net/swift/+bug/1715177 | 16:39 |
| openstack | Launchpad bug 1715177 in OpenStack Object Storage (swift) "Container cache management is racy" [Undecided,New] | 16:39 |
| notmyname | from the 7 o'clock hour this morning | 16:39 |
| timburke | ah. try doing a delete to the container first | 16:39 |
| timburke | (i'm assuming. haven't actually tried it myself) | 16:39 |
| notmyname | I'm using the provided repro script which claims to break against devstack | 16:39 |
| notmyname | I'm not using devstack | 16:40 |
| notmyname | so it's entirely possible that my machine is too fast to see the bug | 16:40 |
| notmyname | I've definitely seen issues come to light on overburdened VMs before | 16:40 |
| timburke | i could be wrong, but i don't think that it actually requires concurrent requests. i think a container DELETE, followed by a container HEAD, followed by a container PUT, followed by an object PUT may have the PUT fail | 16:44 |
| timburke | let's find out! | 16:44 |
| timburke | nope, i stand corrected. unless there's something funny with the cache lifetimes... | 16:49 |
| notmyname | yeah, I think a slow VM may turn the repro script into concurrent actions, and it's not on my dev box ;-) | 16:49 |
| timburke | where *is* the cache time determined? | 16:50 |
| rcernin | kota_: timburke: do you knwo guys of IBM development progress on swift3? aparently this works in Red Hat provided openstack-swift-plugin-swift3. Do you know any background on swift3 with IBM? | 16:51 |
| notmyname | timburke: I think the trick is to follow that x-backend header value. it's ignored of course, but where it's actually used in the proxy is where it's setting the memcache ttl | 16:52 |
| timburke | rcernin: i know that IBM has expressed various levels of interest in swift3 development over time, but i don't know anything about how they use, deploy, or even package it | 16:58 |
| timburke | i know cbartz proposed https://review.openstack.org/#/c/475873/ not too long ago... | 16:58 |
| patchbot | patch 475873 - swift3 - Introduce auth middleware using account metadata. | 16:58 |
| *** tesseract has quit IRC | 17:25 | |
| *** rcernin has quit IRC | 17:25 | |
| *** hseipp has quit IRC | 17:29 | |
| *** alenavolk has joined #openstack-swift | 17:37 | |
| *** klrmn has joined #openstack-swift | 18:00 | |
| *** alenavolk has quit IRC | 18:01 | |
| *** alenavolk has joined #openstack-swift | 18:05 | |
| *** alenavol_ has joined #openstack-swift | 18:11 | |
| *** alenavolk has quit IRC | 18:11 | |
| *** alenavolk has joined #openstack-swift | 18:15 | |
| *** cschwede_ has quit IRC | 18:16 | |
| *** alenavol_ has quit IRC | 18:45 | |
| *** alenavolk has quit IRC | 18:45 | |
| *** pcaruana has quit IRC | 18:49 | |
| *** alenavolk has joined #openstack-swift | 18:49 | |
| *** alenavol_ has joined #openstack-swift | 18:49 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: doc migration: update the doc link address[1/3] https://review.openstack.org/500775 | 19:14 |
| *** chsc has joined #openstack-swift | 19:17 | |
| *** itlinux has quit IRC | 19:35 | |
| *** alenavol_ has quit IRC | 19:36 | |
| *** alenavolk has quit IRC | 19:36 | |
| *** alenavolk has joined #openstack-swift | 19:41 | |
| *** alenavol_ has joined #openstack-swift | 19:41 | |
| *** alenavol_ has quit IRC | 19:45 | |
| *** alenavolk has quit IRC | 19:45 | |
| *** mat128 has quit IRC | 19:51 | |
| *** mat128 has joined #openstack-swift | 19:51 | |
| *** mat128 has quit IRC | 19:52 | |
| *** mat128 has joined #openstack-swift | 19:54 | |
| therve | clayg, Hi! | 20:05 |
| clayg | heyoh! | 20:05 |
| therve | Here to chat about https://bugs.launchpad.net/swift/+bug/1715177 if you have a few minutes | 20:05 |
| openstack | Launchpad bug 1715177 in OpenStack Object Storage (swift) "Container cache management is racy" [Undecided,Opinion] | 20:05 |
| clayg | oh wow! thanks for jumpin on! | 20:06 |
| clayg | Do you have more than one proxy node (sometimes called "the controller"???) | 20:06 |
| therve | clayg, Yeah it's a single node devstack | 20:08 |
| therve | So no in fact | 20:08 |
| clayg | oh... well then how the heck do you get a 404 after a container PUT!? regardless of what's going on in that other thread... i could spin up a devstack and try to duplicate there... pretty strange! doesn't match my experience! | 20:09 |
| therve | Well, from what I can tell, there is nothing that protect cache setting | 20:09 |
| clayg | do you have a request log maybe? i'm not in a good spot to break off what I have up and start up a devstack node and play around with it more today than I have already... but if you're stumped I could try to look again w/i 24 hours | 20:09 |
| therve | So the put_container sets the cache, but the get_container overrides it with the wrong answer, IIUC | 20:10 |
| therve | Sure I have it running, I can paste you whatever you need | 20:10 |
| clayg | "protect" - we cache non-existance for sure - so if you 404 a GET well set the non-existence for I think ... idk 1/10 the existence check? But that code I pointed to (container PUT) will reset the cache - so a object PUT after container PUT should find the memcache entry for the container empty and refresh it and find the container. | 20:11 |
| clayg | yeah if you have the request LOG, ts1 container put, ts2 container get (with 404, *after* the PUT!?), ts3 object 404 | 20:12 |
| clayg | does it happen *every* time? I only ran it like maybe a half-a-dozen - if it's 1/1k maybe it would be reproducible and I just missed it? | 20:12 |
| timburke | i'm thinking if you do something like DELETE container, concurrently HEAD / PUT container, PUT object, there's a window where the GET gets a 404 but hasn't woken up yet, PUT goes through and invalidates the (not actually set) cache, GET wakes up and records the 404 | 20:14 |
| therve | clayg, No it happens about 3 out of 5? | 20:14 |
| therve | clayg, http://paste.openstack.org/show/620436/ | 20:14 |
| clayg | timburke: yeah that's what I thought initially, but the example script provided didn't include any DELETE | 20:14 |
| therve | timburke, Yeah no delete involved | 20:15 |
| clayg | therve: definitely not the same for me in my test - very strange - thanks for providing more information! | 20:15 |
| therve | clayg, FTR removing the time.sleep was important | 20:15 |
| therve | Otherwise you don't see the issue | 20:16 |
| therve | clayg, http://paste.openstack.org/show/620437/ cleaned up with just the requests | 20:19 |
| *** mat128 has quit IRC | 20:22 | |
| notmyname | FYI https://www.openstack.org/summit/berlin-2018 | 20:22 |
| *** mat128 has joined #openstack-swift | 20:23 | |
| therve | Nice | 20:23 |
| clayg | this sequence is somewhat damning... https://gist.github.com/clayg/ecaa44089599cfb7ace34ee758f467a5 | 20:23 |
| clayg | GET 404, PUT 201, PUT 404 | 20:24 |
| clayg | the interesting bit is that by my reading the PUT started before, but finished *after* - the GET | 20:25 |
| clayg | then the object PUT 404's | 20:25 |
| clayg | therve: I think it's it's not nothing! | 20:25 |
| therve | clayg, Yeah I try not to cry wolf :) | 20:26 |
| therve | It's possible the answer is "please don't do that" though. I think I can find some kind of workaround | 20:27 |
| tdasilva | therve, clayg: fwiw, i'm not able to reproduce it on my own dev SAIO, but it was easy to reproduce with tripleo | 20:27 |
| therve | tdasilva, With the script on the undercloud? | 20:27 |
| tdasilva | yes | 20:27 |
| tdasilva | i'm using clayg's script | 20:27 |
| therve | Ah thanks for trying | 20:27 |
| clayg | tdasilva: i'm noticing a huge part of the request time is spent negotiating with keystone in this log snippet - so that's one commonality between different environments where we *can* and *can not* reproduce | 20:28 |
| therve | tdasilva, Do you have memcache in your saio? | 20:28 |
| tdasilva | therve: yes | 20:28 |
| tdasilva | therve: i use this for my saio: https://github.com/thiagodasilva/ansible-saio | 20:28 |
| clayg | anyway - my guess now is that container put should do it's cache invalidation as close as possible to the end of the request... (instead of *before* make_request) | 20:29 |
| tdasilva | clayg: and yeah, some of the differences between tripleo (especially undercloud) and a typical saio is keystone and also 1 replica vs 3 replicas | 20:29 |
| clayg | therve: are you in a situation you could hack that up in swift.proxy.controllers.container around L184? | 20:29 |
| clayg | just move clear_info_cache after make_requests and restart your proxies | 20:30 |
| therve | clayg, Sure | 20:30 |
| timburke | clayg: so with the randomly-generated container name, the DELETE isn't necessary. i can reproduce on my vsaio using curl like http://paste.openstack.org/show/620439/ | 20:31 |
| clayg | cool - GL - be sure to post results to the bug report (either positive or negative results would be helpful I think, diffs always help) | 20:31 |
| timburke | gets me output like http://paste.openstack.org/show/620438/ | 20:31 |
| therve | clayg, Seems to do the trick! | 20:31 |
| clayg | sudo apt install uuid | 20:33 |
| * therve spent days on that issue, excited to see a fix with a 2 lines diff :D | 20:35 | |
| clayg | timburke: is your saio running keystone? your script doesn't result in 404 for me? | 20:35 |
| timburke | nope. try it repeatedly. first time i got a 201, very next run (iirc) got the doubled-up 404s | 20:35 |
| clayg | the HEAD (even in the background) is just too quick? maybe my vm forks faster than yours? | 20:35 |
| clayg | timburke: lol - i'd been running it then - *finally* it 404'd - KUDOS! | 20:36 |
| clayg | next 50 all 201'd | 20:37 |
| * clayg shrugs | 20:37 | |
| timburke | nature of race conditions, man | 20:37 |
| clayg | anyway - PUT invaliding before the HEAD returns (and caches non-existance) seems to make sense to me! | 20:37 |
| therve | Yeah it probably helps I have a small VM on my laptop | 20:38 |
| clayg | anyway - from my mind this is ... sort of well understood - i mean it's a cache over an eventually consistent system for goodness sake - but I like the idea of doing the PUT cache invalidation at the last second better than the wider race window - so I'm totally down. | 20:39 |
| therve | Great | 20:39 |
| therve | Do you want me to post the patch? | 20:39 |
| clayg | the more of that we can capture that lp bug #1715177 the better | 20:40 |
| openstack | Launchpad bug 1715177 in OpenStack Object Storage (swift) "Container cache management is racy" [Medium,Confirmed] https://launchpad.net/bugs/1715177 | 20:40 |
| clayg | therve: well ... i mean we're gunna wanna a test to demonstrate the race | 20:40 |
| therve | clayg, Yeah that one is going to be fun to write :) | 20:40 |
| clayg | but you can get the ball rolling with a patch on the bug report or in gerrit | 20:40 |
| therve | OK I'll do that now, and see if I can find a way to test tomorrow | 20:40 |
| clayg | it's a big help just demonstrating the issue! thanks again for sticking with it! | 20:40 |
| timburke | moving the invalidation should narrow the window, but it'll still exist. in general, though, containers are pretty long-lived -- if a client gets a 404 during an object put, it should wait a bit, try again, and if it *still* fails, get noisy, as something's misconfigured | 20:41 |
| clayg | great GL! I'm sure if you get something up on gerrit timburke or mattoliverau would help you with a test - they love that racy concurrency stuff! | 20:41 |
| therve | timburke, yeah that was one of my questions as well. I can certainly make the client code more robust | 20:41 |
| clayg | timburke: If possible in the ideal the recovery code for errors on object PUT would grow container create logic to deal with 404 | 20:42 |
| clayg | therve: thanks again - you too timburke! | 20:43 |
| timburke | clayg: maybe, but not necessarily -- whoever's in charge of putting objects into the container may not have permission to create one | 20:43 |
| *** rcernin has joined #openstack-swift | 20:44 | |
| openstackgerrit | Thomas Herve proposed openstack/swift master: Delay cache invalidation during container creation https://review.openstack.org/500978 | 20:44 |
| therve | Thanks for your help! | 20:44 |
| * therve off for now | 20:45 | |
| *** lucasxu has quit IRC | 20:48 | |
| *** silor has quit IRC | 20:50 | |
| *** mat128 has quit IRC | 21:01 | |
| *** itlinux has joined #openstack-swift | 21:13 | |
| *** catintheroof has quit IRC | 21:43 | |
| *** rcernin has quit IRC | 21:53 | |
| *** itlinux has quit IRC | 22:55 | |
| notmyname | mattoliverau: are you still on for the 0700 meeting? | 22:57 |
| mattoliverau | notmyname: oh yeah, it's wednesday. Sure, I'll have a look at what I can add/remove from the adjenda | 23:06 |
| notmyname | ok, thanks | 23:07 |
| notmyname | IMO it's really just prep for next week | 23:07 |
| mattoliverau | yup +1 | 23:07 |
| mattoliverau | oh and morning :) | 23:08 |
| acoles | mattoliverau: I won't make the 0700 this week, just arrived in SF so will definitely be asleep at that time | 23:21 |
| mattoliverau | acoles: wheres the dedication man :P Yeah that's fine, enjoy your sleep, being jetlagged means you probably looking forward to it ;) | 23:23 |
| mattoliverau | if it ends up being just me, I'll talk to myself again :P | 23:23 |
| acoles | mattoliverau: the dedication is somewhere over greenland ;) | 23:23 |
| mattoliverau | lol | 23:24 |
| *** kei_yama has joined #openstack-swift | 23:25 | |
| *** catintheroof has joined #openstack-swift | 23:26 | |
| zaitcev | I dunno, I don't think I can make a 1 a.m. meeting | 23:28 |
| zaitcev | Anyway, I was making a great progress with cleaning up PUT+POST. I assumed that we actually want it merged, so it needed proper error catches and such. | 23:29 |
| zaitcev | There's not much to discuss about it short of actually reviewing. | 23:29 |
| *** catintheroof has quit IRC | 23:31 | |
| mattoliverau | zaitcev: 1am it too much, but that's why we also have the original meeting in your timezone where we can discuss PUT+POST. But I can tell people to go review it in the 0700 meeting if you want :) | 23:31 |
| *** chsc has quit IRC | 23:32 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!