| openstackgerrit | Pete Zaitcev proposed openstack/swift master: Drop unnecessary dereferencing https://review.openstack.org/464096 | 00:13 |
|---|---|---|
| *** dja_ has quit IRC | 00:37 | |
| *** dja has joined #openstack-swift | 00:38 | |
| *** patchbot has quit IRC | 00:54 | |
| *** patchbot has joined #openstack-swift | 00:55 | |
| *** kei_yama_ has joined #openstack-swift | 01:09 | |
| *** NM has quit IRC | 01:15 | |
| *** catintheroof has quit IRC | 01:18 | |
| *** tovin07_ has joined #openstack-swift | 01:20 | |
| *** winggundamth has joined #openstack-swift | 02:04 | |
| *** winggundamth_ has joined #openstack-swift | 02:24 | |
| *** winggundamth has quit IRC | 02:27 | |
| *** catintheroof has joined #openstack-swift | 02:40 | |
| *** catintheroof has quit IRC | 03:57 | |
| *** hoonetorg has quit IRC | 04:13 | |
| *** hoonetorg has joined #openstack-swift | 04:26 | |
| *** adriant has quit IRC | 04:44 | |
| *** klrmn has quit IRC | 04:49 | |
| *** jaosorior_away has quit IRC | 04:58 | |
| *** ChubYann has quit IRC | 05:25 | |
| *** jaosorior has joined #openstack-swift | 05:41 | |
| *** tovin07_ has quit IRC | 06:01 | |
| *** pcaruana has joined #openstack-swift | 06:14 | |
| *** rcernin has joined #openstack-swift | 06:18 | |
| *** cbartz has joined #openstack-swift | 06:36 | |
| *** gabor_antal has quit IRC | 06:41 | |
| *** jamielennox is now known as jamielennox|away | 06:53 | |
| *** cshastri has joined #openstack-swift | 06:54 | |
| *** hseipp has joined #openstack-swift | 06:58 | |
| *** gabor_antal has joined #openstack-swift | 06:59 | |
| *** tesseract has joined #openstack-swift | 07:12 | |
| *** hseipp has quit IRC | 08:01 | |
| *** cshastri has quit IRC | 08:42 | |
| *** cshastri has joined #openstack-swift | 08:53 | |
| *** rcernin has quit IRC | 09:05 | |
| *** cshastri_ has joined #openstack-swift | 09:08 | |
| *** d0ugal has quit IRC | 09:08 | |
| *** cshastri has quit IRC | 09:11 | |
| *** rcernin has joined #openstack-swift | 09:14 | |
| *** d0ugal has joined #openstack-swift | 09:16 | |
| *** silor has joined #openstack-swift | 09:25 | |
| *** d0ugal has quit IRC | 09:25 | |
| *** d0ugal has joined #openstack-swift | 09:47 | |
| *** bkopilov has quit IRC | 09:54 | |
| *** winggundamth_ has quit IRC | 10:05 | |
| *** geaaru has joined #openstack-swift | 10:05 | |
| *** winggundamth has joined #openstack-swift | 10:06 | |
| *** cshastri_ is now known as cshastri | 10:21 | |
| *** G-DUO-CO has quit IRC | 11:07 | |
| *** G-DUO-CO has joined #openstack-swift | 11:09 | |
| *** silor has quit IRC | 11:20 | |
| *** hseipp has joined #openstack-swift | 11:28 | |
| *** kei_yama has quit IRC | 11:31 | |
| *** chlong has quit IRC | 11:36 | |
| *** zul has joined #openstack-swift | 11:42 | |
| *** hseipp has quit IRC | 12:03 | |
| *** NM has joined #openstack-swift | 12:23 | |
| *** NM has quit IRC | 12:28 | |
| *** jamielennox|away is now known as jamielennox | 12:30 | |
| *** xinli has joined #openstack-swift | 13:14 | |
| *** NM has joined #openstack-swift | 13:26 | |
| tdasilva | good morning | 13:28 |
| -openstackstatus- NOTICE: The Gerrit service on http://review.openstack.org is being restarted to address hung remote replication tasks. | 13:38 | |
| *** klamath has joined #openstack-swift | 13:45 | |
| *** klamath has joined #openstack-swift | 13:45 | |
| *** rcernin has quit IRC | 13:55 | |
| *** zul has quit IRC | 14:03 | |
| *** rcernin has joined #openstack-swift | 14:05 | |
| *** zul has joined #openstack-swift | 14:07 | |
| *** winggundamth has quit IRC | 14:07 | |
| clayg | good morning! | 14:08 |
| *** chlong has joined #openstack-swift | 14:08 | |
| timburke | clayg: https://review.openstack.org/#/c/463876/ | 14:11 |
| patchbot | patch 463876 - swift - Open-code eventlet.listen() | 14:11 |
| *** rcernin has quit IRC | 14:17 | |
| *** _JZ_ has joined #openstack-swift | 14:23 | |
| *** rcernin has joined #openstack-swift | 14:25 | |
| *** cshastri has quit IRC | 14:33 | |
| *** jistr is now known as jistr|afk | 14:44 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: domain_remap: be more careful about client-path mangling https://review.openstack.org/464216 | 15:01 |
| openstackgerrit | Tim Burke proposed openstack/swift master: domain_remap: stop mangling client-provided paths https://review.openstack.org/464217 | 15:01 |
| clayg | zaitcev: rledisez: now that eventlet's listen is setting SO_REUSEPORT maybe it's time to dust off patch 137659 | 15:04 |
| patchbot | https://review.openstack.org/#/c/137659/ - swift - Support of the Linux socket option SO_REUSEPORT (ABANDONED) | 15:04 |
| *** xinli has quit IRC | 15:04 | |
| clayg | I think I had something in the follow up that allowed for smooth upgrade (which is still interesting for me, I only package eventlet 0.19 currently) | 15:05 |
| zaitcev | clayg: yeah... well, wait. Is it possible to know if a function accepts an argument? Or try: | 15:06 |
| *** rcernin has quit IRC | 15:06 | |
| clayg | so the problem IIRC was in process management - if we *can* bind with SO_REUSEPORT there was no reason not to - the change was mostly about making `swift-init proxy reload` send HUP and start up new servers right away (expecting to be able to bind to a port that's already in-use) | 15:07 |
| clayg | but for in order for the new processes to bind to in-use port - the *existing* processes have to have had bound with SO_REUSEPORT - which they would not have in an upgrade - so swift-init reload had to degrade gracefully (try to bind but if failing, wait for old procs to die then try bind again) | 15:08 |
| clayg | or something - i forget | 15:08 |
| clayg | zaitcev - I can't get tests failing locally? | 15:08 |
| clayg | for i in {1..10}; do SWIFT_TEST_IN_PROCESS=1 nosetests swift/test/functional/tests.py:TestFile -vsx; if [ $? -ne 0 ]; then exit 1; fi; done | 15:09 |
| zaitcev | That sounds about right | 15:09 |
| *** caccola has joined #openstack-swift | 15:09 | |
| zaitcev | clayg: what do you mean, are you trying to get tests to fail in order to test 463876? | 15:10 |
| clayg | timburke: when you were looking at patch 463876 were you able to get tests to fail reliably? | 15:10 |
| patchbot | https://review.openstack.org/#/c/463876/ - swift - Open-code eventlet.listen() | 15:10 |
| clayg | I wanted to both +a that change - but also let temoto know he's good to go with his fix (even w/o zaitcev's fix) | 15:11 |
| clayg | zaitcev: yeah I was hoping I could get a reliable repro | 15:11 |
| timburke | clayg: this has always been my trouble with all of this! i've *never* been able to reliably reproduce this locally. or even *at all*, iirc | 15:11 |
| clayg | wow, that's annoying! | 15:11 |
| zaitcev | clayg: if yes, then first go to https://github.com/eventlet/eventlet/issues/411 and copy-paste the test from there. It fails in a couple of seconds on a box that's vulnerable. However, few of mine are. Needs a very fresh eventlet, not the one that RDO ships. | 15:12 |
| clayg | i verified the eventlet I have installed does set SO_REUSEPORT - i'll try the gist - thanks! | 15:12 |
| timburke | oh yeah...maybe *that* was part of it... | 15:12 |
| clayg | Interesting! Port 44845 dup in 35 tries | 15:13 |
| zaitcev | okay, so you're good to go and try functests | 15:13 |
| *** xinli has joined #openstack-swift | 15:13 | |
| zaitcev | However, that will only blow up if there's a pressure on ports on the box | 15:13 |
| clayg | but most are 200-300 tries - so it's probably only 1/1000 times that spinning up the functests cause a hit :'( | 15:14 |
| zaitcev | quite | 15:14 |
| timburke | i totally hit it with the repro script -- first one was like 51 tries | 15:16 |
| timburke | i'm thinking it's worse on a jenkins host where other ports may already be tied up | 15:17 |
| timburke | (due to other, concurrent jobs) | 15:17 |
| clayg | echo 32768 32868 > /proc/sys/net/ipv4/ip_local_port_range | 15:17 |
| clayg | tests failed on the second try ;) | 15:18 |
| zaitcev | good call | 15:18 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Clean up how PatchPolicies works https://review.openstack.org/460362 | 15:24 |
| zaitcev | I found typos in a commit message. "out gate" instead of "our gate", and "tests tries", listen(("localhost",0) with unbalanced parenthesis. Go me the sloppy programmer. | 15:31 |
| openstackgerrit | Merged openstack/swift master: Fix domain_remap when obj starts/ends with slash https://review.openstack.org/458850 | 15:47 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Clean up how PatchPolicies works https://review.openstack.org/460362 | 15:49 |
| *** xinli has quit IRC | 15:49 | |
| clayg | timburke: i'm sure my response is no surprise on patch 464217 - I'd update the commit message at a minimum if you want a second option as not everyone will have domain_remap weird behavior loaded in their heads | 15:51 |
| patchbot | https://review.openstack.org/#/c/464217/ - swift - domain_remap: stop mangling client-provided paths | 15:51 |
| *** cbartz has left #openstack-swift | 15:52 | |
| *** jistr|afk is now known as jistr | 16:06 | |
| *** pcaruana has quit IRC | 16:12 | |
| *** xinli has joined #openstack-swift | 16:13 | |
| openstackgerrit | Merged openstack/swift master: Open-code eventlet.listen() https://review.openstack.org/463876 | 16:46 |
| *** klrmn has joined #openstack-swift | 16:55 | |
| *** gyee has joined #openstack-swift | 17:03 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: fixup! Add container sharding to Swift containers https://review.openstack.org/450491 | 17:05 |
| *** jaosorior has quit IRC | 17:09 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Clean up how PatchPolicies works https://review.openstack.org/460362 | 17:11 |
| *** JimCheung has joined #openstack-swift | 17:17 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift master: Updated from global requirements https://review.openstack.org/88736 | 17:18 |
| *** hseipp has joined #openstack-swift | 17:20 | |
| *** hseipp has quit IRC | 17:33 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Apply remote metadata in _handle_sync_response https://review.openstack.org/464084 | 17:52 |
| *** Renich has joined #openstack-swift | 18:20 | |
| *** geaaru has quit IRC | 18:36 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Clean up how PatchPolicies works https://review.openstack.org/460362 | 18:36 |
| *** tesseract has quit IRC | 19:00 | |
| *** ChubYann has joined #openstack-swift | 19:05 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Limit number of revert tombstone SSYNC requests https://review.openstack.org/439572 | 19:13 |
| clayg | kota_: I wonder if you could take a look at an old optimization that mahatic had worked on - patch 439572 | 19:18 |
| patchbot | https://review.openstack.org/#/c/439572/ - swift - Limit number of revert tombstone SSYNC requests | 19:18 |
| clayg | kota_: I think the change is good small improvement; but maybe for the ec duplication case there should be more improvement? | 19:19 |
| zaitcev | http://logs.openstack.org/96/464096/1/check/gate-swift-tox-xfs-tmp-py27-ubuntu-xenial/fd5d6dc/console.html <==== wildly mysterious | 19:26 |
| clayg | this one looks like the listen_zero thing? AssertionError: 'HTTP/1.1 400' != 'HTTP/1.1 201' | 19:34 |
| clayg | this one too: AssertionError: Expected 'HTTP/1.1 201', encountered 'HTTP/1.1 400' | 19:34 |
| zaitcev | It's a concern. Seemed completely safe though, and of course everything passes elsewhere. | 19:35 |
| zaitcev | Also, there's another py27-on-xenial gate. What is difference between the two? This one has "tmp-xfs" in its name. | 19:36 |
| clayg | no I mean... AFAIK the listen_zero fixed that error? can you tell if that change was applied to that job? | 19:37 |
| zaitcev | oh | 19:37 |
| zaitcev | but the console also has things like | 19:37 |
| zaitcev | File "/home/jenkins/workspace/gate-swift-tox-xfs-tmp-py27-ubuntu-xenial/swift/obj/diskfile.py", line 555, in __getitem__ | 19:37 |
| zaitcev | return self.policy_to_manager[policy] | 19:37 |
| zaitcev | KeyError: StoragePolicy(0, ['zero'], is_default=True, is_deprecated=False, policy_type='replication') | 19:37 |
| zaitcev | This is just insane | 19:37 |
| clayg | so @timburke is trying to fix the KeyError thing over in patch 60362 | 19:39 |
| patchbot | https://review.openstack.org/#/c/60362/ - nova (stable/havana) - Allow resizes and migrates during live upgrade (MERGED) | 19:39 |
| clayg | ... but I think i sorta derailed he and @acoles - maybe you could give it a quick look? | 19:39 |
| zaitcev | that patch number seems very, very low | 19:39 |
| zaitcev | Maybe patch 460362? | 19:39 |
| patchbot | https://review.openstack.org/#/c/460362/ - swift - Clean up how PatchPolicies works | 19:39 |
| timburke | only i'm still not sure that it actually accomplishes that... | 19:39 |
| clayg | sorry copy and paste error! thanks! | 19:40 |
| timburke | but yeah, i want to sort that out in the near future | 19:40 |
| rledisez | clayg: REUSEPORT upstream, I want that :) it was my first patch submission to swift | 19:40 |
| clayg | zaitcev: is the gate not going to process patch 464096 | 19:40 |
| patchbot | https://review.openstack.org/#/c/464096/ - swift - Drop unnecessary dereferencing | 19:40 |
| clayg | rledisez: you're not home already? hotel? plane!? | 19:40 |
| zaitcev | clayg, it's stuck, yes. But it's not an important patch. I can easily let it hang around for a week or two while we're making sure the gate works. | 19:41 |
| rledisez | clayg: i'm at hotel, leaving to airport in 30 minutes | 19:41 |
| timburke | clayg: what test was that AssertionError from? i think i saw that wort of a message while running down https://review.openstack.org/#/c/461860/ | 19:42 |
| patchbot | patch 461860 - swift - Fix intermittent failure in test_POST_but_expired | 19:42 |
| clayg | timburke: i just grabbed it out of the zaitcev 's log there ... but I had seen similar stuff when looking at the listen_zero thing (in unittests!) | 19:44 |
| clayg | timburke: not sure about patch 461860 | 19:49 |
| patchbot | https://review.openstack.org/#/c/461860/ - swift - Fix intermittent failure in test_POST_but_expired | 19:49 |
| clayg | which request blows up exactly? | 19:56 |
| *** JimCheung has quit IRC | 20:04 | |
| *** JimCheung has joined #openstack-swift | 20:19 | |
| clayg | timburke: sorry, nm you're probably on the plane - i found the gate log in your comment on the patch - but I still have a hard time believing the guest was paused for 1000 seconds?! (assuming the log 201 != 400 means one of the PUT's outside were returning 400 for delete_at_timestamp "in the past" compared to test_time) | 20:28 |
| *** xinli has quit IRC | 21:15 | |
| openstackgerrit | Merged openstack/swift master: Drop unnecessary dereferencing https://review.openstack.org/464096 | 21:26 |
| zaitcev | So, it was intermittent after all | 21:31 |
| *** NM has quit IRC | 21:32 | |
| *** Guest44841 is now known as asettle | 21:33 | |
| openstackgerrit | Alistair Coles proposed openstack/swift master: Clean up how PatchPolicies works https://review.openstack.org/460362 | 21:59 |
| *** geaaru has joined #openstack-swift | 22:32 | |
| *** klrmn has quit IRC | 22:52 | |
| *** chlong has quit IRC | 23:00 | |
| *** JimCheung has quit IRC | 23:23 | |
| *** JimCheung has joined #openstack-swift | 23:30 | |
| *** _JZ_ has quit IRC | 23:30 | |
| *** NM has joined #openstack-swift | 23:36 | |
| *** klamath has quit IRC | 23:36 | |
| *** klrmn has joined #openstack-swift | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!