| *** abhirc_ has quit IRC | 00:00 | |
| *** abhirc has joined #openstack-swift | 00:01 | |
| *** abhirc has quit IRC | 00:06 | |
| *** dmsimard is now known as dmsimard_away | 00:10 | |
| *** ho__ has joined #openstack-swift | 00:13 | |
| ho__ | good morning guys! | 00:13 |
|---|---|---|
| openstackgerrit | Clay Gerrard proposed openstack/swift: Cleanup object-replicator handoff/rebalance partition syncing https://review.openstack.org/158927 | 00:18 |
| *** david-lyle is now known as david-lyle_afk | 00:25 | |
| *** km has joined #openstack-swift | 00:32 | |
| openstackgerrit | Pete Zaitcev proposed openstack/swift: Clean up a couple of deprecated uses of e.message https://review.openstack.org/158937 | 00:48 |
| zaitcev | toe-to-toe with torgomatic ^_^ | 00:48 |
| torgomatic | :) | 00:49 |
| torgomatic | they're everywhere! | 00:49 |
| *** dmorita has joined #openstack-swift | 00:52 | |
| *** zhill has quit IRC | 00:52 | |
| *** abhirc_ has joined #openstack-swift | 00:56 | |
| openstackgerrit | Hisashi Osanai proposed openstack/swift: Enable Object Replicator's failure count in recon https://review.openstack.org/138342 | 01:04 |
| *** AbyssOne_ has quit IRC | 01:05 | |
| *** AbyssOne_ has joined #openstack-swift | 01:07 | |
| *** chmouel has quit IRC | 01:08 | |
| *** openstack has joined #openstack-swift | 01:12 | |
| zaitcev | awww, Sam | 01:14 |
| zaitcev | Put "recheck no bug" and the check failed, although it was succeeded before. Only needed the verify step. | 01:15 |
| *** AbyssOne__ has joined #openstack-swift | 01:16 | |
| *** delattec has quit IRC | 01:16 | |
| *** AbyssOne_ has quit IRC | 01:16 | |
| *** wolsen has quit IRC | 01:16 | |
| *** tristanC has quit IRC | 01:16 | |
| *** morganfainberg has quit IRC | 01:16 | |
| *** ondergetekende_ has quit IRC | 01:16 | |
| *** abhirc_ has quit IRC | 01:16 | |
| torgomatic | zaitcev: interesting; I should update my script to take that into consideration then | 01:17 |
| torgomatic | I think recheck and reverify were equivalent when I wrote it | 01:17 |
| zaitcev | I'd be afraid for infra guys to put a hit on my head if I scripted "recheck no bug". | 01:18 |
| torgomatic | it'll pick up bug numbers if elastic recheck writes one down | 01:18 |
| *** wolsen has joined #openstack-swift | 01:20 | |
| *** tristanC has joined #openstack-swift | 01:20 | |
| *** morganfainberg has joined #openstack-swift | 01:20 | |
| *** ondergetekende_ has joined #openstack-swift | 01:20 | |
| *** Manshoon has joined #openstack-swift | 01:21 | |
| openstackgerrit | Merged openstack/swift: Fix failing tox -e py26 tests https://review.openstack.org/158784 | 01:23 |
| *** Manshoon has quit IRC | 01:26 | |
| *** dmorita_ has joined #openstack-swift | 01:31 | |
| *** dmorita has quit IRC | 01:31 | |
| *** gyee has quit IRC | 01:35 | |
| *** panbalag has quit IRC | 01:35 | |
| *** ekarlso has quit IRC | 01:35 | |
| *** mordred has quit IRC | 01:35 | |
| *** anteaya has quit IRC | 01:35 | |
| *** kei_yama has quit IRC | 01:35 | |
| *** greghaynes has quit IRC | 01:35 | |
| *** mrda has quit IRC | 01:35 | |
| *** cschwede has quit IRC | 01:35 | |
| *** jbonjean has quit IRC | 01:35 | |
| *** j_king_ has quit IRC | 01:35 | |
| *** AbyssOne has quit IRC | 01:35 | |
| *** dosaboy has quit IRC | 01:35 | |
| *** zackmdavis has quit IRC | 01:35 | |
| *** dfg has quit IRC | 01:35 | |
| *** charz has quit IRC | 01:35 | |
| *** mandarine has quit IRC | 01:35 | |
| *** swifterdarrell has quit IRC | 01:35 | |
| *** thurloat has quit IRC | 01:35 | |
| *** btorch has quit IRC | 01:35 | |
| *** ndk has quit IRC | 01:35 | |
| *** straycat has quit IRC | 01:35 | |
| *** hurricanerix_ has quit IRC | 01:35 | |
| *** gyee has joined #openstack-swift | 01:39 | |
| *** kei_yama has joined #openstack-swift | 01:39 | |
| *** panbalag has joined #openstack-swift | 01:39 | |
| *** greghaynes has joined #openstack-swift | 01:39 | |
| *** ekarlso has joined #openstack-swift | 01:39 | |
| *** cschwede has joined #openstack-swift | 01:39 | |
| *** mrda has joined #openstack-swift | 01:39 | |
| *** jbonjean has joined #openstack-swift | 01:39 | |
| *** straycat has joined #openstack-swift | 01:39 | |
| *** j_king_ has joined #openstack-swift | 01:39 | |
| *** AbyssOne has joined #openstack-swift | 01:39 | |
| *** mordred has joined #openstack-swift | 01:39 | |
| *** anteaya has joined #openstack-swift | 01:39 | |
| *** dosaboy has joined #openstack-swift | 01:39 | |
| *** zackmdavis has joined #openstack-swift | 01:39 | |
| *** dfg has joined #openstack-swift | 01:39 | |
| *** charz has joined #openstack-swift | 01:39 | |
| *** mandarine has joined #openstack-swift | 01:39 | |
| *** swifterdarrell has joined #openstack-swift | 01:39 | |
| *** thurloat has joined #openstack-swift | 01:39 | |
| *** btorch has joined #openstack-swift | 01:39 | |
| *** ndk has joined #openstack-swift | 01:39 | |
| *** hurricanerix_ has joined #openstack-swift | 01:39 | |
| *** sendak.freenode.net sets mode: +v swifterdarrell | 01:39 | |
| *** abhirc has joined #openstack-swift | 01:40 | |
| *** abhirc has quit IRC | 01:40 | |
| *** gyee has quit IRC | 01:40 | |
| *** panbalag has quit IRC | 01:40 | |
| *** ekarlso has quit IRC | 01:40 | |
| *** mordred has quit IRC | 01:40 | |
| *** anteaya has quit IRC | 01:40 | |
| *** kei_yama has quit IRC | 01:40 | |
| *** greghaynes has quit IRC | 01:40 | |
| *** mrda has quit IRC | 01:40 | |
| *** cschwede has quit IRC | 01:40 | |
| *** jbonjean has quit IRC | 01:40 | |
| *** j_king_ has quit IRC | 01:40 | |
| *** AbyssOne has quit IRC | 01:40 | |
| *** dosaboy has quit IRC | 01:40 | |
| *** zackmdavis has quit IRC | 01:40 | |
| *** dfg has quit IRC | 01:40 | |
| *** charz has quit IRC | 01:40 | |
| *** mandarine has quit IRC | 01:40 | |
| *** swifterdarrell has quit IRC | 01:40 | |
| *** thurloat has quit IRC | 01:40 | |
| *** btorch has quit IRC | 01:40 | |
| *** ndk has quit IRC | 01:40 | |
| *** straycat has quit IRC | 01:40 | |
| *** hurricanerix_ has quit IRC | 01:40 | |
| *** km has quit IRC | 01:40 | |
| *** acorwin has quit IRC | 01:40 | |
| *** david-lyle_afk has quit IRC | 01:40 | |
| *** delattec has joined #openstack-swift | 01:40 | |
| *** gyee has joined #openstack-swift | 01:41 | |
| *** kei_yama has joined #openstack-swift | 01:41 | |
| *** panbalag has joined #openstack-swift | 01:41 | |
| *** greghaynes has joined #openstack-swift | 01:41 | |
| *** ekarlso has joined #openstack-swift | 01:41 | |
| *** cschwede has joined #openstack-swift | 01:41 | |
| *** mrda has joined #openstack-swift | 01:41 | |
| *** jbonjean has joined #openstack-swift | 01:41 | |
| *** straycat has joined #openstack-swift | 01:41 | |
| *** j_king_ has joined #openstack-swift | 01:41 | |
| *** AbyssOne has joined #openstack-swift | 01:41 | |
| *** mordred has joined #openstack-swift | 01:41 | |
| *** anteaya has joined #openstack-swift | 01:41 | |
| *** dosaboy has joined #openstack-swift | 01:41 | |
| *** zackmdavis has joined #openstack-swift | 01:41 | |
| *** dfg has joined #openstack-swift | 01:41 | |
| *** charz has joined #openstack-swift | 01:41 | |
| *** mandarine has joined #openstack-swift | 01:41 | |
| *** swifterdarrell has joined #openstack-swift | 01:41 | |
| *** thurloat has joined #openstack-swift | 01:41 | |
| *** btorch has joined #openstack-swift | 01:41 | |
| *** ndk has joined #openstack-swift | 01:41 | |
| *** hurricanerix_ has joined #openstack-swift | 01:41 | |
| *** sendak.freenode.net sets mode: +v swifterdarrell | 01:41 | |
| *** delattec has quit IRC | 01:42 | |
| *** vt102 has quit IRC | 01:49 | |
| *** gyee has quit IRC | 01:56 | |
| *** jamielennox is now known as jamielennox|away | 01:58 | |
| *** km has joined #openstack-swift | 01:59 | |
| *** acorwin has joined #openstack-swift | 01:59 | |
| *** david-lyle_afk has joined #openstack-swift | 01:59 | |
| *** IRTermite1 has joined #openstack-swift | 02:00 | |
| *** jamielennox|away is now known as jamielennox | 02:06 | |
| *** km has quit IRC | 02:10 | |
| *** acorwin has quit IRC | 02:10 | |
| *** david-lyle_afk has quit IRC | 02:10 | |
| *** km has joined #openstack-swift | 02:16 | |
| *** acorwin has joined #openstack-swift | 02:16 | |
| *** david-lyle_afk has joined #openstack-swift | 02:16 | |
| *** abhirc has joined #openstack-swift | 02:16 | |
| *** jamielennox is now known as jamielennox|away | 02:17 | |
| *** jamielennox|away is now known as jamielennox | 02:27 | |
| *** reed has quit IRC | 02:30 | |
| *** reed_ has joined #openstack-swift | 02:30 | |
| *** jamielennox is now known as jamielennox|away | 02:38 | |
| openstackgerrit | Merged openstack/swift: Clean up a couple deprecation warnings https://review.openstack.org/158806 | 02:40 |
| *** jamielennox|away is now known as jamielennox | 02:47 | |
| *** reed_ is now known as reed | 02:48 | |
| *** reed has quit IRC | 02:48 | |
| *** reed has joined #openstack-swift | 02:48 | |
| *** swat30 has quit IRC | 02:54 | |
| *** RackerShagz has joined #openstack-swift | 02:56 | |
| *** swat30 has joined #openstack-swift | 02:59 | |
| *** jrichli_ has joined #openstack-swift | 03:03 | |
| *** bkopilov has quit IRC | 03:09 | |
| *** bkopilov has joined #openstack-swift | 03:09 | |
| *** devlaps has joined #openstack-swift | 03:14 | |
| *** RackerShagz has quit IRC | 03:21 | |
| *** jamielennox is now known as jamielennox|away | 03:22 | |
| *** Manshoon has joined #openstack-swift | 03:22 | |
| *** Manshoon has quit IRC | 03:27 | |
| *** jamielennox|away is now known as jamielennox | 03:33 | |
| *** devlaps has quit IRC | 03:40 | |
| *** reed has quit IRC | 03:55 | |
| *** abhirc has quit IRC | 04:04 | |
| *** ho has joined #openstack-swift | 04:08 | |
| *** ho__ has quit IRC | 04:08 | |
| *** sluo_wfh has joined #openstack-swift | 04:08 | |
| *** sluo_wfh has quit IRC | 04:15 | |
| *** annegent_ has joined #openstack-swift | 04:20 | |
| *** sluo_wfh has joined #openstack-swift | 04:23 | |
| *** ho__ has joined #openstack-swift | 04:28 | |
| *** ho has quit IRC | 04:28 | |
| *** doxavore has quit IRC | 04:34 | |
| *** SkyRocknRoll has joined #openstack-swift | 04:44 | |
| *** ppai has joined #openstack-swift | 04:44 | |
| *** sluo_wfh is now known as sluo_laptop | 04:44 | |
| *** zaitcev has quit IRC | 05:09 | |
| *** jrichli_ has quit IRC | 05:20 | |
| *** Manshoon has joined #openstack-swift | 05:23 | |
| *** Manshoon has quit IRC | 05:28 | |
| openstackgerrit | Merged openstack/python-swiftclient: Fix crash with -l, -d /, and pseudo folders https://review.openstack.org/154648 | 05:39 |
| *** annegent_ has quit IRC | 05:47 | |
| *** annegent_ has joined #openstack-swift | 05:48 | |
| *** annegent_ has quit IRC | 05:53 | |
| *** zhill has joined #openstack-swift | 05:55 | |
| *** ppai has quit IRC | 05:57 | |
| *** ppai has joined #openstack-swift | 06:01 | |
| *** nshaikh has joined #openstack-swift | 06:26 | |
| *** silor has joined #openstack-swift | 06:27 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift: Imported Translations from Transifex https://review.openstack.org/159012 | 06:32 |
| *** IRTermite1 has quit IRC | 06:35 | |
| *** zhill has quit IRC | 07:06 | |
| *** mrda is now known as mrda-away | 07:32 | |
| *** ekarlso has quit IRC | 07:48 | |
| *** haomaiw__ has quit IRC | 08:04 | |
| *** rledisez has joined #openstack-swift | 08:12 | |
| *** acoles_away is now known as acoles | 08:29 | |
| *** nellysmitt has joined #openstack-swift | 08:40 | |
| *** ho__ has quit IRC | 08:41 | |
| *** joeljwright has joined #openstack-swift | 08:45 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-swiftclient: Updated from global requirements https://review.openstack.org/89250 | 08:48 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift: Updated from global requirements https://review.openstack.org/88736 | 08:48 |
| *** ekarlso has joined #openstack-swift | 08:52 | |
| *** jistr has joined #openstack-swift | 08:53 | |
| *** kei_yama has quit IRC | 08:58 | |
| *** km has quit IRC | 09:01 | |
| *** ppai has quit IRC | 09:20 | |
| *** ppai has joined #openstack-swift | 09:32 | |
| *** geaaru has joined #openstack-swift | 09:36 | |
| *** jordanP has joined #openstack-swift | 09:44 | |
| *** dmorita_ has quit IRC | 09:56 | |
| *** joeljwright1 has joined #openstack-swift | 10:20 | |
| *** joeljwright has quit IRC | 10:20 | |
| *** erlon has joined #openstack-swift | 10:45 | |
| *** aix has joined #openstack-swift | 11:02 | |
| openstackgerrit | Christian Schwede proposed openstack/python-swiftclient: Release connection after consuming the content https://review.openstack.org/149043 | 11:03 |
| openstackgerrit | Christian Schwede proposed openstack/python-swiftclient: Add connection release test https://review.openstack.org/159076 | 11:03 |
| *** Manshoon has joined #openstack-swift | 11:26 | |
| *** Manshoon has quit IRC | 11:30 | |
| *** EmilienM is now known as EmilienM|afk | 11:39 | |
| *** aix has quit IRC | 12:11 | |
| *** aix has joined #openstack-swift | 12:43 | |
| *** ppai has quit IRC | 12:44 | |
| *** sandywalsh has quit IRC | 12:49 | |
| *** annegent_ has joined #openstack-swift | 12:58 | |
| *** EmilienM|afk is now known as EmilienM | 13:03 | |
| *** annegent_ has quit IRC | 13:03 | |
| *** abhirc has joined #openstack-swift | 13:03 | |
| openstackgerrit | Merged openstack/swift: Clean up a couple of deprecated uses of e.message https://review.openstack.org/158937 | 13:09 |
| openstackgerrit | Merged openstack/python-swiftclient: Fix crash when stat'ing objects with non-ascii names https://review.openstack.org/147846 | 13:10 |
| *** dencaval has joined #openstack-swift | 13:27 | |
| *** sandywalsh has joined #openstack-swift | 13:28 | |
| *** nellysmitt has quit IRC | 13:48 | |
| *** nexusz99 has joined #openstack-swift | 13:48 | |
| *** nexusz99 has quit IRC | 13:48 | |
| *** nellysmitt has joined #openstack-swift | 13:49 | |
| *** khivin has quit IRC | 13:50 | |
| *** nellysmitt has quit IRC | 13:53 | |
| openstackgerrit | Merged openstack/python-swiftclient: Release connection after consuming the content https://review.openstack.org/149043 | 13:54 |
| openstackgerrit | Donagh McCabe proposed openstack/swift: Support HTTP_X_SERVICE_IDENTITY_STATUS in keystoneauth https://review.openstack.org/156634 | 13:56 |
| *** bill_az has joined #openstack-swift | 13:57 | |
| *** abhirc_ has joined #openstack-swift | 14:09 | |
| *** khivin has joined #openstack-swift | 14:10 | |
| *** abhirc has quit IRC | 14:11 | |
| *** abhirc_ has quit IRC | 14:11 | |
| *** annegent_ has joined #openstack-swift | 14:17 | |
| *** mahatic has joined #openstack-swift | 14:19 | |
| *** NM has joined #openstack-swift | 14:24 | |
| *** nshaikh has quit IRC | 14:24 | |
| *** annegent_ has quit IRC | 14:42 | |
| *** zul has quit IRC | 14:43 | |
| *** zul has joined #openstack-swift | 14:44 | |
| *** david-lyle_afk is now known as david-lyle | 14:48 | |
| *** vt102 has joined #openstack-swift | 14:49 | |
| *** zaitcev has joined #openstack-swift | 14:52 | |
| *** ChanServ sets mode: +v zaitcev | 14:52 | |
| *** Manshoon has joined #openstack-swift | 14:54 | |
| *** openstackgerrit has quit IRC | 15:08 | |
| *** openstackgerrit has joined #openstack-swift | 15:08 | |
| *** zul has quit IRC | 15:09 | |
| *** zul has joined #openstack-swift | 15:09 | |
| *** mordred has quit IRC | 15:22 | |
| *** mordred has joined #openstack-swift | 15:22 | |
| *** annegent_ has joined #openstack-swift | 15:28 | |
| *** dmsimard_away is now known as dmsimard | 15:33 | |
| *** SkyRocknRoll has quit IRC | 15:38 | |
| *** Manshoon has quit IRC | 15:40 | |
| *** jrichli has joined #openstack-swift | 15:41 | |
| *** Manshoon has joined #openstack-swift | 15:41 | |
| *** abhirc has joined #openstack-swift | 15:48 | |
| *** nellysmitt has joined #openstack-swift | 15:49 | |
| *** nellysmitt has quit IRC | 15:54 | |
| *** MasterPiece has joined #openstack-swift | 15:56 | |
| tdasilva | acoles: while talking about the current fsync issue, I got pointed to this article: https://lwn.net/Articles/559147/ seemed like an interesting idea to get rid of having to rename all together | 16:03 |
| *** annegent_ has quit IRC | 16:04 | |
| notmyname | tdasilva: that' interesting | 16:12 |
| *** annegent_ has joined #openstack-swift | 16:12 | |
| *** annegent_ has quit IRC | 16:12 | |
| tdasilva | notmyname: yeah, different way of looking at the problem...could help us out... | 16:12 |
| notmyname | tdasilva: is that what you wanted to bring up in the meeting today? | 16:13 |
| tdasilva | notmyname: problem is I'm not sure if linkat syscall is available in python....found this: https://github.com/eisenlaub/python-linkat | 16:13 |
| notmyname | tdasilva: is this only with 3.11+ kernels? | 16:13 |
| *** annegent_ has joined #openstack-swift | 16:14 | |
| tdasilva | notmyname: still trying to figure that out | 16:15 |
| tdasilva | notmyname: I think so...I think the linkat syscall has been there for a while, but the O_TMPFILE flag was added in 3.11 | 16:16 |
| notmyname | which looks like the interesting part :-) | 16:16 |
| notmyname | thanks for looking | 16:16 |
| tdasilva | yeah.. | 16:16 |
| notmyname | tdasilva: so when will you be able to get the 3.11 kernel into RHEL? ;-) | 16:17 |
| tdasilva | touche | 16:18 |
| portante | notmyname: 3.11? That is old, what about 4.0? | 16:18 |
| portante | ;) | 16:18 |
| notmyname | heh | 16:19 |
| *** eranrom has joined #openstack-swift | 16:20 | |
| notmyname | tdasilva: I'd be fine with support in swift for it with a graceful fallback. it would be interesting to see if it's just better in a security way or if it's more performant too. | 16:20 |
| tdasilva | notmyname: just going back to your previous question, I had actually added the topic to the meeting schedule prior to running into this, I guess I just wanted to follow up on where we were wrt that patch | 16:21 |
| notmyname | :-) | 16:21 |
| notmyname | ok | 16:21 |
| tdasilva | notmyname: agreed, will keep looking | 16:21 |
| openstackgerrit | Christian Schwede proposed openstack/python-swiftclient: Add connection release test https://review.openstack.org/159076 | 16:23 |
| openstackgerrit | Christian Schwede proposed openstack/swift: Set connection timeout in container sync https://review.openstack.org/156943 | 16:31 |
| *** cpallares has joined #openstack-swift | 16:32 | |
| tdasilva | anybody else seeing probe tests fail on master? | 16:32 |
| notmyname | tdasilva: just started running them. working so far for me | 16:35 |
| peluse | tdasilva, I'm at a good point to take a break, will run real quick as well | 16:36 |
| *** reed has joined #openstack-swift | 16:38 | |
| eranrom | Greetings. I have a question regarding error_limit / error_occured and ConnectionTimeout. | 16:38 |
| eranrom | Looking at both PUT and GET flows, it appears that a timeout during connection is not accounted as an error. This e.g. manifests in proxy/controller/base.py->_get_source_and_node, where a timeout during a connection attempt to a storage node simply continues to the next node. | 16:38 |
| eranrom | I assume that this is intetional, but wanted to make sure and validate why this is the case. Afterall if a node is down for a long time, why keep trying it... | 16:40 |
| peluse | eranrom, someone with more history should chime in but I think the idea is that if the node is down for a long, long time, someone will make a ring adjustment to remove it | 16:42 |
| peluse | so keep trying til that happens... | 16:43 |
| peluse | and maybe a switch or something will be reboted in the meantime :) | 16:43 |
| *** openstackstatus has joined #openstack-swift | 16:43 | |
| *** ChanServ sets mode: +v openstackstatus | 16:43 | |
| peluse | tdasilva, so I haven't run probe on master in a while but right out of the shoot (w/multiple repl policies defined) here's what I get: | 16:44 |
| peluse | FAILED (SKIP=1, errors=3, failures=1) | 16:44 |
| peluse | how bout you? | 16:44 |
| notmyname | ya, I got 3 errors too | 16:44 |
| peluse | well, there's something to be said for consistency at least | 16:45 |
| peluse | tdasilva, did you already narrow down the suspect patch? (thought I saw something yesterday) | 16:46 |
| tdasilva | i get only one :/ | 16:46 |
| peluse | tdasilva, do you have just one policy? | 16:46 |
| tdasilva | FAIL: test_merge_storage_policy_index (test.probe.test_container_merge_policy_index.TestContainerMergePolicyIndex | 16:46 |
| tdasilva | peluse: no, the default 2 | 16:46 |
| notmyname | eranrom: timeouts incremet the error count and then a node will be error limited | 16:47 |
| eranrom | pleuse: right, but given the fact that being error limited holds for a short amount of time (I think 10 secs is the default), then accounting for the errors would limit the rate at which the retries are done - perhaps this would be considered as a minor optmization. Still, I wonder if I am missing something here... | 16:47 |
| tdasilva | peluse: sorry, i think i gave you the wrong info: I looked at the saio swift.conf | 16:47 |
| peluse | yeah, that one fails for me plus test_object_delete_is_replicated() and test_main() | 16:47 |
| notmyname | eranrom: right. it's error limited for a time and then that expires. in which case it will be error limited again | 16:51 |
| notmyname | be back in 10 minutes. | 16:51 |
| eranrom | I have just noticed that there is a bug titled: unreachable nodes are not error-limited. so thanks and sorry for the interruption. | 16:51 |
| *** rdaly2 has joined #openstack-swift | 16:52 | |
| *** doxavore has joined #openstack-swift | 16:52 | |
| cschwede | acoles: do you mind if i quickly remove the unneeded return values (see Joels comments) on https://review.openstack.org/#/c/153177/3 ? | 16:53 |
| cschwede | i removed them locally and tests are still working, so i could easily submit this as a new patchset | 16:54 |
| acoles | cschwede: no, please push over a new patchset, i didn't have time to do that today | 16:55 |
| acoles | cschwede: thanks! | 16:55 |
| openstackgerrit | Christian Schwede proposed openstack/python-swiftclient: Add tests for uploads deleting DLO segments https://review.openstack.org/153177 | 16:55 |
| cschwede | acoles: you’re welcome - done :) | 16:55 |
| acoles | cschwede: great | 16:56 |
| *** annegent_ has quit IRC | 16:57 | |
| notmyname | eranrom: what bug? got a link? | 16:58 |
| eranrom | notmyname: https://bugs.launchpad.net/bugs/1094242 | 16:58 |
| openstack | Launchpad bug 1094242 in OpenStack Object Storage (swift) "swift-proxy: unreachable nodes are not error-limited" [Undecided,Opinion] | 16:58 |
| notmyname | peluse: cschwede: will you be able to look at https://review.openstack.org/#/c/158821/ today? | 16:59 |
| cschwede | notmyname: i’ll do it tomorrow morning, I’m leaving for dinner right now (but will be back for the meeting later). would like to review it with a clear head | 17:01 |
| notmyname | peluse: looks like tsg has 2 patches outstanding. one marked WIP and one with a merge conflict | 17:02 |
| notmyname | cschwede: thanks :-) | 17:02 |
| notmyname | cschwede: have a good evening | 17:02 |
| cschwede | notmyname: thanks, see you later | 17:02 |
| eranrom | notmyname,peluse: well, seems like this is addressed in v2.2.2. I confess I was looking at an earlier version of the code... | 17:03 |
| peluse | eranrom, ahh cool | 17:03 |
| eranrom | ok thanks very much gotta drop | 17:04 |
| peluse | notmyname, yes, the multi-phase put should be updated anytime now (heard from him late last night). | 17:04 |
| peluse | notmyname, and the DB update one I think will be abandoned in favor of another from acoles | 17:05 |
| *** eranrom has left #openstack-swift | 17:05 | |
| notmyname | ok | 17:05 |
| notmyname | I was looking through the priority reviews page | 17:05 |
| peluse | notmyname, yes to the request above. I already took a look at it and want to poke a bit more but I like it | 17:06 |
| *** atan8 has joined #openstack-swift | 17:06 | |
| notmyname | peluse: I'm moving the trello card for GETs to "done". the patches mentioned have landed. anything else, I think, will need to be a different card. do you agree? | 17:08 |
| *** atan8 has quit IRC | 17:09 | |
| notmyname | mahatic: good morning. looks like cschwede left some good feedback on your patch. anything else going on? | 17:12 |
| mahatic | notmyname, yup :) I'm on it | 17:12 |
| notmyname | tdasilva: are you tracking down the probetests failure? | 17:12 |
| notmyname | mahatic: great :-) | 17:12 |
| mahatic | notmyname, good morning too! and should anything else be going on? | 17:13 |
| notmyname | mahatic: I was just looking at the opw schedule. looks like it ends march 9. so I think you're perfectly timed | 17:13 |
| peluse | notmyname, sounds good. I think the rest of GET (sifting through available headers, etc) is already on another card, let me check real quick | 17:13 |
| notmyname | peluse: thanks | 17:13 |
| notmyname | mahatic: I'd expect your patch to land this week | 17:14 |
| mahatic | notmyname, yup. what will be next after the patch? | 17:14 |
| notmyname | mahatic: which kinda raises the bigger question of "what next?". | 17:14 |
| peluse | notmyname, yeah, its already in TODO but will be hard for anyone to work on until I get at least some of the reconstructor stuff landed because of changes in diskfile that are needed... | 17:15 |
| peluse | notmyname, I need to work on breaking some of that out I guess | 17:15 |
| *** Manshoon has quit IRC | 17:15 | |
| notmyname | mahatic: I think that's up to you, in part because its dependent on your willingness and ability to keep working on swift | 17:15 |
| mahatic | notmyname, yeah :) I can pitch in some other on going work, like you mentioned in the schedule :) | 17:15 |
| notmyname | peluse: ok. will you be around for the meeting to go over the status of things? | 17:15 |
| notmyname | peluse: might be worth discussing breaking up the reconstructor work or landing and iterating | 17:16 |
| notmyname | mahatic: I'd love to see you do that! | 17:16 |
| mahatic | notmyname, sure! I intend to keep working on. | 17:16 |
| notmyname | yay | 17:17 |
| *** Manshoon has joined #openstack-swift | 17:17 | |
| peluse | notmyname, yup | 17:17 |
| *** annegent_ has joined #openstack-swift | 17:18 | |
| notmyname | kk | 17:18 |
| *** Manshoon has quit IRC | 17:18 | |
| *** Manshoon has joined #openstack-swift | 17:18 | |
| *** annegent_ has quit IRC | 17:18 | |
| *** annegent_ has joined #openstack-swift | 17:21 | |
| *** tsg_ has joined #openstack-swift | 17:21 | |
| *** jamielennox is now known as jamielennox|away | 17:22 | |
| tdasilva | notmyname: just got back, yeah I'll continue looking | 17:22 |
| *** jistr has quit IRC | 17:22 | |
| notmyname | tdasilva: ok thanks! | 17:23 |
| *** nellysmitt has joined #openstack-swift | 17:24 | |
| tdasilva | notmyname: mmm..just fyi, probetests on a 'brand new' instance of vagrant-swift-all-in-one runs fine :\ | 17:24 |
| openstackgerrit | Elvis Teixeira proposed openstack/swift: Fix possibility of data leakage for commom X-Versions-Location https://review.openstack.org/159191 | 17:25 |
| *** Manshoon_ has joined #openstack-swift | 17:25 | |
| *** Manshoon_ has quit IRC | 17:25 | |
| *** Manshoon_ has joined #openstack-swift | 17:26 | |
| notmyname | tdasilva: interesting | 17:26 |
| *** Manshoon has quit IRC | 17:28 | |
| *** jamielennox|away is now known as jamielennox | 17:30 | |
| *** annegent_ has quit IRC | 17:34 | |
| *** annegent_ has joined #openstack-swift | 17:35 | |
| *** rledisez has quit IRC | 17:37 | |
| peluse | tdasilva, what?? | 17:38 |
| *** gyee has joined #openstack-swift | 17:38 | |
| tdasilva | peluse: strange | 17:39 |
| notmyname | ok, commute time. be back online later | 17:40 |
| *** annegent_ has quit IRC | 17:40 | |
| tdasilva | notmyname: careful on that bike :P | 17:40 |
| jrichli | +1 | 17:41 |
| notmyname | :-) | 17:41 |
| *** Manshoon_ has quit IRC | 17:41 | |
| *** annegent_ has joined #openstack-swift | 17:43 | |
| doxavore | I'm getting a lot of 499's - basically every request for something that is read through a Ruby app... is it safe to assume as of 2.2.2 that it's an issue with the HTTP client somehow not informing Swift of an expected disconnect? | 17:44 |
| openstackgerrit | Alistair Coles proposed openstack/swift: Enable in-process functional test policy to be configured https://review.openstack.org/159205 | 17:45 |
| acoles | peluse: ^^ maybe helpful? | 17:46 |
| *** jordanP has quit IRC | 17:46 | |
| *** EmilienM is now known as EmilienM|afk | 17:49 | |
| openstackgerrit | Alistair Coles proposed openstack/swift: Enable override of container update headers https://review.openstack.org/158821 | 17:51 |
| *** gvernik has joined #openstack-swift | 17:52 | |
| acoles | peluse: ^^ i cleared out that spurious comment, no other changes | 17:52 |
| *** aerwin has joined #openstack-swift | 18:00 | |
| *** annegent_ has quit IRC | 18:00 | |
| openstackgerrit | Stuart McLaren proposed openstack/python-swiftclient: Retry download after streaming has started https://review.openstack.org/159208 | 18:01 |
| jrichli | my master probetests failed with 3 errors | 18:01 |
| *** jamielennox is now known as jamielennox|away | 18:02 | |
| jrichli | but the feature/crypto that got a merge from main a couple days ago has no errors from probe | 18:03 |
| jrichli | going to lunch now, tho | 18:04 |
| *** tsg_ has quit IRC | 18:09 | |
| *** Anticimex has quit IRC | 18:12 | |
| clayg | doxavore: are you doing and video progressive download stuff? client disconnects aren't nessecarily an indication of bad behavior - sometimes client's request a lot and only download what they need? | 18:12 |
| clayg | doxavore: either way I'm not aware of anyhting that changed wrt how client disconnects are logged/handled recently - what version did you upgrade *from*? | 18:13 |
| doxavore | oh, this is a new install. was just stating which version we're on :> | 18:13 |
| doxavore | this is reading the entire file | 18:14 |
| doxavore | the ruby code is able to read it down into a local file, and md5's match what are in Swift, so it's getting the whole thing... | 18:14 |
| peluse | acoles, cool | 18:15 |
| clayg | what do you have in front of swift - ssl termination - web proxy - anything like that? can you duplicate the client disconnect response with another client (e.g. curl) | 18:15 |
| *** jamielennox|away is now known as jamielennox | 18:15 | |
| acoles | tdasilva: thanks for that link, will take a look | 18:15 |
| doxavore | clayg: nothing between the ruby code and the Swift proxy servers. curl does not log it as a disconnect, which is why i'm leaning toward ruby code, but it's just using stdlib stuff which I'd expect should work correctly out of the box... | 18:17 |
| clayg | i really can't imagine how it can get the last chunk and still cause the proxy logging middleware to raise generator exit - very interesting | 18:18 |
| doxavore | it does include a Connection: close header, but that's all the more reason I'd expect it to _not_ get an unexpected disconnect if anything... | 18:18 |
| *** Anticimex has joined #openstack-swift | 18:18 | |
| clayg | hrmmm idk - if you can get a small ruby script that will reproduce I think it'd be interesting to look at | 18:19 |
| doxavore | if the client reads exactly how much data is expected (eg Content-Length) and then closes, would that still raise a generator exit? | 18:19 |
| doxavore | i'll see if i can pull together a quick script to see it as well | 18:20 |
| clayg | can you tell what version of eventlet you're running? | 18:20 |
| clayg | re reading exactly content-length - that's exactly what is expected - 'Connection: close' is definately not enough to exibit the issue | 18:21 |
| doxavore | eventlet 0.13.0-1ubuntu2, on ubuntu 14.04.2 | 18:22 |
| clayg | doxavore: does the size of the object matter - can you replicate the issue just as well with small (8 byte - 10K) objects? | 18:22 |
| doxavore | yes, it appears to be every size | 18:23 |
| clayg | doxavore: well good that it's easy for you reproduce - getting someone else to reprodcue it is probably the next step :D | 18:23 |
| notmyname | hello. I survived biking once more :-) | 18:23 |
| clayg | notmyname LIVES! | 18:23 |
| doxavore | clayg: ok, i'll pull together a script shortly that should be able to showcase it | 18:24 |
| doxavore | thanks | 18:24 |
| notmyname | clayg: well, my legs don't quite agree with that. turns out san francsico has hills! | 18:24 |
| clayg | notmyname: have you been followig the revisting 6 mo release cycle thread - i tried to read it - but they just keep talking | 18:24 |
| acoles | tdasilva: notmyname : fwiw i see probetest failures when rolled back to commit 6a6f7d5 | 18:25 |
| notmyname | clayg: ya, it's one of the ones I'm currently not ignoring | 18:25 |
| *** aix has quit IRC | 18:25 | |
| acoles | notmyname: starting commute, may be a few mins late for meeting | 18:26 |
| *** acoles is now known as acoles_away | 18:26 | |
| notmyname | clayg: the summary from what I can tell is "big massive integration at the end of six months is hard, maybe we should change something". and the opinions are basically "do it faster" and "some poeple want it slower" and "six moths is good enough" | 18:26 |
| clayg | acoles: i ran probetests on some branch just yesterday and they all passed, not sure exactly how much of master I had in there at the time | 18:27 |
| tdasilva | acoles, notmyname : torgomatic posted this message yesterday: <torgomatic> looks like f578a35100f5dcd0046c79e810441633d28f55ff broke some probe tests | 18:27 |
| tdasilva | I checked out the commit before that, and probe tests pass | 18:27 |
| clayg | notmyname: that's pretty good acctually - thanks! | 18:27 |
| notmyname | clayg: I suspect it will result in more of the swift-style releases, actually. (or maybe I'm just hoping for that.) people are feeling pain with rapidly changing APIs and exponential complexity of integration at an arbitrary time | 18:28 |
| notmyname | tdasilva: look at torgomatic being ahead of us | 18:28 |
| clayg | oh well that looks like kota and i's fault - but that was acctually the starting point from where I was running probetests - so they work with my setup! | 18:28 |
| clayg | which test is broke? | 18:28 |
| notmyname | I'l have to rerun them. give me a few minutes | 18:29 |
| tdasilva | clayg: I get inconsisten results, one run 1 tests fails, another run 3 tests fails, here's one: FAIL: test_merge_storage_policy_index (test.probe.test_container_merge_policy_index.TestContainerMergePolicyIndex | 18:29 |
| *** rdaly2 has quit IRC | 18:31 | |
| *** rdaly2 has joined #openstack-swift | 18:33 | |
| *** MasterPiece has quit IRC | 18:33 | |
| *** MasterPiece has joined #openstack-swift | 18:35 | |
| notmyname | File "/home/swift/swift/test/probe/test_object_metadata_replication.py", line 150, in test_object_delete_is_replicated | 18:38 |
| clayg | that's crazy? i didn't think that test barely hits object replication. | 18:38 |
| notmyname | File "/home/swift/swift/test/probe/test_object_handoff.py", line 135, in test_main ==> AttributeError: 'NoneType' object has no attribute 'http_status' | 18:38 |
| clayg | oh - i'm sure you guys are all running rsync, so that's probably the difference - crap | 18:38 |
| notmyname | and again that one | 18:39 |
| peluse | same errors here BTW | 18:39 |
| notmyname | clayg: ya, I switched back to rsync a couple of weeks ago because of probe tests | 18:39 |
| *** aerwin has quit IRC | 18:40 | |
| clayg | k, i just stashed what I was working on am flipping back to rsync | 18:41 |
| notmyname | hmm...the AttributeError might be from swiftlcient. i havne't updated that in a while | 18:41 |
| clayg | yeah that's weird | 18:41 |
| notmyname | hmm...there's some other squirlly code in there | 18:45 |
| notmyname | but not the line that failed | 18:45 |
| clayg | do any of yous guys have more than one region in your rings? | 18:46 |
| clayg | test_object_handoff just passed for me with rsync - you say it's sort of intermittent? | 18:46 |
| notmyname | yes, I do | 18:46 |
| *** Manshoon has joined #openstack-swift | 18:47 | |
| tdasilva | clayg: yeah, the weird part is that I setup a vagrant_all_in_one and it worked ok | 18:47 |
| clayg | i don't normally do, but I do for the moment since i've been working on kota's stuff | 18:47 |
| clayg | gd | 18:47 |
| notmyname | tdasilva: once? | 18:47 |
| notmyname | tdasilva: you're the only one I've heard say "intermittent". always fails for me | 18:48 |
| clayg | well test_object_handoff passed twice, so i'm running the full suite with rsync now, i have four devices in two regions in my default storage policy - well see | 18:48 |
| tdasilva | notmyname: i've run probetests on this new instance of vagrant_saio and it passed twice | 18:48 |
| tdasilva | notmyname: on my other vm, i've run multiple times on master and sometimes it fails with just the error I posted above, and sometimes with the same 3 errors you are seeing | 18:49 |
| clayg | the *Attribute* Error? | 18:49 |
| tdasilva | clayg: yes | 18:50 |
| clayg | tdasilva: ok, and how many regions do you have in the object ring on the one that always fails | 18:51 |
| *** jkremer has joined #openstack-swift | 18:51 | |
| *** ElvisTeixeira has joined #openstack-swift | 18:51 | |
| tdasilva | clayg: http://ur1.ca/jsuki | 18:52 |
| tdasilva | clayg: one sec, checking on the regions | 18:52 |
| notmyname | rerunning with updated client. if that fails, I'll reset to one region | 18:52 |
| tdasilva | clayg: one region | 18:53 |
| *** nellysmitt has quit IRC | 18:53 | |
| clayg | so looking at test_object_handoff specifically it seems like with ssync and more than one region there should be random chance the "assert handoff 404's" would fail given efficent replication | 18:54 |
| clayg | notmyname: so but you're seeing those same "assert handoff no longer there" errors with rsync and multi-region? cause that's the same setup i have now... | 18:56 |
| *** kota_ has joined #openstack-swift | 18:58 | |
| notmyname | clayg: I'll check after the meeting | 18:59 |
| notmyname | meeting time in #openstack-meeting | 18:59 |
| *** annegent_ has joined #openstack-swift | 19:01 | |
| *** acoles_away is now known as acoles | 19:04 | |
| *** annegent_ has quit IRC | 19:06 | |
| *** Manshoon has quit IRC | 19:11 | |
| *** Manshoon has joined #openstack-swift | 19:12 | |
| *** silor1 has joined #openstack-swift | 19:18 | |
| *** annegent_ has joined #openstack-swift | 19:18 | |
| *** Manshoon_ has joined #openstack-swift | 19:18 | |
| *** Manshoon_ has quit IRC | 19:19 | |
| *** [1]gvernik has joined #openstack-swift | 19:19 | |
| *** Manshoon_ has joined #openstack-swift | 19:19 | |
| *** silor has quit IRC | 19:20 | |
| openstackgerrit | Alistair Coles proposed openstack/swift: Enable in-process functional test policy to be configured https://review.openstack.org/159205 | 19:21 |
| *** gvernik has quit IRC | 19:22 | |
| *** [1]gvernik is now known as gvernik | 19:22 | |
| *** tsg_ has joined #openstack-swift | 19:22 | |
| *** Manshoon has quit IRC | 19:22 | |
| *** rdaly2 has quit IRC | 19:23 | |
| *** rdaly2_ has joined #openstack-swift | 19:23 | |
| *** reed has quit IRC | 19:25 | |
| *** reed has joined #openstack-swift | 19:28 | |
| *** Manshoon has joined #openstack-swift | 19:31 | |
| clayg | probetests still aren't failing for me | 19:34 |
| *** Manshoon has quit IRC | 19:34 | |
| *** gvernik has quit IRC | 19:34 | |
| clayg | acoles: wow - thanks for https://review.openstack.org/159205 - that's huge? | 19:34 |
| zaitcev | but... that's good, right? | 19:34 |
| *** Manshoon_ has quit IRC | 19:34 | |
| *** Manshoon has joined #openstack-swift | 19:35 | |
| acoles | cschwede: clayg: huge == bad?? | 19:35 |
| acoles | cschwede: sorry, not you :) | 19:36 |
| *** jrichli has quit IRC | 19:36 | |
| zaitcev | Nice, portante's legacy is carried on, I love that in-process thing. | 19:36 |
| tdasilva | acoles, peluse: http://ur1.ca/jsusu | 19:36 |
| clayg | acoles: oh I think i misread the feature, it still tests the default policy | 19:36 |
| *** jrichli has joined #openstack-swift | 19:36 | |
| acoles | clayg: yeah, so its a start, as in you can point it at different swift.conf with different defaults | 19:37 |
| clayg | yeah that's interesting | 19:37 |
| acoles | clayg: but it simple to pass in a policy in an env var | 19:37 |
| acoles | clayg: i just pushed what i had so far | 19:37 |
| acoles | clayg: another thought i had was to somehow iterate through all policies in a swift.conf, running test suite against each one | 19:38 |
| notmyname | ouch (via twitter): "Found a bug in #pecan. Looks like it's freaking #OpenStack subproject. Not worth it." | 19:38 |
| acoles | tdasilva: thx | 19:39 |
| mahatic | Would someone know how to access the first tuple in this list? http://paste.openstack.org/show/182076/ | 19:40 |
| zaitcev | notmyname: The question is if you want to chase mentally weak people like that by unbundling Swift from OpenStack. I don't have a sense if they command a good chunk of paying userbase. | 19:40 |
| * mattoliverau is going to find breakfast | 19:40 | |
| acoles | mattoliverau: enjoy! | 19:41 |
| tdasilva | mattoliverau: blue bottle :P | 19:41 |
| mattoliverau | tdasilva: lol | 19:41 |
| tdasilva | short walk | 19:41 |
| clayg | lol @ "not worth it" | 19:42 |
| *** annegent_ has quit IRC | 19:42 | |
| notmyname | mahatic: https://gist.github.com/anonymous/a74c06041cd73e1ff6de | 19:42 |
| notmyname | not sure what's you're looking for? | 19:42 |
| *** annegent_ has joined #openstack-swift | 19:44 | |
| mahatic | notmyname, thanks. [hosts[0]] worked. I was passing it to a function. | 19:45 |
| clayg | zaitcev: if it was "found a bug in #swift" we'd chase them down - doubt it would mean dissaccoiating from openstack - but even if you hate commiting to openstack you could at least open a bug | 19:45 |
| cschwede | peluse: acoles: https://review.openstack.org/#/c/158821/ LGTM, I’ll add my +2/A tomorrow morning - too tired now, don’t want to miss anything there | 19:45 |
| acoles | cschwede: thanks, yes, do it with a fresh mind | 19:45 |
| notmyname | clayg: yup. (and I do chase those people down) | 19:45 |
| zaitcev | clayg: Sorry, but opening bugs is a burden. | 19:45 |
| notmyname | fortunately, "found a bug in swift" is a pretty rare thing to see on twitter :-) | 19:45 |
| notmyname | it's mostly, "openstack is hard to install" | 19:46 |
| zaitcev | it is | 19:46 |
| zaitcev | and it is | 19:46 |
| notmyname | :-) | 19:46 |
| clayg | wait - i want to go back to "opening bugs is a burden" | 19:46 |
| zaitcev | The worst part for me is always Neutron and OVS. | 19:46 |
| zaitcev | clayg: Is that a surprise for you? | 19:47 |
| cschwede | zaitcev: sounds familar to me… | 19:47 |
| notmyname | dfg: hurricanerix_: glange: I just added all 3 of you to https://review.openstack.org/#/c/126923/. We talked about it at length at the hackathon, and we'd love to have your take on it | 19:47 |
| notmyname | tdasilva: ^ | 19:47 |
| tdasilva | notmyname: thx | 19:48 |
| zaitcev | you only need to look at shit like this to realize that life is too short to file bugs: https://github.com/kennethreitz/requests/issues/2214#issuecomment-72941896 | 19:50 |
| clayg | i'm still not getting any probetests failures :\ | 19:50 |
| notmyname | zaitcev: ya, I've seen that kind of thing before in some other projects. it's not healthy | 19:51 |
| *** aerwin has joined #openstack-swift | 19:52 | |
| zaitcev | I even looked at HTTP/2 because of the attitudes of the crew that owns Requests. In theory, HTTP/2 includes a sane 100-Continue, guaranteed (thanks to their "frames"). | 19:52 |
| zaitcev | So you don't need to argue with assholes about what "is HTTP" and what "is in RFC". | 19:53 |
| zaitcev | Sadly, HTTP/2 requires SSL. Which actually may be a good thing, dunno. | 19:53 |
| clayg | what does you disagreeing with the design decision of the project have to do with the length of life or the difficulty of reporting the issue/bug/feature to the upstream project? | 19:56 |
| zaitcev | Because I read my e-mail and every time bug updates, I need to read that. | 19:56 |
| zaitcev | Also, to file a good bug report one must provide a good data. Maybe a reproducer, too. | 19:57 |
| zaitcev | That's 3 hours of work at best, maybe a day. | 19:58 |
| *** silor1 has quit IRC | 19:58 | |
| *** kota_ has quit IRC | 19:58 | |
| openstackgerrit | Mahati proposed openstack/swift: Ring checker in swift-recon https://review.openstack.org/153617 | 19:58 |
| zaitcev | People who argue that filing bug reports is painless seem to think that it takes a minute to file one. | 19:58 |
| *** dencaval has quit IRC | 20:00 | |
| clayg | it's not so much as I think it's painless as I think it's the *least* you can do for an open source project you're using - and you don't always have to file a good bug report - sometimes you can just capture when you first encounted the issue and others can add more info | 20:01 |
| *** annegent_ has quit IRC | 20:01 | |
| *** mahatic has quit IRC | 20:03 | |
| *** geaaru has quit IRC | 20:03 | |
| clayg | i've *tried* to follow H302 a couple times when starting personal projects or new modules, but utlimately always find reason I don't want to do it anymore (most of the it's stdlib, something like collections.defaultdict or datetime.datetime) | 20:03 |
| clayg | i don't get it - still not seeing the probetest failures :'( | 20:05 |
| *** joeljwright has joined #openstack-swift | 20:09 | |
| *** joeljwright1 has quit IRC | 20:10 | |
| clayg | ... wish I knew what I needed to do to get probetests failing | 20:12 |
| *** annegent_ has joined #openstack-swift | 20:13 | |
| *** Manshoon has quit IRC | 20:13 | |
| clayg | i'm going back to ssync, with multiple regions - it seems like that setup should have a good random chance of failing | 20:13 |
| tdasilva | clayg: mm..how can i help? | 20:14 |
| clayg | idk, you said it worked ok for you on a vagrant-swift-all-in-one - maybe something about the way that thing builds it's saio is sheilding me from the failure? | 20:15 |
| clayg | oh hey! i just got a failure with ssync! unexpeted 200 in test_object_delete_is_replicated | 20:16 |
| clayg | that's a thing right? | 20:16 |
| clayg | tdasilva: can you run `swift-config /etc/swift/object-server/1.conf* | grep -i sync` on the machine you've seen the failure and confirm you're running rsync (not ssync?) | 20:17 |
| *** annegent_ has quit IRC | 20:21 | |
| *** annegent_ has joined #openstack-swift | 20:21 | |
| *** Manshoon has joined #openstack-swift | 20:21 | |
| clayg | zaitcev: look it's no so bad -> https://bugs.launchpad.net/swift/+bug/1425679 | 20:22 |
| openstack | Launchpad bug 1425679 in OpenStack Object Storage (swift) "swift-object-info should try harder on tombstones" [Undecided,New] | 20:22 |
| *** Manshoon has quit IRC | 20:24 | |
| tdasilva | clayg: my understanding is that rsync is default, right? here's 1.conf: http://ur1.ca/jsv9f | 20:24 |
| *** Manshoon has joined #openstack-swift | 20:24 | |
| clayg | tdasilva: yeah rsync is the defalt | 20:25 |
| tdasilva | clayg: i haven't really changed my saio setup much | 20:25 |
| tdasilva | only added the filter to obj. versioning, but it's not even on the pipeline right now | 20:26 |
| clayg | oh god damnit, there was a bunch of test cases missing for efficient replication | 20:26 |
| clayg | what's kota's handle? | 20:27 |
| *** annegent_ has quit IRC | 20:27 | |
| *** ElvisTeixeira has quit IRC | 20:27 | |
| clayg | what maybe this is just ssync - what was the bug with ssync delete's aren't replicated? | 20:28 |
| peluse | clayg, that was a one liner wrong timespamt | 20:30 |
| peluse | #link https://review.openstack.org/#/c/155497/ | 20:30 |
| jrichli | clayg: it was kota_ today at the meeting. I assume usually just kota | 20:31 |
| *** cpallares has quit IRC | 20:32 | |
| *** cpallares has joined #openstack-swift | 20:32 | |
| *** marzif_ has joined #openstack-swift | 20:33 | |
| clayg | well there's definately something wrong with ssync currently, the follow up cleanup i was working on to efficient replication fixes *that* issue - i'm still not sure i'm able to duplicate the rsync issue? | 20:34 |
| openstackgerrit | Clay Gerrard proposed openstack/swift: Cleanup object-replicator handoff/rebalance partition syncing https://review.openstack.org/158927 | 20:35 |
| openstackgerrit | Clay Gerrard proposed openstack/swift: Sync local regions first, and always shuffle nodes https://review.openstack.org/159241 | 20:35 |
| clayg | if anyone is able to reliably duplicate the probetest failure - could you check if https://review.openstack.org/158927 makes a difference? | 20:35 |
| clayg | phhpthththht sam already said it fixed probetests for him :'( | 20:37 |
| tdasilva | nice..running now | 20:37 |
| marzif_ | clayg, in regards https://bugs.launchpad.net/python-swiftclient/+bug/1425283 I've tried today and HTTP(S)_PROXY env are honored, not sure the bug make sense, probably only to override proxy related envs | 20:37 |
| openstack | Launchpad bug 1425283 in python-swiftclient "proxy argument needed in client.Connection " [Undecided,New] | 20:37 |
| clayg | marzif_: yeah that makes sense to me, but for the time being you can work around with os.environ['http_proxy'] = 'http://myproxy.org' or whatever? | 20:38 |
| acoles | clayg: i see probetest errors with ssync http://paste.openstack.org/show/182098/, all good with rsync | 20:40 |
| acoles | clayg: just applying https://review.openstack.org/158927 | 20:40 |
| marzif_ | clayg, yes, that works | 20:40 |
| *** jrichli_ has joined #openstack-swift | 20:44 | |
| *** jrichli has quit IRC | 20:45 | |
| openstackgerrit | paul luse proposed openstack/swift: Erasure Code Reconstructor https://review.openstack.org/131872 | 20:46 |
| *** Manshoon_ has joined #openstack-swift | 20:50 | |
| tdasilva | clayg: probetests pass for me too with your patch 158927 | 20:50 |
| acoles | clayg: ^^ ditto all good with https://review.openstack.org/158927 and ssync | 20:51 |
| *** marzif_ has quit IRC | 20:51 | |
| notmyname | clayg: likewise | 20:51 |
| *** Manshoon has quit IRC | 20:51 | |
| jrichli_ | clayg: ditto | 20:52 |
| *** MasterPiece has quit IRC | 20:55 | |
| acoles | clayg: and i checked the failing tests are passing with rsync and 158927 | 20:56 |
| *** MasterPiece has joined #openstack-swift | 20:57 | |
| clayg | ok sweet! so you guys will probably let me squeak that one in too! | 20:57 |
| peluse | clayg, any clue what the fix is in there? | 20:58 |
| clayg | lol! | 20:58 |
| clayg | peluse: that's a great question :\ | 20:58 |
| clayg | peluse: i'd bet it's the dangling elif | 20:59 |
| peluse | yes! | 20:59 |
| *** acoles is now known as acoles_away | 21:00 | |
| clayg | with a single region delete_objs is false-y, but sync_method != rsync either - so you don't clean up the suffixes | 21:00 |
| peluse | I'll pull it down as well and review/test here | 21:00 |
| doxavore | clayg: while reproducing the client disconnect in a smaller script, i found a bug in this code, so it's entirely our fault. thanks though! | 21:01 |
| clayg | ok, so i'm pretty sure all this "probetests is failing everywhereahz@ahahah!!!!!11!11" is acctually limited to probetests is failing if you're using ssync with a single region - which is sort of conflicting with some of the reports people seemed to be giving in channel here so idk | 21:02 |
| clayg | doxavore: DOH! | 21:02 |
| clayg | doxavore: can you give me a cluse though? a) I don't think a client should be able to get all the bytes and still make us 499 b) when this happens to next guy I'd love to know what to look out for | 21:03 |
| jrichli_ | clayg: it was failing for me before that patch, and I was using rsync and default number of regions. | 21:03 |
| tdasilva | same here | 21:03 |
| peluse | clayg, it was failing for me with rsync, testing now with your patch | 21:03 |
| peluse | if it passes I'll actually red the code :) | 21:03 |
| peluse | read | 21:04 |
| clayg | well DAMNIT tdasilva jrichli_ peluse - just because all of you see the same thing doesn't mean you're not all wrong and i'm still right :P | 21:04 |
| * clayg is just kidding | 21:04 | |
| jrichli_ | lol | 21:04 |
| clayg | i'm sure I'm confused about something - but i'm missing it :\ | 21:04 |
| *** Manshoon has joined #openstack-swift | 21:04 | |
| clayg | maybe i should just be happy we have a change that makes it seem to work - I think that dangling elif was definately a bug with 1 region ssync - not sure what the 1 region rsync bug was :\ | 21:05 |
| *** Manshoon has quit IRC | 21:05 | |
| peluse | clayg, excellent point earlier BTW :) | 21:05 |
| *** Manshoon has joined #openstack-swift | 21:05 | |
| jrichli_ | it's always good to be happy | 21:05 |
| doxavore | clayg: this case seemed to stem from a misunderstanding of Ruby's IO.copy_stream, used to copy data from one stream to another. due to the bug in this particular code, it was requesting the file for every chunk (~16kb by default), but internally handling the start/end offsets. so when it finished writing the file the md5's still matched up, but every 16kb chunk would make a request to the proxy, read up to (previous | 21:06 |
| doxavore | position + 16kb), then stop | 21:06 |
| doxavore | everything looks great to the end user who sees their file's md5 matches up, but behind the scenes it made a _lot_ of partial requests | 21:06 |
| *** Manshoon_ has quit IRC | 21:08 | |
| clayg | doxavore: oh! that's pretty cool. | 21:10 |
| peluse | clayg, so my probes pass now as well (rsync) so I guess I'll go ahead and take a look at your patch.... | 21:11 |
| *** dmsimard is now known as dmsimard_away | 21:12 | |
| clayg | i'm afk for a bit - someone see if they an get torgomatic to drop some knowledge on us since he seems to be ahead on this one | 21:12 |
| mattoliverau | clayg: if it was a dangling elif then we're not going to hear the end of it are we (you did tell us you hate them at hackathon) :p | 21:23 |
| clayg | mattoliverau: yeah then I should have said something when I review'd it :P | 21:29 |
| *** zhill has joined #openstack-swift | 21:34 | |
| peluse | BTW, its not the danlgins elif for the rsync fix at least - there's more to it than that (was curious so backed just that out) | 21:39 |
| openstackgerrit | Thiago da Silva proposed openstack/swift: WIP: initial pass at refactoring the PUT method https://review.openstack.org/156825 | 21:41 |
| torgomatic | clayg: I dropped all the knowledge I had; I just "git bisect"-ed until I found what broke it | 21:45 |
| *** rdaly2_ has quit IRC | 21:46 | |
| *** tsg_ has quit IRC | 21:47 | |
| *** tsg_ has joined #openstack-swift | 21:48 | |
| *** tgohad has joined #openstack-swift | 21:52 | |
| *** tsg_ has quit IRC | 21:53 | |
| notmyname | torgomatic: from wikipedia: "The release initially included Linux kernel 3.13, but this was updated to 3.16..." | 21:54 |
| *** Manshoon has quit IRC | 22:07 | |
| *** tgohad has quit IRC | 22:07 | |
| *** Manshoon has joined #openstack-swift | 22:08 | |
| *** NM has quit IRC | 22:08 | |
| *** Manshoon has quit IRC | 22:08 | |
| openstackgerrit | Joel Wright proposed openstack/python-swiftclient: Reduce memory usage for download/delete and add --no-shuffle option to st_download https://review.openstack.org/130339 | 22:08 |
| *** Manshoon has joined #openstack-swift | 22:08 | |
| notmyname | joeljwright: thanks for all your work on swiftclient | 22:11 |
| notmyname | joeljwright: were were talking in the swift team meeting today about the next release for swiftclient. that patch you just submitted is one thing we think should land before a release | 22:11 |
| notmyname | joeljwright: what do you think? | 22:11 |
| *** akhivin has joined #openstack-swift | 22:12 | |
| klrmn | clayg: is it my imagination, or does BrainSplitter.put_container, if called more than once on any given brain, try to create two containers with the same name but different policies? | 22:13 |
| *** vt102 has quit IRC | 22:16 | |
| zaitcev | "Any unexpected Exception will now attempt to abort any downloads in progress before re-raising the exception." -- the code looks fine, BUT WHY | 22:16 |
| zaitcev | Clearly I'm missing something obvious | 22:17 |
| clayg | klrmn: the brainsplitter originally came from the reconciler probe tests - if you're cleaning things up i think it's probably worth while to try and pull that out of there into something else that makes more sense - something about singularity of purpose | 22:18 |
| *** echevemaster has joined #openstack-swift | 22:20 | |
| klrmn | clayg: yah, i had that one flagged as needing more attention | 22:21 |
| klrmn | clayg: i was contemplating whether all of the container and object level tests needed to happen on multiple policies | 22:22 |
| klrmn | clayg: (some do, some don't) | 22:22 |
| *** mrda-away is now known as mrda | 22:24 | |
| *** cl__ has joined #openstack-swift | 22:39 | |
| *** cdelatte has joined #openstack-swift | 22:40 | |
| cl__ | hello, does anyone know if swift ever breaks an object into piece when the object is too big to be fit in a partition? | 22:41 |
| cl__ | clayg, glange , mattoliverau :) | 22:43 |
| mattoliverau | cl__: not by default, if you use the swift client it can break it up for you (if you use the --segment-size upload option). | 22:49 |
| mattoliverau | It will then upload it as a DLO. | 22:49 |
| *** Manshoon has quit IRC | 22:50 | |
| *** tsg has joined #openstack-swift | 22:52 | |
| cl__ | mattoliverau, yup. Thanks a lot! | 22:58 |
| *** aerwin has quit IRC | 23:06 | |
| *** erlon has quit IRC | 23:11 | |
| *** jkremer has quit IRC | 23:12 | |
| *** reed has quit IRC | 23:13 | |
| *** gyee has quit IRC | 23:20 | |
| openstackgerrit | Thiago da Silva proposed openstack/swift: WIP: new attempt at single-process https://review.openstack.org/159285 | 23:22 |
| tdasilva | clayg: ^^^ single process bypassing much of the backend connections in the PUT path of the proxy, very simple | 23:23 |
| tdasilva | clayg: assumes 1-replica paco system like we talked about during hackathon | 23:23 |
| akhivin | Hello everyone. | 23:23 |
| akhivin | Just one question. Why after I installed swift as described in SAIO tutorilal. 4 tests are failing. It are ACL tests | 23:26 |
| akhivin | Should I install keystone? | 23:26 |
| openstackgerrit | Pete Zaitcev proposed openstack/python-swiftclient: Unindent a chunk of st_list https://review.openstack.org/154653 | 23:26 |
| zaitcev | ! [remote rejected] HEAD -> refs/publish/master/formatting (no changes made) | 23:29 |
| zaitcev | haven't seen that before... | 23:29 |
| zaitcev | Whatever. | 23:30 |
| notmyname | akhivin: what tests are you running? | 23:38 |
| *** km has joined #openstack-swift | 23:39 | |
| akhivin | notmyname, I am about unit tests | 23:41 |
| akhivin | notmyname, $HOME/swift/.unittests as noted in tutorial | 23:42 |
| *** chlong has quit IRC | 23:43 | |
| *** chlong_ has quit IRC | 23:44 | |
| notmyname | akhivin: hmm. what tests are failing? | 23:47 |
| *** chlong has joined #openstack-swift | 23:48 | |
| akhivin | for example | 23:48 |
| akhivin | FAIL: test_acls_propagate_to_sysmeta (test.unit.common.middleware.test_tempauth.TestAuthMultiplePrefixes) | 23:49 |
| akhivin | FAIL: test_acl_syntax_verification (test.unit.common.middleware.test_tempauth.TestAuthMultiplePrefixes) | 23:50 |
| *** doxavore has quit IRC | 23:50 | |
| akhivin | notmyname, is it a common issue or I missed something? | 23:54 |
| notmyname | akhivin: nope. it should work | 23:55 |
| akhivin | notmyname, so I will try to understand | 23:56 |
| akhivin | notmyname, thanks | 23:56 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!