| *** bkopilov_ has quit IRC | 00:04 | |
| *** itlinux has quit IRC | 00:12 | |
| *** itlinux has joined #openstack-swift | 00:15 | |
| *** bkopilov_ has joined #openstack-swift | 00:17 | |
| *** itlinux has quit IRC | 00:18 | |
| *** ukaynar has quit IRC | 00:18 | |
| *** ukaynar has joined #openstack-swift | 00:18 | |
| *** ukaynar has quit IRC | 00:23 | |
| *** kiennt has joined #openstack-swift | 00:33 | |
| *** psachin has joined #openstack-swift | 00:36 | |
| *** bkopilov_ has quit IRC | 00:47 | |
| *** tovin07_ has joined #openstack-swift | 00:50 | |
| openstackgerrit | Merged openstack/swift master: added .htaccess redirect file for docs https://review.openstack.org/488202 | 00:53 |
|---|---|---|
| openstackgerrit | Merged openstack/swift master: 2.15.0 authors and changelog update https://review.openstack.org/485839 | 00:54 |
| *** ukaynar has joined #openstack-swift | 00:56 | |
| *** catintheroof has quit IRC | 00:57 | |
| *** two_tired has joined #openstack-swift | 00:57 | |
| *** bkopilov_ has joined #openstack-swift | 01:00 | |
| *** catintheroof has joined #openstack-swift | 01:18 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift master: Updated from global requirements https://review.openstack.org/88736 | 01:24 |
| *** catintheroof has quit IRC | 01:25 | |
| *** JimCheung has quit IRC | 01:25 | |
| *** JimCheung has joined #openstack-swift | 01:26 | |
| notmyname | https://review.openstack.org/488250 is for the 2.15.0 swift release | 01:29 |
| patchbot | patch 488250 - releases - swift 2.15.0 release | 01:29 |
| *** JimCheung has quit IRC | 01:30 | |
| *** JimCheung has joined #openstack-swift | 01:45 | |
| *** JimCheung has quit IRC | 01:49 | |
| *** itlinux has joined #openstack-swift | 02:01 | |
| *** itlinux has quit IRC | 02:03 | |
| *** itlinux has joined #openstack-swift | 02:26 | |
| *** kei_yama has quit IRC | 03:36 | |
| *** kei_yama has joined #openstack-swift | 03:45 | |
| *** nakagawamsa has joined #openstack-swift | 03:49 | |
| *** kei_yama has quit IRC | 03:49 | |
| *** gkadam has joined #openstack-swift | 03:55 | |
| *** gkadam_ has joined #openstack-swift | 03:57 | |
| *** gkadam_ is now known as gkadam-afk | 04:00 | |
| *** gkadam has quit IRC | 04:00 | |
| *** gkadam-afk has quit IRC | 04:00 | |
| *** gkadam has joined #openstack-swift | 04:00 | |
| *** itlinux has quit IRC | 04:07 | |
| *** links has joined #openstack-swift | 04:20 | |
| *** nakagawamsa has quit IRC | 04:44 | |
| *** mathiasb has joined #openstack-swift | 04:49 | |
| *** psachin has quit IRC | 04:50 | |
| *** ukaynar has quit IRC | 05:05 | |
| *** psachin has joined #openstack-swift | 05:08 | |
| *** ukaynar has joined #openstack-swift | 05:09 | |
| *** gyee has quit IRC | 05:13 | |
| *** kiennt has quit IRC | 05:44 | |
| *** kiennt has joined #openstack-swift | 05:44 | |
| *** kiennt has quit IRC | 05:50 | |
| *** kiennt has joined #openstack-swift | 06:05 | |
| *** rcernin has joined #openstack-swift | 06:15 | |
| *** two_tired has quit IRC | 06:20 | |
| *** klrmn has quit IRC | 06:21 | |
| *** ukaynar has quit IRC | 06:22 | |
| *** hseipp has joined #openstack-swift | 06:42 | |
| *** cschwede_ has joined #openstack-swift | 07:03 | |
| *** openstackgerrit has quit IRC | 08:18 | |
| *** xrb has joined #openstack-swift | 08:20 | |
| acoles | clayg: re patch 487168, at some point I was concerned about there being no aggregate stats dump at end of a run_once, in fact if run_once finishes within a stats_interval then there may be no aggregate stats dump. timburke and I discussed some final 'cleanup' call to the daemon to trigger a final aggregation, but this seemed a possible alternative. I just played with the idea hence WIP. | 08:20 |
| patchbot | https://review.openstack.org/#/c/487168/ - swift - WIP try synchronous recon aggregation (ABANDONED) | 08:20 |
| acoles | clayg: it fell by the way (bigger fish to fry) because with run_once you *do* get per disk stats and so long as you have the background daemon running then any run-once per disk stats *will* be rolled up by that. | 08:22 |
| acoles | good morning | 08:23 |
| acoles | clayg: and congrats - multiprocess reconstructor is a thing - that is great! | 08:24 |
| *** cshastri has joined #openstack-swift | 08:35 | |
| *** mvk has quit IRC | 08:36 | |
| *** xrb has quit IRC | 09:00 | |
| *** xrb_ has joined #openstack-swift | 09:00 | |
| *** JimCheung has joined #openstack-swift | 09:03 | |
| *** JimCheung has quit IRC | 09:07 | |
| *** ChubYann has quit IRC | 09:34 | |
| *** skudlik has joined #openstack-swift | 09:53 | |
| *** tovin07_ has quit IRC | 10:07 | |
| *** kiennt has quit IRC | 10:30 | |
| *** catintheroof has joined #openstack-swift | 11:48 | |
| *** rcernin has quit IRC | 11:55 | |
| *** Dinesh_Bhor is now known as Dinesh_Bhor|afk | 12:01 | |
| *** Dinesh_Bhor|afk is now known as Dinesh_Bhor | 12:02 | |
| *** catintheroof has quit IRC | 12:14 | |
| *** dosaboy has quit IRC | 12:14 | |
| *** rcernin has joined #openstack-swift | 12:16 | |
| *** xrb_ has quit IRC | 12:31 | |
| *** xrb has joined #openstack-swift | 12:31 | |
| *** xrb has quit IRC | 12:31 | |
| *** xrb has joined #openstack-swift | 12:32 | |
| *** JimCheung has joined #openstack-swift | 12:39 | |
| *** JimCheung has quit IRC | 12:43 | |
| *** catintheroof has joined #openstack-swift | 12:50 | |
| *** gkadam has quit IRC | 12:52 | |
| *** dosaboy has joined #openstack-swift | 12:57 | |
| *** jaosorior has quit IRC | 13:04 | |
| *** ukaynar has joined #openstack-swift | 13:04 | |
| *** links has quit IRC | 13:16 | |
| *** ukaynar has quit IRC | 13:18 | |
| *** psachin has quit IRC | 13:35 | |
| *** rcernin has quit IRC | 13:38 | |
| *** jistr is now known as jistr|mtg | 13:45 | |
| *** neonpastor has joined #openstack-swift | 13:45 | |
| *** hseipp has quit IRC | 13:54 | |
| *** rcernin has joined #openstack-swift | 14:01 | |
| *** gyee has joined #openstack-swift | 14:19 | |
| *** AlexeyAbashkin has joined #openstack-swift | 14:20 | |
| *** SkyRocknRoll has joined #openstack-swift | 14:20 | |
| *** itlinux has joined #openstack-swift | 14:24 | |
| *** skudlik has quit IRC | 14:41 | |
| *** jistr|mtg is now known as jistr | 14:46 | |
| *** Dinesh_Bhor has quit IRC | 14:46 | |
| *** ukaynar has joined #openstack-swift | 15:00 | |
| *** rcernin has quit IRC | 15:13 | |
| *** klrmn has joined #openstack-swift | 15:14 | |
| *** links has joined #openstack-swift | 15:23 | |
| notmyname | good morning | 15:32 |
| *** itlinux has joined #openstack-swift | 15:40 | |
| *** itlinux has quit IRC | 15:48 | |
| *** klrmn has quit IRC | 16:09 | |
| *** ukaynar has quit IRC | 16:16 | |
| *** xrb has quit IRC | 16:24 | |
| *** xrb has joined #openstack-swift | 16:25 | |
| *** JimCheung has joined #openstack-swift | 16:31 | |
| timburke | good morning | 16:33 |
| *** xrb has quit IRC | 16:37 | |
| *** cschwede_ has quit IRC | 16:37 | |
| *** xrb has joined #openstack-swift | 16:37 | |
| *** cschwede_ has joined #openstack-swift | 16:37 | |
| *** openstackstatus has quit IRC | 16:41 | |
| *** openstack has joined #openstack-swift | 16:44 | |
| *** openstackstatus has joined #openstack-swift | 16:44 | |
| *** ChanServ sets mode: +v openstackstatus | 16:44 | |
| *** ukaynar has joined #openstack-swift | 16:46 | |
| *** ukaynar has quit IRC | 16:46 | |
| *** ukaynar has joined #openstack-swift | 16:47 | |
| *** tonanhngo has joined #openstack-swift | 16:48 | |
| *** tonanhngo_ has joined #openstack-swift | 16:52 | |
| *** tonanhngo_ has quit IRC | 16:52 | |
| *** tonanhngo has quit IRC | 16:52 | |
| *** tonanhngo has joined #openstack-swift | 16:54 | |
| *** catinthe_ has joined #openstack-swift | 16:54 | |
| *** catintheroof has quit IRC | 16:58 | |
| *** ukaynar has quit IRC | 17:02 | |
| *** klrmn has joined #openstack-swift | 17:12 | |
| -openstackstatus- NOTICE: The Gerrit service on review.openstack.org is being taken offline for roughly 5 minutes to perform a database backup and reconfiguration | 17:13 | |
| *** itlinux has joined #openstack-swift | 17:14 | |
| *** ukaynar has joined #openstack-swift | 17:15 | |
| *** gyee has quit IRC | 17:23 | |
| *** gyee has joined #openstack-swift | 17:25 | |
| *** skudlik has joined #openstack-swift | 17:30 | |
| *** cshastri has quit IRC | 17:36 | |
| *** chsc has joined #openstack-swift | 17:38 | |
| *** chsc has joined #openstack-swift | 17:38 | |
| *** cschwede_ has quit IRC | 17:43 | |
| *** ChubYann has joined #openstack-swift | 17:53 | |
| *** ukaynar has quit IRC | 18:00 | |
| *** ukaynar has joined #openstack-swift | 18:01 | |
| *** ukaynar has quit IRC | 18:01 | |
| *** gyee has quit IRC | 18:03 | |
| *** gyee has joined #openstack-swift | 18:06 | |
| *** catinthe_ has quit IRC | 18:08 | |
| *** catintheroof has joined #openstack-swift | 18:10 | |
| *** openstackstatus has quit IRC | 18:11 | |
| *** openstack has joined #openstack-swift | 18:14 | |
| *** openstackstatus has joined #openstack-swift | 18:14 | |
| *** ChanServ sets mode: +v openstackstatus | 18:14 | |
| *** gyee has quit IRC | 18:15 | |
| clayg | acoles: timburke: thinking mainly for dev - it would be very desirable if run once (no overrides) resulted in a "complete" recon dump - for all daemons! | 18:41 |
| clayg | Dunno if we have that or I regressed it with worker config... | 18:41 |
| *** openstackstatus has quit IRC | 18:42 | |
| *** openstack has joined #openstack-swift | 18:44 | |
| *** openstackstatus has joined #openstack-swift | 18:44 | |
| *** ChanServ sets mode: +v openstackstatus | 18:44 | |
| *** ukaynar has joined #openstack-swift | 18:52 | |
| *** SkyRocknRoll has quit IRC | 19:10 | |
| acoles | clayg: agree. that's what prompted me to start on https://review.openstack.org/#/c/487168/ (IDK if you saw in scrollback) but I really didn't get far with that train of thought | 19:14 |
| patchbot | patch 487168 - swift - WIP try synchronous recon aggregation (ABANDONED) | 19:14 |
| *** SkyRocknRoll has joined #openstack-swift | 19:23 | |
| *** ukaynar has quit IRC | 19:32 | |
| *** noxdafox has quit IRC | 19:36 | |
| *** noxdafox has joined #openstack-swift | 19:42 | |
| *** JimCheung has quit IRC | 19:46 | |
| *** JimCheung has joined #openstack-swift | 19:46 | |
| *** SkyRocknRoll has quit IRC | 20:05 | |
| *** catintheroof has quit IRC | 20:08 | |
| *** SkyRocknRoll has joined #openstack-swift | 20:17 | |
| *** ukaynar has joined #openstack-swift | 21:09 | |
| *** ukaynar has quit IRC | 21:10 | |
| *** SkyRocknRoll has quit IRC | 21:11 | |
| *** ukaynar has joined #openstack-swift | 21:15 | |
| *** ukaynar has quit IRC | 21:19 | |
| *** skudlik has quit IRC | 22:03 | |
| *** itlinux has quit IRC | 22:05 | |
| timur | I think I might take up this fight: https://bugs.launchpad.net/python-swiftclient/+bug/1159951. Propagating X-Delete-At and X-Delete-After headers to segments seems reasonable to me. Although, so does removing segments on expiration of a large object -- although, I realize that may lead to unexpected results for folks that use same segments across multiple objects | 22:13 |
| openstack | Launchpad bug 1159951 in python-swiftclient "swift client header is not used for uploading large object" [Undecided,In progress] - Assigned to Tong Li (litong01) | 22:13 |
| notmyname | FYI https://bugs.launchpad.net/openstack-ansible/+bug/1707220 | 22:13 |
| openstack | Launchpad bug 1707220 in openstack-ansible "Swift Erasure Code fails with liberasurecode 1.4.0 on CentOS" [Undecided,New] | 22:13 |
| timur | timburke: have you thought about this at all? | 22:13 |
| timur | notmyname: what's the best way to sort python-swiftclient bugs? heat or importance? | 22:15 |
| notmyname | timur: IMO importance. but I'm not sure there's a "best" way that's always right | 22:15 |
| timur | I used heat as a measure of people complaining | 22:15 |
| timburke | timur: some attempt was made in https://review.openstack.org/#/c/310075/ though i clearly hadn't done a good job of it. the trouble as i recall is that we need to do a HEAD before *every* POST in case any expiry is *already set* and we should clear it on the segments (if any) | 22:16 |
| patchbot | patch 310075 - python-swiftclient - Expire segments when expiring large objects | 22:16 |
| timur | timburke: yes, it will be terrible | 22:16 |
| timur | but I don't think there is any alternative? | 22:18 |
| timur | maybe we can resurrect that review? | 22:19 |
| clayg | FROM THE DEAD | 22:21 |
| timburke | timur: go for it! | 22:22 |
| timur | hehe alright. I'll see what I can do with yours and Joel's comments. Seems like a useful thing for us to fix | 22:22 |
| timburke | hell, we already do a HEAD before every PUT... maybe we could (ab)use the --leave-segments option to skip it on POST, too? or make it --ignore-segments or something... | 22:23 |
| timburke | notmyname: isn't BadStatusLine usually when something dies down in eventlet? those "ERROR WSGI" lines seem suspicious: http://logs.openstack.org/25/485225/5/check/gate-openstack-ansible-os_swift-ansible-func-centos-7/8ad31e6/logs/openstack/swift-proxy/swift/proxy-error.log.txt.gz#_Jul_24_19_22_55 | 22:33 |
| timburke | no clue why upgrading *libec* would fix it, though... | 22:34 |
| notmyname | timburke: yeah, the ERROR WSGI do make it sound like something in eventlet | 22:35 |
| notmyname | timburke: but scroll up? the traceback does look like a libec thing | 22:36 |
| notmyname | no, sorry, not libec | 22:37 |
| notmyname | like a large object thing. maybe ACL issues? | 22:37 |
| notmyname | but yeah, don't know how libec figures into that | 22:37 |
| timburke | ? no... makes me think we shouldn't be logging at exception around https://github.com/openstack/swift/blob/2.15.0/swift/common/request_helpers.py#L506-L507 (at least, not when it's on account of a 403) | 22:38 |
| timburke | i'm trying to remember zaitcev's observations that led to https://github.com/openstack/liberasurecode/commit/0962144 and https://github.com/openstack/liberasurecode/commit/960cdd0 ... was it that *nothing* worked on that box, or only *some* things didn't work? when things went wrong, what did the client see? i could totally understand SIGILL killing the proxy, causing the client to get a bad status line ... but i would've thought *eve | 22:46 |
| timburke | rything* would be bad? | 22:46 |
| zaitcev | timburke: the symptom was that the proxy crashed on start. HOWEVER, a EC policy was configured, so it's possible that nothing would happen unless that. | 22:51 |
| zaitcev | [422171.621114] traps: swift-container[11669] trap invalid opcode ip:7f531ddf2d11 sp:7ffe0583cb00 error:0 in liberasurecode.so.1.4.0[7f531ddef000+d000] | 22:51 |
| zaitcev | actually, not just proxy | 22:51 |
| timburke | zaitcev: thanks. that was kinda what i figured... was looking for anything that could have landed between 1.4.0 and 1.5.0 that would've maybe addressed bug 1707220 | 22:52 |
| openstack | bug 1707220 in openstack-ansible "Swift Erasure Code fails with liberasurecode 1.4.0 on CentOS" [Undecided,New] https://launchpad.net/bugs/1707220 | 22:52 |
| clayg | timburke: we'll have somehing in PT about protability and libec | 22:52 |
| zaitcev | I think this is different because it failed during operation, not when starting. But who knows how they compiled it. Maybe it wasn't optimized enough to crash right away and survived until the time the chunk sizes were calculated. | 22:53 |
| timburke | notmyname: further observations... four failed tests, four instances of "Removing dead child" in the proxy error log... | 22:57 |
| notmyname | timburke: STOP KILLING THE CHILDREN! | 22:58 |
| timburke | notmyname: hey, it's not the poor parent's fault! it just wants to keep from having dead children piling up everywhere! | 22:59 |
| clayg | point is - it does sound like segfaults? invalid op code does *totally* remind me of the non-portable build issue with how libec does supported flag detection | 23:06 |
| clayg | so... maybe someone needs to rebuild something for some distro somewhere with better flags - or we need to stop occasionally/randomly using vms' with different cpu flags than what we're building with... | 23:07 |
| clayg | https://bugs.launchpad.net/openstack-ansible/+bug/1707220 - they already have confirmation it's fixed in the newer version? Are they thinking backport or something? Is it just FYI? | 23:10 |
| openstack | Launchpad bug 1707220 in openstack-ansible "Swift Erasure Code fails with liberasurecode 1.4.0 on CentOS" [Undecided,New] | 23:10 |
| clayg | oh, that's not against any project I know anything about - i should probably shut up | 23:13 |
| *** chsc has quit IRC | 23:15 | |
| *** tonanhngo has quit IRC | 23:22 | |
| *** tonanhngo has joined #openstack-swift | 23:28 | |
| *** tonanhngo has quit IRC | 23:33 | |
| *** tonanhngo has joined #openstack-swift | 23:36 | |
| *** tonanhngo has quit IRC | 23:41 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!