| *** Jeffrey4l has joined #openstack-swift | 00:03 | |
| *** ppai has quit IRC | 00:05 | |
| clayg | omg how have i not seen mat the woliver before!? what is a wolly-ver you ask? apparently a burly australian beast that's good at code. | 00:22 |
|---|---|---|
| clayg | anyone else use vsaio that would like to help beta xenail support -> https://github.com/swiftstack/vagrant-swift-all-in-one/pull/50 | 00:24 |
| clayg | tdasilva: there is a *tiny* little green link "Mark as duplicate" in one of the grey blocks on the RHS of the lp layout | 00:32 |
| clayg | tdasilva: i didn't know about it till notmyname poitned it out to me | 00:32 |
| clayg | tdasilva: but I used to to mark lp bug #1314799 as a duplicate of lp bug #1560574 | 00:32 |
| openstack | Launchpad bug 1560574 in OpenStack Object Storage (swift) "duplicate for #1314799 GET/HEAD should stop continuing to search more nodes if a tombstone is reached" [Medium,In progress] https://launchpad.net/bugs/1560574 - Assigned to Thiago da Silva (thiagodasilva) | 00:32 |
| openstack | Launchpad bug 1560574 in OpenStack Object Storage (swift) "GET/HEAD should stop continuing to search more nodes if a tombstone is reached" [Medium,In progress] https://launchpad.net/bugs/1560574 - Assigned to Thiago da Silva (thiagodasilva) | 00:32 |
| clayg | i guess technically it should go the other way around - but know you're working on that | 00:33 |
| clayg | i don't know why your it didn't link your review to the bug? | 00:33 |
| mattoliverau | clayg: can't trust those woliver's | 00:36 |
| *** klamath has quit IRC | 00:37 | |
| *** nexusz99 has joined #openstack-swift | 00:37 | |
| kota_ | good morning | 00:45 |
| kota_ | it looks like we got great progress while I was in holiday yesterday. | 00:45 |
| mattoliverau | kota_: o/ | 00:47 |
| kota_ | mattoliverau: o/ | 00:47 |
| *** gyee has quit IRC | 01:01 | |
| *** jamielennox is now known as jamielennox|away | 01:03 | |
| *** manous has joined #openstack-swift | 01:03 | |
| *** nexusz99 has quit IRC | 01:11 | |
| *** nexusz99 has joined #openstack-swift | 01:12 | |
| clayg | timburke: playing with proxy protocol change it *does* feel like a general solution to https is in there somewhere? | 01:12 |
| timburke | clayg: right? like, this seems to be *exactly* the sort of thing this was designed for | 01:16 |
| *** nexusz99 has quit IRC | 01:16 | |
| clayg | something in the wsgi environ probably - the proxy protcol is sorta weird in that it's protocol agnostic - we can't have stud "insert a header" | 01:19 |
| *** abhitechie has quit IRC | 01:19 | |
| clayg | but in our protocol hander we could update the wsgi envrion... but I'm not 100% sure what's appropriet? | 01:19 |
| *** jamielennox|away is now known as jamielennox | 01:22 | |
| *** tqtran has quit IRC | 01:34 | |
| clayg | timburke: i don't know about *designed* -> https://gist.github.com/clayg/0864f5a18be7995acc85cd03814dcedd | 01:37 |
| clayg | looks like kludge to me | 01:37 |
| clayg | i agree it's out of scope - but i have to admit it's interesting | 01:37 |
| zhengyin | good morning | 01:58 |
| zhengyin | kota_, mattoliverau:morning | 01:58 |
| kota_ | zhengyin: o/ | 01:58 |
| kota_ | oh, patch 373537 has been landed | 02:03 |
| patchbot | https://review.openstack.org/#/c/373537/ - swift - Use separate headers for versioned_writes' stack a... (MERGED) | 02:03 |
| kota_ | timburke: does that mean I have to rewrite patch 360933? | 02:03 |
| patchbot | https://review.openstack.org/#/c/360933/ - swift - Add functional tests for new versioned_write mode | 02:03 |
| kota_ | timburke: the commit message in patch 373537 says, "x-versions-mode is now ignored". Hmm, my functional will probably be ignored :/ | 02:04 |
| patchbot | https://review.openstack.org/#/c/373537/ - swift - Use separate headers for versioned_writes' stack a... (MERGED) | 02:04 |
| clayg | swifterdarrell: got a traceback with the PROXY protocol probably worth fixing -> https://gist.github.com/clayg/80368d1aed6a512e962bc6ea07d54e0b | 02:08 |
| clayg | timburke: tdasilva: holy crap you guys merged the versioned writes thing! | 02:08 |
| clayg | kota_: man that sucks! we need to get some functional tests on that bad boy! | 02:10 |
| clayg | timburke: tdasilva: ^^^ patch 360933 | 02:10 |
| patchbot | https://review.openstack.org/#/c/360933/ - swift - Add functional tests for new versioned_write mode | 02:10 |
| clayg | oh tim already +2'd it - nice work timburke | 02:10 |
| kota_ | calyg: never mind, I'll try to rework on that today. | 02:12 |
| kota_ | clayg: thanks adding "recheck no bug" ;-) if i understand correctly, it will fail :P | 02:13 |
| *** abhitechie has joined #openstack-swift | 02:30 | |
| *** jamielennox is now known as jamielennox|away | 02:37 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift: EC Fragment Duplication - Foundational Global EC Cluster Support https://review.openstack.org/219165 | 02:43 |
| kota_ | rebased. | 02:44 |
| *** jamielennox|away is now known as jamielennox | 02:51 | |
| *** vinsh has joined #openstack-swift | 02:58 | |
| *** vinsh has quit IRC | 03:03 | |
| *** david-lyle has quit IRC | 03:04 | |
| *** _JZ_ has quit IRC | 03:05 | |
| kota_ | curious, my functional tests for versioned_writes passed even if i don't care about the new name spacce | 03:29 |
| *** chsc has joined #openstack-swift | 03:48 | |
| mattoliverau | kota_: that's strange. The new x-history-header functionality existed previously, you just had to pass in X-Versions-Mode to select stack or history.. so I guess it depends on how your setting the new mode. | 03:56 |
| mattoliverau | *x-history-location i mean | 03:57 |
| kota_ | mattolivearu: i have an idea for that, my funcitonal may be skipped if something changed in /info | 04:06 |
| *** chsc has quit IRC | 04:06 | |
| kota_ | mattoliverau: that func test handle the allowed_versions_mode to know if the test should run or not | 04:07 |
| kota_ | will figure out if my assumption was true or not. | 04:08 |
| *** klrmn has quit IRC | 04:11 | |
| *** nadeem has joined #openstack-swift | 04:12 | |
| kota_ | mattolivearu: my thought is true, http://logs.openstack.org/33/360933/3/check/gate-swift-dsvm-functional-ubuntu-xenial/ea464b2/console.html#_2016-09-23_02_32_00_786757 | 04:29 |
| kota_ | ok, let's make sure how we can find the availability of history mode from the newest master. | 04:30 |
| *** vinsh has joined #openstack-swift | 04:30 | |
| *** tqtran has joined #openstack-swift | 04:32 | |
| kota_ | ah, that was changed as "allowed_flags" to describes the location headers | 04:36 |
| kota_ | flags???? Is that correct word? not sure because I'm not a native Engish guy. | 04:36 |
| *** tqtran has quit IRC | 04:37 | |
| kota_ | anyway, probably I can track the new change for a short time. | 04:37 |
| *** abhinavtechie has joined #openstack-swift | 04:37 | |
| *** aagrawal has joined #openstack-swift | 04:39 | |
| *** abhitechie has quit IRC | 04:40 | |
| *** vinsh has quit IRC | 04:40 | |
| *** abhinavtechie has quit IRC | 04:43 | |
| openstackgerrit | John Dickinson proposed openstack/swift: authors/changelog updates for 2.10.0 https://review.openstack.org/375235 | 04:43 |
| notmyname | I'll let that sit overnight, but I need to land it asap in the morning to get the final SHA for 2.10.0 (all times my local tz) ^ | 04:44 |
| notmyname | please look it over and let me know if/where I goofed | 04:44 |
| *** abhitechie has joined #openstack-swift | 04:51 | |
| *** aagrawal has quit IRC | 04:55 | |
| kota_ | notmyname: thanks for working on the release | 05:07 |
| kota_ | notmyname: right now, I have a concern for the new versioned_write changes recently released | 05:07 |
| kota_ | notmyname: with the change, it seems we have a regression on the response header from versioned container | 05:08 |
| kota_ | notmyname: I'm now working on that to figure out it, hopefully, I'd like to add it before new release. | 05:08 |
| zaitcev | https://wiki.openstack.org/wiki/Swift/PriorityReviews looks pretty okay | 05:09 |
| kota_ | zaitcev: yeah, all items in "should land" have been closed. | 05:13 |
| kota_ | ah, it might be not a regression actually, just a bug in functests??? | 05:15 |
| kota_ | X-History-Location is in the response header from history versioning container | 05:16 |
| kota_ | it's time to loop back to look at the test client in functional test how it handels the container metadata. | 05:16 |
| *** zaitcev has quit IRC | 05:33 | |
| *** manous has quit IRC | 05:33 | |
| *** manous has joined #openstack-swift | 05:46 | |
| *** SkyRocknRoll has joined #openstack-swift | 05:55 | |
| *** hosanai has quit IRC | 05:55 | |
| *** dmorita has quit IRC | 05:59 | |
| openstackgerrit | gecong proposed openstack/swift: Use ConfigParser instead of SafeConfigParser in Python 3 https://review.openstack.org/375260 | 06:11 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift: Add functional tests for new versioned_write mode https://review.openstack.org/360933 | 06:12 |
| kota_ | tdasilva, timburke, clayg: I fixed and rebased the versioned_writes history mode functional tests addressed on the api change. | 06:13 |
| *** rcernin has joined #openstack-swift | 06:15 | |
| *** aswadr_ has joined #openstack-swift | 06:25 | |
| *** hseipp has joined #openstack-swift | 06:40 | |
| *** abhitechie has quit IRC | 06:54 | |
| *** nexusz99 has joined #openstack-swift | 07:01 | |
| *** rledisez has joined #openstack-swift | 07:22 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift: Add functional tests for new versioned_write mode https://review.openstack.org/360933 | 07:27 |
| *** geaaru has joined #openstack-swift | 07:30 | |
| *** nexusz99 has quit IRC | 07:33 | |
| *** nexusz99 has joined #openstack-swift | 07:33 | |
| *** nexusz99 has quit IRC | 07:34 | |
| *** nexusz99 has joined #openstack-swift | 07:34 | |
| *** amoralej|off is now known as amoralej | 07:36 | |
| *** nexusz99 has quit IRC | 07:36 | |
| *** abhitechie has joined #openstack-swift | 07:49 | |
| *** mariusv has joined #openstack-swift | 07:56 | |
| *** dmorita has joined #openstack-swift | 07:59 | |
| *** tqtran has joined #openstack-swift | 08:01 | |
| *** abhinavtechie has joined #openstack-swift | 08:01 | |
| *** sams-gleb has joined #openstack-swift | 08:02 | |
| *** asettle has joined #openstack-swift | 08:02 | |
| *** dmorita has quit IRC | 08:04 | |
| *** abhitechie has quit IRC | 08:05 | |
| *** tqtran has quit IRC | 08:05 | |
| *** asettle has quit IRC | 08:07 | |
| *** asettle has joined #openstack-swift | 08:09 | |
| *** asettle has quit IRC | 08:09 | |
| *** asettle has joined #openstack-swift | 08:09 | |
| *** abhitechie has joined #openstack-swift | 08:25 | |
| *** abhinavtechie has quit IRC | 08:27 | |
| *** nadeem has quit IRC | 08:29 | |
| *** admin6 has left #openstack-swift | 08:34 | |
| *** acoles_ is now known as acoles | 08:35 | |
| *** jordanP has joined #openstack-swift | 08:38 | |
| *** abhinavtechie has joined #openstack-swift | 08:44 | |
| *** abhitechie has quit IRC | 08:47 | |
| *** asettle has quit IRC | 08:49 | |
| *** asettle has joined #openstack-swift | 08:53 | |
| *** asettle has quit IRC | 08:54 | |
| *** asettle has joined #openstack-swift | 08:54 | |
| acoles | kota_: hi, is there anything i need to work on for the release while the rest of you sleep? i just saw you mention something in scrollback about a regression?? | 08:58 |
| kota_ | hi acoles | 08:58 |
| kota_ | actually, it dosn't regression but an api changed in the versioned writes | 08:59 |
| acoles | it tickles me that notmyname is landing patches from a plane, looking down upon us minions :D | 08:59 |
| acoles | kota_: ok | 08:59 |
| kota_ | acoles: older than the recent versioned write change, both stack mode and history mode return x-verions-location in the response | 08:59 |
| kota_ | acoles: due to the change x-versions-location to x-history-location on the history-mode, now history-mode versioned container doesn't return "x-versions-location" anymore | 09:00 |
| kota_ | and I hit a bug on the my written functional test which assuming "if the response includes x-versions-location, the versioning is available" | 09:01 |
| acoles | clayg: timburke notmyname thanks for getting patch 346865 landed, I completely agree with your decisions. I went for the ALL process rather than ZBF process because I thought ALL was the only one guranteed to run in all configs, maybe I am wrong but from staring at code I thought I could see a way in which there might not be a ZBF process forked. | 09:02 |
| patchbot | https://review.openstack.org/#/c/346865/ - swift - Delete old tombstones (MERGED) | 09:02 |
| kota_ | acoles: actually, it was changed to x-history-location on the master. | 09:02 |
| kota_ | ah but, if you could take a look to patch 360933, that is perfectly great, anyway ;-) | 09:03 |
| patchbot | https://review.openstack.org/#/c/360933/ - swift - Add functional tests for new versioned_write mode | 09:03 |
| *** openstackgerrit has quit IRC | 09:03 | |
| kota_ | acoles:^^ | 09:03 |
| *** openstackgerrit has joined #openstack-swift | 09:04 | |
| acoles | kota_: OIC. so do you think we should try to include that patch ^^ in the release? | 09:04 |
| kota_ | acoles: I'd like to do that because it helps users to work for making sure the new capability on the new swift release. | 09:05 |
| kota_ | i mean, it gives the way to test versioned write history mode api via functional test tools | 09:06 |
| acoles | clayg: also, once I was home with a glass of wine in my hand, I realised exactly what you said, that the rate of the two processes would be so different IRL that the dup's might not even occur (ZBF invalidates hash, replicator cleans up, ALL auditor never finds the tomb) | 09:06 |
| acoles | kota_: yeah, ok I will take a look today | 09:06 |
| kota_ | acoles: thanks! | 09:07 |
| *** Shadow_aok has joined #openstack-swift | 09:07 | |
| Shadow_aok | hi | 09:07 |
| acoles | mahatic: nice work on patch 346865 | 09:07 |
| patchbot | https://review.openstack.org/#/c/346865/ - swift - Delete old tombstones (MERGED) | 09:07 |
| Shadow_aok | could someone tell me which configuration files swift-proxy needs to run, except for proxy-server.conf ? | 09:08 |
| Shadow_aok | i found some others files in my swift folder, looks like they belongs to a storage server and have be put there by mistake (account-server.conf, object-server.conf, proxy-server.conf). | 09:09 |
| acoles | Shadow_aok: proxy server (and other servers) also needs swift.conf | 09:11 |
| acoles | the other files are config for swift account/container/object servers and other daemons like the replicators and auditor | 09:12 |
| openstackgerrit | Tuan Luong-Anh proposed openstack/swift: Replace six iteration methods with standard ones https://review.openstack.org/375344 | 09:19 |
| Shadow_aok | +acoles> thanks, so they are not needed on a server working exclusively as a proxy for swift ? | 09:20 |
| *** nexusz99 has joined #openstack-swift | 09:20 | |
| acoles | Shadow_aok: correct | 09:21 |
| *** nexusz99 has quit IRC | 09:22 | |
| *** nexusz99 has joined #openstack-swift | 09:23 | |
| Shadow_aok | thanks | 09:23 |
| acoles | Shadow_aok: you *may* also have a container-sync-realms.conf file which is needed on the proxy, but that is for the optional container sync feature | 09:24 |
| Shadow_aok | what does this optional feature does ? Never heard of it. | 09:26 |
| Shadow_aok | we're runing kilo by the way | 09:26 |
| Shadow_aok | http://docs.openstack.org/developer/swift/overview_container_sync.html | 09:26 |
| Shadow_aok | looks interesting | 09:26 |
| Shadow_aok | it's only used for mirroring a container on the same storage server or for replication between the storage servers ? | 09:27 |
| *** nexusz99 has quit IRC | 09:27 | |
| *** kei_yama has quit IRC | 09:30 | |
| *** aagrawal has joined #openstack-swift | 09:32 | |
| *** abhinavtechie has quit IRC | 09:35 | |
| *** abhitechie has joined #openstack-swift | 09:36 | |
| mahatic | acoles: clayg: thank you for seeing the patch 346865 through! it got in! | 09:37 |
| patchbot | https://review.openstack.org/#/c/346865/ - swift - Delete old tombstones (MERGED) | 09:37 |
| *** aagrawal has quit IRC | 09:38 | |
| mahatic | notmyname: I'm wondering if that ^ fix should be called out in changelog or does it come under one of those things which should obviously be working (and some of the users been thinking it was) but hadn't been? ;) | 09:40 |
| * mahatic notes to catch up on irc discussions sometime over the weekend | 09:43 | |
| mahatic | I'm here only briefly. Have a nice weekend everyone! | 09:43 |
| acoles | mahatic: i just left a comment on changelog patch to include your fix, notmyname probably authored the changelog before your patch landed | 09:46 |
| acoles | mahatic: have a good weekend | 09:46 |
| acoles | Shadow_aok: container sync can be used between clusters | 09:48 |
| acoles | as well as within a cluster | 09:48 |
| *** abhitechie has quit IRC | 09:48 | |
| mahatic | acoles: thanks! you're on top of the things, as usual :) | 09:48 |
| acoles | Shadow_aok: there's some more info here re container sync http://docs.openstack.org/developer/swift/admin_guide.html#geographically-distributed-swift-considerations | 09:49 |
| * acoles afk | 09:50 | |
| openstackgerrit | zheng yin proposed openstack/swift: Add test_PUT/DELETE_bad_timestamp in obj/test_server https://review.openstack.org/375377 | 10:17 |
| *** abhitechie has joined #openstack-swift | 10:32 | |
| *** abhitechie has quit IRC | 10:41 | |
| Shadow_aok | thanks | 10:50 |
| *** nexusz99 has joined #openstack-swift | 11:00 | |
| kota_ | while thinkg the versioning write via new api, I'm realizing to doubt if the consistency model could be safe (or not). | 11:18 |
| kota_ | because we have 2 ways to set the versioning container (location and mode) but the setting will be merged into a header. does it break container metadata semantics? | 11:19 |
| kota_ | I'm not sure right now, not yet deep dive into it but if i understand correctly, a request with x-versions-location to a container which keeps x-history-location will success and the x-history-location setting will be removed | 11:21 |
| kota_ | because, in previous version, we can overwrite the x-versions-mode via a POST request. | 11:21 |
| *** aswadr_ has quit IRC | 11:22 | |
| kota_ | and I'm now wondering what's happen when a request with x-remove-versions-container to a container which keeps x-history-location. | 11:22 |
| kota_ | it could delete the history location? | 11:22 |
| acoles | kota_: i'm still coming up to speed on the change but was wondering similar about consistency | 11:24 |
| kota_ | and more, if we request a PUT container with x-versions-container and x-remove-history-location, what's happen actually? 400 BadRequest? Intuitively, that's a way to change the versions location setting though. | 11:24 |
| kota_ | acoles: yeah, imho, we should discuss *BEFORE* the previous patch landed to the master, though. | 11:25 |
| kota_ | acoles: mine is just functional tests following the newest master. | 11:25 |
| kota_ | at least, imho, it break container semantics if we can delete the x-versions-location via a PUT/POST request with x-history-location. | 11:27 |
| kota_ | ah, it looks like sort of "a PUT container with x-versions-container and x-remove-history-location" scenario is cared. | 11:33 |
| kota_ | https://github.com/openstack/swift/blob/master/swift/common/middleware/versioned_writes.py#L680-L693 | 11:35 |
| kota_ | interesting, looking at https://github.com/openstack/swift/blob/master/swift/common/middleware/versioned_writes.py#L727-L732 | 11:40 |
| kota_ | it think, it means, a PUT container with x-versions-container and x-remove-history-location can work as if setting x-versions-container. ok. sounds cool | 11:41 |
| kota_ | However, from https://github.com/openstack/swift/blob/master/swift/common/middleware/versioned_writes.py#L734-L743, maybe we can delete x-versions-location via a request with x-remove-history-location | 11:43 |
| kota_ | i need timburke (or tests) for sure. | 11:43 |
| * kota_ is thinking probably i cannot decide what is a better way to improve this by notmyname tag the release. | 11:46 | |
| * kota_ is also thinking i need clean thought/eyes for this becuase it's getting late so it means... | 11:48 | |
| kota_ | it looks like, dev_stack is using older versions write, https://github.com/openstack-dev/devstack/blob/master/lib/swift#L492-L493 :/ probaly that's why my func test ware skipped | 12:06 |
| *** Shadow_aok has quit IRC | 12:06 | |
| kota_ | on the other hand, vagrant-swift-all-in-one seems to follow the versioned_write change https://github.com/swiftstack/vagrant-swift-all-in-one/commit/23379d753d2f5778abfdf3caf3b86bcd3350f065 | 12:06 |
| *** zul has quit IRC | 12:07 | |
| kota_ | nice, clayg. so i can wait the result from swiftstack gate. I think we can run the functests in the swiftstack environment. | 12:07 |
| kota_ | thank you swiftstack. | 12:07 |
| *** catintheroof has quit IRC | 12:08 | |
| *** asettle has quit IRC | 12:13 | |
| kota_ | ok, wrote down some notes for reviewers. finish up today. | 12:15 |
| kota_ | i may be online tomorrow morning to follow if something progress on versioned_writes. | 12:16 |
| kota_ | see you guys. | 12:16 |
| *** zul has joined #openstack-swift | 12:32 | |
| *** amoralej is now known as amoralej|lunch | 12:33 | |
| *** SkyRocknRoll has quit IRC | 12:38 | |
| *** rcernin has quit IRC | 12:38 | |
| *** daemontool has joined #openstack-swift | 12:40 | |
| *** nexusz99 has quit IRC | 12:42 | |
| *** nexusz99 has joined #openstack-swift | 12:42 | |
| *** asettle has joined #openstack-swift | 12:43 | |
| *** pcaruana|afk| has joined #openstack-swift | 12:44 | |
| *** nexusz99 has quit IRC | 12:47 | |
| tdasilva | good morning | 12:48 |
| *** zhengyin has quit IRC | 12:52 | |
| *** StraubTW has joined #openstack-swift | 12:54 | |
| *** ChanServ sets mode: +v tdasilva | 12:55 | |
| *** StraubTW_ has joined #openstack-swift | 12:55 | |
| tdasilva | acoles: i just caught up with the scroll back and noticed you found a bug with versioned writes and DLO manifests | 12:55 |
| *** nikivi has joined #openstack-swift | 12:55 | |
| *** david-lyle has joined #openstack-swift | 12:56 | |
| tdasilva | acoles: in terms of the consistency model, I also had some questions about that, but I think it is ok. Basically x-history-location and x-version-location are mutually exclusive and they get written down with the same sysmeta | 12:56 |
| *** StraubTW has quit IRC | 12:59 | |
| *** nikivi has quit IRC | 12:59 | |
| *** pcaruana|afk| has quit IRC | 13:00 | |
| *** pcaruana has quit IRC | 13:01 | |
| *** pcaruana has joined #openstack-swift | 13:02 | |
| *** rcernin has joined #openstack-swift | 13:06 | |
| acoles | tdasilva: ok, good to know. TBH I'm just catching up with history mode. | 13:08 |
| acoles | tdasilva: re DLOs, not sure if I'd call it a bug or a continuation of the anomaly with DLOs. Creating delete markers for something that was never versioned seems weird though. | 13:12 |
| tdasilva | acoles: agree | 13:13 |
| tdasilva | acoles: would you have a min. to help me understand something the consistency engine | 13:14 |
| tdasilva | acoles: i'm looking at https://github.com/openstack/swift/blob/master/test/probe/test_container_failures.py#L45 | 13:15 |
| acoles | tdasilva: sure, back in a few mins | 13:15 |
| *** vint_bra has joined #openstack-swift | 13:15 | |
| *** nikivi has joined #openstack-swift | 13:19 | |
| *** rcernin has quit IRC | 13:26 | |
| *** zhengyin has joined #openstack-swift | 13:29 | |
| *** amoralej|lunch is now known as amoralej | 13:30 | |
| *** acoles has quit IRC | 13:30 | |
| *** acoles_ has joined #openstack-swift | 13:32 | |
| *** ChanServ sets mode: +v acoles_ | 13:32 | |
| acoles_ | tdasilva: back (bouncer problem) | 13:34 |
| *** nikivi has quit IRC | 13:34 | |
| *** tongli has joined #openstack-swift | 13:36 | |
| *** vint_bra1 has joined #openstack-swift | 13:36 | |
| *** ekarlso_ has joined #openstack-swift | 13:37 | |
| *** vint_bra has quit IRC | 13:39 | |
| *** vinsh has joined #openstack-swift | 13:39 | |
| *** takashi has joined #openstack-swift | 13:40 | |
| *** kragniz has quit IRC | 13:48 | |
| *** kragniz has joined #openstack-swift | 13:48 | |
| *** rcernin has joined #openstack-swift | 13:53 | |
| tdasilva | acoles_: sorry, just saw your msg now | 13:58 |
| *** vinsh has quit IRC | 14:00 | |
| tdasilva | acoles_: i'm trying to make sense of that test I mentioned earlier, and i guess i don't understand why after sending a delete to two primary nodes (one primary was killed), when replication runs the container is still around | 14:00 |
| tdasilva | so is the expectation that because a PUT went in before replication caught up, that sort of invalidates the DELETE? | 14:01 |
| tdasilva | i mean, a PUT for an object | 14:02 |
| acoles_ | tdasilva: i was just reading the test, and guessed that might be your question. I had never studied that behavior before, but I believe the answer is that a container is not considered deleted if it has objects | 14:02 |
| acoles_ | tdasilva: yes, so the object PUT increments the container object counts (even on "deleted" container servers), and then the container is no longer deleted. I just reproduced this myself. | 14:03 |
| tdasilva | acoles_: hmm..very interesting. this sort of throws a wrench at my fix for bug 1560574 | 14:04 |
| openstack | bug 1560574 in OpenStack Object Storage (swift) "GET/HEAD should stop continuing to search more nodes if a tombstone is reached" [Medium,In progress] https://launchpad.net/bugs/1560574 - Assigned to Thiago da Silva (thiagodasilva) | 14:04 |
| acoles_ | tdasilva: but the containers do retain their deleted timestamps, so when the object count drops to zero the container does disappear. | 14:04 |
| tdasilva | heh, even more interesting | 14:04 |
| tdasilva | so in my bug fix, the code change is happening in a very generic place in the proxy. patch 371150 | 14:05 |
| patchbot | https://review.openstack.org/#/c/371150/ - swift - Return 404 on a GET if tombstone is newer | 14:05 |
| tdasilva | so i guess it will need to specialize for objects only | 14:06 |
| *** nexusz99 has joined #openstack-swift | 14:07 | |
| *** nexusz99 has quit IRC | 14:07 | |
| *** daemontool has quit IRC | 14:12 | |
| *** daemontool_ has joined #openstack-swift | 14:12 | |
| *** daemontool__ has joined #openstack-swift | 14:13 | |
| acoles_ | tdasilva: hmmm, looks that way | 14:14 |
| *** daemontool_ has quit IRC | 14:17 | |
| acoles_ | tdasilva: I'm wondering if somewhere around here https://github.com/openstack/swift/blob/44a861787a60346aded8be2a54ef4e16accccbb6/swift/proxy/controllers/base.py#L1203-L1203 you could change the sort to use the source_headers which AFAICT includes the good and bad sources, so sort on timestamp in headers would put tombstones ahead of "good sources" ?? just a thought | 14:19 |
| *** sams-gleb has quit IRC | 14:20 | |
| *** silor has joined #openstack-swift | 14:20 | |
| tdasilva | acoles_: reading... | 14:21 |
| openstackgerrit | Donagh McCabe proposed openstack/swift: Document access control lists (ACLs) https://review.openstack.org/374215 | 14:26 |
| tdasilva | acoles_: was your suggestion for fixing this issue with containers or just a better way of figuring out what the correct response should be, it doesn't solve the issue of making a specialized fix for objects right? | 14:29 |
| *** vinsh has joined #openstack-swift | 14:29 | |
| tdasilva | acoles_: i guess i'm not following your thinking | 14:29 |
| *** vinsh_ has joined #openstack-swift | 14:30 | |
| *** vinsh has quit IRC | 14:30 | |
| acoles_ | tdasilva: i only looked quickly so could be wrong :) but that piece of code is specific to object GETs, so could be a better place to implement an object specific fix for the bug without messing with account/container GETs | 14:36 |
| acoles_ | tdasilva: currently it chooses a good source if there is one, but it looks like it could be made to check for a 404 response that is newer than any good source | 14:37 |
| tdasilva | acoles_: i'm not seeing how that code is specific to object GETs. I'm following the code flow: GETorHEAD_base > get_working_response > _get_source_and_node() | 14:42 |
| tdasilva | acoles_: I could try to make use of server_type like in L1186 | 14:44 |
| *** acoles- has joined #openstack-swift | 14:45 | |
| *** acoles- is now known as acoles | 14:45 | |
| *** ChanServ sets mode: +v acoles | 14:45 | |
| *** asettle has quit IRC | 14:46 | |
| acoles | tdasilva: yes, my mistake, I saw it was in ResumingGetter and some comments about objects so associated it with object GETs | 14:49 |
| tdasilva | acoles: no worries, i'm still looking through... | 14:49 |
| acoles | tdasilva: so yes you could use the server type to switch to some conditional behaviour | 14:50 |
| acoles | or write some server type specific source sorting functions | 14:50 |
| *** acoles_ has left #openstack-swift | 14:51 | |
| *** dmorita has joined #openstack-swift | 15:00 | |
| *** abhitechie has joined #openstack-swift | 15:01 | |
| *** nexusz99 has joined #openstack-swift | 15:01 | |
| acoles | tdasilva: the EC case might need more thought too - there we have multiple ResumingGetters so you may need to hunt for a newer tombstone across the collection of getters | 15:01 |
| *** jistr is now known as jistr|call | 15:03 | |
| tdasilva | acoles: thanks for the heads up. I had not given much thought to EC yet | 15:04 |
| *** dmorita has quit IRC | 15:05 | |
| acoles | tdasilva: np. I get easily confused by those code paths because we end up with parallel getters each of which may perform a sequential search through the nodes until reaching a successful response. | 15:05 |
| acoles | so some unsuccessfuls responses get buried in the getters with just the successful one being exposed to the EC handler | 15:06 |
| *** rvasilets___ has joined #openstack-swift | 15:07 | |
| tdasilva | acoles: by parallel getters, do you mean one for each node thou? | 15:09 |
| tdasilva | i guess not | 15:10 |
| openstackgerrit | Clark Boylan proposed openstack/swift: DNM testing functional testing against ssl https://review.openstack.org/375563 | 15:11 |
| clarkb | clayg: notmyname if ^ works would be good to talk about how you think we should be selecting the protocol in that job. Maybe we can have it look at keystone via openstack's openrc? | 15:12 |
| acoles | tdasilva: one for each fragment. each starts with a unique node from the node iter, but may progress to another node until finding a success response just like replica policy. | 15:12 |
| tdasilva | oh i see | 15:12 |
| tdasilva | right | 15:13 |
| acoles | so each getter may have a good source plus may have seen a 404 along the way | 15:13 |
| tdasilva | acoles: true, but even in that case we want the 404 to bubble up (if it's newer) | 15:13 |
| *** tuan_luong has joined #openstack-swift | 15:14 | |
| tdasilva | so it shouldn't change much in the code... | 15:14 |
| tdasilva | sounds like a EC probe test would be helpful | 15:14 |
| tdasilva | :) | 15:14 |
| acoles | if you change the getter to favor the 404 if it is newer, then the EC object controller may also need some changes so that the 404 from that one getter is favored over success from all the other frag getters. IDK right now if the EC controller would just try to fetch another frag in case of the 404 bubbling up. | 15:16 |
| acoles | tdasilva: fortunately I don't think it would be huge churn because the EC controller groups all responses into buckets, so the 404 would end up in bad_bucket which could be inspected for newest 404 timestamp and compared to time of the good_bucket | 15:18 |
| *** rledisez has quit IRC | 15:18 | |
| acoles | s/good_bucket/best_bucket/ if you are reading the code | 15:19 |
| *** tuan_luong has quit IRC | 15:19 | |
| *** rcernin has quit IRC | 15:19 | |
| tdasilva | acoles: yeah, the question i now have is if you only have let's say a minimun number of frags that are 404 and other frags that are OK and enough to build the object (but are older) should the client get a 404? | 15:19 |
| tdasilva | does the fact that there's any one single fragment with a newer 404 make it so that the entire object should 404? | 15:21 |
| *** tuan_luong has joined #openstack-swift | 15:22 | |
| *** klrmn has joined #openstack-swift | 15:22 | |
| *** _JZ_ has joined #openstack-swift | 15:23 | |
| acoles | as I understand the discussion on the bug report, that would be the desired outcome, i.e. newer tombstone trumps older object | 15:24 |
| *** nexusz99 has quit IRC | 15:25 | |
| tdasilva | but how does quorum come into play? or primary over handoff nodes? sorry...just thinking out loud | 15:25 |
| acoles | the tombstone is going to eventually replace the frags after reconstructor | 15:26 |
| *** jistr|call is now known as jistr | 15:26 | |
| acoles | so the object is destined to be a tombstone | 15:27 |
| *** jistr is now known as jistr|biab | 15:28 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift: Updated from global requirements https://review.openstack.org/88736 | 15:31 |
| *** klrmn has quit IRC | 15:31 | |
| *** jistr|biab is now known as jistr | 15:36 | |
| *** zhengyin has quit IRC | 15:38 | |
| *** chsc has joined #openstack-swift | 15:47 | |
| *** hseipp has quit IRC | 15:55 | |
| *** nexusz99 has joined #openstack-swift | 15:58 | |
| *** nikivi has joined #openstack-swift | 16:01 | |
| *** tqtran has joined #openstack-swift | 16:04 | |
| *** tqtran has quit IRC | 16:08 | |
| *** bkopilov has quit IRC | 16:14 | |
| *** jistr is now known as jistr|afk | 16:18 | |
| *** nexusz99 has quit IRC | 16:21 | |
| *** dmorita has joined #openstack-swift | 16:27 | |
| *** dmorita has quit IRC | 16:29 | |
| *** dmorita has joined #openstack-swift | 16:29 | |
| *** manous has quit IRC | 16:32 | |
| *** jordanP has quit IRC | 16:36 | |
| notmyname | good morning | 16:37 |
| *** Suyi_ has joined #openstack-swift | 16:38 | |
| notmyname | acoles: thanks for the comments. I'll get on that first thing. it shows that I finished writing that up when I was at home, late, and tired ;-) | 16:39 |
| notmyname | sorry for the egregious typos | 16:39 |
| acoles | notmyname: actually I envisioned you typing it while enjoying the in flight beverages ;) | 16:40 |
| notmyname | lol | 16:40 |
| notmyname | but I was stuck back in the cattle car. no nice beverages there ;-) | 16:40 |
| acoles | notmyname: oh yeah, domestic. I'm used to being in cattle, but for longer, so they try to make you numb to it | 16:43 |
| clayg | heyoh! | 16:49 |
| *** nikivi has quit IRC | 16:49 | |
| notmyname | FWIW (since someone asked via pm), no I'm not planning on using reno release notes for this release. tackle that next time | 16:50 |
| clayg | clarkb: auth endpoint is currently all configurable piecemeal swift/test/sample.conf | 16:52 |
| clarkb | clayg: ya just thinking th catalog has that info for us in devstack land. though my test change failed so I may misunderstand how that job is running | 16:53 |
| clayg | clarkb: i'm guessing when that test suite started the options were host and port - then it grew auth protocol and ssl prefix options over the years without anyone ever making it just take a blasted url ;) | 16:53 |
| clayg | clarkb: yeah, i'm curious too - the swift functional tests wouldn't know about keystone endpoint to pull the catalog from without being configured to know about it? | 16:54 |
| clayg | which job are you testing? | 16:54 |
| clarkb | the dsvm functional test | 16:54 |
| *** nexusz99 has joined #openstack-swift | 16:54 | |
| clayg | and those acctually run *our* functest? not... tempest? | 16:55 |
| clarkb | yup | 16:55 |
| clarkb | at least I thought it did. It is running something in swiftnot tempest | 16:55 |
| notmyname | clayg: IIRC those run the in-process functests | 16:55 |
| * notmyname could be wrong | 16:56 | |
| clayg | notmyname: i'm staring at logs - i think it's telling me it's running `.functests` | 16:56 |
| acoles | gate-swift-dsvm-functional-ubuntu-xenial runs swift func tests | 16:57 |
| acoles | against devstack | 16:57 |
| clayg | clarkb: I think trying to trick this file into saying ssl = yes would be the most likely thing to work -> http://logs.openstack.org/63/375563/1/check/gate-swift-dsvm-functional-ubuntu-xenial/82ed5ea/logs/etc/swift/test.conf.txt.gz | 16:58 |
| clayg | although it's weird that it's returning 400 Bad Request? | 16:58 |
| *** jistr|afk is now known as jistr | 16:59 | |
| clayg | idk, i guess the code change would mostly have the same effect | 16:59 |
| clayg | as it turns out I'm currently working on spinning some ssl in my saio - i'll double check functest configuration | 16:59 |
| clarkb | clayg: that would be great. thanks | 17:00 |
| *** zaitcev has joined #openstack-swift | 17:02 | |
| *** ChanServ sets mode: +v zaitcev | 17:02 | |
| *** vint_bra1 has quit IRC | 17:03 | |
| *** klrmn has joined #openstack-swift | 17:05 | |
| *** vint_bra has joined #openstack-swift | 17:05 | |
| *** tuan_luong has quit IRC | 17:08 | |
| *** vint_bra1 has joined #openstack-swift | 17:08 | |
| *** vint_bra has quit IRC | 17:08 | |
| *** vint_bra has joined #openstack-swift | 17:09 | |
| *** catintheroof has joined #openstack-swift | 17:09 | |
| *** adu has joined #openstack-swift | 17:12 | |
| *** vint_bra1 has quit IRC | 17:13 | |
| *** nexusz99 has quit IRC | 17:17 | |
| *** geaaru has quit IRC | 17:17 | |
| openstackgerrit | Alistair Coles proposed openstack/swift: Make in-process mode not skip history versions functional tests https://review.openstack.org/375633 | 17:18 |
| openstackgerrit | John Dickinson proposed openstack/swift: authors/changelog updates for 2.10.0 https://review.openstack.org/375235 | 17:21 |
| notmyname | acoles: ^ | 17:21 |
| *** daemontool__ has quit IRC | 17:21 | |
| acoles | notmyname: got it | 17:22 |
| acoles | timburke: why does history mode versioning write delete markers for deletes that 404? | 17:23 |
| acoles | notmyname: timburke I just reviewed Kota's patch 360933 which he was keen to have in the release, but the new tests are skipped in jenkins jobs. patch 375633 fixes the skips in at least one voting job. | 17:24 |
| patchbot | https://review.openstack.org/#/c/360933/ - swift - Add functional tests for new versioned_write mode | 17:24 |
| patchbot | https://review.openstack.org/#/c/375633/ - swift - Make in-process mode not skip history versions fun... | 17:24 |
| acoles | your call as to whether this gets in or not, needs more +2's of course | 17:25 |
| timburke | acoles: thanks; i'll take a look at them today | 17:25 |
| acoles | timburke: thanks, and it goes without saying that patch 375633 could just be squashed into kota_'s | 17:27 |
| patchbot | https://review.openstack.org/#/c/375633/ - swift - Make in-process mode not skip history versions fun... | 17:27 |
| *** rvasilets___ has quit IRC | 17:28 | |
| acoles | hehe ^^ nice truncation, who'd want to skip the fun? | 17:31 |
| timburke | re: deletes that 404 -- seemed like the most-sane thing was to go ahead and record the delete. fwiw, i still don't understand why we let a delete 404 at all (when the container exists)... it's led to https://bugs.launchpad.net/python-swiftclient/+bug/1213179 and https://twitter.com/wolever/status/697234620966895616 and https://review.openstack.org/#/c/368264/ | 17:33 |
| openstack | Launchpad bug 1213179 in python-swiftclient "swiftclient logs all ClientException (404) at ERROR (with traceback)" [Critical,Confirmed] - Assigned to Bartek Żurawski (bartekzurawski1) | 17:33 |
| patchbot | patch 368264 - swift3 - Fix the deletion of non-existent keys | 17:33 |
| timburke | also, it'll simplify my patch to add versioning support to swift3 :-) | 17:34 |
| acoles | timburke: but if nothing got deleted (and copied to versions) what is the delete marker used for - feels like we're using an inode to log a request? | 17:45 |
| acoles | whether the status code of delete is 404 or not is orthogonal, or am I missing something? | 17:46 |
| timburke | i feel like that's rather the point of history mode -- to have an accurate history of the requests made. and i feel like this combined with symlink-based versioning will ultimately help us Do The Right Thing during failures | 17:48 |
| *** nexusz99 has joined #openstack-swift | 17:50 | |
| *** tqtran has joined #openstack-swift | 17:50 | |
| *** amoralej is now known as amoralej|off | 17:52 | |
| notmyname | my gut reaction is that we don't need to rush to have "just" tests added into a release | 17:53 |
| notmyname | yes, they're important, and yes they should pass, and yes we should land them asap | 17:53 |
| notmyname | acoles: timburke: kota_: ^ | 17:53 |
| notmyname | but I'm not sure that means they need to land now. the reason for getting them in a release would be to use them to test backports that may be developed in the future | 17:54 |
| *** silor has quit IRC | 17:54 | |
| *** nexusz99 has quit IRC | 17:55 | |
| *** psachin has joined #openstack-swift | 17:55 | |
| timburke | notmyname: and i'm not sure this is a good candidate for back-ports. particularly as the feature is already spread across multiple commits | 17:59 |
| notmyname | timburke: no, I wasn't meaning to backport this (although we could). I mean that the only reason to land the tests *right now* to be in the release is so that in 6 months when we do have asome other backport to newton, these tests are run | 18:00 |
| notmyname | and TBH I'm not sure how much value that is | 18:01 |
| timburke | ah, gotcha | 18:01 |
| clayg | timburke: I don't think the bugs about swiftclient's bad logging behavior have anything to do with the API difference between S3 and Swift WRT DELETE requests for keys that don't exist | 18:01 |
| notmyname | point is, I don't think we need to rush to get this (or these) test-only patches in to this release | 18:01 |
| *** abhitechie has quit IRC | 18:01 | |
| clayg | notmyname: is this the *release* release - or just the rc? | 18:01 |
| *** arch-nemesis has joined #openstack-swift | 18:02 | |
| notmyname | clayg: very very likely to be the real, actual release | 18:02 |
| timburke | clayg: it makes it decidedly worse when a user is coming from a system where a DELETE should always 204 (like S3) | 18:02 |
| notmyname | if there's something major, we might be able to land it next week, but I think that could have ripple effects for other projects | 18:02 |
| clayg | notmyname: interesting... because lp bug #1624088 doesn't seem to be going anywhere :\ | 18:03 |
| openstack | Launchpad bug 1624088 in OpenStack Object Storage (swift) "EC missing durable can prevent reconstruction" [Critical,Confirmed] https://launchpad.net/bugs/1624088 | 18:03 |
| notmyname | clayg: ...yeah... :-( | 18:04 |
| *** mariusv has quit IRC | 18:04 | |
| notmyname | clayg: that would be a good candidate for a backport | 18:04 |
| clayg | notmyname: yup | 18:04 |
| *** mariusv has joined #openstack-swift | 18:04 | |
| notmyname | clayg: if this were any other swift release than the one to be included in a time-based openstack integrated release, then I'd hold off on the swift release to include that | 18:05 |
| clayg | notmyname: but idk, we stink at backports -> https://review.openstack.org/#/c/363855/ | 18:05 |
| patchbot | patch 363855 - swift (stable/mitaka) - Stop complaining about auditor_status files | 18:05 |
| clayg | timburke: you have another one too right? | 18:05 |
| acoles | timburke: hmmm, I'm going to have to ponder that some more (delete history) to grok it | 18:05 |
| tdasilva | notmyname: is the deadline for this week just for an RC or for the offical release? | 18:05 |
| acoles | notmyname: agree re tests | 18:05 |
| clayg | tdasilva: notmyname says it's probably the real deal | 18:06 |
| tdasilva | clayg: i mean, the deadline from the "openstack release schedule" | 18:06 |
| clayg | tdasilva: i'm sure notmyname could explain it better than me | 18:07 |
| tdasilva | https://releases.openstack.org/newton/schedule.html | 18:07 |
| notmyname | the final final openstack release is next thursday | 18:07 |
| timburke | clayg: do i? i guess https://review.openstack.org/#/c/347538/ would be nice, but it's certainly not critical | 18:07 |
| patchbot | patch 347538 - swift - Store SLO Etag and size in sysmeta | 18:07 |
| *** raba has quit IRC | 18:08 | |
| tdasilva | notmyname: not the week after? | 18:08 |
| clayg | timburke: i ment backports | 18:08 |
| acoles | good night | 18:08 |
| notmyname | tdasilva: our deadline is thursday. I think the overall release is the next week | 18:08 |
| *** acoles is now known as acoles_ | 18:09 | |
| notmyname | so the question is do we want to hold of on tagging to get a patch for bug 1624088 to land before thursday, or do we tag now and backport when that bugfix is available | 18:10 |
| openstack | bug 1624088 in OpenStack Object Storage (swift) "EC missing durable can prevent reconstruction" [Critical,Confirmed] https://launchpad.net/bugs/1624088 | 18:10 |
| notmyname | I'd vote for tag now and backport | 18:11 |
| *** psachin has quit IRC | 18:11 | |
| notmyname | I suppose a third option is tag now and also tag next week if a patch lands | 18:11 |
| notmyname | timburke: clayg: tdasilva: what do you think? | 18:12 |
| *** silor has joined #openstack-swift | 18:12 | |
| tdasilva | notmyname: can we tag rc now and if it lands by next thu then issue a new RC | 18:12 |
| timburke | clayg: oh yeah, those. patch 363855 and patch 362514. it *would* probably be nice to do them while we're still within 6 mos of stable/mitaka being cut -- not sure they really count as "critical" per http://docs.openstack.org/project-team-guide/stable-branches.html | 18:12 |
| patchbot | https://review.openstack.org/#/c/363855/ - swift (stable/mitaka) - Stop complaining about auditor_status files | 18:12 |
| tdasilva | if not, call the current RC *the* release | 18:12 |
| patchbot | https://review.openstack.org/#/c/362514/ - swift (stable/mitaka) - Ignore auditor status files to prevent replicator ... | 18:12 |
| notmyname | tdasilva: yes | 18:13 |
| tdasilva | notmyname: what's your concern with that plan thou? what's the drawback... you've had it set for a while now that you wanted to release today, so you probably have more info on how the whole schedule works than I might realize... | 18:15 |
| clayg | timburke: I don't think I'd read those guidelines before - thanks! | 18:17 |
| *** nikivi has joined #openstack-swift | 18:18 | |
| *** silor1 has joined #openstack-swift | 18:18 | |
| notmyname | tdasilva: ah, I got some clarification | 18:19 |
| timburke | so, notmyname, since i think you're the only one with the right bits in gerrit, you might want to look at them real quick ;-) | 18:19 |
| notmyname | I'd been pushing for today as a deadline because (1) it's a deadline that we can work towards and gives us a tiny bit of room on the other side and (2) I kept getting bugged about "when will swift release be. other people have already tagged" | 18:20 |
| notmyname | however, it doesn't seem like there will be any cross-project impact if we tag today and next week | 18:20 |
| notmyname | timburke: the stable guidelines? | 18:20 |
| *** silor has quit IRC | 18:22 | |
| *** silor1 is now known as silor | 18:22 | |
| *** nikivi has quit IRC | 18:23 | |
| *** adu has quit IRC | 18:25 | |
| notmyname | clayg: what are your thoughts on tagging vs getting patch for bug 1624088 in the next few days? | 18:25 |
| openstack | bug 1624088 in OpenStack Object Storage (swift) "EC missing durable can prevent reconstruction" [Critical,Confirmed] https://launchpad.net/bugs/1624088 | 18:25 |
| *** adu has joined #openstack-swift | 18:26 | |
| *** silor has quit IRC | 18:26 | |
| *** nikivi has joined #openstack-swift | 18:27 | |
| timburke | notmyname: the (my) two outstanding patches for mitaka | 18:29 |
| notmyname | timburke: right. on my list :-) | 18:29 |
| *** nikivi has quit IRC | 18:33 | |
| *** silor has joined #openstack-swift | 18:33 | |
| *** dmsimard has left #openstack-swift | 18:33 | |
| openstackgerrit | Thiago da Silva proposed openstack/swift: Return 404 on a GET if tombstone is newer https://review.openstack.org/371150 | 18:45 |
| *** arch-nemesis has quit IRC | 18:50 | |
| clayg | notmyname: i don't know - we could just update the reconstrcutor to send "x-backend-frag-prefs: '[]'" or some bs like that and the probetest will pass | 19:00 |
| clayg | notmyname: I guess I *sorta* think it'd be better to land that than ship with the known critical bug - but I also sorta think there's like some other (maybe less bad?) bug hiding in ust doing that | 19:01 |
| clayg | or at least acoles seemed to convince me at one time that just always returns non-durable data isn't the best solution either | 19:01 |
| clayg | something like you can have more than parity but less than data non-durable frags and then no one is happy | 19:02 |
| clayg | but I don't know if I've ever seen non-durable data in the cluster that wasn't reconstructable (i haven't proved that it's even possible; but I think it could happen - it's just a really specific packet has to fail to a bunch of nodes) | 19:04 |
| clayg | idk | 19:04 |
| clayg | idk | 19:04 |
| clayg | idk | 19:04 |
| *** arch-nemesis has joined #openstack-swift | 19:06 | |
| *** david-lyle_ has joined #openstack-swift | 19:10 | |
| *** david-lyle has quit IRC | 19:13 | |
| *** catintheroof has quit IRC | 19:17 | |
| *** silor has quit IRC | 19:28 | |
| *** nexusz99 has joined #openstack-swift | 19:38 | |
| timburke | clayg: oh sure, could totally happen. could be that the proxy just keels over for unrelated reasons between sending out all the frags and sending the commit | 19:40 |
| timburke | what i'm wondering though, is what the .durable really *tells* us. it means that on an isolated node, we know that there are (should be, i suppose) enough fragments on other nodes to reconstruct the original object | 19:42 |
| *** nexusz99 has quit IRC | 19:43 | |
| timburke | but if we're contacting the other nodes anyway...why don't we just trust what they tell us? i think you're right that there'd be a bug in *just* sending the extra header and trying to reconstruct whatever comes back, though. i think the reconstructor will need the same sort of "well, ok, what *can* i return?" smarts that the proxy got | 19:43 |
| clayg | wow, running functests with insecure = true is tons of fun (where fun ~= warnings about insecure ssl certs - because there's tons of them!) | 19:45 |
| *** McMurlock1 has joined #openstack-swift | 19:46 | |
| *** nadeem has joined #openstack-swift | 19:47 | |
| clayg | timburke: nope, if the proxy keels over at just the right moment - you could end up with a bunch of .data files that aren't durable - it's really hard to end up with half your nodes having .data files the other half not and no durable - it requires a transmission failure on just the right packet to more than parity nodes - then the proxy would timeout waiting on the 100-continue to send the durable | 19:54 |
| clayg | I'm mainly thinking that scenario is less likely than failure to rebuild because not enough durables encountering missing frags enough to trap the durable - but i suppose that's acctually pretty unlikely too | 19:56 |
| clayg | i don't really like the x-backend-frags-prefs api - it's too much! the matrix of what could possibly be sent and what could possibly be responded with is too huge. | 19:57 |
| clayg | I don't even think it represents the data model all that well - while the storage model can represent lots of different timestamps and frag indexes of various durability the system doesn't really exercise that much richness in practice | 19:58 |
| *** McMurlock1 has quit IRC | 19:58 | |
| clayg | it's mostly just the one primary frag index and either the current timestamp and durable or also another timestamp on top (that is probably totally reconstructable) | 19:58 |
| clayg | so I'm not sure how much "copy what the proxy does into the reconstructor" is really going to buy us :'( | 19:59 |
| *** nadeem has quit IRC | 20:03 | |
| *** nadeem has joined #openstack-swift | 20:04 | |
| *** david-lyle_ is now known as david-lyle | 20:06 | |
| notmyname | timburke: I just discovered a new trick! | 20:08 |
| openstackgerrit | John Dickinson proposed openstack/python-swiftclient: add pypy to the bindep "test" profile https://review.openstack.org/375697 | 20:09 |
| notmyname | timburke: that ^ | 20:09 |
| notmyname | clayg: timburke: so I'm thinking that I should go ahead and tag, then, and if there's a patch to fix that next week we can land it and have 2.10.1. otherwise, backport it as it's available | 20:12 |
| clayg | how many headers do we allow? 1024? | 20:21 |
| openstackgerrit | John Dickinson proposed openstack/python-swiftclient: add pypy to the bindep "test" profile https://review.openstack.org/375697 | 20:22 |
| notmyname | clayg: 90 | 20:22 |
| notmyname | clayg: oh, headers, not metadata keys | 20:22 |
| notmyname | ummm.. | 20:22 |
| notmyname | dunno | 20:23 |
| tdasilva | isn't it like 13something? | 20:26 |
| clayg | tdasilva: 90 + 30 + extra_headers | 20:27 |
| clayg | no... 90 + 36 + extra_headers | 20:28 |
| mattoliverau | So we need to do that releasenotes/notes yaml thing in the swift repo like we did for the swiftclient repo to get release notes/change log on that OS website? | 20:29 |
| clayg | notmyname: I bumped down the severity of lp bug #1624088 | 20:29 |
| openstack | Launchpad bug 1624088 in OpenStack Object Storage (swift) "EC missing durable can prevent reconstruction" [High,Confirmed] https://launchpad.net/bugs/1624088 | 20:29 |
| notmyname | mattoliverau: yeah, someone else asked about that. are we using reno for this one? my initial reaction is to wait, but it's likely not too hard to push it in | 20:30 |
| notmyname | clayg: ack. thanks | 20:30 |
| mattoliverau | Kk | 20:30 |
| notmyname | clayg: keeping our perfect record of not releasing with an open critical bug ;-) | 20:30 |
| tdasilva | that we know of | 20:31 |
| tdasilva | ;) | 20:31 |
| clayg | notmyname: it's more about admiting truthfully that we have a bug - it's bad - but it's not worth a) dopping everythig we're doing until it's fixed or b) blocking the release | 20:32 |
| notmyname | clayg: right. I'm just joking :-) | 20:32 |
| clayg | so... it's by defintion not critical - but it's high - so hopefully we can figure out either a) who's thinking about it looking at it or b) if it should be medium | 20:33 |
| notmyname | at this rate, we'll get it down to "wishlist" pretty soon | 20:33 |
| *** Jeffrey4l_ has joined #openstack-swift | 20:35 | |
| *** Jeffrey4l has quit IRC | 20:35 | |
| *** adu has quit IRC | 20:36 | |
| *** tongli has quit IRC | 20:37 | |
| clayg | does anyone know how to call .functests and get failfast behavior? ostestr says --until-failure but that gives some other bs error | 20:39 |
| openstackgerrit | John Dickinson proposed openstack/swift: authors/changelog updates for 2.10.0 https://review.openstack.org/375235 | 20:43 |
| notmyname | mattoliverau: tdasilva: reno-ified ^ | 20:43 |
| clayg | whoa reno! | 20:49 |
| timburke | notmyname: oh, one question on https://review.openstack.org/#/c/375697/ -- do we know which profile infra is using for the gate? is that documented somewhere? | 20:50 |
| patchbot | patch 375697 - python-swiftclient - add pypy to the bindep "test" profile | 20:50 |
| timburke | or do they just use the tox env (assuming one exists) / default profile (if it doesn't)? | 20:51 |
| notmyname | timburke: clarkb told me it would work (in -infra) and no it doesn't use the tox env in the gate. but I couldn't find the definition of the job that is calling bindep in the project-config repo | 20:52 |
| notmyname | timburke: yes! I found it | 20:54 |
| notmyname | openstack_project_config/jenkins/scripts/install-distro-packages.sh:48 | 20:54 |
| notmyname | I think. let me linkify that for you | 20:54 |
| notmyname | https://github.com/openstack-infra/project-config/blob/master/jenkins/scripts/install-distro-packages.sh#L44-L52 | 20:55 |
| timburke | yep, got it. although really, the bit we're interested in is up at https://github.com/openstack-infra/project-config/blob/master/jenkins/scripts/install-distro-packages.sh#L33 | 20:56 |
| *** nikivi has joined #openstack-swift | 21:12 | |
| clayg | clarkb: k, i got functests passing against ssl omm - the only annoything was using the insecure self signed cert | 21:12 |
| clarkb | clayg: ok, any thoughts on how we should be able to selectively do that depending on whether or not the env has ssl configured? (thinking with devstack we have the catalog which may be useful) | 21:13 |
| clayg | well that's not 100% the *other* annoying part was configuring haproxy to not be so pedantic about http headers | 21:15 |
| clayg | clarkb: well isn't access to the devstack catalog (in keystone right?) also conditional on the ssl'd ness? | 21:16 |
| clayg | clarkb: I mean *really* the functests are going to use what ever protocol is returned from as the storage url in the catalog - the "configuration" of the swift func test suite is "how do you want me to talk to auth to get your storage url" | 21:17 |
| clarkb | gotcha | 21:17 |
| clayg | clarkb: presumably it'd be tons easier if you could give the func tests test.conf a url instead of half a dozen different params for host port ssl insecure etc | 21:17 |
| clarkb | ya | 21:18 |
| clarkb | and maybe a CA cert to use for trusting things | 21:18 |
| clayg | yah, some of tests use httplib - we'd have to look at how urllib3 does it's thing where it creates the ssl context and somehow plubms thought that cert option from requests | 21:18 |
| clayg | clarkb: anyway - let me know if you have something you need me to look at | 21:22 |
| *** StraubTW_ has quit IRC | 21:22 | |
| clarkb | clayg: I don't really have anything specific. Am just trying to make a push towards more ssl by default in our testing | 21:22 |
| clarkb | clayg: so if you end up making those changes to swift I can circle back around and teach the job to use them | 21:23 |
| clarkb | otherwise its probably a matter of me teaching it to use the existing set of flags | 21:23 |
| clayg | well - i have a diff going on supression the warnings when you turn on insecure - and to make insecure work with newer ssl's - i'll add the url thing - but I'm not that hot-to-trot figuring out how to plumb the cacert thing - dunno if it'll help you in your endeavours or not (you'll still have to detect ssl auth endpoint and change the test.conf) | 21:25 |
| clarkb | that will help a bit, just have to set insecure potentially. | 21:26 |
| clarkb | then maybe later switch to setting the cert to trust | 21:26 |
| *** nexusz99 has joined #openstack-swift | 21:26 | |
| clarkb | this is likely not the most important place to be testing ssl, but ideally we would have it enabled in devstack by default so if the jobs that used devstack just worked with that would be great | 21:27 |
| clarkb | but its looking like we will have to do this piecemeal regardless so likely not a rush | 21:28 |
| clayg | clarkb: what would be the url you'd normally put in? http://keystone:35357/auth/v3.0 or something? do you have an e.g. | 21:29 |
| clarkb | clayg: looks like https://127.0.0.1/identity | 21:30 |
| clayg | mainly questioning if you prefix or version are normally included in the url - yah k | 21:30 |
| clarkb | and then for admin things https://127.0.0.1/identity_v2_admin | 21:31 |
| clarkb | (I do not know why admin uses v2 explicitly) | 21:31 |
| *** thumpba has joined #openstack-swift | 21:31 | |
| *** dmorita has quit IRC | 21:36 | |
| *** dmorita has joined #openstack-swift | 21:36 | |
| *** natarej has joined #openstack-swift | 21:43 | |
| *** nadeem has quit IRC | 21:45 | |
| *** thumpba has quit IRC | 21:46 | |
| *** nexusz99 has quit IRC | 21:50 | |
| *** vint_bra has quit IRC | 22:06 | |
| *** nadeem has joined #openstack-swift | 22:07 | |
| clayg | awhat on *earth* could this comment mean? https://github.com/openstack/swift/blob/cb33660848e8576ff46334e7976fedd5fca972db/test/sample.conf#L9 | 22:10 |
| clayg | how does swift work with keystone if allow account managemnet is not true? | 22:10 |
| clayg | and why would you phrase it as a should not? | 22:10 |
| clayg | so lp bug #1530254 makes it sound like allow_account_management should be false - but I'm guessing that only works because the account was already created (like tests ran twice?) | 22:13 |
| openstack | Launchpad bug 1530254 in OpenStack Object Storage (swift) "test.functional.tests.TestAccount testPUT fails when using keystone auth" [Undecided,Fix released] https://launchpad.net/bugs/1530254 - Assigned to Charles Hsu (charles0126) | 22:13 |
| clayg | i would imagine on a clean cluster w/o allow_account_management the proxy app would reject the autocreate PUT account with 405 and no tests would fail unless the account was already created | 22:14 |
| clayg | ... meanwhile I'm guessing the user being used for the test was a reselleradmin? so instead of 403 on PUT account they were getting the 202? | 22:14 |
| clayg | sigh, i gotta finish getting my devstack box up to snuff :'( | 22:15 |
| openstackgerrit | Merged openstack/swift: authors/changelog updates for 2.10.0 https://review.openstack.org/375235 | 22:30 |
| notmyname | yay. submitting that to be tagged at 2.10.0 | 22:30 |
| zaitcev | Guys, who is running swift3 in OpenStack nowadays? I see timburke and kota_ were committing recently. My problem is that we don't have any tarballs in https://tarballs.openstack.org/swift3/ . | 22:30 |
| notmyname | FYI https://review.openstack.org/#/c/375755/ | 22:32 |
| patchbot | patch 375755 - releases - swift 2.10.0 release (newton) | 22:32 |
| clayg | zaitcev: a lot of our deploymens turn on swift3 for some app or another - timburke is definately the point man from our side when issues come up there? | 22:33 |
| clayg | zaitcev: when i talk with cschwede he seems to also have folks doing swift3? | 22:33 |
| zaitcev | clayg: I meant the OpenStack infrastructure, sorry. You know, who do I ask to make tarballs cut and uploaded. | 22:36 |
| clarkb | zaitcev: you need to make sure the job is on the project then it happens automagically | 22:36 |
| timburke | i'm not even clear on how kota_ adds a tag right now :-/ the closest thing i'm familiar with is how notmyname now has to push patches to openstack/releases now for swift & swiftclient | 22:38 |
| kota_ | good morning | 22:39 |
| openstackgerrit | Clay Gerrard proposed openstack/swift: Better SSL support in functests https://review.openstack.org/375759 | 22:41 |
| kota_ | zaitcev, timburke: i'm following http://docs.openstack.org/infra/manual/drivers.html | 22:42 |
| kota_ | for swift3 release, it might be older than the newest way for official project | 22:43 |
| *** cdelatte has joined #openstack-swift | 22:43 | |
| zaitcev | So, all we have is "After a tag is created the release build will generate a source code tarball and may publish it to a repository such as PyPI" | 22:44 |
| timburke | kota_: oh, nice! then yeah, we should totally go put the tarball thing in some yaml file | 22:44 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift: Updated from global requirements https://review.openstack.org/88736 | 22:46 |
| timburke | zaitcev: i'd expect that if we had openstack-publish-jobs in https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/projects.yaml then it *should* Just Work for new tags | 22:46 |
| kota_ | timburke: in project config? yeah, i saw similar thing in pyeclib but didn't on swift3 yet | 22:46 |
| timburke | kota_: http://tarballs.openstack.org/pyeclib/ still just has master, though. pypi-jobs might still be nice to have, though -- once https://pypi.python.org/pypi/swift3 actually exists | 22:48 |
| zaitcev | Somehow '{name}-tarball' job is missing from swift proper, yet tarballs get created. | 22:49 |
| kota_ | ok, that's different from tarball, do you know which job is for tarball site? | 22:49 |
| zaitcev | maybe openstack-publish-jobs is what's needed | 22:51 |
| kota_ | it seems 'opensstack-server-release' or 'openstack-publish' | 22:51 |
| kota_ | zaitcev: yeah, thanks | 22:52 |
| kota_ | let me make sure the script | 22:52 |
| *** _JZ_ has quit IRC | 22:58 | |
| kota_ | zaitcev, timburke: it looks like openstack-server-release-jobs is what we're looking for? https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/python-jobs.yaml#L693-L697 | 23:00 |
| *** nikivi has quit IRC | 23:01 | |
| kota_ | zaitcev: that includes {name}-tarball job | 23:01 |
| kota_ | i think, the reason why the tarball site keeps only master branch is python-jobs has only {name}-branch-tarball job, https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/python-jobs.yaml#L637 | 23:02 |
| zaitcev | You're in a maze of little jobs, all alike. | 23:04 |
| kota_ | exactly, complicated | 23:05 |
| *** nadeem has quit IRC | 23:05 | |
| *** chsc has quit IRC | 23:05 | |
| *** dmorita has quit IRC | 23:07 | |
| *** dmorita has joined #openstack-swift | 23:08 | |
| *** dmorita has quit IRC | 23:12 | |
| *** _JZ_ has joined #openstack-swift | 23:15 | |
| kota_ | ah, wait, swift3 has {name}-tarball job??? | 23:16 |
| *** nexusz99 has joined #openstack-swift | 23:16 | |
| clayg | lol | 23:17 |
| *** dmorita has joined #openstack-swift | 23:18 | |
| *** nexusz99 has quit IRC | 23:21 | |
| *** _JZ_ has quit IRC | 23:26 | |
| kota_ | hmm... I need more deep dive or asking for openstack infrat team later, no idea right now. | 23:32 |
| kota_ | looking at scroll back, notmyname seems to like tag now and if something happens in the next week, it may tag again, anyway, the functests for versioned writes is now not rushed. | 23:44 |
| kota_ | no time in this morning, will look at this night or early in the next week. | 23:44 |
| kota_ | acoles: thanks for reviewing and the follow-up for in-process testing looks awesome! | 23:45 |
| *** dmorita has quit IRC | 23:54 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!