| openstackgerrit | Merged openstack/swift master: replication: Allow databases_per_second to be a float https://review.opendev.org/728571 | 01:28 |
|---|---|---|
| *** gyee has quit IRC | 01:58 | |
| *** rcernin has quit IRC | 02:12 | |
| *** mattia has quit IRC | 03:11 | |
| *** rcernin has joined #openstack-swift | 03:17 | |
| openstackgerrit | Merged openstack/swift master: py3: Fix expirer container generation https://review.opendev.org/735262 | 03:20 |
| openstackgerrit | Merged openstack/swift master: py3: (Better) fix percentages in configs https://review.opendev.org/734721 | 03:20 |
| *** psachin has joined #openstack-swift | 03:29 | |
| *** manuvakery has joined #openstack-swift | 04:14 | |
| *** djhankb9 has joined #openstack-swift | 04:32 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** djhankb has quit IRC | 04:33 | |
| *** djhankb9 is now known as djhankb | 04:33 | |
| *** evrardjp has joined #openstack-swift | 04:33 | |
| *** zaitcev has quit IRC | 05:13 | |
| *** timss has quit IRC | 05:36 | |
| *** timss has joined #openstack-swift | 05:36 | |
| *** rpittau|afk is now known as rpittau | 06:44 | |
| *** rcernin has quit IRC | 07:59 | |
| openstackgerrit | Merged openstack/swift master: proxy: Stop killing memcache entries on 5xx responses https://review.opendev.org/735359 | 08:07 |
| openstackgerrit | Merged openstack/swift stable/ussuri: Fix stable gate https://review.opendev.org/736829 | 08:11 |
| openstackgerrit | Merged openstack/swift master: s3api: Add basic support for ?tagging requests https://review.opendev.org/735173 | 08:11 |
| openstackgerrit | Merged openstack/swift feature/losf: Merge remote-tracking branch 'gerrit/master' into feature/losf https://review.opendev.org/735381 | 08:11 |
| *** manuvakery has quit IRC | 08:45 | |
| *** mugsie has quit IRC | 08:48 | |
| *** mugsie has joined #openstack-swift | 08:48 | |
| *** rcernin has joined #openstack-swift | 08:54 | |
| *** rcernin has quit IRC | 09:01 | |
| *** tkajinam has quit IRC | 09:21 | |
| *** rcernin has joined #openstack-swift | 09:23 | |
| *** rcernin has quit IRC | 09:58 | |
| *** rpittau is now known as rpittau|bbl | 10:22 | |
| *** rcernin has joined #openstack-swift | 10:54 | |
| *** rcernin has quit IRC | 11:08 | |
| *** rcernin has joined #openstack-swift | 11:53 | |
| *** tkajinam has joined #openstack-swift | 12:04 | |
| *** rcernin has quit IRC | 12:06 | |
| *** cschwede has joined #openstack-swift | 12:06 | |
| *** ChanServ sets mode: +v cschwede | 12:06 | |
| *** rpittau|bbl is now known as rpittau | 12:19 | |
| *** manuvakery has joined #openstack-swift | 12:32 | |
| *** rpittau is now known as rpittau|brb | 13:54 | |
| *** rpittau|brb is now known as rpittau|afk | 14:00 | |
| *** psachin has quit IRC | 14:01 | |
| *** tkajinam has quit IRC | 14:13 | |
| *** ccamacho has quit IRC | 14:48 | |
| clayg | timburke: k, the complete & abort are still working going on like idk 19 hours? | 14:58 |
| *** cschwede has quit IRC | 15:32 | |
| *** timss has quit IRC | 15:32 | |
| *** djhankb has quit IRC | 15:32 | |
| *** jv has quit IRC | 15:32 | |
| *** godog has quit IRC | 15:32 | |
| *** cschwede has joined #openstack-swift | 15:32 | |
| *** timss has joined #openstack-swift | 15:32 | |
| *** djhankb has joined #openstack-swift | 15:32 | |
| *** jv has joined #openstack-swift | 15:32 | |
| *** godog has joined #openstack-swift | 15:32 | |
| *** tepper.freenode.net sets mode: +v cschwede | 15:32 | |
| timburke | clayg, nice! sounds like i oughta just take the time limit out, then. thanks for testing it! | 15:41 |
| clayg | well, it definitely stopped in one case at >24 hours - and started erroring with NoSuchUpload | 15:41 |
| clayg | I think it might be near exactly 24 hours - but I haven't experimentally confirmed that yet | 15:42 |
| clayg | we'll know by the end of today - but I think leaving the code as is with a constraint that's at or around 24hours is going to be *pretty* close to what we've observed from aws | 15:42 |
| clayg | and also extending the grade to abort - which is still maybe kinda weird - but I'm observing the 204 on completed uploads | 15:43 |
| clayg | s/grade/grace period/ | 15:43 |
| timburke | yeah, abort-on-failed-complete deserves its own patch, i think | 15:47 |
| *** zaitcev has joined #openstack-swift | 15:53 | |
| *** ChanServ sets mode: +v zaitcev | 15:53 | |
| *** manuvakery has quit IRC | 16:42 | |
| *** ChanServ has quit IRC | 16:59 | |
| *** gmann is now known as gmann_afk | 17:13 | |
| *** ChanServ has joined #openstack-swift | 17:37 | |
| *** tepper.freenode.net sets mode: +o ChanServ | 17:37 | |
| clayg | if i'm an operator configuring per-policy settings in my [proxy-server:policy:N] settings and for some reason I don't want to specify a particular option (say, concurrency_timeout)... | 17:51 |
| clayg | Am I expecting that this policy will get the default value for when the option is not specified (i.e. don't configure this value AT ALL) - or that it falls through and uses the value I configured in the [app:proxy-server] section (i.e. don't override this policies configured value) | 17:53 |
| clayg | I guess it kinda already uses proxy section configured value, so hopefully that's what everyone expects (that's what *I* expected) but I think we can do that a little more DRY | 18:05 |
| openstackgerrit | Tim Burke proposed openstack/swift master: swift-container-info: Don't show all shard ranges when there are many https://review.opendev.org/737056 | 18:12 |
| clayg | or maybe not, maybe i just didn't understand how it worked already 😁 | 18:15 |
| timburke | yeah, i'd also expect it to fall through to the [app:proxy-server] setting, and only go to the default we have in code if it's not configured there | 18:15 |
| timburke | i feel like option overlays like that are fairly common and expected | 18:17 |
| timburke | though there might be some unexpected edge cases if you put a value in [DEFAULT] then skip it in the per-policy section 🤔 | 18:18 |
| zaitcev | timburke: sorry to poke you about it but could you look at https://review.opendev.org/#/c/704435/ | 18:24 |
| patchbot | patch 704435 - swift - Mark a container reported if account was reclaimed - 1 patch set | 18:24 |
| zaitcev | timburke: clayg is nose down in EC waterfall, mattoliverau is in sharder, and Romain and I disagree. So... | 18:25 |
| clayg | timburke: I think paste's loadapp will populate the base conf with DEFAULT values by the time we're building the settings | 18:26 |
| *** cschwede has quit IRC | 18:41 | |
| zaitcev | itertools.islice() why do I want to know if something is a lice | 19:12 |
| openstackgerrit | Tim Burke proposed openstack/swift master: staticweb: Work better with double slashes https://review.opendev.org/737069 | 19:38 |
| openstackgerrit | Tim Burke proposed openstack/swift stable/ussuri: py3: (Better) fix percentages in configs https://review.opendev.org/737070 | 19:55 |
| openstackgerrit | Tim Burke proposed openstack/swift stable/train: py3: (Better) fix percentages in configs https://review.opendev.org/737071 | 20:09 |
| openstackgerrit | Tim Burke proposed openstack/swift master: swift-container-info: Don't show all shard ranges when there are many https://review.opendev.org/737056 | 20:12 |
| *** gmann_afk is now known as gmann | 20:15 | |
| timburke | zaitcev, rledisez: i'm going to poke at a probe test for p 704435 -- idk that i can make heads or tails of it without seeing how things behave when we've reaped in a not-settled state | 20:21 |
| patchbot | https://review.opendev.org/#/c/704435/ - swift - Mark a container reported if account was reclaimed - 1 patch set | 20:21 |
| *** gyee has joined #openstack-swift | 20:46 | |
| openstackgerrit | Merged openstack/swift stable/train: Use ensure-pip role https://review.opendev.org/736845 | 21:00 |
| *** ormandj has joined #openstack-swift | 21:17 | |
| ormandj | hi... we just did an upgrade to train from stein, and have seen HEADs start failing with 403s | 21:17 |
| ormandj | happened right after upgrading the proxy servers | 21:19 |
| ormandj | i don't see anything in the notes indicating why that might have occurred | 21:21 |
| *** patchbot has quit IRC | 21:26 | |
| *** patchbot has joined #openstack-swift | 21:26 | |
| ormandj | ah, should add, using s3 | 21:27 |
| timburke | ormandj, is s3_acl enabled or disabled? was this using keystone for auth? | 21:39 |
| timburke | for a moment i thought it might befrom when we changed the default `location` config, but i see that happened in stein... | 21:41 |
| ormandj | timburke: keystone is auth, yes | 21:41 |
| ormandj | s3_acl set false (well, default) | 21:41 |
| ormandj | we are getting back 401s from keystone on s3token posts | 21:42 |
| ormandj | in the debug logs on swift proxy, tl;dr see calling s3api middleware, calling s3token middleware, connecting to keystone sending this json *stuff here*, keystone reply error status 401 | 21:44 |
| ormandj | and looking at the keystone logs with insecure debug on for testing | 21:44 |
| ormandj | keystone.exception.Unauthorized: The request you have made requires authentication. | 21:45 |
| ormandj | (and returns a 401) | 21:45 |
| ormandj | gets/puts/etc seem to be fine | 21:45 |
| timburke | in the 403, does it give the expected string to sign? does it match whatever debug output you can get from the client? | 21:47 |
| timburke | were these tagged versions from stein and train? and train's still running py2, yeah? | 21:49 |
| zaitcev | oh | 21:50 |
| zaitcev | I don't know if this is relevant, but I had a bad time trying to figure out what options modern keystone takes. My old config would not work. | 21:51 |
| ormandj | we hadn't upgraded keystone yet | 21:54 |
| ormandj | all was still on stein | 21:54 |
| ormandj | had just upgraded the proxy servers and boom, HEADs failing | 21:54 |
| ormandj | train is py3 afaik. | 21:54 |
| ormandj | hi swift-proxy 2.23.1-0ubuntu0.19.10.1~cloud0 all distributed virtual object store - proxy server | 21:55 |
| zaitcev | Look if yours looks like this: http://www.zaitcev.us/things/swift/proxy-server-train.conf | 21:56 |
| zaitcev | This is how I managed to make mine work and now I'm afraid to breath on it. | 21:57 |
| ormandj | we have authtoken before s3api | 21:58 |
| ormandj | we also have s3token after s3api | 21:58 |
| ormandj | you don't have that in your pipeline | 21:58 |
| zaitcev | Hmm. I may be wrong about that. | 21:58 |
| zaitcev | I only mean the ever-changing parameters for authtoken. | 21:59 |
| zaitcev | brb | 21:59 |
| ormandj | well, s3 auth is working for put/get/etc it appears | 21:59 |
| ormandj | only HEADs are failing | 21:59 |
| timburke | ormandj, the pipeline order should be good, fwiw -- i merged something similar recently for devstack: https://review.opendev.org/#/c/731003/ | 22:00 |
| patchbot | patch 731003 - devstack - swift: Fix s3api/keystone interaction (MERGED) - 1 patch set | 22:00 |
| ormandj | funny, i ran into that when trying to figure out this 'bug' | 22:01 |
| ormandj | i think we were working off: https://docs.openstack.org/swift/train/middleware.html#deployment in 'note' | 22:02 |
| timburke | i should update that note -- somewhere around when we started working with keystone v3, authtoken *had* to be left of s3token | 22:05 |
| timburke | it's weird -- there haven't been *that many* changes to s3api, and i can't think of how any of them would break *just HEADs*: http://paste.openstack.org/show/795015/ | 22:12 |
| ormandj | yeah... it's baffling me too | 22:12 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Start to decouple Object GET path https://review.opendev.org/733911 | 22:13 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Add concurrent primary feeder to EC GET requests https://review.opendev.org/711342 | 22:13 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Make concurrency timeout per policy and replica https://review.opendev.org/737096 | 22:13 |
| timburke | and as long as everything's ascii, i can't imagine how py2/py3 would be any trouble | 22:13 |
| clayg | alecuyer: i'm not sure if you can get a sense for p 737096 w/o docs - i'm curious if you were thinking of something like that | 22:14 |
| patchbot | https://review.opendev.org/#/c/737096/ - swift - Make concurrency timeout per policy and replica - 1 patch set | 22:14 |
| clayg | i still need to write something up about the non-durable frag thing in p 711342 | 22:14 |
| patchbot | https://review.opendev.org/#/c/711342/ - swift - Add concurrent primary feeder to EC GET requests - 10 patch sets | 22:14 |
| clayg | but i'm done for now | 22:15 |
| clayg | happy father's day weekend! | 22:15 |
| openstackgerrit | Merged openstack/swift master: Clean up some proxy tests https://review.opendev.org/735741 | 22:16 |
| openstackgerrit | Merged openstack/swift stable/ussuri: py3: Fix expirer container generation https://review.opendev.org/736007 | 22:16 |
| ormandj | timburke: yeah, i wouldn't think it's a unicode weirdness thing, i just shot you something more interesting from keystone w/ insecure_debug on | 22:19 |
| timburke | ormandj, just to sanity check: s3api and s3token were coming from swift, both before and after the upgrade, right? | 22:19 |
| timburke | (just limiting my search space) | 22:19 |
| ormandj | correct, we had migrated from swift3 to s3api i think in the prior release, but these are from swift, yes | 22:20 |
| ormandj | pipeline didn't change between stein -> train either | 22:20 |
| ormandj | timburke: https://github.com/openstack/keystone/blob/stable/stein/keystone/api/s3tokens.py#L78-L92 <-- we see this is where the error from keystone is coming from, back when we were on stein keystone (we upgraded to train there too) | 22:22 |
| timburke | which really seems to indicate that s3api did *something* different when it generated the string to sign... :-/ | 22:24 |
| timburke | looking through the diff... | 22:24 |
| timburke | was this 2.21.1 -> 2.23.1? | 22:25 |
| ormandj | let me look | 22:25 |
| ormandj | 2.19.1-0ubuntu1~cloud0 to 2.23.1-0ubuntu0.19.10.1~cloud0 | 22:26 |
| ormandj | one sec, let me confirm i'm not missing something :) | 22:27 |
| ormandj | hah, i was, sorry, yes 2.21.1 to 2.23.1 | 22:28 |
| timburke | https://github.com/openstack/swift/blob/2.23.1/swift/common/middleware/s3api/s3request.py#L133 feels a little weird when size is None... not sure why i did that... | 22:29 |
| ormandj | i'm seeing some changes in s3token that i don't quite grok not being familiar with the flow: https://github.com/openstack/swift/commit/4aa71aa25caed34f36fafe2de025425aa1d1e0b2#diff-97403e0c0b4d9e25851c3dd00b04cb57 | 22:32 |
| timburke | we used to get an actual token ID back from keystone and shove that in the env; that's what made a pipeline like `... swift3 s3token authtoken keystoneauth ...` work. but we stopped even looking for it once we realize that wasn't going to work in a v3-only world | 22:36 |
| timburke | (plus the doubled-up keystoen requests wasn't great) | 22:36 |
| ormandj | copy | 22:37 |
| *** gyee has quit IRC | 22:42 | |
| timburke | the hashing input thing only affects v4 signatures, btw -- it's something i should maybe investigate, but maybe not-so-relevant for this | 22:42 |
| ormandj | ah, so probably not this then | 22:43 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!