| openstackgerrit | Tim Burke proposed openstack/swift master: py3: (Better) fix percentages in configs https://review.opendev.org/734721 | 00:01 |
|---|---|---|
| openstackgerrit | Tim Burke proposed openstack/swift master: WIP: run probe tests on CentOS 8 https://review.opendev.org/690717 | 00:26 |
| openstackgerrit | Tim Burke proposed openstack/swift master: py3: (Better) fix percentages in configs https://review.opendev.org/734721 | 00:27 |
| openstackgerrit | Tim Burke proposed openstack/swift master: py3: (Better) fix percentages in configs https://review.opendev.org/734721 | 00:46 |
| *** threestrands has joined #openstack-swift | 01:19 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: WIP: run probe tests on CentOS 8 https://review.opendev.org/690717 | 01:28 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Use ini_file to update timeout instead of crudini https://review.opendev.org/734729 | 01:28 |
| *** manuvakery has joined #openstack-swift | 02:40 | |
| *** rcernin has quit IRC | 02:53 | |
| *** rcernin_ has joined #openstack-swift | 02:54 | |
| *** rcernin_ has quit IRC | 03:07 | |
| *** rcernin_ has joined #openstack-swift | 03:19 | |
| *** rcernin_ has quit IRC | 03:24 | |
| *** rcernin has joined #openstack-swift | 03:25 | |
| *** psachin has joined #openstack-swift | 03:40 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: WIP: run probe tests on CentOS 8 https://review.opendev.org/690717 | 03:53 |
| openstackgerrit | Tim Burke proposed openstack/swift master: WIP: run probe tests on CentOS 8 https://review.opendev.org/690717 | 04:12 |
| openstackgerrit | Tim Burke proposed openstack/swift master: WIP: run probe tests on CentOS 8 https://review.opendev.org/690717 | 04:19 |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-swift | 04:33 | |
| *** manuvakery has quit IRC | 04:49 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Add py3 probe tests on CentOS 8 https://review.opendev.org/690717 | 05:10 |
| *** m75abrams has joined #openstack-swift | 05:16 | |
| timburke | sorry for all the noise, but i think that's actually getting to a good place finally! | 05:23 |
| *** renich has quit IRC | 05:52 | |
| *** djhankb has quit IRC | 06:08 | |
| *** rcernin has quit IRC | 06:44 | |
| *** rcernin_ has joined #openstack-swift | 06:44 | |
| *** rcernin_ has quit IRC | 06:50 | |
| *** rcernin_ has joined #openstack-swift | 06:53 | |
| *** rcernin_ has quit IRC | 06:59 | |
| *** psachin has quit IRC | 07:12 | |
| *** rpittau|afk is now known as rpittau | 07:21 | |
| *** ccamacho has joined #openstack-swift | 07:24 | |
| *** ravsingh has joined #openstack-swift | 08:24 | |
| godog | timburke: thanks for the tip on --use-slo, I missed the option! using it I'm also seeing the json manifest so that's at least consistent, I feel like I'm missing something obvious but I've put more information/debug here https://phabricator.wikimedia.org/T254852#6209792 | 09:10 |
| *** rpittau is now known as rpittau|bbl | 10:19 | |
| *** tkajinam has quit IRC | 10:53 | |
| *** threestrands has quit IRC | 11:16 | |
| *** m75abrams has quit IRC | 11:16 | |
| *** zigo has quit IRC | 11:41 | |
| *** rcernin_ has joined #openstack-swift | 11:48 | |
| *** rcernin_ has quit IRC | 11:53 | |
| *** zigo has joined #openstack-swift | 11:57 | |
| *** rpittau|bbl is now known as rpittau | 12:05 | |
| *** ravsingh has quit IRC | 12:20 | |
| *** ravsingh has joined #openstack-swift | 13:07 | |
| *** mikecmpbll has joined #openstack-swift | 13:12 | |
| openstackgerrit | David Sariel proposed openstack/swift master: WIP: Let developers/operators add watchers to object audit (simplified) https://review.opendev.org/706653 | 14:04 |
| *** ccamacho has quit IRC | 14:11 | |
| timburke | godog, any chance you've got an explicit allowed_headers set on the object-servers? if so, does it include X-Static-Large-Object? https://github.com/openstack/swift/blob/master/etc/object-server.conf-sample#L135 | 14:35 |
| timburke | fwiw, i could repro similar bad behavior when i tried restricting mine to just X-Delete-At: http://paste.openstack.org/show/794582/ | 14:40 |
| godog | timburke: hah! that's a good smoking gun, I have indeed allowed_headers set and x-static-large-object isn't there | 14:41 |
| godog | I'm adding now and trying again, anything else that should be included ? | 14:41 |
| timburke | X-Delete-At, X-Object-Manifest, X-Static-Large-Object are all fairly necessary -- we should consider forcing them to be allowed regardless of config | 14:42 |
| timburke | the other defaults that i linked to are mainly for the sake of clients -- they should all be safe, and some clients may expect them to be available | 14:43 |
| timburke | out of curiosity, how are you using swift? | 14:44 |
| godog | sounds good re: headers, x-delete-at x-object-manifest were there already | 14:45 |
| godog | in this particular case we're deploying thanos with s3api + swift as its object storage | 14:45 |
| godog | our main use case for swift though is media storage, that's a separate swift cluster and a much bigger deployment though no s3api | 14:46 |
| godog | and no SLO usage either, which explains why we've never run into this before! | 14:47 |
| timburke | cool! i don't think i'd heard of https://thanos.io/ before, seems interesting | 14:47 |
| godog | yeah fairly happy with it so far but it isn't in full production yet | 14:48 |
| godog | timburke: can confirm adding x-static-large-object to allowed_headers worked, thank you so much for your help! | 14:49 |
| godog | happy to followup with a bug, where should I file it ? | 14:49 |
| timburke | https://bugs.launchpad.net/swift | 14:51 |
| timburke | thanks in advance! | 14:51 |
| godog | no problem -- thanks again! | 14:52 |
| *** ccamacho has joined #openstack-swift | 14:54 | |
| openstackgerrit | David Sariel proposed openstack/swift master: WIP: Let developers/operators add watchers to object audit (simplified) https://review.opendev.org/706653 | 14:55 |
| *** gyee has joined #openstack-swift | 15:54 | |
| *** rpittau is now known as rpittau|afk | 16:07 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Add py3 probe tests on CentOS 8 https://review.opendev.org/690717 | 16:12 |
| *** mikecmpbll has quit IRC | 16:15 | |
| *** mikecmpbll has joined #openstack-swift | 16:18 | |
| *** ravsingh has quit IRC | 16:36 | |
| *** gyee has quit IRC | 16:42 | |
| *** gyee has joined #openstack-swift | 17:08 | |
| clayg | timburke: i'd prefer you don't update p 733919 - just spin improvements on top | 17:19 |
| patchbot | https://review.opendev.org/#/c/733919/ - swift - s3api: Allow CompleteMultipartUpload requests to b... - 3 patch sets | 17:19 |
| clayg | my only reasoning there is because I'd already packaged it - it's not a big deal if it changes before it lands; but if the changes you're suggesting are mostly non-material (comments and stuff) - it seems slightly more clean for me if those are follow on improvements | 17:20 |
| timburke | yeah, that was my thinking, too, for the most part. the comment might be worth fixing, but the abort change seems to belong in its own patch | 17:20 |
| timburke | i'd be fine with fixing the comment in the abort change, too | 17:22 |
| clayg | yeah all seems orthogonal to the complete mpu fix | 17:28 |
| clayg | timburke: so my first attempt to evaluate p 728298 on my vsaio didn't go great 🤔 | 17:54 |
| patchbot | https://review.opendev.org/#/c/728298/ - swift - proxy-logging: Use swift.backend_path if available - 3 patch sets | 17:54 |
| clayg | i run with force_swift_request_proxy_log = true - so I was getting my swift requests logged with statsd metrics - but when I restarted my proxy after checking out the patch I lost ALL metrics | 17:55 |
| *** mikecmpbll has quit IRC | 18:06 | |
| *** mikecmpbll has joined #openstack-swift | 18:07 | |
| clayg | force_swift_request_proxy_log doesn't seem to effect behavior on the patch - i'm just not getting any metrics... I did notice s3 logs PUT as 200 vs 201, I think i need to make some mixed GET/PUT s3/swift load | 18:08 |
| clayg | on master if you turn off force_swift_request_proxy_log you don't get any stats... so that's probably not great or well understood | 18:08 |
| clayg | and so now it looks like it DOES double count GETs 😞 | 19:49 |
| timburke | clayg, how does it behave for SLOs? like, normal, everyday SLOs over the swift api? | 19:50 |
| clayg | man, i'm still on normal PUT/GET 🤷 | 19:51 |
| clayg | what part of SLO are you interested in? And this is relevant to p 728298? | 19:51 |
| patchbot | https://review.opendev.org/#/c/728298/ - swift - proxy-logging: Use swift.backend_path if available - 3 patch sets | 19:51 |
| timburke | i'm wondering if it *also* double-counts bytes | 19:52 |
| clayg | it's totally possible my expectations are off - i've never really tried to validate our statsd emissions - i'm leaning how master works while also seeing striking differences with this change | 19:52 |
| clayg | if it also double counts... what? like the HEADs from the manifest upload? | 19:52 |
| timburke | nah, GETs on download -- like, once for the segment subrequest, then again for the respone to the client | 19:53 |
| clayg | i don't know that I have that kind of resolution - i'm just looking at timing.count - you're thinking about the xfer metrics? | 19:55 |
| clayg | i'm happy to run the experiment - but i don't know if I'm setup for it | 19:56 |
| timburke | clayg, so i think SLO *also* double-counts: http://paste.openstack.org/show/794599/ | 20:21 |
| timburke | it printed out 2048, 2048, 692 for each of the segments, then 4788 for the total client response :-/ | 20:22 |
| timburke | (and the manifest is unaccounted-for) | 20:23 |
| clayg | where "double" is each subrequest gets logged with metrics - and the client request also gets logged with metrics | 20:24 |
| clayg | I mean, I don't see xfer:4788 in there more than once | 20:24 |
| timburke | yeah, that's what i mean. but i feel like there's a decent analogy to s3api -- when i run `aws s3 ls s3://bucket`, i see http://paste.openstack.org/show/794600/ -- 28 bytes for the (swift) container listing, then 361 for the (s3) response to the client | 20:31 |
| timburke | it *does* make the stats a little hard to reason about, though -- you get an upper-bound on how much data was actually sent to the client or how many total client requests were made (rather than an exact count) | 20:33 |
| timburke | i suppose there's a reason we mostly try to drive *our* xfer graphs from firewall data | 20:34 |
| clayg | turning off force_swift_request_proxy_log resulted in more or less the expected results for GETs | 20:36 |
| clayg | but my PUTs are still entirely missing 😞 | 20:36 |
| clayg | I think it must have *something* to do with 201/200 - but I haven't teased it out yet | 20:37 |
| timburke | curious. fwiw, i see http://paste.openstack.org/show/794601/ for an ordinary upload | 20:41 |
| clayg | it's like they're getting logged as container PUTs 🤔 | 20:42 |
| clayg | that's an s3 upload!? | 20:43 |
| timburke | yes, that was an s3 upload with my patch | 20:45 |
| timburke | aws s3 cp tox.ini s3://bucket/swift/ | 20:45 |
| clayg | ugh, bonkers man - that's definitely not what I'm seeing 😞 | 20:51 |
| clayg | what do you have s3_acl set to? | 20:52 |
| clayg | or check_bucket_owner? | 20:53 |
| clayg | turning both of those off fixed my PUTs being logged as container PUTs 🤷 | 20:53 |
| timburke | oh! that's probably it -- i had it off | 20:53 |
| clayg | i think the 'if backend not in environ' can cause some dirty? | 20:54 |
| seongsoocho | morning !! | 20:57 |
| timburke | hrm... might need to think more about all of this. but at least now i've gone to the trouble of writing a little statsd listener! | 20:58 |
| timburke | almost meeting time! | 20:58 |
| kota_ | morning | 20:59 |
| *** threestrands has joined #openstack-swift | 21:55 | |
| *** rcernin_ has joined #openstack-swift | 21:59 | |
| *** rcernin_ has quit IRC | 22:04 | |
| *** rcernin has joined #openstack-swift | 22:04 | |
| *** rcernin is now known as rcernin|brekkie | 22:05 | |
| *** rcernin|brekkie has quit IRC | 22:19 | |
| *** mikecmpbll has quit IRC | 22:44 | |
| *** tkajinam has joined #openstack-swift | 22:45 | |
| *** djhankb has joined #openstack-swift | 23:02 | |
| *** rcernin|brekkie has joined #openstack-swift | 23:09 | |
| *** rcernin|brekkie has quit IRC | 23:10 | |
| *** rcernin has joined #openstack-swift | 23:10 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!