| openstackgerrit | Merged openstack/swift master: Use TOX_CONSTRAINTS_FILE https://review.opendev.org/760712 | 00:28 |
|---|---|---|
| *** gyee has quit IRC | 01:02 | |
| zaitcev | Tried everything, including explicit --overcloud-ssh-key xxx and it still fails: http://people.redhat.com/zaitcev/tmp/20201103_3/ostk_overcloud_deploy.txt | 02:35 |
| *** rcernin has quit IRC | 02:48 | |
| *** rcernin has joined #openstack-swift | 02:49 | |
| *** psachin has joined #openstack-swift | 03:46 | |
| *** dsariel has joined #openstack-swift | 04:17 | |
| *** dsariel has quit IRC | 04:21 | |
| *** psachin has quit IRC | 05:27 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-swift | 05:33 | |
| *** rcernin has quit IRC | 08:05 | |
| *** rcernin has joined #openstack-swift | 08:37 | |
| *** rcernin has quit IRC | 08:56 | |
| *** rpittau|afk is now known as rpittau | 09:05 | |
| *** m75abrams has joined #openstack-swift | 09:20 | |
| *** rcernin has joined #openstack-swift | 09:58 | |
| *** rcernin has quit IRC | 10:13 | |
| *** klamath_atx has quit IRC | 15:41 | |
| timburke | renich's right that we should add service_token_roles_required to the sample conf, though... | 15:50 |
| *** klamath_atx has joined #openstack-swift | 15:54 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: saio: Stop processes more forcefully in resetswift https://review.opendev.org/761439 | 16:15 |
| *** djhankb has quit IRC | 16:40 | |
| *** djhankb has joined #openstack-swift | 16:40 | |
| zaitcev | timburke: Did you have any PTL level communication from the new pop-up group raildo/gmann? | 17:17 |
| timburke | zaitcev, the what now? not that i recall... if i do, want me to send them your way? what's the pop-up group about? | 17:19 |
| *** m75abrams has quit IRC | 17:20 | |
| zaitcev | timburke: That was my question also. I hoped they'd explain to you what they want. https://governance.openstack.org/tc/reference/popup-teams.html#secure-default-policies | 17:21 |
| zaitcev | they had a PTG meeting here https://etherpad.opendev.org/p/consistent-and-secure-default-policies-wallaby | 17:23 |
| timburke | given our lack of integration with policy.json and the like, i'm not sure there's much for us to do... though it'd probably be good for us to add suport for a "reader" role | 17:25 |
| zaitcev | Oh, that's where's linked | 17:26 |
| zaitcev | I actually started from a Red Hat bug about the reader role | 17:26 |
| *** rpittau is now known as rpittau|afk | 17:35 | |
| gmann | timburke: zaitcev main idea for policy popup team is to add reader role as well the scope_type (https://docs.openstack.org/keystone/latest/contributor/services.html#authorization-scopes) | 18:00 |
| gmann | as swift has different way on RBAC, I am not sure how those can be added or scope is also needed or not in swift case. | 18:01 |
| *** klamath_atx has quit IRC | 18:09 | |
| *** klamath_atx has joined #openstack-swift | 18:09 | |
| *** mugsie has quit IRC | 18:18 | |
| *** mugsie has joined #openstack-swift | 18:21 | |
| ormandj | so, interesting point of note, we finally chased down the weird no-account-rewrite stuff re: s3token | 18:55 |
| ormandj | we had a user pummeling the service with simultaneous range requests for the same object, which was leading to keystone being cranky, and the LB was returning a 502 when it couldn't hit a backend | 18:55 |
| ormandj | instead of retrying, it looks like it just 401/403s the request | 18:55 |
| ormandj | (swift) | 18:55 |
| ormandj | https://github.com/openstack/swift/blob/master/swift/common/middleware/s3api/s3token.py#L237-L240 | 19:02 |
| openstackgerrit | Tim Burke proposed openstack/swift master: probe: Use ostestr as test runner https://review.opendev.org/761459 | 19:26 |
| timburke | ormandj, good to have an answer! fwiw, you might want to look into https://github.com/openstack/swift/blob/2.26.0/etc/proxy-server.conf-sample#L642-L654 | 19:35 |
| timburke | (big thanks to sorrison for making that work so much better!) | 19:36 |
| ormandj | let me look | 19:43 |
| ormandj | we have that enabled :) | 19:45 |
| ormandj | 300s timeout | 19:45 |
| ormandj | i think the issue is it's hitting multiple backends at the same time | 19:46 |
| ormandj | since it's making dozens of requests at the exact same ms | 19:46 |
| ormandj | ie: need file, issue curl requests for 1G of file split into 4k chunks, using Range for each, all at once | 19:46 |
| ormandj | we're not currently using stickiness on the LB so they get spread across our proxies | 19:46 |
| ormandj | it might just be a weird timing thing, we've _only_ seen this with this one particular client | 19:47 |
| ormandj | either way, adding retry functionality to the keystone connection in s3token on 5xx failures would make a lot of sense | 19:47 |
| timburke | ah... yeah, i could see them all getting a cache-miss and hitting keystone at the same time then :-/ | 19:56 |
| timburke | still, it sucks that the auth can't handle dozens of concurrent requests :-( | 19:57 |
| ormandj | yeah, keystone has been a lot of 'fun' for sure. it has gotten better over the years though | 20:04 |
| ormandj | but, good news is, good clients can compensate for cranky services | 20:05 |
| ormandj | we're looking to see what optimization we can do there too | 20:06 |
| ormandj | root@keystone01:/var/log/apache2# grep '04/Nov/2020:10:44:07' keystone-access.log |grep POST | wc -l | 20:15 |
| ormandj | 63 | 20:15 |
| ormandj | root@keystone01:/var/log/apache2# grep '04/Nov/2020:10:44:07' keystone-access.log |grep GET | wc -l | 20:15 |
| ormandj | 35 | 20:15 |
| ormandj | root@keystone01:/var/log/apache2# | 20:15 |
| ormandj | so just under 100 requests in that second | 20:15 |
| kota_ | good morning | 20:57 |
| timburke | almost meeting time! | 20:58 |
| timburke | acoles, clayg meeting ping | 21:03 |
| acoles | I'm there :) | 21:03 |
| clayg | i'm *still* trying to get EC multipart/byterange responses loaded into my head 😞 | 21:04 |
| rledisez | zaitcev: well, for now we still have some objects laying around on Swift, but yeah, clearly we will spend less time on Swift (actually, me and alecuyer are moving to a new project) | 21:04 |
| *** zaitcev is now known as zaitcev|doc | 21:38 | |
| *** zaitcev|doc is now known as zaitcev|dr | 21:38 | |
| *** rcernin has joined #openstack-swift | 21:41 | |
| *** acoles has quit IRC | 21:58 | |
| *** acoles has joined #openstack-swift | 21:59 | |
| clayg | i fixed my calendar so i should get notified at the right time next week 😁 | 22:01 |
| clayg | rledisez: that's cool you guys get to stick together! I'm sure the team working on object storage will miss you. Are you excited about what's next? | 22:02 |
| rledisez | clayg: yeah, it looks very challenging (in the good way :)). it's still storage, we are joining a team writing a clustered NVMe over Fabric target (the project comes from an acquisition: https://www.exten.io/ ) | 22:04 |
| rledisez | we will miss Swift for sure. and mostly because of you guys (I mean, the software is great, but you are awesome). But I'll still be around while we have our petabytes on Swift :) | 22:06 |
| *** rcernin has quit IRC | 22:09 | |
| *** rcernin has joined #openstack-swift | 22:09 | |
| *** rcernin has quit IRC | 23:04 | |
| *** rcernin has joined #openstack-swift | 23:12 | |
| *** rcernin has quit IRC | 23:16 | |
| *** rcernin has joined #openstack-swift | 23:16 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: probe: Use ostestr as test runner https://review.opendev.org/761459 | 23:17 |
| *** zaitcev|dr is now known as zaitcev | 23:48 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Continue logging deprecation warnings for run_pause https://review.opendev.org/761474 | 23:57 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!