| *** diablo_rojo has quit IRC | 00:06 | |
| timburke | hmmm... that's right, there were p 695780 and p 695781 that i was playing with last week.... | 00:23 |
|---|---|---|
| patchbot | https://review.opendev.org/#/c/695780/ - swift - s3api: Fix blank delimiter handling - 1 patch set | 00:23 |
| patchbot | https://review.opendev.org/#/c/695781/ - swift - s3api: Fix prefix/delimiter/marker quoting - 3 patch sets | 00:23 |
| tdasilva | clayg, timburke: so my theory for the `DELETE ?version-id=null` when history-location is enabled is because legacy is handling the delete, but during the PUT (while moving data) to archive container, it doesn't strip query parameters from the request, so object versioning is now getting a PUT `?version-id=null` which triggers a 400, legacy then converts it to a 412 | 00:33 |
| tdasilva | wondering if we should avoid handling `?versions` and `?version-id` requests all together to containers with history-location to avoid confusion from users. | 00:35 |
| tdasilva | although for situations like slo IIRC we ignore the manifest queries for non-slo objects | 00:37 |
| donnyd | So I am hoping there is some hope for my swift servers. I had a drive fail a while back and it would seem that this particular drive held an object for 95% of what is stored in swift | 01:20 |
| donnyd | I tried to run an audit, but after day 5 it was still not complete | 01:21 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Latch shard-stat reporting https://review.opendev.org/675014 | 01:21 |
| donnyd | this is the error that was thrown for all of the missing objects | 01:21 |
| donnyd | Failed fo fetch object /AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_b6b/periodic/opendev.org/openstack/tripleo-common/stable/stein/tripleo-ci-centos-7-scenario003-standalone/b6bc096/logs/ara_oooq/result/0728965b-f40e-4922-8bb0-58b30bc64910/index.html at all! | 01:21 |
| donnyd | There are 10 drives in each server and there are two servers. I only had a single drive failure | 01:22 |
| donnyd | I am not even sure where to being with it. Any pointers would be super appreciated | 01:24 |
| timburke | sure sounds like a ghost-listing -- object's deleted but still shows up in listings | 01:25 |
| donnyd | Well the objects have an expiration timer set | 01:25 |
| timburke | first thing i'd do is run something like `swift-get-nodes /etc/swift/object.ring.gz AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_b6b/periodic/opendev.org/openstack/tripleo-common/stable/stein/tripleo-ci-centos-7-scenario003-standalone/b6bc096/logs/ara_oooq/result/0728965b-f40e-4922-8bb0-58b30bc64910/index.html` to figure out where swift *thinks* that object should be | 01:26 |
| timburke | poke around on disk to see if there are any data files around -- if it's expired but not yet cleaned up, that could certainly do it | 01:27 |
| donnyd | Server:Port Device10.0.10.12:6000 sde | 01:27 |
| donnyd | Server:Port Device10.0.10.11:6000 sde | 01:27 |
| donnyd | Server:Port Device10.0.10.12:6000 sdc [Handoff] | 01:27 |
| donnyd | Server:Port Device10.0.10.11:6000 sdf [Handoff] | 01:27 |
| timburke | there should be some ls commands toward the end that'll give you (give or take) the path to look in | 01:28 |
| timburke | something like `ls -lah ${DEVICE:-/srv/node*}/sde/objects/318/776/4fb3c4be96a096b79904e1daf750e776`, though your particular swift_hash_path_prefix/suffix should mean it's different | 01:29 |
| donnyd | root@swift1:/home/ubuntu# ls -lah ${DEVICE:-/srv/node*}/sde/objects/561/f32/8c7a67c6d86b56f9627ad8ae62d5ef32 | 01:29 |
| donnyd | total 16K | 01:29 |
| donnyd | drwxr-xr-x 2 swift swift 4.0K Oct 1 08:18 . | 01:29 |
| donnyd | drwxr-xr-x 35 swift swift 4.0K Nov 12 08:17 .. | 01:29 |
| donnyd | -rwxr-xr-x 1 swift swift 746 Oct 1 08:18 1569917935.31814.data | 01:29 |
| donnyd | it looks to me like there is a thing there | 01:30 |
| timburke | cool, we've got a .data! swift-object-info /srv/.../1569917935.31814.data should tell you some info about it | 01:31 |
| timburke | including the expiration (which i'm guessing is in the past) | 01:31 |
| donnyd | https://www.irccloud.com/pastebin/jJYEC6DK/ | 01:32 |
| donnyd | date +%s | 01:33 |
| donnyd | 1575509573 | 01:33 |
| timburke | right, so it should've expired more than a month ago :-/ | 01:34 |
| timburke | let's check on the expiring-objects queue | 01:34 |
| timburke | swift-get-nodes /etc/swift/account.ring.gz .expiring_objects | 01:34 |
| timburke | should give you some curl commands | 01:34 |
| donnyd | https://www.irccloud.com/pastebin/YxD6whky/ | 01:36 |
| timburke | heh, sorry -- was getting my dev environment into a state that more-closely resembles yours ;-) we'll edit them a bit, get them more like `curl 'http://10.0.10.11:6002/sdf/380/.expiring_objects?limit=5&format=json' | python -m json.tool` | 01:39 |
| timburke | 7000 expiry buckets! wow! | 01:40 |
| timburke | i suppose that happens when it's *all* supposed to be expiring ;-) | 01:40 |
| donnyd | We knew the logs in swift thing would be interesting | 01:40 |
| timburke | :D | 01:40 |
| timburke | so the containers are also done as timestamps -- i'm trying to figure out how far behind we are by looking at just the first few containers | 01:41 |
| donnyd | https://www.irccloud.com/pastebin/A2NMf87f/ | 01:41 |
| timburke | ok, so ~2 months behind... | 01:43 |
| donnyd | So it never actually expired anything | 01:43 |
| donnyd | This whole swift log thing has only been running for like 5 | 01:43 |
| timburke | are the object-expirers running? | 01:44 |
| donnyd | Ummm.. is that a separate thing.. I am swift dumb, so I do apologize | 01:45 |
| donnyd | https://www.irccloud.com/pastebin/Bx3rmLoz/ | 01:45 |
| timburke | yep, gonna need another service running | 01:46 |
| donnyd | https://www.irccloud.com/pastebin/X0M38BUV/ | 01:46 |
| timburke | is there a swift-object-expirer unit? | 01:46 |
| donnyd | systemctl enable --now swift-object-expirer | 01:47 |
| donnyd | Failed to enable unit: Unit file swift-object-expirer.service does not exist. | 01:47 |
| donnyd | https://docs.openstack.org/swift/latest/overview_expiring_objects.html | 01:48 |
| donnyd | So maybe I should read this ^^^^ | 01:48 |
| donnyd | there is nothing about object expirations in the object-server.conf | 01:49 |
| donnyd | Does it need to be there or is it on by default? | 01:49 |
| timburke | sorry, this is a bit of a rough edge: historically there's been a /etc/swift/object-expirer.conf but recently we've been trying to move toward having it configured via object-server.conf | 01:51 |
| donnyd | ok, well I can give the object-server.conf way a go first | 01:52 |
| timburke | just make sure you've got dequeue_from_legacy_queue enabled | 01:52 |
| donnyd | So I need to set that value to true? | 01:53 |
| timburke | yep | 01:53 |
| donnyd | how will i know if its working? | 01:55 |
| timburke | you'll want it on both servers, so processes=2, and set process=0 on one, =1 on the other. probably want to be pretty aggressive on the concurrency value for both | 01:55 |
| donnyd | i set that to 12 | 01:56 |
| timburke | if you've got a graph for object-server requests, you'll see a good spike in DELETEs. i know you've got good disk usage graphs; that'll definitely come down | 01:56 |
| timburke | you can also watch logs to verify that it's making progress; by default every 5 minutes it should be dropping some stats | 01:57 |
| timburke | you might want to kick up the reclaim_age for now until things have caught up | 01:58 |
| donnyd | higher or lower number then? | 01:58 |
| donnyd | I assume lower | 01:59 |
| timburke | higher reclaim age -- that way we'll only remove work items if we actually did the work | 01:59 |
| timburke | keep an eye on the expiring objects account stats -- the object count there doesn't represent data on disk, but objects to be expired | 02:00 |
| timburke | it'll come down, but not to zero -- there should be some plateau | 02:00 |
| donnyd | Is there a good way to check that | 02:01 |
| timburke | basically, we want that GET to have containers for today, maybe even the last couple days, but not two months ago | 02:01 |
| donnyd | check across the whole cluster | 02:02 |
| tdasilva | would `swift-recon --expirer` help? | 02:02 |
| donnyd | ```swift-recon --expirer | 02:03 |
| donnyd | =============================================================================== | 02:03 |
| donnyd | --> Starting reconnaissance on 2 hosts (object) | 02:03 |
| donnyd | =============================================================================== | 02:03 |
| donnyd | [2019-12-05 02:02:40] Checking on expirers | 02:03 |
| donnyd | -> http://10.0.10.11:6000/recon/expirer/object: HTTP Error 400: Bad Request | 02:03 |
| donnyd | -> http://10.0.10.12:6000/recon/expirer/object: HTTP Error 400: Bad Request | 02:03 |
| donnyd | [object_expiration_pass] - No hosts returned valid data. | 02:03 |
| donnyd | [expired_last_pass] - No hosts returned valid data. | 02:03 |
| donnyd | =============================================================================== | 02:03 |
| timburke | probably! honestly, i haven't used swift-recon much :P i need to look more at what tools we provide | 02:03 |
| timburke | once you've got the expirers running, it'll probably provide more-useful info | 02:03 |
| donnyd | how can I know for sure they are running? | 02:04 |
| timburke | watch logs | 02:04 |
| tdasilva | donnyd: are you running tripleo swift? | 02:04 |
| donnyd | is there a seperate process that is spawned? | 02:04 |
| donnyd | now | 02:04 |
| donnyd | no tdasilva | 02:04 |
| tdasilva | or just upstream? | 02:04 |
| donnyd | that one | 02:04 |
| donnyd | just vanilla swift | 02:04 |
| timburke | i'm afraid i've gotta go, though -- kids waiting at home! i'm sure tdasilva can help :-) | 02:04 |
| donnyd | thanks for the help timburke | 02:05 |
| donnyd | its much appreciated | 02:05 |
| tdasilva | donnyd: `swift-init all start` is kinda of brute force, but at least makes sure it's running everything | 02:05 |
| tdasilva | how do you start other processes? | 02:06 |
| donnyd | systemctl | 02:06 |
| tdasilva | but upstream swift doesn't provide systemd configuration files for swift, does it? what am i missing? | 02:07 |
| donnyd | they were installed via package | 02:13 |
| donnyd | and the packages do ship unit files | 02:14 |
| mattoliverau | if you're using packages, are is there an expirer package then? or does the exirer appear in systemctl? | 02:17 |
| mattoliverau | ie, does it appear in systemctl list-units | 02:18 |
| mattoliverau | just so we can see what it's called to start it | 02:18 |
| tdasilva | iirc, in RH packages, object expirer was packaged with container-server package. zaitcev, cschwede would know for sure | 02:20 |
| mattoliverau | oh, of course it is :P | 02:20 |
| mattoliverau | well that wont work when we introduce the general task queue ;) | 02:20 |
| tdasilva | actually it's in the proxy | 02:27 |
| tdasilva | donnyd: so you will probably need to run expirer from your proxy nodes, in case you are not running paco nodes | 02:27 |
| tdasilva | https://cbs.centos.org/koji/rpminfo?rpmID=177802 | 02:27 |
| tdasilva | so i guess `systemctl openstack-swift-object-expirer` start should work? | 02:28 |
| tdasilva | well, you can also install the proxy package in all your other nodes | 02:29 |
| donnyd | So there needs to be a separate unit and should not run on the swift nodes themselves | 02:36 |
| tdasilva | donnyd: no no, i was just trying to figure out why you couldn't start object-expirer | 02:37 |
| tdasilva | you should be able to run expirer in your object-server nodes | 02:37 |
| donnyd | ok, then maybe i just need to install that package | 02:37 |
| tdasilva | welll...are you able to run: `systemctl start openstack-swift-object-expirer` ? | 02:38 |
| donnyd | no | 02:39 |
| tdasilva | ok, just checking, 'cause before you had tried `systemctl start swift-object-expirer`, looks like you were missing openstack in the name | 02:39 |
| tdasilva | then try install the proxy package | 02:40 |
| donnyd | I am missing the openstack name if I am on centos | 02:40 |
| tdasilva | ah, i see | 02:40 |
| donnyd | oh yes, having the package installed is super helpful | 02:41 |
| donnyd | LOL | 02:41 |
| donnyd | so now its good | 02:41 |
| tdasilva | donnyd: can you check what other swift services you have running | 02:42 |
| tdasilva | just afraid if you ddidn't have expirer running, you might not have other consistency engine services running | 02:42 |
| donnyd | https://www.irccloud.com/pastebin/m7QZUVeH/ | 02:43 |
| donnyd | tdasilva: I am afraid its likely I am a terrible swift operator | 02:43 |
| donnyd | I have been a ceph guy forever and this is my first go with swift | 02:44 |
| donnyd | much mo betta | 02:45 |
| donnyd | ps aux |grep expire | 02:45 |
| donnyd | swift 47575 0.2 0.0 120236 36508 ? Ss 02:41 0:00 /usr/bin/python2 /usr/bin/swift-object-expirer /etc/swift/object-expirer.conf | 02:45 |
| tdasilva | and you have rsync and memcache also running, right? | 02:45 |
| donnyd | Oh yes and now I see a bunch of deletes coming in | 02:46 |
| donnyd | yes | 02:46 |
| tdasilva | cool | 02:46 |
| tdasilva | hopefully `swift-recon --expirer` returns better info this time ?? | 02:46 |
| donnyd | I am using memcached from my controller. is that ok? | 02:46 |
| donnyd | https://www.irccloud.com/pastebin/mdm317Sl/ | 02:47 |
| tdasilva | you want to run it on the same nodes as the proxy nodes, possibly you are running proxy nodes from controller nodes too ? | 02:47 |
| donnyd | Yes, my controller has the proxy | 02:47 |
| tdasilva | mm...not sure why recon is returning that | 02:48 |
| donnyd | Oh well i ran the command from a swift storage node, not the proxy | 02:49 |
| donnyd | the proxy gives me a totally different error | 02:50 |
| donnyd | I will give it a little bit to catch up | 02:51 |
| donnyd | At least we are making some progress :) so thanks a bunch for helping me | 02:51 |
| tdasilva | np, hope it works out | 02:54 |
| donnyd | tdasilva: Me too. I have a bunch of build logs from the community in this swift and would like to get it normalized and back to work | 02:55 |
| donnyd | I am going to let this thing work at expiring all the old data and then run the audit again to see whats the what | 02:56 |
| tdasilva | ok, sounds good, definetely recommend playing with swift-recon tool more, and swift-dispersion-report might also be helpful | 02:58 |
| tdasilva | you want to make sure replicators and updaters are doing their work | 02:59 |
| donnyd | i do see them running in the logs and on the machines | 03:00 |
| donnyd | well replicator for sure | 03:00 |
| *** gyee has quit IRC | 03:03 | |
| *** diablo_rojo has joined #openstack-swift | 03:06 | |
| *** tkajinam is now known as tkajinam|lunch | 03:40 | |
| mattoliverau | So now that we have the null hidden containers merged. I wonder when we should think about moving sharding over to using it. Of course we'll always support the legacy. But seeing as the sharder just looks at containers on disk, there is no harm in updating stats for legecy over to the root container. and otherwise, when using null prefixed containers, the starts is magically done via the updaters, which will be much | 03:43 |
| mattoliverau | simpler. | 03:43 |
| mattoliverau | I don't mean we should jump on it.. but it would clean parts up a bit :) | 03:44 |
| mattoliverau | anyway that can wait until someone feels inclined or I find more upstream time. | 03:44 |
| * mattoliverau was reading p 682138 great work! | 03:45 | |
| patchbot | https://review.opendev.org/#/c/682138/ - swift - Allow internal clients to use reserved namespace (MERGED) - 38 patch sets | 03:45 |
| *** diablo_rojo has quit IRC | 03:46 | |
| * mattoliverau should stop reading cool patches, cause it makes me want to get distracted and planning things I want to hack on :P | 03:46 | |
| tdasilva | mattoliverau: I think there's a good distinction between .account and hidden containers that we might want to maintain | 03:47 |
| tdasilva | i guess one could start using hidden container in .accounts, but i'm not sure what the benefit of that would be | 03:47 |
| tdasilva | good distinction in use cases that is | 03:48 |
| tdasilva | since sharded containers stats get rolled up into the root container, i wonder if you moved them to hidden container, if you would get double counting | 03:50 |
| mattoliverau | oh yeah | 04:06 |
| mattoliverau | that's true | 04:07 |
| mattoliverau | your right. we want the stats to migrate to the root container not just the account | 04:07 |
| mattoliverau | so null containers don't work | 04:07 |
| mattoliverau | without horrible hacks.. and we already have hacks for the .accounts | 04:07 |
| mattoliverau | tdasilva: thanks, I can sigh some relief :) | 04:08 |
| tdasilva | heh | 04:08 |
| mattoliverau | tdasilva: where you hacking from today? find a nice place in Syd? | 04:09 |
| tdasilva | yeah, found a co-working place in Surry Hills, know the area? | 04:09 |
| mattoliverau | oh a little. that's near central isn't it? china town is near there. If head over that way theres some great food! including a great Korean BBQ, just ask notmyname ;) | 04:12 |
| mattoliverau | and csmart if he's online :) | 04:13 |
| * csmart waves | 04:13 | |
| csmart | also ask tonyb | 04:13 |
| csmart | :-) | 04:13 |
| mattoliverau | haha yeah :) | 04:15 |
| tonyb | say want now? | 04:15 |
| csmart | tonyb: we're talking about Korean BBQ in Sydney... | 04:15 |
| csmart | \#goodtimes | 04:15 |
| tonyb | csmart: "Mikal's paying!" | 04:16 |
| tdasilva | Yang San Park ? | 04:16 |
| tonyb | tdasilva: Correct | 04:16 |
| csmart | tonyb: haha | 04:17 |
| *** zaitcev has joined #openstack-swift | 04:17 | |
| *** ChanServ sets mode: +v zaitcev | 04:17 | |
| tonyb | tdasilva: https://mjolner.com.au/sydney is kinda funky | 04:20 |
| tdasilva | nice, that's not too far | 04:22 |
| tdasilva | tonyb: saved! | 04:23 |
| tonyb | tdasilva: I have a few faves for coffee but not super close to where you are | 04:23 |
| tdasilva | tonyb: please send my way, I can move around :) | 04:24 |
| tonyb | tdasilva: Will do | 04:24 |
| tdasilva | tonyb: especially if you have anything in the northern beaches area | 04:24 |
| tonyb | tdasilva: Sadly not | 04:28 |
| mattoliverau | StevenK: Your a Sydney local. Maybe you have good recommendations for tdasilva :) | 04:29 |
| StevenK | Coffee? But I don't drink it :-P | 04:31 |
| tdasilva | what? how come I didn't know about some many aussies in the swift channel? | 04:34 |
| tdasilva | we should definitely have a hackathon here | 04:34 |
| tonyb | tdasilva: If you like coffee then beanhunter https://www.beanhunter.com/australia is good as a reasearch tool | 04:34 |
| mattoliverau | StevenK: not just coffee, but just general places for tdasilva to visit while he's in town. Ie good food etc? :) | 04:37 |
| tdasilva | yeah, any place to grab a bite during lunch, break from work type of thing | 04:37 |
| tonyb | tdasilva: https://goo.gl/maps/QkYxqCEhP2zgS5jN7 is one of my faves for cofee and https://goo.gl/maps/EqWYkqELdD9xoEfF9 for brekfast | 04:38 |
| StevenK | Good food? I recommend Melbourne | 04:39 |
| tonyb | StevenK: be nice | 04:39 |
| mattoliverau | you can tell the Rackspace Australia office is in Circular Quay, or at least was when we were all employed there :P | 04:39 |
| tonyb | mattoliverau: true | 04:39 |
| StevenK | tonyb: I'm allowed to scoff at Sydney's food scene :-P | 04:39 |
| tonyb | StevenK: You are | 04:39 |
| StevenK | There's a good Japanese hot pot in the city which is amazing | 04:40 |
| tdasilva | lol | 04:40 |
| tonyb | StevenK: being a git ... I didn't actually say it was good food only that it was a fave for breakfast ;P | 04:40 |
| tonyb | StevenK: I like plenty of things are are bad ;p | 04:40 |
| StevenK | I was speaking in general :-P | 04:40 |
| tonyb | StevenK: fair | 04:41 |
| tdasilva | mmm...seems like everyone i talk to prefers melbourne over sydney | 04:41 |
| * StevenK reaches for the smashed avo joke | 04:42 | |
| StevenK | tdasilva: There's Muglan near Town Hall, which is pretty great | 04:42 |
| tdasilva | ah, that's close to the korean bbq and thai place notmyname mentioned | 04:43 |
| mattoliverau | tdasilva: there is a just a city rivalry between the 2. | 04:44 |
| tdasilva | i can tell :) it's funny | 04:44 |
| mattoliverau | oh the thai place was nice.. if it's the one I think he means. Glad he remembers where it was :P | 04:44 |
| StevenK | My external memory is not co-operating about where this hot pot restaurant is, but it's probably fairly close by to the places you've found | 04:47 |
| mattoliverau | tdasilva will be there for a few weeks I suspect, so you have time :) | 04:48 |
| StevenK | Also, there's a good reason to visit out west, but mattoliverau can't say publically :-P | 04:48 |
| mattoliverau | :) | 04:49 |
| mattoliverau | tdasilva: is swiftstack sends you down to canberra to meet with csmart (a customer of yours) let me know so I can come up ;) | 04:50 |
| mattoliverau | *if | 04:50 |
| mattoliverau | if I can get past these fires :P | 04:50 |
| tdasilva | mattoliverau: what town do you live in again? | 04:50 |
| mattoliverau | surf beach | 04:50 |
| StevenK | It's about four hours in a car from Sydney | 04:51 |
| mattoliverau | more with the northern roads closed atm :P | 04:51 |
| StevenK | Ugh, I keep forgetting there are a bunch of fires around | 04:51 |
| tdasilva | smoke is crazy in Sydney today | 04:53 |
| mattoliverau | yeah, it was really bad here this morning. seems wind is pushing it out to the sea now (at least for the fire close to me) | 04:54 |
| mattoliverau | but there seems to just a few near sydney.. just a few :P | 04:55 |
| mattoliverau | https://www.rfs.nsw.gov.au/fire-information/fires-near-me | 04:55 |
| tdasilva | wow | 05:03 |
| mattoliverau | yeah, we're kinda on fire. and summer has only just started. | 05:04 |
| mattoliverau | tdasilva: at least there's no shoveling snow for you this time of year ;) | 05:05 |
| tdasilva | true that, back home they just got their first snow fall of that year and it was a mere 35cm ;) | 05:09 |
| tdasilva | move a few km to the west and it was close to 50 | 05:10 |
| mattoliverau | wow | 05:21 |
| zaitcev | I'm in central Texas, we only get the frozen water rarely. | 05:23 |
| *** threestrands has joined #openstack-swift | 05:24 | |
| *** pcaruana has joined #openstack-swift | 05:38 | |
| *** pcaruana has quit IRC | 05:42 | |
| mattoliverau | zaitcev: sounds like here ;) | 05:45 |
| *** rcernin has quit IRC | 06:09 | |
| *** ccamacho has quit IRC | 06:54 | |
| *** pcaruana has joined #openstack-swift | 07:17 | |
| *** rcernin has joined #openstack-swift | 08:02 | |
| *** tesseract has joined #openstack-swift | 08:21 | |
| *** rpittau|afk is now known as rpittau | 08:32 | |
| *** mattoliverau has quit IRC | 08:34 | |
| *** mattoliverau has joined #openstack-swift | 08:38 | |
| *** orwell.freenode.net sets mode: +v mattoliverau | 08:38 | |
| *** threestrands has quit IRC | 09:07 | |
| *** tkajinam|lunch has quit IRC | 09:09 | |
| *** ccamacho has joined #openstack-swift | 09:42 | |
| *** mattoliverau has quit IRC | 09:54 | |
| *** ccamacho has quit IRC | 09:54 | |
| *** sorrison has quit IRC | 09:54 | |
| *** mugsie has quit IRC | 09:54 | |
| *** donnyd has quit IRC | 09:54 | |
| *** ccamel has quit IRC | 09:54 | |
| *** ormandj has quit IRC | 09:54 | |
| *** beisner has quit IRC | 09:54 | |
| *** rpittau has quit IRC | 09:54 | |
| *** gregwork has quit IRC | 09:54 | |
| *** toanster has quit IRC | 09:54 | |
| *** rledisez has quit IRC | 09:54 | |
| *** godog has quit IRC | 09:54 | |
| *** seongsoocho has quit IRC | 09:54 | |
| *** abelur has quit IRC | 09:54 | |
| *** d34dh0r53 has quit IRC | 09:54 | |
| *** openstackgerrit has quit IRC | 09:54 | |
| *** hugokuo has quit IRC | 09:54 | |
| *** evrardjp has quit IRC | 09:54 | |
| *** dasp has quit IRC | 09:54 | |
| *** StevenK has quit IRC | 09:54 | |
| *** manuvakery has quit IRC | 09:54 | |
| *** onovy has quit IRC | 09:54 | |
| *** kota_ has quit IRC | 09:54 | |
| *** peluse has quit IRC | 09:54 | |
| *** Anticimex has quit IRC | 09:54 | |
| *** coreycb has quit IRC | 09:54 | |
| *** clayg has quit IRC | 09:54 | |
| *** nottrobin has quit IRC | 09:54 | |
| *** spotz has quit IRC | 09:54 | |
| *** dabukalam has quit IRC | 09:54 | |
| *** timur has quit IRC | 09:54 | |
| *** tesseract has quit IRC | 09:54 | |
| *** pcaruana has quit IRC | 09:54 | |
| *** Jeffrey4l has quit IRC | 09:54 | |
| *** hoonetorg has quit IRC | 09:55 | |
| *** irclogbot_3 has quit IRC | 09:55 | |
| *** benj_ has quit IRC | 09:55 | |
| *** corvus has quit IRC | 09:55 | |
| *** edausq has quit IRC | 09:55 | |
| *** efried has quit IRC | 09:55 | |
| *** dosaboy has quit IRC | 09:55 | |
| *** szaher has quit IRC | 09:55 | |
| *** MooingLemur has quit IRC | 09:55 | |
| *** rcernin has quit IRC | 09:55 | |
| *** persia has quit IRC | 09:55 | |
| *** joeljwright has quit IRC | 09:55 | |
| *** fungi has quit IRC | 09:55 | |
| *** gmann has quit IRC | 09:55 | |
| *** zaitcev has quit IRC | 09:55 | |
| *** paladox has quit IRC | 09:55 | |
| *** jistr has quit IRC | 09:55 | |
| *** mvkr has quit IRC | 09:55 | |
| *** patchbot has quit IRC | 09:55 | |
| *** tonyb has quit IRC | 09:55 | |
| *** kukacz_ has quit IRC | 09:55 | |
| *** aluria has quit IRC | 09:55 | |
| *** rickflare has quit IRC | 09:55 | |
| *** dcourtoi has quit IRC | 09:55 | |
| *** cschwede has quit IRC | 09:55 | |
| *** timburke has quit IRC | 09:55 | |
| *** tristanC has quit IRC | 09:55 | |
| *** mathiasb has quit IRC | 09:55 | |
| *** noonedeadpunk has quit IRC | 09:55 | |
| *** baffle has quit IRC | 09:55 | |
| *** viks___ has quit IRC | 09:55 | |
| *** csmart has quit IRC | 09:55 | |
| *** f0o has quit IRC | 09:55 | |
| *** jrosser has quit IRC | 09:55 | |
| *** tdasilva has quit IRC | 09:55 | |
| *** mgagne has quit IRC | 09:55 | |
| *** lifeless has quit IRC | 09:55 | |
| *** mahatic has quit IRC | 09:55 | |
| *** early has quit IRC | 09:55 | |
| *** ab-a has quit IRC | 09:55 | |
| *** DHE has quit IRC | 09:55 | |
| *** alecuyer has quit IRC | 09:55 | |
| *** cjloader has quit IRC | 09:55 | |
| *** ChanServ has quit IRC | 09:55 | |
| *** mattoliverau has joined #openstack-swift | 10:04 | |
| *** sorrison has joined #openstack-swift | 10:04 | |
| *** ccamacho has joined #openstack-swift | 10:04 | |
| *** abelur has joined #openstack-swift | 10:04 | |
| *** godog has joined #openstack-swift | 10:04 | |
| *** rledisez has joined #openstack-swift | 10:04 | |
| *** seongsoocho has joined #openstack-swift | 10:04 | |
| *** toanster has joined #openstack-swift | 10:04 | |
| *** gregwork has joined #openstack-swift | 10:04 | |
| *** rpittau has joined #openstack-swift | 10:04 | |
| *** beisner has joined #openstack-swift | 10:04 | |
| *** ccamel has joined #openstack-swift | 10:04 | |
| *** donnyd has joined #openstack-swift | 10:04 | |
| *** mugsie has joined #openstack-swift | 10:04 | |
| *** ormandj has joined #openstack-swift | 10:04 | |
| *** evrardjp has joined #openstack-swift | 10:04 | |
| *** hugokuo has joined #openstack-swift | 10:04 | |
| *** d34dh0r53 has joined #openstack-swift | 10:04 | |
| *** openstackgerrit has joined #openstack-swift | 10:04 | |
| *** tesseract has joined #openstack-swift | 10:04 | |
| *** rcernin has joined #openstack-swift | 10:04 | |
| *** pcaruana has joined #openstack-swift | 10:04 | |
| *** zaitcev has joined #openstack-swift | 10:04 | |
| *** timur has joined #openstack-swift | 10:04 | |
| *** spotz has joined #openstack-swift | 10:04 | |
| *** dabukalam has joined #openstack-swift | 10:04 | |
| *** efried has joined #openstack-swift | 10:04 | |
| *** paladox has joined #openstack-swift | 10:04 | |
| *** jistr has joined #openstack-swift | 10:04 | |
| *** f0o has joined #openstack-swift | 10:04 | |
| *** Jeffrey4l has joined #openstack-swift | 10:04 | |
| *** persia has joined #openstack-swift | 10:04 | |
| *** jrosser has joined #openstack-swift | 10:04 | |
| *** mvkr has joined #openstack-swift | 10:04 | |
| *** hoonetorg has joined #openstack-swift | 10:04 | |
| *** baffle has joined #openstack-swift | 10:04 | |
| *** csmart has joined #openstack-swift | 10:04 | |
| *** joeljwright has joined #openstack-swift | 10:04 | |
| *** orwell.freenode.net sets mode: +vvvv mattoliverau rledisez zaitcev joeljwright | 10:04 | |
| *** ab-a has joined #openstack-swift | 10:04 | |
| *** patchbot has joined #openstack-swift | 10:04 | |
| *** tonyb has joined #openstack-swift | 10:04 | |
| *** irclogbot_3 has joined #openstack-swift | 10:04 | |
| *** benj_ has joined #openstack-swift | 10:04 | |
| *** corvus has joined #openstack-swift | 10:04 | |
| *** edausq has joined #openstack-swift | 10:04 | |
| *** tdasilva has joined #openstack-swift | 10:04 | |
| *** mgagne has joined #openstack-swift | 10:04 | |
| *** kukacz_ has joined #openstack-swift | 10:04 | |
| *** lifeless has joined #openstack-swift | 10:04 | |
| *** mahatic has joined #openstack-swift | 10:04 | |
| *** early has joined #openstack-swift | 10:04 | |
| *** rickflare has joined #openstack-swift | 10:04 | |
| *** aluria has joined #openstack-swift | 10:04 | |
| *** dosaboy has joined #openstack-swift | 10:04 | |
| *** viks___ has joined #openstack-swift | 10:04 | |
| *** fungi has joined #openstack-swift | 10:04 | |
| *** dcourtoi has joined #openstack-swift | 10:04 | |
| *** szaher has joined #openstack-swift | 10:04 | |
| *** DHE has joined #openstack-swift | 10:04 | |
| *** MooingLemur has joined #openstack-swift | 10:04 | |
| *** mathiasb has joined #openstack-swift | 10:04 | |
| *** cschwede has joined #openstack-swift | 10:04 | |
| *** alecuyer has joined #openstack-swift | 10:04 | |
| *** timburke has joined #openstack-swift | 10:04 | |
| *** orwell.freenode.net sets mode: +vvvv tdasilva mahatic cschwede timburke | 10:04 | |
| *** tristanC has joined #openstack-swift | 10:04 | |
| *** noonedeadpunk has joined #openstack-swift | 10:04 | |
| *** cjloader has joined #openstack-swift | 10:04 | |
| *** gmann has joined #openstack-swift | 10:04 | |
| *** ChanServ has joined #openstack-swift | 10:04 | |
| *** orwell.freenode.net sets mode: +o ChanServ | 10:04 | |
| *** peluse has joined #openstack-swift | 10:04 | |
| *** dasp has joined #openstack-swift | 10:04 | |
| *** StevenK has joined #openstack-swift | 10:04 | |
| *** manuvakery has joined #openstack-swift | 10:04 | |
| *** onovy has joined #openstack-swift | 10:04 | |
| *** kota_ has joined #openstack-swift | 10:04 | |
| *** Anticimex has joined #openstack-swift | 10:04 | |
| *** coreycb has joined #openstack-swift | 10:04 | |
| *** clayg has joined #openstack-swift | 10:04 | |
| *** nottrobin has joined #openstack-swift | 10:04 | |
| *** orwell.freenode.net sets mode: +vv kota_ clayg | 10:04 | |
| *** rdejoux has joined #openstack-swift | 10:15 | |
| *** pcaruana has quit IRC | 10:42 | |
| *** rcernin has quit IRC | 12:52 | |
| *** pcaruana has joined #openstack-swift | 12:54 | |
| *** diablo_rojo has joined #openstack-swift | 14:00 | |
| *** pcaruana has quit IRC | 14:38 | |
| *** rpittau is now known as rpittau|afk | 15:03 | |
| ormandj | clayg: timburke: moving ratelimit after s3token fixed the PUT issue, so we're in the home stretch now. last remaining issue is how the 498 is handled that swift returns. we're seeing an exception | 16:06 |
| ormandj | Dec 5 15:21:06 sw-proxy01.blah proxy-server: Returning 498 for PUT to AUTH_40a4e27e114a40b0829395945766c9f8/ormandj/test86 . Ratelimit (Max Sleep) Max Sleep Time Exceeded: 60.71 (txn: tx2d6d88fe30e54bb7a1ab6-005de92062) | 16:06 |
| ormandj | Dec 5 15:21:06 sw-proxy01.blah proxy-server: 500 Internal Server Error: #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/dist-packages/swift/common/middleware/s3api/s3api.py", line 229, in __call__#012 resp = self.h | 16:07 |
| ormandj | andle_request(req)#012 File "/usr/lib/python2.7/dist-packages/swift/common/middleware/s3api/s3api.py", line 265, in handle_request#012 res = handler(req)#012 File "/usr/lib/python2.7/dist-packages/swift/common/middleware/s3api/controllers/obj.py", l | 16:07 |
| ormandj | ine 128, in PUT#012 resp = req.get_response(self.app)#012 File "/usr/lib/python2.7/dist-packages/swift/common/middleware/s3api/s3request.py", line 1378, in get_response#012 headers, body, query)#012 File "/usr/lib/python2.7/dist-packages/swift/co | 16:07 |
| ormandj | mmon/middleware/s3api/s3request.py", line 1362, in _get_response#012 raise InternalError('unexpected status code %d' % status)#012InternalError: 500 Internal Server Error (txn: tx2d6d88fe30e54bb7a1ab6-005de92062) | 16:07 |
| ormandj | yuck, sorry on paste. | 16:07 |
| ormandj | https://github.com/openstack/swift/blob/stable/stein/swift/common/middleware/s3api/s3request.py#L1362 | 16:07 |
| clayg | 😂. We’re not in good shape when a 500 & a traceback is considered “progress” | 16:08 |
| ormandj | haha, i've been using swift a while, my yardstick has changed dimension :p | 16:08 |
| ormandj | 503 Slow Down <- appropriate s3 response when rate limiting, so somehow 498 from swift should be getting translated to this. looking at https://github.com/openstack/swift/blob/stable/rocky/swift/common/middleware/s3api/s3response.py#L650 it looks like that should be accounted for | 16:11 |
| ormandj | i'm not sure where the 500 is coming from, i haven't spent a ton of time tracking that down yet. | 16:11 |
| ormandj | what we end up seeing is a 500 issued to the client for that request | 16:16 |
| *** tesseract has quit IRC | 16:39 | |
| *** gyee has joined #openstack-swift | 16:41 | |
| *** openstackgerrit has quit IRC | 16:42 | |
| timburke | yeah, it's the 'unexpected status code' traceback -- i should really try to quiet that down... | 16:47 |
| timburke | i think we'd want to add the 498-handling around https://github.com/openstack/swift/blob/2.23.0/swift/common/middleware/s3api/s3request.py#L1365-L1373 | 16:48 |
| clayg | timburke: 👍 | 17:06 |
| ormandj | yeah, what's odd to me, it's showing 500 | 17:30 |
| ormandj | i would have expected 503, as that's what the 498 slow down should be xlated to in s3response | 17:30 |
| *** openstackgerrit has joined #openstack-swift | 17:35 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Latch shard-stat reporting https://review.opendev.org/675014 | 17:35 |
| *** ccamacho has quit IRC | 17:46 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: s3api: Better handle 498/429 responses https://review.opendev.org/697535 | 17:54 |
| *** mvkr has quit IRC | 19:31 | |
| timburke | :-( one of these things is not like the other... https://github.com/openstack/swift/blob/2.23.0/swift/common/middleware/s3api/s3api.py#L295-L296 | 19:59 |
| timburke | NameError waiting to happen | 19:59 |
| *** rcernin has joined #openstack-swift | 20:03 | |
| *** dosaboy has quit IRC | 20:04 | |
| ormandj | timburke: hilariously, your proposed solution is what we just figured out to implement a few minutes ago after hours of adding log lines because i have no idea what i'm doing :p | 20:08 |
| ormandj | timburke: on the note, the rate limiting seems to have to be after the s3api middle ware, or it breaks entirely for s3 | 20:09 |
| ormandj | i should be more clear, the clients of s3 do not get notification of rate limiting | 20:09 |
| ormandj | so they can't implement exponential backoff/etc | 20:09 |
| ormandj | also, does rate limiting even work if not after s3api, for s3api requests? it didn't seem to in our testing, at least we never saw the rate limiting logs being produced in our debug output | 20:10 |
| *** rcernin has quit IRC | 20:18 | |
| *** zaitcev_ has joined #openstack-swift | 20:18 | |
| *** ChanServ sets mode: +v zaitcev_ | 20:18 | |
| *** zaitcev has quit IRC | 20:22 | |
| ormandj | timburke: FWIW, we just backported to stein and deployed in our dev env and verified the rate limiting is functional | 20:43 |
| ormandj | client is seeing the 503s and behaving | 20:43 |
| ormandj | https://privatebin.net/?362eea6af579b0fb#2JrfH9ykGVPrytTUpczZDYzn4rkxhoAMvGndeTUZYDzy | 20:46 |
| ormandj | (was doing 1000 parallel puts) | 20:47 |
| timburke | glad to hear it's doing better :-D | 21:00 |
| *** dosaboy has joined #openstack-swift | 21:09 | |
| *** rdejoux has quit IRC | 21:26 | |
| *** persia has quit IRC | 21:43 | |
| *** persia has joined #openstack-swift | 21:45 | |
| *** dasp_ has joined #openstack-swift | 21:52 | |
| *** dasp has quit IRC | 21:55 | |
| mattoliverau | morning | 22:08 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Set swift_source in account_quotas middleware https://review.opendev.org/697578 | 22:18 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Set swift_source more in s3api middleware https://review.opendev.org/697580 | 22:23 |
| *** rcernin has joined #openstack-swift | 22:28 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Set swift_source more in versioned_writes https://review.opendev.org/697581 | 22:33 |
| mattoliverau | set swift_source in all the things :P ^ | 22:38 |
| *** zaitcev_ is now known as zaitcev | 22:42 | |
| zaitcev | a good practice | 22:42 |
| timburke | all of them! | 22:52 |
| *** tkajinam has joined #openstack-swift | 23:07 | |
| *** mvkr has joined #openstack-swift | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!