Thursday, 2020-09-24

« Wednesday, 2020-09-23 Index Friday, 2020-09-25 »
*** gyee has quit IRC00:27
*** MooingLe1ur has quit IRC01:24
*** MooingLemur has joined #openstack-swift01:25
openstackgerritMerged openstack/swift stable/victoria: Update .gitreview for stable/victoria  https://review.opendev.org/75275002:22
*** rcernin has quit IRC02:38
*** rcernin has joined #openstack-swift02:42
*** psachin has joined #openstack-swift03:27
openstackgerritTim Burke proposed openstack/pyeclib master: Fix liberasurecode-src tox target  https://review.opendev.org/75393603:58
openstackgerritTim Burke proposed openstack/pyeclib master: Use liberasurecode-src tox target in gate  https://review.opendev.org/75393804:03
openstackgerritTim Burke proposed openstack/pyeclib master: Use liberasurecode-git tox target in gate  https://review.opendev.org/75393804:32
*** evrardjp has quit IRC04:33
*** evrardjp has joined #openstack-swift04:33
openstackgerritMerged openstack/swift stable/victoria: Fixing S3 credential caching region support  https://review.opendev.org/75383904:40
*** psachin has quit IRC04:45
openstackgerritTim Burke proposed openstack/pyeclib master: Fix liberasurecode-git tox target  https://review.opendev.org/75393604:55
openstackgerritTim Burke proposed openstack/pyeclib master: Use liberasurecode-git tox target in gate  https://review.opendev.org/75393804:55
*** m75abrams has joined #openstack-swift05:06
openstackgerritTim Burke proposed openstack/pyeclib master: Use liberasurecode-git tox target in gate  https://review.opendev.org/75393805:37
openstackgerritMerged openstack/pyeclib master: Fix liberasurecode-git tox target  https://review.opendev.org/75393605:51
*** baojg has quit IRC07:29
*** baojg has joined #openstack-swift07:30
*** djhankb has quit IRC07:31
*** djhankb has joined #openstack-swift07:31
*** rcernin has quit IRC08:35
*** rcernin has joined #openstack-swift08:41
*** rcernin has quit IRC08:57
*** baojg has quit IRC09:04
*** baojg has joined #openstack-swift09:41
*** adriant has quit IRC09:52
*** adriant has joined #openstack-swift09:53
*** baojg has quit IRC10:22
*** baojg has joined #openstack-swift10:23
*** rcernin has joined #openstack-swift10:35
*** rcernin has quit IRC10:56
*** tdasilva has quit IRC13:15
*** tdasilva has joined #openstack-swift13:17
*** ChanServ sets mode: +v tdasilva13:17
*** baojg has quit IRC13:21
*** rcernin has joined #openstack-swift13:36
*** jv has quit IRC13:50
*** rcernin has quit IRC13:51
*** jv has joined #openstack-swift13:54
*** baojg has joined #openstack-swift14:21
*** baojg has quit IRC14:46
*** baojg has joined #openstack-swift14:47
openstackgerritTim Burke proposed openstack/swift master: Add a new URL parameter to allow for async cleanup of SLO segments  https://review.opendev.org/73302615:05
*** tdasilva_ has joined #openstack-swift15:12
*** ChanServ sets mode: +v tdasilva_15:12
*** tdasilva has quit IRC15:14
*** m75abrams has quit IRC15:31
*** baojg has quit IRC15:45
*** gyee has joined #openstack-swift15:59
openstackgerritTim Burke proposed openstack/pyeclib master: Use liberasurecode-git tox target in gate  https://review.opendev.org/75393816:54
timburkeclayg, hmm.... https://github.com/openstack/swift/blob/2.26.0/swift/common/middleware/staticweb.py#L431-L43517:25
claygand we normally miss it because of env.get('REMOTE_USER') in __call__ ?17:27
timburkeyup -- https://github.com/openstack/swift/blob/2.26.0/swift/common/middleware/staticweb.py#L544-L54617:27
claygwell spotted - curious about your methodology - i poked at this for awhile and got stuck17:28
timburkeheh. wrote up a little middleware like http://paste.openstack.org/show/798350/ and put it between most every middleware in my pipeline :-)17:29
timburkegrepping for asdf and running the unauthed req gave me logs like http://paste.openstack.org/show/798351/17:31
*** baojg has joined #openstack-swift17:32
timburkeseems weird that we copy the env just to update user agent and source :-/17:35
timburkelooks like that `tmp_env = dict(env)` behavior goes *way* back: https://github.com/openstack/swift/commit/eac721b003ae16931f0deb1f1457d5cbca263d7417:38
claygfucking *brilliant* 🤩17:43
timburkelol i was feeling like "this is so stupid and brutish"17:50
zaitcevhttps://blocksandfiles.com/2020/09/24/seagate-cortx-object-storage-software/ Kynetic debacle was not enough for Seagate18:33
openstackgerritTim Burke proposed openstack/pyeclib master: Move liberasurecode-git behavior into all tox envs with compilelibs marker  https://review.opendev.org/75393818:47
openstackgerritTim Burke proposed openstack/liberasurecode master: Be willing to write fragments with legacy crc  https://review.opendev.org/73895919:05
ormandjwhen a HEAD requests fails (hits conn_timeout) on a proxy, due to a busy backend disk, should we not see a retry to another location (3x replication)?19:09
*** openstackgerrit has quit IRC19:14
timburkeormandj, i would definitely expect HEADs to try alternate locations on a timeout19:25
ormandjtimburke: can you think of any reason they wouldn't? i grep the path in the logs and definitely do not see a retry19:26
ormandjthis may be a part of a composite operation like a mass delete or something because i don't see the HEAD in the lb logs19:30
timburkewere you looking at proxy or object logs? or both?19:32
ormandjproxy19:32
ormandjsee the failed HEAD to the ip/port of one of the storage node drives (we're on servers_per_port now)19:33
ormandjconn_timeout (at 20s currently) then no access attempts again for the same object19:33
ormandjlooking at LB logs (in front of the proxies) i don't see any actual HEAD requests for that object so i'm guessing it's part of another operation the proxy is expanding into multiple calls19:35
ormandjthen i look at the storage node logs, and it gets really crazy. i just looked for the object in question, and found zillions of deletes. (storage node that one of the timeouts was for) for that same object, all with the same txn19:41
ormandjscratch that, was different objects, it's definitely part of a mass delete19:43
timburkemight be related to s3api's multi-delete behavior -- i know we do a HEAD before delete to check if it's an MPU (in which case we need to add a flag so SLO sill clean up the segments)19:46
ormandjyeah, i guess that HEAD isn't retried then19:46
timburkei take it the proxy logged a 503 for the HEAD?19:47
ormandjlooking now19:48
ormandjSep 24 19:37:21 BLAH proxy-server: ERROR with Object server x.x.x.x:7021/sdaa re: Trying to DELETE /AUTH_idstuff/veeam/Veeam/Archive/extendedretention/lotsofstuff/00000000-0000-0000-0000-000000000000/blocks/anotherid/lotsofobjectgunk.blk: ConnectionTimeout (20.0s) (client_ip: x.x.x.x)19:50
ormandjdoesn't really log much of anything, unlike normal client requests19:50
ormandjbut if you grep for that object.blk file you'll not see access again after the timeout19:51
ormandji had to go onto the storage node to even find the txn by grepping for the object name20:02
ormandjand once i had the txn i was able to see it was part of a mass delete20:02
timburkewacky -- i would've expected a normal access line like `Sep 24 19:54:07 saio proxy-server: 127.0.0.1 127.0.0.1 24/Sep/2020/19/54/07 HEAD /v1/AUTH_test/bucket/not-really-there%3Fsymlink%3Dget HTTP/1.0 404 - aws-cli/1.18.140%20Python/3.6.9%20Linux/4.15.0-46-generic%20botocore/1.17.63%20StaticWeb - - - - txa04ff5bb91d44b0691143-005f6cf95f - 0.0193 SW - 1600977247.688429356 1600977247.707708120 0`20:02
ormandjtimburke: yeah, exactly, that would have made it a lot easier to hunt down for sure ;)20:02
ormandjit seems like all of these re: mass delete look like this20:03
timburkeis force_swift_request_proxy_logturned on, or off/not present?20:03
ormandjlooking20:03
ormandjno, defaulted to false20:03
ormandjshould that be on?20:03
timburkedepends on what you're interested in i suppose. though if having it off means we don't log proper subrequests, that sure seems like a bug...20:04
ormandjhttps://opendev.org/openstack/swift/src/branch/master/etc/proxy-server.conf-sample#L593  <-- we basically mirror these on every update, then just apply our changes to override things we need to specific to our environment20:05
ormandjwell, we definitely want enough logging to be useful hah, so we can turn that one. the account info would be very helpful in addition to 'fixing' this so we'll flip the bits20:06
timburkemight want to look at log_msg_template -- recently we merged some fixes so that {account} works as you'd hope for s3 requests20:07
ormandjrecently as in ussuri?20:08
timburkevictoria -- https://opendev.org/openstack/swift/commit/b2efd185c20:10
ormandjwell, guess when we do that upgrade we'll reap the benefits, until then, we can do this other one20:11
*** zamba has quit IRC20:41
*** zamba has joined #openstack-swift20:54
*** baojg has quit IRC21:53
*** baojg has joined #openstack-swift21:54
*** baojg has quit IRC21:58
*** baojg has joined #openstack-swift21:59
*** openstackgerrit has joined #openstack-swift22:42
openstackgerritTim Burke proposed openstack/swift master: Add a new URL parameter to allow for async cleanup of SLO segments  https://review.opendev.org/73302622:42
*** rcernin has joined #openstack-swift22:58
*** rcernin has quit IRC23:14
*** rcernin has joined #openstack-swift23:14
« Wednesday, 2020-09-23 Index Friday, 2020-09-25 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!