| openstackgerrit | Tim Burke proposed openstack/swift master: Add some functional CORS tests https://review.opendev.org/533028 | 00:13 |
|---|---|---|
| openstackgerrit | Tim Burke proposed openstack/swift master: s3api: Pass through CORS headers https://review.opendev.org/710330 | 00:13 |
| openstackgerrit | Tim Burke proposed openstack/swift master: s3api: Allow CORS preflight requests https://review.opendev.org/710355 | 00:13 |
| openstackgerrit | Tim Burke proposed openstack/swift master: WIP: s3api: Allow MPUs via CORS requests https://review.opendev.org/720098 | 00:13 |
| seongsoocho | MPUs means Multipart Uploads?? | 01:35 |
| kota_ | seongsoocho: yes | 02:29 |
| seongsoocho | kota_: thanks :-) | 02:29 |
| * kota_ is looking at https://review.opendev.org/#/c/427911/ that is not merged | 02:30 | |
| patchbot | patch 427911 - swift - Replace MIME with PUT+POST for EC and Encryption - 37 patch sets | 02:30 |
| kota_ | I'm not sure if we don't need the patch or just fixed something at eventlet... | 02:30 |
| *** lifeless_ is now known as liffeless | 02:40 | |
| *** liffeless is now known as lifeless | 02:40 | |
| *** psachin has joined #openstack-swift | 03:26 | |
| *** threestrands has joined #openstack-swift | 03:29 | |
| openstackgerrit | Charles Hsu proposed openstack/python-swiftclient master: Support v3 application credentials auth. https://review.opendev.org/699457 | 03:54 |
| *** gyee has quit IRC | 04:02 | |
| *** evrardjp has quit IRC | 04:37 | |
| *** evrardjp has joined #openstack-swift | 04:37 | |
| openstackgerrit | Charles Hsu proposed openstack/python-swiftclient master: Support v3 application credentials auth. https://review.opendev.org/699457 | 04:41 |
| openstackgerrit | Charles Hsu proposed openstack/python-swiftclient master: Support uploading Swift symlinks without content. https://review.opendev.org/694211 | 04:48 |
| openstackgerrit | Charles Hsu proposed openstack/python-swiftclient master: Allow users to list containers/objects by size. https://review.opendev.org/708074 | 05:57 |
| *** early has quit IRC | 06:04 | |
| *** early has joined #openstack-swift | 06:09 | |
| *** ccamacho has joined #openstack-swift | 07:01 | |
| *** rpittau|afk is now known as rpittau | 07:34 | |
| *** dtantsur|afk is now known as dtantsur | 08:28 | |
| *** threestrands has quit IRC | 08:45 | |
| *** tkajinam has quit IRC | 08:50 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift master: Imported Translations from Zanata https://review.opendev.org/720419 | 09:55 |
| timss | timburke: thank you for your response! The auditors were running, altough the object-auditor may be poorly tuned in relation to the number of stored files (100M+) and therefore not touching the files at a reasonable rate. | 10:05 |
| timss | Recent object-auditor log stats: http://paste.openstack.org/show/792213/ | 10:05 |
| timss | Even if this is the case though, how could simply opening the files (open()) during replication lead to XFS corruption (I/O error), and not just quarantined files or errors if there's any bitrot/read issues? | 10:06 |
| timss | Either way it seems to me that at this point that `xfs_repair -L` is our only option for the disks affected. Would really like to find out what the initial trigger for the filesystem corruption is though. | 10:06 |
| timss | Upgrades (and expansion) has been on the agenda for some time, but yeah.. :) Great to hear Xenial has received such good support though! | 10:07 |
| *** rpittau is now known as rpittau|bbl | 10:19 | |
| *** rpittau|bbl is now known as rpittau | 12:07 | |
| *** psachin has quit IRC | 12:16 | |
| *** gyee has joined #openstack-swift | 14:42 | |
| timburke | good news is, since you're on 2.7.0 you should already have https://github.com/openstack/swift/commit/fd86d5a95 -- but yeah, you probably want to tune the auditors better, more than a month between passes is likely to cause troubles | 14:54 |
| timburke | if you haven't already, you might think about turning up the `concurrency` setting so you have an audit process per disk | 14:55 |
| timburke | my working theory is not that the open() caused the xfs corruption, but rather that the bitrot was already there but undetected | 14:56 |
| timburke | i could be wrong though; someone like rledisez has a ton more experience than me in dealing with that sort of thing ;-) | 14:57 |
| timburke | kota_, ah, you were thinking of that patch... afaik that hasn't caused troubles with py3, but if we ever want to move to a golang object server (or really, anything not-eventlet), i think we'll need to address that | 15:02 |
| kota_ | timburke: even on py3 using eventlet would be safe? | 15:03 |
| timburke | yeah | 15:04 |
| kota_ | that's good news, thanks | 15:04 |
| timburke | now go to sleep ;-) | 15:05 |
| kota_ | ;-) | 15:12 |
| * kota_ was drinking at home. | 15:12 | |
| timss | timburke: good thing to know about that commit, cheers. I will look into tuning the concurrency, but should I expect much additional CPU load from this or mainly IO? These servers are a bit disk-heavy which is why I'm curious :) | 15:28 |
| timss | And you may be onto something. I'm guessing this could happen if the bitrot occured in the XFS metadata sector for files that has not been touched for a very long time. Something one has to take note of I suppose, that while Swift is perfectly able to ensure integrity of its data (files) during normal operation, if you're hit by bitrot in the underlying filesystem metadata you may have to get your | 15:29 |
| timss | hands dirty. | 15:29 |
| timburke | yeah, i'd expect an increase in both CPU and IO -- honestly, i'm not quite sure which will dominate, it probably depends on hardware | 15:37 |
| rledisez | timss: depending on your pattern of objects (big objects or small objects) you should expect more IO or more CPU (well, a bit of both). MD5 computation might be CPU intensive if you dont throttle the auditor. But if you manly have small objects, the IO will dominate | 15:38 |
| rledisez | as timburke said, if you're running a large cluster, you should expect XFS corruption on a regular basis, it's just a matter of triggering it (the auditor is a good tool for that) | 15:39 |
| rledisez | timburke: about "anything non-eventlet", do you know if gevent was considered at some point? | 15:42 |
| timburke | rledisez, i've not looked into it | 15:44 |
| timburke | but, y'know, that could be cool too ;-) | 15:45 |
| rledisez | so, I started to look into it on my "spare" time. i'm trying to isolate everything eventlet-related to a module swift.common.coroutines so that at some point I can try to replace it with a gevent version. it's not moving fast as it's not a priority. but gevent is known for its good performance, being more "transparent" than eventlet. and it would be an easier path than a rewrite (and also, it's grpc compatible :)) | 15:47 |
| timburke | https://github.com/gevent/gevent/blob/v1.2a2/src/gevent/pywsgi.py#L517 has me worried already :-/ | 15:48 |
| timburke | wire protocols + py3 = sad panda | 15:49 |
| timburke | bah, got confused by the change in tag format -- but it's still like that on current releases: https://github.com/gevent/gevent/blob/1.5.0/src/gevent/pywsgi.py#L519 | 15:51 |
| timss | timburke: rledisez: sounds reasonable, I'll try to increase it in steps and see how it looks, thanks! | 15:51 |
| rledisez | timburke: It seems reasonable for HTTP, no? I would say that if we only send "correct" HTTP through the wsgi server it should be fine | 15:53 |
| timss | rledisez: XFS corruption in this instance (metadata leading to FS issues and not just silent data bitflips) has not been on my radar up until now but it explains a lot. | 15:54 |
| timburke | https://bugs.python.org/issue33973 -- it means we can't get rid of https://github.com/openstack/swift/commit/93b49c5e4 any time soon | 15:55 |
| rledisez | hum, ok, I see the issue now… | 15:57 |
| timburke | backend servers should be fine, but at the proxy... clients can be terrible | 16:07 |
| *** rpittau is now known as rpittau|afk | 16:18 | |
| *** dtantsur is now known as dtantsur|afk | 16:36 | |
| *** evrardjp has quit IRC | 16:37 | |
| *** evrardjp has joined #openstack-swift | 16:37 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Add some functional CORS tests https://review.opendev.org/533028 | 17:06 |
| openstackgerrit | Tim Burke proposed openstack/swift master: s3api: Pass through CORS headers https://review.opendev.org/710330 | 17:06 |
| openstackgerrit | Tim Burke proposed openstack/swift master: s3api: Allow CORS preflight requests https://review.opendev.org/710355 | 17:06 |
| openstackgerrit | Tim Burke proposed openstack/swift master: WIP: s3api: Allow MPUs via CORS requests https://review.opendev.org/720098 | 17:06 |
| *** rcernin has quit IRC | 17:37 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Add some functional CORS tests https://review.opendev.org/533028 | 17:41 |
| openstackgerrit | Tim Burke proposed openstack/swift master: s3api: Pass through CORS headers https://review.opendev.org/710330 | 17:41 |
| openstackgerrit | Tim Burke proposed openstack/swift master: s3api: Allow CORS preflight requests https://review.opendev.org/710355 | 17:41 |
| openstackgerrit | Tim Burke proposed openstack/swift master: WIP: s3api: Allow MPUs via CORS requests https://review.opendev.org/720098 | 17:41 |
| openstackgerrit | Merged openstack/swift master: Imported Translations from Zanata https://review.opendev.org/720419 | 19:58 |
| *** ccamacho has quit IRC | 20:38 | |
| openstackgerrit | Julien Lutran proposed openstack/swift master: s3api: Add stats for s3api requests https://review.opendev.org/716016 | 20:45 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Add some functional CORS tests https://review.opendev.org/533028 | 20:48 |
| openstackgerrit | Tim Burke proposed openstack/swift master: s3api: Pass through CORS headers https://review.opendev.org/710330 | 20:48 |
| openstackgerrit | Tim Burke proposed openstack/swift master: s3api: Allow CORS preflight requests https://review.opendev.org/710355 | 20:48 |
| openstackgerrit | Tim Burke proposed openstack/swift master: WIP: s3api: Allow MPUs via CORS requests https://review.opendev.org/720098 | 20:48 |
| *** renich has joined #openstack-swift | 22:08 | |
| *** rcernin has joined #openstack-swift | 22:39 | |
| *** tkajinam has joined #openstack-swift | 22:39 | |
| *** renich has quit IRC | 23:08 | |
| *** renich has joined #openstack-swift | 23:09 | |
| *** threestrands has joined #openstack-swift | 23:14 | |
| *** threestrands has quit IRC | 23:15 | |
| *** threestrands has joined #openstack-swift | 23:15 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!