21:00:13 <timburke> #startmeeting swift
21:00:14 <openstack> Meeting started Wed Mar  4 21:00:13 2020 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:15 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:17 <openstack> The meeting name has been set to 'swift'
21:00:21 <timburke> who's here for the swift meeting?
21:00:28 <mattoliverau> o/
21:00:29 <kota_> hi
21:00:32 <seongsoocho> o/
21:00:35 <clayg> I/
21:00:43 <alecuyer> o/
21:00:47 <rledisez> hi o/
21:01:04 <tdasilva> o/
21:01:12 <timburke> agenda's at https://wiki.openstack.org/wiki/Meetings/Swift
21:01:32 <timburke> i haven't got much planned, so straight to updates!
21:01:39 <timburke> #topic concurrent EC
21:01:44 <timburke> clayg, how's it going?
21:02:14 <clayg> Ok, I’ll push up my wip after the meeting. Folks can see t new approach.
21:03:02 <alecuyer> teasing us
21:03:07 <clayg> I’ll probably do a new change?  Might be easier to A/B
21:03:36 <timburke> do you think it'll be in a ready-to-merge state, or still largely experimental?
21:03:48 <timburke> (asking to set expectations, as much as anything)
21:03:52 <clayg> alecuyer: no, it’s very WIP a lot of duplication that’ll need to be removed after we get it working
21:04:13 <clayg> Not ready to merge.
21:04:15 <timburke> 👍
21:04:23 <clayg> I’ll want some feedback
21:04:48 <timburke> anything else you need now?
21:05:03 <clayg> Not now. Next week. 👍
21:05:19 <timburke> sounds good
21:05:20 <timburke> #topic 503 delays
21:05:37 <timburke> i wasn't sure if there was any other discussion needed here or not
21:05:51 <clayg> I’m good on this topic.
21:06:13 <clayg> I think the whole idea is hostile to well behaved clients
21:06:21 <clayg> I’m over it.
21:07:41 <timburke> i guess the idea is that the client should ensure its own request smearing?
21:08:31 <rledisez> i still see a point in it cause as a public cloud operator, we have no control on clients
21:08:47 <timburke> my main questions are, would something as simple as sleep(random.random() * conf.max_unavailable_wait) work? defaulting it to zero, of course. and where would we want it to live? in the proxy, or somewhere in middleware?
21:09:10 <timburke> in the proxy-server app, i mean
21:09:43 <rledisez> I would say on the left of the pipeline because many things can happen without proxy-server being involved
21:10:14 * timburke nods
21:10:45 <timburke> probably ratelimit (for separation of concerns) or catch_errors, yeah? or its own thing
21:11:23 <clayg> ratelimit seems reasonable
21:11:24 <rledisez> ratelimit makes sense
21:12:40 <mattoliverau> +1
21:12:49 <timburke> oh yeah, i should look at the intereaction between ratelimit and s3api some more...
21:13:00 <clayg> Hahaha
21:13:38 <timburke> i think it's not *terrible* following https://review.opendev.org/#/c/704659/ ?
21:13:38 <patchbot> patch 704659 - swift (stable/stein) - s3api: Better handle 498/429 responses (MERGED) - 1 patch set
21:14:02 <timburke> er, https://review.opendev.org/#/c/697535/ for the master version
21:14:02 <patchbot> patch 697535 - swift - s3api: Better handle 498/429 responses (MERGED) - 1 patch set
21:14:09 <clayg> Merged!
21:14:15 <timburke> anyway
21:14:18 <timburke> #topic losf
21:14:27 <clayg> I remember when things merged.
21:14:29 <timburke> rledisez, how's it going?
21:14:42 <rledisez> fine I guess, just an update about what's going on for LOSF
21:14:59 <rledisez> as discussed at the last PTG in Shangai, the topic is not really lots of small files, but instead lots of big disks (having a consequence lots of files)
21:14:59 <rledisez> this topic is back on our short-term roadmap cause we are experimenting with new hardware (36x14TB, 2 SSD, 96GB of RAM)
21:15:08 <clayg> 🍿
21:15:38 <rledisez> we examined different option that could replace LOSF without having to maintaining them:
21:15:38 <rledisez> XFS with rtdev option => not stable
21:15:38 <rledisez> XFS with Intel CAS => not stable
21:15:38 <rledisez> ZFS with metadata device => too much space needed for metadata + few operational issues
21:15:41 <kota_> wow
21:16:25 <rledisez> in the end, we concluded that LOSF stays the best option. we made some changes that was discussed in Shangai:
21:16:25 <rledisez> - new key format so that we can save a lot of CPU when doing replication
21:16:25 <rledisez> - new option to store the LevelDB in a new path (on a fast device like SSD/NVMe)
21:16:25 <rledisez> - store metadata in the LevelDB (TODO)
21:16:46 <rledisez> and when I say we made some change, I mean alecuyer did it ;)
21:17:15 <timburke> go alecuyer!
21:17:29 <timburke> that all sounds great :D
21:17:47 <rledisez> that's it for me on LOSF
21:18:04 <alecuyer> heh, I guess :) So, I don't want to spend meeting time now on this ,but if anyone may be hitting these same issues we have, and want to guide the design/dev process, let's talk
21:18:06 <clayg> It’s back on!!!
21:18:24 <rledisez> so we're still involved in it, we just had to figure some stuff first
21:18:59 <timburke> what were the issues you were seeing with the new boxes that brought it back around? RAM constraints still?
21:19:32 <rledisez> yeah, the pattern will be the same => small files. if it was not working on 36x6TB, it will for sure not work on 36x14TB
21:19:38 <alecuyer> inodes not fitting in the VFS cache
21:19:51 <rledisez> also IO starvation because of replication because of cache miss in the VFS cache
21:20:36 <timburke> make sense
21:20:38 <rledisez> We have around 3 millions of objects per TB of disk. You can imagine the number of inode it implies (around 10M per TB of disks)
21:20:59 <rledisez> well, maybe 8M, not 10
21:21:26 <timburke> #topic open discussion
21:21:38 <timburke> anything else to bring up today?
21:22:07 <rledisez> zaitcev: I got your message, I loaded the two reviews. will have a look this week, I promess :)
21:22:33 <zaitcev> rledisez: so, what can I help you with, then? You have any reviews for me?
21:22:55 <timburke> oh, this was an interesting thread: http://lists.openstack.org/pipermail/openstack-discuss/2020-March/012950.html
21:23:01 <zaitcev> Hopefully not the whole LoSF
21:23:20 <clayg> Do we need to auto retest failed tests in the gate before we fail the job?
21:23:25 <rledisez> zaitcev: yeah, that would be unfair trading
21:23:27 <kota_> oh what
21:23:40 <tdasilva> rledisez: are you still investigating those proxy level performance improvements?
21:23:52 <timburke> clayg, yeah, i should look into that. i'm pretty sure there's a way to configure that...
21:24:16 <rledisez> tdasilva: yes, I had to pause for the last 2 weeks, busy doing OVH-stuff, but I want to get back on it next week
21:24:56 <tdasilva> rledisez: cool, looking forward to it....
21:24:58 <rledisez> tdasilva: I already have a review "ready" (I have a weird eventlet random failure in test). I'm now on MD5 replacement
21:25:21 <tdasilva> rledisez: can you share gerrit # ?
21:25:25 <rledisez> timburke: I admit I skipped the thread. in few words, any conclusion came out?
21:26:49 <rledisez> tdasilva: https://review.opendev.org/#/c/697653/ actually this one is OK, I just need to rebase it
21:26:49 <patchbot> patch 697653 - swift - Replace all "with Chunk*Timeout" by a watchdog - 5 patch sets
21:26:51 <timburke> no real conclusion, just putting forward an idea of spreading the PTL responsibilities more (in part due to concerns about projects not having any self-nominees)
21:27:02 <timburke> i don't expect *that* to be an issue for us ;-)
21:27:26 <rledisez> tdasilva: this one has the random failure : https://review.opendev.org/#/c/704892/
21:27:26 <rledisez> running the test individually pass, running all of them at once fails
21:27:26 <patchbot> patch 704892 - swift - proxy: stop sending frags to PyECLib with a Queue - 1 patch set
21:27:43 <timburke> right! i meant to dig into that but never got to it...
21:28:30 <zaitcev> sounds like a challenge
21:29:28 <timburke> so, i just realized we might want to poke more at libec -- see if we can avoid needing to do matrix inversion on every call to decode...
21:29:56 <timburke> should be eminently cacheable
21:30:00 <kota_> hmm
21:30:48 <timburke> https://github.com/openstack/liberasurecode/blob/master/src/backends/isa-l/isa_l_common.c#L211-L222 seems like we shouldn'
21:30:59 <timburke> t need to do it on every call
21:31:25 <tdasilva> just fyi, Vianney pinged me recently about quadiron lib patches, they are still waiting for reviews. I've been lacking on that, so if anyone has a chance
21:31:33 <tdasilva> s/lib/libec
21:32:01 <tdasilva> i think patch chain starts here: https://review.opendev.org/#/c/635603/1
21:32:01 <patchbot> patch 635603 - liberasurecode - fix: data access when having non-zero metadata size - 1 patch set
21:32:44 <timburke> my trouble is that when i start doing libec reviews, i end up wanting to rewrite sizable parts of it all ;-)
21:33:40 <rledisez> did anyone already looked into the glacier api, maybe to add it to s3api?
21:33:59 <zaitcev> Sounds ambitious.
21:34:05 <timburke> oh, and i should reach out to libphazr... i think that first patch would break them, but they don't have any CI set up with us...
21:38:15 <timburke> all right
21:38:31 <timburke> thank you all for coming, and thank you for working on swift!
21:38:36 <timburke> #endmeeting