21:00:52 #startmeeting swift 21:00:53 Meeting started Wed Feb 6 21:00:52 2019 UTC and is due to finish in 60 minutes. The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:54 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:56 The meeting name has been set to 'swift' 21:01:03 who's here for the swift team meeting? 21:01:05 o/ 21:01:12 me 21:01:28 hi o/ 21:01:29 o/ 21:01:37 hello o/ 21:01:46 mattoliverau: tdasilva: clayg: ping 21:01:49 zaitcev: ping 21:01:57 what 21:01:59 welcome back mr. notmyname ! 21:02:08 it's good to be back 21:02:18 hello all 21:02:20 also, I don't think you've ever called me mr before. please stop ;-) 21:02:49 let's get started then. lots to go over this week 21:02:55 #link https://wiki.openstack.org/wiki/Meetings/Swift 21:03:20 I've been gone a couple of weeks, so I've been trying to catch up with whatever I missed 21:03:37 so if you had a question or needed me for something and I haven't responded, please ping me again 21:03:52 #topic housekeeping 21:04:03 a few general things up front 21:04:10 first, big news! 21:04:36 rledisez has agreed to be a core reviewer. (that happened about 2 minutes before the meeting started, so I haven't had a chance to click the buttons yet) 21:04:44 yay! \o/ 21:04:58 congrats! 21:05:03 woot! 21:05:07 welcome aboard rledisez! 21:05:11 congrats :! 21:05:36 thx for trusting me, i'll try to reach your high levels of skills on swift 21:05:41 o/ 21:05:59 rledisez: congrats! 21:06:48 rledisez: I'm happy you agreed, and I know you'll continue to do great to help manage and maintain the project 21:07:11 rledisez: congrats! 21:07:21 (now the other housekeeping stuff to talk about will be much less exciting) 21:07:31 notmyname: I will. should I swear somewhere that i'll protect the codebase and so on? :D 21:07:57 i didn't take the time while I was traveling to get the stable branches tagged, so I'm still working on that this week. it should be Real Soon Now (tm) 21:08:04 rledisez: lol 21:08:40 so expect that soon (and if you see a backport that doesn't have a review on it, that would be really helpful) 21:09:37 I'd like to propose a new thing for us to do to help manage patches (it wasn't my idea originally, so I know it's pretty good ;-) 21:09:43 we've got the priority reviews page 21:09:51 i think there's just https://review.openstack.org/#/c/629402/ -- i keep meaning to take a look at it... 21:09:52 patch 629402 - swift (stable/pike) - object-server can 409 in response to x-if-delete-at - 1 patch set 21:10:05 but we've also got a lot of patches that people have been running in production before they have landed upstream 21:10:14 I added a new section to the priority reviews page 21:10:18 #link https://wiki.openstack.org/wiki/Swift/PriorityReviews 21:10:25 so that we can write those patches down there 21:10:52 makes sense 21:11:05 I do *not* think that patches listed there should necessarily be higher priority than others, but as reviewers, we can definitely consider the fact that they meet the "works in prod" bar 21:11:18 that's not nothing! 21:11:38 so.... if you've got patches that your'e running in prod that haven't landed ( rledisez and alecuyer, I think you've got some), please update the wiki with a link to them 21:11:39 how should we fill this page? should we just put links (with irc nicknames), or do we talk about it here first? 21:11:50 rledisez: just update. no need to wait for a meeting 21:11:51 ok, i got my answer ^ 21:11:52 i get the impression https://review.openstack.org/#/c/633671/ maybe belongs there... 21:11:53 patch 633671 - swift - Fix decryption for broken objects - 6 patch sets 21:12:07 yeah, is that the one that zigo is running? 21:12:12 yeah 21:12:23 great! so we've got stuff to fill in already :-) 21:12:38 ok, last bit of housekeeping on the agenda is to raise awareness of the state of our gate 21:12:52 timburke has been looking at this lately and saying our functests are bad and we should feel bad 21:13:02 and zaitcev (?) found an issue today with probe tests 21:13:08 it's sad-face. but hopefully getting better? 21:13:16 I think it was someone else 21:13:49 like I said, i've just been trying to catch up, but it's something that seems to be a problem recently. timburke, didn't you say 13 of 15 reviews you left were "recheck"? 21:13:53 something crazy like that? 21:14:04 somewhere around there 21:14:12 https://review.openstack.org/#/c/634456/ should definitely help, though; https://review.openstack.org/#/c/634976/ too 21:14:13 patch 634456 - swift - func tests: Be willing to retry PUTs (MERGED) - 3 patch sets 21:14:14 patch 634976 - swift - Fix flakey symlink setup/teardown (MERGED) - 1 patch set 21:14:20 so that's the bad news (flakey tests in the openstack gate). but the good news is that it's all in our own tests! this means we can actually fix them pretty easily 21:14:30 wooooohoooo! 21:14:35 i also put up https://review.openstack.org/#/c/634362/ and https://review.openstack.org/#/c/634449/ 21:14:35 patch 634362 - swift - Fix flakey func test setup - 1 patch set 21:14:36 patch 634449 - swift - Fix flakey func test teardown - 1 patch set 21:14:46 notmyname: how'd you phrase it? "The best possible bad news"? 21:14:55 heh 21:15:12 timburke: would you say most of these are because the test environment is pretty resource constrained? or is it just bad test code? 21:16:07 "easily" -- one of the func test failures i observed was assertion failures about container listings following an object PUT... i'm still not sue how best to address that... it's an eventually consistent system! 21:16:15 some were just bad test code - like we'd assume a 201 on a result that retries on 503 where the second attempt might 202 - but I mean... you don't notice that stuff when everything "just works" 21:16:36 pretty sure it's mostly because of resource constraints in the gate 21:17:36 if anyone is wondering how they can help, first step would be to review some of the patches that timburke linked. otherwise, rerunning stuff in your saio and finding intermittent errors and digging in to them is good 21:18:05 i don't think i've *ever* gotten a 503 in a functest on my laptop, even when the fans make it sound like it wants to take off 21:18:31 ok, I'm at the end of my housekeeping list. any other general info things to bring up before we move on to a coupe of other topics? 21:18:41 do you know the configuration of the test environment (CPU, RAM) ? (On SAIO I don't think I ever get errors on my laptop, but i could lower the VM specs) 21:19:25 honestly I don't. maybe we could find it in logs. or ask clarkb or fungi 21:19:35 ok will do 21:19:41 things like http://logs.openstack.org/71/633671/6/check/swift-dsvm-functional/28d266d/zuul-info/inventory.yaml might have some semi-useful info? 21:20:10 aha! "cloud: ovh" -- it's *your* fault ;-) 21:20:18 alecuyer: lowering your VM stats is likely a great idea. or maybe you know of a hosting provider that would give you a lot of small, cheap VMs.... ;-) 21:20:23 woops :-) 21:20:29 lol 21:20:38 * clayg snickers 21:20:40 I will try that and tell you what results I get 21:20:48 i take that back -- that one passed 21:21:06 ok, let's move on 21:21:21 #topic losf feature branch 21:21:37 last week, kota_ and tdasilva and rledisez and alecuyer all met in europe and talked about LOSF 21:21:45 from what I hear, it was great 21:22:06 +1 21:22:09 +1 21:22:15 rledisez and alecuyer want some feedback from the community, but they also need to write some more docs and tests 21:22:15 yes it was :) 21:22:33 kota_/NTT has been running the code and testing it out, too 21:22:45 thx again kota_ for the effort you and your colleague put into testing losf 21:23:02 yup 21:23:04 so, from what I gather, the question was raised about making a formal feature branch for tracking it (instead of keeping it in a few existing gerrit patches) 21:23:32 currently the newest code is at https://github.com/alecuyer/swift/tree/master-losf 21:23:33 actually, right now, it mostly happen on the github of alecuyer 21:23:43 kota_: was there slides!? tdasilva do you have slides!? 21:23:45 that follows recent master branch in swift upstream 21:23:47 the specific question I have for this meeting is "do we need to make feature/losf?" and more generally, what's the state of things 21:24:06 rledisez started a nice doc: https://docs.google.com/document/d/1KoLsqWiXv9u2rnMZ5Rh73Js-lwP5h2qRj9CuFJ2IJJc/edit# 21:24:10 NTT slides are awesome :-) 21:24:23 noice 21:24:41 alecuyer wrote the doc, I just read it ;) 21:24:47 clayg: I have, but not published. I'll send you if you want to look at. 21:24:52 sorry, i meant alecuyer! 21:24:53 #link https://docs.openstack.org/infra/manual/testing.html has details on our test environments 21:24:56 rledisez: any chance that's linked on the ideas page? errr... I mean that for alecuyer then 21:24:56 Group effort :) Kota also wrote some parts of it 21:25:00 fungi: thanks 21:25:03 kota_: yes pls 21:25:23 kota_: even better, link to it on the ideas page! 21:25:27 it is 21:25:27 notmyname: Yes I think I did, checking (on one existing line) 21:25:39 great 21:25:43 search for "losf design doc" 21:25:52 clayg: alright, i'll send by E-mail. I'm not sure it could be okay to be public. 21:25:54 #link https://wiki.openstack.org/wiki/Swift/ideas 21:26:02 yesir 21:26:18 alecuyer: are we still likely to want something along the lines of https://review.openstack.org/#/c/561631/ as a pre-req? 21:26:19 patch 561631 - swift - WIP - abstract FS operations in replicator/reconst... (ABANDONED) - 2 patch sets 21:26:40 kota_: from OVH point of you, it can be public, we have nothing to hide on LOSF (code is already public so…) 21:27:04 timburke: indeed, currently I have that as the single pre-req on my branch. I still remember we discussed how it was not great ;) but currently, we depend on it 21:27:07 rledisez: exactly, it is. 21:27:48 alecuyer: I also heard that FOSDEM resulted in some promising stuff with regards to grpc and eventlet/golang 21:28:03 ORLY? 21:28:04 i really like the core *idea* -- just wanted to think more about how to build the abstraction 21:28:11 clayg: YARLY! 21:28:32 Yes somebody working on ember-csi has a workaround, which I was looking at earlier tonight, but not done with that and haven't tested, but it is promising 21:28:35 but, woking code > ideas 21:29:13 something to discuss in another time but here it is :https://github.com/embercsi/ember-csi/blob/master/ember_csi/workarounds.py#L46 21:29:20 alecuyer: why container storage abstraction be messing with eventlet? 21:29:29 clayg: "openstack", IIRC 21:29:32 oh yeah, alecuyer i was supposed to connect you online with geguileo 21:29:33 clayg: grpc messing with eventlet 21:29:45 he hangs out on #openstack-cinder 21:30:03 tdasilva: and that person was going to patch eventlet? or grpc? there were going to be patches upstream somewhere, right? 21:30:08 tdasilva: he showed me the code at fosdem on his phone 21:30:18 notmyname: he says he will propose a patch upstream in grpc 21:30:29 notmyname, alecuyer: yep 21:30:30 nice! that's great news 21:30:38 well it says native threads - so you know it's good 21:31:10 nice 21:31:25 ok... so do we need a feature/losf branch? or do we need to wait for a couple of other things first? or some other idea? 21:31:54 IMO, it sounds like with 2 companies working on it, and some promising work otherwise, it might be time to make the feature branch 21:32:09 notmyname: is there strict requirements for having a feature branches? (and what would it be) 21:32:20 I supports creating a branch because it helps us to follow the newest master at least 21:32:39 Well it'll be better to have the code closer to the project, tho FB come with a bunch of overhead and regular master merging 21:32:47 the downside is that we know it basically needs a "core sponsor" (likely kota_ at this point, and rledisez as he comes up to speed) just to make sure it doesn't get too stale 21:32:53 I also prefer to see it on gerrit than on github, just to have all the tests running etc… 21:33:11 rledisez: there's no strict requirements. just the social constraints we ourselves put on it 21:33:50 but I don't want a feature branch if nobody is going to look at it for a few months and nobody is going to keep it rebased against master 21:33:56 *that's* the big cost 21:34:12 ooh yeah... making sure we know how to run with losf *in the gate* is a great reason to have a branch... 21:34:26 feature branch == merge commit at some point yeah? if we want to split it up to cleanup/code/doc changes the merge commit that bring them in together has a small, but non-zero value 21:34:30 so "do we make a feature branch" is basically asking "are we ready to work on it as a group and move it towards master" 21:35:09 it's not proposed-master branch yet in the feature branch, is it true? 21:35:14 kota_: correct 21:35:27 this would be a development feature branch. not a review branch for an actual merge 21:35:40 i think the past feature branch is to speed up the new change set w/o taken care of the master broken. 21:35:41 oic 21:35:47 yup 21:36:05 my concern is making sure someone will be able to continually thing about it as it relates to master (including continually merging master into it, or rebasing) 21:36:14 * mattoliverau is a little distracted, and needs to take the toddler to a swimming lesson (first one so I need to be there or she won't get in) +1 to FB so long people are working on it.. I'd like to have it closer and a 1st class work (IE with Gerrit and zuul) 21:36:31 we have said last week with rledisez that I will try to keep this rebased on master weekly 21:36:32 mattoliverau: understood 21:36:37 I always hated feature branches, and hummingbird collapsed, but acoles made it work for encryption, so whatever 21:36:41 the point about having gate tests with a feature branch is great 21:36:45 (wherever it lives) 21:37:24 zaitcev: what's the downside of the feature branch? 21:37:25 zaitcev: hummingbird's issues were 99% social, not technical. and it's not "bad" that it didn't get merged. all the other feature branches we've done have ended up with pretty great things included in swift 21:37:26 and we had sharding... and s3api... 21:37:39 Yes, sharding 21:38:12 clayg: mostly that I am too stupid to figure it out 21:38:18 haha! 21:38:26 oic, s3api also had the feature branch 21:38:35 alecuyer: one logistical issue with a feature branch is that only a core reviewer can propose merge commits to it. you don't rebase the branch against master. master is merged in every so often 21:38:51 like, how to add my own review and then not make it so git-review overwrites other reviews 21:38:53 i think there was another before sharding 21:38:58 kota_: yeah... i think we skipped the review branch, though? 21:39:02 storage policies were the first 21:39:11 tdasilva: storage policies? 21:39:14 yep 21:39:20 notmyname: right, I'll have to learn how to work properly with that 21:39:26 ok ok, here's the full doc on it ;-) 21:39:27 #link https://wiki.openstack.org/wiki/Swift/feature_branches 21:39:40 thanks :) 21:39:45 timburke: oh, might be. only one big patch proposed to the master maybe. 21:40:38 zaitcev: it should work more-or-less like master, particularly on the feature (as opposed to review) branch 21:40:40 based on who's been involved in LOSF so far, the core maintainer doing merges from master would likely be kota_. and rledisez could start doing it soon, too. 21:40:56 so let me rephrase the question 21:41:50 I might make the effort, imo. Also I may need the rledisez's help tho. 21:41:53 question in two parts (1) kota_ and rledisez are you willing to watch over a feature/losf branch and merge master in to it once every week or two? (2) for everyone else, are you ready to more actively watch the losf work and participate in discussing it? 21:43:47 notmyname: tbh, I have no idea how much effort it takes, but as it's a goal for OVH to have it merged, I should take the time needed for that. with the help of kota_ I think we can manage to merge master on a regular basis. for the other part, i'll need to read the doc 21:43:57 I'd certainly be ready to start advocating that since kota and rledisez are going to be working on the feature branch we NEED to more actively watch the losf work and participate in discussing it 21:43:58 rledisez: kota_: got it 21:44:26 clayg: got it 21:44:36 and timburke sitting next to me, got it (he said the same thing) 21:44:38 it'd be great to have a better picture of what that means realistically by the time we get to denver 21:44:58 i mean... unless it was gunna be done by then? 21:45:00 clayg: that is an excellent point 21:45:09 plus, this definitely falls under the "Open patches that are running in production somewhere" heading... 21:45:17 clayg: lol... right ... alecuyer: right? done by denver? ;-) 21:45:21 clayg: i like your optimism ;) 21:45:27 k 21:45:29 yup, we have a couple of month in Denver. 21:45:33 notmyname: hehe sure ;) 21:46:01 ok, I'll get the feature branch set up this week 21:46:04 timburke: i was shy to add it on the priorityreviews page, but I'll do it if you say so ;) 21:46:19 thanks notmyname 21:46:23 wooooo!!! FB time 21:46:27 notmyname: thanks 21:46:28 ok, moving on 21:46:34 one more topic in the last 15 minutes 21:46:35 I'm doing the FB dance - y'all are missing out 21:46:45 (thanks, everyone for being patient) 21:46:58 #topic quadiron for libec 21:47:09 vr42: thank you for being patient with us. we had a lot to cover 21:47:10 hello 21:47:18 no worries 21:47:20 tdasilva: can you into vr42 to us please? and the topic? 21:47:30 sure 21:48:06 everyone, vr42 is from Scality and he got in touch asking us about the possiblity of adding QuadIron to libec 21:48:28 nice 21:48:32 #link https://github.com/scality/liberasurecode/tree/feat/quadiron-fnt 21:48:37 QuadIron is an EC lib designed for managing a large number of parities 21:48:46 cool 21:48:50 this is the original project page: https://github.com/scality/quadiron 21:49:00 some slides here: https://www.slideshare.net/Scality/quadiron-an-open-source-library-for-number-theoretic-transformbased-erasure-codes 21:49:11 #link https://github.com/scality/quadiron 21:49:12 thanks 21:49:16 #link https://www.slideshare.net/Scality/quadiron-an-open-source-library-for-number-theoretic-transformbased-erasure-codes 21:49:21 (for the meeting notes) 21:49:26 ok 21:49:32 one interesting use case is for instance: 21:49:39 take your data, split in 100 21:49:50 generate 200 parities in systematic mode 21:49:59 that means you have 100 parities in clear 21:50:04 100 parities encoded 21:50:06 100+200!? wow 21:50:15 100+100 21:50:18 oic 21:50:22 for an overhead of 2, you can lose 100 drives 21:50:25 now you really need that FB, huh ;) 21:50:30 But we can't store several fragments on one device, right? Our naming scheme does not allow it if partition number is the same. Or does it 21:50:30 kota_: lol, "only" 200 total ;-) 21:50:56 we do support systematic and non-systematic 21:50:57 zaitcev: in this case, I don't think using it in swift is as much of a concern. they're "just" integrating with libec for now 21:51:16 yes, liberasurecode is a nice abstraction for us to integrate 21:51:34 vr42: I think it's awesome that you're integrating with libec 21:51:40 that's exactly why it was written 21:52:00 I've got a few questions (from a maintainer perspective, not ec/tech perspective) 21:52:06 yes 21:52:30 zaitcev: we put the frag_index in the on-disk filename - node can have multiple frags if stuff has to move around but the proxy won't ever write them down that way. 21:52:40 first off, what actually needs to be changed in the code to add quadiron? libec should probably allow new libraries without needing to change it's own code (but if not, that should probably be changed anyway) 21:53:10 I had to change a few things 21:53:14 actually for the better 21:53:20 I think 21:53:38 second, since we maintain libec, how do we maintain quadiron integration? how do we test it? can gate tests be added? what sort of lonter-term participation is scality envisioning? 21:53:39 if you want to look at the commit log: https://github.com/scality/liberasurecode/commit/1d414d625ce6b8117e6f2267ea05406cd1e46934 21:53:50 hard coded k+m upper limit, I suppose at leaset. 21:53:57 yes I changed that 21:54:19 and polished the APIs required to manage vectors of fragments rather than bitmaps (it was already there, just polished it) 21:54:29 seems like it might be a good time to address https://github.com/openstack/liberasurecode/blob/master/src/erasurecode.c#L614 ... 21:54:55 third, are there any concerns about legal entangelments with quadiron? patents, trademarks, copyrights, licenses, etc? 21:55:15 no as far as we know it is patent free 21:55:21 nice 21:55:35 we use a basic multiplicative FFT on a prime field 21:55:56 it is known in the field for decades 21:55:57 how do we test the integration? 21:56:09 is it possible to integrate gates tests for it? 21:56:11 so, QuadIron itself is tested by CircleCI 21:56:29 it's open source? 21:56:32 we have a fork of liberasurecode, we can also set a CircleCI on it 21:56:37 ok 21:56:47 QuadIron is open-source 21:56:59 so the merged fork would have zuul tests in it so they can run with the other stuff in the gate (right?) 21:57:34 I have a question: in our PR, we have 2 different change sets 21:57:50 1) is the minimal changes we had to do to libec to make it working (polishing, etc) 21:57:59 2) is the quadiron backend code itself 21:58:12 should we make 2 PRs ? or 2 different commits in the same PR ? 21:58:15 notmyname: we've done that sort of thing before (though after the fact), for isa-l and jerasure: https://review.openstack.org/#/c/604391/ 21:58:16 patch 604391 - liberasurecode - Install Jerasure and ISA-L libs (MERGED) - 5 patch sets 21:58:35 timburke: +1 21:59:02 vr42: does the first patch make sense on its own? would you have proposed it anyway? 21:59:07 vr42: liberasurecode is maintained under openstack namespace that means github's pr is not a way to push to upstream. 21:59:21 it's just a note. 21:59:39 personally, I love the idea of integrating more ec libraries into libec. I think we should encourage that all the time. I'd love for scality to contribute this. 21:59:53 notmyname: +1 22:00:00 I actually found a bug here https://github.com/scality/liberasurecode/commit/1d414d625ce6b8117e6f2267ea05406cd1e46934#diff-47d435f29c8b3e47130b1fbf03f1ef60R226 22:00:00 does anyone else have concerns or questions about the mechanics of adding quadiron support to libec? 22:00:10 for backends with headers 22:00:30 it was a case not managed 22:00:30 vr42: if proposed as two commits, it'll create two reviews (essentially, PRs) in gerrit, with the second depending on the first. that's fine. if you'd prefer it as a single commit, i'm sure we can work with that, too 22:00:38 vr42: as a reviewer I do prefer a user or caller, even a sample one, to be there for an api change. If you need a new method, for instance, it needs an explanation how it's used at the very least. But other than this, if you can separate cleanups into a separate commit, that's better. 22:00:39 ok 22:00:39 vr42: if the first patch set would be good anyway, then it should likely be two separate patches. if not, then it should likely be squashed 22:01:03 zaitcev: I agree 22:01:23 ok 22:01:41 we're running over our time a little bit, so let me try to sum up what happens next 22:01:43 good docs on adding dependency in gerrit: https://docs.openstack.org/infra/manual/developers.html#adding-a-dependency 22:01:52 vr42: ^^^ 22:01:58 ok thanks 22:02:08 I'm not hearing any major concerns with scality proposing this integration (in fact, people are suggesting helpful ways to do it well) 22:02:25 so vr42 should make the patch proposals to libec 22:02:41 one thing the community will be looking for is the ability to test this code in the gate 22:02:44 ok nice 22:03:00 vr42: let me know if you need help the ansible gate jobs, but what timburke linked above should be a good pointer 22:03:08 and for the time being vr42 should consider the quadiron integration idea as a welcome one 22:03:29 +1 22:03:33 tdasilva: I will reach out to you 22:03:42 tdasilva: in case 22:04:04 If you have any questions on QuadIron itself, feel free to ask 22:04:05 are there any other questions about this? or concerns you'd like to raise now? 22:04:38 in forum.zenko.io 22:05:09 thanks folks 22:05:10 of course, if someone wakes up tomorrow and has another question or concern about it, feel free to pm me about it 22:05:34 vr42: thanks for coming (and being patient). and I'm looking forward to the quadiron integration! 22:05:38 #topic other 22:05:45 anything else from anyone else this week? 22:05:52 (yes, we're 5 minutes over on time already) 22:06:01 x up if you've got something to say 22:06:07 (ie type an x) 22:06:21 all right, let's call it done 22:06:34 thank you everyone from coming this week. thank you for your work on swift! 22:06:38 #endmeeting