21:00:01 #startmeeting swift 21:00:04 hello, world 21:00:06 Meeting started Wed Mar 22 21:00:01 2017 UTC and is due to finish in 60 minutes. The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:07 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:09 The meeting name has been set to 'swift' 21:00:12 Hello. 21:00:12 who's here for the swift team meeting? 21:00:15 hello 21:00:15 o/ 21:00:17 o/ 21:00:21 hello! 21:00:26 o/ 21:00:29 o/ 21:00:33 hello o/ 21:00:34 hi 21:00:36 hello 21:00:55 o/ 21:01:41 welcome 21:01:45 agenda is at... 21:01:48 #link https://wiki.openstack.org/wiki/Meetings/Swift 21:01:58 lots of bullet points, so let's see how fast we can go 21:02:06 #topic follow-up from last week 21:02:18 a few things we talked about last week had some follow-up for this week 21:02:39 https://review.openstack.org/#/c/444604/ has landed, closing https://bugs.launchpad.net/swift/+bug/1657246 21:02:39 Launchpad bug 1657246 in OpenStack Object Storage (swift) "Proxy logs wrong request method when validating SLO segments" [Critical,Fix released] - Assigned to Janie Richling (jrichli) 21:02:54 thanks timburke jrichli mahatic and zaitcev for that 21:03:10 I'll get a backport for that proposed soon 21:03:42 also, rledisez (and alecuyer) said they'd get an etherpad started so we can discuss object server compat testing 21:03:46 and that's done too 21:03:51 #link https://etherpad.openstack.org/p/swift-object-server-tests 21:04:38 we asked for review on https://review.openstack.org/#/c/445160/ since it introduces a new api semantic (and it's a lot easier to get it right the first time) 21:05:10 we got some reviews (from clayg and rledisez), but please keep an eye on it as thurloat works on it 21:05:29 and lastly... 21:05:43 https://etherpad.openstack.org/p/composite_rings is for discussion about the best way to expose composite rings 21:05:56 there's some discussion going on there, and it would be good for everyone to read over it 21:06:02 (and leave comments!) 21:06:17 that's all the follow-up stuff I had. did I forget anything? 21:07:03 ok, I'll take that as a "no" 21:07:08 #topic FYI things 21:07:19 a few things to be aware of (and participate in) 21:07:46 timburke: you've noticed some nightly gate failures recently. can you share any more about that? any links? 21:07:54 or ideas on finding the issues? 21:08:04 timburke: yeah, are there more stable failures? 21:08:29 no clue on the reasons. but there's been like 5 this month? 21:08:29 are just like flakey unittests that we already patched on master? 21:08:30 stable failures? 21:08:38 around tox py27 xfs? 21:09:06 http://logs.openstack.org/periodic-stable/periodic-swift-python27-newton/fc260fd/ 21:09:14 http://logs.openstack.org/periodic-stable/periodic-swift-python27-newton/51cf2e4/ 21:09:20 http://logs.openstack.org/periodic-stable/periodic-swift-python27-newton/aa54677/ 21:09:27 http://logs.openstack.org/periodic-stable/periodic-swift-python27-mitaka/50c0108/ 21:09:35 http://logs.openstack.org/periodic-stable/periodic-swift-python27-newton/687f8d9/ 21:09:50 timburke: how did you see/find these? is there a graph or list or soemthing somewherE? 21:09:58 for reference, we would previously see one failure roughly every two months 21:10:51 notmyname: go subscribe to http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-stable-maint then set up filters so everything not-swift is automatically marked as read? 21:10:54 ah, it's in stable branch with periodic check 21:11:03 timburke: ah 21:11:10 maybe it is an intermittant falure. I can start running py27 unit tests on newton in a loop, but assume your already doing that thing 21:11:54 mattoliverau: actually, I think that would be really helpful 21:12:12 I'm not sure that timburke has done much more than read the test output 21:12:15 kk, I'll start setting it up now :) 21:12:19 mattoliverau: thanks :-) 21:12:36 timburke: thanks for noticing and bringing it up 21:12:43 +100 21:12:55 other FYI things... 21:13:04 as much as anything, i just figure it'd be good if people took a look at the failures and report back if they see anything familiar. "oh yeah, i remember tracking down that intermittent failure in patch X. maybe we should backport that" 21:13:29 the TC has a newly proposed tag for "never-breaks-compat". https://review.openstack.org/#/c/446561/ 21:13:40 clayg: thanks for commenting on it (and I agree with what you said) 21:14:14 but the patch proposer assumed it would be something that would apply to swift, but that's a choice that we need to make 21:14:36 so read over it and be aware of it. think about if that's something you think we should apply to swift or not 21:14:55 also, the boston summit is coming up soon 21:15:04 I've learned more how the process for sessions will work 21:15:24 this week (THIS WEEK) please add anything you want to talk about at the summit to https://etherpad.openstack.org/p/BOS-Swift-brainstorming 21:15:44 on the never-breaks-compat: i think they really ought to break that down between client API and operator config. as it is, i'd still rather like to drop post-as-copy 21:16:11 early next week I'll take what's there and use the provided session submission tool to propose sessions 21:16:27 timburke: i had assumed it was limited to client api - you should put that in a review comment - makes sense to me 21:16:36 timburke: +1 21:16:52 the summit sessions are supposed to be stuff that can't be discussed elsewhere because in other places there aren't the right people 21:17:12 that means that the boston sessions should be more ops focussed (eg dev design sessions for the PTG) 21:17:45 and finally, note that we're in the middle of time changes around the world. 21:17:47 US has passed, EU is this weekend, AUS is next weekend. Japan doesn't do this sillyness. meeting time is 2100UTC (which doesn't change) 21:18:07 notmyname: how many things do you have to select how long should a topic be (i.e. how much ground do we try to cover) 21:18:11 but also note that affects our tx overlap 21:18:31 clayg: 40 minute sessions (with a very tiny possibility of having a double session) 21:18:57 any questions on all those FYI things? 21:20:04 there's not much getting added to https://etherpad.openstack.org/p/BOS-Swift-brainstorming 21:20:14 yeah 21:20:23 I'm not sure how I feel about that, honestly 21:20:24 notmyname: I think there must be some trepidation about what's appropriate 21:20:51 because on the one hand, the previous summit sessions that aren't dev design sessions are largely "ops feedback" 21:20:54 but i'm looking at the list of people attending ... i know what I want to talk about those people - let's just plan to talk about that! 21:20:58 so this etherpad isn't actually too bad 21:21:21 but I think you're right too -- trepidation and uncertainty about what is appropriate to propose 21:22:02 if we don't propose anything, we won't have much relevant swift stuff at the summit. if we propose a lot of stuff, we might not get it accepted, but we'll likely have some 21:22:22 notmyname: one question 21:22:26 so if you're not sure if something is appropriate, then write it down. worst that can happen is that it doesn't get selected 21:22:29 kota_: yes? 21:22:39 timburke: so I wonder , would the backwards compat thing mean we couldn't start to version DLO, just in case someone out there relied on the current behaviour? 21:22:56 notmyname: I'm still not getting all for the forum which say like "that should be long-term future discussion" 21:23:05 acoles: I think eventually the tags just signal intent + whatever you can manage to test for 21:23:28 acoles: If tempest magically tested that behavior and we needed/wanted to change it I think we just make a case like we would have to do anyway 21:23:55 acoles: i'm torn. i could see an argument for it falling into the security-fix category 21:24:04 notmyname: I always want to land all my patch asap so it may be in Pike so some ideas still be in the list may be in Pike or soon, so it's a reason I did not write so much there 21:24:08 (but then i'm biased toward wanting it to land :-) 21:24:10 kota_: I think the reason for saying that is because they imagine that the forum would be requirements gathering for the Q or R cycles instead of stuff that will be worked on right now 21:24:17 kota_: :-) 21:24:22 tdasilva: cschwede: can you guys come to BOS to talk about ring management? 21:24:41 kota_: yeah, I'm with you. land all the patches right now :-) 21:24:54 clayg: not sure yet 21:24:56 timburke: right! well let's land it quickly :) 21:24:57 yeah, and if we could use the forum like as the ptg or past hackathon, it seems to be easy to write something for everyone 21:24:59 tdasilva: cschwede: I feel like a forum ops summit might be a good place to do that!? 21:25:14 clayg: funny enough, it will probably be easier for cschwede to be there than me....trying to figure out funding atm... 21:25:31 kota_: I don't think the highly separated (1) requirements (2) planning (3) develop (4) deploy schedule that's implied from some openstack projects applies very well to how we've been successful in swift 21:26:05 tdasilva: cschwede: esp. if we can get word out to people that might be doing openstack deployment kind of stuff - or etcd config management stuff - get some fresh ideas on what ring management might look like from someone that doesn't "know better" 21:26:20 clayg: but, yeah, i will try to make it at least one day 21:26:30 oh man, gotta get me some etcd configs 21:26:34 clayg: yeah, i was interested in that whole etcd config mgmt stuff 21:26:36 notmyname: k, thanks 21:26:39 notmyname: I had thought the current etherpad was for ops-feedback-type-sessions, not swift-devs-ptg-type session proposals? was I wrong? 21:26:40 clayg: but it's a good idea to discuss this with ops, i'll put this on the etherpad 21:27:01 tdasilva: or we'll come to you! 21:27:30 acoles: y'all should come over here ;) 21:27:36 acoles: works for me! :) 21:27:42 acoles: my impression is the same as you. more like ops/deployment stuff. but that's historically been intertwined with dev sessions within the swift community (IMO). so it's hard to draw a distinct line 21:27:44 acoles: nobody knows - i think if we have to cut we prefer sessions that can take feedback/requirements - but that's design/planning too 21:28:31 IMo if there's something our community needs to talk about in order to make swift better, then write it down. 21:28:40 who is selecting, this the openstack powers to be? or us? 21:28:45 notmyname: +1 21:28:46 notmyname: clayg: I just hadn't added topics in the usual fashion that would be dev focussed 21:28:46 ie topics. 21:29:00 yeah, the Powers That Be. (I think that's the TC and User Committee) 21:29:11 I worried to put something up that I need to be there, but then can't get approval or funding to turn up 21:29:26 notmyname: OHHHHH, they'll pick 'em from the etherpad - or notmyname has to submit something? 21:29:43 mattoliverau: if we've got something in swift that can only happen with one person, then that's another problem, TBH. (ie write it down anyway) 21:29:55 clayg: no, they will pick from submitted sessions 21:30:16 sorry I think i zoned out 21:30:18 we as a team need to write down the stuff we want to talk about, then someone submits those via the sessions submission tool 21:30:21 notmyname: do you submit sessions? 21:30:26 yeah 21:30:47 it doesn't have to be me, but yeah, I was planning on doing it 21:30:58 notmyname: could you name a session "EC feedback" and then if no one has feedback we just talk about the reconstructor rledisez's patches? 21:31:24 sure 21:31:26 :-) 21:31:37 clayg: :-) sounds nice 21:31:44 I like the "let's write down what we need to talk about in Boston" - then we'll see what happens with the TC process 21:31:51 exactly 21:31:52 or foundation/user-group whatever 21:31:54 they're *trying* 21:32:00 if it sucks - well tell them and why 21:32:24 and if, worst case, we can only end up talking about things in the hallway (or tdasilva's house), then at least we've got a list of stuff we can use to start the conversation 21:32:41 k i'm set 21:32:45 ok :-) 21:32:49 kk sounds like a plan 21:32:57 any more questions? I want to move on to the next topic 21:33:09 * tdasilva 's house offers food with conversation 21:33:48 #topic generic diskfile implementation (ie OVH LOSF) 21:34:00 #info LOSF = Lots Of Small Files 21:34:16 i liked where the entrypoints patch was going near as I can tell 21:34:21 awesome work there 21:34:25 alecuyer is working on optimizing swift for lots of small files 21:34:30 and added them item to the agenda 21:34:35 he's got an etherpad up... 21:34:41 #link https://etherpad.openstack.org/p/swift-losf-base 21:34:56 and would like to give a very brief overview of the direction 21:35:02 right. We are trying, with rledisez, to make it easier to add alternative diskfile implementations 21:35:23 so there is the patch clayg just mentionned 21:35:38 #link https://review.openstack.org/#/c/436406/ 21:35:52 that one? 21:36:08 #link https://review.openstack.org/#/c/447129/ 21:36:10 that one 21:36:14 thanks 21:37:09 and another point. When working on the prototype for LOSF, I had to deal with callers outside of diskfile that use full path as arguments (path to objects, partition, etc) 21:37:11 alecuyer: so what's the current state, and where do you need feedback? how can the rest of us help? 21:37:22 that may not make sense for non-file based backends 21:37:48 So I had to emulate listdir, and parse paths to extract information 21:38:19 It would be easier if the functions passed "location" arguments, such as policy, partition, ohash 21:38:42 I have started to patch some functions in DiskFileManager and in callers to do that. 21:38:56 The LOSF "manager" functions would have the same prototypes 21:39:19 So I was wondering what if you think that would be a correct approach 21:40:27 alecuyer: from a high-level (ie you talking about it and me not having looked at the code), i like it. it sounds like the right approach 21:41:10 alecuyer: I would slighly prefer the interface optionally support 'suffix' as a param than have all the callers have to ohash[-3:] 21:41:39 the https://etherpad.openstack.org/p/swift-losf-base etherpad would be a great please to leave feedback (but since etherpads are hard for collaboration, please leave your nick next to what you type) 21:41:41 alecuyer: similarlly it might make sense to just cache some state on the instance and take less params - let implementations use what they need 21:41:48 clayg, ok. We have seen suffix will be an issue because it is "hardwired" at least in the replicator , I think 21:42:15 notmyname, if the approach sounds right I will post patches and link to them in the etherpad 21:42:24 cschwede: acoles: based on your part power increase and ec work, I'd love to see your feedback 21:42:29 alecuyer: it's not a terrible place to split up the partition - were you going to rework get_hahes to an entirely different format? 21:42:37 alecuyer: i'm not sure, but i'm wondering if we should look at a tier above for abstraction, since it is called *diskfile*. My fear is that we might try to change it to fit something it wasn't made for 21:43:06 notmyname: and tdasilva as well, he's a diskfile expert! 21:43:16 tdasilva: who would come along and implement a "Diskfile" for something that's not a local disk?!?!? 21:43:30 ;-) 21:43:31 rofl 21:43:35 touché 21:43:41 alecuyer: for context tdasilva maintains the swift-on-file diskfile; i mainained a out-of-tree diskfile implementation in another life - but I think it *mostly* had to reimplement stuff needed for replication 21:43:44 but...it is still files! 21:44:03 clayg, we have not done so at the moment. We're not going for a full rewrite, initially at least ! 21:44:21 thanks for the context :) 21:44:40 alecuyer: that sounds good to me - so anyway - I think "suffix" as a concept may contine to exist - even if it's not strictly needed by your implementation 21:44:47 clayg: current idea is « just » to plug a new disk file without disrupting everything around (not patching replication that’s your job clayg ;)) 21:45:13 I'm suggesting you continue to send it down - but ignore it - rather than try to get rid of it ... maybe.... maybe not 21:45:24 rledisez: understood 21:45:32 in the prototype, the patches to replication were minimal to get it working (for replication), I believe < 50 lines 21:45:42 yeah, that's good to keep in mind. the problem being solved here is to optimize a real problem, not to make the ultimate abstraction layer for anything :-) 21:45:52 alecuyer: my first reaction is that if there are improvements to the interface that will help then we should consider them (like not passing full paths). if possible, small tactical patches would be good 21:46:22 acoles, that is what we had in mind, yes, small patches to build on 21:46:28 alecuyer: the etherpad makes reference specifically to _get_hashes - and I think the interfaces used outside of the object server are the most coupled with the existing implementation 21:46:52 not to the interfaces used by the object server ... it sounds very fine/correct 21:47:47 i think i lost part of the sentence 21:47:55 clayg: _get_hashes is an example. an actually, we need very few functions patched as the coupling between callers and diskfilemanager is about 10 functions 21:48:07 alecuyer: i was just trying to say that the less changes you have to make to the interface in the object server - the stronger you should feel you're doing it right 21:48:12 and most of them are ok for what we need 21:48:26 clayg, right, thank you for clarifying :) 21:48:40 rledisez: ok, yeah and the REPLICATE verb is the outlier - that's part of the "replication" interface IMHO 21:48:58 other stuff should be "internal" - but jesus - it's not like we ever tried to documnt what the f'ing interface is 21:49:22 there's annoying coupling everywhere - the *type* of the exception raised on certain conditions is part of the "interface" for christ sakes 21:49:46 ^ good point, we should note that alecuyer 21:49:51 ok, but it sounds great - can't wait to see a patch! 21:50:00 I think there is a comment about it -the exception 21:50:01 +1 21:50:36 if we could keep discussing it in the etherpad and in IRC, that would be great 21:50:44 ok ! thank you all for the feedback and please do send more in the etherpad 21:50:46 alecuyer: thank you for working on this. i'm really excited about it 21:50:50 rledisez: alecuyer: try not to get *too* caught up on a crusade to define the perfect diskfile interface - KISS 21:51:05 #topic open discussion 21:51:05 yes 21:51:08 alecuyer: don't assume that because code exists on master that it is universally considered to be perfect :) there has been a lot of history in diskfile, so please challenge anything you find that seems wrong 21:51:12 ok, a couple of things here 21:51:18 acoles: +100 21:51:22 about patches, our goal is to land that as fast as we can to build the new diskfile on clean bases. so these modifications and the pkg_resources will be submitted very soon 21:51:52 first, i had an idea that I was curious about. "call your patches" 21:52:16 if you're about to review or look at a patch in gerrit, say somehting in IRC. like, "I'm looking at patch 345345" 21:52:33 then, within an hour or two, leave a comment in gerrit on the patch 21:52:45 call your patch, leave a comment 21:52:52 I think this will do a couple of things 21:53:02 rledisez: we'll try to keep up! Obviously any help you guy apply to the review backlog makes a huge difference to our throughput - if you think something is not important and you see reviews going on there you need to find some way to communicate that 21:53:03 first, it will help us all know who's looking at what 21:53:21 triage takes a village - everyone can get caught up chasing dragons and lose sight of the prize 21:53:49 and second, I hope it will help reduce the perceived "cost" of leaving a comment 21:54:04 notmyname: but what if I don't want to admit how long it takes me to review things sometimes :'( 21:54:09 ie leaving a comment is just a thing you do and small comments are ok 21:54:26 what if I say in channel I'm going to review something and then don't because I'm a dirty lazy liar! 21:54:34 clayg: it's ok. I've learned that in a group, if you feel a certain way, probably three other people do too :-) 21:55:34 notmyname: I don't think I follow how this reduces the cost of leaving a comment? 21:55:55 * jungleboyj is curious about that as well. 21:56:08 notmyname: I've historically been a big advocate of "try to comment with clear instructions to make the patch meragable" - that can a non-trivial amount of work 21:56:21 because if leaving a comment, even a "I looked at it and didn't find anything yet, +0", if that becomes common, then it's easier for everyone to leave a comment 21:56:28 notmyname: I tend to be a hypocrite about it too 21:56:52 because if you don't ever comment until the comment is perfect, it's like not submitting code until it's perfect -- it's impossible and leads to not much getting done 21:56:54 notmyname: but it "seems" like a reasonable goal? And from someone on the other side - clear instructions help *a lot* 21:57:01 definitely 21:57:33 Hmmm, I leave a lot of imperfect comments. :-) 21:57:37 notmyname: ok, yeah I kinda like that.... we should update the review guidelines! 21:57:56 we can't all be in the same room all the time, so "hey I'm looking at Foo" is something that can be helpful to coordinate 21:58:09 and it might even lead to a "oh! I was looking there too, and I found ..." 21:58:28 timburke: does this sound right to you? Maybe the difference is the +0 and the -1 21:58:31 FWIW I sometimes leave a 'part-complete' review and say its only partial. usually it means I have not yet worked through 100's of lines of test changes, but have some comment on the real code. 21:58:51 notmyname: Doesn't seem like a bad idea. I know I have collided with people on many occasions . 21:58:56 -1 "fix this and we're mergable" +0 "you can either pretend I didn't leave this here or read it - my review is WIP" 21:59:07 WIP review :-) 21:59:30 we need real and imaginary numbers for votes :) 21:59:38 because even if you don't get to come back to your review, it's a starting point for the next reviewer! 21:59:44 clayg: i can get behind it. i certain will leave +0s when i've done a partial review, not found anything blatantly wrong, but definitely had some questions raised 21:59:45 acoles: multi-demensional reviews? 22:00:14 ok, we're at time 22:00:19 notmyname: the review history is *super* important to me when I come to patch that's been at it awhile 22:00:20 notmyname: +100i 22:00:43 I saw that PavelK had a patch question in -swift about a patch. I wanted to get to it here, but we should look at it in -swift 22:00:43 notmyname: sometimes I wish we could a review "summary" or something :P 22:00:52 thank you for your work on swift 22:00:57 #endmeeting