19:00:12 #startmeeting swift 19:00:14 Meeting started Wed Mar 25 19:00:12 2015 UTC and is due to finish in 60 minutes. The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:15 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:00:17 The meeting name has been set to 'swift' 19:00:34 who's here for the swift meeting? 19:00:37 o/ 19:00:38 o/ 19:00:42 hello 19:00:45 hi 19:00:49 yo 19:00:56 o/ 19:01:38 torgomatic: you here? 19:01:47 o/ 19:01:59 šŸ™‹ 19:02:30 ok, let's get started :-) 19:02:50 here 19:03:05 for this week, we're looking at EC status and the merge to master 19:03:28 #link http://goo.gl/uRzLBX 19:03:41 the starred patches there are the top things left for EC 19:03:51 hello, i'm here 19:03:54 so I want to go over them each 19:04:00 #topic EC patches 19:04:10 patchbot get ready 19:04:13 #link https://review.openstack.org/#/c/167595/ 19:04:39 this one is for functesting and needs guidance for import loops 19:04:52 tdasilva: jrichli: any comments on that one? 19:05:20 no, not besides the need for help with the import statement 19:05:27 notmyname: it looks like tdasilva is also trying to push that over to master? 19:05:43 tdasilva: I'm *great* at circular imports - what's going on? 19:05:56 * clayg thinks tdasilva might respond in other channel 19:06:08 code in the module is importing stuff form the __init__ 19:06:09 clayg: probably a good idea, right? 19:06:28 clayg: https://review.openstack.org/#/c/167595/2/test/functional/swift_test_client.py 19:06:48 if we move the import statement to the top, it breaks :/ 19:07:02 AttributeError: 'module' object has no attribute 'functional' 19:07:32 I can help with this one if clayg is busy on other patches 19:07:33 tdasilva: actually iā€™m working on some other ideas, i will try submit sth tomorrow morning 19:07:51 cschwede: on this patch? 19:07:51 cschwede, notmyname: ok 19:08:00 notmyname: yes, on this patch 19:08:00 cschwede: tdasilva: yeah it's just a circular import normally fixable with some shuffling of things to different modules 19:08:25 clayg: alright...i can also keep working on it 19:08:50 cschwede: sounds like tdasilva and I are going to suss it out after the meeting - if you wanna post a gist of what you've got now is probably the time 19:09:09 clayg: give me a few minutes, checking if my idea makes sense 19:09:22 oooh, real time fix :) 19:09:22 ok, sounds like there's a plan for that one. let's move on to the next patches 19:09:31 cschwede: drop it in -swift 19:09:43 next up to talk about 19:09:43 cschwede: you don't have to go do all that - just throw it over the wall - tdasilva and I will get whatever you're getting at 19:09:45 #link https://review.openstack.org/#/c/164950/ 19:09:55 clayg: this is your patch 19:09:58 tell me more about it 19:10:07 what does it need? what's next step? who's reviewing it? 19:10:46 well like we did with the ECDiskFile we (I?) want to simplify the proxy changes for ec by moving the important stuff for PUT to the ECObjectController (like we've already done for GET) 19:10:54 peluse: tdasilva: and I have been talking about this one 19:10:57 ok 19:11:33 in particular there's a thing I'd like to change in PUT to try and send down the node-index sooner and it could be complicated - so it'll look better if we do it outside of the replicated PUT path 19:12:12 torgomatic: sorry everything you so gracefully tried to make flow through one code path is being ripped into two - but I think it'll acctually be better for both replicated and EC if these code paths are seperate - but simpler (more directly tuned to what's going on) 19:12:27 s/tried to make/successfully made/ 19:12:39 heh. and torgomatic just left the room at that comment ;-) 19:12:40 clayg, so I'm not clear on what you want to do with that patch as it stands now? wait til post beta? something smaller instead before beta? 19:12:45 lol 19:12:47 peluse: good question 19:12:48 * clayg things it's a bad sign that torgomatic just walked out :P 19:12:55 ahhhh 19:13:16 peluse: I want to do it now! like today! after the diskfile the proxy was my target this week 19:13:25 peluse: I think it'll make next week go easier 19:13:29 well allrighty then! 19:13:30 my 2cents.. I think we should land before ec. 19:13:30 ok 19:13:36 ok :-) 19:13:45 peluse: the only reason it's so big is because we already landed a bunch of stuff all up in the proxy's replicated PUT path that's EC specific! 19:13:51 so in the order of the current chain under review, right after those? 19:14:14 (we'll get to that big chain in a moment) 19:14:20 well - did we already do everything up to the reconstructor? because that should be a higher priority than the proxy refactor 19:14:28 notmyname: it's getting smaller! 19:14:32 :-) 19:14:39 clayg, almost, there's 2 that I D on I think 19:14:45 ok, good to know. 19:15:04 we'll tackle this one right after the reconstructor chain lands 19:15:18 that's what I was thinking made sense 19:15:27 ok, next up... 19:15:36 which means https://review.openstack.org/#/c/159637/25 right? 19:15:41 not yet 19:15:43 #link https://review.openstack.org/#/c/166576/ 19:15:50 multi range from torgomatic 19:15:55 ahhh, OK 19:15:58 works on my machine (tm) 19:16:00 but there's another depending on that one 19:16:02 peluse: notmyname has his own way of doing things - it's best to just go with it 19:16:11 heh 19:16:12 clayg is learnign ;-) 19:16:26 torgomatic, I can bang on that next, I have some recon stuff to push that I need to wait on a few other things for 19:16:37 torgomatic: peluse: I don't care about multi range gets - is that even requried to make functional tests pass? I'm not going to review it. Someone should tho. 19:16:45 I want to see multi-range get in, but certainly lower priority 19:16:54 what I'm concerned about is the dependet patch 19:16:57 well, maybe I won't then 19:17:01 #link https://review.openstack.org/#/c/167406/ 19:17:06 see peluse needs to work on the reconstructor - i've got refactoring shit I need to get ready before I try to get this stuff on master 19:17:13 yeah, OK 19:17:16 who has time to review *multi*-range GETS!? 19:17:22 ;P 19:17:28 it's p 167406 though 19:17:35 torgomatic: tell me about that one 19:17:45 patch 167406 19:17:46 notmyname: https://review.openstack.org/#/c/167406/ 19:17:48 i like the commit title for patch 167406 a lot! 19:17:48 clayg: https://review.openstack.org/#/c/167406/ 19:17:53 i'll review it 19:17:57 mattoliverau: thanks 19:18:04 mattoliverau: multi-range gets or the get path error handling? 19:18:05 but how much better is the question? 19:18:10 that one... so you've got this ECAppIter pulling in a bunch of streams from the object servers, and yielding up bytes for the client 19:18:14 peluse: any better is better - i'll take it 19:18:36 multi-range.. but both really :) 19:18:39 torgomatic: oh yeah that one - yeah that one needs to land 19:18:47 mattoliverau: YES PLEASE! THANK YOU! 19:18:48 if one of those streams fails horribly, then the client connection should break. hanging forever is bad 19:18:52 which mean multi-range is needed 19:19:04 hence the patch 19:19:08 torgomatic: yeah that sucks too hard - we need to merge that 19:19:21 ok, so mattoliverau is on that chain. we'll need another too 19:19:21 *fine* 19:19:23 * clayg rolls eyes 19:19:25 lol 19:19:28 ;P 19:19:35 cschwede, was looking I think right? 19:19:41 šŸ˜ 19:19:47 at least on the nd one 19:19:48 yes 19:19:51 cschwede: thanks 19:19:53 notmyname: no we don't - if sam and torgomatic both like it I'll propose it to master - good enough for me 19:20:04 well mattoliverau and torgomatic 19:20:05 heh 19:20:07 heh 19:20:15 whatever 19:20:19 ok, we've got a plan there. next up 19:20:21 #link https://review.openstack.org/#/c/143791/ 19:20:25 does that mean sam can +2 something and torgomatic approve it? 19:20:38 this one is container sync with internal client 19:20:52 it's like donagh and acoles 19:21:10 the container sync one is good - i touched it last - but I'll +2 it 19:21:16 ok 19:21:17 I think torgomatic considered it reasonable enough too 19:21:32 yeah, I just don't have enough crap set up to functionally test container sycn 19:21:35 notmyname: yesterday I was hacking on fixing the account-reaper which has a similar issue - but that one I think is going to be different 19:21:36 so why hasn't it landed? ;-) 19:21:43 ok 19:22:05 again, I'm ok with a beta not having container sync, but it's certainly nice to have 19:22:08 torgomatic: just checkout vagrant-swift-all-in-one and set your localrc to have an ec policy - it's like two options - container sync (and the probetest for container sync) work out of the box 19:22:19 notmyname: i see no reason we can't merge it - it's minor 19:22:44 kk, torgomatic will test it and approve when he's nice (after clayg's +2) 19:23:08 next up is... 19:23:08 #link https://review.openstack.org/166754 19:23:18 kota_: this one is yours 19:23:24 it has a merge conflict 19:23:32 notmyname: ah, ok 19:23:41 it's relatively small (compared to the others) 19:23:47 yes 19:23:50 kota_: what's the summary? why does this one need to be in? 19:24:06 yeah but this 'small' function can bust the ECrecon very easily which is why i asked nobody mess with it until after ECrecon is done since its working now 19:24:10 and this is a refactor, not a fix 19:24:20 this is for ensuring .data includes index on ECDiskfile. 19:24:22 oh yeah only put the fi on the file name if it's a data file - that comes in somewhere else in the next change too (pretty sure) 19:24:38 kota_: yeah that ts_to_fname interface is not quite right 19:24:39 ok, so needs to be after reconstructor? 19:24:41 not urgent because it is for safety, I think. 19:24:44 ok 19:24:50 shall I un-star it? 19:24:52 yeah, like I said in the comments... would like to ignore it until after ecrecon and all its D's are in 19:24:52 notmyname: ya 19:24:56 kota_: we should just blow up hard if a ecdiskfilewriter can't add the fi to the datafile 19:25:09 no non-fi'd datafiles in ec - ever! 19:25:24 rock on baby 19:25:24 peluse: wants the D's 19:25:31 peluse: clayg: ok with un-staring it? 19:25:33 damn it 19:25:39 yes :) 19:25:46 lo*L* 19:25:57 but thanks kota_ :) 19:26:00 * clayg is having way to much fun 19:26:03 must be all the stress 19:26:09 peluse: no worries :) 19:26:10 notmyname: ok do the next one! 19:26:20 ok, next (and last) patch chain 19:26:20 #link https://review.openstack.org/159637 19:26:23 this is the big one 19:26:33 it's the reconstructor - so you know - that'd be nice to have :P 19:26:37 where do we stand? 19:26:41 ya, it's kinda important 19:26:54 acoles wants me to do a test audit on the suffix hash canges 19:26:57 seems reasonable 19:27:01 I think its in clayg court - auditing some tests? 19:27:03 oh, yeah 19:27:09 ok, and it sounds like acoles is on it too 19:27:18 i think he's done 19:27:18 then the ssync changes are missing a bunch of tests IMHO - but some of them may just come in with reconstructor - not sure 19:27:20 and me too 19:27:27 notmyname: yeah we're all over it 19:27:28 however, acoles is out for another 12+ hours, so should someone else look 19:27:30 yes ssync is missing stuff 19:27:41 I think I own the first one (multi-fi hash suffix) a diff - then it's back on the review train 19:27:43 acoles had some he was dusting off that we can add 19:27:48 i will finish the review of that patch also in ~ 13-14 hrs 19:27:56 and then there some ec recon ssync changes that have no coverage I need to get in 19:28:22 #link https://review.openstack.org/#/c/165188/ 19:28:29 that's the dependent patch 19:28:33 that's the one that needs some help 19:29:05 that one needs some tests - I think the mulit-fi stuff is getting *really* close - we're down to nits on the tests - i have that one under control I think 19:29:07 yeah, it needs full rebase after the one we just talked about it ready 19:29:26 and then we're still missing GET specfici FI request capability, right clayg? 19:29:40 ok, I'd like to see the first one land soon so that we can get patch 165188 landed either today or tomorrow 19:29:41 notmyname: https://review.openstack.org/#/c/165188/ 19:29:44 did I read something right that you pulled that temporarly a few days aho? 19:29:47 I'll do that right after the meeting - with the one change I have ready for acoles - then if anyone with review bandwidth could start picking apart the pre-req ssync changes for the reconstrutor I think we're good 19:30:18 which would get us to https://review.openstack.org/#/c/131872/ -- the reconstructor itsekf 19:30:37 so, peluse you're out for the rest of the week? how will we make progress? 19:30:39 yup, I have more tests to push but can't til my next D up is rebased 19:30:42 peluse: yeah I totally pulled that - it's not helpful - i've been talking with torgomatic about the probetests we need for handling those kinds of failures and we'll figure out what the code needs to do 19:30:46 sheeeeeeit 19:30:59 peluse: you can't leave 19:31:09 notmyname, the reconstructor is simple - the hard stuff is what was broken out that clayg and acoles have been working on :) 19:31:18 peluse: that's not true at all 19:31:21 or take your computer on you 20th anniversary trip ;-) 19:31:34 great idea if I don't want to see my 21st :) 19:31:34 swift + wine sounds good 19:31:39 peluse: also there is a ton of prints and stuff in there - you need to start imaging we may want to merge that code at some point 19:31:40 lol 19:31:49 I already took them out 19:31:53 just haven't pushed yet 19:31:54 peluse: ok :P 19:31:58 OHHHHHHkay 19:31:58 because of waiting on D rebase 19:32:02 lol 19:32:08 D == dependency (sp?) 19:32:11 yes 19:32:18 the word is too long for me :) 19:32:39 peluse: ok, so if for the time you have left, you get that patch ready for others to take over this week, that would be awesome 19:32:47 if there's anything else to do 19:32:54 well... 19:33:03 i mean i know that the code works - so it's just cleanup and fixups from reviews and stuff 19:33:14 I could the FI support patch (right above me) and then push what I have 19:33:20 when I was looking at it I found the tests suite sort of hard to follow/maintain (personally) 19:33:30 which one? 19:33:40 peluse: obj.test_reconstructor (unittests) 19:33:47 peluse: the probetests are sorta great 19:33:56 yeah, I've cleaned them up a little but they could use some clayg magic wand work :) 19:34:02 lol 19:34:28 we know where we are on this patch chain? 19:34:39 tdasilva: you buying into that pharse - I feel it's more like I just seagull a bunch of other peoples hard work :\ 19:34:46 notmyname: yeah - *behind* 19:34:49 heh 19:35:02 * notmyname feels clayg has his name on a lot of these :- 19:35:04 ) 19:35:04 and also freaking out about no peluse !!!!! 19:35:09 bah 19:35:13 2 days! 19:35:24 peluse: sorry - tell me to shut up and get over it 19:35:32 well, 4 if you count Sat and Sun :) 19:35:39 notmyname: tell me that for peluse 19:35:47 clayg: get back to work! 19:35:53 'essir! 19:36:00 or come to rickhouse Thu at 4ish for a quick drink and I'll tell you in person 19:36:04 ok, so for everyone, any questions on the outstanding work or what you need to be looking at? 19:36:11 peluse: tomorrow!? 19:36:12 * notmyname wants to make sure we make the most of the time 19:36:15 yes! 19:36:26 lol that's great! 19:36:39 where else would I go on vacation but 1 block from swiftstack? 19:36:53 haha 19:37:13 everyone ok with the outstanding patches before we move on to the next topic? 19:37:26 phew 19:37:32 * clayg bets the next topic is still ec related 19:38:02 FYI I'm going to rebase https://review.openstack.org/#/c/159637/25 now so I can push ECrecon latest (sorry last EC thing) 19:38:12 clayg is right 19:38:44 peluse: well do you have an hour? can you not do it *right* now? i was going to change p 159637 19:38:57 OK, I'll wait :) 19:38:59 peluse: well... i mean you can do it now - i'll rebase up the patch change to whatever you push 19:39:08 peluse: well just keep cleaning up whatever you've got going 19:39:13 peluse: and give me a few 19:39:16 notmyname: ok go! 19:39:25 lol 19:39:30 ok, next up (and yes still with EC) 19:39:44 the other thing I wanted to mention at the start 19:39:58 right now, the feature/ec branch passes all functional tests!! 19:40:05 that's really cool. huge progress 19:40:11 woo 19:40:24 well unless you have Ec as a default then I'm seeing 3 failures that look 'odd' 19:40:26 notmyname: that is HUGE - nice work torgomatic yuan et. al. 19:40:37 peluse: works for me with EC as default 19:40:44 peluse: mm..works for me too 19:40:47 OK, must be me then - ignore me 19:40:48 peluse: I think you might be out of date - they landed some stuff 19:40:57 * clayg ignores peluse the abandoner 19:41:02 lol 19:41:11 #topic merge to master 19:41:31 so there's still a lot to do before merge to master happens, but I wanted to touch on it briefly this week 19:41:43 the important points so far are this: 19:42:07 1) master will freeze during this process, so that there isn't unecessary merge conflicts 19:42:27 2) we'll refactor the feature/ec branch to a new patch chain that will be proposed 19:42:51 I'll -2 the first one to plug up the merge until it's ready 19:43:17 then we'll all review the chain and add review votes. when everything is approved, we'll uplug it and let it go 19:43:45 I want to figure out if we can do it (like storage policies IIRC) as a single merge commit so that it's easy to see it being added as one commit 19:44:24 excellent 19:44:25 3) clayg will be point on doing the branching/rebasing patch chain magic 19:44:29 thanks clayg 19:44:41 * peluse wishes he could give some sort of award to clayg for managing the merge chain 19:44:45 lol 19:44:50 peluse: he accepts scotch 19:44:56 done! 19:45:00 whoa - nice 19:45:09 notmyname: How do we work at feature/ec branch during the process? 19:45:17 kota_: we don't :-) 19:45:24 notmyname: ok 19:45:24 people told storage policies merge could have been worse - i hope I can make ec at least that good 19:45:37 so you are freezing both master and feature/ec, right? 19:45:46 we have to, yeah 19:45:52 kota_: but you can review the merge change - and even submit changes (espeically to the docs - which will be at the end) 19:45:53 tdasilva: yes. feature/ec becomes a historical artifact 19:46:03 yes, docs. I wanted to bring that up 19:46:14 historical - or hysterical ? 19:46:23 yes :- 19:46:24 heh 19:46:24 ) 19:46:28 there's some doc already up there, I'll add/updeate when I get back 19:46:32 thanks 19:46:39 it shall be done :) 19:46:50 I expect the merge to start on monday or tuesday. when the current patch chains get merged 19:47:00 * peluse likes docs for some reason even though he cant type for shit 19:47:01 monday is much better than tuesday, though 19:47:15 so looking forward, here's the schedule we have 19:47:20 peluse: yeah totally the docs are great! but we as individuals all sort of suck at docs - they were best when crowd sourced - so I'd really like to see everyone contributing to give us the best docs at the end of the chain 19:47:23 this week, finish up feature/ec work 19:47:29 monday is tuesady for me :P 19:47:39 mattoliverau: better if it's your monday ;-) 19:47:41 check out the patch - build the docs - make fixes as you go and git review +1 - in the end everyone should have their hands in them 19:47:51 next week, start the merge to master 19:48:07 clayg, good point - if anyone wants to add more doc content please feel free (don't think it has to be me) 19:48:10 april 10 (which is only 2 weeks away!!) we shoot for a release candidate 19:48:15 also vagrant-swift-all-in-one has an "autodoc" command that will rebuild the sphinx as you hack so you can review them in your browser - you should try it out! 19:48:24 cool 19:48:30 does autodoc write them too? 19:49:29 or just 'python setup.py build_sphinx' 19:49:50 and look in the doc/build folder 19:49:57 april 10 gives us a chance for an RC2 if needed 19:50:08 and gives a chance for integrated testing across kilo 19:50:13 I think we should plan on an RC2 being needed 19:50:14 the kilo release date is april 30 19:50:18 and upside if its not 19:50:25 notmyname: +1 better to make the 10th with some trepidation and have a chance to rc2 on the 15th-20th or so 19:50:42 peluse: it sticks them into a public container on your saio 19:50:45 and prints out the url 19:50:58 so, any questions from anyone on the plan from here to april 30? 19:51:17 notmyname: quick dumb question: you said clayg will be a new patch chain to merge to master, but you also said that you would like to see it being added as one commit, how's that? seems contradictory? 19:51:21 mattoliverau: well you have to keep typing uparrow-enter after you save your changes - autodoc just watches the mtime on the source dirs - builds - and reuploads 19:51:24 it's great 19:51:36 cool 19:51:38 tdasilva: reviewed as a patch chain, and ideally merged as one merge commit 19:51:44 tdasilva: i have no idea what he's talking about - he misremember's how storage policies went in 19:51:48 lol 19:51:53 tdasilva: maybe we can figure out how to do it better this time 19:52:09 *maybe* i'm the one who's misremembering (but I dobut that - obviously) 19:52:12 tdasilva: there are a bunch of commits from all authors, and one final merge commit in the git history 19:52:13 near term point is that the patch chain will be submitted to gerrit for review 19:52:27 cschwede: *REALLY*? 19:52:35 tdasilva: like these from jenkins: https://github.com/openstack/swift/commits/master 19:52:51 clayg: how painful will the refactor for review work be? 19:52:55 do you guess? 19:52:55 clayg: iirc, yes 19:53:23 "Merge storage policies feature commit chain into master" - how did we do that?! 19:53:36 notmyname: I think it will drive me drinking! 19:53:44 I can enable that :-) 19:53:49 notmyname: but it's mostly just tedious - trying to keep unittests and pep8 passing along the way 19:53:54 ok 19:54:33 anything else from anyone? 19:54:39 tdasilva: clayg: that was the SP merge: https://github.com/openstack/swift/commit/53d4d21 19:55:11 so notmyname should know how he did this ;) 19:55:11 OK, later on people! 19:55:29 peluse: have fun! 19:55:35 thanks!! 19:55:36 ok, notmyname says he knows something I don't - I stand corrected 19:55:36 thanks everyone 19:55:39 peluse: congrats!!! 19:55:43 thanks for working on swift 19:55:49 #endmeeting