19:00:56 #startmeeting swift 19:00:57 Meeting started Wed Apr 15 19:00:56 2015 UTC and is due to finish in 60 minutes. The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:58 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:01 The meeting name has been set to 'swift' 19:01:11 who's here for the swift team meeting? 19:01:14 o/ 19:01:16 o/ 19:01:18 o/ 19:01:21 o/ 19:01:24 Here 19:01:29 hi 19:01:34 o/ 19:01:42 o/ 19:01:47 hello 19:02:04 good group here 19:02:05 hello 19:02:13 clayg: hurricanerix: courtesy ping 19:02:37 peluse is out at a meeting. torgomatic has another commitment 19:02:50 ok, let's get started 19:02:53 agenda is at 19:02:57 #link https://wiki.openstack.org/wiki/Meetings/Swift 19:02:57 o/ 19:03:08 #topic EC status 19:03:44 had to get the ling 19:03:46 *link 19:03:47 #link https://github.com/openstack/swift/commit/e910f7e07d05dd2c6ada939d5704c3d4944c24b0 19:03:52 there it is! on master! 19:03:59 thank you everyone 19:04:02 great! 19:04:14 cool! 19:04:20 awesome!! :) 19:04:32 I've said this many times before, and I'll keep saying it: this is a huge feature that has been a whole-community effort. thank you 19:05:08 as a follow up to that, the feature/ec and feature/ec_review branches are gone 19:05:19 feature/ec has been tagged for historic purposes 19:05:30 but you can no longer submit reviews to them 19:05:44 and the rest of the pending changes have landed on master 19:05:51 we have a 2.3.0rc1 tag 19:06:09 unless major issues are discovered, then this will be our 2.3.0 release and included in openstack kilo on april 30 19:06:29 master is unfrozen 19:06:30 any questions on ec, the RC, or the release? 19:06:41 just one minor q 19:06:50 notmyname: one thing. 19:06:59 wbhuber: first 19:07:06 notmyname: ok, waiting 19:07:31 in the doc link, development_saio.rst, that has yet to include python_pyeclib as a dependency 19:07:57 isn't that what requirements.txt is for? 19:08:15 yes, but requirements.txt comes after installing swift 19:08:17 I thought the docs only listed out like system packages which need to be installed 19:08:52 wbhuber: i'm not sure I follow - can you do a patch or open a doc bug? 19:08:53 hmm..looks like it doesn't actually have a step to install swift's requirements.txt 19:08:56 it does for swiftclient 19:09:01 i could open a doc bug 19:09:04 wbhuber: at the moment, we didn't have rpm format pkgs generally available for pyeclib and liberasurecode yet 19:09:15 only Debian packages 19:09:17 wbhuber: looks like an oversight. a bug (in swift) or evena patch to the .rst file would be great! 19:09:17 i see 19:09:19 notmyname: well the setup.py will do it - it's all pbr'd and setup.cfg - probably won't work unless you pip install -e . 19:09:22 so we decided to stick with pypi 19:09:55 there's a patch but not implemented, i think - https://review.openstack.org/#/c/169990/6..20/doc/source/development_saio.rst 19:10:12 so I have given up on setup.py develop - it just never works with any version of setuptools you're likely to have installed on any release that's reasonable to develop on 19:10:21 ok 19:10:23 clayg: womm 19:10:26 cschwede and I recently fixed vagrant-swift-all-in-one by using pip install -e . 19:10:42 notmyname: you almost cirtainly would have pip installed pyeclib by hand then 19:10:52 clayg: yup. totally did 19:10:53 clayg: yeah, and i can confirm that a fresh vagrant-saio has pyeclib installed 19:10:55 or pip install -r requirements.txt - same different 19:10:59 yeah 19:11:03 ok nifty 19:11:16 wbhuber: ok, so there's a gap in the SAIO docs. can you open a bug or submit a patch? 19:11:25 cschwede: and it doesn't installed it *explicitly* - it just lets' setuptools/pbr/pip do it - but you cant spell it setup.py develop - you have to say pip install -e . 19:11:38 pbr hasn't trained you all to do this yet!? 19:11:38 i'll submit a patch 19:11:43 wbhuber: thanks! 19:11:50 kota_: you have a question? 19:12:00 notmyname: ok 19:12:02 notmyname: I know current reconstructor doing decode/encode because of the jerasure bug and we need to bump it later than Kiko. If you have a schedule for that, could I have it? 19:12:22 * notmyname looks to tsg_ 19:12:28 kota_: thanks I was about to bring up update to 1.0.7 19:12:45 notmyname: that might be one requirements update that we'll need to take in 19:12:58 (pyeclib-1.0.7) 19:13:04 also 1.0.7 will allow xor 3-3 yeah!? 19:13:10 tsg_: pre kilo? 19:13:11 clayg: correct! 19:13:12 notmyname: sorry, i am here =) 19:13:21 hurricanerix: hi! 19:13:32 notmyname: 1.0.7 has a number of bugs ironed out, so I recommend pre-K, yes 19:13:41 notmyname: not before kilo - once liberty opens 19:13:55 tsg_: clayg: you just said 2 different things :-) 19:14:00 tsg_: how are we going to pull that off!? 19:14:35 I can investigate bumping a dependency pre-kilo, but I don't know much about it right now 19:14:45 notmyname: tsg_: kilo will *work* with 1.0.7 right? but it *requires* at least 1.0.6 - that's a done deal at this point? 19:14:50 clayg: the pypi 1.0.7 release for pyeclib should be available by this afternoon - for requirements, I was hoping we could be explicit 19:15:06 clayg: correct .. at least 1.0.6 19:15:15 ok. 19:15:27 phrased a different way, will kilo break with 1.0.6? 19:15:35 or, what will break? 19:15:43 notmyname: 1.0.7 isn't even out yet! we've *all* been using 1.0.6 19:15:57 seems that way 19:16:02 clayg: liberasurecode 1.0.7 was waiting on a couple of fixes .. that's released 19:16:03 it's as good as we need, and 1.0.7 will be even better! 19:16:13 Kevin and I working on 1.0.7 for this afternoon 19:16:30 tsg_: is there something in 1.0.6 that needs to hold the release? 19:16:46 notmyname: some reconstructor related fixes in the jerasure backend 19:16:52 those are important 19:16:56 ie so we can patch/backport/rc2 to get 1.0.7 19:17:22 tsg_: jerasure bugs or liberasurecode bugs? 19:17:25 tsg_: you mean getting rid of the encode/decode hack? that's a liberty dev cycle fix. 19:17:51 notmyname: bugs in the "jerasure" backend in liberasurecode, yes 19:17:53 notmyname: if we could get 1.0.7 during rc timeline, it would be better to me. 19:18:29 kota_: tsg_: why do you guys care? if you don't like what's in kilo repackage? 19:18:34 kota_: what's preventing using 1.0.7 today? 19:18:54 notmyname: a) it's not released b) no one will have time to package it for the kilo release 19:19:13 clayg: it's important the the dependency is working at release. I'm not convinced at this point, but I want to get the info 19:19:25 notmyname: I found decode/encode thing is affecting our backend. 19:19:31 notmyname: (except us - we'll probably re-package swift & pyeclib a few weeks into liberty - but we always do that) 19:19:38 is there any real risk in using 1.0.6? 19:19:48 right, that's what I've been trying to ask 19:20:01 there is no risk 19:20:14 kota_: will you be able to use the newer version of the dependency? 19:20:17 the reconstructors bug has been already fixed in that rev 19:20:18 notmyname: cschwede: sounds like kota_ has some issue with the hack on his plugin? 19:20:29 notmyname, tsg_: because we use unique info at encoding to each fragment so when re-encoding the fragment it's not original fragment on our backend. 19:20:54 notmyname: ok kota_ has a real issue - the hack is acctually casuing a problem 19:21:04 kota_: will you be able to deploy liberasurecode 1.0.7 with swift 2.3.0 (with >=1.0.6 in the transitive dependency list) 19:21:22 kota_: I would suggest if we can't bump the depencency (we probably can't) - we *could* roll a rc2 that includes a config option to do use encode/decode *or* rebuilt 19:21:27 s/rebuilt/rebuild 19:22:01 notmyname: kota_'s problem isn't 1.0.6 - it's the hack - the question is could he repackage and deploy swift kilo+ instead of swift kilo or do we need an rc2 19:22:02 clayg: sounds reasonable, thanks 19:22:13 so far, I'd like to leave it as-is, if kota_ can use the newer version with the swift code 19:22:21 notmyname: +10000000 19:22:33 oh, wait, is there code in *swift* that is breaking things? or just the dependency 19:22:55 notmyname: kilo is beta - it wasn't tested with all backends - there's been an issue discovered with a non-tested backend - requires a bug fix that we want to do anyway 19:23:07 clayg: you don't have to over-sell me ;-) 19:23:18 notmyname: the code would have to change - the 1.0.7 only helps kota_ because it allows us to remove the hack that's breaking him 19:23:29 swift code or liberasurecode? 19:23:42 swift code 19:23:45 sounds like kota needs a change in swift code 19:23:48 ok 19:24:06 calyg, acoles: exactly, thanks 19:24:07 well he *wants* a change - that we want to - the question is if he *needs* it in kilo and we have to spin an rc2 19:24:17 right. I understand now 19:24:18 thanks 19:24:48 so, kota_, do you absolutely require an rc2? ie if this isn't in kilo, are you stuck without EC for 6 months until the openstack liberty release? 19:25:05 kota_: what is the problem with kilo (ec beta) having the hack - do you not expect to release/test with swift during liberty anyway? what if we find a new issue 4 weeks from now? 19:25:14 or can you deploy eg kilo(swift2.3.0)+ a few commits 19:25:24 clayg: exactly my question 19:25:52 notmyname, clayg: ah...thinking... 19:25:58 kota_: :) 19:26:06 kota_: no rush, it's cool ;) 19:26:26 * notmyname would be happy to work with kota_ to set up some 3rd party testing running the NTT EC library 19:26:36 notmyname: what is the possiblity of an rc2 anyway? I'd love to know if I work hard and find a bug I can fix it - but only if I do it before XYZ 19:26:59 clayg: looks like most of the other openstack projects are doing an RC2 next monday I think 19:27:00 notmyname: oh - that's an awesome idea! 19:27:26 notmyname: I mean the 3rd part testing - not doing an rc2 monday 19:27:29 :-) 19:27:37 notmyname: but it's good to know we have a little bit of time 19:27:50 notmyname: I'd be just as happy getting it "right enough" on the first go 19:27:55 kota_: what do you think? 19:28:21 notmyname: i'm also looking into acoles etag issue - trying to understand why the scenario he described which *should* break - is acctually not causing a problem for me in practice 19:28:27 clayg: if we make rebuild configurable but not bump to 1.0.7 then we'd have to make sure you can't select rebuild when using 1.0.6 right? 19:28:37 acoles: "have to" 19:28:40 clayg: wel that could just be because i am wrong 19:28:44 yes, me too. it's beta, and I'd really prefer not to do an RC for a non-open driver that the community doesn't have visibility in to 19:28:47 acoles: it's *beta* - you might loose some data 19:29:35 clayg: also, as much as we put "beta" on it, someone will be using it for prod data somewhere 19:29:44 so fix stuff if we can 19:29:53 clayg: notmyname i agree with you both! 19:29:56 notmyname: you're going to give me an ulcer 19:29:56 kota_: ok, can we come back to this? I want to move on in the meeting 19:30:15 well, I undestood it's beta and we could do testing as 3rd party so maybe not urgent for the rc, but I'd like to fix it as possible 19:30:34 notmyname: thanks to consider that :) 19:30:36 kota_: yes, I agree with that!!! 19:30:54 kota_: I'd totally love a patch to swift master this afternoon as soon as tsg_ has the new version available 19:31:05 kota_: ok - so your best bet is to find a bug in swift that *requires* a rc2 - then we can slip in the reconstruct config option ;) 19:31:06 but that wouldn't be in kilo 19:31:11 yes 19:31:13 clayg: yes 19:31:27 clayg thanks! 19:31:43 yeah that's the other thing - swift development will continue no matter what goes in kilo 19:31:46 ok, so as things stand now, there is no plan for an RC2. if somethign is found, we'll have one 19:31:59 yay! 19:32:15 tsg_: and when you get upstream done, please submit a requirements update. no terrible rush on that, but sooner is better than waiting eg until july 19:32:16 notmyname: just got a final nod on a patch for 1.0.7 from Kevin - we will have pyeclib uploaded by 2pm PDT 19:32:19 thanks 19:32:30 notmyname: will do 19:32:33 thanks 19:32:35 mornign on 19:32:40 #topic summit planning 19:32:52 notmyname: i'd like to make sure we do a good job testing swift with the new version of pyeclib 19:32:53 #link https://etherpad.openstack.org/p/liberty-swift-summit-topics 19:32:59 clayg: of course! 19:33:23 that etherpad is our scratch pad for summit planning 19:33:38 we will have 6 fishbowl sessions and 10 working sessions 19:33:51 and all day friday for "open" 19:34:10 fishbowl == larger session like previous summits 19:34:28 working == smaller session more similar to a room at a hackathon. focused discussion about code 19:34:48 friday will be like friday in paris. open ad-hoc discussions 19:35:28 any questions on logistics before we talk about some of the proposals? 19:36:27 no lets talk proposals :) 19:36:29 ok, let's go though some of the proposals. I want to get a general sense of popularity of them 19:36:42 encryption. acoles and jrichli 19:36:53 current state of on-disk encryption in swift 19:37:10 we talked about this at the hackathon, and I'm happy to see work kick off again for it 19:37:11 If you put the word encrytion in, every man and his dog will probably turn up like last summit 19:37:26 yeah, that leads to my question about it: 19:37:39 acoles: jrichli: do we need to have a second fishbowl one for it? 19:37:51 i would be up for it 19:37:59 you mean in addition to a working 19:38:01 ? 19:38:03 notmyname: *two* fishbolw? 19:38:13 working session schedule titles will be intentionally obfuscated to prevent so many looky-lous from attending and not helping 19:38:15 that makes a fishtank no? :) 19:38:17 sure 19:38:41 yeah, that's what I meant. as second session that is a fishbowl in addition to the existng proposed working session 19:38:49 notmyname: is that like *our* plan? or are other people doing this? 19:38:58 clayg: doing what? 19:39:07 intentionally obfuscated 19:39:22 that's a global thing for the summit 19:39:30 lol 19:39:33 lets call it 'various minor improvements' 19:39:43 lol 19:39:45 lol 19:39:51 acoles: no you title it "said every andriod app evar" 19:39:52 eg they don't want 300 people in a "docker" session 19:40:06 THERE'S A DOCKER SESSION!? 19:40:08 seriously, we must have a session where we can talk details and work out issues 19:40:10 acoles: "metadata of metadata" might scare people off :-) 19:40:14 clayg: not for you! 19:40:15 ie smaller session 19:40:25 well, except storelets. we'll get to that in a minute 19:40:31 jrichli: you're learning ;) 19:40:37 we should setup a honeypot session with docker in the title… 19:40:42 :-) 19:40:56 oh goodness - this is going to be a great summit - i can't wait to see all of ya'll! 19:41:00 lol 19:41:21 notmyname: encrytpion needs a working seession too 19:41:24 ok, I added one 19:41:32 yeah, it's there 19:41:39 cschwede: test framework session 19:41:55 notmyname: how much will the working sessions overlap - can I come to all of them (or the ones that don't lock the door "quick before clayg") 19:41:56 is this a newbie thing or something with new ideas onw hat to do? 19:42:05 clayg: no overlap 19:42:22 I meant it as a newbie thing 19:42:31 jrichli: ah ok. cschwede put hsi name on it 19:42:34 title looks like newbie thing - but cschwede's a nice guy - he'll probably let us coopt it a bit at the end 19:42:51 notmyname: intro for newbies, howto add tests for your patches and tests to improve? 19:42:52 jrichli: cool. we might want to add some other "intro to swift" things to it too 19:42:55 yes, sorry i wasn't actually there when you called out my name earlier today 19:43:02 jrichli: cschwede: what do you think of that? 19:43:13 so it could be (a) heres what we have for newbies and (b) what do we need to improve? 19:43:17 sure, as long as we can get deep and dirty into functests! 19:43:21 heh ok 19:43:22 jrichli: i just volunteered, remove me if you like to lead the session! 19:43:31 notmyname: given "all newbie swift" and "how tests work today - how we'd like you to make them better" - i'd be interested in option 2 19:43:37 cschwede: that must be a joke 19:43:53 clayg: kk 19:44:01 cschwede: I suggested it because I need the info :-) 19:44:01 ops feedback session 19:44:05 mattoliverau: put it down 19:44:18 mattoliverau: I'll work on getting that one also added to the ops track session 19:44:20 jrichli: k :) 19:44:21 I did, cause I think its useful 19:44:35 a new feature of the schedule this summit will be that one session can be listed in 2 tracks 19:44:40 eg swift+ops 19:44:47 killa 19:44:49 I put me down to run it, which I'm happy to do, but also cause I didn't want to volenteer anyone else :) 19:44:56 ok :-) 19:45:01 i love mattoliverau doing it 19:45:05 Large Containers 19:45:07 cool 19:45:12 i love mattoliverau doing that one too 19:45:13 do it 19:45:16 I love this one, because I'd really love to see progress here in liberty!!! 19:45:17 all mattoliverau all the time really 19:45:21 i like large containers... 19:45:23 lol 19:45:28 fishbowl or working, though? 19:45:43 umm... hrd to say - probably fishbowl 19:45:45 idk 19:45:58 bigger group discussing desing and presenting ideas or smaller one talking about the code you've got and with a whiteboard? 19:46:05 * notmyname doesn't know if a whiteboard is available 19:46:14 i want to talk about some with guys high bandwidth and pick some things we think are a) doable b) useful c) moving us to the ultimate goal 19:46:20 mattoliverau: so far it's just your code, so what do you think? 19:46:22 either really 19:46:33 mattoliverau: then what's your preference? 19:46:38 gotta make a choice! 19:46:44 jrichli: i'm really sorry - i used to always say "ya'll" but then in califiornia it's all "you guys" - which seems exclusionariy :\ 19:46:45 i’d prefer smaller and in-depth talks for it 19:46:54 how about working.. unless there is a fishbowl free and I see how much more of the POC i can get done now that EC it out :) 19:47:09 ok 19:47:15 ok working for now 19:47:29 changing policies 19:47:33 s/ok/so/ 19:47:35 whoot! 19:47:36 yeah! 19:47:36 clayg: no worries. I grew up in Delaware and said "you guys" 19:47:38 daisuke isn't here. kota_ do you know of it? 19:48:01 notmyname: are there any obvious candidates to fill the 6 fishbowls - then everyhting else is working? 19:48:03 notmyname: oh... either seems fine to him, I think. 19:48:14 acoles: next, one I think :-) 19:48:19 k 19:48:30 kota_: thanks 19:48:33 storelets 19:48:36 FISHBOWL! 19:48:39 I think this is a fishbowl session 19:48:47 large please! 19:48:54 yeah 19:49:03 increase part_power 19:49:14 it has a variable name in the title. probably a smaller session ;-) 19:49:33 yup 19:49:43 notmyname: yeah cschwede already has like the hard ring part written - so really it's just figuring out how to merge that and what the next patch would be 19:49:45 acoles: I love that you submitted this one. also, good luck, sounds hard ;-) 19:49:53 (we might be able to do the first part before vancouver) 19:49:56 well, ok then 19:50:26 ec follow on work for next steps directly related to ec 19:50:34 I like this one and want it as a workign session 19:50:57 +1 19:51:02 ok 19:51:09 I added a few at the bottom right before the meeting 19:51:15 ec overview 19:51:22 a fishbowl session 19:51:36 +1 19:51:47 as a summary and ops guide and etc. different than the next steps 19:52:06 python-swiftclient working session 19:52:10 to get some next steps there 19:52:12 notmyname: be great if we could sneak in some early performance charictizations 19:52:18 sigh poor swiftclient 19:52:23 clayg: yeah. start running those tests!! 19:52:59 ya, that's why I want to have a working session on swiftclient. so that we do have a plan for the next few months with it 19:53:05 any objections? 19:53:27 * notmyname looks at clock and realizes this meeting has gone by quickly 19:53:39 nope sounds good 19:53:41 acoles: how do you feel about a fast-post session? 19:53:52 +1 19:53:53 working session 19:53:54 ok 19:54:02 happy to lead if you like 19:54:03 global cluster improvements 19:54:14 acoles: done :-) 19:54:31 global cluster improvements is kinda nebulous, but there are a few things to look at 19:54:44 mabe ;-) 19:54:48 +1 19:54:57 kota_: yeah, I knew YOU'D like it 19:55:31 last one there is tape storage 19:55:49 +2! 19:55:51 there have been a few things talked about, but not yet in the dev community. I'd like to try to get someone to talk on it 19:56:01 specifically, IBM and BDE 19:56:08 anyway, just and idea 19:56:24 so, for all of those, this isn't a final list at all, and it's totally unordered 19:56:43 notmyname: isnt' there some specs up that folks wanted to talk about too? 19:56:48 notmyname: should they like... be here? 19:56:48 please keep adding stuff, and if your name is there to lead, make and etherpad and make an outline 19:56:57 clayg: yeah, good point 19:57:08 probably could be one submission for every spec, really 19:57:10 symlinks? 19:57:24 acoles: CRAZYTOWN 19:57:26 symlinks+containeralias 19:57:34 add it! 19:57:46 ya, simlinks should be there! 19:57:52 keep adding stuff this week 19:57:53 i'd like a session on that 19:57:59 cschwede: they may be different things - the PUT == full replace nature of object's makes symlinks at the object layer simpler 19:58:24 I don't have a final date yet, but I'd expect that we'd have a really good idea of sessions by the end of next week 19:58:26 clayg: yes, different in implementation, but user-api should be similar and aligned i think 19:58:36 cschwede: VERY good point 19:58:39 later, we'll rank them and schedule them 19:58:55 anything else to bring up in the last 2 minutes? 19:59:07 keep adding your ideas for the summit to the etherpad 19:59:48 I'll add an entry for the specs that are there 20:00:20 again thank you, everyone, on your hard work getting EC and 2.3.0 done! 20:00:26 I think we'll need a virtual tdasilva so he can join in.. unless the baby to be will keep him too busy ;) RE:summit 20:00:27 see you next week (and on IRC) 20:00:37 mattoliverau: :-) 20:00:45 mattoliverau: ipads with video chat 20:00:50 #endmeeting