21:00:58 <notmyname> #startmeeting swift 21:00:59 <openstack> Meeting started Wed Jul 22 21:00:58 2015 UTC and is due to finish in 60 minutes. The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:01:01 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:01:03 <openstack> The meeting name has been set to 'swift' 21:01:06 <notmyname> who's here for the swift meeting? 21:01:08 <wbhuber> here 21:01:09 <minwoob_> Here 21:01:09 <jrichli> o/ 21:01:11 <mattoliverau> o/ 21:01:11 <torgomatic> ䷣ 21:01:12 <hurricanerix> \o/ 21:01:12 <ho> o/ 21:01:15 <blmartin> o/ 21:01:16 <kota_> yey 21:01:21 <robefran> o/ 21:01:30 <MooingLemur> 🐄 21:01:32 <tdasilva> hello 21:01:48 <notmyname> welcome! 21:01:58 <notmyname> agenda for the meeting is at 21:01:59 <notmyname> #link https://wiki.openstack.org/wiki/Meetings/Swift 21:02:16 <janonymous> o/ 21:02:17 <notmyname> there's a few things to go over, but mostly I'd like to get an update of a few things 21:02:22 <notmyname> #topic ops meetup 21:02:33 <notmyname> first up, the openstack ops meetup is upcoming 21:02:34 <redbo> hi 21:02:44 <notmyname> #link https://www.eventbrite.com/e/openstack-ops-mid-cycle-meetup-tickets-17703258924 21:02:50 <notmyname> #link https://etherpad.openstack.org/p/PAO-ops-meetup 21:03:07 <notmyname> this is a midcycle thing that is for people who run openstack 21:03:41 <notmyname> normally the conversation is around other projects than swift, but I do try to go to them all to represent and answer questions and listen 21:03:51 <clayg> notmyname: thanks! 21:03:54 <notmyname> if you're interested in attending, I wanted to make sure you had the info 21:04:16 <notmyname> this next one is in august in the Bay Area in california 21:04:39 <notmyname> ah, yes. palo alto 21:05:22 <notmyname> if you're interested and want more info, I'll do my best to help 21:05:29 <notmyname> #topic ongoing work 21:05:37 <notmyname> ok, for the stuff that's going on 21:05:48 <notmyname> first, I want to set the context: 21:06:05 <notmyname> top priority in my mind right now is to take care of the outstanding EC bugs 21:06:11 <notmyname> when that happens, I want to cut a swift release 21:06:18 <cutforth> sorry, i'm late 21:06:37 <notmyname> if we do that soon, we should be able to get another release for liberty with no problem rgiht at the end of the cycle 21:07:19 <notmyname> then there's the encryption work, which I'd love to have in liberty if possible, and the version/copy middleware refactors (which encryption depends on) 21:07:25 <notmyname> so let's take each of those in turn 21:07:29 <notmyname> first up, EC bug status 21:07:36 <notmyname> #link https://bugs.launchpad.net/swift/+bugs?field.tag=ec 21:07:43 <notmyname> the bug page looks pretty good, actually 21:07:54 <notmyname> but there are a few that are still open 21:08:09 <notmyname> peluse had a conflict 21:08:13 <notmyname> clayg: can you give an overview of the current EC bug focus 21:08:15 <notmyname> ? 21:08:42 <clayg> what'd i do 21:08:57 <clayg> just pauls thing mainly 21:09:13 <clayg> other than 503's on GET (you know minor stuff) it's startin' to look pretty good 21:09:20 <notmyname> good 21:09:25 <clayg> kota_: is still tracking down stuff with EC ranged requests 21:09:32 <notmyname> is there one where reviews should be focused? 21:09:35 <notmyname> thanks kota_! 21:09:40 <clayg> ummm.... 21:09:43 <notmyname> or are there any without patches yet? 21:09:44 <minwoob_> I have a couple that need review. 21:09:44 <kota_> : ) 21:09:50 <minwoob_> https://review.openstack.org/#/c/196848/ 21:10:19 <clayg> minwoob_: oh you added a probetest! 21:10:20 <minwoob_> https://review.openstack.org/#/c/198909/ 21:10:35 <notmyname> minwoob_: thanks. 21:10:48 <notmyname> I want to keep the EC related reviews starred (I just starred those 2) 21:11:43 <clayg> is enospc one really EC related - it's just like better logging or something? 21:12:00 <clayg> ... no tests 21:12:18 <notmyname> I'll follow up with charz about https://bugs.launchpad.net/swift/+bug/1468708 which is the only EC bug that needs more info right now 21:12:20 <openstack> Launchpad bug 1468708 in OpenStack Object Storage (swift) "ssync receiver get an odd SSYNC command and raise a exception" [Undecided,Incomplete] 21:12:33 <uvirtbot> Launchpad bug 1468708 in swift "ssync receiver get an odd SSYNC command and raise a exception" [Undecided,Incomplete] 21:12:34 <uvirtbot> Launchpad bug 1468708 in swift "ssync receiver get an odd SSYNC command and raise a exception" [Undecided,Incomplete] https://launchpad.net/bugs/1468708 21:12:49 <notmyname> minwoob_: why is that one tagged EC? 21:12:50 <torgomatic> thank you, everybot 21:13:09 <clayg> torgomatic: lol! 21:13:09 <redbo> I liked how the one bot triggered the other bot. 21:13:20 <redbo> I wonder if there's some way to start a war 21:13:23 <minwoob_> notmyname: I think I opened it up when I was looking at another EC bug. 21:14:05 <notmyname> minwoob_: seems like a pretty general sort of thing and not EC specific 21:14:38 <clayg> notmyname: minwoob_: yeah take the ec tag off that one 21:14:44 <notmyname> clayg: minwoob_: done 21:15:13 <notmyname> ok, anything else to cover on EC? everyone know what to look at there? 21:15:53 <notmyname> clayg: anything else on this topic? 21:15:58 <clayg> notmyname: the ec bug list is where its at 21:16:04 <notmyname> ok, good 21:16:06 <clayg> notmyname: we had some great stuff merged 21:16:15 <clayg> notmyname: I really think EC beta is in good shape 21:16:20 <notmyname> yup. thank you, everyone, for looking at all of that 21:16:35 <clayg> notmyname: well keep testing and polishing and come Tokyo... it'll practially be a thing! 21:16:46 <notmyname> the last question on EC will be what is "required" for a new release vs what can wait until later 21:17:07 <notmyname> I'll be happy to work with clayg and peluse and kota_ on that later 21:17:24 <clayg> notmyname: yeah i'd like to know more about what you mean there 21:17:55 <clayg> notmyname: every release from now on will have something that makes ec better in one way or another, so you can't make releases wait just cause ec is improving 21:18:34 <wbhuber> notmyname: what is required also is probably a better understanding of how performance improves using ec vs replication over time 21:18:44 <notmyname> of course. the current EC beta is pretty broken for people who aren't tracking master. the question is if there is another known thing that hasn't been patched that would prevent people from using EC 21:19:01 <notmyname> wbhuber: yeah, the perf comparisons are also an ongoing thing 21:19:05 <clayg> oh - has there not been a swift release since Kilo?! 21:19:09 <notmyname> clayg: yeah 21:19:20 <clayg> notmyname: oh oic 21:19:23 <notmyname> err..right. there has been nothiing since kilo 21:19:39 <clayg> notmyname: well pauls patch is going to be huge - and kota's patches fix some stuff that's really importnat to get out of GA 21:19:47 <notmyname> ok 21:19:48 <clayg> but the big blockers that were breaking reconsttruction are fixed 21:20:01 <clayg> if other things are ready for a tag - let's do it - EC is still beta 21:20:06 <notmyname> ok, good 21:20:25 <notmyname> let's chat this week in IRC about it and perhaps we can do a release really soon, then! 21:20:41 * notmyname is happy about that 21:20:49 <clayg> me too 21:20:55 <notmyname> ok, next up for ongoing work 21:21:05 <notmyname> encryption 21:21:19 <notmyname> jrichli put together a list of the big stuff this week. thanks for that 21:21:29 <jrichli> np 21:21:34 <notmyname> #link https://etherpad.openstack.org/p/swift_encryption_issues 21:22:02 <notmyname> so, as I understand it, there's of course a lot ot do, but the big issue is that there are some actual design choices that need to be made 21:22:06 <notmyname> jrichli: can you expand on that? 21:22:35 <jrichli> yes, it sort of depends on what issue you want to talk about 21:22:54 <jrichli> footers is hard because it will involve some refactoring of EC 21:23:03 <clayg> lets' talk about 38 patches to versioned writes middleware and - how many core reviews? 21:23:19 <notmyname> clayg: yup. htat one's next 21:23:25 <tdasilva> sorry about the 38 21:23:51 <notmyname> jrichli: the refactoring is because encryption needs footers and that's EC-only right now 21:23:52 <jrichli> conditional GETS are hard because we need info saved-off with the object in order to decrypt the etag: not only iv, but crypto-alg used 21:24:01 <notmyname> I think torgomatic has been wanting to unify that too 21:24:07 <clayg> oh nm, everyone's looked at it at one point :'( 21:24:23 <notmyname> jrichli: that's actually something I'm curious about 21:24:28 <jrichli> notmyname: correct. I have talked with torgomatic on this, and I have identified some steps that I can take towards the real footers support. Also, acoles_away will be back next week to help with that. 21:24:40 <notmyname> ok 21:25:12 <clayg> jrichli: I get the fake footers - and I'm keen on moving the middleware extractions forward 21:25:19 <clayg> jrichli: I don't understand the conditional get issue 21:25:29 <notmyname> right now you're mentioning a few existing features that might not be possible with encryption. like conditional gets 21:25:35 <clayg> jrichli: can you write up a tl;dr and a everything-you-need-to-know plus *OPTIONS* 21:25:58 <clayg> like realistically we could solve this by a) b) c) which suck for x) y) z) - now what? 21:26:03 <jrichli> clayg: whaa? 21:26:18 <jrichli> clayg: oh, i see 21:26:29 <clayg> jrichli: most of the time when I have a hard problem that I can't decide how to solve it's because all of my solutions suck 21:26:43 <clayg> but most of the time it's just a tradeoff thing - so you pick the least sucky 21:26:54 <jrichli> do I put this in the etherpad, or maybe keep that one a pristine list, and make a new one for expansions? 21:26:58 <clayg> sometimes it takes rubber ducking it - or getting some more eyes to see which way is going to suck the less 21:27:18 <clayg> ML, gist, etherpad, email it to notmyname and make him sort it out - w/e 21:27:24 <notmyname> jrichli: I'd love for it in the current etherpad 21:27:27 <kota_> or spec? 21:27:39 <clayg> oh god - not a spec 21:27:40 <mattoliverau> or in sky writing 21:27:46 <jrichli> lol 21:27:50 <notmyname> not in the spec for now. this is more about figuring out the ideas 21:27:59 <kota_> k 21:28:01 <jrichli> ok, i will add options details to etherpad for the issues 21:28:15 <notmyname> jrichli: thanks. I think that would be really helpful for everyone 21:28:32 <notmyname> any other question on encryption right now? 21:28:53 <clayg> notmyname: three is enough 21:29:03 <jrichli> I think we need a merge from main sometime soon 21:29:09 <jrichli> s/main/master/ 21:29:24 <notmyname> jrichli: it's good to do that very often. I can help with that. or peluse or acoles 21:29:27 <jrichli> oops - my clearcase is showing 21:29:28 <redbo> just don't start one then abandon it 21:29:37 <clayg> if anyone uses vagrant-swift-all-in-one and wants to take a "quikc peek" at encyrption try the crypto branch and whatever-is-the-lastest-patch-jrichli-says-to-use-for-encrytpion-review and give it whirl 21:29:37 <notmyname> redbo: gotta commit 21:30:13 <jrichli> latest is at patch 203454 21:30:13 <patchbot> jrichli: https://review.openstack.org/#/c/203454/ 21:30:24 <notmyname> ok 21:30:44 <clayg> jrichli: can you update the priority reviews wiki - we need a "end-of-the-chain" link 21:31:05 <jrichli> clayg: will do 21:31:19 <notmyname> thanks 21:31:28 <notmyname> ok, now for the 38-patch-set middleware refactoring 21:31:38 <notmyname> #link https://review.openstack.org/#/c/134347/ 21:32:04 <notmyname> this is a requirement for the copy middleware and the copy middleware is also a requirement for encryption (and good for other things too, I think) 21:32:07 <clayg> so one part of that which has been tricky to review is the upgrade matrix testing 21:32:22 <clayg> i think it's good now 21:32:23 <notmyname> clayg: do we have that in an etherpad or something anywhere? 21:32:34 <notmyname> the upgrade matrix? ie what to look for? 21:32:36 <clayg> umm... tdasilva? 21:32:43 <clayg> no i didn't write it down 21:32:49 <tdasilva> nope 21:33:03 <clayg> tdasilva: we should write that down 21:33:26 <tdasilva> clayg: yeah, i'll think of something 21:33:40 <clayg> tdasilva: you know what I mean? like w and w/o versions from no middleware to middleware with container option on/off to middleware on/off 21:34:01 <notmyname> that would be great to have 21:34:19 <clayg> notmyname: we have better than that tho - we have working code now I think! 21:34:20 <tdasilva> clayg, notmyname: understood, I can work on that 21:34:29 <notmyname> lol 21:34:42 <notmyname> tests pass! merge it! 21:34:47 <mattoliverau> lol 21:35:25 <notmyname> the one after the versioning middleware patch is the copy middleware 21:35:26 <notmyname> #link https://review.openstack.org/#/c/156923/ 21:35:34 <notmyname> is there anything to say about that before the versioning one lands? 21:35:41 <notmyname> tdasilva: ? 21:36:07 <tdasilva> notmyname: not really...just needs eyes... 21:36:11 <notmyname> ok 21:36:29 <tdasilva> ppai left a todo there 21:36:31 <notmyname> tdasilva: so you'll work on typing up the different test options in the next few days? 21:36:36 <notmyname> for the versioning one 21:36:42 <tdasilva> which was based on a conversation i had with torgomatic 21:36:46 <notmyname> ok 21:36:47 <mattoliverau> pfft it's only had 14 patchsets, early days yet :P 21:36:56 <notmyname> *groan* 21:36:59 <tdasilva> hahaha 21:37:08 <clayg> no no no - we need these patches 21:37:16 <notmyname> just look at the "update from global requirements" one ;-) 21:37:20 <notmyname> it's at 200 I think 21:37:26 <clayg> don't need that one 21:37:28 <notmyname> right 21:37:29 <mattoliverau> wow 21:37:42 <tdasilva> notmyname: i'll work on the matrix tomorrow 21:37:46 <notmyname> thanks 21:37:50 <tdasilva> will add a link as a comment on the pathc 21:38:13 <notmyname> goal before the next meeting is to merge the versioning middleware, with tdasilva's help by describing the different test scenarios to think about 21:38:24 <notmyname> ok, next up 21:38:29 <notmyname> hummingbird 21:38:36 <notmyname> hurricanerix: redbo: what's going on in goland? 21:39:02 <clayg> i was in texas last week at my mom's place and this hummingbird was guarding the feeder - like dive bombing anyone else that tried to get close to it 21:39:10 <redbo> Things are trucking along. 21:39:53 <notmyname> redbo: are there any plans yet to show relative performance differences between the 2? something in tokyo would be awesome 21:40:07 <notmyname> request rates, resource usage, latency, etc 21:40:12 <clayg> something in austin would be even better 21:40:16 <redbo> Yeah, so vote for our talk, and we'll give you all the numbers you want :) 21:40:17 <notmyname> indeed :-) 21:40:27 <notmyname> umm..show the numbers no matter what ;-) 21:40:35 <redbo> like.. pie charts. and pictographs 21:40:47 <notmyname> pictographs of pie charts? 21:40:56 <redbo> each pie chart represents 1,000 pie charts 21:41:28 <notmyname> but seriously, soemthign that can show the difference and methodology so others can reproduce it is crucial 21:41:33 <redbo> Yeah, I'd like to have something moderately scientific by Austin. 21:41:40 <MooingLemur> https://xkcd.com/688/ 21:41:43 <notmyname> awesome 21:41:43 <clayg> well wait - we don't just need picture graphs 21:41:56 <notmyname> clayg: he said "moderately scientific" 21:42:11 <clayg> yeah we need to get some method and goals written down so we're all in the same page 21:42:26 <redbo> well, I'd like moderately scientific graphs, and also production graphs, which would probably be hard to reproduce. 21:43:04 <notmyname> "build a 50PB cluster" can't be step 1 ;-) 21:43:06 <clayg> in vancouver the plan was to use this cycle to validate that we want to pursue this in tree - i'm seeing rackspace make some solid incremental improvements to the go code base and I've more or less been able to "run it" barring a few missing features 21:43:42 <clayg> but I'm no where near being able to support it - I don't think any one is - the python and go object servers are continuing to diverge 21:44:15 <clayg> we still don't have any idea what a integration validation testing framework would look like (I think dfg said he had some python tests?) 21:44:34 <notmyname> looks like there are some CI jobs that you're running redbo 21:44:46 <clayg> really?! 21:44:53 <redbo> It's currently running one of our clusters. If you can tell which one it is, you win. 21:45:04 <dfg_> ha 21:45:08 <notmyname> not openstack CI 3rd party stuff, jsut a bot that comments 21:45:20 <redbo> Yeah, I just run swift functional tests with it. 21:45:21 <dfg_> how do you mean "continuing to diverge" ? 21:45:44 <clayg> dfg_: just that there's some pretty interesting behavioral changes that are being added to the go lang object servers 21:46:03 <dfg_> it should only be with replication 21:46:05 <notmyname> I'm working with the openstack infra people this week to get the per-branch test jobs. redbo I'd love to get those as more normalized jobs 21:46:26 <clayg> like how the page cache is used, and when the fsyncs are done, when errors are returned if a disk is busy 21:46:34 <notmyname> there's also been the multiple servers per port stuff on the python side (ie different deployment methods) 21:46:35 <dfg_> oh that :) 21:46:37 <redbo> That'd be fine. Right now it's just a jenkins box that runs go unit tests and swift functional tests backed by hummingbird. 21:46:50 <notmyname> redbo: ok, i should have more info for you next week 21:47:34 <clayg> dfg_: was that about all the awesome tricks you guys are adding to the go code, or about swifterdarrls awesome tricks he's been adding to the python code? 21:48:41 <redbo> most work right now is going into trying to make replication better. 21:49:06 <dfg_> clayg: no. i thought you were talking about the asyncronous container updates. 21:49:09 <clayg> how are you measuring and quantifying "better" 21:49:22 <dfg_> replication passes mostly. 21:49:23 <clayg> dfg_: well i guess that's one thing that both servers will have soon 21:49:29 <redbo> What a handsome question, I'm glad you asked 21:49:33 <dfg_> clayg: ya thats what i thought 21:49:35 <clayg> dfg_: although I don't think they decided on the same config options or behavior :'( 21:49:54 <dfg_> clayg: ya- we made it non configurable. 21:50:14 <redbo> Reducing replication pass times (checkmark), and increasing the percentage of things that successfully sync (no checkmark) 21:50:17 <notmyname> dfg_: what's the state of the hummingbird functionality. what's "mostly"? 21:50:36 <dfg_> notmyname: it should be 100% for object server. 21:50:41 <dfg_> and replication 21:50:53 <notmyname> ok, great 21:51:11 <dfg_> oh- for what we do. we don;t have storage policies for example 21:51:17 <notmyname> have you had a chance to publish the obejct server tests yet? you mentioned somthign at the last summit 21:51:18 <notmyname> ah 21:51:27 <notmyname> yeah, that's a big deal ;-) 21:51:39 <dfg_> ya- we know. we just haven;t done it yet 21:52:12 <dfg_> there's a ton of work to be done and we're "prioritizing" :) 21:52:19 <clayg> notmyname: it's never going to XXX% the same 21:52:33 <dfg_> ya 21:52:38 <clayg> notmyname: people will make the things they care about work and the things they don't wont 21:52:45 <notmyname> right 21:53:00 <notmyname> so in austin we should expect to see some sort of test plan and some numbers that we can try to duplicate in our own clusters 21:53:02 <clayg> notmyname: but what I would like to have is a place where I can put a test that shows - given this - the api does that 21:53:14 <dfg_> notmyname: ya thats sounds reasonable 21:53:28 <notmyname> I'm ok with limited fucntionality that's shown to be better. no need to require "must be perfect" before saying it's a real thign 21:53:45 <clayg> notmyname: and then I can setup some stuff to point those tests at a and b and start to quantify things that are known to be different - or write a test case when I think they sould be the same 21:54:24 <notmyname> I think we all want the same thing 21:54:41 <clayg> notmyname: do we? 21:54:59 <notmyname> clayg: well, it sounds like it 21:55:11 <dfg_> we can talk about that in austin :) 21:55:15 <clayg> dfg_: redbo: in addition to numbers that are heavy in the methodology - are you interested in a test suite that will allow the community to collaborate on the implemeatino of the go server? 21:55:18 <glange> http://www.usatoday.com/story/news/nation-now/2014/07/15/six-californias-tim-draper/12661161 <-- that's what we want in Texas 21:55:26 <notmyname> the first thing is being able to have more than "we said it was faster" and actually be able to have something that is reproducable 21:55:46 <dfg_> notmyname: +1 21:55:48 <notmyname> clayg: that's a great question 21:55:54 <clayg> notmyname: well there's like a whole deployment issue thing 21:55:58 <redbo> clayg: a test suite to allow collaboration? how do you mean? 21:56:14 <clayg> notmyname: like I can't just take my existing chef & puppet and point it at the hummingbird branch and deploy 21:56:38 <notmyname> why is that a requirement for contributing to it? 21:56:42 <redbo> Oh. Maybe. I don't know how any of that stuff works. 21:57:04 <dfg_> clayg: ya- i just don't want to put the cart in front of the horse i guess. i'd like for the code base to be more fleshed out before we get a ton of new stuff added 21:57:06 <notmyname> maybe I'm confusing the prod/vSAIO issues 21:57:18 <dfg_> and we have very specific things we need solved 21:57:42 <notmyname> dfg_: I'd love to put those on the wiki so we can see where the areas of focus need to be 21:57:45 <clayg> notmyname: redbo: I think i'm saying that if there's a place to write tests that can run on both servers - someone will write a test that works on python on fails on go (storage policies, ec, w/e) - then they'll want to make a patch that fixes the test 21:58:11 <dfg_> notmyname: i think we should wait for the hackathon. would be more productive i think 21:58:18 <notmyname> clayg: got it 21:58:35 <notmyname> dfg_: there's a lot of days between now and the hackathon. no need to add delays 21:58:39 <redbo> Oh, yeah. That'd be nice. We kind of have that, but for some reason it grew weird arms where it wants to restart processes and talk to both servers at the same time and stuff. 21:58:39 <clayg> notmyname: see what dfg_ said - I think this is basically still a rackspace experiment? Maybe merged to mainstrain is not a Tokyo goal? 21:58:47 <mattoliverau> I think we are going to run out of time 21:58:51 <notmyname> mattoliverau: :-) 21:59:11 <dfg_> notmyname: and in those days I want to get better answers :) 21:59:18 <clayg> mattoliverau: well we have BOTH dfg and redbo talking to us - it's like Christmas in July! 21:59:23 <redbo> So in conclusion, vote for me 21:59:32 <clayg> lol - love it! 21:59:33 <mattoliverau> clayg: +1 21:59:49 <tdasilva> so is the plan still to "use this cycle to validate that we want to pursue this in tree" or to have it merged but tokyo timeframe? 21:59:58 <notmyname> that's a happy note to end on. we're out of time 22:00:15 <notmyname> tdasilva: validate. not merge, IMO 22:00:33 <notmyname> if it bears fruit, then figure out what it means to have it "upstream" 22:00:44 <notmyname> thanks everyone for coming. thanks for working on swift! 22:00:47 <notmyname> #endmeeting