19:00:38 <notmyname> #startmeeting swift
19:00:39 <openstack> Meeting started Wed Jan 14 19:00:38 2015 UTC and is due to finish in 60 minutes.  The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:40 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:00:43 <openstack> The meeting name has been set to 'swift'
19:00:46 <notmyname> who's here for the swift team meeting?
19:00:49 <mattoliverau> o/
19:00:52 <tdasilva> hello
19:01:09 <cschwede> o/
19:01:24 <acoles> \o
19:01:27 <notmyname> clayg: peluse: torgomatic: complementary ping
19:01:29 <cutforth> howdy
19:01:36 <torgomatic> hi
19:01:47 <brnelson> o/
19:01:53 <clayg> how lucky that i'm in this channel for a change
19:02:02 <notmyname> :-)
19:02:20 <peluse> yo yo
19:02:23 <notmyname> hello, everyone, in whatever timezone and on whatever day it is for you
19:02:54 <notmyname> I think this should be a fast meeting, but there are a few important things to review or go over
19:03:05 <notmyname> #topic ring placement changes
19:03:16 <notmyname> clayg: torgomatic: looks like (most of) the patch chain has landed
19:03:24 <notmyname> what's current status and next?
19:03:29 <notmyname> just the report patch from clayg?
19:03:31 <torgomatic> enough of it to be useful, at least
19:03:56 <notmyname> clayg: I haven't studied your long gerrit comment yet. any highlights?
19:04:06 <clayg> on which one - the dispersion-report?
19:04:26 <notmyname> https://review.openstack.org/#/c/141452/10 you say "Sigh, I'm not sure we're entirely done with this :("
19:04:29 * clayg didn't realize at first we already have a dispersion report that has nothing to do with ring placement
19:04:32 <notmyname> adding overload
19:04:47 <clayg> notmyname: oh yeah that... i'm sorta bummed how overload ends up working in practice
19:04:53 <notmyname> how so?
19:05:27 <clayg> notmyname: well i guess most of the time you want the old as-unique-as-possible placement until you're either a) doing a topology migraiton or b) have mostly full disks
19:06:06 <clayg> it's nice to have the option now, but in the general sense I'm still seeing rings that don't look entirely unreasonable getting more balance and less dispersion unless you crank overload pretty high
19:06:10 <notmyname> it's as if we can't write a single deployment config that works for everyone!
19:06:33 <clayg> but I know in the general sense that a really high overload is not really what you want because lots of full disks sucks pretty bad
19:06:41 <peluse> you can please all of the people some of the time and some of the people all of the time but....
19:06:42 <notmyname> hmm
19:06:44 <mattoliverau> So can't you leave the overload at 0 until migrating. I just think we need good documentation on it.
19:06:50 <clayg> notmyname: seriously - lets go back to requires 5 zones and 100 disks minimum
19:06:54 <notmyname> lol
19:07:43 <notmyname> clayg: for what mattoliverau said. is that what you'd recommend to people? leave it at 0 until it's needed? and docs?
19:07:47 <clayg> notmyname: also torgomatic has a few rings in his pocket that don't seem to do anything reasonable until you add a bit of overload - and that seems wonky - but he has his debugging stuff now so it's gunna be great
19:08:01 <clayg> notmyname: i think acctually a little overload is great thing
19:08:39 <notmyname> ya, that makes sense to me, from how I understand it
19:08:39 <clayg> notmyname: I think it's easier to give up dispersion when you're facing full disks than to try and add overload once you've already got a bunch of weight
19:09:03 <notmyname> torgomatic: what is the current state of docs for overload?
19:09:18 <torgomatic> notmyname: I wrote some words in the ring overview doc
19:09:33 <clayg> notmyname: the part that bugs me is that we should be able to calculate some of this up front and tell people when they're gunna have a bad time - giving them knobs and letting them fight it out with ring placement turns out to be a real try-and-see mess
19:09:39 <notmyname> torgomatic: anything on config options? man page updates?
19:09:46 <clayg> notmyname: i also wrote words for the dispersion change
19:09:52 <notmyname> ok
19:10:04 <torgomatic> notmyname: no, I didn't update the man page; probably should, though
19:10:06 <clayg> mattoliverau: but more words is find - i'm just not sure what to say at this point
19:10:10 <torgomatic> there are no config options
19:10:40 <mattoliverau> swiftstack just needs to write another in depth blog post on it :P
19:10:43 <notmyname> torgomatic: right. config isn't what I meant. something in the deployment guide or whatever doc has that
19:10:49 <notmyname> mattoliverau: another book...
19:11:04 <mattoliverau> notmyname: lol, yeah or that.. so typey typey
19:11:50 <notmyname> clayg: looks like https://review.openstack.org/#/c/145970/ is the current end of that patch chain (ie 2 patches not landed). anything else expected?
19:12:16 <clayg> i don't think the problem is a lack of words really; but if we don't know the problem trying to explain to people why it's hard probably doesn't hurt
19:12:34 <notmyname> yup. that makes a lot of sense
19:12:42 <clayg> the idea tho is that I'm going to be able to use these knobs in the controller to just "always do the right thing" and I'm not sure what that is yet :\
19:12:57 <notmyname> I have confidence in you ;-)
19:13:21 <clayg> notmyname: I think mattoliverau was going to update the commit message on that one for me ;)
19:13:30 <clayg> but so far yeah - that's the best I've got
19:13:34 <mattoliverau> clayg: lol
19:13:39 <notmyname> anything else expected after the 145970 patch lands? specifically related to overload or ring dispersion?
19:14:28 <notmyname> ie anything currently in progress that you haven't pushed yet?
19:14:47 <clayg> cschwede: you gotta any knowledge to drop on us regarding ring placement?
19:15:23 <cschwede> clayg: nothing that you don’t know yet
19:15:36 <swift-deployer> For container sync, I was asking if the community thought there were existing Swift technology limitations inhibiting its adoption for public cloud (enterprise scale) and relative effort to address if so.
19:15:56 <notmyname> swift-deployer: that's not the topic right now
19:16:12 <notmyname> swift-deployer: let's come back to that at the end of the meeting
19:16:18 <swift-deployer> I apologize.
19:16:22 <swift-deployer> Thank you.
19:16:56 <notmyname> clayg: so no more expected patches? (because I'm driving towards the next release)
19:17:24 <clayg> *I* want to talk about limitations on container-sync!  https://review.openstack.org/#/c/103778/
19:17:33 <clayg> notmyname: no more ring patches from me until i get smarter
19:17:42 <notmyname> ok, thanks
19:17:53 <clayg> notmyname: the ring debugger bits are useful - i have other patches that I think would look good in the next release - but I'm all ring'd out
19:18:04 <notmyname> clayg: torgomatic: thanks for working on this and everyone for getting it merged
19:18:16 <notmyname> #topic next release
19:18:25 <notmyname> with the ring placement changes!
19:18:42 <notmyname> clayg: I saw that the other patch landed too. the container replication one
19:19:10 <clayg> notmyname: cschwede for working on it too!  and also for tricking torgomatic and me into breaking it in the first place!  (where breaking it ~= making it not suck when adding failure domains)
19:19:27 <notmyname> thanks cschwede!
19:19:36 <clayg> notmyname: oh did it?  with the large out of date or whatever - yeah that's good then
19:19:49 <clayg> notmyname: maybe I don't have any other known bugs with fixes to merge
19:19:53 <notmyname> anything else from anyone on patches that should land before cutting a release?
19:19:58 <cschwede> you’re welcome! and thank you too for working on this!
19:20:06 <acoles> notmyname: clayg: yes that has merged (large out of date containers)
19:20:06 <notmyname> after https://review.openstack.org/#/c/145970/
19:20:18 * clayg group hugs everyone
19:20:37 <notmyname> I think after https://review.openstack.org/#/c/145970/ lands then we cut a release to get it out there for everyone. ie next week. 2.2.2
19:20:37 * clayg wonders if we should make I survived working on the ring t-shirts?
19:21:04 <notmyname> everyone ok with that? a 2.2.2 release next week?
19:21:05 * peluse wishes he has contributed now :(
19:21:27 <notmyname> peluse: oh, I'm getting to you. you're up next ;-)
19:21:47 <peluse> well, I meant 'had' so I could get a ring t-shert :)
19:21:48 <cschwede> clayg: yeah! i’m in for it! ;)
19:21:52 <torgomatic> notmyname: what about https://review.openstack.org/#/c/144432/
19:22:08 <torgomatic> nm, I guess that's included by dependency
19:22:12 <notmyname> ya
19:22:31 <notmyname> that and the child patch both land before a release
19:23:31 <notmyname> I hear no objections or other patches that need to land....
19:23:56 <notmyname> I'll talk to ttx later and get the machinery set up. go go gadget openstack process
19:24:11 <notmyname> (sorry for the american childrens tv show nostalgia)
19:24:24 <notmyname> ok, next up
19:24:29 <notmyname> #topic ec status
19:24:36 <peluse> rock n roll
19:24:38 <notmyname> peluse: do we have read/write done yet
19:24:53 <peluse> #link https://trello.com/b/LlvIFIQs/swift-erasure-codes
19:24:57 <peluse> is all up to date....
19:25:16 <peluse> reconstructor still being overhauled but coming along very nicely, should be able to push something soon
19:25:28 <notmyname> ok
19:25:34 <peluse> tsg got the eventlet guys to do another relase so we can unblock the completion of PUT so that's coming very soon
19:25:43 <notmyname> and clayg has been doing ring stuff and not GETs. and that's ok
19:25:53 <peluse> he already has a review for the new eventlet version
19:26:06 <clayg> eventlet 0.16.1 FTW!
19:26:11 <peluse> and yeah, when clayg gets done messing around with rings, that'll be the GET side of things
19:26:12 <notmyname> ah, interesting. I overheard something from -infra people about a new eventlet problem. I need to check if there are any problems there
19:26:36 <peluse> yeh, it was something about building from source
19:26:50 <clayg> notmyname: something fixed back in oct-ish was on greenlet's __del__ going maximum recursion - all fixed up in the new hawtness
19:27:10 <notmyname> cool
19:27:26 <peluse> yeah, what we needed was the ability to do more than 1 100-cont and that was there but then the unrelated build problem, thus the new one
19:27:44 <notmyname> peluse: the EC section of https://wiki.openstack.org/wiki/Swift/PriorityReviews is up to date too?
19:27:51 <peluse> anyway, that's about it.  reviews page is also up to date, Yuan has a few that are WIP and map back to trelli
19:27:52 <notmyname> gotcha
19:28:01 <notmyname> great, thanks
19:28:08 <peluse> trello that is
19:28:16 <notmyname> trelli is the plural?
19:28:57 <notmyname> #topic swiftclient
19:29:02 <notmyname> /justbecause
19:29:35 <notmyname> anything here? there's a couple of outstanding bugs listed on the priority reviews page. and has anyone looks at openstack-sdk work recently?
19:29:37 <mattoliverau> clayg: you now what your commit message RE: 145970 ;)
19:30:15 <clayg> mattoliverau: man, that is a *fine* commit message - you're like a poet
19:30:20 <notmyname> lol
19:30:38 <clayg> notmyname: i'm not sure the openstack-sdk thing is going to pan out :\
19:31:11 <notmyname> certainly not something we're spending much time on, as a group
19:31:19 <mattoliverau> clayg: why thank you, I used mostly your own words from comments you wrote inline, so there is a poet inside of you somewhere :P
19:31:46 <notmyname> #topic open discussion
19:32:18 <clayg> notmyname: it's just gunna be hard and we still have a lot of work to do from swiftclient.services on up; plus I think the dependency is gunna be annoying and we never dicussed a deprecation strategy for the existing swiftclient.client except to nebulous idea that we'd try to stop using it as the sdk depends got better ish?
19:32:32 <notmyname> I'm working on setting up hackathon details. invites should be public next week
19:32:42 <acoles> notmyname: there's at least one other swiftclient bug fix i would add to the list if thats ok https://review.openstack.org/125759
19:32:44 <notmyname> clayg: oh I totally agree
19:32:48 <notmyname> acoles: yes
19:32:58 <brnelson> do you have the hackathon location yet?
19:33:00 <clayg> notmyname: I think acoles and I have some outstanding patches to swiftclient that would be pretty good - joel is trying to fix something with downloading huge containers
19:33:11 <clayg> oh - hi acoles !
19:33:23 <acoles> clayg: howdy, yes there's joel's stuff too
19:33:26 <notmyname> brnelson: yes. san francsico
19:33:51 <clayg> acoles: sorry i haven't been working on fast-POST; fwiw at some point I decided you were right about everything and the crushing blow to my ego has been enough to keep easily distracted by other things
19:34:27 <acoles> clayg: lol. hey, you may yet be proven right, i sometimes wake up in a cold sweat about it all :)
19:34:47 <notmyname> swift-deployer: what were you asking about container sync?
19:34:55 <acoles> clayg: i still have bunch of tests to write
19:35:13 <swift-deployer> I was asking if the community thought there were existing Swift technology limitations inhibiting its adoption for public cloud and relative effort to address if so.
19:35:14 <clayg> acoles: maybe while you're still being frustated by lack of useful contribution you could update the spec with your current line of work so I can at least approve that as the most likely path forward to eventual success
19:35:46 <clayg> *I* have opinions about how to make container sync suck less - they are very similar to my opinions on how to make the reconciler scale better
19:35:47 <notmyname> swift-deployer: mostly it's along the lines of cluster interconnect capacity.
19:36:07 <clayg> oh... well yeah... you need lots of bandwidth - duh
19:36:08 <notmyname> clayg: (1) think real hard (2) type in better code
19:36:20 <acoles> clayg: will do, i was holding off doing so in case i crashed and burned with the actual code
19:36:26 <swift-deployer> can you please elaborate on that?
19:37:20 <notmyname> swift-deployer: if you've got a billion objects or lots of TB/PB to sync, it can take a long time.
19:37:36 <clayg> acoles: nah, it's gunna be great, even if the code is hard it'd be worth it to have a plan written down that I believe should theoritically be solveable - my previous thinking I have finally proven to myself was flawed - it was an interesting proof, but not terribly so since it only proved I can't do it the way I wanted :'(
19:38:40 <clayg> notmyname: if anyone is interested in making container-sync and the reconciler faster they should consider how many 404's you get when you send a PUT with an x-timestamp and what happens if you get a 409 (I already have that x-timestamp) from a node in the proxies connect_put_nodes
19:39:04 <clayg> notmyname: then review https://review.openstack.org/#/c/103778/
19:39:40 <acoles> clayg: so are you ok with me pushing a revised spec over your version?
19:40:12 <clayg> acoles: my version was shit, everything in there is garbage, I'm an idiot - anything is better than what's there, what's there won't work
19:40:28 <clayg> acoles: so - yeah knock yourself out bro!
19:40:39 <clayg> acoles: i'd consider it a kindness
19:41:01 <acoles> clayg: ok that sounds like a great commit message :D
19:41:07 <notmyname> lol
19:41:14 <notmyname> clayg: I added it to the priority reviews page
19:41:20 <mattoliverau> I think clayg needs a hug
19:41:40 <acoles> clayg: actually fwiw somewhere some good stuff cross fertilised imho
19:41:50 <clayg> notmyname: the x-timestamp thing?  meh, it doesn't need +2's as much as more people telling me I'm an idiot
19:42:04 <clayg> I know what the problem is - i just need more eyes to flesh out the solution
19:42:12 <notmyname> mattoliverau: I'll have to take your hug back to clayg next week ;-)
19:42:29 <mattoliverau> notmyname: you do that
19:43:03 <notmyname> ok, let's call it
19:43:11 <notmyname> thank you everyone for coming and participating
19:43:15 <notmyname> thank you for working on swift
19:43:31 <notmyname> (I get to say nice things about you in my LCA talk today)
19:43:39 <mattoliverau> yay
19:43:39 <notmyname> #endmeeting