21:00:00 <notmyname> #startmeeting swift
21:00:01 <openstack> Meeting started Wed Nov  2 21:00:00 2016 UTC and is due to finish in 60 minutes.  The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:02 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:04 <openstack> The meeting name has been set to 'swift'
21:00:08 <notmyname> who's here for the swift team meeting?
21:00:11 <timburke> o/
21:00:12 <jrichli> here
21:00:14 <dmorita> Hi
21:00:17 <mmotiani> hi
21:00:18 <hurricanerix> o/
21:00:19 <ntata> hello
21:00:19 <mattoliverau> o/
21:00:20 <cutforth> o/
21:00:20 <hosanai> o/
21:00:21 <sgundur> hi
21:00:21 <pdardeau> hi
21:00:22 <nadeem> o/
21:00:31 <bkeller`> o/
21:00:33 <tdasilva> hi
21:00:41 <kota_> hi
21:00:46 <torgomatic> .
21:01:04 <notmyname> welcome everyone
21:01:34 <notmyname> I hope those who were in Barca got home ok, and for those not there, you were missed
21:01:56 <notmyname> agenda fro this week is at
21:01:57 <notmyname> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:02:05 <notmyname> #topic Note the time change
21:02:27 <notmyname> we're in that fun part of the year where various governments tell us to change our clocks
21:02:38 <joeljwright> :)
21:02:39 <notmyname> I think the EU changed last weekend. the US changes this coming weekend
21:02:48 <joeljwright> is clayg using that as an excuse?
21:03:04 <notmyname> remember that our meeting is at 2100UTC (which doesn't change)
21:03:42 <notmyname> please note your current timezone and adjust your calendar as needed
21:04:09 <notmyname> for me, next week's meeting will be 1 hour earlier (1pm pacific)
21:04:47 <notmyname> also, something that has helped me is to know that Iceland's timezone is GMT+0 and they don't change throughout the year. so repeating events scheduled at UTC can be set to Iceland time
21:05:00 <notmyname> (eg google calendar doesn't support setting something to UTC)
21:05:05 <mattoliverau> ^i do that thing too
21:05:06 <notmyname> I think Ghana does the same thing
21:05:39 <notmyname> IIRC, the northern winter / souther summer will actually make meeting times slightly easier for most people
21:05:50 <notmyname> ok, that's enough about that :-)
21:06:05 <notmyname> #summit retrospective
21:06:14 <notmyname> let's talk about last week's summit!
21:06:50 <notmyname> I want to cover good things, bad things, things to improve, summary of topics for people who weren't there (each in turn)
21:07:00 <notmyname> so first up...
21:07:09 <notmyname> how was it? what did you like about last week's summit?
21:07:47 <jrichli> I liked having a design summit :-)
21:08:31 <jrichli> the rooms were a good size
21:08:32 <notmyname> I liked the solid two days we had for the design summit. I thought it flowed pretty well (and they had to kick us out at the end)
21:08:38 <acoles> sorry I'm late (just remembered clock change)
21:08:47 <notmyname> acoles: glad you made it :-)
21:08:56 <notmyname> acoles: just talking about the summit. first question: what did you like?
21:09:04 <tdasilva> it was great to see so many devops engaged in the discussion and providing their feedback
21:09:11 <acoles> that ^^
21:09:41 <acoles> really good to have a design discussion and be able to turn to an op and ask their opinion
21:09:52 <mattoliverau> +1
21:09:57 <joeljwright> +1
21:09:57 <notmyname> yeah, having two ops sessions were good
21:10:03 <acoles> e.g. on replication/golang and on keymaster v2
21:10:08 <kota_> +1
21:11:07 <notmyname> TBH I think one of my favorite parts of the week was simply exploring barcelona with other swift people :-)
21:11:38 <jrichli> +1
21:11:40 <tdasilva> yeah, barça was pretty cool, food was great
21:11:57 <joeljwright> I don't remember much exploring… lots of eating and drinking… not much exploring
21:12:03 <acoles> my body has not yet forgiven me for the late nights!
21:12:04 <jrichli> lol
21:12:37 <notmyname> for the conference parts, we had a lot of user stories and swift-related talks on tuesday
21:12:39 <bkeller`> i took some photos of the monday before the summit, which includes many swift folk: https://www.flickr.com/photos/146035486@N03/albums/72157674298108501
21:13:00 <notmyname> wednesday there were cross-project sessions. thursday and friday were design sessions
21:13:27 <notmyname> ah yes. I think acoles remembers that spiral staircase :-)
21:13:49 <acoles> notmyname: are we on to the what was bad topic already? ;)
21:13:53 <notmyname> lol
21:13:56 <joeljwright> :)
21:14:02 <notmyname> ok, what could we have improved this week?
21:14:03 <acoles> I liked that Alexis came to the swift room and engaged in conversation with us over oslo things.
21:14:12 <notmyname> acoles: yes, that was great!
21:14:14 <tdasilva> oh, i missed that
21:14:47 <acoles> Alexis is an HPE colleague who I discovered has worked on oslo config and other things
21:14:56 <tdasilva> cool!
21:15:06 <acoles> so some of us were able to explore exactly how that might ever work out for us
21:15:18 <mattoliverau> Yeah he was cool
21:15:26 <acoles> He also sits about 20 yards from me in Bristol
21:15:43 <acoles> but I had never made the connection with oslo
21:16:23 <notmyname> I hope some of those conversations bear fruit. I learned of some outdated misconceptions I had
21:16:53 <clayg> I was in opensack-meeting
21:17:05 <jrichli> lol
21:17:14 <notmyname> heh
21:17:49 <notmyname> so what are things you think we could work on?
21:18:05 <notmyname> or things you wish had gone better
21:18:25 <notmyname> I thought we could have moved the tables sooner and gotten into the room-ful of discussion just a little sooner
21:18:42 <joeljwright> that ^
21:19:04 <bkeller`> we had the fishbowl saying the first thing to do was always to split the room in half, but then we didn't do it ourselves :p
21:19:10 <notmyname> heh
21:19:13 <clayg> notmyname: we did ok, mid morning on the first day
21:19:36 <notmyname> yeah, I'm not sad about missed opportunity. just that we could have done it maybe 30 min earlier :-)
21:20:01 <clayg> bkeller`: pics are great
21:20:16 <notmyname> and I had higher hopes for some of the cross-project sessions. someone suggested to me that we (swift team) could facilitate a cross-project session at the PTG around something that we've learned and are still learning
21:20:17 <clayg> next time when we say first thing we mean first thing!
21:20:25 <notmyname> review times or prioritization or feature branches or somehting
21:20:35 <tdasilva> i missed not having a summary at the end of the discussion that took place, I feel we should write it down somewhere
21:20:37 <acoles> maybe we could have had 30 mins at end of Friday to recap everything that was covered and the take aways/actions.
21:20:50 <clayg> tdasilva: acoles: did you two plan that?
21:20:51 <acoles> but, people were leaving so would have been incomplete
21:20:58 <acoles> clayg: actually, we did ;)
21:21:06 <clayg> i wish we could have had kota_ on Friday!
21:21:30 <tdasilva> lol
21:21:49 <notmyname> ok, so here's the basic question: what do you want to do differently in atlanta at the PTG? so far I'm hearing "move tables sooner" and "make sure kota_ is there the whole time"
21:21:51 <kota_> clayg: that's what I was thinking I had mistaken :/
21:22:22 <kota_> notmyname: thanks man
21:23:34 <notmyname> ok, so for those who weren't there last week, what would you like a brief summary on?
21:23:50 <jrichli> since PTG takes place of hackathon, maybe we should do the dot method for picking topics for the week?  will we have more than 2 days?
21:23:51 * notmyname will also be adding those to his notes
21:24:14 <notmyname> jrichli: I think we'll have 3 days. in that case, dots will work well
21:24:18 <bkeller`> i did like the dot method, although it took half the day just to get our dots in order
21:25:02 <clayg> maybe at the PTG we could square the dots during the corss project meetings or so
21:25:21 <clayg> I think the part that takes a while is "what did we forget to put on the etherpad that people acctually here want to talk about?"
21:25:47 <clayg> putting stickers up is 15-20 mins once you know what all the papers/topics are acctually about
21:26:33 <mattoliverau> So we just need to do a better job of finalising the etherpad before day 1 of swift sessions
21:27:03 <clayg> but i'm sure the papers and lists do make it easier for notmyname to track down summary's and statuses of the weeks discussion
21:27:43 <notmyname> mattoliverau: I think we're pretty good with that for summits. for hackathons, we're generally more in-the-moment for scheduling. different tactics for different events
21:27:45 <clayg> mattoliverau: i think that would mostly solve it - I think when you have a whole week and people are still jetlagged it's no bigs to spend the first morning on a more social activity - but we don't want to blow the first half of 3 days at the PTG
21:27:46 <notmyname> so no summaries requested from anyone?
21:28:09 <clayg> so... on Monday/Tuesday - we're 100% cross project - except for the little bit of planning
21:28:11 <acoles> we could do dot-like voting on etherpad before summit, but I like having physical reminders of topics and progress, like paper or post-its, when we are in a physical meeting
21:28:23 <mattoliverau> clayg: true, but I'm always jetlagged.. even if it was in Oz :P
21:28:30 <clayg> lol
21:28:37 <notmyname> TBH I'm not too worried about scheduling topics for the PTG. we did ok in Barca, and we'll do fine with the PTG. and we'll figure it out in the months between now and then
21:28:48 <pdardeau> any updates from using encryption and EC out in the wild?
21:29:03 <clayg> acoles: i agree, the physicallity is to good to pass up - I don't think it needs to happen before we have actual boots on the ground at the event - anything before that is premature
21:29:03 <notmyname> IOW, we don't have to keep talking about the right way to to do dots on papers 4 months from now ;-)
21:29:24 <clayg> pdardeau: encryption everything is awesome and fast - EC everything is broken and slow
21:29:31 <notmyname> pdardeau: timburke shares some results in his talk with acoles. basically, if you use AES-NI, there's zero impact
21:29:38 <notmyname> but yeah. what clayg said :-)
21:30:01 <notmyname> much of the "EC is slow" was punted to "things will be better after we do golang repcon"
21:30:03 <notmyname> :-/
21:30:07 <jrichli> OVH gave a talk about migrating and using EC in their 25PB cluster
21:30:31 <kota_> jrichli: +1, that was great.
21:30:49 <clayg> and aside from "we had to stop migrating to EC because rebalance was slow/inefficient" it was basically "EC is great!"
21:30:58 <jrichli> :-)
21:32:11 <pdardeau> any big a-ha! moments?
21:32:27 <jrichli> wrt to saying encryption is awesome and fast, I am not sure that we have heard from anyone who is using it in prod yet
21:32:46 <tdasilva> cschwede had one but he said he would share once he figured out if it was working or not
21:32:54 <acoles> OVH raised the "isolated re-incarnated fragment" problem - is there a launchpad bug for that? I would have liked to have found more time to discuss a solution for that.
21:33:12 <clayg> acoles: oh shit your right
21:33:12 <notmyname> jrichli: we've got people using it in prod. at least one now, and a few evaluating or about to use it
21:33:22 <jrichli> oh, ok - great
21:34:13 <mattoliverau> jrichli: looks a little scared now :P
21:34:26 <notmyname> FWIW, etherpads are at
21:34:28 <jrichli> lol.  no ...
21:34:28 <tdasilva> how was the discussion about barbican?
21:34:28 <notmyname> #link https://wiki.openstack.org/wiki/Design_Summit/Ocata/Etherpads#Swift
21:34:30 <acoles> git says to blame jrichli for encryption :P
21:34:50 <acoles> tdasilva: it was good!
21:34:55 <jrichli> git sometimes lies ;-)
21:34:58 <mattoliverau> lol
21:35:40 <acoles> tdasilva: we had a good discussion of how we might achieve key rotation one day, and identified an incremental roadmap for exploring that
21:36:00 <acoles> ugh, is a roadmap ever incremental? sorry for that!
21:36:38 <acoles> tdasilva: initial goal being to rotate per-account keys, that probably being non-trivial to start with
21:37:25 <tdasilva> so is mathias already looking into storing multiple keys in barbican? or are the per-account keys derived from the root key
21:37:25 <tdasilva> ?
21:37:48 <mathiasb> tdasilva: just a single root key in barbican for the moment
21:38:08 <clayg> acoles: per-account keys would be useful for instant secure delete too - even w/o rotation
21:38:08 <jrichli> I wasn't able to be in that discussion.  Are plans to provide a summary?
21:38:13 <tdasilva> mathiasb: hi!
21:38:47 <notmyname> yes I too would ask that we each go through the etherpads and add notes to the various topics
21:38:56 <notmyname> it will help us all stay together
21:38:57 <mathiasb> jrichli: I put a summary of the discussion here https://wiki.openstack.org/wiki/Swift/ideas/keymaster_v2
21:39:09 <jrichli> great, thx!
21:39:28 <acoles> jrichli: clayg yes writing some notes down might be more accurate than me trying to replay the discussion here, I think I might confuse matters
21:39:28 <mathiasb> I can add them to the etherpad (as well) to make them easier to find
21:39:28 <clayg> notmyname: i'll take it as a TODO to update the hummingbird section of whichever pad
21:39:39 <acoles> mathiasb: thanks!
21:39:52 <notmyname> clayg: thanks! https://etherpad.openstack.org/p/BCN-swift-working-session-5
21:39:59 <notmyname> mathiasb: thanks
21:40:18 <mattoliverau> Only downside to splitting the room was there were alot of awesome discussions I wanted to sit in on, but had to make chocies.
21:40:24 <jrichli> +1
21:40:31 <notmyname> mattoliverau: always
21:40:33 <notmyname> ok, anything else to discuss about the summit?
21:40:36 <mattoliverau> I'll take a look at the etherpads of the discussions I was involved in and see if theres anything I can add to them
21:40:57 <bkeller`> should we consider video recording some of the larger discussions? (like golang had the whole room)
21:41:39 <bkeller`> or i guess it could make people worry about asking potentially silly questions/suggestions
21:41:40 <notmyname> bkeller`: I feel like we should work on simply taking notes first
21:42:06 <joeljwright> bkeller`: I'm more worried about the time taken to watch than reliving silly questions
21:42:16 <clayg> bkeller`: yeah I think a summary based on notes concensus is almost always going to be more useful to me than a video for this kind of thing
21:42:54 <clayg> joeljwright: did you know most youtube videos can be watched at double playback speed!?
21:43:09 <joeljwright> clayg: I did, and it's really funny
21:43:10 <acoles> bkeller`: I'd be concerned that people might feel inhibited
21:43:22 <notmyname> ok, no videos
21:43:48 <bkeller`> yep, that's fine
21:43:55 <notmyname> let's move on to the next topic
21:44:01 <notmyname> last call for summit discussions?
21:44:27 <notmyname> ok
21:44:29 <clayg> bkeller`: on the bright side the *idea* of video recordings of design sessions will be preserved forever in irc logs
21:44:42 <notmyname> #topic patches/bugs before the next release
21:45:05 <notmyname> before the summit, we were trackign a few bugs to land before cutting a release. and wanting to cut a release to get the fixes available
21:45:12 <notmyname> https://bugs.launchpad.net/swift/+bug/1633647
21:45:12 <openstack> Launchpad bug 1633647 in OpenStack Object Storage (swift) "bad fragment data not detected in audit" [Critical,Fix released]
21:45:15 <notmyname> has laneded
21:45:25 <notmyname> but https://bugs.launchpad.net/swift/+bug/1624088 has not
21:45:25 <openstack> Launchpad bug 1624088 in OpenStack Object Storage (swift) "EC missing durable can prevent reconstruction" [Critical,Confirmed]
21:45:25 <clayg> because it was *awesome*
21:45:27 <mattoliverau> nice
21:45:39 <notmyname> patch is https://review.openstack.org/#/c/376630/
21:45:40 <clayg> i forget why that one isn't merged - it's probably awesome too
21:45:51 <notmyname> has a +2 from tdasilva (thanks!). needs another
21:45:57 <clayg> nm, i tae it back Co-Authored-By: Clay Gerrard
21:46:07 <mattoliverau> lol
21:46:11 <acoles> nothing can be perfect
21:46:16 <acoles> ;-)
21:46:26 <notmyname> clayg: he did that ambiguous thing again!
21:46:42 <tdasilva> heh
21:47:24 <clayg> the diff on code is *tiny* - just include some inscrutable json in a header and vioala!  200+ lines of test diff prove it's better
21:47:38 <notmyname> yup :-)
21:47:42 <acoles> thanks kota_ clayg tdasilva for merging 376630, I am part way through preparing the backports
21:47:42 <notmyname> this is the only critical bug that's being tracked right now, so IIRC, when this one lands we can cut a release
21:47:45 <timburke> what could go wrong?
21:48:03 <timburke> notmyname: so what's the timeframe on a client release?
21:48:06 <acoles> or was it timburke I should thank? thanks everyone!
21:48:10 <clayg> sounds good - I'm not sure there's any other critical bugs that we have in fligth to fix?
21:48:11 <notmyname> you read my mind :-)
21:48:36 <notmyname> I will be looking into a client release asap too. the keystone session support that landed is great, and we shoudl release that soon
21:48:45 <mattoliverau> +1
21:49:00 <timburke> that's why i ask :-)
21:49:03 <notmyname> #topic open discussion
21:49:07 <joeljwright> timburke: do we want to try landing the keystone session supoprt for v1 auth too?
21:49:08 <notmyname> anything else to bring up this week?
21:49:18 <notmyname> joeljwright: good question
21:49:19 <timburke> joeljwright: i'd love to
21:49:30 <timburke> but i also don't view it as a blocker
21:49:37 <joeljwright> good point
21:49:45 <tdasilva> can we talk about a specific topic that was discussed during the summit, but still needs a resolution?
21:49:49 <joeljwright> I will still get myself reviewing asap though
21:49:58 <notmyname> tdasilva: yes. what's up?
21:49:59 <mattoliverau> timburke: I was looking at your v1 patch on the plane, I'll try and finish up the review for it.
21:50:15 <timburke> i just knew it'd be a necessary thing before we could ever hope to get rid of all the auth options in shell.py and only ever instantiate Sessions ourselves
21:50:25 <timburke> thanks mattoliverau!
21:50:29 <tdasilva> a few of us (can't remember exactly who, acoles, timburke, jrichli) were talking about how we should go about landing symlinks
21:50:36 <notmyname> ok
21:51:03 <tdasilva> it is ready for review, but it would nice if there were a real client to use it, like versioning or tiering to test out the kinks
21:51:13 <tdasilva> before being made available to users
21:51:31 <tdasilva> so we talked about maybe landing on master and labeling it experimental
21:51:36 <notmyname> tdasilva: as opposed to "just" functional and probe tests?
21:52:40 <tdasilva> no, it has functional tests. It just that it is more of a foundational feature and we think it would be nice if "clients" put it to use
21:52:52 <tdasilva> so that we could find issues with it
21:53:17 <tdasilva> by landing it on master, it would make it easier for features like tiering and versioning to use it
21:53:18 <notmyname> I'm not disagreeing. but what are you hoping to find that won't be found in functests? is it api design?
21:53:33 <tdasilva> but at the same time it opens up for backwards compatibility issues in the future
21:53:40 <acoles> we discussed the backwards compatibility "cost" of discovering later that a feature isn't quite what a yet-to-be-finalised application needed, versus the "cost" of not merging a feature and having it languish as a patch
21:54:04 <notmyname> yeah, makes sense
21:54:11 <notmyname> what consensus did you come to last week?
21:54:56 <tdasilva> we talked about the pros and cons of landing on master as experimental
21:55:04 <acoles> while it is not merged we keep talking (costing time and context swicthing) about it and applications have to base patches on a patch
21:55:21 <tdasilva> not sure we came to a consensus, but personally i'm ok with it
21:55:40 <tdasilva> because of what acoles jsut sais
21:56:03 <notmyname> what are the migration costs? is any data in it versioned? eg "this is a v1 symlink"?
21:56:07 <clayg> tdasilva: lets' just maybe try to get a bunch of eyes on it - like really push folks to "hey!  important foundational thing coming down!  do you bit and try and make it good!"
21:56:08 <acoles> I was leaning towards landing with an experimental label
21:56:34 <acoles> clayg: yes, the more eyes now the better
21:57:05 <mattoliverau> so if it lands as exeperimaental then we need to make sure we note that the API may change in future is documentation then.
21:57:07 <kota_> one thing
21:57:13 <notmyname> kota_: yes?
21:57:33 <clayg> acoles: tdasilva: what's the review/change id?
21:57:38 <acoles> and if it is not already well understood, make sure that people realise this is meant to be the enabling feature for tiering etc (I think, right?)
21:57:41 <kota_> in this week, i found something on isa-l
21:57:50 <kota_> now in discussion on fyi, https://github.com/01org/isa-l/issues/10
21:57:54 <tdasilva> https://review.openstack.org/#/c/232162/
21:57:56 <notmyname> kota_: right. I saw the github issue. thanks
21:58:08 <notmyname> (on the symlinks topic) ok, for now, let's do that. get more +2s on it (like we did with other big stuff)
21:58:24 <kota_> notmyname: oh, you has already had, great.
21:58:50 <notmyname> ah jrichli has a -1 on it
21:59:00 <kota_> sorry breaking the conversation.
21:59:05 <clayg> kota_: so that latest diff to switch the vptr to the cauchy function?  That works even with fragments encoded using the old function?
21:59:15 <notmyname> kota_: no worries
21:59:17 <jrichli> yes, i need to look into those failures
21:59:17 <clayg> the matrix manipulation only comes into play when you're slicing around the errors right?
21:59:38 <clayg> kota_: all the bytes are disk a full filled in matrix?  the inversion issues don't come into play?
21:59:41 <kota_> clayg: you are talking in public :/
22:00:01 <notmyname> jrichli: tdasilva: ok, you two keep looking. when you say it's good (you're ready to +2), let the rest of us know to do more reviews. then when we have many, we'll land it
22:00:06 <clayg> ... i thought that's why we were brining it up?
22:00:11 <clayg> sorry
22:00:31 <notmyname> we're out of time in this room
22:00:38 <notmyname> thanks, everyone, for working on swift
22:00:58 <notmyname> don't forget that meetings are at UTC2100, regardless of your timezone changes
22:01:02 <notmyname> #endmeeting