21:00:09 <notmyname> #startmeeting swift
21:00:10 <openstack> Meeting started Wed Oct 19 21:00:09 2016 UTC and is due to finish in 60 minutes.  The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:11 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:13 <openstack> The meeting name has been set to 'swift'
21:00:19 <notmyname> who's here for the swift team meeting?
21:00:21 <ntata> Hello
21:00:22 <jrichli> o/
21:00:24 <bkeller`> o/
21:00:26 <joeljwright> hello
21:00:29 <pdardeau> hi
21:00:29 <sgundur> hi
21:00:31 <mmotiani_> o/
21:00:32 <kota_> hey yo
21:00:39 <hurricanerix> o/
21:00:57 <tdasilva> hi
21:01:01 <acoles> here
21:01:05 <kmARC> o-
21:01:07 <nadeem> hi
21:01:07 <kmARC> o/
21:01:15 <clayg> ＼(＾O＾)／
21:01:27 <joeljwright> clayg explodes into the meeting!
21:01:33 <jrichli> lol
21:01:35 <notmyname> :-)
21:01:40 <mattoliverau> o/
21:01:45 <notmyname> welcome everyone
21:01:50 <cutforth> o/
21:02:00 <notmyname> just a few things to go over this week
21:02:01 <notmyname> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:02:16 <notmyname> #topic barcelona summit next week
21:02:27 <notmyname> summit is next week
21:02:37 <mattoliverau> \o/
21:02:49 <notmyname> I've taken the items from the etherpad and grouped them
21:03:16 <notmyname> I'm not totally happy with it yet, but I've pushed up a schedule. still is likely to be adjusted
21:03:25 <clayg> wow, next week - i just can't seem to wrap my head around it - it feels *so* soon
21:03:27 <notmyname> basically, here's the etherpads for the week:
21:03:29 <notmyname> https://etherpad.openstack.org/p/BCN-ops-swift
21:03:29 <notmyname> https://etherpad.openstack.org/p/BCN-swift-community-feedback
21:03:29 <notmyname> https://etherpad.openstack.org/p/BCN-swift-working-session-1
21:03:31 <notmyname> https://etherpad.openstack.org/p/BCN-swift-working-session-2
21:03:33 <notmyname> https://etherpad.openstack.org/p/BCN-swift-working-session-3
21:03:34 <clayg> i bet when the ptg shows up I'll *really* be freaked out?
21:03:35 <notmyname> https://etherpad.openstack.org/p/BCN-swift-working-session-4
21:03:37 <notmyname> https://etherpad.openstack.org/p/BCN-swift-working-session-5
21:03:56 <notmyname> clayg: especially since that's only 4 months away, not 6
21:04:01 <clayg> notmyname: that doesn't seem nessecary
21:04:06 <mattoliverau> yeah short cycle this one
21:04:22 <notmyname> first thing to note is the ops session is part of the ops track
21:04:52 <clayg> OMG look at all this stuff?
21:05:13 <joeljwright> notmyname: I think I missed the boat on your organisation (only added a new topic to the etherpad a couple of hours ago)
21:05:23 <notmyname> we'll use the 40 min in the ops track on tuesday, and then have a "part 2" as one of our fishbowl sessions on thursday morning
21:05:36 <notmyname> joeljwright: ok, pm it to me, and I'll make sure to address it
21:05:48 <joeljwright> cool, thanks
21:06:13 <notmyname> so please try to be at the tuesday ops session
21:06:33 <notmyname> our fishbowl sessions are "ops feedback (part 2)" and "community feedback"
21:06:53 <notmyname> the community feedback one being our own retrospective for how we as a community are doing and what we need to adjust, if anything
21:07:19 <notmyname> the working sessions are arranged in blocks of time, like the last several summits
21:07:23 <mattoliverau> great I like that, an overflow for more ops and then just community discussion
21:07:48 <notmyname> on thursday, we start with the 2 fishbowl sessions and then have 3 blocks of time (separated by breaks or lunch)
21:08:04 <notmyname> it's 2+3+1 blocks of time (40 minute blocks)
21:08:18 <mattoliverau> I'll be the really jetlagged guy at the ops session :P I oly arrive Monday after over 30 hours in transit
21:08:42 <notmyname> then on friday, we have scheduled time (2 chunks of 2 40 min blocks each), then lunch, then the free-form community meeting from 2-6pm
21:09:01 <notmyname> so basically, ops stuff on tuesday, swift-specific stuff on thursday and friday
21:09:16 <clayg> w/e werx - thanks notmyname
21:09:35 <mattoliverau> cools
21:09:59 <notmyname> like I said, I think I want to tweak the agenda for each block some from what I've written down (some important topics will be tricky since some people won't be there)
21:10:05 <notmyname> but in general, this is our starting point
21:10:29 <mattoliverau> And we can obviously go add stuff (prep) to the etherpads
21:10:40 <notmyname> also, if you saw on the ML an hour so so ago, the foundation has pushed out draft versions of their logo (mascot) for each project
21:10:50 <notmyname> I've got the draft version for swift, and I'll be sharing that next week
21:11:03 <notmyname> so we can give feedback. they want to have it final by the PTG in february
21:11:14 <kota_> cool
21:11:29 <notmyname> ok, I think that's all I have on the summit agenda. any questions or other comments about next week
21:12:04 <jrichli> in the schedule, the link is to the #2 etherpad even for some that are later
21:12:48 <notmyname> there are 3 scheduled 40 minute sessions that should link to the #2 etherpad
21:13:09 <jrichli> oh, ok.
21:13:20 <kota_> notmyname: added a few comments to https://etherpad.openstack.org/p/BCN-swift-working-session-5
21:13:21 <notmyname> but I think I added the right time blocks in the top of each etherpad
21:13:26 <notmyname> kota_: thanks
21:13:30 <clayg> notmyname: one benifit of not kicking ou the PTL every few months - they get sorta good at this shit
21:13:33 <jrichli> ok, i missed that.  will look again
21:13:45 <kota_> some of my topics there but can't attend at Fri.
21:13:49 <notmyname> oh, also on the #5 etherpad I added a section at the bottom for the "whole afternoon discussion stuff"
21:13:52 <kota_> sorry
21:13:56 <acoles> can we just meet notmyname for breakfast and follow him around each day?
21:13:59 <notmyname> since there's not a place to link to it on the schedule
21:14:14 <notmyname> kota_: good to know. yeah, I didn't work with individual schedules yet
21:14:24 <notmyname> acoles: sounds good to me. :-)
21:15:13 <notmyname> also, in case you haven't been to barcelona before, the culture there is very well adjusted to typical dev schedules ;-)
21:15:36 <notmyname> breakfast is normally about 9-10am and lunch is 3-4 ish and dinner starts at 9 or 10pm
21:15:57 <notmyname> unfortunately, the conference schedule doesn't really reflect that :-)
21:16:08 <mattoliverau> Yeah, I got hungry waiting for dinner when I was there.. thank god for tapas
21:16:18 <notmyname> but don't expect to wake up at 7 or 8 and easily find an open coffee shop to grab a pastry or anything
21:16:26 <kota_> how about siesta? :-)
21:16:33 <notmyname> kota_: of course ;-)
21:16:44 <acoles> notmyname: mathiasb said on the etherpad that he's not there on friday but that's when the keymaster work is scheduled
21:16:59 <acoles> afaik mathiasb is working on the keymaster
21:17:15 <clayg> why is no one hanging around on Friday?
21:17:18 <notmyname> acoles: yeah, that's one thing I need to adjust for
21:17:21 <notmyname> clayg: good question!
21:17:28 <kmARC> Altho I haven't really worked on it yet, but I work together with mathiasb and I will be there on friday
21:17:31 <notmyname> acoles: gotta do the same for kota
21:17:33 <clayg> notmyname: i think we should try to make room for folks while there there
21:17:50 <mattoliverau> kmARC: you have a week to get up to speed :P
21:17:50 <acoles> kmARC: great!
21:17:51 <clayg> i'm somewhat distressed about doing a big block of hummingbird when some of our rax players are missing
21:18:01 <notmyname> clayg: yeah. eg might swap keymaster for golang stuff
21:18:08 <notmyname> right. exactly what I was thinking
21:18:30 <clayg> kk
21:18:46 <clayg> so it's good that we discussed this - notmyname will square it
21:18:46 <notmyname> anything else on the summit next week?
21:18:56 <mattoliverau> we could try and at least trick nadeem into conference calling ;)
21:19:12 <nadeem> I am open to attend the conference call
21:19:17 <clayg> woooo!
21:19:30 <acoles> at 9am CET?? ;)
21:19:30 <notmyname> maybe. I'm not a big fan of that, but maybe
21:19:41 <notmyname> acoles: yeah, that's one reason
21:20:30 <notmyname> ok, moving on to the next topic
21:20:40 <notmyname> #topic big bugs and open patches
21:20:50 <notmyname> thanks for landing the EC fix!
21:20:52 <notmyname> yay
21:21:01 <clayg> acoles: is the bomb
21:21:06 <kota_> acoles is great
21:21:06 <clayg> he has other patches up too
21:21:19 <notmyname> we've got https://bugs.launchpad.net/swift/+bug/1624088 and https://review.openstack.org/#/c/376630/ to address it
21:21:20 <openstack> Launchpad bug 1624088 in OpenStack Object Storage (swift) "EC missing durable can prevent reconstruction" [High,Confirmed]
21:21:21 <clayg> tdasilva: has some good fixes in play too - and has been doing a bunch of great reviews - tdasilva is the bomb
21:21:22 <kota_> clayg:^^
21:21:25 <acoles> like a rash
21:21:37 <clayg> kota_'s huge impact as well
21:21:40 <clayg> so many kudos!
21:21:55 <notmyname> everyone is awesome :-)
21:22:15 <notmyname> so we've landed the EC corruption fix and the backports for it
21:22:33 <notmyname> and I think it would be good to tag a release with this
21:22:36 <clayg> it was really an ssync protocol bug
21:22:55 <notmyname> but tdasilva had mentioned that https://bugs.launchpad.net/swift/+bug/1624088 should be fixed first
21:22:56 <openstack> Launchpad bug 1624088 in OpenStack Object Storage (swift) "EC missing durable can prevent reconstruction" [High,Confirmed]
21:22:59 <acoles> notmyname: I think we need this first, we're only half done on that bug.. https://review.openstack.org/#/c/387655
21:23:01 <clayg> but far to easy to trigger with ec reconstruction - i feel so bad :'(
21:23:29 <notmyname> acoles: ack. that's exactly why I was bringing it up :-)
21:23:47 <notmyname> so yeah, to summarize: what's open (bugs or patches) that needs to land before we tag it for release?
21:23:48 <clayg> acoles: the corruption fix before the optomistic reconstructor fix
21:23:59 <acoles> notmyname: it's different from the one you mentioned
21:24:04 <notmyname> yes
21:24:14 <notmyname> so finish https://review.openstack.org/#/c/387655 first
21:24:14 <clayg> i guess we should go ahead and make the auditor fix criticial - idk
21:24:25 <notmyname> clayg: what's the bug link?
21:24:26 <acoles> clayg: +1 IMHO
21:24:32 <acoles> https://bugs.launchpad.net/swift/+bug/1633647
21:24:33 <openstack> Launchpad bug 1633647 in OpenStack Object Storage (swift) "bad fragment data not detected in audit" [High,Confirmed]
21:24:33 <clayg> are we really going to backport the optimistic reconstructor fix?
21:24:53 <acoles> notmyname: ^^
21:25:29 <notmyname> clayg: do you mean https://bugs.launchpad.net/swift/+bug/1624088 ?
21:25:31 <openstack> Launchpad bug 1624088 in OpenStack Object Storage (swift) "EC missing durable can prevent reconstruction" [High,Confirmed]
21:25:53 <clayg> yeah
21:26:04 <clayg> it's weird all this stuff popped up at the same time
21:26:21 <clayg> i'd like to see newton have some of this fixed
21:26:29 <clayg> it's going to be around for awhile
21:26:36 <notmyname> I'm ok with backporting it
21:26:53 <notmyname> but I agree that the corruption bug takes priority
21:26:57 <notmyname> ie https://bugs.launchpad.net/swift/+bug/1633647
21:26:58 <openstack> Launchpad bug 1633647 in OpenStack Object Storage (swift) "bad fragment data not detected in audit" [Critical,Confirmed]
21:27:13 <clayg> I personally haven't cut the SwiftStack Newton release *because* of these issues - we're waiting until after the summit before we start to vet the 2.10+ release
21:27:42 <clayg> but i'm not sure i'm going to go back and repackage our 2.9 - folks that are running EC will just upgrade
21:28:26 <clayg> but when I saw the user survey that it's so common to use os packages - makes me think more about the official openstack releases
21:28:40 <clayg> *idk* - dunno?  dunno... backport everything.  idk
21:28:47 <acoles> Ideally I'd like to backport the auditor fix for bad frags to mitaka like we did the first patch
21:29:09 <acoles> I do fear that backport of patch https://review.openstack.org/#/c/387655 is not going to be trivial tho
21:29:25 <clayg> acoles: what about optimistic GETS?  probably not right?  but we could backport the reconstructor fix to Newton?
21:29:46 <clayg> acoles: *right* - test fallout it nuts here!
21:29:47 <kota_> acoles: yeah, not so trivial..
21:30:11 <notmyname> we should backport what we can, but one of the advantages of having good upgrade paths is that we don't have to backport everything
21:30:18 <acoles> clayg: I'm more concerned with th auditor for newton because we know we have at least one user who needs it and wasn;t planning on upgrade from mitaka
21:30:25 <kota_> my frastration is xenial still provides liberasurecode 1.1.x ... :/
21:30:28 <acoles> s/newton/mitaka/
21:30:36 <notmyname> kota_: bah!
21:30:44 <acoles> argh auditor -> mitaka
21:30:58 <acoles> optimistic reconstructor -> newton is ok with me
21:31:06 <clayg> acoles: ok, so well try to backport auditor as far as we can
21:31:09 <clayg> i have to bounce early today
21:31:29 <clayg> i'll async stuff later this afternoon.  i'm on tomorrow
21:31:48 <acoles> clayg: k, I should have a little more time for it tomorrow
21:32:06 <acoles> clayg: push anything you might get done
21:32:10 <acoles> kota_: ^^
21:32:11 <notmyname> to to back up just a bit and to summarize...
21:32:14 <kota_> i can take a look for the new version today.
21:32:29 <kota_> and also able to work tommorow
21:32:32 <notmyname> https://bugs.launchpad.net/swift/+bug/1633647 is important and in progress. when it lands on master, we'll see what's possible to backport
21:32:33 <openstack> Launchpad bug 1633647 in OpenStack Object Storage (swift) "bad fragment data not detected in audit" [Critical,Confirmed]
21:32:59 <notmyname> https://bugs.launchpad.net/swift/+bug/1624088 is of seconday importance (but already has a patch with one +2) and should also land before cutting a release
21:33:00 <openstack> Launchpad bug 1624088 in OpenStack Object Storage (swift) "EC missing durable can prevent reconstruction" [High,Confirmed]
21:33:07 <notmyname> similarly, backport as possible
21:33:23 <notmyname> then after those land, tag a release on master
21:33:27 <acoles> notmyname: good summary, agree
21:33:29 <mattoliverau> +1
21:33:35 <kota_> k
21:33:41 <notmyname> ok, good :-)
21:33:53 <notmyname> last question I have about backports is patch https://review.openstack.org/#/c/385779/
21:33:57 <notmyname> from cschwede
21:34:08 <notmyname> I'm not sure why this one needs to be backported. did I forget why?
21:34:46 <tdasilva> notmyname: it's important for our downstream newton release
21:34:53 <acoles> notmyname: was this where someone in channel reported seeing huge IO in a fresh empty cluster?
21:34:54 <notmyname> tdasilva: ah ok. got it
21:35:00 <notmyname> acoles: yeah, that's it
21:35:10 <notmyname> specifically, it seems, red hat
21:35:29 <tdasilva> yeah, has to do with the tripleo default installation
21:35:40 <clayg> *huge* IO?  really?
21:35:43 <tdasilva> running *everything* under the sun on the controller nodes
21:35:48 <clayg> where they *saveing* that IO for something else?
21:35:52 <notmyname> ok, I'll recheck that one because of the gate failures. then land it later
21:35:56 <clayg> oic
21:36:03 <clayg> so they *were* saving that io for something else
21:36:18 <acoles> clayg: ok s/huge/non-zero/ ;)
21:36:30 <notmyname> #topic open discussion
21:36:32 <clayg> *yawn*
21:36:36 <notmyname> anything else to bring up this week?
21:36:45 <notmyname> obviously, there will be no meeting next week
21:36:59 <notmyname> or rather, lots of meetings, just not in IRC
21:37:10 <mattoliverau> :)
21:38:01 <acoles> mattoliverau: must be about time for you to start your journey ;)
21:38:06 * notmyname is calling it
21:38:11 <notmyname> thanks everyone for working on swift
21:38:19 <mattoliverau> acoles: lol, not quite but soon :)
21:38:20 <notmyname> looking forward to seeing (most of) you next week
21:38:25 <notmyname> #endmeeting