21:00:18 <notmyname> #startmeeting swift
21:00:19 <openstack> Meeting started Wed Nov  9 21:00:18 2016 UTC and is due to finish in 60 minutes.  The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:20 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:22 <openstack> The meeting name has been set to 'swift'
21:00:23 <notmyname> who's here for the swift team meeting?
21:00:28 <mmotiani> hi
21:00:39 <timburke> hi
21:00:45 <pdardeau> hey
21:00:47 <tdasilva> hello
21:00:53 <joeljwright> evenin
21:01:01 <mathiasb> hi
21:01:01 <jrichli> hello
21:01:11 <cutforth> o/
21:01:12 <ntata> hello
21:01:13 <torgomatic> .
21:01:18 <kota_> hello
21:01:18 <bkeller`> o/
21:01:23 <nadeem> .
21:01:40 <notmyname> welcome everyone
21:01:48 <clayg> greetings programs!
21:02:20 <notmyname> not a ton on the agenda for today, so plenty of time for open discussion (or ending early?)
21:02:23 <notmyname> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:02:27 <notmyname> clayg: hello, fellow human
21:03:04 <notmyname> first up, reminder that time zones are a thing, and we ignore them for meeting scheduling
21:03:37 <clayg> i like this time better
21:03:40 <notmyname> so if you're here now (and you're in the US) then you did the right thing and got to the meeting at the right time
21:03:47 <notmyname> meeting is at 2100UTC
21:03:56 <notmyname> #topic PTG signups
21:04:01 <notmyname> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:04:03 <notmyname> nope
21:04:12 <notmyname> #undo
21:04:13 <openstack> Removing item from minutes: <ircmeeting.items.Link object at 0x7fac5556de10>
21:04:17 <notmyname> #link https://www.openstack.org/ptg
21:04:19 <notmyname> there
21:04:31 <notmyname> the PTG signups are open
21:04:51 <notmyname> I've asked to have 3 full days days: wed, thurs, fri
21:04:52 <timburke> still waiting on https://review.openstack.org/#/c/272727/ ...
21:05:22 <notmyname> monday and tuesday of that week will be for some cross project stuff (also note that monday is presidents' day in the US)
21:06:01 <notmyname> tickets for the PTG are $100 and there is a hotel block. there's a chance that things could fill up, so book early
21:06:16 <notmyname> any question about the PTG?
21:06:21 <torgomatic> Remember when we had our own hackathons and we didn't have to ask for time from some organizing board?
21:06:25 <clayg> notmyname: I this dream that we all went to the PTG - but we sort of just hole'd up together like it was a regular hackathon Mon-Fri and everyone thought it was weird that we didn't go argue with people on Mon-Tue
21:07:10 <notmyname> yes, yes, I know. everything we used to do was awesome and everything in the future is terrible. but we're doing the PTG, and it will be fine
21:07:11 <notmyname> ;-)
21:07:15 <tdasilva> notmyname: do you know if the tickets are limited?
21:07:21 <torgomatic> that's a good idea; we can wander off during the cross-project times and be cross by ourselves
21:07:22 <notmyname> tdasilva: I do not know
21:07:27 <tdasilva> ok
21:07:35 <torgomatic> (not to avoid other projects, just to get more time)
21:07:36 <ntata> notmyname, do we get some sort of ATC pass to attend PTG?
21:07:44 <notmyname> ntata: no
21:08:06 <clayg> ntata: nope, one hundred bones across the board
21:08:23 <notmyname> but attending the PTG will get you a ticket to the forum in boston in may
21:08:25 <mathiasb> tdasilva: I heard that they are, but I don't have any official to point to
21:08:58 <clayg> ntata: it's like this http://www.gifwave.com/media/37332_movie-awesome-coffee-lego-the-lego-movie.gif
21:09:15 <tdasilva> mathiasb: mmm...interesting. getting refund for hotel reservation is easy, not sure about eventbrite tickets...
21:09:31 <ntata> lol
21:09:42 <pdardeau> will patchbot be there?
21:09:55 <mathiasb> tdasilva: IIRC you can get a full refund up to like february 17th or so
21:09:59 <clayg> pdardeau: that can be our mon-tues project!
21:10:08 <notmyname> clayg: great idea! :-)
21:10:23 <clayg> since notmyname doesn't seem to be invested in fixing him - and it would everyone (who uses patchbot; which is probalby just us?)
21:10:38 <clayg> *would benifit
21:10:47 <notmyname> there's a few others. I'm interested. just get pulled into other things
21:10:53 <clayg> :P
21:10:58 <clayg> i'm just poking fun - sorry
21:11:42 <mathiasb> tdasilva: full refund possible until february 15th https://www.eventbrite.com/e/project-teams-gathering-tickets-27549298694
21:12:12 <notmyname> despite the frustrations of the less time and more money, I do think the PTG will be good for us. don't think of it as replacing midcycles, though. think of it as the dev part of the summits. that's actually what it is
21:12:16 <tdasilva> yep, had just read that there...guess i should read the full description before asking...
21:12:53 <nadeem> will we have a separate midcycle too?
21:12:55 <notmyname> and after the PTG, based on what's avaialble for us as a team in boston, we can figure out if we need to do self-organized midcycles again
21:13:55 <notmyname> anything else about the PTG that I can answer or go find an answer for?
21:14:18 <tdasilva> notmyname: thanks for the info!
21:14:36 <clayg> nadeem: unexpected; expectation is that ptg replaces mid-cycle
21:14:52 <clayg> still 4x travel per year all offical os sponsered
21:15:31 <nadeem> sounds good
21:15:49 <tdasilva> actually...one more PTG question...sorry..if we want to hold cross-project meetings, do we have to say anything? like so it gets planned?
21:15:51 <notmyname> clayg: sortof, but not really. yes, there's still 4x travel and they're now all found-ation related. but the PTG is mroe like the summit instead of a midcycle. we'll see what the new "forum" will be like and if we need to do a midcycle instead of that
21:16:03 <notmyname> tdasilva: that... is a really good question
21:16:16 <tdasilva> i'm thinking about all the golang emails that are going on, that could become a cross-project conversation...
21:16:26 <notmyname> it looks like the mon-tues are pretty set for what projects/teams will have meetings, but I don't know how they'll be scheduled
21:16:48 <notmyname> tdasilva: I'll look in to that
21:16:54 <tdasilva> ok
21:17:21 <clayg> tdasilva: I think mon-tues are more like mid-cycles for horizontal teams - so maybe we go to "infra" if we want to work on golang in CI or "docs" if you want to talk about making godoc output stuff that matches sphinx
21:17:53 <tdasilva> clayg: yeah, that's what i was thinking...was just wondering if we need to announce or ask to have that topic on their agenda??
21:18:29 <clayg> tdasilva: i'm not sure it's clear how adgendas will be set - could very well be a dotacracy - so just how up and say "hey, i want to talk about this!"
21:18:31 <notmyname> I'll start a ML thread
21:18:39 <tdasilva> but yeah, also need to remember about the cross-project vs. inter-project distinction
21:18:53 <notmyname> tdasilva: yeah, exactly
21:18:57 <tdasilva> i'm guessing having a barbican conversation wouldn't count as cross-project
21:19:07 <clayg> tdasilva: maybe we could start a golang-wg and have our *own* little horizontal team that folks can show up to!
21:20:15 <notmyname> moving on
21:20:16 <clayg> i *should* start a dependency-hygiene-wg
21:20:18 <notmyname> #topic release
21:20:25 <tdasilva> lol
21:20:42 <notmyname> https://bugs.launchpad.net/swift/+bug/1633647 and https://bugs.launchpad.net/swift/+bug/1624088 have landed in swift, so I'm working on getting a swift release finished up
21:20:42 <openstack> Launchpad bug 1633647 in OpenStack Object Storage (swift) "bad fragment data not detected in audit" [Critical,Fix released]
21:20:43 <openstack> Launchpad bug 1624088 in OpenStack Object Storage (swift) "EC missing durable can prevent reconstruction" [Critical,Fix released]
21:20:52 <clayg> whoot!
21:20:58 <joeljwright> :)
21:21:06 <notmyname> I've also pushed https://review.openstack.org/#/c/395768/ for a swiftclient release (has the sessions support)
21:21:28 <joeljwright> did the keystone auth v1 plugin make it too?
21:21:31 <notmyname> yep
21:21:36 <joeljwright> :)
21:22:28 <notmyname> I mention this is the commit messages, but FYI I'm reordering the AUTHORS files. turns out, names are terrible, and there's no way to actually do the "right" thing for ordering them. so the new plan is to simply order them by the first character/byte in the author field
21:22:35 <clayg> notmyname: when I glance down the list of *high* priority bugs - this one stands out -> https://bugs.launchpad.net/swift/+bug/1516579
21:22:35 <openstack> Launchpad bug 1516579 in OpenStack Object Storage (swift) "PUTing a 0-byte object and supplying an Etag results in a 503 response" [High,Fix committed] - Assigned to Samuel Merritt (torgomatic)
21:22:39 <clayg> but it's already landed
21:22:49 <clayg> then this one -> https://bugs.launchpad.net/swift/+bug/1554233
21:22:49 <openstack> Launchpad bug 1554233 in OpenStack Object Storage (swift) "Servers-per-port can consume excessive OS threads" [High,In progress] - Assigned to Samuel Merritt (torgomatic)
21:23:19 <clayg> ^ cheap to fix/land and may be hurting adoption of servers-per-port (OTOH, there's a good workaround - so maybe that should only be medium)
21:24:52 <clayg> oh wait, not there was *different* high priority bug with 0 in the title - it was kota's fix for multi ec policy
21:25:16 <notmyname> I feel like we do this every time a release gets close: "but wait! there's one more!" :)
21:25:56 <kota_> clayg: https://bugs.launchpad.net/swift/+bug/1549110 ?
21:25:56 <openstack> Launchpad bug 1549110 in OpenStack Object Storage (swift) "EC: object-reconstuctor got unhandled 0 divison error" [High,Fix released] - Assigned to Kota Tsuyuzaki (tsuyuzaki-kota)
21:26:16 <clayg> i'm not doing well finding bugs today :'(
21:26:20 <clayg> kota_: yes thank you!
21:26:38 <clayg> notmyname: there *really* is tho!?  and the one that came up this morning with the if-none-match
21:26:53 <notmyname> clayg: yeah, you're not wrong :-)
21:27:07 <notmyname> this one? https://bugs.launchpad.net/swift/+bug/1640448
21:27:07 <openstack> Launchpad bug 1640448 in OpenStack Object Storage (swift) "If-None-Match doesn't work on deleted object" [High,Confirmed]
21:27:08 <clayg> notmyname: we can't have "releases" we need to just cut a release every couple of weeks because it really keeps getting better and better
21:27:26 <clayg> i know there's a regression that slips in every *now* and then - but all the more reason to keep going keep going!
21:28:05 <notmyname> so that's why I bring up the topic: what's the stuff we know about that we should land before we cut a release?
21:28:09 <clayg> notmyname: neyway - i don't really care - we're so far behind because of these blocker bugs - we haven't even released a Newton yet :'(
21:28:29 <clayg> notmyname: we'll the
21:29:08 <clayg> kota_: lp bug #1549110 is fix released!
21:29:08 <openstack> Launchpad bug 1549110 in OpenStack Object Storage (swift) "EC: object-reconstuctor got unhandled 0 divison error" [High,Fix released] https://launchpad.net/bugs/1549110 - Assigned to Kota Tsuyuzaki (tsuyuzaki-kota)
21:29:10 <clayg> notmyname: ^ that one
21:29:24 <clayg> i think i'm good with that
21:29:55 <clayg> i don't know of others - but I could be just dropping the ball on something terrible and huge - i need to go back through the open priority bugs
21:30:32 <notmyname> https://bugs.launchpad.net/swift/+bug/1628906 should be confirmed or not on master
21:30:32 <openstack> Launchpad bug 1628906 in OpenStack Object Storage (swift) "swift-object-reconstructor memory leak" [High,Confirmed]
21:32:01 <notmyname> good news https://bugs.launchpad.net/swift/+bug/1491908 can be marked as closed
21:32:01 <openstack> Launchpad bug 1491908 in OpenStack Object Storage (swift) "proxy server reports 202 even though backend didn't accept object" [High,In progress]
21:33:29 <notmyname> ok, here's a summary: there's a few high priority bugs that I want to look at further. I'll be working on the authors/changelog patch for the release, so if anyone sees something that really should land, let me know. we'll hold a release for it
21:33:58 <notmyname> ones that say "memory leak" or have 503s are stuff that sticks out to me
21:34:45 <clayg> notmyname: looking that 202 accepted bug for 409 - i don't see where the change removed the logic in the proxy?
21:35:20 <notmyname> ah? I was going by the commit subject and the comments in LP
21:35:27 <clayg> notmyname: I'll try to confirm but it looks like ctennis' concern is still there -> https://github.com/openstack/swift/blob/a30351bdfc60f827979c5afe76ad40bf3a2b05c3/swift/proxy/controllers/obj.py#L482
21:35:46 <notmyname> ok. thanks. I set it back
21:35:56 <kota_> notmynam: i don't think the memory leak on the master, anyway we need to confirm with the reporter though.
21:36:02 <clayg> but now it's a bit more dubious since container-sync doesnt' rely on that behavior
21:36:51 <clayg> kota_: my opinion of it is we fixed *a* memeory leak - and there are probly more - but we don't have to keep the bug open until we fix all of them - we just need to keep fixing them as we can isolate something
21:37:33 <kota_> clayg: fair enough
21:37:58 <notmyname> anything else to bring up about a pending release?
21:38:32 <notmyname> ok
21:38:36 <notmyname> #topic open discussion
21:38:44 <notmyname> anything else to bring up this week?
21:39:21 <clayg> notmyname: i want to land https://review.openstack.org/#/c/393595/3 and tag a liberasure right away
21:39:40 <clayg> notmyname: kota_: how can we make that happen (EOD tomorrow?)
21:40:11 <notmyname> clayg: as soon as we do that and get the version bumped in swift's requirements, I want to do a release of swift. (even if it's the day after another release)
21:40:38 <notmyname> kota_: after the liberasurecode patch lands, what's necessary to do in pyeclib?
21:40:40 <notmyname> anything?
21:40:53 <kota_> ah
21:40:56 <clayg> notmyname: yes yes all of these things right away - i just don't know how anything works?  tdasilva can you save us???
21:41:13 <tdasilva> i think you will need to also do a pyeclib release, right?
21:41:16 <kota_> notmyname: if we could make a hard coded version requirements in the *pyeclib*
21:41:17 <notmyname> clayg: those aren't the hard parts. just have to paste a SHA into a repo
21:41:41 <kota_> notmyname: it seems better, otherwise pyeclib can allow to use older version.
21:41:42 <tdasilva> and then swift's requirement point to the new pyeclib release
21:41:44 <clayg> tdasilva: IME just building a new liberasure with current pyeclib is good enough for me?
21:41:46 <kota_> of libeasurecode
21:41:52 <kota_> tdasilva: true
21:41:55 <notmyname> ok, so release libearasurecode, bump dep in pyeclib, then release pyeclib, then bump dep in swift, then release swift?
21:42:12 <kota_> notmyname: right
21:42:19 <clayg> tdasilva: like for me I would just update my liberasurecode packages and make drag it forward for all versions of pyeclib - I think swiftstack's pacaking is full on crazy town tho
21:42:39 <notmyname> clayg: yeah, that will work for swiftstack, but we shoudl do the rest to avoid older versions being used
21:42:40 <clayg> but right now I only package tagged liberasurecode releases - i'm totally happy to just "fix that" and move to where I can package any random sha
21:42:55 <clayg> notmyname: totally granting *everyone* knows better than I do
21:43:00 <notmyname> heh
21:43:22 <tdasilva> yeah, i don't think pyeclib actually has a dependency on a specific version of a liberasurecode
21:43:29 <tdasilva> can't find it
21:43:34 <clayg> notmyname: but also fully expressing this needs to happen right away - if I get a tagged liberasurecode release out of the deal I win - if I not - i'm doing something different - and I'll check in when I'mdone
21:43:37 <tdasilva> so clayg's method might be right
21:43:57 <notmyname> tdasilva: IIRC either you or me can do the ec library releases. do you want to do that as soon as the patch lands, or would you like me to?
21:44:05 <tdasilva> the important part is getting downstream packagers to package the new liberasure code asap
21:44:15 <notmyname> clayg: I don't know why we can't tag liberasurecode as soon as the patch lands
21:44:26 <clayg> i am a downstream packager!  i will do it the same day you make the tag - promise!
21:44:39 <tdasilva> clayg: hehehe, true
21:44:52 <tdasilva> i was thinking centos and debian/ubuntu folks
21:44:53 <notmyname> clayg: which unblocks you. and the rest of it can be done independently of the swiftstack packaging
21:45:08 <clayg> tdasilva: i know, but I also don't know how that works
21:45:13 <notmyname> tdasilva: wouldn't that "just" be to tell zigo/onovy and zaitcev?
21:45:31 <clayg> and clayg!  (but i already know)
21:45:41 <kota_> lol
21:45:57 <clayg> oh and kota!?  we need a liberasurecode annouce mailing list or something?
21:46:00 <notmyname> clayg: you taking on the distro packaging work now?
21:46:13 <clayg> i would *totally* subscribe to that - hell I would *send* messages to that!?
21:46:29 <clayg> really i'm not so different from distro packagers tho - that's my point
21:46:45 <clayg> i think this is how compiled software is supposed to be released/packaged/updated?
21:46:48 <clayg> i don't know
21:46:53 <tdasilva> do we need to build a matrix of what libec/pyeclib work with which version of swift???
21:47:01 <clayg> tdasilva: :'(
21:47:06 <tdasilva> or is that implicit in the swift git repo?
21:47:08 <clayg> not before we cut a liberasurecode tag - that's for sure!
21:47:19 <notmyname> tdasilva: no. or rather it's simply "at least the current liberasurecode tag" right?
21:47:28 <clayg> tdasilva: it's not tho - we *only* depend on pyeclib - which has no specific depdendency on liberasurecode version (!?)
21:47:31 <clayg> it's madness
21:47:42 <clayg> worse we don't really handle a *testing* matrix
21:47:49 <tdasilva> yeah
21:47:50 <zaitcev> I thnk we're kinda "old that worked, circa 1.1.1/1.0.1" and "new gen 1.3.1/1.2.0"
21:48:00 <clayg> in my new world on vsaio I only test pyeclib master with liberasurecode master with swift master
21:48:13 <notmyname> only because those things have never to this point had releases that require us to specify version dependencies, right?
21:48:41 <tdasilva> for example, centos guys were asking me if version 1.3.1 of liberasure code worked with mitaka and liberty and older releases....
21:48:42 <notmyname> swift says use a version of pyeclib, and that's fine. and we can set a >= version now in pyeclib for liberasurecode
21:48:43 <zaitcev> no, PyECLib and liberasurecode had good dependencies once Tushar took them off Kevin
21:48:52 <clayg> for me there is some risk that some older version of swift won't work with the latest version of pyeclib/liberasurecode and I'm going to cry
21:48:53 <zaitcev> er. had good version numbers and releases
21:49:00 <clayg> but for now the interface of pyeclib seems to be holding strong
21:49:02 <notmyname> and I'll backport the dependency bump to stable branches
21:49:30 <zaitcev> tdasilva: my understanding is Liberty is okay, anything before that I dunno
21:49:39 <tdasilva> zaitcev: ok
21:49:42 <notmyname> I think we're all ok. there's nothing overly complicated
21:50:00 <tdasilva> does bindep allow to specify a version? https://github.com/openstack/pyeclib/blob/master/bindep.txt
21:50:26 <notmyname> tdasilva: probably? (it better)
21:50:34 <notmyname> surely it does
21:50:38 <clayg> i will be doing a compatiblity test of latest pyeclib/liberasure code going back to some pretty old versions of swift during our QA - I can publish those results?
21:50:49 <notmyname> clayg: that would be great
21:50:50 <tdasilva> http://docs.openstack.org/infra/bindep/readme.html#writing-requirements-files
21:50:56 <clayg> but I want the tag to cut the release from first (maybe that's backwards from some folks workflows)
21:51:00 <tdasilva> clayg, notmyname: +1
21:51:38 <notmyname> clayg: no. liberasurecode is the end of the dependency DAG. we do that one first, and that's the easy part
21:51:54 <notmyname> no to it being backwards. we need to tag liberasurecode first
21:51:55 <clayg> notmyname: %^&*ing A!?
21:51:59 <tdasilva> right
21:52:05 <clayg> I don't have +2 on that patch - if kota_ does he should merge it right now
21:52:14 <clayg> IMHO ;)
21:52:17 <tdasilva> clayg: we can fix that :)
21:52:32 <notmyname> :-)
21:52:44 <clayg> tdasilva: i'm all up in the reviewing pyeclib/liberasurecode - learning curve to be sure - but I'll do best promise!
21:53:05 <tdasilva> lol
21:53:19 <notmyname> what if we simply added "swift-core" to those?
21:53:27 <tdasilva> that works too
21:53:30 <kota_> if it's ok for everyone, i can add +A for that now.
21:53:47 <kota_> or more fresh eyes someone wans?
21:53:49 <kota_> wants
21:54:00 <notmyname> kota_: how do you feel about adding swift-core to the liberasurecode-core list?
21:54:08 <clayg> kota_: I'd love to think we *had* someone would could do that?  tdasilva? timburke? kevin? tsg?
21:54:11 <tdasilva> kota_: any chance to get kevin or tsg to look at that patch?
21:54:23 <notmyname> clayg: yeah, we have several, actually
21:54:39 <kota_> tdasilva: not sure, tsg seems to notify the patch pushed.
21:55:24 <kota_> notmyname, clayg: yeah, more cores are great to liberasurecode, imo
21:55:31 <notmyname> done
21:55:32 <clayg> I'll email tsg and tushar - if we can plan to +A it and make a tag first thing tomorrow - I say we wait until then and see if we can get someone else to look at it too?
21:55:49 <notmyname> clayg: you can +2/+A stuff in pyeclib and liberasurecode now
21:55:51 <kota_> clayg: tsg will get 2 emails
21:55:57 <kota_> JK
21:56:48 <notmyname> anything else to bring up in the next 3 minutes?
21:56:51 <timburke> not related to release-y stuff, but i'd be interested in people's opinions on https://review.openstack.org/#/c/391682/ -- basically, take the concurrent bulk-delete work and apply the same principle to the HEAD requests we do for SLO
21:57:30 <joeljwright> timburke: I'll take a look at that while I'm neck deep in SLO anyway
21:57:37 <notmyname> joeljwright: thanks!
21:57:37 <timburke> thanks!
21:57:59 <kota_> aaaa,
21:58:35 <kota_> clayg, notmyname, tdasilva: that patch depends on https://review.openstack.org/#/c/387879/2
21:58:41 <kota_> note that
21:58:54 <clayg> kota_: that's next
21:59:37 <clayg> kota_: lol - yeah I ment kevin & tsg :P
21:59:44 <notmyname> we're at time. thank you for your work on swift. thanks for coming today
21:59:48 <kota_> that mean, https://review.openstack.org/#/c/393595/3 -> https://review.openstack.org/#/c/387879/2
22:00:06 <kota_> oops opposite...
22:00:23 * cschwede is too late to the party :/ calendar messed up the time change. bummer.
22:00:37 <notmyname> cschwede: :-(
22:00:47 <notmyname> cschwede: use iceland time
22:00:58 <notmyname> #endmeeting