21:00:18 #startmeeting swift 21:00:19 Meeting started Wed Nov 9 21:00:18 2016 UTC and is due to finish in 60 minutes. The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:20 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:22 The meeting name has been set to 'swift' 21:00:23 who's here for the swift team meeting? 21:00:28 hi 21:00:39 hi 21:00:45 hey 21:00:47 hello 21:00:53 evenin 21:01:01 hi 21:01:01 hello 21:01:11 o/ 21:01:12 hello 21:01:13 . 21:01:18 hello 21:01:18 o/ 21:01:23 . 21:01:40 welcome everyone 21:01:48 greetings programs! 21:02:20 not a ton on the agenda for today, so plenty of time for open discussion (or ending early?) 21:02:23 #link https://wiki.openstack.org/wiki/Meetings/Swift 21:02:27 clayg: hello, fellow human 21:03:04 first up, reminder that time zones are a thing, and we ignore them for meeting scheduling 21:03:37 i like this time better 21:03:40 so if you're here now (and you're in the US) then you did the right thing and got to the meeting at the right time 21:03:47 meeting is at 2100UTC 21:03:56 #topic PTG signups 21:04:01 #link https://wiki.openstack.org/wiki/Meetings/Swift 21:04:03 nope 21:04:12 #undo 21:04:13 Removing item from minutes: 21:04:17 #link https://www.openstack.org/ptg 21:04:19 there 21:04:31 the PTG signups are open 21:04:51 I've asked to have 3 full days days: wed, thurs, fri 21:04:52 still waiting on https://review.openstack.org/#/c/272727/ ... 21:05:22 monday and tuesday of that week will be for some cross project stuff (also note that monday is presidents' day in the US) 21:06:01 tickets for the PTG are $100 and there is a hotel block. there's a chance that things could fill up, so book early 21:06:16 any question about the PTG? 21:06:21 Remember when we had our own hackathons and we didn't have to ask for time from some organizing board? 21:06:25 notmyname: I this dream that we all went to the PTG - but we sort of just hole'd up together like it was a regular hackathon Mon-Fri and everyone thought it was weird that we didn't go argue with people on Mon-Tue 21:07:10 yes, yes, I know. everything we used to do was awesome and everything in the future is terrible. but we're doing the PTG, and it will be fine 21:07:11 ;-) 21:07:15 notmyname: do you know if the tickets are limited? 21:07:21 that's a good idea; we can wander off during the cross-project times and be cross by ourselves 21:07:22 tdasilva: I do not know 21:07:27 ok 21:07:35 (not to avoid other projects, just to get more time) 21:07:36 notmyname, do we get some sort of ATC pass to attend PTG? 21:07:44 ntata: no 21:08:06 ntata: nope, one hundred bones across the board 21:08:23 but attending the PTG will get you a ticket to the forum in boston in may 21:08:25 tdasilva: I heard that they are, but I don't have any official to point to 21:08:58 ntata: it's like this http://www.gifwave.com/media/37332_movie-awesome-coffee-lego-the-lego-movie.gif 21:09:15 mathiasb: mmm...interesting. getting refund for hotel reservation is easy, not sure about eventbrite tickets... 21:09:31 lol 21:09:42 will patchbot be there? 21:09:55 tdasilva: IIRC you can get a full refund up to like february 17th or so 21:09:59 pdardeau: that can be our mon-tues project! 21:10:08 clayg: great idea! :-) 21:10:23 since notmyname doesn't seem to be invested in fixing him - and it would everyone (who uses patchbot; which is probalby just us?) 21:10:38 *would benifit 21:10:47 there's a few others. I'm interested. just get pulled into other things 21:10:53 :P 21:10:58 i'm just poking fun - sorry 21:11:42 tdasilva: full refund possible until february 15th https://www.eventbrite.com/e/project-teams-gathering-tickets-27549298694 21:12:12 despite the frustrations of the less time and more money, I do think the PTG will be good for us. don't think of it as replacing midcycles, though. think of it as the dev part of the summits. that's actually what it is 21:12:16 yep, had just read that there...guess i should read the full description before asking... 21:12:53 will we have a separate midcycle too? 21:12:55 and after the PTG, based on what's avaialble for us as a team in boston, we can figure out if we need to do self-organized midcycles again 21:13:55 anything else about the PTG that I can answer or go find an answer for? 21:14:18 notmyname: thanks for the info! 21:14:36 nadeem: unexpected; expectation is that ptg replaces mid-cycle 21:14:52 still 4x travel per year all offical os sponsered 21:15:31 sounds good 21:15:49 actually...one more PTG question...sorry..if we want to hold cross-project meetings, do we have to say anything? like so it gets planned? 21:15:51 clayg: sortof, but not really. yes, there's still 4x travel and they're now all found-ation related. but the PTG is mroe like the summit instead of a midcycle. we'll see what the new "forum" will be like and if we need to do a midcycle instead of that 21:16:03 tdasilva: that... is a really good question 21:16:16 i'm thinking about all the golang emails that are going on, that could become a cross-project conversation... 21:16:26 it looks like the mon-tues are pretty set for what projects/teams will have meetings, but I don't know how they'll be scheduled 21:16:48 tdasilva: I'll look in to that 21:16:54 ok 21:17:21 tdasilva: I think mon-tues are more like mid-cycles for horizontal teams - so maybe we go to "infra" if we want to work on golang in CI or "docs" if you want to talk about making godoc output stuff that matches sphinx 21:17:53 clayg: yeah, that's what i was thinking...was just wondering if we need to announce or ask to have that topic on their agenda?? 21:18:29 tdasilva: i'm not sure it's clear how adgendas will be set - could very well be a dotacracy - so just how up and say "hey, i want to talk about this!" 21:18:31 I'll start a ML thread 21:18:39 but yeah, also need to remember about the cross-project vs. inter-project distinction 21:18:53 tdasilva: yeah, exactly 21:18:57 i'm guessing having a barbican conversation wouldn't count as cross-project 21:19:07 tdasilva: maybe we could start a golang-wg and have our *own* little horizontal team that folks can show up to! 21:20:15 moving on 21:20:16 i *should* start a dependency-hygiene-wg 21:20:18 #topic release 21:20:25 lol 21:20:42 https://bugs.launchpad.net/swift/+bug/1633647 and https://bugs.launchpad.net/swift/+bug/1624088 have landed in swift, so I'm working on getting a swift release finished up 21:20:42 Launchpad bug 1633647 in OpenStack Object Storage (swift) "bad fragment data not detected in audit" [Critical,Fix released] 21:20:43 Launchpad bug 1624088 in OpenStack Object Storage (swift) "EC missing durable can prevent reconstruction" [Critical,Fix released] 21:20:52 whoot! 21:20:58 :) 21:21:06 I've also pushed https://review.openstack.org/#/c/395768/ for a swiftclient release (has the sessions support) 21:21:28 did the keystone auth v1 plugin make it too? 21:21:31 yep 21:21:36 :) 21:22:28 I mention this is the commit messages, but FYI I'm reordering the AUTHORS files. turns out, names are terrible, and there's no way to actually do the "right" thing for ordering them. so the new plan is to simply order them by the first character/byte in the author field 21:22:35 notmyname: when I glance down the list of *high* priority bugs - this one stands out -> https://bugs.launchpad.net/swift/+bug/1516579 21:22:35 Launchpad bug 1516579 in OpenStack Object Storage (swift) "PUTing a 0-byte object and supplying an Etag results in a 503 response" [High,Fix committed] - Assigned to Samuel Merritt (torgomatic) 21:22:39 but it's already landed 21:22:49 then this one -> https://bugs.launchpad.net/swift/+bug/1554233 21:22:49 Launchpad bug 1554233 in OpenStack Object Storage (swift) "Servers-per-port can consume excessive OS threads" [High,In progress] - Assigned to Samuel Merritt (torgomatic) 21:23:19 ^ cheap to fix/land and may be hurting adoption of servers-per-port (OTOH, there's a good workaround - so maybe that should only be medium) 21:24:52 oh wait, not there was *different* high priority bug with 0 in the title - it was kota's fix for multi ec policy 21:25:16 I feel like we do this every time a release gets close: "but wait! there's one more!" :) 21:25:56 clayg: https://bugs.launchpad.net/swift/+bug/1549110 ? 21:25:56 Launchpad bug 1549110 in OpenStack Object Storage (swift) "EC: object-reconstuctor got unhandled 0 divison error" [High,Fix released] - Assigned to Kota Tsuyuzaki (tsuyuzaki-kota) 21:26:16 i'm not doing well finding bugs today :'( 21:26:20 kota_: yes thank you! 21:26:38 notmyname: there *really* is tho!? and the one that came up this morning with the if-none-match 21:26:53 clayg: yeah, you're not wrong :-) 21:27:07 this one? https://bugs.launchpad.net/swift/+bug/1640448 21:27:07 Launchpad bug 1640448 in OpenStack Object Storage (swift) "If-None-Match doesn't work on deleted object" [High,Confirmed] 21:27:08 notmyname: we can't have "releases" we need to just cut a release every couple of weeks because it really keeps getting better and better 21:27:26 i know there's a regression that slips in every *now* and then - but all the more reason to keep going keep going! 21:28:05 so that's why I bring up the topic: what's the stuff we know about that we should land before we cut a release? 21:28:09 notmyname: neyway - i don't really care - we're so far behind because of these blocker bugs - we haven't even released a Newton yet :'( 21:28:29 notmyname: we'll the 21:29:08 kota_: lp bug #1549110 is fix released! 21:29:08 Launchpad bug 1549110 in OpenStack Object Storage (swift) "EC: object-reconstuctor got unhandled 0 divison error" [High,Fix released] https://launchpad.net/bugs/1549110 - Assigned to Kota Tsuyuzaki (tsuyuzaki-kota) 21:29:10 notmyname: ^ that one 21:29:24 i think i'm good with that 21:29:55 i don't know of others - but I could be just dropping the ball on something terrible and huge - i need to go back through the open priority bugs 21:30:32 https://bugs.launchpad.net/swift/+bug/1628906 should be confirmed or not on master 21:30:32 Launchpad bug 1628906 in OpenStack Object Storage (swift) "swift-object-reconstructor memory leak" [High,Confirmed] 21:32:01 good news https://bugs.launchpad.net/swift/+bug/1491908 can be marked as closed 21:32:01 Launchpad bug 1491908 in OpenStack Object Storage (swift) "proxy server reports 202 even though backend didn't accept object" [High,In progress] 21:33:29 ok, here's a summary: there's a few high priority bugs that I want to look at further. I'll be working on the authors/changelog patch for the release, so if anyone sees something that really should land, let me know. we'll hold a release for it 21:33:58 ones that say "memory leak" or have 503s are stuff that sticks out to me 21:34:45 notmyname: looking that 202 accepted bug for 409 - i don't see where the change removed the logic in the proxy? 21:35:20 ah? I was going by the commit subject and the comments in LP 21:35:27 notmyname: I'll try to confirm but it looks like ctennis' concern is still there -> https://github.com/openstack/swift/blob/a30351bdfc60f827979c5afe76ad40bf3a2b05c3/swift/proxy/controllers/obj.py#L482 21:35:46 ok. thanks. I set it back 21:35:56 notmynam: i don't think the memory leak on the master, anyway we need to confirm with the reporter though. 21:36:02 but now it's a bit more dubious since container-sync doesnt' rely on that behavior 21:36:51 kota_: my opinion of it is we fixed *a* memeory leak - and there are probly more - but we don't have to keep the bug open until we fix all of them - we just need to keep fixing them as we can isolate something 21:37:33 clayg: fair enough 21:37:58 anything else to bring up about a pending release? 21:38:32 ok 21:38:36 #topic open discussion 21:38:44 anything else to bring up this week? 21:39:21 notmyname: i want to land https://review.openstack.org/#/c/393595/3 and tag a liberasure right away 21:39:40 notmyname: kota_: how can we make that happen (EOD tomorrow?) 21:40:11 clayg: as soon as we do that and get the version bumped in swift's requirements, I want to do a release of swift. (even if it's the day after another release) 21:40:38 kota_: after the liberasurecode patch lands, what's necessary to do in pyeclib? 21:40:40 anything? 21:40:53 ah 21:40:56 notmyname: yes yes all of these things right away - i just don't know how anything works? tdasilva can you save us??? 21:41:13 i think you will need to also do a pyeclib release, right? 21:41:16 notmyname: if we could make a hard coded version requirements in the *pyeclib* 21:41:17 clayg: those aren't the hard parts. just have to paste a SHA into a repo 21:41:41 notmyname: it seems better, otherwise pyeclib can allow to use older version. 21:41:42 and then swift's requirement point to the new pyeclib release 21:41:44 tdasilva: IME just building a new liberasure with current pyeclib is good enough for me? 21:41:46 of libeasurecode 21:41:52 tdasilva: true 21:41:55 ok, so release libearasurecode, bump dep in pyeclib, then release pyeclib, then bump dep in swift, then release swift? 21:42:12 notmyname: right 21:42:19 tdasilva: like for me I would just update my liberasurecode packages and make drag it forward for all versions of pyeclib - I think swiftstack's pacaking is full on crazy town tho 21:42:39 clayg: yeah, that will work for swiftstack, but we shoudl do the rest to avoid older versions being used 21:42:40 but right now I only package tagged liberasurecode releases - i'm totally happy to just "fix that" and move to where I can package any random sha 21:42:55 notmyname: totally granting *everyone* knows better than I do 21:43:00 heh 21:43:22 yeah, i don't think pyeclib actually has a dependency on a specific version of a liberasurecode 21:43:29 can't find it 21:43:34 notmyname: but also fully expressing this needs to happen right away - if I get a tagged liberasurecode release out of the deal I win - if I not - i'm doing something different - and I'll check in when I'mdone 21:43:37 so clayg's method might be right 21:43:57 tdasilva: IIRC either you or me can do the ec library releases. do you want to do that as soon as the patch lands, or would you like me to? 21:44:05 the important part is getting downstream packagers to package the new liberasure code asap 21:44:15 clayg: I don't know why we can't tag liberasurecode as soon as the patch lands 21:44:26 i am a downstream packager! i will do it the same day you make the tag - promise! 21:44:39 clayg: hehehe, true 21:44:52 i was thinking centos and debian/ubuntu folks 21:44:53 clayg: which unblocks you. and the rest of it can be done independently of the swiftstack packaging 21:45:08 tdasilva: i know, but I also don't know how that works 21:45:13 tdasilva: wouldn't that "just" be to tell zigo/onovy and zaitcev? 21:45:31 and clayg! (but i already know) 21:45:41 lol 21:45:57 oh and kota!? we need a liberasurecode annouce mailing list or something? 21:46:00 clayg: you taking on the distro packaging work now? 21:46:13 i would *totally* subscribe to that - hell I would *send* messages to that!? 21:46:29 really i'm not so different from distro packagers tho - that's my point 21:46:45 i think this is how compiled software is supposed to be released/packaged/updated? 21:46:48 i don't know 21:46:53 do we need to build a matrix of what libec/pyeclib work with which version of swift??? 21:47:01 tdasilva: :'( 21:47:06 or is that implicit in the swift git repo? 21:47:08 not before we cut a liberasurecode tag - that's for sure! 21:47:19 tdasilva: no. or rather it's simply "at least the current liberasurecode tag" right? 21:47:28 tdasilva: it's not tho - we *only* depend on pyeclib - which has no specific depdendency on liberasurecode version (!?) 21:47:31 it's madness 21:47:42 worse we don't really handle a *testing* matrix 21:47:49 yeah 21:47:50 I thnk we're kinda "old that worked, circa 1.1.1/1.0.1" and "new gen 1.3.1/1.2.0" 21:48:00 in my new world on vsaio I only test pyeclib master with liberasurecode master with swift master 21:48:13 only because those things have never to this point had releases that require us to specify version dependencies, right? 21:48:41 for example, centos guys were asking me if version 1.3.1 of liberasure code worked with mitaka and liberty and older releases.... 21:48:42 swift says use a version of pyeclib, and that's fine. and we can set a >= version now in pyeclib for liberasurecode 21:48:43 no, PyECLib and liberasurecode had good dependencies once Tushar took them off Kevin 21:48:52 for me there is some risk that some older version of swift won't work with the latest version of pyeclib/liberasurecode and I'm going to cry 21:48:53 er. had good version numbers and releases 21:49:00 but for now the interface of pyeclib seems to be holding strong 21:49:02 and I'll backport the dependency bump to stable branches 21:49:30 tdasilva: my understanding is Liberty is okay, anything before that I dunno 21:49:39 zaitcev: ok 21:49:42 I think we're all ok. there's nothing overly complicated 21:50:00 does bindep allow to specify a version? https://github.com/openstack/pyeclib/blob/master/bindep.txt 21:50:26 tdasilva: probably? (it better) 21:50:34 surely it does 21:50:38 i will be doing a compatiblity test of latest pyeclib/liberasure code going back to some pretty old versions of swift during our QA - I can publish those results? 21:50:49 clayg: that would be great 21:50:50 http://docs.openstack.org/infra/bindep/readme.html#writing-requirements-files 21:50:56 but I want the tag to cut the release from first (maybe that's backwards from some folks workflows) 21:51:00 clayg, notmyname: +1 21:51:38 clayg: no. liberasurecode is the end of the dependency DAG. we do that one first, and that's the easy part 21:51:54 no to it being backwards. we need to tag liberasurecode first 21:51:55 notmyname: %^&*ing A!? 21:51:59 right 21:52:05 I don't have +2 on that patch - if kota_ does he should merge it right now 21:52:14 IMHO ;) 21:52:17 clayg: we can fix that :) 21:52:32 :-) 21:52:44 tdasilva: i'm all up in the reviewing pyeclib/liberasurecode - learning curve to be sure - but I'll do best promise! 21:53:05 lol 21:53:19 what if we simply added "swift-core" to those? 21:53:27 that works too 21:53:30 if it's ok for everyone, i can add +A for that now. 21:53:47 or more fresh eyes someone wans? 21:53:49 wants 21:54:00 kota_: how do you feel about adding swift-core to the liberasurecode-core list? 21:54:08 kota_: I'd love to think we *had* someone would could do that? tdasilva? timburke? kevin? tsg? 21:54:11 kota_: any chance to get kevin or tsg to look at that patch? 21:54:23 clayg: yeah, we have several, actually 21:54:39 tdasilva: not sure, tsg seems to notify the patch pushed. 21:55:24 notmyname, clayg: yeah, more cores are great to liberasurecode, imo 21:55:31 done 21:55:32 I'll email tsg and tushar - if we can plan to +A it and make a tag first thing tomorrow - I say we wait until then and see if we can get someone else to look at it too? 21:55:49 clayg: you can +2/+A stuff in pyeclib and liberasurecode now 21:55:51 clayg: tsg will get 2 emails 21:55:57 JK 21:56:48 anything else to bring up in the next 3 minutes? 21:56:51 not related to release-y stuff, but i'd be interested in people's opinions on https://review.openstack.org/#/c/391682/ -- basically, take the concurrent bulk-delete work and apply the same principle to the HEAD requests we do for SLO 21:57:30 timburke: I'll take a look at that while I'm neck deep in SLO anyway 21:57:37 joeljwright: thanks! 21:57:37 thanks! 21:57:59 aaaa, 21:58:35 clayg, notmyname, tdasilva: that patch depends on https://review.openstack.org/#/c/387879/2 21:58:41 note that 21:58:54 kota_: that's next 21:59:37 kota_: lol - yeah I ment kevin & tsg :P 21:59:44 we're at time. thank you for your work on swift. thanks for coming today 21:59:48 that mean, https://review.openstack.org/#/c/393595/3 -> https://review.openstack.org/#/c/387879/2 22:00:06 oops opposite... 22:00:23 * cschwede is too late to the party :/ calendar messed up the time change. bummer. 22:00:37 cschwede: :-( 22:00:47 cschwede: use iceland time 22:00:58 #endmeeting