21:00:10 <notmyname> #startmeeting swift
21:00:16 <openstack> Meeting started Wed Jan 17 21:00:10 2018 UTC and is due to finish in 60 minutes.  The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:17 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:20 <openstack> The meeting name has been set to 'swift'
21:00:21 <notmyname> who's here for the swift team meeting?
21:00:25 <kota_> hi
21:00:27 <m_kazuhiro> o/
21:00:29 <mattoliverau> o/
21:00:41 <rledisez> good (morning|afternoon|evening) o/
21:01:09 <notmyname> rledisez: what about the night time?
21:01:36 <rledisez> yeah, my bad good night everybody :)
21:01:43 <notmyname> :-)
21:01:47 <acoles> hello
21:01:55 <joeljwright> hello
21:03:17 <timburke> o/
21:03:22 <notmyname> welcome
21:03:47 <notmyname> there's been a ton of stuff landing since our last meeting. great work!
21:03:53 <notmyname> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:03:55 <mattoliverau> Or you could use UGT time then we can all say good morning
21:04:00 <notmyname> there's the agenda for this week
21:04:18 <joeljwright> good morning!
21:04:50 <notmyname> mostly I want to review stuff for the 2.17.0 release
21:04:55 <torgomatic> hi
21:05:01 <notmyname> #topic next release
21:05:04 <notmyname> #link https://wiki.openstack.org/wiki/Swift/PriorityReviews
21:05:20 <notmyname> nearly everything on there is marked off
21:05:26 <notmyname> patch 532383
21:05:27 <patchbot> https://review.openstack.org/#/c/532383/ - swift - Don't make async_pendings during object expiration
21:05:34 <notmyname> ...is open
21:05:54 <notmyname> as is the data segments in SLOs patch 365371
21:05:54 <patchbot> https://review.openstack.org/#/c/365371/ - swift - Add support for data segments to SLO and Segmented...
21:06:15 <notmyname> unless there's any other major bugs or blockers, I'd like to see those land and then tag the release
21:06:19 <joeljwright> I think timburke addressed all the outstanding issues on 365371
21:06:25 <acoles> I will look at p 532383 again tomorrow, most likely +A
21:06:26 <patchbot> https://review.openstack.org/#/c/532383/ - swift - Don't make async_pendings during object expiration
21:06:35 <notmyname> acoles: thanks
21:06:48 <notmyname> joeljwright: yeah, I think it's just that people have been (rightly) reviewing the expiry patches
21:07:01 <joeljwright> +1
21:07:19 <notmyname> I've got the start of the authors/changelog patch at https://review.openstack.org/#/c/534528/
21:07:19 <patchbot> patch 534528 - swift - WIP
21:07:39 <notmyname> thanks to clayg and timburke for the comments so far
21:08:00 <notmyname> I'll update those today or tomorrow, and add info about the other patches that have landed
21:08:18 <notmyname> any questions on the mechanics or logistics of this release?
21:08:25 <notmyname> torgomatic: we'll get to your question next
21:09:59 <notmyname> ok, let's move on to torgomatic's question then :-)
21:10:04 <notmyname> #topic checkpoint release?
21:10:18 <notmyname> torgomatic: would you like to introduce it?
21:10:36 <torgomatic> right now, we've got to consider rolling cluster upgrades across all past versions, and that makes things more difficult
21:11:21 <torgomatic> I'd like to introduce some cutoffs where we don't allow upgrades from any old version to the latest, but make you go through some intermediate steps first
21:11:27 <torgomatic> or, at least, to talk about doing that
21:12:10 <notmyname> thanks. what does everything think? what questions do you have?
21:12:20 <torgomatic> One way is checkpoint releases: you have some releases C1, C2, etc. and you can upgrade from any version v < C1 to C1, then any version v <= C1 < C2 to C2, and so on
21:12:27 <notmyname> (it's a big question, IMO, so I'd expect us to talk about it for a while)
21:12:30 <torgomatic> (sorry, I have the slow fingers today)
21:13:36 <torgomatic> anyhow, this came up in the context of https://review.openstack.org/#/c/534749/ , where we have a rolling-upgrade compatibility issue with proxies from 2013 and earlier
21:13:37 <patchbot> patch 534749 - swift - Preserve expiring object behaviour with old proxy-...
21:13:41 <acoles> would it lose us the openstack upgrade tag thing? I forget the correct terminology
21:14:08 <notmyname> acoles: that depends on how we role out the idea
21:14:18 <notmyname> *roll
21:14:41 <torgomatic> I think we'd keep it as long as you could perform a rolling upgrade to the latest checkpoint release. (That's a property I very much want to keep, FWIW)
21:16:14 <mattoliverau> Going from any to the latest us obvious is a big selling point, but alot of extra effort as me do more and more releases.
21:16:25 <notmyname> aside from the patch you just linked, are there other places where "upgrade straight to latest" is currently obviously hurting us?
21:16:33 <torgomatic> Checkpoints aren't the only way to go; we could define a window where you can upgrade from the last N minor versions or the last Y years
21:16:58 <torgomatic> notmyname: once we land Pete Zaitcev's improvements to the object PUT path, we still have to keep all that old protocol stuff in the proxy forever
21:17:19 <acoles> one advantage I see is that we are reaching the point where we may not have the collective memory to realise when we let an upgrade issue slip in
21:17:38 <notmyname> those are both very good points
21:18:42 <notmyname> kota_: rledisez: joeljwright: what are your thoughts?
21:19:13 <rledisez> I understand the need, and i'm not against the idea. i'm concerned about for how long (in month/years) would you support an upgrade path?
21:19:20 <joeljwright> it sounds reasonable to me, although we always did staged updates anyway
21:20:02 <rledisez> i would say the minimum is a year, probably 2 is better, or to try to sync with some major distro release interval
21:20:07 <kota_> not opposite opinion but a bit worried about how frequently we should care about the checkpoint. I don't expect it'll happen every release.
21:20:34 <kota_> rledisez: it seems i'm with you.
21:20:37 <mattoliverau> It would be good to have stats on how often, people upgrade.. because having a wide enough gap would be good. Say every 3 or 4 years.. or by versions maybe? (Every 2 major versions)
21:20:40 <notmyname> based on the conversation in -swift a few hours ago, I'd said that we needed to make sure a 2 year-old release has supported upgrades but a 4 year-old one seemed like long enough to not worry too mcuh
21:20:40 <torgomatic> Yeah, a couple years at minimum for upgrades IMO.
21:21:05 <joeljwright> torgomatic: +1
21:21:24 <acoles> I assumed we were talking in terms of years
21:21:43 <notmyname> seems we're all in agreement on the unit to measure it
21:21:51 <mattoliverau> Ever couple of major versions would be easy for people to remember.. and represent major API changes
21:22:08 <notmyname> mattoliverau: for all those major version bumps and API updates we do? ;-)
21:22:27 <mattoliverau> Yeah.. but the difference would be we coild
21:22:31 <mattoliverau> Could..
21:23:28 <notmyname> in my mind, we started with swift 1.x. we got to 2.x with storage policies. we'll hit 3.x with a rewrite of the storage layer (eg protocols, new language, etc)
21:23:45 <notmyname> however, that's getting a bit off-topic :-)
21:24:48 <notmyname> torgomatic: help me out here. fi we decided to make a checkpoint at 2.17 and we agreed to support upgrades for 2 years, how long do we support 2.16 features? how long do we support 2.17? 2.18?
21:24:48 <mattoliverau> Your welcome
21:25:27 <torgomatic> notmyname: user-facing features? Pretty much forever, like we do now.
21:25:44 <notmyname> ( rledisez: LOSF is part of that storage layer update, too ;) )
21:25:53 <notmyname> torgomatic: ok. upgrades then?
21:26:04 <torgomatic> If we made 2.17 a checkpoint, then in a couple of years we'd make another checkpoint at, say, 2.34, and then another one at 2.60 in another couple years, and so on
21:26:06 <notmyname> if someone has 2.16 installed, then what?
21:26:19 <timburke> another way of looking at it: how far back will we feel comfortable removing cruft between 2.17 and 2.18?
21:26:29 <torgomatic> so then in five years' time, that person with 2.16 would have to upgrade 2.16 -> 2.17 -> 2.34 -> 2.60 to get to the latest
21:27:00 <torgomatic> and in this case, after we made 2.34 a checkpoint, 2.34.1 could remove compatibility cruft for anything 2.17 -> 2.18
21:27:22 <notmyname> ok, It hink I got it
21:27:42 <notmyname> the difference with a time-based method is that we support eg anything that was released 2 years ago
21:28:03 <notmyname> calendar-based
21:28:29 <notmyname> so in 2018 we drop support for upgrading from anything released in 2015
21:28:32 <notmyname> that sort of thing
21:28:52 <notmyname> the difference is that deployers would need to be on a (very slow moving) upgrade train
21:28:56 <notmyname> right?
21:29:13 <torgomatic> no, deployers could still wait a long time
21:29:36 <torgomatic> when they did decide to upgrade, they'd just have to upgrade several times to get to the latest
21:29:41 <acoles> we would only be dropping support for upgrading *directly* from 2015 to 2018 releases right?
21:29:42 <notmyname> right
21:29:48 <notmyname> acoles: right
21:29:54 <torgomatic> yes
21:29:55 <acoles> right!
21:30:22 <rledisez> an example: what about the renaming of .durable to #d for EC? should it be converted before moving on with version or will it be maintained forever?
21:30:22 <notmyname> torgomatic: deployers could only upgrade, at most, to something released 2 years after their current version
21:30:37 <torgomatic> notmyname: yes
21:30:57 <notmyname> ok. just making sure the different methods are spelled out for everyone (myself included!)
21:31:00 <torgomatic> rledisez: that one, we'd probably have to keep forever since we don't automatically convert things
21:31:16 <timburke> what all would be in-scope for breaking? backend protocols, sure; client APIs, definitely not -- what about operator-facing things like https://github.com/openstack/swift/blob/2.16.0/swift/common/manager.py#L724-L727
21:31:35 <torgomatic> on-disk data formats live forever, storage-node and replication protocols live a couple years
21:32:02 <rledisez> ^ seems reasonable
21:32:03 * kota_ is wondering, it may make us to remove rolling container db scheme change for creating storage policy???
21:33:00 <acoles> rledisez: torgomatic +! and good clarification, on disk data always supported
21:33:01 <timburke> oh yeah, speaking of JIT migrations -- https://review.openstack.org/#/c/502529/
21:33:02 <patchbot> patch 502529 - swift - Create policy_stat table in auditor if missing
21:33:13 <notmyname> here's what I think we should do: ask any further questions about generally how it works, go and sleep on it, and come back together later to talk about specifics or if we even like the idea
21:33:42 <notmyname> I don't think we need to get into "what about lines 200-210 in this module" right now ;-)
21:33:56 <timburke> notmyname: i was just grepping for 'compat' :-)
21:34:00 <notmyname> heh
21:34:08 <notmyname> no, they're important to ask
21:34:14 <timburke> see what broad classes of things are out there
21:34:37 <notmyname> I want to make sure everyone feels they understand the idea of time-based vs checkpoint releases for dropping compatibility
21:35:01 <notmyname> are there any questions? do you (everyone) feel good enough with the concept to explain it to your boss?
21:35:14 <notmyname> if not, ask now!
21:35:49 <notmyname> if so, let's each go ponder it for a bit. not to try to slow down conversation too much, but this may be an interesting topic for the PTG
21:36:05 <mattoliverau> +1 ptg topic
21:36:05 <notmyname> +1 if youre' good with it
21:36:11 <kota_> notmyname: sorry, i have to wait my boss waking up :P
21:36:28 <joeljwright> :D
21:36:35 <mattoliverau> Lol
21:36:38 <notmyname> :-)
21:36:54 <kota_> notmyname: let me confirm on the mean you said *vs* between time-based and checkpoint
21:37:17 <timburke> i'm still wondering whether something like https://github.com/openstack/swift/commit/ebf0b22 would be in scope for removal post-2.17 or not...
21:37:22 <notmyname> kota_: versus. meaning that you know the differences between the two and how they relate
21:37:57 <kota_> yeah, and... i may be missing the context or the summary here.... (scrolling back to the log...)
21:38:38 <notmyname> kota_: I just want to check that you understand both and can think on it for a while in order to figure out what it means for the codebase
21:38:59 <kota_> notmyname: got it
21:39:05 <notmyname> FWIW, I'm not totally sold on the idea, nor am I rushing to make our next release a checkpoint one. but I love the conversation, and I'm happy to be convinced either way (doing a checkpoint or not)
21:39:06 <kota_> thx
21:39:42 <notmyname> kota_: I normally try to leave out idioms and colloquialisms. it's my fault. thanks for asking me to clarify
21:40:04 <kota_> ok ok. I'm sure I can explain the concept and discuss what we (in company) want!
21:40:15 <notmyname> #topic updates on current work
21:40:16 <timburke> soemthing tells me this discussion will continue into Dublin...
21:40:26 <notmyname> let's review some ongoing work
21:40:52 <notmyname> container sharding
21:40:58 <notmyname> mattoliverau: acoles: ?
21:41:16 <acoles> progress continues...
21:41:33 <notmyname> anything blocking you that the rest of us can help with?
21:41:37 <acoles> since last week we decided to reduce scope on listings while sharding is in progress
21:41:43 <acoles> which has helped
21:41:49 <notmyname> great!
21:42:12 <acoles> I have a long patch chain that would be great to get merged, mattoliverau and timburke are helping with that
21:42:23 <notmyname> ok
21:42:33 <notmyname> mattoliverau: timburke: anything to add?
21:42:47 <mattoliverau> And people should join in discussions on Trello or etherpad if there interested
21:43:16 <acoles> oh, today I started another etherpad to start to doc the internal API changes, maybe help newcomers
21:43:37 <mattoliverau> Nice
21:43:40 <acoles> https://etherpad.openstack.org/p/deep-containers-apis
21:43:45 <acoles> #link https://etherpad.openstack.org/p/deep-containers-apis
21:44:12 <notmyname> thanks
21:44:35 <mattoliverau> My week hasn't been the most shard friendly week, but will hopefully spend more time post release reviews (my time is limited :()
21:44:46 <clayg> i was catching up on checkpoint releases
21:45:01 <notmyname> mattoliverau: and you'll be at lca next week, right? so not too much hands-on-keyboard time there
21:45:06 <acoles> BTW, all sharding docs are linked from https://etherpad.openstack.org/p/deep-containers
21:45:18 <acoles> so that is the mother-ship ^^
21:45:30 <mattoliverau> Yup LCA next week \o/
21:45:35 <clayg> I hate to say that if you're running swift from 2015 upgrading to master isn't exactly something I'd have a high confidence in going great anyway... it's more about putting a name to reality
21:46:15 <notmyname> acoles: I added a link to that parent etherpad to the ideas wiki
21:46:29 <notmyname> kota_: any progress on s3api this week?
21:46:59 <kota_> i update the logger patch according to tim's comment, and waiting his review
21:47:05 <notmyname> ok
21:47:28 <notmyname> m_kazuhiro: rledisez: any updates for us on the task queue?
21:47:29 <timburke> i knew there was another patch i should review!
21:47:30 <kota_> on the functional, I 've started to make sure what's the status of tdasilva
21:47:40 <kota_> since yesterday
21:47:56 <m_kazuhiro> I'm working at patch 517389
21:47:57 <patchbot> https://review.openstack.org/#/c/517389/ - swift - Update object-expirer to use general task queue sy...
21:48:20 <m_kazuhiro> I implemented all tests so I removed 'WIP' from the patch.
21:48:39 <kota_> nice
21:48:48 <notmyname> nice
21:49:10 <notmyname> m_kazuhiro: anything currently blocking you, aside from reviews?
21:50:17 <m_kazuhiro> Nothing. Review is what I want.
21:50:51 <notmyname> rledisez: how's progress on LOSF work?
21:51:35 <rledisez> pretty good. we should make a (big) move to put the code on production in the next 2 or 3 weeks
21:51:58 <mattoliverau> Wow
21:52:02 <rledisez> so, we may be running a whole on production wiht LOSF by the next PTG
21:52:02 <mattoliverau> Cool
21:52:13 <notmyname> that is "wow" :-)
21:52:42 <notmyname> speaking of the PTG...
21:52:43 <rledisez> also, a new developer will join our team in march, he will assist alexandre in upstreaming the code
21:52:46 <notmyname> that's coming up soon
21:52:59 <notmyname> rledisez: cool
21:53:25 <notmyname> the PTG is at the end of february
21:53:44 <notmyname> I'd like to start organizing topics in two weeks
21:54:01 <clayg> rledisez: I haven't heard anything about how alexandre ended up dealing with the eventlet + grpc issues he was dealing with?
21:54:15 <notmyname> also, I'll be out next wednesday. would someone like to run the meeting? or would you like to skip it?
21:54:59 <mattoliverau> Well really you'll just be on my time
21:55:16 <mattoliverau> But I'll be busy so happy to akio
21:55:22 <mattoliverau> *skip
21:56:09 <rledisez> clayg: right now there is no real solution. we pinned a specific version of grpc that works (partialy, but enough for us) with eventlet. that's not a long term solution. either droping eventlet or changing grpc for another protocol is the way to go
21:57:36 <notmyname> I think everyone already logged off. so that answers that about skipping next week :-)
21:57:42 <joeljwright> :)
21:57:58 <notmyname> meet again in two weeks. jan 31, 2100UTC
21:58:12 <notmyname> thanks for your work on swift. thanks for coming today
21:58:16 <notmyname> #endmeeting