21:00:23 #startmeeting swift 21:00:24 Meeting started Wed May 31 21:00:23 2017 UTC and is due to finish in 60 minutes. The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:25 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:27 The meeting name has been set to 'swift' 21:00:28 yay! 21:00:31 who's here for the swift team meeting? 21:00:33 o/ 21:00:36 o/ 21:00:42 yo yo yo 21:00:44 o/ 21:00:47 o/ 21:00:49 hi 21:01:07 o/ 21:01:09 hello 21:01:16 0/ 21:01:20 o/ 21:01:26 hi 21:01:42 I feel like I just talked to half of you ;-) 21:01:48 Lol 21:01:51 jeffli: were you aware of the swift meeting that was 14 hours ago? 21:02:08 jeffli: probably a much better meeting time for your time zone :-) 21:02:19 anyway, that's one of the topics for *this* meeting today 21:02:28 yeah. but I had an unexpected meeting at that time. 21:02:34 me too! 21:02:39 actually, no. I expected it 21:03:10 meeting agenda page is... 21:03:12 #link https://wiki.openstack.org/wiki/Meetings/Swift 21:03:23 notmyname: you had conflicts meetings at midnight? 21:03:30 yeah. sleep ;-) 21:03:43 also video games 21:03:45 pfft, sleep 21:03:48 lol 21:03:59 there's a few things to go over in this meeting today, so let's get started 21:04:09 #topic last meeting follow-ups 21:04:30 ok, meta comment: "this meeting" "next meeting" and "last meeting" are now confusing 21:04:42 har har 21:04:44 personally, I do not want to have a "main" meeting and "alternate" meeting 21:04:56 so I've been referring to them as the 0700 meeting and the 2100 meeting 21:05:04 (this is the 2100 meeting) 21:05:16 ok, so last 2100 meeting follow-ups 21:05:28 ^ nice 21:05:45 again, no status updates on writing things down for the pike goals. but I'll keep it on the agenda to keep guilting myself about it 21:06:15 rledisez: last week you promised charts and graphs about memory consuption. did you and/or jeffli get a chance to work on that? 21:06:27 notmyname: I think I promised you a patch on the uwsgi goal 21:06:36 notmyname: but I haven't done it ... yet 21:06:37 notmyname: not on my side, we were off most of the week 21:06:37 i promise nothing 21:06:42 notmyname: for next we should have them 21:06:43 rledisez: ..and a mailing list discussion 21:06:45 rledisez: that's the ticket! 21:06:54 great, thanks 21:06:58 acoles: ok :-) 21:07:06 me neither 21:07:17 notmyname: same for mailing discussion, we'll start it by the end of this week 21:07:46 the ML discussion was the share that memory info (and the related discussion about which LOSF scheme to use), right? 21:08:08 yes, so that anyone can follow progression and choices and give insight 21:08:20 ok, perfect 21:08:28 so perfect 21:08:32 I'll look for that ML thread from rledisez and jeffli this week 21:08:34 rledisez: is a hero 21:08:37 please let me know if you need any help on it 21:08:40 jeffli: is a hero 21:08:56 Yay to heros 21:09:01 ok, next up 21:09:02 notmyname: is a facilitator of global communication 21:09:22 patch 466255 from tdasilva and cschwede_ 21:09:22 https://review.openstack.org/#/c/466255/ - swift - Make mount_check option usable in containerized en... 21:09:28 * clayg hi-five's mattoliverau for assistance with the running commentary 21:09:59 and we were waiting for comments from cschwede_ 21:10:11 uhm, what do you want to hear? 21:10:14 ...which it looks like we have? 21:10:33 clayg: it's you and me buddy 21:10:43 there was one question about the re-instroduction of the lstat() call? 21:11:13 not that the comment needs to be updated, but are we ok with the additional call? 21:11:15 well, it's adding one lstat, but then it's only happening when you have an unmounted or failed disk 21:11:45 it should not happen when you're running outside containers with healthy disks 21:12:09 I like "the normal case is unaffected" 21:12:38 so clayg already had a +2 on it 21:12:45 and IIRC is still good with it 21:12:47 right, normal usecases should'nt be affected at all (hopefully?) 21:13:04 tdasilva: can you please remove your -2 cork? 21:13:21 zaitcev has a +1 21:13:31 notmyname: sure 21:13:32 so we need another core to review it (zaitcev or otherwise) 21:13:49 i mean if someone has a better idea for containers, i'm happy to hear about it - i didn't see one so far 21:13:49 well, the big question is "were the concerns over the stat call addressed?" 21:14:29 clayg? ^^ 21:15:33 clayg: cschwede_: is there a more 'elegant' solution? the bug clayg linked seemed to detail a better way to solve this 21:15:49 so maybe we could have a short term and long term solution? 21:16:06 I know clayg was also concerned about doing things just for containers 21:16:48 I think mostly the stat syscall (which returns false for everyone except cschwede_'s containers) will mostly be cached/quick - I wouldn't worry about it if I thought there was no way solve the use-case with out it 21:17:10 well, as written in the comments - doing stats has some benefits, because it detects broken disks that are still mounted (probably not always, but at least sometimes) 21:17:58 the docstring implies that if we add the lstat() back in, we might as well use the python version of it instead of our own (but I haven't looked at python's in a long time) 21:18:11 tdasilva: not worried about "doing things just for containers" - worried about "trying to do the *right* thing being slower than just 'doing the first obvious thing that seems to work for containers' (and also then that slowness causing contempt or alienating potential contributions)" 21:18:26 no, that's different afaict - that's another lstat in "normal" cases iirc 21:18:33 notmyname: ^^ 21:18:34 cschwede_: ah ok 21:19:17 ok, so for this meeting, we've got it sorted, and we need another reviewer. shall we ping zaitcev about it? or does someone else want to volunteer? 21:19:34 o/ 21:19:45 tdasilva: thanks 21:20:14 thx Thiago! 21:21:29 ok, last thing in the follow-up section is "composite ring" 21:21:32 and that's landed! 21:21:53 \o/ 21:22:22 \o/ 21:22:28 #topic 0700 meeting update 21:22:42 all right. report on the first 0700 meeting 21:22:54 notmyname: quick question for kota_ and other folks working on composite rings 21:23:01 what's left for the global EC work? 21:23:03 yeah, what's uo? 21:23:03 mahatic: did a great job chairing 21:23:17 just trying to catch up 21:23:18 mahatic: thanks for chairing 21:23:38 tdasilva: follow-ups on https://wiki.openstack.org/wiki/Swift/PriorityReviews 21:23:43 tdasilva: IIRC there are some follow up patches e.g. doc changes 21:23:51 here's a brief summary of the earlier meeting 21:24:00 tdasilva: true for acoles 21:24:03 #link http://eavesdrop.openstack.org/meetings/swift/2017/swift.2017-05-31-07.00.html 21:24:25 the next 0700 meeting will be on june 14. acoles will chair 21:24:32 tdasilva: longer term there are some EC GET path optimisations we may want to make 21:24:43 to do with duplicate frags 21:25:30 tdasilva: and, making CLI binary for composite ring builder may be needed too 21:25:33 there are lots of EC reconstructor optimizations we "may want to make" (regardless of "global ec" or not) ;-) 21:25:34 * acoles apparently has first nick by alphabetical order 21:26:23 ok, for meeting transition follow-ups... 21:26:32 acoles, kota_ thanks! 21:26:40 mattoliverau is going to look at https://review.openstack.org/#/c/289664/ and roll in clayg's comments 21:26:40 patch 289664 - swift - Make eventlet.tpool's thread count configurable in... 21:26:41 That's why I'm changing my nick to zzzmattoliverau :p 21:27:02 and we need to bring up https://review.openstack.org/#/c/371150/ in this meeting now 21:27:03 patch 371150 - swift - Return 404 on a GET if tombstone is newer 21:27:25 notmyname: i'm almost done sending another patchset for that 21:27:31 tdasilva: great! 21:27:56 tdasilva: that's awesome 21:27:57 yeah, the 0700 meeting was like "it hasn't been touched in a month, needs to get in, but everyone who's looked at it is asleep" ;-) 21:28:17 i'd like to add https://review.openstack.org/#/c/302494/ to the list of priority patches that haven't had movement in a month. the only change has been to break out https://review.openstack.org/#/c/464084/ as a separate patch since it's a bit of a change in how the replicator works 21:28:18 patch 302494 - swift - Sync metadata in 'rsync_then_merge' in db_replicator 21:28:20 patch 464084 - swift - Apply remote metadata in _handle_sync_response 21:29:03 adjusting meeting topic to reflect the converstation 21:29:13 #topic patches that need attention 21:29:40 notmyname: I'll be back at https://review.openstack.org/#/c/302494/ in this week 21:29:41 for https://review.openstack.org/#/c/302494/ we need other reviewers 21:29:41 patch 302494 - swift - Sync metadata in 'rsync_then_merge' in db_replicator 21:29:42 patch 302494 - swift - Sync metadata in 'rsync_then_merge' in db_replicator 21:29:48 kota_: thanks :-) 21:30:01 I'd love people to review https://review.openstack.org/#/c/365371/ 21:30:01 patch 365371 - swift - Add Preamble and Postamble to SLO and SegmentedIte... 21:30:26 hang on hang on. we've got several patches to discuss 21:30:34 sorry 21:30:37 :D 21:30:40 first patch 302494 then path 365371 21:30:41 https://review.openstack.org/#/c/302494/ - swift - Sync metadata in 'rsync_then_merge' in db_replicator 21:31:19 for https://review.openstack.org/#/c/302494/ it's been split out ( timburke: because it changes the behavior?) and needs reviews 21:31:20 patch 302494 - swift - Sync metadata in 'rsync_then_merge' in db_replicator 21:31:32 timburke: should we wait until we get reviews on the dependency from kota_? 21:31:33 notmyname: I think timburke and kota_ can tell us when that one is ready 21:31:45 timburke: may already know it's done? 21:32:05 so someone could race kota through the first sanity review - or sweep with +A if he's already commited? 21:32:26 IIRC it's one of those must fix things - the question is in how much follow will be needed once we grok the issue and land the damn thing 21:33:13 clayg: correct, https://review.openstack.org/#/c/302494/ is what we must fix and the other is optional improvement IIRC 21:33:14 patch 302494 - swift - Sync metadata in 'rsync_then_merge' in db_replicator 21:33:14 kota_: timburke: how does that sound to you? 21:33:21 keep it tight that's always my suggestion 21:34:02 probably, I will be able to give my +2 for the first one in near future 21:34:08 great 21:34:15 i'm rather certain the first one is good to go -- i've just meddled with it enough that i'm not sure i'm an unbiased reviewer. the follow-up is a bit more of an open question, but should be ready to merge if we want it 21:34:32 kk 21:34:33 on the second one, I'm not entierly sure if it's *good* thing for us 21:34:40 timburke: second set of eyes is good always - too easy to get code-blind 21:34:58 kota_: timburke: let's focus on landing the critical fix - follow up is... follow up 21:35:02 let's focus on the first one fir this week 21:35:04 yeah :-) 21:35:14 and clayg's thought about keeping it tight is why i split it into two changes 21:35:20 yup, will do at least this week. 21:35:22 timburke: is a hero! 21:35:24 ok, joeljwright's patch 365371 21:35:25 https://review.openstack.org/#/c/365371/ - swift - Add Preamble and Postamble to SLO and SegmentedIte... 21:35:50 this is new functionality to swift. offers some pretty interesting stuff (that joeljwright wants to use in his cluster) 21:36:05 timburke has been looking at it and torgomatic expressed some interest a while back 21:36:18 IIRC this enables being able to download tar files from swift that are dynamically composed 21:36:54 that's certainly my use case - avoiding storing lots of tiny objects that only make sense combined with actual data 21:37:33 but in this case you'll be storing the tiny files and then make a tarfile manifest to download them all at once? 21:37:56 all the tar specific data would be base64 encodeed and stored in the manifest 21:38:13 notmyname: I think joeljwright was contrarsting with storing the pre-amble and post-amble *as* objects - then preamble postamble slo is just... a slo 21:38:20 as pre/postambles to the objects 21:38:24 oh, I see. instead of making individual object of the tar metadata, you dynamically build it 21:38:27 yeah 21:38:44 me totally wants this - currently doing something similar for backups... 21:38:57 cschwede_: does that mean you could review it? 21:38:59 :-) 21:39:02 sure! 21:39:02 :) 21:39:05 nice! 21:39:05 thanks! 21:39:16 joeljwright: is there any write up on the API? 21:39:41 it's all in the SLO docstring atm 21:39:50 the use-case seems more like "instead of storing some SLO segments as objects (or a range of an object) I want them to be data written directly into the manifest" 21:40:14 * tdasilva wonders if gnocchi driver could use this 21:40:39 clayg: yes that's the use-case 21:40:50 joeljwright: maybe a script/snippet that says "here's three examples of different things (two useful one crayz) how you use this new api capability for awesomeness success and winning" 21:41:11 clayg: I'll have a think 21:41:19 joeljwright: so is it *pre/post*amble - or just add inline object support to SLO segments 21:41:36 cschwede_: when you look at it, please keep an eye out for what would be good docs :-) 21:41:37 timburke already created a gist to make tarballs :) Now I just need other examples! 21:41:42 oh cool 21:41:45 notmyname: timburke: do we just recheck to make the doc build refresh? 21:41:46 https://gist.github.com/tipabu/5c96175d26fac1ec1771b5d01c6573a7 21:41:53 clayg: ya 21:41:56 that would be nice to include in the tree then 21:42:13 notmyname: the code snippet? 21:42:23 notmyname: i believe joeljwright's intention is to write a new middleware to do exactly that :-) 21:42:24 oh, I said that before I clicked the link ;-) 21:42:31 I modified timburke's example to use the python tarfile lib: https://gist.github.com/joel-wright/b8282f689ff1a622c035cd2741e8e265 21:42:35 notmyname: maybe in the doc/prose - but it could also just go in a gist referenced in the review 21:42:49 no, I don't want to link to timburke's gists in the swift source tree 21:42:52 well aren't *we* fancy :P 21:42:59 :D 21:43:24 ok, we've got a plan on this patch for this week 21:43:30 notmyname: you do enough linking to my gists as it is 21:43:36 a couple of others I wanted to bring up... 21:43:44 first, https://review.openstack.org/#/c/435152/ 21:43:45 patch 435152 - swift - Do not sync suffixes when remote rejects reconstru... (MERGED) 21:44:18 oh, every segment needs it's own pre-amble post amble 21:44:18 I did some backport releases this week, and I noticed that although we backported this patch and have now (or soon) released it, we haven't actually released this on master. it's post-2.14.0 21:44:20 interesting... 21:44:35 which raises the question of tagging another swift release 21:44:49 which is totally fine. or not. what do you think? 21:44:52 I had imagined [{'preabmle'}, {'object1'}, {'object2'}, {'postamble'}] 21:45:02 seems that if it's important enough to backport, it's important enough to cut a release for 21:45:13 but I'd like input from everyone else 21:45:24 what are your thoughts? 21:45:33 clayg: yeah, i had a similar thought... in particular, i was reminded of things like data uris... 21:45:46 but I could also imagine an api [{'blah1'}, {'object1'}, {'blah2'}, {'blah3'}, {'object2'}, {'blah4'}] 21:46:16 certainly in the context of a tlo, having it in the object dict makes perfect sense 21:46:34 notmyname: I hadn't realised we backported that 21:46:57 https://review.openstack.org/#/q/Ia72c407247e4525ef071a1728750850807ae8231,n,z 21:47:00 yep 21:47:02 clayg: pre/postambles are optional and can be mixed in any combination for each object 21:47:16 notmyname: i thought that was a minor optimization - also not understanding the backport (unlike the *other* suffix hashing *bug* which was both terrible *and* inefficient) 21:48:00 joeljwright: but can I have a segment that is *just* a pre-post amble - like an "inline" segment instead of a segment that is referencing another object or a range of an object 21:48:18 clayg: acoles: ok. I'll let it sit for now 21:48:26 notmyname: yeah I think it's weird that got backported - so what's the question? it's done now right? 21:48:33 :shipit: who cares 21:48:33 yeah, it's done 21:48:40 notmyname: well we have a lot of new stuff recently landed so a release might make sense soon 21:48:40 Hmmmm, zero byte objects are allowed as segments if they have a pre/post amble 21:48:45 lol, you just described our backport policy ;-) 21:48:57 acoles: indeed, but I won't rush for it 21:49:11 ok, last patch on the agenda.. 21:49:13 https://review.openstack.org/#/c/468105/ 21:49:14 patch 468105 - swift - Require that known-bad EC schemes be deprecated 21:49:21 for clayg and timburke to bring up 21:49:36 timburke: is definitely the hero here 21:49:48 just the other day I logged into a lab cluster and noticed those messages being logged 21:50:11 going to be really rough when I upgrade to the version of swift that makes your proxies not start - but hey - at least I'll %^&*ing fix my lab clusters! 21:50:17 oic 21:50:18 * clayg so lazy 21:50:32 I definitely support preventing this known-bad config 21:50:56 sounds good to me 21:50:58 timburke: do you have, off-hand, the releases of when we said it was deprecated? 21:51:31 just ocata? 21:51:38 if i remember right, yeah 21:51:59 changelog has it there (2.13.0) 21:52:17 we backported to newton, but that doesn't mean people upgraded 21:52:45 ok, so we need to check the deprecation policy. but if we land it now (ie "Pike" series), I think we'll be ok. IIRC the deprecation is to keep it as a warning for one openstack cycle and remove it in the next 21:53:22 however, this is also a Big Deal, and there's very little reason to leave that particular issue lurking for operators 21:53:25 be nice if we had some cluster-assisted-policy-migration up in there 21:53:33 yeah, i'm cool with landing it whenever -- i just wanted to be aggressive since that config harms durability 21:54:16 well - it's worse if you have an old liberasurecode - at least now we 503 instead of corrupt data 21:54:19 "oh you lost all your data because you selected some parameters? we knew it was bad, but we still let you do that because we didn't warn people long enough" <<<--- a terrible terrible message and we shoudl feel bad if we do it 21:54:22 notmyname: timburke is there any value in a half way step i.e. force the bad policy to be deprecated in the policy config sense, so that it becomes read only? 21:55:01 oh i like that - instead of refusing to start just auto deprecate - noice 21:55:07 otherwise we go from warning to no data, right? 21:55:19 clayg: a plus, but how does that get handled in the reconstructor? do we go grab another frag, or just skip it and hope we have better luck later? 21:55:21 people be like "where'd my policy go" - and we be like "ftfy" 21:55:24 yeah, that is a good idea. timburke, thoughts? 21:55:33 timburke: yeah 21:56:00 ...how would we handle single-policy deployments? 21:56:30 timburke: read-only 21:56:46 notmyname: there is no such thing as a read-only policy 21:56:56 no, just no-new-containers 21:57:00 timburke: h, you mean you gotta have *one* policy not deprecated 21:57:10 can't deprecate if you only have one policy, right? 21:57:21 yeah, sorry, if you only have one and it gets deprecated, things don't work 21:57:29 will the proxy just not start? 21:57:40 which is the same as what's proposed now 21:57:45 effectively 21:57:53 I wonder if we can special case this though 21:58:06 tdasilva: yeah I think if you do the checks in the right order it could still say "This policy is bad and it's deprecated now; I can't start unless you have at least one not-deprecated policy" 21:58:26 still not *great* - but the whole idea is we're doing something - something mean - and the beatings will continue until your cluster configuration improves! 21:58:35 timburke: can you work through these questions in that patch this week? 21:58:37 also, i'm not sure we'd really be doing operators any favors by hiding the issue -- you need to deal with this, and one of the first steps to dealing with it is to have a good, non-deprecated policy to migrate to 21:59:28 oh! one more note from the 0700 meeting. now that composite rings has landed, cschwede_ is unblocked on part power increase and will be moving forward on it again! 21:59:30 timburke: +1 you need to make a good policy anyway to move on from this terrible place 21:59:31 (also worth remembering: even with the policy deprecated, you'll *still* have new data being written with the bad config) 21:59:57 bah. we're at full time for the meeting 21:59:58 womp womp 22:00:09 that's a whole *other problem that needs addressing 22:00:22 ok, I'm trusting that timburke will keep thinking on this and type up some awesome code for it 22:00:25 :-) 22:00:34 thank you all for coming to this meeting 22:00:42 +1 22:00:44 and for about half of you, it's your second swift meeting in 24 hours! 22:00:59 thank you for your commitment to swift 22:01:03 notmyname: it's great! two in a single day! 22:01:06 #endmeeting