21:00:34 <notmyname> #startmeeting swift
21:00:35 <openstack> Meeting started Wed Aug 30 21:00:34 2017 UTC and is due to finish in 60 minutes.  The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:36 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:38 <openstack> The meeting name has been set to 'swift'
21:00:40 <notmyname> who's here for the swift team meeting?
21:00:45 <mattoliverau> o/
21:00:52 <m_kazuhiro> o/
21:00:57 <kota_> hi
21:01:12 <acoles> hi
21:01:14 <timburke> o/
21:01:15 <mathiasb> o/
21:01:37 <notmyname> good morning, afternoon, and evening
21:01:38 <tdasilva> hi
21:02:29 <notmyname> hmm... are we missing anyone?
21:02:46 <joeljwright> hey everyone!
21:02:55 <timburke> sam, clay
21:03:04 <notmyname> clayg: ping
21:03:19 <timburke> well, clayg's lurking, anyway, but he hasn't said anything :-)
21:03:22 <clayg> is it that time already?
21:03:27 <timburke> yup!
21:03:30 <clayg> thanks for the ping
21:03:32 <torgomatic> it's always when you least expect it
21:03:36 <notmyname> all right, let's get going
21:03:41 <notmyname> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:03:44 <notmyname> agenda is there
21:03:58 <notmyname> several things to go over, but we won't do them in the order of that wiki page
21:04:07 <notmyname> (except for the first one)
21:04:15 <joeljwright> yay!
21:04:17 <notmyname> #topic SLO *ambles
21:04:21 <notmyname> joeljwright: what's up?
21:04:28 <notmyname> where are we and what needs to be done
21:04:41 <joeljwright> had some comments from torgomatic and timburke (thanks!)
21:04:46 <notmyname> patch 365371
21:04:47 <patchbot> https://review.openstack.org/#/c/365371/ - swift - Add Preamble and Postamble to SLO and SegmentedIte...
21:04:56 <joeljwright> I'm still chewing over tim's comments
21:05:11 <notmyname> ok
21:05:15 <joeljwright> and I've pushed up a (very) WIP tlo dependent patch
21:05:22 <notmyname> link it?
21:05:41 <joeljwright> https://review.openstack.org/499260
21:05:42 <patchbot> patch 499260 - swift - WIP: Add TLO middleware
21:05:48 <joeljwright> sorry, took me a while to find it
21:06:05 <joeljwright> this is meant to illustrate the use of the *ambles
21:06:15 <notmyname> joeljwright: is the TLO patch something you want to see land upstream or as an example of the ambles?
21:06:22 <joeljwright> it's a lot like SLO, just validates a different manifest and generates an SLO
21:06:27 <mattoliverau> #link https://review.openstack.org/499260
21:06:28 <patchbot> patch 499260 - swift - WIP: Add TLO middleware
21:06:42 <joeljwright> I'd be happy for either I think
21:06:45 <mattoliverau> Just for the minutes
21:06:56 <joeljwright> if this is something people are interested in I can follow it up
21:06:57 <notmyname> (note for non-English speakers... "ambles" in this case isn't actually a real word. we're just using it for the shortened preamble and postamble)
21:07:36 <joeljwright> if it's not generally interesting then it's standalone (just depends on patch 365371)
21:07:36 <patchbot> https://review.openstack.org/#/c/365371/ - swift - Add Preamble and Postamble to SLO and SegmentedIte...
21:08:21 <mattoliverau> I think the preambles is interesting, but also some concrete examples, so maybe something like tlos for me would help it merge, otherwise it's an interesting feature that users may not know how to use.
21:08:31 <notmyname> aside from further review on the first patch, is there more that you need from the community yet? or anything you're expecting to write for us?
21:08:35 <mattoliverau> *also need
21:08:53 <notmyname> mattoliverau: yeah. that. :-) I think that's what the TLO patch is for
21:08:53 <joeljwright> I guess I'll need to update the general docs
21:08:56 <mattoliverau> Noting I haven't looked at tlos yet
21:08:57 <notmyname> ok
21:09:24 <joeljwright> mattoliverau: I only pushed the tlo one this evening
21:09:30 <joeljwright> and it's not functional yet
21:09:33 <mattoliverau> Well good then :p
21:09:35 <joeljwright> it's more of a taster atm
21:10:04 <notmyname> joeljwright: I know there's been some extra time pressure to get the *ambles patch landed for use at sohonet (or start working on some alternative). what's your current time frame? (IMO the *ambles patch should be reasonable, modulo the timburke and torgomatic and other reviews)
21:10:51 <joeljwright> my time frame is ~3weeks actively working on it, but I will continue to follow it up afterwards if I need to
21:11:06 <notmyname> ok. so that's through the PTG
21:11:07 <notmyname> thanks
21:11:59 <notmyname> if anyone else is interested in this work, please take a look. it adds a very interesting (IMO) functionality to SLOs that makes SLOs efficiently usable for structured aggregations of other objects (eg tar files, zip files, maybe more)
21:12:31 <joeljwright> thanks for looking everyone!
21:12:43 <notmyname> any other questions here from anyone?
21:13:06 <notmyname> ok, moving on
21:13:20 <notmyname> #topic debug_logger interface for errors
21:13:25 <notmyname> timburke: this is your patch
21:13:31 <notmyname> patch 496535
21:13:32 <patchbot> https://review.openstack.org/#/c/496535/ - swift - Simplify testing for logging at error vs exception
21:13:33 <timburke> so it is
21:13:41 <notmyname> what's up? and what do we need to cover in this meeting?
21:14:52 <timburke> i'm not sure i like our logging adapter. feels like a bit much magic in https://github.com/openstack/swift/blob/master/swift/common/utils.py#L1735-L1759 and there's no guarantee that we'll actually get to use it
21:15:37 <timburke> further, it encourages us to *always* log at exception and punt to the adapter to decide whether to include a traceback or not, which breaks from the expectations set by stdlib logging
21:16:11 <timburke> ...which in turn has us logging tracebacks for unexceptional things like trouble contacting memcached
21:16:17 <clayg> @timburke well but that patch doesn't really address that issue...
21:16:46 <clayg> @timburke that patch injects some magic lines into the fake logger so tests have to assert on what kind of traceback they expect or not?
21:17:01 <timburke> nope. that all shook out because i proposed that patch (and all the parents that make us use logger.error/logger.exception per stdlib norms)
21:17:37 <clayg> timburke: the memcache thing is readly fixed - an alternative solution would have been figure out if it could use a swift adapted logger when available and logged the Connection/Timeout stuff at .exception level
21:17:56 <timburke> i hate logging, and i feel like i wasted some time shaving a yak that (maybe?) wasn't there. i don't even remember why this is on the agenda
21:18:07 <notmyname> timburke: heh, so what is it we need to discuss about this patch?
21:18:39 <clayg> i added it to the agenda because I wanted to know how folks think about timburke's idea of injecting extra lines into the fake specifically to deal with how we test logging exceptions...
21:18:49 <notmyname> ah ha! it's clayg!
21:19:21 <clayg> currently we just assertEqual(errors, ['unexpected error was found doing xyz:'])
21:20:00 <clayg> ... but we *could* assertEqual(errors, ['unexpected error was found doing xyz:', '<magic string inserted by fake to represent expected traceback for MyExplosionErrror>'])
21:20:26 <clayg> and I wasn't sure if that was a good idea - but timburke was confident enough to type it up and I felt obligated to form an opinion - but wanted help
21:20:49 <notmyname> got it
21:21:10 <clayg> we could ignore it because it's too hard to form an opinion - we ignore lots of stuff
21:21:11 <timburke> ...except who knows whether that represents reality *any better* because of the automatic traceback-silencing
21:21:35 <notmyname> so (1) does anyone have any immediate "whoa that's great" or "ewww yuk!" feedback? and (2) who can look at it in the next few days to provide more informed feedback
21:21:39 <clayg> timburke: that's a good point!
21:22:00 <acoles> so one outcome is that in future we get a heads up when making assertions about logs and think more about the log level?
21:22:21 <timburke> acoles: that was the goal, yes
21:22:38 <acoles> so long as your test checks all the log lines
21:22:44 <torgomatic> I sort of want to take all the special cases out of our logger's exception() method and have our logging work more like stdlib's logging. Everyone knows how stdlib logging works.
21:23:03 <clayg> torgomatic: I think timburke is on that same page!
21:23:06 <mattoliverau> So to test if there is a traceback.. is there ever a time we actually want a traceback to actually test for it. Other then when we find something unexpected, which by definition is unexpected. I mean do users actually like tracebacks in logs
21:23:19 <acoles> torgomatic: +1
21:23:51 <clayg> no one likes tracebacks in logs except developers - when something blew up for unexpected reasons I like to know why - I also like the process not to die/shutdown/start-over.
21:24:09 <torgomatic> mattoliverau: I might want to test for it sometimes if I'm testing a catchall exception handler, but that's the only one that comes to mind.
21:24:23 <clayg> I think the special casing in logging.exception was in response to mostly the fact that no-one likes tracebacks in lots
21:24:26 <clayg> *logs
21:25:08 <torgomatic> I like the patches that have been going by to stop logging tracebacks when we know how to handle the error.
21:25:17 <notmyname> so what are we going to do? ignore this patch? commit to review it?
21:25:18 <timburke> mattoliverau: there are a few places that we currently test for the traceback. the trouble is that you have to consciously go looking for it
21:25:36 <clayg> timburke: did we decide that currently LogAdapter.exception *does* get executed when using a debug_logger - or not?
21:26:20 <timburke> torgomatic: except i'm not sure they were actually getting logged before! there was exactly one that actually came up in logs i've seen (BadStatusLine when the proxy was talking to a backend server)
21:26:53 <clayg> torgomatic: so on one of those I found a case where moving from exception to error that *had* been hitting the special casing in exception caused the format to change sort of needlessly (we already weren't logging a traceback, now where weren't logging a traceback with slightly less automatic context but differently)
21:27:28 <clayg> seemed likely to annoy some grok patterns - which led to tim's discovery of LogAdapter.exception's weird special case behavior
21:27:50 <notmyname> we need to move on for the sake of time. what's the next thing to do here?
21:28:18 <timburke> mattoliverau: fwiw, those would look like https://github.com/openstack/swift/blob/master/test/unit/proxy/controllers/test_obj.py#L3653-L3654
21:28:31 <clayg> umm... that's why i put it on the agenda - next step wasn't clear for me?
21:28:53 <notmyname> :-)
21:29:11 <timburke> punt to PTG?
21:29:22 <timburke> or grill rledisez! rledisez cares about logs :-)
21:29:35 <rledisez> sorry, i'm very late :)
21:29:35 <kota_> lol
21:29:36 <notmyname> lol
21:29:49 <notmyname> rledisez: and very popular it seems :-)
21:30:09 <timburke> rledisez: (don't actually feel like you have to form an opinion on the spot :-)
21:30:23 <rledisez> sorry, i'll have to check the logs to have a opinion
21:30:32 <clayg> ok, so step 1 was raise awareness , we'll talk about it and timburke will do something sane-ish
21:30:48 <mattoliverau> Sign, phone won't goto the lines in question when following the link. I'll follow up looking today at at least add my 2 cents to the review for what's that worth in the review.
21:30:59 <notmyname> clayg: ok
21:31:00 <notmyname> mattoliverau: thanks
21:31:13 <acoles> thanks for raising awareness, I was not fully aware of these quirks before
21:31:18 <notmyname> #topic adding domain_remap functional test
21:31:26 <notmyname> acoles: did you add this item, or did clayg? ;-)
21:31:34 <notmyname> #link https://review.openstack.org/#/c/435929/7
21:31:35 <patchbot> patch 435929 - swift - Functionnal tests for domain_remap middleware
21:31:51 <acoles> notmyname: I did :) and rledisez has arrived just in time
21:31:55 <mattoliverau> Lol
21:31:55 <notmyname> yay
21:32:07 <notmyname> ok, so what do we need to cover here in the meeting?
21:32:13 <clayg> apchacpahcacpahp romain wrote it, timburke and acoles reviewed it and it's not mergeD!?a;lksdjf
21:32:25 <acoles> I wanted to get wider opinion on a couple of things with this patch
21:32:31 <timburke> well, it's also gonna need a devstack change before it can merge
21:32:34 <timburke> #link https://review.openstack.org/#/c/494014/
21:32:35 <patchbot> patch 494014 - openstack-dev/devstack - Allow both Keystone and Tempauth reseller prefixes
21:32:54 <clayg> it doesn't change any behavior?
21:32:59 <timburke> even *if* you hit +A right now
21:33:02 <notmyname> timburke: gate seems happy with it now
21:33:12 <timburke> notmyname: Depends-On
21:33:22 <notmyname> ah
21:33:26 <acoles> first is, I thought in the past maybe we tried to make functional tests pass against any reasonable cluster - but IDK if I imagined that? with these tests there is an assumption about the config of the cluster
21:34:06 <notmyname> AIUI, functional tests should pass against anything calling itself a swift endpoint. and stuff that isn't default should be skipped based on /info results
21:34:11 <acoles> i.e. storage_domain, which is not published in /info, so the tests will fail unless you have example.com
21:34:37 <clayg> acoles: yeah that's wrong - why is there some many + on it?
21:34:42 <acoles> notmyname: right. so the tests do skip id domain_remap is not in the pipeline
21:34:54 <clayg> oh, that's not *terrible* then
21:34:56 <timburke> is there any reason *not* to publish storage_domain in /info?
21:35:04 <acoles> but we do not publish storage_domain
21:35:06 <clayg> timburke: good question!
21:35:09 <clayg> LAME
21:35:11 <notmyname> rledisez: what do you think?
21:35:12 <clayg> maybe
21:35:19 <acoles> timburke: that's a great question
21:35:21 <notmyname> rledisez: any reason to hide it?
21:35:38 <rledisez> actually, we run this test against our prod clusters
21:35:52 <rledisez> except if it's not the exact same code? i should probably check it
21:36:05 <notmyname> ...and it fails because you don't host on example.com? ;-)
21:36:21 <timburke> hey, you could always list it anyway :-)
21:36:35 <timburke> we accept *multiple* domains now, it's great!
21:36:38 <rledisez> and they succeed (or we don't prod…). so, i guess i have slighty different code on our branch. i'll check it right now
21:36:43 <notmyname> ok
21:37:04 <notmyname> my initial reaction is that there is no reason to hide the storage domain from /info
21:37:21 <notmyname> acoles: what's your second "wider opinion" thing?
21:37:57 <clayg> acoles: if you can demonstrate: given a swift cluster X with domain remap that *works* - these tests fail - that sounds like a test bug (or at minimum a test configuration issue)
21:38:15 <clayg> notmyname: I could imagine exampel.com being in some list for "reasons" and not wanting to advertise it :\
21:38:19 <acoles> notmyname: second thing was about adding domain_remap to the sample proxy pipeline.
21:38:40 <rledisez> acoles, notmyname: i run a different version of this test on our prod cluster, without hardcoded example.com. don't know why i submited this. i'll try to update that
21:38:44 <clayg> notmyname: I think a test.conf thing could also work, or discoverability could be optional - in which case the tests either have to skip or be configurable again
21:39:08 <clayg> rledisez: AWESOME!
21:39:13 <clayg> rledisez: is awesome
21:39:14 <notmyname> clayg: adding storage_domain seems better :-)
21:39:33 <notmyname> acoles: why add it or not add it to the example pipeline?
21:39:53 <acoles> clayg: yeah adding to test.conf seemed like one option but if people are happy to publish storage_domain then that works
21:39:54 <clayg> no... I meant like if I'm "myawesome.org" but I used to be "somecompanyibought.com" I might want to not break the internet and also not advertise the old name?
21:40:07 <clayg> or typos...
21:40:14 <acoles> notmyname: what if anything governs what goes in the sample pipeline? it seems to be used as the basis for the install guide
21:40:19 <timburke> fwiw, adding it was an easy way to have devstack be sure to have it enabled
21:40:34 <clayg> you said "I can't think of I why I don't want to publish.." - I can imagine "reasons" - but idk
21:40:44 <rledisez> there is no need to advertise it, as you can get it from the token (keystone catalog or that tempauth stuff i don't know about)
21:41:18 <rledisez> this is how our version of the test is written for now
21:41:20 <clayg> um... I don't think domain_remap should be in the sample pipeline right now
21:41:29 <notmyname> clayg: I agree
21:41:36 <clayg> acoles: why do you think it should be?  IME most people don't use it
21:41:40 <clayg> I always forget how it works
21:42:11 <acoles> clayg: I don't, but the patch puts it there, and I wasn't sure how to feel about that
21:42:21 <notmyname> the sample pipeline is governed by us. we can do whatever we want there. I think other groups (devstack?) take it as a signal because they don't know the current details of what might be configured
21:42:24 <clayg> wat!?  why?  because of devstack?
21:42:39 <rledisez> clayg: as a number, 30% to 40% of our trafic on public cloud is about domain_remap
21:43:07 <notmyname> if we need special dispensation about getting a config change into devstack that is different than the sample config, I'm happy to fight that for anyone
21:43:09 <acoles> I think the in process func test setup is also driven from the sample conf - but I guess that's ok cos the new tests would just skip
21:43:50 <timburke> clayg: weren't *you* the one that suggested putting it in sample configs? "I think most of the gate tests use the *other* example config? gate job logs seem to maybe confirm?"
21:44:00 <kota_> why not doing like in-process func tests like encryption?
21:44:07 <acoles> oh but IIRC jrichli made it possible to have custom in process pipeline for encryption, so we could have a custom domain_remap in process func test
21:44:09 <notmyname> kota_: that's a good idea
21:44:17 <acoles> kota_: yeah, that! :)
21:45:03 <notmyname> acoles: do you have enough based on this conversation to know what to do in gerrit (/cc rledisez)?
21:45:20 <clayg> timburke: past me has proven to be the biggest idiot I've ever known - don't trust anything he says.
21:45:43 <joeljwright> clayg: I'm sorry, but past me is a way bigger idiot
21:46:05 <acoles> notmyname: yes, thanks everyone. rledisez sorry but I will -1 until we resolve discovering the storage_domain, and I will look at how we could customize an in proc func test job
21:46:13 <rledisez> kota_: i'm not aware of that in-process func test. can i still be able to run the func test on a real cluster?
21:46:37 <acoles> rledisez: no, in process tests setup their own swift services
21:46:55 <notmyname> #topic PTG prep
21:47:04 <notmyname> ok, last topic for today (14 minutes left)
21:47:09 <notmyname> #link https://etherpad.openstack.org/p/swift-ptg-queens
21:47:30 <notmyname> in that etherpad, we've got the currently collected topics of interest
21:47:35 <notmyname> that's great, and thanks for updating it
21:47:44 <notmyname> we've got 1.5 weeks before the PTG
21:48:06 <notmyname> and it would help everyone if we can actually prepare instead of catching up from zero in denver
21:48:25 <notmyname> for example, let's talk about container sharding in denver, but read the current patch before we get there
21:48:28 <notmyname> that's what I mean
21:48:40 <notmyname> so i've added a new line to each topic: "what to review before the PTG"
21:48:49 <timburke> that's just the patch i was thinking i ought to load into my head :-)
21:48:53 <notmyname> some of the obvious-to-me ones I've already filled in
21:48:57 <notmyname> (like contaienr sharding)
21:49:12 <notmyname> but some of the others I don't know, and I need help
21:49:28 <mattoliverau> To answer you question, yes it would help :)
21:49:46 <mattoliverau> Also gives me reading for the long flight ;)
21:49:49 <notmyname> line 76. "suffix hashing protocol". acoles, rledisez, clayg, kota_?
21:49:57 <notmyname> what should we review for that one?
21:50:53 <notmyname> is there something on https://wiki.openstack.org/wiki/Swift/ideas or an existing patch?
21:51:20 <tdasilva> notmyname: I think clayg wanted us to take a look at protbuf
21:51:21 <notmyname> acoles: EC rebalance topic is empty
21:51:24 <clayg> yeah, i'll add the wiki link
21:51:40 <clayg> i'll try to sketch out some patches and add them to the etherpad
21:51:42 <notmyname> tdasilva: do you have any info for the HML topic? or the gnocci driver?
21:51:46 <notmyname> clayg: great!
21:51:48 <rledisez> maybe starting a dedicated etherpad? i know what i would like for this, but i don't know what are clayg plans
21:52:13 <notmyname> kota_: "ring rebalance strategy work" is a topic you added. anything needed more than the linked patch and bug?
21:52:33 <tdasilva> notmyname: no, I need to ping hseipp for HLM and nothing for gnocchi atm
21:52:43 <acoles> notmyname: ?"EC rebalance topic is empty"
21:52:56 <notmyname> m_kazuhiro: the auto tiering topic is on the etherpad. what can we review before the PTG?
21:53:16 <mattoliverau> rledisez: feel free to add both some wiki link and and etherpad.. let's just get the information so it's easy to find :)
21:53:17 <notmyname> acoles: sorry. the "what to review before the PTG". is there more than the linked patch?
21:54:12 <m_kazuhiro> notmyname: There are some patches. I'll add links to the etherpad page.
21:54:21 <notmyname> m_kazuhiro: thank you
21:54:28 <kota_> notmyname: they are different topics between "ring rebalance strategy" and "ec rebalance"
21:55:10 <notmyname> kota_: yes, there are 2 different topics listed on the etherpad
21:55:12 <kota_> the first is for *general* rebalancing issue, that link is for description WHY current rebalance is worse
21:55:14 <acoles> notmyname: OIC. I think clayg had an etherpad going on reconstructor improvements
21:55:47 <clayg> no i did... oh you mean this one -> https://etherpad.openstack.org/p/swift-rebalance
21:56:01 <notmyname> kota_: if there's anything else that would be helpful to think about before we all get to denver, please link to it on the etherpad
21:56:01 <kota_> but I'm feeling the solution for the first one is not straight forward so that I'd like to gather opinions on that at PTG
21:56:25 <clayg> yeah that stuff is >6mo old but still relevent despite the awesome improvements to EC rebalance using mattoliverau's great idea for multi-process strategy
21:56:27 <kota_> and the next one "ec rebalance" is for improvemnt on Reconstructor work for ec
21:56:43 <acoles> clayg: ack I added that
21:56:44 <kota_> notmyname: ok
21:57:25 <notmyname> tdasilva: ok, thanks (HML, gnocci)
21:58:06 <clayg> kota_: yeah your rebalance issue is totally different than the other rebalance topic(s) - was that already ack'd
21:58:08 <notmyname> I think having some prep work for topics listed will be helpful in denver. thanks for filling them out
21:58:51 <kota_> the mattoliverau's link "https://etherpad.openstack.org/p/swift-rebalance" looks like for Suffix Hashing protocol
21:59:28 <kota_> probably, we could use "rebalance" word everywhere in different topics :/
21:59:44 <notmyname> kota_: heh. every problem is a rebalance problem ;-)
22:00:08 <notmyname> we're pretty much at full time for the meeting.
22:00:13 <kota_> it's critical issue to discuss for us ;-)
22:00:13 <acoles> notmyname: do we have zaitcev's PUT POST patch on the list? I believe he will be in Denver
22:00:26 <notmyname> acoles: I added it under the golang topic
22:00:38 <notmyname> please continue to add helpful links to the PTG topics
22:00:52 <notmyname> thank you for your work on swift
22:01:05 <notmyname> thanks for coming today
22:01:07 <acoles> notmyname: ok great
22:01:09 <notmyname> #endmeeting