21:00:33 <clayg> #startmeeting swift
21:00:33 <openstack> Meeting started Wed Apr 19 21:00:33 2017 UTC and is due to finish in 60 minutes.  The chair is clayg. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:34 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:36 <openstack> The meeting name has been set to 'swift'
21:00:38 <clayg> who's here for the swift meeting?
21:00:43 <jungleboyj> o/
21:00:43 <joeljwright> o/
21:00:46 <cschwede> o/
21:00:50 <m_kazuhiro> o/
21:00:52 <rledisez> o/
21:00:52 <mathiasb> o/
21:01:00 <pdardeau_> hi
21:01:00 <kota_> hello
21:01:08 <clayg> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:01:08 <jrichli> o/
21:01:09 <mattoliverau> o/
21:01:23 <kota_> oh, clayg chiars the meeting today
21:01:45 <acoles> hi
21:01:53 <clayg> Great thanks everyone - notmyname isn't here today - there's a few things on he agenda - let's do our best
21:02:12 <clayg> #topic swift 2.14 - let's fix some bugs
21:02:38 <clayg> I tried to grab all of the priority bugs that I'm aware of being worked on from lp and group them here -> https://wiki.openstack.org/wiki/Swift/PriorityReviews
21:02:51 <clayg> I think a few things that people were already looking at and working on have landed - great work!
21:03:06 <clayg> I'd like to know what I'm missing?
21:03:13 <clayg> or if anything on there isn't at the right priority
21:03:50 <clayg> it's just a wiki page - I think really anyone can edit it?
21:03:54 <rledisez> if that’s ok i would like to add https://bugs.launchpad.net/swift/+bug/1683689 as i feel it’s at least a « high » bug
21:03:55 <openstack> Launchpad bug 1683689 in OpenStack Object Storage (swift) "ssync fails to replicate an object that had x-delete-at removed" [Undecided,New]
21:04:01 <timburke> i just did not long ago :-)
21:04:46 <rledisez> there is still work to do on it, i’ll look at it tomorrow. (thx acoles for your work on the unit test)
21:04:48 <clayg> rledisez: I agree that's high! and thank you for working on it.  I think we're still trying to work through the details
21:04:56 <acoles> rledisez: I looked at that today, agree it's high
21:05:01 <clayg> yes, thanks acoles, and rledisez for updating the bug
21:05:37 <clayg> if we know more it might be better/worse than it looks at first - I'll put it and the related change under the HIGH section for now
21:05:51 <clayg> who else is working on some bugs?  Or even just know of one they feel guilt about *not* working on?
21:06:38 <acoles> I put together a fix for EC gets leaking Connection headers
21:06:51 <acoles> oh I see that's on the wiki under medium. great
21:07:01 <clayg> acoles: that's on the list under medium - I think it's a bothersome bug
21:07:27 <clayg> there's no relative priority currently with-in the sections - if stack ranking is something that would help folks I can put my opinion behind it
21:08:07 <clayg> I think the bug severity is sort of objective - I think between one medium bug and the next it just depends on personal priorties - but I'm happy to lend my opinon only if folks think that's useful
21:08:25 <clayg> I think it's ok just to leave medium un ordered and let folks spend there time as they can where they think it's most useful
21:08:36 <mattoliverau> nah its fine, critical are critical and high are high, no point in ordering them.. hopefully there wil never be too many :)
21:08:42 <clayg> if you want to side channel me that something is misprioritized I'd be happy to take that too
21:08:48 <clayg> cool!
21:09:09 <clayg> ummm... also I guess... just if you think of somethign else lets try to get it on there
21:09:32 <clayg> now that the criticals are closed we could cut a release at any moment - notmyname might start that when he gets back
21:09:35 <clayg> next topic?
21:10:13 <clayg> #topic follow-up from last week
21:10:31 <clayg> I think it's just https://review.openstack.org/#/c/448480/
21:10:40 <clayg> which is related to a follow up for a critical/high fix we just landed
21:10:57 <kota_> yup
21:11:08 <clayg> I personally tend to push pretty hard to keep critical bug fixes tight - helps reviews grok thing quickly, find them to be "obviously correct", validate the fix and *backport*
21:11:20 <clayg> but all to often our really bad bugs are in code that is a mess
21:11:34 <kota_> clayg: that patch looks still labeled as WIP?
21:11:37 <clayg> in this case Pavel really wants to follow through and clean up some of the other mess (thank you so much!)
21:11:42 <kota_> patch 448480
21:11:57 <clayg> I don't have a great answer on how to do better, but generally the idea is "after a critical fix try to make some follow through"
21:12:18 <clayg> kota_: afaik he needs some help with tests
21:12:28 <clayg> he might be working on them, or he might be blocked, not sure
21:12:29 <kota_> clayg: oic
21:12:57 <acoles> I only looked at the overview but it seems to address several things - can it be broken up or os that just making work?
21:13:32 <clayg> acoles: great question - it's less work for the *reviewer* to keep changes tight?  I think different authors have different thresholds for how much it "costs" to split things up
21:13:48 <clayg> how we can web accommodating and encouraging
21:14:09 <acoles> I guess I need to review it to know!
21:14:55 <clayg> ok, great - if anyone else has some followup patches that got lost try and track them down and let me know
21:15:18 <clayg> if we have a systemic failure of "make this change do only the minimum fix; and do the rest in the followup" - but then we don't *land* the followup
21:15:21 <clayg> ... could be an issue!
21:15:24 <clayg> next topic?
21:15:41 <acoles> I think we're almost done with the follow ups from EC fragment duplication :P
21:15:56 <clayg> acoles: I think we recently landed some follow from the get_hashes fixes ;P
21:16:10 <acoles> yep, that too :)
21:16:11 <kota_> acoles: :D
21:16:15 <clayg> maybe long tail of the cleanup is just ... long
21:16:18 <clayg> #topic Swiftclient patch to use generic Keystone client
21:16:21 <clayg> joeljwright: you about?
21:16:27 <joeljwright> yeah, I'm here
21:16:34 * clayg has no idea what this is - didn't even read it
21:16:36 <joeljwright> been updating the bullets as this evolved this week
21:16:49 <clayg> i hope timburke and joeljwright will tell us what we need to know/do
21:17:02 <joeljwright> basically it started as a patch to use the generic keystone client rather than version specific ones
21:17:31 <joeljwright> mattoliverau and cschwede gave it +2's but it broke me
21:17:43 <joeljwright> when I started digging I noticed some subtle behavioural changes
21:17:46 <mattoliverau> oh yeah thats right
21:17:48 <joeljwright> so it all got stuck
21:17:59 <cschwede> hmm
21:18:21 <joeljwright> jaosorior has proposed a different approach after talking to they keystone peeps
21:18:27 <joeljwright> but it needs a fair bit of work
21:18:37 <joeljwright> https://review.openstack.org/#/c/456205
21:18:41 <clayg> strangely I sort of do think I knew some of this...
21:18:51 <clayg> maybe I did read something
21:19:16 <joeljwright> the big question really is - we probably need to do this at some point, so how do we handle the possible breakages and subtle behavioural issues
21:19:17 <clayg> mattoliverau: cschwede: can you confirm what joeljwright is saying?  did the original patch need to be dropped?
21:19:17 <joeljwright> ?
21:19:21 <timburke> i think the behavioral changes are mostly OK, though? like, keystoneclient is going to be smart enough to *use a version that seems like it should work* even if it isn't the version you explicitly requested
21:19:50 <cschwede> clayg: i need to have a closer look again first
21:19:53 <joeljwright> timburke: yes, but it's also a **lot** more picky about keystone configuration issues
21:20:00 <clayg> mattoliverau: cschwede: is there any chance we could make python-swiftclient undeniably better even if it has some "subtle behavioural changes" in some cases (not *great* granted, but it's relative yeah?)
21:20:29 <clayg> joeljwright: "how do we handle the possible breakages and subtle behavioural issues
21:20:32 <clayg> "
21:20:36 <joeljwright> I was really worried about client breakage when the user cannot force a keystone reconfig (and it works atm)
21:20:46 <timburke> and with keystone-on-devstack moving to uwsgi (https://review.openstack.org/#/c/456344/ and how it broke both client and server gates)... being able to just use the auth_uri like in https://review.openstack.org/#/c/456791/ seems nice?
21:20:54 <clayg> ^ sounds like a good question?  do we have any example of handling that in the past?  or does swiftclient always work the same for everyone forever?
21:21:00 <timburke> joeljwright: yeah... true
21:21:26 <mattoliverau> it worked in my basic keystone setup, but seems in others its flaky. Like cschwede I'd need to relook. But also lets take a look at the new approach, if its got keystone by in, it might be better
21:21:28 <joeljwright> at the very minimum it seems like major version upgrade and explicit "we changed this" docs
21:21:53 <clayg> joeljwright: sounds like your saying the value of "doing it the right way" is less than the cost of "breaking someone" - i tend to agree
21:21:54 <joeljwright> mattoliverau: +1
21:22:18 <joeljwright> clayg: yes, I want to do it right, but without pissing everyone off :)
21:22:35 <timburke> joeljwright: did you ever get your keystone properly configured? i.e., do we at least have an idea of what to tell people if they're having troubles?
21:22:45 <clayg> joeljwright: I like major version upgrade and start doing way better - esp if it really helps our maintance burden and the cost for the app developer to upgrade is offset with new velocity in making things awesomesause in the future!?
21:23:03 <joeljwright> I got some information from jaosorior and I know what the issue appears to be
21:23:29 <joeljwright> the thing is I can force a fix on mine
21:23:36 <clayg> timburke: that ++ is there a work around for a swiftclient user that upgrades swiftclient (to a major version upgrade) and finds the new version doesn't work with their weird keystone config "out of the box"
21:23:57 <clayg> joeljwright: but you're saying it *always* requries a reconfig of the *keystone*
21:24:05 <joeljwright> clayg: no
21:24:09 <clayg> phew ;)
21:24:10 <timburke> fwiw, we might be forced into a major version bump if/when we land https://review.openstack.org/#/c/448978/ and have our list of supported pythons better reflect what we're actually testing...
21:24:13 <joeljwright> just when you've done it wrong and don't notice atm
21:24:49 <clayg> timburke: seems orthogonal - but if we start seriously talking about a release that bumps a major version we *should* try to find/remember stuff like that
21:25:16 <clayg> joeljwright: but you an change your swiftclient config/options and somehow make it talk to your keystone again - even if it's not seemlessly backwards compatible?
21:25:28 <joeljwright> okay, seems like we have an approach, now we just need to look at how much work is required on https://review.openstack.org/#/c/456205 to 'do it right'
21:25:44 <timburke> clayg: agreed; i just wanted to point out that a major version bump is probably coming *anyway* -- they forced our hand on it last time for exactly this reason
21:25:48 <joeljwright> I'm not sure we can force the old behaviour
21:25:52 <joeljwright> but we can see...
21:26:16 <clayg> joeljwright: can you clarify (perhaps just for me)
21:26:44 <clayg> joeljwright: is there a keystone configuration that you think might be *incompatible* with the new (possibly major) version of swiftclient?
21:27:00 <clayg> or just that the swiftclient user has to update some config/option/kwarg to make it work
21:27:15 <acoles> joeljwright: I was wondering that too - right now the CLI requires explicit flag to use keystone v3. AFAICT that would go away and you get whatever API version is discovered. But could a flag force back to a specific API version?
21:27:28 <clayg> ... and that's only assuming it didn't work seemlessly for them in the first place - which in some cases at least it would?
21:27:31 <joeljwright> clayg: I don't think we can force the old behaviour with any args after switching to the generic keystone client
21:28:01 <joeljwright> it work work or not work, the only solution would be 'use an older version until your keystone works'
21:28:14 <clayg> *gotcha* - that sounds pretty rough
21:28:20 <joeljwright> acoles: I really need to look into that
21:28:32 <timburke> acoles: doesn't require, as i recall -- should be on-by-default if you provide a domain
21:28:59 <joeljwright> clayg acoles: maybe there *are* forcing options, but we need to work that out
21:29:11 <clayg> ok, i'll leave a follow up for next week
21:29:21 <joeljwright> clayg: agreed
21:29:23 <clayg> if we don't know the answer (hopefully the author can help)
21:29:34 <joeljwright> yeah, he seems helpful
21:29:35 <acoles> timburke: ok I was going by the doc examples but I can believe we quietly allow domain to force to v3
21:29:36 <clayg> then we'll make a call to either hold off or get some more folks to start asking questions
21:29:36 <joeljwright> just busy
21:29:57 * clayg would not be surprised if acoles figures everything out as always
21:30:00 <clayg> next topic?
21:30:06 <timburke> joeljwright: i think our method of forcing will basically be, "go instantiate a version-specific plugin" :-/
21:30:15 <acoles> clayg: do not hold breath
21:30:21 <clayg> umm... I think that's it?
21:30:29 <joeljwright> yup
21:30:30 <clayg> i'm freaking good at meetings
21:30:37 <jungleboyj> One quick thing.
21:30:38 <clayg> #topic open discussion
21:30:52 <clayg> jungleboyj: has the floor
21:30:58 <jungleboyj> clayg:  Thank you.
21:31:44 <jungleboyj> The Upstream Institute is looking for representatives from each of the different projects to be available at the training for people who are interested in/curious about Swift.
21:31:59 <jungleboyj> Anyone going to be in Boston on Sunday who is willing to help mentor people?
21:32:24 <clayg> jungleboyj: Sunday... the 7th of March?  or like *this* Sunday?
21:32:37 <clayg> derp May
21:32:41 <jungleboyj> clayg:  Sunday the 7th of May
21:32:48 <clayg> i think i get in late
21:32:54 <acoles> jungleboyj: sorry, get in later
21:33:25 <clayg> jungleboyj: could you send more details to the ML with [swift] - maybe we can move flights?  everyone here should be on the lookout.
21:33:48 <clayg> jungleboyj: do you have limit on the number of swift devs that you could use help from?  Do you have a deadline to get a minimum number of volunteers?
21:33:50 <jungleboyj> clayg:  Sure I can do that.
21:34:19 <jungleboyj> In the past I have covered storage in general.
21:34:29 <timburke> get in around 6 :-/ otherwise i'd be happy to -- i'm looking forward to being around for the project-onboarding stuff wednesday
21:34:35 <jungleboyj> I will also find out what kind of interest participants have indicated in Swift.
21:35:00 <jungleboyj> Will send a note to the ML with details and we can figure out the priority.
21:35:03 <clayg> gotcha, well I'd definately be interested in stopping by even if i'm late, and might try to wiggle my flights - it sounds like ahoot
21:35:39 <clayg> jungleboyj: thanks for raising awareness about that!  look forward to anymore details.
21:35:42 <jungleboyj> clayg:  It really is.  It is a good opportunity to get to meet people interested in learning about OpenStack and develop interest.
21:35:44 <kota_> jungleboyj: can i get the time for that, I may be there at the morning or noon
21:36:13 <kota_> whole day? or afternoon or evening...
21:36:16 <jungleboyj> https://docs.openstack.org/upstream-training/
21:36:30 <clayg> jungleboyj: thanks, sounds like there is definately some intrest - I'll ask to follow up in the next meeting too if anyone has follow up questions that don't get answered int he ML thread
21:36:42 <clayg> @here doesn't anyone else have some open topics they need to bring up?
21:36:49 <mattoliverau> On that note, I wont be at Forum.. no travel for me, even asked (very late) for travel assistance.. no dice :(
21:37:01 <kota_> jungleboyj: thx, it looks whole day
21:37:07 <jungleboyj> kota_: Would be Sunday afternoon.
21:37:09 <clayg> mattoliverau: :\
21:37:43 <jungleboyj> kota_ The more focused discussions will be in the afternoon.
21:38:10 <kota_> jungleboyj: ok, I'll check my schedule in detail later
21:38:10 <jungleboyj> If we know we have someone coming we can hold questions on Swift until someone is there.
21:38:19 <jungleboyj> kota_:  Thanks
21:39:21 <clayg> oh, I have a question - did anyone look at storyboard?
21:39:24 <rledisez> just a quick question about https://etherpad.openstack.org/p/swift-object-server-tests
21:39:25 <rledisez> we’re getting dangerously close to the point we need them, so is that ok if we start writing tests like proxy is tested right now? or do you have some other ideas before we write some code
21:40:53 <mattoliverau> clayg: not yet.. was there ever a response to notmyname's ML email?
21:41:22 <jungleboyj> clayg:  We have a bit on it in the education and I know there is going to be sessions on it in Boston.
21:41:28 <clayg> mattoliverau: not unless they sent it directly to him
21:41:39 <acoles> rledisez: when you say "like the proxy" do you mean like functional tests, but targeted at object server?
21:41:52 <rledisez> acoles: yes, exactly
21:42:48 <clayg> rledisez: I will try to put some ideas on the etherpad - you should definately start writing code if you see a blocker down the road
21:43:13 <mattoliverau> rledisez: if you start with potentially the tests that can be re-purposed, then start adding more, and your starting to need it.. then go for it. Sorry the etherpad hasn't gotten much love
21:43:45 <acoles> rledisez: InternalClient might be useful. we need to avoid re-implmenting proxy behaviour to generate realistic backend requests
21:44:26 <rledisez> acoles: right, good idea
21:46:46 <mattoliverau> for those not following along, the the etherpad is being updated while it seems we are paused here ;)
21:46:55 <clayg> hehhee
21:47:04 <clayg> i'll keep working on that after the meeting
21:47:07 <timburke> mattoliverau: thanks. i thought it got quiet :-)
21:47:36 <clayg> rledisez: thanks for raising awareness about that - it seems like pleanty of people are curious what you plan and will be able to accomplish on that topic
21:47:55 <clayg> hopefully we can translate that curiosity to contribution as you get closer to landing something
21:47:59 <clayg> keep bugging folks!
21:48:08 <clayg> anyone else have some other topics - we still have 10 mins
21:48:18 <m_kazuhiro> Umm... I want opinions or questions about tiering topic <http://forumtopics.openstack.org/cfp/details/37>. If you are interested in automated tiering, please visit the url and leave your opinions or questions..
21:48:35 <rledisez> thx for the feedback. we will propose something small at first and see how it goes
21:48:43 <acoles> rledisez: +1
21:49:31 <clayg> rledisez: IIRC in ATL you had some feedback for m_kazuhiro regarding tiering - the topic will be discussed more in Boston - I think you will be there?
21:50:08 <rledisez> clayg: m_kazuhiro yep, i’ll be there
21:50:14 <timburke> as i mentioned earlier, our gate got busted. fixed following https://review.openstack.org/#/c/457868/ but judging by the failures on https://review.openstack.org/#/c/456791/ the gate-swift-dsvm-functional-ubuntu-xenial job we've got for swiftclient doesn't actually use the in-review swiftclient :-(
21:50:51 <clarkb> timburke: I can help sort that out. We possibly need LIBS_FROM_GIT=python-swiftclient
21:50:56 <timburke> if anyone knows more about dsvm tests and how to actually test the in-review patch, help there would be much appreciated
21:51:01 <timburke> whooo!
21:51:05 <clarkb> ping me after meeting if you like and I can help out with making sure the change in review is what gets tested
21:51:09 <clayg> so swift functests use *released* swiftclient - does that sound so insane?  should they use *master*?
21:51:19 <acoles> timburke: thanks for your efforts to sort out the gate
21:51:45 <timburke> clayg: not insane for the swift gate. definitely insane for the swiftclient gate
21:51:50 <mattoliverau> +1
21:51:58 <clayg> timburke: sounds right - thanks for being awesome
21:52:06 <mattoliverau> and thanks clarkb for always being round when we need you :)
21:52:33 <clayg> mattoliverau: sounds right - also clarkb thanks for being awesome
21:52:47 <clayg> let's end this thing early?  I want to brag to notmyname!
21:53:06 <mattoliverau> clayg: nice work today, you rocked it!
21:53:12 <clayg> #endmeeting swift