21:00:33 #startmeeting swift 21:00:33 Meeting started Wed Apr 19 21:00:33 2017 UTC and is due to finish in 60 minutes. The chair is clayg. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:34 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:36 The meeting name has been set to 'swift' 21:00:38 who's here for the swift meeting? 21:00:43 o/ 21:00:43 o/ 21:00:46 o/ 21:00:50 o/ 21:00:52 o/ 21:00:52 o/ 21:01:00 hi 21:01:00 hello 21:01:08 #link https://wiki.openstack.org/wiki/Meetings/Swift 21:01:08 o/ 21:01:09 o/ 21:01:23 oh, clayg chiars the meeting today 21:01:45 hi 21:01:53 Great thanks everyone - notmyname isn't here today - there's a few things on he agenda - let's do our best 21:02:12 #topic swift 2.14 - let's fix some bugs 21:02:38 I tried to grab all of the priority bugs that I'm aware of being worked on from lp and group them here -> https://wiki.openstack.org/wiki/Swift/PriorityReviews 21:02:51 I think a few things that people were already looking at and working on have landed - great work! 21:03:06 I'd like to know what I'm missing? 21:03:13 or if anything on there isn't at the right priority 21:03:50 it's just a wiki page - I think really anyone can edit it? 21:03:54 if that’s ok i would like to add https://bugs.launchpad.net/swift/+bug/1683689 as i feel it’s at least a « high » bug 21:03:55 Launchpad bug 1683689 in OpenStack Object Storage (swift) "ssync fails to replicate an object that had x-delete-at removed" [Undecided,New] 21:04:01 i just did not long ago :-) 21:04:46 there is still work to do on it, i’ll look at it tomorrow. (thx acoles for your work on the unit test) 21:04:48 rledisez: I agree that's high! and thank you for working on it. I think we're still trying to work through the details 21:04:56 rledisez: I looked at that today, agree it's high 21:05:01 yes, thanks acoles, and rledisez for updating the bug 21:05:37 if we know more it might be better/worse than it looks at first - I'll put it and the related change under the HIGH section for now 21:05:51 who else is working on some bugs? Or even just know of one they feel guilt about *not* working on? 21:06:38 I put together a fix for EC gets leaking Connection headers 21:06:51 oh I see that's on the wiki under medium. great 21:07:01 acoles: that's on the list under medium - I think it's a bothersome bug 21:07:27 there's no relative priority currently with-in the sections - if stack ranking is something that would help folks I can put my opinion behind it 21:08:07 I think the bug severity is sort of objective - I think between one medium bug and the next it just depends on personal priorties - but I'm happy to lend my opinon only if folks think that's useful 21:08:25 I think it's ok just to leave medium un ordered and let folks spend there time as they can where they think it's most useful 21:08:36 nah its fine, critical are critical and high are high, no point in ordering them.. hopefully there wil never be too many :) 21:08:42 if you want to side channel me that something is misprioritized I'd be happy to take that too 21:08:48 cool! 21:09:09 ummm... also I guess... just if you think of somethign else lets try to get it on there 21:09:32 now that the criticals are closed we could cut a release at any moment - notmyname might start that when he gets back 21:09:35 next topic? 21:10:13 #topic follow-up from last week 21:10:31 I think it's just https://review.openstack.org/#/c/448480/ 21:10:40 which is related to a follow up for a critical/high fix we just landed 21:10:57 yup 21:11:08 I personally tend to push pretty hard to keep critical bug fixes tight - helps reviews grok thing quickly, find them to be "obviously correct", validate the fix and *backport* 21:11:20 but all to often our really bad bugs are in code that is a mess 21:11:34 clayg: that patch looks still labeled as WIP? 21:11:37 in this case Pavel really wants to follow through and clean up some of the other mess (thank you so much!) 21:11:42 patch 448480 21:11:57 I don't have a great answer on how to do better, but generally the idea is "after a critical fix try to make some follow through" 21:12:18 kota_: afaik he needs some help with tests 21:12:28 he might be working on them, or he might be blocked, not sure 21:12:29 clayg: oic 21:12:57 I only looked at the overview but it seems to address several things - can it be broken up or os that just making work? 21:13:32 acoles: great question - it's less work for the *reviewer* to keep changes tight? I think different authors have different thresholds for how much it "costs" to split things up 21:13:48 how we can web accommodating and encouraging 21:14:09 I guess I need to review it to know! 21:14:55 ok, great - if anyone else has some followup patches that got lost try and track them down and let me know 21:15:18 if we have a systemic failure of "make this change do only the minimum fix; and do the rest in the followup" - but then we don't *land* the followup 21:15:21 ... could be an issue! 21:15:24 next topic? 21:15:41 I think we're almost done with the follow ups from EC fragment duplication :P 21:15:56 acoles: I think we recently landed some follow from the get_hashes fixes ;P 21:16:10 yep, that too :) 21:16:11 acoles: :D 21:16:15 maybe long tail of the cleanup is just ... long 21:16:18 #topic Swiftclient patch to use generic Keystone client 21:16:21 joeljwright: you about? 21:16:27 yeah, I'm here 21:16:34 * clayg has no idea what this is - didn't even read it 21:16:36 been updating the bullets as this evolved this week 21:16:49 i hope timburke and joeljwright will tell us what we need to know/do 21:17:02 basically it started as a patch to use the generic keystone client rather than version specific ones 21:17:31 mattoliverau and cschwede gave it +2's but it broke me 21:17:43 when I started digging I noticed some subtle behavioural changes 21:17:46 oh yeah thats right 21:17:48 so it all got stuck 21:17:59 hmm 21:18:21 jaosorior has proposed a different approach after talking to they keystone peeps 21:18:27 but it needs a fair bit of work 21:18:37 https://review.openstack.org/#/c/456205 21:18:41 strangely I sort of do think I knew some of this... 21:18:51 maybe I did read something 21:19:16 the big question really is - we probably need to do this at some point, so how do we handle the possible breakages and subtle behavioural issues 21:19:17 mattoliverau: cschwede: can you confirm what joeljwright is saying? did the original patch need to be dropped? 21:19:17 ? 21:19:21 i think the behavioral changes are mostly OK, though? like, keystoneclient is going to be smart enough to *use a version that seems like it should work* even if it isn't the version you explicitly requested 21:19:50 clayg: i need to have a closer look again first 21:19:53 timburke: yes, but it's also a **lot** more picky about keystone configuration issues 21:20:00 mattoliverau: cschwede: is there any chance we could make python-swiftclient undeniably better even if it has some "subtle behavioural changes" in some cases (not *great* granted, but it's relative yeah?) 21:20:29 joeljwright: "how do we handle the possible breakages and subtle behavioural issues 21:20:32 " 21:20:36 I was really worried about client breakage when the user cannot force a keystone reconfig (and it works atm) 21:20:46 and with keystone-on-devstack moving to uwsgi (https://review.openstack.org/#/c/456344/ and how it broke both client and server gates)... being able to just use the auth_uri like in https://review.openstack.org/#/c/456791/ seems nice? 21:20:54 ^ sounds like a good question? do we have any example of handling that in the past? or does swiftclient always work the same for everyone forever? 21:21:00 joeljwright: yeah... true 21:21:26 it worked in my basic keystone setup, but seems in others its flaky. Like cschwede I'd need to relook. But also lets take a look at the new approach, if its got keystone by in, it might be better 21:21:28 at the very minimum it seems like major version upgrade and explicit "we changed this" docs 21:21:53 joeljwright: sounds like your saying the value of "doing it the right way" is less than the cost of "breaking someone" - i tend to agree 21:21:54 mattoliverau: +1 21:22:18 clayg: yes, I want to do it right, but without pissing everyone off :) 21:22:35 joeljwright: did you ever get your keystone properly configured? i.e., do we at least have an idea of what to tell people if they're having troubles? 21:22:45 joeljwright: I like major version upgrade and start doing way better - esp if it really helps our maintance burden and the cost for the app developer to upgrade is offset with new velocity in making things awesomesause in the future!? 21:23:03 I got some information from jaosorior and I know what the issue appears to be 21:23:29 the thing is I can force a fix on mine 21:23:36 timburke: that ++ is there a work around for a swiftclient user that upgrades swiftclient (to a major version upgrade) and finds the new version doesn't work with their weird keystone config "out of the box" 21:23:57 joeljwright: but you're saying it *always* requries a reconfig of the *keystone* 21:24:05 clayg: no 21:24:09 phew ;) 21:24:10 fwiw, we might be forced into a major version bump if/when we land https://review.openstack.org/#/c/448978/ and have our list of supported pythons better reflect what we're actually testing... 21:24:13 just when you've done it wrong and don't notice atm 21:24:49 timburke: seems orthogonal - but if we start seriously talking about a release that bumps a major version we *should* try to find/remember stuff like that 21:25:16 joeljwright: but you an change your swiftclient config/options and somehow make it talk to your keystone again - even if it's not seemlessly backwards compatible? 21:25:28 okay, seems like we have an approach, now we just need to look at how much work is required on https://review.openstack.org/#/c/456205 to 'do it right' 21:25:44 clayg: agreed; i just wanted to point out that a major version bump is probably coming *anyway* -- they forced our hand on it last time for exactly this reason 21:25:48 I'm not sure we can force the old behaviour 21:25:52 but we can see... 21:26:16 joeljwright: can you clarify (perhaps just for me) 21:26:44 joeljwright: is there a keystone configuration that you think might be *incompatible* with the new (possibly major) version of swiftclient? 21:27:00 or just that the swiftclient user has to update some config/option/kwarg to make it work 21:27:15 joeljwright: I was wondering that too - right now the CLI requires explicit flag to use keystone v3. AFAICT that would go away and you get whatever API version is discovered. But could a flag force back to a specific API version? 21:27:28 ... and that's only assuming it didn't work seemlessly for them in the first place - which in some cases at least it would? 21:27:31 clayg: I don't think we can force the old behaviour with any args after switching to the generic keystone client 21:28:01 it work work or not work, the only solution would be 'use an older version until your keystone works' 21:28:14 *gotcha* - that sounds pretty rough 21:28:20 acoles: I really need to look into that 21:28:32 acoles: doesn't require, as i recall -- should be on-by-default if you provide a domain 21:28:59 clayg acoles: maybe there *are* forcing options, but we need to work that out 21:29:11 ok, i'll leave a follow up for next week 21:29:21 clayg: agreed 21:29:23 if we don't know the answer (hopefully the author can help) 21:29:34 yeah, he seems helpful 21:29:35 timburke: ok I was going by the doc examples but I can believe we quietly allow domain to force to v3 21:29:36 then we'll make a call to either hold off or get some more folks to start asking questions 21:29:36 just busy 21:29:57 * clayg would not be surprised if acoles figures everything out as always 21:30:00 next topic? 21:30:06 joeljwright: i think our method of forcing will basically be, "go instantiate a version-specific plugin" :-/ 21:30:15 clayg: do not hold breath 21:30:21 umm... I think that's it? 21:30:29 yup 21:30:30 i'm freaking good at meetings 21:30:37 One quick thing. 21:30:38 #topic open discussion 21:30:52 jungleboyj: has the floor 21:30:58 clayg: Thank you. 21:31:44 The Upstream Institute is looking for representatives from each of the different projects to be available at the training for people who are interested in/curious about Swift. 21:31:59 Anyone going to be in Boston on Sunday who is willing to help mentor people? 21:32:24 jungleboyj: Sunday... the 7th of March? or like *this* Sunday? 21:32:37 derp May 21:32:41 clayg: Sunday the 7th of May 21:32:48 i think i get in late 21:32:54 jungleboyj: sorry, get in later 21:33:25 jungleboyj: could you send more details to the ML with [swift] - maybe we can move flights? everyone here should be on the lookout. 21:33:48 jungleboyj: do you have limit on the number of swift devs that you could use help from? Do you have a deadline to get a minimum number of volunteers? 21:33:50 clayg: Sure I can do that. 21:34:19 In the past I have covered storage in general. 21:34:29 get in around 6 :-/ otherwise i'd be happy to -- i'm looking forward to being around for the project-onboarding stuff wednesday 21:34:35 I will also find out what kind of interest participants have indicated in Swift. 21:35:00 Will send a note to the ML with details and we can figure out the priority. 21:35:03 gotcha, well I'd definately be interested in stopping by even if i'm late, and might try to wiggle my flights - it sounds like ahoot 21:35:39 jungleboyj: thanks for raising awareness about that! look forward to anymore details. 21:35:42 clayg: It really is. It is a good opportunity to get to meet people interested in learning about OpenStack and develop interest. 21:35:44 jungleboyj: can i get the time for that, I may be there at the morning or noon 21:36:13 whole day? or afternoon or evening... 21:36:16 https://docs.openstack.org/upstream-training/ 21:36:30 jungleboyj: thanks, sounds like there is definately some intrest - I'll ask to follow up in the next meeting too if anyone has follow up questions that don't get answered int he ML thread 21:36:42 @here doesn't anyone else have some open topics they need to bring up? 21:36:49 On that note, I wont be at Forum.. no travel for me, even asked (very late) for travel assistance.. no dice :( 21:37:01 jungleboyj: thx, it looks whole day 21:37:07 kota_: Would be Sunday afternoon. 21:37:09 mattoliverau: :\ 21:37:43 kota_ The more focused discussions will be in the afternoon. 21:38:10 jungleboyj: ok, I'll check my schedule in detail later 21:38:10 If we know we have someone coming we can hold questions on Swift until someone is there. 21:38:19 kota_: Thanks 21:39:21 oh, I have a question - did anyone look at storyboard? 21:39:24 just a quick question about https://etherpad.openstack.org/p/swift-object-server-tests 21:39:25 we’re getting dangerously close to the point we need them, so is that ok if we start writing tests like proxy is tested right now? or do you have some other ideas before we write some code 21:40:53 clayg: not yet.. was there ever a response to notmyname's ML email? 21:41:22 clayg: We have a bit on it in the education and I know there is going to be sessions on it in Boston. 21:41:28 mattoliverau: not unless they sent it directly to him 21:41:39 rledisez: when you say "like the proxy" do you mean like functional tests, but targeted at object server? 21:41:52 acoles: yes, exactly 21:42:48 rledisez: I will try to put some ideas on the etherpad - you should definately start writing code if you see a blocker down the road 21:43:13 rledisez: if you start with potentially the tests that can be re-purposed, then start adding more, and your starting to need it.. then go for it. Sorry the etherpad hasn't gotten much love 21:43:45 rledisez: InternalClient might be useful. we need to avoid re-implmenting proxy behaviour to generate realistic backend requests 21:44:26 acoles: right, good idea 21:46:46 for those not following along, the the etherpad is being updated while it seems we are paused here ;) 21:46:55 hehhee 21:47:04 i'll keep working on that after the meeting 21:47:07 mattoliverau: thanks. i thought it got quiet :-) 21:47:36 rledisez: thanks for raising awareness about that - it seems like pleanty of people are curious what you plan and will be able to accomplish on that topic 21:47:55 hopefully we can translate that curiosity to contribution as you get closer to landing something 21:47:59 keep bugging folks! 21:48:08 anyone else have some other topics - we still have 10 mins 21:48:18 Umm... I want opinions or questions about tiering topic . If you are interested in automated tiering, please visit the url and leave your opinions or questions.. 21:48:35 thx for the feedback. we will propose something small at first and see how it goes 21:48:43 rledisez: +1 21:49:31 rledisez: IIRC in ATL you had some feedback for m_kazuhiro regarding tiering - the topic will be discussed more in Boston - I think you will be there? 21:50:08 clayg: m_kazuhiro yep, i’ll be there 21:50:14 as i mentioned earlier, our gate got busted. fixed following https://review.openstack.org/#/c/457868/ but judging by the failures on https://review.openstack.org/#/c/456791/ the gate-swift-dsvm-functional-ubuntu-xenial job we've got for swiftclient doesn't actually use the in-review swiftclient :-( 21:50:51 timburke: I can help sort that out. We possibly need LIBS_FROM_GIT=python-swiftclient 21:50:56 if anyone knows more about dsvm tests and how to actually test the in-review patch, help there would be much appreciated 21:51:01 whooo! 21:51:05 ping me after meeting if you like and I can help out with making sure the change in review is what gets tested 21:51:09 so swift functests use *released* swiftclient - does that sound so insane? should they use *master*? 21:51:19 timburke: thanks for your efforts to sort out the gate 21:51:45 clayg: not insane for the swift gate. definitely insane for the swiftclient gate 21:51:50 +1 21:51:58 timburke: sounds right - thanks for being awesome 21:52:06 and thanks clarkb for always being round when we need you :) 21:52:33 mattoliverau: sounds right - also clarkb thanks for being awesome 21:52:47 let's end this thing early? I want to brag to notmyname! 21:53:06 clayg: nice work today, you rocked it! 21:53:12 #endmeeting swift