15:00:16 <bswartz> #startmeeting manila
15:00:17 <openstack> Meeting started Thu Mar  2 15:00:16 2017 UTC and is due to finish in 60 minutes.  The chair is bswartz. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:18 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:20 <openstack> The meeting name has been set to 'manila'
15:00:27 <bswartz> hello all
15:00:32 <bswartz> welcome back from PTG
15:00:36 <tbarron> hi
15:00:45 <ganso> hello
15:01:06 <xyang2> hi
15:01:08 <bswartz> wow everyone's asleep this morning
15:01:22 <vponomaryov> Hello
15:01:40 <bswartz> vponomaryov cknight gouthamr toabctl markstur: courtesy ping
15:01:45 <tommylikehu> hi
15:01:54 <tommylikehu> sudo ping
15:02:00 <toabctl> hi
15:02:19 <bswartz> well we have a quorum at least
15:02:55 <bswartz> no announcements today
15:03:05 <bswartz> #agenda https://wiki.openstack.org/wiki/Manila/Meetings
15:03:07 <vkmc> o/
15:03:16 <vponomaryov> gates are broken, but keep calm and stay tuned ))
15:03:28 <bswartz> #topic Experimental Features
15:03:58 <bswartz> so one of the major topics from the PTG was that not everyone his happy with how experimental feature development has gone
15:04:33 <bswartz> we discussed some possible changes, but I wanted to hold off making any official decisions until we had an officially recorded meeting (like today)
15:04:50 <bswartz> the concerns fell into 2 areas mostly
15:05:20 <bswartz> (1) was the API mechanism for experimental features and (2) was the development process for experimental features
15:06:03 <bswartz> for (1) the proposal is to use a different rest endpoint for experimental APIs rather than using the current header flag to enable experimental APIs
15:06:40 <bswartz> Using a different rest endpoint has a lot of advantages, and other projects are already doing their experimental features that way
15:07:01 <bswartz> Plus nobody else (AFAIK) has adopted Manila's header mechanism
15:07:30 <bswartz> The downside to this proposal is that there would be a LOT of code change for no immediate benefit
15:08:09 <bswartz> so let me ask you all, does anyone want to drive this proposal during Pike?
15:08:45 <vponomaryov> bswartz: the one who raised it? )
15:08:57 <bswartz> gouthamr suggested this
15:09:19 <bswartz> he's here in the channel -- not sure if he's paying attention
15:09:25 <bswartz> I saw him in the hallway 10 minutes ago...
15:09:57 <bswartz> So here's the thing
15:10:10 <vponomaryov> breaking news, gates for manila project are fixed here: https://review.openstack.org/#/c/440531 all new commits and patch-sets better to set as dependency on this
15:10:15 <ganso> bswartz: I got the impression that gouthamr wanted to solve this problem with solution #2
15:10:41 <bswartz> My feeling is that if we create a new rest endpoint for experimental stuff and we stop using the header, then it's a major change to the manila API and would require us to move to v3
15:10:45 <vponomaryov> ganso; same
15:10:50 <bswartz> ganso: the 2 issues are completely orthagonal
15:11:00 <bswartz> we'll discuss (2) in a moment
15:11:32 <vponomaryov> bswartz: we can acceps this header further, just not having experimental APIs anymore
15:11:40 <vponomaryov> bswartz: having them as (2)
15:11:52 <bswartz> anyways, there are other things I don't like about our current microversion approach, andeven the API-WG doesn't like our current microversion approach
15:12:00 <vponomaryov> bswartz: I see it this way as gouthamr's proposal
15:12:02 <bswartz> so moving to a v3 API would allow us to fix more things
15:12:32 <gouthamr> hey bswartz: was on a call. reading scrollback
15:12:43 <bswartz> however all of that is a bunch of code for what amounts to API structural improvements but not actual value
15:13:35 <gouthamr> vponomaryov: +1
15:13:46 <bswartz> vponomaryov: how we expose experimental APIs in the code and how we developer them are completely unrelated issues
15:14:04 <bswartz> the only exception is if we stop doing them altogether
15:14:17 <bswartz> and I haven't heard anyone suggest that yet
15:14:33 <bswartz> s/developer/develop/
15:15:04 <bswartz> okay let's move on to the second issue since that what people seem to be most interested in
15:15:50 <bswartz> so our current approach is that you can merge anything that doesn't affect the API as long as it passes our normal processes of good code quality, doesn't break anything, has test coverage, etc
15:16:42 <bswartz> and you can merge changes that affect the API if you (a) microversion them and implement backwards comaptibility or (b) microversions, mark them experimental, and don't implement backwards compability
15:17:14 <bswartz> (b) microversion them, mark them experimental, and don't implement backwards compability
15:17:44 <bswartz> new features are implemented as a small number of large commits (ideally 1)
15:18:34 <bswartz> and we deal with uncertainty in the API design by using the experimental tag
15:19:44 <gouthamr> the problem is that a new experimental feature touches non-experimental/stable APIs
15:19:47 <bswartz> some have suggested that keep stuff we're uncertain about out of master, either by creating feature branches, or by simply not merging stuff until we're completely sure about the API design
15:20:05 <bswartz> gouthamr:  those are just a bunch of bugs IMO
15:20:08 <gouthamr> and whatever we say/do to convince ourselves, we break the API *always*
15:20:15 <gouthamr> yes.. that's the problem bswartz
15:20:28 <gouthamr> stable would mean we do all we can to ensure they're stable
15:20:30 <bswartz> gouthamr: we can do things to reduce the number of those kinds of bugs
15:20:32 <gouthamr> we live with bugs.
15:20:35 <gouthamr> :|
15:20:44 <bswartz> but that's a 3rd approach
15:21:00 <bswartz> if that's the only problem we have then let's stop talking about (1) and (2) and focus on that
15:21:31 <bswartz> so we have had examples of experimental changes modifying stable APIs
15:22:01 <bswartz> we have a rule that you MUST SUPRESS THOSE CHANGES unless the client sends the experimental tag
15:22:16 <bswartz> if people actually followed that rule I think we would be in okay shape
15:22:33 <bswartz> so maybe we need to raise awareness about this rule
15:22:45 <bswartz> or maybe we need more tempest tests that are able to catch that class of bugs
15:23:06 <bswartz> gouthamr: would that make you happy?
15:23:10 <gouthamr> no
15:23:29 <gouthamr> my proposal is to leave the stable API and evolve it slowly with microversions
15:23:55 <gouthamr> any new change to the API that comes with the premise "sorry can't maintain backwards compatibility" cannot live alongside manila's API
15:24:25 <gouthamr> so, i propose creating a new endpoint that people can chose to totally ignore in their clouds
15:24:36 <gouthamr> this is not RBAC, no experimental header
15:24:47 <bswartz> gouthamr: you're confusing the mechanism with actually enforcing the rules we already have
15:25:03 <gouthamr> this is a experimental API endpoint that you can keep breaking at whim
15:25:11 <bswartz> using a different mechanism, we will still face the same class of issues unless we get better about enfocing our rules
15:25:30 <gouthamr> not really
15:25:30 <gouthamr> tempest always tests the v2 endpoint
15:25:39 <gouthamr> our tempest tests*
15:25:51 <gouthamr> we will not test the v3 endpoint with the same tests/clients
15:26:05 * gouthamr oops i named them
15:26:22 <ganso> gouthamr: but isn't the code base the same? like, if a new feature, which has an experimental API, touches share/manager.py code, and breaks share creation path, which is a stable API, wouldn't it affect the regular endpoint?
15:26:51 <gouthamr> ganso: sure yes.. which was why the real way of doing this is feature branches
15:27:02 <bswartz> gouthamr: no matter whether we use our existing mechanism or a different rest endpoint, we're not going to have to share-list APIs
15:27:03 <gouthamr> ganso: but the argument against it is the huge tax on the developers
15:27:33 <bswartz> gouthamr: the experimental share list and the stable share list will use the same code, and that code will need if statements in it to see which flavor is being invoked
15:27:33 <gouthamr> ganso: so this is a compromise where we police the behavior properly with "stable API" tests
15:27:52 <bswartz> the problem is that people are forgetting those if tests altogether
15:28:04 <bswartz> and we don't have tests that enforce good behavior
15:28:27 <gouthamr> bswartz: we do... we just change the tests when we change the stable APIs ...
15:28:39 <bswartz> gouthamr: then you got at the source of the problem
15:28:46 <bswartz> we need to make it hard or impossible to do that
15:29:02 <bswartz> this has nothing to do with adding a new endpoint
15:29:04 * gouthamr split tempest repo - make it branchless
15:29:06 * gouthamr runs
15:29:10 <vponomaryov> bswartz: split tests to separate repo )))
15:29:22 * bswartz sighs
15:29:32 <vponomaryov> gouthamr ^_^
15:30:42 * toabctl was driving home with the QA ptl and is now convinced that a extra tempest test repo might be a good thing
15:31:02 <gouthamr> toabctl: he got yoy
15:31:04 <gouthamr> you*
15:31:08 <toabctl> :)
15:31:23 <ganso> how so? what am I missing?
15:31:40 <tbarron> I'm not convinced that the tax on devs working on experimental features in an experimental branch outweighs the tax on all developers in the main branch of doing the work to make sure experimental features don't break core features.
15:32:14 <gouthamr> tbarron: i'm with you on that one... bswartz pointed out that it may discourage some developers from contributing
15:32:15 <vponomaryov> tbarron: just develop such one ))
15:33:10 <ganso> I think it can ultimately increase the risk, in the long run
15:33:10 <vponomaryov> tbarron: it will require architecture rework, not only simple rebases
15:33:30 <gouthamr> you may replace developers with vendor-driven-developers and make better sense of that
15:33:55 <ganso> we may end up developing in feature branches, and we *think* we are happy with it, we merge, and then we find out we aren't, and we don't have the experimental API excuse anymore
15:34:05 <gouthamr> ganso: we shouldn't
15:34:21 <gouthamr> ganso: when we develop in feature branches, i'm hoping we're testing well enough
15:34:34 <tbarron> or they don't merge
15:34:44 <toabctl> gouthamr, why are we testing more in feature branches?
15:35:14 <ganso> gouthamr: why don't we continue to do what we are doing, but increase effort in testing?
15:35:45 <vponomaryov> gouthamr: ok, remind API breakage caused by experimental APIs
15:36:05 <vponomaryov> gouthamr: that really worth such rework of approach for developing new APIs
15:36:07 <gouthamr> toabctl: regression tests
15:36:25 <gouthamr> ganso: how so? we don't have dedicated QA resources...
15:36:45 <ganso> gouthamr: and how would we have with feature branches?
15:36:45 <gouthamr> vponomaryov: i should start tweeting this stuff
15:36:59 <tbarron> currently we can have gate failures caused by experimental features that reduce our velocity on core feature work
15:37:22 <toabctl> true
15:37:27 <bswartz> ping
15:37:32 <vponomaryov> pong
15:37:45 <bswartz> sorry guys my connection dropped
15:37:46 <tommylikehu> bswartz is back
15:37:50 <gouthamr> ganso: testing has always been the responsibility of the developer or a group of developers in manila... only once in a while do we submit test-improvement patches..
15:37:56 <bswartz> it's all my fault for hacking on my network config right before the meeting
15:38:13 <gouthamr> bswartz supposedly typed 20 lines and was ignored :P
15:38:18 <bswartz> looks like I missed about 5 minutes of the meeting and you missed 5 minutes of my ranting
15:38:32 <tbarron> you should be doing that in a parallel experimental meeting
15:38:36 <tbarron> bswartz: ^^
15:38:45 <tommylikehu> tbarron:lol
15:38:53 * gouthamr *calls netapp firewall guys* job well done guys. *hangs up*
15:38:57 <ganso> tbarron: lol
15:39:00 <bswartz> okay I'll summarize my points so we can move on
15:39:21 <bswartz> it seems there's no strong case to be made for changing our experimental API mechanism or development process
15:39:25 <bswartz> this is all about testing and how we enforce the rule we already have
15:39:43 <bswartz> let's fix the bugs we already have and work on preventing new bugs from this class
15:40:00 <gouthamr> ?
15:40:04 <tbarron> well, you missed my remarks about the tax that doing all this stuff imposes on the development and hardening of our core features
15:40:15 <bswartz> the first priority is to fix the existing bugs
15:40:38 <tbarron> but we can take that up in another meeting after we've all read and digested backlog, myself included
15:40:44 <bswartz> the second priority is to protect against the next bug of this type with better testing
15:40:46 <vponomaryov> which ones from endless army of them?
15:41:04 <bswartz> vponomaryov: at least *filing* the bugs and compiling a list would be helpful
15:41:31 <gouthamr> we've been doing that.. and the point of bringing this up is that this is taxing...
15:41:48 <bswartz> I see a lot of complaining and suggesting large changes but nobody is doing the simple work of correcting the existing issues
15:42:03 <vponomaryov> why nobody?
15:42:05 <gouthamr> if we don't change our stance, we're only going deeper into it and causing pain to the end users
15:42:21 <tommylikehu> I am trying to help
15:42:31 <bswartz> gouthamr: I thing the opposite is true -- I won't even consider a major change until our house it in order
15:42:36 <bswartz> is in order
15:42:47 <gouthamr> bswartz: which has been my point
15:43:07 <gouthamr> bswartz: no more experimental features.... come with a baked feature and let's add it
15:43:09 <bswartz> when your house gets really dirty do you move to a new house or do you clean the house your in?
15:43:37 <gouthamr> i understand the desire not to impact migration, replication, share groups
15:43:49 <gouthamr> let's drop the "experimental" tag and evolve these APIs
15:44:15 <ganso> gouthamr: no, fix all the bugs first
15:44:25 <bswartz> ganso: +1
15:44:40 * markstur sneaks in late
15:44:50 <gouthamr> ganso: i'm not interested in fixing bugs caused by not invoking this header
15:44:54 <gouthamr> i don't consider that a bug
15:45:04 <bswartz> gouthamr: these are huge bugs
15:45:09 <gouthamr> i consider such a bugfix that breaks end users
15:45:30 <gouthamr> and is not ever ever supposed to be done
15:45:52 <bswartz> you don't have to fix them but I'm not willing to discuss any changes to how experimental APIs work until our existing APIs follow the rules we laid out when we added experimental APIs
15:46:04 <gouthamr> i don't think these APIs can get any better or change "drastically" without us being able to maintain backwards compatibility
15:46:21 <gouthamr> let's take the pain on doing that rather than shove in more "experimental" features
15:46:28 <ganso> gouthamr: the root cause is testing, if the bugs are in stable APIs, we fix them, even if the bugfix breaks the experimental ones, ok, then we fix the experimental ones, with proper testing, to avoid breaking the stable APIs again... for those that are already in-tree. For new experimental features, they should come packed with lots of tests
15:46:30 <bswartz> people are criticizing the mechanism because we have problems, but the mechanism isn't the cause of the problems, it's people doing the wrong thing that's the cause of the problems
15:47:12 <bswartz> once we start doing the right thing, we can have a production discussion about what the mechanism should be
15:47:17 <bswartz> productive
15:47:25 * bswartz can't type
15:47:42 <vponomaryov> agree with bswartz and ganso
15:47:49 <tommylikehu_> +1
15:47:57 <bswartz> okay I'm going to skip over other topics on the agenda and move to the other critical one
15:48:01 <bswartz> #topic Specs Deadline
15:48:41 <bswartz> in Ocata we agreed to have a low priority spec deadline at milestone-1 and a high priority spec deadline at milestone-2
15:49:02 <bswartz> I propose moving the high priority spec deadline to milestone-1 + 2 weeks
15:49:38 <bswartz> because pike-2 is planned to be 8 weeks long, this would move the deadline 6 weeks earlier
15:49:54 <bswartz> but still give slightly more time to merge high priority specs than low priority specs
15:50:01 <vponomaryov> bswartz: are you sure that 2 weeks will be enough?
15:50:10 <bswartz> it's a modest change compared to some other proposals from the summit
15:50:19 <xyang2> bswartz: can you publish the Manila deadline on the Pike schedule wiki
15:50:32 <tommylikehu_> bswartz: how about m-1+4
15:50:38 <bswartz> xyang2: after we agree to them, yes
15:50:42 <ganso> bswartz: -1 tommylikehu_ +1
15:50:44 <xyang2> ok
15:50:59 <ganso> bswartz: 4 weeks is best I think as well
15:51:00 <bswartz> at the PTG some suggested making both deadlines pike-1
15:51:09 <tbarron> milestone 1 is Apr 10-14, just fyi
15:51:12 <bswartz> or some suggested making the high priority deadline *before* the low priority one
15:51:20 <bswartz> #link https://releases.openstack.org/pike/schedule.html
15:51:39 <tbarron> ty
15:51:59 <xyang2> pike-2 is June 5
15:52:08 <bswartz> so I don't like the idea of having specs still floating around 11 weeks into the cycle
15:52:38 <bswartz> ocata showed that with focus we can agree on stuff pretty quickly and say no to the rest
15:52:56 <bswartz> the whole point here is to get people to stop focusing on low priority stuff pretty early on
15:52:59 <tbarron> +1
15:53:49 <bswartz> so there are 2 basic issues
15:53:57 <ganso> bswartz: we could do it on R-17, right after the summit, some people will be travelling so it will be hard to get agreement if they're unavailable
15:54:11 <bswartz> should the high/low prioritty deadlines be the same or should should one come before the other (and in what order)
15:54:20 <bswartz> and then the question of exactly which week that should be
15:54:41 <xyang2> R-17 is the week before the summit
15:54:43 <bswartz> I liked how things went in ocata, with low priority specs out of the way first, then we switched to high priority ones
15:54:49 <ganso> xyang2: yes, sorry, right before the summit
15:55:04 <ganso> bswartz: I suggest R-17, right before the summit
15:55:06 <xyang2> ganso: people may be traveling already
15:55:22 <bswartz> and remember we're getting a much earlier start on pike than we did in past releases, where the summit was 3 or 4 weeks into the first milestone
15:55:26 <tbarron> i'd kinda like to see how the same process we used in ocata works with a normal lengthy cycle
15:55:35 <tbarron> length
15:55:45 <bswartz> tbarron: -1 because milestone-2 is insanely far away
15:55:54 <bswartz> okay how about this
15:56:01 <tbarron> point taken
15:56:08 <bswartz> high priority deadline Pike-1
15:56:19 <bswartz> low priority deadline even earlier
15:56:27 <tommylikehu_> -1
15:56:29 <ganso> -1
15:56:35 <tbarron> :D
15:56:55 <bswartz> okay what if we make all the specs have the same deadline?
15:57:04 <bswartz> that was proposed in Atlanta
15:57:19 <ganso> bswartz: wouldn't it bring back the problem of not focusing on hi-pri?
15:57:41 <bswartz> ganso: if there's one deadline then high priority would actually mean "review it first" unlike what happened in ocata
15:58:01 <bswartz> people pointed out how crazy it was to save the high priority stuff for after the low priority stuff
15:58:38 <tbarron> it's crazy if we don't start reviewing them until after the low priiority specs clear
15:58:42 <bswartz> come on guys, Pike-1 is 6 weeks away
15:58:48 <tbarron> which is i guess human nature
15:58:51 <vponomaryov> also we should decide on date when we mark specs with "high" and "low" priority tags
15:58:54 <bswartz> does anyone seriously want to spend more than 6 weeks writing and reviewing specs for Pike?
15:58:58 <tommylikehu_> what we learned from ocata is that some of you guys don't have enough time on your own specs.
15:59:37 <xyang2> +1 on having the same deadline to simply things
15:59:52 <bswartz> there's not enough time to hold a vote
16:00:08 <jungleboyj> Argh!  Is the meeting actually at 9 am?
16:00:18 <bswartz> I'm in favor of an earlier deadline
16:00:21 <markstur> early is better, but I think there is a need to allow late hi-prio things when needed (hopefully none)
16:00:30 <xyang2> jungleboyj: ? manila meeting?
16:00:32 <bswartz> since we have to leave this room, anyone who wants a later deadline please contact me in the channel and make your case
16:00:36 <jungleboyj> xyang2:  Yes.
16:00:40 <xyang2> jungleboyj: it starts at 10am EST
16:00:55 <bswartz> #endmeeting