22:00:14 <mtreinish> #startmeeting qa
22:00:14 <openstack> Meeting started Thu Apr  3 22:00:14 2014 UTC and is due to finish in 60 minutes.  The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot.
22:00:15 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
22:00:18 <openstack> The meeting name has been set to 'qa'
22:00:26 <mtreinish> hi who do we have here today?
22:00:28 <sdague> o/
22:00:32 <maurosr> o/
22:00:34 <masayukig> hi
22:00:51 <zhikunliu> o/
22:00:52 <mtreinish> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting
22:00:58 <mtreinish> ^^^ Today's agenda
22:01:05 <dkranz> o/
22:01:42 <mtreinish> ok lets get started
22:01:57 <mtreinish> #topic Summit sessions (mtreinish)
22:02:00 <cyeoh> o/
22:02:16 <mtreinish> so I started an etherpad to track design summit proposals:
22:02:20 <mtreinish> #link https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
22:02:42 <mtreinish> I thought it would be good to do what we did last cycle and have a discussion in the meeting about the sessions
22:03:02 <sdague> what else has been submitted into the tool?
22:03:03 <mtreinish> so if people had ideas for a summit session if you could put them in the etherpad
22:03:24 <mtreinish> sdague: right now there are only 2
22:03:36 <sdague> probably worth transposing them to the etherpad
22:03:45 <mtreinish> "A case for project level functional testing" and "Rally & Tempest integration (future steps)"
22:04:00 <sdague> the first one sounds like an overlap of the one dkranz put in
22:04:05 <dkranz> sdague: It is
22:04:16 <dkranz> sdague: I didn't know about the others
22:04:18 <dkranz> sdague: My bad
22:04:23 <sdague> dkranz: no worries
22:04:55 <cyeoh> I guess we should talk about how we can reduce the code overlap between v2 and v3 tests, but I doubt we need whole session on that
22:04:57 <mtreinish> dkranz: http://summit.openstack.org/cfp/details/134
22:05:25 <sdague> cyeoh: well, a refactoring for maintainability session might be good
22:05:27 <sdague> include that
22:05:30 <mtreinish> cyeoh: yeah probably not, but it could be part of a larger session
22:05:38 <sdague> plus other items people have
22:05:58 <sdague> I still kind of want to do the nice skip decorators, but that's back burner
22:06:21 <mtreinish> so anyway if people have ideas if they can put them on the etherpad and we'll go through it in a couple of weeks during the meeting
22:06:30 <sdague> yeh, probably plan for 3 weeks out
22:06:38 <sdague> as decision time
22:06:43 * mtreinish needs to look at a calendar
22:06:58 <mtreinish> ok that's all I had to talk about on this for now
22:07:01 <mtreinish> so let's move on
22:07:10 <mtreinish> #topic  Blueprints
22:07:21 <mtreinish> so sdague why don't you start of on the specs stuff
22:07:25 <sdague> sure
22:07:28 <sdague> #link https://review.openstack.org/#/q/status:open+project:openstack/qa-specs,n,z
22:07:37 <sdague> there are the open specs we currently have
22:08:22 <sdague> currently all of qa-core has +2 rights, which I think is right
22:08:32 <sdague> though we should probably by convention let the PTL do +A
22:08:42 <sdague> just to make sure they are in on the vote
22:08:45 <sdague> or ack it
22:09:10 <sdague> I think right now we have one that's fully ready
22:09:16 <sdague> https://review.openstack.org/#/c/83264/
22:09:29 <sdague> and I went through the rest of them and provided some feedback on additions
22:09:34 <cyeoh> +1 for the +A policy
22:09:45 <sdague> anyone have objectsions to https://review.openstack.org/#/c/83264/  ?
22:09:58 <mtreinish> sdague: I hope not it's mostly implemented :)
22:10:05 <sdague> otherwise I'll land that one as our first one
22:10:17 <sdague> ok, done
22:10:18 <dkranz> sdague: No, but should this include an 'admin' tag or is that a different bp?
22:10:30 <mtreinish> dkranz: this is just the service tagging
22:10:39 <dkranz> mtreinish: ok
22:10:39 <mtreinish> like compute, image, etc...
22:10:50 <sdague> dkranz: I think admin clean is a different qa-spec
22:10:58 <sdague> can you do a draft on that?
22:11:00 <mtreinish> which all thats left on is going through and finding where we need to tag things
22:11:09 <sdague> I know you were very interested in that yesterday
22:11:39 <dkranz> sdague: Yes
22:12:03 <cyeoh> re: 83264 can you do a hacking check to see if someone does add a service name which does exist in the path
22:12:21 <sdague> #action dkranz to draft no admin qa-spec
22:12:23 <cyeoh> and flag an error if it occurs? Just to save inconsistencies creeping in accidentally
22:12:25 <mtreinish> cyeoh: was that not in there. I intend to it came up during the spec review
22:12:39 <cyeoh> mtreinish: cool!
22:12:41 <mtreinish> or I thought it did
22:12:59 <sdague> well, if not, we can always do an ammendment :)
22:13:14 <mtreinish> no it's in there L52
22:13:30 <mtreinish> it's just not a work item...
22:13:36 <sdague> cyeoh, dkranz: would you guys mind doing a run through the existing specs and providing comments as well
22:13:49 <sdague> I'd like to get us used to this model of making sure we have bases covered
22:13:55 <dkranz> sdague: Will do
22:14:00 <cyeoh> sdague: ok
22:14:01 <sdague> fortunately we've only got a few, so it's pretty quick
22:14:35 <mtreinish> I'm also going get the infra side setup soon
22:14:37 <sdague> #action sdague to reset all non high blueprints, point people at qa-specs
22:14:45 <mtreinish> so we'll publish the specs somewhere
22:14:49 <mtreinish> when they get merged
22:14:56 <sdague> mtreinish: I'm not sure why
22:15:00 <mtreinish> it looks pretty
22:15:04 <sdague> I really feel like the specs are ok in git
22:15:20 <sdague> because the blueprint is really the entry point
22:15:28 <cyeoh> having read through a few long nova ones, I think its easier to read in processed html
22:15:35 <sdague> cyeoh: ok
22:15:50 <sdague> then I'll defer
22:16:19 <mtreinish> sdague: ok is there anything else to discuss on the specs?
22:16:27 <sdague> only one last thing
22:16:50 <sdague> I feel like we should have ~4 specs landed as examples before we open this up to the wide world via ML
22:17:09 <sdague> so will hold off a few
22:17:24 <sdague> ok, done, we can move on
22:17:28 <mtreinish> that seems fair
22:17:47 <mtreinish> ok then let's discuss bps in progress
22:17:55 <mtreinish> zhikunliu: you put something on the agenda
22:18:00 <zhikunliu> yes
22:18:03 <zhikunliu> https://blueprints.launchpad.net/tempest/+spec/cinder-v2-api-tests
22:18:04 <mtreinish> go ahead
22:18:24 <mtreinish> #link https://blueprints.launchpad.net/tempest/+spec/cinder-v2-api-tests
22:18:56 <zhikunliu> we have already create basic cinder v2 tests. Not sure if we should continue to port v1 tests to v2 like nova v2 to v3
22:19:19 <mtreinish> zhikunliu: how much duplication is there between v1 and v2?
22:19:26 <mtreinish> we want to have good coverage for both
22:20:15 <zhikunliu> about 10 test files
22:20:22 <mtreinish> cyeoh and ken1ohmichi have been more plugged in with the nova v2 v3 stuff so they may have some advice
22:20:52 <ken1ohmichi> OK, I will see cinder API tests.
22:21:24 <zhikunliu> thank you!
22:21:39 <ken1ohmichi> np:)
22:21:46 <mtreinish> zhikunliu: so we want to exercise both the v1 and v2 apis. It's just a question of we go about implementing that. Inheritance vs 2 copies for example
22:21:52 <cyeoh> yea I guess it comes down to how similar the APIs look - we tried along the way but weren't that succesfull, but ken1ohmichi has some examples of a direction we'll probably take
22:22:13 <sdague> yeh, the nova v2 api is exceptionally odd though
22:22:20 <sdague> maybe we can do better here
22:22:32 <ken1ohmichi> yes, the sample is https://review.openstack.org/#/c/78137/
22:22:55 <sdague> so I would honestly really encourage this to take place in a qa-spec - https://github.com/openstack/qa-specs  because I think that provides a really good way of reviewing the approach
22:23:07 <sdague> and then we can all be agreed on approach and just review for correctness
22:23:26 <ken1ohmichi> I got it. I will propose it to qa-specs
22:23:40 <ken1ohmichi> as the one of the samples:)
22:23:43 <cyeoh> +1, though I don't think it hurts to have a bit of POC code to point to
22:23:53 <sdague> ken1ohmichi: great
22:23:56 <sdague> cyeoh: agreed
22:24:15 <mtreinish> zhikunliu: ok is there anything else on that bp?
22:24:24 <zhikunliu> no, thanks...
22:24:35 <mtreinish> ok does anyone have any other bps they'd like to bring up?
22:25:14 <sdague> if not, I'd actually like to jump to the tempest branch discussion
22:25:32 <sdague> because I want to make sure we give that enough time, for folks that we're in -qa channel yesterday to see it
22:25:36 <dkranz> sdague: I think I got through classifying about half of the failures
22:25:40 <mtreinish> ok then let's move onto that
22:25:44 <mtreinish> #topic Tempest branching strategy
22:26:14 <mtreinish> sdague: you've got the floor
22:26:30 <sdague> so background
22:26:33 <sdague> #link http://eavesdrop.openstack.org/irclogs/%23openstack-qa/%23openstack-qa.2014-04-02.log
22:26:42 <sdague> starting at about 2014-04-02T13:29:08
22:27:04 <sdague> a question came up about the state of tempest as a validation tool for the openstack brand
22:27:34 <sdague> and out of that conversation I asked the question: does tempest really need branches?
22:28:00 <sdague> because it *should* be testing an API that is invariant between releases
22:28:12 <dkranz> sdague: I think it needs versions, not branches, like client libraries.
22:28:13 <sdague> or changed in ways that are detectable/flaggable
22:28:24 <sdague> dkranz: sure, lets get to that in a sec
22:28:56 <sdague> we have 2 weeks before we would normally set the stable/icehouse branch
22:29:04 <sdague> so now is the time to decide if maybe we don't
22:29:25 <sdague> and use master tempest on stable/icehouse and master going forward, and try to get tempest master to work against stable/havana
22:29:37 <cyeoh> so I'm not saying I'm opposed to it, but my concern is if we branch of the overhead of adding new tests
22:30:06 <cyeoh> having to add them to multiple branches if we want to use them as a validation tool
22:30:08 <sdague> cyeoh: vs. the cost of backporting them?
22:30:27 <sdague> because thinks like defcore/refstack are going to lag
22:30:38 <sdague> at icehouse summit they are going to put out a havana based tool
22:30:43 <sdague> which vendors will want to fix
22:31:14 <cyeoh> they are going to want to fix the tool?
22:31:28 <dkranz> cyeoh: The tool is going to run tempest
22:31:31 <sdague> I am sure they will object to some results
22:31:43 <cyeoh> ok
22:31:53 <sdague> and they may be correct in tempest fixes that should land, and then backport
22:32:36 <sdague> the reason we set a branch today is because until this release we didn't have a feature enable mechanism
22:32:47 <sdague> so branch was a giant feature flag
22:32:54 <mtreinish> sdague: well we did but it was a mess
22:32:57 <sdague> havana nova supports X
22:33:06 <sdague> icehouse nova support X + Y
22:33:29 <sdague> however the feature flag model should mean that new extensions just come in under a feature flag, for instance
22:33:47 <mtreinish> sdague: so the prereq for this on devstack as I said yesterday
22:33:52 <dkranz> sdague: The big advantage of running master is that adding tests always lags the release
22:33:52 <sdague> correct
22:34:01 <mtreinish> we have to make sure we lock the feature set in the devstack config for stable/icehouse
22:34:15 <dkranz> sdague: RIght not there are many tests in master that can and should run when testing havana
22:34:39 <dkranz> sdague: And many that don't due to questionable api changes
22:34:46 <sdague> dkranz: sure
22:34:48 <cyeoh> so just to clarify for me how this would work - if we add Nova API V2 tests during Juno, what would we have to do to ensure they are added for both Juno Nova API testing as well as say Icehouse
22:35:17 <sdague> tempest master would gate on master and stable/icehouse of all the server projects
22:35:17 <mtreinish> cyeoh: but they should given the stable api :)
22:35:21 <dkranz> cyeoh: I would say they should run on both by default
22:35:38 <cyeoh> dkranz: yep that's what I'd like the default to be
22:35:45 <cyeoh> mtreinish: yea, they *should* :-)
22:35:45 <sdague> I've been working on devstack gate branch overrides that would let us specify that behavior
22:35:52 <dkranz> cyeoh: And if a test doesn't run due to a havana bug that was fixed iceshouse we tag it as >havana
22:36:00 <sdague> cyeoh: so it will only be 1 add to tempest
22:36:06 <sdague> and it will gate on both branches
22:36:12 <sdague> so you know if it works or not
22:36:22 <cyeoh> sdague: ok, sounds great then :-)
22:36:30 <dkranz> sdague: So are we going to double the gate jobs ?
22:36:37 <mtreinish> dkranz: we have to
22:36:38 <sdague> for tempest
22:36:48 <dkranz> sdague: Actually triple since havana + icehouse + master
22:36:55 <dkranz> sdague: That is my biggest concern with this approach
22:36:58 <mtreinish> dkranz: this will only be for icehouse forward
22:37:02 <sdague> it's only on tempest
22:37:12 <dkranz> sdague: Ah, ok
22:37:19 <sdague> mtreinish: I think getting havana over is still on the table
22:37:23 <sdague> depending on how much work it is
22:37:32 <mtreinish> it is but the discussion now is about what we do on the 17th
22:37:41 <mtreinish> we still have the havana branch whether we like it or not
22:37:46 <sdague> right, for the 17th, it would be just icehouse + master
22:37:46 <dkranz> mtreinish: I don't see why we have to commit
22:37:51 <sdague> agreed
22:38:17 <sdague> I'll put the havana jobs on nv, and if we get it working, we'll delete the havana branch, and win
22:38:23 <sdague> if not
22:38:28 <sdague> it ages out in 5 months
22:38:54 <dkranz> sdague: But it can't if refstack is using it
22:39:04 <dkranz> sdague: I mean can't age out
22:39:09 <sdague> havana still stops being supported in 5 months
22:39:15 <mtreinish> dkranz: havana goes eol in 5 months
22:39:17 <sdague> openstack policy hasn't changed in that regard
22:39:21 <mtreinish> we just did it for grizzly
22:39:48 <dkranz> sdague: That doesn't mean refstack will stop using it. Have you told them this intent?
22:39:57 <dkranz> sdague: Cause it makes no sense from their standpoint
22:40:09 <dkranz> sdague: Of course it could just be cloned into a refstack repo at that point.
22:40:26 <sdague> dkranz: honestly, I'm not policing them, I have lots of other things to do :)
22:40:36 <dkranz> sdague: RIght :)
22:40:53 <dkranz> sdague: Based on my results so far I think it is very doable to get master to work with havana
22:40:58 <sdague> ken1ohmichi, masayukig any thoughts on this one? I'd like to know if there are other problems we haven't thought about yet
22:41:02 <sdague> dkranz: great
22:41:14 <dkranz> sdague: We just need a tagging strategy for the breaks due to bug fixes in icehouse and api changes
22:41:16 <sdague> I should have gate jobs running nv tomorrow that will let us do that
22:41:39 <sdague> dkranz: example?
22:41:54 <dkranz> sdague: https://etherpad.openstack.org/p/master-against-havana-errors
22:42:26 <ken1ohmichi> honestly, I am not familar with refstack.
22:42:33 <sdague> dkranz: I was wondering about "tagging strategy"
22:42:51 <sdague> ken1ohmichi: refstack is basically a website that some foundation folks will run where you can submit tempest tests
22:42:52 <dkranz> ken1ohmichi: It is the "use tempest to gate use of OpenStack trademark"
22:43:01 <sdague> test results
22:43:04 <sdague> for your product
22:43:16 <sdague> as a piece of trademark use policy
22:43:28 <sdague> they are still working out all the trademark details
22:43:35 <dkranz> sdague: about tagging strategy, I think we should have a tag saying a test is enabled as of some version
22:43:49 <sdague> running a subset of tempest successfully will be a part of what you need to do to get the trademark
22:43:50 <mtreinish> dkranz: that should be handled by config
22:44:00 <sdague> right, I agree with mtreinish
22:44:05 <ken1ohmichi> oh, thanks. I got it little.
22:44:07 <maurosr> sdague: one thing that came to my mind is the maintenance burden, like in this situation we  cant remove nova's xml tests
22:44:20 <dkranz> mtreinish: how? some of the tests have different havana/icehouse behavior due to api change
22:44:20 <maurosr> (not sure if you guys touched this matter, lost my connection)
22:44:27 <masayukig> sdague: Does refstack still uses grizzly?
22:44:28 <sdague> maurosr: that's true, at least not for a while
22:44:38 <sdague> masayukig: they are building on havana atm
22:44:45 <sdague> from what I understand
22:45:04 <mtreinish> dkranz: well those types of changes will be blocked by tempest in the first place
22:45:32 <mtreinish> it really only becomes a problem for when we add new test coverage
22:45:45 <dkranz> mtreinish: which we did in icehouse
22:45:47 <maurosr> since api changes are not supposed to happen I guess that is a 'good break'
22:45:59 <dkranz> mtreinish: And the tests were not blocked. We changed them.
22:46:08 <dkranz> mtreinish: YOu can see examples in that etherpad
22:46:10 <maurosr> it's something that maybe cant work planty right now, but will be good at long term
22:46:11 <mtreinish> dkranz: yes because we default everything on everywhere
22:46:24 <mtreinish> but if we have a static set of what's expected in devstack for icehouse
22:46:30 <mtreinish> then those new tests won't run
22:46:41 <sdague> dkranz: there are definitely details here on what the 2 step is, but I think those are doable
22:46:50 <dkranz> mtreinish: But why shouldn't a new test for a stable API in a previous version run?
22:47:04 <dkranz> against the old version
22:47:14 <dkranz> mtreinish: I agree we can do better icehouse->juno
22:47:26 <mtreinish> dkranz: it should but we'd need a devstack change to add it to the stable job (it's just a shift in the 2 step)
22:47:32 <sdague> so I think what we probably want to do is after stable/icehouse on devstack is set
22:47:52 <sdague> we generate the full extension list and hardcode it in devstack stable/icehouse
22:47:55 <sdague> but it floats on master
22:48:01 <sdague> so stuff is auto added
22:48:07 <mtreinish> sdague: exactly
22:48:19 <sdague> but that's actually after the 17th
22:48:36 <mtreinish> yeah it would have to be because we need the stable branch on devstack
22:48:38 <dkranz> sdague: That's fine but half the problem. The other part is stuff that actually changed, or that works in icehouse but not havana due to an unfixed bug in havana
22:48:48 <sdague> dkranz: sure
22:48:56 <dkranz> sdague: At a minimum we could just skip all the fails I found for havana.
22:49:08 <dkranz> sdague: That would be a better test than what is on stable/havana now.
22:49:19 <sdague> well, I don't want to branch skip
22:49:24 <sdague> I think that logic gets nutty
22:49:36 <sdague> so we either fix it, feature flag it, or the job doesn't vote
22:49:58 <dkranz> sdague: Why? We just classify the test as belonging to a new feature. Just a funny one.
22:50:18 <sdague> because we're already going to have 200 features
22:50:23 <sdague> that are real ones
22:50:28 <dkranz> sdague: And this would be one more.
22:50:46 <sdague> I think if we invent funny ones that we can't double check on real extension lists we're going to get it wrong
22:50:57 <sdague> there is a big who watches the watcher thing here
22:51:02 <dkranz> sdague: Anyway, the goal is to run master against havana with as many tests as we can get to work.
22:51:10 <dkranz> sdague: We need to figure the details but not right now.
22:51:14 <sdague> sure
22:51:21 <ken1ohmichi> sdague: the extension list means API list?
22:51:29 <sdague> ken1ohmichi: yes
22:51:43 <mtreinish> ken1ohmichi: yeah it's the api_extensions config option
22:51:57 <dkranz> sdague: What I am talking about is more like bug skipping than feature config.
22:52:05 <ken1ohmichi> sdague: ok, masayukig has automatic API list feature on gerrit as you know.
22:52:19 <sdague> dkranz: yep, I know, but it complicates things when you need to do branch selection
22:52:24 <ken1ohmichi> sdague: will it be useful it?
22:52:25 <sdague> because tempest doesn't know that
22:52:37 <sdague> ken1ohmichi: I don't know, that's a good question
22:52:51 <dkranz> sdague: no, you would have to tell it. I will propose something when I get through the list.
22:52:52 <sdague> I think not exactly here
22:53:05 <mtreinish> ken1ohmichi: http://git.openstack.org/cgit/openstack/tempest/tree/tools/verify_tempest_config.py#n88
22:53:13 <mtreinish> that's all we need for this
22:53:20 <mtreinish> more or less
22:54:00 <sdague> ok, any other thoughts on this?
22:54:28 <mtreinish> sdague: you should probably send something out the ML announcing this change
22:54:31 <ken1ohmichi> mtreinish: thanks:)
22:54:51 <sdague> mtreinish: yeh, I want to get the first data results tomorrow before hand to make sure the jobs are even doable :)
22:54:56 <sdague> but then I will
22:55:23 <sdague> #action sdague to bring up branchless tempest on mailing list
22:55:57 <sdague> ok, so in the 5 minutes left, I want to give an early congrats to mtreinish on being QA-PTL-elect :)
22:56:11 <mtreinish> actually I was going to move onto critical reviews
22:56:14 <cyeoh> mtreinish: congrats!
22:56:17 <mtreinish> but I'm fine with that too :)
22:56:17 <sdague> officially the hat passes at the release
22:56:20 <sdague> so in 2 weeks
22:56:42 <mtreinish> thanks
22:56:48 <sdague> so I'll try to clean up a few things before abdicating to him :)
22:57:23 <sdague> ok, 4 minutes for critical reviews
22:57:25 <sdague> 3 minus
22:57:26 <mtreinish> who knows I might pull an Andrew Jackson and make you flee your house after it's official in a few hours :)
22:57:32 <mtreinish> #topic Critical Reviews
22:57:44 <mtreinish> so does anyone have any reviews they want to bring up?
22:57:47 <masayukig> https://review.openstack.org/#/c/66882/
22:58:07 <mtreinish> #link https://review.openstack.org/#/c/66882/
22:58:26 <mlavalle> Can core reviewers take a look at these neutron api tests:
22:58:31 <mtreinish> masayukig: ooh testscenarios fun
22:58:36 <mlavalle> https://review.openstack.org/#/c/66541
22:58:36 <mlavalle> https://review.openstack.org/#/c/65943
22:58:37 <mlavalle> https://review.openstack.org/#/c/66259
22:59:08 <mtreinish> mlavalle: sure
22:59:19 <mlavalle> thanks!
22:59:26 <sdague> ok, I think we are out of time
22:59:29 <masayukig> mtreinish: Separate the class vs Merge the class
22:59:55 <mtreinish> ok thanks everyone
23:00:04 <mtreinish> #endmeeting