22:00:56 <mtreinish> #startmeeting qa
22:00:57 <openstack> Meeting started Thu Jul 24 22:00:56 2014 UTC and is due to finish in 60 minutes.  The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot.
22:00:58 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
22:01:00 <openstack> The meeting name has been set to 'qa'
22:01:07 <mtreinish> hi, who do we have here today?
22:01:18 <masayukig> o/
22:01:21 <ravikumar_hp> hi
22:01:29 <mtreinish> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Proposed_Agenda_for_July_24_2014_.282200_UTC.29
22:01:31 <oomichi> hi
22:01:32 <andreaf> hi
22:01:34 <mtreinish> ^^^ Today's agenda
22:01:59 <dkranz> o/
22:02:10 <mtreinish> ok well let's get started
22:02:18 <asselin> hi
22:02:23 <mtreinish> #link Volunteer to run next 2 meetings on July 31st and August 7th (mtreinish)
22:02:29 <mtreinish> d'oh
22:02:36 <mtreinish> #topic Volunteer to run next 2 meetings on July 31st and August 7th (mtreinish)
22:02:55 <mtreinish> so I won't be able to make the next 2 meetings
22:03:07 <dkranz> mtreinish: I can do next week but will be out Aug 7th
22:03:11 <mtreinish> so I'm looking for a volunteer to run them and send the reminder email out
22:03:33 <mtreinish> dkranz: ok, I think having one person do both would be preferable but we can split it up
22:03:43 <mtreinish> especially given the timezone differences
22:03:52 <masayukig> I can't join next 2 meetings.. sorry.
22:04:03 <oomichi> I also cannot make both :-(
22:04:25 <sdague> both the next meetings are iffy for me actually
22:04:37 <mtreinish> masayukig, oomichi: heh, stop doing that, you're beating me to asking you if you can lead the Aug 7th meeting :)
22:04:53 <mtreinish> hmm, ok well if no one can run the Aug 7th one then how about we just cancel it
22:05:12 <mtreinish> so dkranz if you want to take July 31st
22:05:18 <mtreinish> and I'll cancel the Aug 7th meeting
22:05:28 <sdague> +1
22:05:32 <dkranz> mtreinish: sure. If there is no one around to run the meeting it won't be very useful :)
22:05:58 <mtreinish> #info dkranz to run July 31st meeting and send email reminder
22:06:05 <mtreinish> #info Aug 7th meeting is cancelled
22:06:12 <mtreinish> ok let's move on to the next topic
22:06:21 <mtreinish> #topic Mid-cycle Meetup Summary (mtreinish)
22:06:36 <mtreinish> so last week we had the midcycle meetup in Darmstadt
22:06:42 <mtreinish> which I thought was very productive
22:06:49 <mtreinish> #link https://etherpad.openstack.org/p/Qa_Infra_Meetup_2014
22:06:59 <mtreinish> that's the etherpad we were using to track things during it
22:07:30 <mtreinish> but the work items that started out of it were
22:07:43 <mtreinish> the test-accounts alternative to tenant isolation
22:08:02 <mtreinish> converting the scenario tests to use the tempest clients
22:08:16 <mtreinish> and the giant re-architect the gate thread
22:08:26 <mtreinish> which jeblair and sdague pushed out to the ML
22:08:53 <sdague> yeh, that would be great to get folks in on if they haven't yet looked at it
22:09:13 <mtreinish> I just wanted to give a brief overview of what we were working on for those who couldn't attend
22:09:28 <mtreinish> so unless there are any questions I think we can move on
22:10:23 <mtreinish> #topic Blueprints
22:10:34 <mtreinish> ok so this week is the J-2 milestone
22:10:49 <mtreinish> so I wanted to go over the BPs targetted for J-2 and see what's going on for those
22:10:56 <mtreinish> #link https://launchpad.net/tempest/+milestone/juno-2
22:11:28 <mtreinish> masayukig: heh, so for Migrate the Scenario Tests to the Tempest Clients we should probably defer it to J-3 right?
22:11:44 <mtreinish> we probably were a bit overoptimistic last week when we started it :)
22:11:57 <masayukig> mtreinish: yeah, that's right.
22:12:18 <mtreinish> ok next on the list is javelin 2
22:12:19 <andreaf> mtreinish, masayukig: yes - but that's moving forward quiclky at least :)
22:12:29 <mtreinish> sdague: that's a defer too right?
22:12:48 <mtreinish> andreaf: yeah, it is moving quickly, which is great
22:12:48 <sdague> mtreinish: yes, we're close, but it's definitely going to defer
22:13:00 <mtreinish> sdague: ok
22:13:50 <mtreinish> hmm so does anyone know Arthur Svechnikov's nick?
22:14:15 <mtreinish> although I'm pretty sure ceilometer scenarios is also a defer
22:14:33 <masayukig> andreaf, mtreinish: yeah, 10/18 test cases are done or doing, now.
22:14:49 <masayukig> #link https://etherpad.openstack.org/p/tempest-client-scenarios
22:14:53 <mtreinish> masayukig: yeah that is a lot of progress for 1 week
22:15:17 <andreaf> masayukig, mtreinish: I'm trying to get onboard some more collegaues on it
22:15:20 <mtreinish> hmm looking at the rest of the list I think all the other bps are defers too
22:15:25 <dkranz> mtreinish: Isn't your "Add a config file verification tool" "done" ?
22:15:46 <mtreinish> dkranz: well it still needs some more testing coverage
22:15:48 <dkranz> mtreinish: I mean I am refactoring it right now but that is not really related to the blueprint, more to the one from boris
22:15:59 <mtreinish> but functionaly I think I've gotten most of down already
22:16:01 <dkranz> mtreinish: ok
22:16:08 <mtreinish> dkranz: but sure I can close it out
22:16:12 <dkranz> mtreinish: I suggest waiting until I check in the refactor for that.
22:16:19 <mtreinish> unless there are other discoverable options I've missed
22:16:27 <mtreinish> dkranz: ok that's fine I'll defer it too
22:16:29 <dkranz> mtreinish: If there are, I will find them.
22:16:55 <dkranz> mtreinish: ANd include them in the refactoring
22:17:02 <mtreinish> dpaterson: you joined right in time :)
22:17:16 <dpaterson> Sorry I was late, traffic
22:17:20 <mtreinish> dpaterson: any update on https://blueprints.launchpad.net/tempest/+spec/post-run-cleanup
22:17:25 <masayukig> andreaf: great!
22:17:27 <mtreinish> since the j-2 milestone is this week
22:17:28 <dpaterson> yes
22:17:52 <dpaterson> I have most of it done.  The last piece I am working on is the --dry-run.
22:17:59 <dpaterson> I am going to finish that up early next week
22:18:08 <mtreinish> dpaterson: ok have you pushed anything up for review yet?
22:18:19 <mtreinish> it's ok to do it iteratively
22:18:23 <dpaterson> Not yet
22:18:40 <mtreinish> dpaterson: ok, then I should defer it to the next milestone?
22:18:50 <dpaterson> When is that?
22:19:21 <mtreinish> dpaterson: Sept. 4th I believe
22:19:38 <dpaterson> yes, that seems like a better plan
22:19:47 <mtreinish> dpaterson: it's not going to get merged this week so yeah :)
22:19:51 <dkranz> dpaterson: deferring it doesn't mean you can't check it in as soon as you want :)
22:19:58 <dpaterson> yes
22:20:16 <mtreinish> ok the last thing is mriedem's BP for the CLI version check
22:20:19 <dpaterson> I can push code pretty soon.
22:20:25 <mtreinish> dpaterson: ok awesome
22:20:41 <mtreinish> that BP should be finished after https://review.openstack.org/100031 lands
22:20:54 <mtreinish> so we can close that out on schedule if it get's merged this week :)
22:21:26 <mtreinish> but if not it's no big deal
22:21:53 <mtreinish> ok aside from the people who I know aren't here or I don't know their irc handles that should cover all the BPs for J-2
22:22:11 <mtreinish> I'll reach out to those I missed outside the meeting
22:22:16 <mtreinish> but I think they're all defers
22:22:27 <mtreinish> so does anyone have anything else to discuss on Blueprints
22:22:32 <mtreinish> otherwise we can move on
22:22:36 <dpaterson> mtreinish: how is that BP related to the cleanup work I am doing?
22:22:51 <mtreinish> dpaterson: the BP is used for tracking that effort
22:23:20 <mtreinish> the spec proposal you got merged, turned into the blueprint so we can track it's progress
22:23:26 <dpaterson> Add min_client_version decorator for CLI tests?
22:23:38 <dpaterson> That is the title 10031
22:23:57 <mtreinish> dpaterson: oh, sorry I jumped the gun a bit and changed the BP we were discussing before you were finished
22:24:19 <mtreinish> it was a separate thought from your cleanup BP discussion
22:24:25 <dpaterson> no worries
22:24:39 <dpaterson> making sure I wasn't missing something
22:24:41 <dpaterson> tx
22:24:50 <mtreinish> ok if there's nothing else then let's move on
22:25:00 <mtreinish> #topic Specs Review
22:25:12 <mtreinish> so I put one spec in the agenda this week:
22:25:19 <mtreinish> #link https://review.openstack.org/#/c/108858/
22:25:43 <mtreinish> I basically outlined how I think we should break out a portion of tempest's common code to create an OpenStack functional testing library
22:25:57 <mtreinish> this can be re-used for project specific functional testing
22:26:10 <mtreinish> which was a component of the ML thread on changing what we do in the gate
22:26:44 <mtreinish> It's a little sparse on the details right now
22:26:55 <dkranz> mtreinish: LGTM except for my comment
22:26:57 <mtreinish> but if people could take a look and add their thoughts that would be great
22:27:09 <mtreinish> dkranz: yeah I have a response drafted I just need to finish it
22:27:10 <andreaf> mtreinish: I'll add it to my review list
22:27:23 <andreaf> mtreinish: are you thinking of a new olso.xyz
22:27:40 <mtreinish> andreaf: I specifically didn't want it in oslo, because I want it to be under the QA program
22:28:04 <mtreinish> I also wanted to call it mesocyclone, and oslo.mesocyclone just looks weird :)
22:28:35 <dkranz> andreaf: It would also be weird for oslo to have another rest client
22:29:01 <mtreinish> ok that's all I had for my spec review
22:29:16 <mtreinish> does anyone else have a spec proposal that they would like to bring up?
22:29:17 <dkranz> Some people already wonder why it is not unified with the official client
22:29:44 <andreaf> the test-account spec, what shall we do?
22:29:58 <mtreinish> dkranz: yeah, that I don't know, I think it comes down different implementation goals
22:30:01 <andreaf> #link https://review.openstack.org/#/c/98400/
22:30:18 <mtreinish> andreaf: oh, that's a good point I forgot to review that before we started working on it
22:30:29 <dkranz> mtreinish: I don't either but it was discussed at the neutron mid-cycle
22:30:53 <mtreinish> andreaf: I'll take a look at it tomorrow and fix anything that differs from what we've implemented and self approve it
22:30:58 <mtreinish> if no one objects to that
22:31:21 <andreaf> mtreinish: sure I'm ok with that
22:31:34 <dkranz> anderstj: There was a -1 review with some nits
22:31:45 <dkranz> andreaf: ^^^
22:31:51 <dkranz> anderstj: Sorry
22:32:18 <marun> +1 for me on eventual unification
22:32:28 <marun> I think there could be a core that encompasses the goals of the tempest client
22:32:36 <andreaf> dkranz: I'll have a look as well - mtreinish: if you want to post comments I can make a new version as you wish
22:32:38 <marun> and something that uses that core but with end-user niceties
22:32:51 <mtreinish> marun: heh, I'm not sure that was actually on the table :)
22:32:58 <marun> I know it's not on the table now
22:33:03 <marun> that's why I said 'eventual'
22:33:14 <oomichi> andreaf: I also will join to the spec.
22:33:15 <mtreinish> marun: heh, fair enough
22:33:16 <marun> when the unnecessary duplication becomes painful and therefore obvious
22:33:21 <marun> :)
22:33:51 <mtreinish> ok are there any other specs to discuss?
22:34:02 <andreaf> marun: I think trying to make that now would divert us from the primary focus to much
22:34:15 <andreaf> marun: but eventually it may be
22:34:25 <marun> andreaf: yeah, I'm not pushing hard for thinking about it now
22:34:31 <marun> just planting a sead
22:34:33 <marun> seed
22:35:10 <mtreinish> ok let's move to the next topic
22:35:18 <mtreinish> #topic Now that there is agreement to move api/func testing to projects, should we start that (virtually) now? (dkranz)
22:35:33 <mtreinish> dkranz: with the absurdly long topic title, you've got the floor
22:35:37 <dkranz> mtreinish: So this was based on two opinions I have
22:35:51 <dkranz> 1. We are paying too much for "double-entry" bookkeeping
22:36:08 <dkranz> 2. It has never been clear exactly what the content of tempest api tests should be
22:36:44 <dkranz> Right now they are not enough to keep out bugs, but enough to run a long time and trigger flaky bugs
22:36:58 <dkranz> One good answer is more func tests in projects
22:37:29 <dkranz> But having a full set in tempest does not scale
22:37:36 <dkranz> Open for other comments :)
22:37:39 <sdague> dkranz: 100% agree
22:37:53 <sdague> we've grown beyond the point where that's sustainable
22:38:12 <dkranz> sdague: So I am proposing something very simple for now
22:38:28 <dkranz> I actully think this was the original purpose of the smoke tag
22:38:44 <oomichi> if moving API tests to each project, will tempest contain only scenario tests and CLI tests?
22:38:45 <dkranz> It was never envisioned that tempest api would get to where it is now
22:38:56 <dkranz> It grew organically and with the tc command for tempest tests
22:39:08 <dkranz> oomichi: NO, it needs to have cross-proejct api tests
22:39:20 <dkranz> oomichi: The trick is that it is not always obvious how to determine that
22:39:20 <mtreinish> dkranz: I agree too, but I don't think we should be running less in the gate now. Just not grow it much and start to migrate things to the projects as functional tests
22:39:27 <dkranz> oomichi: But I think we should trty
22:39:31 <marun> dkranz: aka integration tests
22:39:42 <mtreinish> oomichi: actually cli tests will be the first thing converted to project functional tests
22:39:59 <dkranz> mtreinish: I am not really opposed to that but we have to define what is supposed to be there
22:40:01 <marun> I would hope that the goal would be migrating responsibility rather than hack/slash, so that we don't lose out in the transition.
22:40:09 <mtreinish> oomichi: because there isn't anything that's cross-project about them, they really are functional tests, not integration tests
22:40:32 <sdague> oomichi: so honestly, I'd start with migrating out the CLI tests
22:40:32 <oomichi> mtreinish, dkranz: ah, I can see it.
22:40:41 <rockyg> Question:  Most of Refstack tests are API tests.  Could QA still be the central agent (at least one +2) while the tests get executed outside of tempest?
22:40:59 <mtreinish> marun: yes, which was basically my comment on the ML thread about the gate rearchitecture
22:41:10 <marun> mtreinish++
22:41:14 <mtreinish> I don't want to see a drop in coverage while we move to something more targetted
22:41:15 <sdague> marun: we're not rushing this one
22:41:29 <rockyg> I think QA still needs some ownership, including repo, just not the infra the tests run on.
22:41:35 <sdague> this is about building momentum towards a better system
22:41:37 <andreaf> tracking which api we touch in either API or scenario tests may help us getting an idea of the kind of coverage we have
22:41:52 <dkranz> mtreinish: fair enough. But we have to define what will be in tempest going forward
22:41:55 <andreaf> it would be good to have something like the stress api tracker for all tests
22:42:12 <mtreinish> dkranz: yes I agree, we need to figure that out before we start doing anything
22:42:16 <marun> I think in an ideal case the projects write new API and scenario tests in-tree, and then migrate into separate repos once they have been stabilized.
22:42:17 <rockyg> Could API test results feed into QA the way refstack results feed into Refstack.org?
22:42:50 <dkranz> mtreinish: And there would be a lot of value in running only nova smoke against other projects. Whether that value has greater costs is debatable.
22:42:53 <marun> Whether QA or qualified project-specific reviewers maintain those 'test' repos would be an implementation detail.
22:42:54 <sdague> marun: or just put a hacking freeze on those tests
22:43:15 <mtreinish> marun: yes I think something like that would be an ideal way to do it, but we need criteria on what kind of test is allowed to do that
22:43:19 <mtreinish> and what should be in tempest
22:43:23 <marun> sdague: I'm not sure what you mean, how would it protect against accidental modification?
22:43:38 <mtreinish> marun: I think he's saying have the pep8 job fail if the test is changed
22:43:53 <marun> mtreinish: I didn't realize that was possible, link?
22:43:54 <sdague> marun: enforce a rule where some part of the tree can only be changed if it's the only part that's changed
22:44:20 <sdague> marun: that code doesn't exist yet, but it would be a thing one could write
22:44:37 <marun> That sounds great if it would allow us to protect API tests against regression.
22:44:46 <marun> sdague: ah, gotcha.  Definitely a worthy initiative
22:44:51 <sdague> marun: yeh
22:45:29 <sdague> honestly, my goal here is to move more responsibility back to the projects, and hopefully help expose some issues in a simpler state of the world that means better overall openstack quality
22:45:30 <dkranz> sdague: yes, that would be a less costly way to get double-entry bookkeeping
22:45:36 <sdague> dkranz: yep
22:45:51 <dkranz> sdague: ++
22:46:08 <marun> sdague: ++
22:46:09 <dkranz> sdague: Let's think about what tests tempest would actually have in that world
22:46:31 <mtreinish> sdague: you sure flip flop on your opinion about hacking :)
22:46:49 <mtreinish> dkranz: so I think give our time constraints we'll have to take that converastion outside the meeting
22:46:52 <marun> isn't it better to be right than hypocritical? ;)
22:46:56 <dkranz> mtreinish: agreed
22:47:04 <marun> that made no sense, sorry
22:47:12 <dkranz> mtreinish: I just wanted to get this discussion going as a follow-on to Sean and Jim's email
22:47:25 <marun> hypocritical + right vs consistent + wrong
22:47:43 <sdague> marun: it's called learning and adapting :)
22:47:56 <mtreinish> dkranz: yeah I plan to start a separate ML thread in response to: http://lists.openstack.org/pipermail/openstack-dev/2014-July/041182.html
22:47:59 <marun> sdague: ++
22:48:23 <mtreinish> although I think we'll inevitably have a summit session titled: "Tempest Scope in the Brave New World"
22:48:33 <rockyg> ++
22:48:38 <andreaf> mtreinish: +1
22:48:41 <dkranz> mtreinish: great. I feel like we have turned a major corner here to make things better
22:48:46 <masayukig> Mtreinish: +1
22:49:10 <oomichi> mtreinish: nice title:)
22:49:24 <marun> mtreinish: ++, look forward to it
22:49:41 <mtreinish> but, I definitely don't want to punt on the discussion until Nov., so I think we should definitely take this to the ML and other places in the mean time
22:49:55 <mtreinish> anyway we've got 10min left so I think we should move on
22:50:04 <andreaf> mtreinish, sdague: the post gate testing part of it opens up space for more complex and possibly long running tests as well - which could live in tempest in the brave new world
22:50:38 <rockyg> andreaf:  +1 integrated tests, scenario tests, etc
22:50:49 <mtreinish> #topic Critical Reviews
22:50:55 <sdague> andreaf: maybe...
22:51:06 <andreaf> sdague :D
22:51:16 <mtreinish> ok does anyone have any reviews they'd like to bring up?
22:51:27 <andreaf> mtreinish: all the reviews related to scenario migration
22:51:34 <mtreinish> andreaf: a link?
22:51:36 <andreaf> especially the base onse
22:51:45 <sdague> andreaf: honestly, getting more debugable functional tests would be might better for overall quality
22:51:49 <andreaf> #link https://review.openstack.org/107552
22:52:10 <andreaf> #link https://review.openstack.org/107562
22:52:56 <mtreinish> #link https://review.openstack.org/107371
22:53:04 <andreaf> sdague: perhaps... but if you can collect big data (using subunit2sql) and watch trend on less stable tests it may give you something interesting
22:53:22 <mtreinish> that is also from the mid-cycle and I think it is ready since I added more unit tests
22:53:42 <afazekas> :)
22:53:55 <andreaf> mtreinish: I'll check it tomorrow
22:54:18 <mtreinish> andreaf: well, I need to get subunit2sql into infra first :)
22:54:19 <andreaf> pre-unit tests it was good already for me
22:54:38 <mtreinish> andreaf: heh, well I found some bugs through unit testing...
22:54:46 <mtreinish> ok are there any other reviews to bring up?
22:55:40 <mtreinish> ok then let's move on
22:56:04 <mtreinish> #topic Proposal of temp review policies (andreaf)
22:56:23 <andreaf> there are three things I wanted to propose
22:56:48 <andreaf> New scenario tests shall use tempest client. I guess people could rebase on in-flight changes?
22:57:03 <andreaf> until the migration bp is completed
22:57:12 <marun> hmmm
22:57:24 <mtreinish> so I'm fine with that one, except we have some long standing reviews adding scenarios
22:57:35 <mtreinish> I'd hate to churn on that again to rebase and change the client usage
22:58:01 <mtreinish> I'm thinking mostly around heat
22:58:03 <andreaf> mtreinish: ok - well we left the original base classes in for now
22:58:29 <andreaf> mtreinish: yes I started working on the existing heat ones, but it may take a bit longer
22:58:42 <mtreinish> andreaf: but for submissions that are newer than last week I definitely agree
22:58:53 <mtreinish> andreaf: < 2 min so what are the other 2? :)
22:58:58 <andreaf> mtreinish: ok makese sense
22:59:04 <andreaf> New API tests shall not include return code validation. I think introducing return code validation should be a separate patch as it can be reviewed and tested independently
22:59:08 <andreaf> New scenario / API neutron tests on hold (no +A) until full job voting on neutron and tempest
22:59:25 <mtreinish> so we actually have an agreement on that last one from several meetings back
22:59:32 <mtreinish> I can dig up the logs
22:59:34 <andreaf> ok
23:00:08 <mtreinish> the second one I'm not sure about, mostly because I'm not sure what the goal is
23:00:10 <andreaf> I though the last proposal was to make the job voting on neutron only but I think it should be on tempest as well
23:00:28 <andreaf> ok we can take it offline time's up
23:00:33 <mtreinish> andreaf: yep
23:00:38 <mtreinish> ok thanks everyone we're at time
23:00:42 <mtreinish> #endmeeting