17:00:47 <mtreinish> #startmeeting qa
17:00:48 <openstack> Meeting started Thu Mar  3 17:00:47 2016 UTC and is due to finish in 60 minutes.  The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:49 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:53 <openstack> The meeting name has been set to 'qa'
17:00:58 <mtreinish> hi, who's here today?
17:01:01 <dpaterson> o/
17:01:06 <tosky> hi
17:01:12 <jlanoux> hi
17:01:13 <jordan__> hi
17:01:18 <mtreinish> today's agenda:
17:01:20 <mtreinish> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Proposed_Agenda_for_March_3rd_2016_.281700_UTC.29
17:02:06 <mtreinish> oomichi, andreaf: around?
17:02:46 <mtreinish> ok, let's get started
17:02:55 <mtreinish> #topic Specs Reviews
17:03:00 <oomichi> mtreinish: hi
17:03:03 <ylobankov> hi
17:03:05 <mtreinish> #link https://review.openstack.org/#/q/status:open+project:openstack/qa-specs,n,z
17:03:39 <mtreinish> we've got a few open spec reviews (I think some of that is the fallout from the code sprint last week)
17:03:58 <mtreinish> it'd be good if people put some eyes on those when they get a chance
17:04:11 <mtreinish> does anyone have any open spec review they'd like to discuss?
17:04:31 <jordan__> I don't (just to say I am reading this�
17:04:43 <oomichi> mtreinish: one thing
17:04:55 <oomichi> mtreinish: what is "Move reintegrate-tempest-lib spec to implemented" ?
17:05:11 <mtreinish> oomichi: the bp is closed so that is just house keeping to say its done :)
17:05:15 <oomichi> the code of tempest-lib is already merged into tempest now. so what is that?
17:05:40 <mtreinish> oomichi: it just moves the file into the implemented subdir
17:05:59 <oomichi> mtreinish: ah, ok. the spec should be approved soon :-)
17:06:51 <mtreinish> jordan__: which one are you reading?
17:07:03 <jordan__> (reading the channel)
17:07:14 <mtreinish> ah, ok
17:07:41 <mtreinish> does anyone have anything else to discuss on this topic?
17:08:05 <oomichi> nothing from me
17:08:56 <mtreinish> ok, lets move on then
17:09:09 <mtreinish> #topic Priority Items
17:09:17 <mtreinish> #link https://etherpad.openstack.org/p/mitaka-qa-priorities
17:09:35 <mtreinish> so IIRC mitaka-3 is this week
17:10:08 <mtreinish> looking at the priority list there are 3 items listed for M3
17:10:49 <mtreinish> tempest cli, resource configuration, and finish plugin decomposition for non base layer projects
17:11:32 <dpaterson> I am getting back on wrapping run next week
17:11:54 <dpaterson> that should close cli bp when completed
17:11:59 <mtreinish> I'm assigned that last 1 and it's still in progress. We're down to just trove, heat, ceilo, and sahara in tree iirc
17:12:09 <mtreinish> dpaterson: ok, cool
17:12:29 <mtreinish> for tempest that is, I need to check the plugins in devstack and grenade too
17:12:47 <oomichi> service client thing also should be shifted to M3 now, will make sumarry for the next meeting
17:12:52 <mtreinish> I still hope to have that finished by the end of the cycle
17:13:00 <mtreinish> oomichi: ok, thanks
17:13:08 <jordan__> I thought ceilo was done
17:13:14 <oomichi> mtreinish: yeah, that should be done
17:13:15 <jordan__> there's at least a WIP somewhere
17:13:34 <mtreinish> jordan__: they have the patches up to add a plugin
17:13:39 <mtreinish> but nothing to remove it from the tempest side
17:13:58 <mtreinish> if the plugin patches landed we can go ahead and remove the tempest code
17:14:40 <oomichi> ironic code still exist in tempest side even if they implemented it in project side.
17:15:01 <jordan__> mtreinish, do you have a link to that plugin patch ?
17:15:14 <oomichi> does that(removing the code from temest) depend on each project policy or something?
17:15:36 <mtreinish> oomichi: sigh, I forgot about that. Ok, 1 more project to add to the list of people I need to harass
17:15:49 <devananda> oomichi: we have to leave it there for stable branch testing
17:16:08 <oomichi> devananda: yeah, that is a nice point.
17:16:09 <mtreinish> devananda: no you don't, you just need to install the plugin against tempest in the venv
17:16:17 <mtreinish> and add that to your stable jobs
17:16:33 <mtreinish> jordan__: so I just checked the base plugin landed already in ceilo, but there are no tests
17:16:34 <devananda> mtreinish: stable branches of ironic do not have the plugin
17:16:45 <jordan__> ok
17:16:55 <mtreinish> jordan__: https://github.com/openstack/ceilometer/tree/master/ceilometer/tests/tempest
17:16:59 <devananda> mtreinish: oh, i see. install master ironic in a venv alongside tempest, and use that to test stable ironic? ... interesting
17:17:05 <mtreinish> devananda: yes
17:17:13 <mtreinish> devananda: that's how tempest does it because it's branchless
17:17:22 <devananda> jroll: ^
17:17:38 <mtreinish> devananda: it's part of the reason I always advocate making the tempest plugin live in a seperate project (it makes that much easier)
17:18:06 <jroll> hi
17:18:21 <devananda> mtreinish: huh, well, I believe we followed the docs, which didn't suggest that. </sigh>
17:18:21 <jroll> devananda: mtreinish: yeah, this is on my list to investigate this week
17:18:25 <oomichi> mtreinish: interesting idea
17:18:52 <mtreinish> devananda: I can add that to the docs (I guess I forgot to add that suggestion when I wrote them)
17:19:01 <jroll> but yeah, once we fix this bit, we can drop the ironic code from tempest
17:19:10 <devananda> mtreinish: it makes sense. thanks :)
17:19:19 <oomichi> this kind of thing will be helpful for the other projects also
17:20:26 <mtreinish> ok, is there anything else to discuss on this topic?
17:20:43 <oomichi> the one point is that we need to run stable tests if changing master tempest-like tests on each project
17:20:56 <oomichi> on the above mtreinish's idea.
17:21:29 <oomichi> for checking tempest-like test change works for stable branch
17:21:47 <mtreinish> oomichi: well I thought we decided in yvr that it was a project level decision to define there own stable testing policy for this
17:22:12 <jordan__> yes
17:22:26 <oomichi> mtreinish: yeah, that should
17:23:22 <oomichi> but the stable team is different from project core team now, that also is difficult to maintain each project repo
17:23:40 <mtreinish> like we were making a bright line on things we were going to worry about (and those were things with in tree tests)
17:24:09 <oomichi> yeah, nice to discuss it on your doc patch ;-)
17:24:55 <jlanoux> mtreinish: jordan__ Now that tempest_lib is back in Tempest, what's going on with the migration? Shall we still migrate things in tempest.lib?
17:24:56 <mtreinish> well sort of, the stable team is even smaller than qa (I'm involved in both) and it's the same self service approach for the most part
17:25:15 <jordan__> jlanoux, yes. as far as i know
17:25:36 <jordan__> that's what I read in the spec
17:25:44 <mtreinish> jlanoux: yes, to continue getting the advantages of code updates you should change tempest_lib to tempest.lib
17:25:59 <mtreinish> but the old tempest-lib repo isn't ever going away, it just won't get updates
17:26:07 <jlanoux> mtreinish: jordan__ ok - thanks
17:26:51 <mtreinish> ok, I think we should move on to the next topic now
17:26:59 <mtreinish> #topic Tempest
17:27:17 <mtreinish> tosky: you put an item in the agenda for this right?
17:27:30 <tosky> mtreinish: dmellado did it, he couldn't attent now
17:27:41 <tosky> so I can talk about it
17:27:44 <tosky> attend*
17:27:58 <tosky> if you don't have other points for that topic
17:28:20 <mtreinish> tosky: go ahead, you're on the agenda so you get to go first :)
17:28:59 <tosky> so briefly: this could be just a downstream (packaging) problem, but an opinion is highly appreciated
17:29:32 <tosky> in RDO we are starting to package test code in separate subpackages (usually service-tests), this include the code which uses the Tempest Plugin interface
17:30:02 <jordan__> (I have to go now, sorry. Opinion needed here: https://review.openstack.org/#/c/248355/)
17:30:15 <tosky> and what happens is that the entry point for the plugin interface, coming from setup.cfg, ends up in a common package with the code (like, python-<servicename>)
17:30:47 <mtreinish> jordan__: ok, no worries
17:30:59 <mtreinish> #link https://review.openstack.org/#/c/248355/
17:31:04 <tosky> if you have just that one, without <servicename>-tests, and you run testr list-tests (or tempest cli), then testr complains with an exception
17:31:24 <mtreinish> tosky: right because you added an entrypoint without any code for that path
17:31:42 <tosky> exactly: the exception was added explicitly: https://bugs.launchpad.net/tempest/+bug/1474765
17:31:43 <openstack> Launchpad bug 1474765 in tempest "Discovery ignores plugin errors" [Medium,Fix released] - Assigned to Marc Koderer (m-koderer)
17:31:43 <mtreinish> so the load_tests hook tries to import that code during discovery and it can't
17:32:52 <tosky> the possible annoyance is that if you just install openstack-tempest, you basically need to manually install all the packages with tests even if you don't need to use them, and having the tempest rpm depend on all the -tests package is a bit complicated to maintain
17:33:30 <tosky> so the question is whether you could suggest a solution, or if you think it's not a real problem (just documentation on our side), etc
17:33:49 <mtreinish> tosky: I'm not sure we want to change that. I think being explicit and failing when you can't use a plugin because it doesn't work for whatever reason is better
17:34:00 <tosky> yes, I understand that
17:34:23 <mtreinish> when you think about using this in any automated testing if you install a broken plugin and it just logs a warning and moves on you'll get successful results even though your tests didn't run
17:34:57 <mtreinish> I think we can try to make this more clear in the tempest docs and the runtime failure more obvious (if it's not already)
17:35:15 <mtreinish> but I think erring on the side of failure is the correct behavior
17:35:48 <tosky> yes, I guess there is no way to distinguish an intended failure (because the code is not installed by choice)
17:36:05 <tosky> and it's not possible to split the entry points in a subpackage
17:36:35 <tosky> okidoki, thanks for clarifying this
17:37:05 <mtreinish> tosky: well you could patch it out of sahara and add a separate setup.cfg to your tests package
17:37:22 <tosky> uhm , with just the entry point
17:37:45 <mtreinish> but this kinda of complexity is the other part of why I advocate doing tempest plugins in seperate projects/repos
17:37:45 <tosky> not sure it's possible with rpm, the splitting of packages is done at "installation" time
17:38:01 <tosky> I see, right
17:38:38 <mtreinish> tosky: yeah, I think you'd have to get a bit clever in the spec file. It would require moving a bunch of stuff around
17:38:47 <mtreinish> it's doable I think, but probably not worth it
17:39:15 <tosky> right
17:41:05 <mtreinish> ok, is there anything else to discuss on tempest this week?
17:42:23 <mtreinish> ok, then let's move on
17:42:41 <mtreinish> #topic Devstack + Grenade
17:42:52 <mtreinish> does anyone have anything to discuss on devstack or grenade this week?
17:43:03 <mtreinish> dtroyer, sdague: ?
17:43:11 <mtreinish> sc68cal: ?
17:43:49 <sc68cal> I think we have a patch up to do a DVR multinode grenade job
17:43:56 * sc68cal digs up link
17:44:04 <sc68cal> https://review.openstack.org/#/c/250215
17:44:17 <sc68cal> oh. It merged.
17:44:24 <mtreinish> sc68cal: heh
17:44:44 <mtreinish> well I was gonna a make a joke about the failure rate on that job, but I guess we can look it up now :)
17:45:09 <sc68cal> I am hoping to be pleasantly surprised
17:46:10 <mtreinish> ok, is there anything else on this topic?
17:46:16 <sc68cal> that's all I have so far. Devstack neutron refactor is still in progress. I'm at the point where I need to create the networks that tempest expects for running tests
17:46:31 <sc68cal> without making it as messy as neutron-legacy
17:47:29 <mtreinish> sc68cal: oh fun, I tried digging into the neutron-legacy side for that a month or so ago when we had a gate failure on tempest's network usage
17:47:54 <mtreinish> so I'm really looking forward to not staring at it for an hour and coming up with basically nothing to show for it :)
17:48:23 * sc68cal 's irc connection is lagging
17:48:37 <mtreinish> ok, lets move on
17:48:48 <mtreinish> #topic Critical Reviews
17:49:01 <mtreinish> does anyone have any open reviews they'd like to get extra eyes on?
17:50:17 <mtreinish> well I have 2 quick ones this week:
17:50:23 <mtreinish> #link https://review.openstack.org/287308
17:50:36 <mtreinish> #link https://review.openstack.org/283266
17:50:50 <mtreinish> although dims is the only other active +2 on that 2nd link
17:52:32 <mtreinish> ok, if there aren't any other reviews I'll open the floor
17:52:44 <mtreinish> #topic Open Discussion
17:52:59 <jlanoux> quick question: what is the status on nova network?
17:52:59 <mtreinish> in the last ~7min does anyone have a topic not on the agenda they'd like to discuss?
17:53:27 <mtreinish> jlanoux: like when is it going away?
17:53:58 <jlanoux> mtreinish: yes or is it ok to move nova network calls to neutron ones in Tempest?
17:54:23 <mtreinish> jlanoux: we need to support both for the forseeable future. It's not even deprecate yet
17:54:37 <jlanoux> mtreinish: okidoki
17:54:40 <mtreinish> and we also have to support the stable branches
17:54:52 <jlanoux> ok - thanks
17:54:57 <tosky> quick question (maybe) when a set of API tests is moved outside tempest to a project repository
17:55:05 <mtreinish> so even if it were to go away in newton (which would be 6months too fast) we couldn't get rid of it in tempest until mitaka-eol
17:55:25 <jlanoux> ok
17:55:35 <oomichi> ssh timeout thing.
17:55:42 <oomichi> can we discuss it now?
17:55:45 <tosky> do you suggest, or at least would you not be against, a procedure which involve importing the history of that code in the project repository, instead of a dump of the last version? Did anyone do it?
17:56:38 <mtreinish> tosky: you should use master tests of everything just venv isolate tempest and the plugin and run it against a stable service
17:56:51 <mtreinish> oomichi: we can, but there are only ~3min left :)
17:57:09 <tosky> mtreinish: it's not about running it, it's about the procedure of moving the code
17:57:15 <oomichi> the ssh timeout happens on scenario tests only without api tests now.
17:57:50 <oomichi> but as https://review.openstack.org/#/c/283795/ discussion, have we already seen the reason why that happens on scenario tests only?
17:57:55 <mtreinish> oomichi: we don't run ssh on api tests in the default configuration
17:58:04 <mtreinish> tosky: I'm not sure I understand what you're asking about
17:58:23 <oomichi> mtreinish: ah, I see the point.
17:58:34 <tosky> mtreinish: when you move the code, you usually just dump the last version of the moved code in a review; I'd like to keep the history instead
17:58:35 <jlanoux> oomichi: you can look at the non-voting ssh job for data
17:59:11 <oomichi> jlanoux: ah, thanks. will dig it later deeply
17:59:18 <mtreinish> tosky: oh, so editing history is never the right thing to do
17:59:32 <jlanoux> oomichi: np
17:59:36 <mtreinish> it messes everything up, and it doesn't work in the upstream system
17:59:37 <tosky> mtreinish: and that's not the thing I want to do :)
17:59:50 <oomichi> jlanoux: so the conculusion is just network problem, right?
17:59:54 <mtreinish> tosky well we've got 10 secs, so let's pick this up in -qa
17:59:57 <tosky> ack
18:00:05 <mtreinish> ok, we're at time
18:00:06 <oomichi> jlanoux: not ssh problem
18:00:07 <mtreinish> thanks everyone
18:00:10 <mtreinish> #endmeeting