17:00:51 <mtreinish> #startmeeting qa
17:00:52 <openstack> Meeting started Thu Aug 22 17:00:51 2013 UTC and is due to finish in 60 minutes.  The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:53 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:56 <openstack> The meeting name has been set to 'qa'
17:01:10 <mtreinish> hi who's around for the meeting?
17:01:17 <mlavalle> I am
17:01:30 <afazekas> hi
17:01:33 <tkammer> hi
17:01:50 <mtreinish> ok let's get started then
17:01:53 <mtreinish> here's the agenda:
17:01:56 <mtreinish> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting
17:02:14 <mtreinish> So i'm up first on the agenda
17:02:20 <mtreinish> #topic testr-status
17:02:43 <mtreinish> So I've been working through some of the bugs on parallel tempest
17:02:49 <mtreinish> we're in the home stretch
17:03:02 <mtreinish> right now all that's left are the 3 bugs from last week
17:03:10 <mtreinish> and neutron doesn't work at all in parallel right now
17:03:22 <mtreinish> but once I sort out the neutron fail we might be able to greenlight this
17:03:54 <mtreinish> if we hit too many random fails we can alway just pull it out and go back to serial
17:04:08 <afazekas> Yes there is many nova-neutron   "race" issue ..
17:04:23 <mtreinish> but we've pretty much reached the point where this won't go any further unless we start gating on it
17:05:00 <mtreinish> afazekas: well right now we don't run with tenant isolation on neutron
17:05:18 <mtreinish> and also the neutron smoke run hits resource name conflicts and fails
17:05:47 <afazekas> mtreinish: I would be very surprised if those would be tenant isolation related things .. (may be some of them)
17:06:19 <mtreinish> afazekas: I don't think the resource conflict is a tenant isolation thing. But we need it for the non neutron specific tests to pass
17:06:37 <afazekas> mtreinish:  yes
17:07:06 <mtreinish> ok does anyone else have anything else about testr parallel?
17:07:45 <mtreinish> ok then before I move on I was just curious how many people are running with parallel?
17:07:49 <mtreinish> or at least tried it
17:08:00 <afazekas> mtreinish: sooner or later I try to split the test to more devstack node, can I add tox entry for that if it is necessary ?
17:08:34 <mtreinish> afazekas: I'm not following. Split which tests?
17:09:32 <afazekas> AFAIK the subunit is able to distribute the test cases to multiple node,  sooner or later I will try is it working with tempest
17:09:51 <dkranz> Here. Sorry I was in the wrong channel :(
17:10:08 <afazekas> I think it will require multiple similar system to be installed
17:10:26 <mtreinish> oh, yeah that's something we can look into down the road, but for right now we need to concentrate on getting parallel single node working
17:10:43 <afazekas> ok
17:11:13 <mtreinish> ok then does anyone have anything else to add on testr parallel?
17:12:24 <mtreinish> ok then moving on.
17:12:29 <mtreinish> #topic stress test status
17:12:41 <mtreinish> mkoderer told me he can't make today's meeting
17:12:53 <mtreinish> so does anyone else have something they want to bring up about stress tests
17:12:57 <mtreinish> otherwise we'll move on
17:14:22 <mtreinish> ok then let's move on
17:14:26 <mlavalle> mtreinish: I want to report on https://blueprints.launchpad.net/tempest/+spec/fix-gate-tempest-devstack-vm-quantum-full
17:14:35 <mtreinish> #topic Other blueprint status
17:14:41 <mtreinish> mlavalle: ok go ahead
17:14:51 <mtreinish> we really should add neutron testing status to the agenda for next week
17:15:10 <mlavalle> mtreinish: achieving progress. Right now I am working on this bug https://bugs.launchpad.net/tempest/+bug/1182384
17:15:12 <uvirtbot> Launchpad bug 1182384 in nova "SecurityGroups Tests : invalid id must be valid uuid " [High,In progress]
17:15:26 <afazekas> #action Add the neutron status to the next meeting Agenda
17:15:40 <mlavalle> last night I added a comment to this bug. I would like feedback from the team on that
17:16:02 <mlavalle> it doesn't have to be in this meeting, but I need it to move ahead
17:16:47 <mtreinish> mlavalle: ok looking at the bug it's more of the response code differences between nova network and neutron
17:16:55 <mtreinish> or is there something more too it?
17:17:20 <mlavalle> no, it is the response codes.
17:18:22 <mtreinish> mlavalle: ok we can move this on to the qa channel after the meeting because I think there is a more detailed discussion to be had about this
17:18:37 <sileht> /17
17:18:37 <mlavalle> that's what I want
17:18:59 <mtreinish> mlavalle: :)
17:19:06 <mtreinish> ok is there anything else on neutron?
17:19:16 <mlavalle> not on my side
17:19:44 <mtreinish> ok then does anyone else have a blueprint they'd like to bring up
17:19:53 <mtreinish> #link https://blueprints.launchpad.net/tempest/havana
17:21:11 <mtreinish> ok let's go to the next topic then
17:21:15 <mtreinish> #topic critical review
17:21:25 <mtreinish> Does anyone have any reviews that they would like to bring up
17:22:33 <dkranz> mtreinish: It would be nice to get https://review.openstack.org/#/c/36367/ in
17:22:40 <afazekas> In the heat topic I have 3
17:22:50 <dkranz> mtreinish: Heat stuff waiting for +A
17:23:07 <dkranz> OK, wait for next topic.
17:23:09 <mtreinish> dkranz: ok I'll take a look after the meeting
17:23:23 <mtreinish> I really don't have much experience with heat though
17:23:38 <afazekas> https://review.openstack.org/#/c/36367/ this is good according to the jenkins
17:23:55 <afazekas> https://review.openstack.org/#/c/36367/ this will run with new job
17:24:11 <mtreinish> afazekas: that doesn't get run with jenkins right now
17:24:20 <mtreinish> because we exclude the slow tag
17:24:41 <afazekas> there is two change set
17:25:02 <dkranz> mtreinish: I don't either.
17:25:07 <mtreinish> afazekas: you pasted the same link as dkranz twice
17:25:08 <afazekas> the auto-scaling is not running as it submintted now
17:25:21 <afazekas> sorry copy paste issue :)
17:25:34 <afazekas> https://review.openstack.org/#/c/34866/
17:26:25 <mtreinish> well this segways into the next topic, so before that does anyone else have any non-heat related reviews to bring up?
17:27:45 <mtreinish> ok lets move on to the heat topic
17:27:48 <mtreinish> #topic Heat test merge
17:27:54 <mtreinish> afazekas: take it away
17:28:40 <afazekas> So there is 2 very old heat rlated changeset which is not merged since long, and did not get negative response as well
17:29:09 <mtreinish> afazekas: those are the 2 you pasted before?
17:29:43 <afazekas> yes
17:29:59 <afazekas> and there is tox related change from dkranz
17:30:24 <dkranz> mtreinish: I am going to mark that last one not WIP.
17:30:37 <mtreinish> afazekas: well I'd like to see dkranz get in before anything that uses the slow tag
17:30:53 <dkranz> mtreinish: The issue was where we put the slow heat tests to be recognized .
17:30:57 <mtreinish> so we can add a nonvoting (at first) slow tag job on the check queue
17:30:58 <afazekas> dkranz: Imho you should remove the '\.orchestration' from the regexp
17:31:12 <dkranz> mtreinish: I am going to push all the stuff for the slow heat job tomorrow.
17:31:25 <mtreinish> dkranz: ok
17:31:29 <dkranz> afazekas: But then we will get non-heat tests that happen to be marked slow
17:31:39 <mtreinish> dkranz: do we have any of those?
17:32:04 <dkranz> mtreinish: Not yet, but we don't want the slow heat job to start picking up new stuff by accident.
17:32:22 <dkranz> mtreinish: I think we should put the slow heat scenario tests in their own subdir of scenario.
17:32:29 <afazekas> dkranz: will be on the slow job, I am not sure we want to prevent this in this way. Now non of the other test cases has the slow tag .
17:32:51 <mtreinish> well, I'm just wondering even if we have non slow heat jobs do we really want that to be a separate run
17:33:05 <mtreinish> sorry, slow non-heat tests
17:33:20 <afazekas> dkranz: the auto-scaling test case not submited into the scenario/orchestration
17:33:25 <dkranz> mtreinish: It is a problem to have a job that runs all slow tests that happen to show up.
17:33:40 <dkranz> afazekas: I know. I'm suggesting we change that.
17:33:51 <dkranz> afazekas: sbaker was OK with that.
17:34:16 <dkranz> afazekas: And then leave the regexp as is.
17:34:27 <dkranz> I can't see any harm in that.
17:34:29 <mtreinish> dkranz: ok, I'm fine with splitting it out as heat specific. We can always revisit the discussion when/if we have a non-heat slow test
17:34:33 <afazekas> dkranz: for me both cases is ok, I just would like to see some progress
17:34:42 <dkranz> And it dosn't leave bombs for the future.
17:34:56 <mtreinish> dkranz: the other thing I suggest is that we default it to be parallel
17:35:08 <mtreinish> I don't think we should add new jobs that aren't parallel
17:35:12 <dkranz> mtreinish: OK.
17:35:34 <dkranz> mtreinish: I'm still a slight bit concerned that tests only have to pass once in parallel to become part of the gate.
17:35:56 <dkranz> mtreinish: But I guess that's where we are at this point.
17:36:28 <mtreinish> dkranz: yeah, I mean once we move everything to parallel we'll have 3 runs of parallel
17:36:36 <mtreinish> but it's an issue
17:36:59 <mtreinish> which is why I want to get things merged soon(liek end of this week, first thing next week) for parallel
17:37:37 <dkranz> mtreinish: OK, I will change my regexp off of WIP, submit the slow test job stuff, and get sbaker to move the test to orchestration subdir.
17:37:51 <mtreinish> dkranz: ok sounds good
17:37:55 <afazekas> dkranz: I am also concerned about this kind of issues does not gets higher priority https://bugs.launchpad.net/tempest/+bug/1205344
17:37:56 <dkranz> mtreinish: There is already a slow job in api so it will run
17:37:57 <uvirtbot> Launchpad bug 1205344 in nova "mkfs error in test_stamp_pattern" [High,Confirmed]
17:38:41 <mtreinish> ok is there anything else to talk about on this topic?
17:39:09 <dkranz> mtreinish: Don't think so.
17:39:15 <mtreinish> ok then let's move on
17:39:18 <mtreinish> #topic Devstack independent tempest usage
17:39:23 <mtreinish> afazekas: this one is yours too
17:39:38 <dkranz> afazekas: Is this Tal's script?
17:39:45 <dkranz> afazekas: The topic?
17:40:18 <afazekas> Long time ago we discussed would be nice to add something for help use tempest for non-gating, with a not devstack installed openstack
17:40:28 <afazekas> dkranz: yes
17:41:07 <afazekas> Tal stated a script for helping https://review.openstack.org/#/c/42920/
17:41:37 <afazekas> I would like craftily some details how to move forward
17:41:38 <tkammer> s/Tal/tkammer :)
17:41:44 <dkranz> afazekas: Why did you +1 instead of +2?
17:42:12 <afazekas> dkranz: I just wanted to discuss it first :)
17:42:24 <mtreinish> afazekas: so I'm confused by the script it takes a tempest.conf to configure openstack? Isn't that the opposite of what we talked about before?
17:43:11 <afazekas> mtreinish: it creates users for testing
17:43:34 <afazekas> mtreinish: it will get options for more use cases
17:43:47 <mtreinish> tempest already does that with tenant isolation
17:44:15 <afazekas> mtreinish: does it in every case now ?
17:44:28 <tkammer> mtreinish, it uses both the tempest.conf file for "default" parameters such as the username/password (and more in the future if needed), and for other stuff it updates that conf file for later use of Tempest itself
17:44:48 <mtreinish> afazekas: everything but keystone which needs some work
17:45:57 <afazekas> mtreinish: the script has options, and can be extended with more automation, or make optional some unwanted behavior
17:46:48 <mtreinish> afazekas: ok, I'll need to take a more detailed look to get a better idea of what it's doing.
17:46:58 <afazekas> mtreinish: So the question is , is the script in this form  got for merge and extending with futures when it is needed, or it should do little more ?
17:47:43 <mtreinish> afazekas: well, I'm not exactly sure what it's doing now so I can't really answer that.
17:48:23 <mtreinish> how about we defer this till next week so everyone can get a chance to look at it in detail?
17:48:32 <mtreinish> (or maybe just me :) )
17:48:47 <afazekas> mtreinish: I think it should do just basic things first and evolve
17:49:03 <afazekas> mtreinish: ok :)
17:49:51 <mtreinish> #action review https://review.openstack.org/#/c/42920/ for discussion during next week's meeting
17:49:58 <mtreinish> ok then let's move on
17:50:08 <mtreinish> #topic Adding test plans for new features into the tempest repo
17:50:16 <mtreinish> mkollaro: this is you
17:50:54 <mkollaro> so, I was wondering if we could add test plans for the new features in some nice .rst or .md format to the tempest repo
17:51:11 <mkollaro> would that be a problem?
17:51:18 <mtreinish> mkollaro: what do you mean by test plans?
17:51:50 <mkollaro> mtreinish: test #1, upload file to swift; test #2, remove file from swift
17:52:29 <mtreinish> mkollaro: so you mean human readable descriptions of the tests?
17:52:32 <mkollaro> mtreinish: we are currently writing these kinds of test plans for the havanna release and we want to share them on upstream
17:52:51 <notmyname> mkollaro: hopefully a little more detailed than that :-)
17:52:51 <mkollaro> mtreinish: yes, but only of new stuff (I wouldn't want to document those ~800 existing tests)
17:52:58 <mkollaro> notmyname: of course :D
17:53:09 <mtreinish> mkollaro: oh, so if it's only new tests then I'd say a blueprint is the right place for it
17:53:28 <mkollaro> mtreinish: yeah, but the interface to them sucks
17:53:32 <mtreinish> you can link the blueprint to a wiki page and include as much detail as you want on that
17:53:56 <mkollaro> mtreinish: git has a history and it's easy to work with multiple people
17:54:11 <mtreinish> mkollaro: so do wikis
17:54:47 <mkollaro> well, if you can recommend some place, sure...but simple .md files could be parsed and put into some systems that would make my manager happy :D
17:54:55 <mkollaro> wikis are a bit harder to parse
17:55:36 <mkollaro> test plans are just pseudo code, it's better to use git for them
17:55:48 <mtreinish> that's fair, but our procedure for this kind of plan documentation is blueprints and if more detail or collaboration is needed we put it on the wiki
17:56:02 <mtreinish> but you can link to a git repo with your plan in .md
17:56:18 <mkollaro> mtreinish: what procedure? I barely ever see any test plans written in upstream
17:56:19 <mtreinish> I'm just don't think this kind of thing should live in the tempest repo
17:56:31 <mkollaro> mtreinish: nobody wants to use that and so nobody uses that
17:57:11 <mkollaro> mtreinish: yeah, if tempest doesn't want it we'll put it into a separate github repo, but I guess it would be nicer to have it in tempest
17:57:19 <mtreinish> mkollaro: so we're almost out of time (~3min) can we take this discussion to the -qa channel after the meeting
17:57:26 <mkollaro> ok
17:57:41 <mtreinish> #topic open discussion
17:57:53 <mtreinish> so in the last ~2min does anyone have anything else?
17:58:26 <dkranz> mtreinish: Nothing from me.
17:58:41 * afazekas nothing from my side, I will not be here after the meeting
17:59:22 <mtreinish> ok then I guess we'll end it here today
17:59:25 <mtreinish> thanks everyone
17:59:27 <mtreinish> #endmeeting