17:02:48 <sdague> #startmeeting qa
17:02:49 <openstack> Meeting started Thu Sep 12 17:02:48 2013 UTC and is due to finish in 60 minutes.  The chair is sdague. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:02:50 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:02:52 <openstack> The meeting name has been set to 'qa'
17:03:00 <sdague> ok, who's around for the QA meeting?
17:03:02 <mkoderer> Hi folks :)
17:03:02 <mtreinish> hi
17:03:07 <afazekas> hi
17:03:16 <tkammer> hi
17:03:18 <sdague> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting - agenda
17:03:30 <fnaval_> hi
17:03:30 <sdague> ok, first topic
17:03:50 <sdague> #topic Welcome mkoderer and giulivo to tempest-core
17:03:52 <giulivo> :)
17:03:58 <mkoderer> :)
17:03:59 <afazekas> welcome
17:04:02 <jaypipes> afternoon
17:04:14 <sdague> thanks for all the great work guys, very happy to have you part of tempest-core
17:04:20 <mkoderer> thank you very much - I am very happy
17:04:27 <adalbas> hi
17:04:28 <ravikumar_hp> welcome mkoderer:
17:04:41 <mtreinish> yes, great work guys
17:04:46 <dkranz> indeed
17:04:47 <mkoderer> giulivo did such a great job, I am happy for him too
17:05:25 <sdague> you guys should have all the permissions you need now, if you have any questions on things feel free to just put it out on #openstack-qa
17:05:44 <mkoderer> I already have my first +A
17:05:45 <sdague> ok, next topic
17:05:48 <sdague> mkoderer: nice :)
17:05:57 <mkoderer> have = gave
17:06:04 <sdague> #topic Summit planning (sdague)
17:06:20 <sdague> so mostly I just wanted to set up some time frames here
17:06:49 <sdague> while the summit website is open for submissions, I don't intend to really have us look at them or start organizing until Oct
17:07:04 <sdague> so probably the first week of Oct we'll put summit content on the agenda
17:07:15 <ravikumar_hp> sdague: makes sense
17:07:37 <sdague> between now and then if interesting ideas come up that you think we should consider, please feel free to file it. But we'll hold until Oct to really make decisions there
17:07:37 <mtreinish> sdague: just curious how many slots do we have this time?
17:07:41 <sdague> we have 13
17:07:53 <sdague> they are spread out through the week in 3 blocks
17:08:10 <sdague> in a room where Infra will be, so we won't conflict this time
17:08:17 <dkranz> sdague: :)
17:08:43 <sdague> ttx created a more broken up schedule, which I think is really good
17:09:00 <dkranz> sdague: Is there a link to it somewhere?
17:09:33 <sdague> let me ping him to see if he has a public version of it, I just have a google spreadsheet link which I'm not sure he intended to be public
17:09:51 <sdague> I'll also see him next week, I'll ask him there
17:10:06 <sdague> ok, last administrative topic before content
17:10:11 <sdague> #topic Leader for next week's meeting - sdague on a plane back from LinuxCon during this time
17:10:20 * jaypipes sad he won't be at summit ;(
17:10:23 <jaypipes> :(
17:10:24 <sdague> can I get a volunteer to run the meeting next week
17:10:26 <jaypipes> no winky..
17:10:28 <sdague> jaypipes: bummer
17:10:32 <jaypipes> yeah...
17:10:35 <jaypipes> budget poop.
17:10:35 <mtreinish> sdague: I can do it again
17:10:51 <mkoderer> mtreinish: +1
17:10:53 <mtreinish> jaypipes: that sucks, I'm not sure I can go either
17:10:54 <sdague> #info mtreinish to run openstack-qa meeting next week
17:11:06 <sdague> ok, thanks mtreinish
17:11:11 * jaypipes will be there in spirit :)
17:11:13 <sdague> :)
17:11:46 <sdague> it actually brings up the question of an intersession qa meetup, which we should maybe consider later.
17:12:03 <jaypipes> ++
17:12:14 <sdague> but maybe we bring that up after we get through the rest of the agenda
17:12:22 <jaypipes> yes, sorry for tangent
17:12:27 <sdague> no worries :)
17:12:32 <sdague> #topic Blueprints (sdague)
17:12:49 <sdague> ok, I'm still catching up from being out, are there updates to blueprints people would like to report on?
17:13:08 <sdague> I'll try to get things cleaned up from LinuxCon next week from the show floor as well
17:13:19 <dkranz> sdague: The blueprint about client library stability has basically been implemented by mordred.
17:13:31 <dkranz> sdague: Not sure if it has merged yet.
17:13:37 <sdague> dkranz: cool, is it landed, or are there reviews that need eyes?
17:14:05 <dkranz> sdague: I think it is waiting on some other devstack change. I can dig up the patch link after the meeting.
17:14:16 <sdague> dkranz: ok, that would be great to make happen before Havana releases, so if we know of reviews that need eyes on that, please let me know
17:14:18 <sdague> great
17:14:21 <dkranz> sdague: But it was reviewed favorably
17:14:27 <sdague> any other blueprints?
17:14:28 <dkranz> sdague: Will do
17:15:02 <sdague> I will have to say that everyone owes mtreinish a beer over getting parallel tempest into the gate
17:15:10 <sdague> that was a huge win last week
17:15:26 <sdague> I don't think we'd have survived feature freeze day without it
17:15:31 <dkranz> sdague: Yes, very much so.
17:15:53 <dkranz> sdague: It is interesting that even that slight stress has uncovered a lot of things.
17:15:56 <sdague> It's pretty awesome that tempest tests now take only slightly longer than nova unit tests :)
17:16:06 <afazekas> +1 beer  to mtreinish :)
17:16:06 <jaypipes> ++mtreinish
17:16:17 <mtreinish> thanks everyone, I'll always accept free beers :)
17:16:24 <afazekas> :)
17:16:26 <joel-coffman> \quit
17:16:27 <mkoderer> :)
17:16:30 <sdague> yep, the test env has definitely started to act more like real systems, which is very good. shook out a bunch of new races.
17:16:48 <sdague> ok, anything else on blueprints?
17:17:05 <sdague> mlavalle: you around?
17:17:12 <mlavalle> sdague: yep
17:17:24 <sdague> #topic neutron testing status (mlavalle)
17:17:40 <sdague> ok, you have the floor, where do we stand on neutron?
17:17:51 <mlavalle> sdague: so, I've been working over the past few days in adding network isolation to enable parallel runs
17:17:55 <mlavalle> https://review.openstack.org/#/c/45578/
17:18:24 <sdague> great, lets get eyes on that review
17:18:45 <mlavalle> mtreinish: what do you think? are we good to test it?
17:18:55 <sdague> mtreinish, especially as you did the last refactor there, I'd like your opinion on it
17:19:14 <mtreinish> mlavalle, sdague: ok I'll take a look at the latest revision
17:19:18 <sdague> great, thanks
17:19:27 <mtreinish> testing it is fun because to enable it to actually run requires a devstack change
17:19:33 <mtreinish> which can't happen with this patch :)
17:19:42 <sdague> mtreinish: ok, what's the devstack change we need?
17:19:50 <sdague> and is that proposed anywhere?
17:19:56 <mtreinish> sdague: tenant isolation is disabled if neutron is enabled
17:20:06 <mtreinish> sdague: not yet I was going to put it together after the meeting
17:20:12 <sdague> mtreinish: ok, cool
17:20:26 <afazekas> mtreinish: we can add second change just for testing  based on this which reads the config in a different way :)
17:20:30 <sdague> maybe we need another non-voting job on neutron that sets that
17:21:05 <mtreinish> afazekas: yeah that would work too (we could just hard code it to true)
17:21:10 <sdague> much like we did with testr initially
17:21:36 <sdague> any volunteers to make the new devstack gate job for that?
17:21:57 <mtreinish> sdague: I'm not sure that's really needed (unless you're talking about a neutron parallel run)
17:22:03 <dkranz> sdague: The preferred way (by infra) to do this now is by using the experimental queue in zuul.
17:22:22 <sdague> mtreinish: that's what I mean, neutron parallel job
17:22:49 <dkranz> sdague: Instead of creating non-voting jobs
17:22:50 <sdague> dkranz: ok, I haven't looked at the experimental queue
17:22:54 <sdague> dkranz: ok
17:23:01 <dkranz> sdague: That's what I did for heat.
17:23:07 <sdague> I'll have to catch up on that bit, sounds cool
17:23:15 <sdague> ok, anything else on neutron testing?
17:23:26 <mlavalle> sdague: once we get that running, I'll get back to https://blueprints.launchpad.net/tempest/+spec/fix-gate-tempest-devstack-vm-quantum-full. We've made good progress fixing issues, so this weekend i'll send an email to the ML
17:23:35 <sdague> what are still our blocking issues on full gate
17:23:41 <sdague> ok, just what I was about to ask :)
17:23:46 <sdague> great
17:23:51 <dkranz> sdague: Some one is working to make all the tests on stable/grizzly work.
17:23:56 <mlavalle> with an assessment of how close we are to be done
17:24:08 <dkranz> sdague: Do we want to try to turn on the full gate there once that is done?
17:24:08 <sdague> #info mlavalle to send status update on full gate for neutron
17:24:29 <mlavalle> that's all I have
17:24:30 <sdague> dkranz: honestly, stable/grizzly is the past, I'm not interested in fixing it until master is fixed
17:24:30 <dkranz> sdague: I mean all neutron (quantum) tests
17:24:42 <dkranz> sdague: He is fixing both
17:24:44 <sdague> ok
17:24:57 <dkranz> The question is whether to turn on the full gate in grizzly once he is done.
17:25:18 <sdague> if it works, I think that's fine
17:25:38 <dkranz> sdague: OK, I'll tell him we would like to do that and to keep us posted on progress.
17:25:47 <sdague> sounds good
17:26:03 <sdague> ok, next topic
17:26:08 <sdague> #topic whitebox test removal (sdague)
17:26:27 <dkranz> sdague: Wasn't it done already :)
17:26:34 <mtreinish> sdague: https://review.openstack.org/#/c/46116/
17:26:37 <sdague> so I think we came to a general concensus on irc the other day to pull whitebox and add nova unit tests
17:26:54 <sdague> dkranz: well that's only 1/2 of it :)
17:27:16 <dkranz> sdague: Less than half, work-wise.
17:27:20 <sdague> because we still need a volunteer to actually do the unit test enhancements on nova so that we cover it
17:27:22 <afazekas> there is pending keystone whitebox test change on gerrit as well
17:27:52 <adalbas> sdague, i can volunteer to take a look on what is covered in nova.
17:27:59 <sdague> great
17:28:23 <sdague> #info adalbas to port whitebox tests into nova unit tests
17:28:31 <dkranz> Normally I would say wait to remove whitebox but it is breaking the nightly build.
17:28:51 <sdague> afazekas: so I think we saw with all the whitebox tests that what they were testing was doable in the unit tests by the projects
17:29:04 <mtreinish> dkranz: yeah that's why I removed it, because it doesn't do anything right now except generate errors
17:29:07 <sdague> and because the db schemas changed so fast, tempest was really not the best place for them
17:29:19 <sdague> enhance it closer to the projects
17:29:41 <dkranz> sdague: I think we all agreed on that.
17:29:44 <sdague> yep
17:30:08 <sdague> ok, so my suggestion is -2 any new whitebox patches that come in if you see them, on that kind of direction
17:30:21 <sdague> unless there are objections
17:30:32 <dkranz> sdague: Should send to the ml as well.
17:30:39 <sdague> yes, I'll send to ML
17:31:05 <sdague> #action sdague to send ML email about change in approach for whitebox testing
17:31:37 <sdague> ok, next topic
17:31:51 <sdague> mkollaro: you around/
17:32:06 <dkranz> sdague: She is sick today I just found out.
17:32:09 <sdague> ok
17:32:23 <sdague> so let's skip that and just do reviews, then open discussion
17:32:30 <sdague> #topic Critical Reviews (sdague)
17:32:41 <sdague> ok, critical outstanding reviews that we need eyes on?
17:32:49 <sdague> links appreciated
17:32:58 <dkranz> sdague: This is the client lib stability one https://review.openstack.org/#/c/41931/
17:33:17 <sdague> great, up in a tab for me, I'll check it later
17:33:22 <sdague> other ones?
17:33:27 <adalbas> this one from afazekas https://review.openstack.org/#/c/35165/
17:33:54 <adalbas> it might fix a lot of rechecks related to the parallel runs
17:34:09 <sdague> ok, great
17:34:36 <sdague> afazekas: did we land the client changes to support that already? (I remember reviewing it, but I don't remember if it merged)
17:34:58 <sdague> I'll review that later today as well
17:35:05 <sdague> and other eyes would be good
17:35:06 <afazekas> sdague: the xml version got an approval from you
17:35:12 <sdague> afazekas: great
17:35:34 <sdague> afazekas: yeh, I just couldn't remember if I was the first or second +2 :)
17:35:42 <afazekas> :)
17:36:03 <adalbas> i think it got merged
17:36:06 <sdague> other reviews people think we need extra eyes on?
17:36:46 <mkollaro> sdague: I'm here
17:37:00 <sdague> oh, hey, we can do your topic now
17:37:13 <sdague> #topic OpenStack qa repo for test plans that can be pointed at from blueprints (mkollaro)
17:37:13 <mkollaro> sure
17:37:18 <sdague> mkollaro: you have the floor
17:38:11 <mkollaro> we want to share the test plans for new features with upstream, I created a repo for them https://github.com/mkollaro/openstack-testplans
17:38:59 <mkollaro> I was wondering what it would take to get it put under some openstack managed group, so it wouldn't be just mine project and people from the community could contribute to it
17:39:08 <mkollaro> *my
17:39:35 <mkollaro> an example of a test plan is here https://github.com/mkollaro/openstack-testplans/blob/master/glance/multilocation.rst
17:40:02 <mkollaro> they would be linked in the blueprints
17:40:05 <dkranz> sdague: This was an attempt to address the issue of not enough info in blueprints
17:40:26 <sdague> so is there a reason not to use the wiki for these?
17:41:04 <sdague> I guess that's my only question, the plusses / minusses of doing this in git vs. the openstack wiki
17:41:38 <mkollaro> that would be another option...but AFAIK devels don't like to use the wikis too much and I'd like them to work with us around this more
17:42:10 <mkollaro> handling test plans as code is better IMHO, since it pretty much _is_ code
17:42:12 <sdague> I also understand that while the setup is manual, is the intent to turn the actual plan into something like a scenario test (post setup)
17:42:15 <ravikumar_hp> can we add in whiteborad of blueprint so that all in one place
17:42:43 <mkollaro> blueprints don't have formatting and nobody is using them
17:43:09 <mkollaro> it seems very hard to get devels to use the whiteboard in blueprints
17:43:43 <mkollaro> I hoped that the combo of git+rst text files would make contributions more likely
17:43:59 <mkoderer> is there a way to links such a test cases in a blueprint?
17:44:13 <sdague> mkollaro: I guess this raises the idea about whether if we enhanced some of the markup in the scenario tests if these could be just scenario tests in tempest, plus a preamble that explains the setup
17:44:55 <giulivo> mkollaro, is the point of this being tracking the existing cases or both the existing and wanted cases?
17:44:57 <mkollaro> mkoderer: AFAIK there is no special field for it, but you can paste the link on the whiteboard
17:44:59 <sdague> because our community is very much about automation, so I feel like just documents for test execution doesn't deliver on the idea of repeatability, though I definitely get the need to have more details
17:45:35 <mkoderer> mkollaro: ok sure a special field would be nice :)
17:46:06 <ravikumar_hp> sdague:+1
17:46:55 <rockyg> Newbie here. There may be easy to make wiki test cases executable but it would need to be a project
17:47:04 <mkollaro> sdague: I'm not sure if they should be used only for scenario tests and I'm primarily intending them as a means of communication, not documentation
17:47:29 <mkollaro> giulivo: not the existing ones, there are almost 1k test cases, that would be impossible
17:47:54 <sdague> mkollaro: the problem is test plans that aren't actually test code aren't things we can gate on
17:47:56 <giulivo> how is this: blueprints map ot plans, a blueprint also has a link to descriptive wiki, but cases are tracked in the blueprint todoitems?
17:48:08 <dkranz> I think what Sean is asking is why not just provide code that works, rather than pseudo-code instructions?
17:48:20 <sdague> dkranz: right exactly
17:48:25 <mkollaro> sdague: yes, but I'm afraid there is no way to do that :(
17:48:41 <sdague> I could imagine that document could actually be a scenario test, with extensive markup in it
17:48:52 <sdague> mkollaro: why not?
17:49:33 <mkollaro> before we get to executable code, we need to figure out a test plan...I spent hours of talking to the devel before I wrote that one I linked
17:49:56 <mkollaro> after we got code, the test plan can be pretty much removed as far as I care
17:50:01 <rockyg> Sdague: yes. That is what cucumber test tool does
17:50:40 <sdague> mkollaro: so if it's transient like that, then I think etherpad or wiki is better than git
17:50:55 <sdague> because then we can mark it as done once it's landed in code
17:51:10 <giulivo> that's why I wanted to use the todoitems
17:51:27 <mkollaro> the problem is, nobody really writes down some expected behavior of a feature or even some basic description...this could help with that
17:51:52 <rockyg> Mkollaro:why not submit plan as big to be executed and reviewed?
17:51:53 <sdague> rockyg: right, but we already have a test framework, so while I like cucumber in theory, I think in practice it's not the right direction for us right now
17:52:18 <rockyg> Sorry. Bug.
17:52:20 <sdague> mkollaro: so I think I understand the problem, but I'm not sure another git repo is going to solve it.
17:52:37 <mkollaro> sdague: wikis wouldn't be so easily parsable, there is this idea of creating some replacement of our (redhat) current test plan system or upload it there with some script
17:53:41 <sdague> mkollaro: ok, how would you expect this would become part of OpenStack validation eventually?
17:53:42 <mkollaro> this could be easily doable with the rst format, if we ever want that
17:53:56 <mkollaro> sdague: what do you mean by validation?
17:54:16 <sdague> the gate, or other tools we make available to people to verify openstack is working
17:54:41 <mkollaro> sdague: I never meant this as a validation thing, more like a communication channel
17:55:29 <dkranz> sdague: I think what we want here is a way to extract information from a developer and "add it to the blueprint" in some other way than literally doing that.
17:56:02 <sdague> ok, so I feel like in OpenStack our culture is that we communicate with code, which is why we don't just have HACKING.rst, but actually executable hacking checks
17:56:25 <sdague> I think this is probably a good design summit topic
17:56:36 <mkollaro> sdague: yeah, but we need to figure out first how to write the test, typically together with the devel
17:56:39 <dkranz> sdague: Yes, but if I talk to a dev and take notes, code does not come out right away. ANd some one else might right the actual code.
17:56:53 <dkranz> write
17:57:34 <dkranz> We are not going to solve this in our remaining two minutes
17:57:40 <sdague> agreed
17:58:04 <sdague> ok, so should we take it to the mailing list? or do we want to let this simmer until summit?
17:58:17 <mkollaro> but in case I decided to use wikis, where could I put the test plan?
17:58:40 <sdague> mkollaro: we'd make a namespace in the OpenStack wiki for them.
17:59:09 <dkranz> sdague: Can any one do that?
17:59:12 <sdague> so in our last 2 minutes, mostly I want to figure out where we want to continue the conversation
17:59:17 <sdague> dkranz: yep
17:59:25 <mkollaro> I don't really see why a wiki would be an improvement over git+rst, but if you really prefer that, sure
18:00:16 <sdague> mkollaro / dkranz: want to take this to the ML?
18:00:28 <rockyg> How about proposal on ml to flesh out topic for summit?
18:00:48 <dkranz> sdague: OK, we'll think about it.
18:01:19 <sdague> cool, sounds good. whatever forum you want to, I think there are interesting ideas here, just need to explore it more
18:01:26 <sdague> ok, with that we're over, so I'll end up things
18:01:30 <sdague> #endingmeeting
18:01:35 <sdague> #endmeeting