17:00:24 <davidkranz> #startmeeting qa
17:00:25 <openstack> Meeting started Thu Mar 21 17:00:24 2013 UTC.  The chair is davidkranz. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:25 <donaldngo_hp> hi
17:00:26 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:28 <openstack> The meeting name has been set to 'qa'
17:00:41 <davidkranz> agenda at https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Proposed_Agenda_for_Mar_21_2013_meeting
17:00:59 <davidkranz> Jay is on a plane.
17:01:28 <davidkranz> Any one from quantum here?
17:01:42 <dwalleck> mlavalle is
17:01:44 <mlavalle> davidkranz: yes, me
17:02:14 <davidkranz> mlavalle: Given that we are at RC, what does it meant that the quantum tests are failing?
17:02:56 <mlavalle> davidkranz: I would need to take a look at the tests failling
17:03:02 <zyluo> davidkranz, do you have a jenkins log link of a failure?
17:03:11 <mlavalle> can we do that between today and tomorrow?
17:03:11 <davidkranz> ONe sec..
17:03:37 <davidkranz> http://logs.openstack.org/24917/2/check/gate-tempest-devstack-vm-quantum-full/942/console.html
17:03:46 <sdague> quantum full has never worked
17:03:52 <mlavalle> davidkranz: taking a look
17:03:54 <davidkranz> I noticed this because I ran devstack/quantum today for the first time.
17:04:18 <davidkranz> sdague: I know, but we are supposed to be at release. Are the tests wrong?
17:04:23 <sdague> which is why we only gate on quantum smoke
17:04:43 <sdague> davidkranz: part of it is that quantum doesn't do everything that nova networks does (I think)
17:05:00 <zyluo> 56.414 | ERROR: No server with a name or ID of 'ex-vol-inst' exists.
17:05:12 <zyluo> Does this line have any thing with the error?
17:05:17 <davidkranz> sdague: But these are quantum test, right? And we are at release.
17:05:37 <davidkranz> So either the tests are wrong or quantum is broken.
17:05:44 <sdague> davidkranz: not exactly
17:06:01 <sdague> we always run the same tempest tests on all the full runs
17:06:14 <sdague> quantum means quantum was setup by devstack instead of nova-network
17:06:29 <sdague> it doesn't mean only quantum tests were run
17:06:50 <davidkranz> sdague: I understand. But I get the same errors running vanilla quantum/devstack.
17:07:13 <davidkranz> sdague: ANd it is the quantum tests that are failing.
17:07:25 <sdague> davidkranz: oh, well that's different
17:07:28 <davidkranz> sdague: The failing tests are not even run in the devstack-vm-full
17:07:35 <sdague> I thought it was compute tests
17:07:36 <davidkranz> sdague: Only in the quantum full.
17:07:47 <mlavalle> davidkranz: the Quantum test that I see failling in the log is tempest.tests.network.test_networks.NetworksTest
17:08:17 <sdague> davidkranz: can we handle this by making it a high priority bug on quantum, and engaging the PTL on it?
17:08:50 <davidkranz> sdague: Yes.
17:09:24 <davidkranz> mlavalle: Are we sure these tests are supposed to work?
17:09:51 <mlavalle> davidkranz: that test is not supposed to work in its current state
17:10:16 <mlavalle> davidkranz: because it asumes v1 of the Quantum API, which is not in the codebase anymore
17:10:22 <davidkranz> mlavalle: Are there any tests that are failing that are supposed to work?
17:10:40 <davidkranz> mlavalle: It will be unfortunate if we cut stable/grizzly with zero quantum coverage.
17:10:43 <ravikumar_hp> mlavalle: is the tests tempest.tests.network.test_networks.NetworksTest worked earlier?
17:11:07 <mlavalle> davidkranz: It never worked under v2 of the API
17:11:28 <mlavalle> davidkranz: I have a patchset that I pushed a couple of weeks ago
17:11:38 <mlavalle> that addresses this issue
17:12:03 <sdague> mlavalle: review?
17:12:05 <mlavalle> davidkranz: it was abandoned by Gerrit for lack of review
17:12:42 <mlavalle> sdague: I can resubmit and have the Quantum PTL review it and some of you
17:12:55 <davidkranz> mlavalle: Can you take this to the quantum team and get a solution?
17:13:38 <mlavalle> davidkranz: yes, I will and will resubmit the patchset
17:13:53 <sdague> mlavalle: please make sure there is an issue tracking it as well
17:14:15 <mlavalle> sdague: I also documented the bug a few weeks ago
17:14:39 <mlavalle> give me a minute and will give you the bug
17:14:59 <sdague> mlavalle: if it was this change, that still didn't pass quantum full - https://review.openstack.org/#/c/22927/
17:15:30 <mlavalle> sdague: yes it was that patchset
17:15:59 <mlavalle> you gave a -1, because you asked me whether version 1 had also to be supported
17:16:15 <mlavalle> I responded that v1 wasn't in the codebase anymore
17:17:35 <sdague> mlavalle: ok, sorry about that, we've had some email delay issues
17:17:50 <sdague> but it doesn't explain why it still didn't let quantum full pass
17:18:17 <mlavalle> sdague: so here's what I propose. I resubmit the patchset, and have it reviewed by the Quantum PTL and some of you
17:18:20 <sdague> if the api change is the only issue
17:18:23 <sdague> yep
17:18:32 <sdague> lets see if we can get to quantum full passing as well
17:18:38 <davidkranz> mlavalle: I propose we table this for now so we can discuss the summit and we can resolve this after that, or outside of this meeting.
17:19:04 <mlavalle> davidkranz: perfect
17:19:17 <davidkranz> I think we should wait for these tests before cutting stable/grizzly if it will be soon.
17:19:47 <sdague> davidkranz: sure
17:19:59 <sdague> soon really means the end of this week though
17:20:07 <sdague> because havana is now open on most trees
17:20:21 <sdague> which means we might need to start taking destabalizing tempest patches to support havana
17:20:34 <davidkranz> sdague: Yeah. I guess we can backport the quantum stuff.
17:20:47 <sdague> I think everything except ceilo is in release branches now (keystone went this morning)
17:20:55 <mlavalle> davidkranz: I can work today and tomorow on this
17:20:57 <davidkranz> sdague: How about we cut it tomorrow.
17:21:15 <davidkranz> sdague: Are you going to give the go-ahead to ci to make the branch?
17:21:24 <andreaf> hi sorry I'm late
17:22:19 <mtreinish> davidkranz: I thought that we were cutting it today?
17:22:33 <sdague> davidkranz: sure we can do it tomorrow
17:22:41 <davidkranz> mtreinish: Did you look at the above discussion?
17:22:49 <sdague> ttx needs to do the actual tag for us, and I need to sort out ci switch over
17:22:58 <sdague> so I'll take the todo to make it happen tomorrow
17:23:07 <davidkranz> sdague: Great. Thanks.
17:23:17 <davidkranz> #topic havana summit
17:23:34 <davidkranz> https://etherpad.openstack.org/havana-qa-infra-summit-brainstorm
17:24:01 <davidkranz> I added the list of sessions that were submitted at the bottom with some comments and recommendations.
17:24:37 <davidkranz> If any one objects to my "Y" or "N" please say so.
17:25:07 <mtreinish> davidkranz: the Y's and N's look fine to me.
17:25:10 <ravikumar_hp> davidkranz: what is ?
17:25:21 <mtreinish> Although, I'm not sure there is that much left to talk about with the multiple api versions
17:25:22 <ravikumar_hp> davidkranz: ?        Tempest - Gap Analysis - Identify new testsdevelop Ravikumar Venkatesan
17:25:32 <davidkranz> ravikumar_hp: That is for discussion. THere is not room for all of them.
17:25:48 <sdague> davidkranz: I agree with all your Y & N on there
17:26:18 <davidkranz> mtreinish: There is the major issue of what principles we use to decide which tests to run, and which are part of the gate.
17:26:34 <ravikumar_hp> i think whitebox may be developer scope
17:26:36 <davidkranz> mtreinish: In most cases the new api will have a lot that was also in the old api.
17:26:59 <davidkranz> mtreinish: But the implementation may be different.
17:27:09 <davidkranz> And we can't keep everything in the gate forever.
17:27:17 <davidkranz> So I still think there are issues to discuss.
17:27:17 <ravikumar_hp> why tempest should take care of whitebox?
17:27:28 <mtreinish> davidkranz: very true, I guess I was just thinking about it from the implementation side
17:27:45 <dwalleck> I think its because there's some differing in wording about whitebox testing
17:27:49 <davidkranz> mtreinish: I'll put this session in a place that is better if it does not take the full time.
17:28:00 <sdague> davidkranz: so there is probably an infrastructure topic we want to do about optimizing what's run in gate that might be the multi api version bits
17:28:07 <dwalleck> I know jaypipes would call logging into instances and doing validation whitebox
17:28:09 <andreaf> ravikumar_hp: we cannot provide good coverage only via API driven tests
17:28:33 <sdague> yeh, whitebox testing session should probably go in as well
17:28:39 <ravikumar_hp> andreaf: core services has whitebox tests
17:28:39 <dwalleck> But for me that's critical path testing. I can't release/deploy without testing the things I actually create
17:28:43 <sdague> does someone want to own that
17:28:48 <davidkranz> sdague: You mean combining it with Strategies for Gating in a growing project?
17:28:52 <dwalleck> I will
17:28:53 <Rackspace-Sam> I agree with adreaf, an api only test is not good coverage
17:29:05 <dwalleck> or Sam and I will :)
17:29:06 <sdague> davidkranz: possibly
17:29:17 <patelna_> we need revisit what is our charter and what tests will be developed, run where? Devstack is not a solution
17:29:25 <Rackspace-Sam> Yeah, We can own that one if it's ok with everyone. :-)
17:29:31 <Rackspace-Sam> the whitebox one I mean.
17:29:53 <sdague> Rackspace-Sam: cool
17:30:40 <sdague> patelna_: I think there are plenty of places to discuss that in our scope conversations
17:30:57 <davidkranz> Does any one have a strong opinion about the "?" cases to include or reject?
17:31:12 <sdague> though honestly, the last couple of summits have lots of people complain about stuff like that, promiss to do things, and never show up with code. so I'm not actually sure it's a productive conversation to have.
17:31:15 <patelna_> agree...also revisit nightly test driven environment
17:31:22 <davidkranz> Also, at the last summit we had some QA group discussion outside of the scheduled summit sessions.
17:31:27 <davidkranz> I presume we will do that again.
17:31:34 <ravikumar_hp> davidkranz: I thnik Reliability and Scalability Testing  - to be Y
17:31:35 <davidkranz> So there will be time for other discussions.
17:31:37 <sdague> yeh, stuff like that we can take into uncoference
17:31:53 <sdague> ravikumar_hp: I'm N on that
17:32:12 <davidkranz> sdague: And we have the beer session as well :)
17:32:15 <sdague> because I don't know how it would generate work we would do
17:32:30 <Rackspace-Sam> I like reliability as an unconference as well. :-)
17:32:34 <dwalleck> yeah, I can do that as an unconference
17:32:44 <patelna_> let's plan to have unconference session ...discuss nightly process and multi-node config as the stacks have grown now
17:32:52 <dwalleck> I have some fun stuff I can show that might be helpful :)
17:32:56 <sdague> remember, the point of design summit sessions is that every 6 months we get a chance to get together and discuss how to plan out the last 6 months
17:33:03 <sdague> sorry, next 6 months
17:34:06 <davidkranz> I think Monty's session (the last one) could cover some of this as well.
17:34:14 <sdague> yep, that one is good
17:34:24 <sdague> I think gap analysis should move to Y
17:34:57 <sdague> and I like the metrics and statistics one as well
17:34:59 <Rackspace-Sam> Gap analysis is cool, but maybe an unconference topic too?
17:35:23 <sdague> Rackspace-Sam: well, how many did we decide to push to uncoference? :)
17:35:39 <Rackspace-Sam> LOL good point. :-)
17:35:57 <dwalleck> sdague: Yeah, I'm biased, but I think there's a lot of interest info we can be gathering
17:36:01 <sdague> the way I think about the split is "unconference" tempest sessions should be "this is a neat idea, though I don't know how we'd address it in Havana in a systematic way"
17:36:12 <davidkranz> We have six "?" and room for three in the schedule
17:36:40 <sdague> davidkranz: is that after adding white box?
17:36:59 <davidkranz> No. That has to be submitteed as a session.
17:37:02 <sdague> ok
17:37:21 <sdague> and what's with line 182
17:37:27 <sdague> that just has your name with a Y :)
17:37:50 <ravikumar_hp> free spech by david
17:37:53 <Rackspace-Sam> LOL
17:37:55 <ravikumar_hp> speech
17:37:58 <davidkranz> Don't know how that happened
17:38:50 <mtreinish> davidkranz: wasn't that the strategies for gating line?
17:38:51 <sdague> davidkranz: so how many slots do we get again? just making sure the Ys all line up
17:39:08 <davidkranz> mtreinish: Yes.
17:39:15 <sdague> ok, that's important
17:39:19 <sdague> let's get that back in
17:39:26 <davidkranz> mtreinish: I cut instead of copy when I pasted that above.
17:39:58 <sdague> what is the clients bit?
17:40:01 <davidkranz> I just reverted
17:40:11 <davidkranz> sdague: We have 12 slots total
17:40:20 <sdague> also, I feel like we should circle around with Dean on grenade
17:40:35 <sdague> as I'd like to see that rolled in more
17:40:57 <davidkranz> There are now 12 Y
17:41:26 <sdague> dwalleck: what was your proposal on stats?
17:42:09 <dwalleck> sdague: The possibility of instrumenting the tests to gather timing/build time/etc further metrics for analysis
17:42:10 <sdague> to understand if that's something we summit time for
17:42:12 <dwalleck> In short
17:42:18 <davidkranz> Now we have 13
17:42:28 <sdague> dwalleck: so actually, can we just agree yes
17:42:39 <sdague> and take the conversation of how to the ML?
17:43:00 <davidkranz> That sounds reasonable
17:43:21 <dwalleck> sounds good
17:43:49 <sdague> so if we take off metrics
17:43:59 <davidkranz> mtreinish: OK, we will fold API versions to the general gating discussion.
17:44:03 <sdague> I'd like to hold one slot to try to get dean in on grenade
17:44:07 <sdague> for upgrade testing in the gate
17:44:19 <davidkranz> sdague: OK
17:44:37 <mtreinish> davidkranz: ok sure that should be fine
17:44:40 <sdague> then I think we've got 12 (and we can juggle later if we need to)
17:44:50 <Rackspace-Sam> I think its a good plan
17:45:14 <sdague> and do the performance one as unconference
17:45:19 <davidkranz> sdague: Right now there are 11.
17:45:27 <ravikumar_hp> sdague: looks good. all interesting and relevant topics
17:45:34 <davidkranz> sdague: Which do you propose changing to Y
17:45:40 <davidkranz> sdague: OK
17:45:57 <davidkranz> sdague: Are you going to submit that session?
17:46:07 <davidkranz> Who is going to submit whitebox?
17:46:10 <sdague> davidkranz: I'll circle with dtroyer to get him to submit it
17:46:16 <davidkranz> sdague: k
17:46:20 <sdague> I think Rackspace-Sam said he would do whitebox
17:46:38 <davidkranz> Rackspace-Sam: Can you do that after the meeting?
17:46:39 <Rackspace-Sam> Yeah. I'll submit whitebox testing/testing beyond the API. :-)
17:46:45 <Rackspace-Sam> absolutely
17:46:50 <davidkranz> Rackspace-Sam: Thanks.
17:47:00 <davidkranz> Anything else on the summit schedule?
17:47:16 <sdague> only and FYI
17:47:30 <sdague> I figured I'd do a Tempest 101 session as an early unconference
17:47:52 <davidkranz> sdague: Sounds good.
17:47:53 <sdague> davidkranz: also, don't schedule any of my talks for 5:20 on Tues, because I'm giving a User Summit talk :)
17:48:08 <davidkranz> sdague: Noted.
17:48:20 <davidkranz> Any one else with any kind of scheduling issue?
17:48:40 <Rackspace-Sam> davidkranz: It's already submitted. It's called Beyond the API, End-2-End testing of OpenStack. Do you want us to submit it with a different title to be more distinct?
17:48:50 <davidkranz> I will make sure our 3 overlap slots with process are not of mutual intersest.
17:48:55 <davidkranz> Rackspace-Sam: Not necessary.
17:49:20 <Rackspace-Sam> cool
17:49:29 <davidkranz> #topic Open Discussion
17:49:57 <Rackspace-Sam> Excellent. I had a bit of an announcement/update for the community if I could. :-)
17:50:16 <davidkranz> Rackspace-Sam: Please
17:50:30 <Rackspace-Sam> Cool. Thanx David
17:50:56 <Rackspace-Sam> we have been working really hard on how to share back with the community a lot of what we have been doing internally for testing in our pipeline after tempest has gated OpenStack
17:51:26 <Rackspace-Sam> and been talking a lot with Monty and the OpenStack CI team about getting Rackspace and others Ci closer to the open CI pipe
17:52:17 <Rackspace-Sam> and we are planning on open sourcing our full internal functional testing engine/test suites that we use in our pipeline. We will be placing this on stack forge early next week
17:52:31 <Rackspace-Sam> and will follow up with a heads up on the ML
17:52:31 <sdague> Rackspace-Sam: cool
17:52:38 <davidkranz> Rackspace-Sam: That's great!
17:52:48 <mtreinish> Rackspace-Sam: very cool
17:52:48 <Shree-HP> davidkranz> in the last few meeting, we discussed some tests on Multinode config testing especially with Quantum. Are we planning discuss this as part of unconf topic? that or some more on fuzzy logic for negative testing or test automation for build readability in Jenkins. etc?
17:52:58 <andreaf> Rackspace-Sam: great thanks!
17:53:21 <davidkranz> Shree-HP: I think this is covered by one of the session topics.
17:53:44 <mtreinish> one quick think from me: on https://review.openstack.org/#/c/24805/ cyeoh suggested only using one format for bugs in skip messages
17:53:59 <sdague> Rackspace-Sam: you guys want to do an Infra session at summit on that?
17:54:03 <mtreinish> I was wondering if people had preferences on the format (or with should stick with the multiple formats like we have now)
17:54:04 <patelna_> @Rackspace-Sam - good to hear your contributions
17:54:17 <Rackspace-Sam> sdague: Yes. We would love to a session if everyone is interested
17:54:18 <sdague> it sort of sits 1/2 way between QA and Infra
17:54:35 <sdague> but I'd like a discussion regardless to figure out how we can integrate it
17:54:45 <Rackspace-Sam> Exactly
17:54:47 <sdague> even if we need to drop a QA topic we just agreed to :)
17:54:49 <davidkranz> Shree-HP: Maybe " FITS testing of public clouds Monty Taylor "
17:54:51 <patelna_> cool request
17:55:26 <Shree-HP> davidkranz> thanks will check with Monty
17:55:32 <davidkranz> sdague: Maybe we can get Process to take it?
17:55:36 <Rackspace-Sam> our goal really is just to share what we have been working on and get teh community concensus on where it might fit in the open, what the best use of hte code base is, etc...
17:55:48 <davidkranz> Shree-HP: Otherwise unconference is fine.
17:56:00 <sdague> Rackspace-Sam: well submit, if we need to fiddle we can
17:56:22 <patelna_> we like to hear from Rackspace-Sam on what he has done
17:56:38 <Rackspace-Sam> ifSo you want us to submit a topic specifically on our internal test engine?
17:56:51 <sdague> Rackspace-Sam: well in what you guys are putting to stackforge
17:57:08 <sdague> so that we can figure out the right ways to pull parts in
17:57:17 <davidkranz> Rackspace-Sam: Doing a walkthrough could be really helpful as an overview.
17:57:22 <Rackspace-Sam> We will have several of our core test frameworks devs with us at the conference
17:57:52 <patelna_> +1
17:58:09 <sdague> davidkranz: if we needed to drop a session to fit this, Avoiding duplication between unit and tempest tests David Kranz   is probably the one to drop, because I doubt we'll have that many project devs in the room to help with it
17:58:14 <davidkranz> Rackspace-Sam: I will beg THierry for another slot.
17:58:41 <sdague> and that was mostly about education I thought back to the projects
17:58:48 <Rackspace-Sam> We could actually use our engine that we are planning on dropping at stack forge as a case study in part of our topics. Whatever you guys think is best. :-)
17:59:14 <ravikumar_hp> Rackspace-Sam: what is the timeframe
17:59:19 <sdague> Rackspace-Sam: yeh, submit a session, we can fiddle with the calendar later
17:59:33 <Shree-HP> Rackspace-Sam> I am mainly interested in the negative fuzz logic for testing negative tests and the multinode tests if u have that at rackspace
17:59:45 <Rackspace-Sam> cool. I'll submit something about our frameworks Case Study
17:59:47 <sdague> dammit, too much good stuff to discuss :)
17:59:57 <Shree-HP> we are developing something similar at HP and comparing the approach will be a good excercise
18:00:28 <ravikumar_hp> Rackspace-Sam: can we discuss in unconference session
18:00:41 <Rackspace-Sam> We have a good bit of various tests, the framework is plug-in and extension based, we will be releasing the engine, the OpenStack impl of that engine and a sub-set of the tests initially. We will then be releasing all of our OpenStack related test cases in the framework before the summit happens.
18:01:38 <Shree-HP> Rackspace-Sam> looking forward to it. thanks
18:01:47 <sdague> ok, got to run to another meeting now
18:01:48 <davidkranz> I think we are out of time....
18:01:54 <sdague> see you guys in -qa
18:01:58 <davidkranz> Thanks every one. Good meeting.
18:02:01 <sdague> where more folks should hang out
18:02:02 <davidkranz> #endmeeting