17:02:09 #startmeeting OpenStack QA 17:02:10 Meeting started Thu Mar 14 17:02:09 2013 UTC. The chair is sdague. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:02:11 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 17:02:13 The meeting name has been set to 'openstack_qa' 17:02:24 ok, who's here for QA meeting? 17:02:30 I am 17:02:30 hi 17:02:31 o/ 17:02:31 here 17:02:32 here 17:02:45 here too 17:02:50 #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting 17:02:58 And sam who's looking over my shoulder 17:03:04 threw a couple of things up on the agenda for today 17:03:11 dwalleck: you should give him a whole computer :) 17:03:30 hi 17:03:30 maybe lets start with the easy one - point #3 17:03:35 Let's put the summit to last 17:03:46 #topic cutting stable branch 17:03:53 davidkranz: sounds good 17:04:06 We're finishing something up, hopefully he'll get his soon 17:04:10 so lets start off with the concrete one of when do we cut stable branch for grizzly 17:04:25 davidkranz: when did that happen last time? 17:04:30 doink, sorry for being late guys 17:04:36 sdague: I think not until late. 17:04:46 jaypipes: no worries 17:04:56 THe problem is that the projects are rolling with respect to this issue. 17:05:27 davidkranz: right, so the projects are in hard freeze now. And once we cut stable we could still fix issues on the stable branch 17:05:42 but maybe we should pick a date regardless? 17:06:16 sdague: ++ 17:06:21 THere are more useful grizzly tests going in. 17:06:32 THe danger of waiting is a havana change breaks us. 17:06:51 davidkranz: right, so how about we pick 1 week from now to branch stable 17:06:56 From a test standpoint I think it would be better to wait a bit. 17:07:05 no one should be landing anything funny in masters until after then 17:07:09 davidkranz: one week from now sounds good. 17:07:12 sdague: :) 17:07:22 and we can still let people propose changes to stable after it's cut 17:07:29 just review it with a more critical eye 17:07:29 So be it. 17:07:38 want an official vote on it? 17:07:42 sdague: ++, 1 week sounds fine to me too 17:07:55 #startvote stable cut for tempest is March 21st 17:07:56 Unable to parse vote topic and options. 17:08:03 #vote yes 17:08:11 ok, apparently I don't know how to make the vote work right 17:08:16 #vote yes 17:08:18 #vote yes 17:08:27 #vote yes anyway :) 17:08:31 sdague: need to include the valid vote options at end... 17:08:37 hehe 17:08:48 anyone *opposed* to one week from now? 17:08:50 ok, well you'll have to teach me some time :) 17:09:14 brb with some other folks 17:09:29 ok, we'll consider that one resolved 17:09:33 jaypipes: Who has the magic decoder ring to do it? 17:09:55 davidkranz: to do the cut? 17:09:59 Yes. 17:10:08 davidkranz: we just work with the CI team, IIRC 17:10:12 and maybe ttx 17:10:16 jaypipes: k 17:10:22 ok, I can take it as a todo to go figure out what we need to do 17:10:28 sdague: thx 17:10:38 #todo sdague to sort out process for stable branching on March 21 17:10:46 #action sdague to sort out process for stable branching on March 21 17:11:04 #topic Pending test for multiple server create adds 1 minute to gate. Is that OK? This is the "can't gate on everything issue". 17:11:24 that one should be a reasonably concrete discussoin 17:11:28 sdague: gotta link? 17:11:40 davidkranz: do you have the link to that review? 17:11:49 https://review.openstack.org/#/c/23511/ 17:11:53 #link https://review.openstack.org/#/c/23511/ 17:11:59 doh, jinx 17:12:09 This test should be part of a full suite, but seems overkill for a gate. 17:12:24 yeh, I'm on the fence. Seems like a good test to have, but I can also understand not wanting to increase the gate right now 17:12:27 I think it should not be a gated test 17:12:34 We are looking at death by 1000 cuts for gate time. 17:12:48 should go into a separate folder like - stress 17:13:02 ravikumar_hp: well I think it's a valid API test, so it should be in core tempest 17:13:14 ravikumar_hp: As wel will discuss later or at the summit, these tests should be part of tempest. 17:13:15 but it might be something we should flag as not run on normal gate jobs 17:13:24 But we need a way to only gate on a subset of tests. 17:13:38 anyway, I could go either way, davidkranz / jaypipes you guys have a pov on it 17:14:00 * jaypipes thinks if it is part of the official compute API, it needs to be in the gate. 17:14:02 How about we compromise for now and drop the xml? 17:14:26 jaypipes: I think it is an extension 17:14:31 let me double check 17:14:35 It is an extension. 17:15:02 hmm... 17:15:08 ok, so how about we hold it for now 17:15:18 so don't put it into stable 17:15:25 I don't see the test cases checking to see if the extension even is supported in the compute endpoint. 17:15:37 yeh, that's probably a bigger issue 17:15:44 if it's truly an extension, that is... 17:15:46 jaypipes: We don't do that for most of the "extensions" 17:15:47 which I don't know that we do right anywhere in tempest 17:15:55 davidkranz: touche 17:16:00 ok, so I think we're coming to the agreement to hold for now 17:16:05 That issue is really for the summit. 17:16:10 ya 17:16:11 and we can reopen discussion post summit / stable 17:16:15 I'll -2 it 17:16:16 sounds good. 17:16:55 sdague: next topic? 17:16:59 #topic Blockers / Bugs 17:17:16 any specific bugs or blockers or critical reviews we need the team to look at? 17:17:23 I think we are in pretty good shape unless some one has somlething specific. 17:17:28 (after this we'll dive into design summit topics) 17:17:45 I'll have some time to get through reviews today 17:17:45 sdague: Just one thing. 17:17:45 yeh, just wanted to give folks a moment to speak up 17:18:05 Should the keystone v2/v3 thing be handled in the same way as mtreinish did with images? 17:18:12 https://review.openstack.org/#/c/24013/ <-- rate limit handling is not good now 17:18:33 davidkranz: I think the multiple directory approach is good 17:18:39 so I'd be in favor of it 17:18:47 ravikumar_hp: Can you recommend that? 17:19:12 ravikumar_hp: Sorry to be such a pain about those submissions. 17:19:23 afazekas: ok, I'll take a look at that one 17:19:31 #link https://review.openstack.org/#/c/24013/ 17:19:43 davidkranz: sorry . I missed it. what you want me to do 17:19:44 #action need review eyes on https://review.openstack.org/#/c/24013/ to deal with rate limitting 17:19:57 afazekas: that code looks excellent. I'll review fully shortly. 17:20:10 ravikumar_hp: Model the v2/v3 identity tests the same way the glance tests are handling v1/v2 17:20:26 ravikumar_hp: You are involved in those tests, right? 17:20:30 davidkranz: sure. we are resubmitting 17:20:44 ravikumar_hp: Thanks! 17:20:45 ravikumar_hp: yeh, mtreinish provided a model on doing multiple versions of the api in tree for glance, just follow that lead 17:21:00 sdague:ok 17:21:22 ravikumar_hp: if there are any questions about it feel free to ping me too 17:21:28 ok, any other bugs/reviews that need attention? 17:21:35 mtreinish: odf course 17:21:45 sdague: https://review.openstack.org/#/c/24433/ 17:22:14 andreaf: cool, I'll take a look later today 17:22:28 thanks 17:22:31 ok, next topic, summit 17:22:37 #topic Design Summit Sessions 17:22:52 so there are already a number submitted at http://summit.openstack.org 17:23:07 also mtreinish and I were brainstorming others at https://etherpad.openstack.org/havana-qa-infra-summit-brainstorm 17:23:30 as a starting point for pruning down to what we think we really want to focus on there 17:24:38 I'd love some commentary other thoughts on the etherpad to figure out if we don't need the sessions (it's not really a discussion, we're good with an existing solution), or if there are other thoughts about things to add 17:25:24 sdague: I think it may take a few minutes for others to digest :) 17:25:31 yep, that's fair :) 17:25:41 I was hoping people were off reading :) 17:25:53 yes still reading. :-) 17:26:29 realize, it's a big brain dump, so think of it as scribble notes :) 17:26:36 but it seemed better to start from that 17:26:48 sdague: formal processes and best practice are +++ if they make it easier to contribute tests and reduce the review cycles for new contributors (like myself ^^) 17:27:02 andreaf: much agreed. 17:27:03 sdague: lot of topics 17:27:20 Many of these are already in summit submissions. 17:27:23 I like the idea in general of dashboards, especially for results/failures 17:27:27 But many are not. 17:27:46 davidkranz: yeh, I'm seeing that there is some overlap. Can you mark the dups in the etherpad? 17:27:53 One of the problems we've had internally as well is handling all the data we generate 17:28:01 sdague: On it. 17:28:04 sorry, I was running low on time, so I just did the brain dump but not the dup check 17:28:25 sdague: not sure if we can adddress all those in summit . we need to prioritze - 17:28:30 +1 the idea on moving towards working with dashboards, metrics, etc... 17:28:35 something that I'd like to see is whitebox tests: what is the future of that area, how to we support whitebox tests in a multinode cluster, do we want that to grow as part of tempest 17:29:07 andreaf: yeh, right the whitebox discussion was another good one to have 17:29:19 you want to either direct submit it, or put it on this list? 17:29:54 I'll put it on the list so people can comment on it 17:29:58 I'm not sure how I fee about Tempest only supporting API validation. That might be okay for gating, but it's going to miss a lot of important defects 17:30:29 dwalleck: right, that's the Tempest scope conversation, right? 17:30:41 I'd also like to start thinking about things like logstash --> NoSQL --> Elastic Search for dashboarding/mining automated test logs 17:30:55 dwalleck: Who suggested it should only support API validation? 17:31:08 Nearly all the defects I end up finding having nothing to do with the API. They almost all have to do with the artifacts of the API call, like the instance, it's networking, etc 17:31:32 sdague: right 17:31:39 dwalleck: I agree 17:31:56 this is more about just giving a guideline of what's in play and what's not for havana 17:32:01 sdague: i would to like test coverage process for new releases . right now it is adhoc based . we need to have coverage for all blueprints if applicable 17:32:10 +1 on discussing Tempest Scope in general and what that focus should be 17:32:43 +1 on most of our found defects involving artifacts generated by the API and the integration(s) between those artifacts 17:33:42 +1 on being able to run the whitebox-type tests against a multi-node environment 17:34:32 As a "newbie", I find the proposal to discuss foundational topics very useful: Best Practice Guide and Tempest Scope 17:34:35 And we should probably define better what we mean by white box. I think we may all be thinking slightly differently there 17:34:55 dwalleck: agreed. 17:35:10 mlavalle: Hello and +1 17:35:16 mlavalle : +1 ..It will be awesome if we can have a Tempest 101 17:35:19 yeh, whitebox is probably a topic by itself 17:35:19 mlavalle: +1 17:35:58 jaypipes: +1 on the mutli-node testing 17:36:24 ok, I guess maybe it's worth a point of order to figure out how we move from the artifacts and discussions we've got going now to a schedule. Do people want some more time poking at the etherpad? How do we want to prioritize the topic lists? 17:36:35 malini: I think that is what http://summit.openstack.org/cfp/details/87 is, at least in part. 17:37:22 davidkranz / jaypipes / dwalleck you guys did this before, so curious how that next step goes 17:37:31 sdague: Perhaps there should just be more summit session submissions and then we sort 17:37:39 sure 17:37:54 Thierrey said we will probably have 9-12 slots to be decided soon. 17:37:54 sdague: looking over the proposed QA sessions at http://summit.openstack.org/, the only really glaring omission I think is the whitebox testing... unless dwalleck's "Beyond the API - End to End Testing of OpenStack" is really all about whitebox testing? 17:38:12 dwalleck: ? 17:38:14 sdague: why don't we let people vote on their 3 top topics on the etherpad over the next few days? 17:38:15 ok, so I'd appreciate any markup in the etherpad to catch dupes 17:38:27 mlavalle: well, I think we need them in the proper tool 17:38:30 dwalleck: shall that be our whitebox testing session? 17:38:51 sdague: or whatever tool is apporpriate 17:39:03 jaypipes: I was referring to lower level instance validation, but I don't actually consider that to be white box 17:39:23 dwalleck: ok 17:39:28 But yeah, it seems like the right time to talk about that general topic 17:39:30 dwalleck: just checkin :) 17:40:00 ok, cool 17:40:04 cool :) 17:40:14 anything terrible on my list people want to throw out? 17:40:15 IMO, we should be moving towards tempest being a real upstream acceptance test. 17:40:29 With subsets for gating, etc. 17:40:46 sdague: no.... you proposed good topics. Thanks for the effort 17:40:47 right, well we can talk about that in the scope session 17:41:00 sdague: I'm not sure that http://summit.openstack.org/cfp/details/117 belongs in the QA track 17:41:32 jaypipes: Agreed. Nor design summit 101. 17:41:33 jaypipes: I would agree with you on tat 17:41:47 sdague: but there isn't an Ops track... 17:41:54 yeh, so the qa-core team probably need to scrub 17:42:07 davidkranz: yeah, loic's one should be in Process I think. 17:42:17 +1 on the 117 no really belonging in the QA track (at least without better details) :-) 17:42:31 jaypipes: I might well refuse it now. 17:42:39 RAXSam: yeah, it's a great session proposal acutally, but just not in QA :) 17:42:43 davidkranz / jaypipes: you guys have edit super powers on that? (I can help if you give me powers) 17:43:00 sdague: no, I can not review, only view 17:43:01 japypipes: Exactly...the topic is cool, just wrong slot. :-) 17:43:25 ttx: any chance of getting an Operators track at the summiit? 17:43:27 sdague: I think there is only one person which is me at the moment. 17:43:34 ok 17:43:36 all hail davidkranz 17:43:48 don't make him mad, or he'll kill you session! 17:43:50 jaypipes: You pushed it at me last time :) 17:44:04 jaypipes is the master of delegation :) 17:44:14 davidkranz: I am not complaining :) 17:44:50 OK, so we just need people to push new sessions based on the etherpad and then we can sort. 17:45:01 yep, think that's right. 17:45:03 We can have one more discussion of this next week I think. 17:45:09 ++ 17:45:13 Based on the actual submissions. 17:45:36 Hopefully we will now next week how many sessions we have. 17:45:43 ^^^ know 17:45:45 ok, so. I'll take the list I had, excepting dupes, and put them in 17:45:53 then we can prioritize next week 17:45:55 sdague: Thanks. 17:45:57 sdague: sounds good. 17:46:18 ok. the last agenda topic was on best practices. 17:46:21 The one question I had for now is whether the OPenStackTM thing should go in process or tempest. 17:46:30 I could see it either way. 17:46:40 davidkranz: Process, IMO. 17:46:46 yeh, Process 17:46:55 OK, we'll see if they accept it :) 17:47:03 #topic Guidelines for Tempest commits 17:47:07 last topic on agenda 17:47:17 davidkranz: you talking about http://summit.openstack.org/cfp/details/16 ? 17:47:48 jaypipes: No, the item in the etherpad about using tempest as a gate for openstack trademark. 17:48:02 davidkranz: k, the two are similar 17:48:10 davidkranz: might be worth combining. 17:48:17 davidkranz: check with mordred 17:48:24 yeh, that sounds reasonable 17:48:24 jaypipes: Yes, at the implementation level. 17:48:40 jaypipes: Will do. 17:48:42 davidkranz: you going to take the todo and circle with mordred / ttx on it? 17:48:51 sdague: Yes. 17:49:21 #action davidkranz to discuss with mordred / ttx on where tempest testing for openstack trademark should go 17:49:47 ok, last thing I wanted to make sure we brought up was just pointing lots of people (especially new folks) at - https://wiki.openstack.org/wiki/Gerrit_Workflow#Committing_Changes 17:49:55 ++ 17:50:01 having sane commit messages is really useful 17:50:14 and I've been -1ing a bunch of patches because they didn't have them 17:50:15 sdague: this one too: https://wiki.openstack.org/wiki/GitCommitMessages 17:50:38 ah, yeh, that one is better 17:50:42 #link https://wiki.openstack.org/wiki/GitCommitMessages 17:51:01 so if you are a new contributor, or have new contribs on your team, please go take a look at that 17:51:21 #topic open discussion 17:51:30 it's free for all time :) 17:51:34 anything else from folks 17:52:03 * afazekas if I can get passport in time I will be on the summit 17:52:12 afazekas: cool 17:52:35 awesome 17:53:28 anything else from folks? 17:53:34 I did not have time to follow-up anymore on enabling ssh tests, sorry about that - I'll pick it up again as soon as I can 17:53:45 going once... 17:53:52 going twice... 17:54:03 adios! 17:54:07 ok, lets call it. Thanks all 17:54:12 #endmeeting