17:01:16 #startmeeting qa 17:01:17 Meeting started Thu May 16 17:01:16 2013 UTC. The chair is sdague. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:01:18 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 17:01:21 The meeting name has been set to 'qa' 17:01:26 sdague: I was thinking more along Voltron lines... :) 17:01:34 ok, who's here for QA meeting 17:01:37 jaypipes: awesome :) 17:01:38 o/ 17:01:38 I am 17:01:43 * tkammer waves 17:01:58 davidkranz is away... 17:02:15 hi 17:02:16 appologies for not pulling an agenda together in advance, was down in NYC at a Linux Foundation event the last couple of days 17:02:22 no worries. 17:02:24 so here's my thoughts on agenda 17:02:35 1) blueprints 17:02:49 2) any critical reviews/bugs to look at 17:02:52 3) open meeting 17:03:02 #topic blueprints for H1 17:03:36 #link https://launchpad.net/tempest/+milestone/havana-1 17:04:00 for me, the launchpad cleanup is coming along good. will finally complete that tomorrow 17:04:10 great job. 17:04:17 +1 17:04:19 the tempest restructure is about 1/3 in, 1/3 in review, and 1/3 to go 17:04:29 currently blocked on this volumes race we uncovered 17:04:43 sdague: yeah, I saw that. anything I can do to help? 17:04:52 which giulivo and jgriffith are helping get to the bottom of 17:05:00 I've got it sdague 17:05:06 jaypipes: right now, I don't think so. we've got all the right folks on it 17:05:08 giulivo: awesome 17:05:13 excellent. 17:05:14 the scheduler updates the volume status every minute, which is too long for the tests execution 17:05:21 ah 17:05:22 giulivo: nice find 17:05:40 giulivo: so do we have a way to tune that for gate? 17:05:48 Is Bagashree Lokare here, by any chance? 17:05:49 look for "Received volume service update" in the sched log 17:05:55 not sure, probably it is configurable behaviour 17:06:07 haven't checked yet 17:06:13 ok, cool. well I'll count that as problem sorted, we just need to get the right fix together 17:06:16 nice job on that 17:06:34 any other blueprints people are working for for H1 that they want to report in on? 17:07:34 ok... I'll take that as a no :) 17:07:59 next week we'll plan to do rollcall on that, because any non completed (or close ones) we'll need to push to h2 17:08:00 IMHO https://blueprints.launchpad.net/tempest/+spec/ssh-auth-strategy this should be approved 17:08:04 sdague: hoping that Shree will set a priority and status on her 5 blueprints 17:08:31 afazekas: sure 17:08:38 https://blueprints.launchpad.net/tempest/+spec/add-basic-heat-tests this too 17:09:02 afazekas: can you ping that author to but the detailed specification in the wiki instead of etherpad? 17:09:09 etherpads are too easy to get borked 17:09:24 then I'll happily move it to an approved state 17:09:26 afazekas: ++ on ssh auth strategy. But I would love to see the blueprint title by more descriptive of the actual task. As it sounds, it seems like the blueprint is about the Keystone endpoints, but it's about SSHing into a VM (whitebox testing) 17:09:36 RAX-Sam: hola. 17:09:47 RAX-Sam: wish Daryl a happy birthday from us. :) 17:09:48 Hey Jay 17:09:51 belated... 17:10:00 afazekas: I'm good with https://blueprints.launchpad.net/tempest/+spec/add-basic-heat-tests as well 17:10:03 sdague: I'll ask him 17:10:05 :-D He is here just having trouble connecting to Freenode... :-) 17:10:28 afazekas: thanks 17:10:28 ah :) 17:10:44 ok, any other blueprint updates? 17:11:20 ok, next topic 17:11:31 #topic critical reviews 17:11:56 any reviews we need to jump on that aren't getting enough eyes right now? 17:12:12 doh had two blueprints to ask about: https://blueprints.launchpad.net/tempest/+spec/add-snapshot-tests 17:12:21 and https://blueprints.launchpad.net/tempest/+spec/set-gate-attribute 17:12:39 sdague: yeah, I also still have a few bluepriont-related things.. 17:12:41 the second one looks tricky as it gets into the labeling 'discussion' 17:12:56 jaypipes: oops, sorry for jumping the gun :) 17:13:03 no worries :) 17:13:04 #topic blueprints 17:13:30 sdague: https://blueprints.launchpad.net/tempest/+spec/add-scenario-tests seems to be overlapped with your in-progress https://blueprints.launchpad.net/tempest/+spec/tempest-repo-restructure 17:13:31 giulivo: I think we have enough rought concensus on the gate attribute for now. If we need to make a change in the future, we can 17:13:47 jaypipes: partially 17:13:56 I think that blueprint was for some new scenario tests 17:13:56 so how to use the smoke flag ? 17:14:07 Do we need services related flags ? 17:14:09 sdague: and https://blueprints.launchpad.net/tempest/+spec/quantum-quota-basic-tests and https://blueprints.launchpad.net/tempest/+spec/quantum-quota-extension-test seem to also be identical. 17:14:23 jaypipes: yes, there are massively duplicated quantum ones 17:14:33 sdague: on the scenario tests, then, I would like to see more details on the former. 17:14:37 mlavalle, were you going to try to condense those 17:15:04 sdague: Yes, I was too busy this week, moving my family from Houston to San Antonio 17:15:24 sdague: But I will follow up with this 17:15:25 jaypipes: I'm happy with that as feedback to the author on the scenario tests before it moves to approved, would you like to provide it? 17:15:28 mlavalle: thanks 17:15:58 sdague: OK to approve https://blueprints.launchpad.net/tempest/+spec/add-logging-configuration? It's in code review now... 17:16:12 jaypipes: +1 on that 17:16:15 sdague: yes on scenario. 17:16:22 will provide feedback on whiteboard. 17:16:43 #action jaypipes to provide feedback on https://blueprints.launchpad.net/tempest/+spec/add-scenario-tests 17:16:50 great, thanks 17:17:00 we lost one of giulivo's 17:17:08 sdague: logging config BP approved. 17:17:08 #link https://blueprints.launchpad.net/tempest/+spec/add-snapshot-tests 17:17:13 jaypipes: thanks! 17:17:38 I'm good with approving https://blueprints.launchpad.net/tempest/+spec/add-snapshot-tests 17:17:47 any objections? 17:17:57 looking... 17:18:00 it is "slow" 17:18:14 giulivo: we may have to not put the 'gate' tag on it 17:18:16 I think this could be experimental for the labeling thing 17:18:23 yeah indeed 17:18:30 sdague: I renamed it to add-volume-snapshot-tests 17:18:36 sdague: need to be specific :) 17:18:38 jaypipes: great 17:18:44 sdague: and +1 from me. 17:18:45 yes, that's goodness 17:18:45 giulivo: did you turned of the secure delete option ? 17:18:51 ahh, better :-) I was confused there for a sec too 17:18:53 IIRC, giulivo is already pretty much done with that 17:19:00 afazekas: secure delete is off in the gate 17:19:26 without secure delete it is not too slow AFAIK 17:19:45 giulivo, sdague: k, updated status of add-volume-snapshot-tests 17:19:53 jaypipes: great, thanks 17:20:09 other blueprint issues .... ? 17:20:26 well, just the Shree ones, but doesn't look like Ravi is here to provide feedback 17:20:31 so no :) 17:21:16 sdague: giulivo I'm looking at adding a forced check of capacity update on your race condition 17:21:24 ok, I'll take a todo to send out an email asking about H1 status by the end of the week 17:21:50 #action sdague to email openstack-qa to get remaining updates on blueprint (especially quantum ones) this week 17:22:07 jgriffith: nice, thanks 17:22:37 ok... so probably enough on the blueprints front 17:22:43 ya, sounds like it. 17:22:45 #topic outstanding important reviews 17:23:03 now is the time to pimp reviews that need more eyes 17:23:49 anyone have items? 17:24:26 https://review.openstack.org/#/c/28505/ 17:24:37 I did a run down through the review queue earlier in the week, and hit my opinions on most of them. The gate tags I still need to take a look on, but I'll do that after we get through the cinder bug 17:25:16 afazekas: ok, I'll take a look at that 17:25:41 sdague: the cinder is probably configuration issue about the periodic tasks (and with the jitter config) 17:26:00 any other reviews? 17:26:11 * jaypipes actually had some time this week to do some reviews \o/ will do some more today. 17:26:11 going once.... 17:26:18 jaypipes: awesome 17:26:27 #topic open discussion 17:26:42 ok, any other topics people want to bring up? 17:26:49 this gate tag... 17:26:57 fire away 17:27:06 could you give us a quick summary of the discussion so far and any decisions made? 17:27:39 so... since grizzly rc1 our tempest run time has grown from 35 - 40 mins to 45 - 50 mins 17:27:48 and continues to grow as new tests come in 17:28:08 yes, noted. 17:28:20 we've gotten grumbles and pushback from some of the nova team about the gate getting too long to merge complicated changesets 17:28:28 which might have rebase conflicts 17:28:46 so a knob that we could have, is make an explicit gate tag 17:28:55 sdague: sorry, how does having a rebase conflict have to do with tempest runtime? 17:29:04 can we run multiple nosetests process in the same time ? 17:29:26 jaypipes: because when you have a 5 patch series in zuul 17:29:33 competing with other people's 5 patch series 17:30:08 afazekas: You should be able to 17:30:08 anything that causes a gate reset, be it a bad merge, a flakey test, a flakey thing in infra, gets compounded 17:30:17 sdague: k, understood. 17:30:20 it's O(1) in theory, but not in practice 17:30:27 gotcha 17:30:39 it also means that time to first feedback on a patch, for reviewers to get to it quick, is long 17:30:50 If you programatically loaded the test groups and spin up nosetest processes for each module, you'd get some pretty good relief 17:31:02 sdague: so the solution to this is to improve the runtime using parallelization (poor man's or otherwise), no? 17:31:03 afazekas: there are lots of other approaches 17:31:21 jaypipes: parallel clearly gets us wins 17:31:23 It is the lowest cost approach 17:31:28 dwalleck: ah, hi there :) happy belated b-day! 17:31:48 jaypipes: thanks man! 17:31:53 I guess the question is will parallel, poor man's or otherwise, ever guaruntee that we stay ahead of desired test growth? 17:32:23 sdague: so, I've recommended this a couple times, but it seems to me splitting the gate into an XML and a JSON run would give us approximately a 50% reduction in runtime 17:32:33 jaypipes: well, it wouldn't :) 17:32:38 it will help 17:32:39 sdague: and that seems like it would be very easy to do. 17:32:52 jaypipes: we have other than just nova tests 17:33:07 I'm totally cool with anyone else running after any other approaches 17:33:08 afazekas: sure, but big chunks of time are in nova compute tests :) 17:33:15 jaypipes: less than you think :) 17:33:23 We should split in competent and/or directory bases 17:33:42 the gate tag was something we could get to quickyl 17:33:42 sdague: my concern with the gate tag is that it means one more thing we need to keep track of -- kind of like skips. 17:33:52 jaypipes: sure 17:34:01 long term it just opens a question 17:34:11 but I suppose things have gotten to the point where something needs to be done ASAP. 17:34:20 so I will support any solutions, like you said. 17:34:24 do we restrict content in tempest to what can run in a gate time limit 17:34:54 I like the idea of having another knob so we can say it doesn't need to be 17:35:00 sure. 17:35:16 that, combined with a full daily run 17:35:22 giulivo: exactly 17:35:37 because I do believe that any code in tempest does need to run at least once a day 17:35:43 otherwise it bit rots 17:35:53 eg: the old stress tests 17:35:59 maybe at that point we could even turn the know to 15mins 17:36:08 s/know/knob/ 17:36:12 agreed, sdague 17:36:13 giulivo: right, but we can make that decision later 17:36:25 right now we don't have a tool to even debate that 17:36:36 anyway, so the gate tag seemed prudent and easy 17:36:59 however if someone else wants to sign up for solving it a different way by H1, I'll back off of promoting it 17:37:08 but we are talking about needing a solution for H1 17:37:18 still maybe there should be some agreement on when to use smoke on a testcase? 17:37:20 because we're running long right now 17:37:33 giulivo: yes, we need some time auditing things 17:37:53 oh so setting two barriers you mean 17:38:14 my proposal is gate tag now. Spin up periodic jobs again for full. (all H1) 17:38:14 sdague: understood. I would volunteer to work on a the XML/JSON split, but unfortunately, I cannot promise to make the 5/30 deadline. 17:38:24 sdague: sounds like a good plan for H1 17:38:30 I think that parallelization (poor man's or otherwise) will help short term, but it doesn't solve a long term problem. Ultimately I figure it should be a multi-part solution. I.E. Poor man's parallelization, split XML & JSON, maybe carve up further into multiple jobs (gate job, daily job, etc...) being very careful about tests that create expensive resources, etc... 17:38:31 then push for real parallel for H2 with testr 17:38:35 sdague: Somehow we should document what covered by which test case in order to select correctly test cases for a shorter run 17:38:56 then see what other optimizations we can make for H3 to get as much in the gate as we can 17:39:11 RAX-Sam: agreed. my poor man's parallel == Split XML and JSON runs ;) 17:39:17 afazekas: you volunteering? 17:39:20 really poor man ;) 17:39:23 heh 17:39:33 afazekas, ++ ! 17:39:38 hey, poor man's parallel is fine, I just haven't seen volunteers for it :) 17:40:07 sdague: I will give a stab at PMP with XML/JSON 17:40:12 jaypipes: awesome 17:40:16 thanks! 17:40:18 sdague: #action me. 17:40:34 I don't suppose it will be any code patches to tempest itself... just devstack-gate 17:40:37 #action jaypipes to do PiMP XML / JSON tests 17:40:41 :) 17:40:42 hehe 17:41:32 jaypipes: actually, now that the d-g running is in tox, you'll need to do it in tempest + zuul config 17:41:40 that's actually probably something to fyi folks on 17:41:57 sdague:after I can figure out what is the normal format of doing that, yes. But first I should add same doc stings to the base classes 17:42:10 devstack-gate is now just calling tox -e'....' into tempest to run tests 17:42:19 so we don't have to patch d-g when we move things around 17:42:27 sdague: no problem. I have enough experience with that in th epast couple weeks doing the chef cookbook stuff ;) 17:42:46 currently we have -esmoke and -efull 17:42:58 afazekas, docstrings in the base classes for which purpose? 17:43:00 jaypipes: the 'challenge' is the proper log out output 17:43:02 jaypipes: coolio 17:43:15 afazekas: I wouldn' 17:43:21 t be changing any log output... 17:43:42 yeh, it would be separate jobs 17:43:47 giulivo: documenting the functions used by the test cases 17:44:02 ok, additional topics? 17:45:15 going once... 17:45:29 going twice... 17:45:33 http://lists.openstack.org/pipermail/openstack-qa/2013-February/000219.html 17:45:34 sold. 17:45:59 afazekas: suggestion... 17:46:24 afazekas: that email is likely too long to have an effective conversation about all the points. I would recommend splitting into separate emails for each point. 17:46:43 jaypipes: ok 17:46:53 afazekas: possibly putting that original email in an etherpad and referring to it from the followup single-topic emails. 17:46:54 afazekas: yeh, we've gotten active on our ML recently, so I think if it came in new in chunks, we'd get a good discussion 17:47:02 ++ 17:47:20 afazekas: because they are all good points, just got lost in the shuffle too easily in a big email like that 17:47:39 I see :) 17:47:39 ok, with that I think I'll call it a meeting. We can take additional topics to #openstack-qa or the mailing list 17:47:44 #endmeeting