17:00:18 <jaypipes> #startmeeting qa
17:00:19 <openstack> Meeting started Thu Jan 24 17:00:18 2013 UTC.  The chair is jaypipes. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:20 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:22 <openstack> The meeting name has been set to 'qa'
17:00:33 <afrittoli> hi
17:00:44 <mtreinish> <-- here
17:00:52 <jaypipes> afazekas, davidkranz, mtreinish, sdague, afrittoli: morning
17:01:01 <jaypipes> mlavalle: morning miguel
17:01:19 <jaypipes> jeblair, clarkb, mordred: morning to you guys too :)
17:01:24 <mlavalle> jaypipes: good morning to everybody
17:01:28 <koolhead17> morning jaypipes :)
17:01:30 * afazekas is here
17:01:31 <davidkranz> jaypipes: Morning, barely
17:01:35 <jaypipes> koolhead17: mornin!
17:01:39 <jaypipes> davidkranz: :)
17:01:40 <afrittoli> morning everyone
17:02:18 <jaypipes> OK, so shall we start with a status report of open reviews?
17:02:25 <jaypipes> #topic Open reviews
17:02:39 <jaypipes> #link https://review.openstack.org/#q,status:open+project:openstack/tempest,n,z
17:03:20 <jaypipes> mtreinish: let's start with https://review.openstack.org/#/c/20042/
17:03:24 <jaypipes> dwalleck: mornin.
17:03:38 <mtreinish> jaypipes: ok
17:03:49 <dwalleck> jaypipes: Heya. Sorry, I'm back from the wormhole I've been stuck in
17:03:58 <jaypipes> mtreinish: I know there's been some discussion with a number of folks about the xml/json split
17:03:59 <davidkranz> dwalleck: Hi Daryl!
17:04:06 <jaypipes> dwalleck: no worries :)
17:04:20 <mtreinish> I should probably abandon that since the whole xml split discussion
17:04:28 <jaypipes> mtreinish: including the email between sdague, jeblair and yourself
17:04:43 <jaypipes> mtreinish: want to summarize the conclusion/decision on that?
17:04:48 * jaypipes curious
17:05:02 <mtreinish> jaypipes: I don't think we've reached a conclusion yet. :)
17:05:03 <davidkranz> jaypipes: I think there are a few issues along the lines of "should we wait for testr" or do something simple now for speedup.
17:05:50 <mtreinish> davidkranz: yeah that seems to be the consensus. We also don't have any data on how much of a speedup this gives.
17:05:50 <jaypipes> davidkranz: I see. and what are people's thoughts? wait for testr or move forward with something?
17:06:00 <sdague> sorry, mostly not here because of other meetings, but I'll throw in a few
17:06:01 <davidkranz> It's hard to answer that given the uncertainty about when testr wil be on line for real with no flakies.
17:06:19 <mtreinish> I think it would help to know how fast we need to get full to use it for gate
17:06:28 <sdague> if jeblair is ok with gating at the current time running, I'd say turn on the gate now
17:06:42 <sdague> and it will get better with testr
17:06:44 <dwalleck> I'm out of sync, but I don't see why splitting it now would hurt.
17:07:21 <sdague> testr for g-3 is still the plan
17:07:21 <mtreinish> dwalleck: jeblair said that since we would be duplicating jenkins jobs it puts too much strain on the ci resources
17:07:24 <dwalleck> Just my opinion, but I don't see the need to run the XML job on each run. That's something I have configured to run daily in a separate job
17:07:31 <davidkranz> I am still concerend of a negative reaction from PTLs on having 20+ minutes of nova testing added to all projects.
17:07:34 <sdague> it's more about configuration explosion
17:07:40 <davidkranz> dwalleck: We could put it in the hourly run.
17:07:52 <sdague> it's got to be in gate
17:07:59 <sdague> otherwise you chase bugs for weeks
17:08:06 <sdague> look at postgresql
17:08:06 <jaypipes> agreed.
17:08:17 <sdague> I basically play cat and mouse to keep that running
17:08:20 <sdague> it's mostly not
17:08:29 <dwalleck> I'd be concerned if the PTL's concern is increasing run time for the sake of good testing. If it's really necessary, we could find a way to make it work
17:08:30 <davidkranz> sdague: Right. But we can't put *everything* in gate in the long run or perhaps even sooner. We have to make choices.
17:09:01 <davidkranz> I'm fine with turning it on now but that is only a short-term solution I think.
17:09:37 <sdague> davidkranz: so I'd say lets get that floated to the PTLs
17:09:38 <jaypipes> what are the remaining steps needed to get testr doing the gate?
17:09:55 <sdague> jaypipes: first the testtools conversion
17:10:11 <sdague> Ivan's got some reviews out there for that WIP, assistance on those would be good
17:10:21 <sdague> after that it can run on testr or nose
17:10:29 <sdague> then it's just shaking out the flakies
17:10:38 <sdague> and figure out how to do attrs in testr
17:10:43 <jaypipes> sdague: this one? https://review.openstack.org/#/c/20364/
17:10:52 <sdague> lifeless and cyeoh are going to pound some of that out at LCA next week
17:11:04 <sdague> jaypipes: yep
17:11:05 <jaypipes> sdague: ok, excellent.
17:11:37 <afazekas> sdague: if we import testtools everywhere it will be testtools dependent
17:11:46 <jaypipes> sdague: can we work with jeblair to get a job (non-voting) added to the gate that runs tempest with testr?
17:12:00 <lifeless> o/
17:12:01 <sdague> jaypipes: yes, once it gets "close"
17:12:07 <jaypipes> sdague: instead of nosetest-dependent? :)
17:12:10 <sdague> it's a little too broken right now
17:12:15 <jaypipes> lifeless: mornin.
17:12:30 <sdague> but this https://review.openstack.org/#/c/20364/ is the important next step
17:12:40 <sdague> so eyes on that is important
17:13:07 <jaypipes> ok, will do that review today.
17:13:11 <dwalleck> I'll take a peek this afternoon
17:13:49 <jaypipes> there's a number of reviews from nayna, ravi, and rajalakshmi that deserve a review.
17:14:06 <jaypipes> they are not big reviews, so if we could hammer through those, that would be great
17:14:18 <jaypipes> gets them off the review list, which is getting long
17:14:50 <afazekas> The nose import just used for the skip exception and for the attributes
17:14:55 <davidkranz> I asked ogelbukh to review the container-sync change. But we could approve it without that.
17:15:08 <afazekas> they are very small code pieces
17:15:31 <jaypipes> afazekas: can you elaborate on what you don't like about testtools?
17:15:34 <sdague> afazekas: yeh, skip is easy, attr is more interesting because of how testr works. So that will require testr changes as well
17:15:43 <sdague> but lifeless and cyeoh will figure out something :)
17:15:46 <davidkranz> dwalleck: Can you give the final review for https://review.openstack.org/#/c/19460/ based on your comments?
17:16:01 <dwalleck> Sure, will pull that up now
17:17:03 * sdague sadly needs to drop, battery running out in another meeting
17:17:07 <afazekas> jaypipes: I do not have any problem with it, I just wanted to say we are not really nose dependent at the moment, and we don't use really nose specific features
17:17:10 <jaypipes> sdague: ok, ciao
17:17:14 <chunwang> I want to know whether the race problem is resolved when tempest changed to parallel tests.
17:17:37 <afrittoli> regarding skip and attr it would be great if whatever we do we do not lose compatibility with nose
17:18:07 <dwalleck> chunwang: I think we should only have race conditions in tests if people built assumptions/dependencies into their test. I know I've avoided that like the plague
17:18:16 <jaypipes> chunwang: that is what we are currently discussing... we are hoping that a move to base our test cases on testresources.ResourcedTestCase and testtools.TestCase will enable parallel execution
17:18:58 <davidkranz> afazekas: 'attr' is nose-specific
17:19:14 <jaypipes> afrittoli: we should be able to keep "compatibility", yes, but nose's multi-process plugin is badly broken.
17:19:19 <afazekas> davidkranz: probably it just 5 line decorator
17:19:35 <jaypipes> afrittoli: thus, we've been investigating and prototyping using testresources for parallel execution.
17:19:40 <chunwang> ok, got it.
17:19:47 <afazekas> basically it is the same as classname.attribute = "something"
17:19:52 <jaypipes> right.
17:20:09 <jaypipes> and I think we all agree the @attr decorator is quite useful\
17:20:18 <jaypipes> for indicating which tests are smoke, negative, etc
17:20:21 <afrittoli> jaypipes: for non-gating runs it may be worth still using nose, it comes with some nice features such as xmlunit plugin
17:20:34 <dwalleck> jaypipes: ++. I love my attrs
17:20:53 <donaldngo> I like the xmlunit output as well
17:20:54 <davidkranz> The question I think is whether we use testr and take full advantage of lifeless in our midst, or try to remain compatible with various other runners.
17:21:05 <jaypipes> afrittoli: AFAIK, testr does as well... but regardless, we don't plan on making tempest unrunnable in nose...
17:21:19 <jaypipes> afrittoli: it's just the parallel plugin part that we can't use
17:21:55 <davidkranz> jaypipes: Sure. But if testr works great and in parallel, why would some one want to use nose?
17:21:57 <afrittoli> jaypipes: great thanks
17:22:04 <jaypipes> davidkranz: I would say we SHOULD take advantage of having lifeless in our midst, but continue to allow tempest to be run (in a single process) with nose
17:22:10 <dwalleck> jaypipes: testr is also compatable with the basic python unittest class as well, right?
17:22:13 <mtreinish> davidkranz: debug output...
17:22:25 <jaypipes> davidkranz: meh, I personally won't, but if someone really like nose...
17:22:34 <jaypipes> dwalleck: yes
17:22:35 <davidkranz> mtreinish: Maybe testr should support debug output...
17:22:48 <mtreinish> davidkranz: it has it, it's just a bit harder to follow then nose
17:22:50 <jaypipes> davidkranz: it does, using addDetail()
17:23:03 <afazekas> jaypipes: Probably __init__ usage instead of setup_packege could fix the nose parallel, I would'n be surprised if the testresource addition also fixed it
17:23:03 <jaypipes> right, mtreinish
17:23:18 <davidkranz> So let's stay the course and if a compelling testr-only issue comes up we can re-evaluate.
17:23:43 <jaypipes> afazekas: https://github.com/nose-devs/nose/issues/550
17:24:19 <jaypipes> afazekas: I reported the bug 6 months ago, then 4 months ago when they moved to Github issues, and still no answer.
17:24:20 <uvirtbot> Launchpad bug 6 in launchpad ""next 10 entries" at bottom of page" [Medium,Invalid] https://launchpad.net/bugs/6
17:24:27 <mtreinish> jaypipes: that sure got a lot of attention
17:24:43 <jaypipes> mtreinish: :( yeah.
17:25:22 <jaypipes> anyway, so to summarize on this particular topic, we are aiming to have parallel execution of tempest with testr --parallel by the G-3 milestone release.
17:25:37 <afazekas> jaypipes: Probably I can trace what happened, if you need it
17:25:40 <jaypipes> and we aim to keep things nose-compatible for single-process runs.
17:26:05 <ravikumar_hp> jaypipes: testr in conjuction with testtools or base Python unittests
17:26:17 <jaypipes> afazekas: I'm pretty sure testr is a better solution, to be honest... and the lack of feedback on nose issues is a big problem for me.
17:26:18 <chunwang> then is there a target date to ask all new script to follow testtools format?
17:26:57 <jaypipes> chunwang: so there is a patch here: https://review.openstack.org/#/c/20364/
17:27:11 <jaypipes> chunwang: that changes the base test case classes to support testtools
17:27:43 <jaypipes> chunwang: I'd imagine that we will make a step-wise process, with the first phase meaning no changes to individual test case classes (only to base classes)
17:28:13 <jaypipes> chunwang: and then slowly make changes for resource tracking in a way that will allow testr --parallel to work most effectively
17:28:21 <chunwang> jaypipes: ok
17:28:29 <jaypipes> chunwang: short answer: probably will be happening gradually over next few weeks
17:28:34 <afazekas> jaypipes: I can live without nose.
17:28:38 <lifeless> dwalleck: testr is a meta-runner - it runs a language specific runner; for openstack thats subunit.run, which subclasses testtools.run, and we treat incompatability with stock unittest as bugs [with a couple of opinionated differences :)]
17:29:11 <lifeless> dwalleck: so you can most definitely run tests with regular unittest - or just testtools.run, for interactive drop-into-a-debugger hacking
17:29:27 <dwalleck> Cool, that was my understanding. Thanks!
17:29:34 <lifeless> dwalleck: I do that all the time; I have plans to add a tunneled-debug mode to subunit and testr, just ETIME>
17:29:42 <afazekas> Can someone confirm the gate VM's has just only one CPU ?
17:29:59 <jaypipes> jeblair: ^^ ?
17:30:04 <dwalleck> For the rax VM, it's a 4 GB instance, right?
17:30:11 * dwalleck goes to check his charts
17:30:46 <jaypipes> dwalleck: not sure
17:30:56 <afazekas> I am guessing the flaky issue happens, because  the guest VM  waiting on disk I/O in kernel mode, and it causes time-outs in the network layer too
17:31:11 <dwalleck> From what I saw in the logs it was a 4 GB instance....RAX 4 GB instance should have 2 vCPUs
17:31:15 <afazekas> I guess the wait time is more than 5 sec very frequently
17:31:17 <jaypipes> k
17:31:41 <jaypipes> getting back on track here...
17:31:42 <dwalleck> Though if I sold my group on bumping to an 8 GB that would be 4 cores
17:32:51 <jaypipes> do we have any other business to discuss besides open reviews and testr?
17:33:12 <jaypipes> personally, I'm a bit disappointed we still don't have good quantum coverage...
17:33:37 <ravikumar_hp> jaypipes: we are planning to add
17:33:40 <mlavalle> jaypipes: I am implementing one of the BP's you approved a couple of weeks ago
17:33:47 <davidkranz> jaypipes: There are some quantum tests but they seem to not run anywhere.
17:33:50 <chunwang> When parallel execution implemented, I suppose the stress test case in tempest will be easier. Is there any plan for tempest to cover more stress test cases or any performance related test?
17:34:06 <jaypipes> mlavalle: that is good news, thx!
17:34:11 <davidkranz> jaypipes: Not even ini the quantum run on tempest commits.
17:34:15 <jaypipes> mlavalle: are you coordinating with ravikumar_hp?
17:34:36 <mlavalle> jaypipes: I haven't talked to him
17:34:43 <mlavalle> but I'll reach out
17:34:45 <jaypipes> mlavalle: please do :)
17:34:49 <ravikumar_hp> jaypipes: I think to start with we have some coverage for quantum that some tests need to be gated tests
17:35:10 <jaypipes> ravikumar_hp: agreed completely. but which ones?
17:35:57 <ravikumar_hp> jaypipes: I will check and work wiith mlavalle
17:36:17 <ravikumar_hp> identity gaps and new tests and gated tests
17:36:27 <afrittoli> jaypipes: some more basic quantum tests were merged today https://review.openstack.org/#/c/20023/  those should be good candidate for gating
17:36:32 <dwalleck> I have tests that I wouldn't call stress, more like benchmarks, that I'd like to submit somewhere. They're of the form I build x servers in y time with z success rate, or that previous scenario and then resized as well and that success rate, and so on
17:36:34 <jaypipes> ravikumar_hp: k, sounds good.
17:36:49 <jaypipes> afrittoli: ok. are they smoke tests or not?
17:37:06 <mlavalle> jaypipes: this is the BP I am implementing right now https://blueprints.launchpad.net/tempest/+spec/quantum-basic-api
17:37:15 <jaypipes> ebcause because the devstack-vm-quantum-gate seems to only be executing smoke tests... http://logs.openstack.org/20182/5/check/gate-tempest-devstack-vm-quantum/2605/console.html
17:37:27 <jaypipes> mlavalle: understood.
17:37:53 <jaypipes> dwalleck: want to add those in a new directly somewhere?
17:37:56 <ravikumar_hp> jaypipes: yes. Those tests are hp ..
17:37:58 <afrittoli> jaypipes: yes I'd say so they're basic ops, but they have no smoke attr
17:37:58 <jaypipes> directory...
17:38:17 <ravikumar_hp> right now it is not tagged as gated tests
17:38:30 <dwalleck> jaypipes: Yeah, I can do that, and then figure if there's somewhere else better they belong
17:39:14 <davidkranz> chunwang: I don't think you want to run stress tests in the ci environment.
17:39:16 <jaypipes> All: so I think the first step here is to have a devstack-vm-quantum-tempest-full job that gets run on commits to tempest.
17:39:30 <davidkranz> chunwang: Also, the logs are too strewn with errors to make them useful.
17:39:42 <jaypipes> davidkranz: yes indeed.
17:39:43 <davidkranz> chunwang: errors that are not really errors, that is.
17:40:17 <davidkranz> chunwang: We are told the log issue will get some attention after g-3.
17:40:31 <jaypipes> OK, so then I will ask some of the CI folks to work with afrittoli, ravikumar_hp and mlavalle to enable a FULL tempest run with a quantum-enabled devstack VM on commits to tempest.
17:40:40 <jaypipes> jeblair: ping
17:40:53 <clarkb> jaypipes: pong (he is on australia time at the moment)
17:41:04 <jaypipes> clarkb: ah, of course..
17:41:37 <clarkb> I can write up the change to do full tempset with quantum on tempest commits
17:41:38 <jaypipes> clarkb: think you can work with those guys to change the gate-tempest-devstack-vm-quantum to run all the tests, not just smoke?
17:41:48 <jaypipes> clarkb: thanks man, appreciated.
17:42:18 <jaypipes> that will at least get the full test suite running against an env with quantum... even if the quantum network API tests are not fully done yet
17:43:17 <jaypipes> alright, anything else from folks before we wrap up here?
17:43:22 <jaypipes> #topic open discussion
17:44:05 <afrittoli> any plan to enable xmlunit for periodic runs?
17:44:21 <dwalleck> One minor thing. For the Tempest devstack jobs, would it be possible to output the xunit results and have Jenkins format them? Looking through a failed Tempest reasons is a bit painful right now
17:44:25 <davidkranz> jaypipes: How do we move on the issue of turning on the gate for all projects?
17:44:41 <davidkranz> jaypipes: We should get PTL input but what is the best way to do that?
17:45:10 <chunwang> davidkranz: absolutely. actually what I mean is whether there is any attempt to use tempest for any more stress test cases, enhance current cases and add something like performance test...
17:45:34 <davidkranz> chunwang: That would be desirable.
17:46:29 <afrittoli> dwalleck: +1 also jenkins trends help finding out flaky tests
17:46:32 <lifeless> afrittoli: if xmlunit is junitxml compatible output, you can do that by post-processing the subunit stream testr captures.
17:46:40 <jaypipes> davidkranz: not sure, to be frank. I think we need to send an email to the -dev list that basically says "we are proposing to turn on the full tempest gate for all projects. This means an increased time of X minutes."
17:46:56 <lifeless> afrittoli: however! CI team have found jenkins is too slow at loading xmlunit data, so they just convert directly to html
17:47:05 <jaypipes> davidkranz: if we are confident that the flakies have been solved, I can send that email out.
17:47:40 <lifeless> afrittoli: so AIUI - and clarkb can confirm - we don't, and can't with jenkins today, use its unit test tracking features.
17:47:52 <davidkranz> jaypipes: OK, great. The issue is whether we should do anything else  to decrease time in the short term like skipping some slow tests or a job breakout.
17:48:27 <davidkranz> jaypipes: I'm fine going with what we've got and waiting for testr.
17:48:30 <jaypipes> davidkranz: I say we propose the full gate now, and gradually improve the runtime of tempest.
17:48:45 <davidkranz> So be it.
17:48:51 <dwalleck> I think skipping tests to decrease run time would be a risky move. Would re-categorizing tests based on priority and severity make more sense?
17:49:01 <jaypipes> davidkranz: and so it shall be.
17:49:01 <afrittoli> lifeless: oh, that's a pity, but thanks for the clarification
17:49:04 <clarkb> lifeless: thats correct, jenkins keeps everything on disk and asking it to track lots of info like that turns it into molasses
17:49:06 <jaypipes> dwalleck: agreed
17:49:14 <ravikumar_hp> dwalleck: +1
17:49:32 <jaypipes> OK, I will send the email to the -dev list and PTLs then.
17:49:33 <afrittoli> dwalleck: +1
17:49:41 <jaypipes> anything more to discuss today?
17:49:59 <dwalleck> portland sounds cold
17:50:03 <jaypipes> #action jaypipes to write email to -dev list proposing full tempest gate
17:50:30 <jaypipes> OK, ciao everyone.
17:50:32 <jaypipes> #endmeeting