17:00:14 <jaypipes> #startmeeting qa
17:00:15 <openstack> Meeting started Thu Feb 14 17:00:14 2013 UTC.  The chair is jaypipes. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:16 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:18 <openstack> The meeting name has been set to 'qa'
17:00:22 <jaypipes> morning tempestuous QAers.
17:00:35 <afazekas> good morning
17:00:39 <ravikumar_hp> hi .
17:00:43 <mtreinish> morning
17:00:43 <mlavalle> good morning
17:00:47 <mkollaro> morning
17:01:01 <timello> hey
17:01:03 <davidkranz> Hi
17:02:13 <jaypipes> so, we've been hammering through a number of reviews this morning, which is great
17:02:31 <jaypipes> pleased to see afazekas' many cleanup patches (mostly) going through. thx Attila!
17:02:39 <afazekas> :)
17:02:57 <jaypipes> An announcement: mtreinish and cyeoh are now members of QA core. Congrats and thank you to them both.
17:03:13 <davidkranz> Indeed.
17:03:14 <ravikumar_hp> congrats to them . They deserve it
17:03:20 <afazekas> Congrats
17:03:23 <mtreinish> thanks
17:03:47 <davidkranz> jaypipes: I held off on the review days thing because of expecting new members.
17:03:55 <jaypipes> davidkranz: no probs.
17:03:57 <davidkranz> jaypipes: I'm not on the fence about whether we really need it.
17:04:02 <sdague> <- here
17:04:04 <davidkranz> ^^^ now
17:04:06 <jaypipes> davidkranz: think you can hammer it out today or tomorrow?
17:04:25 <jaypipes> davidkranz: the rotation wiki page, that is.
17:04:51 <davidkranz> jaypipes: Yes.
17:04:57 <sdague> honestly, I think that if people just set asside a little time every day, it fixes itself
17:05:11 <jaypipes> sdague: easier said than done, IME
17:05:12 <sdague> we've got a policy on our team that every member should spend an hour a day on reviews
17:05:20 <davidkranz> sdague: That is where I was as of this morning.
17:05:41 <sdague> jaypipes: fair, I guess I'm just used to nova patch queue size :)
17:05:48 <davidkranz> jaypipes: How about we give it a try. If we are not meeting the 24-48 hour turnaround we can take measures.
17:05:50 <jaypipes> sdague: as much as I appreciate your team's commitment to reviews and tempest, I cannot commit to the same.
17:06:10 <jaypipes> sdague: I think a rotation puts some structure and expectations into the mix that are useful.
17:06:30 <jaypipes> davidkranz: sorry, give what a try? hour a day, or rotation?
17:06:36 <sdague> jaypipes: fair, though hopefully with some more +2s in the mix it will help
17:06:43 <davidkranz> jaypipes: Non-rotation.
17:06:44 <jaypipes> sdague: absolutely.
17:07:17 <jaypipes> davidkranz: we can try it, but without assignment, I fear that it will be too easy to let others do the reviewing...
17:07:29 <davidkranz> I could go either way.
17:07:30 <jaypipes> davidkranz: but I'm fine if that's what the group wants
17:08:05 <davidkranz> jaypipes: I suggest we see how it goes for the next week and then consider it again.
17:08:16 <jaypipes> davidkranz: why don't we try the "hour a day" approach for the next two weeks and discuss if we need to do a rotation after that if it's not working?
17:08:23 <jaypipes> davidkranz: doh, jinx.
17:09:02 <jaypipes> davidkranz: I think 2 weeks, with a published email today to the -dev and -qa lists saying expectation (or wish) is for an hour a day on reviews, and expectation of no more than 48 hours until an initial review.
17:09:13 <sdague> we have a general policy though to make sure nothing gets all the way through with only IBM eyes on things, so you may see code we originate be sitting with +2s looking for someone else to give it the +A. I find it's more healthy when you make sure a broader set of eyes see things.
17:09:22 <sdague> jaypipes: I think that's fair
17:09:30 <jaypipes> ok
17:09:31 <davidkranz> If we all do "hour a day" I predict the actual need will be less. But we'll see.
17:09:40 <sdague> yeh, in reality it will be less
17:09:52 <jaypipes> #action davidkranz to send ML email about above expectations/wishes
17:09:56 <andreaf> hi I'm here too
17:10:04 <jaypipes> andreaf: mornin.
17:10:06 <sdague> the hour a day is across all projects for us, just a general guideline to make sure people don't forget about reviewing
17:10:17 <davidkranz> sdague: Ah. OK.
17:10:46 <jaypipes> so, on a related note to the rotation...
17:11:08 <jaypipes> we had discussed at a previous meeting that we need more subject matter experts on areas other than nova and glance.
17:11:21 <jaypipes> Has anyone made any progress here? Espcially with Quantum?
17:11:36 <mlavalle> jaypipes: I am implementing https://blueprints.launchpad.net/tempest/+spec/quantum-basic-api
17:11:47 <mlavalle> it includes 5 use cases
17:11:50 <jaypipes> We have a number of reviews for quantum stuff, including Gavin's quotas stuff that are kind of hanging around
17:11:56 <mlavalle> making very good progress
17:12:04 <davidkranz> jaypipes: Dan W also told me we could pull Nachi in for reviews.
17:12:07 <sdague> mlavalle: I think the real thing we are looking for is reviewers on quantum patches from the quantum community
17:12:12 <jaypipes> mlavalle: ok, good. I'd love more reviews from you and other Quantum SMEs please! :)
17:12:15 <davidkranz> jaypipes: I had asked him for names.
17:12:21 <jaypipes> davidkranz: k
17:12:24 <sdague> dtroyer_zz and I basically do the same thing on devstack
17:12:24 <mlavalle> I will be pushing code for 3 use cases early nexte week
17:12:42 <sdague> make sure that a quantum person +1s things on a quantum patch for function before we touch it
17:12:43 <mlavalle> and code for the other 2 use cases a few days after taht
17:12:51 <davidkranz> I don't think you have to be Core in Quantum to do a tempest review as a subject "expert"
17:13:03 <sdague> davidkranz: agreed
17:13:15 <sdague> any developer that contributes to quantum is welcomed
17:13:24 <jaypipes> mlavalle: regarding Quantum, I haven't seen the full tempest suite run successfully with a quantum environment yet. Perhaps it is a good idea to hold off on new test cases/features/use cases until we get a green successful run?
17:13:39 <sdague> yeh, that would be worth while
17:14:07 <davidkranz> I was glad to see some keystone v3 tests going in.
17:14:24 <davidkranz> We should also consider ceilometer.
17:14:27 <jaypipes> mlavalle: I'm referring to this job, FYI: https://jenkins.openstack.org/view/Tempest/job/gate-tempest-devstack-vm-quantum-full/
17:14:45 <jaypipes> mlavalle: currently, it's not a gate, but I would really, really like it to be!
17:15:01 <mlavalle> jaypipes: ok, I will take a look at it
17:15:04 <jaypipes> mlavalle: first step would be to fix failures that occur, then add new use cases and API call tests, IMHO
17:15:04 <davidkranz> jaypipes: +1
17:15:11 <jaypipes> mlavalle: awesome, I really appreciate it!
17:15:24 <mlavalle> jaypipes: and reach out to you with suggestions
17:15:25 <sdague> davidkranz: agreed, we should probably try to drum that up at summit. Make sure that core projects (which ceilo will be in havana most likely) ensure they have folks looking at tempest reviews as well
17:15:38 <sdague> mlavalle: awesome, thanks
17:15:48 <jaypipes> mlavalle: you may work with sdague and dtroyer_zz on devstack-related issues with that Jenkins job. May not all be related to Tempest itself or Quantum... just FYI
17:15:59 <jaypipes> sdague: ++
17:16:09 <mlavalle> jaypipes; ok, I will reach out to them
17:16:19 <sdague> yeh, it might be more complicated, but feel free to ping me in #openstack-qa, I try to answer reasonably quickly
17:16:32 <mlavalle> jaypipes: but I will keep developing the BP. I don't want to loose momentum
17:16:44 <jaypipes> mlavalle: understood. and totally fine. :)
17:16:49 <sdague> are we done with the SME discussion? I wanted to also kick around tempest scope as a discussion topic
17:17:13 <jaypipes> sdague: yes, I think so, unless anyone has anything more to say about that?
17:17:16 <jaypipes> 5
17:17:17 <jaypipes> 4
17:17:18 <jaypipes> 3
17:17:20 <jaypipes> 2
17:17:21 <jaypipes> 1
17:17:26 <jaypipes> sdague: go for it.
17:17:30 <jaypipes> #topic tempest scope
17:17:43 <sdague> ok, so we've got 2 patches in the queue to open up new top level directories for tempest tests
17:18:02 <sdague> https://review.openstack.org/#/c/21930/
17:18:17 <sdague> https://review.openstack.org/#/c/20901/
17:18:24 <sdague> which I think are good
17:18:29 <sdague> or the general ideas are good
17:18:41 <sdague> but I wanted to make sure the rest of qa-core is good with tempest increasing scope
17:18:53 <jaypipes> sdague: yeah, not a huge fan, frankly... at least of 21930
17:18:56 <sdague> the idea is these would be seperately runnable top level test groups, like stress is
17:19:14 <sdague> jaypipes: well, jog0 was starting with something rought
17:19:22 <jaypipes> sdague: 21930 isn't tests at all..
17:19:28 <jaypipes> sdague: tests assert things.
17:19:34 <sdague> jaypipes: correct, atm
17:19:35 <afazekas> I think in tempest we should use python primary for testing
17:19:36 <jaypipes> sdague: those don't do anything at al...
17:19:48 <sdague> all those things I agree with, I put them in the review
17:19:59 <sdague> jaypipes: turns out, if you ran those commands, the script would fail
17:20:09 <sdague> because nova client throws exceptions random places
17:20:19 <sdague> which is the thing that needs testing
17:20:28 <sdague> not in the way jog0 proposed
17:20:40 <sdague> but the idea of having a test suite for the clients seems like a good idea
17:20:56 <jaypipes> sdague: s/clients/CLI incantation of the clients/
17:20:59 <sdague> where the clients get execed, and we make sure they do the right thing, and don't stack trace
17:21:02 <davidkranz> afazekas: I think I agree. I sent a message to the list last night regarding the Python novaclient issue.
17:21:06 <jaypipes> sdague: we already have tests that use novaclient library.
17:21:06 <afazekas> I woild like to have test soute for all command , including the *-manage
17:21:20 <afazekas> and it should be able to reuse existing ssh connection
17:21:31 <sdague> jaypipes: yes, this is cli testing
17:21:41 <sdague> sorry, meant to be explicit there
17:21:46 <donaldngo_hp> are the clients not being tested anywhere else?
17:21:47 <davidkranz> jaypipes: Isn't there a difference between having tests that use novaclient and "testing novaclient"?
17:22:00 <jaypipes> sdague: ya... I don't have a problem with that, per-se, I just would like it to be: a) pythonic and b) tests -- i.e. assertions
17:22:04 <davidkranz> jaypipes: Coverage, e.g.
17:22:08 <sdague> jaypipes: I agree
17:22:16 <sdague> my review comments said that :)
17:22:21 <jaypipes> davidkranz: yes, of course, no disagreement.
17:22:24 <afazekas> donaldngo_hp: devstack exercises  does some testing via bash
17:22:32 <jaypipes> afazekas: right
17:22:38 <jaypipes> afazekas: "testing"
17:22:45 <jaypipes> afazekas: without asserting anything.
17:22:46 <sdague> right, but that's sort of accidental testing in devstack
17:22:55 <davidkranz> I think the question is whether we want tempest to really test the client.
17:23:00 <sdague> because it's basically just sanity checking that devstack isn't totally screwed up
17:23:12 <davidkranz> And, if so, is it the cli or Python API, or both?
17:23:33 <sdague> davidkranz: so right now this is trying to go after a specific class of fails that have been found by manual testing
17:23:34 <afazekas> IMHO cli would cover the client library
17:23:44 <sdague> which is the cli exploding horribly
17:23:55 <mtreinish> davidkranz: I'd say both would be useful. I think there are things not covered by the cli in the lib.
17:24:17 <sdague> and given that is a surface lots of people first experience openstack with, not exploding horribly is a nice goal
17:24:24 <davidkranz> sdague: So the is really a regression test.
17:24:33 <sdague> davidkranz: sure...
17:24:43 <davidkranz> sdague: That is fine but different than being step one towards a complete test.
17:24:48 <jaypipes> I don't disagree that both CLI and client lib tests are useful.
17:25:01 <davidkranz> sdague: That may have not been clear.
17:25:38 <ravikumar_hp> Those CLI and client lib tests may not be gated tests
17:25:55 <sdague> ok, so can we get general agreement that: 1) if they are in python 2) if they are in a seperate directory 3) if they are actually tests, it should be ok to let them in?
17:26:02 <davidkranz> I'm not sure what the real value of having client tests in tempest is though. Shouldn't there be unit tests in those projects?
17:26:08 <sdague> ravikumar_hp: right now it's not going to be gate
17:26:29 <sdague> davidkranz: a bunch of the commands can't be exercised unless there is a working openstack
17:26:32 <donaldngo_hp> davidkranz: yes I'm suprised that the CLI project repos dont have tests themselves
17:26:40 <sdague> donaldngo_hp: they do have unit tests
17:27:19 <afazekas> We could make tempest as multi backend tool (xml,json,library,ec2, cli), (witch configurable backend), but is too big work for now. Now we could have the best benefit/cost ratio if we focusing to the cli tests in addition
17:28:08 <sdague> afazekas: yeh, that's my thinking. I also didn't want to send these off to a new tree, because we're only just getting critical mass on reviewers on tempest, and I fear less eyes on another test effort
17:28:25 <donaldngo_hp> wouldn't it be a good idea to tie the CLI functional symantic tests to the CLI repos themselves?
17:28:30 <sdague> grenade being a good example, which is a neat idea, but stalled out in another tree
17:29:03 <sdague> donaldngo_hp: maybe, the infrastructure to set them up with credentials and such would then need to be duplicated out of tempest to all the clis
17:29:17 <jaypipes> sdague: and devstack..
17:29:26 <sdague> jaypipes: yep, and devstack
17:30:05 <sdague> it moves tempest umbrella from "testing openstack API" to "testing live openstack in various useful ways"
17:30:08 <sdague> but I'm ok with that
17:30:17 <sdague> as long as it's easy to just execute the parts of that you want
17:30:30 <afazekas> +1
17:30:32 <davidkranz> sdague: I think that was always part of the goal. To eventually be an acceptance test.
17:30:44 <sdague> davidkranz: ok, cool, then this would seem in scope
17:31:44 <andreaf> +1 in extending the scope / acceptance test
17:31:45 <sdague> jaypipes: you on board?
17:31:55 <jaypipes> sdague: yeah, I'm fine with an expanded scope in this way, just would like to see everything runnable in a consistent manner
17:32:02 <sdague> jaypipes: +1
17:32:07 <cjd_> sorry for the quick detour.  what time did/does the qa meeting start?
17:32:15 <sdague> I'll make sure to work with jog0 to get it there
17:32:19 <sdague> cjd_: 32 minutes ago
17:32:31 <cjd_> ok.  thanks :~\
17:32:41 <davidkranz> cjd_: 17:00 UTC
17:32:58 <sdague> ok, I think that topic is done, unless someone else has something on it
17:33:00 <cjd_> got it.  as you were :)
17:33:34 <donaldngo_hp> will the CLI test the latest version of the CLI?
17:33:46 <sdague> donaldngo_hp: I think that's the intent
17:34:02 <sdague> tempest follows trunck
17:34:06 <sdague> follows trunk
17:34:52 <sdague> ok, next topic?
17:35:03 <davidkranz> Parallel testing?
17:35:03 <afazekas> SKIPED tests ?
17:35:11 <sdague> davidkranz: on parallel testing
17:35:18 <andreaf> +1 Parallel testing
17:35:23 <sdague> cyeoh is working to get a non-voting job that will enable testr for that
17:35:35 <sdague> so we can debug the final issues in qa
17:35:40 <sdague> in CI
17:36:02 <sdague> I was talking with him about it yesterday, have to figure out if the review went in yet
17:36:30 <sdague> we'd then run testr version of tempest full on tempest project checkins for a while in non-voting to figure out what else needs fixing
17:36:43 <sdague> there are still some fails because of state leaks between tests
17:36:45 <jaypipes> sorry y'all, need to hop on an emergency call... sdague, please close out the meeting when finished. thnks!
17:36:52 <sdague> jaypipes: will do
17:37:11 <sdague> so that's comming along
17:37:30 <andreaf> sdague: a good outcome of this exercise would be a set of guidelines on handling resources
17:37:32 <sdague> I was hoping we'd be fully lit by g-3, but looks like it might come on later
17:37:47 <sdague> andreaf: yeh, I think based on what needs fixing we'll get that
17:37:56 <sdague> plus afazekas's work on the resource allocator
17:38:07 <sdague> which will massively simplify things
17:38:22 <andreaf> sdague: I agree
17:38:22 <sdague> I think that's that on that front
17:38:40 <sdague> afazekas: you want to take on SKIPPED tests?
17:39:29 <afazekas> sdague: jaypipes is not  here so we will not answers anyway..
17:39:41 <sdague> afazekas: ok
17:39:57 <sdague> fwiw, I reenabled resize tests in devstack / tempest
17:40:05 <sdague> turns out we disabled them by default
17:40:16 <sdague> and resize got broken in nova because of it
17:40:25 <sdague> so those are back in the mix now
17:40:42 <sdague> it was I think 4 nova fixes to get it working again
17:42:15 <mtreinish> afazekas: was this about the resource thing and generic_setup for the flags?
17:43:17 <afazekas> maybe
17:43:30 <sdague> ok, so other topics?
17:43:35 <sdague> we seem to be winding down
17:43:49 <davidkranz> Counting...
17:44:14 <andreaf> one question still about parallel execution
17:44:34 <sdague> andreaf: go for it
17:44:39 <andreaf> is there any guideline yet about where is the best place to deal with common resources?
17:44:45 <afazekas> Does anyone interested in systemtap scripts, for performance analysis (adding to tempest repo)?
17:44:47 <andreaf> with the current setup
17:44:54 <davidkranz> andreaf: common to what?
17:45:04 <sdague> afazekas: that might be a cool option
17:45:17 <sdague> afazekas: will that work on the ubuntu kernel that's in CI?
17:45:18 <andreaf> common to all tests within a class
17:45:42 <sdague> andreaf: so I think we're going to need to hash some of that out with lifeless at summit
17:45:43 <afazekas> sdague: probably, depends on the code
17:46:07 <sdague> once we have working tempest testr we can more easily look at where we'd get better optimizations out of testr if it behaved differently
17:46:17 <davidkranz> andreaf: That depends on whether the tests in the class are supposed to run in parallel or not.
17:47:44 <davidkranz> andreaf: If the tests are run in parallel they can't share class resources except in a "read only" way.
17:48:18 <afazekas> davidkranz: why not ?
17:48:56 <davidkranz> afazekas: HOw can two parallel tests both poke the state of something independently?
17:48:59 <sdague> I think it will become more clear with the implementation
17:49:20 <sdague> so I would wait for that review and runner to be out there, then we can poke on it
17:49:29 <andreaf> sdague: ok I'll wait for the non gating parallel run and then dig into it
17:49:39 <afazekas> davidkranz: They can aquire a resource from resource pool, but they cant work on the same resource at the same time
17:50:10 <davidkranz> afazekas: Of course resource pools are fine.
17:50:36 <afazekas> ok
17:50:38 <davidkranz> afazekas: Thought I am not sure how the locking works.
17:50:49 <davidkranz> afazekas: In Tempest, that is.
17:51:16 <sdague> ok, so any last topics?
17:51:20 <davidkranz> We don't need to pursued that right now.
17:51:22 <sdague> hot reviews that need more eyes?
17:52:42 <Nithya> https://review.openstack.org/#/c/18631/
17:53:01 <Nithya> This change is abandoned.
17:53:37 <Nithya> I would like to own and submit a patch
17:53:41 <Nithya> is that possible?
17:54:00 <afazekas> davidkranz: most people just hope the gloabal interpreter lock save them from conflicts :), now we do not have anything for mulithread resource sharing, but it can change in the future
17:54:10 <sdague> Nithya: yes, you can: review -d 18631 and build your own version from it and submit
17:54:27 <Nithya> sdague: Thank you
17:54:39 <sdague> davidkranz / afazekas: testr is process not thread based, so that won't be a concern
17:57:07 <sdague> ok, any last topics?
17:57:12 <sdague> 10
17:57:13 <sdague> 9
17:57:16 <sdague> 8
17:57:18 <sdague> 7
17:57:19 <sdague> 6
17:57:22 <sdague> 5
17:57:24 <sdague> 4
17:57:27 <sdague> 5
17:57:29 <sdague> 2
17:57:31 <sdague> 1
17:57:36 <mlavalle> Have a nice day, y'all
17:57:38 <sdague> ok, let's call it a day
17:57:43 <sdague> thanks everyone
17:57:51 <andreaf> thanks have a nice day / evening
17:58:01 <sdague> #endmeeting
17:58:36 <sdague> jaypipes: can you run #endmeeting
17:58:43 <sdague> I don't think I can do it because I'm not #chair
18:00:19 <bdpayne> OSSG meeting starting shortly… waiting for previous meeting chair to end that meeting
18:02:08 <bdpayne> jaypipes … you around to end the meeting?
18:03:44 <jaypipes> #endmeeting