17:02:31 <davidkranz> #startmeeting qa
17:02:32 <openstack> Meeting started Thu Feb 21 17:02:31 2013 UTC.  The chair is davidkranz. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:02:33 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:02:35 <openstack> The meeting name has been set to 'qa'
17:03:10 <davidkranz> First, remember that summit.openstack.org is open for proposals.
17:03:16 <fnaval> hi, i'm new.
17:03:24 <davidkranz> fnaval: Welcome!
17:03:32 <fnaval> davidkranz: thanks
17:03:35 <giulivo> I take the chance, I'm also new
17:03:44 <davidkranz> There are currently two proposals in the QA track.
17:04:00 <davidkranz> I encourage every one to submit proposals for topics of interest.
17:04:04 <Ravikumar_hp> davidkranz: i am going to add 2 proposals
17:04:17 <davidkranz> Ravikumar_hp: Excellent
17:04:21 <sdague> davidkranz: yeh, I've got a few, but it will take me a week to sort it out
17:04:30 <dwalleck> sam and I have a few as well
17:04:34 <sdague> still burned out from the rush of nova reviews
17:04:39 <davidkranz> sdague: We still have some time. I just wanted to remind folks.
17:04:45 <sdague> so my brain's a little broken right now :)
17:05:09 <davidkranz> #topic Managing Reviews
17:05:34 <davidkranz> Looking at the list, we seem to be diong better on timely reviews.
17:06:20 <davidkranz> There is currently only one review without a -1 that is older than yesterday.
17:06:33 <afazekas> https://review.openstack.org/#/c/22112/ <- this ?
17:06:51 <davidkranz> afazekas: Yes.
17:06:59 <Ravikumar_hp> davidkranz: sometimes new reviewers added very late in review and it goes to some more patches
17:07:47 <davidkranz> Ravikumar_hp: Sure. I mentioned it because last week we said we would evaluate after two weeks whether we needed some kind of review days for core reviewers.
17:08:06 <Ravikumar_hp> suggest 3 reviewers from start to end
17:08:25 <davidkranz> Ravikumar_hp: I thought two Core reviewers was sufficient.
17:08:39 <sdague> yeh, 2 cores is fine
17:08:41 <dwalleck> davidkranz: ++
17:09:05 <davidkranz> OK. I have a few topics but does any one else have a topic to bring up first?
17:09:07 <sdague> though I highly encourage other folks to review that aren't core. I definitely take that input into account
17:09:28 <mlavalle> davidkranz:  I want to report on the assignment the team gave me last week. The Jenkins problem with Quantum: https://jenkins.openstack.org/view/Tempest/job/gate-tempest-devstack-vm-quantum-full/
17:09:32 <davidkranz> sdague: Yes, I agree. But the core reviewers are the ones on the hook.
17:09:53 <davidkranz> mlavalle: Please.
17:10:07 <mlavalle> davidkranz: the problem is test_network.py. It's a year old test that uses the minimal network REST client provided  by Tempest. The client is for version 1.1 of the Quantum API, which is not available in DevStack anymore.
17:10:30 <mlavalle> davidkranz: I can patch the client to use Quantum API version 2. But before doing that, I want the team's feedback. For a Jenkins gate test, shouldn't we use the Quantum Python client instead of the minimal REST client in Tempest?
17:10:40 <mlavalle> davidkranz: I can patch the client to use Quantum API version 2. But before doing that, I want the team's feedback. For a Jenkins gate test, shouldn't we use the Quantum Python client instead of the minimal REST client in Tempest?
17:10:47 <sdague> mlavalle: no
17:11:02 <sdague> because that hides bugs that the clients paper over
17:11:13 <sdague> tempest should always have it's own REST client implementations
17:11:15 <mtreinish> mlavalle: you should use the restclient. we are testing the api not the project client implementations
17:11:15 <dwalleck> sdague: ++
17:11:37 <davidkranz> sdague: I agree, but the tempest rest client could be a copy of the "real" one as was done pretty much with glance.
17:11:59 <sdague> davidkranz: we only copied one piece for chunked encoding upload
17:12:02 <mtreinish> davidkranz: the glance one is only there for one thing that httplib2 couldn't do
17:12:03 <mlavalle> davidkranz: ok in that case I will upgrade the rest client to use version 2 of the Quatum API
17:12:05 <afazekas> Are we going to rewrite  all tests in tempest/tests which using the libraries ?
17:12:07 <sdague> because it's kind of complicated
17:12:33 <dwalleck> davidkranz: It really should be more than a copy. Logging, serialization, metrics. There's a lot of data we can get from clients
17:12:33 <davidkranz> My point was that tempest should not need to reinvent the client code.
17:12:35 <sdague> afazekas: what other tests use client libraries, I thought it was only glance
17:12:42 <afazekas> smoke
17:12:50 <davidkranz> It just needs to "own" its own client code.
17:12:52 <afazekas> whitebox
17:13:16 <sdague> I think that core tempest should not use the clients
17:13:19 <mlavalle> davidkranz: I should have a patch ready for the Quantum rest client early next week
17:13:25 <sdague> I'm ok with other directories of tests that do
17:13:33 <davidkranz> mlavalle: OK, great.
17:13:49 <sdague> we're adding other things like cli tests, so we have precidence
17:14:59 <chunwang> sdague: what does cli test mean here?
17:15:04 <davidkranz> dwalleck: I don't disagree. But the client is a really good starting point in most cases.
17:15:21 <davidkranz> Anyway, that is up to the contributor of the tempest client.
17:15:48 <afazekas> sdague: someone should announce it on the ML, before I staring to eliminate these client library anomalies ..
17:15:51 <andreaf> hi
17:15:53 <davidkranz> chunwang: Testing the cli for the real clients in tempest.
17:16:01 <dwalleck> davidkranz: I partially agree. However, it would be better to have consistency across projects
17:16:11 <sdague> afazekas: sure
17:16:31 <davidkranz> dwalleck: Yes, reducing it to a previously unsolved problem :)
17:16:46 <sdague> chunwang: comand line, jog0 has been working to add tests that run the openstack command line tools to make sure they don't blow up
17:16:58 <sdague> which they do under some surprising circumstances
17:17:28 <davidkranz> sdague: That reminds me of the negative testing with fuzz issue?
17:17:41 <davidkranz> Any one have any status on that?
17:17:57 <sdague> davidkranz: as far as I know no ones worked on it
17:18:14 <dwalleck> matt tesauro is leading that up from my group
17:18:36 <davidkranz> dwalleck: Any status worth mentioning?
17:18:37 <dwalleck> I believe he has a working prototype, bouncing it around a bit before pushing it in
17:19:03 <davidkranz> dwalleck: Cool. I'd love to get a look at it.
17:19:04 <dwalleck> He'll actually be online for the sec meeting next. I can ping him to see what his schedule looks like
17:19:11 <sdague> it sort of raises another topic. I was hoping in the havana cycle we could lean on launchpad a bit more and actually set h-1, h-2, h-3 goals for blueprints we wanted to land
17:19:30 <davidkranz> sdague: +1
17:19:34 <sdague> dwalleck: would be nice to see that in progress to understand how it fits with the rest of things
17:19:36 <chunwang> sdague: do you mean the test for sth like nova-client?  Does it mean the tempest script will call the nova command line directly then validate the result whether as expected?
17:19:41 <sdague> incremental is good
17:20:05 <sdague> chunwang: yes, it's a seperate directory, so it won't be in the main tempest tests, but yes
17:20:07 <jaypipes> sdague: ++
17:20:22 <mtreinish> sdague: ++
17:20:41 <jaypipes> as for the discussion on the regular python client libraries from before... we have had that discussion probably 15 times over the last couple years.
17:20:43 * sdague tries to figure out what the value of sdague is now
17:20:49 <jaypipes> people want it both ways.
17:21:02 <mtreinish> sdague: 3?
17:21:04 <sdague> jaypipes: well cli tests kind of give us both ways :)
17:21:15 <chunwang> sdague: got it.
17:21:37 <andreaf> sdague: ++
17:21:45 <mtreinish> sdague: along that note are we going to branch a grizzly version of tempest?
17:22:03 <sdague> mtreinish: yes, we should at milestone proposed
17:22:05 <davidkranz> mtreinish: Wr have to
17:22:09 <jaypipes> mtreinish: another thing we always say we're going to do, then do it, and nobody maintains it.
17:22:12 <sdague> so there is a version that works on stable
17:22:27 <sdague> jaypipes: well it's more important that it doesn't change
17:22:33 <sdague> it's still used on stable gate jobs
17:22:37 <mtreinish> ok, because I didn't think we did for folsom
17:22:51 <afazekas> Can we backport tests to the Folsom too ?
17:22:51 <davidkranz> Right. It's purpose is to prevent regressions on stable releases.
17:22:56 <sdague> oh, maybe not
17:23:01 <sdague> but the gate is only smoke
17:23:07 <sdague> for < grizzly
17:23:15 <sdague> so the chance of break is smaller
17:23:33 <davidkranz> sdague: For grizzly we should make it behave just as it did before becoming a stable branch.
17:23:58 <davidkranz> sdague: Gating on all stable/grizzly projects.
17:24:01 <mtreinish> nm, it was juat hidden on github ui for folsom
17:24:34 <sdague> davidkranz: right, so I assume we cut stable at rc1 for the rest of the projects?
17:24:44 <sdague> actually, I don't know what previous policy was there
17:25:22 <davidkranz> sdague: That would be reasonable.
17:26:03 <davidkranz> IMO, we should not spend time backporting tempest changes to stable branches without a compelling reason.
17:26:41 <davidkranz> I would like to bring up the topic of how we are going to test both v2 and v3 of keystone.
17:27:13 <davidkranz> Though we will have a similar issue with other projects that have more than one version, perhaps glance.
17:27:36 <davidkranz> Has any one thought about that?
17:27:40 <mtreinish> davidkranz: for right now all the glance tests are v1
17:28:00 <davidkranz> mtreinish: That is not a good situation if v2 is part of grizzly.
17:28:13 <sdague> davidkranz: it would be good to test both versions I think
17:28:31 <sdague> nova's going to have the same desire in havana as we cut a v3
17:28:42 <sdague> which is going to change some of the tempest tests as we normalize return codes
17:28:44 <mtreinish> davidkranz: yeah it would be good to test both versions. I'll start working on that.
17:29:04 <davidkranz> sdague: Right. But it is tough to do with this being specified in tempest.conf unless we run a separate gate for each version, which doesn't really scale.
17:29:09 <afazekas> note: our glance endpoint definition does not contains a version..
17:29:10 <andreaf> davidkranz: that's a good topic. in general how are we going to test multiple configurations on nova side? e.g. using autoassigned IPs or not, using libvirt driver or xen
17:29:52 <davidkranz> I believe this is actually a big architectural issue (and performance one) that we have not addressed.
17:29:57 <andreaf> davidkranz: for gate it doesn't scale, but perhaps we need other jobs to verify different configs
17:29:59 <sdague> davidkranz: ok, well lets go experiment a bit and figure out if we have a compelling way to address it
17:30:06 <Ravikumar_hp> davidkranz: can we consult Keystone PTL
17:30:21 <chunwang> Will the tempest developer team still recommend user to use latest master for test of different version after grizzly version branch?
17:30:27 <Ravikumar_hp> we are adding V3 tests as the functioanlity is getting complete
17:30:30 <davidkranz> Ravikumar_hp: Sure, but I think this is really a tempest issue.
17:30:52 <jaypipes> davidkranz: the keystone endpoint should support BOTH v2 and v3 APIs simulataneously.
17:31:14 <sdague> jaypipes: agreed
17:31:23 <Ravikumar_hp> yes. it does
17:32:01 <davidkranz> The question is do we run all the tests in both versions, plus new tests for the new version?
17:32:06 <jaypipes> davidkranz: so IMHO, we should just have the tempest keystone identity client query the endpoint base URI and get the v2 and v3 API root endpoints from the returned 301
17:32:25 <sdague> davidkranz: we should run whatever the components support
17:32:28 <davidkranz> jaypipes: Sounds good.
17:32:36 <sdague> eventually the deprecate the old apis, then we can pull them out
17:32:36 <jaypipes> davidkranz: we run all tests for whatever versions are returned in the 301
17:32:41 <afazekas> as I mentioned on the ML, the services (clients), should not decide the endpoint on their own, they should get the base url from the constructor
17:33:05 <jaypipes> afazekas: they should get the base URI from the uri value in the config file :)
17:33:28 <afazekas> jaypipes: one for v2 and one for v3 ?
17:33:41 <davidkranz> jaypipes: So we just need to change the code that processes the admin_url and remove the /2 in the config?
17:34:01 <jaypipes> afazekas: no, the root URI is auth.example.com:5000/
17:34:16 <davidkranz> jaypipes: Are all OpenStack APIs going to behave the same way with multiple versions?
17:34:18 <jaypipes> afazekas: have the rest_client subclass' tack on the v2.0/ and v3/
17:34:19 <afazekas> yes, and it gets back service endpoint
17:34:28 <jaypipes> davidkranz: everything other than swift should.
17:34:35 <davidkranz> jaypipes: :)
17:35:21 <davidkranz> So we just need to find out from the keystone devs when this will be ready to work. Or does it already?
17:35:22 <sdague> davidkranz: yeh, it's supposed to in the spec. nova does it in theory, though we only have one API version right now
17:35:36 <jaypipes> davidkranz: should work already.
17:35:45 <dolphm> jaypipes: +1, v3 auth merged yesterday
17:35:45 <jaypipes> davidkranz: glance, too.
17:35:54 <afazekas> jaypipes: now every client acquires a token, and parses the token for endpoint, do we really need to do it in the "clients" ?
17:36:12 <jaypipes> afazekas: no, we should cache it.
17:36:18 <davidkranz> So now we just need to know who is planning to make this change.
17:37:14 <afazekas> We should cache it with global  cache used be all rest clients  to hide design glitches ?
17:37:23 <jaypipes> afazekas: but I will say... that the more we write code in these tempest rest clients, the more similar they end up looking like the darn upstream python client libs, so other than fuzz testing, I'm really beginning to question the need for them, frankly.
17:37:48 <jaypipes> afazekas: especially with certain folks' push to make them object-oriented/resource-exposing.
17:38:07 <jaypipes> afazekas: which essentially will just make them identical to the upstream python client libs, for good or bad.
17:39:12 <afazekas> jaypipes: If we could eavesdrop the traffic done by the upstream clean, we would not need to implement our own clients in most cases
17:39:29 <jaypipes> afazekas: we can easily eavesdrop.
17:39:29 <afazekas> jaypipes: If we could eavesdrop the traffic done by the upstream client, we would not need to implement our own clients in most cases
17:39:48 <jaypipes> afazekas: monkeypatch the base request call with a simple decorator.
17:40:44 <afazekas> jaypipes: but in this case we need to modify code, which supposed to an analyses code
17:41:08 <jaypipes> afazekas: I'm not following you, could you elaborate what you mean by that?
17:42:06 <afazekas> If we using the client libraries, we should be try to use them, as it is considered not modified , and we are able to check the traffic, at the same time
17:43:12 <jaypipes> afazekas: I'm talking about adding a simple decorator on the base client class' request() method that listens and records (with fixtures.Fixture.addDetail()) on the incoming and outgoing HTTP request calls
17:43:38 <jaypipes> in other words...
17:43:47 <jaypipes> import novaclient.v1_1.client as c
17:43:52 <jaypipes> from mock import patch
17:43:57 <afazekas> after module load, you would hook them..
17:44:02 <afazekas> it is good
17:44:07 <jaypipes> patch(client.request, some_listener)
17:44:18 <jaypipes> that's all...
17:44:32 <jaypipes> nothing more than a simple listener/recorder, nothing changing the way the client worked.
17:44:36 <afazekas> First it is sounds good..
17:46:07 <afazekas> We can discuss the details later
17:47:28 <davidkranz> So is there a proposal to change tempest policy in favor of "real" clients?
17:47:48 <afazekas> It needs detailed plan before implementing anything
17:48:11 <davidkranz> afazekas: I meant a proposal in concept.
17:48:20 <davidkranz> afazekas: Or policy.
17:48:21 <afazekas> May be it will not fly , because of some unforeseen detail
17:49:07 <davidkranz> afazekas: In order to evaluate that we would need a real requirement list for the client to determine if real clients could meet it.
17:49:31 <afazekas> davidkranz: lets consider it just an idea, until we do not have more detailed plan
17:49:48 <davidkranz> I'm not sure the list of objections to real clients was ever explicit.
17:49:50 <jaypipes> davidkranz: I think it's a good discussion for the summit.
17:49:58 <davidkranz> jaypipes: +1
17:50:18 <davidkranz> jaypipes: Can you or afazekas submit one?
17:50:24 <afazekas> may be they are not letting us to make really bad thing, and some case is not testable
17:50:46 <afazekas> like violating pre-conditions
17:50:56 <jaypipes> davidkranz: the more we add to the tempest rest_client(s), the more they look virtually identical to the upstream libs. and until fuzz testing is handled in an automated/grammar-based way, the only way we can do negative testing is to use the tempest rest client and not the upstream libs.
17:50:57 <davidkranz> afazekas: We have already excepted negative tests with fuzzing.
17:51:31 <davidkranz> jaypipes: I understand.
17:51:36 <jaypipes> davidha: exactly, so IMHO, we can't get rid of the tempest rest client until we have a grammar-based tool for negative testing
17:51:46 <afazekas> do we have a good fuzz-er anywhere ?
17:52:08 <davidkranz> afazekas: Daryl is going to report on that. See above in this meeting.
17:52:40 <afazekas> ok
17:52:55 <davidkranz> jaypipes: Agreed. I was thinking about new tests and new projects coming into integration.
17:53:10 <davidkranz> jaypipes: Whether we make them write their own client.
17:53:23 <davidkranz> jaypipes: tempest client, that is.
17:53:47 <jaypipes> davidkranz: yeah... good point. not sure... I'd lean on the side of saying no.
17:54:13 <davidkranz> jaypipes:  :) At the beginning of this meeting we told mlavalle "yes" I think.
17:55:03 <davidkranz> Obviously this needs more discussion.
17:55:14 <davidkranz> Any other issues before we close?
17:55:43 <mkollaro> does anybody know about any swift destruction tests?
17:56:06 <mkollaro> simulating disk failures and such
17:56:13 <davidkranz> mkollaro: I think all we know about is what is in Tempest now.
17:56:21 <mkollaro> :/
17:57:13 <afazekas> https://review.openstack.org/#/c/22112/ <- please review it :)
17:58:44 <davidkranz> OK, ready to close...
17:59:10 <davidkranz> Thanks all.
17:59:20 <davidkranz> #endmeeting