17:01:38 <sdague> #startmeeting qa
17:01:39 <davidkranz> Here
17:01:39 <openstack> Meeting started Thu Jun 20 17:01:38 2013 UTC.  The chair is sdague. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:01:40 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:01:42 <openstack> The meeting name has been set to 'qa'
17:01:59 <sdague> #topic Blueprint Status
17:02:16 <sdague> please check in with any updated info on your blueprints using the #info tag
17:02:35 <sdague> we'll give everyone a minute or two on that
17:03:34 <mtreinish> sdague: no blueprint link this time?
17:03:45 <sdague> #link https://blueprints.launchpad.net/tempest
17:03:52 <sdague> sorry, I'm slow :)
17:04:08 <sdague> #link https://launchpad.net/tempest/+milestone/havana-2
17:04:24 <sdague> those are the havana 2 blueprints, which we are only 3 weeks away from, and there are a lot there
17:05:18 <davidkranz> I'm going to try to get some progress on the stress tests.
17:05:19 <sdague> davidkranz: what's need to close out the stress test one
17:05:23 <sdague> ok, great
17:05:27 <davidkranz> Volume was just added.
17:05:45 <davidkranz> sdague: The most important thing is to create the job that will run it
17:05:55 <davidkranz> sdague: And set a periodic build.
17:06:08 <davidkranz> sdague: RIght now there is nothing making sure it actually works.
17:06:29 <davidkranz> sdague: But there is an issue that these tests are really targeted to run in an environment not supported upstream.
17:06:30 <sdague> davidkranz: ok, great, is that still doable for havana-2?
17:07:03 <davidkranz> sdague: I am checking on that and will update accordingly.
17:07:07 <sdague> davidkranz: well I think we had a policy that if we can't run it in CI some how (periodic or otherwise) it can't really be in tempest
17:07:13 <sdague> because otherwise we get bit rot
17:07:31 <mtreinish> that was a big issue with the old stress tests.
17:07:44 <sdague> right, exactly
17:08:05 <davidkranz> mtreinish: Unfortunately the reason the old stress tests could not be run still remains.
17:08:12 <davidkranz> mtreinish: All the bogus log errors.
17:08:25 <sdague> davidkranz: well we can run it as a non enforcing job
17:08:56 <sdague> also, if there are specific bugs for each error in the logs, they are likely to get addressed
17:09:01 <davidkranz> sdague: We need to do that anyway but the real issue if false positives if we can't check the log for legitimat errors
17:09:15 <davidkranz> typing too fast...
17:10:00 <davidkranz> sdague: I filed a whole bunch of bugs a few months ago
17:10:09 <sdague> davidkranz: sure, but we can call out the false possitives to code around them, much like skips in our tempest code
17:10:15 <davidkranz> sdague: Some were fixed, some not.
17:10:37 <davidkranz> sdague: Yes, we can do that but it is very fragile.
17:10:38 <sdague> ok, well lets leave it at the following
17:11:03 <sdague> #action davidkranz to see if stress tests can be completed for h2 milestone
17:11:20 <davidkranz> sdague: OK
17:11:32 <sdague> mtreinish: I know I put testr later in the agenda, but you want to talk about it now, given that it's also a critical blueprint?
17:11:50 <mtreinish> sdague: sure
17:12:05 <davidkranz> mtreinish: Is there any issue with the admin tests running in parallel?
17:12:22 <davidkranz> mtreinish: There is no tenant isolation there.
17:12:30 <mtreinish> so I've got a patch pending for testr that partitions on test classes and it's giving runtime numbers that are what we expect
17:12:40 <mtreinish> davidkranz: not that I've seen in my runs
17:12:43 <davidkranz> mtreinish: Great!
17:13:30 <sdague> mtreinish: any word from lifeless yet about his evaluation on the patch?
17:13:31 <mtreinish> I'm waiting on lifeless to review the testr patch before I push out a change that moves run_tests and tox over to testr
17:13:40 <sdague> ok, cool
17:14:05 <mtreinish> once it gets merged we should be able use testr for everything
17:14:24 <mtreinish> I'm sure it will shake loose some races but we debug those as they come up
17:14:43 <sdague> yep
17:14:50 <sdague> that will be great
17:14:51 <mtreinish> but at least locally things have been fairly stable on my dev box
17:15:07 <davidkranz> mtreinish: Maybe we should keep it work in progress and rerun it a bunch of times before merging?
17:15:09 <sdague> mtreinish: you want to update that blueprint to good progress?
17:15:15 <mtreinish> if anyone cares I've put some initial numbers up here: https://etherpad.openstack.org/testr-numbers
17:15:27 <sdague> #link https://etherpad.openstack.org/testr-numbers initial tester numbers
17:15:46 <mtreinish> also the testr patch is here if anyone wants to try it: https://code.launchpad.net/~treinish/testrepository/testrepository
17:15:50 <sdague> #info https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest now Good Progress, just waiting for test patch inclusion
17:15:51 <davidkranz> mtreinish: I would hesitate to put that in the gate based on one successful run.
17:16:17 <sdague> davidkranz: yeh, probably we'd want to make a separate tox rule for a full run
17:16:28 <sdague> and run it in shadow mode for a week or two
17:16:32 <sdague> just to make sure we are good
17:16:37 <davidkranz> sdague: Sounds good to me.
17:16:41 <mtreinish> davidkranz: yeah that's a fair point
17:16:50 <mtreinish> sdague: yeah sounds like a good idea
17:17:09 <sdague> ok, anything else on blueprints?
17:17:18 <sdague> and, anyone here besides just us 3 :)
17:17:18 <mtreinish> also testr has '--until-failure' and '--analyze-isolation' which we could maybe setup as a periodic too
17:17:47 <davidkranz> mtreinish: Do you have a feel as to why the speedup falls sharply after 2x?
17:18:03 <davidkranz> mtreinish: Of course 2x would be great :)
17:18:33 <sdague> davidkranz: I think one thing that came up is that it's not using the timing database to optimize
17:18:41 <afazekas_> hi
17:18:46 <mtreinish> davidkranz: not really, I was happy with the numbers but I haven't looked into optimizing them further yet
17:18:53 <mtreinish> I'm sure there is room for improvment
17:19:06 <sdague> hey afazekas
17:19:13 <davidkranz> mtreinish: Sure, just curious. At some point it will slow just due to load.
17:19:20 <sdague> afazekas: anything from you on blueprints before we move to reviews?
17:19:21 <afazekas_> hey sdague
17:20:21 <sdague> ok, reviews
17:20:28 <sdague> #topic Reviews that need attension
17:20:54 <afazekas_> I will try to figure out why I have the issue with glance+swift+wc2, after that I will try to reproduce the real ec2 issue in a loop (as background job), at the same time I will work on the resource leak ,  and add some docs
17:21:03 <sdague> looks like we actually have a lot of reviews with 1 +2 on them
17:21:14 <afazekas_> s/wc2/ec2/
17:21:16 <davidkranz> sdague: Well, there is https://review.openstack.org/#/c/33689/  :)
17:21:39 <davidkranz> sdague: I think a few are waiting for non-red-hat approval.
17:21:55 <mtreinish> davidkranz: sdague started a ml thread on that one
17:21:55 <sdague> davidkranz: sure, I'll take a look later today
17:22:25 <davidkranz> mtreinish: I know. Should be fun. We've been down this road before. Swift folks just don't agree with the API stability policy.
17:22:28 <sdague> davidkranz: yeh, on the swift one there is a thread here: http://lists.openstack.org/pipermail/openstack-dev/2013-June/010689.html
17:23:22 <sdague> ok, I need to go figure out why my patch died on pg
17:23:30 <sdague> didn't notice that one yet
17:23:41 <sdague> ok, any other reviews of note?
17:24:16 <sdague> ok, so the last thing on the agenda which we didn't yet talk about was multiple api versions - https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Weekly_QA_Team_meeting
17:24:28 <sdague> #topic Multiple API versions
17:24:57 <sdague> there was an ML thread on this, we think it's resolved enough there, or is there need for more discussion?
17:25:17 <mtreinish> sdague: that fail should just be a recheck. I opened a bug for it here: https://bugs.launchpad.net/devstack/+bug/1192990
17:25:19 <uvirtbot> Launchpad bug 1192990 in devstack "Keystone certificate error with heat" [Undecided,New]
17:25:25 <davidkranz> sdague: I don't think it is quite resolved there but we could continue the discussion there.
17:25:38 <sdague> ok, we'll take the discussion there
17:26:01 <davidkranz> My issue is how we test new and old versions that bhavior has not changed without duplicating the tests.
17:26:06 <sdague> #topic Open Discussion
17:26:14 <sdague> davidkranz: I don't think you can
17:26:30 <davidkranz> sdague: That is a big problem given the size of nova, e.g.
17:26:32 <sdague> duplicating is probably the right way from an issolation perspective, it also lets you remove old versions over time
17:26:33 <mtreinish> davidkranz: yeah there is no guarentee something won't change between versions so we just have too test both
17:26:56 <sdague> davidkranz: nova is duplicating the code paths internally for the same reason
17:27:05 <davidkranz> mtreinish: Yes but I hope we can find a way to duplicate dynamically but not statically.
17:27:21 <sdague> davidkranz: the resource structures are going to change as well
17:27:22 <afazekas_> _interface='xml'
17:27:23 <davidkranz> mtreinish: The test code will be the same if the behavior has not changed.
17:27:38 <sdague> davidkranz: the reason for the new api is because it did change
17:27:47 <mtreinish> davidkranz: not necessarily things like return codes will probably change
17:27:50 <sdague> if it didn't change, you wouldn't need a bump
17:27:56 <davidkranz> sdague: Every api changed?
17:28:01 <afazekas_> I am not sure the current '_interface'  solution is the nicest way how to avoid code duplication, but it works
17:28:31 <afazekas_> the services could be responsible for the error code checking
17:28:35 <davidkranz> We can continue this on the list but a real proposal is needed.
17:28:49 <sdague> davidkranz: probably not every, but enough move around that it's easy to miss a test bug inside a complicated if / else around versions
17:28:56 <afazekas_> 'services' ie. rest_clients
17:29:09 <davidkranz> sdague: OK, I did not realize it was this extensive.
17:29:12 <sdague> things like tenant ids in urls going away
17:29:22 <davidkranz> sdague: That is a localized change.
17:29:38 <davidkranz> sdague: Obviously the rest client will change.
17:29:44 <sdague> yeh, but the problem is that you can come from 2 points of view
17:29:57 <sdague> assume it is all the same, and put complexity into the tests when it's different
17:30:09 <sdague> or assume it's all different, and factor out common bits when it's the same
17:30:30 <sdague> I feel like not knowing how same or different it will end up, option 2 is safer
17:30:41 <davidkranz> sdague: I hope it is the first. Otherwise users who upgrade will suffer a lot.
17:30:56 <sdague> davidkranz: that's why it's a major version bump
17:31:06 <sdague> and why v2 will still be in havana
17:31:23 <davidkranz> sdague: If history is a guide, once we get more stable there will be a lot of resistance to changes in major versions as well unless they are really necessary.
17:32:07 <sdague> davidkranz: honestly, I don't expect v3 to be the final word, if the interfaces don't evolve over time, they don't stay relevant
17:32:18 <sdague> but that's a meta discussion :)
17:32:28 <davidkranz> sdague: Right.
17:32:34 <sdague> ok, anything else from folks
17:33:07 <davidkranz> Nope.
17:33:11 <mtreinish> nothing from me
17:33:40 <sdague> ok, thanks all
17:33:43 <sdague> #endmeeting