17:00:06 <mtreinish> #startmeeting qa
17:00:07 <openstack> Meeting started Thu Jan 30 17:00:06 2014 UTC and is due to finish in 60 minutes.  The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:08 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:11 <openstack> The meeting name has been set to 'qa'
17:00:18 <mtreinish> hi who's here today?
17:00:21 <sdague> o/
17:00:50 <mtreinish> heh, ok this'll be a quick one I guess
17:01:00 <mtreinish> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting
17:01:07 <mtreinish> ^^^ today's agenda
17:01:15 <dkranz> here
17:01:15 <andreaf> hi
17:01:19 <rahmu> hi
17:01:37 <mtreinish> so it looks like today's agenda is just the boiler plate one
17:01:41 <mtreinish> but let's dive into it
17:01:47 <mtreinish> #topic Blueprints
17:02:04 <mtreinish> so does anyone have any blueprints to bring up that need attention
17:02:11 <mtreinish> or that have a status update
17:02:26 <andreaf> ok I'll start here
17:02:40 <andreaf> I have an implementation of the multi auth bp here https://review.openstack.org/#/c/65311/
17:02:43 <giulivo> hi
17:03:34 <andreaf> I think it would be good to get that through in icehouse as keystone v2 is going to be deprecated
17:04:04 <mtreinish> andreaf: can your massive refactor wait until after mine? :) https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/config-cleanup,n,z
17:04:09 <mtreinish> it's clash of the titans
17:04:23 <dolphm> o/ identity api v2 has been officially deprecated as of icehouse-m1
17:04:29 <andreaf> it's quite a large patch,  a pain to rebase, so if anyone has ideas on how to split it ...
17:04:41 <andreaf> mtreinish: yes I think so
17:04:48 <sdague> dolphm: what is the status of the official python clients supporting v3?
17:05:29 <andreaf> mtreinish: I saw your patch both together will make the rest client __ini__ down to one parameter :)
17:05:45 <dolphm> sdague: python-keystoneclient supports it quite well from a library perspective; python-openstackclient exposes it quite well on the CLI, barring some initial authentication/configuration hurdles which represents a significant barrier to adoption
17:06:03 <sdague> dolphm: so if you use python-novaclient
17:06:09 <sdague> does it call keystone v3?
17:06:20 <dolphm> not today
17:06:23 <andreaf> sdague: I fear not
17:06:29 <sdague> what's required to make that happen?
17:06:40 <andreaf> most clients allow token / base url as input
17:06:44 <andreaf> but not nova / cinder
17:06:50 <andreaf> I filed a bug for that
17:06:59 <sdague> because I'm a little concerned about use wildly diverging here from the way people are going to use this in the field
17:07:08 <andreaf> I think all clients shall rely on keystone client for auth
17:07:10 <dkranz> This is really going to put the "client libraries should work with old openstack releases" to the test...
17:07:26 <mtreinish> sdague: well for the api tests at least there shouldn't be an issue
17:07:42 <sdague> mtreinish: sure, as long as we decide to run the v2 stuff still
17:07:50 <sdague> both ways that is
17:08:08 <andreaf> https://bugs.launchpad.net/python-novaclient/+bug/1262843
17:08:09 <mtreinish> sdague: yeah I thought that was the intent having one job with v2 and the other with v3
17:08:19 <dolphm> andreaf: we're working on exposing some neat library features (mostly taking advantage of requests "sessions") to make it much easier for other clients to fully adopt identity api v3 in a couple lines of code
17:08:19 <mtreinish> at least while v2 is still in tree
17:08:23 <sdague> right, we do start to run up against job limits
17:08:33 <sdague> so I'm cool with the cleanups this provides
17:08:41 <sdague> and the plugability
17:08:47 <dkranz> sdague: +1
17:08:49 <sdague> but I don't actually think this is icehouse critical
17:09:04 <sdague> so if it doesn't all make it, that's fine.
17:09:18 <mtreinish> sdague: I bumped it up from low to high(not critical) yesterday
17:09:33 <mkoderer> hi, sry I am late
17:09:59 <sdague> because there is clearly a missing piece in all this with the fact that we can't actually run a v3 job
17:10:05 <sdague> because we'll be using v2 all over the place
17:10:20 <sdague> or v3 only job
17:11:07 <sdague> so lets get as much as makes sense in, but this really seems to beg the question of the overall matrix, which I think is going to be a summit discussion
17:11:48 <mtreinish> yeah we need to iron out the details on the matrix
17:12:04 <mlavalle> Hi, sorry for the delay
17:12:10 <andreaf> sdague: r3 v only job I agree, but at least we'll be stressing  v3  a bit more
17:12:16 <sdague> andreaf: sure
17:12:37 <mtreinish> andreaf: ok is there anything else on the keystone v3 blueprint?
17:12:53 <andreaf> andreaf: no
17:13:00 <andreaf> mtreinish: no
17:13:07 <mtreinish> heh, ok
17:13:10 <andreaf> can't make sense in my typing :P
17:13:19 <mtreinish> does anyone else have a blueprint to bring up
17:13:19 <mtreinish> ?
17:13:24 <dkranz> mtreinish: negative test
17:13:28 <mkoderer> yep
17:13:36 <dkranz> mtreinish: It is waiting for core reviews
17:13:36 <mtreinish> dkranz, mkoderer: ok go ahead
17:13:42 <mkoderer> please review https://review.openstack.org/#/c/64733/
17:13:59 <mkoderer> I think we should merge it and fix the nits later
17:14:12 <sdague> I'm ok with that
17:14:27 <sdague> let's give people till end of day to raise objections
17:14:30 <dkranz> So it has three +1 and just needs some +2
17:14:38 <sdague> I'll take a look this afternoon
17:14:45 <mkoderer> sdague: great
17:14:46 <sdague> if there are none, I say we merge it
17:14:47 <dkranz> sdague: Thanks
17:14:52 <sdague> and fix the rest in tree
17:14:54 <mtreinish> mkoderer: yeah I've been meaning to take a closer look at it. The only concern I had is with the jsonschema being in .py files
17:15:00 <mtreinish> why not just make it a .json file?
17:15:22 <dkranz> mtreinish: The problem is that the tests have to reference a schema by name
17:15:22 <sdague> yeh, that is the one cleanup I'd kind of like first
17:15:23 <mkoderer> that was dkranz idea ;)
17:15:37 <dkranz> sdague: The json was moved to separate files
17:15:40 <sdague> dkranz: use descriptive file names
17:16:05 <dkranz> sdague: You mean a separate file for each schema?
17:16:08 <sdague> yeh, I'd still rather they be real json.
17:16:11 <sdague> dkranz: sure
17:16:14 <sdague> whatever makes sense
17:16:25 <dkranz> sdague: I didn't realize you were thinking of hundreds of new files for thies
17:17:05 <sdague> Also if they are real json, they are much more easily syncable from nova
17:17:27 <mtreinish> dkranz: yeah if we eventually have an automated method for generating them then it isn't a big deal
17:17:32 <mkoderer> sdague: ok well then let's move it
17:17:39 <sdague> oh, actually they aren't
17:17:54 <sdague> well, I think we should get to real json as separate files anyway :)
17:18:05 <sdague> so if you guys make that change, I'm +2, and we do the rest in tree
17:18:06 <dkranz> sdague: I don't object, just want to make sure we accept having a separate file for each item at api.openstack.org
17:18:18 <sdague> dkranz: yep
17:18:28 <dkranz> sdague: ok, I'm in
17:18:41 <rahmu> and are each of these files written manually or autogenerated?
17:18:54 <mkoderer> ok, I think I will fix the little nits right away and push it tomorrow
17:19:14 <dkranz> rahmu: Manually, at least for now. From what would you generate them?
17:19:14 <mtreinish> rahmu: right now they're manual but eventually there will be a script to manually create them
17:19:46 <dkranz> mtreinish: I still don't understand from what you could generate them
17:19:54 <rahmu> okay, I saw mention of syncing from nova, and didn't understand
17:20:01 <dkranz> mtreinish: They are the source from which other interesting things could be generated
17:20:17 <dkranz> mtreinish: But it would be better if they were defined in projects rather than tempest
17:20:21 <mkoderer> I don't think that you can generate them completely
17:20:27 <sdague> dkranz: you should be able to reuse a big part of the validation schema from nova, no?
17:20:34 <sdague> for the pieces that are done there
17:21:06 <dkranz> sdague: maybe, but I am not convinced. I would say it should really be the other way around
17:21:09 <sdague> I realize that today the efforts are pretty discreet
17:21:13 <mtreinish> dkranz: if the schema is queriable then all the details can be queried from nova
17:21:36 <sdague> dkranz: right the only tempest specific pieces would be the logic to go after interesting boundaries, right?
17:21:37 <mkoderer> mtreinish: I don't think that you get all the needed details
17:21:38 <dkranz> mtreinish: Our schema has more information, containing what is in nova as a subset
17:22:01 <sdague> dkranz: ok, so we definitely need to figure out how to converge those more, but we can do that in tree
17:22:08 <sdague> I accept this is a good starting point
17:22:10 <dkranz> mtreinish: The nova methodology is flawed because it did non consider all the use cases for schema
17:22:11 <mkoderer> such as expected result code for malicious input
17:22:33 <mtreinish> dkranz: I'm looking at it, you have: name, method, url, and the json schema in the files
17:22:34 <sdague> dkranz: ok, so lets figure out how to get that handled on the nova side as well
17:22:47 <dkranz> sdague: great
17:22:56 <mkoderer> ok
17:22:57 <mtreinish> if the schema is queriable what can't be done with a script?
17:22:57 <dkranz> sdague: Of course the other zillion apis have nothing at all
17:22:59 <sdague> because nova will need to be the authority on their API
17:23:05 <sdague> dkranz: agreed
17:23:16 <sdague> but at least with nova we should try to find a way to leverage the work over there
17:23:24 <sdague> then hopefully other interfaces will follow
17:23:31 <dkranz> mtreinish: It is not script or no script, just that the nova schema just describes payload
17:23:52 <mkoderer> mtreinish: it doesn't care about result codes
17:23:53 <dkranz> mtreinish: It's not enough to generate tests
17:23:54 <sdague> #action mkoderer to break out json schemas into .json files
17:24:12 <mkoderer> sdague: ok cool
17:24:15 <dkranz> mtreinish: They simply did not consider the more general use cases for schemas
17:24:35 <sdague> ok, I think we can move on  :) lets get the change, get this landed, realize we're going to need summit time to work out some of the details on schema reuse
17:24:45 <dkranz> sdague: yes
17:24:51 <mkoderer> sdague: +1
17:24:54 <mtreinish> ok yeah that's fair
17:25:08 <mtreinish> oh I see now it's not just the jsonschema in the dict you're adding to it
17:25:10 <sdague> but it's definitely moving us in the right direction
17:25:15 <sdague> progress
17:25:27 <sdague> similar thing....
17:25:30 <dkranz> sdague: The nova schema is not in files  as real json :)
17:25:32 <mkoderer> mtreinish: exactly..
17:25:37 <sdague> mtreinish's config patch
17:25:45 <giulivo> sorry guys a question from an outsider, when you say that nova will be an authority over the other apis, it means we expect to continue to have facilities in the nova api to create resources of other type (like networks or images), and not to strip it from nova, is that correct_
17:25:51 <mtreinish> ok well I've got a quick status update on the config bp
17:25:53 <sdague> please take a look, because it does a ton of clean up
17:25:57 <mtreinish> oh sdague beat me to it
17:26:13 <mtreinish> #link https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/config-cleanup,n,z
17:26:27 <sdague> and I'd like some eyes on it to make sure it's fine, then probably squash and bulk merge it
17:26:32 <mtreinish> I have one more patch on top to add
17:26:35 <dkranz> giulivo: NO, it has nothing to do with that.
17:26:55 <sdague> mtreinish: is that the patch that finally removes all the self.config assignments?
17:26:57 <mtreinish> I'm holding off because of rebase complications because masayukig approved some
17:26:58 <giulivo> ok so I must have missed something, sorry
17:27:02 <sdague> I hadn't see them yet
17:27:03 <dkranz> giulivo: Just that the api schema should be defined in a project, not in tempest ideally
17:27:04 <mtreinish> sdague: yeah
17:27:19 <mtreinish> it removes the references from manager.py and test.py
17:27:38 <sdague> honestly, unless there is an objection, I'm just going to +A that through once it looks like it works
17:27:51 <sdague> because it's going to be a rebase monstrosity
17:28:24 <sdague> and I'd rather spend our time on useful things, not rebase chasing that series
17:28:26 <mtreinish> sdague: yeah well if I push things now the sha1's on the ones in the queue will change because there is already some of that
17:28:58 <mtreinish> I was trying to make things manageable I hate giant patches
17:29:23 <mtreinish> ok I don't have anything else on this
17:29:35 <mtreinish> let move to the next topic
17:29:35 <sdague> mtreinish: get a working top patch, then we can figure out how to best get this in
17:29:44 <mtreinish> sdague: ok
17:29:52 <mtreinish> #topic Neutron testing
17:30:02 <mlavalle> Hi
17:30:03 <mtreinish> mlavalle, salv-orlando: you around?
17:30:18 <mtreinish> are there any updates on neutron testing?
17:30:20 <mlavalle> api tests development is going well
17:30:40 <mlavalle> we now have a community of debs actively engaged
17:30:50 <mlavalle> devs ^^^
17:31:06 <mlavalle> and I have been doing a lot of code reviews
17:31:09 <sdague> mlavalle: nice job
17:31:31 <sdague> is there any understanding on why the isolated jobs remain at a really high failure rate?
17:31:50 <salv-orlando> mtreinish: the patches are still waiting for the neutron gate to get back to a point where we can merge anything
17:31:58 <mlavalle> I haven't worked on that, but salv-orlando might have insight
17:32:01 <mtreinish> sdague: I thought that was the kernel namespace bug
17:32:10 <mtreinish> salv-orlando: which patches?
17:32:26 <salv-orlando> the patches for parallel testing
17:32:40 <salv-orlando> and for enabling full isolation
17:33:02 <salv-orlando> but since we are stuck on this kernel bug we can't make any progress until that is solved.
17:33:16 <sdague> salv-orlando: does that really account for all the fails?
17:33:30 <sdague> I know that's an impact
17:33:43 <sdague> but I was curious if it was the complete problem, or just one of a couple
17:34:10 <salv-orlando> sdague: not all the fails. Bug 1253896 astir lurks. But bug 1273386 is a lot more frequent
17:34:12 <sdague> also, any word on the new kernel ?
17:34:27 <sdague> and if that looks stable enough to use more broadly
17:34:33 <salv-orlando> sdague: we see the failure that too. And that's bad.
17:34:40 <sdague> ok
17:34:52 <sdague> yep, so at least we know that's not the answer
17:34:57 <mtreinish> sdague: I saw patches, from russellb I think, adding it to nodepool
17:35:08 <salv-orlando> mtreinish: experimental jobs now run on 3.11
17:35:21 <sdague> right, I wanted to know if we had enough data to sort it if it fixed
17:35:24 <mtreinish> salv-orlando: ok cool
17:35:27 <sdague> and it sounds like we do not
17:35:41 <sdague> or we do have enough data.... and it did not
17:36:03 <salv-orlando> yes sdague: we hit the bug at the 3rd recheck attempt
17:36:10 <sdague> ok
17:36:13 <afazekas> BTE anybody else see similar issues: https://bugs.launchpad.net/neutron/+bug/1274121
17:36:51 <russellb> salv-orlando: any need to keep the test nodes with 3.11?
17:36:52 <giulivo> I wonder if there are plans to upgrade later this year to the new LTS, current ubuntu is preventing other stuff from being tested too
17:37:07 <salv-orlando> afazekas: first time i see that failure
17:37:20 <sdague> giulivo: so I'm sure the 14.04 test story will be a Juno topic
17:38:00 <sdague> ok, probably worth moving on
17:38:03 <salv-orlando> russellb: it might be worth keeping them for a while, but since the stack trace is exactly the same I guess there's little we can do.
17:38:17 <salv-orlando> Actually.. I would try to push back 3.2.0-57 in experimental nodepool
17:38:26 <sdague> salv-orlando: that's a good idea
17:39:08 <sdague> because the other possibility is this existed a lot longer than we realized, and we just didn't have the tooling to see it
17:39:09 <giulivo> sdague, I'll try harder to join this time
17:39:15 <salv-orlando> because internally I've been unable to reproduce either on 3.2.0-57 and 3.2.0-58 but going back in time the first time we saw this issue it was with kernel-3.2.0-54
17:40:00 <sdague> yeh, I expect this may be a case of a problem that we didn't know we had until recently, and would explain overall neutron gate stability
17:40:10 <sdague> giulivo: cool
17:40:25 <sdague> mtreinish: ok, what's next? :)
17:40:43 <sdague> actually, before that, who's volunteering to handle moving the experimental kernel?
17:40:43 <mtreinish> #topic Bugs
17:40:58 <sdague> let's make sure we get a volunteer on that
17:40:59 <mtreinish> oh what's the fancy undo command?
17:41:50 <sdague> #info need volunteer to put experimental nodepool on 3.2.0-57 kernel
17:41:56 <sdague> ok, I guess we move on
17:42:02 <mtreinish> ok
17:42:15 <mtreinish> so does anyone have any bugs that need attention
17:42:20 <NithyaG__> Hi,  I have a query on Nova tests. I would like to know if its a good idea to get a few additional tests on live migration added, though its only single node deployment in gate as of today, it will be good to build the test repository. I have raised a few bugs today for adding test cases. (both API and scenario tests)
17:42:24 <mtreinish> or are there any critical bugs not getting any attention
17:42:55 <sdague> NithyaG__: so the problem is we don't really have any way to validate them
17:44:02 <sdague> guess no one needs bug attention
17:44:05 <sdague> moving on :)
17:44:05 <andreaf> sdague: how's the work on multinode gate proceeding?
17:44:09 <mtreinish> ok then let's move on
17:44:15 <sdague> anderstj: no one is working on it
17:44:28 <sdague> andreaf: that should have been
17:44:37 <andreaf> sdague: oh I see
17:44:39 <dkranz> sdague: Is there a description of the issues with multinode anywhere?
17:44:50 <dkranz> sdague: Or some unimplemented blueprint or plan?
17:44:57 <andreaf> andreaf: we already have a few live migration and resize tests in
17:45:05 <sdague> dkranz: not that I know of, no one has ever taken that blueprint as their primary task
17:45:06 <andreaf> sdague: I meant
17:45:20 <sdague> andreaf: so resize we can do in the gate
17:45:23 <dkranz> sdague: ok, I get it
17:45:26 <sdague> because you can resize to yourself
17:45:42 <sdague> we added a flag to nova to let that happen in the gate
17:45:59 <mtreinish> andreaf: I've never run live migration and I've never heard of anyone using it
17:46:06 <mtreinish> I'm sure the code has a lot of bit rot
17:46:11 <mtreinish> since it doesn't get run in the gate
17:46:14 <andreaf> sdague: oh, ok
17:46:35 <mtreinish> ok we can save this for later if there is anything else on it
17:46:39 <sdague> yep
17:46:40 <mtreinish> #topic      Critical Reviews
17:46:52 <mtreinish> does anyone have any reviews that they would like to bring up?
17:47:24 <sdague> only the ones we talked about in blueprints
17:47:35 <mtreinish> yeah I wasn't going to bring those up again
17:48:01 <mtreinish> I guess there aren't any others then
17:48:03 <mlavalle> mtreinish: over the weekend I will put together a list of Neutron api tests that need core reviewers a ttention. I will send it to the ML
17:48:19 <sdague> sounds great
17:48:34 <afazekas> may be I missed the blueprint parts, but this https://review.openstack.org/#/c/65311/  definitely should get more ayes
17:48:36 <mtreinish> mlavalle: ok sounds good. But I'm hesitant to merge anything until the neutron gate is in a better state though
17:48:37 <sdague> also, given the long run times due to the concurency drop
17:48:53 <sdague> please keep an eye on new tests that take a long time, we shouldn't be adding them right now
17:49:14 <rahmu> it would be great to get some input on this one: https://review.openstack.org/#/c/62887/ We'd like to know how to avoid hardcoding the projects' names
17:49:20 <sdague> hopefully some nova fixes will go in and let us run faster again, but until then, lets not kill ourselves too hard
17:49:23 <dkranz> sdague: So what about a test that takes 5s?
17:49:24 <mtreinish> #action sdague to send an email to list about test run time and reviewing
17:49:28 <andreaf> afazekas: thanks yes I mentioned that during the bp part
17:49:32 <mlavalle> ok, I'll be careful with that. I'll focus on api tests that don't take long
17:49:33 <sdague> dkranz: 5s isn't troubly to me
17:49:44 <sdague> 30s is
17:49:55 <dkranz> sdague: ok, reasonable
17:50:20 <sdague> afazekas: we talked about 65311 previously
17:50:27 <mtreinish> #link https://review.openstack.org/#/c/62887/
17:50:30 <dkranz> sdague: I am actually suspicious that our setup time is getting large and could be improved
17:50:37 <mtreinish> rahmu: I wonder if that should really be a bp instead
17:50:56 <dkranz> sdague: The sequential output of tests that do almost nothing is quite slow
17:50:58 <rahmu> mtreinish: that's the kind of attention this patch also needs :)
17:51:03 <giulivo> well I've one which surely is going to take time but it's the first and only test for this feature I think: https://review.openstack.org/#/c/67616/
17:51:10 <dkranz> sdague: I am going to look at that
17:51:34 <sdague> dkranz: so I'm actually working on something with lifeless's help that I think makes it a little easier to see what's going on
17:51:45 <sdague> https://review.openstack.org/#/c/70111/
17:51:51 <mtreinish> giulivo: yeah that will add some run time probably :)
17:52:03 <sdague> oh, on that front, the basic ops test
17:52:07 <sdague> that we run on 3 images
17:52:14 <sdague> why do we do that again?
17:52:20 <giulivo> mtreinish, I can remove the gate tag but we didn't have anything about backups so ...
17:52:38 <mtreinish> giulivo: removing the gate tag won't do anything actually
17:52:57 <dkranz> sdague: I will poke around some on these issues. It has been a while since we looked at such things
17:52:58 <giulivo> sorry yeah , add slow
17:53:30 <mtreinish> sdague: it was a feature add recently for running with different image types which in the gate really isn't that useful
17:53:42 <sdague> right, I see the poc value
17:53:49 <sdague> but in the gate, I think we should turn that off
17:53:55 <mtreinish> I think andreaf made it configurable
17:54:01 <sdague> it's not really catching anything, and just takes up time
17:54:08 <mtreinish> so we should just use it for one image type
17:54:13 <sdague> I was also wondering about the loader
17:54:19 <sdague> so where we skip tests
17:54:31 <sdague> skip whole classes
17:54:35 <andreaf> mtreinsh, sdague: yes it is configurable which images and which flavours
17:54:44 <sdague> is it possible to make the load just not register those classes?
17:55:05 <sdague> andreaf: ok, could you propose something which defaults it back to only 1?
17:55:17 <andreaf> sdague: ok will do
17:55:21 <sdague> thank you
17:55:36 <sdague> mtreinish: thoughts on doing class level skip by the loader?
17:55:57 <mtreinish> sdague: I don't think that's how skip works. I think it just catches the exception and goes from there. I have to double check
17:56:07 <sdague> mtreinish: I know that's not how skip works
17:56:21 <mtreinish> sdague: what exactly are you proposing?
17:57:10 <mtreinish> sdague: how about we save the discussion for after the meeting
17:57:25 <mtreinish> because we're down to ~3 min?
17:57:26 <sdague> is it possible to have test_discover not discover a set of classes if we decorate them correctly
17:57:28 <sdague> sure
17:58:02 <mtreinish> sdague: oh I'm not sure on that, maybe
17:58:18 <sdague> it would give us less chance of completely unbalancing testr
17:58:28 <giulivo> I have this crazy idea that we should slowly drop from nova api what should be done using $service api and as a consequence, remove duplicated tests
17:58:30 <mtreinish> we can set patterns to match
17:58:38 <mtreinish> so maybe we can play games there
17:58:43 <sdague> yep
17:58:49 <sdague> ok, worth exploring
17:59:06 <sdague> anything else from folks?
17:59:52 <mtreinish> ok I guess we'll end here since we're out of time
17:59:56 <mtreinish> thanks everyone
17:59:57 <sdague> ack
17:59:59 <sdague> thanks everyone
17:59:59 <mtreinish> #endmeeting