17:00:19 <mtreinish> #startmeeting qa
17:00:19 <openstack> Meeting started Thu Aug 29 17:00:19 2013 UTC and is due to finish in 60 minutes.  The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:20 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:22 <openstack> The meeting name has been set to 'qa'
17:00:28 <afazekas> hi
17:00:31 <mkoderer> hi
17:00:31 <mtreinish> hi who's around for the meeting?
17:00:41 <mkoderer> google hangout? https://plus.google.com/hangouts/_/0d06aa2bdbfda14c66339811d843480446a54ab3?hl=en ?
17:00:51 <giulivo> me
17:01:09 <giulivo> guys, I would prefer we not use hangout
17:01:12 <mtreinish> here's today's agenda: https://wiki.openstack.org/wiki/Meetings/QATeamMeeting
17:01:15 <mtreinish> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting
17:01:24 <mkoderer> just for video
17:01:26 <mkoderer> ok np
17:01:45 <mtreinish> lets get started I guess it's a pretty long agenda today
17:01:54 <mtreinish> #topic testr status
17:02:01 <dkranz> Here
17:02:16 <mtreinish> so the big news on testr is that we are now running in parallel for the gate
17:02:24 <mtreinish> and everything has been made parallel by default
17:02:35 <mtreinish> so at this point its just bug triage and fixes
17:02:45 <dkranz> mtreinish: Very cool!
17:02:45 <mtreinish> to make things as stable as they were running serially
17:03:24 <mtreinish> giulivo: you wanted to talk about a bug day related to testr?
17:03:48 <giulivo> mtreinish, yeah maybe we can send to the -dev list an announcement for a bug triage day?
17:04:12 <giulivo> let's say something in mid september? would that work for you guys?
17:04:29 <mtreinish> giulivo: sure that's probably a good idea, it's been a while since we last had one
17:04:34 <mtreinish> and the bug list is getting biggert
17:04:40 <dkranz> giulivo: Sure
17:05:02 <giulivo> ok I'll do that tomorrow at last
17:05:17 <mtreinish> giulivo: do you want to take on the effort of organizing it and pushing that forward?
17:05:54 <giulivo> are there preferences for a particular day? a week day I suggest rather than during the weekend, apart from that, I'd pick something from the 2nd week of sept
17:06:34 <mtreinish> giulivo: that's fine with me but we can discuss the details on -qa or the ML
17:06:46 <tkammer_> Hi, sorry for being late
17:06:55 <dkranz> tkammer_: np, welcome
17:07:08 <mtreinish> #action giulivo to setup a bug day for sometime in sept.
17:07:13 <mlavalle> I am late as well, sorry
17:07:20 <giulivo> mtreinish, fine with me
17:07:42 <mtreinish> ok and dkranz you want to talk about writing tests in parallel and a process for approving them in the gate
17:08:19 <dkranz> mtreinish: Some of the recent problems made it clear that it is not obvious some times
17:08:30 <dkranz> when a test will interfere with some other
17:08:46 <dkranz> We have locks within a test class, and tenant isolation, but other issues can occur
17:09:04 <dkranz> So my thought was to have a way to mark a test to not fail the build if that test fails
17:09:23 <dkranz> Then after some time it is removed and the test becomes part of the gate.
17:09:29 <mkoderer> I think we will have the same problems for stress tests that run in parallel
17:09:42 <dkranz> mkoderer: But stress tests are not gating
17:09:49 <mkoderer> thats right
17:09:53 <dkranz> mkoderer: It is the gate, and blocking builds, that concerns me.
17:10:02 <mkoderer> dkranz: correct
17:10:18 <dkranz> I am going to send a message to the list with some guidance.
17:10:29 <mtreinish> dkranz: I'm not opposed to having a nonvoting set of jobs first, it's just in my experience nonvoting jobs don't ever get any attention
17:10:43 <afazekas> dkranz: how to prevent adding racy codes to nova or neutron ?
17:11:03 <mtreinish> and flaky tests are going to happen
17:11:19 <dkranz> afazekas: We can't, but nova developers are constantly aware of this issue.
17:11:35 <dkranz> In tempest we cover all apis together which is more difficult.
17:11:37 <mtreinish> I'd rather see them gate and catch more things
17:11:56 <mtreinish> even if it causes headaches sometimes with flaky fails
17:12:00 <dkranz> mtreinish: ok, as long as our fearless ptl signs on
17:12:08 <giulivo> dkranz, is the suggestion to avoid tagging the tests with 'gate' or to use some different particular tag?
17:12:25 <dkranz> giulivo: My idea was a different tag
17:12:44 <mtreinish> dkranz: because flaky fails are bugs, albeit sometimes obscure ones
17:12:58 <mtreinish> dkranz: ok yeah sdague can weigh in on this when he's back next week
17:13:11 <dkranz> mtreinish: I know. I guess I'm just a little more conservative
17:13:22 <dkranz> But I can go with the flow
17:13:34 <mtreinish> dkranz: heh, ok
17:13:36 <dkranz> That's it
17:13:54 <mtreinish> I still think moving forward we should have a wiki page or a section in the docs about potential parallel issues
17:13:58 <afazekas> I do not see this kind of issues fixed in fast way: https://bugs.launchpad.net/neutron/+bug/1211915, https://bugs.launchpad.net/tempest/+bug/1205344
17:13:59 <uvirtbot> Launchpad bug 1211915 in neutron "Connection to neutron failed: Maximum attempts reached" [High,Confirmed]
17:14:41 <mtreinish> lets move on, afazekas we can talk about that during the neutron topic
17:14:50 <mtreinish> #topic stress tests status
17:14:56 <mkoderer> ok
17:15:02 <mtreinish> mkoderer: you're up
17:15:09 <mkoderer> so we have a decorator now @stresstest
17:15:29 <mkoderer> what is left is to identify good candidates
17:15:43 <mkoderer> to get this decorator
17:15:54 <mtreinish> mkoderer: do you have a list of criteria for what makes a good stress test candidate
17:15:55 <mkoderer> maybe someone could help me with this task
17:16:02 <mtreinish> so people can help you find them
17:16:14 <mkoderer> mtreinish: yes would make sense to have such a list
17:16:25 <mkoderer> I will prepare it and put it into the README
17:16:31 <afazekas> mkoderer: https://bugs.launchpad.net/tempest/+bug/1205344
17:16:33 <giulivo> mkoderer, mtreinish maybe the common create_get_delete tests we have around for the various resources could be a start?
17:16:34 <uvirtbot> Launchpad bug 1205344 in nova "mkfs error in test_stamp_pattern" [High,Confirmed]
17:16:55 <mtreinish> giulivo: yeah those would probably work well
17:17:06 <mtreinish> mkoderer: yeah that's probably a good place to put it
17:17:19 <afazekas> The tests cases  which caused flaky issues are usually good candidates IMHO
17:17:20 <mtreinish> and maybe start a thread on the ML about doing this
17:17:31 <dkranz> mkoderer: If we think this is important there also might be things that are close but must need a tweak to be able to be stress candidates.
17:17:52 <mkoderer> dkranz: yes right
17:17:53 <dkranz> mkoderer:    "just need a tweak"
17:18:14 <mkoderer> so ok first step preparing a definition
17:18:32 <mkoderer> and I am happy about any patch that introduces new stress tests :)
17:18:47 <mtreinish> mkoderer: ok cool
17:18:50 <mtreinish> good work on this
17:18:53 <mtreinish> is there anything else?
17:19:04 <mkoderer> don't think so
17:19:14 <mtreinish> ok then lets go to the next topic
17:19:20 <mtreinish> #topic neutron status
17:19:26 <mtreinish> mlavalle: you're up
17:19:50 <mlavalle> mtreinish: I've been updating progress in https://etherpad.openstack.org/gate-tempest-devstack-vm-quantum-full
17:20:47 <mtreinish> #link https://etherpad.openstack.org/gate-tempest-devstack-vm-quantum-full
17:20:55 <mlavalle> We have fixed several things. By next week I will compare what is in the eteherpad with the logs
17:20:55 <mtreinish> ok that's got a lot of details
17:21:14 <mlavalle> and will provide a summary for the team in this meeting
17:21:31 <mtreinish> mlavalle: ok, cool
17:21:45 <mlavalle> so we can asses how far we are from fixing that gate job
17:22:19 <mtreinish> one other thing is that since everything else is in parallel now we probably won't push this voting until that works with neutron
17:22:24 <mtreinish> even if the issues are all fixed
17:22:34 <dkranz> mtreinish: Yes
17:22:43 <mlavalle> cool with me
17:23:05 <mlavalle> I will aslso be pushing code this weekend for managing networks in the isolation code
17:23:10 <mtreinish> so https://bugs.launchpad.net/tempest/+bug/1216076 should probably be priority
17:23:12 <uvirtbot> Launchpad bug 1216076 in tempest "Neutron jobs won't pass tempest running in parallel" [Critical,Confirmed]
17:23:33 <mtreinish> mlavalle: ok, cool hopefully that will be all thats required for making neutron work in parallel
17:23:48 <mlavalle> i'm crossing my fingers
17:23:54 <mlavalle> ;-)
17:24:35 <mtreinish> ok is there anything else about neutron that needs to be brought up?
17:24:41 <mtreinish> afazekas: you had a bug earlier?
17:24:43 <mlavalle> nope. I'm done
17:26:11 <dkranz> mtreinish: Next topic?
17:26:18 <mtreinish> ok yeah
17:26:26 <mtreinish> #topic Update on slow heat gating job
17:26:29 <afazekas> mtreinish: there is mysql side lock wait issue like issue which causes gate issues on single thread
17:26:30 <mtreinish> dkranz: this one is yours
17:26:45 <dkranz> afazekas: Are you looking at that?
17:27:54 <afazekas> dkranz: I did not seen myself yet..
17:28:06 <dkranz> afazekas: Is there anything to discuss now?>
17:28:30 <afazekas> dkranz: no
17:28:37 <dkranz> afazekas: ok
17:28:43 <afazekas> let's continue with heat
17:28:51 <dkranz> The slow heat status is that everything is in place.
17:29:11 <dkranz> THere is a job on the new "experimental" queue.
17:29:25 <mtreinish> dkranz: ok cool
17:29:28 <dkranz> The current problem is that the autoscale test cannot run because there is a missing image needed.
17:29:49 <mtreinish> dkranz: is that a devstack issue? or a devstack-gate problem?
17:29:54 <dkranz> sbaker and lifeless are working on getting that image into devstack
17:30:21 <mtreinish> dkranz: ok, once that gets solved is the plan to add things to gate queue?
17:30:26 <dkranz> Once the tests run we can move them into the gate for all projects
17:30:41 <mtreinish> dkranz: ok very cool
17:30:44 <dkranz> I would recommend non-voting at first :)
17:31:00 <mtreinish> good work on this, it'll be nice to get coverage for another project in the gate
17:31:18 <dkranz> mtreinish: Yes. sbaker will let us know when we can proceed
17:31:29 <dkranz> mtreinish: I also put the job as experimental in devstack for debugging
17:31:36 <dkranz> That's it.
17:31:56 <mtreinish> dkranz: ok so one quick thing how do we run things in the experimental queue?
17:32:09 <mtreinish> because this is new as of this week right?
17:32:19 <dkranz> mtreinish: With a "check experimental" in a review comment. Yes this is new.
17:32:40 <dkranz> mtreinish: You can see the definition in layout.yaml in zuul
17:32:48 <mtreinish> #info to run experimental jobs leave a "check experimental" in a review comment.
17:32:57 <mtreinish> dkranz: ok cool thanks
17:33:06 <mtreinish> then lets go to the next topic
17:33:10 <mtreinish> #topic critical reviews
17:33:21 <mtreinish> ok does anyone have any reviews that they would like to bring up
17:34:13 <dkranz> mtreinish: Just that there are a lot of patches with +2 that are waiting for approval.
17:34:14 <afazekas> tkammer?
17:34:20 <tkammer_> yeah
17:34:39 <mtreinish> dkranz: ok yeah, I've been a bit lax with the reviews because of all the parallel triage stuff
17:34:47 <mtreinish> I'll take some time later today to go through those
17:34:56 <dkranz> mtreinish: I understand. We are light reviewers
17:35:14 <mtreinish> dkranz: well just 1 this week :)
17:35:15 <tkammer_> afazekas, are we talking about the auto config review yet?
17:35:19 <dkranz> mtreinish: And a bunch are waiting for non-red-hat
17:35:25 <mtreinish> and I'm being furloughed next week so I'll be out
17:36:06 <mtreinish> ok if there aren't any reviews lets go to the next topic
17:36:07 <dkranz> mtreinish: Sorry to hear that. I will be around next week but out on Thursday so will miss next meeting.
17:36:18 <afazekas> tkammer_: this is the part if you have review related question, you can ask it.
17:36:29 <mtreinish> #topic Testing client library compatibility
17:36:37 <mtreinish> dkranz: this is yours too
17:36:40 <tkammer_> o.k then, I want to bring up the review of
17:36:47 <dkranz> tkammer_: Go ahead
17:36:53 <tkammer_> https://review.openstack.org/#/c/42920/
17:37:02 <tkammer_> I wanted to ask a couple of things, but first
17:37:16 <tkammer_> I want to know what is the prefered way of implementing this
17:37:16 <mtreinish> tkammer_: you've got the whole next topic about this
17:37:26 <tkammer_> mtreinish, o.k then, I'll wait :)
17:37:56 <dkranz> So the library issue is what I sent to the stable-branch ml and got no response.
17:38:12 <dkranz> We say that libraries are supposed to work with older releases but nothing tests that.
17:38:37 <dkranz> I proposed to add a gate job to each client lib that runs the last stable release but with master for client libs.
17:38:59 <dkranz> And runs just the scenario tests or anything else that uses client libs.
17:39:12 <mtreinish> dkranz: that sounds like a good thing to have
17:39:21 <dkranz> I think that is straightforward technically but wanted input.
17:39:26 <mtreinish> that should just be the cli tests and scenario at this point
17:39:32 <dkranz> mtreinish: Yes
17:39:45 <mtreinish> dkranz: I'm all for it
17:40:02 <dkranz> mtreinish: I wanted to make sure there were no gotchas with this approach.
17:40:26 <mtreinish> dkranz: there is probably something on the devstack side with the global requirements stuff in there
17:40:27 <dkranz> mtreinish: I don't know whether to interpret silence as consent or just no one wanted to respond.
17:40:54 <dkranz> mtreinish: I think it is really just running a stable branch job but pointing the ENV for clients to master.
17:41:08 <giulivo> dkranz, I think it's great, it'd be nice to tests latest clients with latest two releases maybe rather than just the latest
17:41:26 <dkranz> giulivo: Well, that doesn't really scale.
17:41:39 <dkranz> giulivo: I also proposed to add this test to the bitrot jobs for all older releases.
17:41:39 <clarkb> guitarzan: dkranz mordred has been owrking on something like that
17:41:49 <mtreinish> dkranz: you might want to start a separate ML thread on dev and stable maint about this to get a wider pool of opinions
17:41:52 <clarkb> when he gets back next week you may want to see what he had in mind
17:42:12 <dkranz> clarkb: OK. I didn't get any reponse on my mail to the stable-maint list
17:42:20 <dkranz> clarkb: Didn't know mordred was out
17:42:33 <giulivo> dkranz, for all older releases doesn't scale I understand but my idea is that someone has got the havana clients and logs on a grizzly release
17:42:52 <dkranz> giulivo: Yes, that is what this would be testing
17:43:09 <giulivo> ok sorry for the bad wording than
17:43:52 <mtreinish> dkranz: ok is there anything else on this topic?
17:43:57 <dkranz> mtreinish: Nope
17:44:10 <mtreinish> ok then lets move on
17:44:15 <mtreinish> #topic Devstack independent tempest usage
17:44:24 <mtreinish> afazekas: this is listed as yours but its about tkammer_ patch
17:44:30 <mtreinish> #link https://review.openstack.org/#/c/42920/
17:44:44 <mtreinish> so tkammer_ go ahead
17:44:50 <tkammer_> mtreinish, thanks
17:45:10 <tkammer_> I want to know what is the preferred way of implementing this, Python vs BASH
17:45:29 <tkammer_> I currently implemented it as BASH since I thought to go as the devstack guys did
17:45:43 <tkammer_> but if there is no objections, Python is an option as well
17:45:57 <mtreinish> tkammer_: well normally we do things in python. We only use bash if there is a particular reason to do it in bash
17:46:16 <mtreinish> it makes it easier to review I think (at least for me)
17:46:17 <afazekas> mtreinish: looks like someone copied it from the prev agenda :)
17:46:28 <tkammer_> O.k, that is why I wanted to ask in advance prior for me developing this into something more complex
17:46:30 <mtreinish> afazekas: oops :)
17:46:59 <mtreinish> tkammer_: at least for me complex bash code can be harder to read
17:47:16 <mtreinish> tkammer_: also using python will let you call the client libs directly
17:47:35 <mkoderer> +1 for python
17:47:58 <tkammer_> True, but than again, there is not much difference between calling the clients and calling the CLI :) just easier (for me) to parse the data later on :)
17:48:39 <dkranz> tkammer_: You might find some useful utilities or ideas from https://github.com/stackforge/anvil
17:48:51 <dkranz> tkammer_: A python version of devstack basically
17:49:04 <mtreinish> tkammer_: the cli output really isn't the easiest to parse. (Look at the cli tests) But calling the libs just gives you a python object.
17:49:21 <tkammer_> dkehn, interesting, thanks!
17:49:25 <tkammer_> mtreinish, I agree
17:49:31 <tkammer_> I prefer myself python
17:49:44 <tkammer_> but was not sure what is the correct way to implement this :)
17:50:08 <tkammer_> O.k, I will convert this into python and start working on a more complex initialization
17:50:10 <dkranz> tkammer_: Well there are a few votes for python and none for bash...
17:50:39 <tkammer_> dkranz, I think that says it all :)
17:50:57 <afazekas> IMHO we should start with basic python version, extended  it step by step..
17:51:11 <mtreinish> tkammer_: ok is there anything else on this topic?
17:51:37 <tkammer_> mtreinish, not on this topic, but I would like to comment about my initial use of tempest
17:51:45 <tkammer_> is it alright to raise it now?
17:51:48 <tkammer_> or should I wait?
17:52:13 <mtreinish> tkammer_: we've got another topic to go and <10min so can you hold it for after the meeting?
17:52:18 <dkranz> tkammer_: I think sending that to the qa list would be good value
17:52:33 <dkranz> tkammer_: And easier to type in an email :)
17:52:42 <tkammer_> dkranz, mtreinish o.k, I will wait with it :)
17:52:51 <mtreinish> ok then lets go to last topic on the agenda
17:52:55 <mtreinish> #topic BP proposal: rework "skip test" functionaly
17:52:57 <dkranz> tkammer_: Or discuss in #openstack-qa
17:53:04 <mtreinish> mkoderer: this is yours
17:53:15 <dkranz> mtreinish: Sorry, I have to run out.
17:53:21 <mtreinish> dkranz: ok np
17:53:30 <mkoderer> I remeber that we discussed about skipping interface one of the last meetings
17:53:33 <dkranz> mtreinish: Talk to you in two weeks
17:53:41 <mtreinish> dkranz: yep cya
17:53:43 <mkoderer> and I had a disussion about it with giulivo
17:54:01 <mkoderer> #link https://review.openstack.org/#/c/43490/
17:54:25 <mkoderer> so would it make sense to add a blueprint to change all needed things in that area
17:54:46 <mtreinish> mkoderer: that commit is just to make skip messages easier to parse for the skip_tracker
17:54:48 <mkoderer> e.g. instead of duplicating code like "@testtools.skip('Skipped until the Bug #1170718 is resolved.')"
17:54:50 <uvirtbot> Launchpad bug 1170718 in nova "nova list --ip filter shows all instances" [Undecided,Fix released] https://launchpad.net/bugs/1170718
17:55:11 <mkoderer> I would like to have a decorator @skip(bug=12345)
17:55:42 <giulivo> mkoderer, I think that would be nice and could also help the accounting of the skips
17:55:57 <mtreinish> mkoderer: that's an interesting idea, but I think having the flexibility in the message is needed
17:56:12 <giulivo> maybe we should ensure bug= accepts a list as argument :P
17:56:33 <mkoderer> mtreinish: the field "bug" could be optional
17:56:42 <mkoderer> and we add a field "message"
17:56:48 <mkoderer> I think we could find a way
17:57:11 <mkoderer> so maybe I will brain storm a bit what we can do in that area
17:57:17 <mkoderer> and put it on the ML?
17:57:27 <afazekas> Sooner or later we will have zillion of decorators, can we have one decorator, can we have single decorator on all test classes for centralizing decorator magic ?
17:57:35 <mtreinish> mkoderer: I'm not opposed to this idea, I like it. But we've got a lot of other things on the bp list already
17:57:47 <mtreinish> afazekas: I'm fine with having lots of decorators
17:57:56 <mtreinish> we may want to centralize their definition though
17:58:00 <mtreinish> like a utils file
17:58:04 <afazekas> What if we would have json or csv file with test_method name, and bug numbers for skipping ?
17:58:13 <mtreinish> mkoderer: yeah starting a ML thread would be the right way to do it
17:58:29 <mtreinish> and you can open up the bp too for that discussion as well
17:58:34 <mkoderer> mtreinish: yes it's a question of prioritization
17:59:03 <afazekas> mtreinish: ok
17:59:03 <mkoderer> just wanted to know if you are interested in general
17:59:17 <mtreinish> mkoderer: everyone
17:59:27 <mtreinish> 'is always interested in improvments
17:59:37 <mtreinish> it's just a matter of time to get to everything
17:59:40 <mkoderer> :) ok - I will put on the ML
17:59:53 <mtreinish> ok so we're basically out of time
17:59:58 <mtreinish> #endmeeting