17:03:58 <sdague> #startmeeting qa 17:03:59 <openstack> Meeting started Thu Jul 11 17:03:58 2013 UTC. The chair is sdague. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:04:00 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 17:04:02 <openstack> The meeting name has been set to 'qa' 17:04:35 <sdague> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting 17:04:46 <sdague> ok, lets get rolling with the agenda 17:04:46 <dwalleck> And me 17:04:56 <sdague> #topic QA Program Business 17:05:25 <sdague> so QA is now an official program, which means we need a PTL and a program description 17:05:52 <sdague> I believe I'm still the only PTL nominee as of this meeting, so is there any objection to closing the nominations? 17:06:01 <mtreinish> sdague: nope, congrats 17:06:20 <dwalleck> nope, sounds like a good idea to me 17:06:21 <mlavalle> sdague: congrats Mr. PTL 17:06:34 <sdague> ok, thanks :) 17:06:34 <afazekas> sdague: congrats 17:06:51 <sdague> ok, topic 2 in QA 17:07:04 <sdague> #info sdague officially now QA PTL 17:07:17 <sdague> ok, next topic that is 17:07:22 <sdague> #topic QA Program Description 17:07:35 <sdague> I think we've got a pretty fruitful email thread going on on this one 17:07:57 <sdague> so I'd mostly like to just highlight that, and ask people to participate in it if they have other ideas 17:08:26 <sdague> I'll try to assimilate it all tomorrow into a mission statement we can take forward to the TC 17:08:35 <sdague> and get another round of feedback on it 17:08:51 <mtreinish> sdague: ok, sounds good to me 17:09:11 <sdague> if you haven't been on the ML thread it's here - http://lists.openstack.org/pipermail/openstack-qa/2013-July/000602.html 17:09:13 <sdague> #link http://lists.openstack.org/pipermail/openstack-qa/2013-July/000602.html 17:09:22 <afazekas> sdague: IMHO tools should be generally useabel both for upstream and downstream usage 17:09:38 <sdague> afazekas: ok, you want to get back on the thread there? 17:09:49 <sdague> I think that's the right place to drive the discussion 17:09:57 <sdague> as it lets everyone participate 17:10:15 <afazekas> sdague: yes 17:10:21 <sdague> cool, great 17:10:27 <dwalleck> afazekas: Agreed. A tool is just a tool. It should be flexible enough to use anywhere 17:10:55 <sdague> ok, so lets get to Blueprints 17:11:10 <sdague> #topic Blueprints 17:11:35 <sdague> #link https://launchpad.net/tempest/+milestone/havana-2 17:12:09 <mtreinish> sdague: when is the h2 deadline? 17:12:13 <sdague> can people provide #info updates to blueprints they are working 17:12:18 <sdague> h2 is a week from now 17:12:33 <sdague> #info h2 happens July 18 17:13:16 <sdague> let's at least ping folks on the high and up ones 17:13:23 <sdague> mtreinish: how's testr looking? 17:13:42 <mtreinish> sdague: so I've respun the testrepository patch based on lifeless's review 17:13:42 <sdague> afazekas: how's the leak detection looking? 17:13:45 <afazekas> I have small POC code , where I can add new detecors/monitors by 5-8 lines https://review.openstack.org/#/c/35516/, I before I am starting puppuliation it with new detectors, I would like to knw is the basic frame ok or not 17:13:47 <mtreinish> it's pending for review: https://code.launchpad.net/~treinish/testrepository/testrepository-group-regex/+merge/173749 17:14:14 <mtreinish> I also have a tempest patch using that approach to move over run_tests.sh and not gating tox jobs to use testr --parallel pending 17:14:25 <sdague> afazekas: ok, I'll take a look this afternoon on the patch 17:14:36 <mtreinish> I can push it as WIP but it won't work unless you patch testr with those commits 17:14:43 <sdague> #info review requested on afazekas' leak detector patch https://review.openstack.org/#/c/35516/ 17:15:13 <sdague> #info testr work awaiting review on testr code change - https://code.launchpad.net/~treinish/testrepository/testrepository-group-regex/+merge/173749 17:15:26 <sdague> mtreinish: yeh the WIP might be nice for folks to just see 17:15:30 <afazekas> mtreinish: can we add temporary testr fork to tempest, until your patch not merged ? 17:16:05 <mtreinish> afazekas: I guess we could, but I'm reluctant to do that if lifeless comes back with more changes it'll just diverge more when I make the changes 17:16:08 <sdague> afazekas: I think we should avoid that, as we'll be doing dual maintenance until testr gets the features 17:16:14 <mtreinish> sdague: ok, I'll push it out after the meeting 17:16:37 <afazekas> ok 17:16:38 <sdague> adalbas: you have an update on the grenade enablement in the gate? 17:17:15 <sdague> I think that's the last high priority blueprint that's up there that we've got folks around for 17:18:34 <sdague> ok, well lets move on to reviews 17:18:53 <sdague> #topic Critical Reviews 17:19:07 <afazekas> https://review.openstack.org/#/c/33211/ 17:19:09 <sdague> ok, critical reviews people want to get more eyes on? 17:19:16 <sdague> #link https://review.openstack.org/#/c/33211/ 17:20:01 <sdague> afazekas: ok, I'll relook at that 17:20:05 <sdague> as I'm the only one blocking it 17:20:06 <afazekas> I do not really see why we need this kind of tests in tempest 17:20:18 <afazekas> sdague: ok, thx 17:20:56 <afazekas> https://review.openstack.org/#/c/35165/ 17:21:19 <sdague> #link https://review.openstack.org/#/c/35165/ 17:21:34 <sdague> ok, I'm openning up tabs to go take a look post meeting 17:21:39 <sdague> any other critical reviews? 17:21:54 <afazekas> https://review.openstack.org/#/c/36367/ 17:22:19 <dwalleck> Is this supposed to be some sort of state-machine like verification? 17:22:20 <sdague> #link https://review.openstack.org/#/c/36367/ 17:22:22 <afazekas> The real heat scenario tests needs a lot of time, how we will be able to handle it 17:22:39 <afazekas> Probably the same will be true for the ceilometer tests 17:23:14 <sdague> afazekas: so we were actually discussing heat and ceilo in -infra yesterday 17:23:38 <sdague> I think we're going to try to handle ceilo by decorators on existing tests, dhellmann_ liked the idea, and was going to propose something 17:24:02 <afazekas> cool 17:24:27 <sdague> for heat, I think we might want to make a separate heat gate job. But we'll see. 17:24:50 <sdague> I guess a big part of landing those heat tests is if the testr patch is accepted and we can get that in within a week 17:24:58 <sdague> if so we can just hold the heat patch until it's ready 17:25:10 <sdague> otherwise we should come up with an alternative that lets them be in the gate 17:25:13 <afazekas> ok, BTW: I tried to but together what I know about the heat tests .. : https://wiki.openstack.org/wiki/Blueprint-add-basic-heat-tests , I would like confirm it with heat devs 17:25:23 <sdague> afazekas: great 17:25:35 <sdague> #link https://wiki.openstack.org/wiki/Blueprint-add-basic-heat-tests heat test information 17:25:59 <sdague> ok, any other reviews? 17:26:28 <sdague> #topic Stress Tests in CI 17:26:51 <sdague> was this a dkranz thing? or a jog0-away thing? 17:27:15 <sdague> that's not promissing that neither are around, maybe we skip and see if jog0-away or someone else pops back up in a bit 17:27:28 <afazekas> mkoderer: ^^^ 17:27:35 <mtreinish> sdague: dunno, but I setup a periodic for the stress tests last week 17:28:04 <mtreinish> there is a tox job that gets used for it so if there are new stress tests just add them to the tox job 17:28:08 <sdague> mkoderer: is that your thing? 17:28:14 <mkoderer> no it's not 17:28:18 <sdague> which I also see at the end of the agenda 17:28:34 <sdague> ok, well why don't we take your topic now too, as it seems like it's related 17:28:44 <mkoderer> I know that dkranz wanted to do something with CI and stress tests 17:28:55 <mkoderer> yes sure 17:29:07 <sdague> mkoderer: or should we just take it to the mailing list because dkranz isn't around, and we want to get him in on it as well? 17:29:08 <afazekas> IMHO we should add more stress periodic jobs in the future.. 17:29:21 <hemna> so a while back we wrote a cinder stress test to bang on our 3par drivers. we recently published it on github 17:29:32 <afazekas> As I remember an another stress test was proposed on the mailing list 17:29:51 <sdague> yeh, I really think like we've got the proliferation of different but similar approaches 17:29:59 <hemna> we've been running it to find issues w/ our drivers and cinder and nova 17:30:37 <mkoderer> sdague: I spoke to dkranz we can discuss it here now 17:30:44 <sdague> mkoderer: go for it 17:30:52 <afazekas> IMHO, nowadays the openstack has too many race issue, so we should put more effort on the stress testing .. 17:30:57 <mkoderer> so we are planning a stress test for the new backup function in cinder 17:31:08 <mkoderer> so question .. .what is the right tool? ;) 17:31:10 <sdague> hemna: so how similar, or different is it from the stress stuff in tempest today? 17:31:20 <mkoderer> I think tempest stress test looks quite good 17:31:32 <hemna> I haven't seen the tempest stress stuff yet 17:31:43 <sdague> hemna: can you take a look? 17:31:52 <hemna> sure 17:31:58 <mkoderer> so I already put a new test for review 17:32:01 <sdague> it would be nice if we could figure out a way to have everyone playing in one stress test tree 17:32:03 <hemna> here is our stress tool FWIW : https://github.com/terry7/openstack-stress.git 17:32:11 <mkoderer> https://review.openstack.org/#/c/36652/ 17:32:11 <hemna> sdague, +1 17:32:12 <mtreinish> hemna: they're here: https://github.com/openstack/tempest/tree/master/tempest/stress 17:32:13 <sdague> so we don't duplicate infrastructure 17:32:25 <afazekas> the tempest test stuff is simpler, but configurable. The tool I saw on the ML was more complex, but less flexible 17:32:29 <hemna> mtreinish, thanks 17:32:43 <afazekas> I see good values in both tool 17:32:46 <sdague> right, I guess are they resolvable to meet everyone's needs? 17:32:57 <sdague> because maintaining 2+ tools for this is silly 17:33:02 <hemna> yah 17:33:02 <sdague> if we can avoid it 17:33:14 <afazekas> sdague: It would't be an easy job 17:33:29 <hemna> we wrote ours just as a quick and dirty tool to get the job done, and then we found it useful for finding issues 17:33:44 <sdague> hemna, mkoderer: I guess can I ask you two to get together over the next week and figure out if we could merge these efforts in some way? 17:33:45 <afazekas> we would need to increase the complexity 17:33:47 <hemna> I'd be willing to work on a stress tool, as it has great benefit for us as well as the community as a whole 17:34:00 <afazekas> cool 17:34:01 <adalbas> sdague, sorry, just joined now. 17:34:02 <mkoderer> sdague: yes sure! 17:34:25 <sdague> afazekas: well we could also spin the stress directory out of tempest if it would be dragging too much complexity there 17:34:34 <sdague> as a program we don't need to be just one git tree 17:34:58 <sdague> and we'll already be 2 really soon, grenade is going to fall under this program, dtroyer and I just need to work out a groups thing in gerrit 17:35:14 <afazekas> :), It will not be a too big difference 17:35:18 <adalbas> i have an update on grenade job in the gate, sdague 17:35:29 <sdague> adalbas: ok, lets come back around in a minute on that 17:35:53 <sdague> hemna: ok, you up for taking some time this week with mkoderer to figure out if things could merge together? 17:36:08 <sdague> I don't want to volunteer folks for things unless they have the time :) 17:36:15 <hemna> so the tempest test, is there a way to focus the testing on one component? say I want to stress cinder only 17:36:20 <afazekas> So the tempest stress tool can similar stress, but in different way. 17:36:21 <hemna> or does it run against the entire OS suite ? 17:36:56 <hemna> sdague, yah I'll take a look at the tempest stress tool and see if it can do what my tool does 17:37:04 <sdague> hemna: cool, thanks 17:37:05 <mkoderer> hemna: currently there are just 2 test inside 17:37:11 <afazekas> hemna: the basic concept of the tempest stress tool you can define worker threads with peridic stress job 17:37:30 <afazekas> for cinder it would be great if it would read the cinder's log file as well 17:37:36 <hemna> I would suppose there would be 'plugins' for each OS component 17:37:45 <sdague> #todo hemna and mkoderer to look at the possibility of making https://github.com/terry7/openstack-stress.git and https://github.com/openstack/tempest/tree/master/tempest/stress be able to become one approach 17:38:08 <hemna> and possibly parameterize the run to say...I want to test these OS components only... 17:38:10 <sdague> we'll put this back on the agenda for next week to check in on what people found 17:38:18 <afazekas> hema: https://github.com/openstack/tempest/blob/master/tempest/stress/actions/volume_create_delete.py 17:38:18 <hemna> sdague, ok 17:38:22 <dwalleck> I'd definitely like to see some more flexibility in stress testing. What I'm interested in is combinations/randomization of load. That's when things get interesting 17:38:22 <sdague> #todo revisit stress testing next week 17:38:33 <hemna> I don't see why it can't meet everyone's needs in 1 tool/codebase 17:38:47 <sdague> hemna: we should be able to poke just one component, I think that should be a design point 17:38:57 <hemna> I've found race conditions in my FibreChannel code in Nova due to my tool. been very helpful 17:39:04 <sdague> cool 17:39:08 <hemna> sdague, yah I agree. 17:39:17 <sdague> ok, great 17:39:29 <sdague> ok, lets move onto the last topic then 17:39:33 <sdague> #topic Full Open Stack networking gate job 17:39:49 <sdague> anyone around from neutron, or anyone working on that about? 17:40:17 <sdague> you know I think I'll need to start asking folks to put their names next to agenda items :) 17:40:19 <afazekas> I added this topic to the agenda 17:40:24 <sdague> ok, cool 17:40:29 <sdague> afazekas: the floor is yours 17:40:45 <afazekas> What is missing to have a full neutron voting gate job ? 17:41:25 <afazekas> As I see tens of test cases are failing for ~7 reason 17:41:38 <sdague> the biggest issue is API issues where neutron causes different error codes when called via nova 17:41:46 <nati_ueno> hi are you talking about gating issues now? 17:41:58 <sdague> nati_ueno: no, though I was about to bring that up 17:42:11 <afazekas> Can we skip those with bug until they are fixed ? 17:42:13 <sdague> because we're about to be non voting on neutron at all 17:42:16 <nati_ueno> sdague: gotcha 17:42:24 <sdague> afazekas: the problem is no one is working to fix them 17:42:32 <mlavalle> sdague, afazekas: i've been working on fixing some of those issues 17:42:40 <sdague> ok, sorry :) 17:42:47 <mlavalle> in fact, I have a review pending 17:42:49 <afazekas> Do we track / group those issues in anyway ? 17:42:54 <mlavalle> https://review.openstack.org/#/c/35724/ 17:43:04 <mlavalle> that is aimed at one of those issues 17:43:15 <afazekas> I reported one today .. 17:43:33 <sdague> mlavalle: great 17:43:50 <mlavalle> sdague: there is also Jordan Pittier working on this 17:43:54 <sdague> #info please review - https://review.openstack.org/#/c/35724/ 17:44:12 <mlavalle> sdague; I will ping him to see what progress he has achieved 17:44:22 <mlavalle> on the isues 17:44:26 <mlavalle> he is working on 17:44:27 <sdague> mlavalle: yes, jordan seemed to get blocked recently though, as he wend back down the path of special casing tempest, which I really don't agree with 17:44:50 <sdague> nova v2 api can't return different values with neutron in the backend if we want to provide a seemless migration path for people 17:44:51 <mlavalle> sdague: I will review the email thread as see how can I help 17:44:58 <sdague> mlavalle: cool, thanks 17:45:48 <afazekas> IMHO we should have an etherpad or wikipage or blueprint or something, where we track the full neutron gate blocker issues 17:46:15 <mlavalle> afazekas: We already have one ether pad. I will resend it to the ML 17:46:34 <afazekas> mlavalle: cool, thank you 17:46:55 <sdague> mlavalle: great, do we have a blueprint for this? 17:47:13 <mlavalle> sdague: no, I've been working off of the etherpad 17:47:23 <sdague> so can you make a blueprint as well 17:47:27 <sdague> link it to the etherpad 17:47:32 <mlavalle> sdague: sure 17:47:39 <sdague> but we can track it as a high priority item to revisit each week 17:47:39 <afazekas> cool :) 17:47:48 <sdague> great 17:48:08 <mlavalle> sdague; please add it as a todo to this meetings log 17:48:09 <sdague> so one other thing... we're actually about to turn off voting on neutron entirely 17:48:24 <sdague> #todo mlavalle create a blueprint for full neutron gate tests 17:49:09 <sdague> because there is a neutron flakey bug with at least 124 rechecks on it 17:49:22 <sdague> #link https://bugs.launchpad.net/neutron/+bug/1194026 17:49:25 <uvirtbot> Launchpad bug 1194026 in neutron "check_public_network_connectivity fails with timeout" [Critical,Confirmed] 17:49:34 <sdague> nati_ueno: you were poking at this last night, any updates on it? 17:49:38 <afazekas> sdague: that neuron recheck bug is very annoying, may be enough to skip just that one 17:49:58 <nati_ueno> sorry, no. I'm writing stress code for floating ip 17:50:00 <sdague> afazekas: so the concern was that the neutron team couldn't reproduce locally 17:50:13 <sdague> which means if we skip it in the gate, it's impossible to debug 17:50:23 <sdague> so instead the approach was to make neutron non voting 17:50:42 <sdague> nati_ueno: ok, no problem, someone said you were working on it earlier 17:50:47 <afazekas> sdague: my problem I frequently trigger an another issue when I try to reproduce a race issue .. :( 17:50:51 <nati_ueno> Sorry for inconvenience. Mark and I'm OK for non-vonting now 17:51:02 <afazekas> But I will try it 17:51:24 <nati_ueno> Mark and Gary and me is at least working on this issue as a top priority 17:51:25 <afazekas> sdague: BTW: https://review.openstack.org/#/c/33932/ 17:51:53 <afazekas> I usually using tracers what needs to run from the beginning 17:51:59 <sdague> afazekas: right, I think part of the issue is neutron logging doesn't seem very sufficient in the gate, which is making it harder to debug. Might end up needing some enhancements to neutron to make that better to get to problems quick in the gate 17:52:07 <afazekas> That is the reasion why that change would be helpfull for me 17:52:44 <sdague> afazekas: ok, let me ponder that 17:53:00 <sdague> the reality is, when we hit these kinds of races in nova, we can usually get really close to the issue with just the debug logs 17:53:01 <afazekas> I have doubts this time tracing just single process is enough, maybe I will need to use sytemtap .. 17:53:20 <sdague> so I'd hope we could do the same with neutron 17:53:59 <sdague> ok, I think we're good there for now 17:54:07 <sdague> #topic Open Discussion 17:54:17 <sdague> any other items? 17:55:27 <sdague> going once 17:55:35 <sdague> going twice 17:55:47 <sdague> ok, thanks for joining everyone, we'll see you on #openstack-qa 17:55:53 <sdague> #endmeeting