22:00:13 #startmeeting qa 22:00:14 Meeting started Thu Jun 26 22:00:13 2014 UTC and is due to finish in 60 minutes. The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot. 22:00:15 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 22:00:17 The meeting name has been set to 'qa' 22:00:23 hi who's here today? 22:00:35 hi 22:00:38 #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Proposed_Agenda_for_June_26_2014_.282200_UTC.29 22:00:40 hi 22:00:44 ^^^ Today's agenda 22:00:59 hi 22:01:01 hi 22:02:03 sdague: you around? 22:02:04 ok well let's get started 22:02:05 Any one here? 22:02:17 #topic Juno specs proposal cutoff date? (mtreinish) 22:02:20 hi 22:02:50 so this is something I've been thinking about because the reviews on the qa-specs repo have been moving slowly 22:03:19 as a way to try and increase velocity there we set a cutoff date for when we stop accepting specs targetted for juno 22:03:42 when is that? 22:04:03 asselin: I was thinking the end of July, but I wanted some feedback before we started enforcing it 22:04:08 mtreinish: I don't think we should stop accepting specs until there is not time to implement 22:04:08 does it sound like a good idea? 22:04:14 mtreinish: oh, yeh 22:04:29 mtreinish: what is the purpose? 22:04:50 dkranz: I'm just trying to think of ways to motivate reviews on the specs repo 22:05:09 it goes both ways because a bunch of reviews have sat there with -1's and no response for a while 22:05:19 mtreinish: nova just did a specs review day, that was handy 22:05:20 mtreinish: I think that is an issue for the core team. Something is making us not want to review them even as we do other stuff 22:05:39 sdague: I think that is a great idea 22:06:03 dkranz: sure, honestly, I'll say my overall review level has dropped in the last few weeks as I've been fire fighting a lot 22:06:17 sdague: yeah I think that's a good idea, although we don't have nearly the same amount of specs proposals for qa-specs 22:06:56 mtreinish: sure, but if we carve out a day, we can go through them all. We could even post a hangout or something and talk through them as well in a social way 22:06:59 mtreinish: but we have enough to notice it is an issue 22:07:08 sdague: +++ 22:07:47 a little talk can go a lot further than communicating in gerrit with +1/-1 22:08:08 ok, I think we can do a spec review day, hopefully it will start moving things there 22:08:21 I'll send a ML thread out about doing 22:09:08 #action mtreinish to schedule a specs review day 22:09:37 ok, I guess we'll give this a try and we can revisit having a proposal deadline if we still have a problem 22:09:55 great 22:09:57 mtreinish: great 22:10:04 let's move on to the next topic then 22:10:14 #topic Specs Review 22:10:23 :) 22:10:29 ok does anyone have any specs that they would like to bring up for discussion? 22:10:51 dkranz: heh, I'm glad someone noticed that :) 22:10:57 #link https://review.openstack.org/#/c/91725/ 22:11:42 dpaterson: I think Matt was just asking for clarification 22:11:46 dpaterson: ok I gave it a -1 because of an inconsitency on how the preserved state will be defined 22:12:25 you mentioned both a JSON file and using the config file. You should only need one, and if it's JSON you should clarify what the format is 22:12:51 let me take a look, state is json only 22:13:18 dpaterson: you can follow up with me in -qa after the meeting 22:13:33 are there any other specs to discuss? 22:14:30 ok then let's move on 22:14:30 mtreinish: we are making no so much progress on the generate config spec 22:15:02 dkranz: yeah I noticed that. Do you want to just start writing a separate one 22:15:32 mtreinish: I may do that. I've been trying to wait because boris-42 told me they were actually doing stuff. 22:16:12 yeah, I mean we're already into J-2 and I was going to mark that bp as essential when the spec was approved 22:16:20 I just want to make sure we have enough time for implementation 22:16:28 mtreinish: Right 22:16:48 But Juno does not really have as much meaning for tempest 22:17:05 BUt we should get going. I have a feeling they may be coding away. 22:17:11 we still frame our dev cycle on the same cadence though, even if the milestone doesn't mean anything 22:17:44 although I will tag a release then too 22:17:49 mtreinish: I will ping him again 22:17:53 dkranz: ok 22:18:07 mtreinish: I don't really want to work on a spec only to have some one else submit code right after it is done. 22:18:31 dkranz: but then we'll -2 the code if doesn't align with the spec we worked on 22:18:42 the whole point of this was to decide a technical direction before code is submitted 22:18:45 so we don't waste time 22:18:56 mtreinish: I know that, but not every one buys into that., 22:19:11 mtreinish: anyway, I will talk to him and decide 22:19:34 ok, are there any other specs? Otherwise let's move on to the next topic 22:20:15 #topic Blueprints 22:20:28 #link https://blueprints.launchpad.net/tempest/+spec/nova-api-test-inheritance 22:20:36 so I wanted to discuss what we should do about this BP 22:20:44 since the V3 api isn't going to exist in the same form 22:21:09 I don't think reorganizing the nova api tests like proposed in that bp makes much sense anymore 22:21:15 agreed 22:21:27 right 22:21:40 especially as we're still working out what the nova api evolution strategy is 22:22:01 masayukig: is Ken'ichi around? 22:22:11 no.. 22:22:32 sdague: yeah, I'm was going to close the BP, but I wanted to bring it up first 22:22:36 oh, he's just come! 22:23:01 masayukig: ok cool, I just wanted to make sure he wasn't opposed to closing it, since he's the owner 22:23:31 once the nova api evolution strategy is worked out we can rework the tempest test structure based on that 22:23:35 IMO we need to get v3 out of base classes 22:23:47 mtreinish: Is it not certain that v3 is gone? 22:24:34 dkranz: I think it's going to exist, just not as v3 22:24:41 dkranz: it is agreed that it's not going to be what it is 22:24:55 but what the strategy going forward is still.... under discussion 22:25:28 sdague: ok, but the current structure tries to map two monolithic apis. And that is definitely off the table for v3, right? 22:25:31 there are 2 competing definitions of micro versions at the moment (one is mine, so it's still evolving) 22:25:39 dkranz: maybe 22:25:44 sdague: Is this written up somewhere? 22:25:50 yeh, nova-specs 22:25:58 sdague: ok, I should look at that 22:26:13 I think in the meantime the status quo for tempest is ok 22:26:14 https://review.openstack.org/#/c/101648/ and https://review.openstack.org/#/c/101648/ 22:26:17 oops 22:26:29 https://review.openstack.org/#/c/96139 22:26:37 sdague: got it 22:26:43 we won't do any more of the refactor until the nova direction is decided 22:26:48 I would agree, status quo until this shakes out 22:27:07 sdague: ok. I see dan smith -1 both :) 22:27:13 mtreinish: agree 22:27:13 oomichi: any opposition? ^^^ since it's your bp 22:27:18 oomichi: ok, cool 22:27:46 ok then for the other bps are there any updates? 22:28:07 #link https://blueprints.launchpad.net/tempest/+spec/branchless-tempest 22:28:12 sdague: ^^^ anything on that 22:28:17 since it's marked as essential :) 22:28:33 nothing new 22:29:02 It's still just the finishing up the feature matrix stuff right? 22:29:09 yes 22:29:20 ok 22:29:26 honestly, what it might be worth doing is closing out that as done 22:29:39 and creating an new spec/bp with the extensions part of the feature matrix 22:29:50 then it would be documented at least 22:30:22 sdague: ok, that makes sense to me. I'll defer to you on how to organize it 22:30:29 because realistically we haven't really had angry villagers yet 22:30:54 do you want to write up a spec for the extensions and optional features part of the feature matix 22:31:17 I'll close out the current bp when that's up 22:31:19 sdague: I don't think any one will be angry until they have to backport something to icehouse due to it. 22:31:53 mtreinish: works for me 22:32:02 sdague: ok cool 22:32:14 I'll make tomorrow my writing day, as I think I don't have anything on fire atm :) 22:32:15 #action sdague to draft a spec for finishing the feature matrix 22:32:34 ok are there any other bps to discuss? 22:33:17 #topic Grenade 22:33:31 sdague: the floor is yours 22:34:07 We got the volume support for javelin2 in 22:34:25 EmilienM is going to be looking at using javelin2 tomorrow 22:34:57 ok cool 22:35:04 there is also a devstack change that merged https://review.openstack.org/#/c/101004/ 22:35:13 did the patch merge? I know "we" approved it 22:35:25 which *should* help on the service not starting on the new side 22:35:29 mtreinish: good question 22:35:47 nope 22:35:50 sdague: has the devstack change made a noticeable difference? or is too soon to tell? 22:35:56 https://review.openstack.org/#/c/100105/ 22:36:06 mtreinish: we'll need ES back on track to see 22:36:16 right now I think we're close to 24hrs behind 22:36:39 yeh, that sucks 22:36:44 we've become very dependent on ES 22:37:02 yep 22:37:10 Bug 1331274 - Logs are not getting created for services in Grenade 22:37:11 Launchpad bug 1331274 in grenade "Logs are not getting created for services in Grenade" [Undecided,New] https://launchpad.net/bugs/1331274 22:37:14 is the tracking bug for that 22:37:38 though realistically, that seems to flatline within our signal range 22:37:42 so we might be fixed there 22:38:05 that's probably about it 22:38:08 ok, cool 22:38:20 does anyone else have something to discuss about grenade? 22:38:52 #topic Neutron Testing 22:38:57 mlavalle: around? 22:40:07 ok we can come back to this topic if he shows up 22:40:34 salv-orlando email had a good update on the status of parallel full neutron jobs 22:40:46 which was what we discussed during last weeks meeting 22:40:59 let's move on 22:41:02 #topic Bugs 22:41:04 ping me if you need more info on the matter. 22:41:10 mtreinish: one question about that 22:41:26 salv-orlando: yeh, status on that would be cool actually 22:41:38 mtreinish: I'm going to be at the mid-cycle for neutron. Is there anything we want me to pay attention to from our standpoint? 22:41:40 salv-orlando: heh I figured it was too late for you 22:41:53 it’s never too late. These are my productive hours… so 22:42:14 the status is simple we have identified the failures which occur in the full job but not in the smoketests 22:42:33 funny thing we noticed was that these failures occured on smoke tests - but only in the full job 22:42:45 and then we realized we never enabled parallelism on the smoke job 22:43:16 which is a shame because back in april parallel tests had less than 5% failure rate 22:43:39 salv-orlando: oh yeah I guess we never did 22:43:41 what we are seeing are two things: a race in nova due to the nova/neutron notification mechanism 22:43:58 mtreinish: we had that patch back in january that was lost after that massive gate outage 22:44:13 and there is a patch for that - for which I need to add a UT 22:44:51 the other problem we’re seeing are notifications not sent at all from neutron; which could mean two things - either the notification system has some bug which shows up only in parallel jobs - or the l2 agent is broken again 22:45:22 that’s all. I’m looking logs and by next monday I will have root cause for all bugs and hopefully patches 22:45:26 one last thing 22:45:37 salv-orlando: I could see either likely being the case :) 22:46:36 the lock wait timeout error… that’s another error that is more likely to happen with a parallel job. We have already people working on fixing those - but you know how it is with an opinionated developer community. So even if those failures are not fixed yet I would still enable the job and make it voting 22:46:59 mostly because we’re now merging DVR patches which do a lot of changes in L2 and L3 agents - I expect a lot of regresssions as well 22:47:06 and I’d love to catch them at the gate 22:47:12 now it’s really all 22:47:18 salv-orlando: that sounds good to me 22:47:30 salv-orlando: yeh, honestly, I'm kind of ok with the idea of full job on neutron and not on others 22:47:53 that you proposed, which I know mtreinish didn't like :) 22:47:58 sdague: I dunno asymetrical gating always makes me nervous 22:48:04 sdague: that’s a debate we already had in the past asymettric vs symmetric 22:48:10 sdague: +1 Least of evils IMO 22:48:24 everytime we've tried we end up blocking things 22:48:25 mtreinish: sure, though if we are looking at potentially a big regression in neutron 22:48:37 let’s take it offline - I thing we should go symmetric again once the failure rate is such that we don’t cause a lot of failures in other projects. 22:48:44 yeh 22:49:23 salv-orlando: ok yeah we can discuss it later 22:49:25 let's move on then 22:49:35 I did start the bugs topic 22:49:48 but given time let's move on to critical reviews :) 22:49:57 #topic Critical Reviews 22:50:08 so does anyone have any reviews that they would like to bring up? 22:50:46 the only one I have that just needs another +2 is this spec readme update: 22:50:49 #link https://review.openstack.org/102314 22:51:15 and probably this one to stop dumping tracebacks in the logs on ping retries: 22:51:18 #link https://review.openstack.org/95815 22:52:04 https://review.openstack.org/#/c/101702/ - mostly I wanted to get concensus on this hacking extension locally before spending any effort on cleaning it up 22:52:25 there are currently about 250 assertTrue/assertFalse in our code base with no msg param 22:52:34 sdague: I'm +1 on doing that 22:52:49 I also think assertEqual/NotEqual might be useful to add there too 22:53:10 I seem to keep tripping over these things in debugging, True != False is no good 22:53:11 but sometimes it's fine without it, so just True/False is probably a good starting point 22:53:27 we can always add another rule if we need it 22:53:47 yeh, assertEqual is often ok, I wouldn't hard enforce on that 22:53:55 but assertTrue is completely undebuggable 22:54:03 when it's false 22:54:37 sdague: just make sure to put something in HACKING.rst when you add the rule for real 22:54:45 yeh 22:54:59 ok, does anyone else have any reviews? 22:55:22 https://review.openstack.org/#/c/98065/ 22:55:56 afazekas: I thought we skipped the LBaas test 22:56:13 and I'm not comfortable merging anymore tests for the L7 neutron stuff until we get the full job gating 22:56:22 mtreinish: it was unskipped 22:56:55 well it is a fix for an existing test 22:56:55 afazekas: ok, I'll take a close look at that then 22:56:56 mtreinish: It fixes the lbaas test issue: http://logstash.openstack.org/#eyJzZWFyY2giOiJCYWRTdGF0dXNMaW5lIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDAzODIzMzU0OTEyfQ== 22:57:15 sdague: yeah, I thought it was a fix/unskip 22:57:44 afazekas: do you have a bug link? 22:58:08 https://bugs.launchpad.net/tempest/+bug/1326607 22:58:09 Launchpad bug 1326607 in tempest "TestLoadBalancerBasic fails in Tempest" [Undecided,In progress] 22:58:33 ok, thanks 22:59:04 ok well with 1 min. left, I just wanted to remind people about the midcycle meetup 22:59:16 if you're planning to attend please put your name on the wiki page 22:59:26 #link https://wiki.openstack.org/wiki/Qa_Infra_Meetup_2014 22:59:45 and if you have ideas for a good topic to discuss face to face ping me 22:59:46 #link https://review.openstack.org/#/c/79549 23:00:01 tpatil_: ok, I'll take a look 23:00:06 ok well that's time 23:00:09 thanks everyone 23:00:11 #endmeeting