22:00:59 #startmeeting networking_third_party_testing 22:01:00 Meeting started Wed Jan 15 22:00:59 2014 UTC and is due to finish in 60 minutes. The chair is mestery. Information about MeetBot at http://wiki.debian.org/MeetBot. 22:01:02 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 22:01:02 Hello, yes I am 22:01:04 The meeting name has been set to 'networking_third_party_testing' 22:01:19 #link https://etherpad.openstack.org/p/multi-node-neutron-tempest Neutron 3rd Party and Muilti-Node Testing Etherpad 22:01:35 We'll use the etherpad for notes today and to update progress, so please sign in with your name there if you can. 22:02:00 o/ 22:02:36 So, first off, has anyone succesfully completed the entire 3rd party testing requirement? :) 22:03:19 not yet :-( 22:03:35 still working on it 22:03:38 luqas: I suspect most are still in various stages here, you're not alone. 22:03:51 partially, not sure if we are on the right track 22:04:06 adding what we did to the etherpad 22:04:20 for comments 22:04:22 hemanthravi: Thanks! 22:04:33 So, banix, you had some questions you posed to the list, should we start with those for discussion now? 22:04:52 Is the instructions mentioned towards the end of http://ci.openstack.org/third_party.html a good place to start? 22:04:57 mestery: sure 22:05:39 I have followed the instructions there: So have a Jenkins running with Gerrit plugin 22:06:01 banix: Yes, I believe those are the instructions folks have been and should be following as a starting point. 22:06:33 The instructions for configuring Jenkins are reasonable. The part I am not sure about is the reporting back to Gerrit 22:06:59 banix: OK, what's got you hung up there? 22:07:22 last line of the instructions on the web page above says: "This job will now automatically trigger when a new patchset is uploaded and will report the results to Gerrit automatically." 22:08:09 So, I've seen a few other testing rigs reporting back, can anyone comment on banix's question here that has that part working already? 22:08:14 So I am assuming if all the steps in the Build phase return without error a +1 is reported? 22:09:09 banix: do the jenkins steps I added to the etherpad seem right? 22:09:38 hemanthravi: let me have a look please 22:10:43 I believe you can do the step 5 items in different way; what hemanthravi has outlines is one way of doing it 22:11:27 Alternatively, you could have a multi-node setup, and all that but I believe for the first try those steps sound reasonable 22:11:55 I agree, steps 5 could differ slightly, but the general approach looks reasonable to me as well. 22:11:57 for 5.5, do we specify the list of tests to be run or is there a standard set of test 22:12:07 I am not sure about how to do step 6. Being cautious to avoid reporting/voting incorrectly 22:12:35 hemanthravi: I think you can run the standard tempest tests. 22:13:02 hemanthravi: I would think the tests could be limited to say your plugin only 22:13:46 remember some disc about filters for the tests to be run, is this a config outside jenkins? 22:14:22 lugas: are you setup the system that posts reviews for Midokura? 22:14:36 Currently there is push to consolidate all network tests in one directory, one could specify that to nosetests to run networking tests (suggestions?) 22:14:37 hemanthravi: no you can specify the directories 22:14:52 yes, but currently I fake the answer to post +1 votes always 22:14:52 You can decide which subset of Tempest tests to run to work your plugin out 22:15:06 got it, thanks 22:15:23 you can configure that in the gerrit trigger plugin 22:15:26 luqas: how do you do that? where do you specify that? 22:15:49 in the gerrit plugin and not the Jenkins job configuration? 22:16:05 So we've noticed that devstack is a bit flaky. So I was wondering is it ok to vote -1 if devstack fails? 22:16:40 shashank_: What do you mean devstack is flakey? devstack itself fails? 22:16:47 yep 22:17:10 shashank_: That shouldn't happen, I think you need to figure out why devstack is failing, maybe it's in your local.conf file or something. 22:17:42 We were actually hitting a db migration bug 22:17:52 We worked around it by enabling lbaas 22:18:15 shashank_: Cool! So maybe that was your problem, and not a devstack issue itself? 22:18:36 shashank_: yes that is not a devstack issue 22:18:42 mestery: devstack to use should be stable version, not master? 22:18:56 shashank_: I think the fix is submitted but not merged or may be got merged recently 22:19:06 shivh: I would suggest the latest, e.g. track upstream. But maybe I'm missing a nuance somewhere. :) 22:19:50 My concern was what if such bugs creep into devstack and cause stack to fail 22:19:52 banix: in manage jenkins -> gerrit trigger 22:20:04 mestery: top of the master (head) may be unstable, somewhere there should be a good known version (stable) 22:20:06 you set the reporting values 22:20:33 shivh: True, but you also run the risk of an older version of devstack not being able to support deploying the latest code, right? 22:20:44 So I was a bit concerned about stacking before testing each patch 22:21:07 mestery: I see your point. maybe head is fine to use. 22:21:08 luqas: I see that; so you set the value for successful, etc; but how is it decided that the test was successful is I guess what I am missing 22:21:22 shashank_: You can vote 0 and just report with a message. 22:21:37 shivh: I think there's a balance, and if you look at what -qa and -infra do, they have to strike that balance all the time. 22:21:39 cool… 22:21:47 shivh: But I think this is something we need to all be aware of. 22:22:00 @banix, it depends on the exit value of the script that you run 22:22:12 * mestery nods in agreement with shashank_. 22:22:37 I see. Thanks shashank and luqas 22:23:22 So I think just to be safe, one could set the values for reporting to 0 or +1 for all cases 22:23:42 Just don't want to incorrectly vote -1 on patches and block their review 22:24:28 banix: Yes, we've seen that happen already. But I think -infra is limiting the voting ability of the testing setups until they can prove themselves, which will alleviate that problem as well. 22:25:02 mestery: That's good to know! 22:25:21 banix: :) 22:25:52 regarding the tempest test, network related is enough? 22:26:54 With respect to testing a plugin, I would thing any change to neutron directory could lead to failure and therefore should trigger a test but I am thinking that to begin with I will limit the trigger to the plugin I am concerned with. 22:27:18 Yeah… Even I am curious as to which tests need to be run 22:27:38 So, those are differences: Waht tests to run, and what changes trigger the running of the tests. 22:27:40 so voting with 0 is allowed? 22:27:52 The latter was suggested as changes to your plugin/MechanismDriver along with all Jenkins changes. 22:27:58 The former woudl be the Tempest tests, or a subset of those. 22:28:03 Makes sense? 22:28:04 That would be another way of limiting the impact of my voting…. 22:28:37 mestery: where/what are Jenkins changes? 22:28:47 I don't think the testing rigs voting "0" would have much value, unless it was only temporary. 22:29:01 banix: I mean, changes submitted by the Jenkins user (e.g. translation changes as an example). 22:29:23 mestery: thanks; 22:29:49 banix: Cool. 22:30:14 so that means we can trigger by who submitted the patch set? 22:30:34 banix: I believe so, yes. 22:33:07 OK, so is there anything else people want to discuss here? 22:33:58 is there a criteria to decide, when the test setup can start voting? 22:34:45 hemanthravi: That's a very good question actually. I think the idea is that the test rig has to be running ok for a while, a contact from that company/project has to be particiapting upstream in meetings. 22:34:58 Thsoe are the two things which I remember from an email to the list a while back. 22:36:56 So, multi-node testing would come later on? Anybody working on that? 22:37:49 banix: That's ... complicated. 22:37:55 :) 22:38:00 As anita indicated in email, that doesn't exist yet upstream. 22:38:07 All gate testing is done on a single node if you can believe it! 22:38:22 Now, I see no reason why this shouldn't work, after all, Tempest runs against an OpenStack cloud. 22:38:38 There may be issues with deploying this for testing, etc. But that's all things we can work through together perhaps. 22:38:40 I see 22:39:15 sounds good 22:40:04 OK, good. :) 22:40:17 banix: There could be problems, but if there are, lets all work through them together. 22:40:33 mestery: Yes of course. 22:40:59 So I think from what I have heard so far, Jenkins with Gerrit plugin seems the way to go. 22:41:06 * mestery nods. 22:41:08 I think so. 22:41:30 a last question regarding the stability of devstack 22:42:04 sometimes when stacking, at the end, it gives some errors, when setting up the default config 22:42:07 mestery: thanks for the clarifications - much needed. 22:42:27 can it be because of some race condition? 22:43:41 luqas: I would verify it's not related to your plugin, try ensuring you can get consistent devstack runs outside of the test harness. 22:43:46 OR are you saying you've already done that? 22:44:39 if there are race conditions others may have experienced it. Maybe ask the question on ML. 22:44:46 it happens sometimes with our pluggin that I have to unstack a couple of times before everything is setup correctly 22:45:10 Good point shivh. 22:45:12 I have to be sure that this has never happen of plain devstack without our plugin 22:45:23 luqas: That would be a good datapoint to have. 22:45:42 FYI: I am consisently stacking with the latest master and Neutron and I've never seen it fail with a race condition. 22:45:45 mestery: ok, i?ll check that 22:45:51 luqas: Cool. 22:47:24 OK, anything else before we wrap this meeting up? 22:48:35 OK, lets keep communicating in channel (#openstack-neutron and #openstack-qa) and on the ML. 22:48:50 Thanks everyone, and good luck as we move to the 3rd party testing deadline next week! 22:48:52 #endmeeting