17:00:56 <mestery> #startmeeting networking_third_party_testing
17:00:57 <openstack> Meeting started Thu Dec 12 17:00:56 2013 UTC and is due to finish in 60 minutes.  The chair is mestery. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:58 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:01:00 <openstack> The meeting name has been set to 'networking_third_party_testing'
17:01:12 <mestery> #link https://etherpad.openstack.org/p/multi-node-neutron-tempest Etherpad masquerading as an agenda
17:01:29 <mestery> So, I do not have an official meeting setup for this, depending on what we accompliush and how often we have this, I can set that up next.
17:01:40 <mestery> For now, have a look at the agenda on the etherpad link at the top.
17:01:55 <mestery> Please add things to the agenda, as well as the general etherpad.
17:02:45 <mestery> OK, so I think we're all here because we are individually looking at how to handle the new third-party testing requirement for Neutron plugins and drivers.
17:03:01 <mestery> I'm hoping we can use this meeting to facilitate sharing information, hurdles, and workarounds in getting to that goal.
17:03:15 <mestery> So, a first question: How are people coming along? Does anyone have this setup and working yet?
17:03:53 <rossella_s> we are just starting thinking about it
17:04:02 <aveiga> mestery: we just finished getting our Jenkins setup
17:04:05 <hajay__> we have a somewhat functional setup where we have our n/w controller running on a separate ssystem and all openstack/devstack-driven projects on a differetn system
17:04:23 <Izik_Penso> No, we just started working on it
17:04:37 <mestery> OK, so it looks like people are mostly just starting now.
17:04:39 <aveiga> however, I was actually interested in using this to help test others' plugins in a different environment, since I doubt many people have a dual-stacked l2-provider setup
17:04:39 <irenab> we are finalizing the requirements
17:04:41 <gduan> We are planning
17:04:55 <mestery> aveiga: Interesting, and valid point.
17:05:07 <emagana> mestery: Hi!
17:05:12 <mestery> emagana: Howdy :)
17:05:31 <mestery> Are people using their own Jenkins instances for this with the plugin to read the upstream gerrit stream?
17:05:39 <mestery> That's what we're going to do on our end.
17:06:29 <anteaya> o/
17:06:31 <mestery> I think that approach at least allows for an easier integration with the upstream gerrit for reading and posting +1/-1 back from what I understand.
17:06:34 <mestery> anteaya: Hi!
17:07:00 <mestery> OK, so how about this: Everyone can carve out a section on the etherpad (maybe at the bottom) to list what they are doing for testing.
17:07:15 <Sukhdev> Hi
17:07:18 <mestery> With the idea that we can share that info and people can come up with a common model for this, as much as possible
17:07:21 <mestery> Sukhdev: Hi.
17:07:36 <mestery> For those who joined late: https://etherpad.openstack.org/p/multi-node-neutron-tempest <--- Etherpad with agenda and information.
17:07:46 <emagana> mestery: Good Idea!
17:08:02 <anteaya> Sukhdev: did you get that issue sorted out with you testing nova patches?
17:08:09 <mestery> Also, the etherpad has a section for multi-node testing, because for the most part, I expect everyone here will be doing multi-node testing, so it made sense to combine things a bit.
17:08:24 <SumitNaiksatam> hi
17:08:26 <hajay__> regarding tempest runs, are there plans to run selective tests against a plugin. for eg. test-extensions today expects all plugins to support all extensions
17:08:37 <Sukhdev> <anteaya>: yes, thanks
17:09:05 <anteaya> also regarding etherpad use, please click the top right hand coloured button and input your name beside your colour
17:09:07 <anteaya> thanks
17:09:10 <anteaya> Sukhdev: thank you
17:09:20 <mestery> hajay__: Not that I am aware of. Can you add that under the issues seciont I just added to the etherpad?
17:09:33 <anteaya> Sukhdev: if you have any suggestions on how others might avoid the same issue in future, that would be nice to share
17:09:35 <mestery> anteaya: Thank you for reminding folks to identify themselves on the etherpad.
17:09:40 <anteaya> sure
17:09:56 <hajay__> mestery: sure. thanks. out of curiosity do we have any plugin that passes all tempest tests?
17:09:57 <rossella_s> mestery: will multi-node be required or optional?
17:10:09 <Sukhdev> I have some specific questions - should I ask here or put it on ehterpad?
17:10:13 <mestery> rossella_s: Multi-node is up to the vendor I think.
17:10:24 <mestery> Since we're talking networking, I just assume multi-node is more interesting. :)
17:10:37 <mestery> Sukhdev: Ask away, and we can add issues or info into the pad.
17:10:40 <rossella_s> mestery: I agree but it's harder :)
17:10:58 <mestery> rossella_s: Agree :)
17:11:08 <hajay__> mestery: multi node == multiple network controllers right?
17:11:10 <Sukhdev> I have basic setup working with Jenkins and Gerrit trigger - have specific questions - here we go:
17:11:23 <emagana> rossella_s: harder even using devstack?
17:11:24 <mestery> rossella_s: Although, for the most part, I am thinking we can just spin up a multi-node devstack and run Tempest against that, Is that what you were thinking?
17:11:35 <mestery> hajay__: Multi-node is multiple compute instances.
17:11:38 <Sukhdev> 1) What kind of traffic are we expecting - this will determine how many VMs I need to allocate
17:11:52 <anteaya> rossella_s: also be aware that as of yet -infra has no structure for multi-node testing, though we are aware there is a need
17:12:08 <mestery> Sukhdev: I think the Tempest tests just use pings for verification that things have spun up correctly.
17:12:21 <aveiga> mestery: are these tests required to run under devstack?
17:12:22 <SumitNaiksatam> Sukhdev: in general, isn't this what each vendor will determine?
17:12:23 <mestery> anteaya rossella_s: Yes, thus my combining the two here.
17:12:32 <anteaya> right
17:12:33 <mestery> aveiga: Not required, no, but likely easier for most.
17:12:34 <emagana> anteaya: so, you mean that in current tempest tests, is all in one node?
17:12:39 <Sukhdev> 2) I was trying to use the devstack script used for present tempest gate. I can not find it - does any one have any clue?http://ci.openstack.org/devstack-gate.html
17:12:43 <aveiga> I actually have a few nits to pick with devstack, since it's not a friendly player with v6
17:12:48 <mestery> emgana: Yes, there is no multi-node gate testing at this point.
17:13:03 <mestery> aveiga: I think we should file bugs and fix those issues in devstack :)
17:13:06 <anteaya> emagana: yes, one node currently for tempest test with -infra check and gate
17:13:27 <rossella_s> emagana: anyway I think even with devstack, multi node it tough
17:13:32 <emagana> mestery: No wonder, why we have so many bugs! No complaining just raising a good point to improve!
17:13:34 <mestery> Sukhdev: devstack itself comes from it's own git repo, right?
17:13:35 <aveiga> mestery: agreed, but stretched too thin at the moment
17:13:39 <dkehn> Sukhdev: https://github.com/openstack-infra/devstack-gate
17:13:50 <mestery> emagana: Agreed, it's on the list of things to address soon.
17:14:15 <mestery> emagana: But multiple-nodes is slightly complicated inthe gate due tothings like IP address needs, and the functionality of hte underlying public cloud these thigns run on, etc.
17:14:17 <emagana> rossella_s: agree, just try with single node and failed most of the tests :-(
17:14:39 <rossella_s> emagana: I sympathize
17:14:45 <rossella_s> we should get there anyway
17:14:55 <mestery> IMHO, and marun and I chatted about this, but running everything on a single node means that node is incredibly CPU starved at different points.
17:15:12 <mestery> Anyways, that's a different point I think, though tangentially related to this discussion of multi-node testing.
17:16:25 <emagana> mestery: so, recommendation is to use a baremetal node instead of VM?
17:16:45 <mestery> emagana: That is up to the plugin maintainers who are doing the third-party testing :)
17:16:57 <Sukhdev> 3) I saw an email from salvatore regarding the patches that we (vendors) are suppose to test - has anybody been able to set up Jekinks to pick patches that impact e.g. neutron.db, neutron.api, etc - how do you create such filter?
17:16:58 <mestery> For example, we will not use bare metal, we will use VMs for the Cisco plugin testing.
17:17:11 <mestery> emagana: But our plan is to spinup a multi-node devstack environment and run Tempest against that.
17:17:15 <aveiga> honestly, I think multi-node should be required.  If you can't pass packets between instances, then what's the point?
17:17:31 <emagana> mestery: got it!
17:17:33 <mestery> aveiga: Agreed, and it's easier for the third-party stuff for sure, as it's under control of vendors.
17:17:45 <emagana> aveiga: +1
17:17:46 <anteaya> Sukhdev: if your next issue is filtering, make a note in the etherpad and I can try to follow up to get you some answers
17:17:49 <mestery> Sukhdev: I have not. Can you add that under the issues section?
17:17:57 <mestery> anteaya: Awesome, thanks for the help there!
17:18:01 <anteaya> sure
17:18:21 <rudrarugge> We are starting with a single node to get this going at Contrail
17:18:21 <Sukhdev> I will - thanks
17:18:54 <Sukhdev> At Arista, we are also strating with single node first
17:19:21 <rudrarugge> We have the same question regarding filtering of tests
17:19:49 <emagana> rudrarugge: I almost ask the same question.
17:20:01 <emagana> does anyone know how to filter tests?
17:20:03 <clarkb> infra uses Zuul to communicate between gerrit and jenkins. Jenkins is capable of filtering based on file and branch and so on. Not sure about the gerrit jenkins plugin
17:20:20 <clarkb> *Zuul is capable of filtering based on file and branch and so on
17:20:32 <anteaya> also be sure to look at smokestack code: https://github.com/dprince/smokestack
17:20:39 <rudrarugge> We ran 1100 tempest tests and passed 800. 300 failed but are unrelated to our plugin. We wanted to be able to not run these tests
17:20:47 <anteaya> that will be my first stop trying to address the filtering question
17:21:02 <mestery> good data points here rudrarugge!
17:21:13 <mestery> I just updated the etherpad to add a section around what people's setups look like.
17:21:15 <rudrarugge> Thanks anteaya
17:21:22 <mestery> Please add your plugin/vendor link there, and we can flesh that out as well.
17:21:26 <clarkb> rudrarugge: oh filtering in that manner. tempest tests are tagged with attributes, I am sure the qa team can give you answers on filtering those
17:21:32 <mestery> The idea would be to share this info with others again as datapoints.
17:21:35 <Sukhdev> Can we all agree to share our knowldge on filtering issue on etherpad, please?
17:21:46 <anteaya> zuul repo: http://git.openstack.org/cgit/openstack-infra/zuul/tree/
17:21:48 <mestery> +1 to that Sukhdev.
17:21:52 <rossella_s> +1
17:22:03 <emagana> +1
17:22:04 <rudrarugge> +1
17:22:33 <mestery> Sukhdev: Can you drive the filtering discussion on the mailing list? Start a thread for that please, or continue the existing one if it's there.
17:23:06 <Sukhdev> <mestery>: I will do that
17:23:31 <mestery> OK, what to discuss next?
17:23:36 <Sukhdev> Next issue -
17:23:41 <emagana> Sukhdev: And please, any data points on the filtering, copy them to the etherpad
17:23:44 <Sukhdev> baremetal vs VM
17:23:49 <mestery> emagana: +1 to that too!
17:23:58 <mestery> Sukhdev: I think that is a question left up to the implementor, right?
17:24:09 <mestery> Sukhdev: Are you looking for recommendations?
17:24:21 <Sukhdev> if we are suppose to do devstack for each patch, baremetal will not scale - do you guys agree?
17:24:42 <mestery> Yes
17:24:46 <rudrarugge> Yes
17:24:54 <irenab> yes
17:24:56 <mestery> VMs can spin up on demand and scale, which is why we're going that route.
17:25:12 <mestery> Now, if your plugin depends on some special HW, you are likley left with bare metal as your only option.
17:25:23 <mestery> But for the most part, I would expect everyone to be able to run in virtual environments.
17:25:31 <mestery> But again, it's really plugin dependent.
17:25:39 <SumitNaiksatam> folks, i think we should first focus on getting an end to end workflow sorted out
17:25:47 <amotoki> hi! i think baremetal vs vm is not directly related to testing. it is up to your plugin.
17:25:48 <SumitNaiksatam> starting from getting the triggers
17:25:54 <mestery> amotoki: Agree.
17:25:59 <Sukhdev> This is what I am thinking about doing - create a VM from master branch and everytime there is patch, clone the VM, apply patch, run tests and kill the VM - what do you guys think?
17:26:00 <jbrendel> Sumit: Agree
17:26:00 <SumitNaiksatam> to actually voting back
17:26:01 <mestery> SumitNaiksatam: Also agree.
17:26:16 <SumitNaiksatam> we can then think of how we can scale better, etc.
17:26:24 <ivar-lazzaro> SumitNaiksatam: +1
17:26:32 <SumitNaiksatam> lets first flesh the complete workflow on the etherpad
17:26:34 <mestery> SumitNaiksatam: I'm hoping we can document that on the etherpad.
17:26:42 <SumitNaiksatam> i see that some folks are further along than otheres
17:26:53 <mestery> SumitNaiksatam: I have a rough example of that already on the pad.
17:26:57 <SumitNaiksatam> for those who are, can you please document up to the point that you have reached?
17:27:15 <mestery> SumitNaiksatam: I have added a section at the bottom for vendors/open source plugins to document that there.
17:27:17 <SumitNaiksatam> mestery: it might be beneficial to get the exact stpes
17:27:23 <SumitNaiksatam> mestery: great
17:27:25 <mestery> SumitNaiksatam: AGreed.
17:27:57 <SumitNaiksatam> if we can collectively get to the same point, it will help to make much better progress
17:28:01 <SumitNaiksatam> and ask better questions
17:28:18 <SumitNaiksatam> and we will all meet the I2 deadline :-)
17:28:18 <Sukhdev> I have been playing around with this in a VM - I have it almost working except for devstack and filtering issue - I will try to post what ever I can
17:28:22 <luQAs> SumitNaiksatam: you mean which tempest test pass and/or fail?
17:28:35 <SumitNaiksatam> Sukhdev: awesome!!
17:28:35 <mestery> Yes, lets all document on the etherpad to share information.
17:28:45 <emagana> Sukhdev: It will be very useful!
17:29:07 <mestery> There are two things here: 1) The setup to get the environemt up and running. 2) Ensuring you can get a clean Tempest run with your plugin.
17:29:08 <rudrarugge> we will also update the document
17:29:10 <mestery> Both are important.
17:29:15 <SumitNaiksatam> rudrarugge: great, thanks
17:29:25 <itzikb> I have a basic question: Should every patch be verified
17:29:34 <mestery> #2 can be worked on in parallel and in fact may expose bugs in your plugin if you're not already running Tempest tests against it.
17:29:35 <rudrarugge> mestery: agreed on  the 2 things
17:29:44 <SumitNaiksatam> mestery: the latter part is in paralle
17:29:49 <emagana> itzikb: I think just your plugin patches!
17:29:49 <SumitNaiksatam> mestery: you said it
17:30:13 <mestery> itzikb: Not every patch, there is an email from salv-orlando with what he recommended to filter against, see earlier discussion in the meeting logs for this meeting.
17:30:33 <mestery> OK, so lets focus on documenting where everyone is at on the etherpad.
17:30:35 <Sukhdev> <mestry> regarding you point 2, tempest tests are failing in stable/havana, but, they pass on master branch -
17:30:35 <SumitNaiksatam> emagana: i don't think it will be just your plugin patches
17:30:47 <SumitNaiksatam> emagana: although you may choose to do that
17:30:51 <mestery> And sending emails to openstack dev tagged as [neutron] [third-party-testing] when appropriate.
17:31:09 <SumitNaiksatam> mestery: thats a good suggestion on the "filter" :-)
17:31:13 <mestery> emagana: You need to run against other patches as well, anything from Jenkins for example.
17:31:47 <itzikb> @mestery: Just Neutron ?
17:32:00 <emagana> SumitNaiksatam:  I do agree that testing avery patch will be very good for the benefit of the third party plugin but we will end up re-creating the whole Infra set-up which I dont think will be easy!
17:32:02 <Sukhdev> please read an email from Salvatore regarding what patches should we be testing - it is very clear
17:32:22 <SumitNaiksatam> emagana: there is middle ground between testing every patch and only your plugin
17:32:34 <SumitNaiksatam> emagana: but again i think that is the vendor's choice
17:32:46 <amotoki> we need to run tests for patches which changes at leat your plugin and COMMON tests.
17:32:55 <mestery> amotoki: Agreed.
17:32:56 <emagana> SumitNaiksatam: Understood!
17:33:11 <mestery> OK, so is there anything else to discuss today?
17:33:11 <anteaya> emagana: actually it is very easy to recreate the infra set-up locally
17:33:18 <anteaya> if you choose to do that it would save you a lot of time
17:33:21 <mestery> Or should we focus on filling out the etherpad before meeting next week?
17:33:25 <anteaya> clarkb: ^
17:33:39 <emagana> anteaya: looking forward to get there! Not sure how much VMs will need
17:33:45 <anteaya> you can run zuul jenkins and nodepool locally
17:33:57 <mestery> anteaya: I think that's a good idea actually.
17:33:58 <anteaya> you can set as many vms as you want
17:34:09 <clarkb> anteaya: emagana: right there are other folks doing it and it is getting easier all the time. I would actually suggest using the infra toolchain to solve many of these problems. zuul, devstack-gate, and nodepool in particular
17:34:20 <Sukhdev> <anteaya>: can you post this on etherpad, please?
17:34:41 <rossella_s> anteaya: is there a page that explains how to do that?
17:34:48 <emagana> clarkb: Yes, please, If you have already a cookbook, we will just follow it!
17:34:59 <clarkb> rossella_s: there is! http://ci.openstack.org/running-your-own.html
17:35:19 <rossella_s> clarkb: thanks!
17:35:30 <mestery> clarkb: Thanks for that!
17:35:50 <clarkb> that document covers A-Z and you may not need all of hte pieces
17:36:06 <clarkb> devstack-gate, zuul, and nodepool are individually documented if you want to use parts piecemeal
17:36:09 <mestery> Thanks for that link clarkb!
17:36:22 <mestery> I think that will help get people going quickly.
17:37:50 <marcol> clarkb: thanks for the link
17:38:02 <anteaya> Sukhdev: posted
17:38:11 <anteaya> rossella_s: yes, linked to instructions in the etherpad
17:38:27 <rossella_s> anteaya: thanks
17:38:29 <anteaya> which is clarkb's link
17:38:31 <mestery> OK, should we each circle back now, spend some time digesting this, ask questions on-list and in-channel before next week's meeting then?
17:38:31 <anteaya> :D
17:38:46 <rossella_s> yees :)
17:39:01 <mestery> OK, lets do that.
17:39:33 <emagana> the magic of a link! all is easier!
17:39:34 <mestery> I'll setup another IRC meeting for next week during an Asian friendly time spot so amotoki and gongysh can join. :)
17:39:37 <Sukhdev> <anteaya>: thanks
17:39:43 <mestery> Thanks for joining us this week everyone!
17:39:44 <anteaya> mestery: great idea
17:39:49 <anteaya> Sukhdev: :D
17:39:56 <hemanthravi> thanks
17:39:57 <amotoki> mestery: really appreciated.
17:40:02 <mestery> Lets see what progress we can make individually and as a team on this to make the testing coverage better for all of Neutron and it's plugins!
17:40:13 <mestery> amotoki: No problem, and thanks for joining us this week!
17:40:16 <mestery> #endmeeting