21:04:05 #startmeeting Networking 21:04:05 here 21:04:05 Meeting started Mon Nov 18 21:04:05 2013 UTC and is due to finish in 60 minutes. The chair is markmcclain. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:04:06 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:04:08 The meeting name has been set to 'networking' 21:04:13 #link https://wiki.openstack.org/wiki/Network/Meetings 21:04:22 #topic Announcements 21:05:11 Special agenda this week to go over priorities for Icehouse 21:05:22 #info Icehouse release schedule is available 21:05:27 #link https://wiki.openstack.org/wiki/Icehouse_Release_Schedule 21:05:58 Icehouse-1 is rapidly approaching: December 5th 21:06:45 whoops sorry i'm late. 21:06:57 Also one modification to the agenda: going to insert Plugin/Driver testing after Bug topic 21:07:05 I'm just happy you made it back to the US in one piece arosen. :) 21:07:05 arosen: no worries 21:07:14 mestery: haha no kidding ;) 21:07:27 sounds like there is a good story somewhere 21:07:32 #topic Bugs 21:07:40 #link https://bugs.launchpad.net/neutron/+bugs?search=Search&field.importance=Critical&field.status=New&field.status=Confirmed&field.status=Triaged&field.status=In+Progress 21:08:10 we're still carrying 3 critical bugs 21:08:38 I'm currently working on: https://bugs.launchpad.net/neutron/+bug/1230407 21:08:41 Launchpad bug 1230407 in neutron "VMs can't progress through state changes because Neutron is deadlocking on it's database queries, and thus leaving networks in inconsistent states" [Critical,Confirmed] 21:09:23 I'm inclined to close: https://bugs.launchpad.net/neutron/+bug/1236439 21:09:26 Launchpad bug 1236439 in ubuntu-release-notes "switch to use hostnames like nova breaks upgrades of l3-agent" [Undecided,Fix released] 21:09:35 I know mestery looked into this one 21:10:00 markmcclain: Yes, was reading that one, looks like it was release noted in Ubuntu already. 21:10:09 ok 21:10:23 last one: https://bugs.launchpad.net/neutron/+bug/1243726 21:10:26 Launchpad bug 1243726 in neutron "tempest failure: No more IP addresses available on network" [Critical,Confirmed] 21:10:31 markmcclain: Is that related to that other one you assigned me to look at before the Summit? 21:10:36 yes 21:10:38 * mestery can't recall the number, searching now. 21:10:39 OK 21:10:54 armax: you have 1243726 and then released it 21:11:09 bug 1243726? 21:11:10 Launchpad bug 1243726 in neutron "tempest failure: No more IP addresses available on network" [Critical,Confirmed] https://launchpad.net/bugs/1243726 21:11:18 was that intentional since the code merged? 21:11:59 I am still looking into the root cause 21:12:20 ok.. should we put you as the assignee on the bug? 21:12:25 the partial fix to nova was to address an infinite loop 21:12:42 ok 21:13:13 ah..sorry I confused bug 1243726 for bug 1235435 21:13:15 Launchpad bug 1243726 in neutron "tempest failure: No more IP addresses available on network" [Critical,Confirmed] https://launchpad.net/bugs/1243726 21:13:16 Launchpad bug 1235435 in nova "'SubnetInUse: Unable to complete operation on subnet UUID. One or more ports have an IP allocation from this subnet.'" [Medium,Fix committed] https://launchpad.net/bugs/1235435 21:14:15 I will take bug 1243726 21:14:16 Launchpad bug 1243726 in neutron "tempest failure: No more IP addresses available on network" [Critical,Confirmed] https://launchpad.net/bugs/1243726 21:14:23 armax: thanks 21:14:55 Anyone have any other critical/high bugs that are rated too low? 21:15:06 jog0 has shared a list of gate offenders 21:15:17 two at least are neutron 21:15:23 that is on the agenda under tempest 21:15:30 we can talk about them now if you want 21:15:32 ok, forget about it 21:15:36 let's talk at the right time 21:15:59 markmcclain: let's move to the next topic then 21:16:30 yeah.. we'll cover it then 21:16:58 #topic Plugin/Driver Testing 21:17:09 #link http://lists.openstack.org/pipermail/openstack-dev/2013-November/019219.html 21:17:40 #info 3rd Party Plugin/Driver Requirements are now in effect 21:18:51 Any questions about it? 21:18:59 what about lb? 21:19:16 technically not 3rd party, but we don't have integration testing in the gate at present... 21:19:27 (or rather, ml2+lb) 21:19:39 markmcclain: This applies for all plugins, not just core plugins right? 21:19:55 mestery: for services as well 21:20:03 marun: yes we will need to add load balancing testing 21:20:07 mestery: yes 21:20:10 technically we'd first have some tests for services, and then we'll enforce them for drivers 21:20:22 markmcclain: i think marun is talking abut linux bridge 21:20:32 enikanorov: I am 21:20:47 I'm suggesting we should target full integration testing of all plugins, 3rd party and oss. 21:20:47 too many LBs :) 21:21:00 marun: agreed 21:21:00 sorry, linuxbridge then 21:21:05 but sure we'll add tests from loadbalancing (and actually started working on them already) 21:21:20 *for 21:21:20 we should be able to add a linuxbridge variant to the gate list 21:21:42 we're currently working on getting stability for I1 21:21:45 Why add LB tests if it's going to be deprecated soon? 21:21:53 and then in I2 we can add another gate check 21:21:55 I think adding more tests which exercise the LB agent when used with ML2 would be good. 21:22:03 mestery: I meant ml2+lb, sorry 21:22:09 seems ml2+linuxbridge could run internally as part of the gate 21:22:10 marun: OK, cool, no worries. 21:22:11 ml2+linuxbridge driver, sorry 21:22:25 rkukura: we need separate jobs, is all. that's my point. 21:22:33 rkukura: right it can be run as one of the gate checks 21:22:36 anyway, moving on 21:22:42 didn't mean to derail 21:22:52 marun: no worries 21:22:53 Is the assumption that new core features will be having tests as well (e.g. VPNaaS)? 21:22:54 marun: Not derail, that's a good topic IMHO. 21:23:11 pcm_: yes 21:23:13 for icehouse-2? 21:23:33 pcm_: the scenarios should probably land by icehouse-1, if possible 21:23:40 but I agree it seems not that easy 21:23:59 and possibly not super-high priority 21:24:20 One more thing... 21:24:37 I'd say the screnarios for VPNaaS should follow us getting the core tests stable 21:24:56 s/VPNaaS/L3 Services/ 21:25:04 I'll want to do vendor service, so wondering if there will be core there as reference. 21:25:33 pcm_: no mention in the email, but vendors should be participating in creation of representative integration tests. 21:25:44 ++ 21:25:45 pcm_: It shouldn't all be on the opensource side of things. 21:25:52 pcm_: i guess the simple way is to start with API tests for the service, if there's none 21:25:55 the tests are community work item 21:26:44 that we'll all need to contribute 21:26:54 marun: what was the other item you had? 21:27:08 markmcclain: sorry, the bit about vendor participation in test creation was it 21:27:18 markmcclain: gotcha. 21:27:35 marun: thanks for making sure that was raised 21:27:40 :) 21:27:57 I heard no comment/concern on the integration testing policy draft. Can we consider it a done deal? 21:28:08 * marun keeps beating the dead horse 21:28:15 yes 21:28:17 * mestery assumed it was a done deal. :) 21:28:46 ok so it's up to the PTL to cast it in the meeting minutes. 21:29:14 #info 3rd Party Plugin/Driver Test Requirements in effect 21:29:47 * mestery notes the date and time. :) 21:30:05 what is 3rd party plugin/Driver? ml2+linux bridge is third party plugin/driver? 21:30:40 gongysh: any plugin or driver that is not the reference implementation for Neutron 21:30:52 includes vendor plugins/drivers 21:31:19 ML2+Linuxbridge is something we maintain that we'll need to add a gate job for 21:31:48 #topic Icehouse Project Plan 21:32:46 I'm sure the summit was a whirlwind for everyone, so I wanted to map out the high level plan for this cycle 21:33:26 Several sessions and conference talks really upon a few critical areas we need to address this cycle: 21:33:36 Stability, Testing, and Parity 21:34:12 These are important within our community and even more so outside of our community 21:34:15 markmcclain: nova is re-evaulating nova-network at icehouse-2 or Jan 23rd 21:34:32 jog0: good to know 21:35:18 so ideally these would be goals for milestone 2 as well 21:35:54 yeah we certainly have a lot of work to get done 21:36:17 I've mapped out 10 or so high level area for us to work on this cycle: 21:36:17 https://etherpad.openstack.org/p/NeutronIcehouseProjectPlan 21:37:05 The first 5 all work address stability, testing, and parity 21:38:33 markmcclain: This is a very good high level plan, nice work. 21:39:07 mestery: thanks 21:39:47 I'll be working with of the subteam leads to fill in the details for there subteam to make sure we're working towards the main objectives 21:39:54 Clarifying: (7) and (8) delivering in Icehouse, (9) and (10) post-Icehouse? 21:40:10 Otherwise this is ambiguous as an "Icehouse plan" 21:40:35 geoffarnold: so on 7 I expect there will be Icehouse deliverables 21:41:09 they team needs to distill out into clearer blueprints so that we can better track progress 21:41:25 19 minutes of meeting time remaining 21:41:41 8) I expect the teams will be able to improve testing in their area and deliver the new features listed 21:41:45 no service-chain-insertion in Icehouse? 21:42:14 I'd prefer it if the plan explicitly called out those areas where there are Icehouse deliverables (by I-2, right?) and those that are explicitly out of scope for Icehouse integration 21:42:16 9) are features that may or may not make the cut because they involve a bit of speculative work or require legal clearances to release code 21:42:26 fyi, i've posted a link to that etherpad that outlines a preliminary plan for the parity effort from beagles (I don't see him on here) 21:43:07 thanks marun. I think we can shift the discussion of the plan to etherpad/mailing list given that we all agree at least on priorities for items 1-7? 21:43:08 I'm unhappy about a "may or may not make the cut" in a release cycle focussed on stability 21:43:15 the 10 are subteams that need to keep working with an eye towards J because there's consensus on a vague direction but not on how to do implement 21:43:56 geoffarnold: all of the items in that bucket are new features 21:44:28 salv-orlando: +1 21:44:45 stability needs to focus on what we have today as the new features are not drop in replacements 21:44:58 I'll be sending a message to mailing list and we can discuss further 21:45:25 markmcclain: this sounds like a really ambitious list, I am concerned the last 5 will reduce the amount of work being done on the first 5 21:45:26 Only other concern I have is new subteam work being higher priority than existing subteam work, but the order may not matter there. 21:46:05 Can we also call out all API changes as early as possible? 21:46:09 jog0: that's why I listed the others first as dev and review resources should be focused on 1-4 21:46:11 -5 21:46:37 Can I use the 'shiny' adjective? 21:46:41 markmcclain: what about just dropping some of the bottom ones until after I-2? 21:47:22 I would say 1-6 are must have; 7 and 8 are expected new features due to community request; 9 and 10 are the shiny new? 21:47:49 salv-orlando: +1 to your organization thought. 21:47:58 salv-orlando: makes sense to me, although 8 sounds big 21:48:10 and could distact from 106 21:48:13 1-6* 21:48:29 how about saying the subteams should focus on bugfix/parity stuff only until 1-5 are deemed ready? 21:48:36 jog0: most the blueprints for 8+ will be targets for I3 or J depending the speed of the everything elese 21:48:41 jog0: each sub-item of 8 has a sub team of at least 3 devs independently from the devs busy on core stuff (and btw ml2 is core stuff) 21:48:53 markmcclain: cool, thanks for the clarification 21:48:58 marun: ++ 21:49:02 990495 21:49:25 ok.. we've got 10mins and anteaya has a lots of stuff to cover 21:49:28 I am not sure what 4) "nova-neutron integration" covers. is it related to nova-network deprecation? 21:49:48 amotoki: nova/network/neutronv2/api.py 21:49:49 amotoki: nova-parity will cover the items to deprecate nova-net 21:50:04 or the module from hell 21:50:09 nova-neutron integration is the pain and performance points in the integration between the systems 21:50:09 got it. 21:50:34 I'll send out the email and we can resume on ML 21:50:40 #topic Neutron Tempest 21:50:53 okay 21:51:10 starting with the bugs that are causing rechecks 21:51:26 #link https://bugs.launchpad.net/swift/+bug/1224001 21:51:31 Launchpad bug 1224001 in neutron "test_network_basic_ops fails waiting for network to become available" [Critical,Fix released] 21:51:44 salv-orlando: how is this one going? 21:51:48 well, let's start by reopening it. 21:51:58 it should have never been marked as fix released. 21:52:05 okay, let's do that 21:52:31 The manifestation is different from the one that originally caused the high number of failures. From the logs I clearly see the floating IP being configured. 21:52:33 I've reopened 21:52:38 thank you 21:52:52 anything more on this one now? 21:52:57 or shall i move on? 21:53:00 So I guess the failure in connectivity is due to the vm not booting, but I did not find a trace in the logs for that 21:53:16 salv-orlando: what more do you need to address this bug? 21:53:24 anteaya: time? 21:53:27 okay 21:53:34 let's talk next week, next 21:53:36 https://bugs.launchpad.net/neutron/+bug/1251448 21:53:37 Launchpad bug 1251448 in neutron "BadRequest: Multiple possible networks found, use a Network ID to be more specific. " [High,New] 21:53:51 again salv-orlando 21:54:09 thoughts on this one? 21:54:17 investigating this currently. The problem is that some tempest tests fail to delete a network under some circumstances, leading with multiple networks with the same name. 21:54:33 anything you need to help you with this one? 21:54:37 I can fix tempest for this, but the real issue is understanding why in some cases the network does not get deleted. 21:54:43 hmmm, that seems like a dumb bug. should be easy to always use ids in tempest 21:54:44 also can you delegate this one to anyone? 21:55:01 marun: correct. This would be a tempest fix - an easy one. 21:55:08 marun: are you able to help salv-orlando with this one? 21:55:16 since he has the other one as well? 21:55:22 salv-orlando: When I run tempest tests locally I consistently have leftover networks, ports, subnets, and routers. Not sure if it's related or not. 21:55:29 not sure masking the underlying issue is the best way forward 21:55:33 anteaya: I can take it 21:55:34 But there's an underlying neutron issue - the networks from time to time do not get deleted (remember we still do not have parallel testing so if there are two nets with the same name it's weird) 21:55:40 marun: great thanks 21:55:43 marun - all yours then 21:56:02 salv-orlando: unless you want to keep it? 21:56:10 5 minutes left, can I cover the other bugs? 21:56:22 salv-orlando: i wonder if we temporarily add db uniq constraint for the network name 21:56:27 just for catching purposes 21:56:45 that might help 21:56:48 enikanorov: names are not unique 21:56:51 enikanorov: idk frankly 21:56:51 right 21:56:52 can change 21:57:04 next bug? 21:57:05 *can't change that without pain for people 21:57:13 i say next bug 21:57:16 https://bugs.launchpad.net/neutron/+bug/1235435 21:57:17 enikanorov: -1, i'd say we fix the symptom and worry about the root cause separately 21:57:19 Launchpad bug 1235435 in nova "'SubnetInUse: Unable to complete operation on subnet UUID. One or more ports have an IP allocation from this subnet.'" [Medium,Fix committed] 21:57:44 incomplete and unassigned for neutron 21:57:49 marun: that was suggestion how to catch it, not how to fix it 21:57:55 is this one still showing up? 21:57:57 tests are failing 21:57:58 enikanorov: fair enough 21:57:59 yes 21:58:12 cool, can you point me to them, plz? 21:58:22 see the log stash link at teh bottom of the bug report 21:58:27 k 21:58:47 last 2 21:58:49 https://bugs.launchpad.net/nova/+bug/1249065 21:58:50 Launchpad bug 1249065 in nova "Tempest failure: tempest/scenario/test_snapshot_pattern.py" [Undecided,Confirmed] 21:58:59 https://bugs.launchpad.net/neutron/+bug/1251784 21:59:02 Launchpad bug 1251784 in nova "nova+neutron scheduling error: Connection to neutron failed: Maximum attempts reached" [Undecided,New] 21:59:21 looks like salv-orlando is triaging this one 21:59:29 both unassigned 21:59:35 I'm out of time, thanks 21:59:39 more info on the agenda 21:59:49 I am triaging 1249065 but no progress so far 22:00:07 Sorry we ran short on time 22:00:11 salv-orlando: can you delegate? 22:00:21 anteaya posted lots of good info so make sure to read 22:00:27 I want to highlight a few things: 22:00:40 anteaya: I will reassign in the next few hours. 22:00:47 salv-orlando: thanks 22:00:49 Code sprint in Montreal in January.. please let her know if you're interested 22:00:53 isn't 1251784 a duplicate? 22:01:35 marun: you're referring to bug 1211915? 22:01:37 Launchpad bug 1211915 in neutron/havana "Connection to neutron failed: Maximum attempts reached" [High,Fix committed] https://launchpad.net/bugs/1211915 22:01:42 salv-orlando: ye 22:01:43 s 22:01:54 jog0 said it's a different manifestation; perhaps different root cause. 22:01:59 sorry, I missed the meeting, will catch up with the minutes 22:02:17 salv-orlando: ah, ok 22:03:37 Ok talk to everyone on IRC and ML and have a great week 22:03:40 #endmeeting