21:01:38 <markmcclain> #startmeeting Neutron 21:01:38 <salv-orlando> aloha 21:01:39 <openstack> Meeting started Mon Nov 25 21:01:38 2013 UTC and is due to finish in 60 minutes. The chair is markmcclain. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:01:40 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:01:43 <openstack> The meeting name has been set to 'neutron' 21:02:29 <markmcclain> #link https://wiki.openstack.org/wiki/Network/Meetings 21:02:38 <markmcclain> #topic Announcements 21:02:49 <markmcclain> #info Icehouse-1 is December 5th 21:03:12 <emagana> hi all! 21:03:19 <markmcclain> It's next week which means code needs to merged by Tuesday Dec 3rd 21:03:26 <Swami> hi 21:03:37 <markmcclain> #link https://launchpad.net/neutron/+milestone/icehouse-1 21:03:45 <safchain> hi 21:03:49 <colinmcnamara> hi 21:03:59 * salv-orlando regrets the week off taken during I-1 21:04:20 <markmcclain> the first milestone always seems to feel rushed 21:04:35 * mestery nods in agreement. 21:04:43 <markmcclain> and even more so since many will be taking a few days off at the end of this week 21:05:24 <markmcclain> #topic Bugs 21:05:27 <markmcclain> anteaya: around? 21:05:53 <markmcclain> #link https://bugs.launchpad.net/neutron/+bugs?search=Search&field.importance=Critical&field.status=New&field.status=Confirmed&field.status=Triaged&field.status=In+Progress 21:06:31 <armax> I'd add bug #1254236 to the list 21:06:33 <uvirtbot> Launchpad bug 1254236 in neutron "TestNetworkQuotaBasic tearDownClass fails while deleting port" [Undecided,New] https://launchpad.net/bugs/1254236 21:06:37 <markmcclain> not sure if anteaya is active at the moment.. she has gone through and listed the gate blocking bugs in the agenda 21:06:44 <clarkb> I am going to mention really quickly that the way jenkins processes test logs after tests run has been updated to speed up the neutron unittest jobs. py26 is ~40 minutes now and py27 is ~27 minutes 21:06:49 <armax> I found out about it last week 21:06:50 <armax> it's getting hot pretty fast 21:06:55 <markmcclain> clarkb: awesome 21:06:57 <clarkb> shouldn't see anymore job timeouts on those 21:07:02 <sc68cal> woooo! 21:07:13 <markmcclain> armax: thanks 21:07:15 <nati_ueno> cool! 21:07:17 <armax> clarkb: thanks 21:07:29 <armax> clarkb: so do you think that review https://review.openstack.org/#/c/58022/ is no longer worth it? 21:08:30 <clarkb> armax: correct, we should keep an eye on it but I think we are well below variability in test times 21:08:45 <clarkb> *well below the timeouts considering variability in test times 21:09:06 <armax> clarkb: ok sounds good 21:09:15 <markmcclain> salv-orlando: is working on https://bugs.launchpad.net/swift/+bug/1224001 21:09:17 <uvirtbot> Launchpad bug 1224001 in neutron "test_network_basic_ops fails waiting for network to become available" [High,In progress] 21:09:33 <salv-orlando> I think I and otherwise guy merged the patches that should fix it 21:09:41 <markmcclain> cool 21:09:43 <salv-orlando> we hope it won't show up again, but I know it will 21:09:52 <salv-orlando> (more later when we discuss parallel testing) 21:10:12 <markmcclain> k 21:10:35 <markmcclain> https://bugs.launchpad.net/neutron/+bug/1251448 21:10:37 <uvirtbot> Launchpad bug 1251448 in neutron "BadRequest: Multiple possible networks found, use a Network ID to be more specific. " [High,New] 21:10:43 <markmcclain> marun: any progress? 21:11:00 <marun> mlavalle seems to have narrowed the issue to not having tenant isolation turned on 21:11:19 <marun> is there a reason why any of the gates turn it off? 21:11:41 <markmcclain> when it was activate earlier it actually caused more gate failures 21:11:52 <marun> i thought that was parallelization 21:11:55 <marun> :\ 21:12:10 <marun> damned if we do, damned it we don't 21:12:43 <markmcclain> yeah well as we work on improving the test situation we will uncover a few issue along the way 21:13:12 <beagles> what are we considering the fundamental error there? 21:13:15 <salv-orlando> marun, markmcclain, on this note of tenant isolation I might be dumb, but the 'isolated' job we run 21:13:20 <beagles> that the tearDown fails? 21:13:33 <marun> it is a cleanup issue, yeah 21:13:34 <salv-orlando> which set allow_tenant_isolation = True - aren't they using isolation? 21:13:37 <markmcclain> beagles: yes 21:14:06 <salv-orlando> or does bug 1251448 occur only on non-isolated jobs? 21:14:08 <uvirtbot> Launchpad bug 1251448 in neutron "BadRequest: Multiple possible networks found, use a Network ID to be more specific. " [High,New] https://launchpad.net/bugs/1251448 21:14:14 <markmcclain> sdague or jog0: ^ 21:15:03 <marun> salv-orlando: but if any job doesn't use isolation, then we have potential races in cleanup 21:15:13 <armax> it looks like it occurs on non-isolated jobs 21:15:40 <salv-orlando> armax: that was what I was looking for. If we can confirm that it does not happen with isolation, then the issue is nailed 21:15:44 <marun> so the question is, are the non-isolated jobs going to continue? 21:15:55 <sdague> salv-orlando: http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiIHBvc3NpYmxlIG5ldHdvcmtzIGZvdW5kLCB1c2UgYSBOZXR3b3JrIElEIHRvIGJlIG1vcmUgc3BlY2lmaWMuIChIVFRQIDQwMClcIiAgICAgQU5EIGZpbGVuYW1lOlwiY29uc29sZS5odG1sXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzODU0MTQxMjYzNjcsIm1vZGUiOiJ0ZXJtcyIsImFuYWx5emVfZmllbGQiOiJidWlsZF9uYW1lIn0= 21:15:57 <salv-orlando> marun: should be discontinued, this is my understanding 21:16:08 <sdague> yes, only "non issolated" jobs 21:16:19 <marun> so yeah, not sure we want to bother fixing. isolation is the ideal fix 21:16:34 <salv-orlando> sdague: thanks for showing me how to the query - I was hopelessly trying to do that by myself 21:16:34 <sdague> marun: so we're not fully isolated on the isolated jobs right now 21:16:56 <marun> sdague: ? 21:17:20 <sdague> last I knew the API tests aren't actually properly issolated, because the load pushed neutron over the edge 21:17:31 <sdague> mtreinish has the details 21:17:40 <marun> sdague: ah, ok 21:17:55 <marun> so it will be necessary to ensure name uniqueness 21:18:03 <markmcclain> no that we've activated minimize polling we might was to look and see if the load is still too high 21:18:11 <markmcclain> s/was/want/ 21:18:15 <mtreinish> sdague: yeah the neutron api tests don't have isolation turned on 21:18:36 <mtreinish> last time I did that the gate for neutron failed basically 100% of the time 21:18:56 <mtreinish> it needs a rebase but: https://review.openstack.org/#/c/53459/ 21:19:43 <salv-orlando> mtreinish, sdague, marun, markmclain and whoever else: I'm working on that too as a part of the parallel testing effort 21:19:52 <markmcclain> cool 21:20:00 <marun> the polling switch has made a difference in runtimes at least 21:20:12 <markmcclain> Any other critical bugs we need to discuss? 21:20:13 <marun> i'm seeing 20-25% less execution time 21:20:22 <jog0> so neutron gate is failing at 33% right now 21:20:32 <jog0> .gate-tempest-devstack-vm-neutron 21:21:42 <markmcclain> jog0: good to know.. was hoping for a lower number with all of the recent changes 21:22:17 <jog0> there were some outstanding patches in the nova side that should help 21:22:21 <jog0> let me hunt those down 21:22:41 <markmcclain> me and arosen pinged the nova room earlier with those 21:22:42 <arosen> https://review.openstack.org/#/c/57711/ , https://review.openstack.org/#/c/57069/6 21:23:00 <arosen> jog0: ^ 21:23:03 <jog0> arosen: thanks thats it 21:23:12 <jog0> arosen: I will review those 21:23:18 <markmcclain> jog0: thanks! 21:23:22 <arosen> jog0: awesome thanks 21:23:38 <markmcclain> Any other critical bugs the team needs to track? 21:23:53 <markmcclain> #topic Docs 21:24:01 <jog0> https://bugs.launchpad.net/neutron/+bug/1250168 https://bugs.launchpad.net/neutron/+bug/1251448 https://bugs.launchpad.net/neutron/+bug/1235435 21:24:04 <uvirtbot> Launchpad bug 1250168 in neutron "gate-tempest-devstack-vm-neutron-large-ops is failing" [Critical,Confirmed] 21:24:07 <markmcclain> #undo 21:24:07 <openstack> Removing item from minutes: <ircmeeting.items.Link object at 0x359b710> 21:24:14 <jog0> are all gate bugs, none are listed as critical ATM 21:24:28 <jog0> not sure where the root causes of those bugs are though 21:25:06 <armax> Partial fix for bug 1235435 is review 57069 21:25:07 <uvirtbot> Launchpad bug 1235435 in nova "'SubnetInUse: Unable to complete operation on subnet UUID. One or more ports have an IP allocation from this subnet.'" [Medium,In progress] https://launchpad.net/bugs/1235435 21:25:20 <armax> one of the two that aaron pointed you at 21:25:32 <armax> s/aaron/arosen 21:26:06 <markmcclain> bug 1235435 is because isolations isn't turned on 21:26:08 <uvirtbot> Launchpad bug 1235435 in nova "'SubnetInUse: Unable to complete operation on subnet UUID. One or more ports have an IP allocation from this subnet.'" [Medium,In progress] https://launchpad.net/bugs/1235435 21:26:15 <markmcclain> salv-orlando: wanted to talk about parallel testing 21:26:17 <jog0> armax: cool thanks 21:26:30 <markmcclain> work may help solve that issue 21:26:33 <salv-orlando> markmcclain: I'm fine either way if you want to move it oto open discussion 21:26:43 <salv-orlando> and continue with the normal agenda 21:26:49 <jog0> markmcclain: we just saw bug 435 or the query we have is bad https://review.openstack.org/#/c/58352/ 21:26:50 <uvirtbot> Launchpad bug 435 in ethereal "ethereal uses gksu instead of gksudo for root menuitem" [Low,Invalid] https://launchpad.net/bugs/435 21:26:57 <jog0> anyway thanks 21:26:57 <armax> markmclain: not in all cases 21:27:11 <armax> but we can talk about it offline if you want to learn more 21:27:31 <markmcclain> ok.. we can discuss the actually bug in teh room after the meeting, lets keep moving so that we don't run out of time 21:27:48 <markmcclain> #topic Docs 21:27:55 <emagana> hi 21:28:11 <markmcclain> Metering docs have merged.. thanks to sfchain 21:28:22 <markmcclain> emagana: anything else to add that's not on the agenda? 21:28:28 <emagana> yes! 21:28:56 <emagana> just a head ups about a Doc Bug Day 21:29:19 <emagana> #info https://wiki.openstack.org/wiki/Documentation/BugDay Dec 20th 21:29:43 <emagana> It could be great to have some neutrons participating 21:29:53 <safchain> I also added a section about the ml2 scenarios, with to start 2 scenarios with the l2 population mechanism driver 21:30:16 <salv-orlando> emagana: fine but I really don't feel comfortable around electrons 21:30:39 <markmcclain> emagana: thanks for the reminder about the doc bug day 21:30:46 <mestery> salv-orlando: heh 21:30:47 <emagana> salv-orlando: keep neutral.. I mean neutron 21:31:02 <emagana> nothing else markmcclain 21:31:07 <markmcclain> thanks for the update 21:31:10 <salv-orlando> emagana: so we don't have network bugs open against either the admin-guide or netconn-api? 21:31:11 <markmcclain> #topic Nova Parity 21:31:16 <markmcclain> #undo 21:31:17 <openstack> Removing item from minutes: <ircmeeting.items.Topic object at 0x35d8110> 21:31:33 <emagana> Yes, we do! 21:32:01 <emagana> I did not believe we want to discuss all of them, at least we have a high priority ones 21:32:14 <emagana> as we had with metring part 21:32:24 <emagana> s/metring/metering 21:32:46 <emagana> just wanted to keep it short, but we can go over more details if needed, advise! 21:33:10 <markmcclain> we still have a lot of material to cover, so let's keep moving for now 21:33:15 <markmcclain> #topic Nova Parity 21:33:51 <beagles> somebody is playing my song 21:34:00 <markmcclain> yep 21:34:36 <beagles> I was on PTO last week so I apologize for my silence... however before I left someone proposed creating a draft list of milestones related to parity 21:34:54 <markmcclain> yes will be helpful to track progress 21:35:00 <beagles> I've posted a link to the ethepad in the agenda... it was a complete off the cuff affair,but it fits the schedule (I think) 21:35:03 <dkehn> agreed 21:35:04 <markmcclain> and we can fill it in with action items 21:35:16 <markmcclain> #link https://etherpad.openstack.org/p/icehouse-neutron-nova-parity-prelim-outline 21:35:17 <beagles> I'd appreciate suggestions and refinements 21:35:37 <salv-orlando> thanks a lot beagles 21:36:01 <beagles> my pleasure: I've mentioned some things in there that warrant conversations offline 21:36:07 <salv-orlando> do you reckong we are on schedule with your plane? 21:36:10 <beagles> or out-of-the-meeting 21:36:11 <salv-orlando> plane/plan 21:36:22 <beagles> we are a little behind but not dire I think 21:36:48 <markmcclain> ok.. we can discuss offline farm out the work if necessary to get us back on track 21:37:26 <markmcclain> Anything else for this week? 21:37:46 <sc68cal> ipv6? 21:38:23 <markmcclain> sc68cal: yeah we'll have to look at gaps there too 21:38:29 <markmcclain> #topic Tempest 21:39:02 <markmcclain> Lots of updates to the etherpad 21:39:04 <markmcclain> #link https://etherpad.openstack.org/p/icehouse-summit-qa-neutron 21:39:22 <markmcclain> mlavalle or salv-orlando who wants to update first? 21:39:43 <salv-orlando> I'd let mlavalle go first - he's the topic lead, I think 21:39:59 * salv-orlando at least that's what the wiki page says :) 21:40:02 <mlavalle> markmcclain: as indicated in the ether pad, I have taken responsibility for the api testing 21:40:19 <markmcclain> ok 21:40:40 <mlavalle> my plan is to have a plan by this week, so we can let other contrabute to this effort 21:41:13 <markmcclain> great… I'll follow up with you later in the week 21:41:18 <mlavalle> I also worked over the weekend on https://bugs.launchpad.net/neutron/+bug/1251448 21:41:20 <uvirtbot> Launchpad bug 1251448 in neutron "BadRequest: Multiple possible networks found, use a Network ID to be more specific. " [High,New] 21:41:42 <mlavalle> I've updated the but in launchpad with my analysis 21:41:48 <markmcclain> great thanks 21:42:01 <mlavalle> it seem to me it's a race condition 21:42:18 <mlavalle> when deleting ports 21:42:36 <markmcclain> yeah the test changes have definitely revealed several races 21:42:53 <mlavalle> so by the time the tea tries to delete a subnet, it finds ports still active 21:43:11 <mlavalle> so a net and a subnet are left alive in the demo user 21:43:21 <mlavalle> that cascades to the other tests after that 21:43:42 <markmcclain> yeah.. that will do it 21:44:00 <mlavalle> tonight I will attempt to create a fix….. i might ask a lot of questions tomorrow 21:44:07 <markmcclain> mlavalle: sounds great 21:44:12 <salv-orlando> mlavalle: I'll be around 21:44:16 <mlavalle> that's all I have today 21:44:22 <markmcclain> mlavalle: thanks 21:44:23 <salv-orlando> in terms of "items" pertaining to testing 21:44:44 <salv-orlando> on the etherpad we have at least three we need to discuss here: parallel, full, and grenade testing 21:45:08 <salv-orlando> as I do things without any logic, let's start from the bottom 21:45:13 <markmcclain> ok 21:45:28 <salv-orlando> grenade - the issue is simple, there's no grenade testing, and we're looking for an owner 21:45:41 <salv-orlando> last week I said "we have no owner" and somebody said "hold on" 21:45:46 <salv-orlando> does this mean we have an owner now? 21:46:23 <markmcclain> I don't think we've had anyone step up 21:46:30 <salv-orlando> silence == no owner, I guess. markmcclain, can we prioritise finding a volunteer? 21:46:38 <beagles> nobody wants to dive on the grenade? 21:46:44 <dkehn> cue crickets 21:46:50 <markmcclain> #action markmcclain to find grenade lead 21:47:03 <markmcclain> I work on that this week 21:47:04 <salv-orlando> full neutron devstack test 21:47:17 * beagles nods 21:47:20 <sc68cal> Does grenade piece also include the OVS -> ML2 migrations? 21:47:21 <salv-orlando> This work item is a mix of issues with neutron and features missing from neutron. 21:47:22 <marun> markmcclain: please coordinate with jlibosva 21:47:32 <markmcclain> marun: will do 21:47:34 <sc68cal> not sure if that's out of scope for grenade 21:47:38 <salv-orlando> At the moment we have rossella_s performing an assessment 21:48:00 <markmcclain> sc68cal: the ml2 team will need to add migration scripts 21:48:02 <salv-orlando> she's not online, but as she's in my timezone I will ping her tomorrow 21:48:04 <beagles> salv-orlando, there is an overlap with the parity related activities outlined for this week 21:48:34 <markmcclain> beagles can you coordinate with rossella_s? 21:48:37 <mestery> sc68cal: Yes, we're aware of the migration requirements from OVS to ML2, but have no specific plans at the moment. 21:48:38 <salv-orlando> beagles: I am aware of that; I thought I was good to get someone started anyway on the assessment of what needs to be done to get that test passing 21:48:42 <beagles> absolutely :) 21:48:45 <mlavalle> salv-orlando: a couple of months ago I put together (with the help of others) this https://etherpad.openstack.org/p/gate-tempest-devstack-vm-quantum-full 21:48:47 <markmcclain> thanks 21:48:53 <beagles> salv-orlando, no argument there :) 21:49:06 <salv-orlando> mlavalle: thanks for that I will add it to other etherpad 21:49:13 <salv-orlando> and finally… parallel testing 21:49:38 <mlavalle> salv-oralando: rosella_s can ping me if clarification is needed 21:49:49 <markmcclain> salv-orlando: you've been digging into this one right? 21:49:51 <salv-orlando> mlavalle: I will ell there 21:50:08 <salv-orlando> Basically there are 3 issues, 1 in devstack/tempest, and 2 in neutron 21:51:05 <salv-orlando> devstack/tempest give neutron only 16 public IPs. this is fine for nova-network, but neutron uses a public IP for each tenant router. If you add the fact that some tests do not cleanup routers, you can easily run out of IPs 21:51:06 <EmilienM> salv-orlando: i could help with grenade, even if i have never contributed yet 21:51:24 <salv-orlando> armax has a patch for 2 test cases leaving a router behind 21:51:36 <salv-orlando> I have a patch proposing to start defaulting the public network to /27 21:51:53 <salv-orlando> any questions before we move to the neutron side? 21:52:10 <markmcclain> we have 8 mins, so I think we have to move on 21:52:15 <mlavalle> salv-orlando: I think the number of ip's is configurable in the tempest conf file 21:52:41 <armax> salv-orlando: the patches you're talking about are: https://review.openstack.org/#/c/57805/ and https://review.openstack.org/#/c/58054/ 21:52:43 <salv-orlando> mlavalle: this is about the public network created by devstack I think, but we can take this detail offline I think 21:52:52 <mlavalle> cool 21:53:06 <lifeless> SpamapS: hey - whats the bug number about that race with new networks? 21:53:12 * salv-orlando is funny to say offline while you'll still be online talking about this 21:53:17 <lifeless> SpamapS: thinking it would be topical to raise it here as something tempest should be testing 21:53:55 <salv-orlando> I think we're out of time for the tempest discussion - I will update on neutron issues separately, unless you feel it's important 21:54:26 <markmcclain> yeah.. I think we discuss on this on the mailing list 21:54:42 * markmcclain fingers won't work today 21:55:08 <markmcclain> #topic Subteams 21:55:31 <markmcclain> We've got several new subteams in addition to our existing ones. 21:55:43 <markmcclain> Please check the agenda for updates 21:56:05 <markmcclain> #topic Open Discussion 21:56:28 <markmcclain> Any items need to cover in 4 minutes or less? 21:57:48 <nati_ueno> Is lbbas api still experimental? 21:58:09 <markmcclain> no with Havana that api will need to follow the deprecation cycle 21:58:31 <nati_ueno> markmcclain: Thanks 21:58:45 <nati_ueno> it looks there is a patch which change beheivior 21:58:49 <nati_ueno> of LBaaS API 21:59:01 <markmcclain> link? 21:59:13 <nati_ueno> https://review.openstack.org/#/c/53821/ 21:59:47 <nati_ueno> health monitor was cascate deleted before 21:59:56 <nati_ueno> This patch changes that beheivior 22:00:06 <markmcclain> thanks for the link.. I'll follow up with teh sub-team 22:00:12 <nati_ueno> markmcclain: Thanks 22:00:19 <markmcclain> we're out of time for this week 22:00:24 <markmcclain> I want to thank salv-orlando, armax, arosen, jog0, marun, anteaya for digging into the gate problems 22:00:40 <jog0> markmcclain: thank you 22:00:50 <jog0> to everyone fixing them 22:00:58 <armax> I hope I just don't add more ;) 22:01:05 <arosen> ditto :) 22:01:10 <markmcclain> Have a great week and for everyone traveling for the holidays later this week.. safe travels 22:01:12 <markmcclain> #endmeeting