21:02:16 <markmcclain> #startmeeting Networking
21:02:17 <openstack> Meeting started Mon Jan  6 21:02:16 2014 UTC and is due to finish in 60 minutes.  The chair is markmcclain. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:02:18 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:02:21 <openstack> The meeting name has been set to 'networking'
21:02:23 <mestery> Ha!
21:02:33 * mestery can see his breath as he types in this meeting.
21:02:49 <markmcclain> #link https://wiki.openstack.org/wiki/Network/Meetings
21:02:51 <enikanorov> cold, heh?
21:02:55 <dkehn> I'm guessing anything south of the mason  dixon line
21:03:52 <markmcclain> so it's been two weeks since our last meeting and many took time off during the holidays
21:03:53 <mestery> Almost -30 air temperature this morning, at least -50 with wind chill, all temps F. As cold as I can remember it.
21:04:40 <markmcclain> mestery: that just sounds painful
21:04:47 <mlavalle> mestery: where do you live?
21:04:49 <Sukhdev> Wow!!
21:04:55 <salv-orlando> mestery: if next week in Montreal is anything like that, I think I'll just die. That is way below my operating temperature. Seriously I have a label which says "operate strictly between 0C an 40C"
21:05:11 <mestery> mlavelle: Minnesota
21:05:12 <dkehn> welcome to the polar vortex
21:05:26 <markmcclain> Icehouse-2 is Jan 24rd
21:05:30 <markmcclain> oops 23rd
21:05:57 <markmcclain> The tempest/Neutron sprint is next week
21:06:01 * mestery hopes someone brings salv-orlando a toque: http://en.wikipedia.org/wiki/Toque#Canadian_usage
21:06:40 <markmcclain> #topic Bugs
21:06:54 <salv-orlando> mestery: believe it or not I have a Toronto Maple Leafs toque, somewhere
21:07:11 * mestery loves the Maple Leafs. :)
21:07:12 <dkehn> mestery: its a balmy 28F in montreal
21:07:26 <mestery> dkehn: Psssh. That's for amateurs. :)
21:07:40 <salv-orlando> dkehn: oh yeah? I'm so packing my swimsuit and sunglasses then
21:07:54 <markmcclain> so we're still tracking the same critical bugs as before Christmas
21:07:55 <dkehn> http://www.weather.com/weather/extended/CAXX0301?par=yahoo&site=www.yahoo.com&promo=extendedforecast&cm_ven=Yahoo&cm_cat=www.yahoo.com&cm_pla=forecastpage&cm_ite=CityPage
21:08:10 <dkehn> salv-orlando: I'll meet you by the pool
21:08:22 <salv-orlando> markmcclain: I have updated the status of bug 1253896
21:08:22 <enikanorov> markmcclain: at least one should go away
21:08:31 <markmcclain> salv-orlando: ok
21:08:37 <markmcclain> enikanorov: which one?
21:08:50 <enikanorov> "timeout waiting for thing" one
21:09:19 <markmcclain> great
21:09:23 <enikanorov> the fix has been committed to nova
21:10:43 <markmcclain> enikanorov: do you have a link for the fix?
21:10:49 <markmcclain> I want to link it into https://bugs.launchpad.net/neutron/+bug/1250168
21:10:56 <enikanorov> let me find
21:11:14 <markmcclain> nati_ueno: looks like we have reviews ready for https://bugs.launchpad.net/neutron/+bug/1112912
21:11:43 <markmcclain> nati_ueno: still seems to failing jenkins
21:11:53 <markmcclain> have you been able to triage it?
21:12:02 <nati_ueno> markmcclain: I got it. I'll fix it in this afternoon
21:12:16 <markmcclain> great
21:12:38 <nati_ueno> markmcclain: Jenkins looks working for me (Dec16) but may be rebase needed
21:12:45 <enikanorov> markmcclain: https://review.openstack.org/#/c/64383/
21:12:56 <pcm_> I've getting Jenkin's failures on my code review (only docstring changes in latest version).
21:13:03 <markmcclain> nati_ueno: please rebase
21:13:06 <markmcclain> enikanorov: thanks
21:13:09 <pcm_> Can someone help me off-line to devug
21:13:09 <nati_ueno> markmcclain: gotcha
21:13:15 <pcm_> debug?
21:13:38 <markmcclain> pcm_: ask around in the channel after the meeting
21:13:45 <pcm_> or do I need to rebase (was done last week).
21:13:50 <markmcclain> if I didn't have to run I'd hang around and help
21:13:50 <pcm_> markmcclain: OK
21:14:06 <markmcclain> pcm_: I'd try that
21:14:13 <pcm_> markmcclain: OK.
21:14:40 <markmcclain> marun: this bug is still open
21:14:40 <markmcclain> https://bugs.launchpad.net/neutron/+bug/1192381
21:14:45 <markmcclain> can we consider it closed?
21:14:54 <marun> markmcclain: I think so, yes.
21:15:10 <markmcclain> ok
21:15:12 <marun> markmcclain: there is a follow-on blueprint that will ensure eventual consistency: https://blueprints.launchpad.net/neutron/+spec/eventually-consistent-dhcp-agent
21:15:25 <marun> markmcclain: but the best we can do without a refactor has already been committed
21:15:33 <markmcclain> ok
21:15:45 <markmcclain> what milestone should I target for the bp?
21:16:06 <marun> markmcclain: probably icehouse-3
21:16:10 <markmcclain> ok
21:16:24 <marun> markmcclain: hopefully progress can come sooner but i don't want to rush it
21:17:09 <markmcclain> yeah we definitely want to make sure we maintain stability
21:17:27 <markmcclain> Any other critical bugs the team needs to discuss?
21:17:42 <salv-orlando> do we skip our favourite bug?
21:17:46 <salv-orlando> bug 1253896?
21:18:15 <markmcclain> we did
21:18:24 <markmcclain> https://bugs.launchpad.net/neutron/+bug/1253896
21:18:42 <markmcclain> looks like you did some research on it late last week
21:18:44 <salv-orlando> come on, we're actually not fixing this bug just because we love talking about it.
21:19:05 <salv-orlando> seriously, failures in non-isolated jobs are down to 0% which is good for the gate
21:19:12 <markmcclain> yeah
21:19:46 <salv-orlando> however we want to make isolated and parallel jobs the default solution, so it's not so good that just the isolated jobs has a failure rate of about ^%
21:19:50 <salv-orlando> sorry 6%
21:20:02 <salv-orlando> and that's up ftom about 2.5% before christmas
21:20:07 <markmcclain> ugh
21:20:22 <marun> do we want to get isolated/parallel working as currently defined?
21:20:36 <marun> it's currently a really beastly stress test that is unrepresentative of real-world usage for the most part
21:20:50 <salv-orlando> marun: yes, that is correct, but I would like to move this discussion in the tempest part
21:20:51 <marun> (because of running on a single node that is cpu/io bound like crazy)
21:21:05 <marun> salv-orlando: fair enough
21:21:32 <markmcclain> We'll return to parallel tests in tempest section
21:21:43 <salv-orlando> for the current situation, we need to look at the logs. If we conclude that the reasons for the failure are the same that are causing failures in the parallel jobs, then we should just wait for the patches to merge; but I doubt that.
21:22:05 <salv-orlando> It would be great if we can get some fresh eyes to look at the logs, as I won't have much time during this week.
21:22:08 <markmcclain> yeah I am interested to know what caused the failure rate to triple
21:22:31 <markmcclain> Any other bugs the team needs to discuss?
21:22:35 <salv-orlando> I am too. Note that I've been taking 24 hours samples - the first on Dec 23 and the second on Jan 2
21:23:14 <salv-orlando> that's is all. I would be happier if I somebody else volunteers to look at the logs and provide his/her feedback
21:23:22 <salv-orlando> especially if that somebody is coming to montreal next week.
21:24:01 <markmcclain> I've got training tomorrow, but I'll try to spend time digging if anyone has spare cycles before then feel free update the bug
21:24:05 * salv-orlando is sure people are now rushing to logs.openstack.org to check the logs
21:24:50 <markmcclain> #topic Docs
21:25:04 <markmcclain> emagana is out today but has filled in the report
21:25:15 <markmcclain> #topic Tempest
21:25:24 <annegentle_> markmcclain: can I bring up a docs item?
21:25:28 <markmcclain> #undo
21:25:29 <openstack> Removing item from minutes: <ircmeeting.items.Topic object at 0x3072fd0>
21:25:36 <markmcclain> annegentle_: yes!
21:25:39 <annegentle_> markmcclain: we're having a discussion on the mailing list about  a new networking -only guide
21:25:45 <annegentle_> #link http://lists.openstack.org/pipermail/openstack-docs/2014-January/003582.html
21:25:54 <annegentle_> that just goes to a mid-thread discussion
21:26:04 <annegentle_> but since Edgar has been out I haven't been able to ping him about it
21:26:33 <annegentle_> so, just wanted to put it on your radar... I'm hesitant to add another guide what with all the reorg we've been doing (augggh) but wanted to see what you all think, would it be useful, better, worse?
21:27:13 <markmcclain> I missed this thread, so thanks for raising it
21:27:30 <sc68cal> +1
21:27:55 <annegentle_> I agree with Tim Bell that "if the image management gets it's own book, why not networking"
21:28:14 <annegentle_> but, it's a pile of work, with real underlying teaching needs, so we need to find a good "owner"
21:28:18 <sc68cal> subscribed to the BP - i'll pitch in from a lot of doc that I wrote up internally while I was learning Neutron
21:28:24 <mestery> I'm all in favor of a guide for networking.
21:28:37 <annegentle_> anyway, feel free to discuss amongst yourselves, comment on the blueprint, etc. Ohh thanks sc68cal
21:28:54 <annegentle_> markmcclain: thanks for letting me pop in :)
21:29:03 <annegentle_> carry on
21:29:05 <salv-orlando> my 2p at first glance is that the Neutron community has struggled a bit to handle the current workload
21:29:37 <annegentle_> salv-orlando: yep really it'd be best to have a tech writer take it
21:29:51 <annegentle_> salv-orlando: if you know of anyone, I could even ask for a contract
21:30:48 <salv-orlando> annegentle_: exactly my point is that we don't have a doc guy so far; just people taking turns in playing that role. I'll let you know if I hear of somebody interested
21:31:00 <annegentle_> k thanks
21:31:19 <markmcclain> ok.. everyone feel free to catchup on the thread and chime in on the mailing list
21:31:25 <markmcclain> annegentle_: thanks for pointing this out
21:31:38 <markmcclain> annegentle_: anything else since edgar is out this week?
21:31:40 <amotoki> note that it is a thread in docs ML
21:31:52 <markmcclain> amotoki: good reminder
21:32:19 <markmcclain> #topic tempest
21:32:19 * mestery didn't know there was a docs mailer and wonders why that is in fact ... :(
21:32:49 <markmcclain> Let's circle back to parallel testing before we dive into Tempest
21:32:58 <markmcclain> salv-orlando: want to update on parallel testing?
21:33:06 <salv-orlando> sure markmcclain
21:33:39 <salv-orlando> http://lists.openstack.org/pipermail/openstack-dev/2013-December/023109.html
21:34:02 <salv-orlando> We have a bunch of patches which are aimed at solving the structural problems we found in the OVS agent.
21:34:15 <salv-orlando> They're are all listed in the email linked above
21:34:37 <markmcclain> http://lists.openstack.org/pipermail/openstack-dev/2014-January/023289.html
21:34:38 <salv-orlando> While running parallel tests we noticed a set of new issues.
21:35:24 * salv-orlando is looking for a link, sorry
21:35:52 <salv-orlando> these issues have all been tagged with neutron-parallel: https://bugs.launchpad.net/neutron/+bugs?field.tag=neutron-parallel
21:36:20 <salv-orlando> a single issue is causing 90% of tests to fail and has to do with an error on port quota check.
21:36:24 <salv-orlando> I have a fix for it.
21:36:39 <markmcclain> ok
21:36:55 <salv-orlando> But sdague rightly asked me to make sure that I'm not just gaming the test and hiding what would be a fundamental issue inneutron
21:37:06 <salv-orlando> but I'm no that anyway
21:37:35 <markmcclain> https://review.openstack.org/#/c/64217/
21:37:38 <salv-orlando> on the other issues, I think most of them are just because the tests are not parallel-safe, or do not take into account things might work differently with neutron
21:37:47 <salv-orlando> except for one error: the ssh protocol banner error
21:38:06 <salv-orlando> I think Nachi had a good hint that it might depend on the metadata server being slow or failing
21:38:12 <salv-orlando> nati-ueno: ^^^
21:38:36 <salv-orlando> because that would explain why ping works and ssh not even if iptables rules are correctly configured on the l3-agent
21:39:03 <salv-orlando> So that's the current situation. What I would love to see is people picking up all the bugs tagged with neutron-parallel
21:39:09 <salv-orlando> and squash all of them by next week
21:39:20 <salv-orlando> that's all from me. Questions?
21:39:51 <markmcclain> it would be really nice to have the parallel bugs solved before we all meetup
21:40:16 * marun gets with the squashing
21:40:24 <markmcclain> marun:  thanks
21:40:43 <markmcclain> salv-orlando: thanks for working through the parallel issue
21:41:31 <salv-orlando> I think it's now time to move to what marun said: the load imposed by tempest
21:41:40 <salv-orlando> happy to talk about that?
21:41:48 <markmcclain> yeah we can discuss
21:42:06 <nati_ueno> salv-orlando: Yes I faced the errors when metadata server didn't working well
21:42:14 <salv-orlando> I think if I interpret marun correctly, tempest with isolation and parallelisms bring the cpu on the gate close to 100%
21:42:19 <nati_ueno> salv-orlando: problem is the ssl certificate configration via metadata server
21:42:22 <marun> i think it's reasonable to do stress testing, but watching neutron fall over in a scenario that no self-respecting operator will allow is crazy.
21:42:47 * mestery agrees with marun.
21:42:48 <salv-orlando> I think this is because every test creates and wires a network with dhcp enabled, and attach is to a router (which is wired as well)
21:43:11 <salv-orlando> now some history, deriving from conversations with mtreinish
21:43:25 <salv-orlando> this has been deemed good, because it stressed the neutron a bit
21:43:59 <salv-orlando> since there is no other form of stress testing (the large ops jobs uses a fake driver, so it stresses just the api server)
21:44:18 <markmcclain> I would agree the stress has been good
21:44:20 <salv-orlando> it is arguable whether stress testing should be part of the current test suite, whose aim is functional
21:44:27 <salv-orlando> and that decision is not from me.
21:44:43 <markmcclain> also agree that functional and stress should be different conversations
21:45:21 <salv-orlando> From my side, I've noticed that in particular creating and wiring a router for every test put such a load on the l3 agent that caused long delays and potential timeouts; so I had a  patch for making router creation only optional
21:46:07 <markmcclain> salv-orlando: so it's our agent not handling the load
21:46:44 <salv-orlando> markmcclain: I would say that if you ask to wire 50 routers in a minute, it's understandable that the agent might take more than 60 seconds to process the load
21:47:18 <markmcclain> yeah agreed
21:47:24 <marun> is that a reasonable usage scenario?
21:47:45 <markmcclain> in a large cloud maybe
21:48:06 <salv-orlando> anyway, I would just like to get a consensus between neutron and qa teams
21:48:24 <salv-orlando> on how to behave wrt network resource creation for each test
21:48:46 <salv-orlando> I am happy with anything :)
21:49:15 <marun> salv-orlando: i think providing stable behavior under parallel execution is the goal.
21:49:21 <markmcclain> ++
21:49:27 <rkukura> Seems to me that the goal for functional (tempest) tests should be to run as quickly as possible, and with minimal resources (i.e. in single VM). Stress testing is different.
21:49:29 <salv-orlando> But I guess we can't do anything better than moving the discussion on the mailing list
21:49:38 <marun> rkukura: +1
21:49:43 <marun> salv-orlando: agreed
21:50:01 <salv-orlando> ok marun, can we count on you to start the thread?
21:50:01 <markmcclain> yeah I'd like to give folks an opportunity to weigh in
21:50:12 <marun> salv-orlando: can do
21:50:18 <markmcclain> marun: thanks
21:50:19 <salv-orlando> marun: thanks
21:50:29 <marun> np
21:50:36 <markmcclain> ok 10 minutes left
21:51:00 <markmcclain> mlavalle: Any items that you included on the agenda that need to be highlighted?
21:51:20 <mlavalle> nope, everything is in the agenda, let's move on
21:51:34 <markmcclain> mlavalle: thanks for providing the update
21:51:47 <mlavalle> np
21:52:03 <markmcclain> Any other Tempest items to discss?
21:52:19 <markmcclain> #topic IPv6
21:52:25 <sc68cal> hello
21:52:38 <markmcclain> I'm still working on the tail-f failure
21:52:51 <markmcclain> I've been trying to track down the system maintainer
21:53:08 <markmcclain> I found one person, but he was the point of contact
21:53:12 <markmcclain> *was not
21:53:21 <sc68cal> Thanks - that's pretty much blocking all of our progress, since it seems to scare off reviewers with the big scary -1
21:53:30 <salv-orlando> markmcclain: I was wondering if it's ok to ask the infra team to temporarily revoke voting rights to the service user
21:53:50 <markmcclain> salv-orlando: that's an interesting proposition
21:53:55 <markmcclain> I'll discussion with them
21:54:21 <sc68cal> We're also looking more deeply into what would need to be done for VIF attributes for hairpinning
21:54:22 <markmcclain> sc68cal: yeah I've changed my gerrit view so that I can see who gave -1s
21:54:29 <markmcclain> ok
21:54:34 <sc68cal> we're finding that it's a libvirt only behavior
21:54:50 <markmcclain> ok
21:55:06 <markmcclain> Anything else?
21:55:13 <sc68cal> So is it worth having a VIF attribute if it's really only one compute driver using it?
21:55:33 <sc68cal> otherwise that's it for me
21:55:44 <markmcclain> that's probably best asked on ML, so that we give everyone a chance to weigh in
21:55:51 <markmcclain> Thanks for the update
21:55:53 <markmcclain> #topic ML2
21:56:00 <markmcclain> rkukura or mestery:?
21:56:23 <mestery> We plan to drive the port delete issue to closure this week at the meeting, there has been a long-standing thread on that since before the holidays.
21:56:36 <mestery> Also, the plan for this week is to refocus on the TypeDriver enhancements as well.
21:56:45 <markmcclain> some closure on that will be good
21:56:54 <mestery> We're seeing a few more Ml2 MechanismDrivers pop up, which is also pretty cool!
21:57:00 <markmcclain> awesome
21:57:18 <mestery> That's about it for ML2, more detail in the Wednesday meeting on IRC.
21:57:35 <markmcclain> thanks for updating
21:57:36 <rkukura> nothing to add
21:57:43 <markmcclain> #topic CLI
21:58:09 <markmcclain> On Friday, a version of dependent lib was released that broke the neutronclient
21:58:21 <markmcclain> we rushed out a temporary fix
21:58:34 <markmcclain> and released 3.3.3 which is compatible with the newer version of cliff
21:58:50 <markmcclain> #action markmcclain to send out email on long term client fix
21:58:55 * anteaya notes verification voting is turned off for tail-f, see ml
21:59:22 <anteaya> also new accounts have verification voting off by default
21:59:42 <markmcclain> anteaya: thanks for clearing that up
21:59:50 <anteaya> np, sorry I'm late
21:59:50 <markmcclain> #topic Open Discussion
22:00:00 <anteaya> montreal weather: http://www.weather.com/weather/tenday/Montreal+Canada+CAXX0301
22:00:05 <pcm_> dumb question...there's been some meeting date/time changes. Is there a wiki with all meetings listed (versus scanning ML)?
22:00:18 <sc68cal> pcm_: https://wiki.openstack.org/wiki/Meetings
22:00:25 <pcm_> sc68cal: Thanks!
22:00:31 <anteaya> we are doing waves of cold and warm, so far looking warm for code sprint
22:00:42 <Swami> Hi Folks, I have posted an updated version of "Distributed virtual Router" blueprint for review. Please review it and provide your feedback.
22:00:43 <marun> oh, on the issue of ml2....
22:00:48 <enikanorov> markmcclain: neutron multihost. is it planned? I guess it should be on nova-parity list?
22:00:52 <Swami> link: https://docs.google.com/document/d/1iXMAyVMf42FTahExmGdYNGOBFyeA4e74sAO3pvr_RjA/edit
22:01:03 <marun> requiring the agent to be up to bind ports synchronously.  good? bad? discuss!
22:01:11 <enikanorov> i also remember a patch from gongysh implementing multihost
22:01:15 <markmcclain> enikanorov: yes there are multiple groups working on solutions to tackle it
22:01:42 <rkukura> marun: Sounds like a question for openstack-dev
22:01:42 <enikanorov> markmcclain: do you knwo who can I contact with to discuss it?
22:01:42 <mestery> marun: Is that with OVS and/or LB? I would think it has to be up for a synchronous bind, no?
22:01:47 <markmcclain> enikanorov: the original patch from gongysh I don't think would be accepted right now
22:01:57 <enikanorov> that's for sure
22:02:05 <markmcclain> I'll encourage the teams working on solutions to post updates to the ML
22:02:15 <enikanorov> ok, thanks
22:02:42 <marun> rkukura: fair enough
22:02:47 <rkukura> marun, mestery: I think this is the "fail to bind if agent isn't currently alive" vs. "there has been an agent on that node sometime in the past so it will eventually work, so lets just bind now"
22:02:58 <markmcclain> marun: that's definitely a ML discussion since we're over time
22:03:08 <gongysh> enikanorov: I think the DVR is a direction to replace multihost.
22:03:15 <marun> mestery: neutron is a distributed  app.  we need to act accordingly
22:03:44 <mestery> marun: Lets discuss on ML :)
22:03:55 <markmcclain> Thanks to everyone for stopping in this week… remember we have the IRC channel and ML to discuss items between meetings
22:03:55 <marun> mestery: :)
22:04:00 <markmcclain> #endmeeting