21:00:17 #startmeeting quantum team meeting 21:00:18 Meeting started Mon Sep 24 21:00:17 2012 UTC. The chair is danwent. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:19 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:20 The meeting name has been set to 'quantum_team_meeting' 21:00:50 http://wiki.openstack.org/Network/Meetings 21:01:25 hi 21:01:26 hello 21:01:26 hi 21:01:32 Ok, so announcements 21:01:41 Official folsom release is this week. 21:02:23 I think its technically end of week (27th?), but we need things all wrapped up before that obviously 21:03:14 so, first want to talk through any and all outstanding issues, then focus on docs 21:03:30 #topic possible folsom bugs 21:03:32 https://bugs.launchpad.net/quantum/+bugs?field.tag=folsom-rc-potential 21:03:49 salv-orlando: https://bugs.launchpad.net/quantum/+bug/1054387 21:03:51 Launchpad bug 1054387 in quantum/folsom "oom while executing unit tests" [Critical,Fix committed] 21:04:25 this was unit tests only, correct? we'll need to pull it over the the stable branch, but it does not require a new RC, correct? 21:04:27 If we merge it for folsom release we might avoid angry developers 21:04:42 but won't affect production 21:04:48 so, no new RC needed 21:04:53 ok, thanks 21:05:00 https://bugs.launchpad.net/quantum/+bug/1053799 21:05:02 Launchpad bug 1053799 in quantum "zmq support for quantum agents (pull fix from openstack-common)" [High,In progress] 21:05:02 I'm a bit concerned running the unit tests went from ~3 minutes to ~18 minutes, and thats with an SSD 21:05:24 rkukura: yes, me too. is there an alternate direction we could go in? 21:05:36 rkukura: fixing sqlite in memory db 21:05:38 rkukura: perhaps resetting or clearing the in-memory db between plugin runs? 21:05:48 danwent: does not work 21:05:59 hmm.. the unit tests got faster for me 21:05:59 salv-orlando: i figured you would have explored such things 21:06:03 maybe putting the DB in some sort of RAM FS 21:06:06 clear_db destroys the tables but the memory is not freed 21:06:27 rkukura: that is doable 21:06:29 helo 21:06:35 ok, can we file an issue to explore speeding up unit tests? 21:06:42 +1 21:06:43 rkukura: want to take that one? 21:06:45 +1 21:06:57 or anyone really, but rkukura mentioned it :) 21:07:06 I'd prefer someone more familiar with sqlite do it, but I can if necessary 21:07:24 rkukura: can you at least create the issue and make sure it get assigned to someone? 21:07:29 sure 21:07:30 I don't think you need to run the tests with root permission to mount a ram fs so, it should be ok 21:07:37 thx 21:07:46 next issue: https://bugs.launchpad.net/quantum/+bug/1053799 21:07:49 Launchpad bug 1053799 in quantum "zmq support for quantum agents (pull fix from openstack-common)" [High,In progress] 21:08:13 there was a fixed merged for this, but it didn't seem to fully work, so we never pulled it to the RC 21:08:36 I don't see this as a release blocking issue, though we can consider pulling it in sometime later once other changes require a stable update. 21:08:44 +1 21:08:51 i understand that there may be another patch. i have yet to see/check it 21:09:21 we'll need to be careful resynching openstack-common into the stable branch 21:09:36 garyk: yes, eric has a patch out. i think the problem is that no one really knows that code in openstack.common in order to feel comfortable reviewing it this late in the cycle 21:09:43 yes. it seems openstack-common code is not tested enough. 21:09:53 danwent: ok 21:10:07 danwent: will you make q-iptables for folsom ? 21:10:27 anyway, i'm mainly voicing it hear to make sure that no one is screaming for it to be in initial folsom drop 21:10:35 zykes-: we will be discussing that issue next 21:11:28 next issue isn't really a bug, so much as an unfortunate result of not getting real quantum security groups in folsom: https://bugs.launchpad.net/quantum/+bug/1053819 21:11:30 Launchpad bug 1053819 in quantum "nova security group iptables rules conflict for overlapping subnets" [High,Confirmed] 21:11:57 any changes here would be in nova, which means its even harder for us to potentially change it in Folsom, even if we wanted to. 21:12:51 the one thing we could do in quantum would be to add a flag that check to prevent overlapping subnets, if desired by the administrator. This could prevent misuse that leads to unexpected security issues. 21:13:31 I did some initial inspection to figure out what changes might be required in nova. They seem doable with a day or so of work, but certainly non-trivial in terms of code changed and testing required. 21:14:04 so clearly we're going to have to make this clear in docs. the question is what if anything we do in terms of code. 21:14:20 Do u want to introduce the namespace into nova if we fix it into nova? 21:15:09 gongysh: with all the due respect I guess that is a bit unlikely 21:15:15 gongysh: the design I sketched out in the bug would effectively make nova security groups namespace aware. I'm not a big fan of that approach in general, but it was the only option I saw in terms of fixing it in nova. 21:15:36 danwent: you're adding a new security group driver? 21:15:37 i wrote it up more as a thought experiment than as an actual proposal to make the changes. 21:16:35 salv-orlando: yes, I thin it would amount to a new driver that leverages a lot fo the code from the existing iptables driver. I was trying to figure out a way to minimize the amount of new code, and make sure that what new code was required would only be used if someone turned this capability on explicitly. 21:16:58 but my general conclusion was that this was non-trivial 21:17:02 Does all this depend on the hybrid OVS VIF driver? 21:17:13 in addition, we have a problems with removing a namespace when combining with iptables. It is not solved yet (with l3-agent). 21:17:15 I said it was unlikely because this kind of change would scare people anyway so late in the release cycle (albeit being self contained) 21:17:20 usability killer ;p 21:17:57 rkukura: the whole quantum + sec.group feature depends on the hybrid driver, methings 21:18:05 salv-orlando: not technically 21:18:13 oh yes 21:18:16 secgroups with work bridge driver just fine 21:18:27 that's right. 21:18:30 I'm just wondering if filtering at the router makes more sense for many tenants? 21:18:43 bob, i think we could do something similar for the bridge driver, but we'd have to make it look something like the hybrid driver.. pretty yucky 21:19:11 Is the security group concept more relevant when different tenants share the same L2 net? 21:19:22 Let's do it on the quantum side. I think our quantum.conf has a configuraiton item "usenamespace=true|false" alike. 21:19:22 amotoki: agreed, that issue isn't fixed yet either, though garbage namespaces aren't a huge issue. 21:19:44 Documenting it seems enough. 21:19:49 rkukura: in many cases, filters on routers could be used, though there are some cases where security groups are "defense in depth" 21:20:06 so no security groups ? 21:20:29 But they have a major cost in requiring all those veths, etc., so I'm wondering if a router-based solution might be better in some use cases. 21:20:52 gongysh: well, we can't fix it on the quantum side until we fully implement security groups there (this was targeted for F-3, but not completed). But what we can do on the quantum side is prevent people from getting into the bad state of using nova security groups and then allowing overlapping IPs. 21:22:01 rkukura: i agree that a router based approach is probably a better way to acheive many of the use cases people are trying to do, but security groups have a very well-definied semantics that they are applied on the port/vNIC, so I think router-based filtering will need to be another abstraction all together. 21:23:29 danwent: Seems like something to discuss at the summit 21:23:35 ok, so options are 1) doc only 2) doc + quantum flag to prevent overlapping subnets 3) continue to explore other options 21:23:48 well, really, we have to doc in any of the three cases :) 21:24:21 I guess overlapping ip's are useless then ;) 21:24:33 flag in 2) means a new flag or usernamespace one? 21:24:37 zykes-: would be great if your could make helpful comments 21:25:32 danwent: if it can't be used it should be in the config file at least stating that don't use this unless you have a firewall driver thing at least as you stated above.. 21:25:37 gongysh: that's a good question. it seems a bit odd to mix agent mechanism (namespaces) with API semantics (non-overlapping), so I would tend to say a separate flag, but we do tend to get flag sprawl 21:26:31 zykes-: yes, I would like to have an explicit enablement where the person has to acknowledge that they are not using nova security groups before enabling floating IPs. 21:26:48 the only problem with that is that we'd be changing the default behavior of quantum at the very last minute. 21:27:04 danwent: the person shouldn't be needing firewall at all if they need... 21:27:23 if they need quantum they can't use firewalling i guess with overlapping 21:28:14 yes, basically, they can either use nova security groups, or overlapping IPs, but not both at once. However, there currently is no good way to say "prevent quantum from using overlapping IPs". 21:28:44 also, this constraint is not enforced anywhere in the API - so that would be another todo 21:29:00 The only way to limit the API to admin-access and rely on the admin to not screw things up, which is not a great model. 21:29:35 salv-orlando: I assume this could be done at the db_plugin_base layer… though I haven't investigated the size of the diff. 21:29:56 so my proposed way forward would be to have someone explore the complexity of adding a flag that prevents overlapping subnets. 21:30:08 danwent: diff could be slow - we have the same check for different subnets on the same network, should be extended to all subnets 21:30:19 danwent: I take this 21:30:40 If that is easy, will we also go without namespaces by default? 21:31:31 i'm not a huge fan of disabling namespaces by default for two reasons: 1) its a last minute change in default behavior and 2) its not inline with how quantum will work long-term (this security group issue should be a folsom only thing) 21:31:33 salv-orlando: sounds reasonable. i don't like the fact that one has to restart a service to update a flag 21:31:41 for security issue, I prefer to go without namespace by default. 21:31:55 that said, it disabling overlapping by default is the "secure" thing to do. 21:32:03 danwent: I agree. Sounds we are in a kind of deadlock 21:32:16 well, i think the default needs to go to security 21:32:28 or rather, the "tie goes to the more secure approach". 21:32:54 do folks have significant concerns with that? 21:33:18 +1 for secure option 21:33:26 danwent: I would have loved to see namespaces by default - but I too understand security concerns take priorirty 21:33:26 i'm on the fence here, but when that's the case, it seems best to be conservative 21:34:00 I will help Savl to do reviewing. 21:34:09 Implications for l3 agent? I think it's now by default 1 router per agent, or if you want 1 router per tenant 21:34:12 ok, so this way we can have a note in the config file by enabling overlapping IPs that makes it very clear that enabling this is not compatible with nova security groups. 21:34:15 sorry 1 agent per tenant 21:35:22 l3-agent is currently n-routers per agent. explicit configuration of a router-uuid is needed to tie an agent to a single router. 21:35:44 sorry to interrupt, why not move the usernamespace flag into plugin, so that agent and plugin can use them together. 21:36:12 gongysh: in theory you could allow overlapping IPs, but still not want to use namespaces 21:36:14 usernamespace imply overlapping, right? 21:36:56 gongysh: not quite, as you could still support overlapping IPs by implementing multiple routers using l3-agents on different physical hosts (thereby not needing namespaces). 21:37:15 what you said is correct only if there's just one l3-agent in a deployment. 21:38:16 ok, so salvatore, can you create a bug for this and tag with folsom-rc-potential 21:38:44 sure - the bug would for the API related changes + change in configuration file. Or should I add something else? 21:38:47 if we're going to change the default behavior, we clearly would need a new RC for that, as we can't make that change post release. 21:39:10 salv-orlando: I'd scope it to that. We'll want this change to be as surgical as possible. 21:39:39 danwent: +1M - anything beyond the 10 lines diff would be too much 21:39:53 hopefully you're right :) 21:39:58 not including tests :P 21:40:01 ok: it seems a serious stuff now. 21:40:12 ok, let's keep moving, as we have a lot of ground to cover on docs as well. 21:40:30 garyk: https://bugs.launchpad.net/quantum/+bug/1055384 21:40:32 Launchpad bug 1055384 in quantum "dnsmasq - Stderr: 'Option "-no-hosts" is unknown, try "ip -help".\n'" [High,In progress] 21:40:45 danwent: i updated the comments. hope that this suffices 21:40:54 garyk: ok, great. will re-review 21:41:02 other +2 already was salv-orlando ? 21:41:07 if so, he can quickly +2 as well. 21:41:14 yes 21:41:20 great. 21:41:22 yeah, that one gave me a headache today... I was wondering why my resolv.conf changed each time I restarted quantum services... 21:41:33 assuming we have a change coming for the overlapping thing, it seems like this is a good fix to also include in a new RC? 21:41:51 danwent: definetely 21:41:56 +1 21:42:07 ok, let's plan on that. 21:42:14 or actuall -1 in my case :) 21:42:16 https://bugs.launchpad.net/quantum/+bug/1051744 21:42:18 Launchpad bug 1051744 in quantum "remove default value of 'local_ip' of 10.0.0.3 in ovs_quantum_plugin.ini " [Medium,Fix committed] 21:42:23 https://bugs.launchpad.net/quantum/+bug/1053633 21:42:35 these last two bugs are fairly minor. 21:42:54 i'll move past them unless someone things they really need to be discussed 21:43:10 #topic quantum docs 21:43:33 API docs are in good shape, though we always welcome more eyeballs to reivew: https://github.com/openstack/openstack-manuals/tree/master/doc/src/docbkx/openstack-network-connectivity-admin 21:43:39 salv-orlando: anything to mention on this end? 21:44:15 our real focus needs to be getting the admin guide in shape… its improving, but still have many rough spots 21:44:32 it has been converted to docbook, meaning the source lives here: https://github.com/openstack/openstack-manuals/tree/master/doc/src/docbkx/openstack-network-connectivity-admin 21:44:34 Yes - I am completing the L3 extensions documentation, which is bait long 21:44:47 salv-orlando: ok, great. thanks for handling that 21:44:55 bit long - and might appreciate some help with the provider networks extension 21:45:12 I'm doing the provider networks section 21:45:21 rkukura: ok, great 21:45:33 salv-orlando: that's the best kind of helping :) 21:45:37 I'm doing quota part. Left is translation for docbook 21:45:48 nati_ueno: is that in Api guide or admin guide? 21:46:00 danwent: IMO, it is API guide 21:46:04 salv-orlando: do you have anything started on provider networks I should pick up? 21:46:13 a bare skeleton 21:46:18 nati_ueno: well, utlimately i think its both, but i was just asking what part you were working on. 21:46:24 * vish1 has a minor thing to mention during open discussion 21:46:39 danwent: he's doing the API extensions documentation 21:46:42 vish1: will be a bit. do you just want to mention it now? 21:46:44 danwent: Ah, I got it. I'll push them sepalately 21:47:03 danwent: How can I decide, which part is admin part? 21:47:56 nati_ueno: admin part would be the part when you say what are quotas and how to configure the global limits 21:47:56 nati_ueno: I think describing the flags for setting default quotas, admin commands for modifying quotas, user commands for viewing quoats, would all be in admin guide. 21:48:17 danwent: salv-orlando: I got it :) 21:48:31 Ok, so we still have a good number of gaps remaining. 21:48:48 Are we covering provider API extensions in API guide? 21:48:52 I have some that i've noticed listed on the meeting agenda. 21:49:11 rkukura: yes, i think that is what salv-orlando was asking for help on 21:49:29 OK, I can look at that along with the admin guide part 21:49:33 bug 1055822 for the delight of my dreams! 21:49:33 Launchpad bug 1055822 in quantum "Ensure subnet do not overlap when IP overlapping is disabled" [Critical,Confirmed] https://launchpad.net/bugs/1055822 21:49:44 salv-orlando: ok, thanks. 21:49:58 http://wiki.openstack.org/Network/Meetings 21:50:21 so I am planning doing some diagrams around different deployment models 21:50:21 do we have a few minutes to discuss the ipnets cleanup tool? 21:50:29 garyk: open discussion? 21:50:36 danwent: ok 21:50:37 or do you need to run? 21:50:45 no i'm good 21:50:51 just want to get people to sign up for some docs sections, then we'll have open discussion 21:50:57 http://docs.openstack.org/api/openstack-network/2.0/content/notifications.html 21:50:59 ok, sorry 21:51:00 fedora/red hat innstall 21:51:04 install 21:51:05 seems I have to do notification. 21:51:23 danwent: i'll take that 21:51:30 gongysh: hehe, apparently. you able to do that? 21:51:36 garyk: thanks. 21:51:37 ok. 21:51:39 danwent: sure: https://review.openstack.org/#/c/13539/ 21:51:40 bye folks :) 21:51:43 I will fix that part. 21:52:12 gongysh: I have a patch for removing that section actually 21:52:21 garyk: one open question i had was whether there is too much config in the ubuntu install section, and perhaps some of that should be pulled out into a section that can be shared with fedora. 21:52:25 reason is: I believe it is not API doc. 21:52:35 would be good to get your input after you've had a chance to look 21:52:44 We do not have a REST interface for subscribing to notifications, do we? 21:52:50 salv-orlando: good point on notifications 21:54:33 danwent: not sure. i'll go over it tom and be see what can be done 21:54:33 danwent: I already have a patch under review for removing that section 21:54:33 salv-orlando: they are not official quantum api 21:54:33 garyk: i'm not sure either, so yeah, just wanted to let you know it was an option if that is what you think would be right. 21:54:33 they're just internal at the moment - not open 3rd party services 21:54:33 salv-orlando: which patch 21:54:33 ? 21:54:33 rkukura: you already said you were working on provider nets in admin guide, correct? 21:54:33 yes 21:54:33 https://review.openstack.org/#/c/12877/ 21:54:33 rkukura: 21:54:33 thanks 21:54:33 sounds admin and api guides now 21:54:33 salv-orlando: fyi, i still have you down for writing more about API policies 21:54:42 rkukura: the list keeps growing :) 21:54:52 there's a Section 9 on rootwrap 21:55:01 salv-orlando: why do u remove the quota part? 21:55:22 both distros seems to have rootwrap integrated already, so i'm not sure if people see this as necessary. unless someone does and volunteers to do it, i was planning on removing it. 21:55:31 gongysh: not removed, but moved to API extensions section 21:55:32 to an admin, rootwrap should "just work", I think. 21:55:52 no complaints? ok, good :) 21:56:12 gongysh: do you have a pointer to the stand alone CLI reference? 21:56:22 salv-orlando: sorry. I got it at the end. 21:56:32 seaching... 21:56:38 Chapter 7 was going to be a CLI reference, but instead we can just point to this external doc. 21:57:14 There is also a chapter 10 that is empty, but is supposed to cover operationsl topics like logging (e.g., setting log levels), notifications, and quotas 21:57:27 sounds like nati_ueno will take the quotas stuff for the admin guide. 21:57:31 danwent: http://docs.openstack.org/cli/quick-start/content/index.html 21:57:35 any volunteers on notifications and logging? 21:57:39 gongysh: thx 21:57:50 gongysh: I will insert that at best places in admin guide. 21:58:08 notifications & logging going once... 21:58:13 twice... 21:58:19 since no work for notification at API, I can help with it at admin 21:58:26 gongysh: ok, great :) 21:58:43 on time? 21:58:57 I also think we need a section talking about namespaces vs. non-namespaces, though I'm not sure where it should go. 21:59:07 any volunteers to make sure that content is covered well in the admin guide? 21:59:30 when is the deadline for the docs? 21:59:32 gongysh: asking for a time frame? ideally changes are checked in in the next few days. 21:59:54 I mean you are counting once ... twice... 21:59:58 we can continually update docs, but we need to make sure we have "core" functionality done by the release at the end of week 22:00:03 gongysh: ah :) 22:00:44 do we have core reviewers for the manuals project? 22:00:52 rkukura: yes, good question 22:01:04 annegentle and diane are the main docs folks helping us on that end. 22:01:15 you should have a thread with both of their emails. 22:01:23 danwent: I'll work for namespace section. 22:01:26 though I have not understand what to be written completely. 22:01:31 amotoki: that would be great. 22:02:10 ok. as people identify more gaps in the docs, please send out a note to the list if you think you're not qualitifed to fill the gap yourself. 22:02:15 #topic open discussion 22:02:27 vishy: garyk 22:02:39 netns cleanup 22:02:40 did you have open discussion? 22:03:03 danwent: i wanted to discuss running the netns cleanup as a background task. is this the purpose? 22:03:17 garyk: yes. thanks for bringing that up again. markmcclain around? 22:03:23 garyk: it was designed that way 22:03:33 specifically, should we put a cron job in the packaging? 22:03:57 yes.. because we removed the auto cleanup code late in the game 22:04:08 so a cron job would be good 22:04:11 ok. thanks 22:04:25 But be careful not to remove the newly created namespace. 22:04:32 adam_g: did you see that comment? 22:04:42 gongysh: it's designed to prevent that 22:04:48 gongysh: can you please elaborate 22:04:49 markmcclain: is that for dhcp only, or for l3-agent as well? 22:04:56 it will cleanup both 22:05:30 I think markmcclain can do the explanation better than me. 22:05:48 markmcclain: ok. i'll try and take a look at the code. it makes me a bit nervous that we enable something so late in the game that could potentially introduce issues (not that I don't trust your code :P, i'm just paranoid) 22:05:56 markmcclain: did you see gongysh's point ^^^^ 22:06:17 danwent: understandable… just don't use the —force flag 22:06:26 ok, now i see you answer. 22:06:33 hehe, sounds like good advice :) 22:06:43 #todo #danwent pass this info on to ubuntu packages team as well 22:06:47 garyk: yes… I can take at stab at namespaces 22:07:07 ok, any other open discussion? 22:07:11 sorry, we're way over time 22:07:23 danwent, vishy: "nova flating ip proxy" shoudl be moved after foslom? 22:07:34 amotoki: unfortunately, I think so 22:07:44 np. 22:07:58 it will be very helpful to have that right away in grizzly 22:08:03 ok, thanks folks 22:08:10 final week of Folsom! 22:08:10 thanks and goodnight 22:08:15 #endmeeting