16:00:54 #startmeeting containers 16:00:54 Meeting started Tue Mar 1 16:00:54 2016 UTC and is due to finish in 60 minutes. The chair is adrian_otto. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:55 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:58 The meeting name has been set to 'containers' 16:01:54 the link to the agenda is not available, will add it later 16:02:01 o/ 16:02:01 #topic Roll Call 16:02:05 Adrian Otto 16:02:06 o/ 16:02:07 Thomas Maddox 16:02:07 o/ 16:02:07 o/ 16:02:09 o/ 16:02:10 Ton Ngo 16:02:10 o/ 16:02:11 murali allada 16:02:11 o/ 16:02:12 o/ 16:02:13 Perry Rivera o/ 16:02:15 Corey O'Brien 16:02:26 o/ 16:02:33 Rob Pothier 16:03:54 o/ 16:04:07 hello strigazi dimtruck thomasem levi_b madhurik himanshu_garg Tango Kennan muralia dane_leblanc hongbin juggler coreyob jward rpothier and sew 16:04:15 let's begin 16:04:20 #link https://wiki.openstack.org/wiki/Meetings/Containers#Agenda_for_2016-03-01_1600_UTC Our Agenda 16:04:20 good localtime adrian_otto and all 16:04:30 thanks! 16:04:36 #topic Announcements 16:04:45 any announcements from team members? 16:04:50 adrian_otto: just one from me 16:04:54 ok 16:05:02 I have one after 16:05:28 ok, proceed dimtruck 16:05:28 copy_logs patch landed (#link https://review.openstack.org/#/c/278127) so if you have any patches in progress that are failing intermittenly in functional tests, rebase and you'll get bay node logs 16:05:45 terrific! 16:05:47 +1 16:05:57 thanks for all help coreyob hongbin, adrian_otto 16:06:03 hongbin: go ahead 16:06:08 #link http://lists.openstack.org/pipermail/openstack-dev/2016-February/087771.html 16:06:22 i18n is looking for liasion for magnum-ui 16:06:45 anyone interested is welcome to volunteer 16:07:03 one note: - The liaison should be a core reviewer for the project and understand the i18n status of this project. 16:07:31 o/ 16:07:45 dimtruck: I think most importantly, the liasion needs to know the release cycle of magnum-ui 16:08:06 suro-patz: are you letting us know you are present, or expressing interest in I18n? 16:08:46 or both? 16:09:23 I think we not limited to core reviewer, if someone have ability to do that 16:09:56 +1 16:09:59 Kennan: i read that as cores only :( maybe a clarification from i18n is needed? 16:10:18 thanks hongbin for raising that for discussion. Let's table it for now and let our contributors consider that for a bit. 16:10:28 sure 16:10:35 adrian_otto: I just got into the room - letting know my presence 16:10:41 thanks suro-patz 16:10:43 welcome 16:10:58 #topic Review Action Items 16:11:13 (I have a gap here, checking to see which actions need follow-up) 16:11:16 one moment please 16:11:41 (none) 16:11:44 that explains the gap 16:11:58 #topic Blueprint Review 16:12:04 Essential Blueprint Review 16:12:11 We have two docs blueprints 16:12:33 #link https://blueprints.launchpad.net/magnum/mitaka Our Mitaka Blueprints 16:12:57 #link https://blueprints.launchpad.net/magnum/+spec/magnum-troubleshooting-guide Troubleshooting Guide (Tango) 16:13:17 #link https://blueprints.launchpad.net/magnum/+spec/user-guide User Guide (Tango) 16:13:21 We have 1 patch merged last week, and another still under review 16:13:39 We might want to consider adjusting the priority of the install instructions blueprint 16:13:50 I see dimtruck signed up for 2 new sections, so that's very helpful 16:13:55 Tango: glad to hear that!! 16:14:02 yup! i'll push those in later this week! 16:14:08 thanks dimtruck! 16:14:40 I am still working on image management, had to do some internal work last week so didn't make as much progress as I would have liked 16:15:03 I think install instruction is also important, we have repeated request on this 16:15:04 hongbin: do you have that link to the recent blueprint for install docs? 16:15:12 finding it 16:15:16 thanks 16:15:34 #link https://blueprints.launchpad.net/magnum/+spec/magnum-installation-guide 16:16:06 ok, I'm marking that one as high, because it's surfaced as a gap for operators 16:16:24 does anyone think that one should be upgraded to Essential? 16:16:46 I think it should be essential :) 16:16:54 +1 16:17:02 +1 for me 16:17:03 ok, marking it as such 16:17:05 +1 16:17:40 strigazi: I know you took a look at this, right? 16:17:58 yes, I'm working on it 16:18:21 ok, I'm marking you as the assignee. 16:18:34 this means that I'll seek updates from you at the team meetings for progress on it 16:18:42 ok 16:18:43 any contributors are free to join you in working on it 16:19:35 tx Tango for your remarks and progress on the other Essentials. 16:19:41 Blueprints, Bugs, Specs, and other work items to be discussed as a team 16:19:55 Revisiting form last week 16:19:56 1) swarm related heat templates (for storage support) https://review.openstack.org/#/c/275034/ 16:20:26 wanghua: you indicated you would revisit this 16:20:57 I'm willing to reverse my -2 vote if we can make it work with selinux enabled. 16:21:01 adrian_otto: redhat folks still not find root cause, they thought volume plugin issue, and volume lugin said docker plugin issue 16:21:15 ok, should I nudge this to next week? 16:21:57 please let me know if/when you'd like me to revisit this. Ping me on IRC anytime. 16:21:58 adrian_otto: I think we could make it more if updates from redhat folk 16:22:11 I like to go on investigate it 16:22:25 ok. 16:22:30 adrian_otto: i think it is needed in magnum feature for storage support 16:22:54 agreed 16:23:13 so I just want to remind you that I'm not opposed to the feature, just the details on the proposed implementation. 16:23:32 sure I have bring our options to those folks 16:23:36 ok, let's come back to this one again next time 16:23:39 but seems hard to debug for them now 16:23:52 2) [suro-patz] Looking for consensus on https://review.openstack.org/#/c/275003/, to let https://review.openstack.org/#/c/267134/ proceed 16:24:25 This is about improving async between conductors and COEs 16:24:28 I am looking for the spec review to conclude 16:25:16 so that we can focus on review for the phase-0 implementation 16:25:17 ok, so this is essentially about adjusting call to cast throughout the relevant conductors 16:25:33 and you have an idea about how to stage the work 16:25:54 the phase-0 implementation is up for review for last 1 month 16:25:56 do any team members have concerns about the proposed approach? 16:26:11 if not, let's finish up this review and let it proceed 16:26:16 https://review.openstack.org/#/c/275003/ - is the code 16:26:24 In overall, it looks over complicated. 16:26:40 I won't -1 on that. Just raise some concerns 16:27:05 hongbin: async implementations are not simple, that's why people implement sync first 16:27:09 hongbin: can you suggest a way to reduce the complexity to a point where it's better? 16:27:21 hongbin: I am open to suggestion 16:27:31 adrian_otto: Don't have any idea right now 16:27:40 ok, so I'm going to +2 this patch now 16:27:42 suro-patz: is it just used by Magnum now for asynch ways? 16:28:13 Another concern is 16:28:20 kennan: nova achieves async too 16:28:43 It relies on several unfinished features from other project/libraries 16:28:45 but the COE-interaction is specific to Magnum 16:28:46 suro-patz: you mean asycn instances actions and opeartions in nova ? 16:28:59 hongbin: those dependencies are for phase-1 16:29:02 on taskflow 16:29:21 if you have better alternatives for them, please suggest : hongbin 16:30:02 power of rejection comes with the responsibility of providing alternatives :) 16:30:03 I would say not rely on unfinished features 16:30:34 hongbin: in my study, I could not find any other fitting alternatives 16:30:34 it might be okay to use a new feature as long as it is accompanied by a commitment form the upstream project to support us 16:30:44 OK 16:30:52 Then, np from me 16:31:03 hongbin : +1 the only concern i have is the dependence on the unfinished features from other projects libraries and not sure how it will go since either us or someone/owner of that lib to finish it 16:31:04 but I do appreciate the caution 16:31:17 btw, for those enhancements I have bounced the ideas with respective teams, and I am willing to do those first 16:31:41 I will clear the dependency and then come back to implement phase-1 16:31:42 suro-patz : i think you have highlighted them in the spec as well 16:31:51 the need for mysql driver etc 16:31:55 good, if we can track down who's doing that work, and work closely than we can mitigate the risk to a degree 16:31:56 vilobhmm: yes 16:32:00 I think exploer is welcoming if it is proved to be useful and effiency 16:32:23 kennan: thanks 16:32:26 and the spec offers a way to fall back to the current (sync) mode in the early phase 16:32:40 also this feature is implemented as optional by configuration 16:32:56 so we can decide when it's the right time to merge code for the later part based on the stability we observe from the experimental mode 16:32:57 adrian_otto: absolutely 16:33:26 the implementation is divided across phases to reduce risk 16:33:43 put as an option for people to try to out with safe fall back 16:34:11 thanks suro-patz 16:34:22 the phase-1 will start picking up speed as community feels comfortable with the feature and the dependencies are esolved 16:34:26 resolved 16:35:00 suro-patz: I want to kind remind just make sure the new feaure not bring breaks for old code 16:35:01 ok, I'm happy to work out any concrete objections while we are reviewing the WIP reviews on the implementation 16:35:42 Kennan: of course 16:35:46 let's check on the next work item 16:35:49 3) thomasem to get Libvirt/LXC Nova gate test to voting 16:35:54 all the UT will be run in both the modes 16:35:54 hey 16:36:03 thomasem: sorry to call you out on this 16:36:09 So, back digging into that now. Finding some issues with libvirt_lxc reboots 16:36:21 Not a problem. I specifically worded it that way in the agenda so I would be pinged. :) 16:36:23 the gate nodes reboot? 16:36:32 No, the nova instance reboots 16:36:43 during func test? 16:36:54 It falls through to a hard reboot (being a container) and winds up trying to mount an already mounted volume. 16:36:56 Race condition sort of thing 16:37:07 ick 16:37:10 So, trying to find more deterministic ways to perturb that issue so I can get better information around it. 16:37:26 The gate stuff is riddled with race condition problems. So, enumerating those, and trying to address one-by-one. 16:37:28 ok 16:37:36 Anyway, my status is: In Progress 16:37:42 and probably will be for a while on-and-off. :) 16:37:49 should I place an AI for follow up next week? 16:37:57 Keeping track of things here: https://etherpad.openstack.org/p/lxc_driver_devstack_gate 16:38:13 Sure. I'll hopefully have more information then, maybe even a patch up. 16:38:14 do we have a bug ticket open as well? 16:38:32 if not I can help get that done for you 16:38:54 Actually, no. We need to open one against openstack-infra, I think. 16:39:14 We have one for udevadm settle 16:39:23 lemme find that one 16:39:47 #action thomasem to update team on progress refactoring our gates to use the libvirt-lxc nova driver 16:39:54 sound fair? 16:40:02 So, it's nova's gate right now. 16:40:10 Magnum's gate is a larger and different effort 16:40:16 adrian_otto: thomasem : Maybe it worth to explain to the team why libvirt-lxc needs to be enabled (not everyone attend the midcycle ) 16:40:17 #undo 16:40:18 Removing item from minutes: 16:40:47 There's a few implications for libvirt LXC and Magnum 16:40:50 One 16:40:53 irc://chat.freenode.net:6667/#action thomasem to update team on progress on libvirt-lxc work 16:41:04 #action thomasem to update team on progress on libvirt-lxc work 16:41:09 there, that's less specific 16:41:16 We're considering trying to use libvirtLXC to improve Magnum's gates once we feel it's stable enough 16:41:31 Since there's such dramatic resource problems with Magnum gate using VMs 16:41:44 ok, we have just a little time for a big topic 16:41:47 thanks thomasem 16:42:04 adrian_otto : https://review.openstack.org/#/c/259201/ - Resource Quota - Introduce Quota Table got merged last week 16:42:04 furthermore, Libvirt/LXC being stable in Nova would enable Magnum using containers for infrastructure instead of VMs 16:42:06 No problemo 16:42:08 carry on 16:42:20 #topic Bay node OS support in Magnum 16:42:20 Sending a ML would be great I think 16:42:37 #link http://thread.gmane.org/gmane.comp.cloud.openstack.devel/77324 ML DIscussion (spans Feb/Mar month boundary) 16:42:51 adrian_otto : may be we cab get back to this topic when we are done with "Bay node OS support in Magnum" 16:44:20 well the reason I added this for discussion today is because I want to caution us about focusing too much on image/template support 16:44:37 as it could derail more compelling features 16:45:17 we all want options to allow alternatives 16:45:28 adrian_otto: so you think it is costly to support additonal OS? Could you elaborate the reason? 16:45:36 but the question is which reference implementation(s) should our magnum dev team assume 16:46:11 Take for example the time that Tango spends working on making the Fedora/Atomic one work 16:46:36 if similar efforts were duplicated for others, that could be a material issue for us 16:46:43 There has been suggestion to move away from Atomic 16:47:06 I think Ton's work can be re-use for other OSs 16:47:21 Basically, there are the same troubleshooting steps 16:47:40 we almost has nothing specific to Atomic 16:48:03 the magnum dev team has not expressed any religious view with respect to which OS we support, but rather an interest in streamlining our focus to have a reference implementation, and a suitable way to plug in alternatives 16:49:12 but we do have religious for no vendor lock-in :) 16:49:13 i think we're already seeing the pain of having multiple os support with kubernetes. atomic and coreos versions don't even have consistent feature support 16:49:22 hongbin: agreed 16:49:48 coreyob: what is the pain? Could you elaborate? 16:50:17 we basically had to develop the TLS for kubernetes feature twice, once for atomic and once for coreos right? 16:50:22 adrian_otto: my argument is two OSs support could prevent vendor lock-in 16:50:25 the biggest problem today is separate config files for Atomic and CoreOS 16:51:00 coreyob: We have the Heat conditional features, which could consolidate the templates from different OSs in the future 16:51:05 hongbin: yes, I see that. What I don't see is the engineering (dev) commitment to accompany that desire for diversity. 16:51:18 hongbin we discussed that at the midcycle and it doesn't work with versioned heat templates 16:51:36 I think right now, atomic have coreos did have some different (which means not have one template can work all cases) 16:51:59 coreyob: Why it doesn't work for versioning? Example? 16:52:03 instead of debating which one is better, I think we need to be debating how many os distros we want to support for each coe 16:52:33 but I welcome contributions for that for os, like 3-rd party pipeline and repos maintain ? 16:52:43 Kennan yes 16:52:56 or even in our tree but not as a part of our gate 16:53:17 I think how to consolidate templates is a technical issue 16:53:28 How many OSs to support is directional question 16:53:33 i think starting with a single distro per coe is a great way to handle a reference implementation 16:53:42 then add more as demand surfaces 16:53:45 my idea on this is to allow upstream OS projects to have a clear Magnum integration procedure, and ask if they would like to carry that responsibility to keep it updated, and working as Magnum evolves. 16:54:13 hongbin: it's not just templates all shell scripts need to be consolidated as well 16:54:13 ^^ was especially useful for the RHEL usecase 16:54:23 eghobo: yes 16:54:25 Since that's a licensed thing and all of that 16:54:33 coreyob hongbin perhaps we can limits some popular os ? to archieve a common goal ? 16:54:58 like systemd related and upstart related OS (since shell care about it ?) 16:55:07 Kennan: As long as the OSs is more than one. I am happy 16:55:17 also it's not about popularity, it's about stability 16:55:18 I'm not particular to which os distro we choose as long as there is only 1 we have to maintain a gate for 16:55:48 do we really want test/catch all Docker/Kub bugs at Atomic 16:56:13 i don't want to have to develop features for 6 coe+distro combinations. I'd rather do just 3 (1 per COE) 16:56:55 coreyob: We could have two distro for k8s 16:57:05 coreyob: since it is a popular COE 16:57:50 So it seems we have 2 decision points: (1) proceed on a design for the plugin mechanism, (2) which OS 's to focus on as reference implementation for each COE 16:57:58 coreyob: We don't have to populate multi-distro to other COEs, if we cannot find a volunteer to maintain 16:58:01 if you think we should do coreos for kubernetes I'm fine with dropping atomic instead. I think it'd be easier to have 2 base images (atomic and ubuntu) instead of 3, but I'd much rather have 1 per coe 16:58:55 If we have developers willing to support, it should be OK 16:59:00 ok, let's plan to follow up on this topic again 16:59:12 as we have reached the end of our scheduled meeting time for today 16:59:13 so hongbin are you ok with atomic+swarm, coreos+kubernetes, and ubuntu+mesos as the reference implementations? 16:59:15 yes need further discsussion about it 16:59:31 we can also continue that ML discussion 16:59:59 coreyob: kuberntes should have an additonal distro 17:00:07 our next team meeting will be on Tuesday 2016-03-08 at 1600 UTC. Thanks for attending everyone. 17:00:08 adrian_otto last question, what's the cut date for M relase for Magnum ? 17:00:17 thanks adrian 17:00:21 Kennan: will follow up in #openstack-containers 17:00:23 thanks all! 17:00:31 #endmeeting