16:00:54 <adrian_otto> #startmeeting containers
16:00:54 <openstack> Meeting started Tue Mar  1 16:00:54 2016 UTC and is due to finish in 60 minutes.  The chair is adrian_otto. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:55 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:58 <openstack> The meeting name has been set to 'containers'
16:01:54 <adrian_otto> the link to the agenda is not available, will add it later
16:02:01 <dimtruck> o/
16:02:01 <adrian_otto> #topic Roll Call
16:02:05 <adrian_otto> Adrian Otto
16:02:06 <strigazi> o/
16:02:07 <thomasem> Thomas Maddox
16:02:07 <levi_b> o/
16:02:07 <madhurik> o/
16:02:09 <himanshu_garg> o/
16:02:10 <Tango> Ton Ngo
16:02:10 <Kennan> o/
16:02:11 <muralia> murali allada
16:02:11 <dane_leblanc> o/
16:02:12 <hongbin> o/
16:02:13 <juggler> Perry Rivera o/
16:02:15 <coreyob> Corey O'Brien
16:02:26 <jward> o/
16:02:33 <rpothier> Rob Pothier
16:03:54 <sew> o/
16:04:07 <adrian_otto> hello strigazi dimtruck thomasem levi_b madhurik himanshu_garg Tango Kennan muralia dane_leblanc hongbin juggler coreyob jward rpothier and sew
16:04:15 <adrian_otto> let's begin
16:04:20 <adrian_otto> #link https://wiki.openstack.org/wiki/Meetings/Containers#Agenda_for_2016-03-01_1600_UTC Our Agenda
16:04:20 <juggler> good localtime adrian_otto and all
16:04:30 <adrian_otto> thanks!
16:04:36 <adrian_otto> #topic Announcements
16:04:45 <adrian_otto> any announcements from team members?
16:04:50 <dimtruck> adrian_otto: just one from me
16:04:54 <adrian_otto> ok
16:05:02 <hongbin> I have one after
16:05:28 <adrian_otto> ok, proceed dimtruck
16:05:28 <dimtruck> copy_logs patch landed (#link https://review.openstack.org/#/c/278127) so if you have any patches in progress that are failing intermittenly in functional tests, rebase and you'll get bay node logs
16:05:45 <adrian_otto> terrific!
16:05:47 <Tango> +1
16:05:57 <dimtruck> thanks for all help coreyob hongbin, adrian_otto
16:06:03 <adrian_otto> hongbin: go ahead
16:06:08 <hongbin> #link http://lists.openstack.org/pipermail/openstack-dev/2016-February/087771.html
16:06:22 <hongbin> i18n is looking for liasion for magnum-ui
16:06:45 <adrian_otto> anyone interested is welcome to volunteer
16:07:03 <dimtruck> one note: - The liaison should be a core reviewer for the project and understand the i18n status of this project.
16:07:31 <suro-patz> o/
16:07:45 <hongbin> dimtruck: I think most importantly, the liasion needs to know the release cycle of magnum-ui
16:08:06 <adrian_otto> suro-patz: are you letting us know you are present, or expressing interest in I18n?
16:08:46 <juggler> or both?
16:09:23 <Kennan> I think we not limited to core reviewer, if someone have ability to do that
16:09:56 <madhurik> +1
16:09:59 <dimtruck> Kennan: i read that as cores only :( maybe a clarification from i18n is needed?
16:10:18 <adrian_otto> thanks hongbin for raising that for discussion. Let's table it for now and let our contributors consider that for a bit.
16:10:28 <hongbin> sure
16:10:35 <suro-patz> adrian_otto: I just got into the room - letting know my presence
16:10:41 <adrian_otto> thanks suro-patz
16:10:43 <adrian_otto> welcome
16:10:58 <adrian_otto> #topic Review Action Items
16:11:13 <adrian_otto> (I have a gap here, checking to see which actions need follow-up)
16:11:16 <adrian_otto> one moment please
16:11:41 <adrian_otto> (none)
16:11:44 <adrian_otto> that explains the gap
16:11:58 <adrian_otto> #topic Blueprint Review
16:12:04 <adrian_otto> Essential Blueprint Review
16:12:11 <adrian_otto> We have two docs blueprints
16:12:33 <adrian_otto> #link https://blueprints.launchpad.net/magnum/mitaka Our Mitaka Blueprints
16:12:57 <adrian_otto> #link https://blueprints.launchpad.net/magnum/+spec/magnum-troubleshooting-guide Troubleshooting Guide (Tango)
16:13:17 <adrian_otto> #link https://blueprints.launchpad.net/magnum/+spec/user-guide User Guide (Tango)
16:13:21 <Tango> We have 1 patch merged last week, and another still under review
16:13:39 <adrian_otto> We might want to consider adjusting the priority of the install instructions blueprint
16:13:50 <Tango> I see dimtruck signed up for 2 new sections, so that's very helpful
16:13:55 <adrian_otto> Tango: glad to hear that!!
16:14:02 <dimtruck> yup!  i'll push those in later this week!
16:14:08 <adrian_otto> thanks dimtruck!
16:14:40 <Tango> I am still working on image management, had to do some internal work last week so didn't make as much progress as I would have liked
16:15:03 <Tango> I think install instruction is also important, we have repeated request on this
16:15:04 <adrian_otto> hongbin: do you have that link to the recent blueprint for install docs?
16:15:12 <hongbin> finding it
16:15:16 <adrian_otto> thanks
16:15:34 <hongbin> #link https://blueprints.launchpad.net/magnum/+spec/magnum-installation-guide
16:16:06 <adrian_otto> ok, I'm marking that one as high, because it's surfaced as a gap for operators
16:16:24 <adrian_otto> does anyone think that one should be upgraded to Essential?
16:16:46 <hongbin> I think it should be essential :)
16:16:54 <Tango> +1
16:17:02 <Kennan> +1 for me
16:17:03 <adrian_otto> ok, marking it as such
16:17:05 <juggler> +1
16:17:40 <adrian_otto> strigazi: I know you took a look at this, right?
16:17:58 <strigazi> yes, I'm working on it
16:18:21 <adrian_otto> ok, I'm marking you as the assignee.
16:18:34 <adrian_otto> this means that I'll seek updates from you at the team meetings for progress on it
16:18:42 <strigazi> ok
16:18:43 <adrian_otto> any contributors are free to join you in working on it
16:19:35 <adrian_otto> tx Tango for your remarks and progress on the other Essentials.
16:19:41 <adrian_otto> Blueprints, Bugs, Specs, and other work items to be discussed as a team
16:19:55 <adrian_otto> Revisiting form last week
16:19:56 <adrian_otto> 1) swarm related heat templates (for storage support) https://review.openstack.org/#/c/275034/
16:20:26 <adrian_otto> wanghua: you indicated you would revisit this
16:20:57 <adrian_otto> I'm willing to reverse my -2 vote if we can make it work with selinux enabled.
16:21:01 <Kennan> adrian_otto: redhat folks still not find root cause, they thought volume plugin issue, and volume lugin said docker plugin issue
16:21:15 <adrian_otto> ok, should I nudge this to next week?
16:21:57 <adrian_otto> please let me know if/when you'd like me to revisit this. Ping me on IRC anytime.
16:21:58 <Kennan> adrian_otto: I think we could make it more if updates from redhat folk
16:22:11 <Kennan> I like to go on investigate it
16:22:25 <adrian_otto> ok.
16:22:30 <Kennan> adrian_otto: i think it is needed in magnum feature for storage support
16:22:54 <adrian_otto> agreed
16:23:13 <adrian_otto> so I just want to remind you that I'm not opposed to the feature, just the details on the proposed implementation.
16:23:32 <Kennan> sure I have bring our options to those folks
16:23:36 <adrian_otto> ok, let's come back to this one again next time
16:23:39 <Kennan> but seems hard to debug for them now
16:23:52 <adrian_otto> 2) [suro-patz] Looking for consensus on https://review.openstack.org/#/c/275003/, to let https://review.openstack.org/#/c/267134/ proceed
16:24:25 <adrian_otto> This is about improving async between conductors and COEs
16:24:28 <suro-patz> I am looking for the spec review to conclude
16:25:16 <suro-patz> so that we can focus on review for the phase-0 implementation
16:25:17 <adrian_otto> ok, so this is essentially about adjusting call to cast throughout the relevant conductors
16:25:33 <adrian_otto> and you have an idea about how to stage the work
16:25:54 <suro-patz> the phase-0 implementation is up for review for last 1 month
16:25:56 <adrian_otto> do any team members have concerns about the proposed approach?
16:26:11 <adrian_otto> if not, let's finish up this review and let it proceed
16:26:16 <suro-patz> https://review.openstack.org/#/c/275003/ - is the code
16:26:24 <hongbin> In overall, it looks over complicated.
16:26:40 <hongbin> I won't -1 on that. Just raise some concerns
16:27:05 <suro-patz> hongbin: async implementations are not simple, that's why people implement sync first
16:27:09 <adrian_otto> hongbin: can you suggest a way to reduce the complexity to a point where it's better?
16:27:21 <suro-patz> hongbin: I am open to suggestion
16:27:31 <hongbin> adrian_otto: Don't have any idea right now
16:27:40 <adrian_otto> ok, so I'm going to +2 this patch now
16:27:42 <Kennan> suro-patz: is it just used by Magnum now for asynch ways?
16:28:13 <hongbin> Another concern is
16:28:20 <suro-patz> kennan: nova achieves async too
16:28:43 <hongbin> It relies on several unfinished features from other project/libraries
16:28:45 <suro-patz> but the COE-interaction is specific to Magnum
16:28:46 <Kennan> suro-patz: you mean asycn instances actions and opeartions in nova ?
16:28:59 <suro-patz> hongbin: those dependencies are for phase-1
16:29:02 <suro-patz> on taskflow
16:29:21 <suro-patz> if you have better alternatives for them, please suggest : hongbin
16:30:02 <suro-patz> power of rejection comes with the responsibility of providing alternatives :)
16:30:03 <hongbin> I would say not rely on unfinished features
16:30:34 <suro-patz> hongbin: in my study, I could not find any other fitting alternatives
16:30:34 <adrian_otto> it might be okay to use a new feature as long as it is accompanied by a commitment form the upstream project to support us
16:30:44 <hongbin> OK
16:30:52 <hongbin> Then, np from me
16:31:03 <vilobhmm11> hongbin : +1 the only concern i have is the dependence on the unfinished features from other projects libraries and not sure how it will go since either us or someone/owner of that lib to finish it
16:31:04 <adrian_otto> but I do appreciate the caution
16:31:17 <suro-patz> btw, for those enhancements I have bounced the ideas with respective teams, and I am willing to do those first
16:31:41 <suro-patz> I will clear the dependency and then come back to implement phase-1
16:31:42 <vilobhmm11> suro-patz : i think you have highlighted them in the spec as well
16:31:51 <vilobhmm11> the need for mysql driver etc
16:31:55 <adrian_otto> good, if we can track down who's doing that work, and work closely than we can mitigate the risk to a degree
16:31:56 <suro-patz> vilobhmm: yes
16:32:00 <Kennan> I think exploer is welcoming if it is proved to be useful and effiency
16:32:23 <suro-patz> kennan: thanks
16:32:26 <adrian_otto> and the spec offers a way to fall back to the current (sync) mode in the early phase
16:32:40 <suro-patz> also this feature is implemented as optional by configuration
16:32:56 <adrian_otto> so we can decide when it's the right time to merge code for the later part based on the stability we observe from the experimental mode
16:32:57 <suro-patz> adrian_otto: absolutely
16:33:26 <suro-patz> the implementation is divided across phases to reduce risk
16:33:43 <suro-patz> put as an option for people to try to out with safe fall back
16:34:11 <adrian_otto> thanks suro-patz
16:34:22 <suro-patz> the phase-1 will start picking up speed as community feels comfortable with the feature and the dependencies are esolved
16:34:26 <suro-patz> resolved
16:35:00 <Kennan> suro-patz: I want to kind remind just make sure the new feaure not bring breaks for old code
16:35:01 <adrian_otto> ok, I'm happy to work out any concrete objections while we are reviewing the WIP reviews on the implementation
16:35:42 <suro-patz> Kennan: of course
16:35:46 <adrian_otto> let's check on the next work item
16:35:49 <adrian_otto> 3) thomasem to get Libvirt/LXC Nova gate test to voting
16:35:54 <suro-patz> all the UT  will be run in both the modes
16:35:54 <thomasem> hey
16:36:03 <adrian_otto> thomasem: sorry to call you out on this
16:36:09 <thomasem> So, back digging into that now. Finding some issues with libvirt_lxc reboots
16:36:21 <thomasem> Not a problem. I specifically worded it that way in the agenda so I would be pinged. :)
16:36:23 <adrian_otto> the gate nodes reboot?
16:36:32 <thomasem> No, the nova instance reboots
16:36:43 <adrian_otto> during func test?
16:36:54 <thomasem> It falls through to a hard reboot (being a container) and winds up trying to mount an already mounted volume.
16:36:56 <thomasem> Race condition sort of thing
16:37:07 <adrian_otto> ick
16:37:10 <thomasem> So, trying to find more deterministic ways to perturb that issue so I can get better information around it.
16:37:26 <thomasem> The gate stuff is riddled with race condition problems. So, enumerating those, and trying to address one-by-one.
16:37:28 <adrian_otto> ok
16:37:36 <thomasem> Anyway, my status is: In Progress
16:37:42 <thomasem> and probably will be for a while on-and-off. :)
16:37:49 <adrian_otto> should I place an AI for follow up next week?
16:37:57 <thomasem> Keeping track of things here: https://etherpad.openstack.org/p/lxc_driver_devstack_gate
16:38:13 <thomasem> Sure. I'll hopefully have more information then, maybe even a patch up.
16:38:14 <adrian_otto> do we have a bug ticket open as well?
16:38:32 <adrian_otto> if not I can help get that done for you
16:38:54 <thomasem> Actually, no. We need to open one against openstack-infra, I think.
16:39:14 <thomasem> We have one for udevadm settle
16:39:23 <thomasem> lemme find that one
16:39:47 <adrian_otto> #action thomasem to update team on progress refactoring our gates to use the libvirt-lxc nova driver
16:39:54 <adrian_otto> sound fair?
16:40:02 <thomasem> So, it's nova's gate right now.
16:40:10 <thomasem> Magnum's gate is a larger and different effort
16:40:16 <hongbin> adrian_otto: thomasem : Maybe it worth to explain to the team why libvirt-lxc needs to be enabled (not everyone attend the midcycle )
16:40:17 <adrian_otto> #undo
16:40:18 <openstack> Removing item from minutes: <ircmeeting.items.Action object at 0x9893210>
16:40:47 <thomasem> There's a few implications for libvirt LXC and Magnum
16:40:50 <thomasem> One
16:40:53 <adrian_otto> irc://chat.freenode.net:6667/#action thomasem to update team on progress on libvirt-lxc work
16:41:04 <adrian_otto> #action thomasem to update team on progress on libvirt-lxc work
16:41:09 <adrian_otto> there, that's less specific
16:41:16 <thomasem> We're considering trying to use libvirtLXC to improve Magnum's gates once we feel it's stable enough
16:41:31 <thomasem> Since there's such dramatic resource problems with Magnum gate using VMs
16:41:44 <adrian_otto> ok, we have just a little time for a big topic
16:41:47 <adrian_otto> thanks thomasem
16:42:04 <vilobhmm11> adrian_otto : https://review.openstack.org/#/c/259201/ - Resource Quota - Introduce Quota Table got merged last week
16:42:04 <thomasem> furthermore, Libvirt/LXC being stable in Nova would enable Magnum using containers for infrastructure instead of VMs
16:42:06 <thomasem> No problemo
16:42:08 <thomasem> carry on
16:42:20 <adrian_otto> #topic Bay node OS support in Magnum
16:42:20 <hongbin> Sending a ML would be great I think
16:42:37 <adrian_otto> #link http://thread.gmane.org/gmane.comp.cloud.openstack.devel/77324 ML DIscussion (spans Feb/Mar month boundary)
16:42:51 <vilobhmm11> adrian_otto : may be we cab get back to this topic when we are done with "Bay node OS support in Magnum"
16:44:20 <adrian_otto> well the reason I added this for discussion today is because I want to caution us about focusing too much on image/template support
16:44:37 <adrian_otto> as it could derail more compelling features
16:45:17 <adrian_otto> we all want options to allow alternatives
16:45:28 <hongbin> adrian_otto: so you think it is costly to support additonal OS? Could you elaborate the reason?
16:45:36 <adrian_otto> but the question is which reference implementation(s) should our magnum dev team assume
16:46:11 <adrian_otto> Take for example the time that Tango spends working on making the Fedora/Atomic one work
16:46:36 <adrian_otto> if similar efforts were duplicated for others, that could be a material issue for us
16:46:43 <Tango> There has been suggestion to move away from Atomic
16:47:06 <hongbin> I think Ton's work can be re-use for other OSs
16:47:21 <hongbin> Basically, there are the same troubleshooting steps
16:47:40 <eghobo> we almost has nothing specific to Atomic
16:48:03 <adrian_otto> the magnum dev team has not expressed any religious view with respect to which OS we support, but rather an interest in streamlining our focus to have a reference implementation, and a suitable way to plug in alternatives
16:49:12 <hongbin> but we do have religious for no vendor lock-in :)
16:49:13 <coreyob> i think we're already seeing the pain of having multiple os support with kubernetes. atomic and coreos versions don't even have consistent feature support
16:49:22 <adrian_otto> hongbin: agreed
16:49:48 <hongbin> coreyob: what is the pain? Could you elaborate?
16:50:17 <coreyob> we basically had to develop the TLS for kubernetes feature twice, once for atomic and once for coreos right?
16:50:22 <hongbin> adrian_otto: my argument is two OSs support could prevent vendor lock-in
16:50:25 <eghobo> the biggest problem today is separate config files for Atomic and CoreOS
16:51:00 <hongbin> coreyob: We have the Heat conditional features, which could consolidate the templates from different OSs in the future
16:51:05 <adrian_otto> hongbin: yes, I see that. What I don't see is the engineering (dev) commitment to accompany that desire for diversity.
16:51:18 <coreyob> hongbin we discussed that at the midcycle and it doesn't work with versioned heat templates
16:51:36 <Kennan> I think right now, atomic have coreos did have some different (which means not have one template can work all cases)
16:51:59 <hongbin> coreyob: Why it doesn't work for versioning? Example?
16:52:03 <coreyob> instead of debating which one is better, I think we need to be debating how many os distros we want to support for each coe
16:52:33 <Kennan> but I welcome contributions for that for os, like 3-rd party pipeline and repos maintain ?
16:52:43 <coreyob> Kennan yes
16:52:56 <coreyob> or even in our tree but not as a part of our gate
16:53:17 <hongbin> I think how to consolidate templates is a technical issue
16:53:28 <hongbin> How many OSs to support is directional question
16:53:33 <sew> i think starting with a single distro per coe is a great way to handle a reference implementation
16:53:42 <sew> then add more as demand surfaces
16:53:45 <adrian_otto> my idea on this is to allow upstream OS projects to have a clear Magnum integration procedure, and ask if they would like to carry that responsibility to keep it updated, and working as Magnum evolves.
16:54:13 <eghobo> hongbin: it's not just templates all shell scripts need to be consolidated as well
16:54:13 <thomasem> ^^ was especially useful for the RHEL usecase
16:54:23 <hongbin> eghobo: yes
16:54:25 <thomasem> Since that's a licensed thing and all of that
16:54:33 <Kennan> coreyob hongbin perhaps we can limits some popular os ? to archieve a common goal ?
16:54:58 <Kennan> like systemd related and upstart related OS (since shell care about it ?)
16:55:07 <hongbin> Kennan: As long as the OSs is more than one. I am happy
16:55:17 <eghobo> also it's not about popularity, it's about stability
16:55:18 <coreyob> I'm not particular to which os distro we choose as long as there is only 1 we have to maintain a gate for
16:55:48 <eghobo> do we really want test/catch all Docker/Kub bugs at Atomic
16:56:13 <coreyob> i don't want to have to develop features for 6 coe+distro combinations. I'd rather do just 3 (1 per COE)
16:56:55 <hongbin> coreyob: We could have two distro for k8s
16:57:05 <hongbin> coreyob: since it is a popular COE
16:57:50 <Tango> So it seems we have 2 decision points:  (1) proceed on a design for the plugin mechanism, (2) which OS 's to focus on as reference implementation for each COE
16:57:58 <hongbin> coreyob: We don't have to populate multi-distro to other COEs, if we cannot find a volunteer to maintain
16:58:01 <coreyob> if you think we should do coreos for kubernetes I'm fine with dropping atomic instead. I think it'd be easier to have 2 base images (atomic and ubuntu) instead of 3, but I'd much rather have 1 per coe
16:58:55 <Tango> If we have developers willing to support, it should be OK
16:59:00 <adrian_otto> ok, let's plan to follow up on this topic again
16:59:12 <adrian_otto> as we have reached the end of our scheduled meeting time for today
16:59:13 <coreyob> so hongbin are you ok with atomic+swarm, coreos+kubernetes, and ubuntu+mesos as the reference implementations?
16:59:15 <Kennan> yes need further discsussion about it
16:59:31 <adrian_otto> we can also continue that ML discussion
16:59:59 <hongbin> coreyob: kuberntes should have an additonal distro
17:00:07 <adrian_otto> our next team meeting will be on Tuesday 2016-03-08 at 1600 UTC. Thanks for attending everyone.
17:00:08 <Kennan> adrian_otto last question, what's the cut date for M relase for Magnum ?
17:00:17 <juggler> thanks adrian
17:00:21 <adrian_otto> Kennan: will follow up in #openstack-containers
17:00:23 <juggler> thanks all!
17:00:31 <adrian_otto> #endmeeting