21:01:02 <strigazi> #startmeeting containers 21:01:03 <openstack> Meeting started Tue Oct 16 21:01:02 2018 UTC and is due to finish in 60 minutes. The chair is strigazi. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:01:04 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:01:07 <openstack> The meeting name has been set to 'containers' 21:01:30 <imdigitaljim> o/ 21:01:33 <strigazi> #topic Roll Call 21:01:39 <cbrumm__> o/ 21:01:42 <imdigitaljim> o/ 21:01:43 <strigazi> o/ 21:01:52 <eandersson> \o/ 21:02:10 <flwang> \o/ 21:02:38 <strigazi> #topic Announcements 21:03:21 <strigazi> thanks eandersson for fixing the magnum gate https://review.openstack.org/#/c/610740/ 21:03:27 <eandersson> anytime 21:03:41 <strigazi> Do you know why it used work or why ot broke? 21:03:48 <strigazi> Do you know why it used work or why it broke? 21:03:59 <eandersson> I was looking at some recent changes with oslo.messaging that affects the defaults 21:04:23 <eandersson> Which I think caused it with 9.0.1 21:04:25 <flwang> that's bad 21:04:27 <strigazi> o.m version bump then, thanks 21:05:10 <eandersson> https://github.com/openstack/requirements/commit/29f8a80a71111ea59141c1aeab94bae9cf3f30a9#diff-0bdd949ed8a7fdd4f95240bd951779c8 21:05:16 <flwang> introducing changes in base components and breaking everybody 21:05:31 <strigazi> flwang: it is better than it used to be I think :) 21:05:44 <flwang> better for what? 21:05:54 <strigazi> less breaks/ time unit 21:06:03 <flwang> haha 21:06:11 <eandersson> Yea - it's usually 3rd party libraries now that break things 21:06:13 <eandersson> like evenlet 21:06:15 <canori02> O/ 21:06:20 <flwang> it's true, but i think it's because its losing contributors :D 21:07:04 <strigazi> canori02: o/ 21:07:18 <strigazi> #topic Stories/Tasks 21:08:39 <strigazi> I don't think I have anything from last week to discuss, flwang do you want to discuss the CT rename? 21:09:03 <imdigitaljim> flwang: i found a solution to making a single LB scenario 21:09:15 <imdigitaljim> huge cut in resource usage is nice :) 21:09:54 <strigazi> imdigitaljim: use a single LB for many services? 21:10:02 <imdigitaljim> for etcd/api 21:10:12 <strigazi> oh, right 21:10:27 <strigazi> imdigitaljim: is there a patch? 21:10:33 <imdigitaljim> i think our plan is to submit a v2 driver that is parity with the first one's features 21:10:44 <imdigitaljim> and huge improvements all around 21:10:57 <imdigitaljim> cluster speed/feature rich/compatibility 21:11:44 <flwang> imdigitaljim: cool, we can discuss the single lb later 21:11:50 <imdigitaljim> yeah definitely 21:12:03 <imdigitaljim> i can push up a WIP for you guys soon if you want to see where things are 21:12:03 <flwang> strigazi: yep, i'd like to know you guys opinions about CT rename 21:12:09 <flwang> imdigitaljim: go for it 21:12:41 <flwang> as you know, Catalyst Cloud just released magnum on our prod 21:12:50 <cbrumm__> Nice! 21:12:55 <cbrumm__> congrats 21:12:56 <flwang> and we're starting to get the pain of change/update public templates 21:13:04 <flwang> cbrumm__: hah, thanks 21:13:32 <colin-> that's great, how did the deployment go? 21:13:59 <flwang> colin-: we got some issues, but so far, most of the things are ok 21:14:27 <flwang> a big lesson learn is don't use docker_storage_size with overlay/overlay2 21:14:42 <imdigitaljim> we are considering a redesign (part of the v2) 21:14:46 <imdigitaljim> that decouples that entirely 21:14:50 <flwang> and magnum/heat-container-agent does have the multi region bug 21:15:01 <flwang> imdigitaljim: nice 21:15:04 <colin-> that sounds rough, glad it worked out though 21:15:12 <eandersson> what is the issue with overlay? 21:15:14 <imdigitaljim> so you dont have to worry about kube-tag or other quick changing data 21:15:22 <imdigitaljim> ^ 21:15:28 <imdigitaljim> +1 eandersson 21:15:34 <imdigitaljim> what is the issues with overlay2? 21:15:35 <flwang> imdigitaljim: that's interesting 21:16:24 <flwang> with overlay2 and docker_volume_size(external volume for docker), we have seen some issue with docker, which cause pods can't be created 21:16:31 <flwang> it's not always repeatable 21:16:56 <flwang> we just realized it yesterday with strigazi's help, still need testing though 21:16:59 <flwang> but just FYI 21:17:17 <flwang> btw, our storage backend is ceph 21:17:18 <imdigitaljim> cgroup issue? 21:17:25 <eandersson> Interesting - feel free to share more when you have had time to dig deeper 21:17:32 <flwang> we're using cgroupfs already 21:17:35 <colin-> which cgroup driver? just curious 21:17:38 <colin-> ok 21:18:23 <flwang> I still don't know the root cause, may need help from storage expert or docker expert 21:18:33 <flwang> but i will keep you guys posted if there is any progress 21:18:43 <eandersson> Sounds good - thanks 21:18:48 <tobias-urdin> flwang: sorry for shiming in, regarding the mentioned multi-region bug in heat-container-agent is that addressed or is there a bug tracked somewhere with more info? 21:18:54 <tobias-urdin> we are planning to deploy magnum in another region soon 21:19:49 <tobias-urdin> eandersson: thanks for the review 21:19:56 <eandersson> anytime 21:20:37 <strigazi> tobias-urdin: it is addressed, one moment 21:21:27 <strigazi> tobias-urdin: https://review.openstack.org/#/c/584215/ 21:21:56 <strigazi> tobias-urdin: included here: https://hub.docker.com/r/openstackmagnum/heat-container-agent/tags/ rocky-stable 21:21:57 <flwang> it has been fixed 21:22:20 <strigazi> tobias-urdin: this patch is pending: https://review.openstack.org/#/c/609579/ 21:22:53 <flwang> tobias-urdin: you also need you update your existing deployment 21:23:00 <flwang> i mean Mangum 21:23:06 <flwang> for existing cluster, should be fine 21:23:09 <strigazi> flwang: and heat to queens, right? 21:23:23 <flwang> strigazi: yes, at least Queens 21:23:54 <flwang> I fixed it in Rocky, and I think i have backported all necessary changes into queens 21:24:10 <tobias-urdin> ok thanks! so if I understand correctly when 609579 is merged and backported to stable/rocky we are all good? 21:24:21 <strigazi> tobias-urdin: yes 21:24:33 <tobias-urdin> got it, thanks 21:25:59 <flwang> all good for Magnum 21:26:14 <flwang> tobias-urdin: also need to upgrade Heat 21:27:24 <tobias-urdin> ack 21:28:18 <flwang> tobias-urdin: me and strigazi will have a topic at Berlin to share what we know ;) 21:31:40 <tobias-urdin> flwang: cool, i'll keep an close eye out for the video on that, unfortunately won't attend berlin :( 21:32:06 <flwang> no problem 21:32:20 <flwang> strigazi: i'm good now 21:32:33 <flwang> strigazi: imdigitaljim: anything else you want to discuss? 21:33:27 <imdigitaljim> just throwing it out, were working on inplace upgrades currently 21:33:35 <imdigitaljim> strigazi: has some work thats very close 21:33:39 <imdigitaljim> to what we started with 21:33:59 <imdigitaljim> we're adding new mechanisms but i should also be adding it to the v2 driver 21:34:21 <imdigitaljim> but id like to maybe discuss some basic approaches next time 21:34:22 <strigazi> imdigitaljim: will you push upstream something for v1? 21:34:44 <imdigitaljim> probably not for v1 since its too far back from the updates 21:34:54 <imdigitaljim> it perhaps can be backported 21:35:59 <imdigitaljim> we *are* keeping features of the v1 driver introduced on the v2 21:36:11 <imdigitaljim> so when were done with v2 it should be a minimum everything v1 has 21:36:46 <imdigitaljim> people can still choose v1 for their needs/legacy but because its a driver they can also choose v2 21:36:57 <imdigitaljim> :P 21:37:00 <strigazi> I think we should have a spec or another document for these changes 21:37:14 <strigazi> makes sense? 21:37:39 <imdigitaljim> im sure i can come up with something :) 21:38:18 <strigazi> maybe a spec to sync with nodegroups 21:38:30 <imdigitaljim> yeah i plan to watch your nodegroups changes and improve/port that 21:38:53 <imdigitaljim> (were not using it so i might need you to test it) 21:39:15 <strigazi> sure 21:39:23 <imdigitaljim> i will still be pushing a few for common 21:39:33 <imdigitaljim> that can help any/all drivers 21:40:11 <imdigitaljim> flwang: btw congrats on the prod deploy! 21:41:03 <flwang> imdigitaljim: thanks, it wouldn't happen without you guys help 21:41:12 <flwang> strigazi: +1 for spec for v2 21:41:32 <flwang> I assume it a big change, so better to have a spec 21:41:51 <flwang> or at least doing the code and spec in parallel, so that reviewer can understand the design correctly 21:41:52 <imdigitaljim> yeah totally understandable 21:42:11 <imdigitaljim> some of the goals of the v2 was improved readability as well 21:42:33 <flwang> imdigitaljim: quick question, does the v2 involve any API change? 21:42:39 <imdigitaljim> no 21:42:42 <flwang> nice 21:42:57 <imdigitaljim> core magnum is mostly untouched 21:43:33 <flwang> it's great 21:44:46 <imdigitaljim> there are a some constraints heat >= queens/rocky?, octavia (if used) >= rocky, kubernetes >= v1.11.X 21:45:11 <imdigitaljim> i think heat can be pike if not using octavia though 21:45:50 <imdigitaljim> octavia >= queens* 21:46:01 <strigazi> queens is a reasonable requirement 21:49:34 <strigazi> Anything else for the meeting? 21:49:51 <flwang> strigazi: i'm good 21:50:10 <strigazi> others? 21:50:40 <tobias-urdin> the issue we talked about earlier today is fixed in https://review.openstack.org/#/c/611097/ 21:51:06 <tobias-urdin> nothing fancy :) 21:51:18 <strigazi> thanks +2 21:51:33 <tobias-urdin> should be safe for backports as well 21:52:49 <strigazi> thanks everyone for joining the meeting, see you 21:53:10 <strigazi> #endmeeting