17:00:44 #startmeeting containers 17:00:45 Meeting started Thu Jun 28 17:00:44 2018 UTC and is due to finish in 60 minutes. The chair is strigazi. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:00:46 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 17:00:48 The meeting name has been set to 'containers' 17:00:53 #topic Roll Call 17:01:00 o/ 17:01:00 o/ 17:01:14 o/ 17:01:18 hello 17:01:22 o/ 17:01:38 \o 17:02:26 #topic Review Action Items 17:02:58 strigazi to summarize the meeting in a ML mail DONE 17:04:16 #link http://lists.openstack.org/pipermail/openstack-dev/2018-June/131769.html 17:05:08 I had two more, to create stories but I wanted to confirm before creating noise in the already heavy storyboard 17:05:34 We can discuss them in: 17:05:45 #topic Blueprints/Bugs/Ideas 17:06:15 The first one is pretty clear and it is about moving kubelet/proxy on all nodes. 17:06:46 s/moving/having ? 17:06:47 We had a quick chat with imdigitaljim. I propose to run the minion configuration on all nodes 17:06:59 yes s/moving/having/ 17:07:09 im gonna post a refactor draft here soon on the k8s bootstrapping files, please discuss changes on it. =) if we like it ill continue forward and validate/enhance it. 17:07:30 quick fix minion configuration sounds good 17:07:40 longterm needs to be cleaner 17:08:07 what does quickfix mean? 17:08:19 Which changes? 17:08:46 i would like to postpone the changing to Stein 17:08:50 not in Rocky 17:09:10 thats fine, its just for visibility, doesnt have to be accepted 17:09:11 flwang1: So only small patches? 17:09:40 it's milestone 3 now 17:09:55 i'm not sure if we have enough time to test every corner 17:10:30 ok 17:11:10 adding the minion configuration to master should work though, lets try it out 17:12:40 i just wanna to see a stable/rocky 17:12:48 The current situation that we configure kubelet in the master is a bit comlplicated. If the change is not big we can go for it 17:13:11 besides, k8s 1.11 just released, we may need to bump the k8s version in rocky 17:13:43 flwang1: this is irrelevant 17:13:59 strigazi: if there is a patch in next 2 weeks, i'm happy to review 17:14:24 changing the kube verision is easy and we can't make decisions based on that 17:14:38 yep, 1.11 is not releated, just thinking a loud to avoid i forgot 17:14:39 we did that in the past, config should be generic enough 17:15:37 ok, let's see a patch and we decide in the coming weeks. 17:15:38 Jul 23 - Jul 27R-5Rocky-3 milestone 17:15:49 two weeks 17:15:50 we have about 4 weeks for R-3 17:16:22 or for the worst case, if we can have a patch in good shape by the end of july, i'm happy 17:16:42 The next change would be upgrading the heat templates to pike 17:17:47 how we should make this decision? 17:17:59 flwang1: imdigitaljim what heat version are you running? 17:18:20 strigazi: we will be on queens shortly 17:18:33 should we fork to _v2 drivers for this change? 17:19:07 strigazi: that would be better to physically isolate the changes 17:19:11 so yes 17:19:13 we're still using the juno ones shipped with magnum 17:19:37 a v2 would be good for stability/going forward as well 17:20:10 I'm in favor of v2 but keep the scripts common, thoughts? 17:20:59 yeah that sounds resaonable 17:21:05 scripts is fine 17:21:24 ok then 17:21:25 and if we do that, i would suggest us resondiering the folder structure 17:21:45 flwang1: what do you mean? 17:22:02 not k8s_fedora_atomic_v2? 17:23:07 i'm happy with k8s_fedora_atomic_v2 17:23:26 ok, one moment 17:23:31 would moving to a self hosted control plane be possible in a v2 driver? 17:23:48 ^ 17:24:05 i'm talking about magnum/drivers/common/templates/kubernetes/fragments 17:24:17 i'm a bit weird for me though i haven't got a better idea yet 17:24:26 s/i'm/it's 17:24:31 if you're not familiar with self-hosted 17:24:31 https://coreos.com/blog/self-hosted-kubernetes.html 17:24:38 heres a decent doc to explain 17:24:46 kubernetes is moving towards self-hosted model 17:25:15 but in other words your control plane containers run in k8s instead of as systemd 17:25:16 #link https://storyboard.openstack.org/#!/story/2002750 17:25:27 just the kubelet would be systemd 17:25:57 we could do that, but from experience the benefit is not great 17:26:02 imdigitaljim: we did discussed that way 17:26:40 and we even want to drop master nodes from user's tenant, no mater it's in vm or container 17:27:01 for my part i've enjoyed being able to interact with all components of the control plane via the API, much in the same patern the project encourages elsewhere for self-managing resources 17:27:17 imdigitaljim: and i agree with strigazi, we need to understand the pros and cons 17:27:28 colin-: I agree on it, it is much easier 17:27:29 but 17:28:17 streamlining the process we deliver kubelet is more important and if we do it for kubelet we can do it for all components 17:28:22 control plane updates in place, improved HA, better introspection on the cluster, 17:28:30 ok 17:28:58 control plane access through k8s API 17:29:04 I'm sure a doc could be put together to help weigh the idea 17:29:16 #link https://coreos.com/blog/self-hosted-kubernetes.html 17:29:31 imdigitaljim: colin- cbrumm we had this actually before. 17:30:07 The reason we shifted from was that the way to deliver the same software kubernetes whose being fragmented. 17:30:16 The reason we shifted from was that the way to deliver the same software kubernetes was fragmented. 17:31:27 I'm not against it in principle, but at the moment it would complicate things 17:31:32 ok, I just figured that if a v2 was being considered it might be a good time to bring the control plane in line with kubernetes current recommendations. Use kubeadm etc 17:32:06 but it's not small consideration for sure 17:32:18 those kinds of things really need specs 17:32:19 self-hosted kubernetes would be certainly done with kubeamd. 17:32:21 i think it could cut our bootstrapping code into about 1/4 of what it is 17:32:23 we can't make decision here 17:32:45 flwang1 agreed 17:32:51 ^ 17:34:03 so, we need a detailed proposal and target it to S 17:34:15 that sounds good 17:35:25 I can't commit that we'll get to that immediately, but if we can't then it's only our fault if it slips 17:35:53 #link https://storyboard.openstack.org/#!/story/2002751 17:36:13 thanks strigazi 17:36:41 Jim Bach proposed openstack/magnum master: Kubernetes Bootrapping IDEAS (not intended as PR) https://review.openstack.org/578945 17:38:15 next, I have one thing we need for Rocky and discovered this week 17:39:15 While doing some benchmarks with spark and kubernetes. we fould out that the way magnum is interacting with heat, actually hurts heat a lot. 17:39:24 oh? 17:39:28 what does it do? 17:39:48 In magnum, we don't have locking between the conductor 17:40:18 and when the conductor(s) try to sync the status with heat they do a stack get. 17:40:53 When a stack is CREATE|UDPATE_IN_PROGRESS we only need the status 17:41:08 not the outputs of the stack or any metadata. 17:41:27 Yeah that sounds like overkill 17:41:33 For large stacks, evaluating the stack outputs is expensive 17:41:37 have seen similar patterns in other projects 17:42:01 is there a better query? 17:42:07 and when many processes hit the heat api and queue requestsm rabbit gets overloaded and eventually heat goes down 17:42:25 The better query is heat list and filter of one id 17:42:25 that sounds like a great patch to get done 17:42:26 strigazi: can we use stack list instead of get? 17:42:35 and then matching the id in magnum 17:42:54 filters={"id": some_stack_id} 17:43:08 nice 17:43:24 strigazi: can the fileters accept a list of id? 17:43:29 a list takes a second, a get takes a minute for a cluster with 200 nodes 17:43:38 yeap tested it 17:43:41 but anyway, we should be able to do that 17:43:52 strigazi: cool, have a patch already? 17:43:57 i'm keen to review it 17:44:02 I'll push a patch I have it almost ready 17:44:05 another issue with heat/magnum cleanup. if the cloud-controller-manager allocates something like a loadbalancer and the cluster is deleted.. the LB is dangling and not cleaned up because heat nor magnum know about it as it stands 17:44:19 There is a patch in review 17:44:36 Can you dig it? I can send it you too 17:44:46 Can you dig it? I can send it to you too 17:45:18 strigazi: the heat issue? sure 17:45:30 flwang1: the lbaas clean up 17:45:50 for the heat issue I'll push it 17:45:55 in our testing, we can see the lb is deleted successfully 17:46:08 is this allocated through cloud-controller-manager? 17:46:11 for lbass (created by the cloud provider) 17:46:13 without cloud-controller-manager 17:46:14 using kubernetes 17:46:20 there is a patch 17:46:20 yeah the normal LB's allocated by heat are fine 17:46:29 give me a sec 17:46:35 we haven't tried that way 17:46:48 i'm happy to test and fix it 17:46:50 in other words, kubernetes users need to use load balancers 17:46:57 kubernetes talks to neutron to get a LB 17:46:58 imdigitaljim: i see 17:47:07 it's for service LB not masters LB 17:47:08 #link https://review.openstack.org/#/c/497144/ 17:47:11 neutron/octavia 17:47:14 correct 17:47:33 im not sure what all we can/should do to rectify it, we might want to setup a BP for that 17:47:47 strigazi: thanks, will dig it 17:48:11 it's the linked one, have a look 17:48:16 oh yes 17:48:26 strigazi: ill review this https://review.openstack.org/#/c/497144/3/magnum/drivers/heat/driver.py 17:48:38 err https://review.openstack.org/#/c/497144/ 17:48:48 imdigitaljim: thanks 17:48:55 and quickly one more 17:48:58 great callout for it =) 17:49:10 The keypait issue: 17:49:13 The keypair issue: 17:49:47 #link https://storyboard.openstack.org/#!/story/2002648 17:50:19 if you offer magnum to others and they want you to do things for them it's pretty important 17:50:37 The idea is to create a keypait per stack 17:50:40 The idea is to create a keypair per stack 17:51:36 So when a client wants you to scale or fix their stack with stack update you should be able to do it. 17:52:25 I'll push a patch for it as well. 17:53:06 Apologies for this meeting folks, we can try to make the next one more structured. 17:54:05 no complaints here :) 17:54:10 its complaints here as well 17:54:17 s/its/no 17:54:51 :) 17:55:24 Do tou need to bring up anything else? 17:56:03 i need testing from imdigitaljim for https://review.openstack.org/576029 17:56:23 or any other bless for this one 17:56:26 Nope, I think we've covered everything 17:56:59 flwang: im on it by the end of this week 17:57:11 i should post about it early next week 17:57:13 imdigitaljim: cool, thanks 17:57:28 Thank you guys for hosting this again. It's great to get some solid time with you both 17:57:51 Let's meet in 6 days and 23 hours again :) 17:57:59 sounds like a plan 17:58:33 good to discuss with you guys, very productive 17:58:58 cbrumm: imdigitaljim colin- flwang1 jslater thanks for joining, see you soon 17:59:09 ttyl 17:59:14 ttyl 17:59:17 Have a good evening/day 17:59:28 cheers! 17:59:32 #endmeeting