21:04:18 #startmeeting containers 21:04:18 Meeting started Tue Apr 9 21:04:18 2019 UTC and is due to finish in 60 minutes. The chair is strigazi. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:04:19 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:04:21 The meeting name has been set to 'containers' 21:04:21 #topic Roll Call 21:04:30 o/ 21:04:33 hello 21:04:36 o/ 21:06:10 #topic Stories/Tasks 21:06:11 o/ 21:06:25 o/ 21:06:46 Last week I attempted to upgrade the default version of k8s to 1.14.0 but calico v2 wasn't passing 21:07:01 wasn't passing the conformance test 21:07:13 I have the patch and results here: 21:07:40 https://review.openstack.org/#/c/649609/ 21:08:01 flwang: suggest that the latest calico, may work. I'll give it a go 21:08:23 we use the latest calico 21:08:29 Hey Guys. Whats the latest version of kubernetes I can use on queens version of magnum (6.3.0). I tried with kube_tag=1.11.1-5 and 1.12 and both failed to build. The default 1.9.3 builds fine. 21:08:37 imdigitaljim: i know, that is why I'm not asking :) 21:08:50 ah :D 21:09:05 I mean kube_tag=v1.11.1-5 21:09:05 conformance was passing as well 21:09:13 so you might be right 21:10:24 For upgrades, I did some modifications for the worker nodes and with the heat API works pretty well for worker and it validates the passed nodegroup. 21:10:38 Some more clean up and it will works with the API. 21:10:51 strigazi: https://kubernetes.io/docs/setup/version-skew-policy/ 21:11:03 have you seen that for upgrades? 21:11:16 specifically https://kubernetes.io/docs/setup/version-skew-policy/#supported-component-upgrade-order 21:11:32 o/ 21:11:35 The only missing part is the container registry on clusters 21:12:21 imdigitaljim: yes, but it doesn't enforce it 21:12:22 sorry i'm late, NZ just had a daylight saving 21:13:06 this madness with daylight will end soon, at least in the EU 21:13:36 strigazi: yep 21:13:54 strigazi: so are you still going to do the master upgrade in your existing patch? 21:13:59 yes 21:14:01 or you will propose another one? 21:14:07 this one 21:14:44 flwang: do you want to the 1.14.0, it is calico related 21:14:52 flwang: do you want to the 1.14.0 pathc, it is calico related 21:14:56 also 1.14.1 is out 21:15:38 want to (do)? 21:15:59 flwang: do you want to take the 1.14.0 patch, it is calico related 21:16:06 hehe, sure i can 21:16:42 but i'm busy on the auto scaling regression issue and the upgrade testing/review, is the v1.14.0 urgent for you? 21:17:08 not really really urgent 21:17:22 strigazi: ok, then i can take it, no problem 21:17:32 i said not :) 21:18:18 regarding the *possible* regression with the autoscale. I wasn't able to reproduce. Can you describe it in storyboard? 21:18:44 strigazi: sure, are you using devstack or stable/rocky? 21:18:49 devstack 21:19:00 and are you using the image from opentstackmagnum? 21:19:08 but a in a good machine :) 21:19:21 yes 21:19:34 are you using my patch or a home-made autoscaler yaml? 21:19:52 from the CA repo, not your patch 21:20:39 I don't think this is the issue https://github.com/kubernetes/autoscaler/issues/1870 21:20:43 my code also from ca repo but i'd like to understand the difference, and i think it is a corner case, but we need to figure it out 21:21:24 strigazi: not sure, and I also got a scale down issue which autoscaler and magnum/heat are using different format of UUID 21:21:27 ok, with your patch is it 100% reproducible? 21:21:57 i think it's reproduceible, but i don't think it's 100%, better give it a try by yourself 21:22:07 and that would be really appreciated 21:22:16 ok, where do you test? dsvm? 21:22:23 master branch? 21:22:29 master branch 21:22:32 ok 21:22:44 with all latest code, including the patch NG-5 21:23:03 ok 21:23:28 i will dig it today as well 21:23:40 back to your upgrade patch, did you see all my comments? 21:23:47 cool, I'll check gerrit tmr 21:24:36 now i can the minion upgrade works with those changed i mentioned in the patch, but in my testing, the master node will be rebuilt though i didn't change the image 21:25:38 I am lost in the comments, they are too many. what changes? 21:26:05 for the additional mounts it is fixed. 21:26:14 i suggest you review all my comments, because that took me a lot of time for testing 21:26:30 the additional mounts is for the minion side 21:26:35 i'm talking about the master 21:26:40 sure, I'll address them 21:27:11 so do you mean i shouldn't care about the master behaviour now since you haven't done it? 21:27:33 master is expected to fail atm. 21:27:59 strigazi: it's not "fail", it's being rebuilt 21:28:12 after rebuilt, master is using the new version of k8s 21:28:29 that is kind of a failure :) 21:28:44 I'll fix it 21:28:50 let me explain a bit 21:29:22 I know the issue, it is because of user data 21:29:23 after rebuilt, all components except kubelet will be back soon, and i have to restart kubelet to get it back 21:29:42 it's really like the issue we're seeting for autoscaler's master rebuilt 21:29:50 s/seeting/seeing 21:30:18 yeap, it is the issue we had with cluster_update some months ago and it was fixed 21:30:22 i just wanna highlight that to see if you have any idea 21:30:31 yeap, it is the same issue we had with cluster_update some months ago and we fixed it 21:30:45 which patch fixed it? 21:31:02 with the autoscaler testing, i'm using master 21:31:15 and i also rebased the upgrade patch locally for testing 21:31:23 so i'm wondering which patch you're talking about 21:31:28 no, I mean the cause is the same as in cluster_update in the past. 21:32:03 strigazi: so you mean you fixed it in your existing patch? 21:32:04 https://github.com/openstack/magnum/commit/3f773f1fd045a507c3962ae509fcd57352cdc9ae 21:32:07 no 21:32:28 flwang: let's take a step back. 21:32:53 The current patch for upgrades it is expected to "fail" for master. 21:33:17 The reason is "change of user_data of the vm" 21:33:17 i get that 21:33:36 This reason used to break cluster_update and it was fixed. 21:33:58 I don't know what breaks autoscale, I''ll check. 21:34:10 The reason is "change of user_data of the vm" --- we have to do same thing for master like we did for minion? 21:34:28 to put those scripts "into" heat-container-agent? 21:34:47 ok, i understand 21:34:49 in the current patch in gerrit I'll push a fix for master upgrade. 21:34:55 yes 21:34:58 strigazi: cool 21:36:07 based on my understanding of heat update, the master nodes being rebuilt is still caused by something changed for the master 21:36:16 correct 21:36:30 we just need to figure out what has been changed which slip out our eyes 21:36:47 cool, good to know we're on the same page 21:36:51 for upgrades yes, for the autoscaler to be checked. 21:37:16 dioguerra was doubting the new security group rules from master to nodes 21:37:44 but still failed after revert that one 21:38:01 dioguerra: can you give us more details? 21:38:03 it comes from Ricardo too and I also mentioned it, but I didn't have enough courage to insist 21:38:41 in the patches in gerrit I mentioned that it breaks the pattern we use for ingress 21:39:03 this is the remove of pors 80/443 21:39:41 the other port is ssh which change the default behaviour. 21:39:46 the other port is ssh which changed the default behaviour. 21:40:08 I mentioned this in gerrit as well. 21:40:26 if we confirmed the issue is caused by the security rules, i think we can revisit this part 21:41:33 as I mentione before, in clouds that don't have octavia (like ours) or even if they do, but users don't want to use it. 21:41:57 ingress works with a traefik or nginx or appscode/voyager 21:42:16 strigazi: i can see your point, we maybe able to introduce a config to let cloud provider to config those rules? 21:42:16 using ports 80/443 in the workers 21:43:19 For this, we can open when open when traefik or nginx is used or with another label 21:43:50 same for ssh 21:43:52 strigazi: yep, that would be better and easier 21:44:04 can be cloud wide or with labels 21:44:13 can be cloud/magnum-deployment wide or with labels 21:45:36 yep, we can discuss this later for more details 21:46:11 we can put additional details in storyboard 21:46:19 sure 21:47:55 what is the update on CRUD for nodegroups ttsiouts? 21:49:07 strigazi: ttsiouts: is the ng-6 the last one we need for NG? on server side 21:49:59 is there an NG-6 in gerrit? 21:50:13 https://review.openstack.org/#/c/647792/ 21:50:57 before the new driver, i think this is the last one 21:51:17 ok, good 21:51:32 and client 21:51:43 i'm reviewing the client now 21:51:57 brtknr: I am refactoring the scripts for the deployment of the cluster 21:52:02 I guess we need a microversion 21:52:13 strigazi: do you think we can start to merge upgrade api now? 21:52:16 brtknr: in heat side 21:52:48 flwang: we just need a check for master VS worker and we are good 21:53:42 strigazi: ok, cool, i have done the api, ref, and client, and it generally works with your functional patch 21:54:09 yeap 21:54:09 so as long as your master upgrade work submitted, we can start to do integration testing and get things done 21:54:49 strigazi: thank you for working on this, i know it's a hard one 21:54:51 ttsiouts: sounds good! let me know when its ready for testing :) 21:55:35 :) 21:55:53 just before closing, I want to make a shameless plug 21:56:07 if you use barbican, you may like this one: 21:56:20 https://techblog.web.cern.ch/techblog/post/helm-barbican-plugin/ 21:56:37 install barbican on k8s? 21:56:47 Ricardo wrote an excellent plugin 21:57:24 it can be easily added as a kubectl plugin 21:57:24 strigazi: ah, that's nice, so you still need to have barbican deployed already, right? 21:57:39 yes, you need the barbican API 21:57:51 and then just use barbican as the secret backend for k8s? 21:58:04 strigazi: I like that Kustomize is mentioned :) 21:58:12 that's cool, actually, we already have customer asking that 21:58:27 this plugin is for client side usage. 21:58:48 For KMS there is an implementation in the CPO repo 21:59:40 strigazi: cool 22:01:36 let's end the meeting? 22:02:39 Said once 22:02:51 said twice 22:02:56 i'm good 22:03:01 thanks for joining everyone 22:03:01 thanks strigazi 22:03:05 flwang: cheers 22:03:11 #endmeeting