15:00:32 #startmeeting openstack-helm 15:00:33 Meeting started Tue Nov 28 15:00:32 2017 UTC and is due to finish in 60 minutes. The chair is mattmceuen. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:34 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:37 The meeting name has been set to 'openstack_helm' 15:00:37 o/ 15:00:38 #topic rollcall 15:00:38 o/ 15:00:44 y'all are fast this morning! 15:00:57 I've never seen a group of folks so eager for a meeting ;-) 15:01:12 * srwilkers blinks 15:01:20 hello 15:01:30 hi korzen! 15:01:53 Agenda is here: https://etherpad.openstack.org/p/openstack-helm-meeting-2017-11-28 15:02:05 We'll give folks a couple minutes to filter in and add anything to the agenda 15:02:28 We'll be doing something slightly different today 15:03:12 This will be the first of a monthly meeting focused on CICD and gating-related topics, since those are so critical to QA and require quite a bit of coordination across folks 15:03:21 o/ 15:03:32 And also to understand different CICD use cases, and help share ideas and best practices 15:03:59 * jayahn yeah~ 15:05:10 #topic Round table, overview of CI in OSH 15:05:22 o/ first timer - be gentle.... 15:05:22 o/ 15:05:31 Welcome MarkBaker :) 15:05:45 MarkBaker: o/ 15:06:10 portdirect, want to kick this off with a status / overview of the new zuul3 gate work? 15:06:55 hi 15:07:08 sorry, got distracted 15:07:23 s the zuulv3 work has been taking two prongs 15:07:33 1) splitting of the charts into diff repos 15:07:45 2) converting to ansible (and voting!) gates 15:08:11 this has allowed us to treat-osh infra a bit more roughly than we would have the main repo 15:08:21 and I hope its in quite a good place now 15:08:36 we have some people working on the docs for it now 15:08:52 but it should be a lot more robust than the old kubeadm-aio hack 15:09:16 we have used this as the basis for setting up k8s for osh 15:09:31 and then moved the legacy scripts to run under this cluster 15:10:07 with the next step/stage being importing the 'chart-runner' from osh-infra to allow us to launch all services via ainsible 15:10:20 and check upgrades function as expected 15:10:41 once this is done - I think the biggest item remaining will be to test cluster restarts in the gate 15:10:55 as currently this is the biggest weakness of osh 15:11:05 upgrade gates are going to be a great thing. 15:11:16 agreed 15:11:43 portdirect, docs referred to are at https://docs.openstack.org/openstack-helm/latest/ ? 15:11:44 jayahn: have you peeps been testing node/cluster restarts in your pipelines? 15:11:54 "test cluster restart" means restarting everything? 15:11:55 MarkBaker: these are the current docs 15:11:59 so to summarize a little of that -- the openstack-helm-infra gates set up k8s platform, and today the openstack-helm gates call the legacy scripts to deploy the osh charts 15:12:23 MarkBaker: this is the *very early* wip on the new docs: https://review.openstack.org/#/c/523173/ 15:12:32 but in the near future the openstack-helm gates will be continue to be enhanced 15:12:42 portdirect, thx 15:12:55 portdirect: not in pipeline, but definitly testing it manually to see what problems are there, how to solve it. 15:14:34 before we move on to some other CI topics -- any questions on the existing state of CI in the OSH projects themselves? 15:14:57 just need to learn new CI. 15:14:59 :) 15:15:36 "make dev-deploy" is a neat new feature to learn in osh-infra ;-) 15:15:58 #topic container images 15:16:01 jayahn: i'd really appreate you guys having a look through these: https://github.com/openstack/openstack-helm-infra/tree/master/tools/gate/playbooks/deploy-helm-packages/tasks , https://github.com/openstack/openstack-helm-infra/blob/master/tools/gate/chart-deploys/default.yaml 15:16:39 as they are what I'd like to bring into osh itself for launching/managing charts 15:16:43 SamYaple: you about? 15:16:54 robertchoi80: need to check things portdirect throw us. :) 15:16:59 portdirect: yes 15:17:05 So on images, we now have a new option :) LOCI 15:17:09 okay 15:17:11 Kaspars has started the Jenkinsfile for loci builds - https://review.gerrithub.io/#/c/388686/ 15:17:14 For some of the newer folks to the team -- one of the design goals of OSH is to be able to support multiple container image sets 15:17:35 E.g. Kolla and now LOCI. OSH aims to be largely agnostic. 15:17:40 I am running these internally, and our CI/CD team lead by StaceyF is building out infra for internal builds 15:18:01 okay. gerrithub. 15:18:02 It would be great if we could start cross gating with LOCI 15:18:12 SamYaple: how does that sound? :D 15:18:22 sounds good for sure 15:18:26 portdirect: StaceyF: does gerrithub needs any permission to see logs from ci? 15:18:33 Just a github account 15:18:38 with zuulv3 LOCI can just reconsume OSH jobs 15:19:22 when it fails, it throw us http://10.24.20.18:30085...... which i cannot see 15:19:23 Once we have that in place, I'd like to consider changing the default image to LOCI, and provide over-rides/gates for kolla as well 15:20:01 The Jenkinsfile will be something that you would use in your internal Jenkins and you will just need to update variables to publish them wherever needed. 15:21:01 staceyF: that sounds good. 15:21:26 We will be allowing access to our LOCI images via Artifactory 15:21:39 Should happen this week 15:22:21 Jayahn we'll update the 30085 to our nexus server 15:24:28 Team - I'm going to go out of order on the Agenda -- we have a lot to get through and there a few things I want to make sure we get to today 15:25:07 #topic Cookiemonster from skt 15:25:18 (but hopefully we'll be able to get to everything today) 15:25:28 we got promoted. :) 15:25:28 jayahn, want to give us an overview of cookiemonster? 15:25:32 :-D 15:25:36 robertchoi80 will do 15:26:06 robertchoi80: is our ci guy 15:26:16 all right. I'll briefly explain that 15:26:28 robertchoi80 is StaceyF's new best friend 15:26:37 yes! 15:26:42 yes we met in Sydney :) 15:28:03 it's ha test tool that terminates varios k8s resources (such as deployment or daemonset) on regular interval and duration. 15:28:18 it's similar to chaosmonkey from netflix. 15:28:29 for eg, 15:29:45 it tries to pick randomly one pod from the deployed resources, and if the number of replica is more than 1, it kills one of them. 15:30:03 can you give us a link to cookiemonster -- it's not the first hit on google :) 15:30:20 https://github.com/sktelecom-oslab/cookiemonster 15:30:47 we just uploaded our source code, documentation and more example will come 15:30:57 robertchoi80: what are the primary differences between it and https://github.com/asobti/kube-monkey, https://github.com/linki/chaoskube ? 15:30:59 awesome 15:31:37 very cool 15:32:39 portdirect: sorry, I am not the one who wrote that tool, so I can't answer that right now. I'll give you the answer later by email 15:32:52 that tool = cookiemonster 15:33:05 portdirect: we probably need to do a quick comparison to answer your question. several month ago when we decide to write our own code, these tools were not perfect, so we decided to quickly make our own. 15:33:42 def interested in that comparison. We'd been wanting to use kubemonkey or chaoskube, but haven't gotten too far with that yet 15:33:44 and had master plan to break anything via cookiemonster. :) 15:34:20 okay. we will update with some comparison. 15:34:40 jayahn: sounds good - I think i remember you mentioning being able to target specific applications at the PTG but a simple table would be a really great addition to the readme 15:34:53 we don't have a cookiemonster chart yet, but will definitly make one soon and put on openstack-helm-addons for anyone to use 15:35:03 Thanks robertchoi80 & jayahn -- looks very promising from the readme, and would love to learn more 15:35:08 jayahn: that would be awesome! 15:35:23 * portdirect is lazy and likes cookies 15:35:39 moving on: 15:35:43 #topic Where do we want to go next, how much can we share? 15:35:59 "where do we go next" -- always a good question. 15:36:01 * jayahn i got a biggest difference. cookiemonster is way cooler name than others 15:36:19 * mattmceuen agrees 100% 15:36:39 jayahn: +++++ 15:36:49 however, kube-monkey wont eat my cat 15:37:16 First, from an AT&T perspective, like StaceyF showed with our CI - we want to share everything but the configuration, or anything that's irrelevant outside of AT&T 15:37:48 We will share our Jenkins configurations of any plugins, just no passwords :) 15:38:07 so everyone please feel free to either contribute, or take whatever looks applicable and adapt for your use! 15:38:17 So - I covered where i'd like to see our community gates go a bit above, but I think we also need to work out a set of critera for plugins/alternative backends 15:38:39 eg OpenContrail and friends 15:38:49 anyone from Juniper here? 15:39:18 yes, this is Krishna from Juniper 15:39:29 Welcome Krishna! 15:39:37 Krishna: o/ 15:39:40 Hello Matt 15:40:27 have you peeps worked out what infra will be needed to test opencontrail? can we do it with simplegateway, to provide parity with the current gate tests? 15:41:04 and would this be able to run in the community provided infra? or would a third party gate be required? 15:41:19 i think youve had a few people (andrey?) looking into this 15:42:06 We have done work with the simple GW to provide some parity and should be available to run in the community infra as stated. 15:42:57 Andrey Pavlov has done some tests and there are some pending items which we need to hash out with Andrey. 15:43:15 that is great to hear. do you have that work somewhere for us to have a look at? Is it ready for that? 15:43:39 Krishna: sounds good - could we make documentation a priority, which will help us help you guys get there 15:44:33 from skt perspective, we can share jenkins configuraiton, jenkinsfiles, along with some whitepaper on how we do it. 15:44:33 However, I personally think we (skt) need to learn osh-gate more seriously. 15:44:48 Matt: yes, we will send an emails with the compeleted work 15:44:58 Awesome, thanks Krishna. 15:45:07 Krishna: a PS is always good too ;) 15:45:09 PortDirect: we will make the documentation a priority and post it soon. 15:45:32 to figure out (in long-run), how close/similar we would like to do with osh community gate. 15:45:36 Our goal w.r.t. OpenContrail gating is 1) to make it a first class citizen of OSH QA gates 15:45:55 portdirect: regarding the engaging the PS, we are in discussions for that. 15:46:19 2) to move toward OpenContrail installation on OSH being a first class installation mechanism for k8s that is well-documented and simple to use 15:48:11 Thanks, Krishna. I'm also planning to come to the OpenContrail meeting @ KubeCon, and would like to discuss OSH there too 15:48:56 jayahn -- yeah, please let us know your feedback on the new OSH gating 15:49:43 jayahn -- do you have gating for sona yet? Is that open, nd something we could learn from as well? May be good reusable principles 15:50:13 matt: that's still in our to-do list. 15:50:24 ok cool 15:50:28 should be done in next 2 weeks 15:50:29 just did not have time. :( 15:50:44 we have a release cut this week. super busy. :) 15:50:46 nice! 2 weeks ain't bad 15:51:36 fyi, it will be provided as 3rd party gating. 15:51:45 fanstastic 15:51:57 alrighty -- catching up on topics I skipped :D 15:52:02 #topic Heat Templates 15:52:12 StaceyF take it away 15:52:28 jayahn: thats great - I'd really like to have SONA up there as a 1st class citizen as well - bizarrely it (via CORD) was one of the projects that got me into this.... 15:53:31 I had already mentioned it earlier, we'll be putting up a patchset with all of the heat templates that are used in our Jenkinsfile. 15:53:57 awesome, thanks, I missed that in the flurry 15:54:30 superseed though - I don't think I missed that 15:54:39 #topic SuperSeed 15:55:38 The superseed job is triggered via a change to a Jenkinsfile or seed.groovy file. It updates any pipeline that has a change. We use this in our Development environment for Jenkins to verify that the seed and Jenkinsfile work before putting it into our Prod Jenkins 15:55:59 https://github.com/att-comdev/cicd/tree/master/cicd/SuperSeed 15:56:20 This is the job that will be updated to allow logs accessible by everyone 15:58:26 Thanks StaceyF. Any q's on this topic? 15:58:36 The goal will be to have this one job be in a Jenkins chart that you can kick it off and it builds all of your pipelines for osh 15:59:18 We're about out of time, all -- clearly this focused meeting was a good idea. Do you all want to 1) save the remaining few topics for next meeting, or 2) till the next CI meeting in a month? 15:59:25 I'm leaning toward next meeting 15:59:45 can use some of next meeting time. 15:59:57 cool beans - I'll copy them over 16:00:15 Thanks, everyone! see you in the OSH chat! 16:00:18 #endmeeting