15:00:16 #startmeeting openstack-helm 15:00:17 Meeting started Tue Jul 14 15:00:16 2020 UTC and is due to finish in 60 minutes. The chair is gagehugo. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:18 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:20 The meeting name has been set to 'openstack_helm' 15:00:34 #link https://etherpad.opendev.org/p/openstack-helm-weekly-meeting agenda 15:00:36 Hello 15:00:39 o/ 15:01:23 o/ 15:01:32 \o 15:02:19 we will start in a couple mins 15:04:54 ok 15:05:04 #topic congrats andrii 15:05:12 \o/ 15:05:23 thank you :) 15:05:24 the OSH core team would like to welcome andrii as a core reviewer, thanks andrii for all your hard work! 15:07:01 and I believe andrii has more things to discuss today too :) 15:07:02 \o/ 15:07:06 #topic Improvements in jobs execution time 15:07:16 andrii_ostapenko: the floor is yours 15:08:49 yes. Thank you, gagehugo. I started with small think for now and sorry for commit message. I'm introducing a mechanism to parallelize gate scripts execution that at this point saves 10-20% of execution time. https://review.opendev.org/739659 https://review.opendev.org/739891 15:09:20 if we're going in this direction i have several more optimizations in my mind 15:10:01 like script refactoring and running all tests in parallel at the end 15:10:36 The gates could use cleanup like this, so I'm in favor of something like this 15:10:41 also looking to switch back to kubeadm but move kubeadm-aio image build to osh images and reuse it in osh during the build 15:11:19 this should be evaluated if we can save couple mins there, so rnd for now 15:11:26 also pre-pulling some images in parallel 15:11:55 for now only https://review.opendev.org/739659 https://review.opendev.org/739891 - i'll prepare commits for review later 15:12:34 so if you have some other ideas - please share 15:12:36 I think there was discussion previously about hosting some images in the infra mirror 15:12:46 idk what happened there though 15:12:55 my goal is to have minimum time for compute-kit ~30 mins 15:13:09 That would be amazing 15:13:22 ++ 15:14:02 i don't feel we can have an average 30 mins, but at least we should move in this direction 15:15:05 I'm for it 15:15:06 also i noticed it's a bit worse on 8GB nodes. maybe we have memory consumption spikes due to parallel execution. still it's faster than consequent one 15:15:30 There may be things we can reduce there as well 15:15:34 replica counts 15:15:43 i'll prepare some comparison based on statistics 15:15:45 min memory values 15:16:00 yes. looking at individual charts is the next step 15:16:30 ok cool 15:17:15 minute here, minute there - every minute matters 15:17:36 ++ 15:17:40 yup, might just be a bunch of small tweaks that add up to larger gains 15:17:49 #topic osh-infra-images testing 15:18:00 andrii_ostapenko: this is you as well I believe 15:18:19 https://review.opendev.org/#/q/topic:test-osh-images 15:18:41 it's a mechanism to reuse osh jobs in osh-images on post-review 15:19:19 so what is suggest is to build and upload images during the gates on osh-images and test it there as well 15:20:22 also in future I want periodic builds for loci images, testing and promotion for loci images. we often have a situation when the last latest image is more then 1 month old 15:20:41 because we build them only on commits to corresponding code in osh-images 15:20:48 yeah, we've already hit issues where some images are old and out-of-date with upstream changes 15:21:06 I'm in favor of this as well 15:21:13 it's been some pain points 15:21:43 this topic is ready for review. it provides mechanism and implements it for loci stein and train on gates 15:21:50 If we build on each commit, why do we need to build periodically? 15:22:19 we build on each commit to osh-images to corresponding to particular image code 15:22:35 but don't build for new openstack component merges 15:22:52 Ah okay 15:23:03 I don't think I run into this problem much with the LMA images 15:23:19 yes. we need periodic one only for loci 15:23:25 It's been a bit annoying with things like the requirements image and stable images for openstack 15:23:39 Thanks for clarifying for me 15:23:43 branches get updated with bug fixes or reqs change, but the image is months behind 15:23:52 Makes sense 15:24:08 and suddenly images fail because the reqs match up game breaks 15:24:11 periodic ones will be available if i manage to merge https://review.opendev.org/#/c/740560/ 15:24:35 but now we can at least run test deployment on gates and topic is ready for review 15:24:52 i'm going to implement this logic for other images, not only loci 15:25:09 if we have a corresponding test job in osh or osh-infra 15:25:30 I think thats a good idea. I remember Steve W wanting more robust testing between osh-infra and osh-images too 15:25:51 e.g. for some changes in kibana image we can run osh-infra-logging job 15:26:02 ++ 15:26:53 also i want to get rid of some duplication across the repos https://review.opendev.org/#/c/740530/ 15:27:23 and inherit from infra job and reuse osh-infra-gate-runner instead of having it in both repos 15:27:26 I will go through the chain of patches and take a look 15:27:31 thank you 15:28:05 sounds good! 15:28:14 thanks andrii! 15:28:24 #topic Open Discussion/Reviews 15:28:37 #link https://review.opendev.org/#/c/740825/ node-down detector 15:28:46 #undo 15:28:47 Removing item from minutes: #link https://review.opendev.org/#/c/740825/ 15:28:50 I want to pass on a Q from my team 15:28:53 Radhika has been working on a dockerfile for the node-problem-detector. The dockerfile merged, and she is following that up with a zuul job for the image (https://review.opendev.org/#/c/740825/). The job log shows the image was built successfully, but we don't see it on dockerhub (https://hub.docker.com/u/openstackhelm/) 15:28:55 #link https://review.opendev.org/#/c/740825/ node problem detector 15:29:17 yeah it may need to get added 15:29:26 Is there a missing component, does the repo need to be created manually, will this get fixed when PS merges? 15:29:30 it will be uploaded after merge only 15:29:31 but also there's credentials I believe in the repo 15:29:57 actually it's being uploaded on post-review 15:30:19 with temp tag 15:30:28 and promoted to defined tag after the merge 15:31:25 image is not uploaded to dockerhub in untrusted pre-review pipeline in case if zuul-jobs *-docker-image jobs are used 15:32:24 Upon merging https://review.opendev.org/#/c/740825/ do we have to make a change to the dockerfile to trigger the build? Or will it run once the zuul addition is merged? 15:33:45 to rephrase that - is it merge to the image directory that triggers the build, or any change to osh-images 15:34:04 - node-problem-detector/.* 15:34:10 - zuul.d/node-problem-detector.yaml 15:34:23 so it's defined in zuul.d/node-problem-detector.yaml 15:34:35 when to run it 15:34:42 Ah I see now. ty 15:38:01 Anyone else have anything for this week? 15:38:13 that actually have a relation to testing an image 15:38:31 we'll be able to do it on post-review for this image as well 15:39:16 if there's any job in osh or osh-infra that deploys this chart 15:39:17 Yep :) 15:39:31 but not in pre-review 15:39:46 I'm not sure if it's been included in an osh-infra job yet. But I'll have to double check 15:39:51 might be in monitoring? 15:40:52 https://review.opendev.org/#/c/668754/ 15:41:11 aio-monitoring 15:44:20 Thanks everyone have a good rest of the week 15:44:24 #endmeeting