15:00:46 <portdirect> #startmeeting openstack-helm 15:00:47 <openstack> Meeting started Tue Feb 26 15:00:46 2019 UTC and is due to finish in 60 minutes. The chair is portdirect. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:48 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:50 <openstack> The meeting name has been set to 'openstack_helm' 15:01:12 <mattmceuen> o/ GM 15:01:14 <howell> o/ 15:01:25 <dwalt> o/ 15:01:31 <portdirect> Hey everyone, the agenda is here https://etherpad.openstack.org/p/openstack-helm-meeting-2019-02-26, lets give it untill 5 past for people to trickle in :) 15:01:33 <srwilkers> o/ 15:01:42 <dustinspecker> o/ 15:01:48 <arunkant> o/ 15:02:00 <alanmeadows> o/ 15:02:20 <cheng1_> o/ 15:03:05 <roman_g> o/ 15:05:12 <portdirect> ok - lets go 15:05:17 <portdirect> #topic review turnaround time 15:05:45 <portdirect> for those that are not active on the ml, there is a thread here: http://lists.openstack.org/pipermail/openstack-discuss/2019-February/003035.html 15:05:53 <evrardjp> o/ 15:06:01 <portdirect> I think everyone involved in OSH should spend some time reading it 15:06:17 <portdirect> as theres a lot to parse, and process 15:06:54 <portdirect> as ptl, i think my role here should be at this stage to listen - as i know jayahn would like to write a response 15:07:27 <portdirect> though srwilkers and evrardjp have both been active on that thread, I dont know if theres anything either of you would like to discuss here? 15:08:30 <roman_g> I just wanted to mention that this applies not only to the OSH, but also to Airship. 15:08:32 <srwilkers> i don't have anything else to add here -- i think it's a pretty valuable discussion to add 15:08:38 <srwilkers> have, not add 15:08:39 <srwilkers> sorry 15:08:40 <evrardjp> Not discuss, but I can comment a little -- Thanks everyone active on this thread 15:09:32 <portdirect> evrardjp: ther floor is yours 15:10:34 <evrardjp> I have hope that it has opened eyes in the community , from all the sides. It's just that :) Thanks you all for being part of it, and let's continue working together to make that even better. 15:10:55 <portdirect> ok great, thanks evrardjp 15:10:59 <portdirect> ok to move on? 15:11:01 <evrardjp> alignment takes time, from whatever %COMPANY, %PROJECT you are coming from 15:11:02 <evrardjp> ok 15:11:52 <portdirect> openstack 1st, project 2nd, company 3rd - i think that mantra is so important here 15:12:03 <portdirect> #topic Spec for multiple container distro support 15:12:55 <portdirect> ok so theres been a few patchsets opened recently for adding opensuse support in osh, and up until not to long ago i was running centos based containers 15:13:27 <portdirect> 1st it is great to see renewed interest here - but we probably need to get a plan in place here 15:13:34 <evrardjp> agreed 15:13:49 <evrardjp> the sooner the better, for me at least. 15:13:53 <portdirect> my experience has been so far that over-rides work for everything other than the apache based services 15:14:23 <portdirect> and there we have a couple of options, either using the containers distro specific locations for config files etc 15:15:04 <portdirect> or getting more explicit in how we run these services, so the only 'distro specific' knowedge we need in the charts is some logic to select the right binary name 15:15:18 <portdirect> all this leads to us really needing a spec 15:15:28 <portdirect> (i think at least) 15:15:35 <evrardjp> I agree with the spec 15:16:18 <evrardjp> I think the spec will be refined with inputs of users -- for example, what can solution 1 vs solution 2 bring, can solution 2 do x.... Expertise is required there 15:16:26 <evrardjp> how can we help you portdirect? 15:16:26 <portdirect> does anyone want to volunteer to take it on? (at least the 1st pass) 15:17:06 <portdirect> if you wanted to take a stab at getting it going evrardjp it would be great 15:17:12 <evrardjp> I am fine with writing a spec, but I don't know what you have in mind 15:17:25 <portdirect> otherwise i could write it 15:17:46 <evrardjp> I think I would like to pick your brain to discuss what can be done, so I can document it -- can we do that outside the meeting? 15:17:47 <portdirect> which may be simpler if your still looking for a place to start there? 15:18:19 <portdirect> would it not be best to discuss here? 15:18:32 <portdirect> where all can get involved before going to gerrit? 15:18:47 <evrardjp> it is fine, but I don't want to take most of the meeting for that, so I think it would be better in the channel for example 15:19:15 <evrardjp> we only have an hour :) 15:19:32 <srwilkers> i think getting an initial draft up lets us start piling on feedback evrardjp 15:19:40 <portdirect> should we put some time aside next week to do it in the meeting once people have had time to prep for a bit? I'll get the etherpad up right after this one. 15:19:45 <evrardjp> srwilkers: totally 15:19:47 <srwilkers> i think once that happens, we can get it evolving fairly quickly 15:20:26 <evrardjp> srwilkers: agreed -- I think a sync on our channel can happen outside the meeting, and we put something up for review, and discuss that in the next week meeting 15:20:29 <srwilkers> portdirect: agreed 15:21:01 <evrardjp> or portdirect's approach it just takes us one week after :p 15:21:03 <portdirect> evrardjp: lets use the etherpad - so its not only people on irc that can be part of that initial convo 15:21:16 <portdirect> I'll get that up in the next 40 mins 15:21:20 <evrardjp> yup that's good! 15:21:47 <portdirect> #action portdirect to get meeting etherpad up with section for multiditro container support 15:22:19 <portdirect> #topic Daemonset Overrides 15:22:29 <portdirect> theres been a few things going on here: 15:22:46 <portdirect> 1) some great working looking at bringing the over-rides to ovs 15:23:10 <portdirect> 2) a regression for some projects outside of OSH that use the htk function (airships divingbell) 15:23:38 <portdirect> roman_g: can you take the 1st? and then i think cheng1_ can the 2nd? 15:24:32 <cheng1_> portdirect, yes 15:24:45 <roman_g> the patch form cheng1_ didn't work to resolve the issue, I'm not sure if they are related 15:25:28 <roman_g> portdirect: I think i don't have full understanding. Can I consult with you on topic and then accept the offer? 15:26:00 <portdirect> roman_g: i was meaning explaining the context here :) 15:26:12 <roman_g> ah 15:26:26 <roman_g> >> the secret contents which is used for the host falling under override is identical to the secret used by default 15:26:35 <roman_g> that's it in one line. 15:26:45 <portdirect> and this is related to a shift in helm versions? 15:26:51 <roman_g> Yes. 15:26:55 <roman_g> Update of Helm from version 2.11.0 to 2.12.1 and then to 2.12.3 broke host and label `overrides:` functionality for the Airship Divingbell. 15:27:03 <roman_g> Reverting those patches solves the issue. 15:27:06 <srwilkers> portdirect: yeah -- it failed when we moved from 2.11 to 2.12 15:27:08 <portdirect> if so we should try to identify if this is an issue with the function, or the way it is used in divingbell 15:27:14 <roman_g> I have started bisecting helm commits 15:27:29 <portdirect> i've not seen any reports of ceph/nova breaking - though this may just show a gap in our testing 15:27:48 <roman_g> openstack-helm-infra-airship-divingbell << this gate is failing 15:27:51 <portdirect> (ceph uses a different, but very similar function) 15:27:54 <roman_g> but it's non-voting 15:28:13 <portdirect> yeah - it would be great to get the divingbell maintainer to have a look at it 15:28:36 <roman_g> to make it voting? 15:28:55 <srwilkers> no, to determine why it's failing 15:29:05 <roman_g> it just fails exactly because of this issue. 15:29:12 <roman_g> it fails on overrides test 15:29:13 <portdirect> to try and see why its failing, craig is also the guy who write the function in the 1st place here 15:29:23 <mattmceuen> good point :) 15:29:43 <srwilkers> roman_g: the ask is to determine exactly what is causing the failure. we know where it fails, we know the version change introduced it, but it'd be nice to get the exact root cause of what introduced the regression here 15:30:12 <portdirect> roman_g: i dont think we should be making a project outside of osh voting atm 15:30:20 <srwilkers> absolutely not 15:30:24 <roman_g> agree 15:30:56 <cheng1_> roman_g, could you add the failing log in story, so others who have bandwidth could have a look 15:31:07 <roman_g> Sure, will do. 15:31:24 <roman_g> Anything else I can do? I'm currently bisecting helm commits. 15:32:06 <cheng1_> roman_g, thanks 15:32:22 <mattmceuen> let's reach out to craig on the mailing list if you think we could use his help roman_g 15:33:08 <mattmceuen> I don't think he's here today so he may not be aware of the issue 15:33:13 <roman_g> He knows about the issue for a week or so. I will reach out to him. 15:33:24 <mattmceuen> ok cool - thanks 15:33:42 <roman_g> he actually wrote a comment on that in a version upgrade review 15:34:17 <roman_g> next topic 15:34:22 <roman_g> thank you. 15:35:07 <portdirect> ok - cheng1_ can you discuss what you have been up to? 15:35:16 <portdirect> (im looking forward to this :) ) 15:35:16 <cheng1_> portdirect, sure 15:35:24 <cheng1_> I am trying to support per-host overrides of auto_bridge_add. 15:35:40 <cheng1_> https://storyboard.openstack.org/#!/story/2005059 15:36:34 <cheng1_> with this feature, we can add different network interfaces to ovs/lb bridges. because different host may have different network interface names 15:36:59 <cheng1_> for now, we have to suppose every host has the same network interface names 15:37:41 <cheng1_> I have created two patches in openstack-helm and openstack-helm-infra 15:37:58 <portdirect> cheng1_: this is great - though i wonder two things: 15:38:12 <cheng1_> I tested these two patch together, for ovs an lb cases 15:38:20 <portdirect> a) should we use a configmap (json?) to represent these things rather than env vars? 15:38:51 <portdirect> b) would it make sense to move the values used to drive this functionaility under the 'conf' top level key in the values 15:39:21 <cheng1_> portdirect, for a) I strongly agree with you. I already have implemented it with configmap 15:39:47 <cheng1_> it's in the patch, you can review it 15:40:18 <cheng1_> for b), I am happy to move .network.auto_bridge_add to .conf 15:40:51 <portdirect> cheng1_: on a - awesome :) 15:41:06 <portdirect> for b - this makes sense to me, but im not sure what otheres would think? 15:42:00 <evrardjp> can we leave those opinions for the reviews? 15:42:14 <evrardjp> on/in* 15:42:35 <evrardjp> I am fine either way 15:42:46 <cheng1_> IMO, moving the whole .network to .conf.network will make .conf complicated. because the .network contains some other info 15:43:18 <portdirect> cheng1_: this is my fear - Im not sure if we want to extract that out - and clearly seperate the items we have in there 15:43:21 <cheng1_> So I would like to move .network.auto_bridge_add only. maybe also .network.tunnel.interface 15:43:38 <portdirect> which is currently a strange mix of ingress/k8s svc config 15:43:38 <cheng1_> because we may also need per-host config for tunnel interface 15:43:48 <portdirect> and some low level networking stuff 15:44:21 <portdirect> i think the two you highlight ould be the perfect candates to move 15:45:02 <portdirect> but lets take it to gerrit if theres no one else where with strong opinions :) 15:45:57 <portdirect> #topic Creating separate Elasticsearch indexes for services 15:46:04 <portdirect> srwilkers: this is yours i think 15:46:12 <srwilkers> yep 15:46:54 <srwilkers> i started playing around with creating separate indexes in elasticsearch for the "core" services, if you will -- those being the core openstack services, calico, and the LMA type components 15:47:16 <srwilkers> this provides a few benefits 15:47:46 <srwilkers> first, this reduces the overhead Elasticsearch requires for managing those indexes (adding new indexes, searching for documents, etc) 15:48:14 <srwilkers> it also makes it easier to manage the lifecycle of those indexes with Curator, as it gives us more granular filters for the actions we configure Curator to perform (snapshot, delete, reindex, etc) 15:48:56 <portdirect> so - that sounds great 15:48:57 <srwilkers> the con here is that the fluentd configuration can start to get complicated 15:49:08 <portdirect> how so? 15:49:47 <srwilkers> the list of configured outputs gets longer, which isn't so bad when you think about the size of grafana's values.yaml 15:50:23 <srwilkers> but it'll also allow us to clean up some of the previously defined filters in fluentd that don't seem to be used (things like tagging events with auth.**) 15:51:01 <portdirect> ah ok 15:51:06 <srwilkers> another benefit this gets us is that it makes kibana's visualizations easier -- large indexes have the potential to make kibana choke due to elasticsearch query timeouts 15:51:32 <srwilkers> by splitting things out, it can potentially give us some breathing room wrt timeouts 15:51:40 <portdirect> i think though if this reduces the el bottleneck its worth pursuing 15:51:50 <srwilkers> but that's all i've got here -- feel free to leave reviews and thoughts 15:51:52 <srwilkers> i agree 15:51:59 <evrardjp> link? 15:52:09 <srwilkers> https://review.openstack.org/#/c/639085/ 15:52:37 <srwilkers> thats it for me here portdirect 15:53:00 <portdirect> thx dude 15:53:10 <portdirect> #topic Openstack-Helm Armada periodic jobs failing 15:53:28 <srwilkers> i'll make this quick too 15:53:46 <srwilkers> the openstack-helm armada deploy periodic jobs are failing, and it seems it's due to the neutron helm tests timing out 15:53:50 <srwilkers> i'll look into this today 15:54:24 <srwilkers> thats all i've got here portdirect 15:54:31 <portdirect> ok - i dont think much has changed there recently so would be good to see what regressed 15:55:09 <portdirect> #topic Reviews needed 15:55:36 <portdirect> a few items on the reviews needed list this week 15:55:43 <portdirect> Network Policy changes: 15:55:43 <portdirect> https://review.openstack.org/#/c/637621/ : Kube-state-metrics and openstack-exporter ingress network policy 15:55:43 <portdirect> https://review.openstack.org/#/c/638256/ : Grafana ingress network policy 15:55:43 <portdirect> https://review.openstack.org/#/c/638723/ : Nagios ingress network policy 15:55:43 <portdirect> https://review.openstack.org/#/c/634078/ : Elasticsearch, Fluentd, Kibana ingress network policies 15:55:44 <portdirect> Update Elasticsearch vhost locations: https://review.openstack.org/#/c/638211/ 15:55:44 <portdirect> Conditionally register Elasticsearch snapshot repositories in job: https://review.openstack.org/#/c/638496/ 15:55:45 <portdirect> Enable logging of client IP accessing apache reverse proxies: https://review.openstack.org/#/c/639128/ 15:55:45 <portdirect> Sonobuoy: allow multiple simultaneous chart installations https://review.openstack.org/#/c/636167/ 15:55:46 <portdirect> OVS-DPDK https://review.openstack.org/#/c/626894/ (georgk) 15:55:46 <portdirect> image building changes/fixing publishing: 15:55:47 <portdirect> https://review.openstack.org/#/c/639334/ 15:56:00 <portdirect> please lets try and get some eyes on them 15:56:32 <portdirect> #topic roundtable 15:57:36 <srwilkers> i've got nothing here 15:57:37 <evrardjp> Nothing to add. 15:58:07 <cheng1_> nothing from me. I will try to have a look at the ovs-dpdk patch 15:59:51 <portdirect> ok - thanks all - see you in irc/the ml 15:59:55 <portdirect> #endmeeting