14:00:16 <mnasiadka> #startmeeting kolla
14:00:16 <opendevmeet> Meeting started Wed Mar 20 14:00:16 2024 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:16 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:16 <opendevmeet> The meeting name has been set to 'kolla'
14:00:20 <mnasiadka> #topic rollcall
14:00:23 <mmalchuk> o/
14:00:37 <SvenKieske> o/
14:00:46 <mattcrees> \o
14:01:00 <mhiner> o/
14:01:25 <SvenKieske> aravindh: it would be very welcome if you can rewrite it in python, I don't know about the cli framework you mentioned though.
14:01:49 <SvenKieske> ah they are gone
14:02:09 <Fl1nt> o/
14:02:29 <frickler> \o
14:03:33 <jovial> o/
14:03:39 <mnasiadka> https://www.irccloud.com/pastebin/wjoS0J7a/
14:03:45 <mnasiadka> oops
14:03:50 <mnasiadka> #topic agenda
14:03:50 <mnasiadka> * CI status
14:03:50 <mnasiadka> * Release tasks
14:03:50 <mnasiadka> * Current cycle planning
14:03:50 <mnasiadka> * Additional agenda (from whiteboard)
14:03:51 <mnasiadka> * Open discussion
14:03:55 <mnasiadka> #topic CI status
14:04:05 <mnasiadka> I think it's mainly green, especially after fixing mariadb timeouts
14:04:46 <mnasiadka> jovial: how's kayobe CI fixing?
14:05:51 <jovial> I've submitted a change to Matt's patch. Just awaiting CI.
14:06:15 <mnasiadka> great
14:06:19 <mnasiadka> #topic Release tasks
14:06:47 <mnasiadka> It's R-6 - client libraries release time
14:07:18 <mnasiadka> Any volunteer to switch Kolla to use Caracal UCA?
14:07:53 <mnasiadka> RDO as well for CentOS/Rocky builds
14:09:19 <SvenKieske> I guess I can make that again
14:09:29 <mnasiadka> Fantastic
14:09:29 <SvenKieske> #link https://review.opendev.org/c/openstack/kolla/+/891018
14:09:39 <SvenKieske> just for my personal reference
14:09:42 <mmalchuk> mnasiadka I can
14:09:50 <mnasiadka> great
14:09:53 <mnasiadka> two volunteers
14:10:00 <SvenKieske> mmalchuk: can you do RDO and I do UCA?
14:10:09 <mmalchuk> lets swap
14:10:20 <mmalchuk> I've already made change
14:10:22 <SvenKieske> okay, have to look up the RDO stuff again :D
14:10:30 <SvenKieske> ah alright, that makes sense then
14:10:40 <SvenKieske> no need for duplicate work
14:10:53 <Fl1nt> what does UCA stadn for?
14:11:00 <mnasiadka> Ubuntu Cloud Archive
14:11:09 <Fl1nt> ah! Ok! thx
14:11:20 <mnasiadka> #topic Current cycle planning
14:12:07 <mnasiadka> Let's go through dashboards
14:12:09 <mnasiadka> #link https://tiny.cc/kolla-dashboard
14:12:12 <opendevreview> Maksim Malchuk proposed openstack/kolla master: [release] Use UCA Caracal  https://review.opendev.org/c/openstack/kolla/+/913788
14:12:30 <mnasiadka> two exit code related patches and proxysql bump awaiting reviews
14:13:03 <mnasiadka> I just noticed comments on https://review.opendev.org/c/openstack/kolla/+/905116/2 (thanks frickler )
14:13:06 <mnasiadka> I'll update today
14:14:35 <mnasiadka> #link https://tiny.cc/kolla-ansible-dashboard
14:14:39 <mmalchuk> mnasiadka UCA: https://review.opendev.org/c/openstack/kolla/+/913788
14:14:46 <mnasiadka> Kolla-Ansible dashboard - seems some patches also awaiting reviews
14:15:08 <mnasiadka> mmalchuk: thanks
14:15:27 <mnasiadka> Ok, anybody has any feature/patch that he needs reviews on - or just wants to talk about them?
14:15:57 <SvenKieske> well it would be nice if we have a clear plan with regards to bash and if people would like to have more bash linting (shellcheck)
14:16:04 <SvenKieske> so if we abandon bash (big
14:16:33 <mnasiadka> So, my first problem with shellcheck is that it seems the most widely adopted tool in openstack land is bashate
14:16:40 <SvenKieske> sorry hint enter to fast: +1 from me), I would not put much time in refactoring our bash
14:17:02 <mnasiadka> second of all, I prefer to reduce our bashisms instead of promoting them :)
14:17:45 <mhiner> mnasiadka: yes, what about this small podman fix: https://review.opendev.org/c/openstack/kolla-ansible/+/912521
14:17:48 <mnasiadka> any other comments?
14:17:56 <SvenKieske> I can agree to the second statement, I don't understand the first tbh. that's like saying: we already do pep8 linting on python, no need for any flake8 rules? they really don't conflict with each other
14:18:44 <SvenKieske> and honstly bashate doesn't really do much, you can count the checks it does on 2 hands and they are almost all only about optics, not about actual errors.
14:18:48 <bbezak> o/
14:18:50 <opendevreview> Maksim Malchuk proposed openstack/kolla master: [release] Change RDO's DELOREAN_DEPS repo to Antelope  https://review.opendev.org/c/openstack/kolla/+/913790
14:18:57 <SvenKieske> but agreed, no new bash scripts then?
14:19:06 <mmalchuk> mnasiadka RDO: https://review.opendev.org/c/openstack/kolla/+/913790
14:19:23 <jovial> Just on the podman support in kayobe. Do you think it is OK to land whilst podman is broken on ubuntu (https://review.opendev.org/c/openstack/ansible-collection-kolla/+/911422)? Otherwise how do I proceed?
14:19:38 <mnasiadka> SvenKieske: from my perspective it would be good to have a plan around what we can do to reduce number of bash scripts on the deployment side - and also have a plan for the CI side (if needed)
14:20:20 <mnasiadka> jovial: I think we should have that problem documented for users, that should be fine - and since next release we should be supporting 24.04 - it might get fixed there
14:20:38 <mnasiadka> maybe we should also have a precheck in kayobe/kolla-ansible warning users about possible problems?
14:21:12 <SvenKieske> can we maybe finish one topic before we start the next one?
14:21:31 <SvenKieske> so what about new bash scripts, e.g. https://review.opendev.org/c/openstack/kolla-ansible/+/836941/43/tests/test-container-engine-migration.sh ?
14:21:34 <Fl1nt> I'm a strong support of get ride of bash as much as possible.
14:21:46 <SvenKieske> just a random example
14:22:23 <SvenKieske> imho at first we should document in our dev guide that people please at least not add new bash scripts, bugfixes in existing scripts are fine I guess.
14:22:46 <SvenKieske> I guess I will write a docs patch for that and see if that gets traction
14:22:55 <jovial> What is the replacement for bash?
14:23:01 <Fl1nt> python?
14:23:08 <mnasiadka> there is no replacement for now
14:23:29 <mnasiadka> once we have one, and patches like this one can reuse a ,,framework''
14:23:40 <mnasiadka> then I'm fine with saying no bash, or whatever - adapt to current ,,framework''
14:24:01 <mnasiadka> but now nobody has made any plan if we should continue with CI based on bash scripts
14:25:12 <mnasiadka> and it feels like PTG topic
14:25:20 <mnasiadka> do we all agree here - or is there some other proposal?
14:25:22 <SvenKieske> so I'm back at square one: if we don't have a plan to replace bash (and I don't really get the distinction between code for testing in CI and deployment code, both are equally important to me) we need to somehow ensure bash code quality, no?
14:25:41 <jovial> There seems to be a lot of hate for bash. IMO stringing a load of shell commands together in bash script is a lot more readable than the python equivalent.
14:25:41 <mnasiadka> For me there are two types of bash usage in Kolla-Ansible
14:25:49 <Fl1nt> mnasiadka, CI = zuul so python/ansible/yaml as much as possible meaning almost everywhere.
14:25:50 <SvenKieske> I already added it to the PTG topics and I fear I do disagree on some fundamentals, but maybe let's discuss this at PTG then
14:25:59 <mnasiadka> the CI scripts - I'm fine with them being as they are today
14:26:10 <mnasiadka> But I'm also fine with whatever we decide
14:26:17 <Fl1nt> jovial, bash isn't portable neither it is readable imho
14:26:21 <kevko> \o
14:26:49 <mnasiadka> the bash that we use in some places that gets executed on normal deployment - it might be better to replace that with Ansible/Python - on a case by case basis
14:26:51 <jovial> bash runs on most systems ;-)
14:26:55 <mnasiadka> and that bash probably could do some linting
14:26:59 <mnasiadka> but we already have bashate
14:27:01 <SvenKieske> jovial: I don't hate bash - if that came across like that - I just wrote enough of it to know that it's very very hard to write bash without serious bugs without a linter like shellcheck
14:27:12 <mnasiadka> and I'm not going to tell it's bad or something, it might be limited
14:27:18 <mnasiadka> but we also don't lint a lot of other things
14:27:25 <Fl1nt> jovial, not really, there are so many exemple against that; Like the tests conditions for instance.
14:28:02 <SvenKieske> agreed, let's move the discussion to PTG then because I don't think we can agree on something actionable right now. I'll think about it for PTG to maybe come up with some proposal.
14:28:20 <mnasiadka> yeah, flame wars on PTG
14:28:32 <jovial> Fl1nt, Do you mean running a bash script in alternative shell?
14:28:35 <mmalchuk> I've missed something... do we have any time slots for PTG with Google Calendar links?
14:28:36 <SvenKieske> and please also bring your proposals :)
14:29:36 <jovial> Sounds like an interesting session :)
14:29:42 <Fl1nt> jovial, shell/bash/ksh/whatever bash isn't made to do code, it's made for commands chaining.
14:29:54 <Fl1nt> it's a useful tool sure
14:30:01 <Fl1nt> on daily ops looping :D
14:30:08 <mnasiadka> mmalchuk: https://etherpad.opendev.org/p/kolla-dalmatian-ptg - line 44
14:30:08 <Fl1nt> but for dev, it's horrible
14:30:20 <mnasiadka> and https://ptg.opendev.org/ptg.html
14:31:53 <mmalchuk> mnasiadka last PTG we use some website with planing where I can add google calendar events. do we have it now?
14:32:11 <SvenKieske> I wanted to add to the podman breakage: might it be worth to file a bug against the actual podman package in ubuntu? all I have seen so far is a bug against the linux kernel (which is also technically true, but there is a workaround patch in upstream podman)
14:32:45 <mnasiadka> mmalchuk: google calendar events? maybe it was on the doodle poll, but the timeslots are the same as usual
14:33:29 <mmalchuk> mnasiadka yep. doodle. do we have it now?
14:33:34 <mnasiadka> no, we don't
14:33:39 <mmalchuk> :(
14:33:42 <mnasiadka> doodle was only for a poll when to meet
14:34:00 <mnasiadka> you can ask on #openinfra-events if there's support for .ics file generation via the ptg website ;-)
14:34:12 <mnasiadka> (and maybe even contribute)
14:34:25 <mmalchuk> nevermind. will create events myself
14:34:37 <mnasiadka> jovial: it might make sense to file the bug as SvenKieske says
14:34:41 <mnasiadka> let's go forward
14:34:55 <mnasiadka> #topic Additional agenda (from whiteboard)
14:35:27 <mnasiadka> (SvenKieske): Skyline patches need core reviewers, look finished to me: https://review.opendev.org/c/openstack/kolla-ansible/+/910514 , basically all currently owned by Uwe: https://review.opendev.org/q/owner:uwe.jaeger@valiton.com
14:36:37 <SvenKieske> yeah, Uwe is pretty active actually and responds rather fast if you have feedback for improvements :)
14:36:51 <SvenKieske> but what I looked at myself is in good shape now imho
14:37:26 <SvenKieske> I noticed some stuff just got merged yesterday, thanks
14:39:34 <SvenKieske> and there's also some activity in the other stuff afaik, I need to re-review those myself, thanks so far.
14:39:38 <mnasiadka> (SvenKieske): please also review keystone bugfix: https://review.opendev.org/c/openstack/kolla-ansible/+/912452
14:39:54 <mnasiadka> CI is failing
14:40:01 <SvenKieske> yeah about that one, it's on my todo for today to investigate the upgrade job that fails with this in CI
14:40:19 <mnasiadka> Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Service Unavailable (HTTP 503)\n
14:40:26 <SvenKieske> noticed that myself but was busy, I thought I would have fixed CI until the meeting, sorry.
14:41:08 <jovial> mnasiadka, For the ubuntu packaging issue. There is this one: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2052961, but now much activity. Adding a known issue to the release notes sounds like a good path forward for now.
14:41:15 <SvenKieske> this seems clearly related to the patch, maybe the task is called at the wrong place (to early)? not sure though
14:42:03 <SvenKieske> jovial: this is a bug against the linux kernel source, afaik. it would be better to report this against podman in ubuntu instead, if you want the fix to be backported, that is.
14:42:07 <mmalchuk> strange that all upgrade jobs are failed
14:42:21 <mnasiadka> ok then, next one
14:42:23 <mnasiadka> (ihalomi/mhiner): Patchset that bumped version of docker ( https://review.opendev.org/c/openstack/ansible-collection-kolla/+/910751 )didn't work because we install package python3-docker during tests using package manager and not pip. Debian currently doesn't support release of docker-py higher than 5.0.3, would it be possible to change installation using pip for all cases? Or maybe adding package package by ourselves to osbpo
14:42:23 <mnasiadka> repository as is it done in podman role
14:42:35 <mhiner> tldr: if venv is not used, the docker python package is installed through system package manager and required minimal version 6.0.0 is not available there
14:42:40 <SvenKieske> mmalchuk: yes, strange
14:42:55 <mnasiadka> mhiner: where do we install python3-docker?
14:43:02 <mnasiadka> from rpm/deb?
14:43:38 <mhiner> yes, but only if virtual environment is not used
14:43:46 <mnasiadka> and only on debian
14:44:03 <jovial> SvenKieske, That makes sense. Totally missed that one was against the kernel :facepalm:
14:44:05 <mnasiadka> kevko: how do we get python3-docker bumped in Debian?
14:44:09 <SvenKieske> mhmm, but didn't we switch to use venvs everywhere?
14:44:14 <kevko> mnasiadka: i've built and upload :D
14:44:24 <mnasiadka> mhiner: solves your problem? :D
14:44:39 <SvenKieske> jovial: no problem :) but I guess you won't get an answer there with regards to backports of podman patches ;)
14:44:52 <mnasiadka> if we don't use venvs in all CI - we should alter the CI I guess
14:45:01 <SvenKieske> +1 imho
14:45:14 <kevko> I am a maintainer https://packages.debian.org/sid/python3-podman << it's sid ...but I've asked zigo to add to osbpo deb repo
14:45:17 <SvenKieske> or was that on purpose? (what would that be? local mirror instead of pypi?)
14:45:27 <mhiner> mnasiadka: I guess, since I'm only a middle man in this because Ivan can't attend these meetings :)
14:45:39 <mnasiadka> mhiner: ok, makes sense
14:45:53 <mnasiadka> mhiner: can you check where in CI python3-docker/podman gets installed and we rework this CI to use venvs?
14:45:55 <kevko> oh, sorry python3-docker ... that package i asked also zigo to backport from sid
14:46:32 <kevko> i don't think we should install packages via pip in operating system
14:47:02 <kevko> also new ubuntu and debian distros don't allow installation via pip
14:47:15 <SvenKieske> no we didn't talk about operating system but to use a venv instead? at least to me that implies to not overwrite system-packages?
14:47:50 <SvenKieske> and the issue seems to be that we don't use venv everywhere so system package is used and version there is wrong. and the solution should be to just use venv everywhere
14:48:12 <mnasiadka> anyway, I hope mhiner/ihalomi can find the place and hopefully improve that
14:48:21 <kevko> so you will set ansible_python_interpreter to something local env ?
14:48:23 <SvenKieske> kevko: afaik a proper venv everywhere would sidestep this :)
14:49:07 <Fl1nt> I confirm, it does, that's the way we build everything, container > venvs > pip > kolla > whatever
14:49:22 <kevko> my point is that there are minimum python packages installed in system ... it would be better to handle it in system
14:49:40 <kevko> what i know ..it's only python3-docker and python3-podman
14:49:53 <kevko> what else do we need to run ansible modules against the host ?
14:50:37 <SvenKieske> sysctl stuff springs to mind, there's a current patchset floating around splitting it into it's own role
14:51:07 <Fl1nt> kevko, that's is just for kolla scope, not if you step back then you've got the same for ceph-ansible and many others that will necessarily collide with kolla at some point.
14:51:29 <SvenKieske> I find our distinction with host/container also rather fuzzy, because we tend to mount lot's of host stuff into the containers and stuff bleeds through everything (pid namespaces, net namespaces, user namespaces..)
14:53:06 <kevko> Fl1nt: i don't think
14:53:29 <Fl1nt> kevko, it is, we do HPC and it does collide
14:54:58 <kevko> Fl1nt: with system-wide python packages ?
14:55:00 <mnasiadka> we basically use venv everytime, so it makes sense to use that in CI - especially that we only use python3-docker/podman only on Debian in CI
14:55:07 <Fl1nt> kevko, yes
14:55:10 <opendevreview> Mark Goddard proposed openstack/kayobe master: CI: Test custom routes & rules on EL9  https://review.opendev.org/c/openstack/kayobe/+/899941
14:55:21 <mnasiadka> we install from pip on other distros even if there is no venv
14:55:29 <SvenKieske> mnasiadka: ack +1; imho we should do in CI what we do in actual deployments (where possible)
14:55:50 <mnasiadka> I think I had some patches about that, will try to dig them out later
14:55:52 <kevko> Fl1nt: containerize whatever it is ?
14:55:56 <SvenKieske> that doesn't mean that system packages can't be updated separately :)
14:56:08 <mnasiadka> Sorry, but we need to move to the last topic ;)
14:56:11 <mnasiadka> #topic Open discussion
14:56:13 <Fl1nt> kevko, wdym?
14:56:16 <mnasiadka> Any other topics?
14:56:52 <SvenKieske> just wanted to say thank you all for letting me learn a lot in the past year (it's almost a year since I work almost full time on this)
14:57:52 <mnasiadka> a year, nice - we all learn every day ;)
14:57:56 <mmalchuk> nice work ;)
14:58:10 <SvenKieske> yeah, the learning never stops :)
14:58:29 <frickler> also just a couple of hours left for TC election, if you waited to make your vote, do it now
14:59:10 <SvenKieske> I actually just did that today, because I forgot it yesterday :D
14:59:14 <mmalchuk> frickler I didn't receive email with personal link... maybe google spam filter
14:59:42 <SvenKieske> mmalchuk: it doesn't come from opendev domain, but from a university, kevko had also trouble finding it
14:59:58 <mmalchuk> I know. so spam in progress)
15:00:28 <frickler> mmalchuk: did you actually register? you may want to check the instructions https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/PXG7D3AUTFDOITTTLMJJHPUTXEHNMCMB/
15:00:33 <mmalchuk> thats all, folks I need to go. mnasiadka thanks for chairing
15:00:46 <SvenKieske> mmalchuk: you might want to check for a mail sent 7. of march 2024 from "Condorcet Internet Voting Service (on behalf of Ian Y. Choi) <civs@cornell.edu>"
15:00:51 <mnasiadka> ok then, it's time to finish the meeting
15:00:56 <mnasiadka> thanks for attending!
15:00:58 <mnasiadka> #endmeeting