13:00:24 <bbezak> #startmeeting kolla
13:00:24 <opendevmeet> Meeting started Wed Jun 14 13:00:24 2023 UTC and is due to finish in 60 minutes.  The chair is bbezak. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:00:24 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:00:24 <opendevmeet> The meeting name has been set to 'kolla'
13:00:31 <bbezak> #topic rollcall
13:00:37 <mmalchuk> \o
13:00:39 <mattcrees> o/
13:00:47 <SvenKieske> o/
13:00:55 <kevko> gre\o/
13:00:56 <frickler> and /me is surprised, I'd assumed we'd skip this one, too. but well, I'm here
13:01:34 <bbezak> I recon we can chat about pressing issues, even when some of us are in vancouver sleeping :)
13:01:42 <mmalchuk> frickler only mnasiadka skip this meeting as he said
13:01:53 <hrw> \~o~\
13:02:44 <bbezak> #topic agenda
13:02:46 <bbezak> * Roll-call
13:02:46 <bbezak> * Agenda
13:02:46 <bbezak> * Announcements
13:02:46 <bbezak> * Review action items from the last meeting
13:02:46 <bbezak> * CI status
13:02:47 <bbezak> * Release tasks
13:02:47 <bbezak> * Regular stable releases (first meeting in a month)
13:02:49 <bbezak> * Current cycle planning
13:02:49 <bbezak> * Additional agenda (from whiteboard)
13:02:51 <bbezak> * Open discussion
13:04:01 <bbezak> I don't have much too say about announcements and review actions items from the last meeting
13:04:25 <bbezak> let's go to the CI status then
13:04:33 <bbezak> #topic CI status
13:04:41 <mmalchuk> red
13:05:10 <mmalchuk> rabbitmq issue the main thing
13:05:22 <hrw> just do upgrade to 3.12/26
13:05:28 <mmalchuk> +1
13:05:46 <frickler> 3.12.0? very courageous
13:05:48 <SvenKieske> as said above, there seem to be new IO issues, see e.g. https://zuul.opendev.org/t/openstack/build/e804ca9b01f24a4ca7e1d7edb3e96fb3/log/job-output.txt
13:05:54 <mmalchuk> frickler why?
13:05:59 <frickler> and not very feasible for stable branches
13:06:10 <frickler> because .0 is like "alpha" release
13:06:22 <mmalchuk> tommorow it would be .1
13:06:24 <hrw> frickler: that's why master first?
13:06:42 <mmalchuk> lets pin .*
13:07:30 <hrw> rabbitmq/erlang was always a problem
13:07:37 <frickler> so maybe for master, ok, but that doesn't handle stable branches and upgrade tests
13:08:01 <bbezak> yeah, surely stable branches cannot go with 3.12.0
13:08:04 <mmalchuk> as mnasiadka said we a firefighters) so wi will fix it again and again
13:08:20 <kevko> Arnaud Cogoluègnes replied regarding rabbitmq
13:08:35 <hrw> I do not have hours assigned to kolla anymore so will ignore rmq this time
13:08:52 <hrw> kevko: url?
13:08:56 <SvenKieske> sure, there are, especially for upgrades, quite some preconditions mentioned in the 3.12 release notes, before you can upgrade from 3.11; don't know if we meet them
13:09:03 <frickler> probably in moderation still
13:09:18 <kevko> it is not in mail archive ..but check openstack discuss ..he is suggesting to switch to cloudsmith ..i am preparing email that we already replaced this repo
13:09:59 <frickler> so what about arm then?
13:10:15 <mmalchuk> postpone ?
13:10:16 <hrw> sometimes I wonder when kolla team get tired enough and remove whole aarch64 support and then start cutting distros
13:10:33 <mmalchuk> fix x86_64 first?
13:10:36 <bbezak> SvenKieske: those IO issues looks like zuul issues, maybe it is a case for #opendev channel ?
13:10:42 <kevko> hrw: +1 :D
13:10:57 <mmalchuk> +1 lol
13:11:05 <SvenKieske> bbezak: yeah, was wondering the same; I'll post them there
13:11:21 <bbezak> SvenKieske: thx
13:13:31 * SvenKieske has the impression that there simply does not exist a stable, available mirror for all past erlang/rmq releases, regardless the architecture.
13:14:01 <bbezak> that would a bummer :)
13:14:05 <mmalchuk> it exists but lack of aarch64
13:14:07 <bbezak> and a bit crazy
13:14:09 <hrw> SvenKieske: exactly
13:15:01 <mmalchuk> #opendev sleeps with vancouver)
13:15:02 <SvenKieske> mmalchuk: I dunno, I mean we get the suggestion to use cloudsmith, but we just disabled that because it was broken, no? where is that magic mirror that doesn't remove past patch releases ?
13:15:49 <opendevreview> Michal Arbet proposed openstack/kolla master: Bump proxysql version  https://review.opendev.org/c/openstack/kolla/+/886093
13:16:13 <mmalchuk> SvenKieske we have many suggestions but not choose right)
13:17:06 <hrw> SvenKieske: we have 3 distros and two architectures. and 5-6 distro versions.
13:17:22 <frickler> so the only solution: build erlang+rmq from source within a container?
13:17:30 <hrw> SvenKieske: covering it for rmq/erlang is painful
13:17:40 <SvenKieske> so, fix x86_64 first, or what? fine with me; afaik I don't know of any arm prod systems, but that doesn't mean a lot.
13:17:47 <kevko> https://review.opendev.org/c/openstack/kolla/+/885857
13:17:48 <SvenKieske> hrw: sure; I know :)
13:18:02 <mmalchuk> frickler built from source? kidding?
13:18:04 * hrw off - other duties
13:18:10 <bbezak> somebody contacted ppa admins?
13:18:13 <kevko> me
13:18:17 <bbezak> thx hrw
13:18:20 <SvenKieske> frickler: that sounds at least somewhat viable, though I never build anything in the erlang space.
13:18:42 <kevko> but mail not archived because of message size ... probably my picture of company in footer ? :/
13:19:06 <mmalchuk> lets drop the arm and one problem solved
13:20:03 <SvenKieske> the less nuclear option maybe would be to contact ppa admins and ask if we can help maintain that. as I understood it was "just" a matter of a package being dropped. co maintaining is certainly less a burden than rebuilding everything ourselves?
13:20:13 <kevko> https://paste.openstack.org/show/bUaFtanAO13OvMm8O79Q/  << communication
13:20:25 <bbezak> pretty big decision for that meeting I'd say :), let's not rush it and drop support for now
13:20:36 <kevko> SvenKieske do you have arm64 hosts to build ?
13:21:15 <bbezak> so according to them everything is fine? kevko
13:21:21 <bbezak> "network glitch"
13:21:36 <bbezak> https://cloudsmith.io/~rabbitmq/repos/rabbitmq-erlang/packages/?q=distribution%3Aubuntu+AND+distribution%3Ajammy+AND+version%3A1%3A25*+AND+name%3A%27%5Eerlang-base%24%27
13:22:10 <SvenKieske> I mean when I look at the link posted by bbezak everything looks fine? did someone retry the download?
13:22:32 <SvenKieske> maybe that stuff was affected by the aws us-east-1 outage yesterday(?)
13:22:40 <SvenKieske> dunno where cloudsmith hosts
13:22:52 <kevko> we are using ppa.launchpad.net because armd64 ...but they removed 25*  ..so there is only 26* ... when you switch to cloudsmith ...it's ok ...but no support for arm64
13:23:27 <SvenKieske> kevko: ah thanks for correcting me; I mixed those repos up (we switch too often :D )
13:23:47 <mmalchuk> lets vote for temp swtitch to cloudsmith? until we fix arm
13:24:16 <SvenKieske> sounds reasonable from my newbie perspective *shrugs*
13:24:46 <kevko> Launchpad PPA only provides the most recent release of Erlang 25.3.x. Cloudsmith and its mirror currently provides the most recent patch release in the following Erlang series:
13:25:10 <kevko> https://www.rabbitmq.com/install-debian.html#apt-launchpad-erlang
13:25:57 <kevko> we definitely remove and replace ppa repository
13:26:08 <kevko> because it looks like ppa.launchpad.net only maintain latest erlang versions
13:26:09 <SvenKieske> "The Cloudsmith repository has a monthly traffic quota that can be exhausted"
13:26:18 <SvenKieske> https://www.rabbitmq.com/install-debian.html#apt-cloudsmith
13:26:43 <mmalchuk> IMHO Cloudsmith is the best place. then we can talk how to build/sync arm repos and add it later
13:26:48 <kevko> that's the reason why i used the mirros ...
13:26:49 <kevko> The Cloudsmith repository has a monthly traffic quota that can be exhausted. For this reason, examples below use a Cloudsmith repository mirror. All packages in the mirror repository are signed using the same signing key.
13:27:02 <kevko> in my patch
13:27:16 <kevko> ppa1.novemberain.com
13:27:19 <SvenKieske> yeah, sounds also reasonable
13:27:24 <kevko> https://review.opendev.org/c/openstack/kolla/+/885857
13:27:54 <mmalchuk> +1
13:28:47 <kevko> The logs above mentioned both Cloudsmith and PPA, use one or the other, not both.
13:28:47 <kevko> PPA makes only the latest built version available, in this case, 26.0. This is something we don't control with PPA.
13:28:53 <kevko> ^^ he replied 6 minutes ago
13:28:55 <mmalchuk> no matter wich repo, I mean Cloudsmith - lest be a mirror
13:29:29 <bbezak> so cloudsmith looks better in that regard surely
13:30:27 <kevko> yep
13:31:22 <bbezak> tough case indeed. In any case I would rather have fixed stable branches for x86 for now. master can use 26. On the other hand divergence between master and stable branches is not wise in the longer run
13:32:21 <mmalchuk> as I understand we don't want 26
13:32:45 <mmalchuk> even master use 25 from cloudsmith mirror
13:33:00 <bbezak> not in stable branches, until it is proven to be stable and working upgrade
13:33:56 <kevko> we should use cloudsmith for amd64 and something different for amr64
13:34:14 <kevko> what about this http://binaries.erlang-solutions.com/debian/dists/bullseye/contrib/binary-arm64/Packages
13:34:14 <bbezak> looks like it is a way to go
13:34:25 <kevko> this should work
13:35:44 <bbezak> no jammy arm there?
13:35:53 <kevko> nope
13:36:13 <mmalchuk> debian only
13:36:45 <mmalchuk> http://binaries.erlang-solutions.com/ubuntu/dists/jammy/
13:36:53 <mmalchuk> empty
13:37:21 <bbezak> debian:
13:37:21 <bbezak> erlang:
13:37:21 <bbezak> url: "https://ppa.launchpadcontent.net/rabbitmq/rabbitmq-erlang/ubuntu"
13:37:21 <bbezak> suite: "focal"
13:37:21 <bbezak> component: "main"
13:37:21 <bbezak> gpg_key: "erlang-ppa.gpg"
13:37:29 <bbezak> that is current setting
13:37:29 <SvenKieske> maybe the debian packages "Just work" on jammy? (who am I kidding, but one can hope)
13:37:31 <kevko> hmm we can use debian repo ...but worse is that yammy is based on bookworm if i am correct and bookworm is missing
13:37:34 <bbezak> exactly my point
13:37:35 <bbezak> :)
13:37:46 <bbezak> can you test it kevko ?
13:37:54 <bbezak> in change that is
13:38:37 <kevko> in past hrw built and hosted erlang on his server ..but he already dropped ...
13:38:44 <kevko> maybe we can ask him to build again ? :P
13:39:13 <kevko> well, i don't have arm host ..so i can't :D
13:39:21 <kevko> *can't test
13:39:22 <bbezak> I mean those erlang-solutions.com
13:39:26 <bbezak> binaries
13:39:31 <kevko> i don't have arm64
13:39:52 <bbezak> well I guess in CI it is fine
13:40:04 <kevko> which job is for arm64 ?
13:41:17 <bbezak> kolla-build-ubuntu-aarch64
13:42:03 <bbezak> kolla-build-debian-aarch64,
13:42:07 <bbezak> and then kolla-ansible-debian-aarch64
13:42:13 <bbezak> it has failed in your change
13:42:14 <kevko> https://paste.openstack.org/show/b3mRNUqQi2BVfjRTUhs9/
13:42:25 <bbezak> yeap
13:42:25 <kevko> yeah i know :D
13:42:28 <bbezak> :)
13:44:17 <bbezak> ok I think we have something to test on. Use stable 25 1.11 from Cloudsmith for amd64, and try maybe those erlang-solutions for arm in the meantime if possible
13:44:54 <bbezak> ok, let's move on. we can continue this discussion on IRC and in the change
13:45:15 <bbezak> if that's ok for you
13:45:37 <frickler> +1
13:45:41 <SvenKieske> +1
13:45:43 <bbezak> #topic Release tasks
13:46:34 <bbezak> we need to do switchover master branch from antelope, I saw some changes from hrw. I'll try to look into that later this week if nobody have a time to do so
13:47:15 <mmalchuk> lets merge https://review.opendev.org/c/openstack/kolla/+/886087
13:47:19 <mmalchuk> it has +2
13:48:05 <mmalchuk> no matter that CI failed so)
13:48:31 <kevko> well, i thought that it will pass and second +2 is needed
13:48:34 <frickler> what about the rocky failure? is that also from rmq repo?
13:48:35 <kevko> i will reset my voe
13:49:00 <bbezak> it is bifrost
13:49:05 <bbezak> that failed in rocky
13:49:19 <kevko> frickler: rocky repo should use different repo for erlang ..but didn't check ..ppa is used only for debubtu
13:49:32 <frickler> so more CI issues
13:49:44 <bbezak> hurray
13:50:12 <frickler> epel install failure
13:50:18 <frickler> https://ef634d1e9a5ddf8bf39c-29c48a4a553d1a0c965532e3378dd6a0.ssl.cf1.rackcdn.com/886087/1/check/kolla-build-rocky9/17bbc7d/kolla/build/000_FAILED_bifrost-base.log
13:50:23 <frickler> might just be unstable?
13:50:46 <bbezak> it might, gpg error
13:51:23 <bbezak> let's recheck it then, let's see
13:51:55 <bbezak> ok, let's go further
13:52:04 <bbezak> #topic Current cycle planning
13:52:44 <bbezak> I assume bookworm, that hrw is working on
13:54:02 <bbezak> anyone has a case to discuss within this topic?
13:54:27 <kevko> nope, i will follow hrw changes and will help if it will be needed
13:54:33 <bbezak> thx kevko
13:54:39 <bbezak> #topic Open discussion
13:54:56 <mmalchuk> please review and merge backports: https://review.opendev.org/q/topic:fix-cyborg-service
13:55:11 <bbezak> will do mmalchuk
13:55:13 <mmalchuk> and there lack of Kayobe reviews again
13:55:25 <mmalchuk> https://review.opendev.org/c/openstack/kayobe/+/879554?usp=search
13:55:30 <mmalchuk> https://review.opendev.org/c/openstack/kayobe/+/861397?usp=search
13:55:40 <mmalchuk> please these two at least
13:56:14 <bbezak> will try to look into that
13:56:33 <kevko> bbezak i've added you as reviewer for my patch ..can u check ? https://review.opendev.org/c/openstack/kolla-ansible/+/877413
13:56:54 <bbezak> sure
13:57:26 <mmalchuk> regarding https://bugs.launchpad.net/kolla-ansible/+bug/1965111
13:57:37 <mmalchuk> should we backport?
13:57:55 <mmalchuk> y, x, w
13:58:34 <mmalchuk> IMHO we need to
13:58:49 <bbezak> if it's not breaking anything I don't think it is an issue
13:59:02 <mmalchuk> maybe you right
13:59:23 <kevko> i think this is not breaking anything
13:59:30 <mmalchuk> but it can be cacouse service_name==service_type in most cases
13:59:47 <mmalchuk> but if we face the issue like Cyborg?
13:59:57 <SvenKieske> I can maybe at least review some of that stuff, as we either way wanted to add more service token stuff in light of the last cve.
14:00:29 <mmalchuk> service tokens - the different problem
14:01:02 <mmalchuk> this is about access rules
14:01:21 <opendevreview> Michal Arbet proposed openstack/kolla master: Bump proxysql version  https://review.opendev.org/c/openstack/kolla/+/886093
14:01:24 <SvenKieske> ah okay, I just skimmed it
14:02:04 <bbezak> if you feel that it is a big problem just propose backports, if it is not breaking current deployments
14:02:53 <bbezak> ok I think we can conclude for today, thx for joining and good talk
14:02:55 <bbezak> #endmeeting