13:01:26 <mnasiadka> #startmeeting kolla 13:01:26 <opendevmeet> Meeting started Wed Sep 6 13:01:26 2023 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. 13:01:26 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 13:01:26 <opendevmeet> The meeting name has been set to 'kolla' 13:01:34 <mnasiadka> #topic rollcall 13:01:50 <mnasiadka> o/ 13:01:57 <frickler> \o 13:01:59 <jangutter> o/ 13:02:10 <mhiner> o/ 13:02:11 <SvenKieske> \o/ 13:02:14 <SvenKieske> o/ 13:02:24 <mnasiadka> SvenKieske: well, it has been reviewed multiple times, it's just that this bug got a lot of attention in my company downstream ;) 13:02:51 <SvenKieske> sure, but I'm getting robbed of my +1 stats ;) 13:03:32 <mnasiadka> #topic agenda 13:03:32 <mnasiadka> * CI status 13:03:32 <mnasiadka> * Release tasks 13:03:32 <mnasiadka> * Regular stable releases (first meeting in a month) 13:03:32 <mnasiadka> * Current cycle planning 13:03:34 <mnasiadka> * Additional agenda (from whiteboard) 13:03:34 <mnasiadka> * Open discussion 13:03:39 <mnasiadka> #topic CI status 13:03:46 <mnasiadka> So, let's see 13:03:59 <mmalchuk> o/ 13:04:12 <SvenKieske> there was the rocky infra breakage, but that's fixed, right? 13:04:25 <mnasiadka> deleted wallaby from EM branch status 13:04:28 <mnasiadka> since it's EOL 13:04:35 <mnasiadka> yes, it was rocky infra breakage 13:04:50 <frickler> do we need to mirror more repos? 13:06:09 <mnasiadka> well, rocky is not mirrored at all 13:06:09 <mmalchuk> there was not only rocky issue 13:06:27 <mnasiadka> second thing is we could mirror docker rpm from something that has smaller size 13:06:31 <mmalchuk> also docker and two more 13:06:43 <mnasiadka> grafana is also usually broken every now and then 13:06:57 <mmalchuk> yep. docker and grafana 13:07:42 <mnasiadka> frickler: I was thinking on working on docker, just need to find proper place in system-config for that 13:08:27 <frickler> mnasiadka: let me know if you need help, though I'd have to search for that myself 13:08:40 <mnasiadka> frickler: there's mirror-update role that uses rsync, I doubt docker exposes rsync 13:09:22 <mnasiadka> I'll have a look and start asking stupid questions on #opendev :) 13:09:23 <frickler> mnasiadka: ah, right. I don't know if there is a different solution for rpms 13:09:39 <frickler> we could by starting to mirror docker .debs though ;) 13:09:51 <mnasiadka> frickler: we do mirror them I think, we just don't use them in kolla 13:10:34 <frickler> hmm, o.k. 13:10:36 <bbezak> o/ 13:10:42 <mnasiadka> hmm, zed rocky9/debian periodics failed 13:11:01 <mnasiadka> #link https://zuul.openstack.org/builds?project=openstack%2Fkolla&pipeline=periodic&skip=0 13:11:23 <opendevreview> Michal Arbet proposed openstack/kolla-ansible master: [DNM] Just test something https://review.opendev.org/c/openstack/kolla-ansible/+/893609 13:11:26 <opendevreview> Merged openstack/kolla stable/xena: Add server-status handler to Rocky/Centos Apache conf https://review.opendev.org/c/openstack/kolla/+/893243 13:11:36 <frickler> I also noticed there seem to still be rmq errors, like the latest failure in https://review.opendev.org/c/openstack/kolla/+/893722 13:11:50 <jsuazo> o/ 13:11:53 <SvenKieske> kolla-publish-ubuntu-quay seems also to have issues? 13:13:02 <mnasiadka> SvenKieske: on zed? I don't think so 13:13:17 <mnasiadka> ok, let's move on and see if it fails again 13:13:25 <mnasiadka> #topic Release tasks 13:13:37 <mnasiadka> So, this week we should post a patch with release highlights 13:14:04 <SvenKieske> it has stats here at least: https://grafana.opendev.org/d/c0d59dad13/kolla-failure-rate?orgId=1 13:14:16 <mnasiadka> Anybody wants to have a go at skimming what we could post as marketing highlights? 13:14:36 <mnasiadka> scanning release notes is my usual go 13:14:48 <mnasiadka> SvenKieske: that dashboards surely needs love 13:15:06 <SvenKieske> I fixed what I knew how to fix 13:15:58 <SvenKieske> I still need to check what versions we can bump, but I guess it's not that much 13:15:59 <mnasiadka> bbezak: want to have a go at release highlights? 13:16:13 <mnasiadka> SvenKieske: the whole prometheus stack would be happy :) 13:17:27 <mnasiadka> ok, I'll do the release highlights :) 13:17:36 <mnasiadka> So, when we are at highlights 13:17:41 <SvenKieske> ah right, that are easy bumps, most of the time, did it in the past already 13:17:49 <mnasiadka> one thing is Let's Encrypt 13:17:55 <mnasiadka> kevko: did you update that Kolla patch already? 13:18:26 <mnasiadka> seems not 13:18:35 <mnasiadka> would be nice to get it rolling sooner than later 13:18:40 <mnasiadka> second thing is Podman 13:19:07 <mnasiadka> mhiner: I assume you still have some arm related issues (but I think kevko is looking at them)? 13:19:49 <mhiner> yes, I am also trying to solve it but didnt have much time for that since the last meeting 13:19:52 <SvenKieske> here are also some unadressed podman comments left to do: https://review.opendev.org/c/openstack/ansible-collection-kolla/+/852240 13:20:05 <mnasiadka> I see https://review.opendev.org/c/openstack/kolla-ansible/+/893187 is an easy one 13:21:35 <mnasiadka> SvenKieske: that one - I'm still puzzled why we need ternary and why it doesn't work the same way as for docker role? 13:21:54 <mhiner> SvenKieske: I guess the debian repo comments can be closed and Im waiting on another opinion regarding the podman containers starting/restarting 13:22:47 <SvenKieske> I'm not sure on this ternary stuff. mhiner tested it and said it didn't work 13:22:48 <kevko> mnasiadka: didn't have a time :( ...but i have it half reworked regarding kolla part 13:23:23 <SvenKieske> if we had the time it would be cool to just use always venvs and be done with this ;) 13:23:31 <mnasiadka> kevko: great, hopefully you will have time to finish it off ;) 13:23:39 <kevko> i am experimenting with podman on zuul ... i have some indices ..but still not confirmed 13:23:40 <mhiner> it didnt work because the variable were set, but works when they are commented out like in https://review.opendev.org/c/openstack/ansible-collection-kolla/+/892990 13:23:57 <mnasiadka> ok, I commented on the ansible-collection-kolla patch - sorry, but let's keep the changes only podman related 13:24:18 <mnasiadka> if you need to change anything in the docker role (and I don't know why would you) - it should be a separate patchset 13:24:47 <ihalomi> mnasiadka: its more like refactor since you pointed bad manners in code in podman patchset that were just copied from docker roles 13:24:51 <kevko> btw, it would be nice to merge kolla part of podman 13:25:08 <mnasiadka> I pointed bad manners? I'm fine with copying bad manners :) 13:25:18 <mnasiadka> kevko: I promised to have a look, should have time on Friday 13:25:24 <SvenKieske> well that's the consequence if you want to have the same code patterns everywhere, of course you have then unrelated refactors..that's quite some circular logic here.. 13:25:30 <kevko> because now when i needed to add cross-dependency from kolla-ansible -> kolla .... setup_gate is failing because kolla master don't know how to work with podman ... because not merged 13:25:42 <ihalomi> mnasiadka: not directly you :D but you as reviewers :) 13:25:55 <mnasiadka> SvenKieske: well, adding new functionality should be separate from refactoring - it's a bit insane to do it in one step 13:26:03 <mnasiadka> refactor is refactor - it's not adding new functionality 13:26:28 <SvenKieske> not really, it's insane to replicate known wrong behaviour just because "we have done it this way since forever" 13:26:53 <mnasiadka> well, usually you do a refactor first, and then add new functionality on top of it 13:27:06 <kevko> +1 13:27:27 <mnasiadka> I just value small patchsets that do one thing, not change everything at the same time - because we have no clue what is old, what is new, how should it work, etc 13:27:31 <SvenKieske> yeah, this way you never refactor, because people with money always only pay for features, but not refactoring, so we end up where we are, old stuff rotting. 13:28:28 <SvenKieske> I see maybe 1% of commits doing refactoring and 99% being feature work, where it should be the reverse. these numbers are made up of course. 13:28:32 <mnasiadka> I don't know any people with money that want to invest in Kolla :) 13:28:48 <SvenKieske> really? who's paying you then? ;) 13:28:51 <kevko> +1 :D 13:29:25 <mnasiadka> and this is rather a sidetracking thread, we told Konstantin to refactor a lot of stuff in Kolla in order to add support for Podman, so we shouldn't change approaches for the same feature set in another role 13:29:26 <SvenKieske> sorry for the rant, but it's still circular logic: people say: "don' 13:29:47 <Fl1nt> mnasiadka, my company invest on our team that work with kolla and try to fix things when it break, they're paying time we pass on those patch. 13:29:50 <mnasiadka> SvenKieske: I know you're a fan of ranting, I'm Polish - we rant even when we sleep :) 13:29:59 <SvenKieske> t do this -> people change their code -> now it's done differently than in an unrelated file -> separate your refactoring from feature work.. 13:30:27 <mmalchuk> Fl1nt me too 13:30:30 <SvenKieske> honestly all these issues in the last few weeks are always because we have no written down thing we all agreed upon, no? 13:30:55 <Fl1nt> Here are my two cents 13:31:02 <Fl1nt> since a while now 13:31:10 <mnasiadka> Can we have those two cents in Open discussion section? 13:31:12 <mnasiadka> not now? 13:31:17 <Fl1nt> sure 13:31:20 <mnasiadka> thanks 13:31:40 <SvenKieske> so we are always arguing in circles, "do we need a release note for this?" "is this refactoring"? "should this patch be split up"? There should be no argument about this stuff during code review imho, it should be clear from coding guidelines or a linter. 13:31:45 <mnasiadka> mhiner: let's discuss the a-c-k change in Gerrit, I'll try to do some testing and help with the code 13:31:57 * SvenKieske will shut up for now :) 13:32:16 <mnasiadka> #topic Regular stable releases (first meeting in a month) 13:32:29 <kevko> SvenKieske: feel free to write detailed developer docs and we can agree all together in review 13:32:57 <mnasiadka> frickler: we should wait until the OVN clustering backports are merged, so let's revisit next week (I maybe wrongly assumed you'll want to do the releases) 13:33:14 <frickler> I think we should just stop doing stable releases 13:33:16 <mnasiadka> SvenKieske: yes, channel your ranting powers to a document and Gerrit patchset 13:33:26 <mmalchuk> mnasiadka can I add Xena backport of OVN? 13:33:28 <mnasiadka> frickler: at all? 13:33:33 <frickler> they've become so irregular nobody can rely on them anyway 13:33:34 <mnasiadka> mmalchuk: sure 13:33:35 <frickler> yes 13:34:18 <Fl1nt> just use tags 13:34:28 <mmalchuk> mnasiadka ok. it take some time to solve conflicts 13:34:32 <mnasiadka> frickler: so one release and we're done? this is also a bit weird for people still using pypi for kolla releases 13:34:32 <SvenKieske> I'll try, that will be a fun review I'm sure :) 13:34:34 <frickler> well tags are releases for us 13:34:47 <frickler> stop publishing to pypi, too. issue solved 13:35:04 <mnasiadka> well, that was something I proposed on one PTG, nobody agreed :) 13:35:07 <Fl1nt> TBH, we do only use tags are they ensure readability, discoverability and reproducibility (ish) 13:35:18 <SvenKieske> I agree, but does this need to be announced, for release engineering etc? I'm not sure on the openstack process here 13:35:41 <frickler> I think deployment project are special in that regard 13:35:42 <mnasiadka> #link https://releases.openstack.org/reference/release_models.html 13:35:49 <mnasiadka> those are the release models we can use 13:35:59 <SvenKieske> tags are nice, but the packages on pypi let people assume these are fresh, when they really are not 13:36:00 <frickler> but I can try to find out. also I agree that that should be a PTG decision 13:36:04 <Fl1nt> frickler, why would you stop deploying to pypi, that's exactly the opposite of what people want 13:36:04 <mnasiadka> currently we are cycle-with-rc 13:36:54 <frickler> +trailing? 13:36:55 <SvenKieske> I mean we have regularly people asking about pypi packages, so from a user point of view it might be better to go in the other direction and publish more regularly. 13:37:00 <mnasiadka> yes, +trailing 13:37:11 <mnasiadka> but if we go untagged, then it's probably worse mayhem 13:37:12 <SvenKieske> either way, the current state is not good, I guess everybody agrees on that. 13:37:29 <frickler> I think people using pypi do not get what they expect 13:37:36 <SvenKieske> can the push to pypi not be automated? I really have no clue how much work it is 13:37:37 <mnasiadka> true 13:37:47 <mnasiadka> well, we need to raise a patch in openstack/releases 13:37:55 <frickler> it is automated as part of tagging a release 13:37:59 <Fl1nt> frickler, quite the opposite, they exactly got a release semver that you know what is contained 13:38:06 <mnasiadka> question is why do they not get what they expect 13:38:33 <mnasiadka> because if they expect bugs fixed in a certain bugfix release that gets released every month 13:38:35 <Fl1nt> because kolla stable releases keeps moving 13:38:52 <mnasiadka> then I agree we would need to be better in bug squashing and versions management 13:39:14 <frickler> yes, and there's nobody around to do that 13:39:20 <Fl1nt> I could tell you what we're expecting at my scale, can't talk for everybody 13:39:29 <Fl1nt> frickler, I do 13:39:30 <SvenKieske> well we do regular bugfixes and backports, my understanding was, that these are a) not tagged(?) and b) not published to pypi 13:40:22 <SvenKieske> I'm sure I'm missing something, currently I understand that we miss a patch in openstack/releases, is that all to do? there must be more, no? 13:40:28 <mmalchuk> agree... as backported soo many fixes 13:40:45 <Fl1nt> We've 187 countries, 4 continents, 5 regions per continents, 3 AZ per regions worldwide global cluster, we do not want things to evolve from build to build of a certain release, so we base on snapshot and freeze and pypi 13:41:10 <Fl1nt> so the workflow is as is 13:41:28 <frickler> SvenKieske: you always ask for written docs, now we have them here and you don't read them ;) https://docs.openstack.org/kolla/latest/contributor/release-management.html#stable-branch-lifecycle 13:41:47 <mnasiadka> frickler: that's true, we don't even have people to triage bugs and prioritise them, not speaking about closing them 13:42:39 <mmalchuk> we have them. me? Sven? 13:42:44 <SvenKieske> frickler: well when it comes to openstack docs, I either ask, google, a dev on IRC or chatgpt, because navigating the docs site is...dangerous 13:42:45 <mnasiadka> And I agree it's a discussion for PTG - how to manage that going forwards 13:42:56 <Fl1nt> We build on xena-cos8-13.0.9 using pypi and local build, those build need to be reproducible 100% we can't have build A do diverge from build B, with current kolla structure, it's managable through tags, but yet we still sometimes have sleeps as some package on a tag aren't fixed, so to circumvent that we build against a specific repo snapshot for a specific release build 13:42:57 <mnasiadka> because the current model is a bit broken 13:43:02 <SvenKieske> agreed on PTG discussion 13:44:04 <mnasiadka> ok, let's go to additional agenda from whiteboard, because there's ~15 minutes left 13:44:06 <mnasiadka> #topic Additional agenda (from whiteboard) 13:44:11 <SvenKieske> but thanks for the lifecycle link, but it seems we do not do what's documented there ;) 13:44:37 <mnasiadka> frickler: jobboard needs love :) 13:44:50 <frickler> yes, needs time, too. next time ;) 13:45:08 <mnasiadka> jsuazo - TaaS (tap-as-a-service) 13:45:09 <frickler> you can drop it from agenda for now I think 13:45:23 <mnasiadka> I commented today https://review.opendev.org/c/openstack/kolla/+/885151 13:46:42 <jsuazo> mnasiadka: Thanks! Btw we have tested both changes up to Zed, so everything should be working fine. 13:46:44 <mnasiadka> once Kolla part is sort-of-solved - we can review kolla-ansible part 13:46:57 <jsuazo> noted 13:48:06 <mnasiadka> jsuazo: I'm not saying everything shouldn't be working fine, my question is why are we overriding the version from upper-constraints - and should we even be doing that 13:49:10 <jsuazo> mnasiadka: my comment wasn't related to yours, just mentioning. The version change was done because if left on the constraints, we were having dependency mismatch errors on neutron 13:49:43 <mnasiadka> yes, because in the patch we're trying to install tap-as-a-service from the stable branch, not from pypi 13:49:47 <mnasiadka> question if we should 13:49:51 <mmalchuk> jsuazo bump version in requiremets? 13:50:13 <jsuazo> Older version didn't work when tested, had to up the version to get the expected results. 13:50:23 <frickler> do we really need this in all of neutron? can't it be optional somehow? 13:50:48 <mnasiadka> #link https://review.opendev.org/c/openstack/kolla-ansible/+/885417 13:50:51 <mnasiadka> this is the kolla-ansible part 13:51:08 <jsuazo> frickler: I believe we included a variable to opt in and out, if not we could implement it 13:51:27 <mnasiadka> basically if we have a feature in kolla-ansible - it makes sense to install taas in the image 13:51:57 <mnasiadka> question why taas is in upper-contraints 13:52:03 <frickler> I'm not convinced it makes sense to include any possible feature in all neutron containers 13:52:09 <jsuazo> mmalchuk: Would love to do so, didn't work out how to D; I'll take any pointers 13:52:15 <opendevreview> Michal Arbet proposed openstack/kolla-ansible master: [DNM] Just test something https://review.opendev.org/c/openstack/kolla-ansible/+/893609 13:52:29 <Fl1nt> isn't TAPaaS supposed to be a ghost town? 13:53:07 <frickler> seems it has been revived 13:53:14 <mmalchuk> jsuazo https://opendev.org/openstack/requirements.git 13:53:28 <mmalchuk> jsuazo just make change there 13:53:48 <mmalchuk> jsuazo bump or even drop 13:54:06 <Fl1nt> frickler, last patch is 3 months ago tho 13:54:24 <SvenKieske> I'll need to be in another meeting at 0400 13:54:31 <jsuazo> Fl1nt: might be, but my company is pretty interested on these features, so we are working with one of the remaining members on how revive it 13:54:48 <Fl1nt> jsuazo, ok, good to read that then :D 13:54:57 <jsuazo> we are already working on bug fixes and some new features for it 13:55:11 <mnasiadka> well, anyway - it has been added to u-c long time ago for networking-midonet 13:55:16 <jsuazo> mmalchuk: Thanks! 13:55:46 <mnasiadka> I don't mind installing one lonely pypi package, but overriding u-c seems like an overkill for now 13:56:09 <mnasiadka> But maybe it's just me 13:56:17 <mnasiadka> let's wait for an answer in the patchset 13:56:22 <mnasiadka> #topic Open discussion 13:56:28 <mnasiadka> (a.k.a. ranting time) 13:56:33 <frickler> I actually think we need more u-c overriding, but that's another topic for another day 13:56:37 <mmalchuk> Gnocchi 13:56:44 <mmalchuk> lets build it 13:56:49 <mmalchuk> https://review.opendev.org/q/I3ca4e10508c26b752412789502ceb917ecb4dbeb 13:57:09 <mmalchuk> all fixed, CI passed 13:57:22 <frickler> the version bump should merge on master first IMO 13:57:37 <frickler> https://review.opendev.org/c/openstack/kolla/+/893722 13:57:57 <mmalchuk> not related 13:57:58 <jangutter> jsuazo updating openstack/requirements/upper-constraints.txt means that non-kolla projects like devstack start testing the same thing too, rather than kolla being unique. 13:58:34 <mnasiadka> frickler: once it passes 13:58:36 <mmalchuk> frickler master wes build from the master 13:58:44 <mmalchuk> was* 13:58:50 <mmalchuk> nevermind 13:58:52 <mmalchuk> ok 13:59:11 <mmalchuk> lack of Kayobe review for a week again: 13:59:15 <mmalchuk> https://review.opendev.org/c/openstack/kayobe/+/861397 13:59:19 <mmalchuk> https://review.opendev.org/c/openstack/kayobe/+/879554 13:59:26 <mmalchuk> bbezak ^ 14:00:04 <mmalchuk> and new one https://review.opendev.org/c/openstack/kayobe/+/893688 14:00:11 <mnasiadka> ok 14:00:13 <mnasiadka> time is up 14:00:16 <mnasiadka> see you next week 14:00:18 <mnasiadka> #endmeeting