14:00:17 <bbezak> #startmeeting kolla
14:00:17 <opendevmeet> Meeting started Wed Jan 17 14:00:17 2024 UTC and is due to finish in 60 minutes.  The chair is bbezak. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:17 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:17 <opendevmeet> The meeting name has been set to 'kolla'
14:00:24 <bbezak> #topic rollcall
14:00:28 <mmalchuk> \o
14:00:32 <SvenKieske> o/
14:00:50 <mhiner> o/
14:00:51 <frickler> \o
14:01:48 <jangutter> o/
14:02:08 <mattcrees> o/
14:02:13 <bbezak> #topic agenda
14:02:16 <bbezak> * Announcements
14:02:16 <bbezak> * Review action items from the last meeting
14:02:16 <bbezak> * CI status
14:02:16 <bbezak> * Release tasks
14:02:16 <bbezak> * Regular stable releases (first meeting in a month)
14:02:18 <bbezak> * Current cycle planning
14:02:18 <bbezak> * Additional agenda (from whiteboard)
14:02:20 <bbezak> * Open discussion
14:03:01 <bbezak> #topic CI status
14:03:11 <bbezak> Kolla master is red
14:03:30 <bbezak> fixing it for now with https://review.opendev.org/c/openstack/kolla-ansible/+/905858
14:03:42 <bbezak> however I don't think it is a final approach
14:04:25 <bbezak> as we're using the same var for configuring docker daemon and container images source for kolla-ansible. That maybe not the best approach
14:04:55 <frickler> do you have a patch up in kolla to verify your fix?
14:04:59 <bbezak> it is only seen in Kolla CI invoked k-a upgrade jobs
14:04:59 <SvenKieske> sounds like a good observation to me
14:05:07 <bbezak> https://review.opendev.org/c/openstack/kolla/+/904576
14:05:39 <frickler> thx. and I agree splitting vars sounds reasonable
14:05:41 <bbezak> upgrade jobs finished there, however I've updated commit message, so we need to wait a bit for them to finish second time https://zuul.opendev.org/t/openstack/status#904576
14:06:57 <bbezak> hopefully we can fix it soon for now
14:07:15 <bbezak> and come with more general follow up next
14:07:21 <bbezak> #topic Release tasks
14:07:49 <bbezak> as we wanted to go back to monthly releases, I've pushed most of stable backports last week
14:08:12 <bbezak> and I want to wait with monthly release after this CI fix from above
14:08:27 <bbezak> will raise release patch as soon as we merge it
14:09:51 <bbezak> I think at some point we need to update this spreadsheet - https://docs.google.com/spreadsheets/d/1MS2KEhr5yMXEjD5ywU8peJomBNT0Va8bN8wy8hvodVY/edit#gid=0
14:09:54 <kevko> \o
14:10:04 <bbezak> will add a note for it
14:10:25 <frickler> what's in that sheet? I don't like to open a google doc
14:10:31 <SvenKieske> I didn't even know this spreadsheet is a thing, until now.
14:11:01 <bbezak> link to it is in the kolla whiteboard
14:11:02 <bbezak> https://etherpad.opendev.org/p/KollaWhiteBoard#L156
14:11:28 <bbezak> I'll ask mnasiadka about it when he is back in couple of weeks
14:12:10 <bbezak> #topic Current cycle planning
14:13:19 <bbezak> looking at caracal priorities. I've seen that ravlew started working on his ovn by default enablement
14:13:41 <bbezak> of course that also need a CI revamp a bit, so it'll take some time
14:13:53 <frickler> and /me started discussing about details a bit ;)
14:14:08 <bbezak> cool
14:14:35 <frickler> #link https://review.opendev.org/c/openstack/kolla-ansible/+/904959
14:14:46 <frickler> just for reference
14:14:50 <bbezak> thx frickler
14:15:10 <bbezak> just pasted to the whiteboard https://etherpad.opendev.org/p/KollaWhiteBoard#L233
14:15:41 <bbezak> if anybody have already some patches for priorities please put them in the whiteboard
14:16:05 <mattcrees> I've got a couple of bugfixes on the go, will add them there
14:17:57 <bbezak> just add your nickname there that we're aware who put them :)
14:18:17 <mattcrees> Sure thing
14:19:17 <SvenKieske> I put https://review.opendev.org/c/openstack/kolla-ansible/+/882497 there additionally, want to introduce shellcheck to bring some sanity to our bash scripts ;) need to resolve conflicts and tests though (again)
14:19:36 <bbezak> great, if anobody have something already half-baked from priorities and wanted to discuss the best approach please post questions in irc/changes
14:19:39 <SvenKieske> reviews would still be nice, also some openstack core test now fails, which I still haven't figured out why
14:21:05 <bbezak> thx SvenKieske. I haven't look into that change yet, will do some reading
14:21:59 <bbezak> #topic Additional agenda (from whiteboard)
14:22:46 <bbezak> frickler Stop/disable/cleanup services
14:22:50 <SvenKieske> some users wanted to extend the skyline configurability, I added patches they supplied to the whiteboard just now
14:22:58 <frickler> mnasiadka helpfully added some links to prior art. I need to check those
14:23:14 <bbezak> thx SvenKieske
14:23:36 <frickler> nothing more to add regarding cleanup from me at this point
14:23:44 <bbezak> that would be nice feature, thx frickler
14:24:04 <bbezak> frickler Deploy split glance services
14:24:22 <bbezak> some volunteer needed for that
14:24:53 <bbezak> will add it to priorities
14:25:02 <SvenKieske> I'm sorry, but can someone point me to the correct line on our whiteboard for that? I'm unable to find it.. (need some sleep I guess :D)
14:25:10 <frickler> I would do that, but if anyone wants to help, that would be nice
14:25:15 <frickler> L66
14:25:18 <SvenKieske> found it, ty
14:25:18 <bbezak> https://etherpad.opendev.org/p/KollaWhiteBoard#L66
14:27:06 <frickler> so I'll add that to the priorities list, too. seems it is getting pretty crowded
14:27:34 <bbezak> I don't think we should add regular bugfixes to priorites
14:27:40 <opendevreview> Pierre Riteau proposed openstack/kayobe stable/yoga: Switch IPA builds to CentOS Stream 9 for yoga  https://review.opendev.org/c/openstack/kayobe/+/903242
14:28:27 <bbezak> just ping reviewers and cores to review :)
14:28:39 <frickler> well it may depend on how review intensive they are
14:28:45 <frickler> but I agree in general
14:28:49 <mattcrees> Fair enough, I'll take them back off
14:29:06 <frickler> also set RP+1
14:29:21 <bbezak> exactly
14:29:53 <bbezak> L70 we already covered
14:30:05 <bbezak> mhiner Container engine migration
14:31:11 <mhiner> I would to reach some agreement about the CI tests
14:31:17 <frickler> I'll need to look at the patch mentioned there and the logs
14:31:44 <frickler> #link https://review.opendev.org/c/openstack/kolla-ansible/+/836941
14:31:47 <SvenKieske> in general you can also ping me for reviews, I'll scream if it's too much ;)
14:32:46 <mhiner> alright
14:33:06 <mhiner> additionally i would like for someone to take a look at ansible/roles/container-engine-migration/tasks/cleanup.yml
14:33:22 <bbezak> I haven't look into that change too much unfortunately. lack of logs after migration is pretty important to look into
14:33:28 <bbezak> will try to look into it
14:33:56 <mhiner> first task which deletes old container engine files produces massive logs (80MB, 300K lines)
14:34:31 <frickler> using no_log seems reasonable in that case
14:34:31 <mhiner> is it okay? for now I added no_log: true to suppress it
14:34:59 <SvenKieske> +1
14:35:10 <bbezak> looks pretty scary if some task before will not work as intended :)
14:36:08 <SvenKieske> yeah, the best option would be if this could be filtered or split up somehow, but I don't have looked into the details if that is even possible?
14:36:28 <frickler> the other option would be to leave cleanup as a manual task I guess? just don't add another -yes-i-really-mean-to-copy-this-option-from-the-docs-without-understanding-it
14:37:01 <bbezak> as a CLI command
14:37:24 <mhiner> that is also a option
14:37:54 <mhiner> all the volumes will be moved so theoretically all that is left are some configs from the old container engine
14:39:05 <frickler> if it is just some configs, why 300k lines?
14:40:10 <SvenKieske> can we split this cleanup task in multiple smaller tasks? maybe it's only a few "cleanup" items that account for the majority of the 300k lines?
14:40:16 <mhiner> yeah, it's not just configs, my bad
14:41:07 <mhiner> I still have those logs saved and majority of lines come from deletion in /var/lib/docker/overlay2/
14:42:17 <bbezak> I've added RP1 for this change
14:42:18 <SvenKieske> I guess one line per file/symlink? maybe we can collapse this somehow
14:42:36 <mhiner> yes, that's the case
14:43:30 <frickler> use "command: rf -rf" in this case instead of "file: state: absent", even if that's non-ansiblic?
14:43:32 <bbezak> well, removal of whole directory
14:43:36 <bbezak> above would solve it
14:43:58 <bbezak> I mean the current engine main path
14:44:08 <frickler> anyway, maybe we can continue this discussion on the review after having looked at this in more detail?
14:44:12 <bbezak> if that's what we want to achieve
14:44:22 <bbezak> yeah
14:44:54 <bbezak> #topic Open discussion
14:45:05 <SvenKieske> yeah, I was also thinking about something like `find -bla -delete` and only logging errors from that
14:45:38 <frickler> I have one topic regarding yoga eol
14:46:09 <frickler> iiuc so far we have been waiting for the release team to finish the automation for the new "unmaintained" state
14:46:41 <frickler> but it seems we could move to eol right now regardless of that, do we want that? or does someone still want to keep yoga around?
14:47:52 <bbezak> we still got several clouds on yoga. however definitely we should go to EOL yoga in this cycle
14:48:04 <bbezak> or next at most
14:48:09 <bbezak> imho
14:48:27 <bbezak> as yoga is a stepping stone for CS8/RL9
14:48:33 <SvenKieske> I think there are still some patches for yoga in flight, I guess that list could get updated and we should maybe ask to not propose any new patches for yoga then?
14:48:48 <bbezak> and such migration is taking a while
14:48:51 <SvenKieske> on the other hand I don't know if there is actual harm of patches being proposed and/or merged in yoga
14:49:44 <bbezak> yoga is somewhat a LTS release from CS8/RL9 migration perspective
14:49:52 <SvenKieske> to me it feels it should be the other way around, honestly. Maybe I don't have a good enough understanding of the release process. There should be three phases (like debian):
14:50:10 <frickler> imo it is just useless extra work for reviewers, but I can also just ignore reviews for that branch
14:50:17 <bbezak> it's rather a supporting burden
14:50:46 <bbezak> but good point frickler, we should do EOL of yoga soon'ish
14:50:47 <SvenKieske> 1. declare feature/patch freeze (no new patches proposed); maybe add a tag for that on the branch 2. merge existing patches 3. declare EOL via Tag
14:51:39 <bbezak> I need to check of status of unmaintained branches
14:51:40 <SvenKieske> currently we do: 1. merge existing patches 2. go back to 1. until there are no patches anymore 3. declare EOL via Tag
14:51:40 <frickler> those three things should happen in quick succession IMO, no need to make explicit phases for each
14:52:06 <opendevreview> Pierre Riteau proposed openstack/kolla-ansible master: Drop more remnants of install_type  https://review.opendev.org/c/openstack/kolla-ansible/+/905880
14:52:21 <SvenKieske> well I guess a problem is that new patches are faster to propose than existing patches getting merged
14:53:07 <bbezak> ok, we've got that on our radar then
14:53:11 <frickler> that just needs some concerted reviewer effort. another option is to just abandon unmerged patches
14:53:42 <bbezak> yeah, we're usually either merging or abandoning unmerged patches when doing EOL
14:54:05 <bbezak> and release team is checking if there are no patches awaiting in queue
14:54:29 <SvenKieske> There should be a clear cutoff date or other criteria as a patch submitter so I can know if my patch will be merged, no?
14:54:29 <bbezak> ok, anything else to discuss?
14:54:30 <frickler> so maybe I can propose a release patch for that, but wip it until more feedback comes in and maybe mnasiadka is back, too
14:54:44 <bbezak> yeah, I don't want to do EOL without PTL around
14:55:07 <bbezak> ok, thx frickler
14:55:30 <SvenKieske> maybe it's better if I pester the release team with this stuff?
14:56:22 <SvenKieske> I find it problematic, that there is this kind of no-mans-land/twilight zone, where it's not clear if patches will be merged or get abandoned last minute.
14:56:40 <frickler> usually the predicted dates in the release schedule work well. the current change in methods has sadly lead to delays
14:57:26 <frickler> you never can be sure whether a patch gets merged or rejected anyway
14:57:26 <SvenKieske> alright, if it's only a one-off event I'm fine with keeping everything like it is. I'll watch what happens the next EOL release cycle then :)
14:57:30 <bbezak> ok, let's wait for making decisions for PTL to be back from vacation. I'll add a note for that
14:58:16 <bbezak> ok I think we're done for today, thank your for joining
14:58:24 <bbezak> #endmeeting