14:59:39 <noonedeadpunk> #startmeeting openstack_ansible_meeting
14:59:39 <opendevmeet> Meeting started Tue Oct  3 14:59:39 2023 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:59:39 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:59:39 <opendevmeet> The meeting name has been set to 'openstack_ansible_meeting'
14:59:45 * NeilHanlon needs more coffee
14:59:48 <noonedeadpunk> #topic rollcall
14:59:55 <noonedeadpunk> Yeah, apparently me too :D
14:59:58 <NeilHanlon> hehe
15:00:02 <NeilHanlon> we're just a bit early today
15:00:36 <mgariepy> hey !
15:01:01 <noonedeadpunk> \o/
15:01:06 <jrosser> o/ hello
15:01:15 <damiandabrowski> hi!
15:01:28 <noonedeadpunk> #topic bug triage
15:01:30 <noonedeadpunk> To be frank - I in fact don't have much updates/to say from the last meeting. Was mostly occupied with internal things
15:01:44 <noonedeadpunk> But, I've submitted couple of new bugs we've found during the upgrade
15:02:53 <noonedeadpunk> But lets start from this one
15:02:56 <noonedeadpunk> #link https://bugs.launchpad.net/openstack-ansible/+bug/2036421
15:03:17 <noonedeadpunk> I;ve replied to it, but I'm not sure if we wanna do smth there or just mark as invalid
15:03:30 <openstack-ansib> I want to see if Horizon has been translated into Farsi or not?!
15:03:34 <noonedeadpunk> As I see how this release note can be confusing, but I'm not sure what can be done
15:04:18 <noonedeadpunk> openstack-ansib: you can see horizon translations here: https://translate.openstack.org/iteration/view/horizon/master?dswid=7334
15:05:14 <noonedeadpunk> regarding bug - I'm not sure what we can do at this point... We could add another release note back in the days, but not I guess it's useless
15:05:20 <noonedeadpunk> *now
15:06:40 <noonedeadpunk> So I guess I will mark it as invalid if no objections
15:07:02 <NeilHanlon> that seems reasonable
15:08:29 <noonedeadpunk> #link https://bugs.launchpad.net/openstack-ansible/+bug/2037554
15:08:35 <noonedeadpunk> this one is actually really bad
15:08:46 <noonedeadpunk> and it's not limited to what is described there...
15:09:07 <noonedeadpunk> I've catched another phase of this bug yesterday
15:10:05 <jrosser> should we make that a separate play?
15:10:05 <noonedeadpunk> So it failed on us when we were deploying new compute node
15:10:13 <jrosser> just to make it simple
15:10:31 <noonedeadpunk> As it attempted to do db migration against compute
15:10:40 <noonedeadpunk> jrosser: yeah, I was thinking exactly about that tbh
15:11:08 <noonedeadpunk> except we have quite some complexity in the playbook for nova... I guess that stopped me from doing this originally
15:12:41 <noonedeadpunk> as I was thinking to add some post_tasks or do tasks_from, but that doesn't look too trivial with how we do things here https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/os-nova-install.yml#L54-L58
15:12:43 <ayush> There is another bug here
15:12:45 <ayush> https://bugs.launchpad.net/osops/+bug/2038315
15:13:26 <noonedeadpunk> Ok, moved it to the correct project
15:14:22 <jrosser> common-playbooks/nova.yml is kind of nasty too
15:14:24 <noonedeadpunk> sorry for not describing bugs in a good way - was pretty much in a rush and needed not to forget about them
15:14:42 <jrosser> as it attempts to be common, then a whole bunch of the tasks are conditional for computes only
15:14:46 <noonedeadpunk> yup, it pretty much is
15:14:47 <jrosser> so that could be simplified too
15:14:57 <jrosser> like too much optimisation there makes a mess
15:15:09 <noonedeadpunk> I wonder if we need to have that common playbook
15:15:18 <jrosser> true
15:15:32 <jrosser> perhaps some tidying here would make a neat place to deal with that bug
15:15:36 <noonedeadpunk> as it may be way more plain when moved to os-nova-install
15:15:50 <noonedeadpunk> ok, sounds like a plan then
15:16:17 <noonedeadpunk> hopefully, it will be backportable enough...
15:16:23 <noonedeadpunk> #link https://bugs.launchpad.net/openstack-ansible/+bug/2038315
15:16:40 <noonedeadpunk> So, we have some stable branches borked due to ansible.netcommon breaking galaxy
15:16:55 <noonedeadpunk> I didn't have time to look there today, but we're catching failures internally as well
15:17:01 <noonedeadpunk> Except 2023.1 - it's working fine
15:17:12 <noonedeadpunk> (as we moved to github from galaxy there)
15:17:24 <noonedeadpunk> ayush: did workaround I've posted you helped?
15:17:46 <noonedeadpunk> ie - create user-collection-requirements.yml with  https://paste.openstack.org/show/bC5oFVEOBoOdB9yMb9Dg/ ?
15:18:14 <NeilHanlon> i can probably find some time to look at the galaxy problem today/tomorrow
15:18:17 <noonedeadpunk> damiandabrowski: actually, you've catched that yesterday as well
15:18:42 <jrosser> what actually is the bug?
15:18:55 <noonedeadpunk> that bootstrap fails of some stable branches
15:19:12 <jrosser> oh sure but is there a new netcommon that breaks our old stuff?
15:19:15 <noonedeadpunk> because it can't install ansible.netcommon for $reasons
15:19:20 <jrosser> ahha ok
15:19:30 <noonedeadpunk> yeah, I believe it just breaks galaxy overall
15:20:00 <noonedeadpunk> I believe that's related to them having `v` in tags, while not having in versions or smth like that
15:20:14 <noonedeadpunk> (as ansible wasn't happy either in some scenarios)
15:20:31 <NeilHanlon> "unexpected error when getting available versions of collection"
15:20:38 <NeilHanlon> such a helpful message 🙃
15:20:46 <noonedeadpunk> Skipping Galaxy server https://galaxy.ansible.com/api/. Got an unexpected error when getting available versions of collection ansible.netcommon
15:20:47 <noonedeadpunk> yeah
15:21:18 <NeilHanlon> #link https://github.com/ansible-collections/ansible.netcommon/issues/562
15:21:24 <ayush> @noonedeadpunk : i have not tried, i will try and let you know the results by tmrw
15:21:29 <ayush> mrng
15:22:12 <noonedeadpunk> But, I see no new commits in collection
15:22:21 <noonedeadpunk> And web ui of galaxy doesn't seem broken
15:22:26 <noonedeadpunk> https://galaxy.ansible.com/ui/repo/published/ansible/netcommon/
15:22:45 <noonedeadpunk> So... it's either already fixed... Or API != UI
15:22:50 <NeilHanlon> ope
15:22:51 <NeilHanlon> # https://github.com/ansible/awx/issues/14496
15:22:56 <jrosser> this is "galaxy ng" isnt it?
15:22:57 <NeilHanlon> #link https://github.com/ansible/awx/issues/14496
15:23:13 <noonedeadpunk> "We've had similiar issue today but with 'ansible.netcommon' collection"
15:23:20 <noonedeadpunk> " Removal of default Galaxy credential from organisation settings helped us mitigate the problem."
15:23:24 <NeilHanlon> > However the old-galaxy.ansible.com settings seems to have solved the issue for us ;-)
15:23:40 <noonedeadpunk> Lol
15:23:53 * NeilHanlon adds liquor to his coffee
15:23:59 <noonedeadpunk> This reminds me of some company.... that maintains CentOS Stream...
15:24:08 <noonedeadpunk> What the name was that...
15:24:12 <noonedeadpunk> Anyway
15:24:38 <noonedeadpunk> Might be we can replace galaxy with github or just wait for the fix...
15:25:17 <noonedeadpunk> OR, we need to update ansible version...
15:25:24 <NeilHanlon> i'm gonna ping greg sutcliffe.. get his thoughts :)
15:26:52 <noonedeadpunk> Yeah, would be nice to hear
15:26:57 <NeilHanlon> seems like new ansible won't help
15:27:05 <noonedeadpunk> #link https://bugs.launchpad.net/openstack-ansible/+bug/2037649
15:27:08 <NeilHanlon> #link https://forum.ansible.com/t/awx-cannot-download-collections-roles-from-the-new-galaxy/1251/4?u=neil
15:27:52 <noonedeadpunk> that looks really messy...
15:28:11 <noonedeadpunk> What makes it worse, that in branches like Xena we also had issues with rabbitmq versions...
15:28:19 <NeilHanlon> :(
15:28:19 <NeilHanlon> yeah
15:29:09 <noonedeadpunk> so 2037649 is another race condition bug, but thankfully it affects only vpnaas
15:29:09 <mgariepy> rabbitmq is a mess..
15:29:29 <noonedeadpunk> jamesdenton: you might be interested in https://bugs.launchpad.net/openstack-ansible/+bug/2037649 as well
15:29:58 <noonedeadpunk> I will check what's the best approach, but I guess it would be change of the destination directory to the one inside the venv
15:30:10 <noonedeadpunk> so that symlinking would be not a problem
15:30:18 <noonedeadpunk> #topic office hours
15:30:34 <noonedeadpunk> I failed to book a PTG slots until now :(
15:30:42 <noonedeadpunk> Will try to do that during this week
15:31:11 <noonedeadpunk> Another thing is that we likely should switch from master to tracking stable/2023.2 since all projects have branched already
15:31:40 <noonedeadpunk> Will propose patches soonish
15:31:48 <noonedeadpunk> #action noonedeadpunk book PTG slots
15:32:03 <jrosser> do we have a TODO list for bobcat
15:32:03 <noonedeadpunk> #action noonedeadpunk propose patch to track 2023.2
15:32:10 <jrosser> like actual patches to chase and deal with
15:32:19 <jrosser> seems to be many many open
15:32:54 <jrosser> i think we also need to give the CI some tlc
15:33:11 <jrosser> jobs seem to be running slowly, upgrades are TIMEOUT regularly atm
15:33:32 <noonedeadpunk> I think it's Debian 12, quorum queues at minimum
15:33:39 <jrosser> then there is a fair number of failing to retrieve u-c with https
15:33:39 <noonedeadpunk> we can discuss rest, like Skyline
15:34:15 <jrosser> and I see a number of Rocky repo errors every day still
15:34:33 <noonedeadpunk> But what we tried to scope at the beginning of cycle is here
15:34:35 <noonedeadpunk> #link https://etherpad.opendev.org/p/osa-bobcat-ptg
15:35:02 <noonedeadpunk> Ah, we have openstack-resources role
15:35:26 <noonedeadpunk> #link https://review.opendev.org/q/topic:osa%252Fopenstack_resources
15:35:58 <noonedeadpunk> Ugh, it's failing with same galaxy issue
15:36:33 <NeilHanlon> bleh
15:36:36 <jrosser> oh no curse of the tests repo
15:36:50 <noonedeadpunk> yeah-yeah...
15:37:01 <NeilHanlon> I will look into rocky issues, jrosser. i know of a few lingering problems I'm trying to snuff out.. but also wondering based on what you're saying and I'm seeing if I just need to give the CDN a good ol' flush.
15:37:06 <noonedeadpunk> and modular libvirt would be also a thing nice to have
15:37:13 <NeilHanlon> yea, i have an update on that
15:37:28 <NeilHanlon> It's basically confirmed to be the only thing available in EL10
15:37:50 <NeilHanlon> and ELN composes have begun for what will become CentOS Stream 10
15:37:58 <NeilHanlon> https://composes.stream.centos.org/stream-10/
15:38:31 <noonedeadpunk> so we need to get to bottom of that sooner or later anyway
15:38:49 <noonedeadpunk> And it seems that Debian is not going the same path, at least not with Debian 12?
15:40:32 <NeilHanlon> yeah. i am going to schedule myself some time to come up with a plan, especially around how to migrate/upgrade to modular libvirt
15:41:38 <noonedeadpunk> well. if it's only EL10...
15:41:57 <noonedeadpunk> Maybe no migration needed - you still can't upgrade 9->10
15:42:08 <noonedeadpunk> and then we can wait for 10 for that
15:42:10 <NeilHanlon> true, but i wonder about folks running mixed environments?
15:42:21 <noonedeadpunk> make up some conditions?
15:42:23 <NeilHanlon> i.e., 9 and 10
15:42:41 <NeilHanlon> maybe you're right--we can just not worry about it until we support EL10
15:42:43 <noonedeadpunk> like define services to restart in vars per distro
15:42:54 <NeilHanlon> yeah
15:43:06 <noonedeadpunk> like we have already here: https://opendev.org/openstack/openstack-ansible-os_nova/src/branch/master/vars/redhat.yml#L19-L21
15:43:15 <NeilHanlon> right right
15:43:25 <noonedeadpunk> But I bet it would be way-way more complicated....
15:43:58 <noonedeadpunk> As I'm not sure I understand logic behind this modular thing and how to make tls/non-tls things to work...
15:44:10 <noonedeadpunk> And the order to start/restart
15:45:13 <NeilHanlon> yeah
15:45:23 <NeilHanlon> i'm not looking forward to it, honestly. lol
15:48:13 <noonedeadpunk> so, does the list of Debian 12, quorum queues support, and openstack-resource role (possibly) sounds good enough?
15:48:35 <noonedeadpunk> I was planning also to work on PKI improvement to add vault support, but I'm not sure if I will be able to fit
15:48:46 <noonedeadpunk> And also would better release earlier...
15:50:06 <NeilHanlon> that sounds good to me
15:50:13 <NeilHanlon> would rather less things we're more confident with
15:50:57 <noonedeadpunk> I will also try to work on documentation review. I've already posted some changes, inlcuding some OVN drawing here https://review.opendev.org/c/openstack/openstack-ansible/+/894384
15:51:07 <noonedeadpunk> which would be great to get some feedbacks on...
15:51:24 <noonedeadpunk> And was going to make openstack_user_config examples OVN-friendly
15:51:41 <noonedeadpunk> As I guess most users copying them and running into issues afterwards
15:53:10 <noonedeadpunk> ok, so if that's it
15:53:17 <noonedeadpunk> #endmeeting