19:00:10 #startmeeting infra 19:00:10 Meeting started Tue Jan 30 19:00:10 2024 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:10 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:00:10 The meeting name has been set to 'infra' 19:01:18 #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/L6CVNQQLCEZCK3I7I7FQUC7VV2UWSBHM/ Our Agenda 19:01:24 #topic Announcements 19:01:31 Service coordinator nominations open February 6, 2024 - February 20, 2024 19:01:36 #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/TB2OFBIGWZEYC7L4MCYA46EXIX5T47TY/ 19:01:38 That is one week away 19:01:41 rouhgly 19:02:00 OpenDev PrePTG February 14+15 at 14:00-16:00UTC and 22:00-00:00UTC both days on our meetpad service. 19:02:06 #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/KUZG7DPU77GC3M3MKKTSYY655JLIYV4Z/ 19:02:21 and the Pre PTG is two weeks away 19:02:50 #topic Server Upgrades 19:03:03 #link https://review.opendev.org/c/opendev/system-config/+/905510 Upgrading meetpad service to jammy 19:03:25 fungi: frickler maybe you have time to review those changes? Otherwise I think I may work with tonyb and proceed with things with just my reviews 19:03:49 (which coincidentally is one of the agenda items for the pre ptg, whether or not we want to do more singe reviewer approvals) 19:04:56 #link https://etherpad.opendev.org/p/opendev-bionic-server-upgrades 19:05:13 this etherpad also has notes on a wiki replacement plan. If you haven't read them that may be a good idea. 19:05:43 * frickler makes notes to take a look, also at those reviews 19:06:43 thanks! 19:06:50 #topic Python container updates 19:07:34 Nothing new to mention here. I think we're pretty much stalled out on the zuul oeprator stuff. But it isn't urgent. I may just remove this from the next meeting's agenda 19:08:40 #topic AFS quota issues 19:09:01 somewhere on my todo list that keeps getting overlooked is ubuntu ports cleanup. /me adds it again and maybe it will get looked at 19:10:35 #topic Broken wheel build issues 19:10:48 the wheels are building and being written to afs now 19:10:58 The remaining problem has to do with volume selection for vos release 19:11:04 #link https://review.opendev.org/c/openstack/project-config/+/907150 Will fix centos 8 and 9 stream 19:11:29 In theory that change will have fixed these problems and everything centos 8/9 stream wheel related will be happy 19:11:43 I did notice thatwhile centos 7 was working when this first came up it is not working now 19:11:47 I haven't had a chance to look at it 19:12:57 one thing I noticed is that I'm not sure these wheels were ever really working properly for stream 19:13:15 that may put more weight into the "maybe we should just shut it down side of things" 19:13:49 however if things work now then letting it continue to be happy is least effort 19:14:23 til next time it breaks ;) 19:14:24 #topic Keycloak upgrade struggles 19:14:28 ya exactly :) 19:15:23 recently fungi and I started poking at keycloak upgrades. A little while back keycloak moved from wildfly as its runtime to quarkus. When they did this they didn't really make the container image migration a simple direct one 19:15:40 this resulted in us getting stuck behind on a legacy set of images and now we are trying to figure out how to get onto the new quarkus stuff 19:15:48 but also some useful discoveries 19:16:06 The biggest issue we've run into is that the existing service uses an h2 databse but keycloak considers h2 to not be production worthy so there is no migration path for that 19:16:14 file paths and names and db credentials have all changed 19:16:35 this seems to be a clear sign we should move to a real db so that is what fungi has been doing most recently. Getting mariadb up alongside keyloack 19:16:40 i did try some basic migration which made it into production briefly over the weekend before i rolled it back due to breakage 19:16:58 ya including renaming and moving files 19:16:58 it would be good for someone who has successfully used keycloak.opendev.org recently to test that it's working as expected again 19:17:45 yep wfm 19:18:03 anyway, for a number of reasons, it appears that trying to "migrate" our existing data will be extremely involved (switching database format, upgrading through several keycloak releases, also we should move to latest ubuntu lts while we're at it) 19:18:09 and me 19:18:52 given that the service is used by at most half a dozen people in a non-production-critical path right now, it will probably be less work to set it up from scratch on the latest keycloak images on new ubuntu 19:18:56 ya the end result of all this wheel spinning seems to be we should do a new deployment with that we have laerned then rebuild the currently small set of integration we have using it 19:19:09 then we should be in a much better position for the future and being able to upgrade things 19:19:17 i've got a change underway (close to complete i think) to do that 19:19:28 just adding some more hygiene to the testing now 19:19:46 like switching to group_vars and using keycloak99 for the test node, etc 19:20:39 tests so far are passing, i've been adding some safety like making sure keycloak and mariadb only listen on the loopback address 19:21:23 anyway, hopefully we can have a new keycloak02.opendev.org up by the time for our pre-ptg and start talking about moving forward some more on the sso spec 19:21:30 sounds good 19:21:37 +1 19:22:25 i didn't have anything else for this topic, but maybe folks have questions 19:22:39 plan sgtm, thanks! 19:23:07 #topic InMotion/OpenMetal Cloud RabbitMQ Problems 19:23:22 wanted to follow up on this. frickler's hunch was correct. rabbitmq had a sad 19:23:49 tonyb: can speak more to this than I can but initially a simple docker restart of rabbitmq containers was done to try and get rabbit to cooperate and that didn't help at all 19:24:08 then later a restart using kolla tooling was done and that got things working again 19:24:24 however there are still a number of leaked nodes we need to clean up which likely involve deeper admin work 19:24:50 Yup that one. 19:24:51 one thing I have had to do in the past to make nodes deletable is to reset their task state if they are already in a deleting task state because nova won't retry if task state is already deleting 19:25:10 then there is another issue we've encountered which had to do with a mismatch in cell accounting 19:25:28 melwitt helped the cells issue if we need help again 19:26:08 I'll get more details and then reach out on nova/melwitt 19:26:39 in the next 24-48 hours e should have things "sorted" 19:27:20 thanks! 19:27:29 #topic Open Discussion 19:27:44 That was all I had on the agenda. A reminder to add your content and interest to the pre ptg etherpad 19:29:00 #link https://etherpad.opendev.org/p/opendev-preptg-202402 19:31:18 sounds like that may be everything 19:31:26 a rare short meeting. THank you for your time everyone! 19:31:38 I'll close this out in a few more minutes if nothing else pops up 19:31:41 thanks clarkb! 19:31:59 Thanks all! 19:33:07 thanks clarkb 19:35:21 #endmeeting