19:01:04 <clarkb> #startmeeting infra
19:01:05 <openstack> Meeting started Tue Apr  6 19:01:04 2021 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:06 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:08 <openstack> The meeting name has been set to 'infra'
19:01:11 <clarkb> #link http://lists.opendev.org/pipermail/service-discuss/2021-April/000203.html Our Agenda
19:01:17 <clarkb> #topic Announcements
19:01:30 <clarkb> OpenStack is due to produce its final RCs this week. Being slushy is probably a good thing?
19:01:42 <clarkb> I believe that airship is also in a similar spot of their release process
19:02:04 <clarkb> I think we can still make changes if we need to but we should double check with the openstack release team and airship if we can before zuul restarts etc
19:02:42 <clarkb> #topic Actions from last meeting
19:02:53 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-03-30-19.01.txt minutes from last meeting
19:04:23 <clarkb> ianw had actions to propose planet retirement
19:04:31 <clarkb> I saw that email go out and haven't seen anyone complain.
19:04:45 <clarkb> The bulk of the posts there are superuser and they advertise them elsewhere so I suspect this is fine
19:04:58 <clarkb> ianw also had an action to propose changes to retire pbx which i haven't seen yet
19:05:03 <clarkb> #action ianw Propose changes to retire pbx
19:05:16 <fungi> i think he said he'd be out this week though
19:05:34 <clarkb> ya
19:05:44 <clarkb> #topic Priority Efforts
19:05:50 <clarkb> #topic OpenDev
19:05:55 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/784152 Upgrade Gerrit to 3.2.8
19:06:13 <clarkb> This change will help us keep up with upstream's release. I don't think this is urgent and likely one to wait for release work to finish up
19:06:19 <clarkb> but you can review it now :)
19:07:28 <clarkb> For account inconsistencies I retired 224 accounts last week
19:07:39 <fungi> that's a sizeable dent!
19:07:47 <clarkb> My plan was to let them sit for a few days then run the external id cleanup script against them. Probably tomorrow or thursday to run that script
19:07:58 <clarkb> that should hopefully get us about halfway through the external id conflict list when done
19:10:26 <clarkb> #topic Update Configuration Management
19:10:34 <clarkb> I'm not aware of any new items on this list. fungi are you?
19:10:45 <clarkb> I have seen the storyboard story related to this update occasionally
19:11:35 <fungi> yeah, i got a notification it had some updates, i haven't looked yet
19:11:39 <clarkb> k
19:11:46 <clarkb> #topic General topics
19:11:53 <clarkb> #topic Server Upgrades
19:12:16 <clarkb> The zk work sort of stalled out due to the zuul problems with memory leaks and the now release work. I did draft up a doc about our options though
19:12:22 <clarkb> #link https://etherpad.opendev.org/p/opendev-zookeeper-upgrade-2021 discussion of options for zk upgrades
19:12:40 <clarkb> If infra-root ^ can look that over then give me opinions I will be grateful
19:12:45 <fungi> thankfully the semaphore leak on the scheduler seems to be cleared up now
19:12:52 <fungi> smooth sailing
19:14:11 <clarkb> #topic PTG planning
19:14:33 <clarkb> As mentioned previously I requested PTG time which I expect will be used for config-core bootstrapping and general office hours
19:14:43 <clarkb> I should send email to service-discuss about this and encourage people to join us
19:14:51 <clarkb> Thursday April 22 1400-1600UTC and 2200-0000UTC <- our blocks of time
19:15:50 <clarkb> The next two topic items largely depend on updates from ianw so I'll call them out but don't expect responses
19:15:58 <clarkb> #topic docs-old volume cleanup
19:16:10 <clarkb> Idea was to see if AJaeger had an opinion on this as a sanity check.
19:16:22 <clarkb> #topic planet.openstack.org retirement
19:16:38 <clarkb> As mentioned previously the email to propose retirement went out. The bulk of the posts are superuser
19:16:49 <clarkb> there are some other people getting aggregated there, but overall I suspect this is fine
19:16:57 <clarkb> #topic tarballs ORD replication
19:17:09 <clarkb> This happened. Initial sync was a bit slow (~2 days?)
19:17:29 <clarkb> fungi: you've switched the webserver from reading the RO replica again though and incremental updates are reasoable?
19:18:06 <fungi> yes
19:18:25 <fungi> incremental sync only seems to take a few minutes for reasonable size deltas
19:18:40 <clarkb> excellent, I suspect I can drop this item from next week's meeting agenda, but will dobule check with ianw first
19:19:05 <fungi> i haven't checked that the new replica is healthy
19:19:12 <fungi> been too many other fires
19:19:23 <clarkb> I guess reasonable release times don't imply that?
19:20:10 <clarkb> I tend to associate slow vos releases with either lots of delta or unhappy volumes
19:20:11 <fungi> i assume, but don't know for sure that afs isn't simply skipping ord for some reason
19:20:14 <clarkb> (maybe I shouldnt)
19:20:17 <clarkb> gotcha
19:22:06 <clarkb> #topic Open Discussion
19:22:32 <clarkb> I went through that quickly since it appears to just be fungi and myself. But feel free to bring anything else up now. (or continue previous topic discussion)
19:22:45 <clarkb> If we're quiet I'll stop the meeting at 19:30UTC
19:23:10 <fungi> sounds good. i can be quiet a little longer ;)
19:23:40 <clarkb> feedback on the zk stuff would be great. I can start prep work of booting new instances once we havedecided on the method we want to use
19:23:51 <corvus> clarkb: both of those zk options sound good; i don't really have an opinion
19:23:56 <clarkb> corvus: ok
19:23:59 <corvus> i'd say do whatever feels best to you :)
19:24:25 <corvus> (hopefully "i don't see a technical reason to prefer one over the other" is at least mildly useful input)
19:24:26 <clarkb> my preference currently is to deploy zk04-06.opendev.org and avoid any complicated manual syncing of data
19:24:41 <fungi> wfm
19:24:47 <fungi> i assume we have sufficient quota
19:24:54 <clarkb> since no one seems to object to that plan I'll proceed with that
19:24:57 <clarkb> fungi: I can cehck that too
19:25:00 <clarkb> I believe we do though
19:25:11 <clarkb> they are 4GB instances iirc so only need 12GB memory headroom
19:25:24 <clarkb> and I was doing that with the zuul executor replacements without issue
19:25:57 <clarkb> Then I guess if releases have moved along far enough try and start swapping things fridayish
19:26:44 <corvus> as long as it's you swapping and not the vms running out of ram :)
19:27:00 <clarkb> ++ :)
19:27:06 * fungi pages out for the day
19:27:35 <fungi> (no not really, i still have plenty of active memory)
19:30:27 <clarkb> I'll call it here. Thank you everyone. We'll see you here next week. Then may cancel the week after for the PTG (if people have conflicts, we'll see)
19:30:30 <clarkb> #endmeeting