19:01:48 <clarkb> #startmeeting infra
19:01:49 <openstack> Meeting started Tue Apr 13 19:01:48 2021 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:50 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:52 <openstack> The meeting name has been set to 'infra'
19:01:55 <clarkb> #link http://lists.opendev.org/pipermail/service-discuss/2021-April/000220.html Our Agenda
19:02:39 <clarkb> #topic Announcements
19:02:49 <clarkb> OpenStack should be finishing its release up tomorrow
19:03:12 <clarkb> and I'm told we should be clear of airship concerns by friday (I'll try to start zk on friday as a result)
19:03:20 <clarkb> keep that in mind as you make changes the next couple of days :)
19:04:25 <clarkb> #topic Actions from last meeting
19:04:31 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-04-06-19.01.txt minutes from last meeting
19:04:44 <clarkb> ianw has had an action to push up changes to retire pbx/asterisk server
19:05:06 <ianw> oh, i didn't get to that
19:05:36 <clarkb> #action ianw propose changes to retire the asterisk server
19:05:43 <clarkb> no worries
19:05:46 <clarkb> #topic Priority Efforts
19:05:51 <clarkb> #topic OpenDev
19:07:19 <clarkb> I completed the external id cleanup of about 224 accounts
19:07:35 <clarkb> I have started the cleanup of another 56 and reached out to users about ~7
19:08:10 <clarkb> I'll plan to complete the 56 account external id cleanups thursday. I gave the users I contacted until the 19th before I would proceed with the described plan
19:08:29 <clarkb> overall good progress. Expecting things may become more difficult soon and need to contact users more frequently
19:09:40 <fungi> i suppose we can also get more aggressive with our recency calculations around those accounts too, and maybe just send users notifications of what we're doing in case they need help adjusting afterward
19:10:04 <clarkb> that is not a bad idea
19:10:18 <clarkb> but we can see what it looks like as we whittle it down further
19:11:06 <fungi> possible as the set shrinks further, new patterns will emerge from what remains
19:11:11 <clarkb> ++
19:11:36 <clarkb> definitely, has helped to do it in batches so far to make those sets of users easier to see
19:12:56 <clarkb> #link https://groups.google.com/g/repo-discuss/c/BQKxAfXBXuo Upstream conversation with people struggling with similar problems.
19:12:56 <clarkb> This was the other thing I noticed. Some other gerrit users describing similar issues to what we saw with gerrit GCing
19:13:06 <clarkb> tldr seems to be use more memory (which we have a plan for now) and reduce resources like ssh threads
19:13:24 <clarkb> I think we should probably stick with the server upgrade as step 1 then if we need to reconsider tuning variables like ssh threads afterwards
19:13:34 <clarkb> but wanted to call that out as there is some interesting conversation going on
19:14:22 <fungi> oh, i've also been trying to spend a little time troubleshooting the update-bug hooks by hand-patching the hook script to write output to /home/gerrit2/review_site/tmp/update-bug.log on the server. seems to work now i just need to push some test changes
19:14:54 <clarkb> did you manage to get tracebacks?
19:15:07 <fungi> not yet, no
19:15:27 <fungi> it's only been a few minutes, and also i just realized i need to also redirect stderr ;)
19:16:07 <clarkb> k, mostly just curious to see what is causing the problem :)
19:16:14 <clarkb> #topic Update Config Management
19:16:24 <clarkb> Lets move along. Any config management updates to call out this week?
19:17:52 <fungi> ianw has encoded our vexxhost ipv6 routes workaround into ansible for the gerrit server replacement
19:18:08 <clarkb> and did it with a verbatim yaml content to write out
19:18:12 <clarkb> which seems sane enough
19:18:27 <ianw> yep, i rebooted the server
19:18:59 <ianw> ... and just checking right now it only has one address, wihch is good
19:20:35 <clarkb> sounds like it is working
19:20:40 <clarkb> #topic General Topics
19:20:47 <clarkb> #topic Server Upgrades
19:21:00 <clarkb> as mentioned earlier my plan is to start the zk cluster rolling replacements friday
19:21:12 <clarkb> should be able to do it then with minimal disruption to release processes
19:21:26 <clarkb> the review replacement continues to make progress
19:21:33 <clarkb> any otherreplacements people are working on right now?
19:21:44 <ianw> only removals :)
19:21:52 <clarkb> which are next on the agenda :)
19:21:57 <clarkb> #topic Retire planet.openstack.org
19:22:25 <clarkb> The email notifying of this plan went out. ianw also recorded the source of blogs in a standard format should there be an attempt to resurrect it later
19:22:40 <clarkb> I think fungi has this news going into the openinfra newsletter too
19:22:50 <ianw> if i cloud get some reviews on
19:22:54 <ianw> #link https://review.opendev.org/q/topic:%22planet-static%22+(status:open%20OR%20status:merged)
19:23:04 <fungi> yeah, it's in the openstack section of the foundation newsletter going out tomorrow
19:23:16 <ianw> that will create a planet.openstack.org redirector to the blank project
19:23:49 <clarkb> I'll review that topic after the meeting
19:24:05 <ianw> i guess i should get that in today so it matches whatever is in newsletter?
19:25:01 <fungi> well, the newsletter refers to the copy attached to your announcement anyway
19:25:34 <fungi> as long as we don't approve any new changes to the planet config it should be consistent regardless, right?
19:26:38 <ianw> well yeah,
19:26:43 <ianw> #link https://review.opendev.org/c/openstack/openstack-planet/+/785995
19:26:51 <ianw> does the actual retirement, after we stop puppeting
19:27:15 <clarkb> ya I think we'll be fine in that case as new stuff won't be able to be added once we stop puppeting (and even then I doubt there are many of those changes)
19:28:52 <clarkb> #topic Retiring survey.openstack.org
19:29:07 <clarkb> I have received notice that survey's ssl cert expires in less than a month
19:29:25 <clarkb> I'm not sure this service ever saw much use and wonder if we can just go ahead and plan to turn it off too
19:29:31 <clarkb> fungi: ^ do you have thoughts as I think you were most involved?
19:30:08 <fungi> yeah, we only ever referred to it as a beta, and i think we can basically do same-week (same-day?) announcement we're discontinuing it
19:30:16 <clarkb> cool
19:30:23 <fungi> i can write up something
19:30:40 <clarkb> thanks, do you think you also want to push the cahnges and hit the claenup button too?
19:30:59 <fungi> sure, figured i'd do all that
19:31:05 <clarkb> thanks!
19:31:18 <clarkb> #action fungi announce survey.o.o retirement and push changes to clean it up
19:31:28 <clarkb> #topic docs-old cleanup
19:31:31 * fungi feels like a man of action now
19:31:37 <clarkb> ianw you reported that ajaeger says cleaning this up is fine
19:31:50 <clarkb> anything else to add on this item or just that it can be done and will be done soon (if not already?)
19:32:03 <ianw> yep in my queue
19:32:27 <ianw> we should also put a robots.txt on static as well, that came up when discussing it
19:32:35 <clarkb> ++
19:32:37 <fungi> oh, right-o
19:32:50 <clarkb> I'll pull this item off the agenda then
19:33:08 <clarkb> #topic PTG Planning
19:33:14 <clarkb> The PTG is next week.
19:33:27 <clarkb> From meetpad's perspective it seems we never cleaned up the extra jvb after the last ptg so we should be fine there
19:33:42 <clarkb> Our blocks of time are Thursday April 22 1400-1600UTC and 2200-0000UTC
19:33:53 <clarkb> feel free to skip if that has you staying up all night or waking up terribly early
19:34:06 <clarkb> the first one should be more Europe friendly and the second more apac friendly
19:34:17 <clarkb> #link https://etherpad.opendev.org/p/apr2021-ptg-opendev
19:34:28 <clarkb> That is our etherpad for notes as well as our meetpad room
19:34:59 <clarkb> I sent email to service-discuss@lists.opendev.org and openstack-discuss inviting others to join us. I primarily see these blocks being used for office hours which I described in those emails
19:35:20 <clarkb> hopeflly we can get some config reviewer volunteers in there to aks any questions, projects wanting to look at jobs, etc
19:35:56 <clarkb> One thing I did want to ask is whether or not we thought we would be around for our weekly meeting next week
19:36:21 <clarkb> http://ptg.openstack.org/ptg.html has the schedule and our 1900 block doesn't conflict with official ptg blocks
19:36:36 <clarkb> But I know that we tend to get uscked in and need a break between the all day meeting blocks.
19:36:36 <fungi> seems pragmatic to skip anyway so that the break can actually be a break
19:36:50 <clarkb> yup
19:37:51 <clarkb> why don't we plan to skip, and if we decide we'd rather meet I can get an agenda sent out
19:38:21 <clarkb> this was another reason I put our time on thursday, I was hoping things would be settled down a bit by then (early ptg week tends to be busy)
19:39:04 <ianw> ++
19:39:40 <clarkb> ok, that was what I had on the agenda
19:39:45 <clarkb> #topic Open Discussion
19:39:47 <clarkb> Anything else?
19:40:35 <fungi> progress on the update-bug front. apparently there are new command-line options passed to hook scripts
19:40:44 <fungi> i have a change in progress to add those now
19:41:16 <fungi> oh, also i updated my stack for the openstack/meta-config project addition, it should hopefully be passing tests now
19:41:18 <clarkb> ah so we just need to accept new parameters, that is easy thankfully
19:41:32 <fungi> yeah, argparse doesn't like being given unrecognized options
19:42:18 <ianw> i'm trying to get some new ARM resources donated by OSU OSL (that is surprisingly easy to typo :) online
19:42:40 <clarkb> looks like some openstacksdk struggles?
19:42:49 <ianw> the mirror is up, but we can't talk to it as it seems the cloud launcher is hitting a bug in openstacksdk setting up the security groups
19:43:16 <clarkb> as a workaround you can probably configure the security groups with openstackclient or neutronclient
19:43:26 <clarkb> and then not bother with cloud launcher until the sdk is fixed
19:43:36 <fungi> or downgrade the sdk?
19:43:40 <ianw> yep, or pin it to an older versions
19:44:18 <ianw> i might as well try and fix it, i don't think anyone else is rushing to :)
19:44:39 <clarkb> ++
19:46:26 <ianw> one other one is
19:46:28 <ianw> #link https://review.opendev.org/c/opendev/system-config/+/778116
19:47:05 <ianw> changes have merged to the upstream zuul summary plugin to have  REST endpoint that projects can enable/disable the summary tab
19:47:35 <fungi> ooh neat
19:47:36 <ianw> as a result the plugin is a .jar with .java bits now, so we need to update as we pull from master
19:47:45 <ianw> s/master/main actually in that case
19:48:12 <clarkb> ianw: will that affect 3.2.8 when we update too? or is that just for 3.3 and newer ? I assume 3.2 as well since it is on the plugin side?
19:48:35 <ianw> clarkb: yeah, 3.2.8 as well, i made a note in your update change iirc
19:48:52 <clarkb> ok, just making sure I understand the implications :) should land yours first then
19:49:08 <clarkb> in fact landing yours may be sufficient since I think mine is a noopy change to rebuild the image
19:50:06 <ianw> possibly, yeah
19:50:43 <clarkb> is this specific to zuul results or the more generic results summary tab?
19:50:51 <clarkb> (curious if this implies lots of people using zuul now )
19:52:55 <clarkb> Sounds like that may be all for the meeting. Thank you for your time, we'll skip next week and hope to see you at the PTG!
19:52:58 <clarkb> #endmeeting