19:01:48 #startmeeting infra 19:01:49 Meeting started Tue Apr 13 19:01:48 2021 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:50 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:52 The meeting name has been set to 'infra' 19:01:55 #link http://lists.opendev.org/pipermail/service-discuss/2021-April/000220.html Our Agenda 19:02:39 #topic Announcements 19:02:49 OpenStack should be finishing its release up tomorrow 19:03:12 and I'm told we should be clear of airship concerns by friday (I'll try to start zk on friday as a result) 19:03:20 keep that in mind as you make changes the next couple of days :) 19:04:25 #topic Actions from last meeting 19:04:31 #link http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-04-06-19.01.txt minutes from last meeting 19:04:44 ianw has had an action to push up changes to retire pbx/asterisk server 19:05:06 oh, i didn't get to that 19:05:36 #action ianw propose changes to retire the asterisk server 19:05:43 no worries 19:05:46 #topic Priority Efforts 19:05:51 #topic OpenDev 19:07:19 I completed the external id cleanup of about 224 accounts 19:07:35 I have started the cleanup of another 56 and reached out to users about ~7 19:08:10 I'll plan to complete the 56 account external id cleanups thursday. I gave the users I contacted until the 19th before I would proceed with the described plan 19:08:29 overall good progress. Expecting things may become more difficult soon and need to contact users more frequently 19:09:40 i suppose we can also get more aggressive with our recency calculations around those accounts too, and maybe just send users notifications of what we're doing in case they need help adjusting afterward 19:10:04 that is not a bad idea 19:10:18 but we can see what it looks like as we whittle it down further 19:11:06 possible as the set shrinks further, new patterns will emerge from what remains 19:11:11 ++ 19:11:36 definitely, has helped to do it in batches so far to make those sets of users easier to see 19:12:56 #link https://groups.google.com/g/repo-discuss/c/BQKxAfXBXuo Upstream conversation with people struggling with similar problems. 19:12:56 This was the other thing I noticed. Some other gerrit users describing similar issues to what we saw with gerrit GCing 19:13:06 tldr seems to be use more memory (which we have a plan for now) and reduce resources like ssh threads 19:13:24 I think we should probably stick with the server upgrade as step 1 then if we need to reconsider tuning variables like ssh threads afterwards 19:13:34 but wanted to call that out as there is some interesting conversation going on 19:14:22 oh, i've also been trying to spend a little time troubleshooting the update-bug hooks by hand-patching the hook script to write output to /home/gerrit2/review_site/tmp/update-bug.log on the server. seems to work now i just need to push some test changes 19:14:54 did you manage to get tracebacks? 19:15:07 not yet, no 19:15:27 it's only been a few minutes, and also i just realized i need to also redirect stderr ;) 19:16:07 k, mostly just curious to see what is causing the problem :) 19:16:14 #topic Update Config Management 19:16:24 Lets move along. Any config management updates to call out this week? 19:17:52 ianw has encoded our vexxhost ipv6 routes workaround into ansible for the gerrit server replacement 19:18:08 and did it with a verbatim yaml content to write out 19:18:12 which seems sane enough 19:18:27 yep, i rebooted the server 19:18:59 ... and just checking right now it only has one address, wihch is good 19:20:35 sounds like it is working 19:20:40 #topic General Topics 19:20:47 #topic Server Upgrades 19:21:00 as mentioned earlier my plan is to start the zk cluster rolling replacements friday 19:21:12 should be able to do it then with minimal disruption to release processes 19:21:26 the review replacement continues to make progress 19:21:33 any otherreplacements people are working on right now? 19:21:44 only removals :) 19:21:52 which are next on the agenda :) 19:21:57 #topic Retire planet.openstack.org 19:22:25 The email notifying of this plan went out. ianw also recorded the source of blogs in a standard format should there be an attempt to resurrect it later 19:22:40 I think fungi has this news going into the openinfra newsletter too 19:22:50 if i cloud get some reviews on 19:22:54 #link https://review.opendev.org/q/topic:%22planet-static%22+(status:open%20OR%20status:merged) 19:23:04 yeah, it's in the openstack section of the foundation newsletter going out tomorrow 19:23:16 that will create a planet.openstack.org redirector to the blank project 19:23:49 I'll review that topic after the meeting 19:24:05 i guess i should get that in today so it matches whatever is in newsletter? 19:25:01 well, the newsletter refers to the copy attached to your announcement anyway 19:25:34 as long as we don't approve any new changes to the planet config it should be consistent regardless, right? 19:26:38 well yeah, 19:26:43 #link https://review.opendev.org/c/openstack/openstack-planet/+/785995 19:26:51 does the actual retirement, after we stop puppeting 19:27:15 ya I think we'll be fine in that case as new stuff won't be able to be added once we stop puppeting (and even then I doubt there are many of those changes) 19:28:52 #topic Retiring survey.openstack.org 19:29:07 I have received notice that survey's ssl cert expires in less than a month 19:29:25 I'm not sure this service ever saw much use and wonder if we can just go ahead and plan to turn it off too 19:29:31 fungi: ^ do you have thoughts as I think you were most involved? 19:30:08 yeah, we only ever referred to it as a beta, and i think we can basically do same-week (same-day?) announcement we're discontinuing it 19:30:16 cool 19:30:23 i can write up something 19:30:40 thanks, do you think you also want to push the cahnges and hit the claenup button too? 19:30:59 sure, figured i'd do all that 19:31:05 thanks! 19:31:18 #action fungi announce survey.o.o retirement and push changes to clean it up 19:31:28 #topic docs-old cleanup 19:31:31 * fungi feels like a man of action now 19:31:37 ianw you reported that ajaeger says cleaning this up is fine 19:31:50 anything else to add on this item or just that it can be done and will be done soon (if not already?) 19:32:03 yep in my queue 19:32:27 we should also put a robots.txt on static as well, that came up when discussing it 19:32:35 ++ 19:32:37 oh, right-o 19:32:50 I'll pull this item off the agenda then 19:33:08 #topic PTG Planning 19:33:14 The PTG is next week. 19:33:27 From meetpad's perspective it seems we never cleaned up the extra jvb after the last ptg so we should be fine there 19:33:42 Our blocks of time are Thursday April 22 1400-1600UTC and 2200-0000UTC 19:33:53 feel free to skip if that has you staying up all night or waking up terribly early 19:34:06 the first one should be more Europe friendly and the second more apac friendly 19:34:17 #link https://etherpad.opendev.org/p/apr2021-ptg-opendev 19:34:28 That is our etherpad for notes as well as our meetpad room 19:34:59 I sent email to service-discuss@lists.opendev.org and openstack-discuss inviting others to join us. I primarily see these blocks being used for office hours which I described in those emails 19:35:20 hopeflly we can get some config reviewer volunteers in there to aks any questions, projects wanting to look at jobs, etc 19:35:56 One thing I did want to ask is whether or not we thought we would be around for our weekly meeting next week 19:36:21 http://ptg.openstack.org/ptg.html has the schedule and our 1900 block doesn't conflict with official ptg blocks 19:36:36 But I know that we tend to get uscked in and need a break between the all day meeting blocks. 19:36:36 seems pragmatic to skip anyway so that the break can actually be a break 19:36:50 yup 19:37:51 why don't we plan to skip, and if we decide we'd rather meet I can get an agenda sent out 19:38:21 this was another reason I put our time on thursday, I was hoping things would be settled down a bit by then (early ptg week tends to be busy) 19:39:04 ++ 19:39:40 ok, that was what I had on the agenda 19:39:45 #topic Open Discussion 19:39:47 Anything else? 19:40:35 progress on the update-bug front. apparently there are new command-line options passed to hook scripts 19:40:44 i have a change in progress to add those now 19:41:16 oh, also i updated my stack for the openstack/meta-config project addition, it should hopefully be passing tests now 19:41:18 ah so we just need to accept new parameters, that is easy thankfully 19:41:32 yeah, argparse doesn't like being given unrecognized options 19:42:18 i'm trying to get some new ARM resources donated by OSU OSL (that is surprisingly easy to typo :) online 19:42:40 looks like some openstacksdk struggles? 19:42:49 the mirror is up, but we can't talk to it as it seems the cloud launcher is hitting a bug in openstacksdk setting up the security groups 19:43:16 as a workaround you can probably configure the security groups with openstackclient or neutronclient 19:43:26 and then not bother with cloud launcher until the sdk is fixed 19:43:36 or downgrade the sdk? 19:43:40 yep, or pin it to an older versions 19:44:18 i might as well try and fix it, i don't think anyone else is rushing to :) 19:44:39 ++ 19:46:26 one other one is 19:46:28 #link https://review.opendev.org/c/opendev/system-config/+/778116 19:47:05 changes have merged to the upstream zuul summary plugin to have REST endpoint that projects can enable/disable the summary tab 19:47:35 ooh neat 19:47:36 as a result the plugin is a .jar with .java bits now, so we need to update as we pull from master 19:47:45 s/master/main actually in that case 19:48:12 ianw: will that affect 3.2.8 when we update too? or is that just for 3.3 and newer ? I assume 3.2 as well since it is on the plugin side? 19:48:35 clarkb: yeah, 3.2.8 as well, i made a note in your update change iirc 19:48:52 ok, just making sure I understand the implications :) should land yours first then 19:49:08 in fact landing yours may be sufficient since I think mine is a noopy change to rebuild the image 19:50:06 possibly, yeah 19:50:43 is this specific to zuul results or the more generic results summary tab? 19:50:51 (curious if this implies lots of people using zuul now ) 19:52:55 Sounds like that may be all for the meeting. Thank you for your time, we'll skip next week and hope to see you at the PTG! 19:52:58 #endmeeting