19:01:07 #startmeeting Infra 19:01:08 Meeting started Tue Jan 12 19:01:07 2021 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:09 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:11 The meeting name has been set to 'infra' 19:01:27 #topic Announcements 19:01:53 The foundation's board of director elections are happening this week. If you are a foundation member you shoulve have received an email with your voting instructions in it 19:02:07 Please take a few minutes to go and vote if you are able to 19:02:17 #topic Actions from last meeting 19:02:23 #link http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-01-05-19.01.txt minutes from last meeting 19:02:51 corvus had an action to implement Gerrit WIP support in Zuul. I believe the change was written and I reviewed it. corvus if you are around has that merged yet? 19:03:12 if not, i'd appreciate a link so i can review 19:03:18 ya trying to find that now 19:03:38 #link https://review.opendev.org/c/zuul/zuul/+/769436 looks like it merged according to gerritbot which is where I Found the link 19:03:43 awesome 19:03:51 so we need a restart to pick that up 19:03:53 that means the next step for us in supporting WIP is to restart the zuul scheduler and then test again 19:03:55 yup 19:04:02 at least the scheduler, presumably 19:04:14 ahh, you said those things 19:04:26 * fungi is a bit scattered today 19:04:31 me too 19:04:37 #topic Priority Efforts 19:04:42 #topic OpenDev 19:05:12 First up it was pointed out that we were overdue for service coordinator nominations (and an election if necesasry) 19:05:20 #link http://lists.opendev.org/pipermail/service-discuss/2021-January/000161.html 19:05:34 I sent email dscribing what I think is a reasonable plan for addressing this miss 19:05:55 if you think that plan is flawed in some way please respond to the mailing list and help us set up a new less flawed plan :) 19:06:09 any other election feedback is also appreciated there. 19:06:39 If you are interested in taking on the service coordinator role I'm happy to talk about it if you need more info before committing. Otherwise please send email to the service-discuss list nominating yourself 19:07:01 as mentioned in the email I linked I've done it for a number of cycles now and think that new perspectives would be a good thing to have. 19:07:19 I'm not saying I won't run again if necessary, but do strongly feel that having a bit more rotation would be a good thing 19:07:32 prepare for zbr to volunteer 19:08:04 zbr: you know you want to! 19:08:33 the email I sent also sets up a week for electiosn if they become necessary. And to avoid this problem of missing them in the future I set out dates for the next set of nominations and elections 19:09:05 I'll add them into my calendar reminders if later if no one ends up objecting to that proposed plan 19:09:38 The opendev project update for the foundation's annual report is basically finalized now 19:09:40 #link https://etherpad.opendev.org/p/opendev-2020-annual-report 19:09:52 I think those are due tomorrow so if there are important edits get them in now (or let me know and I can make them) 19:10:40 The last opendev topic I wanted to bring up was the gitea 1.13.1 upgrade. 19:10:42 #link https://review.opendev.org/c/opendev/system-config/+/769226 19:10:43 lgtm 19:11:19 I think this is ready for serious consideration. There is a held gitea test node (somewhere I need to find it again via nodepool) running 1.13.1 to help confirm we're happy with it 19:11:46 my biggest concern is that gitea 1.13 added some big new features like kanban boards to projects so want to make sure we're presenting a gitea that is consistent with our current setup 19:12:07 patch itself looks good - assuming the test node is solid 19:12:25 ya I looked it over (and I think fungi did too?) and it seemed fine 19:12:35 i did, think i already +2'd 19:12:56 if it does look good I should have plenty of time to watch it land and monitor it thursday 19:13:02 trying to get through some project-config backlog today, i noticed this one has a potential for global disruption but would like to merge and watch it closely after the meeting: 19:13:06 #link https://review.opendev.org/760495 Use internal mirror for RAX IAD/DFW 19:13:33 that switches the interface for mirror connections in two regions to use the second nic where bandwidth may be less constrained 19:13:39 ++ 19:13:44 (we're already doing it that way in one) 19:14:16 ++ 19:14:39 might be worth simplifying that to "if nodepool.cloud == 'rax'" at this point 19:14:57 true 19:15:01 ianw: ^ wdyt? 19:15:23 not that I expect us to grow new rax regions or anything 19:15:46 i'm happy to tweak it before approving if folks prefer 19:15:47 yeah, i don't mind; we have used this before for switching in other clouds so it might be useful to keep the cloud/region append just as an example of what to do 19:16:13 i'm good with it as-is too 19:16:58 why don't we land it as is then just to avoid unnecessary churn 19:17:07 wfm, will do 19:17:36 #topic General Topics 19:17:43 #topic Bup and Borg Backups 19:18:01 ianw: this was on my list to check up on after the holidays. Are we completely off of bup at this point for new backups? 19:18:12 and if so should I drop this item from our meeting agendas? 19:18:28 yep, since https://review.opendev.org/c/opendev/system-config/+/766300 19:18:44 i still have to finish the cleanup with https://review.opendev.org/c/opendev/system-config/+/766630/ 19:19:23 i will work on that 19:19:26 thanks 19:19:31 and thank you for working on that 19:19:41 we should be able to start looking at focal nodes now I think 19:19:47 (since this was a big hold up for that iirc) 19:20:03 maybe keep it for one more week as i cleanup the old servers 19:20:12 can do 19:20:23 i can't remember, was borg manually added to the wiki server? if not, i'll try to prioritize that 19:20:33 fungi: I'm not sure 19:20:39 probably not? 19:20:50 fungi: not sure either. i don't remember doing it. i can look into that 19:21:02 it was being backed up with bup 19:21:08 (still is afaik) 19:21:11 it likely still is ya 19:21:16 since the bup bits are still there iirc 19:21:24 (we have to keep them around for backup retention anyway) 19:22:26 maybe give me an action item to confirm wiki being backed up so we don't forget 19:22:41 #action ianw confirm wiki is still backed up after bup to borg migration 19:23:13 #topic openstackid.org scale down 19:24:02 fungi and I conferred with smarcet to confirm that the desired running state for openstackid.org is to scale it down after we scaled it up for the summit 19:24:41 fungi returned openstackid.org to its ansible + puppet managed state and I have shut down the two new servers in vexxhost and removed their A and AAAA dns records 19:24:55 in a day or two when we're happy that we've cleaned things up without disruption I will delete them 19:25:29 we also discovered that smarcet uses docker for openstackid development so it sounds like we can collaborate to convert that over to a ansible + docker-compose + docker deployment 19:25:41 however, that is more of a "yes we can do that" idea at this point 19:26:04 mostly an fyi on that since that service had some changes and now we've undone them. And now everyone else is caught up 19:26:12 #topic Open Discussion 19:26:40 That was what I had written down in notes really quickly after my previous meetings ended. We've got plenty of time to tlk about other topics if we need to 19:27:03 oh! 19:27:13 https://bugs.chromium.org/p/gerrit/issues/detail?id=13930 is worth pointing out 19:27:26 I think I managed to figure out why fedora 33 users are still having trouble with new gerrit and rsa 19:27:57 long story short is upstream openssh has only deprecated ssh-rsa for hostkey signature exchanges. Fedora has disabled it for hostkey stuff and for public key auth 19:28:38 Gerrit does rsa-sha2-* just fine for hostkey stuff but it does not work for pubkey auth because that requires supporting the server-sig-algs kex extension in the server and gerrit's server doesn't seem to support that 19:28:41 oh sorry i got sucked into an issue 19:29:04 fedora 33 users can work around this by enabling ssh-rsa or switching to an ed25119 or ecdsa key for auth 19:29:21 i feel like i'm using fedora 33 and it is working 19:29:31 ohhh, i'm using and ed25119 key 19:29:41 ya this is specific to using rsa keys to auth 19:29:48 ianw: see, you anticipated this 19:30:32 if fedora users have qusetions we can point them to that bug. I'm hoping upstream will say "oh thats an easy fix" and it will magically happen but I think it may be more involved 19:30:45 heh, yep, istr having to merge some changes to our puppet to handle ed keys years ago :) 19:30:46 in particular I think the proper way to fix this is to update mina upstream since other mina sshds will want the same fix 19:31:31 eventually ssh clients should switch their fallback rsa pubkey auth type to rsa-sha2-something. But until that happens I expect this will be a problem for people 19:31:44 related: if anyone knows fedora devs ^ it might be worth suggesting they make that switch 19:31:52 since they are disabling the alternative 19:31:59 well, and also fedora could improve the situation by not still looking for sha-1 with ssh-rsa and instead trying sha-2 first 19:32:09 er, what you also just typed 19:32:25 ya it feels like fedora's disabling of ssh-rsa missed an important step 19:32:46 which was to not fallback to ssh-rsa when doing rsa pubkey auth talking to a server that doesn't do server-sig-algs 19:33:21 https://tools.ietf.org/html/rfc8332#section-3.3 notes that this is the expected end state once rsa-sha2 is sufficiently ubiquitous (and it seems fedora is saying that it is) 19:34:28 is it https://bugzilla.redhat.com/show_bug.cgi?id=1881301 ? 19:34:31 bugzilla.redhat.com bug 1881301 in openssh "openssh-clients do not accept PubkeyAcceptedKeyTypes rsa-sha2-512/256" [Unspecified,Closed: errata] - Assigned to jjelen 19:35:03 ianw: ya I think that is the bug on the fedora side 19:36:28 i'd like to get the zuul summary plugin going 19:36:32 the review stack is @ https://review.opendev.org/q/topic:%22gerrit-admin-user%22 19:37:12 ianw: and that plugin is hosted upstream now too right? 19:37:28 yes, that's right 19:37:57 excellent I'll add that very high on the review todo list once I've got time to do that (probably tomorrow?) 19:38:45 ok, yeah the stuff underneath is to cleanup review-dev, then initalize and populate gerrit during testing, then add selenium testing and take screenshots, and then finally add the plugin 19:39:23 with a little bazelisk stuff for good measure :) 19:41:11 alright anything else? 19:41:14 i am back 19:41:24 (reading backlog) 19:42:22 zbr: just me encouraging folks to make our service coordinator election an election this time 19:42:30 I'll give zbr a coupel of minutes to catch up but then if that is it we can call it a meeting 19:42:45 (was the nick highlight i mean) 19:45:03 sure. ok to call the meeting off. 19:45:28 thanks everyone. Sorry I missed the agenda. I'll do my best to not dismiss the alert until actually done in the future :) 19:45:46 I think what happened was I habitually swiped it away when my phone made noise and had a thing pop up 19:45:52 #endmeeting