19:01:19 #startmeeting infra 19:01:20 Meeting started Tue Jun 1 19:01:19 2021 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:21 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:23 The meeting name has been set to 'infra' 19:01:29 #link http://lists.opendev.org/pipermail/service-discuss/2021-June/000252.html Our Agenda 19:01:37 #topic Announcements 19:02:42 I didn't have any announcements. 19:02:52 welcome to oftc? 19:02:58 It is a later topic in the meeting but ya we are here on oftc now 19:03:13 I know a few of us are still hanging around in a few freenode channels but I'll probably clean those up next week 19:03:15 hope everyone found us okay 19:03:39 and yeah, we can discuss later in the meeting 19:03:39 *clean those up in my client. 19:04:23 But ya welcome and let us know if you have qusetions/concerns/problems with the switch 19:04:27 #topic Actions from last meeting 19:04:32 #link http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-05-25-19.01.txt minutes from last meeting 19:04:47 There were no actions and even if there were I think we ended up fairly swamped by irc and related tasks 19:04:58 a swamp it was 19:05:09 #topic Topics 19:05:24 how topical 19:05:25 The most generic of topics. I reorged the meeting agenda as discussed last week 19:05:32 #topic Switch to OFTC now largely complete 19:05:57 there's some patches up to convert ptgbot 19:05:57 fungi: want to give us an update on where we are with the switch and any outstanding issues we need to clean up? 19:06:25 and someone still needs to follow up on the #openstack-sahara registration 19:06:44 i also have a patch out there to drop #edeploy from the statusbot config 19:07:03 #link https://review.opendev.org/c/openstack/ptgbot/+/793792 ptgbot oftc conversion 19:07:36 #link https://review.opendev.org/c/opendev/system-config/+/793813 drop edeploy channel from statusbot config 19:07:49 on the topic of statusbot, we've seen an unexplained crash but the way it logs fails to capture exceptions/tracebacks, so it's running foreground currently in a root screen session on eavesdrop with an su to the ptgbot account 19:08:08 fungi: and I guess we'll keep running it that way in the hopes of catching the crash? 19:08:27 yeah 19:08:29 likely some unforeseen problem in the hurried python3 conversion we did for it over the weekend 19:09:01 and ianw has been tackling the converged bot/limnoria spec, has a poc up i believe though i haven't had time to look 19:09:05 fungi: for sahara we need them to give accessbot master perms via chanserv? 19:09:13 #link https://review.opendev.org/q/topic:%22limnoria%22+status:open Limnoria bot rewrite 19:09:16 correct on #openstack-sahara 19:09:45 ianw: based on the commits there you're porting the existing supybot config to limnoria? 19:10:07 the pending patch to add #openstack-sahara back to acessbot and gerritbot is... 19:10:18 #link https://review.opendev.org/792857 Revert "Accessbot OFTC channel list stopgap" 19:10:20 that stack (completely uncommented) is intended to port meetbot to python3/limnoria 19:10:35 ianw: nice, I'll try to take a look today 19:10:39 however limnoria is so far exactly the same as supybot 19:10:54 i will clean up that stack iwth actual change comments 19:11:01 ianw: sounds good 19:11:15 it's probably more useful to have a poke at it in #opendev-sandbox atm and see what's right/wrong 19:11:29 ah ok interactive debugging more so than code review 19:11:56 yeah, the other part is 19:11:56 oh, that reminds me, the supybot package on eavesdrop has one of its files edited to basically roll in a newer (but many years old) commit which never made it into ubuntu 19:12:01 #link https://review.opendev.org/c/opendev/system-config/+/793704 19:12:13 which builds a limnoria/meetbot container 19:12:16 which makes the limnoria work slightly more important 19:12:21 fungi: ++ 19:12:31 #link https://review.opendev.org/c/opendev/system-config/+/793719/8 19:12:36 is then installing that 19:12:57 i've tried my best to port over the existing config, but there's *a lot* of options in there 19:13:29 small details all still WIP, but big picture comments definitely useful at this point. 19:13:51 cool, I will attempt to interact with the bot and see if that produces any feedback 19:13:58 also more generally, we introduced some known security risks during the bot conversion by switching some of them to identify where previously they authed at connection time, and by no longer having an atomic means of verifying the nicks they get messages from are authentic 19:14:19 fungi: we should set enforce on our nicks with nickserv ya? 19:14:33 i don't know if supybot had certificate support, but there are options in it for limnoria 19:14:58 ianw: its verification of the nicks interacting with the bots that is the struggle 19:14:59 clarkb: yes, it's already set for our the bot accounts but we should encourage people to set it on their own nicks 19:15:06 if we can set the new converged bot up to do certfp auth, that eliminates one of those risks 19:15:15 other networks have extensions that include with every message the auth/ident status when each message is sent but oftc does not 19:15:40 or if it does, i wasn't readily able to find it 19:16:08 Overall I think the transition went well. We hit a few speedbumps but nothing major and we managed to work through almost all of them. Thank you to all who helped with this process. 19:16:13 granted, the cap freenode was relying on for that also didn't seem to be part of the irc3 standard caps 19:16:29 fungi: it was but then got superceded by another cap that does similar 19:17:07 ahh, yeah i'll admit i only skimmed them to see which ones were implemented in freenode and oftc 19:17:20 alright anything else on this topic or should we move on? 19:17:50 i think we can move on, but please anyone with questions or problems do reach out to us 19:18:02 #topic Gerrit Account cleanup 19:18:06 and i have an oftc-related policy question to bring up during open discussion if there's time 19:18:12 noted 19:18:28 For gerrit account cleanups this is a reminder that I've got a list up of slightly more dangerous but probably safe account cleanups on review 19:18:48 We've all been busy and not able to review that list (and at this point I feel like I should rereview it myself), but if you get time to look over it that would be great 19:19:28 I think doing that cleanup in a multistep process where we disable teh accounts first then sit and wait a while then cleanup the conflicting external ids has worked well and should be relatively safe here. When done I think we end up with about 80 accounts that we should reach out to users about 19:20:01 #topic Refreshing SSL certs 19:20:24 We have made some good progress with these. A number of services are now using LE certs. We also cleaned up certcheck a bit. 19:20:48 The last remaining cert we need to renew is for wiki and I'll be doing that manually after this meeting because we don't have that service in config mgmt 19:21:10 Then we should be good for another year. 19:21:29 thanks! 19:21:30 If you notice old certs hanging around that may be due to apache not cleaning up all child processes when we tell it to reload. I think the risk is low but it does happen 19:21:41 Definitely say something if you see a cert expiring soon 19:21:57 and we do have workarounds we can apply for cases like that if they keep cropping up 19:22:31 #topic Switch Vexxhost to provide only specialized labels in Nodepool 19:22:38 #link https://review.opendev.org/c/openstack/project-config/+/785769 19:22:44 This has been rebased and looks mergeable 19:23:09 fungi: did you want to discuss it further or just ask for reviews and possible +A at this point? 19:23:37 i guess i was hoping for some consensus on either the ml or review that this was something we wanted before approving it 19:23:58 ok so more than just one or two +2s before approving it 19:24:02 but maybe the lack of general interest in the topic is a sign that there's no objection 19:24:25 i can go ahead and approve it after the meeting, it's always easy enough to revert 19:24:58 I added my thoughts to the change in review and ya basically said it should be safe and it is easy to revert if we discover it causes problems somehow 19:25:37 #topic Server Upgrades 19:26:09 Once I've sorted out the wiki ssl cert I plan to set up a hold for change that runs system-config-run-lists and then do in plcae upgrades of the resulting test nodse as a first pass check of that upgrade. 19:26:37 Then based on how the test nodes handle it we can decide if we need to do a snapshot of the prod list server(s) and test that upgrade with real prod like images 19:26:51 that sounds like a reasonable check 19:27:04 I don't expect any trouble. iirc last time we did this we had to modify some mailman pipelines but otherwise it went well 19:28:03 #topic Scheduling project renames 19:28:27 we've still just got the one requested for now, i guess 19:28:34 I still feel like it is a bit early to schedule these as we're still coming up for air from the last fire... Maybe next week we'll have some thoughts based on how playbook updates go? 19:29:03 ya I'd like to think we can do it this month sometime. I'ev got family visiting through the month but should have time for this at some point. But I need to catch up with everything else going on before I can commit to that 19:29:17 i think the osf/* repos are probably wanting to move to something like openinfra/ but i don't know that i have time to push on the patches and coordination for that as it's not at all urgent 19:29:32 fungi: do you think someone else at the foundation might be interested? 19:29:44 maybe jimmy? 19:29:45 not really sure, tbh 19:29:48 I guess we can ask around and see 19:29:53 can ask around, yeah 19:30:00 it would be good to do more than one if we can 19:30:15 should probably be all of those in one go 19:31:02 ok, I think that is all for the agenda. I ran through it fairly quickly. 19:31:05 #topic Open Discussion 19:31:12 fungi: plenty of time for your extra item 19:31:38 yeah, so there's this change... 19:31:46 #link https://review.opendev.org/793999 Rename IRC channel for Octavia 19:32:09 it raises the question of whether we should continue to squat abandoned channels when we move or otherwise clean our bots out of them 19:32:30 hrm my initial though is that keeping things cleaner is a good thing particularly after all the audit work you had to do 19:32:41 given there's no join forwarding on oftc, there's less of an incentive to keep those old channels registered 19:33:12 in a similar vein, i'd like to try to root out other abandoned channels (the ones we log are easy to audit, those we don't log are not so easy) 19:33:14 I'm ++ on renaming and dropping the old for simplicity 19:33:24 yeah; and we're not actually revoking our access; we could conceivably still step in and set a topic if we needed to 19:33:39 also to be polite to the fine folks who run the servers, i feel like we should unregister channels when we stop using them 19:33:51 (no guarantee of course, but mostly just saying we're not likely to box ourselves into a corner and be sad) 19:34:23 if we unregister them, we would theoretically lose control of them 19:34:26 or, should we consider doing all of that 6 months after we stop using them? 19:34:46 yeah, i think a delay before cleaning up is warranted, for sure 19:34:51 doing it with a delay seems reasonable. We still eventually reduce network overhead, but give us an out if things change in the short term 19:34:55 because if we are going to unregister them, then losing privs immediately after the switch might make us sad 19:35:14 and we can always appeal to the network operators in the (very rare, i'm sure) case where we'd need to do something to regain control of a channel we unregistered 19:35:26 i wouldn't want someone to accidentally (or purposefully) hinder a move by squatting the channel right as the move happens 19:36:07 so maybe we should comment out those entries with a date or something, so we remember when to unregister them? 19:36:35 i don't think a heavyweight process is warranted 19:37:21 fungi: ++ 19:37:31 also worth noting we haven't rewritten the channel renaming section of our system-config irc doc yet, so i can take a shot at drafting the new policy there 19:37:37 policy/process 19:37:43 thanks 19:38:01 and big thanks to frickler for rewriting a good chunk of that document 19:38:18 i've already pointed folks to the updated channel registration instructions 19:38:31 so it's definitely come in handy 19:39:50 comment process sgtm :) 19:40:17 if we have time for another unscheduled topic... 19:40:25 we do, go for it 19:41:03 i pushed up a zuul spec to see if the zuul project can achieve consensus to move to matrix; i consider that step 0 before talking to opendev about that 19:41:29 like, my understanding of how we want these relationships to work is for projects to express desires/needs/etc and work with opendev to achieve them 19:41:53 sounds right to me 19:42:00 yup (also Ishould add repsonding to that spec to my todo list) 19:42:00 cool 19:42:23 #link zuul spec about using matrix https://review.opendev.org/793669 19:42:28 there's some positive response already 19:42:52 i think it would be useful for people to weigh in both with their zuul hat and their opendev hat 19:43:16 since most of the folks who haven't weighed in yet are in the overlapping area on the venn diagram :) 19:43:26 ++ 19:44:07 and if that proceeds, then i figure i'll be working on an infra-spec as a followup and talking about it more here 19:44:16 which is probably at least subconsciously intentional since i personally want to know what other folks think before i potentially go clouding the input with my overlapping opinion space 19:45:28 but more generally, i'm comfortable connecting to both irc and matrix from the same unified client for different communities, some of which aren't on irc networks which have matrix bridges 19:46:23 Anything else? if not we can get to breakfast/lunch/dinner 15 minutes early :) 19:46:25 so weechat having a matrix plugin for matrix-only channels and native irc protocol support for non-matrix-accessible channels is going to be necessary for me anyway 19:46:35 fungi: ++ 19:47:31 I'll go ahead and call it there. Thank you everyone. We'll see you here next week 19:47:51 #endmeeting