Tuesday, 2021-05-25

*** ChanServ has quit IRC01:05
*** corvus has quit IRC01:10
*** ChanServ has joined #opendev-meeting01:15
*** weber.freenode.net sets mode: +o ChanServ01:15
*** sboyron has joined #opendev-meeting06:17
*** mordred has quit IRC10:12
*** fungi has quit IRC10:13
*** fungi has joined #opendev-meeting10:14
*** mnaser has quit IRC10:16
*** guillaumec has quit IRC10:18
*** halali_ has quit IRC10:18
*** tobiash has quit IRC10:18
*** sboyron has quit IRC10:18
*** mnaser has joined #opendev-meeting10:18
*** sboyron has joined #opendev-meeting10:19
*** guillaumec has joined #opendev-meeting10:19
*** halali_ has joined #opendev-meeting10:19
*** tobiash has joined #opendev-meeting10:19
*** ChanServ has quit IRC10:21
*** ChanServ has joined #opendev-meeting10:24
*** weber.freenode.net sets mode: +o ChanServ10:24
*** ianw has quit IRC10:27
*** ianw has joined #opendev-meeting10:27
*** mordred has joined #opendev-meeting10:36
*** mordred has quit IRC10:41
*** guillaumec has quit IRC11:00
*** guillaumec has joined #opendev-meeting11:00
*** mordred has joined #opendev-meeting12:01
*** rosmaita has joined #opendev-meeting18:09
*** mnaser has quit IRC18:30
*** mnaser has joined #opendev-meeting18:31
clarkbAnyone else here for the meeting? we will get started in a few minutes18:58
rosmaitao/18:59
fungihowdy19:00
clarkb#startmeeting infra19:01
openstackMeeting started Tue May 25 19:01:25 2021 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.19:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.19:01
*** openstack changes topic to " (Meeting topic: infra)"19:01
openstackThe meeting name has been set to 'infra'19:01
clarkb#link http://lists.opendev.org/pipermail/service-discuss/2021-May/000241.html Our Agenda19:01
ianwo/19:01
clarkb#topic Announcements19:02
*** openstack changes topic to "Announcements (Meeting topic: infra)"19:02
clarkbI didn't have any announcements19:02
clarkb#topic Actions from last meeting19:02
*** openstack changes topic to "Actions from last meeting (Meeting topic: infra)"19:02
clarkb#link http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-05-18-19.01.txt minutes from last meeting19:03
clarkbWe don't appear to have recorded any actions last meeting19:03
clarkb#topic Priority Efforts19:03
*** openstack changes topic to "Priority Efforts (Meeting topic: infra)"19:03
clarkb#topic OpenDev19:03
*** openstack changes topic to "OpenDev (Meeting topic: infra)"19:03
clarkbI think it was fungi that mentioned it is odd we keep this as a separate meeting topic when all the things we do are basically opendev.19:04
fungithe meeting is the opendev meeting ;)19:04
clarkbI'll try to remember when I set up the agenda for next meeting that this needs a reorg19:04
clarkbI don't have any updates that have historically fit under this heading that don't have agenda items later in the agenda19:04
clarkbI think we should move on19:04
clarkb#topic Update Config Management19:05
*** openstack changes topic to "Update Config Management (Meeting topic: infra)"19:05
clarkbThis one is in a similar situation. We tend to talk about these updates in the context of saying updating mailman config management19:05
clarkbI'm thinking maybe instead of putting "priority efforts" upfront we just try to order the agenda item lists to ensure the most important items come first19:05
fungiwfm19:05
clarkb(some weeks that will be our priority efforts and others it will be out of the blue important things)19:05
ianw++19:06
fungiit was originally a section for covering activities related to specs marked as priority19:06
clarkbcool, with that I think we can jump into the agenda items which I've already applied a priority sorting function to19:06
clarkb#topic General topics19:07
*** openstack changes topic to "General topics (Meeting topic: infra)"19:07
clarkb#topic Refreshing non LE certs19:07
*** openstack changes topic to "Refreshing non LE certs (Meeting topic: infra)"19:07
clarkb#link https://review.opendev.org/c/opendev/system-config/+/792789 Redirect ask.o.o19:07
clarkbianw started work on making a landing page for a shtudown ask.openstack.org. This change lgtm though dns for acme challenges needs updating19:07
*** sboyron has quit IRC19:07
clarkbThat switches ask.o.o to an LE cert then serves a tiny index.html pointing people to the internet archive19:08
clarkbSeparately I've got a stack of changes to convert ethercalc, storyboard, and translate to LE certs19:08
clarkb#link https://review.opendev.org/c/opendev/system-config/+/792708 stack of LE certs19:08
ianwthanks, yeah i can finalise that19:08
clarkbThese changes will handle the bulk of the expiring certs.19:09
clarkbWe are left with wiki, openstackid and openstackid-dev19:09
clarkbfungi: for wiki is the easiest thing to buy a cert?19:09
clarkbor is it under sufficient config management to use LE?19:09
fungiyes probably19:09
fungito buy a cert19:09
fungiit is under absolutely zero config management19:10
clarkbok, in the past I've done that about a week before expiry. I can work that out when we get closer to expiration19:10
clarkbFor openstackid and openstackid-dev fungi and I spoke to the foundation sysadmins and they are willing to take on management of those services. However, it is unlikely that will be done before the certs expire19:10
fungithe config management we have for mediawiki is mostly sufficient to stand up a working server, but the data needs reorganizing compared to how it's laid out on the production server19:11
clarkbI told them I would refresh the cert for the prod server one way or another. My plan is to try and LE it if the others I am LEing go well (they should be fine)19:11
*** corvus has joined #opendev-meeting19:11
fungialso "mostly" is important there, i did not get so far as to work out why openid auth was broken on the dev deployment19:11
clarkbfungi: gotcha19:11
fungi(for wiki)19:11
clarkbfor openstackid-dev it sounded like we don't think anything needs a valid cert for that anymore so I'm going to let it just expire19:11
clarkbif we discover after it expires that something does need it we can provision it with LE too (I wanted to avoid provisioning a cert if someone else is going to manage it with their own config management later)19:12
clarkbLong story short we have a plan now and are making progress, we should have things orted well before any certs expire19:13
clarkb#topic Potentially Migrating away from Freenode to OFTC19:13
*** openstack changes topic to "Potentially Migrating away from Freenode to OFTC (Meeting topic: infra)"19:13
clarkbLast week freenodes staff quit due to changes to the organization. Since then there have been changes to things like use policies for freenode too19:13
clarkbthis is precipitated conversations around whether or not we should stick to freenode19:14
clarkbfor many yaers we have maintained a small backup presence on oftc19:15
clarkbincreasing it appears that switching to oftc is going to be a good moev for us, but we are soliciting feedback from our membr projects19:15
clarkb#link http://lists.opendev.org/pipermail/service-discuss/2021-May/000236.html Fungi's call for input19:15
clarkbKata and Zuul have both responded that they are happy to move (zuul would like to stick close to ansible if possible but that isn't critical)19:16
clarkbopenstack seems to be dragging its feet19:16
clarkbfungi: have we heard anything from airship or starlingx?19:16
fungiwell, rather the tc is trying to crowdsource input on an irc preference from all the openstack projects19:16
fungii've heard nothing from either airship or starlingx leadership19:17
funginor openinfralabs, whose ml i also contacted19:17
clarkbfungi: maybe we should ask ildikov and ashlee to prod them?19:17
clarkband I guess have you prod openinfralabs again? (I think you interface wit hthem)19:18
ianwhttps://github.com/ansible-community/community-topics/issues/19 appears to still be ansible discussing things?19:18
fungican't hurt, though for the most part i'm taking a lack of response to mean they're fine with whatever we decide to go with19:18
clarkbianw: ya from corvus' email it seemed ansible was undecided19:18
corvusthat's my understanding19:18
fungiianw: yeah, also the python community, the freebsd community and some other large ones are still trying to decide what to do19:18
corvusi feel like these are all feathers that tip the scale really; i don't think there are many folks that feel very strongly oftc vs libera.19:19
corvusthere are folks that feel strongly "not freenode"19:19
clarkbthe recent updates to freenode policies are concerning so ya "not freenode" seems to be picking up more steam19:20
clarkbfungi: it would probably be good to keep the pressure on so that we can make a decision sooner than later. As stalling puts us in a weird spot. I can reach out to ildikov and ashlee to see what they say about airship and starylingx19:23
clarkb(I cannot type today, or most dats)19:23
fungiyeah, i agree. i mostly expect openstack to take the longest to come to an answer, given larger ships take longer to turn19:24
clarkbcool, anything else to add to this? maybe you want to talk about your bot modifications?19:24
corvusbased on what i've read, we may expect problems from freenode if we vacate.  that's worth planning for.19:24
clarkbcorvus: specifically around repurposing channels right?19:24
corvusyeah.  i expect our idea of closing channels with forwarding messages is incompatible with how the new staff sees things.19:25
clarkbya I guess that means if we switch we need to overcommunicate so that people can discover the new location without relying on messages in irc19:26
corvusyeah.  i think that would be prudent.  if nothing bad comes to pass, there's no downside.  :)19:26
ianwi've seen one incident with something to do with a hacker-news bot repeater channel19:26
ianwbut i now have several moved channels19:27
ianw#cryptography-dev went mute and changed topic, centos seems to now be #centos-has-moved19:28
clarkbya I don't expect much trouble, but being prepared for it can't hurt19:29
corvusi suspect it has more to do with areas of personal interest19:30
fungiright, #openbsd has had a forwarding message in its topic for nearly a week already19:30
corvusso we're probably fine.  but maybe don't rule it out.19:30
clarkb++19:31
fungii agree, it's more likely to come up when moves get press, or are otherwise high-profile and present marketing opportunities or damage control19:31
corvusslightly OT: i'm engaging in a matrix experiment for learnings.19:32
corvusdon't need to discuss now, but just wanted to share19:32
ianw++ experimenting with matrix > talking about matrix :)19:33
clarkbcool. Anything else on this topic or should we move on?19:33
fungisean-k-mooney and yoctozepto have been as well19:33
clarkbfungi: ^ feel free to discuss your updates or whatever other relevant items are in play19:33
fungiwip changes are up with an "oftc" review topic19:33
fungii'm currently wrestling the state machine in accessbot to get it working19:34
clarkb#link https://review.opendev.org/q/topic:oftc Bot updates to make oftc a possibility19:34
fungithe biggest difference there is that the acls on oftc use coarse-grained access levels rather than individual access flags19:34
fungiso our mask handling has to be ripped out and replaced19:34
corvusfungi: are you sure it has to be ripped out?  i *thought* i left some provision for swapping in roles instead of flags19:35
fungialso the lack of identify-msg cap on their ircd means the initial connection stages are subtly different, but enough i'm having trouble getting the identify message to fire at the right time19:35
corvus(but it's entirely possible that it grew more complex since then and overwhelmed whatever provision i thought i left for that)19:36
fungicorvus: if it can also support rbac then that'd be a huge help, but it wasn't especially obvious if it's in there19:36
corvuswhere is the code now? :)19:36
fungiin my editor, i'll push up a broken i progress change19:37
fungier, in progress change19:37
corvusah looks like ./docker/accessbot/accessbot.py ?19:37
fungiyes, that's the current state19:37
fungii thought you were asking if my work in progres on that script was up for review yet, sorry19:37
corvusfungi: i think you're right and i'm wrong.  i think it needs to be ripped out and replaced.19:38
fungicool, i'm about halfway into that, at least19:38
fungithanks for looking19:38
corvussorry for the red herring19:38
fungino worries, that code has a thick layer of dust on it19:38
clarkbWe have a few more topisc to get through, lets get through those and can swing back to this if we still have time19:39
fungianyway, that's all i've got. going to keep plugging on the accessbot work19:40
fungithanks!19:40
clarkb#topic Switch Vexxhost to provide only specialized labels in Nodepool19:40
*** openstack changes topic to "Switch Vexxhost to provide only specialized labels in Nodepool (Meeting topic: infra)"19:40
clarkbfungi: I didn't manage to find this change. But the tl;dr is that the difference in base memory for vexxhost nodes make its problematic for landing things that need more memory then later run elsewhere with less memory19:40
clarkbconsidering the size of vexxhost's pool slice we can (with reasonable safety) shift vexxhost to only providing the specialized resources and not the generic ones19:41
* clarkb looks for the change again19:41
*** halali_ has quit IRC19:41
clarkb#link https://review.opendev.org/c/openstack/project-config/+/78576919:41
fungi#link yes, sorry, i was pulling up the ml thread19:41
clarkbfungi: looks like it needs to be reabsed. Is there any reason to keep it as WIP?19:41
fungi#link http://lists.opendev.org/pipermail/service-discuss/2021-April/000219.html Recent nodepool label changes19:42
fungii didn't want it to merge until it had been discussed, but the ml thread just sort of petered out19:42
clarkbgotcha19:42
fungiand yeah, i'm not surprised it's fallen into a merge conflict in the meantime19:43
clarkbdo you want us to try and pick up the discussion on the mailing list?19:43
clarkbfwiw I'm happy to make that change (and could say so on the mailing list)19:43
ianwdo we have a sense of how many jobs use those expanded node types?19:43
clarkbbasically try and be more flexible in allocating resources to both avoid problems and maybe provide more interesting flavors where they can be useful19:43
clarkbianw: I think it has fluctuated over time. octavia uses the nested virt flavors pretty consistently, but then sean-k-mooney does things with the bigger and nested virt flavors when nova has specific needs iirc19:44
clarkbairship is also a large flavor user for their main end to end test job19:44
*** halali_ has joined #opendev-meeting19:44
fungii switched it out of wip19:45
fungiand will rebase to clear the merge conflict19:45
ianwwe could only set mem= type kernel command lines to limit the ram by setting it in a dib element, right?19:45
clarkbianw: we can but if we do that we need different images19:46
clarkbbecause we want to use more memory in other circumstances19:46
ianwright, yeah.  we used to do that iirc19:46
clarkbya we did it for hpcloud when there were only two clouds and the images were different anyway19:47
clarkbits probably still worth thinking through to see if there is a way to make that happen (like can nova set boot params for us? or have a pre run playbook update the node and reboot if necessary if the time loss isn't too bad)19:47
clarkbbut fungi's change is the simplest thing we can land right now19:47
fungisecond simplest19:48
fungibut less wasteful than just removing that region19:48
ianw++.  i wonder if we can hotplug remove memory19:48
fungiplz_to_be_removing_sticks_of_ram19:48
clarkbI think we can proceed with fungi's change then revert if we find an alternative19:49
clarkb#topic Server Upgrades19:49
*** openstack changes topic to "Server Upgrades (Meeting topic: infra)"19:49
clarkb(running out of time so want to keep going)19:49
clarkbThe mailman ansible stuff seems to have gone well. I don't know that we haev created any new lists yet (but this is tested with zuul) though19:50
*** halali_ has quit IRC19:50
clarkbNext up is taking a snapshot of the server and upgrading it through to focal. One thought I had was before we bother with a snapshot of the server I can test this on a zuul node that we hold19:50
clarkbonce we have this working reliably on the held zuul nodes (can rerun ansible against the focal node too etc) then take an up to date snapshot and make sure it runs happily on our snapshot?19:51
clarkbdo we think running through it on the snapshot is useful if doing it on a zuul node? (how careful do we want to be is the question I guess)19:51
ianw++ .. having gone through quite a few in-place updates with afs i didn't really have any issues19:51
ianwthe only problem was a stuck volume on the afs sever which was barely related19:52
clarkbI'll try to make time for the held zuul node upgrade late this week or next week and we can plan doing the actual snapshot copy after that if we feel it is necessary19:52
fungii feel like having the snapshot on hand as a fallback is probably fine, probably the only actual thing you won't be testing with the in-place zuul upgrade is the newer mailman packages19:53
clarkbfungi: ya and last time we had a few issues related to those iirc19:53
clarkbhad to do with new pipeline entries for mailman iirc19:53
clarkbwe worked through them then, we can probably work through it again19:53
clarkbanother consideration is we may want to avoid mailing list outages while IRC is still somewhat up for discussion19:54
clarkbbut that is a problem to worry about when we are ready to upgrade :)19:54
clarkbianw: any gerrit upgrade updates?19:55
ianwno but i swear i'm getting back to it19:55
clarkbok :) we have all been busy19:55
clarkb#topic Scheduling project renames19:55
*** openstack changes topic to "Scheduling project renames (Meeting topic: infra)"19:55
clarkbrelated to being busy I feel like we're not in a great spot to try and plan for this right now with everything else going on19:55
clarkbIf we can dig out of the backlog and get the playbooks updated we can start scheduling properly19:56
clarkb#topic Open Discussion19:56
*** openstack changes topic to "Open Discussion (Meeting topic: infra)"19:56
clarkbrosmaita: did you have updates or more to discuss about zuul comments in gerrit?19:56
clarkb(wanted to make sure I saved some time for the catch all topic if so)19:56
rosmaitano, i did have a different topic though19:57
rosmaitasomehow got nuked off the agenda :)19:57
fungiwe did have someone come into the opendev channel asking how to set tags on gerrit comments, probably related to running a cinder third-party ci system19:57
clarkbrosmaita: I may have accidentally cleaned up that topic and the old one thinking it was just the old one19:57
clarkbrosmaita: apologies if that is what happened19:57
clarkbrosmaita: go for it though19:57
ianw(happy to run slightly over for missed topics)19:58
rosmaitathanks19:58
rosmaitasorry, my system went unstable for a minute there19:59
rosmaitaquick question about publishing the api-ref19:59
rosmaitahttps://docs.openstack.org/api-ref/block-storage/index.html19:59
rosmaitathat has both v2 and v319:59
rosmaitawe are removing v2 this cycle, so ordinarily would also remove the v2 api-ref19:59
rosmaitabut, it could be useful for people who have procrastinated20:00
rosmaitathe api-ref isn't branched like the regular docs are20:00
fungiright, because the apis aren't versioned the same as the software20:00
rosmaitaso i was wondering if there is a way to have the v2 api-ref built from the stable/wallaby branch or something like that?20:00
clarkbrosmaita: can you change https://docs.openstack.org/api-ref/block-storage/v2/index.html to say (Removed in Xena) instead of (DEPRECATED) and maybe put a warning there but keep the info there?20:01
rosmaitaor is the best think just to say "REMOVED" in big letters at the top of the v2 ref20:01
rosmaitaclarkb: yeah, i guess that's the best thing20:01
clarkbI think the reason the api docs are separated from the projects is that the apis live longer than the master branch20:01
fungiwell, it's removed from the latest version of the software, but not retroactively removed from earlier versions of the software, and that document is in theory about all openstack cinder versions20:02
clarkb(thats probably not hte most accurate statement, but iirc the info that generates those docs isn't directly tied to the code for this reason)20:02
clarkbfungi: ++20:02
rosmaitaok, a note makes sense20:02
rosmaitafor glance we put up a note telling users to generate the v1 api-ref for themselves if they really wanted it20:02
fungirosmaita: we're probably well off-topic for opendev, but consider it like the sdk. should the latest version of the sdk rip out cinder v2 api support and just declare old cloud deployments are no longer valuable?20:02
rosmaitawell, we keep the documentation branched with the software20:03
rosmaitaand you can use an older version of the sdk if you need to20:04
rosmaitaanyway, doesn't matter20:04
fungisort of. it's in the same repository as the software these days (it wasn't always) but in theory only the master branch version is maintained since that's the only one which gets published20:04
fungialso you can't use old versions of the sdk if you have one piece of software which needs to talk to two clouds which have different supported api versions20:04
fungi(there are people who interface with more than one cloud at the same time)20:05
rosmaitaok, we'll put a note on the v2 ref, that will work20:05
clarkbcool, glad we could help20:05
clarkbAnything else? we are over time so I'll end us here if not20:05
fungiultimately, it's more of a question for the openstack tc and tech writing sig though20:05
rosmaitathat's all from me, other than a thank you for being ready for the freenode -> OFTC change20:06
fungii don't have anything else20:06
clarkbthanks everyone!20:06
clarkb#endmeeting20:06
*** openstack changes topic to "Incident management and meetings for the OpenDev sysadmins; normal discussions are in #opendev"20:06
openstackMeeting ended Tue May 25 20:06:23 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)20:06
openstackMinutes:        http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-05-25-19.01.html20:06
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-05-25-19.01.txt20:06
openstackLog:            http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-05-25-19.01.log.html20:06
fungithanks clarkb!20:06
*** rosmaita has left #opendev-meeting20:29
*** corvus has quit IRC21:51
*** corvus has joined #opendev-meeting22:04
*** tobiash has quit IRC22:24

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!