Wednesday, 2023-11-15

clarkbcorvus: oh if we did then the config wasn't removed from static I guess?00:04
corvusyeah working on that now00:04
opendevreviewMerged opendev/system-config master: Add tonyb to statusbot nicks  https://review.opendev.org/c/opendev/system-config/+/90095500:05
opendevreviewJames E. Blair proposed opendev/system-config master: Revert registry.zuul-ci.org  https://review.opendev.org/c/opendev/system-config/+/90096100:05
opendevreviewJames E. Blair proposed opendev/zone-zuul-ci.org master: Revert registry.zuul-ci.org  https://review.opendev.org/c/opendev/zone-zuul-ci.org/+/90096200:06
corvusclarkb: there's cleanup for that, sorry about that!00:06
corvus(i think the fact that the rewrite url points to "corvus" is a pretty good indicator as to the final status)00:07
fungihah, excellent point00:20
fungiwe'll presumably need to manually delete playbooks/roles/static/files/50-registry.zuul-ci.org.conf from the server, since just removing it from configuration management doesn't actually take it away, right?00:21
clarkbcorrect00:22
fungier, i guess really /etc/apache2/sites-{available,enabled}/50-registry.zuul-ci.org.conf00:24
opendevreviewMerged opendev/zone-zuul-ci.org master: Revert registry.zuul-ci.org  https://review.opendev.org/c/opendev/zone-zuul-ci.org/+/90096200:30
opendevreviewMerged opendev/system-config master: Revert registry.zuul-ci.org  https://review.opendev.org/c/opendev/system-config/+/90096102:46
*** benj_6 is now known as benj_02:47
*** benj_5 is now known as benj_02:53
opendevreviewThierry Carrez proposed opendev/irc-meetings master: Move Large Scale SIG meetings one hour later  https://review.opendev.org/c/opendev/irc-meetings/+/90100408:31
*** ykarel|away is now known as ykarel10:06
opendevreviewyatin proposed openstack/project-config master: [Neutron] Update Grafana Dashboard  https://review.opendev.org/c/openstack/project-config/+/90101010:48
opendevreviewMerged opendev/irc-meetings master: Move Large Scale SIG meetings one hour later  https://review.opendev.org/c/opendev/irc-meetings/+/90100413:49
*** d34dh0r5- is now known as d34dh0r5315:00
opendevreviewMerged opendev/system-config master: Add letsencrypt_certs for mirror02.ord  https://review.opendev.org/c/opendev/system-config/+/90095315:02
*** gthiemon1e is now known as gthiemonge15:55
fungiinfra-prod-letsencrypt succeeded after that ^ so we should be back in working order again16:05
fungii'll go ahead and delete the old registry.zuul-ci.org apache config left behind on static.o.o in the wake of 90096116:06
fungidone and reloaded apache on the server16:07
fricklerdo we have the "Move Change" API disabled in gerrit? should there be an UI option for it? I was thinking about the move to unmaintained/ and what should happen with open reviews on stable/(yoga in the current case) 16:07
fungii wasn't aware of its existence, but looks like https://review.opendev.org/Documentation/rest-api-changes.html#move-change documents it in the version we're currently running16:09
fungii can't imagine we would have disabled it16:09
fricklerbut I do remember right, that in order to delete the stable/yoga branch in gerrit, all open reviews need to be abandoned or merged, right?16:10
fungilooks like it would need to be done by an account with abandon permission for the original change's branch16:10
fungiyes, open reviews would have to be closed (merged or abandoned, or possibly moved) before we can delete the stable branch16:11
tonybfungi: Correct.16:11
tonybahem s/fungi/frickler/16:11
fricklero.k., I'll discuss in #-tc first which way we want to handle this, then we can look into how to implement it16:12
fungifrickler: in light of that, i suppose the the process could be 1. eom tag stable branch, 2. create unmaintained branch from tag, 3. move open changes from stable to unmaintained, 4. delete stable branch16:12
tonybI was kinda thikning we could add a tag to make them easy to find, and then abandon them16:12
fricklerfungi: yes, that was my idea for if we do want to keep the reviews instead of abandoning16:13
fungiyes, i was previously assuming (because i didn't know about the move option) that we would 1. eom tag stable, 2. abandon open stable changes, 3. delete stable branch, 4. create unmaintained branch from eom tag16:13
fungifrickler: i agree it seems like a good idea16:14
Clark[m]Ya I don't think we have disabled that api16:15
Clark[m]It should work but I'm not sure we have ever tested it. Could add it to our gerrit testing in system-config though16:15
fungii'm pretty certain we've never tested it, i think it must be a relatively new addition16:17
fungior else we just somehow missed that it existed (it's not often we have need for that exact workflow anyway)16:17
Clark[m]It came with notedb iirc16:19
frickleraccording to https://stackoverflow.com/questions/59796736/how-do-you-move-a-gerrit-change-to-a-different-branch where I found this, it has been there for quite some time16:20
fungithe notedb transition is relatively "recent" to me16:21
fungii mean it wasn't in the 2.x rest api16:21
fungibut looks like they added it in 2.1316:22
tonyboh so a similar timeframe to adding the delete branch api16:23
tonyb.... probably related16:23
clarkbtonyb: https://mirror02.ord.rax.opendev.org/ lgtm at first glance. I think we're ready to update the dns CNAMEs16:30
tonybcool.  I had a little look at it also.  so yeah today I'll play DNS switcheroo 16:34
clarkbinfra-root https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/gerrit/tasks/main.yaml#L161-L177 this is where we currently write out the gerrit -> gitea replication key data on Gerrit. https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/gitea/tasks/main.yaml#L122-L158 this is where we configure the public key in gitea. And this is how16:53
clarkbthe replication plugin documents configuring key selection: https://gerrit.googlesource.com/plugins/replication/+/refs/heads/master/src/main/resources/Documentation/config.md#file-316:53
clarkbgiven all that I think we can use a key type of our choosing then using the ssh config file to select the appropriate key. The tricky part (which was unexpected to me) is that our gitea management wants only one key at a time16:54
clarkbI suspect that we may need to refactor the gitea bit to be a little more flexible16:55
clarkbmaybe track an old key and a new key and only cleanup keys that don't match either. And finally maybe treat a null oldkey as an indication to clean it up too16:56
fungiclarkb: does our gitea management remove old keys? if it's only additive (installing the key if it doesn't exist) then we could probably get by with that for a brief transition16:57
clarkbfungi: it removes keys that don't match the one key it wants in gitea16:57
clarkbor at least that seems to be the intention16:57
fungiah, okay, so no dice16:57
fungianother possibility is that we take the hit to swap keys out and then rereplicate everything after16:58
clarkbI think we can refactor this without too much trouble16:59
fungiwe were talking about needing to do a full replication at some point anyway, after observing missing branches and whatnot16:59
clarkbI think it is buggy as is as well17:00
fungioh, fun17:00
clarkbit checks the number of keys if >0 it tries to delete old keys. But then it only adds new keys if key length is 0 and it doesn't refetch the key list after deletion17:00
clarkbso if we try to swap keys out as is we'll end up with no keys on the first pass then have to wait for the playbook to run again to create the new key17:00
clarkbRoughly I think our process is 1) fix/refactor gitea key management 2) decide on the keytype and size for the new key 3) add new key to gerrit 4) coordinate addition to gitea and config update in gerrit to use new key to swap over17:02
clarkbit doesn't look like Gerrit is currently using a .ssh/config file to set the key info so we're relying on defaults there17:02
clarkbso step 3 above would include that config file17:02
tonybWe aren't in a "rush" for this right?17:15
fungiright, probably won't merge it until a couple of weeks from now17:16
tonybOkay.17:18
opendevreviewClark Boylan proposed opendev/system-config master: Add ssh key rotation to gitea ssh key management  https://review.opendev.org/c/opendev/system-config/+/90108218:18
clarkbsomething like that maybe18:18
clarkbnow to update the gitea 1.21 change to use the release18:18
clarkbhttps://github.com/go-gitea/gitea/pull/27757/commits <- I feel like this is why Gerrit's model is better18:45
JasonFThere are configurations you can mash in github to make that not crap all over the overall git history (e.g. enforcing squash+merge or using CI/review to enforce sane commits)18:47
*** JasonF is now known as JayF18:47
clarkbthe problem is the PR itself is indecipherable18:50
clarkbthere are multiple commits saying it renames a file from foo to foo.sh but the end result of the PR is the file is not renamed18:50
clarkbwhether or not that is in the git hiustory or recorded in a single commit's commit message its completely confusing and a mess18:50
JayFYou can have (and I've seen) the same thing happen in gerrit as a change evolves and the update doesn't include the commit message18:54
JayF(as always with those things, it seems like you find them after they land, but I'm sure I've -1'd a couple for that kinda oversight before, too)18:55
JayFRegardless of tools, creating a good git history require concious effort on the part of everyone in the process. Although I do generally agree that github-style-workflows enable whatever pattern you want them to enable for the most part; even ones that you and I would consider bad patterns. Gerrit is more strongly opinionated.18:55
opendevreviewMerged openstack/project-config master: [Neutron] Update Grafana Dashboard  https://review.opendev.org/c/openstack/project-config/+/90101018:56
opendevreviewClark Boylan proposed opendev/system-config master: Update gitea to 1.21.0  https://review.opendev.org/c/opendev/system-config/+/89767919:00
clarkbJayF: the difference with gerrit is you get the iterative history pretty well laid out in the change history19:00
clarkbso even if the commit itself was missed you'll generally have some indication of why the thing stopped getting renamed for example19:00
clarkband yes it takes discipline. But Gerrit forces you to think about that with a commit per change model19:01
clarkbyou can still get it wrong but you can't ignore it19:01
clarkbinfra-root 897679 is updated for the gitea 1.21.0 release with changelog info accounted for. If that passes CI I'll rebase the force fail change on it and hold a node we can look at19:01
fungithanks!19:03
JayFclarkb: heh, github used to actually have a nastier bug; if you force pushed, the commits used to just completely disappear along with the comments19:04
JayFclarkb: so you could essentially rebase out someone's comment and if they weren't being vigilant, pretty easy to miss19:04
fungii thought that was intentional, and didn't realize github had stopped doing that (good on them for deciding the discussion was worth preserving after all)19:05
JayFI always viewed it as an unintended side effect19:05
clarkbI'm slightly worried the dockerfile refactor has some bug in my update to mimic upstream19:05
clarkbwe should know soon enough19:05
JayFbut I also force-push more than most github users because gerrit taught me how to git better than most other folks (<--- this is why even though I like gerrit personally; I think it's a barrier to entry; you have to know git better than you do to use git{hub,lab} style workflows)19:06
fungiit seems to me that, because those workflows force you to maintain separate forks, you actually have to know more about git19:07
fungior, at worst, you need a grasp of different aspects of git19:08
clarkbI waffle on that my self. I think in our bubble that anyone writing python that is expected to safely, without data loss, manage virtual machines, networks, and storage can be expected to know git. People writing docs probably don't have similar backgrounds and expecting git of them is a bigger order. But also ya I'm not convinced github is inherently easier. I just think it is more19:08
clarkbfamilar19:08
JayFfungi: that's actually not true for everyone; many folks just give access to create branches on the base org and don't require forking19:08
JayFfungi: especially for single-company projects/oss teams (this is how it is setup for armada contributors, for instance, because armadaproject github org has faster CI runners)19:09
clarkba lot of people also conflate github and git which makes it harder to break down the idea that you can use git outside of the github world19:09
JayFclarkb: do not forget about the extensive tooling and docs. There are GH cli/gui apps, and howtos for everything.19:09
JayFYes and yes.19:09
JayFI don't like github the business model. I do think they provide a value to the community we've walled ourselves off from in a set of tradeoffs that I don't always love :)19:10
clarkbmeanwhile I think I'm still one of the only few that actually tracks the github mirrors...19:12
clarkbthere is a comment there against openstackclient I need to respond to19:12
JayFThat is on my list to pick up this cycle; I just ended up with a large number of things on my plate :/ 19:12
JayFand Eventlet sorta shoved itself into my life19:12
JayFrealistically monitoring github will probably die at the altar of eventlet work19:12
clarkbhttps://github.com/openstack/openstacksdk/commit/d17b23f63109a1cac0c6d01ae9e278b6e83bcc02#commitcomment-132631626 fwiw. The issue is they are using python3.6 and need to install python3.8. I plan to respond as soon as I get signed back in19:12
clarkbJayF: the good thing about it is that it is interrupt driven. So I just try to read thigns like ^ if I have time and respond or forward as necessary19:13
fungimy whole existence is interrupt-driven. too bad my brain is single-threaded19:15
JayFI've been working over the last ... well really since taking this GR-OSS job at trying to flip from being interrupt-driven to project-oriented19:15
JayFwith mild success19:15
clarkbthis is interesting the docker file update seems to be failing beacuse it is running a copy instruction before the workdir and make complete so there is no data to copy19:24
clarkboh interesting this is happenign beacuse it thinks the data is cached19:28
clarkbbut it isn't becuase it is new?But why would it be cached if we are starting from scratch on a single use VM to build the image19:29
fungimaybe they only tested upgrades and not new installs?19:34
clarkbI think this is somethign to do with how we are building the images and this change upstream is just exposing it19:36
clarkbdocker image builds take a --no-cache flag optionally19:36
clarkbbut it doesn't make any sense to me that docker would be finding cached data here19:36
clarkbbecause it is a fresh node that we are running on19:37
fungicould it be getting called more than once in the job?19:37
clarkbI don't think so. I'm wondering if it is somehow finding old builds in the intermediate registry. Howver this is the build image in our multi stage builds and we don't push those anywhere iirc19:39
clarkbit should be almost impossible to cache that without running docker image builds twicei n the same env19:39
clarkbI'm going to revert those changes to the image I guess and see if it still explodes19:41
opendevreviewClark Boylan proposed opendev/system-config master: Update gitea to 1.21.0  https://review.opendev.org/c/opendev/system-config/+/89767919:45
opendevreviewClark Boylan proposed opendev/system-config master: Add ssh key rotation to gitea ssh key management  https://review.opendev.org/c/opendev/system-config/+/90108219:46
clarkbhere is where it says a bunch of stuff is cached then it gets to the step wheer I modified things and it fails to copy properly https://zuul.opendev.org/t/openstack/build/382d990a92ba447f99a3b008777f5355/log/job-output.txt#781-79719:47
clarkbcorvus: ^ you don't know off the top of your head why that would be happenign do you? It seems like nothing at all should be cached in those image builds19:48
clarkbI can reproduce the problem locally and adding --no-cache does not help19:54
fungidid someone accidentally check in cache files?19:55
clarkbI think maybe this is part of build setup and because we use RUN git clone ... to create the directory that COPY copies it doesn't know about that directory19:56
clarkbwe need to use ADD19:56
clarkbI'm working on testing this19:56
fungione thing that's come up with mm3 feature parity is that there's no place in postorius to set the footer boilerplate text for each list like you could in mm220:03
fungiinstead you have to create a text file in /var/lib/mailman/core/var/templates/lists/$LIST_ID/$LANG/list:member:regular:footer.txt20:04
fungiwe currently have 5 on lists01: openstack-discuss.lists.openstack.org, starlingx-discuss.lists.starlingx.io, zuul-announce.lists.zuul-ci.org, zuul-discuss.lists.zuul-ci.org, service-discuss.lists.opendev.org20:05
fungii suppose we could stick those files in git and then have ansible install them to the server20:07
fungibut all 5 are empty files, specifically overriding the default footer with nothing in order to make mailman not add a footer to messages20:07
fungiso another option might be to add a config flag for whether we want ansible to create an empty file there20:08
fungiopinions?20:08
clarkbthe existing ones were pulled in via the migration but now end users can't create them via the web ui?20:09
fungicorrect. well, 4 of those were created on import from mm2 data, the starlingx one i added manually yesterday20:09
ildikovHi Team, I have a weird question. I'm trying to "find" the edge-computing ML on the mailman GUI to check on the moderation queue, etc. I think it's on the openinfra.dev domain now, but it doesn't get listed on the web GUI. Can someone help me figuring out what I'm doing wrong?20:10
fungiildikov: lists.opendev.org for that one. if you're logged in, by default the https://lists.opendev.org/ page will default to an owner+moderator+member filter. try clicking "all"20:14
opendevreviewClark Boylan proposed opendev/system-config master: Update gitea to 1.21.0  https://review.opendev.org/c/opendev/system-config/+/89767920:14
clarkbI think ^ is a compromise between what upstream is doing and what we are doing20:15
fungiildikov: if you have a message from the list, you can always double-check the domain part of its e-mail address, that will also be the server name for the management interface and archives20:15
fungii need to step away to drop christine at an appointment, but will be back in 10 minutes20:15
clarkbfungi: if we set no footer file is that not equivalent to setting an empty one?20:15
clarkbI'm wondering if we can just avoid setting it entirely. If not then I think a per list config option in the ansible vars for the lists to copy in an empty file is how I would do it20:16
ildikovfungi: the to address that I used was openstack.org, but I couldn't find the edge-computing ML listed there either20:22
clarkbildikov: the adress you email is edge-computing@lists.openstack.org ? I don't think that shoudl work20:25
ildikovclarkb: I'm trying to access the moderator dashboard of the list20:28
ildikovI can send emails to it20:28
fungiildikov: oh, that's an old address we have forwarding to edge-computing@lists.opendev.org20:28
ildikovfungi: right, that's what I thought20:28
ildikovoh, ok, never mind20:28
ildikovI mixed up opendev with openinfra20:29
ildikovI wish we had more distinctive names :)20:29
fungiyeah, since the edge-computing wg isn't directly a function of the openinfra foundation, it went on opendev (like the openinfralabs ml)20:29
ildikovmy bad, I knew it was something stupid like this, but couldn't figure it out20:30
fungiand i agree, openinfra.dev and opendev.org are very similar names, i mistype one for the other all the time20:30
opendevreviewClark Boylan proposed opendev/system-config master: Add ssh key rotation to gitea ssh key management  https://review.opendev.org/c/opendev/system-config/+/90108220:30
clarkbfwiw we do have distinct names we just setup a redirect to make them more confusing20:30
ildikovfungi: I understand the reason, just my brain failed to process today that there is an opendev.org domain for MLs too20:31
clarkbI feel like this wouldn't be an issue if we didn't have the redirect20:31
fungiclarkb: which redirect?20:31
clarkbrather than openinfra.dev and opendev.org being similarish20:31
clarkbfungi: 'oh, that's an old address we have forwarding to edge-computing@lists.opendev.org'20:31
fungithat was forwarding from an old edge-computing@lists.openSTACK.org address20:32
clarkbyes and ildikov checked lists.openstack.org20:32
clarkbbecause the address ildikov uses is edge-computing@lists.openstack.org20:32
fungiyet a third very-similar domain in that case20:32
clarkbI'm saying if the address people use is edge-computing@lists.opendev.org only then all the ambiguity goes away20:32
clarkbbut we've created a forward that adds ambiguity and makes it unclear20:33
fungiyeah, the main reason we added address forwarding was because people may reply to an older message and we wanted them to not just get a confusing ndr bounced back20:33
fungibut maybe we want to alter our policy for that and instead have exim reject those addresses with explicit error messages that say what the new posting address is20:34
fungifwiw, i still see posts all the time to the old openstack-dev address20:34
ildikovyeah, I would not get rid of the forwarding20:46
ildikovI'm just one of those lazy people who go with the autocomplete in her mail client... and that's in me to fix that :)20:47
fungiclarkb: wrt the templates, there is a default template that gets used, but lists can substitute their own specific template instead. the substitution only occurs if a list-specific template file exists. for the majority of lists, there is no directory under /var/lib/mailman/core/var/templates/lists and so they use the default template instead20:59
fungionly if a list wants to replace the default template (e.g. with empty content so that no footer gets added) will there be a file for them21:00
opendevreviewJulia Kreger proposed openstack/diskimage-builder master: Drop py36 support  https://review.opendev.org/c/openstack/diskimage-builder/+/90109321:14
clarkbfungi: and there is no way to set that template via the ui so we need to manage it out of band for people21:14
clarkbin that case ya I would make it a list attribute in our list of attriutes and have ansibel write the file out21:15
fungiright. in the case of starlingx i did a `mkdir -p ... && touch ... && chown -R x:y ...`21:15
fungioh, good point, we could make it an ansible string, defaulting to empty21:16
clarkbyup21:16
fungithen if someone does want a non-empty but non-mm3-default template, they could put the string in21:16
fungii'll ponder how to go about that. unlike the values we set for lists in inventory currently, this is one that is likely for people to want to apply after list creation21:17
fungithe commit message on 897679 is epic now, but it's passing!\22:15
Clark[m]Ya I'll work out a node hold after this school run22:17
tonyb[tony@thor system-config]$ git show HEAD --no-patch | wc -l22:17
tonyb8222:17
tonyb[tony@thor system-config]$ git diff HEAD^ | wc -l22:17
tonyb64722:17
tonybThat's a pretty good ratio ;P22:17
opendevreviewClark Boylan proposed opendev/system-config master: DNM intentional gitea failure to hold a node  https://review.opendev.org/c/opendev/system-config/+/84818123:09
clarkbI've put an autohold in place for ^23:10
tonyb++23:10
JayFclarkb: I can't find where you were talking about it; are you still digging that gitea dockerfile thing?23:11
JayFclarkb: I had some brain cells fire and had an idea23:11
JayFclarkb: tl;dr basically curious if the build works if env DOCKER_BUILDKIT=0 is set23:11
opendevreviewClark Boylan proposed opendev/system-config master: Add ssh key rotation to gitea ssh key management  https://review.opendev.org/c/opendev/system-config/+/90108223:13
clarkbJayF: it broken with and without buildkit locally. I'm not sure which way its set in the CI. But I think the problem is COPY wants to operate on files it knows about and it doesn't know about these files because they exist due to side effects of RUN commands23:13
clarkbJayF: supposedly you can use ADD to work around that, but I couldn;t make that work with git either (because the flag to include the .git dir wasn't working). I ended up just not bothing with the COPY and modify files in place intead23:14
clarkbinfra-root I think 901082 should be ready for review now. That should make itpossible for us to manage to pubkeys for gerrit in gitea23:14
clarkbthen we can generate a new key, add it to gitea and gerrit then flip gerrit over to it23:14
JayFclarkb: ack, this is usually where I'd say 'makes sense', but it really doesn't, and this is a place where it seems like Dockerfiles have just gotten worse and not better as they have matured :(23:15
JayFbut glad to hear you figured it out23:15
clarkbya it feels a bit overoptimized honestly23:16
clarkblike  ya caching is great but if it means you can't run a RUN command and then COPY the results of that later maybe we've gone too far23:16

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!