| opendevreview | Jeremy Stanley proposed opendev/system-config master: Upgrade to Keycloak 23.0 https://review.opendev.org/c/opendev/system-config/+/907141 | 01:52 |
|---|---|---|
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Upgrade to Keycloak 23.0 https://review.opendev.org/c/opendev/system-config/+/907141 | 03:37 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Upgrade to Keycloak 23.0 https://review.opendev.org/c/opendev/system-config/+/907141 | 05:33 |
| ykarel | frickler, it still shows similar error but for different sig packages | 05:41 |
| ykarel | http://mirror.iad.rax.opendev.org/logs/rsync-mirrors/centos-stream.log | 05:41 |
| ykarel | any idea how to get past to it? not sure if changing to some other mirror would help here | 05:41 |
| ykarel | may be if you have access can try running the sync manually to understand what's actual issue | 05:42 |
| ianw | sigh, not sure there's much we can do if the remote end is hanging up on us ... | 06:23 |
| ianw | there's nothing in the mirror-update logs ~ 2024-02-06T00:08:08 that would suggest local instability | 06:26 |
| ykarel | ianw, so can try other mirror? | 06:28 |
| ianw | ... we can, i mean just look at the git log for plenty of examples :) | 06:28 |
| ykarel | like revert of https://review.opendev.org/c/opendev/system-config/+/868392 | 06:29 |
| ianw | what would be ideal is an upstream mirror with someone we can actually contact when it fails. it feels like we are basically an excellent free monitoring system for someone | 06:29 |
| ianw | we flip between the rax mirror, some facebook one, some university one ... | 06:30 |
| opendevreview | yatin proposed opendev/system-config master: Revert "Revert "Revert "Use rackspace mirror to sync centos stream repos""" https://review.opendev.org/c/opendev/system-config/+/907923 | 06:30 |
| *** ykarel_ is now known as ykarel | 06:46 | |
| opendevreview | Takashi Kajinami proposed openstack/project-config master: Retire heat-cfnclient: End Project Gating https://review.opendev.org/c/openstack/project-config/+/907951 | 07:03 |
| opendevreview | Takashi Kajinami proposed openstack/project-config master: Retire puppet-qdr: Remove Project from Infrastructure System https://review.opendev.org/c/openstack/project-config/+/907954 | 07:09 |
| opendevreview | Takashi Kajinami proposed openstack/project-config master: Retire puppet-qdr: End Project Gating https://review.opendev.org/c/openstack/project-config/+/907951 | 07:10 |
| opendevreview | Takashi Kajinami proposed openstack/project-config master: Retire puppet-qdr: Remove Project from Infrastructure System https://review.opendev.org/c/openstack/project-config/+/907954 | 07:10 |
| ildikov | Hi All, I'm moderating the starlingx-discuss mailing list and got a question from a subscriber who has issues with his mails to the ML getting bounced. Has anyone experienced that before? | 08:21 |
| opendevreview | Dr. Jens Harbott proposed openstack/project-config master: Update reno update to check whether series exists https://review.opendev.org/c/openstack/project-config/+/907963 | 08:25 |
| frickler | ildikov: if you DM me the email address of the subscriber I can check the logs of the mail system | 08:27 |
| frickler | ildikov: just for clarity, are the bounces happening for their submission before they are accepted for the list? or on delivery to some of the subscribers? | 08:34 |
| ykarel | frickler, hi can you check https://review.opendev.org/c/opendev/system-config/+/907923 | 08:34 |
| frickler | ykarel: yes, I saw that, but I don't think that randomly flipping mirror sources is a sustainable task, will let other admins do that if they feel like it | 08:36 |
| ykarel | frickler, yeap agree, but may be atleast can get it sync for now. it would also match the mirror used for 8-stream which is in sync | 08:39 |
| *** liuxie is now known as liushy | 09:42 | |
| apevec | ykarel: we should get access to the primary CS rsync, we'll just need to provide public IP where openinfra is rsyncing from, it is not wide-open for capacity reasons | 10:27 |
| apevec | frickler: is mirror-update running on a specific machine so you could get me whatismyip from there or does it run in multiple machines/clouds providers? | 10:28 |
| tonyb | I'm pretty sure it runs from a single machine that changes infrequently (every couple of years) | 10:29 |
| apevec | tonyb: cool, please get public IP so we can discuss this further on CS infra side! | 10:33 |
| ykarel | apevec, thx yes that should help | 10:33 |
| tonyb | apevec: can do. | 10:35 |
| tonyb | apevec: I'm 95% certain it'd be https://opendev.org/opendev/system-config/src/branch/master/inventory/base/hosts.yaml#L333 | 10:36 |
| frickler | tonyb: apevec: I'll add the remaining 5%, just confirmed the addresses on the host itself | 10:36 |
| apevec | mirror-update02.opendev.org makes sense to run mirror-update :) | 10:37 |
| apevec | thanks frickler ++ tonyb ++ | 10:37 |
| fungi | ildikov: frickler: did the mailing list post bounce problem get sorted out? forwarding a copy of the non-delivery report the user received to one of us might also help in tracking down the cause | 13:48 |
| frickler | I didn't get any further response, so nothing I could do from my side | 13:51 |
| fungi | i'll try to follow up later | 14:14 |
| fungi | thanks! | 14:14 |
| opendevreview | Alfredo Moralejo proposed opendev/system-config master: Use centos hosted mirror to sync CentOS content https://review.opendev.org/c/opendev/system-config/+/908030 | 14:26 |
| opendevreview | Alfredo Moralejo proposed opendev/system-config master: Use centos hosted mirror to sync CentOS content https://review.opendev.org/c/opendev/system-config/+/908030 | 14:29 |
| amoralej | apevec, tonyb ^ | 14:32 |
| fungi | amoralej: already looking at it. are msync and rsync two different hosts? | 14:33 |
| amoralej | yes | 14:33 |
| amoralej | there are separated hosts for stream-9 and previous releases | 14:33 |
| fungi | got it, and we use the latter for stream-9 i see | 14:33 |
| amoralej | exactly | 14:33 |
| fungi | i was going to test this out from the server to confirm connectivity | 14:33 |
| amoralej | yes, please | 14:33 |
| fungi | will take me a few minutes | 14:33 |
| amoralej | actually, will it try to actually sync from some other ci server? | 14:34 |
| amoralej | in that case it will likely fail | 14:34 |
| fungi | nope | 14:34 |
| amoralej | cool! | 14:34 |
| fungi | those scripts aren't really tested, i don't think, which is part of why i'm going to manually test a patched copy from an interactive shell on the server to make sure | 14:34 |
| amoralej | good, that'd be great to confirm connectivity and the directories layout | 14:35 |
| Clark[m] | Note we haven't done that in the past because the existing rules state only public mirrors can sync from those locations. I'm somewhat wary of being an exception to those rules | 14:36 |
| fungi | yeah, i figure it helps to confirm this works regardless of which direction the discussion ends up going | 14:39 |
| amoralej | We explained the issues we've had in the past and the kind of private usage opendev are doing and they agreed on it. We may not be the first case they hit a similar case, let us know your concerns and we can discuss it | 14:41 |
| amoralej | https://pagure.io/centos-infra/issue/1354 is the ticket for centos for the record | 14:43 |
| Clark[m] | The main concern I have is that it will stop working in a few months when someone discovers we broke the rules and we will be right back where we started. It would be better to address the underlying issues | 14:43 |
| Clark[m] | Which means more curation of the second level mirrors which the centos project may not have time and resources for | 14:44 |
| fungi | amoralej: not working, i added inline comments with the error messages | 14:51 |
| Clark[m] | One problem with rsync is you can't easily browse to figure problems like this out. Many of the public mirrors serve http content too and make that less of an issue but msync doesn't seem to? | 14:53 |
| fungi | they do also print a big, bold warning (the gist of which we already know, but for the record): "This service is intended for the sole use of the CentOS worldwide mirror network to synchronize mirrors. Unless you are running or intending to run a listed public CentOS mirror use a mirror listed at https://centos.org/download/mirrors If you intend to populate a mirror for public use please read | 14:54 |
| fungi | the notes at https://wiki.centos.org/HowTos/CreatePublicMirrors If you do use this service then it is implied that you are providing a mirror for public use and giving us authority to publicise such mirror." | 14:54 |
| amoralej | thanks | 14:55 |
| amoralej | lemme check | 14:55 |
| fungi | amoralej: also, ftr, the servers rsync ended up getting the responses back from were centosy8.centos.org and mref1.ue2.stream.centos.org according to the banners they printed | 14:56 |
| fungi | no idea if that matters | 14:56 |
| fungi | we could make the argument that our mirrors technically are for public use, just not safe for *direct* public use; we operate a public ci/cd system and the mirrors are operated in support of the worker nodes running within that context | 14:58 |
| amoralej | that's how we explained the use of this mirror | 15:00 |
| amoralej | fungi, may it be using ipv6 ? | 15:02 |
| amoralej | yep, they found the ipv6 attempt in the log, will take a while to update it | 15:04 |
| fungi | it probably prefers ipv6 unless the remote host only publishes v4 records | 15:04 |
| fungi | an alternative would be to see if rsync has a v4-only option or something in the interim | 15:05 |
| fungi | --ipv4, -4: prefer IPv4 | 15:06 |
| fungi | i'll test that in the meantime | 15:06 |
| fungi | looks like it's working for now if i force it to go over ipv4, will see if it's able to complete | 15:08 |
| amoralej | also, acl for ipv6 address should be in few next minutes | 15:09 |
| fungi | non-stream rsync worked but was a no-op because it was already in sync | 15:12 |
| fungi | stream rsync is running now | 15:12 |
| fungi | i'll test again without -4 afterward to confirm the v6 address access is in place | 15:14 |
| amoralej | ok | 15:14 |
| fungi | stream rsync does seem to be working, pulling some new/changed packages | 15:15 |
| fungi | so this could be a viable option for us, i expect, if reviewers reach a consensus. and in the meantime we'll have the mirrors updated at least | 15:15 |
| fungi | though whether or not we'll get the disconnects we observed earlier remains to be seen, of course | 15:16 |
| amoralej | ok, sure, let's keep the discussion on the review | 15:17 |
| amoralej | thanks fungi! | 15:17 |
| fungi | will do. i plan to comment once testing has finished | 15:18 |
| amoralej | ping me if you find any other issue | 15:20 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: DNM: Try future Keycloak 24.0 https://review.opendev.org/c/opendev/system-config/+/907253 | 15:22 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: DNM: Fail keycloak testing for an autohold https://review.opendev.org/c/opendev/system-config/+/906600 | 15:23 |
| frickler | hmm, is there actually a way to redact comments in gerrit? other than doing weird database stuff? | 16:08 |
| Clark[m] | Yes I think there is an API for it. Sounded like corvus was going to test it on a local test gerrit | 16:12 |
| Clark[m] | The API can delete a comment and replace it's message. So not fully an edit but close enough | 16:13 |
| fungi | amorin: ovh sent us a notification that they couldn't "process our payment" which probably means our time-based credits have run out. anyone recall who we reach out to in order to ask them to refresh that? | 16:27 |
| frickler | oh, that time of year again. I think there was someone else at ovh, let me grep logs | 16:31 |
| frickler | no, amorin fixed it last on 2023-01-04 | 16:34 |
| fungi | oh, thanks for confirming! i'll give him a while to respond, and try reaching out by e-mail soon also if necessary | 16:36 |
| tkajinam | Hi. May I ask someone to add me to https://review.opendev.org/admin/groups/d4fc65b8e79a0c041e0803bb30a98a3e2664decb,members ? | 16:43 |
| tkajinam | context: https://review.opendev.org/c/openstack/project-config/+/905976 | 16:43 |
| clarkb | frickler: tonyb: if you get a chance reviewing https://review.opendev.org/c/opendev/system-config/+/907472 to update gitea to the latest bugfix release would be good | 16:48 |
| opendevreview | Jay Faulkner proposed openstack/diskimage-builder master: Re-enable voting on Gentoo DIB CI job https://review.opendev.org/c/openstack/diskimage-builder/+/907904 | 16:51 |
| fungi | tkajinam: done! | 16:54 |
| tkajinam | fungi, thanks ! | 16:55 |
| fungi | any time! | 17:01 |
| opendevreview | Merged opendev/engagement master: Record raw responses https://review.opendev.org/c/opendev/engagement/+/904298 | 17:35 |
| opendevreview | James E. Blair proposed opendev/system-config master: Add a tool to delete (redact) gerrit comments https://review.opendev.org/c/opendev/system-config/+/908181 | 17:46 |
| opendevreview | James E. Blair proposed opendev/system-config master: Add a tool to delete (redact) gerrit comments https://review.opendev.org/c/opendev/system-config/+/908181 | 18:01 |
| opendevreview | James E. Blair proposed opendev/system-config master: Remove obsolete delete-gerrit-spam.py script https://review.opendev.org/c/opendev/system-config/+/908184 | 18:01 |
| opendevreview | Merged opendev/system-config master: Add a tool to delete (redact) gerrit comments https://review.opendev.org/c/opendev/system-config/+/908181 | 18:15 |
| opendevreview | James E. Blair proposed opendev/system-config master: Fix gerrit-delete-comment script https://review.opendev.org/c/opendev/system-config/+/908186 | 18:29 |
| opendevreview | Merged opendev/system-config master: Remove obsolete delete-gerrit-spam.py script https://review.opendev.org/c/opendev/system-config/+/908184 | 18:33 |
| opendevreview | Merged opendev/system-config master: Fix gerrit-delete-comment script https://review.opendev.org/c/opendev/system-config/+/908186 | 18:50 |
| ildikov | fungi: frickler: apologies, I didn't get the notifications of your pings and got distracted. I just checked and the person's emails didn't get through to the ML, so they bounced before getting posted. | 19:59 |
| clarkb | ildikov: and they didn't end up in moderation? | 19:59 |
| fungi | ildikov: if they can get you (or one of us) a copy of the bounce message, that may help track down the cause | 19:59 |
| fungi | it's possible delivery was rejected by some intermediate mta before reaching us | 20:00 |
| fungi | maybe an office mail gateway or something | 20:00 |
| clarkb | fungi: thoughts on sending in the gitea upgrade? | 21:05 |
| clarkb | I have a school run in an hour or so but otherwise expect to be around | 21:05 |
| clarkb | hapyp to wait for others to review it first though | 21:05 |
| fungi | i can approve it | 21:06 |
| fungi | going to be cooking dinner soon, but nothing too distracting | 21:06 |
| clarkb | fungi: looks liek your figured out the env var for keycloak was KC_DB_PASSWORD? | 21:06 |
| clarkb | and the new test case should confirm that is working so thats all good | 21:07 |
| fungi | i wasn't sure originally because they have separate build-time and run-time envvars for some stuff, but if you don't use the --optimized switch with kc.sh start then it does a build before startintg | 21:07 |
| fungi | so the build-time envvars end up baked into the auto-built runtime, basically | 21:08 |
| fungi | it's so funky, and it does slow down container restarts, but it keeps us from having yet another container we need to build ourselves i guess | 21:08 |
| clarkb | that is weird | 21:11 |
| fungi | yeah, so in summary, there seems to be no runtime envvar to set the database password, but there is a buildtime envvar to bake it into the image, and by not using --optimize so that it rebuilds at every container start, the buildtime envvar effectively acts as a runtime envvar | 21:31 |
| fungi | this is also why i increased the start wait for the service in ansible from one minute to five, the keycloak autobuild at start can take some time to complete, and while it usually finished in under a minute in testing, there were some timeouts i could only attribute to the ansible task giving up too early | 21:43 |
| clarkb | its weird that a startup that has to optimize things is faster than a startup that doesnt | 21:50 |
| clarkb | Iwonder what otpimize actually does here | 21:50 |
| opendevreview | Tristan Cacqueray proposed zuul/zuul-jobs master: Remove the periodic-weekly pipeline https://review.opendev.org/c/zuul/zuul-jobs/+/827369 | 21:53 |
| opendevreview | Merged openstack/diskimage-builder master: Fix various minor issues with Gentoo; make CI pass https://review.opendev.org/c/openstack/diskimage-builder/+/907757 | 21:55 |
| clarkb | fungi: looks like the gitea change will merge right around midway through my school run | 22:02 |
| clarkb | fungi: I can help test the deployment when I get back or you can -W it I suppose if the timing doesn't work | 22:03 |
| fungi | i'm done with dinner and keeping an eye on it | 22:04 |
| fungi | looks like the last non-paused job just finished uploading logs now, so the registry job is unpausing and wrapping up | 22:38 |
| opendevreview | Merged opendev/system-config master: Upgrade gitea to 1.21.5 https://review.opendev.org/c/opendev/system-config/+/907472 | 22:38 |
| fungi | and there it is | 22:38 |
| fungi | deploy is free of any delays at the moment too | 22:38 |
| fungi | and infra-prod-service-gitea deployment is running now | 22:39 |
| fungi | https://gitea09.opendev.org:3000/ is back up and looking good | 22:42 |
| fungi | "Powered by Gitea | 22:42 |
| fungi | Version: v1.21.5" | 22:42 |
| fungi | and i can clone a repo from it | 22:43 |
| fungi | gitea10 is done | 22:43 |
| fungi | seems similarly fine | 22:43 |
| fungi | it's bringing up gitea14 now | 22:48 |
| fungi | and deploy job is done. going through the load balancer to test, everything seems consistent still. should be all set! | 22:49 |
| fungi | basically 10 minutes from merge to deploy. that's pretty amazing | 22:50 |
| fungi | especially since the server upgrades are all serialized | 22:51 |
| clarkb | yup looks good from here | 23:09 |
| opendevreview | Clark Boylan proposed openstack/diskimage-builder master: Write fedora download redirect info to stderr https://review.opendev.org/c/openstack/diskimage-builder/+/908207 | 23:28 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!