| tonyb | Can anyone point me at issues we've seen with CentOS stream (like ping, and the removal of the algo that was used for signing rpms? | 01:29 |
|---|---|---|
| tonyb | nm I found a enough context for my needs | 01:34 |
| Clark[m] | https://bugzilla.redhat.com/show_bug.cgi?id=2037807 and https://bugzilla.redhat.com/show_bug.cgi?id=2184640 fwiw | 02:02 |
| Clark[m] | https://bugs.launchpad.net/tripleo/+bug/2023764 is a problem tripleo attributed to CentOS kernel updates | 02:05 |
| Clark[m] | https://bugs.launchpad.net/tripleo/+bug/1997202 is one where network manager was overwriting resolv.conf. I seem to recall we made a change to more explicitly tell NM to not manage that file but it seems that it also affected buildah | 02:12 |
| Clark[m] | I seem to recall there were problems specifically with openstack stuff when setting up fips jobs but I wasn't following super closely | 02:17 |
| opendevreview | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/887126 | 02:28 |
| opendevreview | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/887126 | 05:58 |
| opendevreview | Caio Cesar Ferreira proposed openstack/project-config master: Add Power Metrics app to StarlingX https://review.opendev.org/c/openstack/project-config/+/886814 | 11:36 |
| opendevreview | Merged openstack/project-config master: Add Power Metrics app to StarlingX https://review.opendev.org/c/openstack/project-config/+/886814 | 12:38 |
| SvenKieske | hey! so I understand we mirror different stuff already, correct? would it be possible to also mirror some (all?) used packages from pypi.org? I don't know how much storage that would need and how much is available. | 16:07 |
| SvenKieske | reason being we are seeing many CI failures due to pypi mirror problems | 16:07 |
| SvenKieske | in kolla-ansible, that is | 16:07 |
| Clark[m] | No. We did mirror pypi.org way back when but it grew too large (on the rate of a gigabyte a day) | 16:08 |
| SvenKieske | e.g. here: https://review.opendev.org/c/openstack/kolla/+/882924/6#message-aad622078f973cee5ff7efc7245b71e08b2e84e1 | 16:09 |
| Clark[m] | As far as I know that growth rate has not changed and may have even gotten worse as AI/ML tools have gotten more popular | 16:09 |
| SvenKieske | ah. good to know! what about mirroring only some packages? or does openstack tend to pull in all of it? :D | 16:09 |
| Clark[m] | The proxy cache system we have in place works reasonably well even though it isn't oerfect | 16:10 |
| Clark[m] | No before we mirrored all of pypi.org we mirrored only the subset from openstack requirements and that creates a bunch of problems | 16:10 |
| SvenKieske | ah | 16:10 |
| Clark[m] | Basically it made this weird dance that had to be done to add any dep (or update a dep whose deps have changed) | 16:10 |
| SvenKieske | I figured it's good to ask, and your answer shows it's really good to ask first :D | 16:10 |
| SvenKieske | yeah, sure, you can't really specify in a sane way which mirror to use for which dependency, I think there's still an open issue for that for pip | 16:11 |
| Clark[m] | Also it always helps to link to job logs when discussing job failures | 16:11 |
| Clark[m] | I'm guessing in this case the issue was pypi failed over to it's stale backend and the index was incomplete | 16:11 |
| SvenKieske | mhm, okay, I don't know enough about pypi internals to comment on that | 16:13 |
| SvenKieske | thanks anyway! | 16:14 |
| Clark[m] | Can you link to the failure? | 16:14 |
| SvenKieske | I'd have to dig that up, I just trusted frickler's comment :) let me look.. | 16:15 |
| Clark[m] | I trust it to but it doesn't characterize the pypi failure just that it had to do with pypi | 16:15 |
| SvenKieske | yeah, well, our error output is not helpful at all, I fear. I guess it is this line: https://zuul.opendev.org/t/openstack/build/2433c9e910a648098e0a32e1b3c41c39/log/kolla/build/000_FAILED_kolla-toolbox.log#847 | 16:18 |
| SvenKieske | but that's already a for loop installing requirements and the error code is just "non zero exit code" because of bash wrappers -.- | 16:19 |
| Clark[m] | That looks like the stale backend for pypi serving old index for pbr that doesn't include the constrained version | 16:19 |
| SvenKieske | the for loop was added because of unrealiable mirrors, but it seems, 5 tries with 120 seconds timeout each are not enough :( | 16:19 |
| Clark[m] | I have encouraged openstack before to take this up with pypi since the use of constraints means openstack hits it often whereas other pypi users just install old software from the old index | 16:20 |
| SvenKieske | can I read up on this stale index stuff somewhere? sounds like a bug to me? | 16:20 |
| Clark[m] | It is a security concern for those other users. For openstack it causes failures due to constraints | 16:20 |
| SvenKieske | but nobody bothered to open an upstream bug, right? :) | 16:21 |
| Clark[m] | There is an old cheese shop/warehouse/pypa bug on GitHub somewhere (pretty sure it's closed because they "fixed" it) | 16:21 |
| SvenKieske | https://github.com/pypi/warehouse/issues/8568 this one? | 16:21 |
| Clark[m] | The tldr is pypi is fronted by a cdn. The cdn requests data from the primary backend when it needs data. If that fails it requests data from a secondary backend. The problem is the secondary backend is often months out of date so you get stale info when this situation occurs | 16:22 |
| Clark[m] | Yes that appears to be the one | 16:22 |
| SvenKieske | I'll try to reopen it :P :D I can be stubborn if it helps.. | 16:23 |
| Clark[m] | That pvr version is about 5 months old implying the secondary backend is at least that stale | 16:25 |
| Clark[m] | *pbr | 16:25 |
| Clark[m] | However it may be package specific. I don't know enough implementation details for that system | 16:26 |
| SvenKieske | I managed to get a response, but I fear I will run in circles with this. | 16:33 |
| SvenKieske | see the github issue for details | 16:33 |
| SvenKieske | Clark[m]: thanks for chiming in on the github issue, maybe we can work out a fix now with pypi :) | 16:54 |
| SvenKieske | I also pinged them again over on libera chat #pypa-dev; I think they won't receive notifications over closed issues otherwise. | 16:57 |
| Clark[m] | I got emailed for the updates to the issue at least. But who knows if they watch them (I get so much email I can't blame anyone for ignoring it) | 16:58 |
| opendevreview | Merged opendev/zone-opendev.org master: Replace ze01-ze03 https://review.opendev.org/c/opendev/zone-opendev.org/+/885513 | 17:42 |
| opendevreview | Merged opendev/system-config master: Replace ze01-ze03 https://review.opendev.org/c/opendev/system-config/+/885508 | 17:55 |
| corvus | i deleted the fact cache for those 3 hosts | 18:03 |
| opendevreview | Merged zuul/zuul-jobs master: Use zuul_workspace_root for prepare/mirror workspace test roles https://review.opendev.org/c/zuul/zuul-jobs/+/886186 | 18:35 |
| opendevreview | Elod Illes proposed openstack/project-config master: Fix release-id fetching for release job https://review.opendev.org/c/openstack/project-config/+/887222 | 18:49 |
| opendevreview | Merged openstack/project-config master: Fix release-id fetching for release job https://review.opendev.org/c/openstack/project-config/+/887222 | 19:56 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: DNM: Test base-test https://review.opendev.org/c/zuul/zuul-jobs/+/887229 | 20:11 |
| corvus | i rebooted ze01-ze03 and they now have apparently working afs trees | 20:13 |
| corvus | and the most recent playbooks runs are green | 20:16 |
| corvus | so i think i'm going to bring up one of those | 20:16 |
| clarkb | sounds good | 20:16 |
| corvus | #log started ze01 on new jammy host | 20:17 |
| corvus | it's running jobs... | 20:18 |
| clarkb | and the afs failure previously was on executor startup | 20:18 |
| clarkb | so that is a really good sign | 20:18 |
| corvus | yep; and a job i just spot checked was successful | 20:20 |
| corvus | #status log started ze02 and ze03 on new jammy hosts | 20:25 |
| opendevstatus | corvus: finished logging | 20:25 |
| fungi | awesome, thanks! | 20:39 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!