| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: registry-tag-remove: role to delete tags from registry https://review.opendev.org/c/zuul/zuul-jobs/+/878614 | 00:32 |
|---|---|---|
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: promote-container-image: use generic tag removal role https://review.opendev.org/c/zuul/zuul-jobs/+/878740 | 00:32 |
| opendevreview | wangxiyuan proposed openstack/diskimage-builder master: Rename openeuler mirror script https://review.opendev.org/c/openstack/diskimage-builder/+/878807 | 02:31 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: registry-tag-remove: role to delete tags from registry https://review.opendev.org/c/zuul/zuul-jobs/+/878614 | 04:05 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: promote-container-image: use generic tag removal role https://review.opendev.org/c/zuul/zuul-jobs/+/878740 | 04:05 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: registry-tag-remove: update docker age match https://review.opendev.org/c/zuul/zuul-jobs/+/878810 | 04:05 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: build-container-image: enhance buildx documentation https://review.opendev.org/c/zuul/zuul-jobs/+/878494 | 05:32 |
| *** eandersson2 is now known as eandersson | 05:56 | |
| frickler | #status log force-merged https://review.opendev.org/c/openstack/horizon/+/875326 to resolve deadlock caused by multi-branch failures | 06:06 |
| opendevstatus | frickler: finished logging | 06:06 |
| frickler | https://review.opendev.org/c/openstack/horizon/+/878751 is running jobs now, so that single force-merge seems indeed to have been enough | 06:06 |
| *** amoralej|off is now known as amoralej | 07:26 | |
| opendevreview | Matthieu Huin proposed zuul/zuul-jobs master: upload-logs: Handle remote dir creation through rsync https://review.opendev.org/c/zuul/zuul-jobs/+/878829 | 10:16 |
| mnasiadka | Ubuntu arm64 mirror seems to be broken for some time - https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_03e/878649/3/check-arm64/kolla-build-ubuntu-aarch64/03ea09e/kolla/build/000_FAILED_base.log - anyone can help? | 10:28 |
| frickler | /afs/.openstack.org/mirror/ubuntu-ports/db/lockfile seems to be stale since Feb 28 | 11:00 |
| *** amoralej is now known as amoralej|lunch | 11:03 | |
| frickler | infra-root: ^^ guess we can simply remove that file? unrelated: wheel releases also haven't happened for 7 days it seems, if someone wants to have a look | 11:04 |
| fungi | frickler: yeah, probably safe to remove, lsof doesn't show any processes with that open, nor is there any running reprepro process at the moment, so it's not hung just got left behind i guess (likely something killed the process). unfortunately it happened much longer ago than our reprepro log retention. i checked dmesg and nothing of consequence was logged around then | 11:43 |
| *** amoralej|lunch is now known as amoralej | 12:19 | |
| frickler | rm: cannot remove '/afs/.openstack.org/mirror/ubuntu-ports/db/lockfile': Permission denied | 12:26 |
| frickler | do I need to do some kerberos stuff for that? | 12:26 |
| frickler | guess yes, will wait until current reprepro run is done | 12:27 |
| frickler | ah, ports ran earlier. done with the rm. next cycle will be in 1.5h, will check logs after that | 12:30 |
| frickler | #status log removed stale lockfile dating Feb 28 /afs/.openstack.org/mirror/ubuntu-ports/db/lockfile | 12:31 |
| opendevstatus | frickler: finished logging | 12:31 |
| fungi | frickler: yeah, presumably you found it already, but i added a "deleting a file" section in our afs docs which should cover that exact case | 13:23 |
| frickler | I must admit that the docs are too complicated to navigate for me most of the time, I just looking into root history on mirror-update | 13:30 |
| frickler | maybe I should at least bookmark some starting pages | 13:30 |
| fungi | i agree the document could probably be far better organized to make things easier to find | 13:45 |
| fungi | https://docs.opendev.org/opendev/system-config/latest/afs.html#deleting-files is the section i was referring tro | 13:46 |
| fungi | s/tro/to/ | 13:46 |
| frickler | what would be the advantage of following those steps instead of simply using "k5start -t -f /etc/reprepro.keytab service/reprepro -- bash" as root@mirror-update ? | 14:57 |
| fungi | mainly that i can do it from my workstation without needing to ssh into a server | 14:58 |
| frickler | ah, o.k., I never installed all that stuff locally | 15:00 |
| fungi | i find it convenient since i can browse afs from my machine easily | 15:01 |
| *** \join_iw19 is now known as \join_iwp9 | 15:26 | |
| *** amoralej is now known as amoralej|off | 16:49 | |
| opendevreview | Sylvain Bauza proposed openstack/project-config master: Stop using Storyboard for Placement projects https://review.opendev.org/c/openstack/project-config/+/878932 | 17:53 |
| clarkb | infra-root I continue to not understand how manage-projects was ever safe with multiple project renames merged in sequence. I've double checked projects.yaml's project listing and gerrit ls-projects and there are no additional projects implying we haven't accidentally recreated old projects after reanming them to a new name | 18:11 |
| clarkb | I thought maybe infra-prod-service-zuul was running and failing on the intermediate change with half old and half new names because the zuul reconfigure request would fail. But this job runs after manage-projects not before. It cannot fail and cause manage-projects to be skipped | 18:12 |
| fungi | did we determine that we're not using a zuul-merged checkout which incorporates all the changes which merged ahead of the change we're deploying? | 18:13 |
| clarkb | manage-projects does depends on infra-prod-base, infra-prod-service-review, infra-prod-service-gitea, and system-config-promote-image-gerrit-3.6 when triggred by system-config. However, these changes are all in project-config and on the project config side manage-projects has no dependencies | 18:14 |
| clarkb | fungi: when corvus joiend the discussion the other day he made it seem like that wasn't the case. But I haven't managed to make a true test case for it | 18:14 |
| clarkb | but ya Ithink that remains the only good explanation I can come up with | 18:15 |
| clarkb | fungi: I think part of the concern with that even it works some of that time is that it ends up ebing a race | 18:15 |
| fungi | oh, right, i keep forgetting that deploy uses the serial manager not the dependent manager, so can't just assume it uses the same kinds of merged refs | 18:16 |
| clarkb | fungi: because it depends on the timing between mergingin gerrit and when the zuul mergers merge the code for the job. If the zuul mergers execute that merge between when gerrit merges the two different changes you would be out of luck | 18:16 |
| fungi | and i guess serial doesn't incorporate changes ahead? or maybe that's what we haven't confirmed yet | 18:17 |
| clarkb | I think it would. The problem is the intermediate change not the last one to run | 18:18 |
| clarkb | if we have two different project rename changes A and B and force merge A first then B the git tree would end up looking liek A <- B. The problem is when the jobs run for A if the do not incorporate the changes from B then we would recreate an old project name | 18:19 |
| clarkb | when B runs it does include A and should be fine | 18:19 |
| clarkb | but the more I dig into this the less I understand of how this ever worked without creating a bunch of orphaned projects | 18:20 |
| fungi | do we maybe only run manage-projects for project entries that have changed rather than for all projects in the config? | 18:21 |
| fungi | since the projects which were deleted are already in the manage-projects cache they don't look like new projects to be created? | 18:23 |
| clarkb | oh you know what | 18:24 |
| clarkb | I think that may be it | 18:24 |
| clarkb | I thought we always checked if the project was in gerrit and took action if not, but if we don't do that then this could be it | 18:25 |
| clarkb | heh we consult the cache for that info not gerrit ls-projects | 18:26 |
| clarkb | fungi: I think you solved it | 18:26 |
| clarkb | https://opendev.org/opendev/jeepyb/src/branch/master/jeepyb/cmd/manage_projects.py#L508-L520 | 18:27 |
| * fungi breathes a sigh of relief | 18:27 | |
| fungi | let's make sure we don't invalidate that cache during a rename! | 18:28 |
| clarkb | ya or just don't rely on it altogether. I'll need to think about this a bit | 18:28 |
| clarkb | my initial impression is that we should thinkabout squashing the changes as the jeepyb cache could be cleared or not have the project in it for some reason (because its a cache!) and this is complicated enough it took me several hours of digging in and talking through it with others to track this down that being explicit seems better than implicit | 18:31 |
| clarkb | but that does make things a bit weird for requesting renames from a user perspective. But we can do a last minute squash ourselves | 18:31 |
| *** JayF is now known as Guest9308 | 19:44 | |
| *** JasonF is now known as jayf | 19:44 | |
| *** jayf is now known as JayF | 19:44 | |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: promote-image-container: do not delete tags https://review.opendev.org/c/zuul/zuul-jobs/+/878612 | 23:56 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: remove-registry-tag: role to delete tags from registry https://review.opendev.org/c/zuul/zuul-jobs/+/878614 | 23:56 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: promote-container-image: use generic tag removal role https://review.opendev.org/c/zuul/zuul-jobs/+/878740 | 23:56 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: remove-registry-tag: update docker age match https://review.opendev.org/c/zuul/zuul-jobs/+/878810 | 23:56 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: build-container-image: expand docs https://review.opendev.org/c/zuul/zuul-jobs/+/879008 | 23:56 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: container-build : add container_promote_method flag https://review.opendev.org/c/zuul/zuul-jobs/+/879009 | 23:56 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!