| ianw | i've closed out https://pagure.io/centos-infra/issue/814 as it seems mirrors have updated | 01:24 |
|---|---|---|
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Add support for Python 3.10 https://review.opendev.org/c/openstack/diskimage-builder/+/844613 | 01:37 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Add Fedora 36 support https://review.opendev.org/c/openstack/diskimage-builder/+/844912 | 02:04 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: containerfile: warn if we don't have a Dockerfile https://review.opendev.org/c/openstack/diskimage-builder/+/846837 | 02:04 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Fix BLS entries for /boot partitions https://review.opendev.org/c/openstack/diskimage-builder/+/846838 | 02:06 |
| opendevreview | Merged opendev/system-config master: gerrit: trigger rebuild for anonymous browsing fix https://review.opendev.org/c/opendev/system-config/+/846809 | 02:10 |
| opendevreview | Merged openstack/diskimage-builder master: Revert "CentOS 9-stream : work around selinux permissions issue" https://review.opendev.org/c/openstack/diskimage-builder/+/846078 | 02:35 |
| opendevreview | Merged openstack/diskimage-builder master: Stop using openstack upper constraints https://review.opendev.org/c/openstack/diskimage-builder/+/845218 | 02:50 |
| opendevreview | Merged openstack/diskimage-builder master: Use plain tox jobs instead of openstack ones https://review.opendev.org/c/openstack/diskimage-builder/+/845349 | 02:50 |
| opendevreview | Merged openstack/diskimage-builder master: Add a warning in satellite configuration https://review.opendev.org/c/openstack/diskimage-builder/+/845650 | 03:25 |
| opendevreview | Merged openstack/diskimage-builder master: Drop tumbleweed job https://review.opendev.org/c/openstack/diskimage-builder/+/845097 | 03:26 |
| opendevreview | Samuel Walladge proposed openstack/project-config master: Add Cinder Dell EMC PowerStore charm https://review.opendev.org/c/openstack/project-config/+/846888 | 04:18 |
| opendevreview | Merged openstack/diskimage-builder master: Add support for Python 3.10 https://review.opendev.org/c/openstack/diskimage-builder/+/844613 | 04:18 |
| opendevreview | Samuel Walladge proposed openstack/project-config master: Add Cinder Dell EMC PowerStore charm https://review.opendev.org/c/openstack/project-config/+/846888 | 04:18 |
| *** ysandeep|out is now known as ysandeep | 04:34 | |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Fix BLS entries for /boot partitions https://review.opendev.org/c/openstack/diskimage-builder/+/846838 | 04:34 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Parse block device lvm lvs size attributes https://review.opendev.org/c/openstack/diskimage-builder/+/839829 | 04:34 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: WIP Support LVM thin provisioning https://review.opendev.org/c/openstack/diskimage-builder/+/840144 | 04:34 |
| opendevreview | Samuel Walladge proposed openstack/project-config master: Add cinder ibm storwize charm https://review.opendev.org/c/openstack/project-config/+/846899 | 04:44 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Add Fedora 36 support https://review.opendev.org/c/openstack/diskimage-builder/+/844912 | 04:47 |
| opendevreview | Samuel Walladge proposed openstack/project-config master: Add Cinder IBM Storwize charm https://review.opendev.org/c/openstack/project-config/+/846899 | 05:04 |
| opendevreview | Samuel Walladge proposed openstack/project-config master: Add Cinder Dell EMC PowerStore charm https://review.opendev.org/c/openstack/project-config/+/846888 | 05:08 |
| opendevreview | Samuel Walladge proposed openstack/project-config master: Add Cinder IBM Storwize charm https://review.opendev.org/c/openstack/project-config/+/846899 | 05:08 |
| *** pojadhav is now known as pojadhav|ruck | 05:11 | |
| * pojadhav|ruck brb | 05:29 | |
| opendevreview | David Ostrovsky proposed zuul/zuul-jobs master: ensure-java role: Bump default java_version to 11 https://review.opendev.org/c/zuul/zuul-jobs/+/846909 | 06:34 |
| *** ysandeep is now known as ysandeep|afk | 06:42 | |
| ianw | i was going to quickly restart gerrit for 846809 but | 06:51 |
| ianw | "ERROR: for gerrit error pulling image configuration: download failed after attempts=6: received unexpected HTTP status: 500 Internal Server Error" | 06:51 |
| ianw | https://hub.docker.com/layers/gerrit/opendevorg/gerrit/3.5/images/sha256-95c8a6d4d9d3716d7ee3616d16b3699a561f26b5f8a3b98cb378a156c04815e0?context=explore looks ok | 06:52 |
| *** marios_ is now known as marios | 06:53 | |
| ianw | ok, it seems to have gone away | 06:57 |
| ianw | # docker inspect f2711d8fdd61 | grep 95c8a6 | 06:57 |
| ianw | "opendevorg/gerrit@sha256:95c8a6d4d9d3716d7ee3616d16b3699a561f26b5f8a3b98cb378a156c04815e0" | 06:57 |
| ianw | ok, restarted, i no longer see the popup | 06:59 |
| ianw | #status log restarted gerrit @ https://review.opendev.org/c/opendev/system-config/+/846809 | 06:59 |
| opendevstatus | ianw: finished logging | 06:59 |
| opendevreview | Merged opendev/statusbot master: twitter: Fix typo on OK tick https://review.opendev.org/c/opendev/statusbot/+/846533 | 07:13 |
| *** jpena|off is now known as jpena | 07:37 | |
| *** pojadhav|ruck is now known as pojadhav|lunch | 07:44 | |
| frickler | interesting long read https://anarc.at/blog/2022-06-17-matrix-notes/ | 07:53 |
| frickler | infra-root: here's a case where zuul seems to have continued to run on PS1 and ignored that PS2 was submitted shortly after PS1 https://review.opendev.org/c/openstack/kolla/+/846554 | 08:11 |
| frickler | not sure if related to gerrit update, will try to find zuul logs | 08:12 |
| *** pojadhav|lunch is now known as pojadhav|ruck | 08:38 | |
| frickler | zuul logs some error from gerrit right after PS2 was received, but iiuc related to a different change. nothing relevant to be seen on the gerrit side. guess we need to wait whether this happens more often | 08:39 |
| noonedeadpunk | fwiw we again catching post_failures due to swift upload timeouts | 08:42 |
| *** ysandeep|afk is now known as ysandeep | 09:10 | |
| *** ykarel_ is now known as ykarel|lunch | 09:19 | |
| *** rlandy|out is now known as rlandy | 10:25 | |
| *** ykarel|lunch is now known as ykarel_ | 11:14 | |
| *** dviroel|out is now known as dviroel | 11:30 | |
| bjonglez_ | hello | 12:49 |
| bjonglez_ | is https://docs.openstack.org/contributors/code-and-documentation/elastic-recheck.html up-to-date? most of the links are dead | 12:49 |
| bjonglez_ | this page is linked from https://docs.opendev.org/opendev/infra-manual/latest/developers.html#automated-testing | 12:49 |
| bjonglez_ | (which in turn is linked from failing Zuul tests in Gerrit) | 12:50 |
| frickler | bjonglez_: it is not up-to-date, we should clean that up | 12:54 |
| fungi | bjonglez_: thanks for pointing that out, yes a lot of that is dead wood we should be trimming. that suite of services was turned off a couple of months ago, and a separate effort is available here: https://governance.openstack.org/sigs/tact-sig.html#opensearch | 12:55 |
| fungi | no longer being run by the opendev sysadmins, it's an effort within the openstack tact sig | 12:56 |
| bjonglez_ | ok, thanks! | 12:58 |
| *** ralonsoh_ is now known as ralonsoh | 13:00 | |
| hrw | morning | 13:00 |
| fungi | bjonglez_: further discussion of the new opensearch.logs.openstack.org or other openstack tact sig efforts would be more on-topic in #openstack-infra | 13:08 |
| fungi | i'll start looking into the indicated documentation updates | 13:08 |
| bjonglez_ | fungi: right, thanks for mentioning it. since part of the doc was opendev, I thought I would raise it here | 13:09 |
| fungi | makes sense! | 13:09 |
| *** dasm|off is now known as dasm | 13:33 | |
| fungi | i've pushed up some changes for the openstack community documentation to address those points, but as for what to do with https://docs.opendev.org/opendev/infra-manual/latest/developers.html#automated-testing i'm open to suggestions | 13:48 |
| fungi | i feel like we're long overdue for moving the openstackisms out of the entire document, not just that section, maybe keeping only a vestigial link to the openstack contributor guide | 13:49 |
| opendevreview | Sylvain Bauza proposed openstack/project-config master: update Review-Priority label for nova related projects https://review.opendev.org/c/openstack/project-config/+/837595 | 14:14 |
| *** ysandeep is now known as ysandeep|afk | 14:19 | |
| clarkb | fungi: most of that content looks fine? Its just the e-r mention and maybe the idea that a QA team exist that is the problem? | 14:27 |
| clarkb | frickler: re ps1 vs ps2. I wonder if zuul did trigger the jobs but then when zuul tried to report back to gerrit it found gerrit was offline due to the restart ianw did? | 14:32 |
| clarkb | frickler: thats my best guess | 14:33 |
| clarkb | oh wait ps2 was pushed june 2 | 14:34 |
| clarkb | *june 20 | 14:34 |
| fungi | clarkb: yeah, i mean i can do a surgical update to address the churn around elastic-recheck bit, but i mean more generally the entire developer.rst has lots of "if you're contributing to an openstack project..." all over the place | 14:39 |
| fungi | information about contributing to openstack projects should probably be in openstack's contributor guide rather than opendev's more generic manual | 14:39 |
| clarkb | ++ | 14:42 |
| clarkb | I think that was added to help transition content out of the doc. It has probably been long enough at this point we don't need to do that anymore | 14:42 |
| fungi | a naive grep counts 27 case-insensitive mentions of "openstack" in that file (some are things like links to wiki.o.o though so not all are necessarily candidates for removal) | 14:42 |
| fungi | but, e.g., sections talking about launchpad blueprints and specs repos... | 14:43 |
| fungi | and openstack-specific bug reporting workflows | 14:44 |
| fungi | also some of the stuff we link out to on the wiki should probably be moved into infra-manual | 14:44 |
| fungi | e.g. information on formatting git commit messages and existing integration we have in gerrit/zuul for them | 14:45 |
| frickler | clarkb: yes, that all happened well after the gerrit update | 14:52 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Fix gerrit deployment dependencies https://review.opendev.org/c/opendev/system-config/+/847034 | 14:52 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Remove Gerrit 3.4 jobs https://review.opendev.org/c/opendev/system-config/+/847035 | 14:52 |
| clarkb | 847034 is a simple change that is safe to land now. 847035 should only be landed once we're happy with 3.5 and don't expect ot revert (though we can revert to our already built images if we really have to later) | 14:53 |
| frickler | and it may as well be unrelated, just the timing makes it suspicous | 14:53 |
| clarkb | frickler: ya my next best guess is that Zuul didn't see an event for that ps or that it decided no jobs should be run for some reason | 14:53 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add Gerrit 3.6 jobs https://review.opendev.org/c/opendev/system-config/+/847040 | 15:04 |
| *** ysandeep|afk is now known as ysandeep | 15:12 | |
| *** dviroel is now known as dviroel|lunch | 15:52 | |
| opendevreview | Clark Boylan proposed openstack/project-config master: Bump jeepyb gerrit image build job to 3.5 https://review.opendev.org/c/openstack/project-config/+/847057 | 15:56 |
| clarkb | 847057 joins 847034 in the list of safe to merge changes | 15:57 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Remove Gerrit 3.4 jobs https://review.opendev.org/c/opendev/system-config/+/847035 | 15:59 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add Gerrit 3.6 jobs https://review.opendev.org/c/opendev/system-config/+/847040 | 15:59 |
| *** marios is now known as marios|out | 16:11 | |
| *** pojadhav|ruck is now known as pojadhav|out | 16:11 | |
| *** jpena is now known as jpena|off | 16:24 | |
| *** dviroel|lunch is now known as dviroel | 16:38 | |
| *** ysandeep is now known as ysandeep|out | 17:37 | |
| frickler | ianw: how long do you think it will take until we can do a dib release? wondering whether it still makes sense to have a workaround for wheel builds | 20:02 |
| fungi | to clarify from the end of the meeting, shortly before the summit i was digging into more reports of audio problems for meetpad from openstack cinder folks, and discovered that one of the most effective workarounds for similar reports had actually become the default in later versions of the service config than the files we forked | 20:03 |
| ianw | frickler: yeah, i merged some stuff yesterday. but in terms of centos 9-stream, it should be working without release | 20:03 |
| fungi | which got me thinking that we should probably try to update all our meetpad configs as there are also other things we've been setting which are now set by default | 20:03 |
| frickler | ianw: oh, o.k., I can unpause builds and see what happens | 20:03 |
| fungi | but diffing them for a year+ of changes is nontrivial | 20:03 |
| fungi | so 1. i'd like to avoid making too much work for myself, but also 2. it would be nice to find a way to make it be less work next time we want to update configs as well | 20:04 |
| ianw | frickler: yeah, so we worked through upstream fixing it, the mirrors updated, then the mirrors were corrupt for a while, but that seems cleared up now to | 20:04 |
| clarkb | ya I don't think you need to diff them as much as just ensure the two or three setting we want are set | 20:04 |
| clarkb | fungi: ^ | 20:04 |
| ianw | frickler: yeah i was just asking about the pause status in matrix :) but yeah, my plan today was to monitor it :) | 20:05 |
| *** dviroel is now known as dviroel|afk | 20:06 | |
| fungi | clarkb: agreed, though just noting that we've taken multiple approaches such as noting the commit from the point where we forked the files in some of them (but not all), grouping our edits together and optionally commenting the same settings out elsewhere which makes it harder to spot when the defaults change, et cetera | 20:06 |
| clarkb | fungi: I think that is because some things were already supported via flags and others werent so had to be added directly to configs | 20:07 |
| clarkb | but ya it definitely makes it confusing | 20:07 |
| fungi | also some of the option names changed | 20:07 |
| fungi | or moved to other files | 20:07 |
| frickler | ianw: you can see that "paused" status on http://nl01.opendev.org/dib-image-list for gentoo, I just unpaused centos-9-stream | 20:08 |
| ianw | frickler: thanks! i'll watch it through today | 20:09 |
| ianw | to the dib release, i was thinking of putting in f36 support which was really nothing this time with https://review.opendev.org/c/openstack/diskimage-builder/+/844912 and doing a release | 20:10 |
| frickler | Ramereth: quite a lot of images seem to be stuck in deleting, you can see the image ids on http://nl01.opendev.org/image-list , can you check this? | 20:10 |
| ianw | we might need to manually clean them up | 20:11 |
| frickler | ianw: iirc the delete attemps show API failures in the log | 20:12 |
| frickler | don't think that we be different with a manual attempt on our side | 20:12 |
| frickler | s/we/will/ | 20:12 |
| frickler | ianw: regarding dib, my main concern were the wheel jobs, if they now work with fresh c9s, that's ok for me | 20:13 |
| ianw | they *should* when the images get out there, will monitor | 20:13 |
| frickler | will check results tomorrow, cu everyone | 20:13 |
| ianw | i'm noticing that nb03 is not responding | 20:14 |
| ianw | i've done a hard reboot. it seems like the server is up, but possibly has no network? | 20:17 |
| fungi | that's the one in linaro? | 20:17 |
| fungi | ianw: check your inbox | 20:17 |
| fungi | reply from equinix on the thread from january | 20:18 |
| ianw | didn't that say things would be going away in a month or so? | 20:19 |
| fungi | oh, end of july, yeah. so maybe not just yet | 20:19 |
| ianw | but yeah, we should have that conversation because if nb03 is going away, we should move the builder to osuosl i guess | 20:21 |
| ianw | it looks like nodepool has also been launching nodes that can't talk. i.e. i guess linaro networking is borked :/ | 20:35 |
| ianw | kevinz: ^ | 20:35 |
| ianw | i'll send mail too, as that seems to be a bit more reliable | 20:35 |
| Ramereth | frickler ianw: should all of those be deleted? | 20:54 |
| Ramereth | I haven't had time to look into why it keeps happening unfortunately | 20:55 |
| ianw | Ramereth: yeah, all of "$ curl -s http://nl01.opendev.org/image-list | grep osuosl | grep deleting" can be removed | 20:57 |
| Ramereth | ianw: k, I'll work on getting those cleaned up later today or at least by tomorrow | 21:10 |
| ianw | frickler: centos 9-stream lg, thanks -> Image file /opt/nodepool_dib/centos-9-stream-0000005774.qcow2 created... | 21:33 |
| *** dasm is now known as dasm|off | 21:38 | |
| ianw | clarkb / fungi: if you have sec to double check https://review.opendev.org/c/openstack/diskimage-builder/+/846837 and https://review.opendev.org/c/openstack/diskimage-builder/+/844912 that will add f36 to dib. nothing functional this time, thankfully, so should be straight-forward | 22:27 |
| Ramereth | ianw: will that url be automatially updated? | 22:33 |
| clarkb | ianw: trade you for https://review.opendev.org/c/openstack/project-config/+/847057 and https://review.opendev.org/c/opendev/system-config/+/847034 :) | 22:38 |
| clarkb | fungi: ^^ | 22:38 |
| fungi | Ramereth: yes, it will update automatically (though perhaps with a few minutes delay for nodepool to notice the change in image list) | 22:39 |
| Ramereth | fungi: sweet, thanks | 22:40 |
| Ramereth | i'm going through them now | 22:40 |
| fungi | thanks! | 22:40 |
| clarkb | ianw: there is a minor whitespace thing I noted in https://review.opendev.org/c/openstack/diskimage-builder/+/844912 but have approved it anyway because zuul and pyymal didn't seem to care | 22:41 |
| Ramereth | alright, I cleared those up. I'll keep an eye on it weekly and do the cleanup until I figure out what's really going on | 22:52 |
| fungi | much appreciated! | 22:55 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Run zuul cluster reboots and updates automatically https://review.opendev.org/c/opendev/system-config/+/846195 | 23:00 |
| clarkb | frickler: ^ thank you for the review I think all your comments have been addressed | 23:01 |
| ianw | clarkb: thanks, and done on the others | 23:13 |
| ianw | doesn't have to be in 846195 but it might be interesting to echo some runtime stats into statsd | 23:14 |
| ianw | just with "echo <stat> | nc graphite.opendev.org" | 23:15 |
| clarkb | ianw: I wonder if we can get timing data out of ansible directly. But then we could send states for time to restart each service? | 23:16 |
| opendevreview | Merged opendev/system-config master: Fix gerrit deployment dependencies https://review.opendev.org/c/opendev/system-config/+/847034 | 23:18 |
| ianw | i know there's been back-and-forth with ansible and timestamps, and confusion over what the time is on the executing host v the remote host | 23:19 |
| clarkb | I guess the alternative is to run 'time' ourselves and measure it directly? | 23:20 |
| ianw | or register the output of date '+%s' in a pre-task, then do again at the end, subtract * 1000 for ms and send to graphite | 23:22 |
| opendevreview | Merged openstack/project-config master: Bump jeepyb gerrit image build job to 3.5 https://review.opendev.org/c/openstack/project-config/+/847057 | 23:22 |
| clarkb | ya that might be better for tracking the whole runtime of the playbook rather than individual tasks but that alone is probably good info to trend time to restart zuul | 23:23 |
| ianw | https://opendev.org/opendev/system-config/src/branch/master/playbooks/zuul/run-production-playbook.yaml | 23:28 |
| ianw | i guess that mostly does the individual task tracking | 23:28 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Remove Gerrit 3.4 jobs https://review.opendev.org/c/opendev/system-config/+/847035 | 23:29 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add Gerrit 3.6 jobs https://review.opendev.org/c/opendev/system-config/+/847040 | 23:29 |
| clarkb | ianw: ya I should be able to look at that and see what we can add. One thing that always bothers me is when you have to mix the logic for metrics directly into the code for the thing you are actually trying to accomplish but probably worthwhile | 23:30 |
| clarkb | 847040 has finally enqueued some 3.6 image builds | 23:30 |
| clarkb | also gitea has a 1.17 rc now | 23:30 |
| clarkb | I should look into a gitea update for the rc so that we are ready once 1.17 releases | 23:31 |
| ianw | clarkb: has it turned out we don't need https://review.opendev.org/c/zuul/zuul-jobs/+/846201 ? | 23:37 |
| clarkb | ianw: correct. ykarel came up with another owrkaround and I think we can proceed with your removal change instead with appropriate notice | 23:39 |
| clarkb | I've updated my review now | 23:39 |
| ianw | ok, i'll send a note out | 23:41 |
| opendevreview | Merged openstack/diskimage-builder master: containerfile: warn if we don't have a Dockerfile https://review.opendev.org/c/openstack/diskimage-builder/+/846837 | 23:42 |
| opendevreview | Merged openstack/diskimage-builder master: Add Fedora 36 support https://review.opendev.org/c/openstack/diskimage-builder/+/844912 | 23:42 |
| clarkb | the 3.6 build fails with error: package org.apache.commons.lang does not exist | 23:52 |
| clarkb | the its-base plugin tries to pull that in | 23:53 |
| fungi | too-old jdk? | 23:53 |
| clarkb | interesting its-base has a stable-3.6 branch I wonder if that fixes it (stable-3.3 to stable-3.5 branches do not exist) | 23:53 |
| clarkb | https://gerrit.googlesource.com/plugins/its-base/+/refs/heads/stable-3.6 | 23:54 |
| clarkb | that claims it will fix this. weird | 23:54 |
| opendevreview | Merged openstack/project-config master: Re-maintenance neutron-fwaas* https://review.opendev.org/c/openstack/project-config/+/825412 | 23:54 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add Gerrit 3.6 jobs https://review.opendev.org/c/opendev/system-config/+/847040 | 23:54 |
| clarkb | And now I need to go kill blackberries with vinegar. I'll pick this back up tomorrow | 23:55 |
| fungi | i thought blackberry was already long obsolete? | 23:55 |
| clarkb | unfortunately the plant versions manages to pop up all over. Birds and other animals do a very good job of spreading them around | 23:56 |
| fungi | yeah, they're all over our yard as well | 23:56 |
| clarkb | but blackberries will pull moisture from both ends of their stalk so you cut an end off and stick it in a jar of vinegar and after a few days it ends up very dead | 23:57 |
| fungi | when i was growing up in the mountains, we'd just lay scrap lumber down across the thickets and walk out into the middle of them to pick the berries | 23:57 |
| clarkb | things have finally dried out here enough for me to do that | 23:57 |
| fungi | i was entirely unaware of the vinegar trick. i'll have to try that here | 23:57 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!