| ianw | looks like the -devel job failure is actually picking up an ansible issue - https://github.com/ansible/ansible/issues/76779. i need to debug a bit more | 00:17 |
|---|---|---|
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Call grub2-switch-to-blscfg with explicit config https://review.opendev.org/c/openstack/diskimage-builder/+/824659 | 00:34 |
| *** odyssey4me is now known as Guest822 | 00:38 | |
| fungi | ooh, i love it when our find bugs in someone else's software | 00:56 |
| fungi | er, when our jobs find bugs | 00:56 |
| *** tobias-urdin5 is now known as tobias-urdin | 02:09 | |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Call grub2-switch-to-blscfg with explicit config https://review.opendev.org/c/openstack/diskimage-builder/+/824659 | 02:56 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Don't install python3-virtualenv on centos9 https://review.opendev.org/c/openstack/diskimage-builder/+/825034 | 03:24 |
| *** artom__ is now known as artom | 03:28 | |
| *** frenzy_friday is now known as frenzyfriday|ruck | 04:36 | |
| *** ysandeep|pto is now known as ysandeep | 05:23 | |
| *** anbanerj is now known as frenzyfriday|ruck | 07:00 | |
| zbr | wdyt about the direction in which https://github.com/pypa/pip/issues/10556 is going? My impression is that pip folks seems to be too sure that they know everything about container building. | 07:39 |
| *** ysandeep is now known as ysandeep|lunch | 08:03 | |
| *** jpena|off is now known as jpena | 08:35 | |
| *** ysandeep|lunch is now known as ysandeep | 09:11 | |
| opendevreview | Sandeep Yadav proposed openstack/diskimage-builder master: Add rhel9 in grub2 element https://review.opendev.org/c/openstack/diskimage-builder/+/825070 | 09:16 |
| *** odyssey4me is now known as Guest925 | 09:22 | |
| *** ysandeep is now known as ysandeep|brb | 11:05 | |
| *** rlandy|out is now known as rlandy|ruck | 11:14 | |
| *** ysandeep|brb is now known as ysandeep | 11:14 | |
| *** dviroel|afk is now known as dviroel | 11:25 | |
| opendevreview | Aurelien Lourot proposed openstack/project-config master: Mirror charm-nova-compute-nvidia-vgpu to GitHub https://review.opendev.org/c/openstack/project-config/+/825089 | 11:27 |
| fungi | zbr: i've been closely following pep 668 and they do seem to want to find solutions for containers as "external package managers" it's just that there are a lot of competing interests to consider | 13:26 |
| fungi | for our purposes, we've been going out of our way to at least run everything as non-root users within our containers, though i expect we do still pip install some things in them as root | 13:27 |
| fungi | (though maybe not, we may be using a non-root user when executing container image builds too, i'd have to look) | 13:28 |
| opendevreview | Szymon Datko proposed zuul/zuul-jobs master: Add support for RPM packages https://review.opendev.org/c/zuul/zuul-jobs/+/825118 | 15:00 |
| zbr | fungi: sometimes using non-root user can probe a real PITA. It seems that in the end they are willing to reconsider it, even if in a bit weird way with an imposed delay of 3 months. | 15:23 |
| zbr | somehow i have the impression that pip will give the root warning even in chroot or if with `--no-deps` which are two other reasons why that warning is overzealous. | 15:25 |
| fungi | yep, but at least it _is_ merely a warning/reminder | 15:34 |
| opendevreview | Tristan Cacqueray proposed opendev/system-config master: Update gerritbot-matrix version to include ChangeReady event https://review.opendev.org/c/opendev/system-config/+/825131 | 15:49 |
| *** ysandeep is now known as ysandeep|out | 16:04 | |
| clarkb | I'm going to approve https://review.opendev.org/c/opendev/system-config/+/823026 now and we can sort out a restart of gerrit later today after that has merged | 16:18 |
| clarkb | This is the 3.3.9 bump | 16:19 |
| *** dviroel is now known as dviroel|lunch | 16:21 | |
| fungi | thanks! | 16:24 |
| clarkb | re pip I think there are bigger issues when it comes to giving people unhelpful info in pip output. Particularly the combo of unreliable CDN and constraints saying it can't find a valid version could be far more clear. A warning explaining that pip as root is not recommended seems reaosnable since it isn't recommended and does have problems when you do that | 16:33 |
| clarkb | fungi: fwiw I think many of our container image builds install as root before switching the user | 16:33 |
| opendevreview | Gustavo Sanchez proposed openstack/project-config master: Mirror cinder-solidfire and cinder-nimblestorage to Github https://review.opendev.org/c/openstack/project-config/+/825138 | 16:37 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Install Limnoria from upstream https://review.opendev.org/c/opendev/system-config/+/821331 | 16:40 |
| clarkb | fungi: ianw ^ I looked closer at limnoria and while they don't seem to have recent proper releases they do checkpoint tags of master and there is one fromthe first of the year. I think we start there rather than master proper | 16:41 |
| fungi | yeah, part of the unhelpful messaging from pip is that it silently ignores sdists with build failures and tries older and older versions until it gets one to build, which can lead to you running much older versions of dependencies than expected if you don't have a sufficiently high lower-bound | 16:49 |
| artom | How would I check whether Zuul can use a particular ansible module? ansible.posix.mount in this case specifically, but the question can remain general | 17:03 |
| clarkb | artom: it can use anything you get out of a `pip install ansible` then there are a couple of additional things like the openstack and kubernetes modules too iirc. But generally anything you get out of pip installing ansible should work | 17:04 |
| fungi | just not on the executor | 17:04 |
| artom | clarkb, huh cool, I remember reading somewhere that Zuul's ansible was limited | 17:04 |
| artom | fungi, ah, that was it, it's limited on the executor | 17:05 |
| clarkb | right well it is limited as fungi points out but the modules are there. Depends more on where you execute them | 17:05 |
| fungi | artom: i assume your question is about *our* zuul deployment, not about your own zuul install somewhere? | 17:05 |
| artom | fungi, heh, yes | 17:05 |
| artom | Yeah, this is being executed (well, attempted to be executed, because it's complaining of a module not found) on the target CI VM. | 17:06 |
| clarkb | I suspect the issue is more that you are trying to use modern ansible collection syntax with zuul and it hasn't updated its ansible for a while (focus has been on scale out scheduelr stuff but expect ansible updates after 5.0 happens) | 17:06 |
| fungi | artom: more generally, you can find what zuul shadows by default from its ansible installation on the executor here: https://opendev.org/zuul/zuul/src/branch/master/zuul/ansible | 17:06 |
| clarkb | https://docs.ansible.com/ansible/2.9/modules/mount_module.html is probably better doc for that module | 17:07 |
| clarkb | (note it doesn't use the collection namespacing stuff) | 17:07 |
| * artom goes to figure out what this newfangled ansible collection stuff is, and why the same module can be references as two different things, and which one will work is apparently a crapshoot? | 17:09 | |
| clarkb | artom: it was/is a whole thing. Ansiblefest 2019 red hat/ansible said "we're doing this" and then everyone has been trying to get up to speed since. But basically they took ansible and split it up into many many repos as collections of ansible modules. A number of them are included by default when you install ansible but many others are not and have to be installed separately | 17:10 |
| clarkb | artom: I don't think absible 2.9 understands the syntax of ansible.posix.mount. It is just 'mount' on that version of ansible | 17:11 |
| *** marios is now known as marios|out | 17:11 | |
| * artom apologises on behalf of his employer. | 17:11 | |
| clarkb | but the collections have that namespacing because you might install a collection that is artom.tools.mount ot whatever | 17:11 |
| jrosser | in later ansibles there is a mapping between the previous unqualified name and the fully namespaced equivalent | 17:11 |
| jrosser | so it should work either way | 17:12 |
| clarkb | jrosser: right but only on newer ansible which zuul doesn't support yet | 17:12 |
| fungi | ahh, so continuing to use "mount" should be forward-compatible, at least for a while | 17:12 |
| clarkb | at least that was my understanding | 17:12 |
| clarkb | fungi: ya | 17:12 |
| jrosser | fungi: yes old code will still work | 17:12 |
| fungi | just not the other way around is, i guess, what clarkb was getting at | 17:13 |
| clarkb | yup | 17:13 |
| jrosser | sure | 17:13 |
| fungi | so "it should work either way" _on newer ansible_ but if you use the old syntax you'll be able to make it work on old and new ansible | 17:14 |
| jrosser | yes, i wasnt too specific there :/ | 17:14 |
| *** dviroel|lunch is now known as dviroel | 17:19 | |
| *** jpena is now known as jpena|off | 17:33 | |
| opendevreview | Merged opendev/system-config master: Update Gerrit images to 3.3.9 and 3.4.3 https://review.opendev.org/c/opendev/system-config/+/823026 | 17:39 |
| opendevreview | Merged opendev/system-config master: Update gerritbot-matrix version to include ChangeReady event https://review.opendev.org/c/opendev/system-config/+/825131 | 17:46 |
| ianw | what is the chance that zuul autoholds aren't autoholding as much as we'd like? | 19:35 |
| ianw | 0000000000 | openstack | opendev/system-config | system-config-run-review-3.4 | .* | 1 | ianw: gerrit-3.4 hold | 19:35 |
| ianw | i put that in yesterday | 19:35 |
| clarkb | ianw: did the change not fail maybe? | 19:35 |
| clarkb | s/change/job/ | 19:35 |
| ianw | https://zuul.opendev.org/t/openstack/build/412485bff9ff4b44b1f67436ad0f9f67/ was the job i'd expect to hit it | 19:36 |
| ianw | yeah i got distracted on ansible issues yesterday, i'll run it through now and if it doesn't hit can debug in a more immediate fashion | 19:51 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [dnm] force fail of gerrit-3.4 jobs https://review.opendev.org/c/opendev/system-config/+/815933 | 19:52 |
| fungi | my workstation crashed toward the end of the meeting, but i'll start bringing up and updating the jitsi servers once i've got it back online | 19:57 |
| fungi | okay, got sidetracked with some deferred package updates since i wanted to make sure i took the opportunity to update the kernel as well, but working on meetpad bits now. i'll start with the jvb servers | 20:50 |
| ianw | ok, it appears the gerrit job hasn't held itself again | 21:01 |
| clarkb | exciting. corvus ^ fyi I half wonder if that is an sos bug in zuul | 21:01 |
| fungi | anything useful about it in the debug log? | 21:02 |
| clarkb | https://etherpad.opendev.org/p/9Cy4GFmrFCK1yLpxgs5V does a gerrit downtime/maintenance/upgrade notification need more info than that? | 21:19 |
| fungi | lgtm. i added a link to the renames list | 21:23 |
| *** dviroel is now known as dviroel|out | 21:26 | |
| corvus | i think the autohold thing is a slight behavior change in zuul-client compared to the old zuul command. requires specifying the full canonical repo name now. | 21:29 |
| clarkb | ah | 21:30 |
| clarkb | in this case opendev.org/opendev/system-config ? | 21:30 |
| corvus | ya | 21:30 |
| fungi | i've upgraded all the jitsi container images on our meetpad/jvb servers and started services back up, if anyone wants to test it out (i'm not currently able to do a/v testing myself, but poked at the browser-accessible features at least and everything seems to be functional) | 21:31 |
| fungi | if anyone else wants to test, please do so | 21:31 |
| fungi | i'll hold off on the status log for it until folks have a chance to confirm it seems to be working | 21:32 |
| clarkb | if anyone else has a mic working I can hop on a call | 21:32 |
| corvus | sure, let me fix my hair | 21:33 |
| corvus | https://meetpad.opendev.org/test2022 | 21:34 |
| fungi | i can see your video stream | 21:35 |
| fungi | and it didn't kill the shared document, that's a good sign | 21:36 |
| fungi | once someone with working mic/speakers joins you'll hopefully be able to confirm audio is working | 21:36 |
| clarkb | oh sorry grabbed a glass of water. I'll join momentarily | 21:38 |
| corvus | omg reaction sounds | 21:38 |
| fungi | i can tell my browser was trying to play some sound effect, just not the audio stream. i need to dig into what the deal is with my system | 21:39 |
| fungi | it has "polls" now too, is that new? | 21:39 |
| fungi | and "break out rooms" | 21:39 |
| fungi | i probably just wasn't paying close attention to available features in the past | 21:40 |
| clarkb | fungi: note it says I am joining muted but then when I join I am not muted | 21:41 |
| fungi | oh, neat | 21:41 |
| fungi | i wonder if something changed with how we need to configure that | 21:41 |
| fungi | we set it to mute video on joining, is that what you're talking about? | 21:41 |
| clarkb | yes that | 21:44 |
| clarkb | it does give me a pop up saying I'm starting muted but then I am not | 21:44 |
| clarkb | which is probably more problematic | 21:44 |
| fungi | starting with your camera muted, specifically? | 21:46 |
| fungi | (not audio?) | 21:46 |
| clarkb | no audio. I don't have a camera | 21:46 |
| fungi | ahh, i thought we configured it to only mute video on joining, not audio | 21:46 |
| clarkb | fungi: yup the .env file on meetpad agrees. I think the regression is that I get a pop up saying I'm starting muted | 21:48 |
| clarkb | But maybe it means my video is muted and I've misread it. I'll reconnect and read carefully | 21:48 |
| fungi | whereby it means muted video but doesn't specify maybe | 21:48 |
| fungi | okay, but you were also able to see corvus's video, and the two of you could hear one another? | 21:49 |
| clarkb | "You have started the conversation muted" is what it says so it is ambiguous | 21:49 |
| clarkb | yes I could see corvus' camera and we could hear each other | 21:49 |
| fungi | i only get the join/part/reaction sound effects, but am fairly certain it's my browser or pulseaudio | 21:49 |
| clarkb | I think we should consider setting the setting to start audio as muted too to make that ambiguity of the message less painful. But I don't think this is regression | 21:49 |
| fungi | thanks | 21:50 |
| fungi | #status log The meetpad.opendev.org services have returned to working order, and are available for use once again | 21:51 |
| opendevstatus | fungi: finished logging | 21:51 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Start meetpad meetings with muted audio https://review.opendev.org/c/opendev/system-config/+/825179 | 21:55 |
| clarkb | I think ^ will make the muting pop up far less ambiguous and confusing | 21:55 |
| clarkb | I'll send out the gerrit maintenance announcement momentarily | 21:57 |
| clarkb | corvus: for the restart to pick up https://review.opendev.org/c/zuul/zuul/+/824864/ there isn't any reason to coordinate that with our gerrit update to the latest 3.3.9 image right? We can do that one as a rolling restart? | 22:17 |
| corvus | clarkb: correct; i'm also thinking maybe we should get 825042 and 825079 in. maybe 824706 as well. | 22:24 |
| clarkb | cool I'll proceed with the gerrit restart when I'm ready for that. Probably in 45 minutes or so. Warning the openstack release team now | 22:25 |
| clarkb | I'll also look at those changes | 22:25 |
| clarkb | infra-root there are a few tripleo changes that look close to merging. I'll give those a few minutes to clear out then docker-compose pull && docker-compose down && docker-compose up -d on review02 | 23:02 |
| *** lajoskatona_ is now known as lajoskatona | 23:04 | |
| ianw | ++ | 23:09 |
| clarkb | down to one job now holding up a bunch of potential merges :) | 23:10 |
| clarkb | if I had known the zuul estimate wasn't too accurate I could've been done by now :) but better safe than sorry when it takes that long to gate those changes | 23:18 |
| clarkb | the job is in the post run activities now looks like | 23:28 |
| clarkb | these tripleo jobs spend a lot of time just moving data around at the end of a job :/ | 23:32 |
| fungi | due to the sheer number of "log" files saved | 23:32 |
| clarkb | 2022-01-18 23:07:43.952093 | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml@master] <- I think that means I don't need to care about it | 23:33 |
| clarkb | I went ahead and ran a pull. mariadb and gerrit images updated. The gerrit image matches what I see in docker hub from today's update. Any objection to a down and up -d now? | 23:34 |
| clarkb | sounds like no? I'll proceed with the gerrit update to 3.3.9 then | 23:36 |
| clarkb | actually let me start a screen | 23:36 |
| clarkb | ok screen started on review. Proceeding with restart in that screen | 23:37 |
| clarkb | web ui is up for me and I see the expected 3.3.9 version in the footer | 23:39 |
| clarkb | #status Log upgraded Gerrit on review02 to 3.3.9 | 23:43 |
| opendevstatus | clarkb: finished logging | 23:43 |
| clarkb | I don't have any changes to push right now but it seems to be working on the web side | 23:43 |
| clarkb | ianw just pushed a change to zuul and that was detected by matrix gerritbot | 23:44 |
| clarkb | and now zuul has merged a zuul change through gerrit. All looking good there | 23:47 |
| *** rlandy|ruck is now known as rlandy|out | 23:50 | |
| clarkb | I have closed the screen | 23:56 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!