| opendevreview | Michael Kelly proposed zuul/zuul-jobs master: helm: Add job for linting helm charts https://review.opendev.org/c/zuul/zuul-jobs/+/861799 | 00:21 |
|---|---|---|
| opendevreview | Merged zuul/zuul-jobs master: Make test-registry-buildset-registery-k8s-<docker|crio> non-voting https://review.opendev.org/c/zuul/zuul-jobs/+/863582 | 00:59 |
| opendevreview | Merged zuul/zuul-jobs master: ensure-podman/skopeo : update for Jammy https://review.opendev.org/c/zuul/zuul-jobs/+/863575 | 00:59 |
| opendevreview | Merged zuul/zuul-jobs master: ensure-pip: use dummy package for testing https://review.opendev.org/c/zuul/zuul-jobs/+/863570 | 00:59 |
| opendevreview | Merged zuul/zuul-jobs master: linter: Use capitals for names https://review.opendev.org/c/zuul/zuul-jobs/+/854933 | 01:01 |
| opendevreview | Merged zuul/zuul-jobs master: Fix ansible-lint name[template] https://review.opendev.org/c/zuul/zuul-jobs/+/861559 | 01:06 |
| opendevreview | Merged zuul/zuul-jobs master: Add names to include tasks https://review.opendev.org/c/zuul/zuul-jobs/+/861560 | 01:08 |
| opendevreview | Merged zuul/zuul-jobs master: Standarise block/when ordering https://review.opendev.org/c/zuul/zuul-jobs/+/861562 | 01:10 |
| opendevreview | Merged zuul/zuul-jobs master: Update to ansible-lint 6.8.2 https://review.opendev.org/c/zuul/zuul-jobs/+/861563 | 01:10 |
| opendevreview | Merged zuul/zuul-jobs master: zuul-tests: autogenerate some more roles https://review.opendev.org/c/zuul/zuul-jobs/+/863578 | 01:10 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: Combine ensure-pip playbooks into a test role https://review.opendev.org/c/zuul/zuul-jobs/+/860296 | 03:09 |
| ianw | clarkb: just a bit confused trying to follow https://review.opendev.org/c/opendev/system-config/+/862152, but also probably haven't fully recovered everything that goes on there from my mental swap partition :) | 04:09 |
| ianw | comment inline | 04:09 |
| Clark[m] | ianw: the first step we do with the build tool makes a single wheel for the package we want to install. My understanding of the `pip wheel` command is that it is supposed to make wheels for everything necessary to install the item(s) listed | 04:16 |
| Clark[m] | So what that second step does is say make wheels for all of our deps | 04:17 |
| Clark[m] | And according to the pypa/pip upstream this is the correct way to have pip make wheels for everything you need to install instead of relying on the cache | 04:17 |
| Clark[m] | So you can give `pip wheel` a package name, an sdist, a requirements file, a wheel file etc and it is supposed to sort out wheels for everything from that | 04:19 |
| *** yadnesh|away is now known as yadnesh | 05:28 | |
| ianw | thanks for clarifying. i indeed had a play with it and agree, all lgtm | 05:37 |
| *** marios is now known as marios|ruck | 05:57 | |
| timburke | anybody else seen something like this lately when running `git review --setup`? seems i can't copy the commit hook https://paste.opendev.org/show/bCTQw3ned8goB6wuK9P8/ | 06:11 |
| timburke | (i mean, *i* can always copy it from one of the many other local checkouts i've got that's already set up -- but i pity the poor newcomers...) | 06:12 |
| frickler | timburke: can you run the scp command manually? for me this is still working fine, fwiw | 06:36 |
| timburke | frickler, nope; same (slightly ominous) "subsystem request failed on channel 0" | 06:37 |
| timburke | tacking on a -v and only looking at the bits following "Server accepts key": https://paste.opendev.org/show/bpf1uud9wNKy5t5molWM/ | 06:39 |
| frickler | timburke: what OS and ssh version is this? the XMODIFIERS env looks weird and I also only see "pledge: filesystem" in my run, lacking to "full" | 06:46 |
| timburke | frickler, Fedora 36, OpenSSH_8.8p1, OpenSSL 3.0.5 5 Jul 2022 | 06:48 |
| timburke | not sure how i picked up XMODIFIERS, but unsetting it things fail the same way | 06:50 |
| frickler | was there some recent change to those versions, which might be related? On Jammy I have "OpenSSH_8.9p1 Ubuntu-3, OpenSSL 3.0.2 15 Mar 2022" | 06:51 |
| frickler | other than that maybe some issue with your network environment. do you have a chance to test from a different host with a different OS? | 06:54 |
| timburke | trying on a jammy VM, things are fine -- must be something with those version differences :-/ | 06:57 |
| timburke | relevant openssl changelog: https://paste.opendev.org/show/b9AWuzI0fFw5Mtturp26/ | 06:57 |
| timburke | and openssh: https://www.openssh.com/txt/release-8.9 | 06:59 |
| timburke | increasing the verbosity (and dropping XMODIFIERS): https://paste.opendev.org/show/bvWvuhdIyuve7ajmXJ9q/ | 07:03 |
| frickler | timburke: ah, that ssh reno mentions switching to sftp. if I do that by adding "-s" to the scp command, I get the same error. so maybe fedora switched the default already. can you check whether you have a flag to switch back to scp mode? | 07:05 |
| timburke | -O seems to have done it -- https://paste.opendev.org/show/b0RLM3cr9NBxZZD85jDC/ | 07:10 |
| timburke | thanks frickler~ | 07:10 |
| noonedeadpunk | hey there! Would be great to get https://review.opendev.org/c/openstack/project-config/+/863158/ landed this week as we want to make stuff in time for Zed (we're trailing so need to release until 16 of December). | 07:13 |
| *** yadnesh is now known as yadnesh|afk | 08:09 | |
| jrosser | timburke: does this help in newer git-review? https://opendev.org/opendev/git-review/commit/5bfaa4a6f355a6820fe16c1aea77a01ba7b97eaa | 08:21 |
| frickler | jrosser: at least that seems to match the issue exactly, thx for the pointer | 08:49 |
| *** yadnesh|afk is now known as yadnesh | 08:53 | |
| *** dviroel_ is now known as dviroel | 09:40 | |
| opendevreview | Tama McGlinn proposed opendev/git-review master: Add message option https://review.opendev.org/c/opendev/git-review/+/864098 | 09:56 |
| opendevreview | Tama McGlinn proposed opendev/git-review master: Add message option https://review.opendev.org/c/opendev/git-review/+/864098 | 10:29 |
| opendevreview | Tama McGlinn proposed opendev/git-review master: Add message option https://review.opendev.org/c/opendev/git-review/+/864098 | 11:13 |
| opendevreview | Tama McGlinn proposed opendev/git-review master: Add message option https://review.opendev.org/c/opendev/git-review/+/864098 | 11:19 |
| opendevreview | Dr. Jens Harbott proposed opendev/git-review master: Fix nodesets for tox and add py310 job https://review.opendev.org/c/opendev/git-review/+/864106 | 11:46 |
| fungi | frickler: timburke: yes, openssh deprecated the old scp protocol and expect "modern" scp servers to speak sftp protocol now | 12:43 |
| fungi | and yeah, that should be addressed with git-review>=2.3 | 12:44 |
| frickler | fungi: the py310 tests for git-review that I tried to add above are failing, likely some request mocking needs to be fixed, do you want to have a look at that? | 12:59 |
| frickler | also interesting coincidence to see so much git-review action happening on the same day | 13:00 |
| fungi | frickler: i had started an attempt at updating to newer gerrit/java/ubuntu for git-review testing, but haven't made a lot of progress. if it's a matter of older gerrit not running on new enough javascript then that may need to be picked back up | 13:03 |
| fungi | there is no "mocking" in the git-review tests. they actually install and run a gerrit server | 13:03 |
| fungi | but since that test setup has to bootstrap a working gerrit account to test with, and that breaks in various ways in newer gerrit versions | 13:04 |
| fungi | ...it's nontrivial | 13:05 |
| frickler | fungi: ah, o.k., then I'll drop py310 from my patch again for now and just fix the CI | 13:05 |
| fungi | https://review.opendev.org/849419 was as far as i got with it | 13:06 |
| opendevreview | Dr. Jens Harbott proposed opendev/git-review master: Fix nodesets for tox jobs https://review.opendev.org/c/opendev/git-review/+/864106 | 13:06 |
| fungi | i guess i could push it forward to the version of gerrit we're running now that the ssh host key issue is addressed | 13:07 |
| frickler | I was just thinking whether the failures might be due to new openssh in 22.04, then, that sounds plausible | 13:08 |
| fungi | oh, that will almost certainly break for the gerrit version we test with there unless some workarounds are added, yes | 13:10 |
| fungi | or unless we can get the account setup worked out for a recent gerrit version | 13:10 |
| frickler | hmm, why does gerrit offer commits from different projects to choose from in the rebase menu? doesn't make any sense at all to make. trying to rebase https://review.opendev.org/c/opendev/git-review/+/864098 on top of 864106 | 13:16 |
| opendevreview | Dr. Jens Harbott proposed opendev/git-review master: Add message option https://review.opendev.org/c/opendev/git-review/+/864098 | 13:16 |
| fungi | i've not noticed that, but i also don't recall ever trying that rebase feature | 13:40 |
| *** slaweq_ is now known as slaweq | 14:09 | |
| *** dasm|off is now known as dasm | 14:10 | |
| opendevreview | Vishal Manchanda proposed zuul/zuul-jobs master: [DNM] Test horizon with removing install chromium task https://review.opendev.org/c/zuul/zuul-jobs/+/864119 | 14:19 |
| *** dviroel is now known as dviroel|lunch | 15:50 | |
| clarkb | frickler: I confirm that rebase behavior when you switch to rebasing on a specific change. That seems worth filing a bug for. I can do that if you prefer (I already have an account set up and everything) | 16:01 |
| clarkb | https://github.com/jitsi/jitsi-meet/pull/11338 added beta local recording support to jitsi meet | 16:17 |
| clarkb | gmann tested this and reported there is an echo in the audio. | 16:17 |
| clarkb | I think recordrtc as a browser plugin is another option that people can try (this is what I thought gmann had tested initially as i didn't realize jitsi meet had recently added this featre) | 16:17 |
| fungi | neat, and good to know they added client-side recording built in, even if it's still suboptimal for the moment | 16:22 |
| jrosser | clarkb: i just tested rescue here and it works for me on bvf instance using two completely different images, centos and ubuntu | 16:26 |
| clarkb | jrosser: do you have to set any properties on the image you use for rescuing? melwitt pointed me to the tempest testing for rescue and in the bfv case it sets a bus and device type on the image | 16:27 |
| jrosser | and i have "hw_rescue_bus='scsi', hw_rescue_device='disk'" properties on all images | 16:27 |
| clarkb | yup I suspect ^ is what is needed | 16:27 |
| melwitt | ack | 16:28 |
| clarkb | jrosser: thank you for testing that. I think that points us in a direction that should hopefully get this working cc mnaser__ | 16:29 |
| jrosser | this is how it looked inside the ubuntu that rescued the centos https://paste.opendev.org/show/bshyaqKvq96PTZl83HT4/ | 16:29 |
| *** marios|ruck is now known as marios|out | 16:30 | |
| clarkb | jrosser: the other thing to check is `blkid` to see if you have colliding labels and your grub config for any root label specifiers. I suspect that this isn't an issue though as those filesystems seem to have done the correct thing | 16:30 |
| opendevreview | Vishal Manchanda proposed zuul/zuul-jobs master: Remove "Install chromium-brower" task to fix CI. https://review.opendev.org/c/zuul/zuul-jobs/+/864119 | 16:31 |
| jrosser | clarkb: https://paste.opendev.org/show/bKnP2L2oLLvZruELEa2d/ | 16:31 |
| clarkb | jrosser: thanks that confirms no colliding label values | 16:32 |
| clarkb | that ubuntu image must be booting by uuid instead | 16:32 |
| clarkb | anyone know if Visahl is on IRC or matric re 864119? | 16:33 |
| fungi | clarkb: vishalmanchanda in #openstack-infra and #openstack-horizon | 16:34 |
| fungi | i think we discussed that problem last week, or at least i remember looking into the job failures | 16:35 |
| fungi | yeah, it was 2022-11-02 12:24:15 in #openstack-infra | 16:36 |
| *** yadnesh is now known as yadnesh|away | 16:42 | |
| *** dviroel|lunch is now known as dviroel | 16:46 | |
| frickler | clarkb: yes, please create a bug report, thx | 16:49 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Cleanup uwsgi build hacks https://review.opendev.org/c/opendev/system-config/+/861925 | 17:10 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Switch python-builder/python-base to pip wheel https://review.opendev.org/c/opendev/system-config/+/862152 | 17:10 |
| clarkb | ianw pointed out that a recent uwsgi release claims to ahve fixed the issues I was trying to workaround with the old change | 17:10 |
| clarkb | I've removed the workarounds and the pip wheel change should exercise uwsgi builds | 17:11 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Install dbus-user-session along with xvfb and firefox https://review.opendev.org/c/zuul/zuul-jobs/+/864137 | 18:22 |
| *** dviroel is now known as dviroel|biba | 19:51 | |
| *** dviroel|biba is now known as dviroel|biab | 19:51 | |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Add summit-track-chairs ML in place of old name https://review.opendev.org/c/opendev/system-config/+/864143 | 20:04 |
| fungi | infra-root: can i get https://review.opendev.org/864143 expedited, the conference organizers are wanting to start using the new ml today so i'll take care of the manual steps after that's deployed | 20:16 |
| ianw | does removing the list delete it? i see you plan to manually migrate subscribers | 20:19 |
| fungi | me running `rmlist` deletes it, but i don't want to do that until the change deploys so that ansible won't recreate it | 20:20 |
| fungi | i've had that happen to me more than once, learned my lesson eventually | 20:20 |
| ianw | :) | 20:21 |
| opendevreview | Merged opendev/system-config master: zuul reboot: pin to bridge01.opendev.org https://review.opendev.org/c/opendev/system-config/+/863568 | 20:35 |
| opendevreview | Merged opendev/system-config master: Remove bridge.openstack.org https://review.opendev.org/c/opendev/system-config/+/863564 | 20:35 |
| clarkb | ya the ansible stuff for mailman doesn't know how to delete things just create them | 20:35 |
| clarkb | fungi: is it ok for me to approve that change and have ansible create the new list? | 20:36 |
| fungi | clarkb: yes, that's the plan. let ansible create the new list and then i'll diff the list configs and set similar options on the one one and move stuff over | 20:37 |
| fungi | feel free to approve if you're good with it | 20:37 |
| clarkb | done | 20:37 |
| clarkb | I wasn' sure if you wanted to create it first too and have ansible simply reflect that config. Approved now | 20:38 |
| fungi | i considered doing it that way, but the more i thought about it this was simpler | 20:38 |
| opendevreview | Ian Wienand proposed opendev/system-config master: inventory: add host keys https://review.opendev.org/c/opendev/system-config/+/862762 | 20:57 |
| opendevreview | Ian Wienand proposed opendev/system-config master: bastion host: add global known_hosts values https://review.opendev.org/c/opendev/system-config/+/862764 | 20:57 |
| ianw | ^ does anyone have thoughts on that? I feel like we've discussed it before ... but perhaps not in the context of the bridge bootstrap steps? | 20:58 |
| fungi | i've suggested it in the past, and am wholly in favor of the idea | 20:59 |
| fungi | the alternative was relying on sshfp records, but you need custom client configuration to have openssh auto-accept unknown host keys based solely on sshfp | 21:00 |
| fungi | i suppose we could add it in system-config the same time we add the ip addresses to the inventory | 21:01 |
| ianw | yeah and istr we hit issues with ports and sshfp records | 21:01 |
| ianw | that approach is perhaps different to others because it reads the inventory in the bridge bootstrap and writes the values from that; rather than actually being in the nested ansible when *ansible* has read the inventory | 21:02 |
| fungi | trying to have more than one sshfp record for different ports yes, and another problem is that openssh won't auto-update known hosts if the host key associated with a name changes (e.g. after server replacement) so sshfp is complicated in that case too | 21:02 |
| ianw | ... that's a really interesting idea about essentially a dynamic known_hosts ... i wonder if you can do that | 21:04 |
| opendevreview | Merged opendev/system-config master: Add summit-track-chairs ML in place of old name https://review.opendev.org/c/opendev/system-config/+/864143 | 21:04 |
| fungi | ianw: one thing which rubs me the wrong way there is the embedded hostnames in the entries. we should probably be populating those from the inventory? | 21:06 |
| fungi | i know it complicates the role though | 21:07 |
| fungi | not a blocker, but it seems... redundant | 21:07 |
| ianw | quite possible -- really that's an artifact of me running ssh-keyscan and > the output to a file, then running an emacs macro that grabbed 3 lines from that and pasted them in below the hostname | 21:08 |
| ianw | so basically that was the raw output from ssh-keyscan | 21:08 |
| fungi | and if they don't match 100% then it will be useless | 21:08 |
| fungi | oh, also renames will still be a problem unless we configure the client to not write its own known_hosts entries as well | 21:10 |
| fungi | because by default, it's going to go "okay i recognize this host key matches the name i'm connecting to, and for extra safety i'll also write my own copy of it with the ip address so that it's recognizable if i don't connect by name some time later" | 21:11 |
| ianw | yeah i was hoping to basically remove /root/.ssh/known_hosts | 21:11 |
| fungi | we can mitigate that by also embedding the ip addresses in the host records, but at that point you're definitely going to want to dynamically generate them from other inventory data | 21:11 |
| fungi | otherwise, you'll end up with stale (redundant) entries in /root/.ssh/known_hosts which won't get updated when we change or replace entries in /etc/ssh/ssh_known_hosts and then the client will quite likely refuse to connect | 21:13 |
| fungi | though that's also worth testing, i suppose | 21:13 |
| ianw | ++ i think that's a good point to ditch those hostnames in there, and write out dynamically the hostname, and potentially public_ipv4,public_ipv6 | 21:14 |
| fungi | i'm honestly not sure what happens if ssh finds host keys in both files and they're completely different keys | 21:14 |
| ianw | i was thinking the /etc because sometimes admin users will bounce through, and so it should allow connecting for everyone | 21:15 |
| fungi | yeah, i like it for that reason as well | 21:16 |
| ianw | the launch-node on jammy can give us the right values now too. the bionic era ssh-keyscan didn't have the flags to output the values | 21:20 |
| fungi | awesome | 21:20 |
| corvus | this all sounds great for in-system -- i like sshfp for remote use (ie, our initial connections to servers from our workstations) | 21:41 |
| corvus | (so i think we should keep putting those records in dns for that reason) | 21:42 |
| opendevreview | James E. Blair proposed opendev/system-config master: Add ssl support to zookeeper-statsd and fix latency handling https://review.opendev.org/c/opendev/system-config/+/864146 | 21:46 |
| corvus | i think that will solve the longstanding mystery of our zero-latency zk cluster :) | 21:46 |
| clarkb | corvus: are some stats only available over ssl? | 21:48 |
| clarkb | ianw: I think the host key stuff is fine, the only thought I've got is we should update our launch node tooling and/or readme to make it easy to update that file too. | 21:50 |
| clarkb | we already write out the inventory file contents, maybe update that to emit the host key data in the block as well? | 21:50 |
| corvus | clarkb: no, that's a 2-in-1 change: 1) add ssl; 2) fix type handling. | 21:51 |
| clarkb | oh i see | 21:51 |
| corvus | they are independent, but when i tested that locally it was as a unit. seemed harmless to leave them combined. | 21:52 |
| fungi | corvus: agreed on also keeping the sshfp records, my point was they don't solve the "our automation needs to validate the keys" problem, but as long as we don't mind adding host keys to the inventory and dns, having both makes things smoother | 21:55 |
| corvus | ++ | 21:55 |
| fungi | we have to add stuff to both files anyway when setting up a new server, so it's already "copy/paste this blob" | 21:56 |
| fungi | (both files being the ansible inventory and the bind zonefile) | 21:56 |
| fungi | oof, that bridge change which merged has triggered 32 deploy jobs | 21:58 |
| fungi | this will be a while | 21:59 |
| ianw | fungi: https://review.opendev.org/c/opendev/system-config/+/820651 and https://review.opendev.org/c/opendev/system-config/+/820320/ I think are getting pretty close to being able to run 31 of those in parallel | 22:11 |
| ianw | maybe not quite; there's still other dependencies with letsencrypt etc. but more parallel than one at a time, anyway :) | 22:11 |
| ianw | corvus: lgtm but is there a reason to not just make it a float always? | 22:19 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [dnm] trigger openafs testing https://review.opendev.org/c/opendev/system-config/+/864148 | 22:36 |
| corvus | ianw: good q. all of the metrics are integers except one, and i honestly don't recall if the statsd-graphite part of the system would behave differently or not there. seems like there's a possibility we could end up with a 1 ~= 0.9999999 situation. in other words, i'm being conservative. | 22:45 |
| ianw | ok, seems fine | 22:48 |
| ianw | ... in things that are not fine, openafs-1.8.8.1 is indeed failing to build on 9-stream; which is holding up the wheel release | 22:48 |
| ianw | i bet this is fixed in 1.8.9pre1 ... but pre releases have not been great for us in the past | 22:49 |
| ianw | ... http://git.openafs.org/?p=openafs.git;a=commit;h=7a181415db1b2142d125714f1dea32879e2ca07d | 22:51 |
| opendevreview | Nikolay Vinogradov proposed openstack/project-config master: Add the cinder-infinidat charm to Openstack charms https://review.opendev.org/c/openstack/project-config/+/863954 | 23:00 |
| *** dasm is now known as dasm|offp | 23:03 | |
| *** dasm|offp is now known as dasm|off | 23:03 | |
| clarkb | ianw: did centos 9 get a new kernel? | 23:09 |
| clarkb | seems odd that it owuld break otherwise | 23:09 |
| clarkb | also left a question inline | 23:11 |
| opendevreview | Ian Wienand proposed opendev/system-config master: openafs: copy dkms log directory https://review.opendev.org/c/opendev/system-config/+/864148 | 23:12 |
| clarkb | ianw: on the etherpad syslogging change what ensures the docker-foo tagged log lines end up in /var/log/containers? Is that just config we put on all hosts? | 23:13 |
| ianw | clarkb: yep, ensure-docker does that -> https://opendev.org/opendev/system-config/src/commit/e9e63f1d524824e0cd7d31e3ae13ff46465de068/playbooks/roles/install-docker/tasks/main.yaml#L47 | 23:14 |
| clarkb | I have confirmed the logs are there in the test job so change lgtm | 23:14 |
| clarkb | thanks | 23:14 |
| opendevreview | Nikolay Vinogradov proposed openstack/project-config master: Add the cinder-infinidat charm to Openstack charms https://review.opendev.org/c/openstack/project-config/+/863954 | 23:26 |
| opendevreview | Nikolay Vinogradov proposed openstack/project-config master: Add the infinidat-tools charm to Openstack charms https://review.opendev.org/c/openstack/project-config/+/863955 | 23:31 |
| opendevreview | Nikolay Vinogradov proposed openstack/project-config master: Add manila-infinidat backend charm to OpenStack charms https://review.opendev.org/c/openstack/project-config/+/863957 | 23:33 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: enable-kubernetes: Fix jammy install, improve pod test https://review.opendev.org/c/zuul/zuul-jobs/+/863810 | 23:40 |
| opendevreview | Merged zuul/zuul-jobs master: test-registry-post: collect k8s logs https://review.opendev.org/c/zuul/zuul-jobs/+/863781 | 23:41 |
| opendevreview | Merged zuul/zuul-jobs master: ensure-kubernetes: move testing into common path https://review.opendev.org/c/zuul/zuul-jobs/+/863940 | 23:42 |
| opendevreview | Michael Kelly proposed zuul/zuul-jobs master: helm: Add job for linting helm charts https://review.opendev.org/c/zuul/zuul-jobs/+/861799 | 23:47 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!