| opendevreview | Merged opendev/system-config master: mirror: Add centos-stream link https://review.opendev.org/c/opendev/system-config/+/818026 | 00:27 |
|---|---|---|
| opendevreview | Ian Wienand proposed openstack/project-config master: grafana: add centos-stream mirror volume https://review.opendev.org/c/openstack/project-config/+/818031 | 00:42 |
| opendevreview | Ian Wienand proposed openstack/project-config master: grafana: update nodepool/dib images https://review.opendev.org/c/openstack/project-config/+/818035 | 01:48 |
| opendevreview | Merged openstack/project-config master: grafana: add centos-stream mirror volume https://review.opendev.org/c/openstack/project-config/+/818031 | 01:52 |
| ianw | ianw: test | 06:18 |
| ianwtest | ianw: testing | 06:20 |
| ianwtest | ianw: testing again | 06:21 |
| ianwtest | ianw: testing again again | 06:23 |
| ianwtest | ianw: testing again | 06:26 |
| ianwtest | ianw: a message | 06:28 |
| ianwtest | ianw: foo | 06:28 |
| *** ysandeep|out is now known as ysandeep | 06:30 | |
| ianw | what a yak shaving adventure. i finally tracked down my f35 crashes to the x11 qxl driver in my vm. so i decided to switch to virtio. i don't know why but latency was terrible with xfce. so now i've switched back to gnome | 06:33 |
| ianw | 3d/opengl passthrough seems to actually work now | 06:34 |
| *** gibi_ is now known as gibi | 08:03 | |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: source-repositories: don't use --git-dir https://review.opendev.org/c/openstack/diskimage-builder/+/818053 | 08:25 |
| ianw | fungi/clarkb: ^ after instrumenting git to narrow this down, ultimately I think it's our error | 08:27 |
| *** ysandeep is now known as ysandeep|lunch | 08:29 | |
| *** ykarel is now known as ykarel|lunch | 09:10 | |
| *** ysandeep|lunch is now known as ysandeep | 09:44 | |
| *** ykarel|lunch is now known as ykarel | 10:26 | |
| opendevreview | Jakub Darmach proposed openstack/diskimage-builder master: Added missing grubby arg DIB_BOOTLOADER_DEFAULT_CMDLINE https://review.opendev.org/c/openstack/diskimage-builder/+/818063 | 10:43 |
| *** ysandeep is now known as ysandeep|brb | 11:07 | |
| *** ysandeep|brb is now known as ysandeep | 11:13 | |
| opendevreview | Andreas Jaeger proposed openstack/project-config master: Iotronic-pythonclient and Iotronic-UI update https://review.opendev.org/c/openstack/project-config/+/817719 | 11:26 |
| opendevreview | Merged opendev/base-jobs master: Remove growroot logs dumping in base-test https://review.opendev.org/c/opendev/base-jobs/+/817289 | 11:31 |
| opendevreview | Merged openstack/project-config master: Create repo for ProxySQL Ansible role https://review.opendev.org/c/openstack/project-config/+/817271 | 11:38 |
| opendevreview | Merged openstack/project-config master: Iotronic-pythonclient and Iotronic-UI update https://review.opendev.org/c/openstack/project-config/+/817719 | 13:41 |
| zigo | Any idea why heat's doc job fails, when it fully builds the docs? https://9348cb89109060032864-cd07d3426af01024dbeaac2f8639d2e0.ssl.cf1.rackcdn.com/817890/2/check/openstack-tox-docs/cbe0bf9/job-output.txt | 13:56 |
| *** pojadhav is now known as pojadhav|brb | 13:59 | |
| *** pojadhav|brb is now known as pojadhav | 14:08 | |
| *** ysandeep is now known as ysandeep|out | 14:34 | |
| fungi | build finished with problems, 4 warnings. | 14:39 |
| fungi | ERROR: InvocationError for command /home/zuul/src/opendev.org/openstack/heat/.tox/docs/bin/sphinx-build -W --keep-going -b html doc/source doc/build/html (exited with code 1) | 14:39 |
| fungi | zigo: i believe -W tells sphinx-build t treat warnings as errors, so exits nonzero | 14:40 |
| zigo | Oh, ok, thanks. | 14:40 |
| fungi | most openstack projects strive for warning-free doc builds, i think | 14:40 |
| zigo | Well, in this case, it looks like it broke the CI ! :) | 14:41 |
| zigo | You are right with the -W ... | 14:43 |
| fungi | zigo: looks like it's the warnigns like "docstring of heat.db.sqlalchemy.types.Json.process_bind_param:19: WARNING: undefined label: types_typedecorator" it's concerned about | 14:50 |
| zigo | Yeah, saw it. | 14:50 |
| zigo | I'm looking for how to fix ... | 14:50 |
| zigo | I'm not sure what this is, but looks like comming from sqlalchemy. | 14:51 |
| fungi | heat/db/sqlalchemy/types.py i think | 14:51 |
| zigo | Yep. | 14:51 |
| zigo | Though types.TypeDecorator isn't on line 19, but 23, so it's kind of werid. | 14:51 |
| zigo | weird. | 14:51 |
| fungi | it might be on line 23 after merging to the current state of the master branch | 14:52 |
| fungi | https://review.opendev.org/c/openstack/heat/+/816592 merged a few days ago and added some blank lines early in that file | 14:53 |
| zigo | Right ! :) | 14:53 |
| fungi | so that would explain the line numbers being skewed from the ones in the change, zuul tests the proposed commit as it would be if it merged to the current state of the repository | 14:53 |
| fungi | not as-written | 14:54 |
| zigo | Maybe this is due to now using a higher SQLAlchemy version... | 14:56 |
| fungi | and yeah, i have a feeling something used to define the types_typedecorator label | 14:57 |
| fungi | use of types.TypeDecorator was added to that script in 2015, so it's not like it's a recent addition | 14:58 |
| fungi | sqlalchemy 1.4.27 released last week, and was updated in upper-constraints.txt yesterday when https://review.opendev.org/c/openstack/requirements/+/817667 merged | 15:00 |
| fungi | so if the problem started showing up shortly after 08:00 utc yesterday, that might be related | 15:00 |
| fungi | nothing in https://docs.sqlalchemy.org/en/14/changelog/changelog_14.html#change-1.4.27 jumps out at me, but since it's documentation-related it could have been from something below the threshold for a changelog entry | 15:03 |
| zigo | Let's ask zeeek? :) | 15:09 |
| fungi | zzzeek, but yeah i'm digging through the commit history first | 15:10 |
| fungi | https://github.com/sqlalchemy/sqlalchemy/commit/605d0d9 might be related | 15:12 |
| fungi | it added :ref:`types_typedecorator` to some docstrings, new in the rel_1_4_27 tag | 15:22 |
| fungi | and that seems to be getting inherited via subclassing in heat, which i think causes sphinx to want to include it, but the types_typedecorator label is defined in the doc/build/core/custom_types.rst file in the sqlalchemy repo | 15:23 |
| fungi | it might be possible for heat to add a types_typedecorator label in its own docs | 15:24 |
| zigo | What exactly is a label ? | 15:24 |
| zigo | Is this a sphinx thing? | 15:25 |
| fungi | its a restructuredtext thing | 15:25 |
| zigo | Oh, it's a .. thingylikethis: right? | 15:25 |
| fungi | yeah | 15:25 |
| zigo | With underscore in front, event. | 15:26 |
| zigo | Let's try, I'd say? :) | 15:26 |
| fungi | you can see the _types_typedecorator label for sqlalchemy's docs in https://raw.githubusercontent.com/sqlalchemy/sqlalchemy/main/doc/build/core/custom_types.rst | 15:28 |
| zigo | Maybe something naive like this will repair? https://review.opendev.org/c/openstack/heat/+/818104 | 15:30 |
| fungi | it's worth testing, as long as that document is included in their docs build. a slightly less hacky solution might be to find a relevant place in the heat docs to add a short section which links to https://docs.sqlalchemy.org/en/14/core/custom_types.html#augmenting-existing-types | 15:33 |
| fungi | since that's what the docstring is trying to reference anyway | 15:33 |
| *** artom__ is now known as artom | 15:34 | |
| *** dviroel is now known as dviroel_ | 15:37 | |
| opendevreview | Artem Goncharov proposed zuul/zuul-jobs master: Add no_log to the log uploading module parameters https://review.opendev.org/c/zuul/zuul-jobs/+/818109 | 15:39 |
| clarkb | I'm testing the growroot log removal in base-test with https://review.opendev.org/c/zuul/zuul-jobs/+/680178 I'll approve the base update if that looks happy | 15:46 |
| clarkb | fungi: did you want to review the dib git repo caching fix before I approve it? | 15:48 |
| fungi | oh, right thanks, i had pulled that up before i got distracted with a meeting | 15:50 |
| clarkb | Once I've caught up on a few things like ^ I'm going to work on an email to openstack-discuss asking openstack to address their zuul config errors | 15:50 |
| clarkb | fungi: the change seems straightforward I'll let you +A if you are happy with it | 15:50 |
| fungi | yeah, for some reason i never noticed git's -C option and always ended up verbosely overriding both --git-dir and --work-tree | 15:53 |
| fungi | i agree the manpage indicates this is far simpler too | 15:53 |
| clarkb | I suspect that it is newer than the ~ubuntu precice centos 5/6 time period this stuff was written | 15:54 |
| fungi | yeah | 15:55 |
| opendevreview | Artem Goncharov proposed zuul/zuul-jobs master: Fix zuul-jobs-test-upload-logs-s3 job https://review.opendev.org/c/zuul/zuul-jobs/+/807132 | 16:20 |
| *** ykarel is now known as ykarel|away | 16:27 | |
| opendevreview | Artem Goncharov proposed zuul/zuul-jobs master: Add no_log to the log uploading module parameters https://review.opendev.org/c/zuul/zuul-jobs/+/818109 | 16:29 |
| opendevreview | Merged zuul/zuul-jobs master: Fix zuul-jobs-test-upload-logs-s3 job https://review.opendev.org/c/zuul/zuul-jobs/+/807132 | 16:34 |
| opendevreview | Merged zuul/zuul-jobs master: Add no_log to the log uploading module parameters https://review.opendev.org/c/zuul/zuul-jobs/+/818109 | 16:45 |
| clarkb | I've approved https://review.opendev.org/c/opendev/base-jobs/+/817290 which updates the base job to remove growroot log checking | 17:04 |
| clarkb | base-test testing seemed happy | 17:04 |
| opendevreview | Merged opendev/base-jobs master: Remove growroot log dumping from the base job https://review.opendev.org/c/opendev/base-jobs/+/817290 | 17:11 |
| fungi | thanks! | 17:24 |
| opendevreview | Merged openstack/diskimage-builder master: source-repositories: don't use --git-dir https://review.opendev.org/c/openstack/diskimage-builder/+/818053 | 17:28 |
| *** marios is now known as marios|out | 17:29 | |
| *** sshnaidm is now known as sshnaidm|afk | 17:33 | |
| clarkb | fungi: when is a good time to talk lists.o.o kernel stuff? | 17:50 |
| clarkb | I need ot eat something and prep for the infra meeting but can probably sneak ^ in now if that works | 17:50 |
| *** tosky is now known as Guest6054 | 18:05 | |
| *** tosky_ is now known as tosky | 18:05 | |
| fungi | ahh, sorry, stepped away for a snack, but maybe after the meeting? | 18:20 |
| clarkb | fungi: now is still good if it works, but ya can do after the meeting too (though going to try a bike ride today if I can since the rain stopped) | 18:20 |
| fungi | oh, sure, i can do it now | 18:21 |
| fungi | i've started a root screen session on lists.o.o | 18:22 |
| clarkb | ok I've joined (had to load my keys) | 18:24 |
| clarkb | fungi: in my homedir there is a dir called kernel-stuff or similar | 18:24 |
| clarkb | amybe we should create one in roots homedir and work from there goign forward | 18:24 |
| fungi | i'll copy it over | 18:28 |
| fungi | and done | 18:29 |
| clarkb | fungi: extract-vmlinux is the tool from the kernel git repo | 18:29 |
| clarkb | you basically give it a vmlinuz file and it writes to stdout the extracted file. So what I did previously was copied the compressed file to this kernel-stuff dir, extract it into a new file in that dir than copied the extracted file over the original in /boot | 18:29 |
| clarkb | the menu.lst stuff was from figuring out the chainload situation | 18:30 |
| fungi | okay so there's a vmlinuz-5.4.0-90-generic.extracted in ~root/kernel-stuff now | 18:33 |
| fungi | as well as a backup of the compressed vmlinuz-5.4.0-90-generic from /boot | 18:33 |
| fungi | i ran it like `bash extract-vmlinux vmlinuz-5.4.0-90-generic > vmlinuz-5.4.0-90-generic.extracted` | 18:33 |
| clarkb | ya I think that is correct. The size looks about right too | 18:33 |
| clarkb | I don't think we were able to come up with any better verification other than size is bigger | 18:34 |
| clarkb | but nwo you can copy that file back over the /boot path that the original was at | 18:34 |
| fungi | file says it's an ELF 64-bit LSB executable rather than a Linux kernel x86 boot executable bzImage | 18:34 |
| clarkb | fungi: amybe double check permissions for the result too | 18:35 |
| clarkb | after it is copied I mean | 18:35 |
| clarkb | that looks good | 18:35 |
| fungi | permissions and ownership remained consistent with the others there, yeah | 18:35 |
| fungi | and the size is in the same range as the earlier extracted one | 18:36 |
| fungi | should we reboot now or test that at a different time? | 18:36 |
| clarkb | up to you I guess. I'd like to get out on a bike ride in an hour and a half or so and that might impact that :) but if you want to debug potentail failed boots I'm fine with that | 18:37 |
| fungi | i'm happy to reboot now | 18:37 |
| clarkb | I won't stop you :) | 18:37 |
| fungi | going ahead then | 18:38 |
| clarkb | it pings but is network unreachable currently. I don't recall if that was a symptom of the failed boot or not | 18:39 |
| *** sshnaidm|afk is now known as sshnaidm | 18:39 | |
| clarkb | oh and now it doesn't pign so maybe it was on its way down and now we wait for it to come back | 18:40 |
| fungi | it was refusing port 22 connections up to a moment ago, yeah | 18:40 |
| fungi | i think so | 18:40 |
| fungi | pinging again now | 18:41 |
| fungi | Linux lists 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | 18:41 |
| fungi | #status log Rebooted main lists.o.o server onto new extracted vmlinuz-5.4.0-90-generic kernel | 18:41 |
| clarkb | its up and I confirm uname reports the new kernel | 18:41 |
| opendevstatus | fungi: finished logging | 18:41 |
| fungi | thanks! that was fairly straightforward | 18:41 |
| clarkb | ya once we sorted out how to make it work it isn't terrible (other than it isn't currently automated) | 18:42 |
| clarkb | it was the figuring it out step that was painful | 18:42 |
| fungi | also 45 processes owned by the list user, according to ps, so everything's running that should be | 18:42 |
| clarkb | I can get list archives too (I needed that for writing down my meeting notes) | 18:43 |
| *** tosky_ is now known as tosky | 18:48 | |
| clarkb | is https://zuul.opendev.org/t/openstack/build/bc0075f04a8941cf82dc536f76d6ecab having a sad for anyone else too? | 18:50 |
| clarkb | that caused the openstack gate to reset and I was going to take a quick look but I don't seem to get logs | 18:50 |
| clarkb | that has me concerned that some of the job updtes that landed recently could affect that but spot checking other failures I get logs for them | 18:50 |
| frickler | clarkb: "malformed URL" | 18:51 |
| frickler | https://storage.bhs.cloud.********.net does it really mean it or try to hide something? | 18:52 |
| clarkb | ya it seems there are **** in the url? | 18:52 |
| clarkb | https://zuul.opendev.org/t/openstack/build/f9f275a756534946850b2ec51541fe2a ok found another one | 18:53 |
| clarkb | corvus: ^ could this be related to the recent zuul-jobs update? seems unlikely given that rax uploads seem fine? | 18:53 |
| fungi | replacing the ******** with ovh fixes it | 18:53 |
| frickler | https://review.opendev.org/c/zuul/zuul-jobs/+/818109/2/roles/upload-logs-base/library/zuul_swift_upload.py | 18:54 |
| frickler | that looks like it could try to obscure the result | 18:54 |
| clarkb | frickler: ya that is the change, but it isn't doing it for rax. I'm really confused as to how that would happen here | 18:54 |
| clarkb | the first build I linked to ran on ze02 | 18:57 |
| clarkb | Return log URL to Zuul is the task that would set the value, maybe it is affected | 18:58 |
| clarkb | log_url: "{{ upload_results.url }}/" is what that task does | 18:58 |
| fungi | strange that only the "ovh" in the url was replaced | 18:59 |
| fungi | i wonder if it mistakes that for a sensitive pattern | 18:59 |
| clarkb | ok rax has a different code path due to the cdn in the library file | 19:00 |
| clarkb | I need to context switch to the meeting, but we should keep looking at this. it is possible it is just coincidence and something change elsewhere but also possibly related | 19:00 |
| clarkb | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/upload-logs-base/library/zuul_swift_upload.py#L92-L101 is the special casing | 19:01 |
| fungi | that change was the first one to touch zuul_swift_upload.py since february | 19:03 |
| opendevreview | Artem Goncharov proposed zuul/zuul-jobs master: Revert "Add no_log to the log uploading module parameters" https://review.opendev.org/c/zuul/zuul-jobs/+/817988 | 19:06 |
| opendevreview | Merged zuul/zuul-jobs master: Revert "Add no_log to the log uploading module parameters" https://review.opendev.org/c/zuul/zuul-jobs/+/817988 | 19:18 |
| clarkb | to followup on the no_log thing ansible has a "feature' where any values in a no_log dict are removed from logs and return values | 20:17 |
| clarkb | in this case ovh is the name of the cloud in the cloud dict so it gets removed from the url that is returned | 20:18 |
| clarkb | rax doesn't suffer this as it uses different names for the cdn | 20:18 |
| ianw | https://landscape.cncf.io/?selected=open-stack is the thing scraping the repo i guess | 20:18 |
| ianw | i don't know how i feel about ranking projects by "latest tweet" | 20:19 |
| clarkb | ianw: its the future! | 20:19 |
| clarkb | :P | 20:19 |
| ianw | also seems zuul could have an icon in just about every box of https://landscape.cd.foundation/ | 20:20 |
| fungi | also "github stars" are a good indication of project health, in case you didn't knopw | 20:21 |
| clarkb | but no github repo so they don't let zuul in | 20:22 |
| fungi | which is amusing, because they let proprietary projects into the landscape with no public source control reference at all | 20:22 |
| fungi | but if it's open source, it has to be in github (isn't that the definition of "open source" now?) | 20:23 |
| opendevreview | Artem Goncharov proposed zuul/zuul-jobs master: Apply no_log on the upload-logs-swift role https://review.opendev.org/c/zuul/zuul-jobs/+/818137 | 20:24 |
| clarkb | things seem happy enough now. Going on that bike ride | 20:31 |
| clarkb | back in a bit | 20:31 |
| fungi | have fun! | 20:40 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Set noop jobs on openstackid repos https://review.opendev.org/c/openstack/project-config/+/818170 | 22:29 |
| opendevreview | Clark Boylan proposed opendev/puppet-openstackid master: Retire this project https://review.opendev.org/c/opendev/puppet-openstackid/+/818172 | 22:41 |
| *** tosky is now known as Guest6070 | 22:42 | |
| *** tosky_ is now known as tosky | 22:42 | |
| opendevreview | Clark Boylan proposed openstack/project-config master: Complete openstackid repo retirements https://review.opendev.org/c/openstack/project-config/+/818176 | 22:52 |
| clarkb | openstackid retirement shoudl clean up the errors with its zuul config ^ :) | 22:53 |
| *** tosky is now known as Guest6073 | 23:07 | |
| *** tosky_ is now known as tosky | 23:07 | |
| ianw | small patch for the caching issue @ https://public-inbox.org/git/YZQ5Zk0ItWvfr8sF@fedora19.localdomain/T/#u | 23:08 |
| ianw | of course already seen a spelling error. i hate emailing patches | 23:09 |
| clarkb | here here | 23:09 |
| clarkb | or is it hear hear? | 23:09 |
| clarkb | english | 23:09 |
| clarkb | ianw: I guess is_empty() does more than just check if it is a directory and is empty? similar with is_directory? | 23:12 |
| clarkb | because --git-dir should be a directory path right? | 23:12 |
| clarkb | oh wait is it that empty_submodule_path.buf is set to some weird value? I bet that is it and it isn't actually a directory and probably doesn't exist at all | 23:13 |
| clarkb | ianw: the typo appears to be in the comment? the C seems valid at least. I wonder if they will nitpick that | 23:15 |
| clarkb | ianw: re the debugging were you recompiling git to add extra debugging or was gdb sufficient? (I'm mostly curious as its been a long time since I had to do debugging like that myself) | 23:16 |
| ianw | clarkb: i recompiled git; i found you can mostly fprintf(stderr, "...") at random points safely | 23:28 |
| ianw | --git-dir should be relative to the top of the repo. i can't image why it's really even an option, but ... git | 23:28 |
| ianw | in this case, the repo was in /home/iwienand/tmp/os/openstack/ ; we work in /home/iwienand/tmp/os/ and then set --gitdir openstack/.git | 23:30 |
| ianw | the error would look like | 23:30 |
| ianw | From https://opendev.org/openstack/openstack | 23:30 |
| ianw | + 7af3bee3a1...c557eac228 master -> master (forced update) | 23:30 |
| ianw | * [new branch] milestone-proposed -> milestone-proposed | 23:30 |
| ianw | Submodule directory '/home/iwienand/tmp/os/manila/' not found | 23:30 |
| ianw | Submodule directory '/home/iwienand/tmp/os/openstack-helm-images/' not found | 23:30 |
| ianw | Submodule directory '/home/iwienand/tmp/os/tripleo-ansible/' not found | 23:30 |
| ianw | with the patch applied | 23:30 |
| ianw | (note the missing "/openstack/") | 23:31 |
| clarkb | aha | 23:31 |
| clarkb | so ya the dir didn't exist at all not that it wasn't empty | 23:31 |
| clarkb | that makes sense given your patch | 23:31 |
| ianw | yeah, that is_empty_dir function ends up conflating "is empty" and "missing" | 23:35 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!