| fungi | good luck on your talk! | 00:00 |
|---|---|---|
| clarkb | oh yes good luck. I thought about trying to participate there, but the timing just wasn't good. The inperson but not inperson wouldn't work for me :) | 00:00 |
| *** dviroel|afk is now known as dviroel | 00:00 | |
| ianw | hrm, i'm guessing the ppa didn't like me adding ~ppa0 as it probably doesn't sort later than the version already there | 00:03 |
| ianw | for vhd-util | 00:04 |
| ianw | this is my fault for not uploading it with this tag in the first place | 00:04 |
| johnsom | Ok, so I know the issue with the unbound logs being empty on centos/rhel. Maybe you folks might have an idea of how to solve it. | 00:13 |
| johnsom | So, this https://opendev.org/openstack/project-config/src/branch/master/nodepool/elements/nodepool-base/finalise.d/89-boot-settings#L71 | 00:13 |
| johnsom | Puts the unbound.log in two different places depending on the OS. | 00:13 |
| johnsom | The filename is the same however. So, when the devstack job comes around, I can't just add the other path to zuul_copy_output because the empty one will copy over the top of the correct one: https://zuul.opendev.org/t/openstack/build/1ebaa9bab9a3424ebe1b1e84372922d6/log/zuul-info/inventory.yaml#131 | 00:15 |
| johnsom | It doesn't look like zuul_copy_output has any merge or rename options. | 00:16 |
| johnsom | Maybe this "null" option could help? I just don't fully understand how it works: https://zuul-ci.org/docs/zuul-jobs/general-roles.html#rolevar-stage-output.zuul_copy_output | 00:18 |
| opendevreview | Ian Wienand proposed opendev/infra-vhd-util-deb focal: Retrigger upload https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/841511 | 00:20 |
| opendevreview | Merged zuul/zuul-jobs master: add-gpgkey: trust incoming key https://review.opendev.org/c/zuul/zuul-jobs/+/840566 | 00:22 |
| *** dviroel is now known as dviroel|out | 00:24 | |
| opendevreview | Merged opendev/infra-vhd-util-deb focal: Retrigger upload https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/841511 | 00:28 |
| johnsom | Maybe the best answer is to stop that nodepool element from creating the /var/log/unbound.log empty file. | 00:29 |
| johnsom | Or just set them both to store it under /var/lib/unbound/unbound.log | 00:37 |
| *** dviroel|out is now known as dviroel | 00:59 | |
| ianw | johnsom: yeah, i think that is a limitation of zuul_copy_output :/ | 01:05 |
| ianw | did you say it was something about selinux and centos not storing it under /var/log/unbound.log? that also might be the sort of thing we merged in centos6 era and might be different now | 01:06 |
| johnsom | ianw Yeah, the patch that added the path change mentions selinux: https://review.opendev.org/c/openstack/project-config/+/506332 | 01:08 |
| johnsom | It is old.... | 01:08 |
| *** rlandy|bbl is now known as rlandy|out | 01:09 | |
| ianw | yeah, that feels like the type of thing that might have changed. not 100% sure how to gate test that though | 01:09 |
| johnsom | Yeah, same here. No idea how to get all the parts together when it's a nodepool element. | 01:09 |
| ianw | yeah it's a bit of a manual process because once we merge it goes into the next build | 01:10 |
| ianw | we could probably just put a node on hold and fiddle manually to see what works | 01:10 |
| johnsom | I need to go make dinner now, so limited availability. I will check the channel log tomorrow though if folks have thoughts on the best approach. | 01:11 |
| johnsom | Yeah, holding a node to try moving the log path would probably be the best approach. If it can open the file in /var/log/unbound.log we know whatever issue existed doesn't anymore. | 01:12 |
| ianw | ok, i need to eat lunch :) i'll take a look | 01:12 |
| johnsom | If we don't force a path it goes in journald, not sure if there is good tooling to pull those logs out that we should just switch to using journald | 01:12 |
| johnsom | ianw Thank you! Drop me a note if there is something I can pick up and continue to work on tomorrow. | 01:13 |
| ianw | yeah it might be better to just dump a "journalctl -u unbound" type thing | 01:13 |
| opendevreview | Merged opendev/infra-openafs-deb jammy: Add build stamp to push to production PPA https://review.opendev.org/c/opendev/infra-openafs-deb/+/841509 | 01:16 |
| opendevreview | Merged opendev/infra-openafs-deb bionic: Add build stamp to push to production PPA https://review.opendev.org/c/opendev/infra-openafs-deb/+/841507 | 01:16 |
| opendevreview | Merged opendev/infra-openafs-deb focal: Add build stamp to push to production PPA https://review.opendev.org/c/opendev/infra-openafs-deb/+/841508 | 01:16 |
| *** ysandeep|rover|out is now known as ysandeep|rover | 01:19 | |
| opendevreview | Merged opendev/infra-openafs-deb xenial: Add build stamp to push to production PPA https://review.opendev.org/c/opendev/infra-openafs-deb/+/841506 | 01:20 |
| *** dviroel is now known as dviroel|out | 01:24 | |
| *** ysandeep|rover is now known as ysandeep|afk | 02:26 | |
| *** diablo_rojo_phone is now known as Guest458 | 02:44 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: Add testing for jammy openafs https://review.opendev.org/c/opendev/system-config/+/841525 | 02:54 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [dnm] holding some centos nodes https://review.opendev.org/c/opendev/system-config/+/841526 | 03:01 |
| *** ysandeep|afk is now known as ysandeep|rover | 04:44 | |
| *** soniya is now known as soniya|ruck | 05:04 | |
| opendevreview | Ian Wienand proposed openstack/project-config master: Set context for unbound.log on selinux systems https://review.opendev.org/c/openstack/project-config/+/841546 | 05:17 |
| *** ysandeep|rover is now known as ysandeep|rover|brb | 05:57 | |
| ianw | johnsom: 173.231.255.77 is a 8-stream, 173.231.255.252 is a 9-stream with ^ manually setup. i think it's easier to keep it in the same location and should fix the collection issue? | 06:12 |
| *** ysandeep|rover|brb is now known as ysandeep|rover | 06:15 | |
| *** ysandeep|rover is now known as ysandeep|rover|brb | 07:50 | |
| *** ysandeep|rover|brb is now known as ysandeep|rover | 08:03 | |
| *** Guest458 is now known as diablo_rojo_phone | 08:03 | |
| *** tweining|off is now known as tweining | 08:12 | |
| opendevreview | Dmitry Tantsur proposed openstack/diskimage-builder master: Switch to the CentOS 9 IPA job https://review.opendev.org/c/openstack/diskimage-builder/+/841558 | 09:43 |
| *** jpena|off is now known as jpena | 09:44 | |
| *** ysandeep|rover is now known as ysandeep|rover|lunch | 10:03 | |
| *** rlandy|out is now known as rlandy | 10:20 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/diskimage-builder master: Adopted dkms element to work on Ubuntu Jammy and nvidia drivers https://review.opendev.org/c/openstack/diskimage-builder/+/841465 | 10:20 |
| *** ysandeep|rover|lunch is now known as ysandeep|rover | 10:34 | |
| *** soniya29 is now known as soniya29|ruck | 11:08 | |
| *** soniya29|ruck is now known as soniya29|ruck|brb | 11:10 | |
| *** dviroel_ is now known as dviroel | 11:33 | |
| *** ysandeep|rover is now known as ysandeep|rover|brb | 12:01 | |
| *** soniya is now known as soniya|ruck | 12:21 | |
| *** ysandeep|rover|brb is now known as ysandeep|rover | 12:28 | |
| opendevreview | Merged openstack/diskimage-builder master: Switch to the CentOS 9 IPA job https://review.opendev.org/c/openstack/diskimage-builder/+/841558 | 13:05 |
| *** marios_ is now known as marios | 13:17 | |
| slittle | Can we get eyes on https://review.opendev.org/c/openstack/project-config/+/840263 | 13:30 |
| slittle | ah... perhaps that should be opendev/project-config | 13:32 |
| fungi | slittle: no, it's correct | 13:33 |
| fungi | we're still in the (lengthy) process of moving all that stuff to the newer opendev project namespace | 13:33 |
| fungi | but untangling it from other things we need to wind down in the openstack namespace is taking a while | 13:34 |
| fungi | in part because it's also a move to a separate zuul tenant, so some things can't go until related repositories switch to the other tenant | 13:35 |
| slittle | List of cores for starlingx-app-sriov-fec-operator-core: Teresa Ho, Greg Waines, Steve Webster, Cole Walker | 13:39 |
| opendevreview | Merged openstack/project-config master: Add SRIOV FEC Operator app to StarlingX https://review.opendev.org/c/openstack/project-config/+/840263 | 13:43 |
| *** ysandeep|rover is now known as ysandeep|rover|mtg | 14:00 | |
| fungi | slittle: i've found and added all four of them. in the future, it's easier for us to just add one person and then let them add the others (since these groups are self-managed by default) | 14:31 |
| clarkb | infra-root I'm lurking the TC meeting then will reboot for some updates and then my plan is to restart zuul-web instances | 15:41 |
| clarkb | this should hopefully fix the scroll to log line issue with current zuul web | 15:41 |
| fungi | i'm around to help | 15:41 |
| clarkb | its a relatively minor thing but something that I hit constantly as I look at job logs so I'm selfishly looking forward to having my browser scroll for me :) | 15:42 |
| fungi | yeah, it comes up for me a lot too. i end up scrolling thousands of lines of log to spot the one that's a slightly different color | 15:43 |
| clarkb | also I'll be listening in on the gerrit hackathon which is running over PDT more than I expected it to | 15:44 |
| opendevreview | Merged openstack/project-config master: Set context for unbound.log on selinux systems https://review.opendev.org/c/openstack/project-config/+/841546 | 15:51 |
| clarkb | johnsom: ianw ^ that will need iamge rebuilds though | 15:52 |
| *** ysandeep|rover|mtg is now known as ysandeep|rover | 15:56 | |
| johnsom | clarkb Right, thanks! Fingers crossed for tomorrow | 16:00 |
| clarkb | johnsom: is thee a particular centos flavor that would be more helpful to get signal on quicker? I can manually trigger a build for that one now | 16:01 |
| clarkb | *is there | 16:01 |
| johnsom | clarkb C9s, but I am fine waiting until tomorrow too. I have C9s test jobs I was using yesterday | 16:01 |
| clarkb | johnsom: I'm impatient and want signal more quickly :) | 16:02 |
| johnsom | Fair enough, ping me and I can run the job for confirmation | 16:02 |
| *** marios is now known as marios|out | 16:05 | |
| clarkb | its queued behind a couple of other builds now but should happen soon | 16:05 |
| *** soniya|ruck is now known as soniya|out | 16:11 | |
| *** ysandeep|rover is now known as ysandeep|rover|out | 16:28 | |
| clarkb | I'm going to restart zuul-web on zuul01 now | 17:11 |
| clarkb | then when it is up again do zuul-web on zuul02 | 17:12 |
| opendevreview | Merged zuul/zuul-jobs master: Switch enable_src_repos to False in configure-mirrors https://review.opendev.org/c/zuul/zuul-jobs/+/839593 | 17:12 |
| fungi | gotta say, from an operations standpoint i *love* the https://zuul.opendev.org/components page | 17:17 |
| clarkb | fungi: we should be able to land https://review.opendev.org/c/opendev/system-config/+/839623 whenever we are ready to do the cleanups in the repo by hand | 17:17 |
| fungi | clarkb: did you restart the fingergw on zuul01 as well? it's reporting the new version | 17:18 |
| clarkb | fungi: yes they share a docker-compose file so both got down and up'd | 17:18 |
| clarkb | I didn't realize ti when I did 01 but I'll be sure to treat 02 the same way | 17:19 |
| fungi | aha, perfect. just making sure the component api wires weren't crossed | 17:19 |
| clarkb | ya I don't think it is a problem but it surprised me too. Not a big deal | 17:19 |
| clarkb | zuul-web debug log doesn't give much indication of its progress in updating system config, but I'm guessing that is normal? cc corvus | 17:20 |
| *** jpena is now known as jpena|off | 17:21 | |
| clarkb | zuul-web does seem to be using a number of cpu cycles so I should be pateitn I guess | 17:26 |
| clarkb | it is logging about all the config files it is loading now | 17:33 |
| clarkb | I expect it will be done very soon | 17:33 |
| clarkb | johnsom: ianw: https://nb01.opendev.org/centos-9-stream-0000005213.log dib doens't like the new selinux context setting | 17:34 |
| clarkb | fungi: zuul01 is up now | 17:34 |
| clarkb | I'm talking to it. Going to test the log viewing | 17:34 |
| clarkb | it works! I'm so happy I managed to track that down | 17:35 |
| clarkb | I'll give it a couple minutes for any problems to show up then restart zuul02's web and fingergew | 17:35 |
| johnsom | clarkb ack, interesting... "chcon: can't apply partial context to unlabeled file '/var/log/unbound.log'" I will pull down the patch and see if I can figure out what chcon wants. | 17:36 |
| clarkb | johnsom: thanks | 17:36 |
| johnsom | That file should probably be labeled | 17:36 |
| clarkb | as far as I can tell zuul-web is happy. Proceeding with 02 now | 17:40 |
| fungi | yep, seems to be working for me | 17:49 |
| clarkb | #status log Updated zuul-web and zuul-fingergw to 6.0.1.dev14 60e59ba67. This fixes scrolling to specific line numbers on log files. | 18:05 |
| opendevstatus | clarkb: finished logging | 18:05 |
| fungi | thanks! so much more convenient that it will scroll to the line number anchors now | 18:06 |
| clarkb | I tested it on https://zuul.opendev.org/t/openstack/build/4293bd9af704494388894f1069c48579/log/job-output.txt#2131 and it handled that relatively large log and the deeper link than some of the more trivial stuff that it was testing on pre merge | 18:07 |
| clarkb | but ya seems to work and I'm happy | 18:07 |
| clarkb | fungi: I rechecked https://review.opendev.org/c/opendev/system-config/+/839623 so that it has a +1 when we are ready to prune the mirrors | 18:19 |
| clarkb | it is not longer approved so it won't land but I think we can approve that whenever someone is ready to do the reprepro cleanups | 18:19 |
| fungi | thanks! | 18:19 |
| clarkb | specifically someone has grabbed and held the lock when we approve that I mean | 18:20 |
| fungi | not that it would hurt if we grabbed the lock afterward | 18:20 |
| clarkb | ya I guess reprepro will just fail and we won't publish anything as a result | 18:21 |
| fungi | will it fail? we're just removing entries right? i need to re-review the change obviously to double-check | 18:21 |
| clarkb | iirc the first repo we tested with failed | 18:22 |
| clarkb | but maybe not | 18:22 |
| clarkb | there were a lot of cleanups and it could've been another thing that caused it to fail | 18:22 |
| opendevreview | Michael Johnson proposed openstack/project-config master: Fix selinux context for unbound.log https://review.opendev.org/c/openstack/project-config/+/841629 | 19:25 |
| johnsom | clarkb ^^^ I think that will fix the error. Though I couldn't reproduce the "partial context" issue locally for some reason. | 19:25 |
| opendevreview | Merged openstack/project-config master: Retire openstack-helm-docs repo, step 3.3 https://review.opendev.org/c/openstack/project-config/+/839427 | 20:05 |
| corvus | clarkb: yeah re zuul-web. i generally check the components page every 10m or so | 20:39 |
| corvus | clarkb: it'll switch from 'initializing' to 'running' | 20:39 |
| clarkb | right, I was mostly just surprised at how quiet it is since the schedulers are pretty verbose. But it all worked out | 20:39 |
| rosmaita | hello ... i'm suddenly seeing a bunch of post failures, can't find pbr 5.9.0 ... are you already aware? https://zuul.opendev.org/t/openstack/build/f8a5a2afa7ea45c0b305729c21e94641 | 20:45 |
| clarkb | news to me | 20:45 |
| rosmaita | sorry to be the one to report it | 20:46 |
| clarkb | pbr probably should be in constraints but that is a separate problem | 20:46 |
| rosmaita | does it look like an outdated mirror situation? | 20:46 |
| clarkb | sort of. Pypi is merely proxy cached these days. It asks pypi for that data if the cache ttl is expired or it doesn't haev it already. If the pypi CDN has an error talking to the pypi backend it will talk to a backup backend which will serve often stale data. I looks like that situation | 20:47 |
| clarkb | you've asked for pbr 5.9.0 which is the lastest version from may 5th. But the index only provided up to version 5.8.0 | 20:47 |
| clarkb | implying you got a stale index which was likely caused by a pypi failover to their backup backend | 20:48 |
| rosmaita | gotcha, thanks for explaining that | 20:48 |
| clarkb | looks like 5.8.1 was also missing from the stale index | 20:48 |
| clarkb | which means what we got was older than february 6, 2022 :/ | 20:48 |
| rosmaita | so i guess the thing to do is wait a bit and then recheck? | 20:49 |
| clarkb | yes, you can also check https://status.python.org/ as large ongoing issues with pypi tend to make it there (it appears clean right now though) | 20:49 |
| rosmaita | ok, i'll make a note of that so i can troubleshoot better next time | 20:49 |
| clarkb | separately it is worth noting that openstack is particularly susceptible to this due to the use of constraints and keeping them up to date with more recent releases | 20:50 |
| clarkb | other pip installs will often just install an older (potentially vulnerable version of the package) and continue on | 20:50 |
| clarkb | I wish that pypi would just return an error rather than failvoer and potential give people insecure packages... | 20:50 |
| fungi | also it can be interesting to note which regions the failures happened in. the usual pattern is that it impacts some specific part of the globe because of the cdn pypi relies on, so you get endpoints in, say, the montreal area returning old data while the rest of the world is unaffected | 20:51 |
| clarkb | It might be worth trying ot have a discussion with pypi about this behavior again. I think for the vast majority of their users and error would be much safer | 20:53 |
| clarkb | More annoying but safer | 20:53 |
| fungi | they're currently not indicating they're aware anything is out of sorts: https://status.python.org/ | 20:56 |
| clarkb | ya the status page doesn't always trip when this trips. But if the status page does trip we tend to see this | 20:57 |
| *** dviroel is now known as dviroel|afk | 21:09 | |
| ianw | johnsom/clarkb: huh, it must be something to do with creating it in a non-selinux context i guess, then maybe we haven't relabeled it since? i did "test" this on a live machine | 21:31 |
| ianw | i wonder about the restorecon; *maybe* that restores context from the "global" db and overwrites what we just did? | 21:32 |
| johnsom | ianw Yeah, I couldn't reproduce it locally either. I went ahead and proposed doing the persistent update to the local selinux via semanage which should resolve that error since it relabels the file. | 21:32 |
| ianw | selinux is hard enough to use when you've got it already working :/ | 21:33 |
| ianw | do you think the restorecon might undo that though? | 21:33 |
| clarkb | ya I'm mostly wondering if it is easier to use the normal path and have devsatck copy it from the various locations or if it is to manage selinux directly | 21:33 |
| johnsom | restorecon gives this: "Relabeld /var/log/unbound.log from unconfined_u:object_r:var_log_t:s0 to unconfined_u:object_r:named_log_t:80" as the output of the -v | 21:34 |
| johnsom | Had to transcribe that as it was a console session on that c9s instance | 21:35 |
| ianw | yeah that named_log_t was the key | 21:35 |
| johnsom | Without semanage stashing it in the local file, chcon would have been undone if something ran a relabel of /var/log | 21:36 |
| johnsom | man semanage-fcontexts for the details of why I went this path | 21:37 |
| ianw | right so it updates the db, which seems right | 21:38 |
| johnsom | clarkb I went down the copy them both path, but the order the zuul copies happened seemed not deterministic and since the nodepool element created a empty file, it ended up getting copied over the top of the one with data. (I was using the zuul config to collect them) | 21:40 |
| clarkb | johnsom: right yuo have to have devstack copy the file to a singular location and then zuul copies that | 21:40 |
| clarkb | there are a number of logs dont tht way iirc | 21:40 |
| johnsom | Hmm, I didn't see that outside of devstack-gate (which I think we don't use). Just the zuul_copy_output settings that collect them (the problem). | 21:42 |
| ianw | we didn't revert anything, so 841629 is required to get builds happy again right? | 21:43 |
| clarkb | ianw: correct something is required at least | 21:43 |
| clarkb | I'm happy to give 841629 a go | 21:43 |
| ianw | yeah i can watch it | 21:44 |
| opendevreview | Merged openstack/project-config master: Fix selinux context for unbound.log https://review.opendev.org/c/openstack/project-config/+/841629 | 21:56 |
| *** prometheanfire is now known as Guest0 | 22:26 | |
| *** rlandy is now known as rlandy|bbl | 23:02 | |
| johnsom | ianw Let me know if you want me to launch a test job for the log file capture. | 23:07 |
| ianw | looks like nb02 built a centos-7 with it | 23:13 |
| ianw | 2022-05-12 22:55:54.713 | + [[ -e /usr/sbin/semanage ]] | 23:13 |
| ianw | 2022-05-12 22:55:54.713 | + semanage fcontext -a -t named_log_t /var/log/unbound.log | 23:13 |
| ianw | 2022-05-12 22:56:06.993 | + restorecon -v /var/log/unbound.log | 23:13 |
| ianw | so we just have to wait for the other builds + uploads | 23:13 |
| johnsom | Ah, cool. Yeah, my test case is C9s | 23:14 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!