| opendevreview | James E. Blair proposed zuul/zuul-jobs master: buildset registry: run socat in new session https://review.opendev.org/c/zuul/zuul-jobs/+/843038 | 00:29 |
|---|---|---|
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: DNM: welfare check: ensure-podman https://review.opendev.org/c/zuul/zuul-jobs/+/843048 | 02:00 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: ensure-podman: Remove kubic from Ubuntu 18.04 https://review.opendev.org/c/zuul/zuul-jobs/+/843049 | 02:00 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: ensure-podman: Use PPA for Ubuntu 20.04 https://review.opendev.org/c/zuul/zuul-jobs/+/843093 | 02:14 |
| *** ysandeep|out is now known as ysandeep|rover | 03:24 | |
| *** ysandeep|rover is now known as ysandeep|afk | 04:24 | |
| *** ysandeep|afk is now known as ysandeep|rover | 04:45 | |
| opendevreview | yatin proposed openstack/diskimage-builder master: [DNM] Test oooq build image without CentOS preshipped repos https://review.opendev.org/c/openstack/diskimage-builder/+/794395 | 07:22 |
| *** ysandeep|rover is now known as ysandeep|rover|lunch | 08:02 | |
| *** ysandeep|rover|lunch is now known as ysandeep|rover | 08:38 | |
| opendevreview | yatin proposed openstack/diskimage-builder master: [DNM] Test oooq build image without CentOS preshipped repos https://review.opendev.org/c/openstack/diskimage-builder/+/794395 | 10:10 |
| opendevreview | Pavlo Shchelokovskyy proposed openstack/diskimage-builder master: Fix openssl example command in dynamic-login https://review.opendev.org/c/openstack/diskimage-builder/+/843128 | 11:13 |
| *** dviroel|out is now known as dviroel | 11:21 | |
| dtantsur | I think I'm seeing something weird in zuul. The 'command' ansible task ends up with stderr merged into stdout. Is it something zuul does? | 11:58 |
| dtantsur | I *seem* to recall that, but want to double-check | 11:58 |
| fungi | "merged" as in when you pull of the task detail you see things which should have been part of the sterr stream showing up the stdout box instead? | 12:10 |
| fungi | er, pull up the task detail | 12:10 |
| fungi | an example would help me undertand what you're seeing | 12:11 |
| opendevreview | Dr. Jens Harbott proposed openstack/project-config master: Add missing volumes to Grafana AFS page https://review.opendev.org/c/openstack/project-config/+/843133 | 12:16 |
| chandankumar | Hello #opendev | 12:30 |
| chandankumar | what is the sync frequency of syncing stream contents from facebook to opendev mirrors? | 12:31 |
| fungi | chandankumar: centos stream you mean? or a stream of something else? | 12:32 |
| chandankumar | fungi: centos stream 9 | 12:32 |
| chandankumar | *facebook mirror | 12:32 |
| chandankumar | what is the sync frequency of centos stream 9 contents from facebook mirror to opendev mirrors? | 12:33 |
| fungi | chandankumar: roughly every 4 hours: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/tasks/rsync.yaml#L47-L54 | 12:34 |
| fungi | er, every 6 hours | 12:34 |
| chandankumar | fungi: thanks :-) | 12:35 |
| chandankumar | I was checking this https://grafana.opendev.org/d/9871b26303/afs?orgId=1 | 12:35 |
| chandankumar | Mirror release times shows 6 hours. | 12:35 |
| chandankumar | ysandeep|rover: ^^ | 12:35 |
| ysandeep|rover | fungi, chandankumar thanks! | 12:36 |
| frickler | mirror.epel shows 3 days, /me goes looking at logs | 12:36 |
| frickler | /afs/.openstack.org/mirror/epel/7/x86_64/repodata/repomd.xml: FAILED | 12:37 |
| frickler | sha1sum: WARNING: 1 computed checksum did NOT match | 12:37 |
| frickler | source is "rsync://pubmirror1.math.uh.edu/fedora-buffet/epel/7/" , maybe someone wants to propose an alternative | 12:38 |
| fungi | that's an interesting sanity check | 12:39 |
| frickler | also a good cue to alert config-core to my patch adding some missing mirror volumes to that page https://review.opendev.org/c/openstack/project-config/+/843133 | 12:39 |
| fungi | so we're comparing the copy of repomd.xml we pulled via rsync to what's being served from the official fedora mirror i guess? | 12:39 |
| frickler | we compare to https://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/repomd.xml , which looks official, yes | 12:40 |
| chandankumar | frickler: Do we want to switch this https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/files/epel-mirror-update#L31 to facebook also? | 12:48 |
| frickler | chandankumar: whatever works, feel free to propose a patch | 13:03 |
| fungi | though whatever you propose, you might want to try to rsync a copy of epel/7/x86_64/repodata/repomd.xml and compare it to what's being served from https://dl.fedoraproject.org/ | 13:05 |
| fungi | just to make sure there's not some broader problem | 13:05 |
| dtantsur | fungi: dunno how to link to a task, check the one before the failed task: https://zuul.opendev.org/t/openstack/build/bb018fbbeda14baab8061ac96cc9b48f/console | 13:33 |
| dtantsur | the first line in stdout comes from stderr | 13:33 |
| frickler | there's a permalink button at the top of the task popup https://zuul.opendev.org/t/openstack/build/bb018fbbeda14baab8061ac96cc9b48f/console#3/1/25/controller | 13:43 |
| frickler | dtantsur: did you try adding "2>/dev/null" to your command? | 13:44 |
| fungi | dtantsur: yeah, i agree that's probably an effect of https://opendev.org/zuul/zuul/src/branch/master/zuul/ansible/base/library/command.py#L460-L467 | 13:47 |
| fungi | or maybe i'm misreading that | 13:49 |
| fungi | the exit json is passing a stderr keyarg still | 13:50 |
| fungi | yeah, i'm i'm reading correctly, stdout and stderr from the command are both interleaved into _log_lines so that it forms the live console log stream, and then the aggregate of _log_lines is passed into the exit json as stdout | 13:55 |
| fungi | er, if i'm reading correctly | 13:55 |
| fungi | so this is essentially a side effect of zuul forking the command module in order to hook the stdout and stderr from commands into the live console stream (something vanilla ansible doesn't support) | 13:56 |
| dtantsur | ouch | 14:24 |
| dtantsur | okay, thanks for explanation | 14:24 |
| fungi | dtantsur: i'm not immediately seeing why we couple sdterr and stdout rather than interleaving them from separate descriptors so they could still be separately preserved in the json. the follow() function takes care of writing the stream to a separate file anyway | 14:30 |
| fungi | maybe someone else remembers why that was necessary | 14:31 |
| fungi | i'm guessing it's something to do with needing to make sure they get ordered and timestamped correctly when you're getting buffered output | 14:32 |
| frickler | maybe zuul could offer an unstreamed_command module for such usecases | 14:37 |
| opendevreview | Merged openstack/project-config master: Add missing volumes to Grafana AFS page https://review.opendev.org/c/openstack/project-config/+/843133 | 14:41 |
| clarkb | fungi: you acn't separate them without losing information. specifically interleaving/timing info | 14:41 |
| clarkb | I think the idea was you can separate them yourself if necessary | 14:41 |
| frickler | clarkb: how would you do that in the general case? although in the current scenario maybe using instance_info.stdout_lines[-1] instead of .stdout might work | 14:45 |
| clarkb | frickler: command 2> record_stderr | 14:46 |
| clarkb | or invert it depending on which aspect you are interested in | 14:46 |
| clarkb | essentialy zuul has made a compromise to collect the least lossy information by default. but that does mean you need to take action if you want it to be lossy | 14:47 |
| frickler | clarkb: ah, sure, I though you were saying that one could separate somehow from what zuul collects | 14:47 |
| fungi | yeah, i think zuul is doing the rough equivalent of 2>&1 so that the system will perform the interleaving more accurately than zuul itself would be able to after the fact when trying to mash two separate streams together | 14:50 |
| frickler | fungi: clarkb: my propose-updates fix would still like to get reviewed if you have time https://review.opendev.org/c/openstack/project-config/+/842853 | 14:51 |
| opendevreview | Merged openstack/project-config master: Add neutron-vpnaas-stable-maint group https://review.opendev.org/c/openstack/project-config/+/842693 | 15:11 |
| *** ysandeep|rover is now known as ysandeep|out | 15:17 | |
| opendevreview | Merged openstack/diskimage-builder master: Fix openssl example command in dynamic-login https://review.opendev.org/c/openstack/diskimage-builder/+/843128 | 15:46 |
| *** dviroel is now known as dviroel|lunch | 15:58 | |
| *** marios is now known as marios|out | 15:58 | |
| clarkb | fungi: you mentioned being willing to install the wiki cert when it is available. It is now available and in the typical location. Let me know if you can't find it | 16:18 |
| fungi | thanks! i can take care of it after i get off this meeting and grab a bite to eat | 16:19 |
| mnaser | o/ how can one request the deletion of a stale feature branch (feature/lbaasv2 on neutron-vpnaas), last change was in.. 2014 | 16:21 |
| clarkb | mnaser: the openstack release team has permissions to manage branches in poenstack projects now | 16:22 |
| clarkb | (new gerrit allowed us to delegate that which is awesome) | 16:22 |
| mnaser | clarkb: oh cool, so i'll ask there then | 16:22 |
| clarkb | well to be clear they could always add branches but now they can delete them too | 16:22 |
| *** jpena is now known as jpena|off | 16:51 | |
| *** dviroel|lunch is now known as dviroel | 16:58 | |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: ensure-podman: Remove kubic from Ubuntu 18.04 and drop 20.04 https://review.opendev.org/c/zuul/zuul-jobs/+/843093 | 17:16 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: buildset registry: run socat in new session https://review.opendev.org/c/zuul/zuul-jobs/+/843038 | 17:16 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: ensure-podman: Remove kubic from Ubuntu 18.04 and drop 20.04 https://review.opendev.org/c/zuul/zuul-jobs/+/843093 | 17:46 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: buildset registry: run socat in new session https://review.opendev.org/c/zuul/zuul-jobs/+/843038 | 17:46 |
| fungi | the last entry in statusbot's debug log is 2022-05-17 16:23:41 | 17:48 |
| fungi | the daemon is still running though | 17:49 |
| fungi | 2022-05-17 16:29:46 <-- opendevstatus (~opendevst@eavesdrop01.opendev.org) has quit (synthon.oftc.net reflection.oftc.net) | 17:50 |
| fungi | ps says it's been running since april 20 though | 17:50 |
| fungi | nothing useful in the docker logs either | 17:52 |
| fungi | #status log Restarted statusbot container on eavesdrop01 since it appeared to disconnect from the IRC server around 2022-05-17 16:30 UTC and did not reconnect on its own (but also logged nothing to explain why) | 17:54 |
| opendevstatus | fungi: finished logging | 17:55 |
| clarkb | corvus: re the nodepool changes remember launchers don't auto update (builders should) | 17:59 |
| opendevreview | Merged openstack/project-config master: Fix python list for requirements propose-updates job https://review.opendev.org/c/openstack/project-config/+/842853 | 18:18 |
| jrosser | if i have a job that passes on on 2nd or 3rd attempts, would those attempts be (potentially) in a different provider from the ones that failed | 18:26 |
| jrosser | i'm wondering if i have something broken with setting up the extra volume on some providers (rax?) | 18:28 |
| frickler | jrosser: yes, that's possible. you should be able to see the retried jobs in the build results to verify | 18:29 |
| jrosser | oh, it keeps the failed ones? | 18:29 |
| frickler | jrosser: do you have a link to a failure? | 18:29 |
| jrosser | heres one that passed on the second attempt https://zuul.opendev.org/t/openstack/build/5d6068be71af4ae1808d5d3e25aee6c4 | 18:30 |
| jrosser | but i'm kind of wondering why the first failed | 18:30 |
| jrosser | ah is that what "view build history" is showing | 18:31 |
| frickler | jrosser: if you look at the "build history" link, you can see the "retry" | 18:31 |
| jrosser | excellent, just what i need | 18:31 |
| frickler | looks like rax vs. inmotion indeed | 18:33 |
| jrosser | i think i've a bug which is only triggered when setting up the extra disk | 18:33 |
| frickler | clarkb: it seems you missed my earlier comment about the project.opendev volume. I don't think it really needs 1TB of quota? https://grafana.opendev.org/d/9871b26303/afs?orgId=1&from=now-7d&to=now&viewPanel=68 | 18:48 |
| clarkb | frickler: what path is that mounted at? I wonder if it is hosting tarballs.opendev.org ? | 18:53 |
| clarkb | In which case it might need some quota, but I agree 1TB seems excessive | 18:53 |
| clarkb | maybe we reset it to 10GB | 18:54 |
| clarkb | I can look into that later today | 18:54 |
| frickler | /afs/.openstack.org/project/opendev.org/ | 18:57 |
| fungi | the project/opendev.org mount is indeed separate from project/tarballs.opendev.org | 18:58 |
| fungi | though there is a tarballs.old in it | 18:58 |
| clarkb | in that case 1GB is probably plenty | 18:59 |
| clarkb | that is about 10x the current usage | 19:00 |
| clarkb | I went ahead and set it to 1GB | 19:39 |
| fungi | thanks! | 19:41 |
| frickler | ianw: I think I did set up the jammy wheel volumes now correctly. if you can double check and then go on that would be great | 19:58 |
| ianw | frickler: will do | 20:00 |
| ianw | thanks, acl's look right, so all lgtm | 20:04 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Install setfacl on test nodes https://review.opendev.org/c/openstack/project-config/+/843210 | 20:05 |
| clarkb | frickler: infra-root ^ turns out this is one of those packages where almost everyone agrees on the package name which is nice | 20:05 |
| fungi | convenient | 20:06 |
| ianw | maybe it should be a dib thing too? i've managed to be using linux for about 30 years and never once to my knowledge manually called setfacl afaik | 20:09 |
| clarkb | ianw: maybe? I feel like the ansible situation is a bit of a corner case but one likely in a CI system | 20:10 |
| fungi | it's taken a while to catch on | 20:10 |
| clarkb | or at least CI system that uses ansible | 20:10 |
| opendevreview | Merged openstack/project-config master: Build and publish wheel mirror for Ubuntu Jammy https://review.opendev.org/c/openstack/project-config/+/842841 | 20:13 |
| ianw | "The acl package is a dependency of systemd, it should already be installed. " | 20:13 |
| ianw | i wonder if it's a reccommendation that we're not pulling in because of ... something (hand wavy something about debootstrap, etc) | 20:13 |
| ianw | seems to require libacl1, but maybe not "acl" | 20:15 |
| ianw | the fault in my thinking is that you'd want to do anything on your system outside of having systemd do it for you. clearly crazy :) | 20:17 |
| fungi | systemd must have its own facl entrypoint | 20:17 |
| ianw | still, given that it falls under "makes system work with ansible" adding it to even minimal dib images i think would be fine | 20:19 |
| ianw | https://bugs.launchpad.net/ubuntu/+source/acl/+bug/1913666 similar sort of bug about it not being in base images | 20:21 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: DNM: test buildset registry jobs for bitrot https://review.opendev.org/c/zuul/zuul-jobs/+/843212 | 20:31 |
| opendevreview | Merged openstack/project-config master: Install setfacl on test nodes https://review.opendev.org/c/openstack/project-config/+/843210 | 20:34 |
| *** dviroel is now known as dviroel|out | 21:11 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!