Thursday, 2022-08-04

opendevreviewIan Wienand proposed openstack/project-config master: grafana: helper script updates  https://review.opendev.org/c/openstack/project-config/+/85195600:04
ianwif anyone has a quick sec for https://review.opendev.org/c/openstack/project-config/+/851956 to update the grafana helper script that would be good00:07
ianwi might reference it in a grafana bug, it's clearly something to do with 9.1.0-beta100:07
ianwthough maybe it would be more helpful to put a node on hold with it00:08
*** undefined_ is now known as Guest704900:09
opendevreviewMerged openstack/project-config master: grafana: Add infra deployment overview  https://review.opendev.org/c/openstack/project-config/+/83321300:11
opendevreviewMerged openstack/project-config master: grafana: add helper script  https://review.opendev.org/c/openstack/project-config/+/83321400:11
opendevreviewMerged openstack/project-config master: grafana: update nodepool-dib status page  https://review.opendev.org/c/openstack/project-config/+/84821200:11
opendevreviewIan Wienand proposed opendev/system-config master: [wip] Revert "Pin version of grafana-oss container"  https://review.opendev.org/c/opendev/system-config/+/85205600:11
ianwhjensas: i see from the bug looking for uuids is a better plan, but yeah that still seems odd00:30
ianw(more an #openstack-dib topic)00:30
ianwif you override it with a label, it should work https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/block_device/level2/mkfs.py#L5800:32
ianwiirc the default names flatten themselves from the tree with _'s, so like mkfs_upper_lower00:33
ianwi've got a node on hold with the grafana issue and filed https://github.com/grafana/grafana/issues/5327501:08
fungiianw: 851956 lgtm other than the sleep may still be too short01:10
ianwyeah arbitrary timeouts aren't good.  really i guess it should probe with curl for maximum reliability01:11
opendevreviewIan Wienand proposed openstack/project-config master: grafana: better loop for container start  https://review.opendev.org/c/openstack/project-config/+/85206201:17
ianw^ should be less lame01:17
opendevreviewIan Wienand proposed opendev/system-config master: sync-project-config: specify delete and no owner/group  https://review.opendev.org/c/opendev/system-config/+/85206301:49
ianwinfra-root: I think that's why grafana is currently failing to deploy; as it has duplicate graphs.  I will do a manual clean on the grafana /opt/project-config dir, which should fix this for now01:50
ianwarrgh, now it's failing for some other reason01:52
ianwon the graph import01:52
opendevreviewOpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml  https://review.opendev.org/c/openstack/project-config/+/85206502:20
ianwok, that wasn't too hard; it was because nothing ever pulls the latest grafyaml image, so it was using an old version02:40
*** Guest7049 is now known as rcastillo02:44
opendevreviewIan Wienand proposed opendev/system-config master: grafana: pull the grafyaml image before running  https://review.opendev.org/c/opendev/system-config/+/85206702:49
ianwit works! https://grafana.opendev.org/d/a0241a42c4/opendev-infra-deployment-overview?orgId=103:02
*** pojadhav|afk is now known as pojadhav03:15
opendevreviewMerged openstack/project-config master: grafana: helper script updates  https://review.opendev.org/c/openstack/project-config/+/85195603:16
opendevreviewMerged openstack/project-config master: grafana: better loop for container start  https://review.opendev.org/c/openstack/project-config/+/85206203:23
ianwclarkb/frickler: re 851094 there isn't a https site, so i think that's made the job fail there.  03:37
opendevreviewIan Wienand proposed opendev/system-config master: Run our base playbook on jammy  https://review.opendev.org/c/opendev/system-config/+/85109403:40
opendevreviewIan Wienand proposed opendev/system-config master: Ensure /etc/apt/sources.list.d exists  https://review.opendev.org/c/opendev/system-config/+/85126603:40
opendevreviewIan Wienand proposed opendev/system-config master: WIP Add a mailman3 list server  https://review.opendev.org/c/opendev/system-config/+/85124803:40
ianwi just s/https/http/ there to run it through again, all lgtm03:41
opendevreviewIan Wienand proposed openstack/project-config master: grafana: pull grafyaml in helper script  https://review.opendev.org/c/openstack/project-config/+/85207103:44
opendevreviewMerged opendev/system-config master: run-production-playbook: rename with original timestamp  https://review.opendev.org/c/opendev/system-config/+/85012303:53
opendevreviewIan Wienand proposed openstack/project-config master: grafana: pull grafyaml in helper script  https://review.opendev.org/c/openstack/project-config/+/85207103:59
opendevreviewMerged openstack/project-config master: grafana: pull grafyaml in helper script  https://review.opendev.org/c/openstack/project-config/+/85207105:04
opendevreviewIan Wienand proposed opendev/system-config master: static: remove unused letsencrypt group variable  https://review.opendev.org/c/opendev/system-config/+/84890606:04
*** ysandeep|out is now known as ysandeep06:04
opendevreviewIan Wienand proposed opendev/system-config master: static: move certs to group, update testing name to static99  https://review.opendev.org/c/opendev/system-config/+/84890806:11
opendevreviewIan Wienand proposed opendev/system-config master: mirror: retwork testinfra testing  https://review.opendev.org/c/opendev/system-config/+/84891206:22
opendevreviewMerged openstack/project-config master: Bump openEuler to 22.03 LTS  https://review.opendev.org/c/openstack/project-config/+/85195706:50
*** tweining|off is now known as tweining06:51
ianwfrickler: i think https://grafana.opendev.org/d/f3089338b3/nodepool-dib-status?orgId=1 is showing us openeuler fails, which i hope ^^ will fix06:52
opendevreviewMerged opendev/system-config master: gitea-lb: test ssl connections during testing too  https://review.opendev.org/c/opendev/system-config/+/84877707:19
opendevreviewMerged opendev/system-config master: paste : move testing host to paste99, remove https hacks  https://review.opendev.org/c/opendev/system-config/+/84868507:19
*** soniya29 is now known as soniya29|afk07:19
fricklerianw: after nobody cared to maintain the devstack support, I'm not sure what use case this has07:20
fricklerprometheanfire: gentoo builds are also failing again with some python related ebuild conflict, maybe you have an idea for that? https://nb01.opendev.org/gentoo-17-0-systemd-0000228574.log07:21
*** jpena|off is now known as jpena07:26
*** ysandeep is now known as ysandeep|afk07:27
*** undefined is now known as Guest707907:43
opendevreviewMerged openstack/project-config master: Normalize projects.yaml  https://review.opendev.org/c/openstack/project-config/+/85206507:59
opendevreviewSlawek Kaplonski proposed openstack/project-config master: [Grafana] Neutron dasboard change Fedora job to Centos 9 Stream  https://review.opendev.org/c/openstack/project-config/+/85211008:03
*** soniya29|afk is now known as soniya2908:07
*** ysandeep|afk is now known as ysandeep08:22
opendevreviewHemanth N proposed openstack/project-config master: Add official-openstack-repo-jobs to openstack k8s charms  https://review.opendev.org/c/openstack/project-config/+/85211708:38
opendevreviewHemanth N proposed openstack/project-config master: Add official-openstack-repo-jobs to openstack k8s charms  https://review.opendev.org/c/openstack/project-config/+/85211708:52
fricklerianw: deployment failed https://zuul.opendev.org/t/openstack/build/3369125194544ac996276a1b8fa35a7e , I'll stop the 20.x builds now to keep them from spinning09:04
frickleralso paused gentoo-17-0-systemd once again while at it09:07
fricklerRamereth: quite a number of images stuck in deleting again, if you get a chance to have a look that would be great09:08
fricklerianw: the nodepool config is broken actually: KeyError: 'openEuler-22-03'09:44
fricklerseems to be missing the "-LTS" suffix09:47
opendevreviewDr. Jens Harbott proposed openstack/project-config master: Fix mismatch for openEuler image name  https://review.opendev.org/c/openstack/project-config/+/85212709:50
opendevreviewDr. Jens Harbott proposed openstack/project-config master: Add nested-virt-ubuntu-jammy nodes  https://review.opendev.org/c/openstack/project-config/+/85213610:10
opendevreviewMerged openstack/project-config master: Fix mismatch for openEuler image name  https://review.opendev.org/c/openstack/project-config/+/85212710:13
opendevreviewIan Wienand proposed opendev/system-config master: run-production-playbook-post : fix log name typo  https://review.opendev.org/c/opendev/system-config/+/85214710:28
ianwfrickler: thanks10:28
*** rlandy|out is now known as rlandy10:35
opendevreviewIan Wienand proposed opendev/system-config master: system-config-run-gitea: increase timeout  https://review.opendev.org/c/opendev/system-config/+/85215410:37
opendevreviewIan Wienand proposed opendev/system-config master: system-config-run-gitea: increase timeout  https://review.opendev.org/c/opendev/system-config/+/85215410:38
ianwclarkb: ^ perhaps we should put some more effort into that and try to trim the import or similar?10:38
opendevreviewMerged opendev/system-config master: run-production-playbook-post : fix log name typo  https://review.opendev.org/c/opendev/system-config/+/85214711:17
opendevreviewMerged opendev/system-config master: Run our base playbook on jammy  https://review.opendev.org/c/opendev/system-config/+/85109411:34
*** dviroel|afk is now known as dviroel11:34
fungireminder, i'm not around much today but i'll try to check in briefly when i can11:52
opendevreviewMerged openstack/project-config master: linter: add name to blocks  https://review.opendev.org/c/openstack/project-config/+/85127611:53
opendevreviewMerged openstack/project-config master: linter: use spaces between filters in jinja  https://review.opendev.org/c/openstack/project-config/+/85127711:54
*** ysandeep is now known as ysandeep|afk12:03
opendevreviewMerged opendev/system-config master: Ensure /etc/apt/sources.list.d exists  https://review.opendev.org/c/opendev/system-config/+/85126612:04
opendevreviewMerged opendev/system-config master: gitea: test loadbalancer to gitea  https://review.opendev.org/c/opendev/system-config/+/84868712:41
opendevreviewMerged opendev/system-config master: static: remove unused letsencrypt group variable  https://review.opendev.org/c/opendev/system-config/+/84890612:41
*** Guest7079 is now known as rcastillo13:03
*** dasm|off is now known as dasm13:15
opendevreviewNeil Hanlon proposed openstack/project-config master: Add rockylinux 9 to nodepool  https://review.opendev.org/c/openstack/project-config/+/85216713:28
*** ysandeep|afk is now known as ysandeep13:45
prometheanfirefrickler: source of the issue seems to be on re-installing portage14:12
prometheanfireI think default python on gentoo is now 3.10, which could be causing the issue14:13
prometheanfireif you search the log for 'One of the following packages is required to complete your request' you will see it offers two options, one that enables py310, one that disables it (on two different packages)14:14
*** frenzyfriday|rover is now known as frenzyfriday|rover|lunch14:16
prometheanfirefrickler: one option is to clear out the binpkg cache as well, in case one of the old packages is messing with things14:17
opendevreviewAlfredo Moralejo proposed zuul/zuul-jobs master: Use RDO wallaby repo to install openvswitch in CS9  https://review.opendev.org/c/zuul/zuul-jobs/+/85217614:36
*** ysandeep is now known as ysandeep|out14:52
fungiaround for a few minutes and looking at the deploy post_failure situation. 852147 seems to have been incomplete, now we're getting "missing required arguments: path" from the reset file time task15:06
fungithough i can't find that task anywhere15:13
fungihttps://zuul.opendev.org/t/openstack/build/2a3f830593a442b0889cf790b382386115:13
funginevermind, it's here: https://opendev.org/opendev/system-config/src/branch/master/playbooks/zuul/run-production-playbook-post.yaml#L104-L11315:15
fungii guess the "src" in there should be "path" instead?15:17
fungiyeah, https://docs.ansible.com/ansible/latest/collections/ansible/builtin/file_module.html#parameters seems to confirm15:17
opendevreviewJeremy Stanley proposed opendev/system-config master: Correct file time task in production post playbook  https://review.opendev.org/c/opendev/system-config/+/85218015:19
fungiinfra-root: brown bag fix ^15:19
fricklerfungi: there's yet another bug there, I'll update your patch15:32
fungithanks!15:33
*** gibi is now known as gibi_pto15:37
fricklerfungi: no, disregard, I just rediscovered https://review.opendev.org/c/opendev/system-config/+/852147, which your patch wasn't based upon yet, which made me quite confused15:39
fungiahh15:41
frickleror rather gerrit's "show blame" wasn't refreshed? now I'm more confused. anyway your fix seems legit15:41
fungii thought i had done it on the tip of the master branch, but i'm quite literally "phoning it in" from a waiting room hours from home15:42
clarkbapproved thank you for fixing that15:44
*** dviroel is now known as dviroel|lunch15:47
*** ysandeep|out is now known as ysandeep|PTO15:51
fungithanks!16:04
fungii'm back on the road, so disappearing again for the next few hours16:04
opendevreviewMerged opendev/system-config master: Correct file time task in production post playbook  https://review.opendev.org/c/opendev/system-config/+/85218016:08
Ramerethfrickler: I'll take a look at that later today16:12
*** jpena is now known as jpena|off16:21
*** marios is now known as marios|out16:35
opendevreviewClark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server  https://review.opendev.org/c/opendev/system-config/+/85124816:40
*** dviroel|lunch is now known as dviroel16:52
*** frenzyfriday|rover|lunch is now known as frenzyfriday|rover16:55
opendevreviewClark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server  https://review.opendev.org/c/opendev/system-config/+/85124817:04
clarkbit helps to git add the new files :)17:05
opendevreviewMerged opendev/system-config master: system-config-run-gitea: increase timeout  https://review.opendev.org/c/opendev/system-config/+/85215417:06
opendevreviewClark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server  https://review.opendev.org/c/opendev/system-config/+/85124817:22
opendevreviewClark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server  https://review.opendev.org/c/opendev/system-config/+/85124818:08
opendevreviewClark Boylan proposed opendev/system-config master: Fix strftime format string in infra-prod jobs  https://review.opendev.org/c/opendev/system-config/+/85220418:37
opendevreviewClark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server  https://review.opendev.org/c/opendev/system-config/+/85124818:39
fricklerha, I just wanted to write that there's yet another error now, and there's the solution already :)18:39
clarkbfrickler: yup I just noticed. It took me a bit to figure it out since it complains about the - but I'm pretty sure that is because %- isn't a valid substitution18:40
opendevreviewClark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server  https://review.opendev.org/c/opendev/system-config/+/85124819:14
frickleropeneuler build is still failing with the new version https://nb02.opendev.org/openEuler-22-03-LTS-0000000003.log19:17
opendevreviewMerged openstack/project-config master: [Grafana] Neutron dasboard change Fedora job to Centos 9 Stream  https://review.opendev.org/c/openstack/project-config/+/85211019:36
opendevreviewMerged opendev/system-config master: Fix strftime format string in infra-prod jobs  https://review.opendev.org/c/opendev/system-config/+/85220419:57
clarkbhttp://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_7e6/851248/27/check/system-config-run-lists3/7e6aa8e/bridge.openstack.org/screenshots/mailman3-main.png Boom we have mm3 running well enough behind a web proxy to render that webpage20:16
ianwgood heavens, sorry about all the bugs in that.  i tested that locally :/20:47
clarkbanyone know of any public mm3 sites that I can look at to generate some selenium screenshot urls without having to hold a node?20:53
fungifedora20:53
fungiclarkb: https://lists.fedoraproject.org/20:54
clarkbthanks20:54
opendevreviewClark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server  https://review.opendev.org/c/opendev/system-config/+/85124820:57
clarkbI feel like there are a million little details that still need to be sorted out. But we are fast appraoching a thing that accomplishes "something"20:57
clarkbI'm largely punting on the exim config too since I think others understand that better than me and can hopefully help out with that20:58
*** dviroel is now known as dviroel|afk20:59
fungiwell, that and we can hopefully get by as we have previously with "here have the defaults"21:11
fungimostly at least21:11
opendevreviewIan Wienand proposed opendev/base-jobs master: opendev-release-python: switch to pypi API key  https://review.opendev.org/c/opendev/base-jobs/+/85222222:10
*** dasm is now known as dasm|off22:11
opendevreviewIan Wienand proposed opendev/system-config master: static: move certs to group, update testing name to static99  https://review.opendev.org/c/opendev/system-config/+/84890822:19
opendevreviewIan Wienand proposed opendev/system-config master: paste: move certificate to group variable  https://review.opendev.org/c/opendev/system-config/+/84891722:19
opendevreviewIan Wienand proposed opendev/system-config master: mirror: retwork testinfra testing  https://review.opendev.org/c/opendev/system-config/+/84891222:19
opendevreviewIan Wienand proposed opendev/system-config master: mirror-update: move testing to mirror-update99  https://review.opendev.org/c/opendev/system-config/+/84891922:19
opendevreviewIan Wienand proposed opendev/system-config master: letsencrypt-acme-sh-install: handle errors better in driver  https://review.opendev.org/c/opendev/system-config/+/69621122:19
opendevreviewIan Wienand proposed opendev/system-config master: letsencrypt: make acme.sh exits clearer  https://review.opendev.org/c/opendev/system-config/+/85043522:19
opendevreviewClark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server  https://review.opendev.org/c/opendev/system-config/+/85124822:19
clarkbianw: for the pypi token, for some reason I thought a different token was generated for each project? I guess that is optional?22:20
ianwclarkb: you can make a token that is scoped to only a given project22:21
clarkbif we approve that now I guess we want to notify the openstack release team to try it first?22:21
ianwor even projectS22:21
clarkbbefore making a ton of releases I mean22:21
ianwwell the openstack side is already done with https://review.opendev.org/c/openstack/project-config/+/84976322:22
clarkboh hrm, we share the user across multiple secrets?22:22
clarkbI guess due to different tenants and things22:22
ianwyeah we have one user, openstackci -- and that now has two upload tokens registered to it -- the one in project-config and the one under review in opendev/base-jobs22:23
opendevreviewMerged opendev/base-jobs master: opendev-release-python: switch to pypi API key  https://review.opendev.org/c/opendev/base-jobs/+/85222222:29
clarkbunrelated to everything else my laptop brightness buttons stopped working. So now I'm echo'ing ints into /sys/class/backlight/amdgpu_bl0/brightness when I take the laptop outside22:29
fungii've always had to do that because the xorg integration piece is missing for the intel backlight in this netbook22:30
fungii have a convenience script i call22:30
ianw"convenience" :)22:31
fungiclarkb: https://paste.opendev.org/show/bDRiQoIJuFFx8Hn0DeG1/22:32
fungii call that as a toggle from a menu option22:32
ianwfungi/clarkb: https://review.opendev.org/c/opendev/system-config/+/852063 is one i found yesterday about project-config pulling.  i feel kind of surprised that we never noticed22:32
fungiianw: interesting! yeah that would be a problem22:34
clarkbfungi: fun. This one worked until very recently22:34
clarkbianw: ya I think our puppet modules were synced that way too. It never really caused problem sand we're almost off of them22:35
fungibut yeah, that script is basically because i got tired of doing what you described22:35
clarkbianw: can you check my question on that22:38
clarkbianw: do you mean the files are owned by zuul? I think that may be the case "sometimes" and there may be anothe rissue here.22:41
clarkbThe 'Update from master' task runs as root delegate dto bridge if we are updating project-config from master. And that modifies a repo managed by zuul in the zuul homedir?22:42
clarkbWe might also need to run that task as the zuul user?22:42
clarkbposted a second question22:44
clarkbif/when we land that fix we should avoid landing any project creations until we are happy that the repo is updating properly22:44
clarkbit is a bit odd that the kernel exposes an interface that shows the max brightness value and we can't get a thing that works universally on percentages22:48
opendevreviewClark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server  https://review.opendev.org/c/opendev/system-config/+/85124822:53
*** rlandy is now known as rlandy|out22:54
ianwclarkb: well the weird thing was on grafana it's owned by 203123:08
ianw$ ls -ld /opt/project-config/23:08
ianwdrwxr-xr-x 17 2031 2031 4096 Aug  4 19:49 /opt/project-config/23:08
ianwwhich is the zuul user on bridge23:09
ianwi guess it is syncing project_config_src which is /home/zuul/src/opendev.org/openstack/project-config ... which was checked out by zuul23:13
opendevreviewNeil Hanlon proposed openstack/project-config master: Add rockylinux 9 to nodepool  https://review.opendev.org/c/openstack/project-config/+/85216723:18
clarkbianw: ya I think its preserving the perms from the source side, not setting them to who is running ansible23:30
clarkbianw: I think we should drop the comment maybe23:30
*** dviroel|afk is now known as dviroel23:30
opendevreviewIan Wienand proposed opendev/system-config master: run-production-playbook-post: become when updating file time  https://review.opendev.org/c/opendev/system-config/+/85222423:30
ianwclarkb: yeah, will respin23:30
ianwi think every part of that playbook-post timestamp update was broken in some way :/  23:31
*** dviroel is now known as dviroel|out23:36
opendevreviewIan Wienand proposed opendev/system-config master: sync-project-config: specify delete and no owner/group  https://review.opendev.org/c/opendev/system-config/+/85206323:37
opendevreviewIan Wienand proposed opendev/system-config master: grafana: pull the grafyaml image before running  https://review.opendev.org/c/opendev/system-config/+/85206723:37
opendevreviewClark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server  https://review.opendev.org/c/opendev/system-config/+/85124823:42
clarkbI think I'm getting close to needing to hold a node again :/ but progress23:45
ianwthank you, it will be very cool to have this updated23:46

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!