| corvus | fyi my reply in gerrit just took like 30-60 seconds. | 00:50 |
|---|---|---|
| ianw | hrm, just saw this; cpu and load average graphs look normal | 02:08 |
| ianw | perhaps there was a bit of a traffic spike around then | 02:09 |
| fungi | backups? | 02:21 |
| ianw | quite likely i guess | 03:52 |
| opendevreview | Merged openstack/project-config master: Add Grafana dashboard for the Tobiko CI https://review.opendev.org/c/openstack/project-config/+/850953 | 04:09 |
| *** ysandeep|out is now known as ysandeep | 05:58 | |
| opendevreview | Ian Wienand proposed openstack/project-config master: project-config-grafyaml: drop job https://review.opendev.org/c/openstack/project-config/+/851954 | 06:20 |
| opendevreview | Ian Wienand proposed openstack/project-config master: grafana: helper script updates https://review.opendev.org/c/openstack/project-config/+/851956 | 06:29 |
| opendevreview | wangxiyuan proposed openstack/project-config master: Bump openEuler to 22.03 LTS https://review.opendev.org/c/openstack/project-config/+/851957 | 06:40 |
| *** jpena|off is now known as jpena | 07:14 | |
| frickler | I'm seeing query errors for all the graphs on the AFS page while the timer panels seem to work fine https://grafana.opendev.org/d/9871b26303/afs?orgId=1 | 07:30 |
| *** ysandeep is now known as ysandeep|mtg | 07:57 | |
| *** anbanerj is now known as frenzyfriday|rover | 07:59 | |
| *** ysandeep|mtg is now known as ysandeep | 08:37 | |
| *** tosky_ is now known as tosky | 08:47 | |
| *** ysandeep is now known as ysandeep|lunch | 09:04 | |
| *** ysandeep|lunch is now known as ysandeep | 10:00 | |
| *** rlandy|out is now known as rlandy | 10:37 | |
| opendevreview | jayaditya gupta proposed openstack/diskimage-builder master: Closes-Bug: 1974350 https://review.opendev.org/c/openstack/diskimage-builder/+/850882 | 10:47 |
| *** bhagyashris is now known as bhagyashris|ruck | 11:11 | |
| *** dviroel|afk is now known as dviroel | 11:23 | |
| *** ysandeep is now known as ysandeep|afk | 12:25 | |
| *** ysandeep|afk is now known as ysandeep | 12:25 | |
| *** dasm|off is now known as dasm | 13:18 | |
| *** pojadhav is now known as pojadhav|afk | 13:31 | |
| *** frenzyfriday|rover is now known as frenzyfriday|rover|lunch | 13:36 | |
| *** frenzyfriday|rover|lunch is now known as frenzyfriday|rover | 14:28 | |
| clarkb | frickler: 'Query error: -1 Request was aborted' is reported by the grafana page. If I look at the graphite metrics browser both metrics are there. I suspect the problem is with the construction of the relative percentage calculation as a result | 14:43 |
| clarkb | maybe we can't refer to things using #A as a refid anymore? | 14:44 |
| *** dviroel is now known as dviroel|lunch | 14:48 | |
| frickler | clarkb: I noticed that other pages seem broken in the same way, did we update grafana recently? | 14:48 |
| clarkb | image: docker.io/grafana/grafana-oss:latest <- we may haev | 14:49 |
| frickler | REPOSITORY TAG IMAGE ID CREATED SIZE | 14:49 |
| frickler | grafana/grafana-oss latest 1e5c4e689218 28 hours ago 296MB | 14:49 |
| frickler | seems like it | 14:49 |
| frickler | docker log has a lot of: level=warn msg="Request Origin is not authorized" | 14:52 |
| frickler | but not sure if related | 14:52 |
| frickler | do we want to try and revert to an older container for now? seems the one on the server got pruned, though | 14:52 |
| Clark[m] | All of the state is in graphite not grafana so in theory that is a safe thing to try | 14:54 |
| Clark[m] | Might be worth root causing the actual failure first if we can. I've got errands I have to do today unfortunately though | 14:54 |
| *** ysandeep is now known as ysandeep|out | 15:07 | |
| odyssey4me | hey folks - not sure who reviews JJB things any more, but there's a patch up which could do with some eyes: https://review.opendev.org/c/jjb/jenkins-job-builder/+/817665 | 15:08 |
| frickler | the latest CI snapshots in ianw's patch look broken the same way https://e5b99a76528ca038c82c-ece6d16851dc34e48db19a7f0145a6c3.ssl.cf5.rackcdn.com/851956/1/check/project-config-grafana/f5d3988/screenshots/afs.png | 15:11 |
| frickler | humm, I tried running with the most recent stable container (9.0.6) while our :latest seems to match 9.1.0-beta1 on https://hub.docker.com/r/grafana/grafana-oss/tags , now all dashboards are gone, even after running with :latest again | 15:23 |
| frickler | I'm hoping they'll get restored with the next hourly deployment | 15:23 |
| frickler | if not, I'll need to dig further or maybe wait for ianw to come to the rescue | 15:24 |
| Clark[m] | Wow they tag beta images with latest? That is not ideal | 15:27 |
| frickler | that's surprising, yes, but you can see it in the docker log, too https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_5ac/848212/3/check/project-config-grafana/5ac2951/docker/grafana-opendev_test.txt | 15:30 |
| frickler | the last working run of that job from a week ago has 9.0.4 | 15:30 |
| fungi | odyssey4me: they have their own channel, #openstack-jjb (not sure if any of the maintainers pay attention in there either though) | 15:32 |
| odyssey4me | fungi right, thanks - will go poking there - thanks! | 15:34 |
| frickler | o.k., restored the dashboard with the command used to import them in the test job. seems to be working fine now with 9.0.4, I assume we will be able to debug further in CI. I'll make a system-config patch to pin the version for now | 15:46 |
| opendevreview | Merged openstack/project-config master: Add OpenStack K8S charms https://review.opendev.org/c/openstack/project-config/+/849996 | 15:47 |
| fungi | thanks frickler! | 15:48 |
| opendevreview | Dr. Jens Harbott proposed opendev/system-config master: Pin version of grafana-oss container https://review.opendev.org/c/opendev/system-config/+/852037 | 15:49 |
| frickler | fungi: clarkb: ianw: ^^ I'll stop at that for today, I can continue tomorrow if needed | 15:54 |
| *** marios is now known as marios|out | 15:57 | |
| Clark[m] | Thanks! | 16:03 |
| *** dviroel|lunch is now known as dviroel | 16:13 | |
| *** jpena is now known as jpena|off | 16:24 | |
| *** frenzyfriday|rover is now known as frenzyfriday|rover|afk | 16:47 | |
| hjensas | ianw: since https://review.opendev.org/c/openstack/tripleo-image-elements/+/846807 tripleo is using separate /boot partition in images. We got this bug reported, https://bugs.launchpad.net/tripleo/+bug/1983467 - for some reason the label is "mkfs_boot" and not just "boot". if I'm not mistaken this is the code to look at https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/block_device/level2/mkfs.py#L58-L | 16:49 |
| hjensas | 59 ? so I would expect label == name unless label is set? | 16:49 |
| opendevreview | Merged opendev/system-config master: Pin version of grafana-oss container https://review.opendev.org/c/opendev/system-config/+/852037 | 18:24 |
| *** frenzyfriday|rover|afk is now known as frenzyfriday|rover | 18:58 | |
| *** rlandy is now known as rlandy|brb | 19:24 | |
| fungi | pip 22.2.2 just went out https://pip.pypa.io/en/stable/news/#v22-2-2 | 19:37 |
| fungi | looks like it's all bug fixes for problems i haven't encountered though | 19:38 |
| *** rlandy|brb is now known as rlandy | 20:07 | |
| *** dviroel is now known as dviroel|afk | 20:34 | |
| *** timburke__ is now known as timburke | 20:46 | |
| *** rlandy is now known as rlandy|out | 21:32 | |
| ianw | frickler/fungi: thanks for looking at it :) not sure i have any rescue steps :/ I tried running grafyaml directly against python3.10 late yesterday (usually i use the container) and unfortunately voluptuous doesn't work with it, and is pinned to an ancient version, and unpinning it doesn't work and it's not an obvious fix | 22:09 |
| fungi | ick | 22:10 |
| ianw | so i mean it's fine in it's 3.8 container, but it's not going to magically get better | 22:10 |
| ianw | you know i have some vague memory from looking at things that the datasource setup we use is deprecated for some reason, i wonder if it finally got pulled out | 22:11 |
| clarkb | fungi: do you know the answer for the https question on https://review.opendev.org/c/opendev/system-config/+/851094/2 ? | 22:50 |
| clarkb | I guess I can update it and see if it works | 22:55 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Run our base playbook on jammy https://review.opendev.org/c/opendev/system-config/+/851094 | 22:59 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Ensure /etc/apt/sources.list.d exists https://review.opendev.org/c/opendev/system-config/+/851266 | 22:59 |
| opendevreview | Clark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server https://review.opendev.org/c/opendev/system-config/+/851248 | 22:59 |
| ianw | sorry just had to stop the v6 dhcp client, reset the duid and restart, twice, to get ipv6 back. just totally normal things that everyone should be able to do | 23:10 |
| ianw | right, so | 23:12 |
| ianw | Deprecation Notice | 23:12 |
| ianw | This data source uses browser access mode. This mode is deprecated and will be removed in the future. Please use server access mode instead. | 23:12 |
| ianw | is what i was thinking of | 23:12 |
| ianw | re grafana | 23:12 |
| ianw | i think the difference is that the browser doesn't talk to graphite, it gets proxied through grafana | 23:13 |
| ianw | that seems unlikley to be it; https://github.com/grafana/grafana/commits/06d3c27bc11c0f83579550d9c6a3cd03453ed1e2/public/app/plugins/datasource/graphite/configuration/ConfigEditor.tsx | 23:18 |
| ianw | still shows the settings | 23:18 |
| *** dasm is now known as dasm|off | 23:26 | |
| *** undefined is now known as Guest7047 | 23:34 | |
| ianw | 9.0.6 works, so it is something to do with the beta | 23:53 |
| fungi | clarkb: we probably can use https there, i'm not opposed (but also it provides little real benefit since the package manager is going to verify checksums from a list signed by a key in its trusted keyring, so at most it provides a questionable amount of privacy) | 23:58 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
