| *** rlandy is now known as rlandy|out | 00:30 | |
| *** JasonF is now known as JayF | 00:54 | |
| *** yadnesh|away|pto is now known as yadnesh | 03:59 | |
| ianw | i've sent a mail about the columns for labels in gerrit 3.6 @ https://groups.google.com/g/repo-discuss/c/gYVD-iBR9Ow/m/p_i5L4_vAQAJ | 04:25 |
|---|---|---|
| *** ysandeep is now known as ysandeep|ruck | 05:42 | |
| frickler | looks like a number of mirror volumes are getting close to being full, also afs01.dfw is > 90% now https://grafana.opendev.org/d/9871b26303/afs?orgId=1&from=now-6M&to=now | 05:46 |
| ianw | a lot of upward slanting lines there | 05:56 |
| ianw | maybe reducing usage here would be a good carrot for pushing on purging xenial work | 05:58 |
| *** marios is now known as marios|rover | 05:58 | |
| *** ysandeep|ruck is now known as ysandeep|lunch | 07:34 | |
| *** jpena|off is now known as jpena | 08:49 | |
| *** priteau_ is now known as priteau | 08:58 | |
| *** ysandeep|lunch is now known as ysandeep | 09:38 | |
| noonedeadpunk | hi everyone! | 10:37 |
| noonedeadpunk | I'm quite stuck with investigating on why job does fail in CI, as I can't reproduce it. And nasty thing that I don't really see the way of how to debug it, as ansible module for lxc_container does have `--quiet` flag hardcoded https://github.com/ansible-collections/community.general/blob/main/plugins/modules/lxc_container.py#L826 | 10:38 |
| noonedeadpunk | So any chance to get a hold on the task? | 10:38 |
| noonedeadpunk | s/task/job/ | 10:38 |
| noonedeadpunk | Patch in topic is https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/868176 and job `openstack-ansible-deploy-hosts_lxc_zfs-ubuntu-focal` (will need to recheck it though) | 10:39 |
| frickler | noonedeadpunk: I'll set up a hold in a minute and recheck the job, please let us know your ssh key, too | 11:44 |
| noonedeadpunk | frickler: https://launchpad.net/~noonedeadpunk/+sshkeys | 11:45 |
| frickler | slaweq: there's still a held node for 865822, I assume that it can be removed now? (re-asking from last week) | 11:48 |
| *** rlandy|out is now known as rlandy | 12:09 | |
| *** ysandeep is now known as ysandeep|afk | 12:54 | |
| frickler | noonedeadpunk: seems your job has passed now, let me know if you want to keep retrying or maybe modify your patch, or if I should cancel the hole | 12:56 |
| noonedeadpunk | ugh, that's annoying... Maybe depends on provider... | 12:58 |
| noonedeadpunk | I will try to recheck more | 12:58 |
| jrosser | i am thinking extra-disk / not extra-disk..... | 12:58 |
| noonedeadpunk | Yeah, seems it's super close to truth | 12:59 |
| fungi | what was the symptom? just the task exiting with a nonzero return code? | 13:03 |
| noonedeadpunk | yup | 13:03 |
| noonedeadpunk | for 1 specific scenario (zfs) | 13:03 |
| noonedeadpunk | Or well, the task is run of lxc_container module, that has hardcoded `--quiet` option in their code | 13:04 |
| *** ysandeep|afk is now known as ysandeep | 13:58 | |
| *** ysandeep is now known as ysandeep|ruck | 13:59 | |
| *** dasm|off is now known as dasm | 14:01 | |
| noonedeadpunk | frickler: it failed now :) | 14:38 |
| slaweq | frickler yeah, sure, sorry that I forgot to tell You that | 14:44 |
| frickler | noonedeadpunk: good, I guess? anyway root@104.130.132.119 is now yours | 14:48 |
| *** ysandeep|ruck is now known as ysandeep | 14:48 | |
| frickler | slaweq: np, cleaned up now | 14:48 |
| noonedeadpunk | yep, good enough | 14:48 |
| slaweq | frickler: thx a lot | 14:48 |
| *** ysandeep is now known as ysandeep|ruck | 14:51 | |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: use-buildset-registry: support microk8s https://review.opendev.org/c/zuul/zuul-jobs/+/867063 | 15:09 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Switch ensure-kubernetes role default to cri-o from docker https://review.opendev.org/c/zuul/zuul-jobs/+/869411 | 15:09 |
| noonedeadpunk | frickler: thanks, you can release hold now! That was very helpful | 15:11 |
| fungi | noonedeadpunk: what did the problem wind up being? | 15:28 |
| noonedeadpunk | we have different path on how to create zfs pools when provider has extra drive that should be used for data | 15:29 |
| noonedeadpunk | and we appeared to have different pool names in tasks depending on the path | 15:29 |
| *** yadnesh is now known as yadnesh|away | 15:39 | |
| opendevreview | Clark Boylan proposed openstack/diskimage-builder master: Clean up tox.ini for tox v4 https://review.opendev.org/c/openstack/diskimage-builder/+/869579 | 16:18 |
| *** ysandeep|ruck is now known as ysandeep|dinner | 16:48 | |
| *** marios|rover is now known as marios|out | 16:52 | |
| frickler | noonedeadpunk: done, thx for the feedback | 16:54 |
| opendevreview | Clark Boylan proposed opendev/base-jobs master: Add opendev nox docs promotion https://review.opendev.org/c/opendev/base-jobs/+/869580 | 16:58 |
| clarkb | corvus: ^ something like that? | 16:58 |
| opendevreview | Clark Boylan proposed opendev/base-jobs master: Add opendev nox docs promotion https://review.opendev.org/c/opendev/base-jobs/+/869580 | 17:22 |
| opendevreview | Clark Boylan proposed opendev/base-jobs master: Fix tox.ini for tox v4 https://review.opendev.org/c/opendev/base-jobs/+/869582 | 17:22 |
| jrosser | clarkb: would you be able to help me looking at some connectivity trouble? | 17:23 |
| clarkb | jrosser: sure | 17:24 |
| jrosser | a traceroute/mtr to 132.185.116.1 from opendev.org/38.108.68.66 would be great if you could do one | 17:24 |
| opendevreview | Merged zuul/zuul-jobs master: Remove kubernetes docker jobs https://review.opendev.org/c/zuul/zuul-jobs/+/869410 | 17:33 |
| clarkb | infra-root reviews on 869580 and 869582 appreciated (fixes tox v4 problems in base-jobs then adds nox jobs to help us move away from tox) | 17:44 |
| *** jpena is now known as jpena|off | 17:45 | |
| *** rlandy is now known as rlandy|brb | 17:54 | |
| clarkb | infra-root meeting agenda is coming together. Please take a look and add anything I've missed over the holidays. Or let me know what I've missed and I'll add it | 18:08 |
| *** ysandeep|dinner is now known as ysandeep|out | 18:17 | |
| *** rlandy|brb is now known as rlandy | 18:29 | |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: use-buildset-registry: support microk8s https://review.opendev.org/c/zuul/zuul-jobs/+/867063 | 18:38 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Switch ensure-kubernetes role default to cri-o from docker https://review.opendev.org/c/zuul/zuul-jobs/+/869411 | 18:38 |
| fungi | jrosser: clarkb: did you ever manage to track down the bbc/gitea connectivity problem? | 20:15 |
| jrosser | fungi the trace routes look pretty dimilar from each side but the throughout is/was just 50-100kbit/s | 20:18 |
| jrosser | *similar | 20:18 |
| fungi | odd | 20:18 |
| jrosser | yes, it seems sensitive to routing/peering at this end. picking a geographically similar server that went via cogent there was no issue | 20:20 |
| jrosser | we had someone in osa irc the other day saying their deployment was taking many hours for no apparent reason, and it’s possibly the same thing, the symptom is just huge slowness | 20:21 |
| fungi | jrosser: yes, i used to see that in my old job wrangling backbone providers and massaging bgp prefs, it wasn't at all uncommon for two backbones to get into an argument over who was going to pay to repair or upgrade a peering interconnect, and then you'd get fun packet loss or latency depending on what intermediate routes your traffic followed | 20:24 |
| fungi | could persist for days or months | 20:24 |
| fungi | we'd add all manner of weird conditional pads/prepends to try to route around those situations | 20:26 |
| opendevreview | Merged zuul/zuul-jobs master: use-buildset-registry: add URL to buildset_registry_namespaces https://review.opendev.org/c/zuul/zuul-jobs/+/867280 | 20:34 |
| opendevreview | Merged zuul/zuul-jobs master: ensure-kubernetes: add microk8s support https://review.opendev.org/c/zuul/zuul-jobs/+/866953 | 20:34 |
| opendevreview | Merged zuul/zuul-jobs master: use-buildset-registry: support microk8s https://review.opendev.org/c/zuul/zuul-jobs/+/867063 | 20:34 |
| ianw | ssh_client.ssh('(ifdown eth0 && ifup eth0) || true') | 22:11 |
| ianw | AttributeError: 'NoneType' object has no attribute 'ssh' | 22:11 |
| ianw | this is a new one trying to launch a new nb04 in osuosl | 22:11 |
| *** dasm is now known as dasm|off | 22:20 | |
| Clark[m] | I feel like this is the issue I hit with paramiko not supporting rsa sha2 | 22:20 |
| Clark[m] | It was hard to debug because ssh failed in this weird way but once I updated paramiko to latest it worked | 22:21 |
| ianw | hrm, interesting. this is *after* it has connected as the cloud user on the image, copied the auth bits and now trying to connect back as root | 22:22 |
| Clark[m] | I don't think I ever got that far previously with the rsa sha2 problem. Hrm | 22:23 |
| fungi | could the failure mode be specific to the user keys? i.e. affecting root's but not the cloud user's? | 22:27 |
| ianw | i'm wondering same ... i'm trying again, maybe it was a fluke | 22:28 |
| ianw | ok, not a fluke ... | 22:36 |
| ianw | but i also launched the mirror node in linaro just before xmas, and that didn't do this | 22:37 |
| ianw | "PermitRootLogin no" | 22:47 |
| fungi | hah | 22:47 |
| ianw | i think the osuosl 22.04 base image might have added that | 22:47 |
| fungi | seems likely | 22:48 |
| corvus | Clark: left a comment on https://review.opendev.org/869580 | 22:49 |
| clarkb | corvus: can I set the var as a variant in the new project-template? I guess that would be a better way to express this if so | 22:51 |
| corvus | clarkb: yeah i like that idea | 22:52 |
| corvus | (also, i didn't -1 because i'm not opposed to making a new job with that default, but i do think i have a small preference for trying to avoid that -- basically trying to stick to the original design) | 22:52 |
| clarkb | ok, I'll spin up a new patchset that does this shortly | 22:52 |
| opendevreview | Clark Boylan proposed opendev/base-jobs master: Add opendev nox docs promotion https://review.opendev.org/c/opendev/base-jobs/+/869580 | 23:03 |
| clarkb | corvus: ^ like that? | 23:03 |
| ianw | ... sigh, i've uploaded a generic image to osuosl, but that doesn't seem to connect | 23:19 |
| clarkb | ianw: could also edit the ssh config as the ubuntu user before flipping over | 23:20 |
| ianw | yeah, that's my next option :) | 23:20 |
| ianw | i may have just missed a image property or wrong format, etc. it seems easier to not debug it though | 23:21 |
| corvus | clarkb: ++ | 23:36 |
| clarkb | ok last call for meeting agenda edits. I'll send that out in about 20 minutes | 23:42 |
| opendevreview | Ian Wienand proposed opendev/system-config master: launch: permit ssh root login on base image https://review.opendev.org/c/opendev/system-config/+/869615 | 23:44 |
| ianw | yeah, the other server is stuck in uefi somehow | 23:50 |
| ianw | (the one with the image i uploaded) | 23:50 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!