| *** akrpan-pure has quit IRC | 00:10 | |
| *** yoctozepto1 has joined #opendev | 00:21 | |
| *** yoctozepto has quit IRC | 00:22 | |
| *** yoctozepto1 is now known as yoctozepto | 00:22 | |
| *** tosky has quit IRC | 00:29 | |
| *** artom has quit IRC | 00:31 | |
| *** DSpider has quit IRC | 00:33 | |
| *** yoctozepto4 has joined #opendev | 01:05 | |
| *** yoctozepto has quit IRC | 01:06 | |
| *** yoctozepto4 is now known as yoctozepto | 01:06 | |
| *** yoctozepto5 has joined #opendev | 01:20 | |
| *** yoctozepto has quit IRC | 01:21 | |
| *** yoctozepto5 is now known as yoctozepto | 01:21 | |
| *** yoctozepto5 has joined #opendev | 01:41 | |
| *** yoctozepto has quit IRC | 01:42 | |
| *** yoctozepto5 is now known as yoctozepto | 01:42 | |
| *** d34dh0r53 has quit IRC | 02:59 | |
| *** d34dh0r53 has joined #opendev | 03:01 | |
| *** iurygregory has quit IRC | 03:10 | |
| *** cloudnull has quit IRC | 03:44 | |
| *** cloudnull has joined #opendev | 04:10 | |
| *** sboyron has joined #opendev | 07:11 | |
| *** DSpider has joined #opendev | 09:27 | |
| *** sboyron has quit IRC | 10:36 | |
| *** JayF has quit IRC | 10:58 | |
| *** jhesketh has quit IRC | 11:02 | |
| *** jhesketh has joined #opendev | 11:02 | |
| *** JayF has joined #opendev | 11:02 | |
| *** tosky has joined #opendev | 11:16 | |
| *** danpawlik has quit IRC | 11:33 | |
| *** danpawlik5 has joined #opendev | 11:33 | |
| *** fbo has quit IRC | 12:17 | |
| *** fbo has joined #opendev | 12:19 | |
| openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Revert "Un-pause Gentoo image builds" https://review.opendev.org/c/openstack/project-config/+/771104 | 15:03 |
|---|---|---|
| fungi | prometheanfire: ^ i've tried to include as much diagnostic info as i can in that commit message | 15:04 |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: Temporarily stop running Gentoo base role tests https://review.opendev.org/c/zuul/zuul-jobs/+/771105 | 15:19 |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: Revert "Temporarily stop running Gentoo base role tests" https://review.opendev.org/c/zuul/zuul-jobs/+/771106 | 15:19 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Correct path in mk-archives-index cronjob on lists https://review.opendev.org/c/opendev/system-config/+/771107 | 15:29 |
| *** mlavalle has quit IRC | 15:50 | |
| *** _mlavalle_1 has joined #opendev | 15:50 | |
| *** tosky has quit IRC | 16:16 | |
| openstackgerrit | Jeremy Stanley proposed opendev/bindep master: ArchLinux: ignore unrelated warnings from pacman https://review.opendev.org/c/opendev/bindep/+/771108 | 16:24 |
| *** cgoncalves has quit IRC | 16:58 | |
| *** cgoncalves has joined #opendev | 17:03 | |
| *** cgoncalves has quit IRC | 17:19 | |
| *** cgoncalves has joined #opendev | 17:27 | |
| prometheanfire | fungi: kk | 17:37 |
| prometheanfire | fungi: it seemed like it was still unpacking if there was a lock on distfiles | 17:41 |
| *** brinzhang has joined #opendev | 18:02 | |
| *** brinzhang_ has quit IRC | 18:04 | |
| fungi | prometheanfire: yeah, i wonder if whatever unpacking was going on died and the parent process didn't notice or something... but dmesg didn't indicate an oom or anything of the sort | 18:33 |
| prometheanfire | fungi: is this a single instance of an issue or a repeated issue? | 18:42 |
| fungi | prometheanfire: it happens consistently. image build starts, it gets to installing six, then sticks like that for 4+ hours and finally nodepool gives up | 18:52 |
| fungi | image builds aren't completing | 18:52 |
| prometheanfire | k | 18:54 |
| *** sgw has quit IRC | 18:55 | |
| *** sgw has joined #opendev | 18:56 | |
| *** brinzhang_ has joined #opendev | 19:10 | |
| *** bodgix_ has joined #opendev | 19:12 | |
| *** bodgix has quit IRC | 19:12 | |
| *** fbo has quit IRC | 19:12 | |
| *** dmellado has quit IRC | 19:12 | |
| *** brinzhang has quit IRC | 19:13 | |
| *** fbo has joined #opendev | 19:13 | |
| *** dmellado has joined #opendev | 19:13 | |
| *** slittle1 has joined #opendev | 19:14 | |
| *** tosky has joined #opendev | 19:24 | |
| fungi | prometheanfire: could we maybe add some additional debugging output to the emerge? | 19:49 |
| prometheanfire | heh, there is a --debug option | 20:01 |
| prometheanfire | it's kinda verbose :P | 20:01 |
| prometheanfire | fungi: would this be a good test? https://dpaste.com/5S8JTJDUY | 20:05 |
| prometheanfire | iirc that's what I was running before when I initially developed the stuff | 20:06 |
| prometheanfire | then I can test locally | 20:06 |
| fungi | prometheanfire: probably? i honestly haven't tried reproducing an image build locally for a while | 20:09 |
| prometheanfire | ok, I'll assume it's right | 20:09 |
| fungi | prometheanfire: i'm also wondering if it could be related to the kernel version on our builder... but also nb01 has now ceased to be reachable so i'm going to see what's happened to it | 20:18 |
| prometheanfire | heh | 20:18 |
| *** sgw has left #opendev | 20:32 | |
| fungi | ahh, my bad, i was trying to reach old servers which we never cleaned up in dns | 20:34 |
| fungi | i'll clean that up | 20:35 |
| prometheanfire | I am having an issue (not the same one though) | 20:35 |
| prometheanfire | https://gist.github.com/prometheanfire/42cfd32e92df5f3a474848e414b2191b | 20:36 |
| prometheanfire | I'm thinking that six isn't installed in the base image for me | 20:38 |
| prometheanfire | it happens before I can install anything | 20:38 |
| prometheanfire | project-config/nodepool/elements/openstack-repos/extra-data.d/50-create-repo-list does it | 20:38 |
| fungi | #status log deleted old aaaa records for nonexistent nb01.openstack.org and nb02.openstack.org servers | 20:39 |
| openstackstatus | fungi: finished logging | 20:39 |
| fungi | looks like somebody cleaned up the ipv4 address records but not ipv6 | 20:40 |
| prometheanfire | gonna try wrapping that import in a try/except | 20:42 |
| prometheanfire | yep, that worked | 20:43 |
| prometheanfire | https://dpaste.com/33V38UK3P | 20:44 |
| fungi | prometheanfire: so as far as replicating the issue, if it comes down to it, we're running this in an ubuntu 18.04 lts vm with docker-compose using this compose file: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/nodepool-builder/templates/docker-compose.yaml.j2 | 20:46 |
| fungi | with defaults, so "image: docker.io/zuul/nodepool-builder:latest" | 20:47 |
| prometheanfire | it'll be a minute, for my testing | 20:47 |
| fungi | i'm not sure if any of those details will be involved in the problem | 20:47 |
| prometheanfire | 2021-01-16 20:47:09.165 | Caching gerrit from https://opendev.org/opendev/gerrit.git in /opt/dib_cache/source-repositories/gerrit_0a56dd139195635d3ada2296d9ddf8ce967dea28 | 20:47 |
| fungi | zuul seems to have caught up on its backlog, so maybe i'll restart the scheduler for gerrit wip support after dinner | 20:52 |
| *** calcmandan_ has joined #opendev | 21:00 | |
| *** calcmandan has quit IRC | 21:00 | |
| *** lbragstad has quit IRC | 21:01 | |
| *** smcginnis has joined #opendev | 21:42 | |
| fungi | looks like the static site volumes are releasing on a normal cadence again | 21:48 |
| fungi | there are still outstanding transactions for some of the mirrors though, so we'll need to consider if we want to try to abort them now that we can make rpc calls in a timely fashion again | 21:48 |
| fungi | we should be able to approve the revert for serving static sites from the writeable path at least, if any other config-core wants to review: https://review.opendev.org/770857 | 21:51 |
| fungi | zuul utilization had a bit of a spike around 20:30z so i'll give it a bit longer before i restart the scheduler so i won't need to reenqueue quite so many builds (we have around 150 nodes in use at the moment according to the zuul dashboard in grafana) | 21:53 |
| mnaser | fungi: I think you’re looking for infra-root perhaps :) — I can’t review that :p | 21:53 |
| fungi | mnaser: d'oh, you're right, that's a system-config repo. sorrt! | 21:53 |
| fungi | er, sorry! | 21:54 |
| fungi | but thanks for looking i guess :/ | 21:54 |
| *** tosky has quit IRC | 21:59 | |
| mordred | fungi: lgtm | 22:15 |
| fungi | thanks! | 22:22 |
| *** tosky has joined #opendev | 22:34 | |
| fungi | we're down around 45 nodes in use now... getting ready to restart the scheduler shortly if it drops a bit more | 22:51 |
| fungi | er, 65 i mean | 22:51 |
| openstackgerrit | Merged opendev/system-config master: Revert "Temporarily serve static sites from AFS R+W vols" https://review.opendev.org/c/opendev/system-config/+/770857 | 23:12 |
| * fungi sighs | 23:25 | |
| fungi | another (smaller) spike around 23:00z, so just over 100 nodes in use at the moment | 23:25 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!