| johnsom | FYI, jobs are failing (retry exceeded). I caught a console on one and all it has in it is: | 00:59 |
|---|---|---|
| johnsom | Build ID dbeff18b68834c63bedeaf60ee98a2b3 not found | 00:59 |
| johnsom | --- END OF STREAM --- | 00:59 |
| johnsom | This is the patch I was watching in zuul: https://review.opendev.org/c/openstack/octavia/+/864308 | 01:01 |
| *** dviroel is now known as dviroel|out | 01:57 | |
| fungi | johnsom: is it just octavia jobs? | 02:04 |
| johnsom | No, I saw it on a neutron job too | 02:05 |
| fungi | 2022-11-12 00:33:42,543 ERROR zuul.AnsibleJob: KeyError: 'semaphores' | 02:09 |
| fungi | here's the traceback: https://paste.opendev.org/show/b8bvT5s5FUkOnE3lZiII/ | 02:10 |
| fungi | i wonder if these started appearing after a zuul service restart | 02:11 |
| johnsom | I know the patch I link didn’t change any job config | 02:11 |
| fungi | it does look like the containers have restarted so that's probably our weekly zuul rolling restart | 02:13 |
| fungi | yeah, the executor where i spotted that in the log is ze02, which restarted at 00:33 utc | 02:14 |
| fungi | https://zuul.opendev.org/components shows the restart progress (we do an executor at a time) | 02:14 |
| fungi | corvus: ^ if you're around, any idea why we'd be looking for a semaphores key in the playbook dict? | 02:15 |
| fungi | i wonder if this is a transitive incompatibility which will subside once the schedulers also restart | 02:16 |
| fungi | looks like that probably came in with https://review.opendev.org/862978 on thursday | 02:34 |
| fungi | looking through that change, i have a feeling it will work for any builds which are enqueued after the schedulers restart in a few hours | 02:36 |
| fungi | given the current zuul activity level is fairly low, i'm inclined to check it again in the morning | 02:37 |
| fungi | by which time all the services should hopefully be running from a consistent point in the repo | 02:38 |
| johnsom | Ack, yeah, I am done for the evening. | 02:39 |
| fungi | everything has updated and looks like jobs are working again | 12:16 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!