| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: containerfile: handle errors better https://review.opendev.org/c/openstack/diskimage-builder/+/817139 | 00:01 |
|---|---|---|
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Revert "centos 9-stream: make non-voting for mirror issues" https://review.opendev.org/c/openstack/diskimage-builder/+/817313 | 00:01 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: containerfile: fix tar extraction https://review.opendev.org/c/openstack/diskimage-builder/+/817317 | 00:01 |
| *** mazzy5098811 is now known as mazzy509881 | 00:10 | |
| opendevreview | Ian Wienand proposed openstack/project-config master: Pause Fedora 34 builds https://review.opendev.org/c/openstack/project-config/+/817318 | 00:11 |
| clarkb | I went ahead and fast approved ^ since I had looked over the related work | 00:12 |
| *** mazzy5098812 is now known as mazzy509881 | 00:18 | |
| clarkb | rosmaita: jrosser_: note I left some comments on your changes unrelated to the zuul trouble | 00:24 |
| opendevreview | Merged openstack/project-config master: Pause Fedora 34 builds https://review.opendev.org/c/openstack/project-config/+/817318 | 00:24 |
| *** mazzy5098814 is now known as mazzy509881 | 00:42 | |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: centos 9-stream: make non-voting for mirror issues https://review.opendev.org/c/openstack/diskimage-builder/+/817312 | 00:44 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: containerfile: fix tar extraction https://review.opendev.org/c/openstack/diskimage-builder/+/817317 | 00:44 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: containerfile: handle errors better https://review.opendev.org/c/openstack/diskimage-builder/+/817139 | 00:44 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Revert "centos 9-stream: make non-voting for mirror issues" https://review.opendev.org/c/openstack/diskimage-builder/+/817313 | 00:44 |
| corvus | clarkb: i think we don't call loadTPCs in the layout update path | 00:58 |
| corvus | i may need to include an event with branch cache ltimes in my test case | 01:00 |
| clarkb | I would agree that scheduler.py seems to loadTPCs when validating, priming and reconfiguring | 01:01 |
| clarkb | but those are the only instances of the loadTPCs calls | 01:02 |
| corvus | clarkb: and i think the cacheConfig call you were looking at is called once every time, but it's caching things onto the tpcs, so if we're reusing them when we expect to have empty ones... | 01:03 |
| corvus | still can't repro locally though | 01:04 |
| corvus | i think i reproduced it | 01:19 |
| Clark[m] | Yay | 01:20 |
| corvus | throwing in a loadTPCs call does fix it. now, i need to try to clean this test up; it's a mess. | 01:28 |
| corvus | (it's just a big pile of changes, reconfigurations, and sleeps that i threw at it until it broke) | 01:28 |
| Clark[m] | Kitchen sink debugging | 01:40 |
| *** diablo_rojo is now known as Guest5475 | 01:59 | |
| opendevreview | Takashi Kajinami proposed openstack/project-config master: Retire puppet-senlin - Step 1: End project Gating https://review.opendev.org/c/openstack/project-config/+/817324 | 02:14 |
| opendevreview | Takashi Kajinami proposed openstack/project-config master: Retire puppet-senlin - Step 3: Remove Project https://review.opendev.org/c/openstack/project-config/+/817327 | 02:20 |
| opendevreview | chandan kumar proposed opendev/system-config master: Enable mirroring of centos stream 9 contents https://review.opendev.org/c/opendev/system-config/+/817136 | 03:31 |
| ianw | https://review.opendev.org/c/openstack/diskimage-builder/+/817312/2 just failed with "Nodeset ubuntu-bionic-2-node already defined" | 04:01 |
| Clark[m] | ianw: the fix for that is corvus' most recent change pushed to zuul | 04:03 |
| Clark[m] | Hopefully we can restart with that fix tomorrow once people review it | 04:03 |
| ianw | thanks, i figured that one, wasn't sure if we'd seen it on other changes. probably a good sign to walk away :) | 04:05 |
| opendevreview | Ian Wienand proposed openstack/project-config master: Set debian-stretch to min-ready: 0 https://review.opendev.org/c/openstack/project-config/+/817338 | 04:11 |
| opendevreview | Ian Wienand proposed openstack/project-config master: Remove debian-stretch nodes and builds https://review.opendev.org/c/openstack/project-config/+/817339 | 04:11 |
| opendevreview | Ian Wienand proposed opendev/system-config master: reprepro: stop mirroring Debian stretch https://review.opendev.org/c/opendev/system-config/+/817340 | 04:12 |
| *** ysandeep|out is now known as ysandeep | 05:33 | |
| akahat|rover | hello | 08:38 |
| akahat|rover | on zuul.opendev.org/ queue: tripleo is stuck | 08:39 |
| akahat|rover | we can see there are some jobs which are pending since 11 hrs | 08:39 |
| akahat|rover | some jobs in queue * | 08:39 |
| *** ysandeep is now known as ysandeep|lunch | 08:41 | |
| soniya29|ruck | tripleo-ci-centos-8-containers-multinode and tripleo-ci-centos-8-standalone-upgrade-victoria are few of them | 08:41 |
| soniya29|ruck | here is the console log:- https://zuul.openstack.org/stream/e3488a9056d2422983bbdc140b6f487e?logfile=console.log | 08:43 |
| *** akahat|rover is now known as akahat|lunch | 08:44 | |
| *** ykarel is now known as ykarel|lunch | 08:51 | |
| *** akahat|lunch is now known as akahat|rover | 09:13 | |
| *** ysandeep|lunch is now known as ysandeep | 09:23 | |
| frickler | corvus: I'm seeing an empty "Queue:" header for every patch in check and other pipelines, is this a known issue? for gate, "Queue: internal" etc. looks o.k. | 09:33 |
| frickler | also that "nodeset already defined" seems to be happening quite often, hopefully we can get that fixed soon, otherwise we should maybe revert to one node until we can | 09:35 |
| frickler | akahat|rover: soniya29|ruck: I don't see anything being stuck, just a lot of patches plus gate resets due to failures | 09:36 |
| akahat|rover | frenzy_friday, https://zuul.opendev.org/t/openstack/status#817233, jobs: tripleo-ci-centos-8-containers-multinode, tripleo-ci-centos-8-standalone-upgrade-victoria | 09:42 |
| akahat|rover | frickler, ^^ | 09:42 |
| akahat|rover | it is running for 12 hr 37 mins | 09:43 |
| akahat|rover | 817106, 817260 this ids jobs already ran.. but they still are in queue. | 09:46 |
| frickler | hmm, the console logs for all the jobs that are still being shown as in progress in the ui for those jobs show "build id not found" | 09:53 |
| frickler | so something is indeed broken, maybe you want to abandon/restore the affected patches, otherwise we'll need to wait for corvus | 09:55 |
| *** ykarel|lunch is now known as ykarel | 09:57 | |
| akahat|rover | frickler, okay we will wait for corvus. | 09:58 |
| *** melwitt is now known as Guest5508 | 10:12 | |
| ysandeep | folks o/ https://zuul.openstack.org/status#heat Some of the jobs are waiting for too long for a node , "Build ID 8b3c99ddb0d94a999def904873717e1d not found" Do we have a known issue? | 11:06 |
| *** dviroel|out is now known as dviroel | 11:16 | |
| opendevreview | Tristan Cacqueray proposed opendev/statusbot master: Add Etherpad backend https://review.opendev.org/c/opendev/statusbot/+/807946 | 11:55 |
| *** soniya29|ruck is now known as soniya29|ruck|afk | 12:23 | |
| Alex_Gaynor | Jobs on pyca/cryptography don't appear to be starting, on https://zuul.opendev.org/t/pyca/status/ I see our queue constantly hanging around at 8. Known issue? | 12:29 |
| frickler | Alex_Gaynor: we have some known issues currently, but I'm not sure whether this is related or now. clarkb, corvus ^^ | 12:30 |
| frickler | s/now/not/ | 12:30 |
| *** ysandeep is now known as ysandeep|afk | 12:32 | |
| *** ysandeep|afk is now known as ysandeep | 13:08 | |
| *** jpena|off is now known as jpena | 13:21 | |
| *** soniya29|ruck|afk is now known as soniya29|ruck | 13:33 | |
| noonedeadpunk | well and jobs are not scheduled for gates for us as well | 13:37 |
| noonedeadpunk | and all gate jobs looks like stuck ones atm | 13:38 |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 14:01 | |
| corvus | i'm looking into the error causing the stuck queues | 14:15 |
| tristanC | the cacti graphs for zookeeper seems correct, though there is a suspicious spike about 9h ago on the network page | 14:17 |
| fungi | frickler: there's already a fix for the "nodeset already defined" problem, hopefully the new patchset makes it in shortly ( https://review.opendev.org/817328 ) | 14:22 |
| fungi | tristanC: openstack periodic and periodic-stable pipelines trigger their jobs at 02:00-02:01 utc, which seems to line up with the start of the burst at http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=70044&rra_id=all | 14:24 |
| corvus | i'm trying to save the debug info i need, but zk-shell isn't helping; i need to write a quick script | 14:41 |
| corvus | okay, i've saved a copy of the zk data, and i think we should restart now. probably the best thing to do is restart on .4, then we'll switch back once the in-flight changes have landed. sound good? | 14:50 |
| corvus | i'm going with that | 14:52 |
| corvus | stopped; deleting state now | 14:53 |
| tristanC | corvus: that sounds good. so you now have a dump of all the zk nodes? | 14:53 |
| fungi | thanks corvus, i agree | 14:54 |
| fungi | i expect we'll lose anything in the event queue with the downgrade to .4, as our saved change queues are all that will get reenqueued | 14:55 |
| corvus | tristanC: yes | 14:57 |
| corvus | starting now | 14:57 |
| corvus | here's my dump script: https://paste.opendev.org/show/810906/ | 14:58 |
| fungi | oh, that's handy | 14:59 |
| corvus | there was a deserialization error, and i'll need to look at the data to figure out what it was. but i also don't know the path, so i grabbed the whole system. | 15:01 |
| corvus | it's something wrong with the config_errors patch i wrote :( | 15:01 |
| corvus | anyway, i don't think it'll be too hard to fix once i find the actual node with the error :) | 15:01 |
| *** ykarel is now known as ykarel|away | 15:12 | |
| corvus | re-enqueing | 15:27 |
| *** soniya29|ruck is now known as soniya29|ruck|dinner | 15:38 | |
| outbrito_ | Do I have to remove +W and re-add to get something back on the gate queue? | 15:57 |
| outbrito_ | (btw, sorry for the newbie question) | 15:57 |
| *** akahat|rover is now known as akahat|lunch | 16:03 | |
| *** akahat|lunch is now known as akahat|dinner | 16:03 | |
| fungi | outbrito_: it depends on which tenant that's in. some tenants will send a rechecked change straight to the gate pipeline if they have sufficient approval votes, others will require a positive vote from the check pipeline and addition of a new workflow +1 | 16:08 |
| corvus | outbrito_: i re-enqueued all the changes that were there before; check the status page now, and if i missed yours, go ahead and reapprove or recheck | 16:08 |
| fungi | i think it's just the openstack tenant which will require a positive check vote before entering the gate pipeline, but also yes make sure it wasn't already reenqueued | 16:09 |
| fungi | it seems like there were a few events the schedulers didn't process, so never got enqueued | 16:09 |
| *** soniya29|ruck|dinner is now known as soniya29|ruck | 16:21 | |
| *** ysandeep is now known as ysandeep|out | 16:51 | |
| *** akahat|dinner is now known as akahat|rover | 16:52 | |
| *** soniya29|ruck is now known as soniya29|ruck|out | 17:01 | |
| *** marios is now known as marios|out | 17:03 | |
| *** rlandy|ruck|mtg is now known as rlandy|ruck | 17:09 | |
| opendevreview | Jeremy Stanley proposed opendev/statusbot master: Add use_ssl option https://review.opendev.org/c/opendev/statusbot/+/807947 | 17:10 |
| opendevreview | Jeremy Stanley proposed opendev/statusbot master: Handle exception for unprivileged commands https://review.opendev.org/c/opendev/statusbot/+/807948 | 17:11 |
| outbrito_ | Yeah, mine was one of them. Just +W again to re-enqueue. Tks (btw, it was on starlingx/openstack-armada-app, openstack tenant) | 17:18 |
| outbrito_ | thanks fungi corvus | 17:18 |
| *** jpena is now known as jpena|off | 17:34 | |
| *** rlandy is now known as rlandy|ruck | 17:40 | |
| opendevreview | Tristan Cacqueray proposed opendev/statusbot master: Introduce a BackendInterface https://review.opendev.org/c/opendev/statusbot/+/807871 | 19:27 |
| opendevreview | Tristan Cacqueray proposed opendev/statusbot master: Add Etherpad backend https://review.opendev.org/c/opendev/statusbot/+/807946 | 19:27 |
| opendevreview | Jeremy Stanley proposed opendev/statusbot master: Add use_ssl option https://review.opendev.org/c/opendev/statusbot/+/807947 | 19:59 |
| opendevreview | Jeremy Stanley proposed opendev/statusbot master: Handle exception for unprivileged commands https://review.opendev.org/c/opendev/statusbot/+/807948 | 19:59 |
| ianw | fungi/clarkb: https://review.opendev.org/c/opendev/system-config/+/817136 has some numbers on the 9-stream mirror, and including source/debug. layout seems different to the way it was done before. thoughts welcome either way | 20:41 |
| fungi | what, red hat changed things in a new release? ;) | 20:42 |
| fungi | and thanks, i've been trying to get around to taking a look at that one | 20:42 |
| *** dviroel is now known as dviroel|out | 21:15 | |
| opendevreview | Merged openstack/diskimage-builder master: centos 9-stream: make non-voting for mirror issues https://review.opendev.org/c/openstack/diskimage-builder/+/817312 | 21:51 |
| opendevreview | Merged openstack/diskimage-builder master: containerfile: fix tar extraction https://review.opendev.org/c/openstack/diskimage-builder/+/817317 | 21:51 |
| opendevreview | Merged openstack/diskimage-builder master: containerfile: handle errors better https://review.opendev.org/c/openstack/diskimage-builder/+/817139 | 21:56 |
| opendevreview | Marco Vaschetto proposed openstack/diskimage-builder master: Allowing ubuntu element use local image https://review.opendev.org/c/openstack/diskimage-builder/+/817481 | 22:11 |
| opendevreview | Merged opendev/statusbot master: Introduce a BackendInterface https://review.opendev.org/c/opendev/statusbot/+/807871 | 22:17 |
| opendevreview | Merged opendev/statusbot master: Add Etherpad backend https://review.opendev.org/c/opendev/statusbot/+/807946 | 22:17 |
| opendevreview | Merged opendev/statusbot master: Add use_ssl option https://review.opendev.org/c/opendev/statusbot/+/807947 | 22:22 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!