| opendevreview | Ian Wienand proposed opendev/grafyaml master: Strip id/uid from .json input https://review.opendev.org/c/opendev/grafyaml/+/833212 | 00:29 |
|---|---|---|
| opendevreview | Ian Wienand proposed openstack/project-config master: grafana: Add infra deployment overview https://review.opendev.org/c/openstack/project-config/+/833213 | 01:01 |
| opendevreview | Ian Wienand proposed openstack/project-config master: grafana: add helper script https://review.opendev.org/c/openstack/project-config/+/833214 | 01:01 |
| opendevreview | Ian Wienand proposed openstack/project-config master: grafana: add helper script https://review.opendev.org/c/openstack/project-config/+/833214 | 01:03 |
| opendevreview | Ian Wienand proposed opendev/grafyaml master: Strip id/uid from .json input https://review.opendev.org/c/opendev/grafyaml/+/833212 | 02:38 |
| opendevreview | Ian Wienand proposed openstack/project-config master: grafana: add helper script https://review.opendev.org/c/openstack/project-config/+/833214 | 02:39 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Handle btrfs root subvolume for fedora extract-image https://review.opendev.org/c/openstack/diskimage-builder/+/833216 | 02:51 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: containerfile: add support for Docker https://review.opendev.org/c/openstack/diskimage-builder/+/832630 | 03:06 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: [dnm] see if this works without podman running as root https://review.opendev.org/c/openstack/diskimage-builder/+/833217 | 03:06 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Update fedora element testing to F35 https://review.opendev.org/c/openstack/diskimage-builder/+/830113 | 03:53 |
| *** ysandeep|out is now known as ysandeep | 05:13 | |
| *** pojadhav- is now known as pojadhav | 05:36 | |
| *** ysandeep is now known as ysandeep|lunch | 07:35 | |
| ykarel | Hi can someone help in finding why zuul is not reporting in stable/yoga patches of neutron | 07:52 |
| ykarel | like https://review.opendev.org/c/openstack/neutron/+/833044 | 07:52 |
| ykarel | in zuul status page i can see the patch appears for few seconds and then disappears | 07:52 |
| ykarel | from zuul logs i think can get what's the issue | 07:53 |
| ykarel | frickler, fungi ^ once you get around | 08:00 |
| *** jpena|off is now known as jpena | 08:13 | |
| frickler | ykarel: infra-root: it seems that zuul did not receive a reconfiguration event from gerrit for the neutron stable/yoga branch. it seems there were some connections issues with gerrit around the time when the branch was created, that may be related. I have no idea how to trigger the reconfiguration manually or how else to fix this | 08:29 |
| frickler | https://paste.opendev.org/show/blAUy5NYueY1T0IwsdOd/ | 08:31 |
| *** ysandeep|lunch is now known as ysandeep | 08:33 | |
| ykarel | Thanks frickler for checking, let's wait for others if they have some idea | 08:35 |
| *** chandankumar is now known as chkumar|rover | 08:42 | |
| *** gibi is now known as gibi_pto | 09:38 | |
| opendevreview | daniel.pawlik proposed opendev/system-config master: DNM Add Opensearch Dashboard information https://review.opendev.org/c/opendev/system-config/+/833264 | 11:09 |
| *** dviroel|out is now known as dviroel|ruck | 11:12 | |
| *** rlandy|out is now known as rlandy|ruck | 11:13 | |
| fungi | frickler: if a branch changes "behind zuul's back" i think it requires a full reconfigure to become aware: https://zuul-ci.org/docs/zuul/latest/operation.html#reconfiguration | 11:55 |
| frickler | fungi: ah, it evens mentions this scenario as example, great. can I do this right now or should we wait for some off-peak time? | 11:57 |
| frickler | the docs also aren't explicit about whether the full-reconfigure would also be done only on one scheduler or all of them | 12:00 |
| fungi | yeah, if it's not urgent we can do it at a time when a 20-minute pause will be less disruptive | 12:03 |
| frickler | lajoskatona: ykarel: ^^ I would do this tomorrow morning then, if you agree | 12:05 |
| *** dviroel is now known as dviroel|ruck | 12:20 | |
| lajoskatona | frickler: sure, thanks | 12:34 |
| lajoskatona | frickler, fungi: perhaps an idea what went wrong to be sure never happen again? | 12:35 |
| ykarel | frickler, ack sounds good | 12:40 |
| fungi | lajoskatona: yes, step 1: fix the internet so that things in texas and montreal never lose packets again | 12:47 |
| fungi | (out gerrit and zuul servers run in different donors' clouds) | 12:48 |
| lajoskatona | fungi: thanks :-) I just checked history, so I see now the begginning of the discussion :-) | 12:48 |
| fungi | s/out/our/ | 12:48 |
| lajoskatona | fungi: ok, I see now, so it was accident / Murphy behind it | 12:50 |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 12:57 | |
| fungi | it's possible with gerrit working on a new event queue system which buffers/stores events server-side, we'll see less impact from disconnects than we do from a stateless socket stream | 13:28 |
| fungi | since zuul would in theory be able to ask for events which occurred while it was out of contact | 13:28 |
| Clark[m] | Yes, https://gerrit.googlesource.com/plugins/events-kafka/ is an example of one such option | 13:33 |
| *** rlandy|ruck|mtg is now known as rlandy|ruck | 14:09 | |
| corvus | no one has run the reconfig yet, right? | 15:22 |
| corvus | i'm going to assume so, and i'm going to run it now. i don't think it's important to wait (we shouldn't miss any events, and i think the pause in launching jobs won't be too long). | 15:25 |
| corvus | the merge jobs are already done, but the scheduler is still reading the results; there's probably an opportunity for optimization there. | 15:42 |
| *** dviroel|ruck is now known as dviroel|ruck|lunch | 15:48 | |
| corvus | the reconfiguration is complete, but it appears it does not invalidate the branch cache, so neutron's yoga branch is still missing. that may get corrected if zuul sees an event on the stable/yoga branch (at which point another full-reconfigure may work). aside from that, i think the only way to correct the issue would be a full shutdown and delete-state. | 15:56 |
| corvus | if there are no open changes, the branch could be deleted and recreated at the same hash; that would probably correct it as well. | 15:57 |
| corvus | i'll work on fixing the actual bug in zuul, but probably won't have that today. | 16:01 |
| fungi | corvus: thanks! and yeah i hadn't tried yet, was out running errands until moments ago | 16:07 |
| fungi | roughly how long did the reconfigure pause queue processing, could you tell? | 16:07 |
| corvus | fungi: almost 20m - your estimate was good :) (maybe 18? we can check the logs to be sure). that was for the openstack tenant; others ran unencumbered during that time. | 16:09 |
| fungi | corvus: 833221 seems to be a dnm stable/yoga change for that project, do you think rechecking it would be the sort of event on the branch you were thinking of? | 16:10 |
| fungi | i've rechecked it just now, in case that helps kick the branch cache | 16:11 |
| corvus | oh hrm, if it already existed then my theory may not work :/ | 16:12 |
| corvus | (though, if was created during the outage, then the theory may still be valid) | 16:12 |
| *** ysandeep is now known as ysandeep|out | 16:15 | |
| frickler | 833221 was created today, the branching happened yesterday around 12UTC. does it help if I create another patch? or abandon/restore the existing one(s)? | 16:35 |
| frickler | if all else fails, recreating the branch also should be feasible. would you do that in gerrit manually or retrigger the release automatism? | 16:39 |
| frickler | can we somehow verify that neutron is the only affected project? | 16:39 |
| fungi | the .gitreview change would have been created immediately after the branch appeared, if we need one created earlier | 16:45 |
| *** dviroel|ruck|lunch is now known as dviroel|ruck | 16:46 | |
| *** marios is now known as marios|out | 16:57 | |
| *** jpena is now known as jpena|off | 17:11 | |
| *** artom__ is now known as artom | 18:10 | |
| *** odyssey4me is now known as odyssey4me|away | 18:16 | |
| *** odyssey4me|away is now known as odyssey4me | 18:16 | |
| frickler | corvus: I'm off now, let me know if I should do anything tomorrow or if you'd rather continue on this yourself | 19:42 |
| *** odyssey4me is now known as odyssey4me|away | 20:09 | |
| *** odyssey4me|away is now known as odyssey4me | 20:09 | |
| *** odyssey4me is now known as odyssey4me|away | 20:24 | |
| opendevreview | Mohammed Naser proposed openstack/project-config master: Add vexxhost/ansible-collection-atmosphere project https://review.opendev.org/c/openstack/project-config/+/833467 | 21:05 |
| mnaser | infra-core: anyone around to push this through to let me do things over the weekend? ^ :) | 21:06 |
| mordred | mnaser: don't hate me for being your own personal yamllint ... but there is an unrelated change in that | 21:17 |
| mnaser | mordred: damnit, did i mess that up | 21:17 |
| mnaser | ugh, auto-save on format | 21:18 |
| mordred | right? stupid autosave | 21:18 |
| *** dviroel|ruck is now known as dviroel|out | 21:19 | |
| opendevreview | Mohammed Naser proposed openstack/project-config master: Add vexxhost/ansible-collection-atmosphere project https://review.opendev.org/c/openstack/project-config/+/833467 | 21:19 |
| mnaser | mordred: ok thanks for catching that | 21:19 |
| fungi | mnaser: lgtm | 21:50 |
| mnaser | fungi: thanks! :) | 21:50 |
| mnaser | fungi: and thanks again =) | 21:55 |
| fungi | my pleasure | 21:58 |
| *** odyssey4me|away is now known as odyssey4me | 22:08 | |
| *** odyssey4me is now known as odyssey4me|away | 22:09 | |
| opendevreview | Merged openstack/project-config master: Add vexxhost/ansible-collection-atmosphere project https://review.opendev.org/c/openstack/project-config/+/833467 | 22:10 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!