| *** rlandy|bbl is now known as rlandy|out | 00:55 | |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: add-gpgkey: trust incoming key https://review.opendev.org/c/zuul/zuul-jobs/+/840566 | 03:42 |
|---|---|---|
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: add-gpgkey: trust incoming key https://review.opendev.org/c/zuul/zuul-jobs/+/840566 | 04:05 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: add-gpgkey: trust incoming key https://review.opendev.org/c/zuul/zuul-jobs/+/840566 | 04:40 |
| *** ysandeep|out is now known as ysandeep | 05:01 | |
| *** pojadhav- is now known as pojadhav | 05:07 | |
| *** marios is now known as marios|ruck | 05:15 | |
| *** jpena|off is now known as jpena | 07:39 | |
| *** ykarel_ is now known as ykarel | 07:48 | |
| *** ysandeep is now known as ysandeep|lunch | 08:07 | |
| *** pojadhav is now known as pojadhav|lunch | 09:01 | |
| opendevreview | Ian Wienand proposed openstack/project-config master: Add opendev/infra-openafs-deb https://review.opendev.org/c/openstack/project-config/+/840594 | 09:13 |
| opendevreview | Ian Wienand proposed openstack/project-config master: Add opendev/infra-openafs-deb https://review.opendev.org/c/openstack/project-config/+/840594 | 09:25 |
| *** pojadhav|lunch is now known as pojadhav|afk | 09:27 | |
| ianw | fungi: ^ i completely admit your were right :) i'm thinking that if we keep xenial -> jammy branches in a separate project, what we've already committed can be reworked to a generic job to upload to the PPA | 09:29 |
| *** ysandeep|lunch is now known as ysandeep | 09:36 | |
| *** rlandy|out is now known as rlandy | 10:25 | |
| *** pojadhav|afk is now known as pojadhav | 10:36 | |
| *** dviroel|out is now known as dviroel | 11:21 | |
| *** ysandeep is now known as ysandeep|afk | 11:46 | |
| fungi | i'm not right all that often, so i'll take it! ;) | 12:25 |
| fungi | and yeah, ultimately this is closer to how the debian packaging is being done anyway (per the README.source file) | 12:26 |
| *** soniya29 is now known as soniya29|afk | 12:27 | |
| frickler | another nice thing about phased updates: it seems reprepro only mirrors the update initially when it contains "Phased-Update-Percentage: 10" and doesn't update when this value gets updated | 12:27 |
| frickler | our mirror is still at 10% for systemd while upstream has reached 70% | 12:28 |
| frickler | but the devstack jammy job worked fine after yesterdays patches | 12:29 |
| *** ysandeep|afk is now known as ysandeep | 12:29 | |
| fungi | interesting. i wonder if newer reprepro behaves differently | 12:32 |
| frickler | seem no: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=984690 | 12:35 |
| TheJulia | anyone around who can hold nodes from a CI job so I can investigate the lost mysteries of why we can't seem to ssh from one node to another ? | 13:21 |
| frickler | TheJulia: sure, patch id and job name? | 13:41 |
| fungi | mysteries of the (ci) universe | 13:41 |
| TheJulia | heh | 13:42 |
| TheJulia | https://review.opendev.org/c/openstack/ironic/+/840530 840530,3 ironic-grenade-multinode-multitenant <-- as requested. I just kicked the job with recheck but it is waiting at the moment. Likely should start in a minute or three and fail in ~1.5 hours | 13:44 |
| frickler | TheJulia: ok, hold is set. it is only evaluated at the moment the job is failing, so that should be fine. but you could let us know our ssh key beforehand already | 13:51 |
| TheJulia | frickler: https://paste.openstack.org/show/bHHmaYhfkP2YOm1qCy1q/ | 13:53 |
| *** soniya is now known as soniya|afk | 14:15 | |
| *** soniya|afk is now known as soniya | 15:11 | |
| opendevreview | Sandeep Yadav proposed openstack/diskimage-builder master: Revert "Set machine-id to uninitialized to trigger first boot" https://review.opendev.org/c/openstack/diskimage-builder/+/840486 | 15:17 |
| clarkb | infra-root looks like the 0600 backup on lists.o.o has failed today and yesterday. But the other backup isn't complaining? | 15:28 |
| clarkb | has anyone looked at that yet? | 15:28 |
| fungi | i have not yet, no | 15:28 |
| clarkb | we have an arm64 jammy image now so I've approved https://review.opendev.org/c/openstack/project-config/+/840394 | 15:31 |
| clarkb | that change adds the nodes. Then once node config is in place we can land the nodeset addition | 15:31 |
| frickler | TheJulia: root@198.72.124.16 + 198.72.124.11 | 15:32 |
| clarkb | frickler: re reprepro I guess even more reason to disable use of phased updates on our side | 15:32 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Borg ignore ansible tmp files https://review.opendev.org/c/opendev/system-config/+/840690 | 15:38 |
| clarkb | I think ^ is the fix for the lists backup failures. It is a race against ansible updating tmp files and trying to back them up. We should be able to safely ignore those files | 15:38 |
| fungi | oh, fun! | 15:40 |
| opendevreview | Merged openstack/project-config master: Add Jammy arm64 test nodes https://review.opendev.org/c/openstack/project-config/+/840394 | 15:41 |
| clarkb | the log files on the host for the vexxhost backup target show the error if anyone wants to dobule check it | 15:42 |
| frickler | clarkb: ack | 15:42 |
| TheJulia | frickler: thanks! | 15:43 |
| *** dviroel is now known as dviroel|lunch | 15:53 | |
| *** marios|ruck is now known as marios|out | 16:01 | |
| TheJulia | frickler: I've pushed a 4th revision of that patch. The prior nodes can be reclaimed, and if the new nodes can be held for the job that is launching now, that would be appreciated | 16:20 |
| TheJulia | tl;dr change accidently squashed system state | 16:20 |
| *** ysandeep is now known as ysandeep|out | 16:28 | |
| clarkb | ok I think the old hold is still there with a max count of 1 so it won't catch the new ps. I'll delete it then create a new one | 16:37 |
| clarkb | TheJulia: frickler ^ fyi since I think frickler's day is ending now | 16:37 |
| opendevreview | Pierre Riteau proposed opendev/irc-meetings master: Bring Blazar meeting one hour forward https://review.opendev.org/c/opendev/irc-meetings/+/840708 | 16:42 |
| opendevreview | Pierre Riteau proposed opendev/irc-meetings master: Schedule Blazar meeting one hour earlier https://review.opendev.org/c/opendev/irc-meetings/+/840708 | 16:43 |
| TheJulia | clarkb: thanks | 16:54 |
| *** dviroel|lunch is now known as dviroel | 17:01 | |
| clarkb | corvus: I'm looking at that nl02 error and I think I may understand it. There is a node in inmotion cloud that failed to delete. Nodepool must be treating it as a leaked node because it has a record of it in zk without any information like node.type (which generates the error). I think that means it generated a simple record for it to clean up the leak | 17:10 |
| clarkb | The correct way to fix this is to delete the instance from the cloud. I have no idea what that will entail right now. On the nodepool side I wonder if we may need to add some messaging around those errors indicating that leaked nodes may not have sufficient info to calculate quota from and that generates the error? | 17:11 |
| corvus | clarkb: perhaps -- but at this point every other driver handles leaked nodes differently, so that would be an openstack-driver-specific issue. just something to keep in mind. | 17:21 |
| corvus | (ie, i think openstack may be the only one that creates bogus nodes) | 17:22 |
| clarkb | how do the others handle it? Just attempt to delete it directly and try again later via a listing against the cloud to make the leaked list? | 17:22 |
| clarkb | anyway I'm not too worried about it currently as the problem seems straightforward and nodepool should continue to handle this just fine (it may get more quota errors than usual though) | 17:23 |
| *** jpena is now known as jpena|off | 17:24 | |
| corvus | clarkb: yep exactly | 17:43 |
| corvus | hopefully we can move the openstack driver to that system at some point | 17:43 |
| clarkb | ++ | 17:44 |
| clarkb | I'm manually booting a jammy arm64 instance now and if it comes up and is ssh'able I'll approve the nodeset change | 17:50 |
| clarkb | it boots! | 17:51 |
| *** artom__ is now known as artom | 17:52 | |
| opendevreview | Merged opendev/base-jobs master: Add Jammy Arm64 nodeset https://review.opendev.org/c/opendev/base-jobs/+/840538 | 18:00 |
| *** dviroel is now known as dviroel|out | 20:43 | |
| ianw | linaro should be back now, i'll double check on that and do a revert of the disable | 21:31 |
| clarkb | ianw: kevinz: did the cert get updated too? I think it expires soon | 21:31 |
| ianw | notBefore=May 5 12:48:39 2022 GMT | 21:34 |
| ianw | notAfter=Aug 3 12:48:38 2022 GMT | 21:34 |
| ianw | looks like yes | 21:34 |
| fungi | lgtm | 22:04 |
| fungi | must have been replaced today once he got back from holidays | 22:04 |
| *** rlandy is now known as rlandy|bbl | 22:36 | |
| opendevreview | Merged openstack/project-config master: Add opendev/infra-openafs-deb https://review.opendev.org/c/openstack/project-config/+/840594 | 22:37 |
| clarkb | TheJulia: I realize that I've spaced and not given you access to those nodes you wanted held. One second | 22:47 |
| clarkb | TheJulia: root @ 172.99.75.148 and 172.99.75.136 | 22:49 |
| clarkb | ianw: https://review.opendev.org/c/opendev/system-config/+/840690 is probably interesting to you (noticed that when debugging the lists.o.o backup failures and this should fix it) | 23:06 |
| ianw | ++ thanks, good find | 23:07 |
| opendevreview | Merged opendev/system-config master: Borg ignore ansible tmp files https://review.opendev.org/c/opendev/system-config/+/840690 | 23:21 |
| opendevreview | Ian Wienand proposed opendev/infra-openafs-deb master: [wip] make jammy packages https://review.opendev.org/c/opendev/infra-openafs-deb/+/840789 | 23:46 |
| TheJulia | clarkb: thanks, you guys can reclaim those machines now | 23:54 |
| TheJulia | confirmed I wasn't (too) insane too | 23:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!