| openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: WIP: Ensure Gentoo Repo directory https://review.opendev.org/c/zuul/zuul-jobs/+/782339 | 00:02 |
|---|---|---|
| fungi | prometheanfire: guess we'll see what the jobs report on that ^ | 00:02 |
| *** tosky has quit IRC | 00:09 | |
| *** sshnaidm is now known as sshnaidm|off | 00:12 | |
| prometheanfire | wfm | 00:13 |
| openstackgerrit | Ian Wienand proposed opendev/glean master: Stop requiring /usr/local/bin links for glean.sh https://review.opendev.org/c/opendev/glean/+/782010 | 00:23 |
| openstackgerrit | Ian Wienand proposed opendev/glean master: Create "legacy" script path https://review.opendev.org/c/opendev/glean/+/782016 | 00:23 |
| openstackgerrit | Ian Wienand proposed opendev/glean master: Run a glean-early service to mount configdrive https://review.opendev.org/c/opendev/glean/+/782017 | 00:23 |
| openstackgerrit | Ian Wienand proposed opendev/glean master: Stop requiring /usr/local/bin links for glean.sh https://review.opendev.org/c/opendev/glean/+/782010 | 00:41 |
| openstackgerrit | Ian Wienand proposed opendev/glean master: Create "legacy" script path https://review.opendev.org/c/opendev/glean/+/782016 | 00:41 |
| openstackgerrit | Ian Wienand proposed opendev/glean master: Run a glean-early service to mount configdrive https://review.opendev.org/c/opendev/glean/+/782017 | 00:41 |
| prometheanfire | fungi: it looks like `Ensure Gentoo Repo directory` was not run | 00:43 |
| prometheanfire | maybe the 'when' should be removed | 00:43 |
| fungi | yeah, possibly | 00:53 |
| fungi | prometheanfire: oh! those are handler definitions | 00:54 |
| fungi | we still need to call them | 00:54 |
| *** hamalq has quit IRC | 00:56 | |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: WIP: Ensure Gentoo Repo directory https://review.opendev.org/c/zuul/zuul-jobs/+/782339 | 00:58 |
| fungi | prometheanfire: ^ like that | 00:58 |
| fungi | explains why there were no conditionals on anything else in that file ;) | 00:59 |
| prometheanfire | heh | 01:07 |
| prometheanfire | ya, was confusing | 01:07 |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: WIP: Ensure Gentoo Repo directory https://review.opendev.org/c/zuul/zuul-jobs/+/782339 | 01:23 |
| corvus | fungi, clarkb: btw, quay.io is currently several hours into a user-visible outage | 01:50 |
| ianw | i had to uninstall a webview update on my phone to get things working too | 02:32 |
| ianw | https://www.engadget.com/android-gmail-stop-running-webview-234125352.html | 02:32 |
| ianw | seems it's one of those days! | 02:32 |
| fungi | corvus: good to know! | 02:50 |
| fungi | prometheanfire: looks like your suggested directory creation is running now, but we still get the same error attempting to test install emacs: https://zuul.opendev.org/t/zuul/build/4fc15f19041448208c8c8297606821a4/console | 02:51 |
| prometheanfire | bah, humbug | 02:57 |
| prometheanfire | fungi: is there a way I could inspect a failed image? | 02:59 |
| fungi | yeah, i can set up an autohold for that and recheck it | 02:59 |
| prometheanfire | lrwxrwxrwx 1 root root 67 May 15 2020 /etc/portage/make.profile -> ../../var/db/repos/gentoo/profiles/default/linux/amd64/17.1/systemd | 02:59 |
| prometheanfire | that should be a symlink and that's what it's complaining about | 02:59 |
| prometheanfire | oh, 17.0 systemd, maybe it should be 17.1 | 03:00 |
| prometheanfire | not sure if 17.0 exists in tree anymore | 03:00 |
| fungi | that would make sense | 03:01 |
| prometheanfire | no, it does (as a dev profile nowadays), a hold would help and a ls on that file to know what it's pointing to would help | 03:01 |
| fungi | yeah, working on it | 03:01 |
| prometheanfire | it is marked as depricated though, so changing to 17.1 is a good idea in any case | 03:02 |
| prometheanfire | looks like the profile in project-config is named 17.1 even if the job is still named 'gentoo-17-0-systemd' | 03:03 |
| *** hemanth_n has joined #opendev | 03:07 | |
| prometheanfire | I wonder if `eselect profile set ${the profile defined} right after the sync needs to happen because the ${PORTDIR} moved. It used to be /usr/portage, but that was long ago... | 03:08 |
| openstackgerrit | Ian Wienand proposed opendev/glean master: Stop requiring /usr/local/bin links for glean.sh https://review.opendev.org/c/opendev/glean/+/782010 | 03:17 |
| openstackgerrit | Ian Wienand proposed opendev/glean master: Create "legacy" script path https://review.opendev.org/c/opendev/glean/+/782016 | 03:17 |
| openstackgerrit | Ian Wienand proposed opendev/glean master: Run a glean-early service to mount configdrive https://review.opendev.org/c/opendev/glean/+/782017 | 03:17 |
| openstackgerrit | Ian Wienand proposed opendev/glean master: Cleanup glean.sh variable names https://review.opendev.org/c/opendev/glean/+/782355 | 03:17 |
| fungi | prometheanfire: what ssh public key should i add to this node for you? | 03:22 |
| prometheanfire | https://github.com/prometheanfire.keys | 03:28 |
| fungi | prometheanfire: ssh root@23.253.213.146 | 03:29 |
| prometheanfire | in | 03:30 |
| prometheanfire | it looks like it cleaned up the build? | 03:30 |
| prometheanfire | ah | 03:31 |
| prometheanfire | lrwxrwxrwx 1 root root 67 Mar 22 15:30 make.profile -> ../../tmp/portage-portdir/profiles/default/linux/amd64/17.1/systemd | 03:31 |
| prometheanfire | after running `eselect profile set default/linux/amd64/17.1/systemd` | 03:31 |
| prometheanfire | lrwxrwxrwx 1 root root 67 Mar 23 03:31 make.profile -> ../../var/db/repos/gentoo/profiles/default/linux/amd64/17.1/systemd | 03:32 |
| prometheanfire | and it's blue, not red | 03:32 |
| prometheanfire | not red is good | 03:32 |
| prometheanfire | so, one more step | 03:32 |
| fungi | eselect profile set default/linux/amd64/17.1/systemd | 03:32 |
| fungi | in a handler? | 03:32 |
| prometheanfire | ya | 03:32 |
| prometheanfire | GENTOO_PROFILE is the bash variable | 03:32 |
| prometheanfire | defined in project-config/nodepool/nodepool.yaml | 03:33 |
| prometheanfire | become: yes of course :P | 03:33 |
| prometheanfire | it may be an artifact in the image build process that sets it to a tmpdir because of DIB caching being that location | 03:34 |
| prometheanfire | https://github.com/openstack/diskimage-builder/blob/580256f3742263d23b5e48956835e0f4df2bac8f/diskimage_builder/elements/gentoo/bin/install-packages#L31 | 03:35 |
| prometheanfire | annoying, having to juggle envvars and paths and stuff | 03:36 |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: WIP: Ensure Gentoo Repo directory https://review.opendev.org/c/zuul/zuul-jobs/+/782339 | 03:37 |
| fungi | like that ^ ? | 03:37 |
| prometheanfire | I don't like the hard set profile, because it does update, any way of updating it based on what's set in project-config/nodepool/nodepool.yaml ? | 03:39 |
| fungi | if we write that to disk somewhere in the image build maybe | 03:39 |
| prometheanfire | looks like it's an envvar | 03:40 |
| prometheanfire | {{ lookup('env', 'GENTOO_PROFILE') | default('default/linux/amd64/17.1/systemd', True) }} | 03:42 |
| prometheanfire | that might work | 03:42 |
| fungi | an envvar passed to/in dib though, a dib element would need to write that to a file | 03:42 |
| prometheanfire | hmm | 03:42 |
| fungi | at the point this error is arising, it's ansible launched by zuul on a booted server instance | 03:42 |
| fungi | the nodepool diskimages data is used in constructing the server image which will be booted | 03:43 |
| prometheanfire | ah | 03:43 |
| fungi | i think we could have somewhere in a dib element (might make sense in dib's gentoo element even) that would write that value to some file on disk, like /etc/nodepool/gentoo.profile or whatever | 03:44 |
| fungi | and then in our configure-mirrors role, we could read that file from the image | 03:44 |
| prometheanfire | ya | 03:46 |
| prometheanfire | eselect profile show on the bad symlink doesn't give any useful data, would have been nice | 03:46 |
| prometheanfire | so, for now hard setting it is ok | 03:47 |
| fungi | so is the earlier directory creation possibly not needed then? | 03:48 |
| prometheanfire | possibly | 03:49 |
| prometheanfire | probably | 03:49 |
| prometheanfire | test turnaround time is quick, see if this works first :D | 03:49 |
| fungi | yah | 03:50 |
| prometheanfire | signed out of that box, if you want to kill it | 03:54 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: borg-backup-server: set SHELL for verification script https://review.opendev.org/c/opendev/system-config/+/782357 | 03:55 |
| fungi | yep, reaping now | 03:55 |
| fungi | thanks | 03:55 |
| fungi | sed: can't read /var/db/repos/gentoo/profiles/profiles.desc: No such file or directory | 04:07 |
| fungi | Error: default/linux/amd64/17.1/systemd is not a valid profile for amd64 | 04:07 |
| fungi | anyway, i need to disappear and recharge on low-power standby mode for a few hours, can pick this up tomorrow | 04:08 |
| fungi | it occurs to me that these might also be commands we can call during image building rather than during mirror configuration | 04:10 |
| prometheanfire | fungi: eselect after sync, sorry I missed it | 04:15 |
| prometheanfire | ya, the eselect thing should be fixed first, in finalize.d or cleanup.d LAST thing, unset the envvar and run the eselect | 04:16 |
| prometheanfire | though... the dir will be missing then | 04:16 |
| prometheanfire | iirc the bind mount could not work as a direct thing | 04:16 |
| prometheanfire | (meaning to the default location) | 04:17 |
| *** ykarel has joined #opendev | 04:28 | |
| *** whoami-rajat_ has joined #opendev | 04:57 | |
| openstackgerrit | Ian Wienand proposed opendev/glean master: Run a glean-early service to mount configdrive https://review.opendev.org/c/opendev/glean/+/782017 | 05:12 |
| openstackgerrit | Ian Wienand proposed opendev/glean master: Cleanup glean.sh variable names https://review.opendev.org/c/opendev/glean/+/782355 | 05:12 |
| *** dmsimard9 has joined #opendev | 05:14 | |
| *** dmsimard has quit IRC | 05:14 | |
| *** dmsimard9 is now known as dmsimard | 05:14 | |
| *** ykarel has quit IRC | 05:23 | |
| *** ykarel has joined #opendev | 05:25 | |
| *** ysandeep|away is now known as ysandeep | 05:52 | |
| *** ralonsoh has joined #opendev | 06:26 | |
| openstackgerrit | Ian Wienand proposed opendev/glean master: Run a glean-early service to mount configdrive https://review.opendev.org/c/opendev/glean/+/782017 | 06:50 |
| openstackgerrit | Ian Wienand proposed opendev/glean master: Cleanup glean.sh variable names https://review.opendev.org/c/opendev/glean/+/782355 | 06:50 |
| *** amoralej|off is now known as amoralej | 07:01 | |
| *** lpetrut has joined #opendev | 07:11 | |
| *** moshiur has joined #opendev | 07:11 | |
| *** slaweq_ has joined #opendev | 07:15 | |
| *** marios has joined #opendev | 07:25 | |
| *** eolivare has joined #opendev | 07:37 | |
| *** ysandeep is now known as ysandeep|lunch | 07:46 | |
| *** slaweq_ is now known as slaweq | 07:53 | |
| *** whoami-rajat_ is now known as whoami-rajat | 08:03 | |
| *** sboyron has joined #opendev | 08:04 | |
| *** fressi has joined #opendev | 08:10 | |
| *** rpittau|afk is now known as rpittau | 08:13 | |
| *** andrewbonney has joined #opendev | 08:24 | |
| *** sshnaidm|off has quit IRC | 08:34 | |
| *** hashar has joined #opendev | 08:41 | |
| *** fressi has quit IRC | 08:57 | |
| *** ysandeep|lunch is now known as ysandeep | 08:57 | |
| *** jpenag is now known as jpena | 08:57 | |
| *** ykarel is now known as ykarel|lunch | 08:57 | |
| *** fressi has joined #opendev | 09:03 | |
| *** tosky has joined #opendev | 09:04 | |
| openstackgerrit | Merged opendev/irc-meetings master: Set cinder-bs meeting duration https://review.opendev.org/c/opendev/irc-meetings/+/782264 | 09:04 |
| openstackgerrit | Merged opendev/irc-meetings master: Add Cinder Festival of Reviews meeting https://review.opendev.org/c/opendev/irc-meetings/+/782269 | 09:04 |
| *** zoharm has joined #opendev | 09:21 | |
| *** ykarel|lunch is now known as ykarel | 10:01 | |
| openstackgerrit | Mark Goddard proposed opendev/system-config master: reprepro: add dist for Ubuntu UCA Wallaby https://review.opendev.org/c/opendev/system-config/+/782396 | 10:10 |
| *** sshnaidm has joined #opendev | 10:13 | |
| *** sshnaidm is now known as sshnaidm|off | 10:13 | |
| *** tosky has quit IRC | 10:13 | |
| *** tosky has joined #opendev | 10:14 | |
| *** DSpider has joined #opendev | 10:48 | |
| *** DSpider has quit IRC | 10:48 | |
| *** yoctozepto has quit IRC | 11:02 | |
| *** fressi has quit IRC | 11:03 | |
| *** yoctozepto has joined #opendev | 11:03 | |
| *** artom has joined #opendev | 11:04 | |
| *** elod has quit IRC | 11:13 | |
| *** elod has joined #opendev | 11:21 | |
| *** fressi has joined #opendev | 11:22 | |
| *** lourot has quit IRC | 11:25 | |
| *** lourot has joined #opendev | 11:52 | |
| *** zoharm has quit IRC | 11:55 | |
| *** brinzhang has quit IRC | 11:56 | |
| *** brinzhang has joined #opendev | 11:57 | |
| *** fressi has quit IRC | 11:59 | |
| *** hashar is now known as hasharLunch | 12:09 | |
| *** jpena is now known as jpena|lunch | 12:31 | |
| *** amoralej is now known as amoralej|lunch | 12:32 | |
| *** hemanth_n has quit IRC | 12:37 | |
| *** ykarel has quit IRC | 12:38 | |
| *** ykarel has joined #opendev | 12:38 | |
| *** fressi has joined #opendev | 12:43 | |
| *** ykarel_ has joined #opendev | 12:44 | |
| *** ykarel has quit IRC | 12:47 | |
| *** mgoddard has quit IRC | 12:53 | |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: WIP: Ensure Gentoo Repo directory https://review.opendev.org/c/zuul/zuul-jobs/+/782339 | 12:58 |
| *** hasharLunch is now known as hashar | 13:17 | |
| *** ykarel_ is now known as ykarel | 13:17 | |
| *** ykarel_ has joined #opendev | 13:22 | |
| *** lpetrut has quit IRC | 13:22 | |
| *** ykarel has quit IRC | 13:24 | |
| *** ykarel_ has quit IRC | 13:26 | |
| *** hemanth_n has joined #opendev | 13:29 | |
| *** amoralej|lunch is now known as amoralej | 13:33 | |
| *** jpena|lunch is now known as jpena | 13:34 | |
| *** hemanth_n has quit IRC | 13:39 | |
| openstackgerrit | Sorin Sbârnea proposed opendev/elastic-recheck master: WIP: Run elastic-recheck container https://review.opendev.org/c/opendev/elastic-recheck/+/729623 | 13:44 |
| *** dmsimard has quit IRC | 13:53 | |
| *** dmsimard has joined #opendev | 13:53 | |
| *** mgoddard has joined #opendev | 13:58 | |
| *** mlavalle has joined #opendev | 13:59 | |
| *** fressi has quit IRC | 13:59 | |
| *** fressi has joined #opendev | 14:01 | |
| *** ysandeep is now known as ysandeep|dinner | 14:20 | |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: WIP: Set Gentoo profile in configure-mirrors https://review.opendev.org/c/zuul/zuul-jobs/+/782339 | 14:30 |
| *** dardelean has joined #opendev | 14:46 | |
| kopecmartin | Hi, I've noticed some time ago that gerrit doesn't update LP bugs when a patch is created/merged/abandoned .. is it a known issue or have I missed a change regarding the bot? thanks | 14:48 |
| *** fressi has left #opendev | 14:51 | |
| fungi | kopecmartin: there is no "bot" in this case. there's a gerrit hook script which gerrit calls on specific events, and it needs to be updated to accommodate changes in recent gerrit versions (the old implementation relied on sql queries for looking up user info, which are no longer possible since new gerrit doesn't use an rdbms backend for account data) | 14:54 |
| fungi | probably the most maintainable replacement would be a nodeless zuul job which triggers on relevant change events instead | 14:55 |
| openstackgerrit | Sorin Sbârnea proposed opendev/elastic-recheck master: WIP: Run elastic-recheck container https://review.opendev.org/c/opendev/elastic-recheck/+/729623 | 14:55 |
| kopecmartin | fungi: thanks for the explanation .. do any projects use the zuul job replacement so that i can check how exactly i can configure it in our projects? | 14:58 |
| mordred | kopecmartin: it hasn't been written yet - it's just the most likely replacement implementation | 15:00 |
| openstackgerrit | Jeremy Stanley proposed opendev/jeepyb master: Stop trying to assign Launchpad bugs https://review.opendev.org/c/opendev/jeepyb/+/782538 | 15:01 |
| fungi | kopecmartin: ^ there's a strawman workaround | 15:01 |
| fungi | the rest of the script should still work (i *think*), as long as we give up automatically assigning bugs to change authors | 15:02 |
| fungi | update_blueprint.py is more fundamentally broken though, since it relies on databse queries to find change ids | 15:02 |
| *** whoami-rajat has quit IRC | 15:03 | |
| fungi | kopecmartin: more generally, we're tracking such things here: https://etherpad.opendev.org/p/gerrit-3.2-post-upgrade-notes | 15:17 |
| kopecmartin | fungi: thanks for the info! | 15:19 |
| fungi | yw | 15:19 |
| *** ysandeep|dinner is now known as ysandeep|away | 15:21 | |
| *** lpetrut has joined #opendev | 15:26 | |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Run update-bug on patchset-created again https://review.opendev.org/c/opendev/system-config/+/782540 | 15:28 |
| *** owalsh has quit IRC | 15:35 | |
| *** owalsh has joined #opendev | 15:36 | |
| *** lpetrut has quit IRC | 15:37 | |
| *** marios has quit IRC | 15:37 | |
| *** frickler has quit IRC | 15:37 | |
| *** Eighth_Doctor has quit IRC | 15:37 | |
| *** mordred has quit IRC | 15:37 | |
| *** irclogbot_3 has quit IRC | 15:37 | |
| *** frickler has joined #opendev | 15:37 | |
| *** marios has joined #opendev | 15:38 | |
| *** dirk has quit IRC | 15:38 | |
| *** irclogbot_2 has joined #opendev | 15:38 | |
| *** dirk has joined #opendev | 15:38 | |
| *** irclogbot_2 has quit IRC | 15:51 | |
| *** irclogbot_3 has joined #opendev | 15:52 | |
| *** hashar has quit IRC | 16:01 | |
| *** hashar has joined #opendev | 16:03 | |
| *** amoralej is now known as amoralej|off | 16:04 | |
| *** mordred has joined #opendev | 16:10 | |
| *** Eighth_Doctor has joined #opendev | 16:18 | |
| *** moshiur has quit IRC | 16:25 | |
| *** marios is now known as marios|out | 16:51 | |
| *** cloudnull has quit IRC | 16:54 | |
| *** cloudnull has joined #opendev | 16:54 | |
| *** marios|out has quit IRC | 16:56 | |
| *** iurygregory has quit IRC | 17:02 | |
| *** rpittau is now known as rpittau|afk | 17:26 | |
| *** iurygregory has joined #opendev | 17:29 | |
| *** hashar is now known as hasharAway | 17:48 | |
| *** jpena is now known as jpena|off | 18:05 | |
| *** andrewbonney has quit IRC | 18:12 | |
| *** hamalq has joined #opendev | 18:18 | |
| fungi | prometheanfire: https://review.opendev.org/782339 indicates that just adding the eselect command is sufficient, we didn't need to create the directory. i'm still unclear on your explanation for why we can't do eselect when we build the image and have to call it at job runtime | 18:22 |
| *** eolivare has quit IRC | 18:24 | |
| openstackgerrit | Merged opendev/system-config master: Set up gitea image provides and requires for gating https://review.opendev.org/c/opendev/system-config/+/781830 | 18:29 |
| fungi | infra-root: dmesg for lists.o.o says there was an oom on 2021-03-04 which killed a python process. since i can't tell which one (our mailman lists are coordinated by a fleet of python daemons) i've restarted all the lists' services to make sure there isn't a site with hung queue processing which hasn't noticed yet | 18:33 |
| fungi | #status log Restarted all Mailman queue processing daemons on lists.o.o in order to mitigate any fallout from a 2021-03-04 OOM event | 18:34 |
| openstackstatus | fungi: finished logging | 18:34 |
| fungi | infra-root: also, still a couple of weeks away, but rackspace has opened a ticket to let us know about a network maintenance in dfw which is going to briefly disconnect the cinder volumes for the wiki and listserv on 2021-04-06 | 18:50 |
| fungi | oh, nevermind it's "loss of connectivity to the ServiceNet Network" so might impact database access for the wiki. not storage | 18:53 |
| fungi | they referred to the servers as "affected volumes" (probably by mistake) | 18:53 |
| fungi | hence my confusion | 18:54 |
| fungi | so, yeah, not likely to noticeably impact anything for us. the wiki will recover from temporary service network disruption | 18:54 |
| *** dtantsur is now known as dtantsur|afk | 18:55 | |
| *** dtantsur|afk is now known as dtantsur|afk|afk | 18:55 | |
| *** dtantsur|afk|afk is now known as dtantsur|afk | 18:55 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Revert "Downgrade scheduler to Zuul 4.1.0" https://review.opendev.org/c/opendev/system-config/+/782589 | 19:12 |
| corvus | fungi: ^ scheduler is still in emergency, so should be okay to approve that, then i'll remove emergency, pull, and do a restart | 19:12 |
| fungi | corvus: thanks! and yes, i'm good with a restart. should it wait until after the opendev meeting? | 19:15 |
| corvus | fungi: yep; would be good to approve that now to get it going through the gate tho | 19:16 |
| fungi | on it | 19:16 |
| fungi | approvidado | 19:16 |
| *** sboyron has quit IRC | 19:22 | |
| prometheanfire | fungi: awesome | 19:30 |
| fungi | prometheanfire: also did you get to the bottom of the iscsi/gcc situation which seemed to be impacting image updates? | 19:32 |
| prometheanfire | fungi: ya, threre's a review for it (updating the version) | 19:32 |
| prometheanfire | https://review.opendev.org/781594 | 19:33 |
| fungi | oh, cool. thanks! | 19:33 |
| fungi | ianw: ^ that'll need a dib release after it merges i guess, to get gentoo images building again | 20:00 |
| *** hashar_ has joined #opendev | 20:00 | |
| *** hasharAway has quit IRC | 20:01 | |
| fungi | prometheanfire: i'm still unclear on your explanation for why we can't do eselect when we build the image and have to call it at job runtime instead. does that not persist in a way that the job would find on the booted server? | 20:02 |
| fungi | or is it that the image build already does eselect (or some equivalent) and it's not being seen by the job? | 20:03 |
| *** hashar__ has joined #opendev | 20:03 | |
| prometheanfire | fungi: it has to be run after the portdir variable is unset and while portage is mounted (bind) or sync'd (emerge-webrsync) to the correct directory | 20:05 |
| prometheanfire | the first (change the bind mount location) may be a thing, if only for eselect, webrsync at that phase inflates the image iirc | 20:05 |
| prometheanfire | really, I should just override the symlink, the profile forms part of the path | 20:06 |
| *** hashar_ has quit IRC | 20:06 | |
| fungi | oh, okay | 20:12 |
| fungi | mainly trying to work out how to avoid the duplication between zuul-jobs and project-config (nodepool) | 20:12 |
| prometheanfire | ya, it was a compromise to get binary package builds working | 20:14 |
| prometheanfire | I should take another look | 20:14 |
| fungi | in jeepyb job failure news, it's apparently our old friend cryptography with "ModuleNotFoundError: No module named 'setuptools_rust' | 20:16 |
| fungi | which is from a playbook in gerritlib | 20:19 |
| *** hashar__ has quit IRC | 20:35 | |
| openstackgerrit | Jeremy Stanley proposed opendev/gerritlib master: Run gerritlib-jeepyb-integration on ubuntu-focal https://review.opendev.org/c/opendev/gerritlib/+/782603 | 20:36 |
| fungi | hopefully that's ^ the simplest solution | 20:36 |
| openstackgerrit | Merged opendev/system-config master: Revert "Downgrade scheduler to Zuul 4.1.0" https://review.opendev.org/c/opendev/system-config/+/782589 | 20:50 |
| fungi | nope, not so easy after all | 20:50 |
| fungi | apparently gerrit's not starting on focal | 20:50 |
| fungi | or it's starting but can't be connected to (connection refused on localhost:29418) | 20:53 |
| ianw | fungi: a java8 v java11 type issue? | 20:53 |
| fungi | mebbe | 20:53 |
| fungi | not much to go on in the logs | 20:53 |
| fungi | oh! | 20:54 |
| fungi | overlooked it the first two times i went through | 20:54 |
| fungi | https://zuul.opendev.org/t/openstack/build/1bfc2a9a84fb49acbf36641f335543b6/log/container_logs/gerrit.log#96 | 20:54 |
| fungi | "WARN com.google.gerrit.sshd.SshDaemon : Cannot format SSHD host key [EdDSA]: invalid key type" | 20:54 |
| fungi | seems like it could be that | 20:54 |
| fungi | though that's just a warning, and it does then proceed to eventually log "INFO com.google.gerrit.sshd.SshDaemon : Started Gerrit APACHE-SSHD-2.4.0 on *:29418" | 20:56 |
| *** gothicserpent has quit IRC | 21:12 | |
| fungi | corvus: infra-prod-service-zuul ran in deploy for 782589 roughly 20 minutes ago. anything else we need to do before a restart? | 21:22 |
| fungi | (aside from the usual capturing of queues) | 21:23 |
| fungi | just restarting the zuul-scheduler container, or do we need to do zuul-web too? | 21:24 |
| corvus | in this case, we'll want to remove zuul01 from emergency, update the compose file since it won't have actually run, then run the image pull playbook, then save queues and restart | 21:24 |
| corvus | i can do that now | 21:24 |
| fungi | oh, right, it's still in emergency | 21:24 |
| fungi | just realized that after noticing the compose file on the server still had image: docker.io/zuul/zuul-scheduler:4.1.0 | 21:25 |
| corvus | yeah; ideally we would have removed it at just the right time, but i'm not sure any of us wanted to watch that closely :) | 21:25 |
| fungi | so true | 21:26 |
| corvus | okay, i did the emergency, cleanup, and pull; i think we're ready for a normal restart now | 21:27 |
| corvus | #status log restarted zuul on commit b268f71b233304dbbf2ce59846e47d0575b6b35b with recent scheduler bugfixes | 21:28 |
| openstackstatus | corvus: finished logging | 21:28 |
| fungi | thanks! | 21:30 |
| corvus | re-enqueueing now | 21:33 |
| *** ralonsoh has quit IRC | 21:36 | |
| corvus | there was quite a delay there at the end, but it completed | 21:43 |
| *** slaweq has quit IRC | 21:52 | |
| *** slaweq has joined #opendev | 21:54 | |
| ianw | fungi: re https://review.opendev.org/c/opendev/system-config/+/584035 ... does removing the group from puppet actually remove the mailing list? i'm feeling like no | 22:20 |
| fungi | ianw: it does not, but it's a necessary first step | 22:20 |
| fungi | otherwise if we retire it by deleting the mailing list configuration, it will get recreated by puppet on the next run | 22:21 |
| ianw | thanks, pretty much what i thought | 22:22 |
| fungi | okay, so i held a node for that failing jeepyb integration job | 22:24 |
| fungi | sadly, the ssh api is listening and not refusing connections | 22:25 |
| fungi | The authenticity of host '[localhost]:29418 ([127.0.0.1]:29418)' can't be established. | 22:25 |
| fungi | Permanently added '[localhost]:29418' (ECDSA) to the list of known hosts. | 22:25 |
| fungi | so it's using an ECDSA host key, seems like | 22:25 |
| fungi | which doesn't at all match the error in the job | 22:26 |
| fungi | maybe it's trying to connect too soon and not retrying? | 22:26 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: grafana/afs : add ubuntu-cloud volume tracking https://review.opendev.org/c/openstack/project-config/+/782620 | 22:27 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: grafana/afs : add ubuntu-cloud volume tracking https://review.opendev.org/c/openstack/project-config/+/782620 | 22:28 |
| fungi | [2021-03-23T20:46:09.112Z] [main] INFO com.google.gerrit.sshd.SshDaemon : Started Gerrit APACHE-SSHD-2.4.0 on *:29418 | 22:29 |
| fungi | and then the "fetch ssh host keys from gerrit" task doesn't start until 20:46:11.436354 according to the console log | 22:30 |
| fungi | and tries 5 times between then and 20:46:11.717998 | 22:31 |
| fungi | i suppose it could wait a little longer between retries? | 22:31 |
| fungi | those retries seem to be baked into the ssh-keyscan tool, and manpage doesn't indicate there's a way to control them | 22:34 |
| ianw | fungi: we have some "wait for gerrit" things in the system-config jobs | 22:34 |
| ianw | https://opendev.org/opendev/system-config/src/branch/master/playbooks/test-review.yaml#L4 | 22:34 |
| ianw | is what i'm thinking of | 22:34 |
| fungi | there's one immediately above it in the playbook i'm editing, though it's for the rest api | 22:35 |
| ianw | i guess this is too; so the rest api responds well before ssh? | 22:35 |
| fungi | maybe? | 22:36 |
| openstackgerrit | Jeremy Stanley proposed opendev/gerritlib master: Run gerritlib-jeepyb-integration on ubuntu-focal https://review.opendev.org/c/opendev/gerritlib/+/782603 | 22:37 |
| fungi | we'll see if that doe sit | 22:37 |
| fungi | does it | 22:37 |
| openstackgerrit | Merged opendev/system-config master: borg-backup-server: set SHELL for verification script https://review.opendev.org/c/opendev/system-config/+/782357 | 22:53 |
| *** slaweq has quit IRC | 23:05 | |
| *** slaweq has joined #opendev | 23:06 | |
| fungi | https://zuul.opendev.org/t/openstack/build/25413b8d2a044bd6aa2f765176b9345b/log/job-output.txt#700-714 | 23:10 |
| fungi | doesn't seem to have helped | 23:10 |
| ianw | fungi: there's no gap between those connection refused messages though? is it pausing a bit? | 23:12 |
| fungi | those are the multiple attempts ssh-keyscan seems to perform | 23:12 |
| ianw | although i'd agree, clearly waiting for the REST API isn't helping | 23:13 |
| fungi | i think ansible only includes the output from the last attempt | 23:13 |
| fungi | or maybe that's output from the first? regardless there's a 7 second delay between that logged output and the error | 23:13 |
| ianw | :/ .. no other suggestion than to turn that up to like 5 second gaps i guess | 23:17 |
| fungi | sure, worth a try, though i'm already suspecting this means i'm misinterpreting the tea leaves | 23:25 |
| openstackgerrit | Jeremy Stanley proposed opendev/gerritlib master: Run gerritlib-jeepyb-integration on ubuntu-focal https://review.opendev.org/c/opendev/gerritlib/+/782603 | 23:25 |
| openstackgerrit | Ian Wienand proposed opendev/glean master: Run a glean-early service to mount configdrive https://review.opendev.org/c/opendev/glean/+/782017 | 23:34 |
| openstackgerrit | Ian Wienand proposed opendev/glean master: Cleanup glean.sh variable names https://review.opendev.org/c/opendev/glean/+/782355 | 23:34 |
| *** hamalq has quit IRC | 23:55 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!