| corvus | i'd like to rolling-restart zuul again. :) | 00:29 |
|---|---|---|
| corvus | (this is so cool -- we're like converging on CD here :) | 00:29 |
| corvus | i'm going to restart the scheduler on zuul01 now | 00:30 |
| clarkb | sounds good. I'm around for a bit longer too | 00:30 |
| corvus | problem preventing 01 from starting; looking into it | 00:32 |
| corvus | you will be amused | 00:33 |
| clarkb | It can't find the api model version for the other components? | 00:35 |
| corvus | sql/database | 00:35 |
| corvus | i could have sworn we landed that change | 00:35 |
| clarkb | oh! | 00:35 |
| clarkb | at least that is an easy fix | 00:35 |
| corvus | easy but a surprising amount of typing.... | 00:38 |
| opendevreview | James E. Blair proposed opendev/system-config master: Move Zuul SQL connection to "database" https://review.opendev.org/c/opendev/system-config/+/826790 | 00:47 |
| corvus | infra-root: ^ our zuul config is broken and needs ^ before we can (re-) start any components | 00:47 |
| ianw | fixing that seems useful... | 00:49 |
| clarkb | corvus: for https://review.opendev.org/c/opendev/system-config/+/826790/1/inventory/service/group_vars/zuul.yaml did we merge in private vars over the top of that somehow? | 00:50 |
| clarkb | Just noting it doesn't ahve a uri or user/passwd info | 00:50 |
| clarkb | (wonderinf if we need to clean anything else up) | 00:50 |
| corvus | clarkb: yes, zuul_connection_secrets -- it is an empty list in system-config so no visible change. | 00:51 |
| clarkb | gotcha | 00:51 |
| corvus | i also have not removed the entry from that in secret hostvars, so as not to break the current system, but we can drop it there after that merges | 00:51 |
| opendevreview | James E. Blair proposed opendev/system-config master: Remove gearman from Zuul https://review.opendev.org/c/opendev/system-config/+/826791 | 00:55 |
| corvus | low-priority followup ^ | 00:55 |
| corvus | infra-root: any objection to me throwing 826790 straight into gate? | 00:56 |
| ianw | no, please have it merged in case I ever need to restart it! :) | 00:58 |
| fungi | wfm | 00:58 |
| clarkb | haha ya no objection from me | 00:58 |
| corvus | enqueued | 00:59 |
| corvus | i'll be back in ~20m | 01:00 |
| clarkb | the system-config-run-review jobs for fungi's gitea testing change show as failed and clicking on the link says the build doesn't exist. I wonder if zuul01 tried to process stuff despite not having a proper db config? Its not a big deal for those changes but calling it out here in case that is something to look into closer | 01:13 |
| clarkb | I need to go help wit hdinner now though | 01:13 |
| corvus | clarkb: it never started | 01:23 |
| corvus | the build uuid from 3.4 cb2813b2a2704273920fba7ac310f936 doesn't show up in any zuul component logs | 01:34 |
| corvus | clarkb: fungi it's a bit hard to tell from the logs, but i suspect something related to container images; like it may not have found the required artifacts or something. | 01:38 |
| corvus | 2022-01-28 01:00:46,727 INFO zuul.QueueItem: [e: f8775b510faa4cefbc3c0d149cb3e566] Job system-config-run-review-3.4 requires artifact(s) gerrit-3.4-container-image provided by build 7a2820c9b3934619a761a7a5092e0f5a (triggered by change 825337 on project opendev/system-config), but that build failed with result "FAILURE" | 01:39 |
| corvus | clarkbfungi ^ yeah that's it. the UI is misleading because that build doesn't exist, but it doesn't represent an operational problem. | 01:41 |
| corvus | i think that will get reported in the message to gerrit | 01:44 |
| opendevreview | Merged opendev/system-config master: Move Zuul SQL connection to "database" https://review.opendev.org/c/opendev/system-config/+/826790 | 01:44 |
| corvus | waiting on deployment of that now | 01:48 |
| *** rlandy|ruck|bbl is now known as rlandy|ruck | 02:02 | |
| Clark[m] | corvus: aha I guess maybe we need to report skipped or something along those lines to reduce confusion | 02:03 |
| *** ysandeep|out is now known as ysandeep | 02:07 | |
| opendevreview | Merged opendev/system-config master: Rebuild Gerrit images particularly for 3.4 https://review.opendev.org/c/opendev/system-config/+/826761 | 02:10 |
| *** rlandy|ruck is now known as rlandy|out | 02:16 | |
| corvus | infra-root: https://zuul.opendev.org/t/openstack/build/8d844f8f4b7d44a195a3ae20291a60a0 the deploy base job failed | 02:40 |
| corvus | i'm not in a position to debug that now | 02:41 |
| ianw | i'll take al ook | 02:42 |
| ianw | fatal: [lists.openstack.org]: FAILED! | 02:43 |
| ianw | E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem. | 02:43 |
| ianw | i'll do a manual run of it to confirm | 02:48 |
| ianw | base has deployed now | 03:25 |
| ianw | sha256:867785204c26492af92bee4f769c36421a77ba9e17bf94c7fd0d823610fb91b9 is the gerrit image promoted by https://zuul.opendev.org/t/openstack/build/57856218ab5b4f7eba86e2e3777d0e8b/console | 03:28 |
| ianw | https://hub.docker.com/layers/opendevorg/gerrit/3.4/images/sha256-3453c3420c87ed05b531e294f5030fe0cb98f5c9f40f69e4484110be02963005?context=explore was pushed by 826761 and that's what i've just ensure is pulled onto gerrit | 03:33 |
| ianw | i'm going to take gerrit down, upgrade docker and restart it, with that image | 03:37 |
| ianw | ... and back | 03:41 |
| *** ysandeep is now known as ysandeep|away | 03:52 | |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: yum-minimal: don't strip -* from releasever https://review.opendev.org/c/openstack/diskimage-builder/+/826244 | 04:07 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Switch 9-stream testing to use opendev mirrors https://review.opendev.org/c/openstack/diskimage-builder/+/821651 | 04:07 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Add 9-stream ARM64 testing https://review.opendev.org/c/openstack/diskimage-builder/+/821653 | 04:07 |
| opendevreview | Eduardo Santos proposed openstack/diskimage-builder master: Fix openSUSE images and bump them to 15.3 https://review.opendev.org/c/openstack/diskimage-builder/+/825347 | 05:19 |
| *** anbanerj is now known as frenzyfriday | 05:42 | |
| *** marios is now known as marios|ruck | 06:15 | |
| frickler | fungi: hrw: did some further debugging on the py27 oauthlib issue. seems the culprit is our wheel mirror, pip fails to detect that it should not install 3.1.1, likely because it is still an universal wheel | 06:24 |
| frickler | the reason that the issue only pops up now is that wheels hadn't been released since end of november due to the broken arm jobs https://zuul.opendev.org/t/openstack/builds?job_name=release-wheel-cache&project=openstack/requirements | 06:24 |
| frickler | .tox/py27/bin/pip install -U oauthlib --extra-index-url https://mirror.gra1.ovh.opendev.org/wheel/ubuntu-20.04-x86_64/ | 06:25 |
| frickler | that shows the failure, without our mirror everything is fine | 06:25 |
| frickler | see also https://github.com/oauthlib/oauthlib/commit/642cc2134deccd7de3a305a3f48a302fbf7e8ae9 which isn't in 3.1.1 yet | 06:27 |
| opendevreview | Merged zuul/zuul-jobs master: pin oauthlib version for python2.7 https://review.opendev.org/c/zuul/zuul-jobs/+/826648 | 06:38 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: yum-minimal: Document why we strip -stream from $releasever https://review.opendev.org/c/openstack/diskimage-builder/+/826244 | 07:05 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Switch 9-stream testing to use opendev mirrors https://review.opendev.org/c/openstack/diskimage-builder/+/821651 | 07:05 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Add 9-stream ARM64 testing https://review.opendev.org/c/openstack/diskimage-builder/+/821653 | 07:05 |
| *** amoralej|off is now known as amoralej | 07:48 | |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: centos: do not use $releasever in .repo files https://review.opendev.org/c/openstack/diskimage-builder/+/826244 | 07:53 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Switch 9-stream testing to use opendev mirrors https://review.opendev.org/c/openstack/diskimage-builder/+/821651 | 07:53 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Add 9-stream ARM64 testing https://review.opendev.org/c/openstack/diskimage-builder/+/821653 | 07:53 |
| *** bhagyashris_ is now known as bhagyashris | 08:01 | |
| *** jpena|off is now known as jpena | 08:13 | |
| dpawlik | fungi, clarkb: hey, is it ok to add logscraper01.openstack.org to our softwarefactory infra? I mean I would like to monitor the host state with prometheus node-exporter + check if services are alive. If it is ok, I will do a account on that host: "sf" or "zuul-sf" and it will be configuring additional things on that hos. | 08:18 |
| dpawlik | fungi, clarkb: ah, I forget to mention: if it is fine to monitor with node exporter, could you open the firewall on pot 9100 for host prometheus.monitoring.softwarefactory-project.io please ? | 08:19 |
| fungi | dpawlik: we don't manage any external firewall there, just update iptables on the server | 08:48 |
| dpawlik | ack fungi | 08:51 |
| *** bhagyashris_ is now known as bhagyashris | 08:53 | |
| *** ysandeep|away is now known as ysandeep | 09:29 | |
| *** dviroel_ is now known as dviroel | 11:03 | |
| *** rlandy|out is now known as rlandy|ruck | 11:12 | |
| *** amoralej is now known as amoralej|lunch | 14:02 | |
| corvus | zuul01 is up | 14:10 |
| *** rcastillo|rover is now known as rcastillo | 14:13 | |
| fungi | thanks for the quick fix! | 14:19 |
| *** amoralej|lunch is now known as amoralej | 14:30 | |
| *** ysandeep is now known as ysandeep|dinner | 14:34 | |
| opendevreview | Neil Hanlon proposed openstack/diskimage-builder master: Add new container element - Rocky Linux https://review.opendev.org/c/openstack/diskimage-builder/+/825957 | 14:47 |
| corvus | further rollout is stalled pending https://review.opendev.org/826898 | 15:26 |
| *** dviroel is now known as dviroel|lunch | 15:31 | |
| *** ysandeep|dinner is now known as ysandeep | 15:44 | |
| *** ykarel_ is now known as ykarel | 15:54 | |
| *** dviroel|lunch is now known as dviroel | 16:23 | |
| clarkb | I've approved that change now | 16:27 |
| clarkb | ianw: thank you for getting that new gerrit image installed | 16:28 |
| corvus | ianw: and thanks for the base playbook fix :) | 16:29 |
| corvus | clarkb: thanks; i'll be afk for a while today, but i'll be around this evening/tomorrow to roll that out | 16:29 |
| clarkb | sounds good. I think I may need to restart zuul executors, I can work with fungi for that since he has done a couple of those recently | 16:31 |
| fungi | well, we can do a graceful restart of the schedulers if we prefer | 16:34 |
| fungi | all the ones i did recently were hard restarts for the sake of expediency/urgency | 16:34 |
| *** jpena is now known as jpena|off | 17:06 | |
| *** marios|ruck is now known as marios|out | 17:08 | |
| fungi | clarkb: if you're cool with 826734 i can go ahead and get that server deleted | 17:10 |
| clarkb | fungi: approved | 17:11 |
| fungi | thanks! | 17:17 |
| opendevreview | Merged opendev/system-config master: Drop wiki-dev03 from inventory https://review.opendev.org/c/opendev/system-config/+/826734 | 17:32 |
| fungi | cool, i'll delete it now | 17:37 |
| fungi | and done | 17:38 |
| *** ysandeep is now known as ysandeep|out | 17:42 | |
| *** amoralej is now known as amoralej|off | 19:07 | |
| *** dviroel is now known as dviroel|brb | 20:46 | |
| clarkb | ok the system-config-run-review-3.4 job is failing to build because it is executing before the image it depends on has been built | 21:53 |
| clarkb | the review-3.3 job is waiting properly | 21:54 |
| clarkb | the good news is that zuul seems to be checking out the depends on in the imgae build properly which means we should be able to do this depends on thing with upstream gerrit changes if we figure out why the system-config-run-review-3.4 job is unhappy | 21:54 |
| clarkb | we list system-config-build-image-gerrit-3.4 as a soft dependency | 21:56 |
| priteau | Hello. Was I the only one who received email an hour ago from Storyboard, but for things that happened yesterday? | 21:58 |
| clarkb | ok its a transitive issue. one of the parent chagne failed to build the image so no we can't build the image in later changes. I'll recheck the bottom of the stack and work our way up I guess | 22:00 |
| clarkb | priteau: I haven't received recent emails from storyboard. fungi may be subscribed to more stuff | 22:00 |
| clarkb | priteau: I do wonder if that means your mail servers were rejecting storyboard emails for a bit | 22:07 |
| clarkb | smtp is a protocol that will retry with backoffs | 22:07 |
| clarkb | fungi: fyi I only rechecked the first change in your gerrit gitea stack because I realized that stack is making prod changes and the bottom of the stack is approved. The first one should be fine it is just a docker image update to remove the gitweb stuff explicitly from the image. But worried about not being ble to watch the others go in | 22:08 |
| corvus | zuul01 is restarted; i'm going to restart zuul02 now, which will cause a web outage | 23:07 |
| corvus | it's up, now the mergers | 23:22 |
| corvus | #status log restarted all of zuul on 930ee8faa3076233614565fcfbf55a4ee74551a7 | 23:25 |
| opendevstatus | corvus: finished logging | 23:25 |
| corvus | i'm going to restart nodepool now | 23:27 |
| corvus | #status log restarted all of nodepool on 1a73a7a33ed63ad919377fae42c14390d8fb9eb5 | 23:31 |
| opendevstatus | corvus: finished logging | 23:31 |
| fungi | thanks! | 23:44 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!