| opendevreview | Merged zuul/zuul-jobs master: emit-job-header: noqa on error ignore https://review.opendev.org/c/zuul/zuul-jobs/+/851163 | 00:06 |
|---|---|---|
| opendevreview | Merged zuul/zuul-jobs master: linters: fix spaces between filters https://review.opendev.org/c/zuul/zuul-jobs/+/851014 | 00:07 |
| ianw | fungi: thanks for working on that! saves us a lot of pain when volumes go missing | 00:07 |
| fungi | okay, rackspace ticked closed out | 00:09 |
| fungi | #status log Replaced 3 Cinder volumes across afs01.ord and backup01.ord servers in order to avoid impact from upcoming scheduled maintenance for the service provider | 00:10 |
| opendevstatus | fungi: finished logging | 00:10 |
| fungi | ianw: no problem, i meant to knock it out weeks ago, but at least we had plenty of notice | 00:11 |
| opendevreview | Merged zuul/zuul-jobs master: linters: add names to blocks https://review.opendev.org/c/zuul/zuul-jobs/+/851015 | 00:22 |
| opendevreview | Merged zuul/zuul-jobs master: linters: update to ansible-lint 6 https://review.opendev.org/c/zuul/zuul-jobs/+/851017 | 00:22 |
| opendevreview | Merged zuul/zuul-jobs master: ansible-lint: disable progressive mode https://review.opendev.org/c/zuul/zuul-jobs/+/851164 | 00:22 |
| opendevreview | Merged zuul/zuul-jobs master: ensure-kubernetes: pull cri-dockerd systemd from tag https://review.opendev.org/c/zuul/zuul-jobs/+/851263 | 00:22 |
| ianw | anyone know who usually releases "hacking"? | 01:30 |
| ianw | it has unrelased changes to use flake8 4.0.1 which would fix running with python3.10 | 01:30 |
| *** rlandy|bbl is now known as rlandy | 01:39 | |
| *** rlandy is now known as rlandy|out | 01:39 | |
| ianw | hrm, do we know about | 01:44 |
| ianw | scp -P29418 iwienand@review.opendev.org:hooks/commit-msg .git/hooks/commit-msg | 01:44 |
| ianw | failing with newer ssh clients? | 01:44 |
| Clark[m] | Yes. It is because scp is deprecated | 01:51 |
| Clark[m] | I think you can override to reenable it. But also maybe git review should fetch it via http | 01:52 |
| ianw | yeah i haven't setup a new repo in a long time, i guess :) | 01:52 |
| ianw | https://bugs.chromium.org/p/gerrit/issues/detail?id=15944 is upstream | 01:55 |
| ianw | oh, mea culpa. i'm actually running git review 2.2.0. looks likes fixes to auto flip the flag were merged in 2.3.0 | 01:58 |
| ianw | in related news; https://review.opendev.org/c/openstack/releases/+/851273 is for hacking | 02:00 |
| opendevreview | Ian Wienand proposed openstack/project-config master: trigger-rtd: fix role syntax https://review.opendev.org/c/openstack/project-config/+/851274 | 02:10 |
| opendevreview | Ian Wienand proposed openstack/project-config master: linter: update some YAML whitespace issues https://review.opendev.org/c/openstack/project-config/+/851275 | 02:10 |
| opendevreview | Ian Wienand proposed openstack/project-config master: linter: add name to blocks https://review.opendev.org/c/openstack/project-config/+/851276 | 02:10 |
| opendevreview | Ian Wienand proposed openstack/project-config master: linter: use spaces between filters in jinja https://review.opendev.org/c/openstack/project-config/+/851277 | 02:10 |
| opendevreview | Ian Wienand proposed openstack/project-config master: linter: update ansible-lint; add auto-download of roles https://review.opendev.org/c/openstack/project-config/+/851278 | 02:10 |
| ianw | i'm watching a translation update job from periodic and it seems to be getting further | 02:14 |
| ianw | Obtaining file:///home/zuul/src/opendev.org/openstack/i18n | 02:15 |
| ianw | error: Multiple top-level packages discovered in a flat-layout: ['glossary', 'playbooks', 'resources', 'checksite'] | 02:15 |
| ianw | i guess this is not compatible with the new setuptools. that should be fixable | 02:15 |
| fungi | sounds about right | 02:40 |
| ianw | https://review.opendev.org/c/openstack/i18n/+/851280 should fix that | 03:00 |
| ianw | ianychoi[m]: ^ | 03:01 |
| opendevreview | Ian Wienand proposed openstack/project-config master: linter: update ansible-lint; add auto-download of roles https://review.opendev.org/c/openstack/project-config/+/851278 | 03:10 |
| opendevreview | Ian Wienand proposed openstack/project-config master: linter: update ansible-lint; add auto-download of roles https://review.opendev.org/c/openstack/project-config/+/851278 | 03:17 |
| opendevreview | Ian Wienand proposed openstack/project-config master: linter: update ansible-lint; add auto-download of roles https://review.opendev.org/c/openstack/project-config/+/851278 | 03:29 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: linters: standardise on newline at end of file https://review.opendev.org/c/zuul/zuul-jobs/+/851288 | 03:55 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: linters: standardise on newline at end of file https://review.opendev.org/c/zuul/zuul-jobs/+/851288 | 04:08 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: linters: use Ansible 5 for ansible-lint https://review.opendev.org/c/zuul/zuul-jobs/+/851289 | 04:08 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: test-requirements: bump to Ansible 5 https://review.opendev.org/c/zuul/zuul-jobs/+/851334 | 04:18 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: test-requirements: bump to Ansible 5 https://review.opendev.org/c/zuul/zuul-jobs/+/851334 | 04:44 |
| *** ysandeep|out is now known as ysandeep | 05:28 | |
| *** pojadhav- is now known as pojadhav | 05:30 | |
| *** ysandeep is now known as ysandeep|brb | 06:03 | |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: linters: standardise on newline at end of file https://review.opendev.org/c/zuul/zuul-jobs/+/851288 | 06:28 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: linters: use Ansible 5 for ansible-lint https://review.opendev.org/c/zuul/zuul-jobs/+/851289 | 06:28 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: Drop py27 tox testing https://review.opendev.org/c/zuul/zuul-jobs/+/851343 | 06:28 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: test-requirements: drop 3.5 dependencies https://review.opendev.org/c/zuul/zuul-jobs/+/851344 | 06:28 |
| *** ysandeep|brb is now known as ysandeep | 06:36 | |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: test-requirements: drop 3.5 dependencies https://review.opendev.org/c/zuul/zuul-jobs/+/851344 | 06:37 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: test-requirements: bump to Ansible 5 https://review.opendev.org/c/zuul/zuul-jobs/+/851334 | 06:37 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: linters: use Ansible 5 for ansible-lint https://review.opendev.org/c/zuul/zuul-jobs/+/851289 | 06:37 |
| *** pojadhav- is now known as pojadhav | 07:15 | |
| opendevreview | Ian Wienand proposed opendev/base-jobs master: linters: update to ansible-lint 6 https://review.opendev.org/c/opendev/base-jobs/+/851348 | 07:16 |
| opendevreview | Ian Wienand proposed opendev/base-jobs master: linters: update to ansible-lint 6 https://review.opendev.org/c/opendev/base-jobs/+/851348 | 07:30 |
| *** jpena|off is now known as jpena | 07:38 | |
| *** jpodivin_ is now known as jpodivin | 08:37 | |
| *** ysandeep is now known as ysandeep|lunch | 10:21 | |
| *** rlandy|out is now known as rlandy | 10:38 | |
| *** ysandeep|lunch is now known as ysandeep | 10:58 | |
| *** dviroel|afk is now known as dviroel | 11:31 | |
| fungi | looks like we have about a week left on the ssl cert for the linaro mirror | 12:21 |
| fungi | er, i mean linaro cloud api endpoint | 12:21 |
| fungi | kevinz: when you get a chance, can you refresh the ssl cert for the linaro cloud api endopint? ideally before wednesday when it expires | 12:23 |
| *** ysandeep is now known as ysandeep|afk | 12:53 | |
| *** dasm|off is now known as dasm | 12:57 | |
| arxcruz|rover | fungi is this related to the post_failure i'm seeing ? | 13:04 |
| fungi | arxcruz|rover: "this" what? | 13:04 |
| arxcruz|rover | the ssl cert | 13:05 |
| arxcruz|rover | fungi i'm seeing some pep and python jobs failing with post_failure | 13:05 |
| fungi | not unless you have a time machine to run your tests a week into the future ;) | 13:05 |
| fungi | and on arm | 13:05 |
| arxcruz|rover | fungi for example https://zuul.openstack.org/build/52cb7e83a58d4cd8ad1e43fa47450bce | 13:05 |
| arxcruz|rover | fungi https://zuul.openstack.org/builds?result=POST_FAILURE&skip=0 | 13:06 |
| arxcruz|rover | it seems to have a bunch today | 13:06 |
| fungi | looks like those start around 12:00z | 13:07 |
| fungi | or maybe that's just when job volume picked up | 13:08 |
| fungi | i suspect a problem with a swift provider, i'll check executor logs | 13:08 |
| fungi | TASK [upload-logs-swift : Upload logs to swift] | 13:12 |
| fungi | WARNING:keystoneauth.identity.generic.base:Failed to discover available identity versions when contacting https://identity.api.rackspacecloud.com/v2.0/. Attempting to parse version from URL. | 13:13 |
| fungi | keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://identity.api.rackspacecloud.com/v2.0/tokens: HTTPSConnectionPool(host=\'identity.api.rackspacecloud.com\', port=443): Max retries exceeded with url: /v2.0/tokens (Caused by SSLError(SSLCertVerificationError(1, \'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired | 13:13 |
| fungi | hah! what a coincidence | 13:13 |
| fungi | openssl x509 confirms it: "Not After : Jul 28 12:00:00 2022 GMT" | 13:14 |
| fungi | i'll take them out of our log upload list for the time being | 13:15 |
| fungi | infra-root: ^ | 13:15 |
| *** ysandeep|afk is now known as ysandeep | 13:15 | |
| fungi | maybe we should be watching ssl cert expirations for all our providers, not just the linaro cloud :/ | 13:15 |
| Clark[m] | That may impact our ability to boot instances there too. May want to file an issue? | 13:16 |
| fungi | yeah, i'm probably not the first to notice | 13:16 |
| fungi | but i'll add that to the list | 13:16 |
| Clark[m] | Yes grafana seems to indicate we aren't booting test nodes there | 13:17 |
| opendevreview | Jeremy Stanley proposed opendev/base-jobs master: Temporarily stop trying to save logs to Rackspace https://review.opendev.org/c/opendev/base-jobs/+/851403 | 13:19 |
| Clark[m] | fungi: ^ lgtm but not in a spot to login to Gerrit and +2 it | 13:20 |
| opendevreview | Merged opendev/base-jobs master: Temporarily stop trying to save logs to Rackspace https://review.opendev.org/c/opendev/base-jobs/+/851403 | 13:21 |
| fungi | #status log Temporarily removed one of our Swift providers from the Zuul build log upload list because their API cert expired, resulting in POST_FAILURE errors starting at 12:00 UTC | 13:22 |
| opendevstatus | fungi: finished logging | 13:22 |
| fungi | arxcruz|rover: ^ hopefully those failures will stop for any builds started after 851403 merged (unfortunately a lot of currently running builds may still break since the upload target is decided at build start) | 13:23 |
| arxcruz|rover | fungi all good, most important is that we know the root cause and it's bein fixed :) thanks | 13:24 |
| opendevreview | Jeremy Stanley proposed opendev/base-jobs master: Revert "Temporarily stop trying to save logs to Rackspace" https://review.opendev.org/c/opendev/base-jobs/+/851404 | 13:26 |
| fungi | looks like it may already be fixed on their side? | 13:27 |
| fungi | Not After : Jun 28 23:59:59 2023 GMT | 13:27 |
| fungi | i'll check it a few times here for a bit to see if it's just sometimes that we get an expired one | 13:27 |
| fungi | but so far it's coming back consistently good | 13:28 |
| Clark[m] | I'm not sure nodepool has noticed yet based on grafana graphs. Though the error spikes seem to have stopped we still haven't cleaned up the deleting node list | 13:29 |
| fungi | https://rackspace.service-now.com/system_status "2022-07-28 07:10:00 | Engineers are investigating an issue related to access to our customer portals and potentially other services. During this time, customers may not be able to access our customer portals and possibly other services. At approximately 08:18 CDT, services have begun to recover. Engineers continue to be engaged to confirm all | 13:30 |
| fungi | systems have once again become accessible." | 13:30 |
| fungi | looks like i pushed the change right before they fixed the issue | 13:31 |
| fungi | i'm consistently getting a 2023 expiration from the api endpoint, so i expect it's good... removing wip from the revert now, but i'll let someone else approve once they're comfortable it's safe | 13:32 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Revert cri-dockerd changes https://review.opendev.org/c/zuul/zuul-jobs/+/851414 | 14:22 |
| *** ysandeep is now known as ysandeep|out | 14:50 | |
| corvus | i'm going to force-merge that change ^ (discussed in #zuul:opendev.org) | 14:51 |
| corvus | erm, apparently we have two separate sections in our docs on force-merging a change, one of them is correct and the other is very wrong | 14:52 |
| opendevreview | James E. Blair proposed opendev/system-config master: Combine / reconcile two force-merge docs https://review.opendev.org/c/opendev/system-config/+/851417 | 14:57 |
| clarkb | fungi: I think something about our copied mailman2 exim config (via the host_vars file) breaks exim under jammy https://zuul.opendev.org/t/openstack/build/d8ac40ef901a4db88a1db527b528e366 That same test is passing with our normal exim config in the base job. I guess next step is to capture the commands' stdout (exim man page indicates -bt results are published to stdout) | 14:58 |
| opendevreview | Merged zuul/zuul-jobs master: Revert cri-dockerd changes https://review.opendev.org/c/zuul/zuul-jobs/+/851414 | 15:02 |
| corvus | #status log force-merged https://review.opendev.org/851414 due to regression in existing jobs | 15:03 |
| opendevstatus | corvus: finished logging | 15:03 |
| corvus | infra-root: friendly reminder that if you have changed your gerrit ssh key to deal with the rsa issue, you may need to do so for your .admin account as well | 15:05 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Add jammy testing https://review.opendev.org/c/zuul/zuul-jobs/+/851418 | 15:14 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Sort supported platforms https://review.opendev.org/c/zuul/zuul-jobs/+/851419 | 15:14 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Support subsets of platforms in update-test-platforms https://review.opendev.org/c/zuul/zuul-jobs/+/851420 | 15:14 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Test ensure-kubernetes on all Ubuntu platforms https://review.opendev.org/c/zuul/zuul-jobs/+/851421 | 15:14 |
| opendevreview | Clark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server https://review.opendev.org/c/opendev/system-config/+/851248 | 15:38 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Add jammy testing https://review.opendev.org/c/zuul/zuul-jobs/+/851418 | 15:40 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Sort supported platforms https://review.opendev.org/c/zuul/zuul-jobs/+/851419 | 15:40 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Support subsets of platforms in update-test-platforms https://review.opendev.org/c/zuul/zuul-jobs/+/851420 | 15:40 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Test ensure-kubernetes on all Ubuntu platforms https://review.opendev.org/c/zuul/zuul-jobs/+/851421 | 15:40 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Revert "Revert cri-dockerd changes" https://review.opendev.org/c/zuul/zuul-jobs/+/851425 | 15:40 |
| *** marios is now known as marios|out | 16:00 | |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Revert "Revert cri-dockerd changes" https://review.opendev.org/c/zuul/zuul-jobs/+/851425 | 16:44 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Install crio on Jammy like Focal https://review.opendev.org/c/zuul/zuul-jobs/+/851434 | 16:44 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Install crio on Jammy like Focal https://review.opendev.org/c/zuul/zuul-jobs/+/851434 | 17:11 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Revert "Revert cri-dockerd changes" https://review.opendev.org/c/zuul/zuul-jobs/+/851425 | 17:11 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Install crio on Jammy like Focal https://review.opendev.org/c/zuul/zuul-jobs/+/851434 | 17:24 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Revert "Revert cri-dockerd changes" https://review.opendev.org/c/zuul/zuul-jobs/+/851425 | 17:24 |
| *** jpena is now known as jpena|off | 17:31 | |
| opendevreview | James E. Blair proposed openstack/project-config master: Synchronize diskimages on builders https://review.opendev.org/c/openstack/project-config/+/851444 | 17:36 |
| *** gibi is now known as gibi_pto | 17:43 | |
| opendevreview | James E. Blair proposed openstack/project-config master: Synchronize diskimages on builders https://review.opendev.org/c/openstack/project-config/+/851444 | 17:55 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Install crio on Jammy like Focal https://review.opendev.org/c/zuul/zuul-jobs/+/851434 | 18:05 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Revert "Revert cri-dockerd changes" https://review.opendev.org/c/zuul/zuul-jobs/+/851425 | 18:05 |
| clarkb | fungi: should I go ahead and approve your swift log location revert change? | 18:26 |
| clarkb | I too confirm ssl seems to work for me at that location | 18:26 |
| fungi | clarkb: if thinks look okay to you, then please do | 18:26 |
| fungi | sadly, i think they fixed it at the same moment i pushed the original change | 18:27 |
| clarkb | done | 18:27 |
| fungi | but at least looking into it wasn't a total waste, it allowed us to confirm things weren't broken on our end | 18:28 |
| clarkb | ++ | 18:28 |
| opendevreview | Merged opendev/base-jobs master: Revert "Temporarily stop trying to save logs to Rackspace" https://review.opendev.org/c/opendev/base-jobs/+/851404 | 18:33 |
| fungi | thanks! | 18:36 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Install crio on Jammy like Focal https://review.opendev.org/c/zuul/zuul-jobs/+/851434 | 18:36 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Revert "Revert cri-dockerd changes" https://review.opendev.org/c/zuul/zuul-jobs/+/851425 | 18:36 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Revert "Revert cri-dockerd changes" https://review.opendev.org/c/zuul/zuul-jobs/+/851425 | 19:00 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Test ensure-kubernetes on all Ubuntu platforms https://review.opendev.org/c/zuul/zuul-jobs/+/851421 | 19:00 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Allow Stackalytics maintainers to rewrite history https://review.opendev.org/c/openstack/project-config/+/851454 | 19:05 |
| opendevreview | Denys Mishchenko proposed openstack/diskimage-builder master: Add download of manifest file. https://review.opendev.org/c/openstack/diskimage-builder/+/851462 | 19:22 |
| *** dviroel is now known as dviroel|afk | 20:24 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server https://review.opendev.org/c/opendev/system-config/+/851248 | 20:45 |
| clarkb | woot ^ finally passes the trivial setup of a node and exim | 21:07 |
| clarkb | I've got some afternoon errands, but I'll try to start adding in service bits tomorrow | 21:07 |
| corvus | fungi: ianw https://review.opendev.org/851417 should be an easy doc update | 21:41 |
| *** dasm is now known as dasm|off | 21:47 | |
| *** rlandy is now known as rlandy|out | 21:51 | |
| opendevreview | Merged opendev/system-config master: Combine / reconcile two force-merge docs https://review.opendev.org/c/opendev/system-config/+/851417 | 21:53 |
| clarkb | I meant to mention the results of the works on arm discussion. It sounds like they can give us (really kevinz) a couple of nodes and then we'll do a sync up in ~6 months to see how it is going and whether it should continue. | 22:20 |
| clarkb | Also there is opportunity/need to write up a blog post for them or something similar to that to show off what it is we are doing with arm | 22:21 |
| ianw | ++ i think the more blog posts/tweets/whatever we do about it the better | 22:21 |
| fungi | if anyone wants to write something but doesn't have a blog to stick it on, the superuser editors are always happy to publish anything related to what we're working on | 22:42 |
| fungi | also the foundation has a pr agency it works with which is good at shopping our articles around to publications with broader audiences/greater reach | 22:43 |
| fungi | and the opensource.com folks are also always happy to carry good articles about our sorts of projects (possibly also a convenient outlet for red hatters, since that's who runs it) | 22:44 |
| fungi | wouldn't be hard to tie this sort of stuff to zuul either, and add to zuul-ci.org/blog which could use more content | 22:45 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!