| opendevreview | Merged zuul/zuul-jobs master: configure-mirrors: fix typo in 9-stream enablement list https://review.opendev.org/c/zuul/zuul-jobs/+/858256 | 00:21 |
|---|---|---|
| opendevreview | Merged openstack/diskimage-builder master: Start running dib-lint again https://review.opendev.org/c/openstack/diskimage-builder/+/855589 | 00:25 |
| *** ysandeep|out is now known as ysandeep | 00:29 | |
| *** dasm is now known as dasm|off | 00:37 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: Abstract name of bastion host for testing path https://review.opendev.org/c/opendev/system-config/+/858476 | 00:52 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Convert production playbooks to bastion host group https://review.opendev.org/c/opendev/system-config/+/858486 | 00:52 |
| ianw | ok, now we've got liniting i'll make a dib release | 00:55 |
| ianw | probably the best thing to merge is clarkbs linter fix to nodepool, then that will pull in a new release | 00:56 |
| ianw | well, switching *back* to a gpg key from a ssh key is a bit of a PITA | 01:05 |
| ianw | for reference, it's "git config gpg.format openpgp" to get back to "regular" signing; and set user.signingkey to the gpg key, not the ssh one | 01:15 |
| ianw | i guess it would be nice if gerrit's signed tag push allowed ssh signatures | 01:15 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Convert production playbooks to bastion host group https://review.opendev.org/c/opendev/system-config/+/858486 | 01:38 |
| *** pojadhav|PTO is now known as pojadhav | 03:08 | |
| *** ysandeep is now known as ysandeep|afk | 03:31 | |
| *** Guest931 is now known as prometheanfire | 03:47 | |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Add Rocky 9 ARM64 functional test https://review.opendev.org/c/openstack/diskimage-builder/+/858606 | 05:08 |
| *** ianw is now known as ianw_pto | 05:09 | |
| *** ysandeep|afk is now known as ysandeep | 05:58 | |
| frickler | I'm still worried by the "infinite" lifetime of ssh keys. also I don't think they're useable without gerrit and gitea support | 05:58 |
| mnasiadka | ianw_pto: I see you mentioned it builds, thanks for adding functest in DIB | 07:28 |
| zigo | Has the IRC channel for murano changed? Where should I go? | 07:36 |
| zigo | Murano has the same defect with oslo.db as Designate and Magnum (ie: autocomit)... | 07:37 |
| zigo | I just fixed it on the Debian side of things. | 07:37 |
| zigo | And forwarded my patch ... | 07:37 |
| *** jpena|off is now known as jpena | 07:38 | |
| zigo | Looks like Vitrage has the issue too (I'm on it...). | 07:38 |
| mnasiadka | Magnum is now merging | 07:47 |
| ianw_pto | frickler: you can choose the length of time to trust the key as in https://review.opendev.org/c/opendev/system-config/+/857542. and i think that github will show things correctly signed even if your key is expired (but not revoked) because it was too confusing that people's old commits looked invalid | 08:21 |
| *** pojadhav is now known as pojadhav|afk | 08:22 | |
| frickler | I guess we also need cryptographically verifiable timestamps on sigs. but yes, I'd expect the valid-until to limit the timeframe where valid signatures can be made with that key, not limit the validity of existing signatures | 08:27 |
| *** tobias-urdin6 is now known as tobias-urdin | 08:49 | |
| opendevreview | Rafal Lewandowski proposed openstack/diskimage-builder master: Added cloud-init growpart element https://review.opendev.org/c/openstack/diskimage-builder/+/855856 | 09:28 |
| opendevreview | Frikin Evgenii proposed openstack/diskimage-builder master: Add variable for check installing python3 in yum element https://review.opendev.org/c/openstack/diskimage-builder/+/856577 | 09:58 |
| *** ysandeep is now known as ysandeep|lunch | 10:06 | |
| *** rlandy|out is now known as rlandy|ruck | 10:29 | |
| *** ysandeep|lunch is now known as ysandeep | 10:58 | |
| *** pojadhav|afk is now known as pojadhav | 11:14 | |
| fungi | zigo: looks like we did log a #murano channel but it had gone unused for so long we removed it from our configuration before the move off freenode | 11:20 |
| opendevreview | Rafal Lewandowski proposed openstack/diskimage-builder master: Added cloud-init growpart element https://review.opendev.org/c/openstack/diskimage-builder/+/855856 | 11:25 |
| fungi | frickler: yes, the general idea with key expiration is that you shouldn't trust a signature made later than the key validity (especially in the case of revocations), and you don't want to encrypt messages to expired or revoked keys obviously | 11:27 |
| fungi | but also, how you tell for sure that a signature was made after the key expired (and wasn't just backdated) is not always a solved problem | 11:28 |
| fungi | i haven't looked into how signed commits are implemented, but as long as the signature is an integral part of the commit itself and factors into subsequent commit hashes, that should be fairly easy to confirm. if signatures can be attached later and subsequent timestamped data doesn't depend on them, then you don't really know for sure when they were made | 11:30 |
| *** frenzyfriday is now known as frenzyfriday|lunch | 11:44 | |
| zigo | fungi: Is murano a dead project ?!? | 11:54 |
| zigo | I wrote this: https://review.opendev.org/c/openstack/murano/+/858638 | 11:56 |
| zigo | I'd like to make sure core reviewers will see it ... | 11:56 |
| frickler | at least murano won't run any jobs until the queue config is fixed. not sure it can be called dead, but it sure smells funny IMO | 12:09 |
| frickler | maybe wait a couple of days for reactions and take it up with the TC if nothing happens? | 12:10 |
| *** ysandeep is now known as ysandeep|afk | 12:26 | |
| *** ysandeep|afk is now known as ysandeep | 12:39 | |
| *** frenzyfriday|lunch is now known as frenzyfriday | 12:46 | |
| opendevreview | Rafal Lewandowski proposed openstack/diskimage-builder master: Added cloud-init growpart element https://review.opendev.org/c/openstack/diskimage-builder/+/855856 | 12:56 |
| opendevreview | Rafal Lewandowski proposed openstack/diskimage-builder master: Added cloud-init growpart element https://review.opendev.org/c/openstack/diskimage-builder/+/855856 | 13:25 |
| *** dasm|off is now known as dasm | 14:05 | |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Fix CORS and endpoint in AWS log upload https://review.opendev.org/c/zuul/zuul-jobs/+/858726 | 14:16 |
| *** ysandeep is now known as ysandeep|out | 14:16 | |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Fix CORS and endpoint in AWS log upload https://review.opendev.org/c/zuul/zuul-jobs/+/858726 | 14:38 |
| slittle1 | please add me as the first core for starlingx-app-sts-silicom-core. I'll set up the rest. Thanks | 15:23 |
| *** marios is now known as marios|out | 15:31 | |
| clarkb | slittle1: give me a couple minutes to load keys and I can do that | 15:34 |
| clarkb | slittle1: you've been added | 15:40 |
| clarkb | infra-root I've discovered via codesearch (so likely incomplete listing) that we're using ansible-version for ansible 2.8/2.9 in a handful of places | 15:43 |
| clarkb | Since support for that is going away reall soon I'll get changes up to remove that asap | 15:44 |
| slittle1 | clarkb: thanks | 15:44 |
| opendevreview | Clark Boylan proposed opendev/base-jobs master: Remove ansible-version: 2.8 https://review.opendev.org/c/opendev/base-jobs/+/858766 | 15:46 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Remove ansible-version: 2.9 https://review.opendev.org/c/opendev/system-config/+/858767 | 15:48 |
| clarkb | infra-root and rlandy|ruck chandankumar https://review.opendev.org/q/topic:cleanup-old-ansible | 15:52 |
| rlandy|ruck | clarkb: thanks - will look into those - we probably needs those values specified elsewhere | 15:54 |
| clarkb | rlandy|ruck: well in this case I'm proposing you don't hardcode them | 15:54 |
| clarkb | that way you'll be kept up to date automatically when the zuul tenant config updates | 15:54 |
| clarkb | whcih is what I had expected happend a while back when we moved everyone to ansible 5 ... | 15:54 |
| clarkb | rlandy|ruck: https://review.opendev.org/c/zuul/zuul/+/857796/ is the zuul change and opendev auto deploys zuul updates each weekend at the latest | 15:55 |
| rlandy|ruck | ok - should be fine - if the CI on that does not fail then we aren't relying on some specified version | 15:58 |
| fungi | clarkb: oh! i did codesearch for ansible_version not ansible-version, oops! | 15:58 |
| fungi | thanks for spotting those | 15:59 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Fix CORS and endpoint in AWS log upload https://review.opendev.org/c/zuul/zuul-jobs/+/858726 | 16:18 |
| clarkb | A change to nodepool just landed which should've updated dib in the builder image. The next hourly deployment should upgrade the builders and then we can rebuild fedora-36 images | 16:22 |
| clarkb | I'm going to go ahead and single core approve the ansible-version removals from base-jobs and system-config | 16:23 |
| clarkb | the base-jobs change only affects base-test | 16:24 |
| clarkb | and in system-config it is self testing | 16:24 |
| opendevreview | Merged opendev/base-jobs master: Remove ansible-version: 2.8 https://review.opendev.org/c/opendev/base-jobs/+/858766 | 16:32 |
| clarkb | slittle1: thinking a bit more about the linux kernel idea. I think if ya'll wanted to push that along further the best next step is to push a change to opendev/system-config that adds the kernel as a test repo in system-config-run-review-base and system-config-run-gitea jobs. That won't give us a direct comparison to prod performance but will give us data we can start to work with | 16:33 |
| *** jpena is now known as jpena|off | 16:40 | |
| opendevreview | Merged opendev/system-config master: Remove ansible-version: 2.9 https://review.opendev.org/c/opendev/system-config/+/858767 | 16:47 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Fix CORS and endpoint in AWS log upload https://review.opendev.org/c/zuul/zuul-jobs/+/858726 | 17:02 |
| corvus | i'm going to launch the tracing server now | 17:11 |
| corvus | our docs don't seem to be up to date: https://opendev.org/opendev/system-config/src/branch/master/launch/README.rst | 17:15 |
| corvus | i get openstack.exceptions.BadRequestException: BadRequestException: 400: Client Error for url: https://dfw.servers.api.rackspacecloud.com/v2/610275/servers, Bad networks format | 17:15 |
| corvus | is there a different env i should run the launch script out of? | 17:15 |
| clarkb | corvus: it is possible that openstacksdk no longer supports rackspace server apis :/ | 17:16 |
| clarkb | corvus: I've got a venv in my homedir that is there for rax cinder voluem management | 17:16 |
| corvus | clarkb: hrm. i guess to verify i should make a venv and pip install openstacksdk? | 17:16 |
| clarkb | corvus: ya or try if ~clarkb/venv which is an older sdk for cinder stuff also fixes the servers api | 17:17 |
| corvus | i'll try yours | 17:17 |
| corvus | that did not immediately blow up, so that's progress | 17:18 |
| corvus | on a lighter note, while i'm waiting... this had me chuckling: `opendev/zone-opendev.org/zones/opendev.org/zone.db` it's basically a path that just says "opendev" and "zone" over and over again :) | 17:18 |
| corvus | "i heard you like opendev and zones" | 17:19 |
| clarkb | we put more dns into your domain and vice versa | 17:19 |
| clarkb | infra-root the builder image and container have updated on nb01 and nb02. I've manually requested a rebuild of fedora-36 | 17:20 |
| corvus | the playbook failed with: AttributeError: module 'platform' has no attribute 'linux_distribution' | 17:22 |
| corvus | i'm trying to work out how fatal that is... | 17:22 |
| clarkb | grepping for platform and linux_distribution returns no results in system-config so the failure is happening in ansible itself I guess? | 17:23 |
| clarkb | ah google says its a python/pip thing | 17:24 |
| clarkb | https://stackoverflow.com/questions/58758447/how-to-fix-module-platform-has-no-attribute-linux-distribution-when-instal | 17:24 |
| corvus | here's the full output https://paste.opendev.org/show/bxh7k10IiyVKC19GOShA/ | 17:25 |
| clarkb | corvus: maybe the ansible in my env is too old and expecting older python3 on the node (I'm assuming it is focal or jammy which is 3.8 or 3.10) | 17:26 |
| clarkb | so ya may need to construct a new env with newer ansible but older opnstacksdk (I hope that doesn't introduce new issues) | 17:27 |
| clarkb | ugh | 17:27 |
| clarkb | oh ya the ansible in that env is pretty ancient | 17:28 |
| clarkb | (I normally only use openstackclient in that env) | 17:28 |
| clarkb | corvus: checking ansible's git log I think ansible 2.7.0 or newer fixes that issue | 17:31 |
| corvus | i will try ansible 2.7.18 and openstacksdk 0.41 | 17:37 |
| clarkb | re linux kernel I'm tailing the fedora-36 image builds and we'd need to handle the large repo there too (it will add significant disk usage to our images if we cache the repo there) | 17:41 |
| clarkb | But I still think starting with the Ci jobs we've got to collect initial data is the way to go if slittle1 wants to pursue it | 17:42 |
| corvus | should this be focal or jammy? | 17:54 |
| clarkb | corvus: I think jammy if it works (it generally does in CI, but this would be our first real one) | 17:55 |
| corvus | (current launch-node default is focal, so that's what's booting now, but it just occurred to me maybe we want to change that?) | 17:55 |
| fungi | the mm3 testing we've been doing has all been on jammy as well, with the expectation that's what the new ml server will run | 17:55 |
| clarkb | put another way, if jammy explodes on you I would proceed with focal and we can sort out jammy separately | 17:56 |
| fungi | updating the launch script sounds fine to me, tes | 17:56 |
| fungi | er, yes | 17:56 |
| clarkb | but if jammy works (and our testing indicates it should) then go for it | 17:56 |
| fungi | agreed, standing up a new service on focal just means one more server we'll have to work out how to upgrade to jammy later | 17:57 |
| corvus | hrm, same issue with ansible | 17:57 |
| corvus | i will try ansible 2.9 | 17:59 |
| corvus | i accidentally installed ansible outside the venv, i'm repairing that now by reinstalling ansible==4.0.0, but be aware that could have caused issues | 18:02 |
| fungi | noted, thanks for the warning! | 18:02 |
| clarkb | The update to using a venv for this can't come soon enough :) | 18:03 |
| fungi | clarkb: do you think we should follow up to the thread from june/july about the ansible 5 default to say that support for older ansible is going away imminently? in case people have pinned it in stable branches or something | 18:04 |
| clarkb | fungi: ++ | 18:04 |
| clarkb | you can like to my topic of changes too to give people examples of what to do | 18:04 |
| fungi | https://lists.opendev.org/pipermail/service-announce/2022-July/000043.html was our last communication on the subject | 18:04 |
| clarkb | the tripleo-ci change appears to be happy so far. One job failed but it is non voting so I'm unsure of how important that is | 18:05 |
| fungi | will do | 18:05 |
| clarkb | the bootstrap bridge job just reported success in the hourly infra-prod jobs fwiw | 18:06 |
| corvus | zomg we have ~root/launch-node-venv ~root/launch-env and i was just about to create ~root/launch-venv | 18:09 |
| corvus | good news and bad news | 18:10 |
| fungi | i think it's time for me to find a lunch-venv in fact | 18:10 |
| corvus | ansible works, but the inital playbook now fails with "userdel: user ubuntu is currently used by process 1559" | 18:10 |
| corvus | that may be a jammy fault | 18:10 |
| fungi | oof | 18:10 |
| corvus | retrying with focal | 18:10 |
| fungi | some service running as that user, or maybe it's the shell itself? | 18:11 |
| corvus | also, it might just be that jammy checks that and focal didn't | 18:11 |
| fungi | entirely possible | 18:11 |
| clarkb | I think we may try to login as root and the older ubuntu images allowed that. But if that fails fallback to ubuntu? But ya I think we can proceed with focal given that and sort this out later | 18:12 |
| corvus | we could add -f | 18:12 |
| corvus | we probably use an ansible task for that though | 18:12 |
| corvus | so "just add -f" may not be simple | 18:12 |
| fungi | announcement for the ansible 2.x removal in jobs has been sent. i'm taking a long-ish late lunch/early dinner a ways up to the north end of the island, so will probably be afk for a couple of hours | 18:20 |
| fungi | bbiaw | 18:21 |
| clarkb | enjoy! | 18:21 |
| opendevreview | James E. Blair proposed opendev/zone-opendev.org master: Add tracing server to DNS https://review.opendev.org/c/opendev/zone-opendev.org/+/858812 | 19:12 |
| opendevreview | James E. Blair proposed opendev/system-config master: Add tracing server to inventory https://review.opendev.org/c/opendev/system-config/+/858813 | 19:13 |
| clarkb | rlandy|ruck: https://review.opendev.org/c/openstack/tripleo-ci/+/858768 did get a +1 from zuul | 20:37 |
| rlandy|ruck | and third party | 20:43 |
| rlandy|ruck | do you need that merged now? | 20:44 |
| rlandy|ruck | otherwise will get a second core on that tomorrow morning (europe time) | 20:44 |
| rlandy|ruck | clarkb: ^^ | 20:44 |
| clarkb | rlandy|ruck: I don't need it to be merged. Just know the zuul deployment may update over the weekend and break your jobs | 20:44 |
| clarkb | tomorrow morning is probably fine | 20:45 |
| rlandy|ruck | ok - will get that sorted tomorrow | 20:45 |
| *** ScottSolkhon[m] is now known as scottsol[m] | 21:22 | |
| clarkb | infra-root I've cleaned up my old fedora-36 node in rax dfw and am booting another with the newly built image | 21:28 |
| fungi | thanks! | 21:29 |
| clarkb | I successfully logged into a new fedora-36 instance in rax booted off the new image | 21:34 |
| clarkb | I have deleted the host now too. I think that ends the fedora-36 safa | 21:34 |
| clarkb | *saga | 21:34 |
| *** rlandy|ruck is now known as rlandy|bbl | 22:05 | |
| corvus | i'd like to restart the zuul web servers to pick up the rest api change for semaphores | 23:02 |
| corvus | there's no internal zk api changes, so running with a version skew shouldn't be an issue | 23:04 |
| Clark[m] | Fine with me. | 23:04 |
| corvus | i can do one at a time to make rollback easier in case something goes terribly wrong | 23:04 |
| Clark[m] | May as well do them rolling since we can. I'm off my computer for a moment but will jump back in a few minutes and can help should that become necessary | 23:05 |
| corvus | 01 is restarting | 23:07 |
| corvus | stopping 02 now | 23:29 |
| corvus | sanity check still looks good, starting 02 now | 23:29 |
| corvus | https://zuul.opendev.org/api/tenant/openstack/semaphores is new info | 23:30 |
| clarkb | neat | 23:32 |
| corvus | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f3f/856338/4/check/zuul-build-dashboard-opendev/f3f5123/npm/html/ is a preview build with a ui for that | 23:33 |
| fungi | awesome--thanks for the update! | 23:33 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!