| fungi | ianw: do you have more details? i'm able to `sudo ~fungi/foo/bin/openstack --os-cloud=openstackci-rax --os-region-name=DFW server list` on bridge01 and `sudo ~fungi/foo/bin/pip freeze|grep openstackclient` reports python-openstackclient==6.0.0 | 00:03 |
|---|---|---|
| ianw | fungi: this is on my local machine, i made a venv, installed python-openstackclient and added a clouds.yaml and did "server list" | 00:04 |
| fungi | which version were you seeing an error with? | 00:04 |
| fungi | something newer than 6.0.0? | 00:04 |
| ianw | $ ./venv/bin/openstack --version | 00:04 |
| ianw | openstack 6.0.0 | 00:04 |
| fungi | odd, i wonder why that version works on bridge | 00:04 |
| fungi | could it be related to the openstacksdk version instead? | 00:05 |
| fungi | or keystoneauth? | 00:05 |
| ianw | actually | 00:08 |
| ianw | raise exceptions.UnsupportedVersion(msg) | 00:08 |
| ianw | cinderclient.exceptions.UnsupportedVersion: Invalid client version '2.0'. Major part should be '3' | 00:08 |
| ianw | it's the volume bit maybe | 00:08 |
| fungi | you need python-cinderclient<8 | 00:09 |
| fungi | 8.0.0 dropped support for volume api v2 | 00:09 |
| fungi | and rackspace doesn't have v3 | 00:09 |
| ianw | ok, i see what i've done, i've commented out the volume_api_version: 2 when testing also | 00:10 |
| ianw | *but* ... rax connections do still work without identity_api_version: 2 | 00:10 |
| fungi | interesting | 00:11 |
| ianw | ... and fails with identity_api_version: 3 | 00:11 |
| ianw | so, it must know how to autonegotiate it | 00:12 |
| ianw | ok, that change is probably just a red-herring. i guess we can remove the explicit pin, if we feel confident openstacksdk knows what to do, but it's a minor point | 00:12 |
| fungi | yeah, we could probably remove the identity_api_version: 3 from the others too, given that | 00:13 |
| ianw | I can replicate the "BadRequestException: 400: Client Error for url: https://dfw.servers.api.rackspacecloud.com/v2/610275/servers, Bad networks format" | 00:13 |
| ianw | it looks like what we send is | 00:13 |
| ianw | -d '{"server": {"config_drive": true, "max_count": 1, "min_count": 1, "networks": "auto", "imageRef": "c2e5b7be-32ea-4f74-bb88-1c9a4104f8ca", "flavorRef": "6", "name": "ianw-test"}}' | 00:13 |
| ianw | so, presumably "networks: auto" is the thread to pull on ... | 00:14 |
| fungi | oh, so this is the api responding with an error because it doesn't understand networks:auto, not the client bombing because it doesn't understand the response? | 00:16 |
| ianw | RESP BODY: {"badRequest": {"message": "Bad networks format", "code": 400}} | 00:16 |
| ianw | i think so | 00:16 |
| fungi | that does look to be the case, yes | 00:16 |
| ianw | hrm, maybe i should first try the same thing on an older client and see if i can pick out the same request and see what it is | 00:17 |
| fungi | ianw: in that case you want openstacksdk<0.99 | 00:20 |
| fungi | which ends up being 0.61.0 | 00:20 |
| fungi | that's what got the error to go away for me | 00:20 |
| ianw | hrm, openstacksdk<0.99 doesn't work with python3.11 | 00:20 |
| fungi | oh, ouch | 00:21 |
| clarkb | correct | 00:21 |
| clarkb | the reason is a bit :( too | 00:21 |
| fungi | yeah, i guess we're still using 3.10 on bridge01 | 00:21 |
| clarkb | it uses some deprecated intepreter introspection method that was finally removed in 3.11 | 00:21 |
| ianw | ... ok, so using a python3.8 venv, and openstacksdk 0.99 | 00:22 |
| ianw | -d '{"server": {"config_drive": true, "max_count": 1, "min_count": 1, "imageRef": "c2e5b7be-32ea-4f74-bb88-1c9a4104f8ca", "flavorRef": "6", "name": "ianw-test"}}' | 00:22 |
| clarkb | 0.99 is what failed right? | 00:22 |
| fungi | yes | 00:22 |
| ianw | (this didn't work, but also it didn't give the same error) | 00:22 |
| ianw | which "failed" are we talking about? :) | 00:23 |
| ianw | anyway, it looks like openstacksdk 0.99 *doesn't* send "networks:" in the server create request | 00:24 |
| fungi | openstacksdk>=0.99.0 is what was sending network:auto and resulting in the network error response | 00:24 |
| fungi | ianw: openstacksdk 0.99 or <0.99? | 00:24 |
| ianw | aah yes sorry correct! <0.99 | 00:24 |
| ianw | let me install 0.99 and just 100% confirm | 00:24 |
| fungi | so 0.61.0 (being the last tag before 0.99.0) | 00:25 |
| ianw | yes correct. so ... | 00:25 |
| fungi | so whatever is happening with that additional parameter seems to have come in with the major refactor from the feature branch | 00:25 |
| ianw | 0.61.0 does *not* send "networks: auto" and 0.99.0 *does* send "networks: auto" and that makes RAX barf | 00:26 |
| clarkb | I'm guessing that and update was made to ste networks: auto if you don't specify any network info | 00:26 |
| ianw | yes, i guess the next step is code inspection, or if that doesn't help, i can bisect | 00:26 |
| ianw | https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/compute/v2/server.py should be it | 00:27 |
| ianw | https://opendev.org/openstack/openstacksdk/commit/29c872460b392885882c2fbba35249d188ecdcef looks like it was the modification | 00:29 |
| ianw | # If user has not passed networks - let Nova try the best. | 00:29 |
| ianw | kwargs['networks'] = 'auto' | 00:29 |
| clarkb | seems like a situation where we want to let nova make its best guess but by passing it explicit info we've invalidated that | 00:30 |
| ianw | ... to the nova source ... | 00:31 |
| fungi | that feature was probably added to nova very recently in 2015 | 00:32 |
| clarkb | "Starting with microversion 2.37, this field is required and the special string values auto and none can be specified for networks. auto tells the Compute service to use a network that is available to the project, if one exists." | 00:32 |
| clarkb | ianw: ^ from the nova compute api docs | 00:32 |
| ianw | heh, yeah, newton release notes ... | 00:32 |
| clarkb | so ya this has essentially broken all nova pre microversion 2.37 | 00:32 |
| fungi | so the sdk should only do that if if detects a sufficient microversion, but either doesn't check or checks incorrectly | 00:32 |
| ianw | well i think we have enough for an openstacksdk patch at any rate | 00:33 |
| clarkb | fungi: its not checking it seems to blindly add the argument | 00:33 |
| fungi | noice | 00:33 |
| clarkb | basically if you don't set networks it sets networks: auto for you | 00:33 |
| fungi | in good news, odds are we can use latest sdk with rackspace if that gets fixed | 00:34 |
| ianw | yeah a quick check for 2.37 microversion gets past that | 00:37 |
| clarkb | ianw: the utils file in there has a few functions that might be helpful for doing this properly | 00:39 |
| clarkb | bsaiclly if supports_microversion(2.37) or something | 00:39 |
| ianw | https://review.opendev.org/c/openstack/openstacksdk/+/865354 | 00:46 |
| ianw | i'll take a poke at testing | 00:46 |
| ianw | i guess that would simplify things down to just needing the older cinderclient | 00:46 |
| clarkb | it would also allow us to update in nodepool potentially | 00:48 |
| fungi | yes, on both counts | 00:49 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Improve launch-node deps and fix script bugs https://review.opendev.org/c/opendev/system-config/+/865320 | 01:53 |
| opendevreview | Jeremy Stanley proposed opendev/zone-opendev.org master: Add lists01 to DNS https://review.opendev.org/c/opendev/zone-opendev.org/+/865360 | 02:21 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Add lists01.opendev.org to the inventory https://review.opendev.org/c/opendev/system-config/+/865361 | 02:22 |
| fungi | ianw: another detail i just spotted... launch-node outputs command lines to use the rackdns tool for setting reverse dns entries in rackspace, but it doesn't seem we copied that to bridge01 (i'll probably just resort to using the webui) | 02:38 |
| ianw | oh, hrm, yeah | 02:42 |
| ianw | i felt like for some reason that stopped working. but maybe it did start working again | 02:42 |
| Clark[m] | That was something that got on the etherpad for bridge iirc | 02:45 |
| opendevreview | Ian Wienand proposed opendev/system-config master: rax: remove identity_api_version 2 pin https://review.opendev.org/c/opendev/system-config/+/865351 | 02:46 |
| ianw | https://github.com/kwminnick/rackspace-dns-cli does not look compelling :/ | 02:49 |
| ianw | probably it would be easier to repurpose the rax dns backup script | 02:49 |
| *** pojadhav|afk is now known as pojadhav | 03:12 | |
| *** yadnesh|away is now known as yadnesh | 04:03 | |
| ianw | that can work, i'll clean it up. i don't want to spend too much time on rax dns tools | 04:04 |
| Clark[m] | Do the existing tools not work? | 04:22 |
| Clark[m] | Maybe we would need python2 on the new bridge? | 04:22 |
| *** ysandeep is now known as ysandeep|ruck | 04:29 | |
| ianw | the existing tool i linked to above is abandoned for a long time -- i think we can get a token and do the post in a much more straight forward fashion | 05:25 |
| *** ysandeep|ruck is now known as ysandeep|ruck|brb | 05:34 | |
| *** ysandeep|ruck|brb is now known as ysandeep|ruck | 05:53 | |
| *** jpena|off is now known as jpena | 08:29 | |
| *** yadnesh is now known as yadnesh|afk | 08:47 | |
| opendevreview | Merged openstack/project-config master: Add Allow-Post-Review flag to OpenStackSDK project https://review.opendev.org/c/openstack/project-config/+/859976 | 09:04 |
| opendevreview | Merged openstack/project-config master: Add post-review pipeline https://review.opendev.org/c/openstack/project-config/+/859977 | 09:07 |
| *** yadnesh|afk is now known as yadnesh | 09:22 | |
| *** ysandeep__ is now known as ysandeep|ruck | 10:39 | |
| *** rlandy|out is now known as rlandy|rover | 11:10 | |
| *** dviroel|afk is now known as dviroel | 11:15 | |
| *** soniya29 is now known as soniya29|afk | 11:23 | |
| *** ysandeep|ruck is now known as ysandeep|out | 11:57 | |
| opendevreview | Merged opendev/git-review master: Add message option https://review.opendev.org/c/opendev/git-review/+/864098 | 12:02 |
| opendevreview | Cedric Jeanneret proposed opendev/base-jobs master: Ensure NetworkManager will not override /etc/resolv.conf https://review.opendev.org/c/opendev/base-jobs/+/865428 | 13:48 |
| *** dasm|off is now known as dasm | 13:59 | |
| opendevreview | Cedric Jeanneret proposed openstack/project-config master: Ensure NetworkManager doesn't override /etc/resolv.conf https://review.opendev.org/c/openstack/project-config/+/865433 | 14:12 |
| opendevreview | Cedric Jeanneret proposed opendev/base-jobs master: Ensure NetworkManager will not override /etc/resolv.conf https://review.opendev.org/c/opendev/base-jobs/+/865428 | 14:16 |
| opendevreview | Cedric Jeanneret proposed opendev/base-jobs master: Ensure NetworkManager will not override /etc/resolv.conf https://review.opendev.org/c/opendev/base-jobs/+/865428 | 14:23 |
| *** ysandeep__ is now known as ysandeep|out | 14:50 | |
| *** dviroel is now known as dviroel|lunch | 15:07 | |
| *** soniya29|afk is now known as soniya29 | 15:48 | |
| *** dviroel_ is now known as dviroel | 16:15 | |
| *** marios is now known as marios|out | 16:31 | |
| *** yadnesh is now known as yadnesh|away | 16:41 | |
| *** jpena is now known as jpena|off | 17:30 | |
| clarkb | fungi: is there a maxking PR for the urls.py update yet? I can go thumbs up it or whatever github wants to bring attention with. Also I can write it if that helps (I alredy have a fork setup in githu and all that) | 17:51 |
| opendevreview | Vishal Manchanda proposed zuul/zuul-jobs master: [DNM] Change browser name chromium-browser->chromium for debian bullseye https://review.opendev.org/c/zuul/zuul-jobs/+/865457 | 18:00 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Add support for nodejs testing on Debian https://review.opendev.org/c/zuul/zuul-jobs/+/865459 | 18:10 |
| fungi | clarkb: there is not, no. been sidelined by other work unfortunately. if you're up for proposing it i won't object, you likely already have a personal fork to push the pr from | 18:33 |
| clarkb | ya I've got all that set up already so its a few steps easier | 18:34 |
| clarkb | fungi: https://github.com/maxking/docker-mailman/pull/580 | 18:42 |
| clarkb | checkout the job list on https://review.opendev.org/864468 | 18:45 |
| clarkb | I wonder if that is a record for us | 18:45 |
| clarkb | ~91 jobs | 18:46 |
| fungi | whoa | 18:47 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Improve launch-node deps and fix script bugs https://review.opendev.org/c/opendev/system-config/+/865320 | 19:44 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Improve launch-node deps and fix script bugs https://review.opendev.org/c/opendev/system-config/+/865320 | 19:47 |
| fungi | clarkb: when you have time, the lists01 inventory and dns additions are pending approval under topic:mailman3 | 19:48 |
| fungi | already got +2 from ianw | 19:48 |
| fungi | new major versions of flake8 and pyflakes today, minor update to pycodestyle (formerly pep8), and yet another patch update for setuptools | 19:55 |
| fungi | it's a python community holiday tradition | 19:55 |
| clarkb | fungi: and we don't expect email to be sent right? | 20:03 |
| clarkb | on the test nodes we caught all outbound email but won't do that on this node | 20:03 |
| fungi | clarkb: we haven't imported any subscribers yet, so there should be no digests to send | 20:04 |
| clarkb | and it doesn't seem to alert list owners when we create the lists right? oh also before we land this, are there any list cleanups we need to do. I should've mentioned that when we landed the previous changes | 20:04 |
| clarkb | basically is our list of lists in sync with mm2 to reduce pruning needed later | 20:04 |
| clarkb | the summit program committee list for example got renamed? | 20:05 |
| fungi | i think i also renamed it in the patch? | 20:05 |
| fungi | checking | 20:05 |
| clarkb | oh it looks like we'll only configure opendev lists to start. The others are commented out | 20:06 |
| clarkb | which means lists.openinfra.dev syncing isn't critical | 20:07 |
| fungi | yeah, but also i did update summit-programming-committee to summit-track-chairs in the commented one too | 20:07 |
| fungi | and in the test version | 20:07 |
| fungi | so that all matches still | 20:07 |
| clarkb | last thing I'm double checking is letsencrypt which I don't see config for in our host vars | 20:08 |
| fungi | oh, i thought we added that when we did the test one | 20:08 |
| clarkb | maybe? I don't see it in the role | 20:08 |
| fungi | does it need to be done separately for production? | 20:08 |
| clarkb | oh because that is done by groups? | 20:08 |
| fungi | lists[0-9]*.opendev.org is in the letsencrypt list in inventory/service/groups.yaml | 20:09 |
| clarkb | ya and we use the mailman3 group to specify the cert names | 20:10 |
| clarkb | I'm now trying to figure out if we need dns updates or if this will fail | 20:10 |
| clarkb | I think it willwork since we already set acme records for the current prod server and those should work here as well | 20:11 |
| clarkb | fungi: I do notice that lists01.opendev.org is not in the borg backup group | 20:11 |
| clarkb | our test server, lists99, is | 20:11 |
| clarkb | I've approved the dns update and +2'd the inventory update. I'm not approving the inventory update as I need to step out now for lunch but feel free to if you can watch it | 20:12 |
| fungi | clarkb: https://opendev.org/opendev/system-config/src/branch/master/inventory/service/groups.yaml#L29 | 20:12 |
| fungi | am i misreading that entry? | 20:13 |
| clarkb | oh heh I was looking for it below where the other entry is | 20:13 |
| clarkb | lists.openstack.org | 20:13 |
| fungi | d'oh! | 20:13 |
| clarkb | so ya don't even need a followup for that one | 20:13 |
| fungi | thanks! | 20:14 |
| opendevreview | Merged opendev/zone-opendev.org master: Add lists01 to DNS https://review.opendev.org/c/opendev/zone-opendev.org/+/865360 | 20:15 |
| *** dviroel is now known as dviroel|afk | 21:25 | |
| *** rlandy|rover is now known as rlandy|out | 22:25 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: rax: remove identity_api_version 2 pin https://review.opendev.org/c/opendev/system-config/+/865351 | 22:35 |
| *** dasm is now known as dasm|off | 23:49 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!