| opendevreview | Merged openstack/diskimage-builder master: Cleanup uses of DIB_IMAGE_ROOT_FS_UUID variable https://review.opendev.org/c/openstack/diskimage-builder/+/905556 | 00:58 |
|---|---|---|
| opendevreview | Merged openstack/diskimage-builder master: Bump upper version of flake8 https://review.opendev.org/c/openstack/diskimage-builder/+/909336 | 00:58 |
| *** promethe- is now known as prometheanfire | 01:30 | |
| *** liuxie is now known as liushy | 02:51 | |
| opendevreview | Birger J. Nordølum proposed openstack/diskimage-builder master: feat: add almalinux-container element https://review.opendev.org/c/openstack/diskimage-builder/+/883855 | 06:48 |
| opendevreview | Thierry Carrez proposed opendev/irc-meetings master: Large Scale SIG now meets monthly https://review.opendev.org/c/opendev/irc-meetings/+/911416 | 09:36 |
| opendevreview | Birger J. Nordølum proposed openstack/diskimage-builder master: feat: add almalinux-container element https://review.opendev.org/c/openstack/diskimage-builder/+/883855 | 09:54 |
| opendevreview | Merged opendev/irc-meetings master: Large Scale SIG now meets monthly https://review.opendev.org/c/opendev/irc-meetings/+/911416 | 10:07 |
| *** mrunge_ is now known as mrunge | 10:22 | |
| opendevreview | Jonathan Rosser proposed openstack/project-config master: Implement ironic-unmaintained-core group https://review.opendev.org/c/openstack/project-config/+/911576 | 13:12 |
| opendevreview | Jonathan Rosser proposed openstack/project-config master: Implement openstack-ansible-unmaintained-core group https://review.opendev.org/c/openstack/project-config/+/911576 | 13:13 |
| opendevreview | Jonathan Rosser proposed openstack/project-config master: Implement openstack-ansible-unmaintained-core group https://review.opendev.org/c/openstack/project-config/+/911576 | 13:23 |
| fungi | #status log Restarted the ptgbot container to force a reconnect to IRC since it came loose during a netsplit at 00:49 UTC | 13:56 |
| fungi | huh, looks like the statusbot may have met a similar fate | 13:57 |
| fungi | 0049 <-- opendevstatus (~opendevst@eavesdrop01.opendev.org) has quit (resistance.oftc.net larich.oftc.net) | 13:57 |
| fungi | yep. i'll restart it too | 13:57 |
| *** tosky_ is now known as tosky | 13:57 | |
| fungi | #status log Restarted the ptgbot and statusbot containers forcing them to reconnect to IRC since both came loose during a netsplit at 00:49 UTC | 13:59 |
| opendevstatus | fungi: finished logging | 14:01 |
| fungi | the meetbot doesn't appear to have been impacted, fwiw | 14:06 |
| frickler | yes, successfully started kolla meeting right now | 14:08 |
| *** dansmith_ is now known as dansmith | 15:02 | |
| TheJulia | Any chance I could get some insight from the opendev wizzards? Specifically curious why https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/909939 doesn't seem to be triggering anything | 15:06 |
| TheJulia | in zuul, that is. | 15:06 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Try switching Rackspace DFW to an API key https://review.opendev.org/c/openstack/project-config/+/911381 | 15:06 |
| jrosser_ | TheJulia: is the depends-on abandoned? | 15:07 |
| fungi | looks like it is indeed, yes | 15:07 |
| fungi | good eye jrosser_! | 15:07 |
| TheJulia | That would do it | 15:07 |
| TheJulia | Thanks guys, I'll make more coffee now | 15:07 |
| fungi | it's late enough in the morning here i think i'd better switch to tea | 15:08 |
| clarkb1 | we got the email from rax finally which is the final confirmation this is coming for our accounts too | 16:39 |
| fungi | it's heading right for us! | 16:40 |
| clarkb1 | fungi: I think corvus was hoping to avoid approving the nodepool rackspace auth change until after we confirmed it was working on nl01? | 16:41 |
| clarkb1 | we have a couple minutes to remove the +A if so | 16:41 |
| clarkb1 | but also its easy enough to revert so meh | 16:42 |
| clarkb1 | but we could use the artifact for the launcher from here: https://zuul.opendev.org/t/zuul/build/e02cb1c8119b4a00bb6c4a02e13585a4/artifacts to test it before merging | 16:42 |
| fungi | oh, sorry! i think i forgot that discussion happened since it wasn't mentioned in the review comments. anyway i unapproved it | 16:42 |
| fungi | clarkb1: technically zuul will upload a container image to quay before it decides not to merge, it just won't tag it in promote | 16:43 |
| fungi | but we could also grab it from the intermediate registry sure | 16:43 |
| clarkb1 | fungi: ya so you can pull it from quay or the insecure registry | 16:43 |
| clarkb1 | I need to figure out this 1 tail | 16:43 |
| fungi | you're number one! | 16:44 |
| fungi | er, you're the duke of new york, you're a-number-one? | 16:44 |
| fungi | (sorry, couldn't help that one) | 16:44 |
| clarkb1 | fungi: other things to note: I'm not sure if openstacksdk will get grumpy on the other launchers trying to read the new clouds.yaml | 16:44 |
| clarkb1 | I think it only loads the clouds.yaml info when you ask it to set up a cloud connection so its probably fine | 16:45 |
| clarkb1 | but if not we can put all the nodepool nodes in the emergency file | 16:45 |
| clarkb1 | oh and the infra-prod-service nodepool job runs hourly so we may have to put at least nl01 into the emergency file while testing | 16:45 |
| clarkb1 | (to prevent it from reverting to the prod released image and restarting the container under us) | 16:45 |
| clarkb1 | in any case I think the system-config change goes in first, then we have to coordinate the project-config update with a launcher update on nl01. Might have some slight bumps along the way but nothing we can sort out | 16:46 |
| fungi | we could also test out the launch_node script before moving onto nodepool tests | 16:47 |
| clarkb1 | oh ya that too | 16:49 |
| clarkb1 | since that is only depending on the system-config update | 16:49 |
| fungi | certainly if things break there, then that's a signal to pause and work out why before we move onto nodepool | 16:51 |
| clarkb1 | ya though note the volume stuff is unahppy already. I would just do a simple server boot and not worry about volumes | 16:53 |
| clarkb1 | I think that gets us 90% of what we need anyway | 16:54 |
| *** clarkb1 is now known as clarkb | 16:55 | |
| clarkb | fungi: I'm not going to be around for the next several hours (maybe longer than that actually), but I'm happy if you want to proceed with the system-config change and start testing some stuff | 17:01 |
| clarkb | fungi: also on the git-review side of things I think we can approve https://review.opendev.org/c/opendev/git-review/+/910268 and the rebase cleanup change. Then maybe make a release then do a followup release for the other changes if you want to get those in too? | 17:01 |
| fungi | sounds good. have fun! | 17:06 |
| fungi | i'll get the launch script tested and revisit git-review changes if nothing major comes up | 17:07 |
| clarkb | thanks! | 17:07 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Move gerrit replication waiting queue aside during project renames https://review.opendev.org/c/opendev/system-config/+/911622 | 17:20 |
| clarkb | this is a quick stab at moving waiting queues aside during the rename process. I'm not sure I like the approch there (maybe we should just delete the queue instead), btu wanted to get something up now so I don't forget also feedback welcome | 17:21 |
| clarkb | that playbook should be tested by the system-config-run-* jobs for gerrit too which is nice | 17:21 |
| fungi | clarkb: not sure if you really meant to mv instead of cp there | 17:23 |
| clarkb | fungi: I did! | 17:23 |
| clarkb | bad copy pasta from the task above | 17:24 |
| fungi | got it | 17:24 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Move gerrit replication waiting queue aside during project renames https://review.opendev.org/c/opendev/system-config/+/911622 | 17:25 |
| opendevreview | Merged opendev/git-review master: Don't make hook script read-only https://review.opendev.org/c/opendev/git-review/+/910268 | 17:44 |
| opendevreview | Merged opendev/system-config master: Transition to Rackspace API keys https://review.opendev.org/c/opendev/system-config/+/911164 | 18:02 |
| opendevreview | Merged opendev/git-review master: Vendor a copy of Gerrit's commit-msg Git hook https://review.opendev.org/c/opendev/git-review/+/910275 | 18:06 |
| fungi | okay, clouds.yaml additions just deployed to bridge, so i'll start trying things out there | 18:30 |
| fungi | launch-node run underway using the new --cloud=opendevci-rax instead of the old --cloud=openstackci-rax and it seems to be interacting with the api normally | 18:32 |
| fungi | completed without error, and i'm able to ssh into the resulting server instance normally | 18:40 |
| fungi | the /usr/local/bin/openstack on bridge is symlinked to /usr/launcher-venv/bin/openstack and is able to openstack server list and openstack server delete just fine too | 18:43 |
| fungi | again with the new --os-cloud=opendevci-rax instead of the old --os-cloud=openstackci-rax | 18:43 |
| Clark[m] | Woot | 18:43 |
| fungi | openstack volume list is working fine too | 18:44 |
| fungi | as a secondary check, i can openstack server list with the new --os-cloud=opendevzuul-rax (instead of the old --os-cloud=openstackjenkins-rax) too | 18:46 |
| fungi | and i see nodepool nodes in dfw for example | 18:46 |
| fungi | so i think this is as thorough of a test as we need to do for this phase | 18:47 |
| fungi | new configuration parameters seem right, api keys are also apparently correct | 18:47 |
| opendevreview | Merged opendev/git-review master: Don't keep incomplete rebase state by default https://review.opendev.org/c/opendev/git-review/+/850061 | 18:49 |
| fungi | how did we want to go about nodepool testing for the next phase? put all nodepool servers except nb01 and nl01 into emergency disable, let 911381 deploy, add nb01 and nl01 to emergency disable, pull experimental nodepool images onto them and restart the containers, then watch for errors? | 18:49 |
| fungi | infra-root: we're talking about tagging the current state of git-review's master branch for a new release (probably 2.4.0). the changes since the last release (2.3.1) can be found here for your convenience: https://paste.opendev.org/show/bOoHHxUEW8odCDEetUXO/ | 19:00 |
| fungi | judging from https://docs.opendev.org/opendev/git-review/latest/releasenotes.html only two of those changes came with release notes. which others do you think we should add release notes for? | 19:01 |
| fungi | (note i highlighted infra-root to get their attention, but anyone who cares is welcome to provide input) | 19:02 |
| fungi | "feat(cmd): add hashtag implementation" and "Add message option" probably at a minimum? "Use GIT_SSH for the SSH executable" maybe as well? i'm on the fence about "Warn rather than fail if HEAD already exists on the remote" and we probably don't need to call out changes like "Add --wip as an alias to --work-in-progress" or "Add classifiers for Python 3.10 and 3.11" but i suppose we can | 19:04 |
| fungi | let me know which of those you think warrant their own release notes and i'm happy to write them up in a catch-up change we can expedite | 19:05 |
| fungi | i'll also try to do a better job in the future of asking contributors to include release notes (or amending their changes myself) | 19:06 |
| fungi | also it looks like i used the wrong markup in my release notes. should have wrapped commands and options in doubled backticks rather than single. i can fix those up in the same change | 19:09 |
| JayF | oooh fixing git-review to use GIT_SSH is going to be great | 19:27 |
| JayF | that needs to be release noted for sure | 19:27 |
| JayF | going to be a HUGE help for people running on WSL/Windows due to the shenanigans (windows ssh can't talk to mingw/git ssh agents) | 19:28 |
| fungi | cool, so that's at least 3 more release notes to add to the existing 2 | 19:31 |
| corvus | fungi: to that i would say "yes" to "wip" and "no" to classifiers | 19:40 |
| fungi | okay, can add a simple mention of the --wip alias | 19:41 |
| fungi | yes on one no on two | 19:41 |
| corvus | (my test for a release note is "might a user change their behavior or usage of the program based on reading this") | 19:41 |
| corvus | fungi: yes no two! | 19:41 |
| fungi | rodger rodger | 19:42 |
| corvus | https://www.youtube.com/watch?v=zSdObe8VRDE&t=60s | 19:43 |
| fungi | in the past we've sometimes mentioned when we started testing some tools on new python versions, which is why i mentioned the classifiers (but sort of embarrassing to have a release that proclaims it's now tested on python 3.11 when 3.12 has been out for a while) | 19:44 |
| fungi | ah, yes, the prop 242 misspelling classic | 19:45 |
| corvus | yeah, my feeling is that if it starts or stops working on some python version due to effort on our part, that warrants a note, but "still works" doesn't make the cut, mostly so that notes don't get long and boring. i don't feel strongly about it, but i have thought about it and those are my thoughts. :) | 19:45 |
| fungi | agreed | 19:45 |
| fungi | we did, for example, add a release note when we dropped python 3.5 support in git-review | 19:46 |
| fungi | but this version is still tested on 3.6 so we're fine there | 19:46 |
| frickler | fungi: if this can wait until tomorrow, I'll be happy to take a look at those, too. out for today now | 19:52 |
| fungi | frickler: yeah, i don't think it's super urgent | 19:55 |
| diablo_rojo | Hello party people :) | 20:34 |
| diablo_rojo | fungi: Clark I would like to introduce you to aquin and his professor profcorey | 20:34 |
| diablo_rojo | We are getting aquin setup to work on horizon and we are hitting that super awesome issues where it tells us to manually setup a remote | 20:35 |
| diablo_rojo | We double checked usernames, sshkeys, icla but still are having no success. | 20:35 |
| diablo_rojo | Any ideas? | 20:35 |
| JayF | profcorey: o/ hey Corey | 20:36 |
| JayF | diablo_rojo: is this windows/wsl by any chance? | 20:37 |
| diablo_rojo | JayF: it is :) | 20:37 |
| JayF | So a possible thing you're hitting is that git-review and the scp command git-review runs may be using different SSH binaries | 20:37 |
| JayF | if you run with enough -vvvv you may get some useful output to share | 20:38 |
| JayF | if that's what's happening, I think you can manually add the hook and it'll fix things up | 20:38 |
| diablo_rojo | We will give the -vvvv a try! | 20:39 |
| profcorey | Hi Jay! | 20:41 |
| aquin | https://paste.opendev.org/show/beCdECA65ygloteaB9aL/ | 20:41 |
| diablo_rojo | JayF: ^ | 20:41 |
| JayF | > ssh://aquintero93@review.opendev.org:29418/opendev/sandbox.git did not work. Description: Host key verification failed. | 20:42 |
| JayF | is the meaningful bit | 20:42 |
| JayF | what does review.opendev.org resolve to, there? | 20:42 |
| aquin | I have a cloned sandbox repo | 20:44 |
| JayF | I have to AFK for a bit (probably 20ish minutes); but I'd suggest: | 20:44 |
| JayF | 1) Validating that review.opendev.org resolves where you'd expect; mine resolves to review02.opendev.org / 199.204.45.33 | 20:44 |
| JayF | 2) Assuming ^ that is true (and only if that looks like you expect), check ~/.ssh/known_hosts and remove anything relating to opendev or that IP | 20:44 |
| JayF | 3) Try again | 20:44 |
| diablo_rojo | JayF: sounds good! Thank you! | 20:44 |
| fungi | diablo_rojo: aquin: profcorey: hi! apologies, you caught me in the middle of an early dinner | 20:51 |
| fungi | i can check the server for errors related to the aquintero93 user | 20:52 |
| fungi | interesting, no matches for any aquintero93 user in the log | 20:53 |
| diablo_rojo | fungi: thank you! We are working through what JayF suggested. | 20:53 |
| diablo_rojo | Huh. | 20:53 |
| fungi | but since it's a "Host key verification failed" error, it probably never got as far as trying to authenticate | 20:53 |
| fungi | usually, openssh will let you know when the host key it sees from the server is unknown, and will prompt you to remember it | 20:54 |
| diablo_rojo | He doesn't actually have a knownhosts file in the dir, just the ssh keys he created and shared with gerrit. | 20:56 |
| fungi | it's possible running `ssh -p 29418 aquintero93@review.opendev.org` would produce a more useful error | 20:56 |
| diablo_rojo | On it. | 20:57 |
| diablo_rojo | Oh that looks much happier | 20:58 |
| fungi | did that get a "Welcome to Gerrit Code Review" banner? | 20:59 |
| diablo_rojo | Correct | 20:59 |
| fungi | did it prompt to remember the host key, or no? | 20:59 |
| diablo_rojo | It did | 20:59 |
| fungi | i wonder if running `git review -s` would "just work" now | 21:00 |
| diablo_rojo | aquin: did you already delete the sandbox repo? | 21:02 |
| diablo_rojo | If not give that a try? | 21:02 |
| fungi | maybe there's something off with ssh's ability to cache host keys when called from git on wsl. as JayF mentioned earlier, https://review.opendev.org/c/opendev/git-review/+/890043 may be a useful workaround for situations where the GIT_SSH environment variable has been set, should be included in 2.4.0 when we release it (likely later this week) | 21:02 |
| aquin | Ok I believed runnign 'git review -s' again simply woked | 21:03 |
| aquin | I'm getting a: 'Creating a git remote called 'gerrit' that maps to: | 21:04 |
| aquin | ssh://aquintero93@review.opendev.org:29418/opendev/sandbox.git' | 21:04 |
| fungi | yeah, that's what it was supposed to do before | 21:04 |
| fungi | in that case, this is probably "fixed" for the environment you've got set up, but if you try to do it again from a new computer you may run into a similar situation | 21:04 |
| diablo_rojo | Huzzah! | 21:05 |
| aquin | Ok perfect thanks a lot! | 21:05 |
| fungi | out of curiosity, what does it say if you enter `echo $GIT_SSH` (without the quotes)? | 21:05 |
| fungi | a blank line, or does it spit out a path to a git executable? | 21:05 |
| fungi | also, what distro is your wsl using? ubuntu (i think that's the default if you don't specify anything when building it)? debian? something else? | 21:07 |
| aquin | fungi: I am using Ubuntu in my WSL | 21:12 |
| fungi | thanks. and does $GIT_SSH appear to be set or no? | 21:13 |
| aquin | fungi: When I use the 'echo' command above, it produces PS C:\Users\aquin\OneDrive\Documents\Projects\sandbox> | 21:13 |
| fungi | okay, so just another prompt, no actual value for that environment variable. in that case change 890043 probably isn't going to help avoid that in the future on its own | 21:14 |
| aquin | Sorry, I should be good to go now though right? | 21:19 |
| diablo_rojo | aquin: so now you can make a branch, make a file add some pizzaz to it, git add it and git commit it and git push. | 21:26 |
| fungi | aquin: absolutely, but if you run into more problems please feel free to reach out here | 21:26 |
| fungi | there are often people around, but we're a global community living in a variety of timezones, so sometimes it may take a while to get an answer, just be patient | 21:27 |
| aquin | diablo_rojo: Sounds good will do! I have to pivot over to some meetings and deliverables at work, but as soon as I get off I will tackle this! | 21:29 |
| diablo_rojo | Sounds good! | 21:30 |
| aquin | fungi: Thank you! | 21:30 |
| diablo_rojo | You can ping me here later or on my other nick diablo_rojo_phone and I will try to help out if I can! | 21:30 |
| jrosser | fungi: mtu is a usual trip hazard with ssh in wsl | 21:31 |
| Clark[m] | fungi: trying to catch up quickly. I think I convinced myself that we don't need to do anything outside nl01. The updated config won't affect them and if builders fail it isn't an urgent issue | 22:05 |
| Clark[m] | fungi: so I think we can land changes in whatever order works and restart nl01 on the intermediate image. Or maybe just approve the nodepool change after your testing on bridge | 22:06 |
| fungi | okay, so basically merge 911381 and, once it deploys, put nl01 in the emergency disable list and pull the experimental nodepool image there and restart onto it? | 22:06 |
| Clark[m] | Ya | 22:09 |
| Clark[m] | The builders will be angry for a bit but as mentioned not a big deal | 22:09 |
| Clark[m] | But if you want out then into the emergency file now and we can sort them out after | 22:10 |
| Clark[m] | nb01 and nb02. nb04 doesn't talk to rax dfw so will be unaffected | 22:11 |
| Clark[m] | Why would ssh under wsl have a different mtu than the windows host? | 22:11 |
| fungi | it's basically a virtual machine, and uses separate addressing on an internal switch that has to be encapsulated, from what i understand | 22:12 |
| Clark[m] | Huh there really shouldn't be a reason for tunneling a single special purpose VM | 22:13 |
| fungi | or at least the networking layer for it is virtualized by default | 22:13 |
| Clark[m] | Should be able to just route or NAT out | 22:13 |
| fungi | well, wsl can have an arbitrary number of machines too, i think | 22:13 |
| Clark[m] | You really only need the tunnel stuff when you want arbitrary addressing that may conflict with other addresses. Maybe they are just being extra cautious | 22:15 |
| fungi | i think there are also ways to change the networking model it uses, not sure | 22:18 |
| fungi | i'd read that it was fairly limited on windows 10 but got more flexible on 11 | 22:19 |
| *** kopecmartin_ is now known as kopecmartin | 22:50 | |
| clarkb | fungi: re git review release notes corvus' rule makes sense to me | 22:52 |
| fungi | cool. if you spot any other changes that could use release notes, let me know | 22:53 |
| clarkb | it just occurred to me that the other place that MFA will impact us is DNS | 22:55 |
| clarkb | thsi is far less of an issue these days since there are few oepnstack.org records | 22:55 |
| clarkb | I'll bring it up on the foundation side too since this is one of those weird shared situations | 22:56 |
| clarkb | I believe that account is shared but we have our own credentials for it so amybe we just need our own MFA setup for it and then we're good. That doesn't address configuring dns records automatically but as mentioned this is less and less cmommon so maybe we can live with that | 22:58 |
| fungi | actually the account is not shared. we have our own dedicated account, the tenant/project is shared though | 23:00 |
| fungi | though i guess that's what you said, more or less | 23:00 |
| clarkb | ya we login using "infra" credentials but then have access to the shared openstack.org domain | 23:00 |
| clarkb | however that ends up being structured. I suppose the most likely scenario there is we set up MFA for the credentials we use and that doesn't affect a nyone else's access to the resources | 23:01 |
| fungi | that's what i'm expecting will happen | 23:01 |
| opendevreview | Birger J. Nordølum proposed openstack/diskimage-builder master: feat: add almalinux-container element https://review.opendev.org/c/openstack/diskimage-builder/+/883855 | 23:07 |
| clarkb | fungi: I'm guessing that at this point we'll test nodepool stuff tomorrow? (thats fine just making sure I'm caught up on where we are at) | 23:11 |
| fungi | yeah, that's my expectation | 23:12 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!