| opendevreview | Ian Wienand proposed opendev/system-config master: launch-node : make into a small package https://review.opendev.org/c/opendev/system-config/+/861284 | 00:11 |
|---|---|---|
| *** rlandy is now known as rlandy|out | 00:26 | |
| opendevreview | Merged opendev/system-config master: Abstract name of bastion host for testing path https://review.opendev.org/c/opendev/system-config/+/858476 | 02:07 |
| opendevreview | Ian Wienand proposed opendev/system-config master: launch-node : make into a small package https://review.opendev.org/c/opendev/system-config/+/861284 | 02:45 |
| opendevreview | Ian Wienand proposed opendev/system-config master: launch-node : make into a small package https://review.opendev.org/c/opendev/system-config/+/861284 | 03:20 |
| opendevreview | Merged opendev/system-config master: bootstrap-bridge: use abstracted hostname https://review.opendev.org/c/opendev/system-config/+/861031 | 03:33 |
| *** Guest0 is now known as osmanlicilegi | 04:11 | |
| opendevreview | Merged opendev/system-config master: Convert production playbooks to bastion host group https://review.opendev.org/c/opendev/system-config/+/858486 | 04:18 |
| ianw | ^ infra-prod-bootstrap-bridge still working after that (https://zuul.opendev.org/t/openstack/build/969af3d0071647b390e22a9ca9c081d7) | 05:11 |
| ianw | tomorrow i'll start a new bridge node and i think we're in a position to transition | 05:11 |
| *** marios is now known as marios|ruck | 05:13 | |
| opendevreview | Merged opendev/system-config master: Run a base test against "old" bridge https://review.opendev.org/c/opendev/system-config/+/860802 | 05:22 |
| *** jpena|off is now known as jpena | 07:39 | |
| *** rlandy|out is now known as rlandy | 10:34 | |
| *** dviroel|out is now known as dviroel | 11:46 | |
| *** rlandy is now known as rlandy|brb | 12:38 | |
| *** rlandy|brb is now known as rlandy | 12:51 | |
| *** dasm|off is now known as dasm | 13:28 | |
| *** dasm is now known as dasm|rover | 13:28 | |
| *** rcastillo_ is now known as rcastillo | 13:30 | |
| fungi | #status log Pruned backups on backup02.ca-ymq-1.vexxhost, reducing volume utilization by 36% (from 93% to 57%) | 13:32 |
| opendevstatus | fungi: finished logging | 13:32 |
| clarkb | thank you | 15:09 |
| fungi | clarkb: not urgent, but when you have a chance i believe you had some feedback in irc for https://review.opendev.org/859977 which hasn't made it into review comments | 15:13 |
| clarkb | hrm I thought I posted them. Maybe I did it on the ACL change only | 15:14 |
| clarkb | ya I think one of the comments from the acls applies here so I've posted that. Otherwise I Think this is good | 15:16 |
| fungi | awesome, thanks | 15:19 |
| *** dviroel is now known as dviroel|lunch | 15:21 | |
| clarkb | re the python image build stuff. Someone left a comment on my pip bug that they do similar but instead install everything using --user and then copy $USER/.local to the prod iamge. This hsa the benefit of not needing to copy the wheel files too | 15:22 |
| clarkb | I havne't been able to come up with a good reason why that wouldn't work. But it is a much bigger/different change for us. In particular we expect things to install globally so would hae to update an image at a time if we do that | 15:23 |
| clarkb | infra-root https://review.opendev.org/c/opendev/system-config/+/862374 is a node cleanup change for gitea-lb01 and jvb02. Once they are out of the inventory we can ask the clouds to delete them | 15:46 |
| *** marios|ruck is now known as marios|out | 15:49 | |
| frickler | seems pypi is going bonkers once again. devstack gate fails like e.g. https://zuul.opendev.org/t/openstack/build/9ea76a6e8d25421b842b3c0fd85ca0a0 and also in ironic | 16:10 |
| frickler | actually the whole buildset pretty much exploded https://zuul.opendev.org/t/openstack/buildset/b8ba06fc11804ea6848522751ab29ca3 | 16:11 |
| JayF | the output was reading like a new pip release that turned on be_even_more_breaky=true or something lol | 16:11 |
| clarkb | oof. That does indeed look like the stale fallbck index data | 16:11 |
| clarkb | JayF: if this is the issue we know about its on the server side. Pypi is fronted with a CDN. If that CDN fails to fetch necessary info from the backend it fallback to a backup backend. The problem is the backup backend isn't kept up to date and you get indexes from months ago | 16:12 |
| JayF | ah, that makes sense | 16:13 |
| clarkb | OpenStack in particular is affected because constraints (for master) are kept up to date with recent releases | 16:13 |
| clarkb | I think it would be great if someone from openstack would file a bug arguing that it is better to fail upfront than serve stale data. The reason is who knows how many security bugs you are installing transparently if you don't use constraints or a similar mechanism | 16:14 |
| clarkb | But I don't have the energy to argue with them that a fundamental piece of their architecture is falwed | 16:14 |
| JayF | I might know the right ears to bend about that. | 16:14 |
| clarkb | when we first ran into this it took us a fair bit of time to even convince them there was a problem. Once we convinced them there was a problem days later they discovered the backup backed had run out of disk a while back and that was why it was super stale... | 16:16 |
| clarkb | you can try running an HTTP PURGE (non standard) request against the project indexes that have this issue which is supposed to tell the CDN to repopulate from the backend (part of the problem here is once it caches from the stale backend it will keep serving that bad data aiui) | 16:18 |
| clarkb | `curl -X PURGE https://pypi.org/simple/os-testr` for frickler's example | 16:19 |
| *** dviroel|lunch is now known as dviroel | 16:19 | |
| fungi | i got a socket timeout pulling a package from pypi at home a few hours ago | 16:19 |
| fungi | and https://status.python.org/ shows a significant error spike on the graphs | 16:20 |
| corvus | i'm looking into some nodepool stuff and happened to notice that we are getting a lot of "No valid host was found" errors from nova on inmotion | 16:30 |
| corvus | i don't plan to do anything else with that. just an fyi in case someone wants to look into tuning that cloud some more. | 16:30 |
| corvus | (the graph shows we're pretty consistently topping out at 50% of our expected capacity) | 16:31 |
| fungi | thanks, sounds like we probably leaked some instances or something | 16:35 |
| clarkb | the last time we had that in inmotion the problem was related to the placement service not being able to allocate to full capacity because of leaked record for nodes long since deleted. I don't recall the exact thing | 16:35 |
| clarkb | melwitt helped with it last time. And iirc it was a known nova/placement issue | 16:35 |
| clarkb | https://docs.openstack.org/nova/latest/admin/troubleshooting/orphaned-allocations.html link to that page was purple not blue so probably related | 16:36 |
| melwitt | ++ I was about to grab that ^ to link you | 16:36 |
| clarkb | I can probably take a look at that a bit later today | 16:38 |
| clarkb | Just need to work through a bit more of my todo list first :) | 16:38 |
| corvus | cool. that sounds consistent with what i'm seeing. | 16:39 |
| clarkb | related infra-root https://etherpad.opendev.org/p/fAVaSBNXEzwsMpfcrKz3 is a draft email I'm working on after discussions happened at the ptg (and elsewhere) that we/I were not included in. If you think that this email is a good productive step I'd like to send it out today | 16:39 |
| opendevreview | Artem Goncharov proposed openstack/project-config master: Add post-review pipeline https://review.opendev.org/c/openstack/project-config/+/859977 | 16:53 |
| clarkb | gtema: ^ I think that needs to be rebased onto the latest patchset of its parent | 17:01 |
| opendevreview | Artem Goncharov proposed openstack/project-config master: Add post-review pipeline https://review.opendev.org/c/openstack/project-config/+/859977 | 17:02 |
| gtema | thks clarkb, haven't noticed now | 17:02 |
| *** dviroel is now known as dviroel|biab | 18:56 | |
| *** dviroel|biab is now known as dviroel | 19:40 | |
| clarkb | the meeting agenda has grown quite long. But please add important topics to it and we'll do our best to get through them | 20:04 |
| opendevreview | Ildiko Vancsa proposed openstack/project-config master: Add starlingx-gerritbot config https://review.opendev.org/c/openstack/project-config/+/862534 | 20:40 |
| ianw | is there a storyboard container? | 20:52 |
| clarkb | ianw: https://opendev.org/opendev/storyboard/src/branch/master/Dockerfile yes but I'm not sure how much exercise it gets | 20:54 |
| ianw | i guess also bugzilla probably isn't an option -- reminded me of https://lwn.net/Articles/910740/ | 20:55 |
| ianw | at least we're not starting from a base of sending patches via email to each other :) | 20:55 |
| fungi | python 3.11.0 was officially tagged an hour or so ago | 21:10 |
| fungi | https://docs.python.org/3.11/whatsnew/3.11.html | 21:10 |
| fungi | (compiling now) | 21:12 |
| clarkb | fungi: a good time to remind people to review https://review.opendev.org/q/status:open+(topic:use-new-python+OR+topic:docker-cleanups) so that we have room for python3.11 images (though I think you have reviewed them?) | 21:21 |
| clarkb | ok infra meeting agenda will go out in about 40 minutes or so. Please add any other updates nowish | 21:21 |
| fungi | i have nothing new to add | 21:28 |
| opendevreview | Merged opendev/grafyaml master: Fix default handling with newer voluptious https://review.opendev.org/c/opendev/grafyaml/+/861484 | 21:31 |
| opendevreview | Merged opendev/grafyaml master: Update Docker images to python 3.10 https://review.opendev.org/c/opendev/grafyaml/+/861475 | 21:31 |
| ianw | clarkb: ^ thanks, i think i reviewed all those | 21:31 |
| opendevreview | Merged opendev/statusbot master: Update container images to python3.10 https://review.opendev.org/c/opendev/statusbot/+/861476 | 21:32 |
| ianw | i'm planning on getting on with the bridge update today. i've abstracted everything i can think of relating to that | 21:33 |
| *** rlandy is now known as rlandy|out | 21:42 | |
| clarkb | ianw: are there more changes that need reviewing related to that? | 21:47 |
| ianw | not at the moment, i'm at the point of bringing up the server | 22:04 |
| *** dasm|rover is now known as dasm | 22:13 | |
| *** dasm is now known as dasm|off | 22:14 | |
| ianw | clarkb: 400: Client Error for url: https://dfw.servers.api.rackspacecloud.com/v2/ ..., Bad networks format | 22:56 |
| ianw | is that one you saw? | 22:56 |
| clarkb | ianw: yes I think so. I used my old venv to work around that iirc | 22:56 |
| clarkb | though I used a paramiko venv so maybe not | 22:57 |
| clarkb | ianw: where are you launching launch.py from? | 22:57 |
| ianw | ok, on that topic; https://review.opendev.org/c/opendev/system-config/+/861284 is a place we could codify that a bit more | 22:57 |
| clarkb | is it the /opt path that the README points too? Ihave/had a change to update that as that location is stale | 22:57 |
| ianw | bastion host | 22:57 |
| clarkb | oh wait I think it was corvus that hit that. I launched in vexxhost beacuse that is where gitea is hosted | 22:58 |
| ianw | but just command line, i can create an env with a older openstacksdk :/ | 22:58 |
| clarkb | I think corvus has a venv that should work | 22:58 |
| corvus | ~root/corvus-venv is what i used | 22:59 |
| *** dviroel is now known as dviroel|out | 23:00 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: launch-node : make into a small package https://review.opendev.org/c/opendev/system-config/+/861284 | 23:03 |
| ianw | thanks, that is working. that venv has 0.41 which i pinned in ^ | 23:03 |
| clarkb | re Gerrit 3.6 I've just responded to a thread that indicates conflicts with checking might be broken in 3.6 | 23:29 |
| clarkb | (somethingto be aware of as we plan to upgrade, hopefully it isn't actually broken) | 23:29 |
| opendevreview | Ian Wienand proposed opendev/zone-opendev.org master: Add bridge01.opendev.org https://review.opendev.org/c/opendev/zone-opendev.org/+/862543 | 23:40 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Switch testing bridge name to bridge01.opendev.org https://review.opendev.org/c/opendev/system-config/+/861112 | 23:40 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Add bridge01.opendev.org to inventory https://review.opendev.org/c/opendev/system-config/+/862544 | 23:41 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Add bridge01.opendev.org to inventory https://review.opendev.org/c/opendev/system-config/+/862544 | 23:46 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Switch testing bridge name to bridge01.opendev.org https://review.opendev.org/c/opendev/system-config/+/861112 | 23:46 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!