| diablo_rojo | fungi, lol got it. Fixing now. | 00:02 |
|---|---|---|
| openstackgerrit | Kendall Nelson proposed openstack/project-config master: Setup OpenInfra Channels https://review.opendev.org/c/openstack/project-config/+/774550 | 00:03 |
| diablo_rojo | Done! | 00:03 |
| fungi | thanks! | 00:03 |
| openstackgerrit | Jeremy Stanley proposed opendev/git-review master: Add missing -p/-P/-w/-W/--license to manpage https://review.opendev.org/c/opendev/git-review/+/774567 | 00:17 |
| *** tosky has quit IRC | 00:20 | |
| openstackgerrit | Merged openstack/project-config master: Setup OpenInfra Channels https://review.opendev.org/c/openstack/project-config/+/774550 | 00:32 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: borg-backup-server: add script for pruning borg backups https://review.opendev.org/c/opendev/system-config/+/774561 | 00:33 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: borg-backup-server: volume space monitor https://review.opendev.org/c/opendev/system-config/+/774564 | 00:33 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: gerrit: only backup accountPatchReviewDb https://review.opendev.org/c/opendev/system-config/+/774569 | 00:33 |
| clarkb | one thing I am realizing is that we'll want to do a prepass on the refs/users/ side of things to avoid the preferred email is missing when we then clean up external ids | 00:45 |
| clarkb | and if we do the refs/users/ side of things first we should end up without new errors | 00:46 |
| fungi | makes sense, yeah | 00:46 |
| clarkb | but looking at the external ids so far the vast majority really do seem to be there are 2-3 accounts with conflicting emails. Only one them was ever used. So we retire the other(s) to avoid preferred email missing errors then remove the conflicting external ids from the retired accounts | 00:47 |
| clarkb | the tricky bit is in deciding which account to retire as sometimes it is the newer one or the older one and in some cases both were used and so you have to apply some judgement to decide which is active now | 00:47 |
| clarkb | one thing I've used for that is looking up whether or not the internet claims people still work at the company that one or both of the accounts has an email address with | 00:48 |
| clarkb | in some cases I think the company doesn't even exist anymore :/ | 00:48 |
| *** openstackgerrit has quit IRC | 00:49 | |
| *** mlavalle has quit IRC | 00:53 | |
| *** rchurch has quit IRC | 00:57 | |
| *** openstackgerrit has joined #opendev | 00:58 | |
| openstackgerrit | Merged opendev/system-config master: Setup OpenInfra Channels https://review.opendev.org/c/opendev/system-config/+/774563 | 00:58 |
| *** rchurch has joined #opendev | 01:00 | |
| clarkb | this is a fun one. I've just found a pair of accounts where one pushed code the other did reviews. The preferred email in one matches the openid emails on the other and the preferred email on the other matches the openid emails from the first one | 01:00 |
| clarkb | its like the perfect set of criss crossed wires :/ | 01:00 |
| clarkb | And I've just written down "I don't know what to do with this situation" in my notes | 01:01 |
| clarkb | we could maybe flip the preferred emails so they match their openids then remove the conflicting mailto? that seems pretty heavy handed. | 01:01 |
| clarkb | Both accounts haven't been used in half a decade though so maybe we can just retire them both and if the user returns we sort it out with them? | 01:02 |
| clarkb | and now time to figure out dinner | 01:04 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: doc: update backup instructions https://review.opendev.org/c/opendev/system-config/+/774570 | 01:18 |
| *** openstackstatus has quit IRC | 01:20 | |
| *** openstack has joined #opendev | 01:22 | |
| *** ChanServ sets mode: +o openstack | 01:22 | |
| fungi | clarkb: yeah, i've run into that when trying to help folks resolve duplicate accounts in the past, but like you say, it may be easiest to just disable both and remove the conflicts from them | 01:27 |
| fungi | i would actually lean toward disabling the one which is being used for change uploading, because the user can always set a username and ssh key via the webui with the one they logged into for reviewing | 01:29 |
| fungi | though one missing piece of info we may want to incorporate is to cross-reference all of them against group memberships, since if one account in a conflict set is a member of a group that's probably the one which should be kept/fixed | 01:31 |
| *** DSpider has quit IRC | 01:43 | |
| *** diablo_rojo has quit IRC | 02:12 | |
| *** dviroel has quit IRC | 02:43 | |
| ianw | le sigh ... anyone know why bridge can't talk to rax any more? | 02:55 |
| ianw | Max retries exceeded with url: /v2.0/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fd79f6e8e90>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) | 02:55 |
| ianw | it works with 5.0.1 in my home directory, nto with 5.2.1 from the container | 02:59 |
| *** TheJulia has quit IRC | 03:26 | |
| *** walshh_ has quit IRC | 03:26 | |
| *** mwhahaha has quit IRC | 03:26 | |
| *** snbuback has quit IRC | 03:27 | |
| *** parallax has quit IRC | 03:27 | |
| *** CeeMac has quit IRC | 03:27 | |
| *** ildikov has quit IRC | 03:28 | |
| *** TheJulia has joined #opendev | 03:28 | |
| openstackgerrit | Merged opendev/system-config master: gerrit: only backup accountPatchReviewDb https://review.opendev.org/c/opendev/system-config/+/774569 | 03:28 |
| *** mwhahaha has joined #opendev | 03:29 | |
| *** ildikov has joined #opendev | 03:29 | |
| *** walshh_ has joined #opendev | 03:29 | |
| *** snbuback has joined #opendev | 03:31 | |
| *** CeeMac has joined #opendev | 03:31 | |
| *** parallax has joined #opendev | 03:31 | |
| ianw | i think we need to look at unattended-upgrades during node creation. i think it's installing every package completely separately and running more inbetween; it's been going for about 40 minutes | 03:33 |
| ianw | it recreated the initfs 10 times according to the logs | 03:35 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add refstack01.openstack.org https://review.opendev.org/c/opendev/system-config/+/774579 | 03:39 |
| fungi | ianw: urllib3 5.2.1? | 03:50 |
| ianw | sorry openstackclient | 03:51 |
| fungi | ahh | 03:51 |
| ianw | although i'm sure the dependencies installed are also many and varied | 03:51 |
| fungi | yeah, first suspicion is rax uses some older ssl parameters which newer <something> has decided is better to throw an opaque error about rather than use | 03:52 |
| fungi | what's the full url? could try passing it to requests.get() in the repl and seeing if it gives any more useful error | 03:57 |
| openstackgerrit | Merged opendev/system-config master: Deploy refstack with ansible docker https://review.opendev.org/c/opendev/system-config/+/705258 | 03:58 |
| fungi | mmm, trying it myself, and it's taking a while to return | 03:59 |
| fungi | Failed to discover available identity versions when contacting https://identity.api.rackspacecloud.com/v2.0/. Attempting to parse version from URL. | 03:59 |
| ianw | yeah, that look like it | 04:00 |
| fungi | ahh, there we go | 04:00 |
| fungi | it's obscured by a retry | 04:00 |
| fungi | however /v2.0/tokens presumably needs a post or something, so requests.get() just receives a 405 | 04:02 |
| openstackgerrit | Merged opendev/system-config master: Add refstack01.openstack.org https://review.opendev.org/c/opendev/system-config/+/774579 | 04:24 |
| *** ykarel has joined #opendev | 04:33 | |
| *** ysandeep|away is now known as ysandeep|rover | 04:48 | |
| *** whoami-rajat__ has joined #opendev | 05:13 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: refstack: add production image and deployment jobs https://review.opendev.org/c/opendev/system-config/+/774586 | 05:19 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: refstack: move non-private variables to public https://review.opendev.org/c/opendev/system-config/+/774587 | 05:19 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: borg-backup: save PIPESTATUS before referencing https://review.opendev.org/c/opendev/system-config/+/774588 | 05:23 |
| *** ykarel is now known as ykarel|mtg | 05:38 | |
| *** marios has joined #opendev | 06:10 | |
| *** ralonsoh has joined #opendev | 07:36 | |
| *** eolivare has joined #opendev | 07:40 | |
| *** fressi has joined #opendev | 07:55 | |
| *** sboyron has joined #opendev | 07:58 | |
| *** rpittau|afk is now known as rpittau | 07:59 | |
| *** hashar has joined #opendev | 08:00 | |
| *** slaweq has joined #opendev | 08:03 | |
| *** jpena|off is now known as jpena | 08:15 | |
| *** andrewbonney has joined #opendev | 08:19 | |
| *** fressi has quit IRC | 08:22 | |
| *** fressi has joined #opendev | 08:23 | |
| *** ykarel|mtg is now known as ykarel | 08:26 | |
| *** DSpider has joined #opendev | 08:43 | |
| *** ralonsoh has quit IRC | 09:02 | |
| *** tosky has joined #opendev | 09:10 | |
| *** marios has quit IRC | 09:21 | |
| *** marios has joined #opendev | 09:24 | |
| *** dtantsur|afk is now known as dtantsur | 09:45 | |
| dtantsur | Hi folks! storyboard gives HTTP 500 on login, is it known already | 09:51 |
| *** dviroel has joined #opendev | 10:58 | |
| *** guilhermesp has quit IRC | 11:10 | |
| *** icey has quit IRC | 11:10 | |
| *** guilhermesp has joined #opendev | 11:11 | |
| *** icey has joined #opendev | 11:13 | |
| *** ysandeep|rover is now known as ysandeep|brb | 11:38 | |
| *** hashar is now known as hasharAway | 11:44 | |
| *** calcmandan has quit IRC | 12:00 | |
| *** calcmandan has joined #opendev | 12:01 | |
| *** whoami-rajat__ has quit IRC | 12:08 | |
| *** andrewbonney has quit IRC | 12:08 | |
| *** guilhermesp has quit IRC | 12:08 | |
| *** ysandeep|brb is now known as ysandeep|rover | 12:09 | |
| *** dviroel has quit IRC | 12:09 | |
| *** whoami-rajat__ has joined #opendev | 12:10 | |
| *** andrewbonney has joined #opendev | 12:10 | |
| *** dviroel has joined #opendev | 12:10 | |
| *** guilhermesp has joined #opendev | 12:10 | |
| *** calcmandan has quit IRC | 12:10 | |
| *** calcmandan_ has joined #opendev | 12:14 | |
| *** jpena is now known as jpena|lunch | 12:18 | |
| *** calcmandan_ is now known as calcmandan | 12:18 | |
| dtantsur | another bad news this morning: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=982182 seems to affect debian-buster images | 12:21 |
| openstack | Debian bug 982182 in grub-pc "grub-pc: Regression on buster 10.8 release with non-interactive update" [Important,Open] | 12:22 |
| dtantsur | at least bifrost fails on installing grub-efi-amd64-signed | 12:22 |
| dtantsur | we may work around by downloading and unpacking it instead (we don't really need it install), but it seems the bug will be triggered by anything that will cause upgrade of grub-pc | 12:22 |
| *** hasharAway is now known as hashar | 12:35 | |
| zigo | dtantsur: Again, I don't think you need to go into unpacking and such, just preseeding grub should be enough. | 12:47 |
| zigo | dtantsur: Do you know how preseeding works? | 12:47 |
| dtantsur | no idea, I'm not the one creating these images | 12:48 |
| zigo | dtantsur: It's as if you pre-answer the prompts from debconf. | 12:48 |
| dtantsur | I don't think *I* can do that | 12:49 |
| zigo | Why? | 12:49 |
| dtantsur | I neither know nor care how this image is booted | 12:50 |
| zigo | It would be something like that (example with grub-pc/chainload_from_menu.lst): | 12:53 |
| zigo | echo "grub-pc grub-pc/chainload_from_menu.lst boolean false | 12:53 |
| zigo | grub-pc grub-pc/chainload_from_menu.lst seen true" | debconf-set-selections | 12:53 |
| *** whoami-rajat__ has quit IRC | 12:57 | |
| *** CeeMac has quit IRC | 12:58 | |
| *** walshh_ has quit IRC | 12:58 | |
| *** walshh_ has joined #opendev | 13:00 | |
| *** CeeMac has joined #opendev | 13:01 | |
| *** iurygregory has quit IRC | 13:04 | |
| *** iurygregory has joined #opendev | 13:06 | |
| *** whoami-rajat__ has joined #opendev | 13:07 | |
| dtantsur | if this is required by debian, it should be built into the images | 13:19 |
| *** jpena|lunch is now known as jpena | 13:21 | |
| *** ykarel_ has joined #opendev | 13:23 | |
| *** ykarel has quit IRC | 13:25 | |
| *** ykarel_ is now known as ykarel | 13:26 | |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: Update upload-logs-swift https://review.opendev.org/c/zuul/zuul-jobs/+/774650 | 13:55 |
| *** ykarel is now known as ykarel|away | 13:56 | |
| *** mlavalle has joined #opendev | 13:58 | |
| *** sshnaidm is now known as sshnaidm|afk | 14:00 | |
| *** ykarel|away has quit IRC | 14:32 | |
| dtantsur | infra folks, would it be possible to build the steps zigo mentions in the images? | 14:46 |
| *** stand has joined #opendev | 14:51 | |
| *** lourot has quit IRC | 15:00 | |
| *** lourot has joined #opendev | 15:00 | |
| *** ysandeep|rover is now known as ysandeep|away | 15:21 | |
| *** hashar is now known as hasharAway | 15:27 | |
| fungi | dtantsur: storyboard's been fixed for a few minutes, i'm still catching up on scrollback | 15:29 |
| fungi | unattended-upgrades apparently decided python-pymysql was a cool thing to automatically remove for no reason, i haven't gotten to the root cause there but i manually reinstalled the package and that seemed to solve things | 15:29 |
| fungi | dtantsur: which files from the grub-efi-amd64-signed package do you need? maybe there's a better source for them? | 15:33 |
| dtantsur | fungi: this one: https://opendev.org/openstack/bifrost/src/branch/master/playbooks/roles/bifrost-ironic-install/defaults/required_defaults_Debian_family.yml#L10 | 15:34 |
| fungi | the *.efi.signed files i guess | 15:34 |
| fungi | and where are you getting that /usr/lib/shim/shimx64.efi.signed file from? | 15:36 |
| dtantsur | from shim-signed | 15:36 |
| fungi | aha, okay, so also installing a package of that | 15:37 |
| fungi | i wonder if you could temporarily stub out /usr/sbin/policy-rc.d to make dpkg not invoke the maintscripts for that | 15:39 |
| fungi | because you don't actually want to muck with the bootloader on the system where you're installing it | 15:40 |
| fungi | i need to prep for a meeting, but can revisit this later | 15:41 |
| *** sshnaidm|afk is now known as sshnaidm | 15:44 | |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: Update upload-logs-swift https://review.opendev.org/c/zuul/zuul-jobs/+/774650 | 15:55 |
| clarkb | fungi: ya in the vast majority of cases it seems pretty clear whihc one is the one to preserve. That one with the criss crossed conflicts was the first really ambiguous one I have run across in the externalid conflict set | 15:57 |
| clarkb | I have not been cross checking against groups, but have looked at reviewedby:id and owner:id and that usually makes it clear | 15:57 |
| *** hasharAway is now known as hashar | 15:58 | |
| *** diablo_rojo has joined #opendev | 16:06 | |
| diablo_rojo | fungi, looks like we only have smcginnis left to drop from the board channel. | 16:07 |
| fungi | i'm not sure public shaming works on him ;) | 16:14 |
| fungi | also i merged your earlier changes for the other channels | 16:14 |
| fungi | diablo_rojo: were you going to push a change to add yourself to the operators list in the accessbot config? | 16:16 |
| diablo_rojo | I thought that I did that? | 16:20 |
| fungi | oh, yep, you used a separate review topic, that's why i missed it | 16:23 |
| diablo_rojo | Oh sorry! | 16:23 |
| diablo_rojo | I suppose I should have made them all the same | 16:23 |
| fungi | no need to apologize, i was spacey last night | 16:23 |
| fungi | anyway, there's already a comment asking you to insert yourself in alpha order | 16:24 |
| diablo_rojo | fungi, whoops. Will do that after the staff call. | 16:27 |
| *** chandankumar is now known as raukadah | 16:33 | |
| dtantsur | FYI test-release-openstack has been broken with cryptography for hours | 16:34 |
| clarkb | dtantsur: you should let the openstack release team know | 16:34 |
| dtantsur | will do | 16:34 |
| dtantsur | fungi: I'm pondering downloading and unpacking the package instead of installing it | 16:35 |
| dtantsur | it's definitely much more annoying but also more robust | 16:36 |
| diablo_rojo | fungi, we are good to go with the board channel! | 16:54 |
| diablo_rojo | Got OP and registered it and guarded it and added the openstackinfra access | 16:56 |
| *** slaweq is now known as slaweq|away | 16:56 | |
| fungi | dtantsur: yeah, it would work, i just wonder if there's a better source for that file, since the package is focused more on deploying it for booting the system on which it's installed | 17:03 |
| fungi | thanks diablo_rojo! so two more patches for that channel i guess and we'll be all set for the next phase | 17:03 |
| clarkb | fungi: dtantsur: is this happening in a dib context? I want to say that dib does the grub install very late and only after the image has been constructed | 17:04 |
| dtantsur | clarkb: no, not this time. this is bifrost, it requires this file to build a deployment environment. | 17:04 |
| *** marios is now known as marios|out | 17:14 | |
| *** jpena is now known as jpena|off | 17:17 | |
| fungi | okay, so update on the storyboard outage. it looks like there was a flurry of python module packages removed in the wake of us uninstalling bup | 17:25 |
| fungi | i still need to spider the dependency tree there, but my guess is that when we installed bup it pulled in python-pymysql as an indirect dependency and the package was marked as automatically installed | 17:25 |
| fungi | once bup was removed, there were no remaining explicitly installed debian packages depending on python-pymysql, so it got cleaned up | 17:26 |
| fungi | i think the puppet apt module just checks to see whether a specified package is installed, and doesn't know/care if it's marked as automatically installed | 17:27 |
| clarkb | fungi: so since we pulled it in automatically first puppet never updated that flag and it got removed when nothing directly depended on it even though we tell puppet to install it explicitly? | 17:28 |
| fungi | that certainly what it seems like, yeah | 17:29 |
| fungi | i expect that bup was installed before puppet checked to see whether it needed to install python-pymysql | 17:29 |
| fungi | in fact, we probably were using python-mysqlclient at the time bup was installed on that server, and then transitioned to python-pymysql for storyboard later | 17:30 |
| dtantsur | it may be an argument for turning storyboard into a package (or a container) | 17:38 |
| clarkb | it is already a container aiui, we just have to update the deployment to use it | 17:39 |
| *** dtantsur is now known as dtantsur|afk | 17:50 | |
| *** rpittau is now known as rpittau|afk | 17:56 | |
| *** eolivare has quit IRC | 18:00 | |
| openstackgerrit | Kendall Nelson proposed openstack/project-config master: Add diablo_rojo to AccessBot Operators https://review.opendev.org/c/openstack/project-config/+/774555 | 18:03 |
| openstackgerrit | Kendall Nelson proposed openstack/project-config master: Setup OpenInfra-Board Channel https://review.opendev.org/c/openstack/project-config/+/774705 | 18:06 |
| openstackgerrit | Kendall Nelson proposed opendev/system-config master: Setup OpenInfra-Board Channel https://review.opendev.org/c/opendev/system-config/+/774706 | 18:08 |
| diablo_rojo | fungi, ^^ all done. | 18:08 |
| *** hashar is now known as hasharAway | 18:23 | |
| *** marios|out has quit IRC | 18:30 | |
| mtreinish | fungi: yeah, I saw that cryptography added a rust extension, but I didn't think they made it mandatory in the recent release | 18:37 |
| mtreinish | setuptools_rust is probably required to be installed before running setup.py because it most likely gets imported at the top of the setup.py, but I haven't looked at how they did the packaging | 18:38 |
| clarkb | mtreinish: the github issue made it sound like they talked about having an alternative C set of bindings but that they didn't have the necessary tools or time to support that too | 18:39 |
| clarkb | also it seems the invovled individuals aren't interested in writing more C | 18:39 |
| clarkb | anyway its only an issue if you don't get a wheel (and that seems to happen more often than it should, possibly due to stale pypi indexes?) | 18:39 |
| clarkb | or if your platform doesn't have a wheel I ugess | 18:40 |
| clarkb | alpine in particular seems affected there because it uses musl | 18:40 |
| mtreinish | ah, yeah looking at their setup.py it looks like it's what I guessed. setuptools-rust is a prereq to run setup.py and they put it in pyproject.toml, but the pyproject.toml is hit or miss in my experience in actually installing required things | 18:42 |
| clarkb | mtreinish: https://github.com/pyca/cryptography/blob/master/pyproject.toml they use a pyproject.toml | 18:42 |
| clarkb | mtreinish: I think it depends on having a new enough pip to support it? | 18:43 |
| mtreinish | I worked around it in my rust/python lib by doing this super hacky thing: https://github.com/Qiskit/retworkx/blob/master/setup.py#L2-L10 which solves it for most cases | 18:43 |
| mtreinish | clarkb: I dunno, the pyproject.toml never seems to work for me locally | 18:43 |
| mtreinish | even with the latest pip | 18:43 |
| clarkb | huh | 18:43 |
| mtreinish | like the last time I tried to remove that try except block CI failed because setuptools rust couldn't be found | 18:44 |
| mtreinish | although I just tested it locally now and it seems to work fine, so I dunno. The last time I tried it was like a year ago | 18:48 |
| *** andrewbonney has quit IRC | 18:52 | |
| fungi | dtantsur|afk: clarkb: yep, we build container images and upload them to dockerhub for every commit which lands to storyboard and storyboard-webclient, we're just overdue for deploying those with ansible instead of installing from git into the system context with puppet | 19:10 |
| fungi | clarkb: mtreinish: i believe the issue ultimately is they published an abi3 wheel, which the older pip shipped with centos 8 doesn't know to look for, so it tries to install the sdist instead, but the version of pip there is also old enough to not know what to do with the pyproject.toml so it doesn't install setuptools_rust either | 19:12 |
| fungi | i think ubuntu-bionic is in the same boat | 19:12 |
| fungi | unfortunately, that's still our default nodeset (we need to talk about switching our default to ubuntu-focal, that's also overdue) | 19:13 |
| clarkb | aha its the abi that causes the sdist to be pulled | 19:16 |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: Update upload-logs-swift https://review.opendev.org/c/zuul/zuul-jobs/+/774650 | 19:17 |
| mordred | fungi, clarkb: latest openstacksdk works just fine talking to rackspace using the clouds.yaml entry on bridge | 19:27 |
| clarkb | could be an osc specific issue then? | 19:27 |
| fungi | yeah, even something as simple as server list is breaking for me | 19:28 |
| fungi | sudo openstack --os-cloud openstackci-rax --os-region-name DFW server show afs01.dfw.openstack.org | 19:28 |
| fungi | on bridge | 19:28 |
| mordred | I'm working on trying osc now | 19:28 |
| mordred | kk - thanks | 19:28 |
| fungi | that spins for a while for me and then complains about the tokens method | 19:29 |
| mordred | urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='identity.api.rackspacecloud.com', port=443): Max retries exceeded with url: /v2.0/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9bcf0bcfd0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) | 19:36 |
| mordred | I think there is some sort of networking issue | 19:36 |
| clarkb | temporary failure in name resolution sounds like dns problems | 19:37 |
| clarkb | weird that it would work with sdk if dns was sad though? | 19:37 |
| mordred | well - I tried sdk from my laptop | 19:38 |
| mordred | BUT | 19:38 |
| mordred | I also tried curl-ing the main keystone url from bridge and that is working | 19:38 |
| mordred | so I'm checking what's going on inside of the container right now | 19:38 |
| fungi | might it be something listed in the catalog isn't actually in dns? | 19:39 |
| mordred | ooh - yeah | 19:39 |
| mordred | I booted a container | 19:39 |
| mordred | and I can't do things in it | 19:39 |
| mordred | root@5fb36de8a300:/# apt-get update && apt-get install curl | 19:39 |
| mordred | Err:1 http://deb.debian.org/debian buster InRelease | 19:39 |
| mordred | Temporary failure resolving 'deb.debian.org' | 19:39 |
| mordred | that's in an osc container running bash on bridge | 19:39 |
| clarkb | aha | 19:39 |
| fungi | ooh, we're running osc from a container! so it's container networking which is at fault? | 19:40 |
| mordred | so I think the issue is "dns is broken inside of containers on bridge" | 19:40 |
| mordred | and we just happened to notice via the osc container | 19:40 |
| fungi | thanks, that seems much more tractable. i forgot we had containerized osc | 19:40 |
| mordred | fungi: yah - openstack on bridge is a wrapper script installed by ansible | 19:40 |
| mordred | cat /usr/local/bin/openstack | 19:40 |
| mordred | https://serverfault.com/questions/642981/docker-containers-cant-resolve-dns-on-ubuntu-14-04-desktop-host | 19:42 |
| mordred | "Apparently the docker0 network bridge was hung up." | 19:43 |
| mordred | might be worth starting with just restarting docker daemon | 19:43 |
| mordred | shall I try that? | 19:44 |
| clarkb | seems reasonable | 19:45 |
| fungi | double-plus good | 19:45 |
| mordred | fixed | 19:46 |
| mordred | I restarted both docker and containerd for good measure | 19:46 |
| mordred | all works now | 19:47 |
| fungi | thanks mordred! i should have remembered there was a container layer there now | 19:48 |
| fungi | my mind immediately jumped to the problem preventing us from using osc to talk to rackspace's cinder | 19:48 |
| mordred | ++ - that will eventually get fixed with the ongoing osc->sdk work | 19:51 |
| ianw | oh cool. yeah the container wrapper has hit me trying to upload image files ... the container can't see them. but i didn't think of container networking | 20:04 |
| clarkb | ianw: -1 for a job name thing on the refstack jobs change | 20:06 |
| clarkb | lunch and exercise next for me, but then I've got the borg changes top of my todo list when I return | 20:06 |
| ianw | clarkb: thanks will look | 20:07 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: refstack: move non-private variables to public https://review.opendev.org/c/opendev/system-config/+/774587 | 20:11 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: refstack: add production image and deployment jobs https://review.opendev.org/c/opendev/system-config/+/774586 | 20:11 |
| mordred | ianw: we should do a $something to mount in a working dir | 20:13 |
| mordred | ianw: maybe we should add something like -v$(pwd):/work or something like that so you could still use it for image uploads | 20:13 |
| diablo_rojo | fungi, so what's next? | 20:30 |
| openstackgerrit | Oleksandr Kozachenko proposed openstack/project-config master: Add zuul-storage-proxy in zuul namespace https://review.opendev.org/c/openstack/project-config/+/772364 | 20:35 |
| *** Dmitrii-Sh has quit IRC | 20:39 | |
| fungi | diablo_rojo: what's next is i need to review your changes ;) | 20:40 |
| *** Dmitrii-Sh has joined #opendev | 20:40 | |
| diablo_rojo | fungi, lol and then I do the rest of the steps in the renaming an irc channel section? | 20:40 |
| fungi | diablo_rojo: pretty much, yeah | 20:42 |
| fungi | and thinking about the channels involved, we probably don't need to do a ton of advance warning (it was already discussed on the ml anyway) | 20:43 |
| fungi | config-core: diablo_rojo is volunteering to help with irc channel management, and is working on some foundation channel moves to the #openinfra channel namespace: https://review.opendev.org/774555 | 20:44 |
| fungi | "/bin/bash: ansible: command not found" huh, that's... unexpected https://zuul.opendev.org/t/openstack/build/e07f15fd4c7f43f09565eeda65217525 | 20:47 |
| fungi | that job was just passing yesterday | 20:47 |
| ianw | " It will also require most users to update the way they install the linter as they now need to mention which version of Ansible they want to use it with." | 20:49 |
| ianw | https://github.com/ansible-community/ansible-lint/releases | 20:49 |
| fungi | did that just happen? | 20:49 |
| ianw | a few hours ago it seems, 5.0.0 release | 20:49 |
| fungi | "Released: about 6 hours ago" | 20:50 |
| fungi | yeah | 20:50 |
| * fungi takes another look at the change for dropping ansible-lint jobs from system-confg | 20:51 | |
| openstackgerrit | Merged openstack/project-config master: Setup OpenInfra-Board Channel https://review.opendev.org/c/openstack/project-config/+/774705 | 20:56 |
| *** whoami-rajat__ has quit IRC | 21:03 | |
| *** d34dh0r53 has quit IRC | 21:07 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Stop running ansible-lint on this repo https://review.opendev.org/c/opendev/system-config/+/733406 | 21:08 |
| *** d34dh0r53 has joined #opendev | 21:08 | |
| ianw | corvus / fungi: ^ that's an update if we don't want to get to the bottom of this | 21:08 |
| corvus | +2 | 21:13 |
| *** hasharAway has quit IRC | 21:14 | |
| ianw | hrm | 21:14 |
| ianw | bash -c "ANSIBLE_INVENTORY_PLUGINS=./playbooks/roles/install-ansible/files/inventory_plugins ansible -i ./inventory/base/hosts.yaml not_a_host -a 'true'" | 21:14 |
| ianw | we are probably relying on ansible-lint to pull in ansible for that | 21:15 |
| fungi | right, i'm guessing it no longer expresses a dep on it | 21:15 |
| fungi | i'm working on reproducing locally now | 21:15 |
| fungi | yeah, reproduces here for me, trying just adding ansible to the deps list for the linters testenv now | 21:17 |
| fungi | that seems to get past the error | 21:19 |
| ianw | yeah, and i mean that probably fixes ansible-lint too | 21:20 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Install ansible for ansible-lint testing https://review.opendev.org/c/opendev/system-config/+/774735 | 21:21 |
| fungi | alternative if we want to keep it | 21:21 |
| fungi | at least the solution seems trivial | 21:21 |
| ianw | well yeah, we're directly calling ansible so need it in the linters tox environment | 21:21 |
| ianw | the usefulness of linting with uncapped ansible when bridge is running, whatever it's running is perhaps questionable | 21:22 |
| fungi | right, previously we were relying on ansible-lint to drag it in for us | 21:22 |
| fungi | and i agree, i waffled on adding a cap there, but figured i'd start with that and see what folks want | 21:22 |
| ianw | i'd like the gate to work :) | 21:23 |
| fungi | up to now we were relying on whatever version of ansible an uncapped ansible-lint depended on | 21:23 |
| fungi | so i figured uncapped ansible was effectively the same | 21:23 |
| ianw | yeah, it's no worse. i feel like i had a change to update bridge ansible | 21:24 |
| fungi | okay, so we may need to squash them | 21:59 |
| fungi | looks like 774735 does also get ansible-lint to run | 21:59 |
| fungi | unfortunately 5.0.0 has lots of new things to say about the content in system-config | 21:59 |
| corvus | maybe just add the requirement to 406? | 22:01 |
| fungi | https://zuul.opendev.org/t/openstack/build/b9c5a49d34904b008403e292cbd3f28d/log/job-output.txt#733-5450 | 22:03 |
| fungi | nearly >4.5k lines of ansible-lint complaining about that repo | 22:04 |
| fungi | s/nearly// | 22:04 |
| fungi | yeah, i'll revise 733406 with that | 22:05 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Stop running ansible-lint on this repo https://review.opendev.org/c/opendev/system-config/+/733406 | 22:09 |
| *** gmann is now known as gmann_afk | 22:13 | |
| ianw | ++ | 22:25 |
| *** Dmitrii-Sh has quit IRC | 22:27 | |
| *** Dmitrii-Sh has joined #opendev | 22:34 | |
| clarkb | is there a tldr for which change(s) I shoudl review to fix ansibel lint stuff (if any) | 22:40 |
| ianw | clarkb: just have to wait for https://review.opendev.org/c/opendev/system-config/+/733406 to go through | 22:41 |
| clarkb | cool I'll catch up on refstack and backup reviews then | 22:41 |
| clarkb | since the -1s appear related to the ansible lint thing | 22:42 |
| ianw | thanks; now we have a plan for pruning i'll add wiki to borg manually; that's the last thing outstanding | 22:43 |
| ianw | fungi: i know we had this discussion, but we need basically all /opt backed up? | 22:43 |
| fungi | ianw: yes, the way i was trying to reorganize it on wiki-dev would put the configuration-managed stuff in a separate path from the precious data, but on the existing server even the deployment of the software is precious because it was done by hand | 22:46 |
| fungi | and everything runs out of git checkouts/submodules at specific refs | 22:47 |
| ianw | np; the user/keys are all setup, i just need to copy the backup script and cron jobs into it | 22:48 |
| clarkb | ianw: does `sudo FOO=bar -s <<'EOF' echo $FOO EOF` do proper var substituation in that heredoc with the env vars set prior to the -s? | 22:50 |
| clarkb | for context I'm looking at https://review.opendev.org/c/opendev/system-config/+/774561/3/playbooks/roles/borg-backup-server/files/prune-borg-backups.sh and trying to replicate that behavior and haven't successfully gotten it to work with my local sudo and shell | 22:50 |
| ianw | clarkb: for this script we don't want to do that because we're passing the variables in via the sudo | 22:50 |
| clarkb | ianw: if I do `sudo FOO=foo -s echo $FOO` it echos a blank newline not foo | 22:51 |
| fungi | in that example your shell is interpreting $FOO (and replacing it with a null string) | 22:52 |
| clarkb | ah | 22:52 |
| clarkb | ok ya echo \$FOO works | 22:52 |
| ianw | yeah, the <<'EOF' stops that and passes it raw into the shell sudo has started | 22:52 |
| clarkb | TIL you can set env vars in the sudo command like that | 22:52 |
| fungi | $FOO needs to be part of the string which the subshell executes under sudo if you're going to rely on sudo's environment options | 22:52 |
| ianw | i admit, it's a little crazy -- but i wanted to have a single prune script and wanted to make sure to run as the user so we don't leave behind root files that might block further backups | 22:52 |
| clarkb | ianw: ya I think the sudo -u is a really good idea | 22:53 |
| clarkb | I just had to reconcile the env var stuff in my own head. Thank you for walking me through that | 22:53 |
| fungi | the resulting syntax is a good bit cleaner than trying to accomplish the same with su -c | 22:53 |
| clarkb | ianw: and checking the test logs we have https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_acb/774561/3/check/system-config-run-borg-backup/acb9e1d/borg-backup01.region.provider.opendev.org/prune-borg-backups.log | 22:58 |
| clarkb | ianw: do we expect it to emit a bit more output tehre (saying which things it is pruning? I can't remember what it says for my local backups) | 22:58 |
| ianw | hrm, it may not if there is nothing to prune ... or the rather hacky way i ran that in testinfra might not be capturing it's output | 22:59 |
| clarkb | ianw: I wonder if those tests all run in order? coudl be the prune runs before the archives happen? | 22:59 |
| ianw | i did think it would run in file sequential order ... but indeed that is another possibility | 23:00 |
| ianw | the html log should tell us | 23:00 |
| ianw | does look like it ran last https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_acb/774561/3/check/system-config-run-borg-backup/acb9e1d/bridge.openstack.org/test-results.html | 23:01 |
| ianw | let me run in noop mode and see what happens when i capture on servers | 23:01 |
| clarkb | ok | 23:01 |
| ianw | yeah, i can see output coming out. it must use another fd | 23:03 |
| clarkb | cool, fwiw I really like the use of testing here to help ensure things are happenign the way we expect :) | 23:04 |
| clarkb | this is a testing only issue for now so I'll leave it up to you if you want to fix it now or in a followup | 23:04 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: borg testing: catch stdout and stderr from test prune correctly https://review.opendev.org/c/opendev/system-config/+/774745 | 23:05 |
| ianw | or indeed we could do ^ ... i always get that mixed up | 23:05 |
| ianw | thanks, good catch | 23:05 |
| clarkb | ianw: left a question on the docs change too | 23:06 |
| clarkb | ianw: for the redirect fix shouldn't 2>&1 and &> be equivalent in bash ? 2>&1 is the sh specific version? | 23:07 |
| * clarkb is learning lots about bash and sudo today | 23:08 | |
| ianw | i had the order wrong though, the file after the 2>&1 | 23:08 |
| ianw | &> is just a shorthand so you don't make the mistake i did :) | 23:09 |
| clarkb | ah | 23:09 |
| clarkb | https://www.gnu.org/software/bash/manual/html_node/Redirections.html confirms the order is important, I feel like I remember that whenever I do it wrong too | 23:09 |
| clarkb | ianw: I've +2'd the stack but didn't approve as I awsn't sure the lint fix is in the gate yet | 23:12 |
| clarkb | looks like 733406 is just about to merge so approving things is probably ok | 23:12 |
| *** Dmitrii-Sh has quit IRC | 23:15 | |
| fungi | yeah, `foo >bar 2>&1` does basically the same as `foo &> bar` | 23:15 |
| fungi | except there's an implied 3>&1 and 4>&1 and so on, i believe | 23:16 |
| fungi | ahh, nope, bash manpage says it's just fd 1 and 2 | 23:17 |
| fungi | also acknowledges you can write it >& but that notation is ambiguous and limits what characters the target can start with | 23:17 |
| *** Dmitrii-Sh has joined #opendev | 23:18 | |
| *** sboyron has quit IRC | 23:18 | |
| * mordred always find &> weird and too new | 23:23 | |
| mordred | or >& or whatever it is | 23:23 |
| ianw | apropos nothing, we mentioned node launch in the meeting ... there was one slight problem with unattended-upgrades | 23:24 |
| ianw | i don't know exactly what it's doing, but it seems to install every single package it wants to update separately. this takes like close to an hour to get through, it rebuilt the initramfs liek 10 times | 23:25 |
| clarkb | ianw: I wonder if it is really built to be run daily and if you boot something that is quite behind it doesn't work well | 23:26 |
| clarkb | its possible we may just want to replace it with apt-get update && apt-get dist-upgrade -y ? | 23:26 |
| ianw | yeah, i think it's supposed to be run on shutdown or something and does it like that to avoid blocking things | 23:26 |
| clarkb | I think I added that bit to the script and it was just an easy way to use existing tooling to get the image up to date before we proceeed | 23:26 |
| ianw | yeah, i think we need that earlier in the launch script. i'll take a look, anyway just remembered that | 23:26 |
| mordred | we should be careful with dist-upgrade -y - probably want to make sure we've applied our apt config first (because we turn off recommends iirc) - and dist-upgrade -y could wind up installing some things that we wouldn't otherwise install | 23:37 |
| mordred | I mean - I thnk it's probably a good idea - just saying we should make sure our apt config stuff has applied | 23:38 |
| fungi | an excellent reminder | 23:39 |
| clarkb | mordred: ya I think the current code runs after applying base configs | 23:39 |
| clarkb | that pulls in our unattended upgrades at least and then we just run that | 23:39 |
| mordred | great | 23:39 |
| clarkb | which was why I seem to recall using it, since it was going to use our existing policy for updates and that was easy mode | 23:39 |
| openstackgerrit | Merged opendev/system-config master: Stop running ansible-lint on this repo https://review.opendev.org/c/opendev/system-config/+/733406 | 23:40 |
| *** tosky has quit IRC | 23:48 | |
| *** DSpider has quit IRC | 23:54 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!