| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [wip] enable-kubernetes: check pod is actually running https://review.opendev.org/c/zuul/zuul-jobs/+/863810 | 00:13 |
|---|---|---|
| clarkb | that was fun. tumbleweed broke sudo so I had to figure out how to do a package update without sudo | 00:19 |
| fungi | i keep an emergency ramdisk boot in my bootloader config which goes straight to a root shell | 00:28 |
| clarkb | I had set a root password thankfully | 00:29 |
| clarkb | I just had to remmber that after my initial panic | 00:29 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [wip] enable-kubernetes: check pod is actually running https://review.opendev.org/c/zuul/zuul-jobs/+/863810 | 00:47 |
| opendevreview | Nikolay Vinogradov proposed openstack/project-config master: Add the cinder-infinidat charm to Openstack charms https://review.opendev.org/c/openstack/project-config/+/863954 | 01:38 |
| opendevreview | Nikolay Vinogradov proposed openstack/project-config master: Add the cinder-infinidat charm to Openstack charms https://review.opendev.org/c/openstack/project-config/+/863954 | 01:39 |
| opendevreview | Nikolay Vinogradov proposed openstack/project-config master: Add the infinidat-tools charm to Openstack charms https://review.opendev.org/c/openstack/project-config/+/863955 | 01:52 |
| opendevreview | Nikolay Vinogradov proposed openstack/project-config master: Add the cinder-infinidat charm to Openstack charms https://review.opendev.org/c/openstack/project-config/+/863954 | 02:45 |
| opendevreview | Nikolay Vinogradov proposed openstack/project-config master: Add the infinidat-tools charm to Openstack charms https://review.opendev.org/c/openstack/project-config/+/863955 | 02:45 |
| opendevreview | Nikolay Vinogradov proposed openstack/project-config master: Add manila-infinidat backend charm to OpenStack charms https://review.opendev.org/c/openstack/project-config/+/863957 | 02:45 |
| opendevreview | Nikolay Vinogradov proposed openstack/project-config master: Add the infinidat-tools charm to Openstack charms https://review.opendev.org/c/openstack/project-config/+/863955 | 02:48 |
| opendevreview | Nikolay Vinogradov proposed openstack/project-config master: Add manila-infinidat backend charm to OpenStack charms https://review.opendev.org/c/openstack/project-config/+/863957 | 02:48 |
| *** yadnesh|away is now known as yadnesh | 04:45 | |
| *** marios is now known as marios|ruck | 05:50 | |
| *** yadnesh is now known as yadnesh|afk | 08:04 | |
| *** jpena|off is now known as jpena | 08:29 | |
| *** yadnesh|afk is now known as yadnesh | 09:12 | |
| *** dviroel|afk is now known as dviroel | 09:48 | |
| opendevreview | Jie Niu proposed openstack/project-config master: Apply cfn repository for code and storyboard https://review.opendev.org/c/openstack/project-config/+/863168 | 10:27 |
| *** pojadhav is now known as pojadhav|afk | 11:47 | |
| opendevreview | Jeremy Stanley proposed zuul/zuul-jobs master: pypi-upload: support twine --skip-existing option https://review.opendev.org/c/zuul/zuul-jobs/+/864004 | 13:16 |
| opendevreview | Jeremy Stanley proposed zuul/zuul-jobs master: pypi-upload: support twine --skip-existing option https://review.opendev.org/c/zuul/zuul-jobs/+/864004 | 13:21 |
| opendevreview | Jeremy Stanley proposed zuul/zuul-jobs master: pypi-upload: support twine --skip-existing option https://review.opendev.org/c/zuul/zuul-jobs/+/864004 | 13:26 |
| *** rcastillo|rover_ is now known as rcastillo | 13:49 | |
| *** rcastillo is now known as rcastillo|rover_ | 13:49 | |
| *** rcastillo|rover_ is now known as rcastillo|rover | 13:51 | |
| *** dasm|off is now known as dasm | 14:02 | |
| *** yadnesh is now known as yadnesh|away | 14:22 | |
| *** gthiemon1e is now known as gthiemonge | 15:43 | |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Skip existing remote artifacts during PyPI upload https://review.opendev.org/c/openstack/project-config/+/864019 | 15:45 |
| *** dviroel is now known as dviroel|lunch | 15:58 | |
| *** marios|ruck is now known as marios|out | 16:25 | |
| *** dviroel|lunch is now known as dviroel | 16:59 | |
| *** timburke_ is now known as timburke | 17:03 | |
| *** jpena is now known as jpena|off | 17:30 | |
| *** dviroel is now known as dviroel|afk | 19:23 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: etherpad: redirect container logs https://review.opendev.org/c/opendev/system-config/+/864060 | 19:54 |
| opendevreview | Merged zuul/zuul-jobs master: pypi-upload: support twine --skip-existing option https://review.opendev.org/c/zuul/zuul-jobs/+/864004 | 20:04 |
| *** rlandy is now known as rlandy|out | 20:19 | |
| *** rlandy|out is now known as rlandy | 20:28 | |
| opendevreview | Nikolay Vinogradov proposed openstack/project-config master: Add the cinder-infinidat charm to Openstack charms https://review.opendev.org/c/openstack/project-config/+/863954 | 20:39 |
| opendevreview | Ian Wienand proposed opendev/system-config master: etherpad: redirect container logs https://review.opendev.org/c/opendev/system-config/+/864060 | 20:46 |
| opendevreview | Merged openstack/project-config master: Skip existing remote artifacts during PyPI upload https://review.opendev.org/c/openstack/project-config/+/864019 | 21:02 |
| fungi | that ^ worked as hoped | 21:21 |
| ianw | heh it's always nice when things don't explode :) | 21:24 |
| fungi | for the curious, https://zuul.opendev.org/t/openstack/build/ab7622e7094842dd82dd72f213450acd/console | 21:26 |
| fungi | "WARNING m Skipping metalsmith-1.5.3-py3-none-any.whl because it appears to already exist" | 21:26 |
| fungi | "Uploading metalsmith-1.5.3.tar.gz" | 21:27 |
| opendevreview | Nikolay Vinogradov proposed openstack/project-config master: Add the cinder-infinidat charm to Openstack charms https://review.opendev.org/c/openstack/project-config/+/863954 | 21:33 |
| opendevreview | Nikolay Vinogradov proposed openstack/project-config master: Add the cinder-infinidat charm to Openstack charms https://review.opendev.org/c/openstack/project-config/+/863954 | 21:47 |
| opendevreview | Nikolay Vinogradov proposed openstack/project-config master: Add the infinidat-tools charm to Openstack charms https://review.opendev.org/c/openstack/project-config/+/863955 | 21:58 |
| clarkb | I'm booting a new vexxhost test node so that I can try the relabel idea | 21:59 |
| clarkb | and I'm pretty sure I've broken it :) a good reason to carefully consider editing labels. But also a good test case of actually rescuing things | 22:05 |
| clarkb | (I updated fstab but not grub) | 22:07 |
| opendevreview | Ghanshyam proposed opendev/irc-meetings master: Update policy popup meeting time https://review.opendev.org/c/opendev/irc-meetings/+/838727 | 22:09 |
| clarkb | ok that boots the rescue node with the rescue image kernel and rootfs | 22:10 |
| clarkb | so the label collision does seem to be at the heart of the issue here. | 22:10 |
| clarkb | ianw: I'm trying to run grub-mkconfig in the rescue node to fix the relabeling in the grub config. It complains that a device isn't known for / because I chrooted into the mounted rescued node rootfs. Do you know what dib does to get around that? | 22:18 |
| clarkb | I can edit it by hand too I guess | 22:18 |
| clarkb | It must do bind mounts | 22:20 |
| *** dasm is now known as dasm|off | 22:28 | |
| ianw | clarkb: hrm, is that because it can't find /dev /sys, etc? | 22:34 |
| clarkb | ya I bind mounted those things and then it worked | 22:34 |
| clarkb | after I broke the bind mounts in a bad way :/ | 22:34 |
| clarkb | thi swhole process does seem very dangerous and full of dragons. But I have no successfully rescued and unrescued a regularly booted node (not bfv) in vexxhost | 22:35 |
| clarkb | * I have now | 22:35 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [wip] enable-kubernetes: check pod is actually running https://review.opendev.org/c/zuul/zuul-jobs/+/863810 | 22:37 |
| clarkb | so ya this is realted to the label as far as I can tell. I think on our end that means we can remove the label and boot by uuid or switch to a different label. Ideally at launch time so that we don't do all the stupid things I just did when trying to do it on a live (test) node | 22:38 |
| clarkb | mnaser__: ^ it might be worthwhile for you to hvae a rescue specific image in your cloud as an option for users? | 22:38 |
| clarkb | one that does not boot using label=cloudimg-rootfs. | 22:39 |
| opendevreview | Michael Kelly proposed zuul/zuul-jobs master: helm: Add job for linting helm charts https://review.opendev.org/c/zuul/zuul-jobs/+/861799 | 22:39 |
| clarkb | now I'm booting a bfv node and will try to stop it and then detach its volume in order to attach it elsewhere | 22:42 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [wip] enable-kubernetes: check pod is actually running https://review.opendev.org/c/zuul/zuul-jobs/+/863810 | 22:46 |
| clarkb | "Cannot detach a root device volume" | 22:48 |
| clarkb | I think that answers that question | 22:48 |
| clarkb | where that leaves us is that the rescue instance problem is solveable on our end for non bfv instances. But I don't know how to do this for bfv instances. Maybe we can shutdown the instance, clone the volume (is this a thing?), attach the volume to another node, edit the volume. Then use the existing volume on a new bfv server. This won't preserve Ips and generally seems ugly | 22:51 |
| opendevreview | Nikolay Vinogradov proposed openstack/project-config master: Add manila-infinidat backend charm to OpenStack charms https://review.opendev.org/c/openstack/project-config/+/863957 | 22:51 |
| jrosser | i have rescue + bfv configured in my clouds | 22:51 |
| clarkb | jrosser: hvae you tested it? | 22:52 |
| jrosser | with very very specific instuctions to users that they can't rescue an instance with the same image they booted it from | 22:52 |
| clarkb | jrosser: the cloud I'm working with does accept rescue on a bfv instance through the api. But it promptly breaks the instance and puts it in an error state | 22:52 |
| clarkb | and you cannot unrescue an error state instance so its a pretty game over moment | 22:52 |
| jrosser | i've not personally used it for a while but we deploy and document it as working | 22:53 |
| jrosser | but it was really awkward to figure out how to make it work | 22:53 |
| clarkb | re using the same image, I wasn't doing that in either case. The problem is the image choices I did make use the same label specifier to boot into as / | 22:53 |
| clarkb | so its a bit more subtle than simply not using the same image (you need disjoint labels or boot from uuid etc) | 22:54 |
| jrosser | iirc that amounts to a similar thing | 22:54 |
| clarkb | it doesn't help if different images use the same label is my point | 22:54 |
| clarkb | so using different images may not always be sufficient | 22:54 |
| jrosser | yes indeed | 22:54 |
| jrosser | we also put the ISO from system-rescue.org in glance and that was useful | 22:54 |
| clarkb | ya a dedicated rescue image that uses a unique uuid (redundant but still important) would be helpful here | 22:55 |
| clarkb | but that doens't explain why bfv is broken. And that isn't something I can debug on this end of the cloud I don't think | 22:55 |
| clarkb | honestly I kinda wish that the api had a "I know what I' mdoing" flag for detaching a root volume so that I can rescue in this way instead | 22:56 |
| clarkb | I think the next thing I can do is double check if openstack has bfv rescue testing somewhere at least | 22:56 |
| jrosser | i can have a go tomorrow and see if mine still works :) | 22:57 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [wip] enable-kubernetes: check pod is actually running https://review.opendev.org/c/zuul/zuul-jobs/+/863810 | 23:05 |
| * clarkb cleans up test resources | 23:05 | |
| opendevreview | Merged opendev/irc-meetings master: Update policy popup meeting time https://review.opendev.org/c/opendev/irc-meetings/+/838727 | 23:15 |
| opendevreview | Merged opendev/irc-meetings master: Update TC weekly meeting Day & time https://review.opendev.org/c/opendev/irc-meetings/+/863939 | 23:20 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [wip] enable-kubernetes: check pod is actually running https://review.opendev.org/c/zuul/zuul-jobs/+/863810 | 23:25 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: enable-kubernetes: Fix jammy install, improve pod test https://review.opendev.org/c/zuul/zuul-jobs/+/863810 | 23:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!