| *** rlandy|bbl is now known as rlandy|out | 01:23 | |
| dpawlik | dansmith: good work. Only one think can be problematic: there is no date, so it will use current date for result that can be done few hours ago | 06:07 |
|---|---|---|
| *** rlandy_ is now known as rlandy | 10:32 | |
| *** dviroel|out is now known as dviroel | 11:11 | |
| xiaolong | hi | 11:24 |
| *** dasm|off is now known as dasm | 13:12 | |
| dansmith | dpawlik: ack, will add a timestamp :) | 13:25 |
| *** soniya29 is now known as soniya29|out | 13:51 | |
| dpawlik | thanks dansmith | 14:38 |
| fzzf[m] | fungiclarkb : when I run df -h , it shows mount on /run/media/root/cloudimg-rootfs， run kpartx -d /dev/loop0 also shows device or resource busy, and I run lsof | grep loop0p2 can't find process. follow is different build from before.... (full message at https://matrix.org/_matrix/media/r0/download/matrix.org/FrPjssLVEUUKrHAdwioAAbep) | 15:03 |
| fungi | fzzf[m]: try grepping lsof output for the mount path rather than the device name. it'll either be a file held open somewhere in the tree being umounted, or maybe just a race with lazy umount | 15:08 |
| fzzf[m] | fungi: I run losf | grep clouding-rootfs, still no results. curious df -h show loop0p2 mount on /run/media/root/cloudimg-rootfs rather than /var/cache/nodepool/dib_tmp/dib_build.3DMuQbbI/mnt. | 15:13 |
| fungi | does it show anything mounted under /var/cache/nodepool/dib_tmp? | 15:14 |
| fzzf[m] | I'm using HDD disk, will this matter? | 15:14 |
| fungi | shouldn't, no | 15:14 |
| fungi | i'll admit i'm not super familiar with the internals of dib, so just approaching this as a typical *nix sysadmin problem | 15:15 |
| fzzf[m] | fungi: /var/cache/nodepool/dib_tmp/dib_build.3DMuQbbI/mnt is empty.but its parent's block-device and hooks stats have file.... (full message at https://matrix.org/_matrix/media/r0/download/matrix.org/DFMPWVxWZvAMqPMSAdOCTboV) | 15:18 |
| clarkb | fzzf[m]: a raw disk file is mounted via loopback dev then a filesystem is created on it and the contents of the dib chroot are copied into it. Then it is unmounted and the loopback device is removed and the file can be converted using qemu-img to qcow2 or vhd etc | 15:21 |
| clarkb | fungi: ^ that was meant for you to | 15:21 |
| clarkb | it isn't doing anything special other than treating a file as a disk | 15:22 |
| fzzf[m] | log show umount mnt. I run umount already show not mounted. but I run kpartx -d /dev/loop0 also show device busy.... (full message at https://matrix.org/_matrix/media/r0/download/matrix.org/ZbuqiJDKTluPqQLLNJQrTduQ) | 15:23 |
| fungi | yep, that much i get. as to exactly where it mounts the loop devices, or what gets executed with them as a chroot i'm less clear | 15:23 |
| clarkb | oh ya I think grub installation happens within a chroot on the loopback dev because grub wants rael partitions or something | 15:24 |
| fungi | fzzf[m]: right, the prior lazy umount has marked it as no longer mounted, but the kernel won't free the references until it's no longer busy | 15:24 |
| clarkb | does dmesg indicate why it might be busy if lsof is empty? that said I would really expect lsof (run as root) to show you what files are open and by what processes | 15:25 |
| fungi | yeah, i have a feeling trying to run it after a lazy umount is hiding that | 15:26 |
| fungi | because the file paths no longer exist | 15:26 |
| fungi | might be a good idea to find all the umount -l/--lazy calls in dib and wrap them in some sort of debug fixture so they can be toggled to hard umounts | 15:27 |
| fungi | that would make debugging things like this easier | 15:27 |
| *** dviroel is now known as dviroel|lunch | 15:28 | |
| fzzf[m] | <fungi> "might be a good idea to find all..." <- I check log. it's use umount -lf. do you mean i use hard mount replace | 15:33 |
| fungi | fzzf[m]: i mean drop the -l/--lazy option, so it would just be umount -f | 15:34 |
| fungi | that way the umount command blocks until the device is no longer mounted. if the problem isn't a race between umount and kpartx commands, then the umount command may just hang indefinitely, but at least then lsof will probably show you what's open where it's still mounted | 15:35 |
| fzzf[m] | fungi: okay. I'll make a try. | 15:35 |
| fzzf[m] | fungi: I get it | 15:36 |
| *** dviroel|lunch is now known as dviroel | 16:35 | |
| elodilles | hi, the usual heads up: since we have some new EOL'd stable/ussuri branch for tripleo, as usual I will run the clean-up script to remove the branches (as far as I see there's no problem with gerrit & zuul) | 17:21 |
| clarkb | elodilles: no problem with gerrit and zuul that I am aware of however the git issues on ubuntu are distracting fungi and I | 17:22 |
| clarkb | more a caution that if something does go wrong with the EOL'ing we may not be able to help immediately. | 17:22 |
| fungi | yep | 17:26 |
| elodilles | clarkb: thanks, ack (yeah, that issue that came with the new git release is not nice... I'm glad we didn't face it at Yoga release time :S) | 17:26 |
| icey | hey, any chance for more reviews on https://review.opendev.org/c/openstack/project-config/+/835430? thanks in advance! | 17:45 |
| clarkb | icey: we're deep in firefighting mode today so not sure if we'll get to it today | 17:46 |
| icey | clarkb: no worries - it's been sitting for a couple of weeks so just hoping to move it forward :) | 17:46 |
| clarkb | fungi and I have both +2'd it now. Once fires die down we can approve it. My main concern with ^ is that the current fire is related to git operations and I'd like to not make the fire bigger if that project improtation process has trbouel | 17:49 |
| clarkb | oh we can rerun the gerrit system-cofnig test jobs to sanity check that. /me does this | 17:49 |
| fungi | oh, yep, i'll remove my +w | 17:49 |
| fungi | er, no, i won't since i didn't +w that ;) | 17:49 |
| clarkb | fungi: I rechecked https://review.opendev.org/c/opendev/system-config/+/827774 whcih should exercise jeepyb for us I think | 17:49 |
| fungi | but yeah, what clarkb said | 17:49 |
| clarkb | oh but maybe it doesn/t jeepyb and gerritlib changes do though | 17:50 |
| clarkb | I'll push up a jeepyb chagne to cover our bases | 17:50 |
| clarkb | remote: https://review.opendev.org/c/opendev/jeepyb/+/837753 DNM want to run jeepyb testing against gerrit | 17:51 |
| sean-k-mooney | by the way woudl use the USE_VENV flag in devstack help with the git issue | 18:13 |
| sean-k-mooney | that would not do gloabl installs | 18:13 |
| clarkb | no | 18:14 |
| clarkb | because it doesn't work | 18:14 |
| sean-k-mooney | i have not hit the issue yet but i was going to play with getting devstack working on my macbookair nativly on debian | 18:14 |
| sean-k-mooney | does it not? | 18:14 |
| clarkb | its missing a lot of stuff iirc | 18:14 |
| sean-k-mooney | hum i was using it to work around path issues on debian | 18:15 |
| clarkb | when I tried to switch devstack to use a global virtualenv to work around the pip 9 cannot uninstall package problem it basically had to be completely rewritten because it didn't actually do half of what was required | 18:15 |
| sean-k-mooney | and it seamed to be getting me past the inital issue i was having but it brok trying to connect to ovs so i did not get that far | 18:15 |
| sean-k-mooney | ok | 18:16 |
| clarkb | https://review.opendev.org/c/openstack/devstack/+/559418 is where that effort tailed off when I realized no one was really interesting in making the big switch | 18:16 |
| clarkb | which is the other problem. I tried to take our time and find the issues and address them with the intent of making that change by defaulta nd it was really unpopular | 18:16 |
| sean-k-mooney | the two main downsides would be co installablity | 18:17 |
| sean-k-mooney | and disk space | 18:17 |
| sean-k-mooney | and i guess time right | 18:17 |
| sean-k-mooney | although time is mitigated by pip cache | 18:17 |
| clarkb | well my proposal there uses a single virtualenv for everything to maintain all that | 18:18 |
| sean-k-mooney | ah ok | 18:18 |
| clarkb | all it did was shift the installs from /usr/local to /some/other/path/.venv | 18:18 |
| sean-k-mooney | you could get the same effect by doign a user install now | 18:18 |
| clarkb | I'm not a fan of user installs. They do some weird stuff | 18:19 |
| sean-k-mooney | we woudl just need to make sure the python path was correct in teh systemd files | 18:19 |
| sean-k-mooney | or make those also user services | 18:19 |
| clarkb | (even locally I make a virtualenv and then symlink to it from /home/foo/bin to get python utilities like tox) | 18:19 |
| sean-k-mooney | well i currently do that too | 18:19 |
| clarkb | sean-k-mooney: ya my change does all that if you want to look at it. I even got grenade working | 18:19 |
| sean-k-mooney | but only because im using nixos | 18:19 |
| sean-k-mooney | and i have no idea how to actully make tox work any other way | 18:20 |
| clarkb | there were problems with ironic jobs because ironic does stuff with python in weird ways | 18:20 |
| sean-k-mooney | ok am do ye have an overall game plan on how to adress the issue | 18:20 |
| sean-k-mooney | i see https://review.opendev.org/c/openstack/devstack/+/837659 | 18:21 |
| clarkb | yes, in #openstack-qa we're landing changes to set the git config to trust the repos that devstack clones as the short term correction. fungi is working on a change to stop trusting the repos and create a package artifact that root can install without access to the repo | 18:21 |
| fungi | yeah, pip install --user is very hacky | 18:21 |
| fungi | it's like halfway between a venv and not | 18:22 |
| sean-k-mooney | ya | 18:22 |
| sean-k-mooney | it also relise on your distor have ~/.local/bin ectra on your path | 18:22 |
| clarkb | to be clear I have no intention of resurrecting my devstack global venv change. I spent many many many hours trying to get ahead of problems just like this one and it was clear no one else thought that work was a priority so I gave up | 18:22 |
| clarkb | if anyone wants to resurrect the changes feel free | 18:23 |
| sean-k-mooney | clarkb: oh im not asking you too | 18:23 |
| sean-k-mooney | i was just wondifn if that could be used in the interim | 18:23 |
| sean-k-mooney | that being virtual envs | 18:23 |
| sean-k-mooney | to bypass the need to use sudo | 18:23 |
| sean-k-mooney | i would not have ask if it was not the fact i seamed to be making progress setinng USE_VENV=True at the weekend | 18:24 |
| fungi | it could have been used, but apparently no amount of warning that `sudo pip install` is a bad idea has motivated significant change in the codebase | 18:24 |
| sean-k-mooney | well i dont know i prefonally dont like to use python packages form distos | 18:25 |
| clarkb | I remember being convinced USE_VENV was a dead end when I did that work since it just missed stuff. Looking at my change I see issues with libguestfs for example that I bet I hit with USE_VENV when If irst looked at it | 18:25 |
| sean-k-mooney | so sudo pip install is only bad if you mix both worlds | 18:25 |
| clarkb | basically since no one ever used USE_VENV and it was never tested it just lacked coverage of what needed to be done | 18:25 |
| clarkb | sean-k-mooney: or if you pip install out of git repos :) | 18:25 |
| fungi | `pip install` into a venv is fine, it's using pip to install into system paths which compete with the distro package managers that causes problems | 18:26 |
| sean-k-mooney | fungi: yep | 18:26 |
| sean-k-mooney | i generally try not to install any python project form the distro to avoid that | 18:26 |
| sean-k-mooney | althogh on nixos i have now been forced to use virtual envs for everything more or less | 18:27 |
| fungi | debian has patched around that problem to some extent by carrying non-upstream changes which provide additional non-distro-package-managed module paths, but red hat just lets them overwrite one another at will | 18:27 |
| sean-k-mooney | not nessiarly a bad thing as im more comfortable using them now | 18:27 |
| sean-k-mooney | well thats because redhat dont really supprot using pip at all | 18:27 |
| sean-k-mooney | its there but your kind of on your own if you use it | 18:28 |
| fungi | debian basically patches distribute in the stdlib to make it so that pip installs into /usr/local instead of /usr, but that's fragile and does things which will soon not be possible in the upcoming stdlib refactors | 18:28 |
| sean-k-mooney | clarkb: so is the plan basically to build wheels and isntall those going forward | 18:28 |
| sean-k-mooney | and use sudo git config --global --add safe.directory as a interim step | 18:29 |
| clarkb | sean-k-mooney: thats the longer term plan fungi is trying to sort out. short term we're telling git to trust the repos that devstack git clones | 18:29 |
| fungi | sean-k-mooney: the near-term plan is to tell git it's okay to run in those directories even though the process owner isn't the file owner | 18:29 |
| sean-k-mooney | ack | 18:29 |
| clarkb | and if anyone wants to resurrect the the global virtualenvs idea so we can avoid this class of problem entirely going forward I would be happy | 18:29 |
| fungi | the mid-term idea i have is to separate package building from installation, though in so doing we lose the editable-mode/develop install feature | 18:29 |
| sean-k-mooney | fungi: its kind of broken already | 18:30 |
| fungi | long-term in my mind would be to stop calling pip as root and use venvs, yeah | 18:30 |
| sean-k-mooney | you can nolonger just sudo pip install -e a different directory | 18:30 |
| sean-k-mooney | e.g. sudo pip install -e /opt/repos/nova | 18:31 |
| fungi | right, and future changes to setuptools in order to support pep-660 editable mode installs are likely to just make the system-wide editables a no-go anyway | 18:31 |
| sean-k-mooney | so you can only use the editbale repos if you edit the one on /opt/stack | 18:31 |
| sean-k-mooney | which means you cant use reclone=true without risk of lossing work | 18:31 |
| fungi | also it's just plain scary to have a system-wide editable mode install backed by files which are owned by an entirely different user | 18:32 |
| sean-k-mooney | well the files and serivce are all the stack user by default right | 18:32 |
| fungi | i get that it's a dev tool and we throw securit yconcerns out the window, but it's also just really fragile | 18:32 |
| sean-k-mooney | or your current user if you dont use stack | 18:32 |
| sean-k-mooney | fungi: it would be a pretty big regression if we have to install the package every time we make an edit | 18:33 |
| sean-k-mooney | that is what eventually force me to start usign /opt/stack/nova for dev and disabel reclone=true | 18:34 |
| fungi | technically you don't have to reinstall the package if you just edit the files where they're installed and then copy them back to the repo, but yes i understand the desire to use editable mode. it's just that i don't think it's going to be viable long-term | 18:34 |
| sean-k-mooney | so personally i woudl prefer to keep editbale installs long term | 18:34 |
| fungi | unless we switch to using one or more venvs for it | 18:35 |
| sean-k-mooney | yep venvs or a user install i think shoudl be our long term play if we are force to stop doign gloabal editbale installs | 18:36 |
| sean-k-mooney | that or modify the system service files | 18:36 |
| sean-k-mooney | so they reinstall when you do a restart | 18:37 |
| sean-k-mooney | i.e. add pre-start=pip install ... | 18:37 |
| sean-k-mooney | you know this is going to break peopel in a lot of intersting ways | 18:40 |
| sean-k-mooney | i dont personally use sshfs or anything like that to use remote git repos form an ide | 18:41 |
| sean-k-mooney | but i know a lot of peopel used to install pycharm and mount a filesystem remotely | 18:41 |
| clarkb | yes, I expect the fallout to be quite widespread. Someone I know that does university IT said "this might be fun" | 18:41 |
| sean-k-mooney | i guess depending on how its mounted it might be ok if it looks like they are the same user | 18:41 |
| clarkb | I think students in multi user environments might do a lot more sharing than you and I :) | 18:42 |
| sean-k-mooney | ha ya perhaps | 18:42 |
| sean-k-mooney | i was just thinking if i will need to do anything with the simple caching i implemtned recently | 18:42 |
| sean-k-mooney | i think i chown all the git dirs after i copy them | 18:43 |
| sean-k-mooney | no i dont i just become the stack user. i think that should be ok | 18:45 |
| sean-k-mooney | but i guess i know what to lookfor if its unhappy | 18:45 |
| sean-k-mooney | clarkb: by the way i dont see why https://review.opendev.org/c/openstack/devstack/+/559418 stalled out | 18:58 |
| sean-k-mooney | oh i see there are comment in the master version | 18:58 |
| sean-k-mooney | https://review.opendev.org/c/openstack/devstack/+/558930 | 18:59 |
| sean-k-mooney | ok today i suspect it woudl be simpler to do since we are python3 only | 18:59 |
| clarkb | sean-k-mooney: ya iirc ianw's indication the tc should be involved was part of it. I tried to get people to say "we're going to make the change and deal ith the fallout" and no one wanted to make that decision | 18:59 |
| clarkb | basically the qa team wasn't willign to assert that themselves (which is fair given the likely widespread impact) | 19:00 |
| sean-k-mooney | i see | 19:00 |
| sean-k-mooney | personally i think it would be a nice win form a clean build env perspecitve | 19:00 |
| clarkb | and there were definitely ironic problems and I waas losing my ability to dig into them | 19:01 |
| sean-k-mooney | it woudl be much easier to clean the system after an install | 19:01 |
| clarkb | so if we merged it we knoew ironic would break or we needed to get someone in ironic to try and care neough to address things | 19:01 |
| sean-k-mooney | ack | 19:02 |
| clarkb | it basically stalled out because there wasn't enough global interest to push it further than it got | 19:02 |
| sean-k-mooney | i mean if it comes down to venvs or no more editble installs | 19:02 |
| sean-k-mooney | that might be more of a motivation | 19:02 |
| sean-k-mooney | losing editable install woudl be a major ux regression | 19:03 |
| sean-k-mooney | not fatal by any means | 19:03 |
| sean-k-mooney | but it would change the cost benifit of alternitives like this | 19:03 |
| fungi | well, unfortunately, it's probably going to come down to merging some git config changes and then ignoring it until it catches on fire yet again for another reason | 19:03 |
| fungi | because being able to merge project changes is more important than fixing the test framework for it | 19:04 |
| sean-k-mooney | i feel this is aproptite https://i.kym-cdn.com/entries/icons/facebook/000/018/012/this_is_fine.jpg | 19:05 |
| fungi | nobody wants to worry about it until it's broken, and then it's a sudden scramble to do whatever it takes to just make it run again | 19:05 |
| sean-k-mooney | well its not jsut the test framework | 19:05 |
| sean-k-mooney | its what most of use use to deploy and test our chagnes | 19:06 |
| fungi | local and ci test framework, but you're still using it to test your development work | 19:06 |
| sean-k-mooney | yep | 19:06 |
| sean-k-mooney | i tried using kolla-ansibel with git repos in the container for a while | 19:07 |
| sean-k-mooney | but i havent really found a good replacment of devstack | 19:07 |
| sean-k-mooney | it does what we want it too for the most part | 19:07 |
| fungi | https://pics.me.me/ill-just-put-this-over-here-with-the-rest-of-30833910.png | 19:07 |
| sean-k-mooney | you have to keep the office organised | 19:08 |
| fungi | i love that he's surprised the fire extinguisher burst into flames, but then checks the label and it says "made in the uk" and he shrugs | 19:08 |
| sean-k-mooney | i mean it makes sesnes hehe | 19:09 |
| sean-k-mooney | when was the last time the uk made things instead of out sourcing to china like everyone else | 19:09 |
| fungi | besides flammable fire extinguishers and wine which doubles as automotive fuel | 19:10 |
| sean-k-mooney | they make good cider however. you have to at least give them that | 19:12 |
| fungi | oh sure! and "real ale" | 19:12 |
| sean-k-mooney | as someone who actull likes cider im alway a littel sad that the craft beer craze did not really carry over to craft cider in ireland at least | 19:13 |
| fungi | we have some great craft cideries here, there's one mere minutes from my parents' house and they live in the middle of nowhere (granted it's a middle of nowhere with a lot of orchards and fresh mountain water) | 19:15 |
| clarkb | the PNW has a number of great cideries too | 19:31 |
| clarkb | seattle cider, anthem cider, I think reverand nats closed due to the pandemic though | 19:32 |
| clarkb | I once tried to make cider and got vinegar | 19:37 |
| clarkb | I've still got it actually because I'm too afraid of digging out the pellicle | 19:38 |
| fungi | "bold rock" is the main cidery near where i grew up | 19:42 |
| fungi | though a few others sprang up there more recently | 19:43 |
| clarkb | fungi: just noting that our rechecked gerrit change's gerrit jobs succeeded but gitea did not due to this issue. | 19:58 |
| clarkb | I believe that to likely be a production issue but need to remember how this all works | 19:58 |
| clarkb | oh wait no | 19:59 |
| clarkb | only gerrit pushes to gitea | 19:59 |
| clarkb | so this is a test only problem I think | 19:59 |
| clarkb | the way production works is that we tell gitea to create a project then add perms for gerrit to replicate to it over the network. Then we create the repo in gerrit and it pushes to gitea as it updates on the gerrit side | 20:00 |
| clarkb | so if we can update gerrit then we should be good. This is just a test only problem when we out of band push system-config into gitea to have some real content. Working on a fix now | 20:00 |
| clarkb | jeepyb testing also looks good. Anotehr thing is this may affect project renaming. we may want to punt on that now just because these fires continue and it may impact that process :/ | 20:09 |
| fungi | that's a good point. if the fires are out by friday then we can go forward with it, otherwise we should postpone | 20:10 |
| *** dviroel is now known as dviroel|afk | 20:22 | |
| *** arxcruz is now known as arxcruz|out | 20:28 | |
| *** timburke__ is now known as timburke | 20:55 | |
| *** dasm is now known as dasm|off | 21:36 | |
| opendevreview | Luciano Lo Giudice proposed openstack/project-config master: Add the cinder-three-par to Openstack charms https://review.opendev.org/c/openstack/project-config/+/837782 | 23:30 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
