| ianw | my preference might be to do that, to avoid ongoing confusion | 00:00 |
|---|---|---|
| fungi | yeah, they're all in afs, i suppose i could just grep -r to build up a file list and then sed them or something | 00:00 |
| ianw | i don't mind doing that seeing as i created the problem | 00:00 |
| fungi | well, it's also not as urgent as correcting the config. i don't mind working on it myself, but would likely save it for tomorrow | 00:01 |
| ianw | i can do it, i think it makes the most sense | 00:03 |
| ianw | my thinking was that although the logs are stored on eavesdrop01.opendev.org, we should just hide that behind meetings.opendev.org (hence the proxy there) | 00:03 |
| ianw | and if the long-term future of eavesdrop-ing is not eavesdrop01 (matrix bot, etc) then we just create a static copy of the logs for archival on afs | 00:04 |
| fungi | yep, i understand and agree with the design | 00:04 |
| ianw | i'll see that change through and modify the incorrect meeting logs, thanks for finding it! | 00:05 |
| fungi | anyway, if you want to work on fixing up the meeting logs and/or channel logs feel free, but don't feel solely responsible. i'm sure if i looked back at the review which updated that file i likely reviewed if not approved it | 00:05 |
| fungi | i'm happy to help more with fixing it up too if you've got other things going on | 00:06 |
| ianw | no worries | 00:09 |
| ianw | unless mordred or someone can tell me how to make the query above not overwhelm the trove server, i don't even know how i'd clean up the existing account_patch_reviews db | 00:10 |
| ianw | ergo, at this point, unless someone has other ideas i think it will drop transitioning it from the review update checklist procedure | 00:11 |
| fungi | yeah, i think if it gets reset to an empty table when we move the service that's entirely fine | 00:12 |
| fungi | we should probably make sure to include that impact in communications, but i have a feeling it won't really affect our users significantly | 00:13 |
| fungi | most of them likely never even noticed it tracked that, and of the ones who did and relied on it, they probably only care about the current reviews they're looking at, so it'll catch back up enough for their needs in hours if not days | 00:14 |
| ianw | that reminds me, eavesdrop01.opendev.org isn't in hte backup list. i think we should add that | 00:16 |
| ianw | http://paste.openstack.org/show/806870/ | 00:22 |
| ianw | is the incorrect logs | 00:22 |
| ianw | http://paste.openstack.org/show/806871/ is the fixup plan | 00:26 |
| fungi | ianw: also i saw that gitea01 backups have been failing. not urgent since we haven't added new repo redirects lately, but i haven't had time to look into it yet | 00:29 |
| ianw | mysqldump: Error 2013: Lost connection to MySQL server during query when dumping table `action` at row: 6968 | 00:32 |
| ianw | i think we have seen this before a bit | 00:32 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Add eavesdrop01.opendev.org to backup group https://review.opendev.org/c/opendev/system-config/+/797552 | 00:49 |
| fungi | ianw: looking at your grep output, i guess there are no absolute urls embedded in the html files? also that's just the meeting logs right? should we do channel logs too? | 00:51 |
| ianw | yeah, i think the text just points to the html | 00:55 |
| ianw | but the html doesn't have any links | 00:55 |
| ianw | i'll grep the channel logs now to see if there's anything other than "this address isn't working" or this discussion :) | 00:56 |
| ianw | ohh, the comments in the end of the meeting, i see what you mean | 00:59 |
| opendevreview | Merged opendev/system-config master: Correct the meeting base URL for our meetbot https://review.opendev.org/c/opendev/system-config/+/797550 | 01:09 |
| opendevreview | Merged opendev/system-config master: centos-mirror: add dry run mode https://review.opendev.org/c/opendev/system-config/+/797364 | 01:12 |
| opendevreview | Merged opendev/system-config master: centos-mirror: exclude ppc64le https://review.opendev.org/c/opendev/system-config/+/797365 | 01:12 |
| fungi | ianw: right, what the bot echoes to the channel. that's how it originally came up, someone tried to follow the link to the minutes that the bot reported in-channel | 02:53 |
| ianw | fungi: ok, all the files where opendevmeet has incorrectly stamped things are @ http://paste.openstack.org/show/806874/ | 03:15 |
| ianw | i think what i'll do is merge and apply that backup change, and run backups. then i can do the sed replacements but feel good we have redundant copies | 03:16 |
| opendevreview | Merged opendev/system-config master: Add eavesdrop01.opendev.org to backup group https://review.opendev.org/c/opendev/system-config/+/797552 | 03:29 |
| *** diablo_rojo is now known as Guest267 | 04:00 | |
| ianw | letsencrypt failed because of nb01 | 04:01 |
| *** ykarel|away is now known as ykarel | 04:37 | |
| opendevreview | Merged openstack/project-config master: Add publish-to-pypi job template for tripleo-operator-ansible https://review.opendev.org/c/openstack/project-config/+/797415 | 05:15 |
| *** marios is now known as marios|ruck | 05:23 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: borg-backup: exclude /var/lib/snapd https://review.opendev.org/c/opendev/system-config/+/797562 | 05:37 |
| ianw | #status log cleaned up and rebooted nb01/nb02 | 05:40 |
| opendevstatus | ianw: finished logging | 05:40 |
| ianw | i am running a manual run of backs on eavesdrop | 05:41 |
| opendevreview | Ian Wienand proposed opendev/system-config master: review02: enable replication https://review.opendev.org/c/opendev/system-config/+/797563 | 05:45 |
| opendevreview | Ian Wienand proposed opendev/system-config master: backups: add review02.opendev.org https://review.opendev.org/c/opendev/system-config/+/797564 | 05:51 |
| *** ysandeep|out is now known as ysandeep | 05:52 | |
| *** odyssey4me is now known as Guest273 | 05:55 | |
| ianw | Wed Jun 23 06:02:08 UTC 2021 Backup finished successfully | 06:26 |
| ianw | ok, i will do the sed replacements now | 06:26 |
| ianw | #status log manually corrected links in recent meetings/logs on eavesdrop01.opendev.org to meetings.opendev.org; see https://review.opendev.org/c/opendev/system-config/+/797550 | 06:44 |
| opendevstatus | ianw: finished logging | 06:44 |
| ianw | just to be clear on the files i updated; http://paste.openstack.org/show/806871/ & http://paste.openstack.org/show/806874/ | 06:46 |
| *** ChanServ changes topic to "OpenDev is a space for collaborative Open Source software development | https://opendev.org/ | channel logs https://meetings.opendev.org/irclogs/%23opendev/" | 06:48 | |
| *** jpena|off is now known as jpena | 06:56 | |
| *** rpittau|afk is now known as rpittau | 07:07 | |
| zbr | what is the best irc channel to discuss pbr issues? | 07:23 |
| *** bhagyashris_ is now known as bhagyashris | 08:27 | |
| zbr | https://bugs.launchpad.net/pbr/+bug/1933311 | 08:48 |
| *** ykarel is now known as ykarel|lunch | 08:52 | |
| *** ykarel|lunch is now known as ykarel | 09:56 | |
| frickler | infra-root: the bot expanding bug urls seems down/gone, not sure which one that was. I'm in meetings today so cannot check myself | 11:28 |
| *** jpena is now known as jpena|lunch | 11:30 | |
| *** dviroel|out is now known as dviroel | 11:37 | |
| *** marios|ruck is now known as marios|ruck|call | 12:00 | |
| *** ysandeep is now known as ysandeep|afk | 12:26 | |
| *** jpena|lunch is now known as jpena | 12:33 | |
| *** marios|ruck|call is now known as marios|ruck | 12:47 | |
| fungi | frickler: it was the meetbot, so now opendevmeet. it was switched from the abandoned supybot framework to limnoria a couple weeks ago, so maybe we're missing the bug expansion plugin? when did you notice it last working? | 12:50 |
| fungi | zbr: officially, #openstack-oslo since pbr is an oslo deliverable | 12:50 |
| zbr | fungi: thanks. so far I identified 3 places for a fix: cpython, setuptools and the newer distutils, a joy. | 12:54 |
| fungi | nice | 12:54 |
| fungi | i have a feeling there will be little interest in fixing distutils, since it's fully deprecated now | 12:55 |
| zbr | kindof, being taken out. but what was not clear to me was which copy is used by pbr itself, as it would much easier to vender our fixed copy in pbr | 12:56 |
| fungi | probably pbr should have a plan for replacing its use of distutils. i count 6 modules in it (aside from tests) which import distutils or something from it | 12:57 |
| * zbr is looking for a link... | 12:58 | |
| mordred | yeah - we probably need to make a pbr modernification plan | 12:58 |
| fungi | https://www.python.org/dev/peps/pep-0632/ | 12:58 |
| fungi | "In Python 3.12, distutils will no longer be installed by make install or any of the first-party distribution. Third-party redistributors should no longer include distutils in their bundles or repositories." | 12:59 |
| fungi | however it goes on to say that setuptools may continue to provide a copy itself | 12:59 |
| zbr | usually i would drop use of pbr, but i still need it for packages that install stuff in no-kosher places, like installing ansible roles (which is not-supported by the modern packaging) | 12:59 |
| mordred | I would not drop use of pbr :) | 13:00 |
| zbr | in fact setuptools has two copies of the broken code inside (for some reason), but i have a patch to fix it, after the reference distutils project is patched. | 13:00 |
| mordred | but I do agree - it's going to need to learn how to co-exist with the new python packaging ecosystem sooner or later | 13:00 |
| mordred | I haven't checked recently - but my understanding of where things are going is that we should be able to reach a point where we can have a pbr that doesn't use setuptools under the covers in any way | 13:01 |
| mordred | which would likely make several of the things pbr wants to do much easier to accomplish (rather than needing to monkeypatch things that dont' necessarily want to be extended :) ) | 13:02 |
| fungi | like acting as a pep-517 build backend? | 13:02 |
| zbr | https://opendev.org/openstack/pbr/src/branch/master/pbr/packaging.py#L555 is the line that ends-up calling the buggy bits | 13:02 |
| mordred | fungi: yeah | 13:03 |
| fungi | https://www.python.org/dev/peps/pep-0517/#build-backend-interface | 13:03 |
| mordred | fungi: my brain kindof has had a pause in the back waiting for that to be ready | 13:03 |
| *** ysandeep|afk is now known as ysandeep | 13:04 | |
| zbr | i do not know pbr internals, but based on debuggin include_pattern seems to point to `distutils.filelist.include_pattern` | 13:06 |
| mordred | also - I haven't dug in far enough to see whether we can get away with not having requirements or version listed in pyproject.toml yet | 13:06 |
| mordred | zbr: interesting that the recursive symlink issue only expresses on osx and not on linux | 13:07 |
| zbr | in fact it does impact both | 13:07 |
| zbr | but the experience may be different from person to person, it is not even infinite because it stops when it reaches maxpath length. | 13:08 |
| zbr | but the feeling is that is infinite, with 100% CPU usage too. | 13:08 |
| mordred | isn't the prefix=ei_cmd.egg_info part of that call supposed to be telling it to find all of the files in the egg_info dir? | 13:09 |
| mordred | that dir is usually quite small - so I wonder if someting in that invocation is not doing that any longer | 13:10 |
| zbr | think about editable installs, that method ends up looking are git root repo, at least that is what I seen | 13:12 |
| frickler | fungi: that's a good question, I was pretty sure it had been working not so long ago, but checking my logs, the latest working response I could find was on the 10th https://meetings.opendev.org/irclogs/%23openstack-nova/%23openstack-nova.2021-06-10.log.html#t2021-06-10T14:54:23-2 | 13:13 |
| mordred | https://github.com/pypa/setuptools/blob/main/setuptools/command/egg_info.py#L225 | 13:14 |
| opendevreview | Florian Haas proposed opendev/git-review master: Doc updates for core.hooksPath support https://review.opendev.org/c/opendev/git-review/+/797643 | 13:15 |
| mordred | zbr: ^^ that's where ei_cmd.egg_info *should* be being set - I wonder if in your editable context it's getting set to something else | 13:15 |
| mordred | zbr: do you have a setup where it would be possible to print out what ei_cmd.egg_info is? | 13:16 |
| mordred | (on the line right before the include_pattern call) | 13:16 |
| fungi | frickler: that sounds like it might coincide with the switch to limnoria in that case: "2021-06-11 06:23:10 UTC meetbot/logging now running from limnoria on eavesdrop01.opendev.org" https://wiki.openstack.org/wiki/Infrastructure_Status | 13:17 |
| mordred | zbr: also - the class we're overriding there has changed upstream | 13:18 |
| mordred | zbr: maybe also try replacing the include_pattern call with: | 13:18 |
| mordred | self.filelist.graft(ei_cmd.egg_info) | 13:18 |
| mordred | and see if that solves your issue? (mostly - you have a reproducible env set up - so if updating the call to the new upstream call fixes it - that'll be good!) | 13:19 |
| mordred | https://github.com/pypa/setuptools/blob/main/setuptools/command/egg_info.py#L593 FTR | 13:19 |
| zbr | apparently it worked, but i also had to install wheel package (probably unrelated) | 13:29 |
| zbr | tbh, i did print the ei_cmd.egg_info and it pointed to a temp directory (already cleaned by the time command finished) | 13:30 |
| mordred | eww | 13:32 |
| mordred | but the graft worked? | 13:32 |
| zbr | yep | 13:32 |
| zbr | well, i did not test if the install was really correct, but at least pip reported success | 13:33 |
| mordred | cool. it should be - that method is an override of the one from setuptools and the content is copied in (there isn't actually a proper hook point so we have to copy-and-extend) - updating it to match what the setuptools method is doing is almost certainly the correct thing to do | 13:34 |
| zbr | mordred: now that I look at implementation of graft, i have reason to say that is also prone to the same bug: https://github.com/pypa/setuptools/blob/main/setuptools/command/egg_info.py#L446-L454 | 13:36 |
| zbr | in the end all of them endup calling https://github.com/pypa/setuptools/blob/a4dbe3457d89cf67ee3aa571fdb149e6eb544e88/setuptools/__init__.py#L212-L221 -- which we know chokes with symlinks-loops | 13:38 |
| mordred | yah - but maybe for now it's not a thing we need to solve because it fixed it for you and it's what setuptools itself is doing | 14:03 |
| *** ysandeep is now known as ysandeep|out | 14:14 | |
| *** marios|ruck is now known as marios|ruck|call | 14:31 | |
| *** sboyron is now known as Guest322 | 14:44 | |
| *** marios|ruck|call is now known as marios|ruck | 14:51 | |
| *** ykarel is now known as ykarel|away | 15:20 | |
| *** gthiemon1e is now known as gthiemonge | 15:49 | |
| *** marios|ruck is now known as marios|out | 15:55 | |
| *** jpena is now known as jpena|off | 16:00 | |
| *** rpittau is now known as rpittau|afk | 16:08 | |
| *** frenzy_friday is now known as anbanerj|rover | 16:19 | |
| *** sshnaidm is now known as sshnaidm|afk | 16:35 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/project-config master: Deprecate OpenStack-Ansible nspawn repositories https://review.opendev.org/c/openstack/project-config/+/797723 | 17:14 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/project-config master: Deprecate OpenStack-Ansible nspawn repositories https://review.opendev.org/c/openstack/project-config/+/797723 | 17:21 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/project-config master: Remove noop jobs for OpenStack-Ansible nspawn projects https://review.opendev.org/c/openstack/project-config/+/797727 | 17:26 |
| opendevreview | Shnaidman Sagi (Sergey) proposed openstack/project-config master: Allow RDO CI on ansible-collections-openstack https://review.opendev.org/c/openstack/project-config/+/797738 | 18:51 |
| sshnaidm|afk | tristanC, mnaser ^^ please take a look | 18:51 |
| opendevreview | Shnaidman Sagi (Sergey) proposed openstack/project-config master: Allow RDO CI on ansible-collections-openstack https://review.opendev.org/c/openstack/project-config/+/797738 | 18:52 |
| opendevreview | Ghanshyam proposed openstack/project-config master: End project gating for retiring scientific-wg repo https://review.opendev.org/c/openstack/project-config/+/797745 | 20:26 |
| opendevreview | Ghanshyam proposed openstack/project-config master: Remove scientific-wg from infra https://review.opendev.org/c/openstack/project-config/+/797747 | 20:38 |
| opendevreview | Ghanshyam proposed openstack/project-config master: Remove arch-wg from infra https://review.opendev.org/c/openstack/project-config/+/797748 | 20:43 |
| opendevreview | Ghanshyam proposed openstack/project-config master: Remove openstack-specs from infra https://review.opendev.org/c/openstack/project-config/+/797750 | 20:47 |
| opendevreview | Ghanshyam proposed openstack/project-config master: Remove project-navigator-data from infra https://review.opendev.org/c/openstack/project-config/+/797751 | 20:51 |
| opendevreview | Ghanshyam proposed openstack/project-config master: Remove enterprise-wg from infra https://review.opendev.org/c/openstack/project-config/+/797752 | 20:53 |
| opendevreview | Ghanshyam proposed openstack/project-config master: Remove workload-ref-archs from infra https://review.opendev.org/c/openstack/project-config/+/797774 | 21:01 |
| opendevreview | Ghanshyam proposed openstack/project-config master: Remove ops-tags-team from infra https://review.opendev.org/c/openstack/project-config/+/797775 | 21:03 |
| opendevreview | Ghanshyam proposed openstack/project-config master: Remove governance-uc from infra https://review.opendev.org/c/openstack/project-config/+/797777 | 21:05 |
| opendevreview | Ghanshyam proposed openstack/project-config master: Remove uc-recognition from infra https://review.opendev.org/c/openstack/project-config/+/797778 | 21:07 |
| *** dviroel is now known as dviroel|out | 21:13 | |
| opendevreview | Ghanshyam proposed openstack/project-config master: End proejct gating for puppet-openstack-specs https://review.opendev.org/c/openstack/project-config/+/797781 | 21:30 |
| opendevreview | Ghanshyam proposed openstack/project-config master: Remove puppet-openstack-specs from infra https://review.opendev.org/c/openstack/project-config/+/797784 | 21:38 |
| ianw | fungi / frickler: yeah, there's some NOTE type things in the Dockerfile about putting that plugin back in | 22:23 |
| JayF | eiddccidrhjinekeuiietvkhibkkvgikvrebdbjrhdvh | 22:29 |
| JayF | whoops, apologies | 22:29 |
| corvus | fungi: did we lose the zuul secret decrypting script on the server move? | 22:30 |
| fungi | JayF: that's one very long password you're going to need to reset, or a very dexterous cat | 22:30 |
| JayF | yubikey that has a super extra sensitive button | 22:31 |
| corvus | my guess is OG yubikey? | 22:31 |
| fungi | corvus: hmm... maybe, though the server may still exist as well. if you need the pypi creds they're also in our usual list | 22:31 |
| JayF | it's a USB-C yubikey, which has the capacitive button on both side, so if you have it in your laptop and shift just right, it activates :| | 22:31 |
| corvus | (there's a limited character set for keyboard compat) | 22:31 |
| fungi | i'll check if the old server can be revived | 22:31 |
| corvus | fungi: don't worry, i think i have a local copy :) | 22:32 |
| fungi | corvus: okay good, because the only zuul* server in rax-dfw now is zuul02.opendev.org | 22:33 |
| corvus | fungi: copied back to zuul02:~root | 22:33 |
| fungi | thanks | 22:33 |
| corvus | we could probably check that into zuul's tree. | 22:33 |
| fungi | yeah, not a bad idea, though it might also be simplified by exposing zuul's decrypt function as an entrypoint | 22:34 |
| fungi | but we have a number of random utility scripts in tools/ if memory serves, so would fit fine in there | 22:35 |
| corvus | considering it requires a zuul install, the entrypoint option may be best actually | 22:36 |
| fungi | alternative is to do it with openssl, but that's more fiddly | 22:40 |
| fungi | i suppose if we add a zuul-decrypt entrypoint, it would make sense to have it be able to fetch keys from zk as well in preparation for when we no longer maintain on-disk copies | 22:47 |
| opendevreview | Ian Wienand proposed openstack/project-config master: nodepool: pause gentoo and tumbleweed builds https://review.opendev.org/c/openstack/project-config/+/797790 | 23:26 |
| ianw | prometheanfire / frickler : ^ i've put some pastes of failures in the above change, full logs are @ https://nb01.opendev.org/ | 23:29 |
| ianw | unfortunately, i haven't seen the dib gate pass in a few days. i'll investigate now but i don't think we have a quick path to deploying a change right now | 23:30 |
| ianw | it looks like on centos8-stream "2021-06-21 10:54:17.806 | > Running scriptlet: grub2-tools-1:2.02-99.el8.x86_64 73/121" just hangs until the job times out | 23:35 |
| opendevreview | Merged openstack/project-config master: nodepool: pause gentoo and tumbleweed builds https://review.opendev.org/c/openstack/project-config/+/797790 | 23:56 |
| fungi | ianw: hanging during boot device detection? | 23:57 |
| ianw | fungi: maybe ... trying to put a node on hold now. i'm not sure if it's doing it on the builders | 23:58 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!