| opendevreview | Ian Wienand proposed opendev/system-config master: grafana: update to oss latest release https://review.opendev.org/c/opendev/system-config/+/825410 | 00:02 |
|---|---|---|
| opendevreview | ZhouHeng proposed openstack/project-config master: Retire neutron-fwaas* https://review.opendev.org/c/openstack/project-config/+/825412 | 00:38 |
| fungi | clarkb: use ${hash} in place of all the ${commit} macros, or just in the file option? | 00:41 |
| Clark[m] | fungi: my change only modified the file option | 00:41 |
| Clark[m] | It is very proof of concept :) | 00:42 |
| fungi | ahh, okay thanks | 00:42 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Drop gitweb dependencies https://review.opendev.org/c/opendev/system-config/+/825337 | 00:45 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Fix mixed spaces and hard tabs in Gerrit config https://review.opendev.org/c/opendev/system-config/+/825338 | 00:45 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Use Gitea for Gerrit's code browser URLs https://review.opendev.org/c/opendev/system-config/+/825339 | 00:45 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: DNM: Fail our Gerrit testing for an autohold https://review.opendev.org/c/opendev/system-config/+/825396 | 00:45 |
| fungi | hopefully i'll have a held node to inspect when i wake up | 00:45 |
| fungi | Clark[m]: it works! https://172.99.69.94/c/x/test-project/+/1/1/file-1.txt | 01:54 |
| opendevreview | Eduardo Santos proposed openstack/diskimage-builder master: Fix openSUSE images and bump them to 15.3 https://review.opendev.org/c/openstack/diskimage-builder/+/825347 | 02:24 |
| *** frenzy_friday is now known as frenzyfriday|ruck | 03:45 | |
| *** mtreinish_ is now known as mtreinish | 05:13 | |
| *** ysandeep|away is now known as ysandeep | 05:36 | |
| *** ysandeep is now known as ysandeep|brb | 06:30 | |
| *** ysandeep|brb is now known as ysandeep | 07:14 | |
| opendevreview | Gregory Thiemonge proposed openstack/project-config master: Adding nested-virt pools for centos-9-stream https://review.opendev.org/c/openstack/project-config/+/825435 | 08:07 |
| chkumar|rover | Hello infra, https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-content-provider&project=openstack/tripleo-common is failing with mirror issue | 08:31 |
| chkumar|rover | Status code: 404 for https://mirror.bhs1.ovh.opendev.org/centos/8-stream/AppStream/x86_64/os/repodata/09255ba7c10e01afeb0d343667190f9c3e42d0a6099f887619abcb92ea0378db-filelists.xml.gz (IP: 158.69.73.218) | 08:31 |
| chkumar|rover | please have a look, thanks! | 08:32 |
| *** jpena|off is now known as jpena | 08:37 | |
| chkumar|rover | It seems all are coming from https://mirror-int.dfw.rax.opendev.org/ | 09:24 |
| opendevreview | chandan kumar proposed opendev/system-config master: Use facebook mirror for CentOS Stream 8 https://review.opendev.org/c/opendev/system-config/+/825446 | 09:48 |
| *** ysandeep is now known as ysandeep|lunch | 09:57 | |
| opendevreview | chandan kumar proposed opendev/system-config master: Use facebook mirror for CentOS Stream 8 https://review.opendev.org/c/opendev/system-config/+/825446 | 10:02 |
| *** ysandeep|lunch is now known as ysandeep | 10:48 | |
| *** rlandy|out is now known as rlandy|ruck | 11:15 | |
| *** priteau_ is now known as priteau | 11:16 | |
| *** dviroel__ is now known as dviroel | 11:25 | |
| *** frenzy_friday is now known as frenzyfriday|ruck | 11:42 | |
| *** bhagyashris__ is now known as bhagyashris | 11:49 | |
| *** ysandeep is now known as ysandeep|brb | 12:06 | |
| fungi | chkumar|rover: mirror.bhs1.ovh.opendev.org is a different mirror, so what do you mean? | 12:46 |
| fungi | oh, i see, jobs running outside rax-dfw are trying to access a mirror url which can only be reached by systems located there | 12:48 |
| fungi | looks like whatever changed happened between the last successful run at 2022-01-20 02:05:51 and the first retry at 2022-01-20 07:59:09 | 12:52 |
| fungi | so roughly a 6-hour window | 12:52 |
| jrosser | i see the same thing on openstack-ansible jobs too | 13:01 |
| fungi | yeah, broadening the search: https://zuul.opendev.org/t/openstack/builds?result=RETRY_LIMIT&skip=0 | 13:06 |
| fungi | looks like it's stuff using centos-8-stream | 13:07 |
| fungi | so maybe an image update happened in that timeframe and it's the result of something which changed in dib | 13:07 |
| fungi | though i think we consume dib from tagged releases, and there's not been a new release of dib in ~1.5 months | 13:08 |
| jrosser | the irony is that my job failing due to this mirror thing was to help understand why /etc/ssh/sshd_config.d seemed missing on the centos-8-stream nodes | 13:13 |
| fungi | looks like new centos-8-stream images uploaded roughly 7-8 hours ago in these regions: iweb-mtl01, ovh-bhs1, ovh-gra1, rax-dfw | 13:14 |
| fungi | the images in these regions are still over a day old: airship-kna1, inmotion-iad3, limestone-regionone, rax-iad, rax-ord, vexxhost-ca-ymq-1, vexxhost-sjc1 | 13:17 |
| fungi | so if we see any failures in the latter, then it's probably not from updated centos-8-stream images in that timeframe | 13:17 |
| fungi | https://zuul.opendev.org/t/openstack/build/e96fa6aeec4449d6b5bae3a2705a5d2c/log/zuul-info/inventory.yaml#40 | 13:18 |
| fungi | so i think that rules out it being a problem in a new image | 13:19 |
| fungi | there were some rpm-related changed which merged to the zuul-jobs repo recently, but hours before the problem seems to have started: https://review.opendev.org/822503 https://review.opendev.org/825118 | 13:23 |
| fungi | nothing relevant changed in openstack/openstack-zuul-jobs or opendev/base-jobs recently either | 13:25 |
| fungi | there was a zuul restart, but it concluded at 21:48 and we had successful runs after that | 13:26 |
| frickler | fungi: isn't that just what we had with cs9 a couple of days ago, syncing from a broken mirror? https://review.opendev.org/c/opendev/system-config/+/825446 | 13:31 |
| fungi | frickler: no, these jobs are failing trying to reach the internal address of the rax-dfw mirror from other providers | 13:32 |
| fungi | something is causing us to not set the correct mirror url on the nodes | 13:32 |
| fungi | oh, or maybe i'm misreading it | 13:34 |
| fungi | frickler: you're right, this one is hitting the iweb-mtl01 mirror: https://zuul.opendev.org/t/openstack/build/ec70ed4cdd76409cafbd7792564e5445/console | 13:35 |
| fungi | i got tunnel-vision after chkumar|rover said "It seems all are coming from https://mirror-int.dfw.rax.opendev.org/" | 13:35 |
| fungi | that's not true at all | 13:36 |
| fungi | i'll check the mirror updates, maybe it's just a stale centos-8-stream mirror | 13:36 |
| fungi | mirrors for that are up to date though: https://mirror.mtl01.iweb.opendev.org/centos/timestamp.txt | 13:38 |
| fungi | but it could be we're successfully mirroring from a stale source this time, instead of one that's refusing connections | 13:38 |
| fungi | currently we mirror that from rsync://mirror.dal10.us.leaseweb.net/centos/8-stream/ | 13:41 |
| fungi | yeah, 404 for the same files there, e.g. https://mirror.dal10.us.leaseweb.net/centos/8-stream/AppStream/x86_64/os/repodata/678056e5b64153ca221196673208730234dd72f03397a3ab2d30fea01392bd87-primary.xml.gz | 13:42 |
| fungi | i missed chkumar|rover's proposed change, saw his comments in here and started troubleshooting | 13:44 |
| * fungi sighs | 13:44 | |
| fungi | this is what i get for trying to fix things before coffee | 13:45 |
| fungi | i do note that the files it's complaining about don't appear on facebook's mirror either: https://mirror.facebook.net/centos/8-stream/AppStream/x86_64/os/repodata/ | 13:46 |
| fungi | seems to match what we're serving: https://mirror.mtl01.iweb.opendev.org/centos/8-stream/AppStream/x86_64/os/repodata/ | 13:47 |
| *** ysandeep|brb is now known as ysandeep | 13:47 | |
| fungi | chkumar|rover: are you sure switching to the facebook mirror will address this problem? everything there looks the same to me as what we're already serving | 13:50 |
| rlandy|ruck | fungi: hi chkumar|rover is out at the moment | 13:57 |
| fungi | same content on cern's mirror too: https://linuxsoft.cern.ch/centos/8-stream/AppStream/x86_64/os/repodata/ | 13:58 |
| rlandy|ruck | fungi: chkumar|rover was debugging with amoralej in their morning | 14:00 |
| fungi | the repomd.xml file is identical across all of those mirrors too | 14:00 |
| rlandy|ruck | checking the LP bug | 14:01 |
| rlandy|ruck | hmmm .. maybe the mirrors updated - we are starting to see some jobs run - rechecking the content provider job | 14:04 |
| rlandy|ruck | https://zuul.opendev.org/t/openstack/stream/8853457aca24402ea93746e3721d2c9d?logfile=console.log | 14:05 |
| fungi | rlandy|ruck: looking through our mirror-update logs, it looks like there may have been a significant mirror update at leaseweb which caused the files in question to be removed at 06:55 utc, and then replaced at 12:57 utc | 14:16 |
| rlandy|ruck | fungi: why all this upheaval with mirrors all of a sudden? | 14:17 |
| fungi | great question. do you know anyone maintaining the primary centos mirror infrastructure? maybe there was a large file change in the past 24 hours | 14:18 |
| fungi | from what i can tell looking at https://static.opendev.org/mirror/logs/rsync-mirrors/centos.log it seems like the indices there were changed to different ones and then changed back | 14:21 |
| fungi | like someone updated content and then undid/reverted it | 14:22 |
| fungi | likely with some number of hours delay as we're several mirrors down the chain from the primary | 14:23 |
| fungi | it's also possible, since we built some new centos-8-stream images during that window, that we'll need fresh images now that the mirrors are back in sync | 14:31 |
| rlandy|ruck | fungi: I'll ask our rdo team - they have some better connections with the centos team | 14:36 |
| fungi | thanks, it could help confirm our hypothesis i guess | 14:36 |
| rlandy|ruck | looks like things are progressing now | 14:37 |
| rlandy|ruck | I see one successful content-provider on c8 | 14:37 |
| fungi | awesome | 14:37 |
| fungi | i've gone ahead and issued a delete for the centos-8-stream image we built ~9.5 hours ago, since it only deployed to roughly a third of our providers | 14:38 |
| rlandy|ruck | I think chkumar|rover is betting on the fact that the facebook mirrors are more stable | 14:39 |
| rlandy|ruck | but in this case | 14:39 |
| rlandy|ruck | it would not have helped, iiuc | 14:39 |
| fungi | well, it might have helped, but maybe by happenstance if it had updated sooner than the one we're copying from currently | 14:39 |
| fungi | the mirror we had a problem with for stream 9 was a completely different operator than the one we're using for stream 8, and failed in a different way (it spontaneously began refusing rsync connections) | 14:40 |
| fungi | in that case switching mirrors was obviously necessary | 14:42 |
| opendevreview | James E. Blair proposed opendev/zone-zuul-ci.org master: Add google site verification https://review.opendev.org/c/opendev/zone-zuul-ci.org/+/825519 | 15:12 |
| *** frenzyfriday|ruck is now known as frenzyfriday | 15:16 | |
| *** bhagyashris is now known as bhagyashris|ruck | 15:17 | |
| *** rcastillo is now known as rcastillo|rover | 15:18 | |
| opendevreview | Merged opendev/zone-zuul-ci.org master: Add google site verification https://review.opendev.org/c/opendev/zone-zuul-ci.org/+/825519 | 15:24 |
| *** ysandeep is now known as ysandeep|away | 15:25 | |
| *** marios is now known as marios|out | 16:03 | |
| clarkb | fungi: re gerrit gitea links, I guess the next step is to see what feedback we get on a gerrit bug and then if they are happy with my patch I can clean it up and push it? | 16:17 |
| fungi | yeah, the prototype seems viable to me, but if you want to check out the links in polygerrit on the held node too that's probably good | 16:32 |
| clarkb | fungi: ya I checked the links on the page you linked to | 16:33 |
| clarkb | it looks like a sha1 to me though I guess I didn't check it is the right sha1 | 16:33 |
| fungi | cool, so no obvious issues | 16:33 |
| clarkb | let me double check the value looks correct | 16:33 |
| fungi | right, it does seem to match the hash on the other link which was already correct | 16:34 |
| clarkb | ya the values appear correct to me. So ya I think we file a bug (has that happened yet?) with an indication we can write a patch and ask for clarification if adding a new value like ${hash} is their preferred method | 16:35 |
| fungi | i did not have time to dig up my google login to file anything yet, it's been a morning full of meetings and fires | 16:35 |
| clarkb | no rush on my end. I'm catching up on a bunch of zuul stuff myself | 16:37 |
| AlekseiPavlov | Hello everyone! I have question about gerrit. We make commit to openstack/cinder, but gerrit doesn't send feedback to https://review.opendev.org/c/openstack/cinder/+/825492 when my build in jenkins finished successful, but send message, if build finished with fail.Why ? | 16:42 |
| clarkb | AlekseiPavlov: what specific feedback are you expecting? ssh event stream events? | 16:44 |
| AlekseiPavlov | no, just a message in Change log | 16:45 |
| clarkb | AlekseiPavlov: are you trying to vote Verified +1 on successful results? cinder limits which accounts can do this via the cinder-ci group https://review.opendev.org/admin/groups/b84ae7e7faf9049af9273e8599c3e9b995a86a94,members which is currently empty | 16:46 |
| clarkb | you need to avoid voting. I assume it is trying to +1 and that is why this isn't working as expected | 16:46 |
| AlekseiPavlov | probably yes, i'll try to configure gerrit plugin to avoid voting | 16:47 |
| AlekseiPavlov | no, it didn't help | 16:49 |
| AlekseiPavlov | https://ibb.co/zfswVS4 i wanna get message like this, but about success build | 16:49 |
| clarkb | AlekseiPavlov: is Jenkins configured to talk to gerrit via ssh or http? | 16:50 |
| AlekseiPavlov | clarkb, thanks | 16:50 |
| AlekseiPavlov | i fix my proble | 16:50 |
| AlekseiPavlov | i fix my problem | 16:50 |
| clarkb | once you have identified the method you can try reproducing directly | 16:50 |
| clarkb | ah ok. what was the issue? | 16:50 |
| AlekseiPavlov | i forgot delete 1 in verify in jenkin's job configuration for gerrit plugin ) | 16:52 |
| fungi | okay, meetings are done. if there's no other fires i'm going to try to help figure out the new pbr test regressions | 16:57 |
| clarkb | fungi: were those mentioned somewhere? | 16:59 |
| fungi | #openstack-oslo | 16:59 |
| clarkb | ah | 16:59 |
| fungi | i reproduced them yesterday with 825380 and am trying to reproduce them locally now | 16:59 |
| fungi | looks like something has started to cause unit tests for >=py37 to fail | 17:00 |
| fungi | some uwsgi related tests and a couple others | 17:00 |
| fungi | oddly py27-py36 are passing | 17:00 |
| fungi | so probably a change in some dep which released a new version dropping 3.6 support and the successful runs are using an older release | 17:01 |
| clarkb | in the case of PBR just about the only dep is setuptools | 17:01 |
| fungi | right | 17:02 |
| fungi | though also there's changes in stdlib, e.g. importlib stuff | 17:03 |
| clarkb | its looking for text in stdout running the package install that indicates a script was installed | 17:04 |
| clarkb | setuptools latest requires python >=3.7 | 17:05 |
| clarkb | https://pypi.org/project/setuptools/59.6.0/ appears to be the most recent that supports python 3.6 | 17:05 |
| clarkb | the code to emit the text that pbr is looking for is in setuptools latest and 59.6.0 so likely something more subtle | 17:07 |
| clarkb | :/ | 17:07 |
| clarkb | unrelated, my git uses unicode symbols like → in checkout output now | 17:09 |
| fungi | ugh | 17:09 |
| fungi | i hope there's a .gitconfig option to disable | 17:09 |
| clarkb | fungi: I suspect #2974 and/or #2973 based on the git log in setuptools | 17:16 |
| fungi | mebbe | 17:18 |
| clarkb | though I suppose it is possible the code is just not running and the issue isn't related to logging | 17:21 |
| clarkb | fungi: I'm fairly certain this is a logging issue beacuse the rpm_version command emits 0.0.0 which is via a print() but it doesn't emit the log.info() output | 17:25 |
| clarkb | that indicates to me that the code is executing and changes to setuptools logging (as indicated with those issue/PR numbers) are likely to blame | 17:25 |
| clarkb | I suspect set_threshold is to blame. They are doing that to scale up distutils.log 1,2,3,4,5 values to 10,20,30,40,50 python logging values. But if the values coming in are already scaled up then you'll end up with 100,200,300,400,500 and nothing will be logged | 17:27 |
| clarkb | hrm except the distutils.log _log method has a guard against that and raises a value error if the level is not one of 1,2,3,4,5 | 17:31 |
| *** jpena is now known as jpena|off | 17:37 | |
| clarkb | ok its definitely related to the setuptools version | 17:48 |
| fungi | downgrading setuptools fixes it? | 17:48 |
| fungi | er, works around it i mean | 17:49 |
| clarkb | yes and now with hacky debugging I think something is setting the theshold to 2 not 1. INFO is 1 | 17:52 |
| clarkb | oh wait INFO is 2 | 17:53 |
| clarkb | verbosity == 1 == INFO == 2 | 17:54 |
| fungi | so we need to up the verbosity? | 17:55 |
| fungi | er, no if it's like syslog's loglevels then we need a threshold lower or equal to the message's loglevel to capture it | 17:56 |
| clarkb | ya something is bumping up the threshold and I'm trying to understand that | 17:57 |
| clarkb | we are going from a theshold of 2 to 3. Info is 2 so get silenced | 17:57 |
| fungi | makes sense | 17:57 |
| clarkb | oh hrm no this is even more complicated. I think something is trying to set it to 2 but failing and so we stay at 3? | 17:58 |
| fungi | in pbr/testr_command.py? | 17:59 |
| clarkb | no in setuptools/distutils | 17:59 |
| clarkb | early on something is calling set_threshold to increase verbosity to an INFO level. But later when we try to actually log the message the Log.threshold value is the original higher value | 17:59 |
| clarkb | hardcoding the default level to INFO from WARN fixes it. I feel like this is some sort of order of oeprations thing introduced by that monkey patching change | 18:07 |
| fungi | so the expectation is this arrived in setuptools 60.2.0 (the first release that commit appeared in) | 18:10 |
| fungi | "Setuptools now relies on the Python logging infrastructure to log messages. Instead of using distutils.log.*, use logging.getLogger(name).*" (from the changelog) | 18:11 |
| fungi | we do a lot of "from distutils import log" in pbr | 18:11 |
| clarkb | distutils.log is still the correct interface according to the legacy migration doc. I don't think the python logging stuff is fully there even though the changelog asys to use it. Also older setuptools won't have proper python logging set up | 18:15 |
| fungi | right, we'd have to maintain some sort of split logic to handle that | 18:15 |
| clarkb | I think that something is creating a new distutils.log.Log() object after verbosity is being set which causes it to fall back to its default value of WARN | 18:16 |
| clarkb | but when I try to log that I get exceptions in some io writer thing because that apparently messes with stdout | 18:16 |
| fungi | clarkb: fwiw, using setuptools 60.1.1 doesn't fix those tests for me, or maybe doesn't fix all of them (could be we have more than one problem) | 18:19 |
| fungi | though it did still give me the same rpm version test error, so i think it's a regression earlier than 60.2.0 where that pr landed | 18:22 |
| fungi | i'm trying the last 59.x.x now for comparison | 18:23 |
| clarkb | fungi: ya commenting out the monkey patch didn't fix it. I have confirmed that the id() of the Log object that set_threshold is called on is different than the id() of the object that log.info is called on later | 18:25 |
| clarkb | the issue definitely seems to be that logging is set up and the threshold is set, then somewhere along the line a new object is create and the updated config is no longer valid | 18:25 |
| fungi | setuptools 59.8.0 doesn't exhibit this, at least. i'll try to bisect | 18:26 |
| fungi | using 60.0.5 next | 18:27 |
| fungi | i have a feeling we're looking at something which changed in either 60.0.0 or 60.1.0 | 18:28 |
| clarkb | 34defd4d420e31e7c4cefe3e44145c6fe8b41eca maybe | 18:29 |
| clarkb | hrm no that is test only | 18:30 |
| fungi | unfortunately my workstation is a bit underpowered/overloaded for running pbr unit tests, so i get a fair number of fixtures._fixtures.timeout.TimeoutException test failures | 18:31 |
| clarkb | you can run it against a single failing test | 18:33 |
| clarkb | test_custom_rpm_version_py_command is the one I've been using | 18:33 |
| fungi | yeah, seeing the same failures in 60.0.5 | 18:33 |
| fungi | what's the stestr syntax for that? | 18:33 |
| clarkb | looks like 60.0.0 - 60.0.3 were straight up broken in setuptools due to a missing variable. and 60.0.4 exhibits this issue | 18:33 |
| clarkb | fungi: I do tox -e py38 -- test_custom_rpm_version_py_command | 18:34 |
| fungi | okay, so it could be anywhere between working 59.8.0 and failing 60.0.4 | 18:34 |
| clarkb | ya | 18:35 |
| fungi | bisecting in projects without a green trunk is... unpleasant | 18:35 |
| *** amoralej is now known as amoralej|off | 18:35 | |
| fungi | maybe backporting the fix from 60.0.4 for the missing variable to earlier releases will help me make progress | 18:36 |
| fungi | and could their commit messages be any more useless? | 18:37 |
| fungi | i guess it's eba2bcd "Add support for 'platsubdir'. Fixes pypa/distutils#85." | 18:38 |
| clarkb | ya I think that is it | 18:38 |
| clarkb | also good idea | 18:38 |
| fungi | at least that's a very simple patch | 18:39 |
| clarkb | 60.0.0 with that patch applied fails with this issue | 18:40 |
| clarkb | I think that means it is between 59.8.0 and 60.0.0 | 18:40 |
| fungi | okay, so now we're down to bisecting commits between those two tags, yeah | 18:40 |
| clarkb | #2896: Setuptools once again makes its local copy of distutils the default. To override, set SETUPTOOLS_USE_DISTUTILS=stdlib. | 18:41 |
| clarkb | I wonder if that is it. The vendored copy is just broken | 18:41 |
| clarkb | yes passes if I set that var | 18:41 |
| clarkb | so ya something about loading distutils is breaking the log level settings | 18:42 |
| fungi | yeah, b6fcbbd0 "Restore local distutils as the default." looks like the only substantive change there | 18:42 |
| fungi | testing with SETUPTOOLS_USE_DISTUTILS=stdlib and latest setuptools now | 18:43 |
| fungi | yep, that solves it | 18:44 |
| fungi | just adding it to the setenv in [testenv] | 18:45 |
| fungi | clarkb: does that seem like an acceptable temporary workaround? | 18:45 |
| fungi | i'll push it up as a wip | 18:46 |
| clarkb | I think its probably ok, this seems to only affect logging | 18:48 |
| clarkb | whcih means pbr will still work elsewhere just less verbosely | 18:48 |
| fungi | https://review.opendev.org/825665 with an actual commit message | 18:51 |
| clarkb | fungi: should we file a bug against setuptools? | 18:51 |
| clarkb | Something along the lines of the switch back to local vendored distutils results in a log level of WARN from distutils.log instead of INFO as was previously the default. | 18:52 |
| fungi | likely. you seem to have a bit better grasp on the specifics of the loglevel bit | 18:52 |
| fungi | a minimal reproducer would likely help | 18:52 |
| clarkb | ya thats the problem with setuptools. I have no idea how to reproduce any of this without pbr | 18:53 |
| clarkb | the whole thing is a giant mess of magic and knowing how to plug in is tricky. Maybe the thing to do is fine a log message that setuptools itself is already emitting | 18:53 |
| clarkb | hrm no because logging seems to work there :/ | 18:57 |
| clarkb | this is specific to the command hooks then? | 18:57 |
| fungi | could be they didn't notice because they don't test them, so missed adding support | 18:57 |
| clarkb | ya I always get lost right at this point. How does the command hook stuff get registered wtih setuptools. I think the thing that makes it confusing is we do a bunch of PBR side processing that emits it into setup() | 19:00 |
| fungi | as soon as i get caught up on list moderating, i'll start putting together the gerrit gitweb config bug report | 19:02 |
| fungi | and then hopefully on to pruning backups | 19:03 |
| clarkb | I'm getting hopelessly lost trying to figure out how our custom command differs from the sdist command | 19:08 |
| clarkb | fungi: ok I can reproduce using the build_scripts command. It emits an info log level entry for chmod | 19:22 |
| clarkb | but 60.0.0 doesn't do that | 19:22 |
| fungi | perfect | 19:24 |
| fungi | at least that gives them something to replicate the problem without risk of them just blaming pbr | 19:25 |
| clarkb | https://github.com/pypa/setuptools/issues/3038 | 19:36 |
| fungi | thanks! | 19:38 |
| fungi | https://bugs.chromium.org/p/gerrit/issues/detail?id=15589 (for the gitweb config issue) | 20:34 |
| *** artom__ is now known as artom | 20:47 | |
| clarkb | lgtm. | 21:00 |
| fungi | also the setuptools distutils workaround for pbr needed a revision, i failed to notice that [testenv:cover] separately overrode setenv | 21:05 |
| clarkb | I'll rereview momentarily | 21:05 |
| clarkb | also getting https://etherpad.opendev.org/p/project-renames-2022-01-24 together for renames | 21:06 |
| clarkb | do we know if there are rename changes yet? | 21:06 |
| clarkb | we'll also append to that the gerrit upgrade plabn | 21:06 |
| corvus | #status log manually moved new rebuilds of old zuul docs into position on project.zuul afs volume | 21:15 |
| opendevstatus | corvus: finished logging | 21:15 |
| clarkb | fungi: should we split https://review.opendev.org/c/openstack/project-config/+/822222 in two or force merge it? | 21:16 |
| clarkb | specifically they have added the project to the zuul projects.yaml config and the projects aren't known by zuul yet | 21:16 |
| clarkb | ianw: I put a rough outlien of the gerrit upgrade process on https://etherpad.opendev.org/p/project-renames-2022-01-24 feel free to update that with more info | 21:17 |
| fungi | clarkb: i'd be okay with trying to split it, i think previously we bypassed testing to merge those | 21:19 |
| clarkb | ok I'll split it then | 21:21 |
| opendevreview | Clark Boylan proposed opendev/project-config master: Record project renames happening January 24, 2022 https://review.opendev.org/c/opendev/project-config/+/825677 | 21:22 |
| clarkb | gmann: fungi: one thing I notice is that skyline is adding the publish to pypi jobs but I think we've established their source code repos won't work that way? | 21:25 |
| clarkb | I'm going to push the updates anyway, but I'm warning openstack here that those jobs may not do what you think they do | 21:25 |
| ianw | clarkb: thanks, looking | 21:25 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Rename skyline/skyline* project to openstack/skyline* https://review.opendev.org/c/openstack/project-config/+/822222 | 21:26 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Configure renamed skyline projects with openstack jobs https://review.opendev.org/c/openstack/project-config/+/825680 | 21:26 |
| fungi | clarkb: yeah, it's probably fine for them to have the pypi release template, with the understanding that there's no point in them making release requests for anything until they're in a state where those jobs will actually work | 21:44 |
| clarkb | infra-root Luca identified an issue with gerrit that could cause runtime errors of plugins. I've tried grepping our error log for some of the strings taht show up in his tracebacks | 21:44 |
| clarkb | There is an unmerged fix which I'll propose to our gerrit builds momentarily so that we are prepared wit htaht should it become necessary | 21:45 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Fixup Gerrit deps https://review.opendev.org/c/opendev/system-config/+/825684 | 21:58 |
| clarkb | infra-root ^ that is the gerrit thing. I've added links to the things and a suggested plan. TL;DR is since we don't seem to be broken right now we can wait until tomorrow when the upstream fix should be landing. Then rebuild our images. If upstream doesn't land the fixup we can alnd this patching fix | 21:58 |
| fungi | thanks! | 21:59 |
| clarkb | Or I suppose if we discover we are broken go ahead and land one or the other depending on how far along it is | 22:00 |
| fungi | yep | 22:00 |
| corvus | howdy, i would like to rolling-restart zuul soon | 22:02 |
| clarkb | no objections from me | 22:03 |
| corvus | (just waiting a few mins for promote to finish) | 22:03 |
| clarkb | will you be restarting executors non gracefully again? | 22:03 |
| clarkb | if so I can warn the release team | 22:03 |
| fungi | sounds good to me | 22:03 |
| corvus | i'm mulling on that ... maybe we have enough time to graceful them today/ | 22:03 |
| clarkb | hard to say, our timeout is 3 hours so it may stretch at least that long? | 22:04 |
| corvus | s///?/ | 22:04 |
| corvus | hrm good point | 22:04 |
| corvus | maybe we should just non-graceful | 22:04 |
| fungi | our timeout is 3+3 hours technically, right? | 22:04 |
| fungi | (pre/run + post) | 22:04 |
| clarkb | fungi: I think post is a shorter timeout but ya its 3 hours + some value | 22:05 |
| clarkb | I can warn the release team | 22:06 |
| corvus | k | 22:06 |
| corvus | thx | 22:06 |
| fungi | oh, i guess at some point the post-run timeout was split out to a separate config option rather than just using the one that covers the other phases | 22:06 |
| clarkb | ya | 22:07 |
| clarkb | fungi: our issue is a dup of https://github.com/pypa/setuptools/issues/3035 | 22:22 |
| clarkb | The debugging on that issue was a little more in depth and shows two instances of the moduel which explains why we set the threshold then it gets ignored | 22:23 |
| gmann | clarkb: yeah, its a TODO for them to work on. we are aware of those limitation, let | 22:24 |
| gmann | let's see when they start working on those | 22:24 |
| clarkb | gmann: ok just calling it out since the jobs are added | 22:26 |
| corvus | okay promotes are done, starting the zuul maint now | 22:28 |
| corvus | pulling images | 22:28 |
| clarkb | fungi: project-rename is the topic you expect for the rename changes right? I went ahead and set those values and also updated the wiki entry so that hopefully people see it there in addition to our docs | 22:29 |
| fungi | yeah, i thhink that's what our documentation says to use | 22:31 |
| clarkb | Ok I think the rename portion of the maintenance is pretty much ready for people to look it over then. | 22:32 |
| clarkb | *of the maintenance etherpad | 22:32 |
| corvus | restarting zuul01 scheduler | 22:33 |
| *** dviroel is now known as dviroel|out | 22:34 | |
| corvus | looking good. i'm going to restart zuul02 scheduler and web now. | 22:44 |
| corvus | they're back up; going to restart mergers now | 22:56 |
| corvus | done; now going to hard-restart executors | 22:57 |
| corvus | #status log performed (mostly) rolling restart of zuul on commit 548eafe0b5729e78ab4024abea98b326678d83d8 | 23:00 |
| opendevstatus | corvus: finished logging | 23:00 |
| fungi | thanks! | 23:01 |
| *** rcastillo|rover is now known as rcastillo|out | 23:01 | |
| *** ysandeep|away is now known as ysandeep | 23:52 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!