| opendevreview | Ian Wienand proposed opendev/system-config master: infra-prod: fix infra-prod-service-zookeeper soft dependency https://review.opendev.org/c/opendev/system-config/+/821504 | 00:03 |
|---|---|---|
| fungi | ianw: thanks for running that down! | 00:04 |
| opendevreview | Merged opendev/system-config master: infra-prod: fix infra-prod-service-zookeeper soft dependency https://review.opendev.org/c/opendev/system-config/+/821504 | 00:17 |
| ianw | heh well i created the problem :) | 00:18 |
| fungi | you can't make config omelette without breaking config eggs | 00:38 |
| opendevreview | Ian Wienand proposed opendev/system-config master: zuul-*: use multiline formatter https://review.opendev.org/c/opendev/system-config/+/821508 | 03:55 |
| *** ysandeep|out is now known as ysandeep | 04:20 | |
| *** frenzy_friday is now known as anbanerj|ruck | 04:35 | |
| ianw | ok, LE periodic job has run again : https://zuul.opendev.org/t/openstack/build/21aa2fe4c07f4710809517fd1f6a3c99 | 06:15 |
| *** bhagyashris_ is now known as bhagyashris | 06:21 | |
| *** ysandeep is now known as ysandeep|intv | 06:25 | |
| *** ysandeep|intv is now known as ysandeep | 08:51 | |
| *** ysandeep is now known as ysandeep|lunch | 08:58 | |
| hrw | morning | 09:14 |
| hrw | is there some doc about migrating to other SSO login? | 09:15 |
| hrw | I managed to lock myself out of login.ubuntu.com so will need to create new account there and regain kolla permissions | 09:16 |
| *** ysandeep|lunch is now known as ysandeep | 09:21 | |
| frickler | hrw: afaict if you lost access to the old login, things get difficult. but other kolla folks should be able to give access to the new account once you created it | 09:37 |
| hrw | ok, thanks | 09:37 |
| hrw | will check how stuff goes | 09:38 |
| hrw | frickler: ok, cannot add work email to new account as it is assigned to old account | 09:48 |
| frickler | hrw: yes, we would need to remove that from the old account first. however this is blocked by some cleanup we still need to perform on existing accounts. maybe clarkb later can tell us something about the current status | 09:49 |
| hrw | ok | 09:49 |
| frickler | hrw: actually I think I could try to remove your email address from the old account, if you want | 09:53 |
| hrw | if you can then it would be great | 09:54 |
| hrw | but - how do you know that I am who I am? | 09:54 |
| frickler | hrw: well if you are not you, then the real hrw would complain later I guess. you also need to verify your email in order to add it to the new account | 09:56 |
| hrw | ;) | 09:56 |
| hrw | right | 09:56 |
| frickler | hrw: o.k., that seems to have worked, please try again | 09:59 |
| hrw | thanks. added and set as preferred | 09:59 |
| hrw | and will have fresh history on launchpad ;D | 10:00 |
| *** rlandy is now known as rlandy|ruck | 11:10 | |
| *** jpena|off is now known as jpena | 11:27 | |
| *** dpawlik7 is now known as dpawlik | 11:56 | |
| opendevreview | Martin Kopec proposed opendev/system-config master: Update Interop doc https://review.opendev.org/c/opendev/system-config/+/816982 | 12:48 |
| fungi | hrw: did something go wrong with the 2fa setup? | 12:55 |
| hrw | fungi: first I forgot to do 2fa, then covid came and forgot even more. then got logged out. | 13:12 |
| hrw | fungi: I am sorting that out. | 13:12 |
| fungi | ahh, okay | 13:14 |
| hrw | fungi: in worst case I will use second account. Kolla PTL is aware of situation (the only team where I have core) | 13:14 |
| opendevreview | Merged zuul/zuul-jobs master: Print version of installed podman https://review.opendev.org/c/zuul/zuul-jobs/+/814604 | 13:45 |
| *** ysandeep is now known as ysandeep|dinner | 14:34 | |
| *** ysandeep|dinner is now known as ysandeep | 15:14 | |
| *** ysandeep is now known as ysandeep|out | 16:31 | |
| clarkb | fungi: ianw looks like certs must've renewed as we didn't get emails indicating the fix worked | 16:40 |
| fungi | yep, confirmed | 16:41 |
| fungi | we have new "not after" dates in the certs now | 16:41 |
| *** marios is now known as marios|ou | 16:44 | |
| *** marios|ou is now known as marios|out | 16:44 | |
| clarkb | I'm working on fixing up my bullseye docker image builds this morning. One thing it is tripping over is installing libffi6 on bullseye. Any idea why distros don't do meta pacakges for libs like that so you can install libffi instead? | 16:50 |
| clarkb | fixable via bindep rules but curious why this has to be as painful as it is | 16:51 |
| fungi | on debian it's usually because transitions need e.g. both libffi5 and libffi6 available for install at least for some short period | 16:53 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Properly build bullseye uwsgi-base docker images https://review.opendev.org/c/opendev/system-config/+/821339 | 16:53 |
| fungi | otherwise everything which links libffi would have to be rebuilt and uploaded at the same time as libffi6, which on its own may not sound complicated, but multiply that by thousands of similar libs and you quickly get yourself deadlocked between multiple library transitions or have to serialize them to the point where you can't finish transitions faster than you start them | 16:55 |
| clarkb | I see | 16:56 |
| fungi | it's quite common for the soname version of a lib to be embedded into the package name, as a result | 16:59 |
| fungi | and usually it's assumed those will only ever be consumed through declared package dependencies anyway, so users aren't inconvenienced by the package name changes | 17:00 |
| fungi | when you're installing them directly to build something against, yeah that becomes annoying | 17:01 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update the hound image to bullseye https://review.opendev.org/c/opendev/system-config/+/821329 | 17:04 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Properly build bullseye uwsgi-base docker images https://review.opendev.org/c/opendev/system-config/+/821339 | 17:08 |
| clarkb | fungi: buster-backports contains packages that are in bullseye right? This means when moving from buster to bullseye I can drop the backports specifier and keep the same version? | 17:11 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update matrix-eavesdrop image to bullseye https://review.opendev.org/c/opendev/system-config/+/821332 | 17:12 |
| clarkb | That ^ update goes with that assumption | 17:12 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update refstack image to bullseye https://review.opendev.org/c/opendev/system-config/+/821335 | 17:16 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update the accessbot image to bullseye https://review.opendev.org/c/opendev/system-config/+/821328 | 17:19 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update limboria ircbot to bullseye https://review.opendev.org/c/opendev/system-config/+/821330 | 17:22 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Install Limnoria from upstream https://review.opendev.org/c/opendev/system-config/+/821331 | 17:22 |
| clarkb | fungi: do you understand why https://zuul.opendev.org/t/openstack/build/a36b14e7e5084fd487b70bddef07c88d/log/job-output.txt#1596 says the wheel failed to build. It seems light on details. But our image builds require we build wheels for everythign on the build image and then we install the wheels on the base image. Currently failing beacuse it tries to setup.py sdist install on the | 17:29 |
| clarkb | base image because the wheel failed | 17:29 |
| clarkb | I'm going to spin up a bullseye container locally and try t odebug | 17:30 |
| fungi | clarkb: yes, for a package to be in buster-backports it must be present in bullseye already (i'm almost positive that's policy) | 17:32 |
| clarkb | I can't reproduce this failed uwsgi build locally on top of our python builder. This is weird | 17:38 |
| clarkb | huh it only failed on python 3.8 on bullseye. Which is what I'm trying to reproduce on. But maybe it was a fluke? I'll recheck and see | 17:40 |
| fungi | i wonder where it gets a 3.8 for bullseye, the python3 provided with bullseye is 3.9 | 17:43 |
| *** jpena is now known as jpena|off | 17:50 | |
| clarkb | fungi: we use the "official" python docker hub images which are based on debian but then build a C python on top of that from source | 17:52 |
| clarkb | the python isn't debians, the rest of the userland is | 17:52 |
| clarkb | heh now the 3.9 job is failing | 17:53 |
| fungi | ahh, okay | 17:53 |
| fungi | that makes more sense | 17:53 |
| hrw | morning | 17:54 |
| hrw | fungi, frickler: my account is restored | 17:54 |
| hrw | reason turned out to be more complex | 17:54 |
| fungi | if you can provide details, that info might be useful to others who run into the same situation trying to use 2fa | 17:56 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update refstack image to bullseye https://review.opendev.org/c/opendev/system-config/+/821335 | 17:59 |
| hrw | fungi: turned out that some 2fa settings were enabled on my account from my Canonical times (when it was mandatory for employees). When I left company, 2fa group was removed along with 2fa being disabled. I shredded access codes etc and now when they enabled 2fa for everyone turned out that some settings left so LP asked me for codes from devices I stopped using 8 years ago. | 17:59 |
| clarkb | ok I guess that means we should prescreen people and ask if they ever had 2fa enabled previously | 18:00 |
| hrw | yep | 18:00 |
| fungi | that's excellent information, thanks! i wondered if it might be something like that | 18:01 |
| hrw | the good part? login.launchpad.net allows to setup 2fa without being in any 2fa groups | 18:01 |
| hrw | so if someone wants to enable 2fa they can do it on their own | 18:01 |
| fungi | i didn't realize login.launchpad.net even still existed, we ported all the openids over to login.ubuntu.com years ago | 18:01 |
| hrw | whatever - I think it is the same backend on two urls | 18:02 |
| clarkb | hrw: oh if they let you opt in directly now then ya we should just push people that direction and not worry about managing this group | 18:02 |
| fungi | agreed | 18:02 |
| clarkb | oh wait lp.net vs ubuntu one though | 18:03 |
| clarkb | ok the bullseye python 3.9 error is the same as the bullseye python 3.8 error. It failed to build a uwsgi wheel then in the base image build we failed as we don't have the bits necessary to build the package from source (it relies on that wheel existing) | 18:05 |
| fungi | clarkb: yeah, looks like it's complaining that the image lacks gcc? | 18:06 |
| clarkb | fungi: thats the later error. Which happens because we don't put build deps in our production deploy image. The expectation is taht the builder image does all of that and produces a wheel then the production image copies that wheel and installs it. This keeps images sizes down as you don't end up with a bunch of build artifacts and dependencies | 18:07 |
| fungi | "Exception: you need a C compiler to build uWSGI" raised when trying to find the gcc version | 18:07 |
| clarkb | fungi: https://zuul.opendev.org/t/openstack/build/abe63905dc58403bb21664597726784d/log/job-output.txt#1596 is the error causing ^ to happen | 18:07 |
| fungi | oh, i didn't realize that was from a different layer operation | 18:08 |
| fungi | i was a bit confused as to why it was bothing to invoke gcc and then later complaining it wasn't there at all | 18:08 |
| clarkb | If the wheel builds as expected then we install later and don't need gcc | 18:08 |
| fungi | however, the compiler error (if there was one) seems not to have been included | 18:08 |
| clarkb | ya exactly | 18:09 |
| fungi | could "UserWarning: Unknown distribution option: 'descriptions'" be causing the build process to exit nonzero after compiling? | 18:10 |
| clarkb | hrm maybe. | 18:10 |
| clarkb | let me cross check against the successful builds | 18:10 |
| clarkb | https://zuul.opendev.org/t/openstack/build/3cfc135feeed4d21a7c503385af3d103/log/job-output.txt#1552 a succesful one doesn't complain at all but I'm not sure if that is because it isn't really printing much when successful or if it is related | 18:14 |
| clarkb | I wonder if it is a bug in their build and doing it with all the CPUs trips a race | 18:16 |
| clarkb | is setuptools too old? | 18:19 |
| clarkb | Problem is this isn't reliably reproduceable so makes it really difficult :/ | 18:20 |
| jrosser | it doesnt look like that has compiled as many source files as a successful run when it fails | 18:27 |
| jrosser | which would make me wonder about a missing header file, something like that | 18:27 |
| fungi | yeah, i wish we could get the (stderr?) output from it | 18:28 |
| clarkb | fungi: if you look it says complete output and gives the number of lines and it appears we have all of them? | 18:29 |
| fungi | that might just be stdout though? | 18:30 |
| jrosser | there are approx 10 lines short of the core and it's not started the embedded plugins at all | 18:30 |
| clarkb | fungi: ya I suppose, Its just doing pip install uWsgi basically and that gets emitted. Maybe I should modify it to do a pip -vvv or whatever the equivalent is | 18:31 |
| clarkb | jrosser: interesting, maybe that does point to the buggy build process then and us hitting a race somehow | 18:31 |
| jrosser | i can give you a paste of a successful build, though thats kind of unhelpful for what the actual bug is | 18:31 |
| clarkb | I can probably get that output from a successful build too, just need to increase verbosity which seems like a good idea either way | 18:32 |
| fungi | another option might be to set an autohold and then check dmesg/syslog for signs of memory exhaustion et cetera | 18:34 |
| fungi | but hopefully increasing verbosity will get us errors in such cases | 18:35 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Properly build bullseye uwsgi-base docker images https://review.opendev.org/c/opendev/system-config/+/821339 | 18:37 |
| clarkb | That ps makes pip very verbose | 18:37 |
| clarkb | and locally that produces a complete build output so we can get it that way if we need it | 18:37 |
| fungi | d'oh! | 18:41 |
| fungi | that looks like one way of doing it, yep | 18:42 |
| clarkb | ya easier than modifying assemble which is in another docker image :/ | 18:43 |
| clarkb | at least since we only need it for one image right now. If we want this by default then we can update assemble directly | 18:43 |
| clarkb | infra-root most changes under https://review.opendev.org/q/hashtag:%2522bullseye-image-update%2522+status:open should be ready for review now. These update a number of our images to bullseye. | 18:43 |
| clarkb | If you'd are able to review and at least +2 that would be great. I'm happy to +A one or two at a time as we are able to verify the end results | 18:44 |
| fungi | clarkb: i'll take a look in a moment. on an unrelated note, should we go ahead with topic:mailman-lists as well? you've +2'd them all and ianw has done so for half. or we could wait for more reviewers | 18:50 |
| clarkb | fungi: I think we can probably proceed carefully unless frickler or corvus indicate an interest. Probably the iptables update is most interseting though it shouldn't affect prod | 18:52 |
| clarkb | basically land a change or two and verify as we go | 18:52 |
| fungi | yeah, the firewall change is the first one in the series, so the rest of it is blocked on that anyway | 18:57 |
| *** sshnaidm is now known as sshnaidm|afk | 19:06 | |
| bqian | hi | 19:55 |
| bqian | any admin can take a look at the Zuul build error in https://review.opendev.org/c/starlingx/update/+/810026 | 19:56 |
| fungi | sure, taking a look now | 19:57 |
| fungi | bqian: looks like your patch-tox-pylint job is complaining "cgcs_patch/patch_controller.py:17:0: E0401: Unable to import 'rpm' (import-error)" | 20:00 |
| bqian | yes, that's what it is complaining. not sure why it is unable to import | 20:01 |
| fungi | have you tried to reproduce it elsewhere? | 20:02 |
| bqian | it runs ok in my local, and it ran ok before patch 6, the patch 6 has not changes in that file | 20:04 |
| fungi | what platform are you testing on locally? | 20:05 |
| fungi | what normally provides the "rpm" python library? | 20:05 |
| fungi | i don't see where the test is installing it | 20:05 |
| bqian | I am not sure about the rpm import, as my changes don't need it | 20:06 |
| bqian | let me check around | 20:07 |
| fungi | bqian: according to this, the job wasn't run for a couple of months before your change tried to call it: https://zuul.opendev.org/t/openstack/builds?job_name=patch-tox-pylint&project=starlingx/update | 20:07 |
| fungi | odds are something happened during that span of time, probably outside of that project | 20:07 |
| fungi | unfortunately we no longer have logs from the last successful run since it was in october | 20:08 |
| fungi | (we expire build logs after a month) | 20:08 |
| ianw | clarkb: if you get a chance for https://review.opendev.org/c/openstack/diskimage-builder/+/820970 and https://review.opendev.org/c/openstack/diskimage-builder/+/820978/5 that tests 8-stream arm64 and fixes up mirror usage, that would be good | 20:09 |
| ianw | just a double check i didn't miss something meaning it's not testing what i think it's testing | 20:09 |
| clarkb | ianw: I should be able to look after lunch | 20:09 |
| ianw | but with that, i think we can do 8-stream arm64 nodes | 20:09 |
| ianw | clarkb: no rush! :) | 20:10 |
| ianw | there's no changes so current dib should work | 20:10 |
| ianw | (famous last words) | 20:10 |
| clarkb | Then it also looks like my pip verbosity change doesn't work on 3.7 because 3.7 bundles old pip in venv | 20:10 |
| fungi | oof | 20:11 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Properly build bullseye uwsgi-base docker images https://review.opendev.org/c/opendev/system-config/+/821339 | 20:22 |
| clarkb | Only still sort of half here, but the rtt on getting new results is long enough that I wanted to get something up while I finish digesting | 20:22 |
| clarkb | fungi: bqian https://pypi.org/project/rpm/ I suspect you need to run that job on a centos or fedora node | 20:24 |
| fungi | right, i'm wondering if what happened recently is that some other change switched the job to use ubuntu | 20:25 |
| clarkb | or maybe pylint started cehcking imports by importing them? | 20:25 |
| fungi | alternatively, there are python-rpm and python3-rpm packages available on bionic | 20:26 |
| fungi | clarkb: the error indicates the import is embedded in another script in that repo | 20:26 |
| fungi | cgcs_patch/patch_controller.py (line 17) | 20:26 |
| fungi | oh, started checking imports is what you said | 20:26 |
| fungi | yeah, could be a new pylint version | 20:26 |
| fungi | or something changed the exclusions list for pylint | 20:27 |
| clarkb | yup | 20:29 |
| bqian | thanks fungi | 20:39 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Properly build bullseye uwsgi-base docker images https://review.opendev.org/c/opendev/system-config/+/821339 | 20:51 |
| fungi | #status log Jitsi-Meet services on meetpad.opendev.org are back in service again following an upgrade to the most recent image builds | 21:05 |
| opendevstatus | fungi: finished logging | 21:05 |
| clarkb | fungi: responded to your question on https://review.opendev.org/c/opendev/gerritbot/+/818493 note a similar reason applies for refstack | 21:08 |
| fungi | thanks! | 21:09 |
| clarkb | of course as soon as I get a working dockerfile up with extra pip verbosity all of the builds succeed :) | 21:19 |
| clarkb | I've rechecked. It really wouldn't surprise me if this is a build system timing issue if changing verbosity changes behavior and it only fails every once in a while | 21:20 |
| clarkb | hound is probably a good next bullseye update: https://review.opendev.org/c/opendev/system-config/+/821329 if you have time to be a second reviewer on that ianw | 21:30 |
| clarkb | ianw: also https://review.opendev.org/c/opendev/system-config/+/820900/ is the bottom of fungi's improve mailman testing stack which includes firewall updates in testing if you have time for that | 22:04 |
| clarkb | I think we should carefully land that one | 22:04 |
| clarkb | Meeting agenda has been updated. Please edit to add anything I missed or let me know and I'll add it in then probably send out in an hour or so | 22:07 |
| opendevreview | Merged opendev/system-config master: Update the hound image to bullseye https://review.opendev.org/c/opendev/system-config/+/821329 | 22:08 |
| ianw | that stack lgtm, only question is if https://review.opendev.org/c/opendev/system-config/+/821144/3 bits would be better in a test playbook | 22:13 |
| clarkb | ya I thought about that too, but in this case it seemed fine since it is tied to the specific check in the test case. But I could go either way | 22:14 |
| clarkb | just waiting for the hourly runs to finish then hound should update | 22:17 |
| clarkb | I'm keeping an eye on it | 22:18 |
| fungi | right, if there's a good way for me to insert a restart playbook between the production deployment playbook and the testinfra run, i'm happy to do it that way | 22:18 |
| clarkb | fungi: ya you can specify a test playbook like gerrit uses | 22:18 |
| clarkb | fungi: look in zuul.d/system-config-run.yaml and the system-config-run-review-base job | 22:19 |
| clarkb | it lists the prod playbooks then a test specific playbook that runs after the prod playbooks | 22:19 |
| fungi | thanks, i'll try to get it working tonight | 22:19 |
| opendevreview | Merged openstack/diskimage-builder master: Test 8-stream aarch64 build https://review.opendev.org/c/openstack/diskimage-builder/+/820970 | 22:23 |
| clarkb | hound is restarting now | 22:38 |
| clarkb | seems to be doing what I expect it to. It is setting up searchers for all the repos | 22:38 |
| clarkb | and it is up, queries work. I think it is happy | 22:41 |
| clarkb | I cannot get the uwsgi builds to fail with extra verbosity. I'll try removing the verbosity and see if they fail again. If so maybe we leave the verbosity in there as a bug fix <_< | 22:42 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Properly build bullseye uwsgi-base docker images https://review.opendev.org/c/opendev/system-config/+/821339 | 22:43 |
| clarkb | ianw: https://review.opendev.org/c/opendev/gerritbot/+/818493 is probably a good next step for docker images if you want to look at another one. The change there bumps gerritbot to bullseye then the child sets the same uid and gid that we set on the service now | 22:45 |
| ianw | clarkb: lg, is there a depends-on for it? | 22:56 |
| ianw | in system-config i mean | 22:56 |
| clarkb | ianw: no we run the eavesdrop playbook hourly for this reason. A number of eavesdrop things happen without easy cross tenant triggering | 22:59 |
| clarkb | similar to why we run zuul and nodepool that way | 22:59 |
| ianw | np, it all looks good to me, yeah we just want to watch it. i did think the eavesdrop test started this and connected to opendev-sandbox, maybe that's something else | 23:00 |
| ianw | does anyone have a problem with the openeuler nodes in https://review.opendev.org/c/openstack/project-config/+/818723 ? | 23:01 |
| ianw | it's been sitting for a while. space is tight, but since we've accepted this in dib, etc. it seems like this has been the ultimate goal | 23:02 |
| ianw | on arm64 if we add 8-stream, 9-stream and this we may be very very tight | 23:02 |
| ianw | /dev/mapper/main-main 787G 526G 262G 67% /opt | 23:03 |
| clarkb | I think we can start moving forward on removing gentoo and opensuse tumblweed which will free up space (mostly tumblweed not sure we ever mirror'd gentoo stuff) | 23:05 |
| clarkb | and then we can followup with opensuse leap 15 removal and centos-8 removals | 23:05 |
| clarkb | oh you don't mean mirrors you mean actual builder room | 23:06 |
| clarkb | sorry I misinterpreted | 23:06 |
| clarkb | I think only centos-8 will be removed on the arm builder side and that is the last thing we said we would remove | 23:06 |
| clarkb | ianw: No objcetions from me but should we add it to more clouds or are you wanting to do a smaller number to start? | 23:07 |
| clarkb | oh i see a comment says one provider to start ok +2 from me | 23:08 |
| clarkb | I didn't approve it as I'm not sure I'll be around long enough to monitor the resulting image builds and ensure we don't run out of disk | 23:08 |
| ianw | yeah i'm not really wanting to do anything :) but probably not a bad idea to debug in one env only first | 23:08 |
| ianw | corvus: https://review.opendev.org/c/opendev/system-config/+/821508 is one to update our logging to use the multilineformatter you put in | 23:09 |
| ianw | the other thought i had was dropping the formatter section, but i wasn't sure if it would fall back to the default correctly if reading a logging config file | 23:10 |
| clarkb | meeting agenda sent now as promised about an hour ago :) | 23:11 |
| ianw | i'm hoping to get a spec on credentials from zuul in shape before then, not quite done | 23:11 |
| clarkb | thanks, it is on the agenda for discussion either way | 23:12 |
| clarkb | ok the uwsgi base images don't build when I drop the verbose flag | 23:12 |
| clarkb | I'm going to propose it with the verbose flag | 23:12 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Properly build bullseye uwsgi-base docker images https://review.opendev.org/c/opendev/system-config/+/821339 | 23:14 |
| clarkb | And yes that is crazy, but meh | 23:14 |
| clarkb | I've approved the gerritbot bullseye update but will do the user update separately just in case | 23:19 |
| opendevreview | Merged opendev/gerritbot master: Update docker image to bullseye https://review.opendev.org/c/opendev/gerritbot/+/818493 | 23:27 |
| ianw | clarkb: hrm ... https://zuul.opendev.org/t/openstack/build/a36b14e7e5084fd487b70bddef07c88d really looks like there was no gcc installed for amd64, but it was for arm64? | 23:28 |
| ianw | this might be the case as arm64 were are more attuned to having these dependencies instead of relying on wheels | 23:29 |
| ianw | that was from ps3 @ https://review.opendev.org/c/opendev/system-config/+/821339 ... is that representitive of the problem? | 23:29 |
| clarkb | ianw: that error is misleading | 23:32 |
| clarkb | there isn't a gcc installed beacuse that is on the python-base image where we copy over the wheels and install the package. The problem is earlier when we try to build the wheel it fails so we don't have a wheel and it tries to setup.py install and fails due to no gcc | 23:33 |
| clarkb | There is something up with the uWSGI build that I cannot reproduce locally and adding verbosity also seems to make go away | 23:33 |
| ianw | ahhh, ok, more like 2021-12-13 17:22:15.939811 in https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a36/821339/3/check/system-config-build-image-uwsgi-base-3.8-bullseye/a36b14e/job-output.txt ? | 23:38 |
| clarkb | yup that happens then we don't get a wheel so it looks for gcc later when we don't want it to have gcc (its done in an effort ot keep image sizes down) | 23:39 |
| ianw | yeah, ok that makes sense | 23:41 |
| fungi | clarkb: what are the odds we tripped over a uwsgi release which only had an sdist and no manylinux wheels matching that interpeter abi? | 23:45 |
| fungi | i guess slim, last release was in early october | 23:45 |
| clarkb | ya the reason we go through this trouble is uWSGI needs a proper build | 23:48 |
| ianw | 2021-12-13 23:21:15.763065 | ubuntu-focal | #13 31.19 [thread 7][gcc -pthread] lib/linux_ns.o | 23:50 |
| ianw | 2021-12-13 23:21:15.864407 | ubuntu-focal | #13 31.23 [thread 5][gcc -pthread] core/yaml.o | 23:50 |
| ianw | 2021-12-13 23:21:15.864455 | ubuntu-focal | #13 31.23 [thread 0][gcc -pthread] core/ssl.o | 23:50 |
| ianw | these seem to be 3 things the successful verbose build creates that don't appear on the failed build | 23:50 |
| ianw | i should say 3 suspicious things | 23:51 |
| clarkb | we don't explicitly add libyaml to the bindep. But libssl-dev is there. Linux binding stuff could also be cleaned up maybe in some cases beacuse we don't hve a kernel on the image? | 23:52 |
| clarkb | https://uwsgi-docs.readthedocs.io/en/latest/Install.html doesn't say anything about those extra bits though | 23:53 |
| fungi | this was working on buster but isn't on bullseye, right? or is now working at random on bullseye? | 23:55 |
| clarkb | ya I haven't seen this failuer on buster. It works randomly on bullseye and fails randomly on bullseye. I'ev done a number of local bullseye builds and havne't been able to reproduce. It always seems to work here | 23:55 |
| clarkb | this is why i suspect a timing issue and I suspect doing more IO for pip verbosity improves that | 23:55 |
| clarkb | initially all I wanted was the pip verbosity to expose what was failing better. But now it seems to just work with that flag :/ | 23:56 |
| fungi | and heisenbugs out when you add the extra verbosity so we have no clue what it's failing on, right | 23:56 |
| clarkb | This is like the 5th rebuild in a row where all three python versions on bullseye built with verbosity set | 23:56 |
| clarkb | but when I remove verbosity at least one fails in every buildset | 23:56 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!