| fungi | clarkb: yes, i think we need new xenial images with working bindep preinstalled, right? | 00:02 |
|---|---|---|
| fungi | i'm just pre-approving it so it can land with a recheck later | 00:02 |
| Clark[m] | Ah ok. I was wondering if we wanted to force merge and approve it but I think either way we need new images so waiting makes sense | 00:03 |
| fungi | or we can shove it past zuul with admin powers and tag a new point release | 00:03 |
| fungi | yeah, that | 00:03 |
| fungi | i'm good with either solution | 00:04 |
| Clark[m] | Ya now that I realize we are waiting for new images either way waiting to land it properly makes sense | 00:04 |
| fungi | the image fixing would need 814677 landed, right? | 00:04 |
| opendevreview | Merged openstack/project-config master: Add ubuntu-bionic-32GB https://review.opendev.org/c/openstack/project-config/+/814683 | 00:05 |
| Clark[m] | Correct | 00:06 |
| fungi | do we want to revise it for your comment about using an inequality instead of an exact version? | 00:06 |
| fungi | otherwise i'll approve ow | 00:07 |
| fungi | now | 00:07 |
| fungi | went ahead and approved it | 00:12 |
| fungi | we can always loosen that later if we want | 00:12 |
| fungi | we'll eventually be dropping xenial anyway eventually | 00:13 |
| fungi | double-eventually consistent | 00:13 |
| Clark[m] | Ya I think approving it as is makes sense for now | 00:17 |
| opendevreview | Merged openstack/project-config master: infra-package-needs: install latest pip https://review.opendev.org/c/openstack/project-config/+/814677 | 00:22 |
| ianw | i can trigger some xenial builds or we can just wait | 00:22 |
| fungi | whichever you prefer, but one of those options definitely sounds like less work | 00:39 |
| ianw | fwiw paste.opendev.org seems happy again | 02:44 |
| opendevreview | Merged openstack/diskimage-builder master: Remove py35 tox jobs https://review.opendev.org/c/openstack/diskimage-builder/+/814680 | 03:51 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ensure-docker: remove Debian Stretch testing https://review.opendev.org/c/zuul/zuul-jobs/+/814695 | 03:55 |
| opendevreview | Merged opendev/base-jobs master: Remove debian-stable nodeset https://review.opendev.org/c/opendev/base-jobs/+/802639 | 03:56 |
| opendevreview | Ian Wienand proposed opendev/glean master: Remove debian-stable, add focal/bullseye testing https://review.opendev.org/c/opendev/glean/+/814697 | 03:59 |
| *** pojadhav|out is now known as pojadhav|ruck | 04:04 | |
| *** ykarel_ is now known as ykarel | 04:40 | |
| opendevreview | Merged openstack/project-config master: Remove fedora-32 disk image config https://review.opendev.org/c/openstack/project-config/+/795644 | 04:40 |
| *** ysandeep|out is now known as ysandeep | 05:08 | |
| chandankumar | ianw: Good morning, please have a look at this https://review.opendev.org/c/zuul/zuul-jobs/+/814516 when free, thanks :-) | 05:12 |
| opendevreview | Sandeep Yadav proposed zuul/zuul-jobs master: multi-node-bridge: repos to install ovs in C9 https://review.opendev.org/c/zuul/zuul-jobs/+/814516 | 05:47 |
| opendevreview | Merged openstack/diskimage-builder master: epel: match replacement better https://review.opendev.org/c/openstack/diskimage-builder/+/813922 | 06:23 |
| opendevreview | Merged openstack/diskimage-builder master: Revert "Allowing ubuntu element use local image" https://review.opendev.org/c/openstack/diskimage-builder/+/814094 | 06:24 |
| opendevreview | Merged openstack/diskimage-builder master: ubuntu-systemd-container: deprecate and remove jobs https://review.opendev.org/c/openstack/diskimage-builder/+/814068 | 06:24 |
| opendevreview | Merged openstack/diskimage-builder master: ubuntu: add Focal test https://review.opendev.org/c/openstack/diskimage-builder/+/814072 | 06:32 |
| opendevreview | Merged openstack/diskimage-builder master: functests: drop apt-sources https://review.opendev.org/c/openstack/diskimage-builder/+/814074 | 06:32 |
| opendevreview | Merged openstack/diskimage-builder master: centos7 : drop functional testing https://review.opendev.org/c/openstack/diskimage-builder/+/814075 | 06:32 |
| opendevreview | Merged openstack/diskimage-builder master: functests: drop minimal tests in the gate https://review.opendev.org/c/openstack/diskimage-builder/+/814078 | 06:32 |
| opendevreview | Merged openstack/diskimage-builder master: Remove extras job, put gentoo job in gate https://review.opendev.org/c/openstack/diskimage-builder/+/814079 | 06:32 |
| opendevreview | Merged openstack/diskimage-builder master: Simplify functests job https://review.opendev.org/c/openstack/diskimage-builder/+/814080 | 06:32 |
| *** ysandeep is now known as ysandeep|afk | 06:34 | |
| *** ysandeep|afk is now known as ysandeep|trng | 07:03 | |
| priteau | Good morning. Is Zuul struggling today? https://review.opendev.org/813786 has been queue for 45 minutes without starting jobs. | 07:19 |
| opendevreview | Merged openstack/diskimage-builder master: Run functional tests on Debian Bullseye https://review.opendev.org/c/openstack/diskimage-builder/+/814081 | 07:22 |
| frickler | priteau: I think that patch was just unlucky to come in right after the daily bunch of periodic jobs fired | 07:29 |
| *** jpena|off is now known as jpena | 07:29 | |
| frickler | infra-root: looking at the grafana data, maybe we could move the periodic trigger a bit earlier to avoid clashing with EU work day start so much? seems we mostly have a pretty idle period from roughly 0-6 UTC | 07:32 |
| priteau | OK, just need to be patient I guess | 07:32 |
| opendevreview | Dr. Jens Harbott proposed opendev/project-config master: Move the daily periodic trigger earlier https://review.opendev.org/c/opendev/project-config/+/814721 | 07:56 |
| *** ykarel is now known as ykarel|lunch | 08:00 | |
| opendevreview | Alfredo Moralejo proposed openstack/diskimage-builder master: Add support for CentOS Stream 9 in DIB https://review.opendev.org/c/openstack/diskimage-builder/+/811392 | 08:07 |
| opendevreview | Alfredo Moralejo proposed openstack/diskimage-builder master: Add support for CentOS Stream 9 in DIB https://review.opendev.org/c/openstack/diskimage-builder/+/811392 | 08:12 |
| *** iurygregory_ is now known as iurygregory | 09:00 | |
| opendevreview | Alfredo Moralejo proposed openstack/diskimage-builder master: [wip] Add support for CentOS Stream 9 in DIB https://review.opendev.org/c/openstack/diskimage-builder/+/811392 | 09:05 |
| opendevreview | Alfredo Moralejo proposed openstack/diskimage-builder master: [wip] Add support for CentOS Stream 9 in DIB https://review.opendev.org/c/openstack/diskimage-builder/+/811392 | 09:07 |
| opendevreview | Alfredo Moralejo proposed openstack/diskimage-builder master: [wip] Add support for CentOS Stream 9 in DIB https://review.opendev.org/c/openstack/diskimage-builder/+/811392 | 09:09 |
| opendevreview | Alfredo Moralejo proposed openstack/diskimage-builder master: [wip] Add support for CentOS Stream 9 in DIB https://review.opendev.org/c/openstack/diskimage-builder/+/811392 | 09:33 |
| lourot | o/ we (openstack charmers) would love to get this one landed, thanks! https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/814126 | 09:38 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add new element rocky https://review.opendev.org/c/openstack/diskimage-builder/+/805800 | 10:12 |
| *** ykarel|lunch is now known as ykarel | 10:15 | |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add new element rocky https://review.opendev.org/c/openstack/diskimage-builder/+/805800 | 10:18 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add new element rocky https://review.opendev.org/c/openstack/diskimage-builder/+/805800 | 10:19 |
| *** ysandeep|trng is now known as ysandeep|afk | 10:19 | |
| opendevreview | Alfredo Moralejo proposed openstack/diskimage-builder master: Add support for CentOS Stream 9 in DIB https://review.opendev.org/c/openstack/diskimage-builder/+/811392 | 10:35 |
| *** dviroel|rover|afk is now known as dviroel|rover | 11:18 | |
| *** jpena is now known as jpena|lunch | 11:25 | |
| *** ysandeep|afk is now known as ysandeep | 12:21 | |
| *** jpena|lunch is now known as jpena | 12:24 | |
| *** pojadhav|ruck is now known as pojadhav|brb | 12:32 | |
| *** pojadhav|brb is now known as pojadhav|ruck | 12:43 | |
| opendevreview | Merged zuul/zuul-jobs master: multi-node-bridge: repos to install ovs in C9 https://review.opendev.org/c/zuul/zuul-jobs/+/814516 | 12:45 |
| fungi | frickler: i have no objection to moving the trigger time for the various periodic pipelines, though also i'll reopen the discussion with the openstack stable maintainers about removing periodic jobs if there's no hope of them succeeding | 12:51 |
| frickler | fungi: I already proposed a patch earlier. but I agree removing known-failing jobs should happen in parallel, too | 12:52 |
| fungi | ahh, i see now, you pushed up 814721 | 12:54 |
| *** timburke_ is now known as timburke | 12:58 | |
| clarkb | I too have no objection with better tuning periodic trigger times. ++ to cleaning up the queues though | 13:51 |
| fungi | yeah, elodilles wanted to keep those failing stable periodic jobs around to see when they get fixed, but i still get ~110-120 failure reports in my inbox every morning so i don't think the number is shrinking | 13:52 |
| elodilles | it shrinked, but unfortunately just a very bit o:) | 14:19 |
| elodilles | and worth to mention that most of the failing periodic jobs came from oslo, rocky and older projects, that is currently about to move to EOL | 14:20 |
| elodilles | at least the team sent a mail to ML about that | 14:21 |
| elodilles | this was the case so far... but for example today I see double amount of failing periodics than the "usual" 115 :S | 14:22 |
| opendevreview | Douglas Viroel proposed zuul/zuul-jobs master: Add FIPS enable multinode job definition https://review.opendev.org/c/zuul/zuul-jobs/+/813253 | 14:22 |
| opendevreview | Merged opendev/project-config master: Move the daily periodic trigger earlier https://review.opendev.org/c/opendev/project-config/+/814721 | 14:24 |
| clarkb | frickler: ^ did you want to write a corresponding change for openstack/project-config too? or should I? | 14:25 |
| *** andrewbonney_ is now known as andrewbonney | 14:26 | |
| frickler | clarkb: on it | 14:31 |
| opendevreview | Dr. Jens Harbott proposed openstack/project-config master: Move the daily periodic trigger earlier https://review.opendev.org/c/openstack/project-config/+/814794 | 14:34 |
| opendevreview | Merged openstack/diskimage-builder master: Update centos element for 9-stream https://review.opendev.org/c/openstack/diskimage-builder/+/806819 | 14:44 |
| fungi | elodilles: the additional failures today are that basically any jobs run on ubuntu-xenial failed because of a problem with how we were installing the latest bindep release from monday into the images when we built them | 14:54 |
| opendevreview | Merged opendev/bindep master: Add old python packaging pin https://review.opendev.org/c/opendev/bindep/+/814647 | 14:54 |
| opendevreview | Alfredo Moralejo proposed openstack/diskimage-builder master: Add support for CentOS Stream 9 in DIB https://review.opendev.org/c/openstack/diskimage-builder/+/811392 | 15:02 |
| elodilles | fungi: indeed, I saw that rocky, queens and pike have the new failures, those are on xenial | 15:07 |
| elodilles | fungi: I've checked now the failure ( https://zuul.opendev.org/t/openstack/build/69d5c1e75d264a40836c3a14be8cca2f/log/job-output.txt#561-579 ) | 15:08 |
| prometheanfire | I suppose making a new bug for gertty traceback (git master) when trying to pull up a diff? or is there an irc channel? | 15:09 |
| fungi | prometheanfire: are you using latest master of gertty? there are fixes which address problems like you're describing | 15:14 |
| fungi | infra-root: proposing to tag opendev/bindep commit 0bd79b1d5dd94570f8e3629b991642e919acdb94 as 2.10.1 | 15:14 |
| clarkb | fungi: checking | 15:14 |
| fungi | i have the tag created locally, just waiting to push it to gerrit | 15:15 |
| clarkb | 0bd79b1d5dd94570f8e3629b991642e919acdb94 as 2.10.1 lgtm | 15:15 |
| fungi | thanks, i'll go ahead and push since this is a fairly important fix for people with old pip versions | 15:16 |
| fungi | and done | 15:16 |
| fungi | i'll send a release announcement once it's up on pypi | 15:16 |
| elodilles | thanks for the fix and the release! | 15:21 |
| prometheanfire | fungi: ya, just installed | 15:28 |
| *** ykarel is now known as ykarel|away | 15:31 | |
| *** marios is now known as marios|out | 15:41 | |
| fungi | prometheanfire: are you seeing the crash on master? | 15:46 |
| prometheanfire | yes | 15:46 |
| fungi | prometheanfire: some specific change you can reproduce it on? i'll see if i can come up with a patch | 15:49 |
| prometheanfire | sure repository: https://opendev.org/ttygroup/gertty | 15:49 |
| prometheanfire | at the commit: 559bcfdb33769dd1b9db499daf7443dae06d09a8 | 15:49 |
| prometheanfire | that's the gertty version installed | 15:49 |
| prometheanfire | https://review.opendev.org/814716 | 15:50 |
| fungi | i mean what change are you pulling up in gertty | 15:50 |
| prometheanfire | when trying to view the diff | 15:50 |
| fungi | yeah | 15:50 |
| prometheanfire | I know, wanted to get you the full info :P | 15:51 |
| fungi | thanks, seeing if i can reproduce | 15:51 |
| fungi | strangely i can look at the diff view for that change with no problem. pbr freeze reports i'm using 559bcfd as well | 15:53 |
| fungi | maybe it's because i'm using unified diff mode | 15:54 |
| clarkb | possibly also a dep issue? | 15:54 |
| fungi | prometheanfire: can you see if it still happens with "diff-view: unified" in your .gertty.yaml? | 15:55 |
| fungi | i'll try switching mine to the default (side-by-side) | 15:55 |
| fungi | doesn't crash for me with sxs mode either | 15:56 |
| prometheanfire | updating gitpython seemed to help | 15:59 |
| fungi | i take it that appeared in the traceback? | 15:59 |
| prometheanfire | from 3.1.20 to 3.1.24 | 15:59 |
| fungi | i've got GitPython==3.1.24 | 15:59 |
| prometheanfire | you could try with 3.1.20 if you care to verify, but 3.1.20 works, not sure why it failed all of a sudden, been a couple of months fine with 3.1.20 | 16:00 |
| prometheanfire | 3.1.20^t{xpak}(03:08:18 PM 08/27/2021) | 16:00 |
| fungi | prometheanfire: could be related to our recent gerrit upgrade "somehow" i suppose | 16:01 |
| prometheanfire | ya, wfm | 16:01 |
| fungi | this is what `pbr freeze` reports for my current gertty venv (running under python 3.10): https://paste.opendev.org/show/810108 | 16:02 |
| fungi | i'll try to force GitPython down to 3.1.20 | 16:03 |
| *** pojadhav is now known as pojadhav|ruck | 16:12 | |
| *** pojadhav|ruck is now known as pojadhav|out | 16:12 | |
| fungi | ugh, internet provider seems to have decided 16:00 utc on a wednesday is the ideal time to take everything offline | 16:13 |
| fungi | after all, who'd be home in the middle of a weekday? | 16:13 |
| fungi | anyway, i'm limping along via cellular modem for the time being | 16:14 |
| fungi | prometheanfire: downgrading to GitPython==3.1.20 doesn't seem to have allowed me to reproduce the crash either | 16:15 |
| fungi | did you happen to upgrade anything else along with gitpython? | 16:16 |
| * prometheanfire shrugs | 16:16 | |
| fungi | and do you still have your earlier traceback? | 16:16 |
| prometheanfire | nope | 16:16 |
| prometheanfire | https://gist.github.com/prometheanfire/2cb6f088c3cc2cba15eacf9ccf889af3 | 16:18 |
| fungi | i wonder if i need to be on python 3.9 instead of 3.10 to reproduce that, it's being raised in the stdlib typing module | 16:20 |
| prometheanfire | could be | 16:21 |
| fungi | i'll recreate my venv with 3.9, just a sec | 16:23 |
| fungi | unfortunately somewhat slower over 3g tin-cans-n-string | 16:25 |
| clarkb | fungi: your island hasn't upgraded to fake 5g yet? | 16:26 |
| fungi | bingo! | 16:26 |
| fungi | TypeError: Protocols cannot be instantiated | 16:26 |
| fungi | prometheanfire: so seems to be python 3.9 + gitpython 3.1.20 | 16:26 |
| fungi | upgrading to latest gitpython fixed it for me too | 16:28 |
| fungi | prometheanfire: looks like that's https://github.com/gitpython-developers/GitPython/issues/1332 | 16:29 |
| fungi | seems python 3.9.6 was fine but 3.9.7 caused it to break, so maybe you upgraded to 3.9.7 recently? | 16:30 |
| fungi | the workaround was included in gitpython 3.1.22 | 16:31 |
| *** jpena is now known as jpena|off | 16:31 | |
| fungi | anyway, i'm switching my gertty back off this ancient 3.9 interpreter back to a nice clean 3.10 ;) | 16:31 |
| *** ysandeep is now known as ysandeep|away | 16:41 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update zuul restart documentation https://review.opendev.org/c/opendev/system-config/+/814808 | 16:50 |
| clarkb | frickler: ^ fyi | 16:50 |
| opendevreview | Jeremy Stanley proposed opendev/bindep master: DNM: checking that jobs still run https://review.opendev.org/c/opendev/bindep/+/814809 | 16:56 |
| fungi | clarkb: not sure if you saw, but 813301 and 813303 update our specs template based on your comments for the mailman3 spec | 17:06 |
| clarkb | fungi: I hadn't I'll take a look once I've got this next change about debugging zuul behaviors pushed | 17:07 |
| fungi | no worries, just making sure you knew they were there | 17:11 |
| prometheanfire | fungi: nice | 17:12 |
| prometheanfire | fungi: yep, 3.9.7_p1(3.9)^t{xpak}(07:33:10 PM 10/15/2021) | 17:12 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add zuul debugging documentation https://review.opendev.org/c/opendev/system-config/+/814812 | 17:13 |
| clarkb | frickler: ^ and now a second change to talk about debugging process a little bit | 17:13 |
| clarkb | that is a bit more hand wavy but hopefully gives enough hints to what I've found useful that others can make use of it | 17:13 |
| clarkb | fungi: I went ahead and approved the spec template updates, thanks for those | 17:16 |
| fungi | sure, i just didn't want to forget to push them up | 17:20 |
| opendevreview | Merged opendev/infra-specs master: Add backups section to template https://review.opendev.org/c/opendev/infra-specs/+/813301 | 17:20 |
| opendevreview | Merged opendev/infra-specs master: Simplify assignees section of template https://review.opendev.org/c/opendev/infra-specs/+/813303 | 17:23 |
| clarkb | fungi your dnm bindep change passed its tox-py35 build. I think we're good there now | 17:25 |
| fungi | yep | 17:26 |
| clarkb | looks like the dib changes all landed after the py35 job was removed so nothing to recheck there | 17:34 |
| clarkb | My doc changes appear to have rendered well enough. I'm not sure I like italics for inline quoted stuff. But I think it is a theme change to change that | 17:42 |
| clarkb | fungi: I checked on https://review.opendev.org/c/starlingx/distcloud-client/+/812130/30/bindep.txt because it failed a bunch of bindep stuff. But the issue is the lack of the newline in that file | 17:56 |
| clarkb | I guess parsely or whatever parser bnidep uses doesn't like that | 17:56 |
| clarkb | I wonder what editor people are using that doesn't add the newline | 17:59 |
| fungi | i see it pretty often, yeah | 18:08 |
| fungi | no idea what sane text editor would do that though | 18:09 |
| clarkb | fungi: I think your docs snippet on decoding zuul secrets is no longer valid due to the up to date secrets living in zk | 18:09 |
| clarkb | I'm not yet sure what the best way to extract the info is though. Maybe export-keys then something sort of python -m json script or jq usage? | 18:10 |
| fungi | reuse our key backups? | 18:10 |
| clarkb | ya that is an option. The only thing is its a giant json file with everython on one line iirc. So you'll need a smart tool to pull out the info you want | 18:10 |
| clarkb | jq should be able to do it I guess | 18:11 |
| fungi | as long as the project wasn't added in the past 24 hours, that's probably easier than trying to extract them from zk | 18:11 |
| fungi | yeah i'd probably use q | 18:11 |
| clarkb | ++ | 18:11 |
| clarkb | and then you can always manually run export-keys too to update the file | 18:11 |
| clarkb | now the next problem becomes figuring out the jq incantation to make this work. its syntax does not come very naturrally | 18:12 |
| fungi | it's sql-like right? | 18:12 |
| fungi | or am i thinking of a different utility? | 18:12 |
| clarkb | I don't think so. You use a python addressing like syntax but with weird quoting iirc | 18:13 |
| clarkb | jq ´.foo?´ -> returns foo if present in the dict and null otherwise | 18:13 |
| fungi | ahh, i'm thinking of something else just called q which lets you query json files as if they're sql databases, i think, but the name makes it very hard to find | 18:14 |
| clarkb | {"keys": {"fullkeypath": {"keys": [{"private_key": "key data here"}]}}} <- is the datastructure we are trying to extra private_key from | 18:17 |
| clarkb | ok I think I have it | 18:20 |
| fungi | okay, it was https://harelba.github.io/q/ i was misremembering, it's for querying sqlite not json | 18:20 |
| fungi | so yeah jq is likely our best bet | 18:20 |
| fungi | s/sqlite/csv/ | 18:20 |
| clarkb | hrm now running into issues with /'s in the dict key names that I have to figure out | 18:24 |
| fungi | because json is web-scale? | 18:25 |
| clarkb | well because shell quoting and weirdness. This is always what I run into with jq when I try to use it. But I figured it out | 18:25 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update zuul secrets decryption docs https://review.opendev.org/c/opendev/system-config/+/814817 | 18:33 |
| clarkb | I've not tested ^ on anything more than a local mockup of secret key backups | 18:34 |
| clarkb | fungi: ^ I think you understand all of this better than I do so maybe carefulyl review it and give it a go and feel free to update with new patchsets? | 18:34 |
| fungi | sure | 18:35 |
| clarkb | corvus: ^ also I'm not sure if that sort of thing might want to live in zuul docs? | 18:39 |
| clarkb | corvus: maybe it would be better for me to psuh something there then opendev can link to it? | 18:40 |
| fungi | i'm still slowed a bit by a broadband outage but will have a look shortly | 19:01 |
| fungi | doesn't help when t-mobile decides to just stop talking to the modem for 5 minutes at a stretch | 19:16 |
| fungi | to my backup cellular modem i mean | 19:17 |
| fungi | whee, a zuul-jobs change approved almost two hours ago is still waiting on a f34 node in the gate. might be time to switch some projects back to testing with f33 until f34 is actually bootable most places | 20:03 |
| opendevreview | Merged zuul/zuul-jobs master: build-python-release: Add flag for whether to build a wheel or not https://review.opendev.org/c/zuul/zuul-jobs/+/795419 | 20:32 |
| clarkb | I'm looking at making bindep handle no trailing newline and the trivial method would be to add the newline if it isn't there already. But I really want to address this in the grammar but I just end up with parsing loops | 20:46 |
| clarkb | anyway I'll have to look at this more closely after the dentist | 20:46 |
| fungi | yay, my internet finally came back | 21:33 |
| ianw | fungi: i don't think we ever had f33 nodes, and f32 is gone -- so ... | 21:38 |
| ianw | 9-stream is closer | 21:38 |
| ianw | something that might be good, and i just notice dthis looking at that rocky patch too, if we could consistently fix our grub to make sure we get serial output | 21:47 |
| fungi | ianw: aha, i hadn't been paying close enough attention to realize f33 never materialized | 21:53 |
| fungi | timburke_: 795419 merged at 20:32 utc, so time to work out next steps for getting a release uploaded (whether we can re-enqueue the most recent tag or a new one is needed will depend on what, if anything, may need to merge to that repo to use the new flag) | 22:00 |
| fungi | happy to talk through options next time you're free | 22:00 |
| timburke_ | fungi, thanks for the ping! i'll recheck https://review.opendev.org/c/openstack/pyeclib/+/795420 and go from there | 22:02 |
| fungi | great, let me know if you need anything at all | 22:03 |
| timburke_ | i don't *think* we need to do any more work on existing tags -- i see tarballs for the current release at both https://pypi.org/project/pyeclib/#files and https://tarballs.opendev.org/openstack/pyeclib/ | 22:06 |
| timburke_ | for better or worse, that library changes so infrequently that we don't really have need of another release any time soon, i think | 22:06 |
| timburke_ | (though getting manylinux wheels building might be a good reason, if i can get that working ;-) | 22:07 |
| fungi | timburke_: cool, i can try reenqueuing the most recent tag to get it uploaded to pypi, if that's good with you | 22:14 |
| fungi | looks like that's 1.6.0 | 22:15 |
| timburke_ | shouldn't be needed -- looks like it's already there | 22:15 |
| fungi | ahh, okay | 22:15 |
| fungi | yeah, i agree it's already there | 22:16 |
| timburke_ | this was all noticed because of an attempt to update the README -- nothing major, but enough to make the test-release-openstack job run | 22:16 |
| fungi | so i guess make a new release request to tag the next version you want, looks like you've landed 16 changes since 1.6.0 | 22:17 |
| fungi | anyway, if you need more help with it, don't hesitate to let us know | 22:18 |
| ianw | fungi: tox-py35 still fails, but now in a different way it seems | 22:28 |
| ianw | https://zuul.opendev.org/t/openstack/build/18fe53973c484558ab279f6b5ada8648 | 22:29 |
| ianw | it's decided to build a wheel for cffi and failed | 22:30 |
| ianw | this might related back to the fact that the wheel mirrors are unhappy | 22:30 |
| ianw | i guess that's what i'm doing now | 22:30 |
| fungi | yeah, presumably no cffi wheel on pypi for that platform/interpreter | 22:31 |
| fungi | checking | 22:31 |
| fungi | cffi!=1.11.3,>=1.8 so it's settled on 1.15.0 | 22:32 |
| fungi | which is the most recent cffi release from a week ago | 22:32 |
| fungi | and indeed, only wheels for python >=3.6 | 22:33 |
| fungi | but hey, they have wheels for python 3.10 at least, if you need those | 22:33 |
| ianw | cp: cannot create regular file '/afs/.openstack.org/mirror/wheel/centos-8-stream-x86_64//a/ansible/ansible-2.9.27-py3-none-any.whl': Disk quota exceeded | 22:34 |
| fungi | ooh | 22:34 |
| ianw | here's problem 1 / X | 22:34 |
| ianw | (X to be determined ...) | 22:34 |
| fungi | 1/0 | 22:34 |
| fungi | ZeroDivisionError: division by zero | 22:35 |
| ianw | cp: cannot create regular file '/afs/.openstack.org/mirror/wheel/centos-8-stream-aarch64//a/ansible/ansible-2.9.27-py3-none-any.whl': Disk quota exceeded | 22:35 |
| ianw | arm64 same issue | 22:35 |
| fungi | same volume, yeah? | 22:35 |
| ianw | the xenial wheels are failing because ... bindep : https://zuul.opendev.org/t/openstack/build/7b1a8d2c36cf4598a712e9c8b6066d92 | 22:36 |
| fungi | https://grafana.opendev.org/d/T5zTt6PGk/afs suggests we still have space on the mirror.centos volume | 22:36 |
| ianw | i think they have their own volumes | 22:37 |
| fungi | oh, right, this is the wheels | 22:37 |
| * fungi scrolls down | 22:37 | |
| fungi | oh, yep | 22:37 |
| ianw | the quota is 5mb | 22:38 |
| fungi | slap full | 22:38 |
| ianw | that seems an oversight ... | 22:38 |
| fungi | 5gb is the default i think? so we probably forgot to set a quota on some of these | 22:38 |
| fungi | focal x86_64 looks similarly problematic, but i guess has no job yet | 22:38 |
| ianw | yeah not 5gb, 5mb | 22:39 |
| fungi | the other wheel volumes have 100gb quota | 22:39 |
| fungi | er, yeah, i meant 5mb, but it's been so long since i needed to ytpe mb that my fingers just corrected it | 22:39 |
| * fungi thinks back to that 5mb hardcard he installed in his computer several decades ago, before hard drives were in vogue | 22:40 | |
| fungi | it was like having 10 floppy disks permanently inserted | 22:41 |
| ianw | 100gb seems too much but i just upped both centos-8-stream's to taht | 22:41 |
| fungi | thankfully, overcommits are just assumed with afs | 22:42 |
| ianw | they are all a 100gb now | 22:42 |
| fungi | thanks! | 22:42 |
| fungi | i'm willing to bet it was me that forgot to set quotas when adding them | 22:43 |
| fungi | and i even remember adding that reminder to the docs so i'd stop forgetting | 22:43 |
| ianw | i feel like this is a "me" problem but happy to share blame :) | 22:43 |
| ianw | now, the problem is that the release job won't release unless all the wheel builds complete | 22:44 |
| fungi | that also seems overly optimistic | 22:44 |
| ianw | so we have the question -- will the broken bindep env stop the wheel build working, stopping the wheel release | 22:44 |
| fungi | will it not try to upgrade bindep? | 22:45 |
| ianw | yes this is an artifact of the way the jobs are structured in zuul, where the release job depends on the build jobs | 22:45 |
| fungi | 2.10.1 should force python 3.5 venvs to get a working version of the packaging module at least | 22:45 |
| fungi | what platform(s) do we still have a broken bindep env on? | 22:46 |
| fungi | the xenial images should be okay now | 22:47 |
| ianw | yeah, i think it's just that the periodic jobs haven't run since | 22:48 |
| fungi | that publish-wheel-cache-ubuntu-xenial failure you linked was before the new image got built | 22:48 |
| ianw | https://zuul.opendev.org/t/openstack/build/18fe53973c484558ab279f6b5ada8648/console is a different issue. that's actually tox not having libffi-devel installed | 22:48 |
| ianw | really, as a generic thing in zuul-jobs, i guess that it should | 22:49 |
| fungi | we realized around 14:00 utc that we could just delete the most recent image and fall back to the working one from the day before until the new one finished building and uploading, so anything which ran on ubuntu-xenial nodes after that no longer exhibited the issue | 22:49 |
| ianw | ahh, yes that's a good idea | 22:50 |
| ianw | actually ... the problem is *glean* should be specifying this in it's bindep | 22:50 |
| ianw | glean doesn't even have a bindep | 22:51 |
| ianw | i think we're up to problem 4? :) | 22:51 |
| fungi | yeah, anything which uses cffi should strongly recommend libffi-dev/libffi-devel and a c compiler, because at some point it will be installed onto a platform which, at least temporarily, has no available cffi binary wheel | 22:52 |
| fungi | as is the case with latest cffi and python 3.5 | 22:53 |
| ianw | i guess it doesn't really -- it doesn't have any requirements -- but it's test-requirements.txt pulling this in | 22:53 |
| ianw | so not required for glean runtime, but yes to run glean tox | 22:53 |
| fungi | of course, once you clear that hurdle, problem 5 is likely that latest cffi has dropped python 3.5 support and won't actually be buildable with it anyway | 22:53 |
| *** dviroel|rover is now known as dviroel|rover|afk | 22:53 | |
| ianw | yeah, problem 6 is if i should try to kick off the periodic wheel build jobs manually | 22:54 |
| fungi | doing an end run around problem 5, the cffi package only includes trove classifiers for python 3.6 and later, though curiously doesn't set requires_python metadata | 22:55 |
| ianw | while we still have xenial hanging around, we really do what glean to be py35 clean | 22:56 |
| ianw | s/what/want/ | 22:56 |
| fungi | thankfully, the cffi changelog doesn't mention dropping python 3.5 support, so maybe it'll still work if you can get the latest version to build and install | 22:56 |
| ianw | do we have bindep-test.txt. i feel like we discussed this ... | 22:57 |
| ianw | since the point of glean is to be the zero-dependency network init tool, it feels a bit odd to flag as requiring things in bindep it doesn't actually for runtime | 22:58 |
| Clark[m] | You can tag things test in the normal bindep file | 22:58 |
| fungi | tangential, but i'd never heard of heptapod until reading the cffi changelog just now, since that's where they moved their source code: https://foss.heptapod.net/ | 22:58 |
| fungi | ianw: yeah, most of our jobs install anything in the default bindep profile as well as the test profile. just add [test] after the package name | 22:58 |
| opendevreview | Ian Wienand proposed opendev/glean master: Remove debian-stable, add focal/bullseye testing https://review.opendev.org/c/opendev/glean/+/814697 | 23:02 |
| ianw | ^ that might get things going | 23:02 |
| fungi | and looking around for information on heptapod led me to discover https://fedeproxy.eu/ which we might want to look closer at | 23:04 |
| fungi | maybe not for us, but seems to be trying to bridge things like defect and task tracking between different code hosting platforms | 23:07 |
| fungi | ianw: you have a typo there, noted inline | 23:08 |
| fungi | oh, hah, loïc dachary is one of the folks who started fedeproxy | 23:11 |
| opendevreview | Ian Wienand proposed opendev/glean master: Remove debian-stable, add focal/bullseye testing https://review.opendev.org/c/opendev/glean/+/814697 | 23:11 |
| fungi | now i'm thinking i already knew about it and had since forgotten | 23:12 |
| *** dviroel|rover|afk is now known as dviroel|rover|out | 23:16 | |
| opendevreview | Merged zuul/zuul-jobs master: Add FIPS enable multinode job definition https://review.opendev.org/c/zuul/zuul-jobs/+/813253 | 23:54 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!