| gmann | fungi: frickler: boxiang has given the access and these patches are passing and ready to review: https://review.opendev.org/q/topic:skyline-setup | 03:56 |
|---|---|---|
| opendevreview | Merged openstack/project-config master: Add openstack-skyline irc channel in access an gerrit bot https://review.opendev.org/c/openstack/project-config/+/825881 | 13:03 |
| fungi | pip 22.0 was just released. looks like it adds ansi color and a non-ascii progress bar to its stdout | 15:05 |
| fungi | hopefully it's smart enough to avoid that when not run under a tty | 15:05 |
| fungi | at least they added a --no-color option to the cli | 15:06 |
| fungi | huh, suddenly ara isn't installed in the system-config-run-eavesdrop job, causing the post-run playbook to fail when it attempts to generate an ara report | 15:21 |
| fungi | i don't see anything we merged recently which would have caused this to start happening | 15:22 |
| fungi | oh! secondary failure | 15:24 |
| fungi | it's actually failing in the run phase, thanks to new pip dropping python 3.6 support from get-pip.py | 15:24 |
| fungi | now to figure out how we've worked around this for earlier instances | 15:25 |
| Clark[m] | I think we've always relief on get-pip.py respecting python requires. But I believe it takes arguments like pip and you can cap it manually | 15:26 |
| fungi | this is what we did for 3.5: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/pip3/tasks/xenial.yaml#L3 | 15:27 |
| fungi | and yeah, the latest version just hard bails with an error on too-old python: https://zuul.opendev.org/t/openstack/build/874d98b2995448a0ae7032a62a485a12/console | 15:28 |
| fungi | er, https://zuul.opendev.org/t/openstack/build/874d98b2995448a0ae7032a62a485a12/console#3/0/4/bridge.openstack.org for the exact task | 15:28 |
| Clark[m] | Oh right because the get-pip.py script itself doesn't run on older python. | 15:34 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Use versioned get-pip.py URL for Ubuntu Bionic https://review.opendev.org/c/opendev/system-config/+/826968 | 15:38 |
| fungi | Clark[m]: yep ^ | 15:38 |
| fungi | that's not going to be easy to get merged | 16:13 |
| fungi | looks like new pip is also breaking on downloading virtualenv? https://f51991741f95b04e3220-c981d66854858f594afcaf7c7ef9de6c.ssl.cf1.rackcdn.com/826968/1/check/system-config-run-codesearch/7d7ab4c/bridge.openstack.org/ara-report/results/286.html | 16:13 |
| fungi | HTML doctype missing or incorrect. Expected <!DOCTYPE html> | 16:14 |
| fungi | recommends --use-deprecated=html5lib or complaining on https://github.com/pypa/pip/issues/10825 | 16:14 |
| fungi | yeah, that issue is blowing up | 16:17 |
| fungi | apparently pip now wants any implementation of the simple api to consist exclusively of valid html5 documents | 16:19 |
| Clark[m] | Apache's auto listings aren't valid html5 I guess? That would only affect our wheel caches though? | 16:32 |
| fungi | aha, i think it's our wheel index causing this. it doesn't start with <!DOCTYPE html> | 16:32 |
| fungi | yep, we can prepend it, i'm sure | 16:32 |
| Clark[m] | This shouldn't be fatal for us is what I'm getting at and if an entire pip install breaks because one of two valid indexes breaks that would be a pip bug | 16:32 |
| Clark[m] | fungi: is the entire install failing or does it fallback and ignore the wheel cache? | 16:33 |
| Clark[m] | But also what a silly change | 16:33 |
| fungi | Clark[m]: it's not mod_autoindex at fault in this case... https://opendev.org/openstack/project-config/src/branch/master/roles/copy-wheels/files/wheel-index.sh#L13 | 16:33 |
| Clark[m] | Aha an easy fix at least, but we should file a pip bug if this is a fatal error | 16:34 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Make wheel indices valid HTML 5 https://review.opendev.org/c/openstack/project-config/+/826969 | 16:39 |
| fungi | Clark[m]: ^ yep | 16:39 |
| fungi | hopefully that one will pass without needing 826968 so we don't need to bypass testing | 16:41 |
| fungi | worst case i'll hand-edit the indices in afs temporarily to get the other merged | 16:42 |
| Clark[m] | We should be able to land a change that uses pip 21 then roll forward from there? | 16:43 |
| fungi | also possible, yeah | 16:44 |
| Clark[m] | I can't get over how silly a change like that is though. It provides no benefit when you just need a list of entities. The bug even notes the built in python parser is sufficient | 16:44 |
| fungi | well, except that if we try to use get-pip.py to do it, we need 826968 | 16:45 |
| Clark[m] | Ya might have to squash two things together I guess | 16:45 |
| fungi | also pressure is mounting in that pip issue to have the "feature" reverted or 22.0 yanked | 16:46 |
| fungi | Clark[m]: good news! we only run the linters job when that file is the only one changed | 16:50 |
| fungi | so we should be able to merge it normally, wait for the wheel mirrors to rebuild (or trigger that sooner), and then recheck the python 3.6 fix | 16:50 |
| Clark[m] | Sounds good I can approve shortly. Still bootstrapping my day | 16:51 |
| fungi | no rush, and thanks! | 16:51 |
| fungi | i need to switch to laundry and cooking anyway | 16:52 |
| clarkb | I have approved the wheel doctype fix | 17:27 |
| fungi | thanks | 17:27 |
| clarkb | I need to go eat breakfast and do weekend things though. Not sure how much more help I'll be today | 17:27 |
| fungi | no worries, i'll get it merged once the indices have regenerated | 17:27 |
| opendevreview | Merged openstack/project-config master: Make wheel indices valid HTML 5 https://review.opendev.org/c/openstack/project-config/+/826969 | 17:30 |
| fungi | okay, probably the most expedient way to update those is to reenqueue the periodic builds for openstack/requirements | 17:37 |
| fungi | i don't think i've ever done zuul-client enqueue for a timer-triggered ref before | 17:38 |
| fungi | i did it like `sudo zuul-client enqueue-ref --tenant=openstack --pipeline=periodic --project=openstack/requirements --ref=refs/heads/master` though the ref shows up in the status page as 0000000 | 17:41 |
| fungi | all the builds are error | 17:41 |
| Clark[m] | I think periodic enqueue and dequeue has been very iffy | 18:04 |
| Clark[m] | Does merging a requirements change trigger rebuilds? It did in the past iirc | 18:04 |
| Clark[m] | But now need to afk | 18:04 |
| fungi | the publish-wheel-cache-* jobs only run in periodic, from what i can see | 18:06 |
| fungi | yeah, nothing in post or promote pipelines for openstack/requirements | 18:08 |
| fungi | anyway, it looks like the pip maintainers have decided an automatic fallback to the prior library on error is called for, so maybe we'll see a new pip release with that in short order | 18:08 |
| fungi | if we want to override mod_autoindex, it looks like the solution is probably to set SuppressHTMLPreamble in our IndexOptions for those vhosts, and then provide a HeaderName directive pointing to a file which contains <!DOCTYPE html> | 18:16 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Override DOCTYPE in wheel cache autoindex https://review.opendev.org/c/opendev/system-config/+/826974 | 19:01 |
| fungi | frickler: that ^ should address the problem you observed | 19:01 |
| fungi | i manually tested it on mirror.sjc1.vexxhost because we currently have max-servers=0 there | 19:01 |
| fungi | i can consider editing the wheel indices in afs for 826969 rather than waiting for the periodic jobs to kick off (which also may fail for related reasons) | 19:14 |
| fungi | i guess it's a question of whether we want to wait for pip 22.0.1 to (hopefully) temporarily make this go away so that we can merge these normally, or assume that it may not happen soon enough to ignore for now | 19:16 |
| fungi | if we don't want to take our chances and wait, then i think it's probably going to be updating the indices in afs to make 826969 take effect sooner, bypassing zuul to merge 826974, waiting for that to deploy, and then rechecking 826968 which should hopefully be able to merge on its own after that and unblock us (unless we find yet still more pip 22.0 has broken which these failures are masking | 19:18 |
| fungi | for now) | 19:18 |
| fungi | i'll give folks who might be around an hour or so to object to that plan, and then assuming no concerns raised i'll start working through it in hopes we'll have things going again for monday | 19:19 |
| fungi | pip 22.0.1 didn't back out the default, only fixed some other bugs, so probably still going to be an issue for us until the above is done. https://github.com/pypa/pip/pull/10847 to add a graceful fallback is still awaiting further review | 19:56 |
| fungi | infra-root: seeing no objections yet, starting in a few minutes i'll bypass zuul to merge 826974 and manually apply 826969 to the files in afs while the first is deploying | 20:26 |
| opendevreview | Merged opendev/system-config master: Override DOCTYPE in wheel cache autoindex https://review.opendev.org/c/opendev/system-config/+/826974 | 20:58 |
| fungi | #status log Bypassed gating to merge https://review.opendev.org/826974 and manually applied the results of https://review.opendev.org/826969 to files in AFS in order to avoid waiting for daily periodic jobs to trigger. | 21:10 |
| opendevstatus | fungi: finished logging | 21:10 |
| fungi | once 826974 hopefully deploys, i'll recheck 826968 and see if it passes | 21:11 |
| fungi | huh... 826974 does not seem to have resulted in a deployment to the mirror servers | 21:14 |
| fungi | nevermind, it ran faster than i expected | 21:16 |
| fungi | checking now to see if that's sorted it | 21:17 |
| fungi | system-config-run jobs aren't bailing straight away this time, which is a good sign | 21:30 |
| fungi | ugh, i think the mirrors need a harder apache restart, i'm still seeing the old doctype from autoindex even though the configuration's in place | 21:36 |
| fungi | yeah, confirmed. i'll go through and restart apache on all of them as soon as i'm done with dinner | 21:38 |
| fungi | running this in a root screen session on bridge: ansible-playbook -f 20 /home/zuul/src/opendev.org/opendev/system-config/playbooks/service-mirror.yaml | 21:51 |
| fungi | okay, that wasn't what i needed. it's the same thing the deploy job does, so didn't actually have any effect | 21:58 |
| fungi | i just looped over all of them with ssh and did `sudo systemctl restart apache2` which seems to have done the trick | 22:04 |
| ianw | sorry had to run around, back now | 22:38 |
| ianw | what a weird mess | 22:44 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Allow the bootloader element with the default block device https://review.opendev.org/c/openstack/diskimage-builder/+/826976 | 22:59 |
| fungi | yay! looks like the index workarounds are successfully working around | 23:06 |
| ianw | fungi: thanks for chasing it down! | 23:08 |
| fungi | ianw: my pleasure! care to approve 826968 now that it's finally passing? | 23:14 |
| fungi | that should get us able to merge other system-config changes again | 23:14 |
| ianw | ++, thanks | 23:15 |
| fungi | once it lands, i'll recheck the change which originally clued me into today's problems (825882). seeing it land should provide some much-needed closure | 23:22 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!