| fungi | so in summary, main.yaml updates are still working because ansible is able to smart-reconfigure the scheduler on zuul02, and in theory this will clear up on its own over the coming weekend with the weekly restart now fixed | 00:00 |
|---|---|---|
| opendevreview | Merged opendev/system-config master: Remove Gerrit 3.4 jobs https://review.opendev.org/c/opendev/system-config/+/847035 | 00:20 |
| opendevreview | Merged opendev/system-config master: Add Gerrit 3.6 jobs https://review.opendev.org/c/opendev/system-config/+/847040 | 01:28 |
| opendevreview | Merged opendev/system-config master: Add Gerrit 3.5 to 3.6 upgrade testing https://review.opendev.org/c/opendev/system-config/+/847195 | 01:29 |
| opendevreview | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/847872 | 02:33 |
| opendevreview | Ian Wienand proposed opendev/grafyaml master: Fix datasource UID https://review.opendev.org/c/opendev/grafyaml/+/848070 | 04:26 |
| opendevreview | Ian Wienand proposed openstack/project-config master: grafana: Add infra deployment overview https://review.opendev.org/c/openstack/project-config/+/833213 | 04:43 |
| opendevreview | Ian Wienand proposed openstack/project-config master: grafana: add helper script https://review.opendev.org/c/openstack/project-config/+/833214 | 04:43 |
| opendevreview | Ian Wienand proposed openstack/project-config master: grafana: Add infra deployment overview https://review.opendev.org/c/openstack/project-config/+/833213 | 04:46 |
| opendevreview | Ian Wienand proposed openstack/project-config master: grafana: add helper script https://review.opendev.org/c/openstack/project-config/+/833214 | 04:46 |
| opendevreview | Ian Wienand proposed opendev/grafyaml master: [dnm] testing screenshot of .json panel https://review.opendev.org/c/opendev/grafyaml/+/848072 | 04:47 |
| ianw | it is still churning through the xfilesfactor changes | 05:00 |
| *** akahat|out is now known as akahat|ruck | 05:15 | |
| frickler | I unpaused gentoo builds now since we seem to be running nodepool with the dib release that should fix it | 05:18 |
| opendevreview | Ian Wienand proposed openstack/project-config master: grafana: add helper script https://review.opendev.org/c/openstack/project-config/+/833214 | 05:23 |
| opendevreview | Ian Wienand proposed openstack/project-config master: grafana: Add infra deployment overview https://review.opendev.org/c/openstack/project-config/+/833213 | 05:29 |
| opendevreview | Ian Wienand proposed openstack/project-config master: grafana: add helper script https://review.opendev.org/c/openstack/project-config/+/833214 | 05:29 |
| opendevreview | Ian Wienand proposed opendev/grafyaml master: Use fixed datasource UID https://review.opendev.org/c/opendev/grafyaml/+/848070 | 06:00 |
| opendevreview | Ian Wienand proposed opendev/grafyaml master: [dnm] testing screenshot of .json panel https://review.opendev.org/c/opendev/grafyaml/+/848072 | 06:00 |
| frickler | ianw: c9s is now starting to show build time data, which I think is a success of your fix, but the build status is now "32"? https://grafana.opendev.org/d/f3089338b3/nodepool-dib-status?orgId=1 | 06:22 |
| ianw | yeah, the 32 i think a result of scaling issues. it should 0 or 1, but can get aggregated to somewhere in between, and there's a x100 happening | 06:45 |
| ianw | i'm waving hands i forget exactly | 06:46 |
| ianw | anyway the .wsp file update is still happening, and it will take a few days for that to percolate out i think, as it rolls up the data into the larger buckets | 06:46 |
| ianw | 0/1 data doesn't represent well. it's a bit of a hack to try and find the last value and continue it over to show the current status | 06:49 |
| *** jpena|off is now known as jpena | 07:39 | |
| opendevreview | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/847872 | 07:52 |
| *** rlandy|out is now known as rlandy | 09:33 | |
| akahat|ruck | frickler, hey o/ | 09:47 |
| akahat|ruck | frickler, we have gate blocker could you put it on the top of gate? https://review.opendev.org/c/openstack/tripleo-ansible/+/848075 | 09:47 |
| akahat|ruck | top of tripleo queue * | 09:48 |
| *** dviroel|out is now known as dviroel | 11:20 | |
| fungi | akahat|ruck: since it looks like that change is still way at the back of you're gate queue, i'll assume your request remains relevant. promoting 848075,1 to the front of the tripleo gate queue now | 11:40 |
| fungi | er, at the back of your gate queue | 11:41 |
| fungi | and done | 11:42 |
| rlandy | fungi: yes - thank you so much! | 11:54 |
| fungi | no problem | 11:55 |
| akahat|ruck | fungi, thank you :) | 11:58 |
| jrosser_ | i'm trying to run zuul_swift_upload.py in it's local test mode but failing like this https://paste.opendev.org/show/bdXfjX8VWqtENkHYdVdD/ | 12:03 |
| jrosser_ | i understand what the error is saying, but what to do about it is less clear to me | 12:03 |
| fungi | can you install ansible.module_utils.zuul_jobs into the venv? | 12:20 |
| jrosser_ | fungi: i've tried pip install -e . in the root of the repo | 12:35 |
| *** pojadhav is now known as pojadhav|afk | 12:52 | |
| fungi | jrosser_: i'm starting to suspect that https://review.opendev.org/742736 may have broken stand-alone running of the script | 13:33 |
| jrosser_ | fungi: right - i asked someone here much more python aware than me to also take a look, and we're both really confused about the state of the code as it stands | 13:34 |
| fungi | jrosser_: i think the except ImportError may also need to catch newer ModuleNotFoundError | 13:35 |
| jrosser_ | for now i think i should be able to run it as an ansible module, which is a little more awkward to add debugging to | 13:35 |
| fungi | maybe try that first? | 13:35 |
| jrosser_ | really nice feature though to be able to dual-stack ansible modules and CLI like that | 13:35 |
| fungi | jrosser_: or try with python 3.7 if you have it handy? | 13:37 |
| fungi | looks like ModuleNotFound error is new in 3.8 | 13:38 |
| fungi | i think the try block doesn't fail when run under zuul because the module has been preinstalled into ansible's namespace | 13:39 |
| fungi | so the except has bitrotted in the face of newer python interpreters | 13:40 |
| fungi | mmm, looking back through the git history for another of my projects which does similar try/except on importing, some cases may have raised ModuleNotFoundError as far back as python 3.6, so maybe this hasn't even worked after 3.5 | 13:42 |
| fungi | making it backward-compatible with 3.5 gets a bit tricky. what i did in the past was to if not hasattr(__builtins__, 'ModuleNotFoundError'): ModuleNotFoundError = ImportError | 13:43 |
| fungi | and then forward-compat with except ModuleNotFoundError | 13:44 |
| fungi | maybe a bit hacky, there are probably cleaner approaches | 13:44 |
| fungi | jrosser_: anyway, i'm increasingly certain that's what's going on | 13:46 |
| jrosser_ | thanks for taking a look - i'll use it as an ansible module | 13:46 |
| fungi | though given the length of bitrot for that, i wouldn't be surprised if there are other problems lurking once the except is modernized | 13:47 |
| clarkb | fungi: jrosser_: initially the swift upload stuff was all in its own module and I bet the cli tooling worked then. But then google and s3 and so on were added and things were refactored to split common code out. I wonder if the cli portion was just never exercised after that refactor. It is probably worth asking in the zuul matrix room to see if anyone there knows how to make it work | 13:56 |
| fungi | right, the change i referenced is what refactored those to centralize the logic into upload_utils | 13:57 |
| fungi | and it tried to add relative imports as a fallback when the role isn't installed under ansible, but that's where things are breaking | 14:01 |
| corvus | none of that is bitrotted -- i use the cli all the time | 14:01 |
| corvus | here's the magic incantation: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/upload-logs-base/library/zuul_ibm_upload.py#L29 | 14:02 |
| clarkb | ah running it out of a dir that allows the python interpreter to dig down itno the roles and find what it needs | 14:02 |
| * jrosser_ makes patch | 14:03 | |
| *** dasm|afk is now known as dasm | 14:03 | |
| fungi | ooh, helpful code comment! would be good to copy that into the other scripts | 14:04 |
| fungi | thanks corvus! | 14:04 |
| opendevreview | Jonathan Rosser proposed zuul/zuul-jobs master: Add cli running example for s3 and swift log uploaders https://review.opendev.org/c/zuul/zuul-jobs/+/848135 | 14:09 |
| fungi | still struggling with working out how to call that from a venv with the dependencies installed | 14:09 |
| fungi | okay, i can do it with a venv outside of tox | 14:11 |
| fungi | cd roles | 14:11 |
| fungi | python3 -m venv foo | 14:11 |
| fungi | foo/bin/pip install -r ../test-requirements.txt | 14:12 |
| fungi | foo/bin/python3 -m upload-logs-base.library.zuul_swift_upload --help | 14:12 |
| fungi | but trying to `tox -e venv -- python -m upload-logs-base.library.zuul_swift_upload --help` with roles as my cwd as a short-cut seems to be second-guessing the import path | 14:14 |
| fungi | probably a tox option or two in a custom testenv for these would work | 14:15 |
| *** pojadhav|afk is now known as pojadhav | 14:35 | |
| clarkb | infra-root I'm sanity checking https://review.opendev.org/c/openstack/project-config/+/848034 and I don't see that project in codesearch results for any other zuul configs implying it is safe to remove from the tenant config and won't create additional errors. Is there anything else you think we should check before removing ap roject from the zuul tenant config like that? | 15:07 |
| clarkb | I suppose that a revert is straightforward and that is why I split the windmill and neutron-classifier removals into two changes as I figure they aren't related to each other so if reverts happen it will be one or the other | 15:07 |
| clarkb | I do think it would be neat to try and land one or both of those changes soon to test some of these assumptions. I have noticed that some openstack projects are starting to push changes up to fix the queue: thing too | 15:08 |
| corvus | clarkb: you could run zuul validate-tenants on it, but i probably wouldn't bother. just throwing it out there as an option. | 15:11 |
| fungi | ahh, okay, i don't need a special testenv definition in tox, this works with the general testenv:venv... | 15:11 |
| fungi | tox -e venv -- python -m roles.upload-logs-base.library.zuul_swift_upload --help | 15:11 |
| clarkb | corvus: ya that seems like overkill (and may haev a bunch of false positives from the other existing errors?) | 15:12 |
| corvus | yep | 15:15 |
| *** dviroel is now known as dviroel|lunch | 15:24 | |
| *** marios is now known as marios|out | 15:48 | |
| *** rlandy is now known as rlandy|biab | 15:51 | |
| opendevreview | Jonathan Rosser proposed zuul/zuul-jobs master: Create the log container and show the url before uploading logs https://review.opendev.org/c/zuul/zuul-jobs/+/848153 | 16:25 |
| *** rlandy|biab is now known as rlandy | 16:27 | |
| *** jpena is now known as jpena|off | 16:34 | |
| *** dviroel|lunch is now known as dviroel | 16:37 | |
| opendevreview | Jonathan Rosser proposed zuul/zuul-jobs master: Create the log container and show the url before uploading logs https://review.opendev.org/c/zuul/zuul-jobs/+/848153 | 16:38 |
| opendevreview | Julia Kreger proposed openstack/diskimage-builder master: DNM: Network Manager logging to Trace for Debugging https://review.opendev.org/c/openstack/diskimage-builder/+/847600 | 17:46 |
| *** undefined_ is now known as Guest3684 | 17:55 | |
| *** Guest3684 is now known as rcastillo_ | 17:55 | |
| *** lbragstad1 is now known as lbragstad | 17:57 | |
| *** rcastillo_ is now known as rcastillo | 17:57 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: WIP Update to Gitea 1.17.0-rc1 https://review.opendev.org/c/opendev/system-config/+/847204 | 18:28 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Move gitea partial clone test https://review.opendev.org/c/opendev/system-config/+/848174 | 18:28 |
| clarkb | infra-root I added the breaking change list from the gitea changelog to that commit message and went through each one and left my notes on them. There is one outstanding item that I believe needs testing via a held node which I'll set up if those two changes end up happy via a new DNM failing change | 18:29 |
| clarkb | Note this is only the breaking changes list and I haven't gotten through the rest of the change log | 18:29 |
| clarkb | also 848174 is reviewable and hopefully mergeable now. It cleans up a testing thing I noticed when adding more testing to this for 1.17.0 | 18:30 |
| clarkb | This release is a big one compared to some of the previous ones | 18:31 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM intentional failure to hold a node https://review.opendev.org/c/opendev/system-config/+/848181 | 19:54 |
| clarkb | I've just put a hold in place for system-config-run-gitea against ^ | 19:56 |
| *** dviroel is now known as dviroel|biab | 20:30 | |
| clarkb | https://158.69.70.31:3081/opendev/system-config shows me the favicon I expect. I think I understand the change better after being able to reread the code change | 20:44 |
| clarkb | basically they used the logo.svg as the favicon.svg and then fell back to favicon.png. But they want to support users having a different logo.svg and favicon.svg so now you must have a separate favicon.svg too. Which we already have | 20:44 |
| clarkb | I'll probably delete that held node today if anyone else wants to check it really quickly. I'll update the parent change that is collecting all this info about the upgrade with this information too | 20:45 |
| opendevreview | Clark Boylan proposed opendev/system-config master: WIP Update to Gitea 1.17.0-rc1 https://review.opendev.org/c/opendev/system-config/+/847204 | 20:47 |
| clarkb | PTG planning is starting to pick up. I expect that fungi and myself will be there. I don't think ianw or frickler are planning to go. | 20:50 |
| clarkb | corvus: do you know if you intend on being there at this point? and if so do you think it makes esnse to try and schedule time for opendev if it is the three of us? | 20:51 |
| clarkb | This isn't urgent but thought I'd start asking the ptg questions in case we do want to plan something. | 20:53 |
| *** timburke__ is now known as timburke | 21:01 | |
| clarkb | neat gitea added a healthcheck https://158.69.70.31:3081/api/healthz | 21:01 |
| corvus | clarkb: thanks; i hadn't realized that was so imminent. i don't know at this point; i'll try to start figuring that out. | 21:01 |
| clarkb | possible we could start using that to check the server is up doing rolling restarts and other operations, but it seems basic enough now that it probably isn't too different from what we are alread doing | 21:02 |
| clarkb | corvus: it is 4 months away which isn't super close but also not super far away | 21:02 |
| corvus | fun fact: (almost) everyone uses healthz because google did. google did because they already had a "/health" | 21:03 |
| clarkb | gitea 1.17 will add support for docker registries too | 21:07 |
| clarkb | (and python packages and php and maven and so on) | 21:08 |
| clarkb | at first I thought it was integrating with the upstream package repos but no it seems to be the actual repo in gitea | 21:08 |
| clarkb | oooh the new go to file feature is actually pretty neat. Try it at https://158.69.70.31:3081/opendev/system-config | 21:11 |
| clarkb | ok I've now skimmed much more of the changelog and I think we're probably in pretty good shape with the WIP change. Of course that can change when the final release happens | 21:18 |
| clarkb | But I suspect the bulk of the work there is done now and we can do final validation after the final release | 21:18 |
| *** timburke is now known as timburke__ | 21:39 | |
| *** timburke__ is now known as timburke | 21:40 | |
| ianw | 874687 stats later the xfilefactor resize is still running on graphite | 21:56 |
| clarkb | 874687 stats later ? | 22:04 |
| *** dviroel|biab is now known as dviroel | 22:09 | |
| ianw | .wsp files it's updated ... it's still going. i don't know how many files there actually are | 22:12 |
| clarkb | oh I see | 22:15 |
| *** dasm is now known as dasm|off | 22:15 | |
| *** rlandy is now known as rlandy|bbl | 22:25 | |
| *** dviroel is now known as dviroel|out | 22:28 | |
| fungi | lots | 22:33 |
| opendevreview | Chris Stone proposed openstack/diskimage-builder master: Removing old grub removal step https://review.opendev.org/c/openstack/diskimage-builder/+/817666 | 22:34 |
| fungi | setuptools 63.0.0 is looming, 63.0.0b1 is on pypi if anyone wants to test things with it | 22:35 |
| clarkb | I can't find a changelog for that version just the previous ones | 22:37 |
| clarkb | even checking out the beta tag it doesn't have changes listed | 22:37 |
| clarkb | https://github.com/pypa/setuptools/blob/v63.0.0b1/CHANGES.rst | 22:37 |
| fungi | notably for me, it supports editable/develop installs when used as a pep 517 build backend | 22:38 |
| fungi | yeah, i think they write their changelogs just before actual releases | 22:38 |
| fungi | it's mostly been pushed so folks can test pep 660 support: https://discuss.python.org/t/16904 | 22:39 |
| ianw | i guess touching the .wsp files has also expanded their lifespan, as we do a cleanup of anything that hasn't updated in ~9 months i think. i don't think that's going to blow things out though, we're < half full 1007G 399G 558G 42% /opt | 23:43 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!