| fungi | has ansible-lint exploded on us? https://zuul.opendev.org/t/openstack/build/a9f991b05eac4a37aab3e98f5b01db8e | 01:25 |
|---|---|---|
| opendevreview | Ian Wienand proposed opendev/system-config master: tools/atc: remove https://review.opendev.org/c/opendev/system-config/+/881937 | 02:04 |
| ianw | fungi: yeah just looking at that. it's a bit weird, as it doesn't look like much has changed | 02:07 |
| ianw | ^^ is also fixed by https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/881938 | 02:08 |
| ianw | i am now getting the same error as the gate locally, so that's something | 02:08 |
| ianw | it must be https://pypi.org/project/ansible-compat/#history | 02:10 |
| ianw | "This release has been yanked" | 02:10 |
| ianw | why does pypi even bother supporting this :/ | 02:10 |
| ianw | (yanking) | 02:11 |
| ianw | yep, 3.0.2 v 4.0.1 | 02:11 |
| ianw | filed https://github.com/ansible/ansible-compat/issues/258 but looking at versions | 02:19 |
| fungi | foo | 02:19 |
| ianw | https://github.com/ansible/ansible-compat/pull/245/commits/e69ec0b813187911712bf524cc08396bf072264e#diff-1aa5764fb9e6d74d6a723ce1662397e6800caae433197d2c982e1299550c687c | 02:31 |
| ianw | that bit doesn't look safe -- ansible-lint calls get_cache_dir and is passing in a string, not Path object | 02:32 |
| fungi | love the pr description... "No description provided." | 02:34 |
| fungi | the commit message isn't a lot better | 02:35 |
| ianw | fungi: https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/881938 should avoid it, and i've made a pull request which may or may not be right | 02:53 |
| fungi | thanks! looks like it worked | 02:58 |
| opendevreview | Marcin Juszkiewicz proposed opendev/system-config master: reprepro: add Debian Bookworm config https://review.opendev.org/c/opendev/system-config/+/881952 | 09:40 |
| opendevreview | Marcin Juszkiewicz proposed opendev/system-config master: reprepro: add Debian Bookworm config https://review.opendev.org/c/opendev/system-config/+/881952 | 09:59 |
| *** dviroel__ is now known as dviroel | 11:36 | |
| opendevreview | Ashutosh Sarode proposed openstack/project-config master: Add Harbor app to starlingx https://review.opendev.org/c/openstack/project-config/+/881960 | 11:37 |
| opendevreview | Ashutosh Sarode proposed openstack/project-config master: Add Harbor app to StarlingX https://review.opendev.org/c/openstack/project-config/+/881962 | 12:01 |
| opendevreview | Ashutosh Sarode proposed openstack/project-config master: Add Harbor app to StarlingX https://review.opendev.org/c/openstack/project-config/+/881962 | 12:06 |
| opendevreview | Ashutosh Sarode proposed openstack/project-config master: Add Harbor app to StarlingX https://review.opendev.org/c/openstack/project-config/+/881962 | 12:37 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: linters: avoid broken ansible-compat https://review.opendev.org/c/openstack/project-config/+/881963 | 12:43 |
| *** sfinucan is now known as stephenfin | 13:04 | |
| opendevreview | Ashutosh Sarode proposed openstack/project-config master: Add Harbor app to StarlingX https://review.opendev.org/c/openstack/project-config/+/881962 | 13:14 |
| opendevreview | Merged openstack/project-config master: linters: avoid broken ansible-compat https://review.opendev.org/c/openstack/project-config/+/881963 | 13:26 |
| opendevreview | Merged openstack/project-config master: Retire puppet-tacker - Step 5: Remove Project https://review.opendev.org/c/openstack/project-config/+/875291 | 13:57 |
| opendevreview | Merged openstack/project-config master: Retire puppet-rally - Step 1: End project Gating https://review.opendev.org/c/openstack/project-config/+/879419 | 14:00 |
| opendevreview | Ashutosh Sarode proposed openstack/project-config master: Add Harbor app to StarlingX https://review.opendev.org/c/openstack/project-config/+/881962 | 15:02 |
| *** Guest74 is now known as atmark | 15:22 | |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Fix the "all" transformation so it actually works https://review.opendev.org/c/openstack/project-config/+/882075 | 15:25 |
| clarkb | fungi: in the commit message for ^ I'm confused because isn't a dry run going to not affect writing to disk? In other words this should be a noop other than it maybe does the work twice throwing it away once? | 15:29 |
| clarkb | or does it short circuit when processing the 0 transformation? | 15:30 |
| fungi | transformation "0" switches on dry run mode | 15:30 |
| clarkb | oh so it is on globally? | 15:30 |
| fungi | as does not specifying any transformations (in which case all are applied) | 15:30 |
| fungi | the way to not dry run but apply all transformations is to supply the list 1 2 3 4 5 6 7 8 9 | 15:31 |
| fungi | the "all" transformation is meant to be a shortcut for that | 15:31 |
| opendevreview | Ashutosh Sarode proposed openstack/project-config master: Add Harbor app to StarlingX https://review.opendev.org/c/openstack/project-config/+/881962 | 15:31 |
| fungi | but since the range it generated erroneously included 0 that caused it to be a dry run when not intended to be | 15:32 |
| clarkb | gotcha | 15:32 |
| fungi | though going back over the comments now i'm not positive that was intentional | 15:38 |
| fungi | er, wasn't | 15:38 |
| clarkb | oh I approved it already | 15:52 |
| fungi | i set it wip, it also failed check | 15:52 |
| fungi | i have an alternative i'm about to push | 15:52 |
| fungi | i think i misread the intent, and we never implemented the thing i wanted it to have, so i'm adding that instead | 15:53 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Add an "apply" transformation which applies all https://review.opendev.org/c/openstack/project-config/+/882075 | 15:54 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Make option indenting a selectable transformation https://review.opendev.org/c/openstack/project-config/+/882080 | 15:54 |
| fungi | clarkb: ^ that should make more sense, sorry | 15:54 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update user agent filter list https://review.opendev.org/c/opendev/system-config/+/882083 | 16:28 |
| clarkb | fungi: noonedeadpunk ^ fyi | 16:28 |
| fungi | thanks! | 16:29 |
| fungi | emergency single-core approved to hopefully get things less overloaded | 16:30 |
| noonedeadpunk | thanks! | 16:54 |
| clarkb | I think there is minimal testing of 882083 so we should keep eyes open for any trouble | 17:12 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Run gitea and static tests when update Apache UA filters https://review.opendev.org/c/opendev/system-config/+/882088 | 17:17 |
| clarkb | actually I'm going to flip the order of those changes | 17:18 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Run gitea and static tests when update Apache UA filters https://review.opendev.org/c/opendev/system-config/+/882088 | 17:18 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update user agent filter list https://review.opendev.org/c/opendev/system-config/+/882083 | 17:18 |
| clarkb | this short term slowdown should make things better longer term since we'll test our updates and deploy them automatically without humans needing to do things manually | 17:19 |
| clarkb | fungi: ^ rereview needed though | 17:19 |
| fungi | done | 17:25 |
| opendevreview | Merged opendev/system-config master: Run gitea and static tests when update Apache UA filters https://review.opendev.org/c/opendev/system-config/+/882088 | 17:37 |
| JayF | Does the opendev hosted zuul have a GITHUB_TOKEN available we can use to prevent API rate limiting against github? | 18:40 |
| JayF | Just checking to see if this has been solved at a high level before we consider something ironic-specific | 18:40 |
| clarkb | JayF: The zuul installation runs as a github application which is subject to api limits | 18:41 |
| JayF | We have a new job (metal3-integration for Ironic \o/) we saw fail on github rate limits | 18:42 |
| clarkb | these have largely only been an issue when doing multiple zuul restarts in a short period of time as zuul tries to figure out the state of the world for github repos from scratch | 18:42 |
| JayF | and it appeared as if we didn't set a GITHUB_TOKEN for that job at all | 18:42 |
| clarkb | JayF: can you link to the job? | 18:42 |
| JayF | yeah looking for it already | 18:42 |
| JayF | > Error: failed to read "cert-manager.yaml" from provider's repository "cert-manager": failed to get GitHub release v1.11.0: rate limit for github api has been reached. Please wait one hour or get a personal API token and assign it to the GITHUB_TOKEN environment variable | 18:43 |
| JayF | https://zuul.opendev.org/t/openstack/build/c597287b94bb45a9880e58287123dd64 | 18:43 |
| JayF | this is a frankenjob of sorts; using metal3-dev-env to spin up the ironic sha under test | 18:43 |
| JayF | so it's a bit of a snowflake as far as OpenStack jobs go | 18:44 |
| clarkb | ok your job failed making requests to github, Zuul didn't. That clarifies some confusion I had | 18:44 |
| JayF | Yeah, I'm basically asking if you all hydrate a credential I can drop in or if I need to manage that on my own | 18:45 |
| JayF | (next question after that would involve how to manage that secret) | 18:45 |
| clarkb | we do not. everything we do happens before your job starts running in a priveleged context using the github zuul application credentials | 18:45 |
| clarkb | if you express your metal3 dependencies as zuul repos then zuul can ensure that is on disk for you when the job starts | 18:45 |
| clarkb | but I'm not sure what exactly is going on there based on the error so hard to make a recommendation | 18:46 |
| JayF | That would essentially require us to do a hard fork of the metal3-dev-env (basically the development stack for metal3 devs), I think | 18:46 |
| clarkb | why? | 18:46 |
| JayF | I don't have a good answer to that. It's just my impression based on how it's laid out. rpittau is the expert on this job; I'm just trying to hydrate some context about our options | 18:47 |
| clarkb | right without knowing what requests are being made there it is hard for me to suggest anything the logging doesn't really say much about the actual operation | 18:48 |
| JayF | you did help, whether you realized it or not | 18:48 |
| clarkb | it is weird to me that you would need to do api requests at all when you can just look at the git repo state (since those look like yaml files you want) | 18:48 |
| JayF | I had the bad assumption that the real fix for that was to just get an API key to github to make it happy | 18:48 |
| JayF | but it sounds like we should try to arrange the job so it does less external fetches at runtime | 18:48 |
| clarkb | JayF: well it may still be depending on what the tool is doing. It is possible the infromation is only included in the github api | 18:49 |
| JayF | and instead lets zuul provide more of the deps it needs (?) | 18:49 |
| clarkb | JayF: yes I suspect that is one way around this | 18:49 |
| clarkb | essentially you would add this github repo to zuul like any other repo and then require that project in your jobs so it is on disk | 18:49 |
| clarkb | then you can grab the contents of that yaml file | 18:49 |
| JayF | ack; thank you, you helped me understand the scope of the problem better. I'll dig in further tomorrow | 18:49 |
| JayF | rpittau: ^ clarkb's suggestion for how to fix that metal3 github rate limit | 18:50 |
| JayF | rpittau: if you don't have time to dig tomorrow; I'll try to make time | 18:50 |
| clarkb | it looks like you are grabbing ~9 yaml files from github. Which should be well under the api limits | 18:50 |
| clarkb | I think they are like 5k requests per hour or something | 18:51 |
| clarkb | it is possible something is happening in the background that is exploding the request count | 18:51 |
| clarkb | looks like `clusterctl` kicks this off and I have no idea if you cna point it to things on disk vs it going off and talking to github | 18:54 |
| fungi | note that caching a local copy of the git repositories you're depending on would make for a more efficient experience for developers running locally too. this is how devstack works, for example | 18:54 |
| fungi | openstackansible too | 18:55 |
| fungi | cloning all your requirements from scratch every time you run a test is not very efficient, regardless of whether it's happening in ci | 18:55 |
| clarkb | https://github.com/kubernetes-sigs/cluster-api/issues/2450 | 18:56 |
| clarkb | its 60 requests per hour without auth. 5k per hour with auth | 18:57 |
| fungi | it's shocking to me that the kubernetes community struggles with very basic challenges like that | 18:58 |
| JayF | It's shocking to almost every other open source developer when they see how well opendev handles basic challenges like that :D | 18:59 |
| JayF | your perspective is broken becauase you all are good at your jobs :D | 18:59 |
| * JayF & | 19:00 | |
| clarkb | if I parse that correctly you can set up a goproxy somewhere (or use someone elses?) | 19:02 |
| clarkb | and that might help | 19:02 |
| fungi | that bug report was opened 3 years ago, and closed about a year ago by adding the ability to provide a github token? yikes | 19:07 |
| fungi | on the theme of things that i'm disappointed but not surprised by, people using text editors which don't differentiate between different kinds of whitespace characters (mainly spaces vs tabs) and not providing easy control over which to use | 19:34 |
| clarkb | I have vim configured to give visual indicators for tabs when used. Similar to what gerrit does in its diff rendering. Makes it super noticeable to me | 19:35 |
| fungi | yeah, me too | 19:38 |
| opendevreview | Merged opendev/system-config master: Update user agent filter list https://review.opendev.org/c/opendev/system-config/+/882083 | 19:44 |
| clarkb | the UA filter appears to have applied to gitea09 | 19:59 |
| clarkb | I need to pop out now but I suspect that gitea14 will be much happier now | 19:59 |
| ianw | ++ looks like that script, again :/ | 20:00 |
| fungi | i'm surprised it's only one script | 20:00 |
| clarkb | making a direct request to gitea14 responds quickly now too | 20:00 |
| fungi | ianw: your ansible-compat pr seems to be missing an update to tests that assert the thing you're reverting | 21:16 |
| fungi | or at least that's what i'm inferring from the ci failure | 21:16 |
| ianw | fungi: ohhh, hrm that could be right | 21:19 |
| fungi | i think that test was literally intended as "make sure nobody reverts this" | 21:20 |
| ianw | i mean, that makes sense | 21:21 |
| clarkb | hahahaha | 21:21 |
| ianw | now i look at the original, runtime.py needs a revert too | 21:21 |
| clarkb | request for AAAA glue records has been made | 21:22 |
| ianw | maybe i'll just make it a full revert as i don't want to try pulling apart the change any further. the cl doesn't give any context to help | 21:22 |
| fungi | it sends a clear signal, but i have a feeling i know how that will be received | 21:24 |
| fungi | still, it's probably a way to make some progress there | 21:25 |
| ianw | i pushed a full revert, see if that works | 21:30 |
| clarkb | infra-root tonyb https://etherpad.opendev.org/p/tk-kBybo141kvSiI_hCb here is a first draft of the fedora feedback email | 21:35 |
| ianw | lgtm at a first glance. the fact that we've missed f37 in nodepool shows how it never gets to the top of anyones todo list | 21:37 |
| fungi | yeah, i take that as a strong sign | 21:39 |
| fungi | corrected a minor typo, but lgtm | 21:43 |
| clarkb | JayF: reading more on the clusterapi thing I think it is already using a goproxy but still hitting the limits. I'm honestly amazed that this suggestion hasn't been implemented yet as this would make the problem go away completely aiui: https://github.com/kubernetes-sigs/cluster-api/issues/3982#issuecomment-1255450787 | 21:49 |
| clarkb | but this is webscale cloudnative you must provide authentication to github so they can track you :) | 21:49 |
| clarkb | JayF: I suspect that you will need to provide a token because this tool (clusterapi/clusterctl) isn't built to look at anything but upstream state and that requires at least some github api interaction | 21:51 |
| clarkb | JayF: https://zuul-ci.org/docs/zuul/latest/config/secret.html#secret is the way to deal with that. The use of a secret will make a job post review only | 21:58 |
| clarkb | tonyb: I've got to pop out now, but let me know what you think and if ou don't object I can get that sent either tonight or tomorrow | 22:48 |
| tonyb | will do. | 22:49 |
| tonyb | clarkb: looks good to me. | 22:52 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!