| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 907509: WIP: Use git native commit with squash merge https://review.opendev.org/c/zuul/zuul/+/907509 | 01:18 | |
| @morucci:matrix.org | Hi, regarding the decrypt secrets subject, alright I understand the point. Then, what about if I propose a new simple Python tool stored in tools/ capable of extracting a given project's private key from the keys dump file (from export-keys) and that file (the decrypted private key) will be suitable to be used with decrypt_secret.py tool ? | 08:50 |
|---|---|---|
| @fungicide:matrix.org | fbo: or just extend the current decrypt_secret.py to optionally work on an export and have it take options for the location of the master key and project to extract | 14:19 |
| @vonschultz:matrix.org | Hello! Sometimes our users upload long relation chains of changes to Gerrit, resulting in one user taking up a disproportionate amount of compute resources, crowding out other users. Can this be managed somehow? I would ideally not forbid the long relation chains entirely, but it would be nice if the system was still reasonably responsive for the other users. I could imagine having a quota for the number of changes in flight per user, or for the number of concurrently running jobs per user, for example, or maybe something involving the Gerrit triggers so that the parent patch needs to have a positive vote from Zuul before a child patch gets its jobs enqueued, or enforce a certain maximum distance from the main branch to enter the pipeline. Or something else entirely. Is there any facility in Zuul that could help make it a bit more fair? | 14:19 |
| @fungicide:matrix.org | > <@vonschultz:matrix.org> Hello! Sometimes our users upload long relation chains of changes to Gerrit, resulting in one user taking up a disproportionate amount of compute resources, crowding out other users. Can this be managed somehow? I would ideally not forbid the long relation chains entirely, but it would be nice if the system was still reasonably responsive for the other users. I could imagine having a quota for the number of changes in flight per user, or for the number of concurrently running jobs per user, for example, or maybe something involving the Gerrit triggers so that the parent patch needs to have a positive vote from Zuul before a child patch gets its jobs enqueued, or enforce a certain maximum distance from the main branch to enter the pipeline. Or something else entirely. Is there any facility in Zuul that could help make it a bit more fair? | 14:28 |
| zuul's "fair queuing" algorithm tries to balance out resource consumption by using a sort of round-robin allocation across shared queues so that changes for one project or queue can't monopolize available node quota and choke out everything else. in your case, are the other users proposing changes for the same project, or for projects which share a named queue with the one that has the long chain of changes? | ||
| @fungicide:matrix.org | it also has an activity window for dependent pipelines, which can dynamically control how many changes are actively tested within a queue, in order to keep one queue from consuming too many resources (particularly when there are build failures causing job resets for other changes down-queue) | 14:31 |
| @fungicide:matrix.org | though if the case you're seeing is in an independent pipeline, that doesn't come into play either | 14:32 |
| @fungicide:matrix.org | since independent pipelines only ever have one active change per queue anyway | 14:33 |
| @fungicide:matrix.org | (every change gets its own queue) | 14:33 |
| @vonschultz:matrix.org | My users are all in the same Gerrit project. | 14:33 |
| @vonschultz:matrix.org | I'm thinking of our check queue, which is an independent pipeline. | 14:34 |
| @fungicide:matrix.org | got it, zuul doesn't currently have any throttles based on what gerrit accounts are associated with changes, and i'm not aware of any mechanism it has to limit the length of relation changes (though if there is one, it would probably result in a confusing behavior for your users, namely that some of their changes would never get tested until they shortened the length of that chain) | 14:35 |
| @fungicide:matrix.org | * got it, i don't think zuul currently has any throttles based on what gerrit accounts are associated with changes, and i'm not aware of any mechanism it has to limit the length of relation chains (though if there is one, it would probably result in a confusing behavior for your users, namely that some of their changes would never get tested until they shortened the length of that chain) | 14:36 |
| @fungicide:matrix.org | i tend to see those as social problems rather than technical ones, which need social solutions rather than technical ones. but i know everyone's environment is different | 14:37 |
| @vonschultz:matrix.org | Ideally everyone's changes _would_ be tested, but if you rebase a long relation chain on master, you don't get everything checked at once, but at some reasonable pace (say, one patch at a time). We could try to manage it with some policy rule, to only rebase one patch at a time in a chain, and only do the next once Zuul has responded to the previous one, but it's a lot more work than just telling git or Gerrit to rebase the entire relation chain. If we can make the desired behavior happen automatically, I'm leaning towards a technical solution. | 14:41 |
| @fungicide:matrix.org | Christian von Schultz: aha, what i described does actually exist, sort of: https://zuul-ci.org/docs/zuul/latest/drivers/gerrit.html#attr-%3Cgerrit%20connection%3E.max_dependencies and https://zuul-ci.org/docs/zuul/latest/tenants.html#attr-tenant.max-dependencies | 14:41 |
| @fungicide:matrix.org | i think that means you can set an upper limit either in the driver or tenant (or both) so that zuul will silently skip any changes which have "too many" other changes they depend on | 14:42 |
| @vonschultz:matrix.org | Looks promising, assuming "number of dependencies" will be the length of the relation chain. | 14:45 |
| @fungicide:matrix.org | that's the way i read it, but hopefully someone more familiar with that option can confirm | 14:45 |
| @fungicide:matrix.org | anyway, if it does work the way i think, just be prepared for users who push longer change series than that to come asking why some of their changes never got test results | 14:46 |
| @vonschultz:matrix.org | Yes. It's not quite the solution of my dreams, but it might still be useful. Thanks. | 14:47 |
| @fungicide:matrix.org | also, this is the release note associated with the introduction of those options, if it helps elucidate further: https://opendev.org/zuul/zuul/src/branch/master/releasenotes/notes/max-dependencies-7e50194dbe23aaaf.yaml | 14:48 |
| @fungicide:matrix.org | introduced recently in zuul 9.2.0 (2023-09-27) | 14:51 |
| @jim:acmegating.com | Christian von Schultz: fungi yes i wrote that feature in collaboration with an acme gating customer. it acts as you both describe, but i will say that the idea was more to protect the scheduler rather than necessarily for test resource consumption. think of it as an emergency limit designed to protect the system; it intentionally does not provide any feedback to users (since doing so would defeat the purpose of protecting the scheduler from denial of service). so you can set it to a lower value for your purpose, but that will increase the calls you get from users wondering why nothing happened. and you simply will not be able to test those changes at all until their parents merge. | 16:43 |
| @jim:acmegating.com | in a system like opendev, i'd probably not set it lower than, say, 100? i think 20-50 is probably about as long as our patch series typically get, and so the 30-100 range is where i think devs would start having friendly conversations about resource consumption.... and then after 100 we're just trying to keep the system from spending all its time querying gerrit. that's just a guess though; we haven't actually gone through that exercise for opendev. | 16:48 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 906765: Fix 10 second delay after skipped command task https://review.opendev.org/c/zuul/zuul/+/906765 | 17:43 | |
| @clarkb:matrix.org | Depending on your workflow the upcoming circular dependency refactor may also help reduce resource needs due to deduplication (I guess that works today but it will become more easy in the future) | 17:56 |
| @clarkb:matrix.org | If you are not using circular deps today then your stacks are probably already independent changes though | 17:57 |
| @clarkb:matrix.org | so maybe not as helpful | 17:57 |
| @jim:acmegating.com | yeah. though submitWholeTopic could be used to turn a linear stack of gerrit changes into a circular dependency. it's a workflow change and a change to the verification regime and bisectability guarantees, but something to consider. | 18:00 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-client] 907627: Handle forward compatability with cycle refactor https://review.opendev.org/c/zuul/zuul-client/+/907627 | 19:26 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: | 19:28 | |
| - [zuul/zuul] 907506: Update web ui for dependency refactor https://review.opendev.org/c/zuul/zuul/+/907506 | ||
| - [zuul/zuul] 906320: Finish circular dependency refactor https://review.opendev.org/c/zuul/zuul/+/906320 | ||
| - [zuul/zuul] 907628: Change status json to use "refs" instead of "changes" https://review.opendev.org/c/zuul/zuul/+/907628 | ||
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: | 20:13 | |
| - [zuul/zuul] 907256: Add --keep-config-cache option to delete-state command https://review.opendev.org/c/zuul/zuul/+/907256 | ||
| - [zuul/zuul] 907506: Update web ui for dependency refactor https://review.opendev.org/c/zuul/zuul/+/907506 | ||
| - [zuul/zuul] 906320: Finish circular dependency refactor https://review.opendev.org/c/zuul/zuul/+/906320 | ||
| - [zuul/zuul] 907628: Change status json to use "refs" instead of "changes" https://review.opendev.org/c/zuul/zuul/+/907628 | ||
| @clarkb:matrix.org | corvus: I'm starting to get a sense for the changes in the stack above. One thing I'm wondering about is whether or not we should do a release nowish | 23:06 |
| @clarkb:matrix.org | I think that probably depends on whether or not we're happy with people upgraded to the intermediate state we are at now. If that isn't a problem then having a more recent checkpoint befoer landing the big update is probably a good idea | 23:06 |
| @clarkb:matrix.org | maybe we do that after the web ui update? so that we're immediately behind the end of the refactor? | 23:08 |
| @clarkb:matrix.org | corvus: did my best to review the UI update. I have a visual suggestion but not quite sure how to reflect that in the code so unfortunately don't have a suggestion on how to fix it | 23:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!