| *** gthiemon1e is now known as gthiemonge | 12:41 | |
| *** pojadhav is now known as pojadhav|dr_appt | 13:50 | |
| *** dasm|off is now known as dasm | 14:30 | |
| *** frenzy_friday is now known as frenzy_friday|food | 14:45 | |
| *** frenzy_friday|food is now known as frenzy_friday | 15:33 | |
| clarkb | for tox v4 what I'ev realized is that projects like nova run the openstacksdk functional job which relies on openstacksdk's tox.ini file | 17:00 |
|---|---|---|
| clarkb | It is possible we might end up with project A relying on B relying on C relying on A compatibility issues though I suspect for tox.ini things its unlikely to loop | 17:01 |
| clarkb | but if we do have loops untangling that next week could be painful and updates should be made now if possible | 17:01 |
| JayF | When is the pin going away? Monday? | 17:01 |
| clarkb | the 22nd is what was announced iirc | 17:02 |
| JayF | That's very rough timing; there'll be a lot of folks gone... | 17:02 |
| JayF | I'll try to make sure Ironic is alright before I'm gone | 17:02 |
| clarkb | yes, but if we wait until janurary it will become "this is to oclose to the release" then once the release is done it will be "but we have this feature to write". We didn't control the timing of the tox release we've merely tried to address the bleeding so that we can move forward. But we have to move forward at some point | 17:03 |
| clarkb | in the meantime any developer fetching openstack and following dev instructions to run ttests before pushing a change will install tox and promptly discover nothing works | 17:03 |
| JayF | Oh, I don't think it's anyone's fault the tminig is unfortunate | 17:04 |
| JayF | it jus is | 17:04 |
| JayF | especially if we're going to need any highly coordinated fixes | 17:04 |
| JayF | honestly it makes me tempted to revise my time off schedule to make sure I'm working 12/22 | 17:04 |
| clarkb | my main concern at this point is the new developer or even old developer who isn't following too closely having trouble doing something that should just work for them | 17:04 |
| clarkb | but also I've written a handful of compatibility fixup changes at this point and there isn't too much to them. | 17:05 |
| JayF | Is it as simple as, install tox 4.0, run tests locally, it's fine? | 17:05 |
| JayF | or are you taking a more programatic approach? | 17:05 |
| clarkb | ya thats basically it. `tox -efoo --showconfig` is also useful as it will comment out things it is ignoring due to compatibility issues | 17:06 |
| clarkb | so ya you run tox and fix what explodes, then run tox --showconfig to see what isn't exploding and fix that. Make a commit and push it | 17:06 |
| JayF | ack; I'll get all Ironic projects tested right now then | 17:06 |
| clarkb | there are a few specific types of issues we've seen so far. The first is you need to convert whitelist_externals to allowlist_externals and that list needs to be complete as it doesn't produce warnings anymore but errors. Next is # are always treated as ini comments and need to be escaped if you don't want them to be a comment. Openstacksdk's problem is listing passenv env vars | 17:08 |
| clarkb | on one line space separated. They need to be on separate lines or comma separated. And ifnally neutron discovered that skipsdist might actually mean skip install and you may need to drop the use of skipsdist | 17:08 |
| gmann | it should be testable locally but have we identified all the changes/fixes we need to do ? or they need to investigate depends on their project ? | 17:08 |
| gmann | I think most of the projects tox.ini are similar | 17:08 |
| clarkb | gmann: it will vary. I've seen different groupings of the above problems | 17:08 |
| gmann | ok | 17:09 |
| clarkb | JayF: fwiw I'm not sure I'd revise time off schedules. Instead I was hoping to bring awareness to the prolem now so that we can head it off as much as possible | 17:13 |
| JayF | I'm going to make sure there's an ironic core around who I'd trust to troubleshoot if anything goes kaboom :) otherwise I'll make sure I have time that day to check up on things | 17:14 |
| gmann | challenge is cross projects fixes and if we have their core around to merge it even we push the fix | 17:19 |
| fungi | that's not really a problem though is it? if there are no core reviewers around to approve tox fixes, they're also not around approving anything else that would break because of tox version incompatibilities | 17:23 |
| clarkb | fungi: my concern raised above is that we've got cros project testing that is affected. For example nova depends on an openstacksdk job which requires openstacksdk to be updated | 17:24 |
| fungi | if a job fails in the forest and there's nobody around to hear it... | 17:24 |
| clarkb | its possible the incidence of that is small enough we can address it without too much trouble | 17:24 |
| clarkb | But also, this sort of situation might be the sort of thing where openstack fixes it rather than trying to rely on individual projects | 17:24 |
| gmann | fungi: is it about cross project, nova gate will be blocked until sdk fix are done | 17:25 |
| clarkb | fwiw sdk has reviewed the chagne, but it looks like osme of their tests are failing | 17:26 |
| fungi | in that scenario, nova will only be blocked if they don't disable the broken jobs, but yes that's mainly an argument in favor of better cross-pollination of core review teams | 17:27 |
| gmann | clarkb: nice, that is good | 17:27 |
| fungi | or if they don't fork the jobs and run their own modified versions | 17:28 |
| JayF | Stable branches are staying pinned to tox<4.0? | 17:30 |
| JayF | Or do we need to test/fix those too | 17:30 |
| gmann | yeah that is main issue, how to pin them? | 17:30 |
| fungi | depends on what you mean | 17:30 |
| gmann | for unit tests | 17:30 |
| gmann | we should pin them instead of fixing all | 17:30 |
| JayF | I don't know what I mean :) I'm asking if tox 4.0 compat fixes should be backported | 17:30 |
| clarkb | the zuul-jobs ensure-tox role applies globally without branching. It is a central zuul role and has no concept of project specific ranching policies | 17:30 |
| JayF | with an implied "please no" :) | 17:30 |
| fungi | for things using ensure-tox you can just override the version in the job definition, and can do it with branch variants | 17:31 |
| clarkb | that means when we unpin in ensure-tox your stable branches will also be unpinned. | 17:31 |
| clarkb | fungi: you can also set a project level var | 17:31 |
| JayF | that still means having to land a change on those project stable branches | 17:31 |
| clarkb | but I'm not sure if those are branch specific | 17:31 |
| gmann | we can pin it openstack-tox-* job at least | 17:31 |
| JayF | which means fighting CI on those stable branches :( | 17:31 |
| JayF | which means I might as well backport my tox fixes... | 17:31 |
| fungi | but also, remember that means telling people who want to work on stable branches that they need to keep an old version of tox on hand to do local testing | 17:31 |
| JayF | yeah, I think backporting the actual fix is the right move | 17:32 |
| JayF | but also unpleasant | 17:32 |
| gmann | openatack-tox-* pin shoould work for most of jobs until few are directly inherited from 'tox' jobs | 17:32 |
| gmann | or can we cap it in tox.ini with max_version or so? | 17:32 |
| JayF | I think this is absolutely a case of burn the candle at both ends: projects should fix tox in stable branches, and we should pin it so they don't have to | 17:32 |
| clarkb | fungi: exactly. I think in this situation the correct thing may be to backport not pin | 17:32 |
| fungi | also it's only the ensure-tox role in zuul-jobs which was pinned and so that's the only pin being removed next week. | 17:33 |
| clarkb | because it is accomodating the external world not internal features | 17:33 |
| JayF | because we don't want CI failing in earlier branches for this reason; but we also want the better dev experience | 17:33 |
| JayF | I view leaving the pin on for stable branches a way to make sure we don't give yet-another-painful-change to projects that are barely maintained | 17:33 |
| gmann | well we did pin for devstack based jobs | 17:33 |
| fungi | so for example, any pinning which was done for tempest isn't tied to what ensure-tox does or doesn't pin anyway | 17:33 |
| gmann | I will say be consistent on stable branch tesitng tox4 or not at all | 17:34 |
| clarkb | and pinning for tempest is less prolblematic for devs beacuse devstack manages all of that for you | 17:34 |
| gmann | either unit or integration testring | 17:34 |
| JayF | I don't think we can or should dictate that, neccessarily. Ironic will backport tox 4.0 compat fixes whether stable is pinned or not, b/c dev experience is crap otherwise | 17:35 |
| gmann | pinning in openstack-zuul-jobs should cover most of that but need to check | 17:35 |
| JayF | but I'm not oppossed to us pinning at a top level so CI always uses tox<4 | 17:35 |
| JayF | but we shouldn't tell projects not to fix stable branches, too | 17:35 |
| JayF | this just removes the time pressure | 17:35 |
| gmann | sure, its up to project if they want | 17:36 |
| fungi | right, luckily we haven't yet run into anything tox v4 requires which breaks v3 | 17:36 |
| fungi | so it should be possible to backport fixes for v4 without breaking the jobs that are still using v3 with the same configs | 17:37 |
| fungi | the whole situation with skipsdist being mutually exclusive of usedevelop is probably the biggest risk i can think of, though that's mainly going to impact job performance/duration not actual functionality | 17:37 |
| fungi | and the alternative workaround is to explicitly add {toxinidir} to your deps list for a given testenv if you really want to not install the project by default but have it in some specific testenvs | 17:38 |
| fungi | relying on usedevelop as a means of making sure the project gets installed is sort of abusing a feature which wasn't quite intended for that purpose | 17:39 |
| fungi | it's really for stating that you need the resulting venv to have the project installed in editable mode rather than normally | 17:40 |
| *** pojadhav|dr_appt is now known as pojadhav | 18:01 | |
| gmann | clarkb: I was testing pinning tox in openstack-zzul-jobs and then if project can unpin it in their zuul.yaml or not but this does not work as setting in template/job definition are taken priority https://review.opendev.org/c/openstack/glance/+/867880/5/.zuul.yaml#295 | 20:20 |
| gmann | clarkb: any other way to do that? | 20:20 |
| clarkb | gmann: no things closest t othe playbook have priority | 20:21 |
| clarkb | well no way around the precedence | 20:21 |
| clarkb | what you can do is inject vars with higher precedence via a job variant | 20:21 |
| clarkb | so when you run the foo-python38 job in your project you can make a variant for it. SImilar to how you might overide a nodeset. In this case you can override the var value | 20:21 |
| gmann | but that needs to be done for many jobs and mainly project zuul.yaml run them via template so they have to explicitly add jobs with that var override | 20:22 |
| gmann | I think I will drop the idea of pin and let project to backport fixes as you all were suggesting | 20:23 |
| gmann | I thought if it is easy way to unpin on project side then we can pin in central place | 20:24 |
| fungi | yeah, this is why the earlier zuul-jobs announcement suggested using depends-on to the wip change which will be removing the tox<4 pin | 20:24 |
| fungi | in order to test using tox v4 without having to make local configuration changes | 20:26 |
| gmann | yeah local config changes to unpin seems more complex than backport fixes :) | 20:26 |
| gmann | If time permit today or weekend, I will try to test/fix projects as many as I can | 20:27 |
| gmann | but let's keep the timeline same as holiday season can be taken as opportunity to fix them as not all developers will be on leave | 20:28 |
| gmann | shifting after holiday seems more disturbance to dev cycle/activities | 20:28 |
| clarkb | ya I think despite the holidays this is possibly the best time todo it. Fewer people are impacted and people who do plan to be around can get something done that helps others | 20:28 |
| clarkb | well best time might be too strong. Of all the bad possibilities it is least bad :) | 20:29 |
| gmann | yeah | 20:29 |
| JayF | we've already landed dozens of the fixes in ironic just today :) | 21:20 |
| *** blarnath is now known as d34dh0r53 | 22:29 | |
| clarkb | heads up https://github.com/tox-dev/tox/issues/2712 is something sfinucan discovered, then I ran into independently | 23:13 |
| *** dasm is now known as dasm|off | 23:59 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!