Wednesday, 2023-02-08

opendevreviewTony Breeds proposed openstack/election master: Remove DPL model projects.  https://review.opendev.org/c/openstack/election/+/87304600:32
opendevreviewchenker proposed openstack/election master: Add Ke Chen candidacy for Watcher PTL  https://review.opendev.org/c/openstack/election/+/87267700:49
opendevreviewchenker proposed openstack/election master: Add Ke Chen's candidacy for Watcher PTL  https://review.opendev.org/c/openstack/election/+/87267701:23
opendevreviewTony Breeds proposed openstack/election master: Remove DPL model projects.  https://review.opendev.org/c/openstack/election/+/87304602:49
opendevreviewMerged openstack/election master: Add Ke Chen's candidacy for Watcher PTL  https://review.opendev.org/c/openstack/election/+/87267703:19
*** JasonF is now known as JayF04:12
*** blarnath is now known as d34dh0r5306:52
*** blarnath is now known as d34dh0r5307:02
opendevreviewMichal Nasiadka proposed openstack/election master: Michal Nasiadka candidacy for Kolla PTL Bobcat/2023.2  https://review.opendev.org/c/openstack/election/+/87306608:35
*** amorin_ is now known as amorin09:09
opendevreviewAxel Vanzaghi proposed openstack/election master: Adding Axel Vanzaghi candidacy for project Mistral  https://review.opendev.org/c/openstack/election/+/87309509:50
*** blarnath is now known as d34dh0r5312:14
opendevreviewAxel Vanzaghi proposed openstack/election master: Adding Axel Vanzaghi candidacy for project Mistral  https://review.opendev.org/c/openstack/election/+/87309513:26
*** dasm|off is now known as dasm|rover13:51
*** tosky_ is now known as tosky14:19
*** dasm|rover is now known as dasm|afk15:36
gmanntc-members: weekly meeting in 5 min from now15:55
gmann#startmeeting tc16:00
opendevmeetMeeting started Wed Feb  8 16:00:17 2023 UTC and is due to finish in 60 minutes.  The chair is gmann. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
opendevmeetThe meeting name has been set to 'tc'16:00
gmann#topic Roll call16:00
dansmitho/16:00
gmanno/16:00
slaweqo/16:00
knikolla[m]o/16:01
noonedeadpunko/16:01
rosmaitao/16:02
gmannlet's wait for couple of min, meanwhile this is today agenda #link https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting16:02
spotzo/16:02
gmannlet's start16:04
gmann#topic Follow up on past action items16:04
gmannJayF to write document on how to remove pypi maintainer/owner access and a draft email for PTLs to use when asking former-contributors to give up their access.16:04
gmannJayF has started working on this, we will talk about it in separate topic we have next16:04
gmann#topic Gate health check16:04
gmannany better news with gate :) ?16:05
gmannwe are seeing lot of timeout in various tempest jobs now a days16:05
dansmithlots of timeouts :(16:05
gmannwhich blocking the things to merged16:05
dansmith*still* trying to get my chunked get patch merged, but it keeps getting nailed16:05
slaweqI noticed at the begining of the week some failure due to failed download of ubuntu images16:05
noonedeadpunkand timouts seems to be related just to some very slow/overloaded provider16:05
gmannyeah16:05
gmannnoonedeadpunk: not sure if that is the only reason16:06
slaweqyeah, timeouts is also something what I saw often recently16:06
dansmithwe're also seeing an occasional failure with the glance test that tries an external http download of cirros, so I have a patch up to make that retry a few times16:06
gmannyes, I need to review that. will do today16:06
noonedeadpunkWell, out of our jobs, they do timout in the middle of execution, on tasks that was not hanging for too long16:06
dansmithit hasn't gotten a clean test yet, but it is passing the tests it's supposed to of course16:07
noonedeadpunkSo it looked like execution is slow overall16:07
dansmithnoonedeadpunk: same16:07
gmannnoonedeadpunk: ohk16:07
dansmithI've also seen timeouts in older release jobs,16:08
dansmithso I don't think it's anything like keystone or nova suddenly do a sleep(1) in every request or anything16:08
dansmithi.e. a performance regression in one of the release-specific services that is causing it to go slower16:08
noonedeadpunkno-no, the jobs that passing do have proper execuiton time16:08
gmannyes, I also did not see any increase in time for passing job 16:09
noonedeadpunkand timouts are quite random on scenario/OS16:09
dansmithare the passing jobs just under the job timeout? or do the timed out jobs end up running way longer for some reason?16:09
gmannwhat i observed is passing jobs are taking time they use to do in previous releases or so16:10
gmannat least not noticeable increase in their time16:10
noonedeadpunkso my bet would be on some overloaded provider. I think it should be possible to identify what provider is that, but I'm not sure we have spare capacity for nodepool to just disable it...16:10
dansmithgmann: okay but it seems to me like they're fairly close to the timeout right?16:11
dansmithlike a full-xena run that did not timeout ran in 1:55 but a full-zed that did was 2:0516:11
dansmithnoonedeadpunk: if nothing is wrong and it's just a slow provider, then perhaps a different job timeout for that provider, or we split up some jobs so they run in less time16:12
gmannbut if you see other passing rate tempest-full-py3 takes ~1.35 in many time16:12
dansmithgmann: on that same patch I was looking at full-py3 took 1:5716:12
dansmithbut okay16:12
noonedeadpunkdansmith: well, we had job timeout set in 3h, and job that takes in average 1.40-2h does timeout16:13
gmannyeah not always. 16:13
slaweqnoonedeadpunk I think it's the same in neutron check queue as well16:13
dansmithis it set at 3h or 2h? because it seems like 2h is the threshold from what I'm seeing16:13
slaweqbut I don't have links to any jobs now16:13
funginoonedeadpunk: zuul logs the provider id if the builds have logs available. if not, i should be able to look it up from the executor's service logs16:14
gmanndansmith: tempest multinode/slow jobs are 3 hr ands rest are 216:14
noonedeadpunkdansmith: in osa we have 3h16:14
dansmithgmann: okay16:14
noonedeadpunkfungi: yup, logs are available - they don;'t reach post timeout16:14
fungiin that case, look in the zuul info log files16:14
fungiyou may also be able to correlate it with dpawlik's opensearch service16:15
gmannrax-iad is recent provider I am seeing taking 3 hr for tempest-slow job which used to take 2 hr https://zuul.opendev.org/t/openstack/build/5f5f48160b0049d99153d0d60909cee7/log/job-output.txt#4916:15
dansmithgmann: I guess my point is.. it seems like some legit full runs are dangerously close to the 2h mark, so either a slight performance regression, or a few extra tests could be causing all the jobs on our slowest provider to tip over that mark lately16:15
gmanndansmith: yeah, agree with that. i have started the test split in this, #link https://review.opendev.org/c/openstack/tempest/+/87305516:16
gmannneed some more work on that, checking some project gate coverage on those extra tests16:17
dansmithgmann: ack16:17
gmannanyways, let's monitor the provider if we can identify the one making things slow16:17
gmannanything else on gate ?16:18
gmann#topic Cleanup of PyPI maintainer list for OpenStack Projects16:18
gmannEtherpad for audit and cleanup of additional PyPi maintainers16:19
gmann#link https://etherpad.opendev.org/p/openstack-pypi-maintainers-cleanup16:19
gmannML discussion16:19
gmann#link https://lists.openstack.org/pipermail/openstack-discuss/2023-January/031848.html16:19
gmannas discussed in last meeting, JayF is working on the email template and steps to remove/give access to openstackci16:19
gmannha has prepared the email template for PTLs to ask additional maintainers to do the required steps #link https://etherpad.opendev.org/p/openstack-pypi-maintainers-cleanup-email-template16:20
gmannadding PyPi steps also here #link https://review.opendev.org/c/opendev/infra-manual/+/87303316:20
gmannplease check and add it in etherpad if any feedback16:20
JayFI will respond to feedback on that PR/etherpad this afternoon.16:20
gmann+116:21
gmannanother thing is to explicitly add PyPi things in governance as policy16:21
gmannnoonedeadpunk: if i remember correctly you wanted to add it in documentation ?16:22
gmannthat will help to clearly document the process on PyPi acess16:22
noonedeadpunkYes, I didn't manage to push patch yet as took a week of vacation16:22
noonedeadpunkBut was planning to do that during this week16:23
gmannperfect, let me add action item just to track16:23
knikolla[m]great! 16:23
gmann#action noonedeadpunk to add PyPi access policy in governance documentation 16:23
noonedeadpunk+116:24
gmannJayF: as you are on top of your email template and steps documentation, do you want me to continue your action item for next meeting?16:24
JayFIt'd be ideal for me if we can action those before next meeting16:24
JayFe.g. get general agreement on email and if the change merges, we can update actions for PTLs16:24
gmannJayF: also, please add this email template/sending to main etherpad also in step1 #link https://etherpad.opendev.org/p/openstack-pypi-maintainers-cleanup16:25
JayFI'm waiting to do that until folks here give a general agreement to the etherpad :)16:25
gmannsure16:25
JayFif it's OK for me to do that; I'm going to do it :D 16:25
gmannJayF to write document on how to remove pypi maintainer/owner access and a draft email for PTLs to use when asking former-contributors to give up their access.16:25
gmannthanks JayF for working on that16:25
gmannanything else on PyPi things?16:25
gmann#topic Recurring tasks check16:27
gmannBare 'recheck' state16:27
gmann#link https://etherpad.opendev.org/p/recheck-weekly-summary16:27
gmannslaweq please go ahead16:27
slaweqnothing really new there16:28
slaweqI updated stats, generally gates aren't very stable as we discussed already16:28
gmannyeah16:28
slaweqand it is visible in number of rechecks needed to merge patches16:28
slaweqnothing else except that16:28
gmannok. thanks for updating16:29
gmann#topic Open Reviews16:29
gmann#link https://review.opendev.org/q/projects:openstack/governance+is:open16:29
gmann6 open reviews out of which 3 are ready to review16:29
gmann#link https://review.opendev.org/c/openstack/governance/+/87223216:29
gmann#link https://review.opendev.org/c/openstack/governance/+/87223316:29
gmannthis is new one #link https://review.opendev.org/c/openstack/governance/+/87276916:29
gmannslaweq I will check it today, opened it in tab16:30
gmannothers are waiting for their dependency to merge first16:30
slaweqthx16:30
gmannthat is all on open reviews16:30
gmanntwo things more from my side16:30
gmann1. elections: nomination are open. tc members whos term are completing in this election and thinking to re-run, please check nomination deadline16:31
gmannalso encourage other members to run even TC or PTL for your known projects. deadline for nomination is Feb 15, 2023 23:45 UTC16:32
slaweq++16:32
knikolla[m]time flies16:33
gmannyeah :)16:33
gmann2. open infra Board syncup call today at 20 UTC details are mentioned in #link https://etherpad.opendev.org/p/2023-02-board-openstack-sync16:33
gmannwhich is zoom call in 3hr 30 min from now16:33
gmannplease plan to attend or feel free to add topics to discussed in etherpad16:34
gmannthat is all from agenda and me for today. we have ~26 min left if anything else to discuss from anyone ?16:34
slaweqot16:34
slaweqit's pretty late for me but I will try to be16:34
gmannack16:34
noonedeadpunkbtw, should we somehow start tracking how projects are progressing with adding upgrade jobs for N-2?16:35
gmannnoonedeadpunk: good point, we do have grenade-skip job which we will update on release and if I am not wrong that will be added in integrated gate template16:36
gmannbut that is only for 4-5 projects. and manila does too. but i agree it will be good to ask other projects to do so and track that16:36
gmann~16 projects have the greande plugin and should add the skip level testing #link https://docs.openstack.org/grenade/latest/plugin-registry.html16:37
noonedeadpunkAs while it's matter of practise for SLURP upgrades, I think we'd better have things and mindset in place sooner then later16:37
gmannyeah16:38
dansmithgmann: how many have it voting?16:38
gmanndansmith: good question.  I have not checked that but yes many of them might be non voting16:39
dansmithAFAIK, pretty much everyone should have it voting in current master right?16:39
gmannyes16:39
slaweqYou mean skip-level job voting?16:39
dansmithyes16:39
noonedeadpunkBut does everyone have grenade-skip in some default template?16:39
gmannnoonedeadpunk: not for this cycle as this is non-SLURP 16:40
dansmithwait what?16:40
gmannthose run only for SLURP releases.16:40
slaweqI though that we agreed to have them non-voting for now as it will be just first SLURP release now16:40
dansmith2023.1 is a slurp no?16:40
slaweq2023.1 will be first slurp16:40
slaweqyes16:40
rosmaitayes, but in the forward direction16:40
noonedeadpunkWell, it's first slurp. so it's indeed can be non-voting16:40
gmanndansmith: i mean the first immediate SLUPR after 2023.1 16:40
noonedeadpunkBut already running at least?16:40
dansmithoh you mean because it's the first, right right16:40
gmannin 2023.1 it is non voting yes as we discussed, but in 2023.3 it should be voting16:41
dansmithack, but opt-in to voting for 2023.1 would be good if the project can swing it16:41
gmann2024.116:41
slaweqso in neutron we have neutron-ovs-grenade-multinode-skip-level and neutron-ovn-grenade-multinode-skip-level jobs but those are non-voting for now16:41
gmannin 2023.2 it will not run. in 2024.1 it should be voting 16:41
dansmithyep, I keep thinking this is the first upgradeable-to but it's really the first upgradeable-from...I'm on the page now :)16:42
noonedeadpunkI don't see them in heat at least https://opendev.org/openstack/heat/src/branch/master/.zuul.yaml#L185-L20216:42
gmannI think we have updated our PTI document also for that but not every project knowing that16:42
noonedeadpunkUnless it's part of some template16:42
noonedeadpunkdansmith: I think it's unofficial upgradable-to as well, as a matter of practise?16:42
gmannnoonedeadpunk: if I remember, except manila and grenade in-tree projects, I do not think any one else staretd 16:43
dansmithnoonedeadpunk: just not required yeah16:43
gmannso this is for 2024.1 and something we can start in next cycle to ask projects to prepare those jobs and rn16:43
gmannrun16:43
noonedeadpunkok, maybe worth writing some ML to encourage projects in doing so?16:43
gmann+116:44
noonedeadpunkok16:44
gmannno harm to do that16:44
dansmithyeah, running it for everyone, encouraging projects to opt-in to voting now would be best, IMHO16:44
gmannsure16:44
dansmithmeaning non-voting in the template16:44
gmannlet me put the data in etherpad about what all project need to add and then I can send it on ML to ask projects16:45
slaweqmaybe we should talk with release team so they can add info about it in their weekly emails about releases? To remind projects e.g. to remove those jobs from gate in 2023.2 cycle16:45
gmann#action gmann to prepare etherpad for grenade skip upgrade job data and send email asking required projects to add job 16:46
gmannslaweq: that also work and even I can add it in my weekly summary too16:46
gmannonce 2023.2 start16:46
slaweqgmann++ thx16:47
gmannnoonedeadpunk: thanks, good reminder. 16:47
gmannanything else for today?16:47
spotznot from me16:47
gmannok, let's close then. thanks everyone for joining. 16:48
gmann#endmeeting16:48
opendevmeetMeeting ended Wed Feb  8 16:48:36 2023 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:48
opendevmeetMinutes:        https://meetings.opendev.org/meetings/tc/2023/tc.2023-02-08-16.00.html16:48
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/tc/2023/tc.2023-02-08-16.00.txt16:48
opendevmeetLog:            https://meetings.opendev.org/meetings/tc/2023/tc.2023-02-08-16.00.log.html16:48
slaweqo/16:48
spotzo/16:49
clarkboh I wanted to call attention to ianw's email about gerrit acls16:50
clarkbBasically gerrit 3.6 (the version we are running on) is the transition version between label functions and submit requirements in gerrit acls. ianw has a change up to begin converting these in our acls16:50
clarkbwe believe the change to be a 1:1 mapping but openstack uses this heavily for things like backport candidates and review priority so be on the lookout for behavior changes after that merges16:51
clarkbhttps://review.opendev.org/c/openstack/project-config/+/867931 is the change16:51
clarkbthis doesn't actually change the functions yet just the copy conditions, so there will be an iterative process as we convert each little aspect of this16:52
noonedeadpunkbtw about that email - it was the time I realised we do have dash-creator. As we currently are using manually create dashboard that's not commited anywhere16:52
clarkbwe should probably also try to get around to enabling hashtags globally since a lot of this could be addressed by hashtags I think16:54
clarkbbackport candidates in particular come to mind16:54
clarkbre timeouts one thing to check is swap use16:55
noonedeadpunkwell. it's a bit tricky, as making a typo in hashtag will lead to missing patch from the dashboard16:56
clarkbin the past it has correlated well to jobs suddenly being slow. Particularly on providers that don't have high iops16:56
clarkbadditionally we can be our own noisy neighbors.16:56
clarkbHaving a bunch of jobs all swapping on the same host is unlikely to be good for performance of the jobs running there16:57
noonedeadpunkso label in that sense is protecting from stupid mistkaes16:57
clarkbnoonedeadpunk: ya thats fair, I personally don't mind how ya'll do it, just want to ensure you have options16:57
JayFclarkb: we explicitly wanted votes over tags for BC so you can BC-117:01
fungialso labels allow for per-reviewer feedback while tags are effectively shared17:04
opendevreviewAmy Marrich proposed openstack/election master: Amy Marrich TC candidacy  https://review.opendev.org/c/openstack/election/+/87316017:27
fungitc-members: please see https://lists.openstack.org/pipermail/openstack-discuss/2023-February/032083.html "Last maintained release of TripleO is Wallaby" since it contains a governance question17:52
noonedeadpunkI assume it's project deprecation, isn't it?17:56
gmannyes, deprecation case and then retire once last maintained rlease is EOL17:57
noonedeadpunkwhen master and futher development is stopped, and stable branches are still maintained. Though I'm not sure how to deal with Zed if it was branched... Likely it can be EOLed, but it's quite weird from governance prespective indeed17:58
funginoonedeadpunk: the governance question is one of ptl elections17:58
fungisee the last paragraph17:58
noonedeadpunkYeah, ok, there were 2 questions then I guess :)17:59
fungithough on rereading, i guess the question is on how to deprecate new development, and not about leadership. so yeah, a pointer to the deprecation process could be in order18:00
noonedeadpunkI'd say no PTL is needed for maintain deprecated project18:00
fungihe does at least say that they'd nominate someone to act as ptl if required, so a semi-question i suppose18:00
gmannwell they can have PTL as they maintain on stable release and still in governance 18:01
gmannreplying with the deprecation process and still they can propose for PTL18:01
noonedeadpunkI'm not sure if they have stable-maint-core as a separate group?18:01
fungiseems like dpl could work, they're planning to handle stable branch testing and security fixes, which means liaisons for infra and security could be warranted, not sure if they issue point releases from their stable branches18:01
noonedeadpunkthough I don't think PTL is a requirement18:01
noonedeadpunkyeah, exactly was thinking about dpl18:02
gmannyeah, dpl can be option too but honestly saying it is same thing for them if 1 person listed as DPL liaison or PTL18:03
fungiright. if they want to spread those roles around then dpl could makes sense18:03
gmannbut until project is in governance even though deprecated we should have some point of contact18:03
gmannsure18:04
fungianyway, just wanted to call that post out since it's a pretty significant change in course for a long-time official project team, and they didn't add [tc] in the subject so it could have gone unnoticed18:04
dansmithseems kinda weird to sort of abandon an existing should-be-supported release right?18:07
dansmithI mean, I'm guessing it's true that there would be few objections, but...that's a bit odd, no?18:07
gmannbut it was not release at all after wallaby which is not caught?18:09
fungii thought they announced a while back that they weren't going to make releases of tripleo every cycle and were consciously tying them to rdo releases which have a different cycle driven by rh's downstream support decisions18:09
dansmithokay, so there have been no releases since wallaby but there are named release branches?18:10
fungiprobably worth asking their team though, rather than speculating here18:10
clarkbthe email does imply zed was maintained though18:10
clarkbbut ya I'm not sure at what level18:10
dansmiththey say "zed release" in the mail18:10
dansmithright18:10
gmannthere releases are not marked as deprecated yet18:10
dansmith"empty commit those branches" seems a bit drastic to me18:10
gmannyeah that was confusing. it is strange that releases were missed ?18:11
dansmithwhy do you say missed?18:12
dansmithbut this says only wallaby: https://releases.openstack.org/teams/tripleo.html18:12
gmanndansmith: I mean xena, yoga, zed rleased are missed right as it is not yet deprecated 18:13
gmanndeprecation process/question should be in xena release itself18:13
noonedeadpunkthey do have Zed branch18:13
noonedeadpunkhttps://opendev.org/openstack/tripleo-common/src/branch/stable/zed18:13
gmannyeah, that is another strange, wallaby and then directly zed they have not deprecated it yet so 18:14
dansmiththey have branches, they refer to "zed release" in their email, but yeah I don't see any releases on that page past wallaby18:14
dansmithso I'm confused18:14
gmannyes18:14
noonedeadpunkThey were targeting to skip couple of releases to catch up with refactoring or smth IIRC18:14
dansmithgmann: are you talking about deprecating the *project*?18:14
gmannhttps://github.com/openstack/tripleo-common/branches18:14
* noonedeadpunk was talkign about deprecating project at least18:14
gmanndansmith: yes, they are not marked as deprecated or release-deprecated means they are supposed to do release18:15
dansmithyeah, I thought they were moving to release-independent though18:15
gmannrelease team monitor that i think if anyone missed release then they propose to mark it release management None or deprecate release18:15
noonedeadpunkI do recall their ML in W though when they said - we're gonna skip couple of releases but will do Z or Y iirc18:15
noonedeadpunkNot sure I will be able t ofind that quickly though18:15
gmannohk18:15
dansmithyeah, something18:15
gmannelodilles: ^^ you know about tripleO release skipping for xena, yoga, and zed also?18:16
dansmithbut still... they say "zed release" in the email, and even if the branches weren't released, it seems odd to destroy them, which is what they're implying I think18:16
noonedeadpunkI'm not sure they've skipped Z?18:16
dansmithmaybe they just meant "zed branch"18:16
noonedeadpunkyeah, dunno...18:16
gmannI cannot find it here https://releases.openstack.org/zed/index.html18:16
gmannyeah, may be zed branch not release18:17
noonedeadpunkwell 18.0.0 tag seem to reffer zed18:17
noonedeadpunkor it's even master....18:18
gmannah, here it is independent releases https://github.com/openstack/releases/blob/3bef29d74c7be4a834518e3b01dd7ee20c79a22e/deliverables/_independent/tripleo-common.yaml18:20
gmann17.0.0 was cut as stable/zed18:20
dansmithso there are releases after wallaby,18:20
dansmithbut they don't show up on the releases page18:20
noonedeadpunklooks quite messy...18:21
gmannyeah. it should show there even it is independent 18:21
gmannbut yes all seems messy18:21
gmannah, we have that in independent so release page is all good https://releases.openstack.org/independent.html#tripleo-common18:22
dansmithokay weird18:23
gmannso after 15.0.0 (stable/wallaby) it is kind of no stable branch things/backport support but they do have new releases18:24
dansmithare they expected to support those releases? I would expect so18:24
dansmithat least not delete the branch before EOL18:24
dansmithif they're not around to support them, then we can't expect them to, but I guess if they're supporting wallaby, it seems uncool to change the support envelope of releases after that18:25
noonedeadpunkI think main question is what we're supposed to do with branched (and likely tagged) Zed?18:25
noonedeadpunkYes, exactly18:25
dansmithnoonedeadpunk: well, the email said they would "empty commit those branches", which to me means leave them up so they're find-able, but effectively destroy (as much as could happen in git) them to avoid people running or packaging those right?18:26
gmannwell, we do not have stable branch support as governance requirement right? but agree that we should have somewhere marking that project-supporiting-branch now stopped supporting it18:26
noonedeadpunkoh, don't we? As I thought release cadence and supporting stable releases is kind of requirement...18:27
dansmithgmann: we do *not* have stable branch support as a requirement? meaning for independent releases or what?18:27
dansmithyeah me too18:27
fungi"destroy" is a bit strong, but yes break them sufficiently that people will hopefully spot the readme file in them that says what's happened18:27
gmannindependent releases, branchless repo are example of no stable branch right18:27
dansmithfungi: yeah I said "as much as could happen in git18:27
fungithey can always grab HEAD^1 if they need it18:27
dansmithfungi: doing that seems not very constructive though.. a final commit in the readme or something, maybe18:27
noonedeadpunkTo be frank, I'd vote to mark zed as EOL for this specific project rather then empty it....18:28
dansmithgmann: okay IDK, I thought releases need to be supported18:28
dansmithnoonedeadpunk: ++18:28
noonedeadpunk+1 ^18:28
gmannagree on marking zed EOL that is confusing18:28
mnaserwow, tripleo winding down18:29
gmanndansmith: release needs to be supported but cutting stable branches are not18:29
dansmithit's also weird to have releases after the last supported one be EOL but the last supported one not18:29
dansmithgmann: ack, but they were released right?18:29
mnaseri'm curious what the different deployment approach is18:29
gmanndansmith: yeah18:29
gmanndansmith: noonedeadpunk: I think EOL/remove zed and mark it deprecated after stable/wallaby? or may be after tag 18.0.018:30
noonedeadpunkmnaser: I've heard from folks they've invested time/resources into deploying openstack in k8s, but that info is not 100%18:30
mnaserah, i've never heard of anyone thinking of doing that before 👀18:31
dansmithgmann: I'm not sure what you mean18:31
dansmithgmann: you mean mark everything post-W as EOL?18:31
dansmithdeleting releases doesn't sound very good to me18:31
gmanndansmith: not deleting. marking deprecated after 'tag 18.0.0' which is last release. deprecating after stable/wallaby does not seems right 18:32
gmannI think this is first case of deprecation for independent release model so taking last release tag seems right way18:33
dansmithgmann: okay, still confused18:33
gmann:)18:33
noonedeadpunkI'm not sure how you can mark everything after some tag as deprecated?18:34
gmannyou mean how to support last tag release as we cannot backport?18:34
noonedeadpunkAs I guess you can deprecate only current HEAD, as you need to drop content from it first18:34
gmannohk, so i mean to record in governance about what is last supported for this deprecation18:35
gmannhere https://github.com/openstack/governance/blob/master/reference/projects.yaml#L80918:35
noonedeadpunkactually you also mae a good point about tag that can't be supported anymore...18:35
gmannso we know when to retire the project 18:35
noonedeadpunkok, yeah, gotcha18:36
gmannit is confusing by saying stable/wallaby is last supported but hey we do have newer release than stable/wallaby :)18:36
dansmithwell, that would be okay if nothing past wallaby was released,18:37
noonedeadpunkOnce we sort this out - we should likely do lessons learned from that to cover gaps that allowed this to happen...18:37
dansmithbut since there were, I'm not sure how you can go back on previous promises/releases18:37
gmannnoonedeadpunk: yeah, this is something needs to cover when projects moving 'from branched to independent release' what is needed in governance18:38
noonedeadpunkmake another tag on the same SHA as already released one saying EOL-18.0.0 ?18:38
gmanndansmith: yeah18:39
noonedeadpunkNot sure how that will help, but at least some indication that smth is wrong with this18:39
gmannbut if we see from 'support' perspective stable/wallaby is the only branch they can support. so may be we just ignore the releases after stable/wallaby ?18:40
noonedeadpunkI mean - we can ignore a lot of things. But we should make it also clear for operators and users that this project died way before they think it did18:40
gmannhumm18:41
gmanncan we move wallaby-em to 18.0.0 hash and that way we can say stable/wallaby last release as their last release and supported18:42
dansmithnoonedeadpunk: that may be the reality, but anyone who has packaged this needs a bit more than that I think18:42
dansmithgmann: wouldn't that involve a regression of, for example, the requirements.txt etc18:42
dansmith?18:42
noonedeadpunkI'm not sure it was ever packaged?18:42
gmanndansmith: right, it may. most probably it will18:43
dansmithnoonedeadpunk: you don't know18:43
dansmithlooks like there were at least some packages in ubuntu at some point18:44
dansmithagreed, not likely widespread, but if we do it now for this, some other project could expect the same18:45
noonedeadpunkwell. another option is likely to say our no on EOLing or emptying Zed branch allowing for rest of community to pick up it's maintenance if needed18:48
fungibasically treat those branches as if they're in extended maintenance state early?18:48
dansmithI definitely think we should ask for not emptying the branches18:48
noonedeadpunkYeah18:48
JayFWe don't have a lot of leverage here, even as technical leadership, unless we are going to provide technical support for keeping CI running and the like, yeah? I don't understand the value of leaving EM branches open if the community around them doesn't exist anymore18:49
fungifrom a governance perspective, you've got the current maintainers saying they're no longer able to continue development on the project, so from the tc's perspective you could i suppose hope someone else offers to take it over18:49
noonedeadpunk^ yes, exactly18:50
JayFI don't see a large queue of contributors itching to contribute to it18:50
dansmithnobody is going to take it over, but I don't see why that has anything to do with not *emptying* branches that already exist18:50
dansmithfor releases that are supposed to have timelines attached to them18:50
noonedeadpunkWell. We're already in situation, where they're going to maintain only N-1 release basically, but not N. And I hate this specific aspect of that18:51
JayFSupposed to really is the key word though; becuase we can't do anything to enforce that tripleo follow this timeline; if the contributors are gone the project is gone, right?18:51
gmannyeah, let's go that way. mark it deprecated, keep zed as it is (for already release versions or in case anyone want to un-deprecate and maintain it),  retire when stable/wallaby is EOL18:51
JayFI don't love it, but I think we might be better off just "ripping the band-aid off"18:51
dansmithnoonedeadpunk: right, the other weird thing is, maintenance of W creates regressions in the W+n release branches too18:52
noonedeadpunkIt's open development after all ;)18:52
gmannand later we can document the process to better handle such cases18:52
dansmithgmann: you mean deprecate X, Y, and Z correct?18:54
gmanndansmith: there is no X, Y right only Z and we can keep Z as it is with master README.rst mentioning about 'this project maintenance is not supported after wallaby'18:56
dansmithgmann: there are numbers between 14 and 17 in the releases thing right?18:57
gmannyeah but we cannot do anything about those, keeping zed open is the way if someone want to pick/maintain it after 18.0.0 (last release)18:57
gmannor from stable/wallaby18:58
dansmithyou mean the only open *branch* since wallaby is zed?18:58
gmannyes18:58
gmanndansmith: https://github.com/openstack/tripleo-common/branches18:58
dansmithright, so that makes sense, but I guess it seems like the "lifetime" of the releases between W and Z have now changed18:58
dansmithprobably doesn't matter that much I guess, but...18:59
gmannyeah, we cannot do anything on that now18:59
gmannits very weird situation  18:59
noonedeadpunkYeah, I'd say that keeping Zed open is likely by far best choice even if we know it will be broken and quite unmaintained. As I assume there might be ppl who used 18.0.0 and will appear one day to step in to maintain these19:01
gmannyeah19:02
noonedeadpunkAt least we shouldn't shut the door until Zed will move to EM, where it can be instantly EOLed19:02
dansmithI guess I'm still confused on the expected support envelope of the intermediate releases on an independent project19:02
dansmithis it not expected to be 18ish months in the same way as our coordinated releases?19:02
noonedeadpunkI'd say confusing thing is that they have independent release cycle but releases named same way as coordinated ones19:03
gmanndansmith: if it is completely independent model project then there is no 'deprecation' it is directly 'retirement' as there is no stable branches there to keep support for19:03
gmannopenstack-health project is example of that which we recently retired 19:03
dansmithnoonedeadpunk: yeah19:04
noonedeadpunkSo it's some mix of both worlds that exists by moving to independent and then somehow branching zed...19:04
dansmithgmann: I mean for the releases, not the project19:04
gmann'deprecation' is only for those who do support old release via stable branhces19:04
dansmithgmann: okay I guess i don't get it.. most consumers care about releases, not so much the branches that back the releases,19:04
gmannyeah, here situation is weird that it is mix of coordinated and independent release 19:04
dansmithso I think they expect support for the 17.x lineage not "does the 17.x have a stable/foo branch?"19:05
noonedeadpunkdansmith: I don't think they can support 17.x without stable/foo19:05
gmannyeah how they will support tag only releases19:06
dansmithnoonedeadpunk: I'm talking about what consumers expect19:06
gmannmeans they cannot backport right19:06
noonedeadpunkah, yes19:06
gmannohk19:06
dansmithi.e. people that deploy a release (from a tarball or package) do not first go look to see if there's a stable/foo branch backing it and then extrapolate the support envelope from that19:06
gmannI will say support for them is to fix and release new tag19:06
gmannsame as any PyPi release?19:07
dansmithgmann: to be clear, I'm not talking about tripleo specifically here, I'm talking about what consumers of "openstack project foo" have in the way of expectations for *any* release they get from "us"19:07
JayFThat cat is already out of the bag19:07
JayFwe explicitly have a list of OpenStack projects that opt-in to stable processes19:08
JayFthere is no clear way for an operator to know how long a given openstack deliverable will be supported without navigating documentation19:08
dansmiththat's my point, I'm trying to say that I'm not sure what the expectation is for tripleo 17.x support19:09
noonedeadpunkdansmith: well. quite a few consumers prefer to track branch and not tags.19:09
dansmith(in a world where anyone other than redhat customers are running tripleo)19:09
gmanndansmith: may be I am also confused with 'support':) for any released software what is mean of 'support' for that release - backport things right?19:09
noonedeadpunkso it's not really black and white19:09
gmannif they are thinking of support, it needs to be with backport branches19:10
noonedeadpunkgmann: not only backport, but fixing at least security issues19:10
gmannyeah19:10
dansmithgmann: yes, I'm saying the current way we have releases, someone may be expecting that grabbing a release from an independent project could *ever* get a bug or CVE fix, but that's probably a bad assumption from just looking at the releases page and seeing releases with semver-ish version numbers19:10
gmannfor example, fix bugx for me and do not give me all new things19:10
noonedeadpunkso if just 18.x contains vulnarabiltiy -- there should be the way to cover it19:11
dansmithnoonedeadpunk: right, and it sounds like there isn't currently because there's no branch for those 19:11
gmanndansmith: yes, from release page it is not clear19:11
gmannwe should mark explicitly 'not supported/no-bug-fix for this release and only new releases' for independent releases  19:12
dansmithgmann: yeah I guess I'm saying, it seems like we maybe need some indication of what to expect19:13
dansmithon a project that does coordinated releases and stable branches, it's pretty clear19:13
TheJuliadansmith: and creation of branches after the fact where applicable/needed has also been frowned upon/outright rejected where it came up and was needed, which is what helped spur bugfix branching to become a thing in ironic19:13
JayFOr even defer to project policy in some cases? I wonder how that kind of verbiage would interact with ironic bugfix releases19:13
gmanndansmith: yeah. that is clearly needed 19:13
dansmithbut for anyone not in that bucket, it probably isn't19:13
JayFwhich also live in a weird intermediate place19:13
dansmithTheJulia: ack, but how do you make sure you're applying bug fix 27 on top of bug fix 26? I mean, I understand the mechanics of it, I'm just saying, I can see it being confusing19:14
JayFit is confusing, and we fix it with documentation and folks who approve our stable stuff knowing bugfix branches exist19:15
TheJuliadansmith: and time consuming, and not *always* needed. It is situational unfortunately19:15
dansmithyeah19:15
JayFI think I cleaned up some of that confusion this cycle by retiring old ironic bugfix branches + changing policy to be more judicious about when we cut them19:15
TheJuliayes, cleaning things up unfortunately is an easy thing to let slip19:16
dansmithrelying on brute force and some stable maintainers to be sure you're not releasing a bugfix that regresses another because you had the order wrong in your head though is kinda unfortunate.. obviously if you're super good and never go outside near busses, then it works :)19:17
JayFI mean, honestly it goes back to the heart of the tripleo discussion19:17
gmannso here is no mention of 'maintained/supported' for independent releases. may be adding 'Status' as 'not maintained' or soemthing  https://releases.openstack.org/independent.html19:17
JayFa project is not code; it is the people who work on it19:18
JayFI'd love to have the problem of Ironic having so many contributors that bugfix branches got to be an issue19:18
TheJulia+++++++++ project == people19:18
JayFbut that's not the spot that we're at now; so I prefer to solve problems that do exist rather than ones that may :/19:18
noonedeadpunkIt's also people who consume it - let's not forget about thouse19:18
noonedeadpunkAnd consumers can convert to maintainers. I don't say they will, but they can19:18
dansmithnoonedeadpunk: yeah, thanks for saying that19:19
TheJulia++19:19
dansmithand the rest of openstack is on the hook for the consumers, IMHO, especially if we stop being responsible in our support policies for one project and that reflects on the others19:19
TheJuliasometimes it also takes such an announcement to get people to think "maybe I should"19:19
JayFTheJulia++19:19
JayFand honestly, if I were to stack rank concerns people have when I try to sell them on openstack "stable support" never even gets on the list :/ 19:20
noonedeadpunkyes, I agree we should try to avoid creating nasty precedent19:20
TheJuliawe can ask/advocate/beg for help, but until there is pain, sometimes people won't do anything when they see others involved19:20
JayFwe've seen that in PTL elections almost every cycle as of late19:20
JayFnobody wants to be a PTL until there is no PTL19:20
JayFsimilar thing here, I guess, I'm coming around to the more popular opinion of leaving zed open for tripleo19:20
* fungi doesn't want to be ptl ;)19:21
JayFbecause we are providing the "stick" of the carrot-and-stick to get people outta downstream and contributing back19:21
fungi(at least not again)19:21
gmannso on triplo discussion 19:21
gmannyeah keeping zed open19:21
TheJuliaSo... different discussion, perhaps for in 39 minutes, "what thoughts or patterns might help encourage people to want to serve in those roles?"19:21
dansmithJayF: that's funny, the biggest thing I hear is "we want to be on release X forever and keep getting backports"19:21
gmannanything we need to discuss with them or in ML thread other then let them proposed as deprecation and we will mark it as stble/wallaby  but keep zed opne ?19:22
JayFTheJulia: that's already on the agenda :)19:22
TheJuliaOh, somebody needs to take notes on that call. I'm limited bandwidth and will be dialed in to start it19:22
TheJulia\o/19:22
JayFdansmith: fair; I filtered those kinds of opinions out because that wish is ... hardly unique to openstack... and doesn't really acknowledge the reality of the environments changing around us (even if our software is 'stable')19:23
dansmithum, okay :)19:23
TheJuliaAnd the rate of evolution/change is forever moving faster19:23
JayFdansmith: I am extremely happy to have a deeper version of this conversation later; I'm really interested in it.19:24
JayFbut now, I need to get lunch before the meeting19:24
TheJulia++++ food19:24
noonedeadpunkdansmith: to be frank, as of today I don't feel huge difference of doing minor openstack upgrade comparing to the major one19:24
fungiyes, everyone wants to be on a stable version of teh software forever except get new features too, but somehow without it breaking anything they rely on19:24
TheJuliafungi: exactly :(19:24
* JayF ==fungi19:24
gmannI am adding it it PTG etherpad to discuss it further and give more thought19:24
noonedeadpunkIt's close to same thing I've heard quite a lot recently, like "I've tried openstack 10 years ago - it was hard and it's still hard, so nothing has changed"19:25
TheJulia"hi, I would like big giant new feature in your minor patchlevel release!" *sigh*19:25
fungibasically everyone wants fewer bugs, and they equate that to not making new releases19:25
fungithe releases aren't the problem though19:25
TheJuliaconsumption side problem?19:26
fungiit's a conflicting desire to want things to not change but wanting them to get better19:26
TheJuliaindeed19:26
fungiand you can't make the software better without changing it in some way. you can either have software which doesn't change and so never gets better, or software which gets better over time but necessarily changes to make that happen19:27
TheJuliaand in that, sometimes steps must be taken, decision points exist, understanding is sometimes needed to appropriately navigate that19:28
TheJuliawhich is challenging in an age where people yearn for simple/easy19:28
noonedeadpunkPeople always want simple solutions to their complex usecases...19:31
TheJuliayup19:31
TheJuliaThat is not just limited to use cases, but perceptions19:31
noonedeadpunkyes, totally19:32
TheJuliaand often, they are the best person to advocate for that change when they don't want to be19:32
TheJuliasince they are best suited to make the case and advocate/convince19:32
* TheJulia will happily discuss this on the call soon19:32
gmannadded 'support' things discussion in PTG etherpad, feel free to reword it as I might not have phrased it well (as still no coffee for me yet ) https://etherpad.opendev.org/p/tc-2023-2-ptg#L5019:42
noonedeadpunkshould we hold tripleo folks from cleaning out stable/zed until the next week at least?19:44
elodillesoh, i see here kind of a long discussion :)19:46
elodillesgmann: i did not remember that tripleo is not releasing after wallaby,19:47
elodillesgmann: though tripleo was always a 'special' project19:47
elodillesgmann: only maintained by RH folks, so basically almost a pure RH project19:47
elodillesgmann: and yes, they had some special handling, like they chose to close ussuri and victoria branches (EOL), while keeping train and wallaby in EM19:48
noonedeadpunkhm... what about moving it to X namespace then?19:48
elodillesgmann: since wallaby they changed release model to 'independent'19:49
noonedeadpunkelodilles: the problem they've branched stable/zed but don't want to maintain it anymore but keep wallaby19:49
gmannnoonedeadpunk: we should move it to x/ only if anyone interested to maintain there19:49
noonedeadpunkwell. they're interested to maintain wallaby :)19:49
gmannelodilles: ok. close ussuri and victoria branches (EOL), while keeping train is another different thing here19:50
elodillesgmann: which means they can release from the tip of the branch anytime, cut branches (like they did with stable/zed - i was surprised first, that this is possible, but we had examples in the past), but *NO* release is possible from those branches19:50
fungialso the tc enacted a policy that you can't "move" a project out of openstack. for awareness of downstream consumers it has to be retired in openstack and then forked to begin development elsewhere outside openstack19:50
fungibecause that's a clear signal to everyone that the project is no longer under the governance and policies of the openstack project19:51
noonedeadpunkmakes sense19:51
gmannnoonedeadpunk: It seems we did not clear agreement here even tc-members discussed here agree on keeping stable/zed open. let me add this in next week meeting agenda and ask tripleo on ML to hold the stable/zed things till we get official agreement in governance 19:51
noonedeadpunk++19:51
elodillesnoonedeadpunk: as soon as they cut a branch in independent release model, it's the team responsibility to maintain / delete those branches (just what happened with ironic's many 'bugfix/*' branches)19:52
noonedeadpunkelodilles: it's is akwardly confusing when independent project cuts a branch with coordinated name19:52
gmannelodilles: but branch with independent release model end up in weird situation  or at least many process about 'support', 'deprecation/retirement' are not covered for that cases19:53
noonedeadpunkso as a user you don't really know much about what release model some project at very least19:53
gmannnoonedeadpunk: yeah, that one19:53
dansmithexactly19:53
JayFhonestly saying "don't make indepedendent releases with the same name as the coordinated release" seems like it'd be reasonable policy19:53
noonedeadpunk++19:54
gmannany ways that is something we can discuss in PTG and find some better process on release as well as governance perspective 19:54
bauzasfrom what I see, there are other contributors but Red Hat who express interest in TripleO development on stable/zed https://www.stackalytics.io/?release=zed&module=tripleo-group19:54
gmannJayF: ++19:54
elodillesJayF: ++19:54
bauzasI'm not 100% sure they'll appreciate an empty commit on stable/zed19:54
noonedeadpunk99cloud contributed not that long ago19:55
elodilleswe did not have a rule for that, but did have examples in the past with stable/$series, so we allowed to cut stable/zed, but i agree, the best is to avoid this in the future19:55
gmannyeah19:55
noonedeadpunkshould be also possible to add a CI test for that19:55
bauzasnoonedeadpunk: all their commits were about docs, but their reviews seemed to me to express interest in the development19:56
noonedeadpunkwell, at least they to consume it19:57
TheJuliaI've started the call19:57
bauzasI don't know what 99cloud is using of tripleo if not none, but saying "only RH was paying attention to the TripleO development" seems to me a wrong assumption19:57
TheJuliaI suspect they were typo fixes19:58
gmanntc-members: call time  https://etherpad.opendev.org/p/2023-02-board-openstack-sync19:58
TheJuliasomeone will need to take a look, but I'm 95% sure they had their own product19:58
JayFIs the email sent to the list just straight up false then? Did tripleo have a meeting and vote on it? Or just a bunch of people in a backroom nodded at each other with red fedoras on19:59
JayFPerhaps asking for the paper trail on how that decision was made is wise from an open governance perspective19:59
bauzasTheJulia: which may be based on bits of TripleO, including THT probably given the reviews20:00
bauzasand yeah, the 4 commits were on docs, but they had like 90 reviews20:00
TheJuliapuppet modules it looks like20:01
TheJuliaor at least the tripleo-puppet stuffs20:01
bauzasyup, tripleo templates and modules20:01
TheJuliaand all look like drive-by +1's in my quick look20:01
bauzasyup, don't know if part of the +1s were gamification, but that's still a valid concern20:01
JayFI think the sentiment that it is mostly driven by redhat is accurate20:01
bauzasand I don't disagree20:02
JayFbut we should still do due dillegence to make sure the entire community was consulted in a public forum20:02
bauzasbut between "mostly" and "entirely" there are nuances20:02
bauzasand the puppet modules or the heat templates may be reused by other companies20:02
TheJuliaindeed20:03
TheJuliaTIL, we have an opendev.org matrix server20:11
fungithat's where zuul's channel is20:18
fungihttps://matrix.to/#/#zuul:opendev.org20:19
fungiwe also have a gerritbot which we can connect to matrix channels20:19
*** dasm|afk is now known as dasm|offline22:05
tonybgmann: In terms of process, how long will it take to move mistral from DPL -> PTL.  Can it be done in <7 days from today?  noting that as of now there isn't actually a change to so.23:13
tonybIf not I propose that we continue with the election with mistral as DPL and switch it afterwards. As that's overall less work anyway23:13
fungiprobably the only gotcha is if more than one person is interested in being ptl23:23
fungibut that doesn't seem likely23:24
fungiand even if it does crop up, the tc can decide that the way they're going to appoint the ptl is to have a special election for it23:24
tonybfungi: Yeah I tend to agree.  I feel like we need an explicit step in the proccess that's like "$date the TC and projects will not alter projects.yaml in a way that alters the configuration of an election, until after the election concludes"23:26
fungithat was the original intent with deciding when the governance repo would be tagged23:26
tonybfungi: Yeah I'm happy to do the special election also it only invalidates my "less work" satement23:27
tonybfungi: Yeah that's true.23:27
tonybfungi: (re tagging).  I'll update the setup-election tool to emit that tagging date so the election officials can communicate that to the TC23:28
fungigood ca23:29
fungill23:29
tonybFWIW: I've captured that at: https://etherpad.opendev.org/p/TC_PTL_Elections2023.2Bobcat#L9723:30

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!