Friday, 2022-09-23

opendevreviewZiyu Bai proposed openstack/releases master: Add Highlights Venus-Dashboard RC1 for Zed  https://review.opendev.org/c/openstack/releases/+/85901102:47
opendevreviewIury Gregory Melo Ferreira proposed openstack/releases master: Creating stable zed for ironic  https://review.opendev.org/c/openstack/releases/+/85828503:53
iurygregoryelodilles, hberaud ironic is ready ^03:53
opendevreviewTony Breeds proposed openstack/releases master: Creating stable zed for requirements  https://review.opendev.org/c/openstack/releases/+/85829405:27
*** amoralej|off is now known as amoralej06:34
priteauGood morning. I have a couple of release management questions.07:40
priteau1) Are you planning to approve the cloudkitty RC2 today? https://review.opendev.org/c/openstack/releases/+/85773307:40
priteau2) Is it expected that https://review.opendev.org/c/openstack/releases/+/858866 is missing some deliverables, like blazar for example?07:42
opendevreviewMerged openstack/releases master: Release networking-odl RC1 for Zed  https://review.opendev.org/c/openstack/releases/+/85714307:46
elodillespriteau: 1, yepp it's time to merge it08:18
elodillespriteau: 2, it's a bit tricky: as our script only adds release note links for *existing* release notes pages. for that a project needs to merge their 'reno-zed' patch: https://review.opendev.org/q/topic:reno-zed08:20
elodillespriteau: so for the 2nd question, it is possible that not all reno page link is added08:24
opendevreviewMerged openstack/releases master: Creating stable zed for ironic  https://review.opendev.org/c/openstack/releases/+/85828508:36
opendevreviewMerged openstack/releases master: Creating stable zed for ironic-python-agent  https://review.opendev.org/c/openstack/releases/+/85828108:37
opendevreviewMerged openstack/releases master: Release designate-tempest-plugin for Zed  https://review.opendev.org/c/openstack/releases/+/85840408:38
opendevreviewMerged openstack/releases master: Release bifrost for Zed  https://review.opendev.org/c/openstack/releases/+/85722308:38
opendevreviewMerged openstack/releases master: Proposes Nova RC2  https://review.opendev.org/c/openstack/releases/+/85850908:38
opendevreviewMerged openstack/releases master: Release sahara-image-elements RC1 for Zed  https://review.opendev.org/c/openstack/releases/+/85715608:38
opendevreviewMerged openstack/releases master: Release sahara-extra RC1 for Zed  https://review.opendev.org/c/openstack/releases/+/85715508:38
priteauelodilles: Indeed, we are missing the zed release note page. That is strange because we merged https://review.opendev.org/c/openstack/blazar/+/858028 on September 1608:40
priteauWhen I build locally the release notes are there.08:41
priteauOur release notes say they have been updated at 2019-09-20 12:03:51: https://docs.openstack.org/releasenotes/blazar/08:41
priteauBut we didn't merge anything that day08:42
priteauEh, sorry, that's 2019, not 202208:43
priteauI only looked at the month and day08:43
priteauelodilles: promote log is here: https://zuul.opendev.org/t/openstack/build/e11787211e644c38ad536962b2ded4f5/console08:46
priteau<<CHANGED>>>f+++++++++ zed.html08:46
priteauOK, I found the issue08:48
priteauhttps://zuul.opendev.org/t/openstack/build/1e58e9fdd9a74d9d956b04b632aa99d9 ran next and deleted zed.html08:48
priteau<<CHANGED>>*deleting   zed.html08:48
priteau<<CHANGED>>*deleting   _sources/zed.rst.txt08:48
priteauSeen at https://zuul.opendev.org/t/openstack/build/1e58e9fdd9a74d9d956b04b632aa99d9/console08:48
priteauIs this expected?08:49
elodillesi see, it's in 'upload-afs-roots: Synchronize files to AFS' task08:52
elodillesto tell you the truth, i don't know08:53
priteauI am pushing a dummy change to master to force refresh08:53
elodillesfirst of all i thought publish-openstack-releasenotes-python3 job publishes the renos08:53
elodilleslike this one: https://zuul.opendev.org/t/openstack/build/535356f8b23a4917a39b080357e4ce4608:54
priteauIt's only used when tagging? https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/zuul.d/project-templates.yaml#L304-L33008:56
priteauBut every commit changing releasenotes can promote08:58
elodillestrue, you are right09:00
elodillespriteau: it worked: https://docs.openstack.org/releasenotes/blazar/zed.html10:18
priteauYes. Do you prefer to update your release reno patch or do a follow-up?10:24
priteauI think other projects are missing too10:24
priteauSorry, not your patch, hberaud's10:25
elodilleswell, since every PTL and release liaison are added to the patch, it probably would spam them every update with mails, so the best is to not update the patch one-by-one, rather when multiple new release notes page appears :) i guess there will be an update for this patch, and later on there will be separate patches, i'm sure10:39
hberaudyeah lets update the missing releases notes with a follow up patch11:35
elodilleshberaud: ++11:47
opendevreviewRajat Dhasmana proposed openstack/releases master: Release cinder-tempest-plugin for Zed  https://review.opendev.org/c/openstack/releases/+/85840111:59
*** amoralej is now known as amoralej|lunch12:30
*** amoralej|lunch is now known as amoralej13:18
elodillesreminder: we'll have a meeting in ~15 minutes13:44
elodilles#startmeeting releaseteam14:00
opendevmeetMeeting started Fri Sep 23 14:00:56 2022 UTC and is due to finish in 60 minutes.  The chair is elodilles. Information about MeetBot at http://wiki.debian.org/MeetBot.14:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
opendevmeetThe meeting name has been set to 'releaseteam'14:00
hberaudo/14:00
armstrongo/14:01
elodilleshello all o/14:01
ttxo/14:01
elodilles#link https://etherpad.opendev.org/p/zed-relmgt-tracking14:01
elodilleswe are at L40814:01
elodillesalmost at the bottom :-o14:02
elodilleslet's start with the topics14:02
elodilles#topic Review task completion14:02
elodilles1st task was:14:03
elodilles#info Process any standing RC1 deadline exceptions.14:03
elodilleshttps://review.opendev.org/q/topic:zed-rc1-deadline+is:open14:03
elodillesthese were mostly the ones that had broken gates14:04
elodillesduring the week we agreed that we should continue with a 'formal' release from this14:04
elodilles* these14:05
elodillesI'll merge them after the meeting if no objection14:05
ttxThe barbican one whould wait14:06
elodillesexcept barbican, as it has still -1 on it14:06
elodillesyepp14:06
ttxthat would be the last standing one, unless dmendiza[m] has an update14:06
elodillesyepp14:06
ttxWe are super late for this one so if we don;t hear on Monday I'd consider bypassing it14:07
opendevreviewMerged openstack/releases master: Creating stable zed for swift  https://review.opendev.org/c/openstack/releases/+/85830114:07
elodillesttx: ack, i'll reach out to the team and inform them about it14:08
elodillesso that's about this task14:08
elodilles2nd task was:14:08
elodilles#info Gracefully release tempest plugins with latest changes available. (elod)14:08
elodilleshttps://review.opendev.org/q/topic:zed-tp-latest14:08
elodilleswe have two -1s14:09
elodilles~half were accepted and merged14:09
elodillesi'll merge the ones without response after the meeting14:10
ttx+114:10
elodilles3rd task14:10
elodilles#info On Monday, generate stable branches for all cycle deliverables that are still missing one. (hberaud)14:11
elodilleshttps://review.opendev.org/q/topic:zed-missing-stable-branches (some deliverables are listed in deliverables that are still missing stable branches, however, those have also RC1 pending patches, so I ignored them)14:11
hberaudI think we can now ignore my comment14:11
elodilleshberaud: ack14:11
opendevreviewMerged openstack/releases master: Creating stable zed for python-openstackclient  https://review.opendev.org/c/openstack/releases/+/85829114:11
elodillesbut then I'll propose the stable-branch cutting patches as well14:11
elodillesi guess we'll need some14:12
hberaudack14:12
elodilles(the rest of the sub-points were only instructions for this task)14:12
elodillesinfo After all the projects enabled in devstack by default have been branched, we can engage with the QA, I18n and Requirements PTLs to finalize the stable branch setup. (elod)14:13
elodillesi haven't processed this one as we still have some unbranched projects14:13
elodilleswhich will be branched after the above task is finished14:14
elodillesso i copy this task to next week as well14:14
elodillesotherwise, should be OK14:15
elodillesnext task:14:15
elodilles#info Ensure that all projects that are publishing release notes have the notes link included in their deliverable file. (hberaud)14:15
elodillesthe patch was proposed:14:15
elodilleshttps://review.opendev.org/c/openstack/releases/+/85886614:15
elodillesthis is usually not complete for some time after RC114:16
elodillesas this needs all the reno-$series patches to get merged14:16
elodillesi see some -1s on the patch already from PTLs14:17
elodillesshould we wait for an update for the patch?14:17
elodillesor merge this today and propose a follow up later on?14:18
ttxchecking what the -1s are for14:18
elodillesmissing reno links basically14:18
ttxah so we should regenerate the patch right14:19
elodillesyes, maybe that's the easiest14:19
hberaudwe could add the missing links through a follow up 14:19
elodillesand i'm sure we can re-iterate this task some time later (as i did in last cycle)14:19
hberaudas we did with prvious series14:19
elodilleswhat about regenerating the links as a 2nd PS for this one, and we merge it ASAP?14:20
hberaudas you want14:21
hberaudboth wfm 14:21
elodillesand later on creating a follow-up14:21
elodillesif needed14:21
hberaudok I'll update the patch14:21
elodilles(just to not merge with lots of -1 on it ;))14:21
elodilleshberaud: sorry, and thanks! o:)14:21
hberaudnp14:21
elodillesthe last task is again, not a task, but a note: Let cycle-with-rc projects iterate on RCs as needed. The final release candidate for each project needs to be prepared at least one week before the final release date.14:22
elodillesso this means the final RC iterations (if needed) needs to be proposed next week14:24
elodillesok, let's move to the next topic:14:24
elodilles#topic Assign R-1 tasks14:25
elodillesplease add your name if you want to work with any of the tasks14:25
elodillesOk, as I see, all taken14:27
elodillesthanks :)14:27
* ttx looks up what he signed up for14:27
elodilles:]14:27
elodilles#topic Review countdown email contents14:28
ttxlooks like the deadline is Friday, so "day before deadline" is Thursday14:28
elodilleshttps://etherpad.opendev.org/p/relmgmt-weekly-emails14:28
elodillesplease review ^^^14:28
elodillesttx: yepp14:28
ttxah no the deadline is Sep 29 on the email14:29
ttxso I'll propose last-minute ones on the 28th14:29
elodillesyes, just wanted to correct,14:29
elodillesthat we discussed this some time ago that it should be 29th September O.o14:30
ttxack14:30
ttxemail lgtm14:31
opendevreviewHervĂ© Beraud proposed openstack/releases master: Add release note links for zed  https://review.opendev.org/c/openstack/releases/+/85886614:31
hberaudrelease notes links updated ^14:31
elodillesttx hberaud : ack, thanks!14:31
elodilleswill send + will review :)14:32
hberaudmail LGTM14:32
elodilles++14:32
elodilles#topic Open Discussion14:32
elodillesanything to mention here?14:32
ttxdo we think we captured all the oslo.db issues?14:33
ttxI saw a few late mentions pass by while I was traveling14:33
elodillesgood question14:34
elodillesat least we know about some14:34
elodilleswatcher and ec2-api were mentioned in the last mails: https://lists.openstack.org/pipermail/openstack-discuss/2022-September/030493.html14:35
ttxwe might want to chase a fix -- although I'll admit I'll have trouble to have cycles to do that next week14:36
elodillesfor watcher i see a patch: https://review.opendev.org/c/openstack/watcher/+/85803214:36
elodillesbut it seems zuul ignores that patch :-o14:36
ttxhmm14:37
elodillesah, broken gate: https://review.opendev.org/c/openstack/watcher/+/85739314:37
elodillesstrange that on master it seems fine14:38
elodillesand its already branched14:39
ttxand the failures look functional14:39
elodillesyes14:40
ttxI guess it could all be linked to DB access failures14:40
elodillesec2-api, the same, here is the fix: https://review.opendev.org/c/openstack/ec2-api/+/85788014:40
elodillesbut the gate is broken14:40
ttxmore data for our "abandoned" report14:41
ttxAt least ec2api fails in a clearer manner14:42
elodillesexactly..14:42
ttxTypeError: missing a required argument: 'public_key'14:42
ttxFile "/home/zuul/src/opendev.org/openstack/ec2-api/ec2api/api/key_pair.py", line 88, in create_key_pair14:43
ttxI'll have a quick look in meeting in case I can fix it quick14:43
ttx17 min14:43
iurygregoryhberaud, https://review.opendev.org/c/openstack/releases/+/858866 is missing ironic-inspector (or at least I couldn't find)14:44
elodillesttx: there is a patch that tried already to fix the key issue i think: https://review.opendev.org/c/openstack/ec2-api/+/85814314:44
ttxstephenfin to the rescue again14:45
elodillesanyway, i've added it on our tasks list for R-114:47
elodillesi'll try to follow up on them, but any help is appreciated14:47
stephenfinThat needs tests. The issue is that nova no longer generates keypairs for you so we need to do it ourselves, client-side. I've yet to verify that the SSH keys I'm generating there match the format of what nova was generating (since nova was using paramiko but we're using the lower-level cryptography lib)14:47
elodillesi really hope these are all that is related to oslo.db 12.1.0 for now :)14:48
stephenfinAlso, some missing imports, clearly (I was testing in the terminal and copy/pasting)14:48
ttxyeah this is non-trivial really14:48
hberaudiurygregory: indeed, I just regenerated it with our script, so, something else is at work, which prevent the generation of this link.14:48
stephenfinelodilles: Sorry oslo.db 12.1.0 has been such a pain. It'll be less painful that SQLAlchemy 2.0 landing on with no warning though. At least not we know where all the tech debt is :)14:49
stephenfins/not/now/14:49
hberaudiurygregory: https://docs.openstack.org/releasenotes/ironic-inspector/zed.html => 40414:49
ttxif all the codebases were functional it would not have been that much of an issue14:49
iurygregory=O14:49
iurygregorylet me check things in inspector 14:49
ttxits just that it woke up areas of the code that were happily sleeping14:49
elodillesstephenfin: fingers crossed :) and thanks for all the work you done to fix all the found issues!!!14:49
hberaudI think its why the links is not added14:50
hberaud/link/14:50
fungiiurygregory: hberaud: someone brought one up yesterday where the release notes promote job hit a failure during rsync, probably intersected with a change for another branch trying to lock the same tree for writing14:51
hberaudI see14:51
fungiin that case we simply reenqueued the change into the promote pipeline14:51
iurygregoryfungi, ohhh14:51
elodillesand today we were informed with the same issue14:51
fungiif someone can put together a list of which builds failed, i'm happy to do the reenqueuing (but be mindful that doing so may roll back documentation if a later change published newer documentation and didn't update the release notes)14:52
elodillesand i saw that the reno-zed patch was merged later than the reno updating job ran14:52
iurygregoryhttps://docs.openstack.org/releasenotes/ironic-inspector/14:52
iurygregoryyeah I don't see Zed .-.14:53
elodillesso after triggering the promote job with a patch, it got updated14:53
iurygregoryhttps://opendev.org/openstack/ironic-inspector/commit/64c881b20fdf694a719e0c7793e72f0345679623 so this one failed to promote I think14:53
fungiyeah, the options are to either reenqueue the change into promote if there wasn't any newer change which did docs builds for that branch, or merge another change to the branch which triggers a release notes build14:55
iurygregoryI don't think we have any changes we can merge in inspector atm, let me check14:56
iurygregoryok, there is one, let's see :D14:56
fungithis was the promote buildset for the commit you linked: https://zuul.opendev.org/t/openstack/buildset/3cf3c8863f0e45ba9370609ac2c4c2b514:57
fungiand it ran promote-openstack-releasenotes14:57
fungisuccessfully14:57
iurygregoryO.o 14:57
fungihttps://zuul.opendev.org/t/openstack/build/edd6954962d646a39836faeddb3ade6f/console#1/0/28/localhost14:58
fungi<<CHANGED>>>f+++++++++ zed.html14:58
iurygregoryenr .-. (no idea why it doesn't show...)14:58
fungithere are a couple of possibilities i can look into. either another build later un-published that file, or our afs vos release cron is failing to publish that update to the read-only replicas14:59
ttxposted a new patch fixing the imports for ec2api -- no idea if it actually works though14:59
ttxhttps://review.opendev.org/c/openstack/ec2-api/+/85814314:59
iurygregoryI'm merging https://review.opendev.org/c/openstack/ironic-inspector/+/85901615:00
iurygregorylet's see if with the new build it will show-up15:00
elodillesmeanwhile, we've run out of meeting time, so let's close the meeting, but please continue the debugging if needed o:)15:01
elodillesthanks everyone for joining o/15:01
fungi/afs/.openstack.org/docs/releasenotes/ironic-inspector/ (the writeable volume) does not include a zed.html, so i think it got unpublished15:01
elodilles#endmeeting15:01
opendevmeetMeeting ended Fri Sep 23 15:01:29 2022 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:01
opendevmeetMinutes:        https://meetings.opendev.org/meetings/releaseteam/2022/releaseteam.2022-09-23-14.00.html15:01
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/releaseteam/2022/releaseteam.2022-09-23-14.00.txt15:01
opendevmeetLog:            https://meetings.opendev.org/meetings/releaseteam/2022/releaseteam.2022-09-23-14.00.log.html15:01
hberaudelodilles: thx15:01
elodillesthanks, too!15:01
armstrongThanks elodilles15:01
elodilles++15:01
hberaudI've to disappear early today, see you MOnday15:01
elodilleshberaud: see you!! o/15:02
elodillesfungi: in earlier case today we saw that 'zed.html' was deleted during the rsync, but i don't see that in this case15:02
fungiiurygregory: https://zuul.opendev.org/t/openstack/build/7c3a7bedb2c3489e842dab0094c369c6/console#1/0/28/localhost15:03
elodillesso it's really weird15:03
elodillesoh, so here, too!15:03
fungithat build ran later, and removed the file because its change's tarball didn't include it15:03
fungi<<CHANGED>>*deleting   zed.html15:03
iurygregoryWOOT?!15:04
iurygregorythis would be expected?15:04
fungiwell, the one which created the zed.html was a change for the master branch, while the one which was missing zed.html and ran later was for the stable/zed branch15:04
fungithe one missing zed.html was https://review.opendev.org/85887015:05
fungi858871 which had a zed.html was enqueued into the gate at 09:22, merged at 09:38 and its promote builds completed at 09:39... 858870 which didn't have a zed.html was enqueued in the gate at 09:21, merged at 09:38, and completed its promote at 09:4015:09
fungiso basically two changes approved at the same time for different branches. if we look into the timestamps inside the logs for their gate and promote builds i expect what we'll find is that 858870 built its release notes before 858871 but then the release notes for 858871 were published before 85887015:12
elodillesso as soon as another patch gets merged, the proper release notes will be present15:16
elodillesif i understand correctly15:17
elodillesmeanwhile, i fast-approved the generated reno-link patch, we'll run the script to generate all the missing reno links later on and propose a follow-up patch some time next week15:20
fungielodilles: sounds probable, yes. basically they built in one order but got published in reverse sequence, i think because we don't have strict ordering control between different branches in promote. clarkb does that match your understanding of supercedent pipelines?15:30
clarkbI thought we did15:31
clarkbif you have change A and B from different branches for the same project in a supercedent pipeline I thought they would still run oen at at time15:32
clarkbthe order is determined by the order events are received15:33
clarkbbut https://zuul-ci.org/docs/zuul/latest/config/pipeline.html#value-pipeline.manager.supercedent says they are grouped by project,ref and maybe that means branches in which case I'd be wrong15:33
opendevreviewMerged openstack/releases master: Add release note links for zed  https://review.opendev.org/c/openstack/releases/+/85886615:36
clarkbEven if we could solve this with the pipeline config what prevents the next zed patch from landing and overwriting things again?15:37
*** amoralej is now known as amoralej|off15:37
clarkbSeems like fixing this from a publication standpoint is probably preferable to avoid that problem?15:38
*** marios is now known as marios|out15:46
opendevreviewMerged openstack/releases master: Release monasca-events-api RC1 for Zed  https://review.opendev.org/c/openstack/releases/+/85713115:51
opendevreviewMerged openstack/releases master: Release adjutant-ui RC1 for Zed  https://review.opendev.org/c/openstack/releases/+/85704815:51
opendevreviewMerged openstack/releases master: Release adjutant RC1 for Zed  https://review.opendev.org/c/openstack/releases/+/85704915:54
opendevreviewMerged openstack/releases master: Release mistral-dashboard RC1 for Zed  https://review.opendev.org/c/openstack/releases/+/85712615:54
opendevreviewMerged openstack/releases master: Release magnum-tempest-plugin for Zed  https://review.opendev.org/c/openstack/releases/+/85841415:54
opendevreviewMerged openstack/releases master: Release cinder-tempest-plugin for Zed  https://review.opendev.org/c/openstack/releases/+/85840115:59
opendevreviewMerged openstack/releases master: Release zaqar-tempest-plugin for Zed  https://review.opendev.org/c/openstack/releases/+/85843115:59
opendevreviewMerged openstack/releases master: Release kuryr-tempest-plugin for Zed  https://review.opendev.org/c/openstack/releases/+/85841316:05
opendevreviewMerged openstack/releases master: Release barbican-tempest-plugin for Zed  https://review.opendev.org/c/openstack/releases/+/85839916:06
opendevreviewMerged openstack/releases master: Release heat-tempest-plugin for Zed  https://review.opendev.org/c/openstack/releases/+/85840916:06
opendevreviewMerged openstack/releases master: Release octavia-tempest-plugin for Zed  https://review.opendev.org/c/openstack/releases/+/85842116:06
opendevreviewMerged openstack/releases master: Release senlin-tempest-plugin for Zed  https://review.opendev.org/c/openstack/releases/+/85842416:06
opendevreviewMerged openstack/releases master: Release solum-tempest-plugin for Zed  https://review.opendev.org/c/openstack/releases/+/85842516:11
opendevreviewMerged openstack/releases master: Release monasca-tempest-plugin for Zed  https://review.opendev.org/c/openstack/releases/+/85841716:14
opendevreviewMerged openstack/releases master: Release murano-tempest-plugin for Zed  https://review.opendev.org/c/openstack/releases/+/85841816:14
opendevreviewMerged openstack/releases master: Release mistral-tempest-plugin for Zed  https://review.opendev.org/c/openstack/releases/+/85841616:14
opendevreviewMerged openstack/releases master: Release manila-tempest-plugin for Zed  https://review.opendev.org/c/openstack/releases/+/85841516:19
opendevreviewMerged openstack/releases master: Release ec2api-tempest-plugin for Zed  https://review.opendev.org/c/openstack/releases/+/85840516:19
opendevreviewMerged openstack/releases master: Release freezer-tempest-plugin for Zed  https://review.opendev.org/c/openstack/releases/+/85840716:19
opendevreviewMerged openstack/releases master: Release cloudkitty-tempest-plugin for Zed  https://review.opendev.org/c/openstack/releases/+/85840216:19
opendevreviewMerged openstack/releases master: Creating stable zed for mistral-extra  https://review.opendev.org/c/openstack/releases/+/85828616:20
opendevreviewMerged openstack/releases master: Creating stable zed for python-mistralclient  https://review.opendev.org/c/openstack/releases/+/85829016:23
opendevreviewMerged openstack/releases master: Creating stable zed for mistral-lib  https://review.opendev.org/c/openstack/releases/+/85828716:25
opendevreviewMerged openstack/releases master: Release networking-generic-switch for Zed  https://review.opendev.org/c/openstack/releases/+/85722416:32

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!