Wednesday, 2022-10-05

fungii'm hoping to be online by 10:30z, though no idea what time the button-mashing is going to begin00:05
*** marios is now known as marios|ruck06:31
hberaudo/07:03
*** amoralej|off is now known as amoralej07:27
elodilles~o~08:18
elodilleshttps://status.python.org/ and zuul look operational08:23
elodillesbased on last years experience we should start the machinery @ 11:00 UTC (or slightly sooner)08:24
hberaud\o/08:46
opendevreviewElod Illes proposed openstack/releases master: Add missing release note links for Zed  https://review.opendev.org/c/openstack/releases/+/86035909:07
elodilleshberaud ttx : i would ask for a last minute reno-link patch review ^^^ o:) (freshly generated)09:09
hberaudack09:40
ttxhola09:45
elodilleshello o/09:46
elodillesthx hberaud 09:46
ttxlate start for the long day!09:46
ttxelodilles: patch looked good to me09:46
elodillesttx: ack, thx \o/09:47
elodillesthis should not conflict with the final RC patch, so +W'd it09:47
elodillesa bit more than 1 hour and we can start with the final RC patch merge09:48
opendevreviewMerged openstack/releases master: Add missing release note links for Zed  https://review.opendev.org/c/openstack/releases/+/86035909:55
fungihow's thinks so far?10:33
ttxI'll soon break for lunch, but feel free to push the big button10:33
hberaudsame thing here, I've to run a couple of short dad taxi and break lunch.10:37
elodillesfungi: https://status.python.org/ and zuul look operational, i hope you see the same from infra side :)10:45
fungiyes, seems everything's good10:46
elodillesttx hberaud : Bon Appétit! :)10:46
elodillesfungi: cool \o/10:46
fungihttps://grafana.opendev.org/d/21a6e53ea4/zuul-status10:49
fungishows things are definitely running, but we've got available capacity10:50
elodillesyepp, i saw the same with zuul, that there were ~60 patches at the check queue and ~20 in gate, which is a bit more than during the usual mornings i would say. but otherwise things looked OK10:55
fungiit's 11:00 utc11:00
elodilles+W'd >>> https://review.opendev.org/c/openstack/releases/+/86001111:00
elodillesjobs started at the gate queue11:01
*** amoralej is now known as amoralej|lunch11:03
fungiand all builds have nodes assigned now11:05
fungilooks like the validate job is expected to take 15 minutes to complete11:06
elodillesi think it will take longer than that as now there are quite many deliverables in the patch11:07
elodillesthe estimate is based on the usual runs, if i'm not mistaken11:07
fungiyeah, likely11:07
fungiit's still on cloudkitty11:15
elodillesi've just checked: the yoga release patch merge took 32 minuntes11:19
fungifreezer11:20
elodillesgetting warmer :-o11:20
elodilles(pun intended. sorry o:))11:20
fungiit is rather cold in my house this morning. i've got a fire going for the first time since winter11:21
fungi18c inside at the moment (12c outside now that the sun's started to come up)11:23
elodilleswe have nice sunny autumn weather here with around 18°C / 65°F, so i'm not complaining :]11:23
fungiit's damp and windy here on the water, 90% relative humidity and wind gusting to 30kph11:28
fungimakes it feel a lot colder to me11:28
elodillesyepp, that should feel cold i guess :S11:29
fungilooks like we're already to monasca11:29
elodillesit's closer to the end now then, but it seems it will take more than 32 mins this time11:30
elodillesplacement11:40
elodillesis that weather still due to Hurricane Ian?11:40
fungioh, no that blew through over the weekend, and then the storm front which pushed it inland moved through11:41
fungithat one's just about cleared out now. hopefully tomorrow i can catch back up on overdue yardwork11:41
elodillesi see. this time in autumn is the best to do some work outside the house. (if it's) not too cold, not too warm, just perfect11:46
fungisame here, as long as we can find a break between storms11:47
elodillessolum. 17 projects left11:48
fungii'm watching watcher11:53
elodillesi've just missed it watching.... now it's already zun11:55
elodillesnow we need to wait for the result :S and then comes the fun part: post jobs :-o11:55
fungiyup11:56
elodilles0 warnings / 0 errors... so far that looks good11:56
opendevreviewMerged openstack/releases master: Zed final releases for cycle-with-rc projects  https://review.opendev.org/c/openstack/releases/+/86001111:56
elodillesthere we have ^^^11:56
elodilles56 mins11:57
funginow it's in release-post11:57
elodillesit almost doubled since previous release (yoga)11:57
fungitag-releases build has started12:01
fungitags are being pushed12:05
fungihttps://pypi.org/project/adjutant-ui/12:08
fungiReleased: less than 20 seconds ago12:08
*** jpodivin_ is now known as jpodivin12:12
elodillesglance is released on pypi as well. though now the pypi releases don't really follow the alphabet12:13
fungithe semaphore for the releasenotes builds in the tag pipeline are probably going to run all day again12:20
fungilooks like we're averaging around 7 minutes per build12:20
elodilleshmmm. i see.12:20
elodillesat least it started: https://zuul.opendev.org/t/openstack/builds?job_name=publish-openstack-releasenotes-python312:20
elodillesaodh is finished so far12:20
fungiyeah, once we have a few we can look at the cadence between completion times to get a more accurate estimate, since that will also take node allocation time into account (which i don't think the duration value includes)12:22
fungitag-releases is done now12:23
elodilles~o~12:25
elodillesbtw maybe this patch and the parent can be approved (if we have reviewers): https://review.opendev.org/c/openstack/openstack-manuals/+/85967812:25
ttx\o/12:26
elodillesbut the one on top of it we should still keep open12:26
elodillesand merge around 14 UTC if i remember correctly (as it needs ~1 hr to refresh the pages)12:27
fungiwith a sample size of one span between releasenotes jobs so far (so not much statistical confidence), i'm estimating completion of those will be around 20:43 utc today12:29
elodilleshmmm, that's quite late then12:29
fungiwell, they ran for nearly that long on yoga as well12:30
ttxi think that's ok12:30
ttxThe important part is that we have all the tarballs up12:30
ttxand the Zed release page accurate12:30
ttxrelease-post just concluded12:30
elodilles\o/12:30
ttxhttps://releases.openstack.org/zed/12:30
fungiyeah, if anyone asks we can just explain that publication of final release notes is ongoing and will complete later in the day12:30
*** amoralej|lunch is now known as amoralej12:31
ttxdunno why the titles are duplicated on releases.o.o those days12:31
ttxprobably a template issue12:31
fungisphinx changed the way it deals with top-level titles, and openstackdocstheme had to undo a hack it had in place to add them12:32
fungithey used to be suppressed12:32
fungiit's possible the release notes site needs a similar adjustment12:32
ttxhmm ok we should replicate that on the customized copy releases.o.o uses12:32
fungier, releases site12:32
ttxLet's see if i can get that fixed real fast12:33
ttxdoes not look too hard12:34
elodillesmeanwhile i've started the missing-releases check (probably a bit early, but so far it looks good)12:34
fungittx: https://review.opendev.org/79889712:36
fungii think that was the docstheme fix12:36
ttxyep that's the one12:36
fungicool. took some hunting to track it down due to lack of obvious keywords in the commit title12:36
opendevreviewThierry Carrez proposed openstack/releases master: Fix title duplication  https://review.opendev.org/c/openstack/releases/+/86041112:38
ttxthat should quickfix it (hopefully it does not rely on other fixes)12:39
elodillesthanks ttx o/12:39
elodillesi'll check the result and +2 it if it looks good12:39
fungisecond span between release notes builds was only 0.5% (2 seconds) longer than the first, so somewhat more confident in my earlier prediction12:40
elodilles:]12:41
fungithe release pipeline has cleared12:42
elodilles\o/12:42
fungii guess it's time to check for missing artifacts?12:44
elodillesi'm already running the check12:44
fungiright on12:44
elodillesbut a double check does not hurt :]12:45
fungihttps://lists.openstack.org/pipermail/release-job-failures/2022-October/thread.html12:47
elodilles did not find python 3 wheel https://tarballs.openstack.org/swift/swift-2.30.0-py3-none-any.whl12:47
fungimonasca releases failed... expected?12:47
elodilleshmmm12:47
fungielodilles: swift-2.30.0-py2.py3-none-any.whl is there though, looks like swift is still producing "universal" wheels12:48
elodillesthe monasca failures are all related to docker image publish12:48
fungiswift-2.30.0-py2.py3-none-any.whl12:49
fungier, pasted the wrong buffer12:50
fungihttps://zuul.opendev.org/t/openstack/builds?job_name=publish-monasca-thresh-docker-image&project=openstack/monasca-thresh12:50
fungilooks like those jobs haven't worked for ages12:50
elodillesfungi: oh, i see :S though i guess that's a problem, becuase py2 support was dropped from swift. but have to double check as i might be wrong12:50
fungielodilles: yes, if they've dropped python 2.7 support then they should remove the universal=1 from their setup.cfg or whatever12:51
elodillesfungi: yes, those jobs are failing for some time now12:51
fungi[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.1.1:exec (package-execution) on project monasca-common: Result of /bin/sh -c cd /monasca-common && /monasca-common/run_maven.sh org.codehaus.plexus.classworlds.launcher.Launcher --quiet -B clean install execution is: '1'. -> [Help 1]12:52
elodillesdid not find python 3 wheel https://tarballs.openstack.org/monasca-events-api/monasca_events_api-6.0.0-py3-none-any.whl12:52
elodillesseems to be the same ^^^12:52
elodillespy2.py3 is there12:52
fungiyeah, same deal12:53
fungii guess we have a growing list of projects to pester about universal wheels12:53
elodillesdid not find python 3 wheel https://tarballs.openstack.org/heat-tempest-plugin/heat_tempest_plugin-1.6.0-py3-none-any.whl12:53
fungior maybe a shrinking list12:53
elodillesand this too ^^^12:53
fungipresumably we noticed all these last release too12:53
elodillesdid not find python 3 wheel https://tarballs.openstack.org/ansible-role-thales-hsm/ansible_role_thales_hsm-4.0.0-py3-none-any.whl12:54
elodillesand this too ^^^12:54
elodilles+ this:12:55
elodillesdid not find python 3 wheel https://tarballs.openstack.org/ansible-role-atos-hsm/ansible_role_atos_hsm-4.0.0-py3-none-any.whl12:55
fungirevisiting swift, it's not clear to me that they've dropped python 2.7 support yet. they still gate on a swift-tox-py27 job12:56
*** kopecmartin is now known as kopecmartin|sick12:56
fungiso that one's probably correct still12:56
elodillesyes, i see that. i was wrong. so py2 is still there for swift12:58
elodillesjust to have these on a collected list: https://paste.opendev.org/show/bkEt3AQypISDeV12ue1j/12:58
elodillesfungi: do we need to do anything regarding these? i guess we can ignore these12:59
fungielodilles: i think the links for their wheels on the release site may end up wrong? but not the end of the world13:01
elodilleson the other hand: these were all the problems i saw. no missing tarballs or anything else13:01
fungithough those projects should review whether they still support python 2.7 and remove the universal flag if not13:02
fungifor the sake of future releases13:02
fungiwith 6 release notes builds complete, a refined prediction puts completion at 19:48:08 utc13:03
fungiso about an hour sooner than my earlier guess13:03
elodillesa bit better then :)13:04
elodillesAFAIS we don't link the whls on releases.o.o, only the tarballs and signatures13:05
fungibut yeah, aside from updated release notes lagging again, i guess it all went smoothly13:05
fungino issues13:05
elodilles++13:05
fungizuul hardly broke a sweat13:05
elodillesnice \o/13:05
fungi"ho hum, 26 on-time releases of openstack"13:06
fungiand that wraps up the english alphabet. now on to the klingon alphabet13:07
hberaud:)13:07
fungior were we doing the sindarin alphabet next?13:07
elodillesboth should do :D13:09
hberaudsindarin is more topical13:09
hberaudlet's wait for the next reboot of star trek13:10
fungisince it looks like my assistance won't be needed to fix anything, and i've reached my geek reference quota for the morning, i'm going to take a quick coffee break13:10
hberaudfungi++13:10
Clark[m]For wheels you could have the check look for the universal flag in the repo and find the correct file. Then warn if any are still doing python213:10
fungiagreed, but hardly seems worthwhile for 5 remaining projects13:11
elodillesfungi: thanks for your help \o/ 13:12
fungimy pleasure, as always13:12
elodilleswe can continue with these i think: https://review.opendev.org/q/topic:www-zed-final13:12
elodillesdo we have anyone around with +W rights to openstack-manuals ?13:13
ttxfor some reason I have that13:14
fungii've also made https://review.opendev.org/856326 no longer wip, though there's no urgency on that13:14
ttxelodilles: shall I approve now?13:15
fungibut ideally we should merge 856326 before we start approving too many library release requests for the new cycle13:16
ttxApproving now since that needs time to be processed13:17
elodillesttx: AFAIK the pages refresh in ~1 hr, so i guess we should merge them13:17
elodillesttx: ack, thanks!13:17
fungidocs publication refreshes normally occur every 5 minutes, but can end up lagging if there are large updates of content for other volumes (i have a feeling the tarballs site publication may still be underway)13:19
Clark[m]Grafana tracks most recent release times13:19
opendevreviewMerged openstack/releases master: Fix title duplication  https://review.opendev.org/c/openstack/releases/+/86041113:20
Clark[m]https://grafana.opendev.org/d/9871b26303/afs?orgId=113:20
fungiyeah, looks like it's all caught up13:21
Clark[m]Ya you can see the spikes associated with the openstack release in the vos release timing graph but seems back to normal now 13:23
*** marios|ruck is now known as marios13:31
elodillesmeanwhile i've added the announcement mails to our review page: https://etherpad.opendev.org/p/relmgmt-weekly-emails13:33
elodilleshberaud ttx : please review it ^^^13:33
elodillesdo i remember correctly that we need a moderator to approve the mail sent to the announce list?13:33
ttxunless your address is whitelisted in advance, yes13:34
elodillesat least i subscibed to the list at Yoga release :)13:36
ttxelodilles: Fixed one thing, lgtm otherwise13:36
elodillesoh, thanks13:37
elodillesand of course i'll need to update the link after the announcement mail was sent13:37
ttxyep13:37
ttxWe'll likely have to pre-create the deliverable files sometimes after we are done zith Z release13:38
ttx(deliverables/antelope)13:38
elodillesttx: i'm planning to do that tomorrow, do you want to have it earlier?13:39
ttxno no13:39
elodilles(some projects are eager to release for antelope, as i remember o:))13:39
ttxTomorrow is good. close enough to our proclamation that we are open for Antelope releases13:39
ttxI just want to avoid conflict from trigger-happy releasers13:40
elodilles:]13:40
elodillesOK, looking at our task list, I see the following: now we need to wait for the pages to refresh, then the 'Mark Zed as released' patch needs to be merged: https://review.opendev.org/c/openstack/releases/+/860029/13:41
elodillesand then the announcement mails13:41
elodillesdo we need anything else than these?13:42
ttxnot that i can think of13:42
elodillesi think not, but let me know if i forgot something13:42
ttxThe title fix is up13:43
elodillesit has merged, but it haven't refreshed yet as I see13:44
elodillesoh, it did13:45
elodillesmy bad, you were right :)13:45
elodilleslooks nice! \o/13:45
elodillescan you please review this one? https://review.opendev.org/c/openstack/releases/+/860029/13:45
elodillesif that is OK, then we can merge it soon, if you agree13:46
*** Guest2330 is now known as diablo_rojo13:46
diablo_rojoThis where the party is? :D13:47
ttxI would wait until 9:30am CT (45 min from now)13:47
ttxThe party is on https://review.opendev.org/c/openstack/releases/+/860029/13:47
elodillesttx: ack13:47
ttxI did +2 it but we should wait for W+113:48
elodillesttx: ack13:48
elodillesthanks!13:48
elodillesit needs 10-15 mins to be picked up by zuul and get it merged, and some time after it to refresh things I guess. 14:30 UTC sounds right to trigger it.13:49
elodillesmaybe if we are so impatient then I can +W it some 5-10 mins earlier :D13:50
elodilles(i'll eat meanwhile a left-over cinnamon roll from yesterday's Cinnamon Roll Day :))13:52
hberaudboth LGTM13:58
elodilleshberaud: thanks o/13:59
fungii so hope cinnamon roll day is a national holiday14:02
elodilles:D14:02
elodillesunfortunately not :(14:02
fungiperhaps a reason to get involved in politics then14:03
diablo_rojoLets make it international14:05
elodilleswell, i think it's on the way to make it international :D14:07
elodillesprobably IKEA advertises it around the globe, so it just needs time... :-o14:08
elodilles(being Cinnamon Roll Day a swedish-finnish invention)14:10
funginorwegians will make their own competing pastry day14:10
fungisomething with fermented fish14:11
diablo_rojoNah, lefse14:11
elodilleshmm, did not know that :-o14:12
fungidiablo_rojo: is it improper to wrap lutefisk in lefse?14:15
diablo_rojoNo. 14:16
elodillesthough it looks similar like our pancake / crêpe. maybe the filling is mostly different :) looking at the salmon and onion filled one... :)14:16
diablo_rojoUsually lefse is eaten with butter and sugar. 14:16
elodillesthen it's definitely closer to our pancake / crêpe.14:17
diablo_rojoYes definitely. 14:18
clarkbkrumkake14:22
diablo_rojoOh yes. 14:22
diablo_rojoThats stuff is awesome. 14:23
diablo_rojoMore like a waffle though14:23
clarkbya I think I only had it the once when norwegian aunt brought it to christmas or something. But was good14:23
elodilleslooks better than a fermented fish :D (not that i don't like fermented fish :P)14:24
elodillesand i haven't tried lutefisk either :/14:25
elodillesanyway, my impatience is growing. I think i'll +W the release patch: https://review.opendev.org/c/openstack/releases/+/86002914:27
elodillesthanks everyone for the reviews!14:27
clarkbit is the suggested time14:27
fungisounds good14:27
fungii'm also on hand to approve the release announcement if necessary14:27
fungiat least if it's in the next ~20 minutes14:28
ttxelodilles: I think you can press the W+1 now to give it time to process before the email send14:28
elodillesttx: sure, +W'd.14:29
elodilles1.5 minutes ahead of the planned time :/14:29
ttxheh14:30
clarkband down to 48 projects needing releasenote jobs to run from about 6314:31
elodillesi haven't checked but i guess fungi's projection about finishing it around 19:30 UTC still holds14:32
fungiwish we could come up with a better mechanism to mitigate cross-branch rsync collisions which didn't end up serializing those builds across different projects14:32
fungii think the last time this came up we talked about the possibility of adding a project-scoped semaphore type in zuul14:35
clarkbzuul estimates 18 minutes until that change lands14:35
clarkbshould be right on time14:35
fungiso that we wouldn't simultaneously run the job for multiple branches of the same project, but could still run builds for multiple projects in parallel14:35
diablo_rojoSqueeee14:36
fungithe estimate for openstack-tox-validate may be inflated since the most recent run was for the full release change which goes a lot longer14:37
clarkbzuul looks at the last 10 runs for the estimate so there will be some impact but shouldn't dominate iirc14:37
clarkbwe'll know soon enough14:38
fungiyeah, i guess if it takes a median rather than a mean it should be fairly accurate14:38
fungias that will basically exclude outliers14:38
fungiit already finished though14:39
funginow estimate has updated to 1 minute remaining14:39
elodilles\o/14:40
opendevreviewMerged openstack/releases master: Mark Zed as released  https://review.opendev.org/c/openstack/releases/+/86002914:40
clarkbya I wonder if it weighs the more recent runs more heavily14:40
elodillesthere we go ^^^14:40
elodilles(it was like 50 mins)14:41
elodilles(the last run of validate job for the final release)14:41
fungiso now we're waiting for the publish-tox-docs-releases build in release-post followed by the volume update for the site, i guess?14:41
elodillesfungi: sounds like it14:42
clarkband then we send the email?14:43
elodillesfungi: i can send the announcement mail meanwhile, so you just need to approve it when you see fit14:43
fungiyeah, can do14:43
elodillesokay, i've sent it14:45
clarkbwe're waiting on a lower priority node request for that job14:46
clarkb(any idea why it is lower priority? I thought all the release stuff was meant to be higher priority)14:47
fungino idea. the release-post pipeline is supposed to be higher priority14:47
clarkbmaybe I've got my priority order backwards14:48
clarkbthat could also explain it14:48
clarkboh yup I think I do. 300 for check, 200 for gate, 100 for release-post14:48
clarkbnevermind I think that is already the higher priority14:48
fungiand it's started14:49
fungii think the volume update may fire sooner than the files get pushed14:49
fungiyeah, it just started14:50
clarkbafter all the release stuff is done one thing I'd like to do in opendev is bump our default ansible version to ansible 6. There are two reasons for this. The first is that zuul is looking at dropping ansible 5 support soon and if we update to 6 early we can run it through its paces and provide feedback if this doesn't work for some reason. The other is that ansible 5 broke14:51
clarkbtask pipelining but ansible 6 should correct this. In theory this makes some jobs a bit quicker.14:51
clarkbMentioning that here since we held off specifically due to the release. But once the release is done I think we'll move ahead on that. I did already test it with devstack and devstack seems happy14:51
elodillesclarkb: ack, sounds good, thanks for waiting with it until the release. I think this will be the right timing to do that!14:53
fungiokay, files have been pushed to afs so i've approved the release announcement out of moderation. it barely missed the 14:55 pulse but site content should appear updated when the 15:00 pulse runs in a few minutes14:56
clarkbit will be a race to see if the 14:55 publish run gets the docs out. One that I think it will lose. Seems likely to publish at 15:0014:56
clarkbfungi: ++14:56
fungiand with that, i have an appointment i need to rush off to14:56
fungiback in an hour or so!14:57
clarkbhttps://lists.openstack.org/pipermail/openstack-announce/2022-October/002061.html fwiw14:57
elodillesfungi: ack, thanks for your help!14:58
ttxclarkb: isn't the releases site already set?15:00
elodillesreleases.o.o is refreshed: https://releases.openstack.org/15:01
ttxAll done!15:01
clarkbttx: the publishing job writes to AFS but then we have to vos release the afs volume which ahppens via cron every 5 minutes15:01
clarkbttx: so there was a couple of minutes delay before you'd actually see the update on the website15:02
clarkbbut I agree it seems updated now just as expected15:02
elodillesi've sent the mail to openstack-discuss: https://lists.openstack.org/pipermail/openstack-discuss/2022-October/030735.html15:02
elodillesthanks everyone for your work! \o/15:04
ttxwoohoo! celebration!15:05
ttxTomorrow we can unblock all the things we held and start breaking $stuff again15:05
elodillesyeah, awesome! breaking stuff part is tempting :D15:07
clarkbI'll get the ansible 6 default up shortly so we can start on that :)15:08
elodilles:D15:08
elodilles(and fingers crossed :X)15:09
*** marios is now known as marios|out15:17
fungiokay, back15:54
*** amoralej is now known as amoralej|off16:39
*** adia is now known as abdi18:09
clarkbdown to 25 queued release note change jobs18:12
clarkbI'm guessing we'll be done by 2100 UTC?18:12
*** abdi is now known as adia18:15
fungiyeah, once we had 6 of them done i approximated completion time at 19:48:0818:17
*** adia is now known as abdi18:53
*** dviroel is now known as dviroel|afk21:17
opendevreviewJay Faulkner proposed openstack/releases master: EOL Queens for all Ironic-program projects  https://review.opendev.org/c/openstack/releases/+/86049921:20
opendevreviewJay Faulkner proposed openstack/releases master: EOL Rocky for all Ironic-program projects  https://review.opendev.org/c/openstack/releases/+/86050021:27
opendevreviewJay Faulkner proposed openstack/releases master: EOL Stein for all Ironic-program projects  https://review.opendev.org/c/openstack/releases/+/86050121:32
opendevreviewJay Faulkner proposed openstack/releases master: EOL Queens for all Ironic-program projects  https://review.opendev.org/c/openstack/releases/+/86049922:19

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!