Wednesday, 2021-04-14

*** tkajinam has quit IRC00:55
*** tkajinam has joined #openstack-release00:56
*** evrardjp has quit IRC02:33
*** evrardjp has joined #openstack-release02:33
*** ykarel has joined #openstack-release04:14
*** vishalmanchanda has joined #openstack-release04:45
*** whoami-rajat_ has joined #openstack-release05:20
*** tinwood has quit IRC06:22
*** tinwood has joined #openstack-release06:25
hberaudgmann: Wallaby release press release is scheduled to go out at 10am CT on Wednesday. That's 15utc. So the plan is to have the release done by then.06:30
*** eolivare has joined #openstack-release06:32
*** knikolla has quit IRC06:34
*** rpioso has quit IRC06:34
*** sboyron has joined #openstack-release06:35
*** knikolla has joined #openstack-release06:36
*** rpioso has joined #openstack-release06:37
hberaudMorning everyone06:48
hberaudWell, the last mile to run is there06:50
hberaudI didn't noticed recent release job failure06:51
hberaudI didn't noticed recent email about RFE|release exception06:51
hberaudNobody complained on https://review.opendev.org/c/openstack/releases/+/78575406:51
hberaudSo, I think that we can let's go!06:52
hberaudttx, elod: ^06:53
hberaudAlso FYI https://review.opendev.org/c/openstack/releases/+/78606907:13
hberaudThis patch is about devstack-plugin-ceph ^ they propose to create the deliverable for W, V, U, T. Since this patch is only for branching purpose, and since devstack plugins are tagless, I think that we can merge it after the final Wallaby, any objections?07:16
hberaudI mean, I don't think that we have to wait for this patch to finalize wallaby07:17
*** xinranwang has quit IRC07:20
ykarelhberaud, can u please review https://review.opendev.org/c/openstack/releases/+/78600507:32
ttxhberaud: no objections07:44
ttxykarel: approved07:45
hberaudttx: thanks07:47
hberaudthen I think that we can proceed with this one => https://review.opendev.org/c/openstack/releases/+/78575407:49
*** tosky has joined #openstack-release07:50
ykarelttx, hberaud Thanks07:54
ykarelttx, hberaud please also check https://review.opendev.org/c/openstack/releases/+/784124/07:54
*** e0ne has joined #openstack-release07:54
*** rpittau|afk is now known as rpittau07:55
openstackgerritMerged openstack/releases master: Release puppet-glance 17.6.0 for stable/victoria  https://review.opendev.org/c/openstack/releases/+/78600507:56
elodhberaud: ack08:00
elodfor both :)08:01
hberaudelod: thanks08:02
elodI mean I have not reviewed them yet :X08:04
hberaudnp08:04
hberaudyou mean "ack" :)08:05
elod:)08:06
elodhberaud: The Release Patch looks good :) +2'd (only) :)08:18
hberaudthanks08:18
elod\o/08:19
*** slaweq_ has joined #openstack-release08:19
*** slaweq has quit IRC08:19
*** ricolin has quit IRC08:39
*** slaweq_ is now known as slaweq08:51
hberaudThe python infra seems fully operational => https://status.python.org/08:54
hberaudI prefer to check before we start to flood them with several new versions of deliverables08:55
hberaudI'm waiting for feedback from our infra team concerning our infra's status08:56
elod++09:12
openstackgerritHervĂ© Beraud proposed openstack/releases master: [doc] check infras status before the final release  https://review.opendev.org/c/openstack/releases/+/78618909:29
*** dtantsur|afk is now known as dtantsur09:45
* hberaud reboot10:13
*** hberaud has quit IRC10:13
*** whoami-rajat_ is now known as whoami-rajat10:17
*** hberaud has joined #openstack-release10:17
hberaudttx: I think that we can start to merge the final release patch, that's work for you?10:28
hberaudBy starting now I think that we will be ready around 15:30 UTC+2 ~ (if everything work fine)10:32
fungii'm around to help keep an eye on things too10:33
hberaudfungi: thanks, how is the situation from the infra team point of view?10:34
hberaudfungi: The python infra seems fully operational => https://status.python.org/10:34
hberaudand the openstack infra too (AFAIK)10:34
fungii'm still waking up, but i don't see anyone complaining about anything broken yet10:34
hberaudok10:35
hberaudnp10:35
hberaudOk I start to merge the patch10:36
fungizuul status looks fine too10:37
hberaudexcellent let's go10:37
fungiyes let's10:40
openstackgerritMerged openstack/releases master: Wallaby final releases for cycle-with-rc projects  https://review.opendev.org/c/openstack/releases/+/78575410:53
hberaud\o/10:53
hberaudalright so now the post-job on that openstack/releases merge is going to perform all the tagging10:53
hberauds/post-job/release-post job10:54
hberaudwhich will, in turn, create a number of release jobs10:54
fungiyep10:54
fungias tag-releases runs, we should see the release pipeline start to fill up with tags for all the projects10:56
hberaudyep10:56
hberaudthey now arrives one by one10:58
hberaudadjutant-ui => Successfully released (FIFO)11:10
*** eolivare_ has joined #openstack-release11:12
hberaudhttps://zuul.opendev.org/t/openstack/builds?job_name=openstack-upload-github-mirror11:14
hberaudhttps://zuul.opendev.org/t/openstack/builds?job_name=release-openstack-python11:14
*** tkajinam has quit IRC11:14
hberaudhttps://zuul.opendev.org/t/openstack/builds?job_name=announce-release11:14
hberaudhttps://zuul.opendev.org/t/openstack/builds?job_name=propose-update-constraints11:14
*** eolivare has quit IRC11:15
fungii've proposed https://review.opendev.org/786205 in case we end up waiting a long time for our static sites to update11:19
hberaudack11:20
hberaudgood idea11:20
hberaudAll jobs seems done there => https://zuul.opendev.org/t/openstack/builds?job_name=openstack-upload-github-mirror11:21
fungiexcellent11:21
hberaudall done up to monasca*11:26
hberaudseems to be going well11:26
hberaudwe could almost "twitch" that11:27
smcginnisLooks pretty smooth.11:29
hberaudyes11:29
hberaudeverything is green for now11:29
hberaudOnly one warning => https://zuul.opendev.org/t/openstack/build/a586d78652824f16878c5c6bfff96ed411:30
ttxlooking good so far!11:31
hberaudalmost there11:31
hberaudNotice that the warning was sucessfully retried11:32
fungiyeah, that propose-update-constraints retry is due to a known issue in limestone where sometimes nodes boot without ipv4 configuration. we access those over ipv6 so zuul is able to reach the node, but the node might try to reach some v4-only site when running a job so we discard it early and let the build retry11:33
hberaudok11:34
hberaudSo, this step is now successfully done!11:34
elod\o/11:34
hberaudNow I'm going to check for missing tarballs11:35
*** eolivare_ has quit IRC11:35
hberaudlet's go!11:36
*** ykarel_ has joined #openstack-release11:37
fungithe last vos release of the tarballs volume occurred at 11:35 and has already completed, so hopefully there's no delay there11:37
hberaudack11:38
hberaudwhich TZ?11:38
hberaudof 11:3511:38
fungiutc, sorry11:38
hberaudnp11:39
fungias far as i'm concerned all times are in utc11:39
smcginnisfungi operates on UTC time. :)11:39
hberaud:)11:39
fungiso now we're just waiting for release note publication to complete, since all those jobs are serialized to prevent rsync collisions over the same paths11:40
*** ykarel has quit IRC11:40
hberaudyes I remember that issue11:40
*** ykarel_ is now known as ykarel11:47
fungilooks like it's slowly churning through those11:48
hberaudcool11:48
openstackgerritMerged openstack/releases master: W,V,U,T branch for devstack-plugin-ceph  https://review.opendev.org/c/openstack/releases/+/78606911:53
hberaudMissing tarballs check is still in progress12:14
yoctozeptohberaud: "Also we need this patch merged before the initialization of Xena (from a release point of view), to allow to create this deliverable for Xena too." - could you explain it to me? /me interested to understand12:16
yoctozeptobecause I thought it's otherwise irrelevant12:16
*** eolivare_ has joined #openstack-release12:16
hberaudyoctozepto: Next week we will run the command "init-series" which I think is based on the existing deliverables during the previous series => https://releases.openstack.org/reference/process.html#week-after-previous-release12:17
hberaudso if the file doesn't exist then I think that the deliverable will be ignored for Xena12:18
yoctozeptohberaud: ah, thanks! makes sense to be in line12:19
*** nweinber has joined #openstack-release12:27
hberaudfungi: Well, the check just succeeded. No missing tarballs were find. I only noticed some python 3 wheel warnings, however py2/3 wheels have been found, so I think that we can ignore them. During Victoria we introduced changes on our script to ignore them (to not consider them as problems). Here are the filtered logs of the check, please double check them http://paste.openstack.org/show/804465/12:31
hberaudFrom my point of view everything seems ok12:31
ttxlooking12:32
ttxyes it looks good12:33
ttxhberaud: do you have a draft for the release email? I can review it now12:34
hberaudyes12:35
hberaudsec12:35
ttxI have a tunnel of meetings 14-17utc12:35
fungiyeah, the consistency check anomalies look related to old releases12:35
hberaudttx: => https://github.com/4383/notes/blob/master/openstack-announce.txt12:36
fungiahh, i see, not old releases, but projects still releasing python 2.7 compatible universal wheels12:37
fungie.g. swift-2.27.0-py2.py3-none-any.whl != swift-2.27.0-py3-none-any.whl12:37
fungigot it12:37
hberaudttx: here is a version more "reviewable/amendable" => https://etherpad.opendev.org/p/wallaby-release-announce12:39
hberaudfungi: ack12:40
ttxhberaud: "A lot of amazing work still being done by the community" sounds a bit negative :) I'd say "6 months. 800 contributors. 17000 changes. Another great release!"12:40
fungieven just s/still// would help12:41
ttxIf you don;t, I'll tweet that :)12:41
hberaudttx: ack12:41
hberaudI'll12:41
ttxok will retweet12:41
hberaudack12:41
hberaudfungi: I suppose that dropping py2.7 from setup.cfg could fix that, isn't?12:42
ttxSo the goal is to send that a bit before 15utc... like between 1430 and 1500.12:42
hberaudok12:42
fungihberaud: yes, well close, dropping universal=1 from the setup.cfg would12:43
hberaudSo, I think that we want to wait a bit for this one too => https://review.opendev.org/c/openstack/releases/+/78601912:43
hberaudisn't?12:43
fungihowever some of those projects (swift for example) may still be intentionally supporting python 2.712:43
hberaudI see12:44
hberaudI'll keep these logs under my elbow and I'll try to have a look during Xena12:44
hberaudAnd see if we can propose a couple of patch to stop to release these artifacts12:45
hberaudFor now I added this topic to our PTG agenda to ensure to not forget that point12:49
ttxhberaud: re: https://review.opendev.org/c/openstack/releases/+/786019 I would wait until the tag job queue empties12:57
ttxor 1430 whatever happens first12:58
hberaudack13:03
*** gthiemonge has joined #openstack-release13:05
fungiyeah, the burndown rate on those release notes jobs is rather slow13:11
hberaud:)13:12
ttxyes I bet they won;t be finished in 75min13:14
*** tkajinam has joined #openstack-release13:14
hberaud+113:14
fungithey spend a lot longer generating release notes than i would have expected13:18
ykarelelod, ttx can u please check https://review.opendev.org/c/openstack/releases/+/78412413:21
fungiaha, the translation building in the release notes jobs takes 4-5x as long as the main release notes build task13:22
elodykarel: looking13:24
fungilooks like it reruns sphinx-build separately for every configured locale, and is also scanning for notes as far back as stable/liberty13:24
hberaudouch13:24
hberaudcan't we ignore these unmaintained branches?13:25
hberaud(for the future runs)13:25
fungiprobably13:28
*** ricolin has joined #openstack-release13:34
fungii guess it's looking at the individual indices in https://opendev.org/openstack/heat/src/branch/master/releasenotes/source and removing those would effectively unpublish their old release notes13:36
fungithe bigger slowdown is that it runs once for each directory under https://opendev.org/openstack/heat/src/branch/master/releasenotes/source/locale13:37
fungibut i'm not sure there's a good way around that if the release notes are really being translated13:37
hberaudif we can already clean up the old things a little, we will save a significant time13:39
hberaudand by example for heat we'll save at least 5 rounds13:40
gmannhberaud: perfect, thanks for update.13:49
hberaudgmann: you're welcome13:50
elodykarel: I have some comments in the patch, can you check those if they are valid?14:01
ykarelelod, checking14:01
ykarelelod, what u mean by "reno suggests minor version bump"14:03
ykarelwrt latest commit current proposed is fine, as latest commit keeps on changing14:03
hberaudsome minor version are also already used on victoria14:04
hberaudso we need to stick to bugfix version14:04
ykarelhberaud, okk Thanks14:08
ykarelelod, hope ^ answers your question14:08
*** slaweq has quit IRC14:16
*** slaweq has joined #openstack-release14:20
*** ykarel_ has joined #openstack-release14:20
*** ykarel has quit IRC14:22
elodhberaud ykarel_ : ok, thanks, +2+W'd14:25
hberaudthanks14:25
ykarel_Thanks elod14:26
elodnp14:26
elodi haven't realized that those versions are already used14:26
hberaudnp14:27
*** ykarel_ is now known as ykarel14:29
hberaudKudos ttx you win your bet!14:30
ttx\o/14:31
hberaudStill 44 jobs14:31
hberaudThat's a beautiful score14:31
ttxhberaud: I would +2A now14:32
hberaudSo, I think that we can continue with https://review.opendev.org/c/openstack/releases/+/78601914:32
hberaudttx: yes14:33
ttxand send ann0uncement in 12 min14:33
hberaudAbout to queue up the release announcement.14:33
hberaudSent14:36
hberaudfungi: it's up to you to moderate this one when the patch will be merged14:37
openstackgerritMerged openstack/releases master: Make tripleo repos release for stable/ussuri branch  https://review.opendev.org/c/openstack/releases/+/78412414:37
fungihberaud: yep, i'm watching for it and will approve14:38
hberaudexcellent14:38
fungikeeping an eye on that change in the gate for now, but the announcement won't get generated until the merge commit gets builds in release-post14:42
fungiso probably at least another 6-7 minutes from now14:42
hberaudsure14:42
openstackgerritMerged openstack/releases master: Mark Wallaby as released  https://review.opendev.org/c/openstack/releases/+/78601914:45
hberaudrelease-post job queued14:46
fungiyeah, it's probably slowed a bit by all the tripleo deliverable releases14:47
ttxthe releases website publication is probably a post job, not a release-post?14:47
fungipublish-tox-docs-releases happens in release-post, i think that's the one14:48
hberaud+114:49
ttx+114:50
*** haleyb has quit IRC14:50
ttx845ec8f is the one we want completed14:51
fungiyup14:51
hberaudyes14:51
ttxwill be a close race to get it in before 15utc :)14:51
fungithe one before it in the queue is from the tripleo stable/ussuri release batch14:51
ttxyeah we should probably have blocked that one14:51
fungiin future, we might want to hold stable branch releases while coordinated release is underway14:52
hberaudagree14:52
fungiall the tags in the release pipeline are getting first rights for available job nodes too14:53
ttxI would hold on announcement until teh website shows it's released14:53
hberaud+114:54
fungiat least there are no more tripleo tags getting pushed at this point, so it should catch up shortly14:54
fungithe publish-tox-docs-releases build for the tripleo release batch finally got a node assigned14:57
hberaudyes14:57
fungiand looks like we have a semaphore for that build to keep it from colliding, so the build we're blocked on is waiting for that one to complete14:58
ttxyeah syncing that docs job sure delayed us :)14:58
elodsorry for the tripleo patch approve :S15:00
hberaudnp that's not your fault elod15:01
elodwell, technically I've approved that just in the wrong time o:)15:01
fungiyep, it's a good reminder to add an extra step to the coordinated release process in the future (freeze all other release activity)15:01
fungibut it's not like you would have known15:01
fungiand it's not a significant delay15:02
hberaudelod: I was not expected this issue too15:02
fungiwe're probably only a couple minutes out from the final release site publication starting15:02
hberaudyes I'll put an extra step to the process to avoid similar situation15:02
fungiafter that, there will be a (hopefully <5min) delay for the batched vos release of static site content to publish it to the read-only volume apache is serving15:03
fungiwe run that every 5 minutes, but it can be delayed by large volumes of data syncing for other static volumes (e.g. tarballs)15:04
openstackgerritHervĂ© Beraud proposed openstack/releases master: [doc] extra steps while coordinated release is underway  https://review.opendev.org/c/openstack/releases/+/78618915:07
hberaudnow queued15:08
hberaudextra steps to prevent similar issue next time ^15:08
ttxa watched job never completes15:09
hberaudCan we call it a schrodinger job?15:10
hberaudlet's go15:12
fungiyay it's started!15:12
fungiif the writes to afs happen in the next minute, then it'll get picked up in the 15:15 vos release15:14
fungino dice. should land in the 15:20 vos release at least15:15
hberaudyup15:15
clarkbI'm watching ya'll watch the jobs. Not sure if that is better or worse :)15:16
fungiwe're at that task now15:18
fungiand written!15:18
ttxETA 2min15:18
fungianother minute and the vos release should start15:18
fungivos release is underway15:20
hberaudWell, successfully updated => https://releases.openstack.org/15:20
hberaudLet's go15:20
ttxyep15:20
ttxgo for it15:20
ttxReady to retweet your formal announcement15:20
fungiyep, vos release finished15:21
hberaudok15:21
fungii'm watching the announce mod queue15:21
hberaudI'm waiting for the annoucement url15:21
ttxhberaud: did you send to openstack-announce yet?15:22
hberaudyes15:22
fungihasn't gotten to the listserv moderation queue yet15:22
ttxyeah I don;t see it yet15:22
hberaudI sent it 30 minutes ago15:23
fungito openstack-announce@lists.openstack.org?15:23
hberaudyes15:24
fungii'm checking the mta logs now15:24
hberaudah15:24
fungiare you subscribed to openstack-announce?15:24
hberaudyes good point15:24
fungiit will reject posts from non-subscribers15:24
fungiit will moderate posts from subscribers15:25
hberaudI just subscribed15:25
*** armstrong has joined #openstack-release15:25
ttxplease resend email15:25
fungiyou'll need to resend the announcement yeah15:25
hberauddone15:27
fungilooking it over quickly now15:27
ttxfungi: it's in queue,I'll let you process it15:28
fungii've approved it15:28
ttxhttp://lists.openstack.org/pipermail/openstack-announce/2021-April/002047.html15:28
fungittx: yep, i was just skimming the message body/headers really quickly for obvious problems15:28
ttxhberaud: ^15:28
hberaudack15:29
hberaudttx: https://twitter.com/4383hberaud/status/138235510097962189115:29
ttxFirst to retweet15:29
hberaudAnnnnddddddd sent to openstack-discuss => http://lists.openstack.org/pipermail/openstack-discuss/2021-April/021834.html15:31
ttxhberaud: congrats!15:31
ttxfor xena we'll try to finish 30 minutes early instead of late :)15:32
fungiexcellent work!15:32
hberaudThank you everyone for all your contributions and all your help during this cycle!15:32
ttxhberaud: you can drop the news on #openstack-tc and #openstack-dev for letting the IRCs know15:32
hberaudack15:33
elodcongrats! :)15:34
clarkbttx: I feel like the allowable window should be measured in a couple of hours :P congrats!15:34
gmannhberaud: ttx and release team for great work!15:39
hberaudOur pleasure!15:40
hberaudOpenstack is an awesome community, all steps are achieved all together15:41
armstrongCongrats team15:44
hberaudarmstrong: Thank you for all your contributions during this series15:45
armstrongAlways ready to do more :)15:46
hberaudI hope ;)15:47
* hberaud afk for awhile, be back later15:49
dtantsuryoctozepto: so what's about release notes?15:50
*** e0ne has quit IRC15:50
*** e0ne has joined #openstack-release15:51
yoctozeptodtantsur: https://review.opendev.org/c/openstack/releases/+/78421015:51
yoctozeptoyou were left out in there15:51
yoctozeptohence no ironic releasenotes in https://releases.openstack.org/wallaby/15:51
yoctozeptolikely due to patch merged late15:51
yoctozeptohttps://docs.openstack.org/releasenotes/ironic/wallaby.html seems working15:52
yoctozeptoso it's a matter of you proposing a fix on releases repo15:52
dtantsurmaybe actually too early: the branches of IPA and ironic were created quite late this time15:52
dtantsurI'll do it now15:52
yoctozeptodtantsur: I meant patch on your side15:52
dtantsurright15:52
yoctozepto:D15:53
*** e0ne has quit IRC15:53
openstackgerritDmitry Tantsur proposed openstack/releases master: Add release note links for ironic and ironic-python-agent  https://review.opendev.org/c/openstack/releases/+/78627315:55
dtantsuryoctozepto: done ^^15:55
yoctozeptodtantsur: great, thanks15:57
yoctozeptohberaud: ^^15:57
yoctozeptowould look better on the release page15:57
*** rpittau is now known as rpittau|afk16:21
hberauddone16:24
openstackgerritMerged openstack/releases master: Add release note links for ironic and ironic-python-agent  https://review.opendev.org/c/openstack/releases/+/78627316:39
*** eolivare_ has quit IRC16:48
*** e0ne has joined #openstack-release16:55
*** clarkb has quit IRC17:08
*** clarkb has joined #openstack-release17:08
*** ykarel has quit IRC17:35
*** dtantsur is now known as dtantsur|afk17:44
*** e0ne has quit IRC17:56
*** ianychoi__ has quit IRC18:08
*** whoami-rajat has quit IRC18:38
*** vishalmanchanda has quit IRC18:44
*** sboyron has quit IRC19:30
*** nweinber has quit IRC20:00
*** armstrong has quit IRC21:55
*** tkajinam has quit IRC22:43
*** tkajinam has joined #openstack-release22:52
*** tosky has quit IRC23:29
*** irclogbot_1 has quit IRC23:50
*** irclogbot_0 has joined #openstack-release23:56
*** irclogbot_2 has joined #openstack-release23:56

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!