Wednesday, 2023-10-04

hberaudo/05:54
ttxhappy release day everyone!06:53
elodilles\o/07:25
elodillesdiablo_rojo: thanks! o/07:25
elodillesfungi: cool, thanks! o/07:26
elodillesas far as i see there's no newly merged reno-2023.2 patch since i've generated the last reno adding patch: https://review.opendev.org/c/openstack/releases/+/89714607:31
elodillesso we are good to go with this one ^^^07:31
opendevreviewElod Illes proposed openstack/releases master: Mark 2023.2 Bobcat as released  https://review.opendev.org/c/openstack/releases/+/89730207:40
hberaud+W'd07:45
elodilleshberaud: thanks o/ i've added PTL-Approved+107:46
hberaudack07:46
opendevreviewMerged openstack/releases master: Add release note links for 2023.2 Bobcat (#4)  https://review.opendev.org/c/openstack/releases/+/89714608:09
ttxBooked our usual meeting slot on the PTG schedule09:17
elodillesttx: thx \o/09:23
hberaudthx09:23
elodillesi've reviewed the patch again ( https://review.opendev.org/c/openstack/releases/+/896971/3 ), PS3 looks OK to me. we are @ T-36mins to start :)09:24
ttxInfra looks good09:29
elodilles++09:34
fungiheyhey stackers!09:46
elodillesgood morning fungi o/09:47
* hberaud school-run, back in minutes09:49
fungishowtime!10:00
elodillesinfra still looks good, status.python.org, too10:00
elodilles+2'd the patch10:00
elodillesttx: your turn! :)10:00
fungihttps://grafana.opendev.org/d/21a6e53ea4/zuul-status?orgId=1 shows a surprising amount of activity for this time of day, but looks healthy enough10:00
elodillesyeah, sime teams are surprisingly active on final release day :)10:01
elodillesttx hberaud : if you are around, it's time to start the machinery o:) https://review.opendev.org/c/openstack/releases/+/896971/310:10
ttxoops sorry10:11
ttxhberaud: around, or should I +a?10:12
ttxI guess i should since he authored the patch10:12
elodilleshe mentioned he has to run to school for his kids, so +A please :)10:13
ttxdone10:13
elodilles\o/10:13
hberaudback10:14
elodillesjobs are queued10:14
elodillesvalidation and pep8 have started already10:15
elodillesvalidation will take like 30+ mins10:15
zigoFYI, Debian Unstable already has Python 3.12 ...10:16
zigoSo, as far as I'm concerned, OpenStack is effectively exactly 1 year late regarding this, as some projects don't even have voting gate on 3.11.10:19
elodilleszigo: sounds cool, especially given that 3.12 is released 2 days ago! :)10:19
elodilleszigo: Caracal has python 3.11 is amongst supported versions, so every projects will have py311 voting job in their gate10:20
elodilleszigo: https://governance.openstack.org/tc/reference/runtimes/2024.1.html10:21
zigoWell, the Python transitions are done in Debian Unstable, not Ubuntu, so we really should be using Debian Testing moving forward, IMO.10:21
zigoEither that, or Ubuntu with a custom interpreter.10:22
fungiin the middle of making the release is hardly the time to debate dependency choices (nor is this the channel for it anyway), but we normally start testing on new python versions once they're available in ubuntu lts. normally the x.y.1 python releases get backported, which will likely happen around january, in time for us to start testing it at the beginning of our next development10:23
fungicycle after 2024.1/caracal10:23
zigofungi: Right, sorry... :)10:23
fungiwe've had rather lengthy debates on testing platform choices too, debian/stable we can do. unstable/rolling distributions are pretty impossible for us (we've had to drop opensuse tumbleweed, fedora, non-lts ubuntu, etc)10:24
fungitoo much changing in the distro means not enough stability for our own testing, and people spend more time trying to fight through transient problems in the platform rather than fixing bugs in the software10:25
fungiwe even mostly stopped running centos jobs for the same reason10:25
fungithe developers want test platform choices to be made by the beginning of a new development cycle, and then have those platforms remain mostly unchanged (aside from minor bug fixes) until the release is done ~6 months later10:27
fungitesting on a 6-month-old frozen version of debian/unstable probably wouldn't help anyone10:27
fungi(which, ironically, is exactly what the non-lts ubuntu releases are too)10:32
ttxbreaking for quick lunch, brb10:42
elodillesbon appétit!10:45
elodillesthe validation job is still running. (it could run 40+ mins long)10:47
fungiyeah, i think for antelope it took just under an hour? anyway, looks like it's on solum now10:48
elodillesyes, but with a minor change in releases repo it became way faster10:49
fungioh, cool i missed/forgot about that10:49
elodilleswell, it was a side effect of changing from 'python3 setup.py sdist' to 'python3 -m build --sdist --wheel' in requirements checking part of validation :)10:52
elodillesanyway, here we can see the previous runs of the validator job on this patch: https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-validate&change=896971&skip=010:53
fungiwe are still calling setup.py --name though, i see constant warnings about it in the output10:53
fungimaybe we should extend the pbr entrypoint to be able to output the project name10:54
elodilleshmmm, i thought we replaced all occurances by now :-o10:55
fungivalidation completed, job is wrapping up now10:55
elodillesfingers crossed10:55
opendevreviewMerged openstack/releases master: Bobcat final rc  https://review.opendev.org/c/openstack/releases/+/89697110:55
fungiand now for the fun part10:55
elodillesthere it is! ^^^ \o/ 10:55
fungitag-releases is already running10:56
elodillesyeah, tagging & publishing the packages & docs10:56
fungiopenstack/adjutant-ui got tagged10:58
elodillesso far so good10:59
fungiand more tags are apearing, looks good so far10:59
fungirelease-openstack-python is underway for adjutant now10:59
fungipython-adjutant 7.0.0 was just uploaded https://pypi.org/project/python-adjutant/#files11:01
fungiwe've got multiple releasenotes jobs running in parallel, so the semaphore removal worked11:04
fungifirst release announcement has gone out to the release-announce ml (adjutant)11:05
elodillesyepp, and the 2nd mail has come, too: barbican -- https://lists.openstack.org/pipermail/release-announce/2023-October/thread.html11:07
fungiwe've had a spike in node requests for the burst of job activity, but plenty of capacity to satisfy those11:08
elodilles++11:09
elodillesyeah, 57 items in the release pipeline & 32 in the tag pipeline11:10
elodillesthere will be some release job failure as i see... monasca. the usual ones i guess...11:11
elodilles(docker image publishing jobs)11:11
fungilooks like tag-releases is cloning everything over the network... "WARNING: Cache directory /opt/git/openstack/nova does not exist, ignoring."11:13
fungisomething to look into11:13
hberauddad taxi again, brb11:17
elodilleshberaud: ACK, o/11:18
elodillesrelease pipeline already shrinking11:19
fungiyep, tag-releases finished11:19
fungi(successfully)11:19
elodilles\o/11:20
fungitrove-dashboard releasenotes failed to build11:24
elodilles:S11:24
fungilooks like a broken sphinx config11:25
elodillesyeah, it failed at rc1 as well11:25
fungiokay, so known issue11:25
elodillesaccording to job history11:26
fungii wonder if it would make sense to find a way to batch up the constraints updates from coordinated releases11:28
fungieach one starts about 40 new jobs in check11:28
elodillesi guess frickler had a similar idea, discussed with the requirements team11:30
elodilless/guess/think/11:30
fungiluckily, most of these aren't resulting in constraints updates11:30
fungiso maybe it's not all that bad11:31
elodillesyeah, just wanted to write this ^^^11:31
elodillesservices are not part of the uppder-constraints.txt in general11:31
fungiright, so far it's been just ceilometer and tap-as-a-service11:32
hberaudback11:32
fricklerwell the idea I had was more related to multiple u-c updates conflicting each other11:36
fricklernot about batching multiple ones into a single commit11:36
fungiyeah, for a moment i thought there were a lot more than there turned out to be11:37
fungiso far the only job failures seem to be known ones for monasca-* image publication and trove-dashboard's broken sphinx config11:42
elodillesyepp11:43
fungithe tag pipeline is finally empty (all releasenotes besides trove-dashboard completed successfully), just waiting on the last few in the release pipeline to clear now11:48
fungithe last release-openstack-python build (skyline-apiserver) just started11:50
fungiand there it is! https://pypi.org/project/skyline-apiserver/#files11:52
elodilles\o/11:52
fungion track to have this phase done before 12:00z11:53
elodilles2 jobs are still in the queue for the last item (skyline-apiserver) in the release pipeline11:56
elodillesthough those are the announce-release & the propose-update-constraints jobs,11:57
elodillesso i think the script that checks for missing releases can be started11:57
elodillesttx: fyi ^^^11:57
ttxon it11:58
fungiyes, everything that's going to get uploaded and published has beem11:58
fungier, been11:58
fungithe straggler jobs are finally running11:59
elodillesso we misses 12UTC with some seconds then :D12:00
elodilles(well, zuul says 2 mins :))12:01
fungibut we're also going at this way earlier than last time, so plenty of breathing room12:01
elodillesyeah, we started earlier + jobs were faster, so far so good12:02
fungiand the release pipeline is finally clear!12:02
elodilles\o/12:03
fungiwe started a little later than planned but still through this phase at approximately 2 hours in12:03
elodillesyepp12:03
elodillesi guess best is to wait until we see that there's no missing releases and then start with the next task: merging '[www] Set 2023.2 Bobcat as released' patch12:05
ttxscript is running12:05
elodilles++12:08
ttxsuccessfully completed12:20
elodilles\o/12:20
fungiawesome12:21
ttxI'd say you can proceed with '[www] Set 2023.2 Bobcat as released' patch12:21
elodillesttx: if i remember well you had +W right for this repo, so could you please add a 2nd +W (so that we don't need to recheck :)) >>> https://review.opendev.org/c/openstack/openstack-manuals/+/896402/212:22
ttxyes I do12:23
ttxdone12:23
elodillespatch is on the gate! thanks!12:24
fungiand all builds running, zuul estimates 11 minutes to completion12:25
elodilles++12:25
fungithough this took closer to 40 minutes for antelope, so the estimate may be skewed12:25
elodilleshmmm, strange, i wouldn't expect a longer run than zuul estimates for this patch12:27
fungioh, because we also needed time for the promote pipeline jobs after it merges12:28
fungihttps://meetings.opendev.org/irclogs/%23openstack-release/%23openstack-release.2023-03-22.log.html#t2023-03-22T13:31:1712:28
Clark[m]Estimates now are only for running jobs so aren't accurate until every job is running.12:29
fungiwell, every build is in progress, but my recollection included the actual publication from the promote pipeline jobs12:29
Clark[m]In the past it wouldn't estimate until all jobs were running. I forget why this changed. Ah12:29
elodillesyeah, i guess in the 40 mins we have more than the jobs finishing times. so the estimate should be precise :)12:30
fungiit was approximately 40 minyres from when we approved the change to when the content showed up on the site12:31
elodilles~5 mins left12:31
fungigate+promote+afs12:31
elodilles++12:31
elodilleshence we started the release process ~1hr earlier now :)12:32
elodilleszuul says 3 mins, but the job is ahead, so it should be less i think12:33
fungiyep, merged just now12:34
elodillesyepp, it's ready \o/12:34
fungipromote jobs running, estimate is 24 minutes12:34
elodillesso now we have to wait until docs.o.o refreshes, right?12:34
elodillesyepp, both jobs are running12:35
fungithese are the jobs that actually push the updated content into the afs writeable volume, and then once they complete we wait for afs to replicate the writeable volume to the read-only volumes. that kicks off every 5 minutes, but can take time to complete depending on how much data needs to be transferred between the fileservers12:36
elodilles++12:36
* fungi takes this opportunity to put the kettle on for another cup12:38
elodillesenjoy your tea! :)12:40
fungithanks!12:40
fungipromote jobs finished12:45
funginevermind, they haven't yet12:46
fungizuul just got impatient because of the lack of output from the current upload task12:46
elodilles:-o12:46
fungi(if console output is silent for too long, the console stream gets terminated and reconnects)12:46
fungior rather the browser closes the websocket and reestablishes it12:47
elodillesah, i see12:52
fungicontent upload into afs has finished, so now we're waiting on the 13:10 vos release cronjob to start replicating it13:07
elodilles\o/13:08
hberaud\o/13:08
fungiunderway13:10
fungireplication of the docs volume completed at 13:1113:11
fungihttps://docs.openstack.org/ redirects to 2023.2 and says it's the currently supported release13:11
elodilles\o/13:12
elodillescool!13:12
fungiand there's a drop-down option for "2024.1 (in development)"13:12
elodillesi think we can proceed with the next patch then: https://review.opendev.org/c/openstack/releases/+/89730213:14
hberaud+113:14
elodillesit needs a +W then :]13:14
hberauddone13:15
elodilles\o/13:16
elodilleslet's wait until that merges13:16
fungii would have expected to see zuul react to that approval vote by now13:18
ttxwas missing PTL-Approved13:19
ttxI just pushed it13:19
fungid'oh!13:19
funginow it's going13:20
hberaudthx13:21
elodilles:)13:22
fungiestimated time to merge is 16 minutes13:26
fungithen promote job to upload into afs, and vos release cronjob13:26
elodillesit should be less than 16 mins, i guess the 'Final release' patch "confused" a bit the estimation :)13:27
Clark[m]Is this the last change that needs processing to complete the release?13:28
elodillesClark[m]: yes, this should be the last one13:28
fungiyeah, and then the announcement e-mail gets sent13:28
elodillesthen we only have 3 tasks that are: 1) inform foundation staff 2) send mail to openstack-announce 3) send mail to openstack-discuss13:29
fungiClark[m]: also i've un-wipped https://review.opendev.org/896949 to put the release notes semaphore back, if you want to approve it when you've got access (there's no rush)13:30
opendevreviewMerged openstack/releases master: Mark 2023.2 Bobcat as released  https://review.opendev.org/c/openstack/releases/+/89730213:30
fungiwoo!13:30
fungii guess publish-tox-docs-releases runs in release-post not promote13:31
elodilleshberaud: for 2 and 3 if you want we can do the review part in advance at the usual place: https://etherpad.opendev.org/p/relmgmt-weekly-emails13:31
hberaudthe etherpad is already up-to-date13:32
elodilleshberaud: ACK, reviewing in a minute!13:34
hberaudI just added the openstack-discuss part at the end of the etherpad13:35
elodillesshould we wait with the mails ~5 o'clock-ish?13:35
fungishould that be "Let me join *in* thanks"?13:35
ttxelodilles: ideally sometimes between 14 and 15utc13:36
ttxso that we beat the press releases13:36
fungithe publish job for the releases site hasn't started yet anyway13:36
hberaudlet send it in 25 min, at 2pm13:36
fungiso it may not actually be updated until after 14:00 utc depending on how long the remaining bits take to happen13:36
hberaudthat's work for you?13:37
fungipublish-tox-docs-releases is finally in progress now13:37
hberaudack13:38
fungizuul estimates 7 minutes to completion13:38
fungiso we may make the 13:45 vos release pulse, more likely 13:5013:38
elodilles"We will meet *at* the Virtual..." ? (i'm not natives speaker, so just asking o:))13:38
elodillesttx: ACK, thanks13:38
fungifwiw, the foundation event organizers are no longer referring to it as "the virtual ptg" (or vptg), they just note that it's happening remotely, on-line13:39
elodillesfungi: oh, thanks for the information. so that means that sentence can be rewritten a bit more than that13:40
fungithough messaging it still rather split. ptg2023.openinfra.dev does say "virtual project teams gathering" up top13:41
fungii'll remind them that needs updating ;)13:41
elodillesabout the 2nd mail, it could be more like a reference to the announcement mail, like we did in the past: https://lists.openstack.org/pipermail/openstack-discuss/2022-October/030735.html13:42
elodilleswhat do you think?13:42
hberaudwfm13:43
elodilleshmmm, maybe it's good as it is13:43
elodillesnever mind13:43
elodilles(sorry o:))13:44
fungiokay, releases site contents have been uploaded into afs, in time for the 13:45 vos release pulse13:44
fungiwhich is running now13:45
fungiand https://releases.openstack.org/ shows bobcat as maintained rather than in development13:45
hberaudelodilles: I updated the etherpad with your suggestion13:45
elodilleshberaud: thx, both mails look good to me o:)13:46
hberaudthx13:46
elodillescool, so now https://releases.openstack.org is also up to date \o/13:47
elodillesno we have to wait a bit until the mails can be sent out :]13:51
elodilless/no /now /13:51
ttxYeah, feel free to send once the clock hits 1400utc... I'll be in a meeting13:51
elodilleshberaud: ^^^13:52
hberaudok, will send it in ~8min13:52
elodilles+113:52
fungii'm around to approve the announcement in moderation13:52
zigoecho "Mail -f announcement.txt" | at "14:00"13:53
zigo:)13:53
fungiit's time!14:00
* fungi makes alarm clock noises14:00
fungii've got the openstack-announce moderation queue up and waiting14:01
hberaudsent14:04
fungiand accepted14:05
elodillesthere we have it: https://lists.openstack.org/pipermail/openstack-announce/2023-October/002073.html14:07
elodillesnice!14:07
hberaudand sent on openstack-discuss14:07
elodillesyepp: https://lists.openstack.org/pipermail/openstack-discuss/2023-October/035266.html14:08
elodillesthanks \o/14:08
fungii've notified the foundation marketing team, they're doing their stuff now14:08
elodilles\o/14:09
fungihttps://social.openinfra.dev/@OpenStack/11117713368900965414:09
fungihttps://www.linkedin.com/feed/update/urn:li:activity:711533680324656742414:09
ttx\o/14:10
fungihttps://www.facebook.com/photo?fbid=706738644834152&set=a.624784356362915 (doesn't seem to be visible without a facebook account, but i'll take their word it's there)14:10
elodilles:)14:11
hberaud\o/14:11
fungihttps://twitter.com/OpenStack/status/170957136800456741014:11
fricklershouldn't that be x.com now? also one should seriously discuss whether to continue using it14:13
fungiyeah, well mastodon was the first place the marketing folks posted to14:14
fungitwitter was the last ;)14:14
fungii expect they'll continue to notify twitter followers about things until that's no longer viable14:14
fungibut it hasn't stopped them from also adopting newer channels14:15
fricklerIMNSHO it hasn't been viable for quite some time, if one takes any form of openness serious14:16
fricklerbut apparently also zoom continues to count as a viable option, so what do I know?14:18
fungiis them posting announcements to twitter any worse than the announcements they post to facebook and linkedin? none of those platforms are anything i feel safe using, but reaching people who do use them still seems to have some utility14:42
fungithey also send press releases to news sites, almost none of which are open platforms (and some of which are paywalled most likely)14:44
fricklerfacebook and linkedin afaict at least still try to combat hatred and racism, yes. doing anything that helps people to continue using x is being actively malicious IMO14:44
fungii see. my distaste for proprieary platforms that snoop on conversations and try to profit off them means i have no idea which ones are less ethical than which other ones14:46
zigofrickler: The line between "combat hatred and racism" and censorship is kind of blured, it's a long debate we don't want to enter ... The only thing we can agree on, is what fungi wrote (that it's all proprieary platform and that we should move to a distributed system...).15:03
fungiwe're getting closer to convincing people to use meetpad, at least. for example the foundation calls with community event organizers are using it now15:04
fungiit came up in the tc meeting yesterday (in zoom, ironically) and there were concerns raised about sound quality of jitsi's local audio recording option, inability to dedicate channels to specific owners so that they can ban abusive participants from calls, lack of international dial-in trunks, no transcript feature... the proponents of proprietary teleconferencing platforms refer to15:08
fungithose as "accessibility" features, and are opposed to moving to "less accessible" (but open) platforms15:08
zigoI never had any issue with the sound quality of Jitsi, nor the recording during Debconf online events ...15:09
fungiwell, i don't think the people complaining have really tried it in a year or two, and we're constantly upgrading to the latest version15:13
fungi"accessibility" has become the new word for "convenience" and there are definitely a lot of community participants who value convenience over free/libre open source principles. they're happy to use open source when it works better for them than any of the proprietary alternatives, but don't want to compromise on features in order to do so15:15
clarkbzigo: gmann tested the recording feature a while back and reported there was an echo. Since then there have been many jitsi updates so it is worth testing again. As for general issues we've found that it consumes resources on the local side (not the server) in relation to the number of participants. Having a beefier machine helps. But also that limit seems to be above ~3015:19
clarkbat this point which covers the vast majority of ptg use cases15:19
fungifwiw, i've been on scs community jitsi sessions with well over 30 people most of whom have their cameras on. granted, i do adjust my browser to stop receiving video streams on those calls, but the workstation it's running on is pretty heavily loaded with other activity15:24
fungireviews on https://review.opendev.org/896943 appreciated, i'll un-wip on monday and we can merge then20:14

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!