| *** chandankumar is now known as chkumar|rover | 05:07 | |
| *** blarnath is now known as d34dh0r53 | 06:56 | |
| *** amoralej|off is now known as amoralej | 07:04 | |
| *** amoralej is now known as amoralej|lunch | 13:45 | |
| ttx | #startmeeting releaseteam | 14:00 |
|---|---|---|
| opendevmeet | Meeting started Fri Feb 10 14:00:01 2023 UTC and is due to finish in 60 minutes. The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| opendevmeet | The meeting name has been set to 'releaseteam' | 14:00 |
| ttx | Ping list: hberaud armstrong elodilles | 14:00 |
| elodilles | o/ | 14:00 |
| ttx | Meeting agenda at: https://etherpad.opendev.org/p/antelope-relmgt-tracking | 14:00 |
| ttx | down to ~line 306 | 14:00 |
| ttx | waiting a minute to see if we get anyone else | 14:01 |
| ttx | #topic Review task completion | 14:02 |
| hberaud | o/ | 14:02 |
| ttx | * Propose autoreleases for cycle-with-intermediary libraries (excluding client libraries) (elod) | 14:02 |
| elodilles | yepp | 14:03 |
| elodilles | i've proposed the patches: | 14:03 |
| elodilles | https://review.opendev.org/q/topic:antelope-final-non-client-libs | 14:03 |
| elodilles | ~half of them got responses and merged | 14:03 |
| ttx | ok looks like we could process a couple PTL+1s more | 14:04 |
| elodilles | i've reviewed all the generated patches and the rest now seems fine to me to merge | 14:04 |
| elodilles | +2'd them | 14:04 |
| elodilles | only one -1 from a team left | 14:04 |
| ttx | OK | 14:05 |
| elodilles | so as we promised in the commit message (those without responses) will be merged today | 14:05 |
| ttx | * Check with the Technical Committee that Zuul job templates have been created to include those runtimes (elod) | 14:06 |
| elodilles | so basically the patches need a 2nd +2 from either of you :) | 14:06 |
| elodilles | ok, so that one: | 14:06 |
| elodilles | patch is proposed but needs to be merged NO SOONER than 2023.1 final release has passed: | 14:07 |
| elodilles | https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/873051 | 14:07 |
| elodilles | this is because the process changed since last cycle | 14:07 |
| ttx | noted | 14:07 |
| elodilles | in connection, another patch is proposed to remove the script for autogenerated BOT patches: | 14:08 |
| elodilles | https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/873051 | 14:08 |
| ttx | hmm same one? | 14:08 |
| elodilles | indeed | 14:08 |
| elodilles | sry, let me search for the right one :S | 14:08 |
| ttx | https://review.opendev.org/c/openstack/project-config/+/873016 ? | 14:08 |
| elodilles | yepp ^^^ that one :) thx | 14:09 |
| ttx | OK I reviewed that one iirc | 14:09 |
| elodilles | thx \o/ | 14:09 |
| ttx | ok last task was: | 14:09 |
| ttx | - Send countdown email (ttx) | 14:09 |
| ttx | which we'll review in a sec | 14:09 |
| ttx | #topic Assign next week tasks | 14:10 |
| ttx | I'm OOO all week so would rather not take tasks | 14:10 |
| elodilles | ack | 14:10 |
| elodilles | all taken, thx hberaud :) | 14:11 |
| hberaud | np | 14:11 |
| ttx | Thanks guys! | 14:11 |
| ttx | #topic Review countdown email | 14:11 |
| ttx | #link https://etherpad.opendev.org/p/relmgmt-weekly-emails | 14:12 |
| ttx | Etherpad shows the areas I edited to facilitate your review | 14:12 |
| ttx | We all agree the branch will be named stable/2023.1 | 14:12 |
| ttx | and I edited the text around cycle highlights since the deadline is the week after now | 14:13 |
| hberaud | yes for the branch name | 14:13 |
| hberaud | LGTM | 14:13 |
| ttx | Also took the liberty of mentioning the early bird deadline as it may save the acute reader a few bucks | 14:14 |
| elodilles | thanks, LGTM, too | 14:14 |
| opendevreview | Merged openstack/releases master: Release final os-win for 2023.1 Antelope https://review.opendev.org/c/openstack/releases/+/872780 | 14:15 |
| elodilles | +1 to mention the early bird | 14:15 |
| ttx | Alright, last topic | 14:15 |
| ttx | #topic Open Discussion | 14:15 |
| ttx | anything to discuss? | 14:15 |
| hberaud | nope | 14:15 |
| elodilles | i've added the 2023.{1,2} just in case :X | 14:15 |
| elodilles | (for the release name) | 14:15 |
| bauzas | hey release team, I have a question | 14:15 |
| elodilles | bauzas: o/ | 14:15 |
| elodilles | shoot :) | 14:16 |
| bauzas | oh snap, you're on meeting | 14:16 |
| bauzas | my bad | 14:16 |
| bauzas | I can wait until the end | 14:16 |
| opendevreview | Merged openstack/releases master: Release final ceilometermiddleware for 2023.1 Antelope https://review.opendev.org/c/openstack/releases/+/872766 | 14:16 |
| ttx | well it's open discussion, so great timing | 14:16 |
| bauzas | lol | 14:16 |
| elodilles | :) | 14:16 |
| bauzas | my question was about how the proposal bot works | 14:16 |
| ttx | it uses magic beans, I think | 14:16 |
| bauzas | https://review.opendev.org/c/openstack/requirements/+/854821 was proposed but I don't see any open change for os-traits 2.10 | 14:16 |
| bauzas | ttx: I need the recipe | 14:17 |
| bauzas | oh wait | 14:18 |
| elodilles | hmmm, then there must be a bug :S | 14:18 |
| bauzas | I think I know the answer | 14:18 |
| bauzas | I don't see any bump yet | 14:18 |
| bauzas | hmmm https://review.opendev.org/c/openstack/releases/+/873106 | 14:19 |
| elodilles | hmmm, we don't have a pypi release either | 14:19 |
| elodilles | :S | 14:19 |
| bauzas | despite this, https://review.opendev.org/q/project:openstack/requirements+os-traits | 14:19 |
| bauzas | elodilles: yup, and the tag isn't present | 14:19 |
| bauzas | the git tag | 14:19 |
| elodilles | neither have we a 'release job failure' reported :-o | 14:20 |
| ttx | the tag is missing in git? | 14:21 |
| elodilles | even that | 14:21 |
| bauzas | yes, I quickly checked on gh, but I'll look at opendev | 14:21 |
| bauzas | https://opendev.org/openstack/os-traits/tags | 14:21 |
| bauzas | see, no 2.10.0 tag | 14:22 |
| * bauzas didn't want to derail the meeting | 14:22 | |
| fungi | probably the highlight from the infra channel is also about this, i haven't looked yet but trying to dig up the log for it now | 14:22 |
| bauzas | fungi: thanks | 14:22 |
| * fungi was ignoring the infra channel to pay attention to this meeting | 14:22 | |
| ttx | It's normal that we don;t have a release on PyPI as those get triggered by the tag | 14:23 |
| bauzas | fwiw, we have a feature patch awaiting for the u-c patch to land in order to have it merged this cycle | 14:23 |
| fungi | so this was the merge commit which should have triggered the tagging job: https://opendev.org/openstack/releases/commit/f2626efa6b9b75ae7e5886292b5b1b0f0b99dfe2 | 14:24 |
| elodilles | yepp | 14:24 |
| ttx | Hrm that pipeline look stuck since Feb 3 https://zuul.opendev.org/t/openstack/builds?job_name=tag-releases | 14:26 |
| elodilles | it seems we have quite many of these silently failing releases :( | 14:26 |
| ttx | The job does not even run so there is no error | 14:27 |
| fungi | the zuul builds interface finds nothing triggered in any pipeline for that merge commit | 14:27 |
| ttx | fungi: no tag-releases job ran for a whole week | 14:27 |
| ttx | and that was a busy week :) | 14:27 |
| elodilles | freezer-tempest-plugin -- Fri, 03 Feb 2023 16:16:11 GMT -- 3.0.0 | 14:27 |
| bauzas | \o/ | 14:27 |
| opendevreview | Merged openstack/releases master: Release final oslo.i18n for 2023.1 Antelope https://review.opendev.org/c/openstack/releases/+/872892 | 14:28 |
| elodilles | was the last one that was really released /o\ | 14:28 |
| opendevreview | Merged openstack/releases master: Release final oslo.metrics for 2023.1 Antelope https://review.opendev.org/c/openstack/releases/+/872784 | 14:28 |
| opendevreview | Merged openstack/releases master: Release final oslo.middleware for 2023.1 Antelope https://review.opendev.org/c/openstack/releases/+/872785 | 14:28 |
| opendevreview | Merged openstack/releases master: Release final oslo.context for 2023.1 Antelope https://review.opendev.org/c/openstack/releases/+/872782 | 14:28 |
| opendevreview | Merged openstack/releases master: Release final oslo.limit for 2023.1 Antelope https://review.opendev.org/c/openstack/releases/+/872783 | 14:28 |
| opendevreview | Merged openstack/releases master: Release final oslo.policy for 2023.1 Antelope https://review.opendev.org/c/openstack/releases/+/872786 | 14:28 |
| ttx | hmm I think we should stop approving stuff :/ | 14:28 |
| opendevreview | Merged openstack/releases master: Release final oslo.serialization for 2023.1 Antelope https://review.opendev.org/c/openstack/releases/+/872789 | 14:28 |
| opendevreview | Merged openstack/releases master: Release final oslo.upgradecheck for 2023.1 Antelope https://review.opendev.org/c/openstack/releases/+/872791 | 14:28 |
| fungi | i don't see any obvious job configuration errors for the releases repo | 14:28 |
| ttx | none of those will tag (or publish) | 14:28 |
| ttx | Ok I think we can move that one off-meeting | 14:29 |
| ttx | anything else before I close the meeting officially? | 14:29 |
| elodilles | this will be enough i think :D | 14:30 |
| elodilles | :/ | 14:30 |
| ttx | alright then | 14:30 |
| opendevreview | Merged openstack/releases master: Release final oslo.vmware for 2023.1 Antelope https://review.opendev.org/c/openstack/releases/+/872793 | 14:30 |
| elodilles | so let's close | 14:30 |
| ttx | #endmeeting | 14:30 |
| opendevmeet | Meeting ended Fri Feb 10 14:30:28 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:30 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/releaseteam/2023/releaseteam.2023-02-10-14.00.html | 14:30 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/releaseteam/2023/releaseteam.2023-02-10-14.00.txt | 14:30 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/releaseteam/2023/releaseteam.2023-02-10-14.00.log.html | 14:30 |
| fungi | looks like nothing has run in the release-post pipeline for a week: https://zuul.opendev.org/t/openstack/builds?pipeline=release-post&skip=0 so it's not just the tag jobs specifically | 14:30 |
| bauzas | I usually hate when someone rushes into our meetings and drops a bomb | 14:30 |
| bauzas | :) | 14:30 |
| ttx | elod: is tag-releases the only post-job that does not run when we merge openstack/releases changes? | 14:31 |
| elodilles | bauzas: especially in friday afternoon :] | 14:31 |
| bauzas | yeah tgif | 14:31 |
| bauzas | g being whatever you appreciate | 14:32 |
| elodilles | ttx: good question | 14:32 |
| ttx | Reference: https://releases.openstack.org/reference/release_infra.html | 14:32 |
| fungi | i don't see any obvious recent changes to job configuration for releases | 14:32 |
| fungi | ttx: see the link i posted | 14:32 |
| opendevreview | Merged openstack/releases master: Release final oslo.service for 2023.1 Antelope https://review.opendev.org/c/openstack/releases/+/872790 | 14:33 |
| opendevreview | Merged openstack/releases master: Release final oslo.versionedobjects for 2023.1 Antelope https://review.opendev.org/c/openstack/releases/+/872792 | 14:33 |
| opendevreview | Merged openstack/releases master: Release final automaton for 2023.1 Antelope https://review.opendev.org/c/openstack/releases/+/872763 | 14:33 |
| opendevreview | Merged openstack/releases master: Release final castellan for 2023.1 Antelope https://review.opendev.org/c/openstack/releases/+/872765 | 14:33 |
| opendevreview | Merged openstack/releases master: Release final stevedore for 2023.1 Antelope https://review.opendev.org/c/openstack/releases/+/872795 | 14:33 |
| fungi | it shows all the builds in the release-post pipeline so you can see what's been run in it historically | 14:33 |
| ttx | publish-release-docs also did not run since Feb 3, so that's the whole pipeline that is stuck https://zuul.opendev.org/t/openstack/builds?pipeline=release-post&skip=0 | 14:33 |
| fungi | right, or at least no longer being matched | 14:34 |
| ttx | (I mean, publish-tox-docs-releases) | 14:34 |
| *** amoralej|lunch is now known as amoralej | 14:35 | |
| elodilles | so it seems that the patch merge did not trigger the post pipeline at all, right? | 14:35 |
| ttx | yeah, il looks like no job is run from the release-post pipeline at all | 14:37 |
| elodilles | i'm looking at it's definition in project-config repo: https://opendev.org/openstack/project-config/src/branch/master/zuul.d/pipelines.yaml#L282 | 14:39 |
| fungi | looks like there may have been a schema/model change impacting that pipeline, or possibly some data corruption in zk: https://paste.opendev.org/show/bcGaLU1wkHZzXgcZ4agi/ | 14:40 |
| fungi | i'll take this to the zuul matrix channel | 14:40 |
| elodilles | fungi: thanks | 14:40 |
| fungi | the good news is i'm pretty sure the tag-releases and publish-tox-docs-releases jobs are idempotent, so as soon as we solve this the next releases change to land should catch everything up | 14:41 |
| elodilles | fingers crossed X | 14:42 |
| ttx | weird that release-post pipeline would not get run as it has the same trigger as the post pipeline, which runs alright | 14:42 |
| ttx | i.e. ref-updated with ref: ^refs/heads/.*$ | 14:42 |
| ttx | one is supercedent and the other is independent butthat's about the only diff | 14:43 |
| bauzas | fungi: others, I get that if those jobs aren't idempotent, the question would be "what happens with a revert then ?" | 14:43 |
| bauzas | I mean, what are the post-jobs of a releases revert patch ? | 14:44 |
| ttx | we do not revert releases | 14:44 |
| ttx | once released, always there | 14:44 |
| bauzas | ok, so technically, if we were needed to tag releases again, a revert and then a revert of revert would do the trick | 14:46 |
| fungi | ttx: yeah, my first guess is something may have changed in the zk schema in a backward-incompatible way, maybe only affecting supercedent pipelines | 14:46 |
| ttx | (post is supercedent, release-post is independent) | 14:47 |
| ttx | beats me | 14:47 |
| fungi | bauzas: the fact that it's a supercedent pipeline means we already skip all queued items between the running one and the most recent to be added, which implies the jobs are assumed to all be idempotent | 14:47 |
| bauzas | TIL | 14:48 |
| fungi | oh, hmm so release-post isn't supercedent? | 14:48 |
| fungi | i'll double-check in a moment | 14:48 |
| fungi | i was misunderstanding which one you meant was which, i guess | 14:48 |
| ttx | https://opendev.org/openstack/project-config/src/branch/master/zuul.d/pipelines.yaml#L286 | 14:50 |
| fungi | first occurrence of "json.decoder.JSONDecodeError: Extra data:" in the scheduler logs appears to be 2023-02-06 17:15:19,025 utc | 14:50 |
| ttx | afaict it's not, but that's the only difference it has with the 'post' pipeline | 14:51 |
| ttx | fungi: that would match, no release over the weekend anyway | 14:51 |
| fungi | last successful use of the pipeline was when 868588 merged 2023-02-03 16:08:40 | 14:58 |
| fungi | so an approximate 3-day window where things appear to have broken | 14:58 |
| fungi | er, started breaking | 14:58 |
| elodilles | (yepp, that's the freezer-tempest-plugin i think) | 14:59 |
| elodilles | (yepp: https://review.opendev.org/c/openstack/releases/+/868588/ ) | 15:01 |
| fungi | we automatically perform a graceful upgrade of all of zuul over the weekend, which is likely when the problem came in | 15:03 |
| fungi | the other pipeline which seems to be impacted is deploy, which explains a seemingly unrelated but lower priority mystery i had on my to do list to look into (we're apparently a few days behind on deploying infrastructure changes) | 15:06 |
| bauzas | fungi: ttx: then when can we expect to have the release-post pipeline to catch up ? | 15:07 |
| fungi | bauzas: it might be a day, if we need to patch zuul to recognize existing data in zk under the prior model | 15:08 |
| bauzas | ack, noted | 15:10 |
| bauzas | fungi: as I said, my only concern is about making sure we're not holding someone else's series to be merged in nova due to a missing lib upgrade | 15:11 |
| bauzas | but we'll freeze on thur | 15:11 |
| bauzas | I just hope the requirements team will be able to catch up the flow of releases bumps | 15:12 |
| fungi | it could be sooner if it's just some data corruption we need to manually clean up | 15:12 |
| bauzas | ++ | 15:24 |
| opendevreview | Brian Rosmaita proposed openstack/releases master: Release final os-brick for 2023.1 Antelope https://review.opendev.org/c/openstack/releases/+/872778 | 15:30 |
| fungi | okay, so the good news is that this is corrupted znodes which are in the process of being cleared now so we should be able to get stuff moving again in a few minutes. the better news is that this is probably a race condition which was identified over a week ago and has a fix at https://review.opendev.org/872482 which will hopefully merge today and get pulled in with our automatic | 15:37 |
| fungi | upgrades tomorrow, hopefully preventing recurrence | 15:37 |
| bauzas | fungi: great news ! | 15:42 |
| elodilles | thanks for the info fungi \o/ | 15:52 |
| fungi | i'll let you know when we're ready to try approving another release request and see if everything which merged previously ends up getting tagged | 15:56 |
| elodilles | fungi: ack, we have a patch that is waiting for approval, so that sounds good \o/ | 16:00 |
| fungi | elodilles: may not even be necessary. apparently the schedulers stored up all the items they had been unable to enqueue while the pipeline cache was broken, and they're running now | 16:01 |
| elodilles | fungi: perfect \o/ | 16:03 |
| elodilles | bauzas gibi : from pypi: os-traits -- Fri, 26 Aug 2022 17:09:34 GMT -- 2.9.0 | 16:25 |
| elodilles | bah | 16:25 |
| gibi | yeah we still need 2.10 | 16:25 |
| elodilles | sry :D | 16:25 |
| elodilles | false alarm :D | 16:26 |
| gibi | I know you are excited :) so much releases :D | 16:26 |
| bauzas | elodilles: don't tell me the past, I want the future :p | 16:27 |
| elodilles | guilty as charged o: ) | 16:30 |
| elodilles | bauzas gibi : nah, there you have it: os-traits -- Fri, 10 Feb 2023 16:30:49 GMT -- 2.10.0 | 16:32 |
| gibi | \o/ then we can all go home and enjoy our weekends ;) | 16:34 |
| gibi | thanks folks | 16:34 |
| bauzas | bravo and kudos for the hard work | 16:34 |
| bauzas | elodilles: I'll look at the requirements patches to see whether the bot posted them | 16:35 |
| elodilles | bauzas: it should be on the way: https://review.opendev.org/q/project:openstack/requirements+topic:new-release+is:open | 16:35 |
| bauzas | the requirements team will enjoy | 16:36 |
| fungi | sorry about the missed problem for nearly a week, but glad it was fixed without having to revert anything | 16:49 |
| opendevreview | Elod Illes proposed openstack/releases master: Create Train and Ussuri EOL tags https://review.opendev.org/c/openstack/releases/+/871857 | 16:50 |
| fungi | and as i said, after this weekend the race condition which caused that corruption should hopefully no longer be present | 16:50 |
| elodilles | ack, thanks | 16:52 |
| elodilles | i time to time check the releases on pypi so it's an unfortunate situation that it wasn't me who realised the issue :/ | 16:53 |
| elodilles | i even checked os-traits for nova team, but this time missed to look pypi /o\ | 16:53 |
| elodilles | could have been noticed earlier... :/ | 16:53 |
| elodilles | release-job-failure mail is so convenient that we forgot to be more careful o:) | 16:55 |
| *** amoralej is now known as amoralej|off | 17:08 | |
| fungi | nah, it's really not all that often we have a bug which causes jobs to not run and not report a failure. easy to overlook unfortunately | 18:56 |
| fungi | the backup in the tag and release-post pipelines reminds me that we still have serialized contention around the release notes and releases site content publication jobs | 19:59 |
| fungi | we're about 4 hours in and still have 7 items in tag and 24 in release-post waiting for their turn to run those jobs | 20:00 |
| opendevreview | Eric Arellano proposed openstack/reno master: Add support for subsections https://review.opendev.org/c/openstack/reno/+/873165 | 20:32 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!