| opendevreview | Dmitriy Rabotyagov proposed openstack/election master: Add Dmitriy Rabotyagov candidacy for TC https://review.opendev.org/c/openstack/election/+/910325 | 08:47 |
|---|---|---|
| opendevreview | Artem Goncharov proposed openstack/election master: Add Artem Gonharov candidacy for TC https://review.opendev.org/c/openstack/election/+/910327 | 09:10 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/election master: Add Dmitriy Rabotyagov candidacy for Vitrage PTL https://review.opendev.org/c/openstack/election/+/910328 | 09:10 |
| opendevreview | Artem Goncharov proposed openstack/election master: Add Artem Goncharov candidacy for TC https://review.opendev.org/c/openstack/election/+/910327 | 09:35 |
| opendevreview | Gregory Thiemonge proposed openstack/election master: Adding Gregory Thiemonge candidacy for Octavia https://review.opendev.org/c/openstack/election/+/910362 | 16:10 |
| opendevreview | Michael Johnson proposed openstack/election master: Adding Michael Johnson candidacy for Designate https://review.opendev.org/c/openstack/election/+/910367 | 16:31 |
| *** dasm is now known as Guest1195 | 16:51 | |
| *** Guest1195 is now known as dasm | 17:34 | |
| gmann | meeting time? | 18:01 |
| rosmaita | pretty sure JayF is around | 18:02 |
| slaweq | gmann I was just thinking if it's just my matrix bridge not working properly again and I don't see anything :) | 18:02 |
| JayF | I am here | 18:02 |
| slaweq | but seems like it works fine :) | 18:02 |
| rosmaita | \o/ | 18:02 |
| gmann | here he is. | 18:02 |
| JayF | Why did my brain think it's 11am for the meeting | 18:02 |
| JayF | wow | 18:02 |
| gmann | slaweq: I refreshed mine for same reason :) | 18:02 |
| JayF | one sec lemme get my doc up | 18:02 |
| * JayF chasing C-3 deadlines and getting distracted | 18:03 | |
| JayF | #startmeeting tc | 18:03 |
| opendevmeet | Meeting started Tue Feb 27 18:03:28 2024 UTC and is due to finish in 60 minutes. The chair is JayF. Information about MeetBot at http://wiki.debian.org/MeetBot. | 18:03 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 18:03 |
| opendevmeet | The meeting name has been set to 'tc' | 18:03 |
| JayF | Welcome to the weekly meeting of the OpenStack Technical Committee. A reminder that this meeting is held under the OpenInfra Code of Conduct available at https://openinfra.dev/legal/code-of-conduct. | 18:03 |
| JayF | Today's meeting agenda can be found at https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee. | 18:03 |
| JayF | #topic Roll Call | 18:03 |
| dansmith | o/ | 18:03 |
| gmann | o/ | 18:03 |
| frickler | \o | 18:04 |
| jamespage | o/ | 18:04 |
| slaweq | o/ | 18:04 |
| rosmaita | o/ | 18:04 |
| JayF | #topic Follow up on tracked action items | 18:04 |
| JayF | I had 3 action items, I have done none of them. | 18:05 |
| JayF | #action JayF reach out to Tony and Elod about follow-up to unmaintained group email | 18:05 |
| JayF | #action JayF to email ML with a summary of gate ram issues and do a general call for assistance | 18:05 |
| JayF | Actually I did the third one | 18:05 |
| JayF | > Find something written about who volunteered to keep things un-maintained status back to Victoria | 18:05 |
| JayF | I was unable to find anything in public ML archives with someone concretely volunteering to take things over back to victoria. | 18:06 |
| JayF | I did find a thread where that was volunteered *in context of heat* | 18:06 |
| JayF | I'll take care of those two other actions as soon as meeting is over here, sorry about that | 18:06 |
| spotz[m] | o/ | 18:06 |
| JayF | #topic Gate Health Check | 18:06 |
| JayF | How is the gate looking? I'll note for Ironic we've been pretty swell. | 18:06 |
| dansmith | lots of rechecks in nova/glance land, AFAICT | 18:07 |
| frickler | what's swell in that context? I only know ocean swell | 18:07 |
| fungi | "pleasant" | 18:07 |
| slaweq | not that bad in Neutron AFAIK | 18:07 |
| frickler | thx fungi | 18:08 |
| fungi | "swell" can be an adjective meaning "very good" | 18:08 |
| JayF | pleasant is a pretty dead-on for getting the connotations | 18:08 |
| JayF | Going to move on, we addressed gate pretty heavily last meeting and I still have the follow-up action from that | 18:08 |
| JayF | #topic Implementation of Unmaintained Branch Statuses | 18:09 |
| JayF | does anyone have an update on this? rosmaita? | 18:09 |
| frickler | there were some issues in stable branches regarding upgrades from yoga | 18:09 |
| rosmaita | i don't have anything new | 18:09 |
| frickler | and people are unclear whether they should be made to work or whether yoga should be treated like being eol in that context | 18:09 |
| frickler | they=grenade jobs | 18:09 |
| frickler | I support the latter option fwiw | 18:10 |
| JayF | For purposes of unmaintained branch policy, I'm pretty sure that's entirely up to the folks doing the work, yeah? | 18:10 |
| gmann | yeah, discussion going on in #link https://review.opendev.org/c/openstack/grenade/+/908826 | 18:10 |
| gmann | I also do not think we need to do upgrade testing from unmaintained to any other maintained branches | 18:10 |
| frickler | well the issue is that the job failures are affecting zed and 2023.1 | 18:10 |
| gmann | we made it optional to tests for EM and for unmaintained either we can just remove or kepe it optional only | 18:11 |
| frickler | and then we have to discuss things like https://review.opendev.org/c/openstack/grenade/+/908826 | 18:11 |
| gmann | frickler: job of yoga to zed upgrade so my point is we do not need to test upgrade from unmaintined to maintained releases | 18:11 |
| JayF | ++ | 18:12 |
| rosmaita | gmann: ++ | 18:12 |
| JayF | And we have a pretty strong workaround if it ever breaks: upgrade to -eom | 18:12 |
| JayF | then upgrade to latest from UM | 18:12 |
| JayF | because until maintenance ended, we proved upgrades worked | 18:12 |
| gmann | and devstack/grenade yoga branches will also not unmaintained so unmaintained maintainers can work there if they need to keep it tested? | 18:12 |
| frickler | other than that I can note that the yoga unmaintainance was completed and elodilles is now preparing patches to move all other older branches back to v | 18:12 |
| JayF | Thank you for the work making this happen | 18:13 |
| gmann | frickler: for devstack/grenade also or we need to do something there? | 18:13 |
| gmann | I think there was one bug logged in devstack or tempest for this | 18:13 |
| frickler | gmann: I don't understand the question. the patches will be for all projects, like we had them for yoga | 18:13 |
| gmann | frickler: ok I did not see unmaintained/yoga for devstack greande or its setup in the scripts. | 18:14 |
| gmann | will check later | 18:14 |
| frickler | well the branch was created | 18:14 |
| frickler | I didn't see much work so far in general to actually make jobs on yoga work | 18:15 |
| frickler | so another question is how much time do we want to give people until we say: "this doesn't work, let's just eol it" | 18:16 |
| gmann | ++, as long as we cut the branches for devstack/grenade then we are good. I also do not think I myself will spend time on setting up the tempest testing there | 18:16 |
| gmann | on time before EOL, maybe giving/open it for a cycle will be enough ? | 18:16 |
| gmann | it does not harm to keep the broken one for a cycle and it will give enough time for people to step up | 18:17 |
| JayF | Yeah, I think the normal 6 month period is what to wait for | 18:17 |
| frickler | a cycle is a long time, the usual time to move zed to eom would be one month after 2024.1 release | 18:17 |
| JayF | and then at next evaluation, if CI is not passing, it gets EOL'd | 18:17 |
| frickler | which is in roughly two months | 18:17 |
| JayF | frickler: Zed is not slurp, so it'd get an automatic unmaintained branch for 6 months AIUI | 18:18 |
| rosmaita | well, this is the first time we're doing this, i think we need to be flexible about the deadlines | 18:18 |
| gmann | a cycle will give enough time for people if they care that is only I was thinking. | 18:18 |
| JayF | rosmaita++ | 18:18 |
| rosmaita | for example, the normal timeline for stable/yoga -> unmaintained would have been november | 18:18 |
| JayF | it's also literally 2 days before C-3 | 18:18 |
| JayF | many people (myself included) are trying to complete things before the end of cycle | 18:18 |
| JayF | I suspect we'll get a much better insight into how much people are concerned about the CI for these UM branches once some of the deadline-crunch is passed | 18:19 |
| frickler | ok, so let's wait and see | 18:19 |
| JayF | frickler: I'm going to send that gate email in my actions after this meeting; if you have a blurb you want me to add about UM branches that might be an easy way to raise awareness | 18:20 |
| JayF | take the opportunity to shine light when we can and hope someone is looking :D | 18:20 |
| frickler | JayF: maybe just mention that now is the opportunity for people to show they care about keeping those branches alive | 18:21 |
| JayF | ++ | 18:21 |
| JayF | #topic Testing runtime for 2024.2 release | 18:21 |
| JayF | #link https://review.opendev.org/c/openstack/governance/+/908862 | 18:21 |
| JayF | gmann added this topic; but I'll note: we really need folks to participate here and vote; I'm not sure we're going to reach community consensus especially around python versions | 18:22 |
| dansmith | the thing I think is missing there, | 18:22 |
| dansmith | is adding 24.04 to "best effort" right? | 18:22 |
| gmann | yes that is one thing | 18:23 |
| dansmith | it seemed like that was liked by multiple people for when it comes out | 18:23 |
| dansmith | otherwise I think I'm good | 18:23 |
| JayF | I think that'd be an ideal change | 18:23 |
| gmann | my point is we should not add when it is not yet released | 18:23 |
| dansmith | so I was waiting for or expecting that respin | 18:23 |
| gmann | once it is released in april then we can add | 18:23 |
| frickler | IMO it should not be added there. we didn't add py3.11 while it was non-voting either | 18:23 |
| dansmith | gmann: so you want to amend the document once it's released? | 18:23 |
| JayF | Honestly not a bad approach, add 24.04 as "best effort to ensure smooth upgrade" once we're sure we can potentially do it | 18:24 |
| gmann | dansmith: we can that time. but now It seems not correct way to add something not yet there | 18:24 |
| rosmaita | well, we should at least say that we anticipate adding it when available | 18:24 |
| dansmith | seems like we've had pushback before when we change that late, so it seems most up-front to go ahead and stake our claim on it being a thing | 18:24 |
| slaweq | how "smooth" it will be if it will be best effort and some projects will test it and others maybe not? | 18:24 |
| rosmaita | otherwise, it looks completely ad-hoc later | 18:24 |
| frickler | we could add 24.04 once devstack jobs are working. iff someone actually implements that in time | 18:24 |
| dansmith | minimal surprises and all | 18:25 |
| gmann | dansmith: sure in that case I will say to add in next cycle 2025.1 | 18:25 |
| JayF | rosmaita: In those cases, I do consider the gerrit conversations as some historical backing. We do have public logs of these meetings, too :) | 18:25 |
| dansmith | rosmaita: yeah, we could add a new thing, we just already have "best effort" but however we make it clear is fine I guess | 18:25 |
| gmann | because 1. it is not yet released 2. not setup in infra/devstack | 18:25 |
| JayF | I am +1 to the idea that it'd be silly to not test against Ubuntu 24.04 once we have it available. I do not thing many people outside of this meeting care about how we document that fact. | 18:26 |
| gmann | making that as target for next cycle seems like difficult. | 18:26 |
| JayF | s/thing/think/ | 18:26 |
| dansmith | gmann: "best effort" means it could be zero if we don't have time or something major comes up that makes it hard, IMH | 18:26 |
| JayF | ++ | 18:26 |
| clarkb | one upside to makring it best effort is it shows the effort would be apprecaited | 18:26 |
| gmann | dansmith: sure but adding something which does not exist yet makes me not comfortable :) | 18:26 |
| clarkb | then peopel know they can push changes for it without being shot down because it doesn't match the doc | 18:26 |
| JayF | clarkb: that's a good perspective I hadn't considered | 18:26 |
| gmann | and I see less push back of updating runtime for best effort | 18:27 |
| jamespage | in the interests of preserving my own downstream sanity I would like to put effort in to making that happen - 24.04 is already consumable via the development release so I can start looking at that sooner rather than later | 18:27 |
| frickler | well we do have a general statement somewhere that we do try to support the latest Ubuntu LTS release | 18:27 |
| gmann | as you mentioned, best effort is optoinal thing so we might not see push back on this if we add later | 18:27 |
| clarkb | fwiw I'm actively trying to clear out old distro releases from nodepool to make room for new ones like ubuntu 24.04 | 18:27 |
| frickler | so that will kind of kick in automatically once it is released | 18:27 |
| JayF | jamespage+++++ like I said in my gerrit comments; I'd rather you all be doing that work upstream than duplicating it downstream :D. I know many times support is not a choice, we're just choosing the venue. | 18:28 |
| dansmith | frickler: I'm just kinda going on past history where people were, um, not pleased to see that changed mid-cycle | 18:28 |
| jamespage | completely concur - holding patches downstream is 100% overhead | 18:28 |
| fungi | we've definitely added ubuntu lts versions while they were still rc in the past, so it's not impossible | 18:28 |
| gmann | best effort or adding non voting python versions are the possible ways to add in middle of the cycle also and advance preparation to make them mandatory in future cycle | 18:28 |
| fungi | though i think openstack has avoided depending on them, we offered them more as a preview | 18:29 |
| frickler | as I said, do make patches in infra and devstack, once that works, we can add it to testing runtime, but now earlier IMO | 18:29 |
| frickler | s/now/not | 18:29 |
| gmann | dansmith: but those were for changing our mandatory expectation of voting testing or so | 18:29 |
| dansmith | gmann: yeah I know | 18:29 |
| gmann | doing best effort of nv python 3.12 should be ok | 18:29 |
| JayF | I am borderline -1 to saying best effort of python 3.12 | 18:30 |
| gmann | frickler: ++ | 18:30 |
| JayF | because we already know that is extremely unlikely to be supported | 18:30 |
| slaweq | frickler++ | 18:30 |
| JayF | see my note on Feb 23 here https://review.opendev.org/c/openstack/governance/+/908862/1/reference/runtimes/2024.2.rst#65 | 18:30 |
| dansmith | JayF: the problem is that we need to get the fails in front of people soon | 18:30 |
| gmann | yeah that is why we are not adding it right. same with 24.04 also, once it is released then we can ad | 18:30 |
| gmann | add | 18:30 |
| dansmith | because things like taskflow are using removed-from-3.12 things currently AFAIK | 18:30 |
| JayF | dansmith: this fails in a way that is unlikely to fail in unit tests, and more likely to fail IRL | 18:31 |
| frickler | you could add a community goals to support py3.12 instead? | 18:31 |
| JayF | dansmith: but as long as we ensure that information is aggregated, I'm OKish with it | 18:31 |
| JayF | frickler: https://review.opendev.org/c/openstack/governance/+/902585 is mostly what this is | 18:31 |
| JayF | frickler: or at least, the largest identified prereq so far | 18:31 |
| gmann | 24.04 is coming in April end or so right so once it is up and we have devstack patches/setup there then we can add? that might be early in the cycle not so late. | 18:32 |
| gmann | dansmith: ^^ does that work for brining failure early so that project can fix | 18:32 |
| dansmith | I honestly don't understand the concern here so I guess I'll just sit on the side and wait | 18:32 |
| frickler | JayF: that's a subset where we don't know how much we dont know yet? | 18:32 |
| JayF | frickler: we do know that the sslutils module in oslo_service is 100% nonworking in python3.12 | 18:32 |
| JayF | frickler: because the entire ssl.wrap_socket() interface is gone | 18:32 |
| jamespage | I did some patches somewhere else to rework code around that interface removal | 18:33 |
| frickler | ok, so if we know that things aren't working with py3.12, how can we even discuss making it required, if only as best-effort? | 18:33 |
| JayF | frickler: that's exactly what I'm trying to say ++ | 18:33 |
| frickler | adding jobs that will just fail all the time makes no sense at all | 18:34 |
| JayF | or that will pass and give false sense of security | 18:34 |
| gmann | yes | 18:34 |
| JayF | which is the more likely outcome given how sslutils is used | 18:34 |
| JayF | So I have a suggestion for getting past the deadlock | 18:34 |
| gmann | I think let's vote on the current change, please do -1 if you think 24.04 in best effort is blocking. | 18:35 |
| dansmith | taskflow literally can't even be imported in 3.12 AFAIK | 18:35 |
| gmann | vote on gerrit i mean | 18:35 |
| JayF | We should land the change, as-is, then post followups: one by someone who cares about Ubuntu 24.04 best effort support, and one by someone who cares about Pythin 3.12 best effort support | 18:35 |
| JayF | and we can vote opn those ideas separately | 18:35 |
| gmann | and if there are many objection then we can have a another version with 24.04 as best effort and vote there | 18:35 |
| JayF | that's the easiest way for us to get to a completed document without mixing the matrix of concerns | 18:35 |
| gmann | that way we can progress on that. as next cycle setup is coming soon we should be ready with runtime | 18:36 |
| dansmith | that's a little tilted in favor of the people who don't want that right? :) | 18:36 |
| frickler | and these followup should IMO be backed by working jobs | 18:36 |
| dansmith | how about as rosmaita said, some sort of statement of "we're going to add 24.04 to this list mid-cycle, so just assume it's coming" | 18:36 |
| JayF | dansmith: That's a reasonable point; I'm just thinking from a "how can we get clean up/down votes" perspective | 18:36 |
| frickler | dansmith: well we don't know when support will actually be implemented | 18:37 |
| frickler | so it may be mid-cycle or much later | 18:37 |
| dansmith | I mean "in the cycle" | 18:37 |
| dansmith | tbh, I don't know when we're going to start with 3.12 unit/functional jobs that fail obviously for people if not as soon as we have a distro that can run them | 18:38 |
| frickler | but much later likely would make more sense to defer to 2025.1 then. so no need to announce now, is my reasoning | 18:38 |
| gmann | "24.04 testing as best effort if it is released and testing setup is completed before m-2 of the 2024.2 cycle" | 18:38 |
| JayF | "OpenStack plans to implement devstack support for Ubuntu 24.04 when released during the 2024.2 cycle. When this is complete, we will offer best efforts to support Ubuntu 24.04 with available time." | 18:38 |
| gmann | how about that ^^ | 18:38 |
| JayF | dansmith: 24.04 is python 3.12? | 18:38 |
| JayF | dansmith: that's a detail I had missed until this moment | 18:38 |
| fungi | yes | 18:38 |
| jamespage | it is | 18:38 |
| gmann | doing these things after m-2 is little hectic | 18:38 |
| dansmith | JayF: that's why I'm so interested in it being on the list, yes :) | 18:38 |
| dansmith | not for any other reason really | 18:38 |
| JayF | oof, then yeah, there's no point in doing it for D | 18:38 |
| JayF | because we can't get python 3.12 working in D unless we think we can eventlet-migrate a large amount of things in one cycle | 18:38 |
| dansmith | um, what? | 18:38 |
| JayF | Eventlet does not expose the interfaces, in python 3.12, for us to replace the oslo_services.sslutils module | 18:39 |
| JayF | because they have changed upstream | 18:39 |
| spotz[m] | So eventlet will hold up the move unless folks can work on their migration faster | 18:39 |
| dansmith | so you are thinking everyone is going to migrate off of eventlet before we can support 3.12? I have some bad news for you in that case... | 18:39 |
| JayF | spotz[m]: that's pretty much what I'm asserting, or at least, we have to partially migate bits | 18:39 |
| JayF | dansmith: this is why I whipped up so much of a frenzy around starting the migration effort, when we ID'd this back in Nov 2023 | 18:40 |
| opendevreview | Tim Burke proposed openstack/election master: Tim Burke candidacy for Swift PTL (Dalmation) https://review.opendev.org/c/openstack/election/+/910383 | 18:40 |
| JayF | We have another important issue still on the agenda | 18:41 |
| dansmith | JayF: AFAIK, the issue you're talking about is only for people exposing an eventlet-hosted SSL server socket, which is not everyone | 18:41 |
| gmann | our runtime deadline also says (somewhere in doc but i remember we follow that since starting): define the things to test if they are available before cycle start and not after or in the middle once they are available. that is why ubuntu LTS are always picked up in next cycle even they are released after 1 month of our cycle start | 18:41 |
| dansmith | and I thought the assertion was that eventlet was working to mitigate all these things | 18:41 |
| JayF | dansmith: sslutils is used for more than that, fwiw | 18:41 |
| gmann | with that reason I think adding 24.04 in 2024.2 cycle not good instead considering it in 2025.1. | 18:41 |
| JayF | We made eventlet *install at all* on python 3.12, and fixed things like Queue and Event for python 3.12 | 18:42 |
| JayF | I'm timeboxing this topic to :45 so we have time for Murano | 18:42 |
| dansmith | again, my reasoning for 24.04 as best effort was to get the manageable 3.12 things to the forefront, like the fact that taskflow won't even import and thus a big chunk of glance isn't either | 18:43 |
| spotz[m] | It definitely seems we have eventlet things that need to be accomplished before the move to 3.12 and 24.04 needs to be bumped unless you can run 2 versions of python in it | 18:43 |
| JayF | well, Dan's point about making more people aware of how broken py3.12 is ... it's a good point | 18:43 |
| dansmith | that's really my only reason for wanting 24.04 in there at all this close to its release | 18:44 |
| dansmith | and really only for unit/functional at this point | 18:44 |
| dansmith | anyway, I said I was going to shut up | 18:44 |
| JayF | I'm glad you didn't that's a good point. | 18:44 |
| JayF | I'm closing this topic for now, moving on. I'm not sure what the specific action is, but I think it's unlikely we'll find general consensus and may just need to majority-vote. | 18:45 |
| JayF | #topic Murano Inactivity, and inclusion Caracal release | 18:45 |
| JayF | #link https://review.opendev.org/c/openstack/governance/+/908859/ | 18:45 |
| jamespage | ls | 18:45 |
| JayF | So right now, we have Murano with strong consensus to mark as inactive | 18:46 |
| gmann | only concern in the doc change (i was doing before murano inactive status) was stopping the release for project moving to Inactive after m-2 | 18:46 |
| JayF | However, myself, and some other TC members, felt we shouldn't release it since we know it has a critical unfixed bug and no support | 18:46 |
| gmann | I modified the doc change now with "not to define any release criteria for project going to inactive after m-2 and let release team decide on those" #link https://review.opendev.org/c/openstack/governance/+/908880 | 18:46 |
| gmann | true, i do not think we should release but I agree on the argument about let's release team handle it like they do/did for any other project with broken gate/code | 18:47 |
| JayF | I am OK with that doc as changed... but was OK with it unchanged, too. I'm curious what frickler would think about that. | 18:47 |
| frickler | regarding the critical bug, we'll still have three "supported" releases containing it, so I don't think it will make much difference to have another one | 18:48 |
| JayF | I'll note that mark-murano-inactive change does not appear stacked properly with the modify-timeline patch anymore | 18:48 |
| frickler | I didn't get to read the updated on the above patch yet | 18:48 |
| frickler | *updates | 18:48 |
| JayF | frickler: Well, we didn't know those releases were bugged when we pushed them out. I think that shifts our responsibilities here a little bit. | 18:48 |
| gmann | frickler: please check and if that looks ok, i can rebase Murano inactive status change | 18:48 |
| JayF | Everytime we release an OpenStack project with known issues, or that's barely working, it hurts what "OpenStack" means for everyone :/ | 18:49 |
| gmann | As we are not worried about Murano releaase, I think we should stack up it on top of doc change handling the case of marking inactive after m-2 | 18:49 |
| frickler | we still don't know how critical that bug actually is or whether it is bogus | 18:49 |
| JayF | fungi: Have you taken the actions we discussed the other day? w/r/t opening that bug? | 18:50 |
| fungi | i've urged the new maintainers to consider switching it to public if they don't have time to fix it | 18:50 |
| JayF | frickler: I will say, it's my understanding it *is* a critical bug, and unlikely to be fixed ever. | 18:50 |
| frickler | so that would make the project dead on the spot? | 18:51 |
| fungi | the specific quote from andy was "With the lack of community participation in Murano, this may not get fixed. I hope to look into the code when I get time, but I'm not sure when that might be." | 18:52 |
| frickler | so is this critical enough that we should consider actually pulling all release of murano? | 18:53 |
| JayF | I do not have access to the bug and haven't seen it. fungi may be the only person in the meeting who has | 18:53 |
| JayF | but if it's 1) bad enough to be nonpublic and 2) has no reasonable timeline to fix | 18:53 |
| JayF | that makes me feel like pulling releases is the right thing | 18:53 |
| fungi | i can say that people who have access to that bug indicated their production approach would probably be to disable a significant amount of its functionality | 18:54 |
| JayF | er, not releasing more | 18:54 |
| JayF | I don't know about *pulling* them from pypi that are already oyut | 18:54 |
| slaweq | JayF++ | 18:54 |
| spotz[m] | I agree with Jay here, any release of OpenStack that has a major bug in it hurts OpenStack. We can't help what's already out there except putting a warning in it or trying to find someone who can fix it and maybe backport it which I think currently it's own issue | 18:55 |
| JayF | All this being said: we're past the second milestone, and unless I'm willing to do the work to keep it out of the release, I prefer ultimately leave it up to the releases team who are doing the work | 18:55 |
| frickler | as a release team member I would prefer that decision not to be delegated to me | 18:56 |
| fungi | if there are tc members who are interested in helping either 1. try to find a fix for murano, or 2. provide authoritative guidance to the murano maintainers on how to proceed with disclosure/release activities related to this, i'm happy to subscribe you | 18:56 |
| dansmith | JayF: walked right into that one | 18:56 |
| JayF | frickler: what form should that be in? A separate resolution, if we don't want to change the overall policy? | 18:56 |
| frickler | but with this new information, I would support actually dropping it from the current release as a special case | 18:56 |
| frickler | I still think we should not change the general policy, yes | 18:57 |
| JayF | Do we agree that logistically that takes the form of a resolution? | 18:57 |
| JayF | If so, I'll take an action to write one up. | 18:57 |
| frickler | probably, yes | 18:57 |
| JayF | #action JayF to write resolution removing Murano from Caracal release | 18:57 |
| JayF | We have 3 minutes left; anything else on this? | 18:57 |
| gmann | I do not think we need to write resolution per each cases of such inactive projects. does not our current polocy cover it? | 18:57 |
| gmann | i mean release policy/criteria ? | 18:58 |
| frickler | not the "remove from release after m-2" part I guess? | 18:58 |
| JayF | gmann: current policy is "inactive things can't release past -2", if releases team doesn't want to change that policy so TC continues to be incentivized to evaluate activity earlier, it makes sense to special case | 18:58 |
| JayF | I can understand releases team not wanting to change a policy that could make TC decide, say before the release, to pull something | 18:58 |
| JayF | we're 3 days before RC, we're pretty darn close to doing that now :D | 18:59 |
| gmann | that is my doc change cover the case of inactive after m-2 and let release team decide | 18:59 |
| JayF | > frickler> as a release team member I would prefer that decision not to be delegated to me | 18:59 |
| gmann | or we want to change that to TC need to decide with resolution or so | 18:59 |
| JayF | I am trying to respect that request | 18:59 |
| JayF | hence the TC resolution so we take the decision | 18:59 |
| gmann | I am confused honestly :) | 18:59 |
| gmann | we do not want TC to decide on release of them as proposed in doc change but need resolution for the same | 18:59 |
| JayF | We do not want to change the document for the general case. Murano is a special case: it's not just inactive, it's potentially insecure | 19:00 |
| frickler | I just want the "no release" decision to be murano specific, not a general policy change | 19:00 |
| gmann | ok | 19:00 |
| JayF | And that's time | 19:00 |
| JayF | thanks everyone o/ | 19:00 |
| JayF | #endmeeting | 19:00 |
| opendevmeet | Meeting ended Tue Feb 27 19:00:49 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:00 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/tc/2024/tc.2024-02-27-18.03.html | 19:00 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/tc/2024/tc.2024-02-27-18.03.txt | 19:00 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/tc/2024/tc.2024-02-27-18.03.log.html | 19:00 |
| spotz[m] | That makes sense or we'll be doing this potentially every release | 19:00 |
| slaweq | IMO it should be more like general policy if we will have cases like Murano in the future | 19:01 |
| JayF | slaweq: I think the entire point is: TC screwed up here. We should've ID'd this earlier | 19:01 |
| gmann | frickler: JayF: i think we should cover such insecure case also in doc as there might be other cases in future instead of writing resolution every time | 19:01 |
| JayF | slaweq: We do not want to remove the time-pressure to do it before M-2 by changing the general policy; but we also don't want Murano to release | 19:01 |
| slaweq | JayF true, ut it can happen for many reasons | 19:01 |
| JayF | gmann: this is sticky because the bug isn't public. We don't *know* it's insecure ;) | 19:01 |
| gmann | slaweq: exactly | 19:01 |
| fungi | granted, the vulnerability report only showed up in january | 19:02 |
| JayF | My main concern is getting us to a point where everyone involved is happy, and murano isn't in the release | 19:02 |
| fungi | but the writing was on the wall, sure | 19:02 |
| gmann | 'I mean if any project are not/do not want to fix security bug reported' | 19:02 |
| JayF | I don't care if I have to leap headfirst through a flaming hoop | 19:02 |
| JayF | :D | 19:02 |
| slaweq | I totally agree that it should be before M-2, what I'm saying is that we should have some kind of additional "safety switch" in case like that, when things will pop up after M-2 | 19:02 |
| spotz[m] | So we as the TC need to make the decision to pull Murano from the C release | 19:02 |
| JayF | slaweq: a resolution is cheap, easy, and definitionally a special case | 19:03 |
| slaweq | but if we want to do it with extra document indiviually when thing like that will happen - that's also fine for me | 19:03 |
| gmann | may be adding such case in general polciy which can cover Murano case also now and future cases also will save more time | 19:03 |
| JayF | slaweq: I do not want to trust myself to write a policy general enough to cover all the cases WITHOUT being general enough to be abused | 19:03 |
| slaweq | once we will have such precedence with Murano it will be kind of easy to do it if needed | 19:03 |
| rosmaita | we can split the difference, special resolution to stop murano now, discuss policy at the PTG | 19:03 |
| JayF | that's a great suggestion | 19:03 |
| slaweq | ++ | 19:04 |
| rosmaita | from my point of view, not fixing a security issue is enough to pull from the release | 19:04 |
| JayF | tc-members: Someone should think about volunteering to be next chair, and then help me do PTG research | 19:04 |
| JayF | rosmaita: I agree, I think the sticky bit here is that technically there's no public security bug for us to point at | 19:04 |
| gmann | rosmaita: and that is why making it in general policy is clear case right | 19:04 |
| slaweq | rosmaita++ exactly, and it's no matter what point of the cycle it is | 19:04 |
| slaweq | ok, I'm going AFK, see You o/ | 19:04 |
| rosmaita | i don't think we need to point to a public bug, if someone is really intereseted, they can volunteer to be murano PTL and get access and fix it | 19:05 |
| frickler | one other thing I meant to mention for general discussion: the current delay in getting election changes merged is IMO not a good feedback to candidates, leaving the status of their submission pending for 10 days | 19:05 |
| rosmaita | frickler: i agree, i guess that's a new thing? iirc, patches were merged as soon as eligibility was determined in the past | 19:06 |
| JayF | rosmaita: I think that eligibility checking was a human doing checks | 19:06 |
| spotz[m] | Yeah, as of right now glancing at the repo we have a TC election but I haven't seen any others. | 19:06 |
| gmann | only 1 day left to close the nomination. if needed I volunteer to help as election official to ianychoi , let's see what he says | 19:06 |
| spotz[m] | I think it's just Ian who has reviewed everyone up until today and TOny | 19:06 |
| JayF | ack | 19:07 |
| JayF | I will take time to look over some of these | 19:07 |
| gmann | msged in election channel | 19:07 |
| spotz[m] | gmann that would be a big help | 19:07 |
| JayF | tonyb: elodilles: (apparently this is the channel I share with both of you?) -- do you all want to send out an email to the ML asking for additional reviewers to join the global UM core group for OpenStack? | 19:09 |
| fungi | the election check jobs do check the required information and the job logs provide links to the evidence returned by those queries. some very minor adjustment to the jobs could surface those urls as inline comments in the review so that the election officials don't even need to go dig them out of the log. manual research shouldn't really be needed | 19:10 |
| JayF | I'm glad you said that, because I would've done it manually | 19:10 |
| fungi | the check job log already does try to do a good job of visually calling attention to those query results to make them faster to spot too | 19:11 |
| frickler | fungi: or add them as job artifacts. I did that recenty for similar results from release jobs | 19:12 |
| elodilles | JayF tonyb : i can do that, yes. i'll be on PTO tomorrow (and i am leaving for the day now) but i try to not forget to do that on Thursday | 19:14 |
| JayF | ack; ty | 19:14 |
| fungi | frickler: the up side to inline comments is that you don't even need to follow a link to the job results, so it saves you an additional page visit or two | 19:15 |
| frickler | fungi: yes, but it will also end up as dead link once logs expire | 19:18 |
| opendevreview | Jay Faulkner proposed openstack/governance master: Resolution: Remove Murano from 2024.1 release https://review.opendev.org/c/openstack/governance/+/910434 | 19:22 |
| fungi | so do test results, for that matter | 19:23 |
| opendevreview | Jay Faulkner proposed openstack/governance master: Resolution: Remove Murano from 2024.1 release https://review.opendev.org/c/openstack/governance/+/910434 | 19:23 |
| fungi | though the links shouldn't end up being dead? | 19:23 |
| JayF | log links time out very quickly | 19:23 |
| JayF | especially on rackspace cloud | 19:23 |
| JayF | I've had log links from CI jobs disappear in <1wk in some cases from rax jobs | 19:24 |
| fungi | they'd be 1. url to the candidate's foundation profile, 2. url to a gerrit query for the candidates merged changes over a specific timeframe | 19:24 |
| JayF | those would be links that don't expire :D | 19:24 |
| fungi | er, actually #2 is a url to a specific gerrit change owned by the candidate merged within the required timeframe, sorry | 19:24 |
| fungi | and for a deliverable repo of the team they've nominated to be ptl of | 19:25 |
| opendevreview | Jay Faulkner proposed openstack/governance master: Resolution: Remove Murano from 2024.1 release https://review.opendev.org/c/openstack/governance/+/910434 | 19:28 |
| clarkb | now that the opendev team meeting is over I wanted to make two observations re the python3.12 and ubuntu 24.04 discussion. The first is that we seem to be in a logjam over whether we should describe where we want to be or where we think we will be at the end of the next cycle. I think the document can be written to capture both pieces of info | 19:43 |
| clarkb | I'm not hearing any objections to people working on python3.12 support and ubuntu 24.04 support. But there is concern that it is unreasonable to expect that across the board. | 19:44 |
| clarkb | I do think there is value in saying "please help us work on this, we want this to be functional" | 19:44 |
| clarkb | and also "due to problems with dependencies (eventlet) we think it is unlikely that universal python3.12 support will be present in the release" | 19:44 |
| *** elodilles is now known as elodilles_pto | 19:46 | |
| clarkb | Then separately the confusion around all of the moving parts and their relationships to get python3.12/Ubuntu 24.04 support in makes me rewonder if it would be a good idea to try and have some lightweight project management within openstack for these larger goals/issues | 19:46 |
| frickler | I was actually having a similar idea of making something like a "known issue" section and mention py3.12 and the needed work there? | 19:46 |
| clarkb | someone who can organize the dependency tree and keep track of where we are in accomplishing things. That way end users and devs both can follow along to track progress. I'm sure there are other issues that could use similar information tracking | 19:47 |
| clarkb | frickler: ya I think that would work well | 19:47 |
| fungi | does seem like it can be made to work. i mean, debian and ubuntu are both packaging openstack services for platforms with python3.12 as the default python3. but there are corner cases unsolved still i'm sure | 19:48 |
| gmann | clarkb: frickler yeah, that is why it is good idea to move this effort as community wide goal in term of fixing as a group/tracking etc | 19:51 |
| clarkb | gmann: I think that is orthogonal | 19:52 |
| clarkb | YOu should absolutely make the statement in the platform support document beacus that is where people will look for this info | 19:52 |
| clarkb | then if you need a community wide goal to drive the work you do that too | 19:52 |
| JayF | frickler: I really like that idea | 19:53 |
| clarkb | part of the problem here is it appears we've got contributors that feel they need to ask permission to work on this stuff | 19:53 |
| JayF | and I agree re: saying our platform is X and organizing the work to make our platform "x" (a goal) is separate | 19:53 |
| clarkb | we know we want to get there eventually. Lets make that explicit and solve the permission problem | 19:53 |
| gmann | clarkb: yeah, i am trying to do that in runtime doc. | 19:54 |
| clarkb | frickler's suggestion also neatly captures the idea that this stuff only happens when people invest in making it happen | 19:54 |
| clarkb | which I hope is a good way to encourage people to help | 19:55 |
| frickler | JayF: I don't want you the overload the gate email, but if you could also drop the https://review.opendev.org/q/topic:%22vwx-unmaintained%22 URL in the unmaintained context, that might be nice, too | 19:57 |
| frickler | and with that I'm really done for today, cheers | 19:57 |
| JayF | o/ | 19:57 |
| opendevreview | Ghanshyam proposed openstack/governance master: Define testing runtime for 2024.2 release https://review.opendev.org/c/openstack/governance/+/908862 | 20:55 |
| opendevreview | Ghanshyam proposed openstack/governance master: Define testing runtime for 2024.2 release https://review.opendev.org/c/openstack/governance/+/908862 | 20:56 |
| opendevreview | Ghanshyam proposed openstack/governance master: Define testing runtime for 2024.2 release https://review.opendev.org/c/openstack/governance/+/908862 | 20:58 |
| gmann | tc-members ^^ added note about 24.04 testing, please check if this version looks ok | 20:59 |
| opendevreview | Merged openstack/election master: Adding Michael Johnson candidacy for Designate https://review.opendev.org/c/openstack/election/+/910367 | 22:12 |
| opendevreview | Merged openstack/election master: Adding Gregory Thiemonge candidacy for Octavia https://review.opendev.org/c/openstack/election/+/910362 | 22:15 |
| opendevreview | Merged openstack/election master: Add Dmitriy Rabotyagov candidacy for Vitrage PTL https://review.opendev.org/c/openstack/election/+/910328 | 22:15 |
| opendevreview | Merged openstack/election master: Add Artem Goncharov candidacy for TC https://review.opendev.org/c/openstack/election/+/910327 | 22:18 |
| opendevreview | Merged openstack/election master: Add Dmitriy Rabotyagov candidacy for TC https://review.opendev.org/c/openstack/election/+/910325 | 22:19 |
| opendevreview | Merged openstack/election master: Tim Burke candidacy for Swift PTL (Dalmation) https://review.opendev.org/c/openstack/election/+/910383 | 22:22 |
| opendevreview | Merged openstack/election master: Adding Axel Vanzaghi candidacy for Mistral https://review.opendev.org/c/openstack/election/+/909901 | 22:23 |
| opendevreview | Merged openstack/election master: Add suzhengwei candidancy for Masakari https://review.opendev.org/c/openstack/election/+/910205 | 22:23 |
| opendevreview | Merged openstack/election master: Slawek Kaplonski candidacy for TC https://review.opendev.org/c/openstack/election/+/910154 | 22:23 |
| opendevreview | Merged openstack/election master: Add jamespage TC candidacy https://review.opendev.org/c/openstack/election/+/910249 | 22:24 |
| opendevreview | Merged openstack/election master: Erno Kuvaja for Telemetry PTL https://review.opendev.org/c/openstack/election/+/910226 | 22:24 |
| opendevreview | Merged openstack/election master: Adding Jon Bernard candidacy for Cinder PTL https://review.opendev.org/c/openstack/election/+/910024 | 22:40 |
| opendevreview | Merged openstack/election master: Adding Tatiana Ovchinnikova candidacy for Horizon PTL https://review.opendev.org/c/openstack/election/+/910272 | 22:40 |
| opendevreview | Merged openstack/election master: Add Pierre Riteau's candidacy for Blazar PTL https://review.opendev.org/c/openstack/election/+/909964 | 22:40 |
| opendevreview | Merged openstack/election master: Adding Riccardo Pittau candidacy for Ironic PTL https://review.opendev.org/c/openstack/election/+/909534 | 22:40 |
| opendevreview | Merged openstack/election master: Add Goutham Pacha Ravi candidacy for TC https://review.opendev.org/c/openstack/election/+/909239 | 22:40 |
| opendevreview | Merged openstack/election master: Adding Amy Marrich candidacy for TC https://review.opendev.org/c/openstack/election/+/909112 | 22:40 |
| opendevreview | Merged openstack/election master: Adding songwenping candidacy for Cyborg https://review.opendev.org/c/openstack/election/+/909313 | 22:41 |
| opendevreview | Merged openstack/election master: Add Hao Wang candidancy for Zaqar PTL https://review.opendev.org/c/openstack/election/+/909562 | 22:41 |
| opendevreview | Merged openstack/election master: [2024.2] Propose dalees candidacy for Adjutant https://review.opendev.org/c/openstack/election/+/909803 | 22:41 |
| opendevreview | Merged openstack/election master: Adding Hemanth Nakkina candidacy for Sunbeam https://review.opendev.org/c/openstack/election/+/909944 | 22:43 |
| opendevreview | Merged openstack/election master: Adding Takashi Kajinami candidacy for Storlets PTL https://review.opendev.org/c/openstack/election/+/910137 | 22:44 |
| opendevreview | Merged openstack/election master: Adding Takashi Kajinami candidacy for Heat PTL https://review.opendev.org/c/openstack/election/+/910138 | 22:44 |
| opendevreview | Merged openstack/election master: Adding Takashi Kajinami candidacy for Puppet OpenStack PTL https://review.opendev.org/c/openstack/election/+/910136 | 22:46 |
| opendevreview | Merged openstack/election master: Adding Eric Zhang candidacy for Venus https://review.opendev.org/c/openstack/election/+/909570 | 22:46 |
| opendevreview | Merged openstack/election master: Add Martin Kopec candidacy for QA 2024.2 PTL https://review.opendev.org/c/openstack/election/+/909871 | 22:49 |
| opendevreview | Merged openstack/election master: Adding Yasufumi Ogawa candidancy for Tacker https://review.opendev.org/c/openstack/election/+/910233 | 22:49 |
| opendevreview | Merged openstack/election master: Sylvain Bauza candidacy for Nova 2024.2 PTL https://review.opendev.org/c/openstack/election/+/910235 | 22:49 |
| opendevreview | Merged openstack/election master: Add Vladimir Kozhukalov candidacy for Openstack-Helm 2024.2 PTL https://review.opendev.org/c/openstack/election/+/909067 | 22:49 |
| opendevreview | Merged openstack/election master: Sylvain Bauza candidacy for TC https://review.opendev.org/c/openstack/election/+/909076 | 22:49 |
| opendevreview | Merged openstack/election master: [2024.2] Add mnasiadka candidacy for Kolla https://review.opendev.org/c/openstack/election/+/909078 | 22:49 |
| opendevreview | Merged openstack/election master: Andriy Kurilin (andreykurilin) PTL Candidacy for Rally https://review.opendev.org/c/openstack/election/+/909079 | 22:49 |
| opendevreview | Merged openstack/election master: Add Hongbin Lu Candidacy for Zun https://review.opendev.org/c/openstack/election/+/909345 | 22:49 |
| opendevreview | Merged openstack/election master: Add Artem Goncharov candidacy for OpenStackSDK PTL https://review.opendev.org/c/openstack/election/+/909450 | 22:49 |
| opendevreview | Merged openstack/election master: Add Brian Haley (haleyb) candidacy for Neutron PTL https://review.opendev.org/c/openstack/election/+/910263 | 22:49 |
| opendevreview | Merged openstack/election master: Add Carlos da Silva candidacy for Manila PTL https://review.opendev.org/c/openstack/election/+/910259 | 22:49 |
| JayF | jamespage: noonedeadpunk: Do you have any way of raising the priority of https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/2026757 -- this is actively breaking Ironic's gate because the older, unbroken package we pin to was pulled from mirrors | 22:57 |
| JayF | We are currently working around via a manual build ( https://review.opendev.org/c/openstack/ironic/+/888121/8/devstack/lib/ironic#3426 ) but that is obviously non-ideal | 22:57 |
| JayF | and before I make a PPA to fix the issue and blog about it, I wanted to see if you all could just ... help me get it fixed? | 22:58 |
| opendevreview | Merged openstack/election master: Remove retired OpenStack-Chef from election https://review.opendev.org/c/openstack/election/+/910261 | 23:05 |
| fungi | JayF: any idea whether anyone attempted a git bisect of upstream sources to figure out what commit introduced the issue? or is reproduction so involved as to render such tactics impractical? | 23:34 |
| JayF | This has been ongoing a while, we punted it (out of sight out of mind) and now it's coming back | 23:34 |
| JayF | AIUI it's fixed in the next version | 23:34 |
| JayF | but because it's LTS, they won't bump package version by policy | 23:35 |
| JayF | and there's dispute which of two commits actually fixes the issue at hand ... and it sorta puttered out there | 23:35 |
| fungi | right, though they could backport a patch in theory, if a working one were identified | 23:35 |
| JayF | I guess? I know probably less about how Ubuntu manages that sorta thing than any other distribution in common use. | 23:35 |
| fungi | but also, it's possible jammy-backports would be an acceptable place to have a full backport of the n version | 23:36 |
| JayF | Yeah -- my question is mainly trying to get ubuntu experts to tell me a way to fix it that is not just for Ironic CI, but that I could point deployers at, too | 23:36 |
| JayF | ideal fix is invisible to LTS; but my preference is anything that doesn't lead to me doing ad-hoc build infra to avoid telling all our users to compile dnsmasq ;) | 23:37 |
| fungi | right, if it's a candidate for inclusion in jammy-backports then that's not too hard of a solution for downstream consumers | 23:37 |
| JayF | TheJulia: fyi ^ | 23:37 |
| fungi | just enable backports in their apt sources and install dnsmasq/jammy-backports instead | 23:37 |
| clarkb | https://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2022q3/016562.html did identify a likely culprit fwiw | 23:37 |
| fungi | though it sounded like an ubuntu package maintainer tried backporting that patch to the version in jammy to no avail (problem still recurred)? | 23:38 |
| JayF | Yeah, which is where the dispute comes in about how it's fixed | 23:38 |
| TheJulia | I quite literally have no spoons to follow it beyond do the immediate needful to unblock the work infront of me. | 23:39 |
| JayF | TheJulia: ++ understood just didn't want you to miss context since it's also what you're working on | 23:39 |
| JayF | TheJulia: I appreciate your short-term fixing while I try to scan for medium/long term :D | 23:39 |
| TheJulia | I think the cause and fixes were identified based upon dnsmasq commit/change logs and since we're not the only ones who have spotted it | 23:40 |
| TheJulia | Anyway, back to the needful short term | 23:40 |
| JayF | If it's a reproducable failure, we just rigged up a source-built-reproducation-machine in Ironic; I can probably branch off that once merged and try a few patches in our CI | 23:40 |
| JayF | see what combo fixes it | 23:41 |
| fungi | oddly, https://packages.ubuntu.com/dnsmasq would seem to indicate the versions in jammy and noble are both based on 2.90 | 23:41 |
| JayF | that is likely the ideal solution | 23:41 |
| fungi | package changelog indicates the version in jammy was updated to 2.90 two weeks ago to address a couple of security vulnerabilities | 23:43 |
| TheJulia | oh, well, hmm | 23:43 |
| JayF | hmmmmmmm | 23:43 |
| TheJulia | we could likely just backout the change anyhow | 23:43 |
| fungi | http://changelogs.ubuntu.com/changelogs/pool/main/d/dnsmasq/dnsmasq_2.90-0ubuntu0.22.04.1/changelog | 23:44 |
| JayF | fungi: look at you, fetching fresh context when we're working on a copy of month old context | 23:44 |
| JayF | A+++++ | 23:44 |
| * TheJulia turns off ipv6 because some intermediate carrier is randomly dropping v6 to review.opendev.org | 23:44 | |
| JayF | TheJulia: I'll stack a removal on top of your existing patch, if you want? | 23:44 |
| JayF | happy to let you do it too | 23:44 |
| TheJulia | JayF: if you could that would be awesome, I have to backport something down to wallaby before I call it a day | 23:45 |
| JayF | yep, should be ezpz | 23:45 |
| fungi | TheJulia: probably zayo (former abovenet backbone), vexxhost has been having no end of trouble with them not relaying ipv6 recently | 23:45 |
| TheJulia | fungi: fun fun! | 23:53 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!