| clarkb | fungi: left a response to one of your comments on the git-review change that are probably worth checking before you make any updates | 00:26 |
|---|---|---|
| clarkb | just to make sure we're on the same page as far as corner case handling needs go | 00:26 |
| opendevreview | Takashi Kajinami proposed openstack/project-config master: Retire puppet-sahara: End Project Gating https://review.opendev.org/c/openstack/project-config/+/910452 | 01:54 |
| opendevreview | Takashi Kajinami proposed openstack/project-config master: Retire puppet-sahara: Remove Project from Infrastructure System https://review.opendev.org/c/openstack/project-config/+/910455 | 02:01 |
| tkajinam | I wonder if anyone has idea about the failure in https://zuul.opendev.org/t/openstack/build/285ea9b517304061ab37510a9b850a49 ? | 02:14 |
| tkajinam | subprocess.CalledProcessError: Command '/bin/bash -c "mkdir -p /afs/.openstack.org/docs/releasenotes/trove/ && /usr/bin/rsync -rtp --safe-links --delete-after --out-format='<<CHANGED>>%i %n%L' --filter='merge /var/lib/zuul/builds/285ea9b517304061ab37510a9b850a49/work/tmp/tmpa5slxi7n' /var/lib/zuul/builds/285ea9b517304061ab37510a9b850a49/work/docs/ /afs/.openstack.org/docs/releasenotes/trove/"' returned non-zero exit status 23. | 02:14 |
| tonyb | tkajinam: I think the root cause is https://zuul.opendev.org/t/openstack/build/285ea9b517304061ab37510a9b850a49/log/job-output.txt#154-155 ... Why the appropriate target dirs don't exist I don't know | 03:47 |
| tonyb | Error 23 from rsync means "Partial transfer due to error" | 03:47 |
| tkajinam | the file is not present in trove itself and I guess it comes from openstackdocstheme | 03:49 |
| tkajinam | I wonder if we can rerun that task somehow to see if that is a transient problem (like short network glitch). | 03:49 |
| tkajinam | or probably we can leave it for now and see it when we create rc release ? | 03:50 |
| tonyb | tkajinam: I believe I can re-enqueue that job, but I've never done it before. So I'd like to check with another infra-root for possible side-effects | 03:51 |
| tonyb | tkajinam: So it can be done tomorrow, and/or whenenver the next patch lands that has a releasenote | 03:53 |
| tkajinam | tonyb, ok, thanks. it's not urgent. It may be enough if we can agree that the problem is likely a transient one, and we can look into it in detail if the same problem is reproduced again. | 03:56 |
| tonyb | tkajinam: Okay. I'll let you know what (if anything happens) | 03:57 |
| tkajinam | tonyb, ack. thanks again :-) | 03:58 |
| frickler | tonyb: tkajinam: that type of failure does happen if two releasenote jobs for the same project run in parallel. see https://zuul.opendev.org/t/openstack/build/49d09944f27948e7bb12c77049fdc643 which ran at the same time | 06:15 |
| frickler | in general the issue should resolve once the next releasenote update is pushed, since the whole content gets regenerated | 06:16 |
| tkajinam | frickler, ah, ok. that makes very clear sense | 07:07 |
| tkajinam | I see a stable/zed patch was merged along with that master patch in a short span so that was the trigger of the collision, I guess | 07:08 |
| tonyb | frickler: Thanks | 07:13 |
| frickler | tonyb: if you still have a moment, https://review.opendev.org/c/openstack/project-config/+/910270 would unblock further tripleo retirement | 07:43 |
| tonyb | frickler: Looking. | 08:15 |
| tonyb | frickler: Are there docs on setting up my account so I can do AFS quota bumps? | 08:16 |
| frickler | tonyb: https://docs.opendev.org/opendev/system-config/latest/afs.html but as I said some time ago not sure how up to date that is | 08:24 |
| opendevreview | Merged openstack/project-config master: Retire TripleO: remove project from infra https://review.opendev.org/c/openstack/project-config/+/910270 | 08:31 |
| tonyb | frickler: Okay thanks. I'll work through that. | 08:32 |
| noonedeadpunk | hey folks, I've spotted a weird thing is happening with gerrit for some time, ragarding Owner/Author/Committer fields. | 09:09 |
| noonedeadpunk | Like I'm pretty sure, that Owner was representing a Gerrit user, Author (or Commutter) was whatever is in commit itself | 09:10 |
| noonedeadpunk | This way it was possible to distinguish organisation/personal/etc commits, depending on what's in your local git config for the repo | 09:10 |
| noonedeadpunk | But now it's all just gerrit users | 09:11 |
| noonedeadpunk | Ie https://review.opendev.org/c/openstack/election/+/910476 - my local `git history` looks correct/different : https://paste.openstack.org/show/bhr9EfFlkE3Koh3wsSfA/ | 09:11 |
| noonedeadpunk | So basically commit author is not respected anymore, which is weird/confusing/bug? | 09:12 |
| noonedeadpunk | and ofc if I donwload patch from gerrit, I still see right author. | 09:13 |
| tonyb | They look the same to me? the review you posted has the same address the paste you shared? | 09:13 |
| noonedeadpunk | so feels like a bug I guess.... | 09:13 |
| noonedeadpunk | um.... | 09:14 |
| noonedeadpunk | So you see author as <dmitriy.rabotyagov@cleura.com> ? | 09:14 |
| tonyb | noonedeadpunk: Yes | 09:14 |
| noonedeadpunk | lol | 09:14 |
| noonedeadpunk | https://ibb.co/5YZyzBN | 09:15 |
| noonedeadpunk | all 3 fields for my view looks same and like this | 09:16 |
| noonedeadpunk | so defenitely a bug in gerrit... | 09:16 |
| tonyb | https://ibb.co/QJ2dPxD | 09:16 |
| noonedeadpunk | yeah, ok | 09:17 |
| noonedeadpunk | thanks for checking on that | 09:17 |
| noonedeadpunk | now it's at least a bit more clear | 09:17 |
| tonyb | noonedeadpunk: It's probably due to changes gerrtt made to 'hide' secondary addresses. gerrot will now only show/search with primary addresses | 09:17 |
| noonedeadpunk | yeah, probably. but it's weird still, as confuses patch owner a lot kinda | 09:18 |
| noonedeadpunk | like having different view for all ppl and owner is not perfect change... | 09:18 |
| noonedeadpunk | anyway | 09:18 |
| noonedeadpunk | good it preserves information | 09:18 |
| tonyb | noonedeadpunk: Yeah If the root cause matches my hunch it's a little annoying | 09:20 |
| noonedeadpunk | oh wait... does it mean that contribution stats per organization can be borked now? | 09:20 |
| noonedeadpunk | like if think about contractors that might perform work for hire for different orgs | 09:20 |
| noonedeadpunk | but probavbly bitegria does not care much about gerrit... | 09:24 |
| noonedeadpunk | and in git author is stored correctly | 09:25 |
| noonedeadpunk | yeah, so slightly annoying, but should not break anything for real | 09:25 |
| noonedeadpunk | ok, cool, thanks for helping out! | 09:25 |
| frickler | noonedeadpunk: tonyb: there was some discussion about this some days ago. iirc there was a related change in gerrit to make secondary addresses more private, since they are not intended to be shown to the public. so the view changes when the account owner is logged in (and thus has access to those addresses and gerrit can all associate them to the same account) vs. for other users | 09:43 |
| noonedeadpunk | now it's slightly vice versa. Also it's kind of pointless to hide an email, that will be published as commit author later on | 09:57 |
| frickler | iiuc the effect is not to hide the email itself, but its association with your account. but I'm also not trying to defend that change, maybe clarkb wants to take this up with the gerrit community | 10:02 |
| noonedeadpunk | But um... You still have change-id in the commit msg... Or it's only git-review thingy? | 10:03 |
| noonedeadpunk | As Iguess not? | 10:03 |
| noonedeadpunk | So you kinda take Change-ID, go to gerrit and voila... | 10:03 |
| noonedeadpunk | Anyway | 10:03 |
| fungi | i think it's only semi-intentional on gerrit's part | 12:57 |
| fungi | basically, there is an (admin) permission to be able to see the secondary addresses associated with an account | 12:57 |
| fungi | when a non-administrator looks at a change that mention's someone else's secondary address, the gerrit webui operating as the other user is unable to query the account associated with that address due to lack of permission | 12:59 |
| fungi | this is similar to how, if a non-admin user looks at a project acl in the gerrit webui, it only shows them the parts of the acl which grant permissions to them because the ui is operating as that user and can only successfully query things that user can access | 13:01 |
| Clark[m] | Yes, I believe the motivation is to avoid leaking account details since you can push a change with arbitrary authorship info and use that to brute force info out of Gerrit. The commit change id to Gerrit change mapping doesn't change this | 14:32 |
| Clark[m] | Gerrit is unlikely to change this behavior as a result. They would likely tell us to give registered users read access to secondary emails. But they removed that intentionally and my inclination is to be cautious there | 14:33 |
| clarkb | I'm going to approve the gitea 1.21.7 upgrade if there are no objections in the next few minutes | 16:08 |
| fungi | sounds great, i'll be around | 16:12 |
| clarkb | gitea change has been approved | 16:21 |
| fungi | thanks | 16:24 |
| clarkb | I'm going to approve the change to remove debian buster nodesetse now | 17:08 |
| clarkb | from opendev/base-jobs | 17:08 |
| fungi | go for it | 17:08 |
| opendevreview | Merged opendev/base-jobs master: Drop debian-buster nodesets https://review.opendev.org/c/opendev/base-jobs/+/910015 | 17:18 |
| clarkb | https://review.opendev.org/c/openstack/project-config/+/910030 should be ready for review/approval as soon as zuul reports back now that ^ has merged | 17:21 |
| clarkb | there are no debian-buster nodes in nodepool according to nodepool list | 17:22 |
| clarkb | we should be good to drop the image uploads and then when those are cleaned up drop the image builds entirely | 17:22 |
| frickler | noticed in the devstack propose-updates job that https://opendev.org/x/fuel-plugin-onos is giving a 500, maybe someone can have a look in gitea? | 17:23 |
| clarkb | frickler: can you link to the failure? | 17:24 |
| frickler | https://zuul.openstack.org/build/be70a176e27e45c5acb34d259c659237 for example. the main issue is just the 500 response for the above url though | 17:25 |
| clarkb | ya but jobs should never talk to gitea? | 17:25 |
| clarkb | I guess there are two problems here 1) why is that a 500 in gitea and 2) why is the job talking to gitea | 17:25 |
| frickler | this one does, because it checks all existing repos for content iiuc. might be possible to rewrite, but that has been in place for 8 years or so | 17:26 |
| frickler | this is the tool https://opendev.org/openstack/devstack/src/branch/master/tools/generate-devstack-plugins-list.py | 17:27 |
| clarkb | frickler: ...ules/context/repo.go:682:RepoAssignment() [E] SyncRepoBranches: Error 1062 (23000): Duplicate entry '1057-Kilo' for key 'UQE_branch_s' | 17:29 |
| frickler | looking at the job history this has actually been going on since early january | 17:29 |
| frickler | so some kind of repo corruption? | 17:30 |
| clarkb | At least as far as gitea is concerned yes. I'm cloning the repo and it hasn't failed yet so Ithink git may be ok with it | 17:30 |
| clarkb | could be something like a tag and a branch with the same name or something like that which is technically valid but may make gitea unhappy? | 17:31 |
| clarkb | fuel has been dead for years at this point. I'm not sure we shoukld invest much time in fixing this | 17:31 |
| clarkb | a cloned copy of the repo checks out clean with `git fsck --verbose --strict` | 17:34 |
| clarkb | looking at the gitea source code it is trying to sync the branches in the git repo with the branches known in the database | 17:37 |
| clarkb | the error 1062 is a mysql/mariadb error | 17:37 |
| clarkb | if I had to guess the db became more strict about duplicate entries in rows either as a result of mysql/mariadb upgrades over time or for gitea schema updates | 17:38 |
| clarkb | I don't know that I'm going to live debug the database right now. But if others want to poke at it I don't have objections | 17:40 |
| fungi | i wonder if there are other repos with the same issue | 17:44 |
| clarkb | can probably determine that with a db query once the table structure is better understood | 17:46 |
| clarkb | ya 1062 occurs when a duplicate value is added to a unique column | 17:48 |
| fungi | maybe the schema is wrong and that column shouldn't be flagged unique | 17:49 |
| clarkb | that could be too. Or it was a new restriction that wasn't enforced until recently | 17:50 |
| clarkb | 6e19484f4d3bf372212f2da462110a1a8c10cbf2 in gitea from end of june 2023 introduced branch synching into the db to reduce the number of git operations required | 17:55 |
| clarkb | so this is relatively new functionality in gitea | 17:55 |
| clarkb | I guess it isn't super surprising that there are bugs in new features | 17:56 |
| clarkb | the error message is interesting because it almost looks like a projectid-branchname pair | 17:56 |
| clarkb | which maybe points to an index? | 17:56 |
| clarkb | ok I think maybe this could explain it: //git's ref-name is case-sensitive internally, however, in some databases (mssql, mysql, by default), it's case-insensitive at the moment | 17:57 |
| clarkb | remotes/origin/Kilo and remotes/origin/kilo exist in that repo | 17:57 |
| clarkb | frickler: ^ fyi I'm like 99% sure that is the issue | 17:57 |
| clarkb | also lol | 17:58 |
| clarkb | they could've made the db table column case sensitive | 17:58 |
| clarkb | so ya this is a bug in gitea | 17:58 |
| fungi | hilarious | 18:00 |
| clarkb | I'll file a bug later today when I get over the "this is just bad" feeling I currently have around it | 18:01 |
| opendevreview | Merged opendev/system-config master: Upgrade gitea to 1.21.7 https://review.opendev.org/c/opendev/system-config/+/909941 | 18:01 |
| clarkb | I feel like this is the sort of thing where strong code review culture pushes back and says no you can't merge this as is you haev to address the fundamental issue you've already identified | 18:05 |
| clarkb | the hourly jobs are running and when those are complete we should do a rolling upgrade of gitea | 18:05 |
| clarkb | looks like newer changes to gitea enforce the branch to db synchronization as git push time | 18:06 |
| clarkb | whcih means new instances of this issue would likely lead to failed syncs to gitea | 18:06 |
| clarkb | the instance frickler identified is from way way back though so we get this behavior instead. Both are suboptimal | 18:06 |
| clarkb | gitea deployments should start soon. The hourly jobs are just finishing up | 18:19 |
| frickler | case handling issue, nice. also regarding the "is this still needed" question: do we want to define a policy to be able to drop seemingly unused content? how would we identify that? how would we archive repos? tarball of .git? | 18:30 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: prepare-workspace-git: Add ability to define synced pojects https://review.opendev.org/c/zuul/zuul-jobs/+/887917 | 18:31 |
| clarkb | frickler: my question was more along the lines of "the git repo is still there and is accessible, the only thing that doesn't work is fancy rendering of the contents" | 18:31 |
| clarkb | frickler: I think we can live with that for repos in this situation | 18:31 |
| frickler | the start of the issue matches with when we upgraded gitea from 1.20 to 1.21 https://review.opendev.org/c/opendev/system-config/+/897679 | 18:34 |
| clarkb | frickler: ya Ithink that is because this db table was added in june 2023 | 18:35 |
| clarkb | and so didn't end up in a gitea release until 1.12 | 18:35 |
| clarkb | *1.21 | 18:35 |
| clarkb | the giteas have upgraded and they look good from the web ui side | 18:36 |
| clarkb | I have also successfully cloned system-config against the updated servers | 18:36 |
| fungi | i think i'd be hard-pressed to come up with a metric for determining when a repo is no longer of historical interest | 18:36 |
| frickler | yeah, that's my issue, too, but seems I just misunderstood the "not sure we shoukld invest much time in fixing this" comment | 18:38 |
| clarkb | right I wasn't suggesting we remove that content. I was saying the functionality that exists is sufficient for retired repos. | 18:38 |
| clarkb | not ideal but sufficient | 18:38 |
| frickler | I guess I'll just put a skip for the specific repo into the devstack script then for now | 18:39 |
| clarkb | frickler: I think all retired repos can be skipped | 18:40 |
| clarkb | and should be? | 18:40 |
| clarkb | there isn't anything we can propose to those repos at this point as they should be read only | 18:40 |
| clarkb | zuul is happy with https://review.opendev.org/q/topic:%22drop-buster%22+status:open now. Reviews on those three changes would be great then I can approve and step through them | 18:41 |
| clarkb | also I said I would do the lodgeit db upgrade today | 18:41 |
| fungi | well, technically retired openstack repos aren't read-only, they have push access for tc members in case there's need for mass updating them for consistency | 18:41 |
| clarkb | fungi: true, but I think the fuel repos should all be read only as they predate that change | 18:42 |
| frickler | hmm. so maybe that script should look at governance data instead of checking all repos. guess I'll have to think about that | 18:42 |
| fungi | but that's only for things still in the openstack/ namespace, right | 18:42 |
| corvus | wait like literal actual push access? | 18:42 |
| clarkb | but even then this job doesn't need to propose to technically not read only openstack/ repos | 18:42 |
| clarkb | corvus: no I think its force merge ability through the ui not git push --force | 18:43 |
| corvus | okay that's cool | 18:43 |
| fungi | corvus: submit access, sorry: https://opendev.org/openstack/project-config/src/branch/master/gerrit/acls/openstack/retired.config#L13 | 18:43 |
| corvus | (my concern is that it should not be easy to accidentally or on purpose update a repo without audit :) | 18:44 |
| frickler | that job only pushes to devstack. it generates a list of all devstack plugins that exist as a reference in the doc | 18:44 |
| corvus | (no matter its lifecycle) | 18:44 |
| fungi | yeah, i mis-spoe | 18:44 |
| fungi | mis-spoke | 18:44 |
| clarkb | frickler: ah I see it is pushing in the other direction. Still don't need to check all those dead repos there | 18:45 |
| fungi | gitea's still working for me now that the deploy job has finished | 18:45 |
| clarkb | looking at MARIADB_AUTO_UPGRADE again I see "If MARIADB_AUTO_UPGRADE is set, and the .my-healthcheck.cnf file is missing, the healthcheck users are recreated if they don't exist, MARIADB_HEALTHCHECK_GRANTS grants are given, the passwords of the healthcheck users are reset to a random value and the .my-healthcheck.cnf file is recreated with the new password populated." in the docs now | 18:46 |
| frickler | so the question is how to identify "dead" when devstack explicitly also wants to include non-governance-covered content | 18:46 |
| clarkb | I'm going to dobule check that on the held node and cross check prod before approving | 18:46 |
| clarkb | I don't think having those users get created is a problem based on the docs, but I want ot make sure I understand if it iwill happen or not | 18:46 |
| clarkb | https://mariadb.com/kb/en/using-healthcheck-sh/ has more info. I think the old 10.4 containers don't do this but the 10.11 containers do? | 18:49 |
| clarkb | so that is telling us it will upgrade not just the base database stuff but the container related healthcheck things that are expected in the db | 18:49 |
| clarkb | again should be fine, but want to undersatnd | 18:49 |
| clarkb | ya prod running 10.4 doesn't have that healthcheck stuff but the held node which upgraded to 10.11 does. Any concerns with that being added as part of this upgrade? | 18:51 |
| clarkb | this is actually a neat feature we could take advatnage of in the future so ya I think this is an improvement and not one we should try to disable | 18:53 |
| fungi | sounds fine to me, no objection. i agree it could be useful | 18:54 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: prepare-workspace-git: Add ability to define synced pojects https://review.opendev.org/c/zuul/zuul-jobs/+/887917 | 18:56 |
| corvus | clarkb: ++ | 18:59 |
| gmann | yeah, TC +2 rights on retired repo is only for openstack/ namespace repo | 19:02 |
| clarkb | corvus: did you want to reack the change before I approve it? | 19:04 |
| corvus | done | 19:08 |
| clarkb | thanks I'll approve that as soon as I'm done with this gitea bug report | 19:17 |
| clarkb | currently trying to figure out how to permalink in the github file browser | 19:18 |
| clarkb | https://github.com/go-gitea/gitea/issues/29477 frickler here is the upstream gitea issue | 19:27 |
| clarkb | maybe double check I scrubbed that log paste properly (pretty sure I did. The only IP in there is 127.0.0.1 and we don't care about the repo names being public) | 19:28 |
| clarkb | I've just approved the lodgeit db upgrade change | 19:33 |
| clarkb | gitea reports that we should just fix this ourslves by updating the db to use a case sensitive collation method. But maybe also gitea 1.22 will fix this... | 19:38 |
| clarkb | my concern is that if we fix this by hand we may end up creating more work for us in the future if their fix conflicts with ours somehow | 19:38 |
| clarkb | ideally we see how they are fixing it and then manually apply that if we need it sooner than 1.22 | 19:38 |
| clarkb | our gitea db seems to default to latin1 character set and latin1_swedish_ci collation. Which I think are defaults. Its amazing that in 2024 we're still fighting this stuff | 19:48 |
| clarkb | that said I think sticking to latin1 for db names (and maybe even branch names) is probably a good idea. Its just the _ci collation that is a problem | 19:49 |
| clarkb | ah nevermind the column is utf8 mb4 | 19:50 |
| clarkb | so gitea must be setting that at a table level but not setting a db default | 19:51 |
| clarkb | wow I feel like I just opened a can of worms with this gitea thing | 20:01 |
| clarkb | not mysql isnt' really supported it just magically works | 20:01 |
| clarkb | and we should be using a case sensitive db instead | 20:01 |
| frickler | I was just reading "gitea doctor convert" ;) | 20:03 |
| opendevreview | Merged opendev/system-config master: Upgrade the lodgeit mariadb to 10.11 https://review.opendev.org/c/opendev/system-config/+/909471 | 20:05 |
| clarkb | that is waiting behind the hourly jobs | 20:11 |
| clarkb | https://paste.opendev.org/show/bf5uxZYIvklE5RlVXvMg/ is a new paste after the database upgrade | 20:23 |
| clarkb | https://paste.opendev.org/show/bWhZZH97IMLv44eeiWlB/ is an old paste that still loasd for me | 20:23 |
| clarkb | also /var/log/containers/docker-mariadb.log lgtm | 20:24 |
| opendevreview | Jeremy Stanley proposed opendev/git-review master: Don't make hook script read-only https://review.opendev.org/c/opendev/git-review/+/910268 | 20:24 |
| opendevreview | Jeremy Stanley proposed opendev/git-review master: Vendor a copy of Gerrit's commit-msg Git hook https://review.opendev.org/c/opendev/git-review/+/910275 | 20:24 |
| clarkb | and the backup of the system tables is small as expceted and shouldn't cause problems going forward | 20:25 |
| fungi | we should be using a case-sensitive db with gitea, except a case-sensitive db can't handle certain git repositories where refs differ only by case... so basically gitea has declared such repositories out of scope? | 20:27 |
| fungi | no, scratch that | 20:27 |
| fungi | a case-sensitive db would be fine | 20:27 |
| fungi | this sb is also capable of being case-sensitive or case-insensitive depending on the schema, right? | 20:28 |
| fungi | so seems like we could be using a case-sentitive db, except gitea configured it to not be | 20:28 |
| clarkb | yes I'm writing a response and getting frustrated | 20:29 |
| clarkb | latest problem is some columns are utf8 with no mb3 or mb4 specifier. I believe that is because they are mb3 and that distinction wasn't solid until 10.6 | 20:29 |
| clarkb | but we're on 10.4 or something | 20:29 |
| fungi | i guess we should put this on hold until we upgrade to mariadb 10.11? | 20:32 |
| clarkb | fungi: not that is orthogonal | 20:39 |
| clarkb | it just means utf8 == utf8mb3 | 20:39 |
| clarkb | and you'd need to convert one way or another | 20:39 |
| opendevreview | Tristan Cacqueray proposed zuul/zuul-jobs master: logjuicer: update to version 0.9.8 https://review.opendev.org/c/zuul/zuul-jobs/+/910549 | 20:40 |
| fungi | clarkb: oh, got it, by "problem" i thought you meant it had surfaced another bug | 20:44 |
| clarkb | sorry no its a problem beacuse it makes understanding this even more complicated than it needs to be | 20:49 |
| clarkb | we have utf8mb4 and utf8 columns in our existing db. Mariadb docs no logner talk about utf8 because it is utf8mb3 now | 20:50 |
| clarkb | I wrote a response that took much too long to write bceuase mysql makes this complicated | 20:51 |
| clarkb | and now lunch | 20:52 |
| clarkb | I identified a new concern in that response which is that something seems to be choosing utf8 character sets because our default is latin1 (this is good), but it implies that even if we chagne database and table defaults gitea may still choose the wrong thing | 20:53 |
| clarkb | then when they add a new table like they did with the branch table we could be right back where we started if they don't fix that | 20:53 |
| clarkb | I tried to call this out | 20:53 |
| clarkb | basically if they don't fix their table and column creation (which may already be done I'm not sure) then we'll run into this problem repetetively | 20:54 |
| opendevreview | Jeremy Stanley proposed opendev/git-review master: Don't make hook script read-only https://review.opendev.org/c/opendev/git-review/+/910268 | 20:55 |
| opendevreview | Jeremy Stanley proposed opendev/git-review master: Vendor a copy of Gerrit's commit-msg Git hook https://review.opendev.org/c/opendev/git-review/+/910275 | 20:55 |
| clarkb | thinking out loud here becuase I fail at eating lunch. I think for the 1.22 upgrade post upgrade we'll want to manually run the doctor command against each backend in sequence | 21:01 |
| clarkb | that should make text and varchar columns utf8mb4 (instead of utf8mb3) and fix the collation | 21:02 |
| clarkb | but I think we should probably punt on this issue until we can use the automated tooling to correct it for us | 21:02 |
| clarkb | we may also need to manually update the default charset and collation on the gitea schema | 21:04 |
| clarkb | this shouldn't convert anything but will act like belts and suspenders when new tables show up | 21:04 |
| clarkb | oh nevermind the doctor command alters the bd | 21:06 |
| clarkb | so ya make a note that the gitea 1.22 upgrade should include some database maintenence. I deally via their doctor command as that should automate the whole thing (I think we'll be fine with utf8mb3 to utf8mb4 conversions as 4 byte is suepr set of 3 byte should just be a matter of time and doing it) | 21:32 |
| clarkb | we will probably want to hold a 1.21 test node and upgrade it to 1.22 manually and figure out how to run the doctor command out of the container as part of that process. All things that can be done when we get to that upgrade | 21:33 |
| clarkb | infra-root anyone have a moment for https://review.opendev.org/c/openstack/project-config/+/910030/ ? I'm hoping I can remove buster from nodepool today | 21:35 |
| clarkb | then maybe tomorrow run the mirror cleanup manual steps | 21:36 |
| fungi | lgtm | 21:38 |
| clarkb | I'll approve that if a nodepool list says there are no buster nodes currently (there shouldn't be) | 21:39 |
| clarkb | and there are not | 21:39 |
| clarkb | fungi: can you also review https://review.opendev.org/c/opendev/system-config/+/910032 it is the change that removes buster mirroring. Less urgent but related | 21:39 |
| fungi | oh, yep | 21:40 |
| fungi | also lgtm | 21:41 |
| clarkb | the git-review updates lgtm as well | 21:48 |
| clarkb | fungi: re None beraking type annotations that feels like a bug in type annotation implementations/static checking. Haskell has a Nothing and Maybe type | 21:48 |
| clarkb | essentially what we're saying is that the type of that value Maybe Nothing or Maybe String | 21:48 |
| opendevreview | Merged openstack/project-config master: Drop debian-buster image uploads from nodepool https://review.opendev.org/c/openstack/project-config/+/910030 | 21:49 |
| fungi | heh, "maybe" | 21:49 |
| clarkb | and that should be easy to express (I fully believe it may not be easy though just saying its a problem with mypy not the code) | 21:49 |
| fungi | i like it | 21:49 |
| clarkb | `nodepool image-list | grep buster` returns an empty list now. I'm going to approve the change that cleans up builds | 22:13 |
| corvus | in rust it's Option, and in python typing, it's Optional. that makes it not any harder than normal python typing, so don't let the tail wag the dog on using None. having said that, please don't take this an an endorsement from me of type checking in python (because it isn't) :) | 22:13 |
| opendevreview | Merged openstack/project-config master: Remove debian-buster image builds from nodepool https://review.opendev.org/c/openstack/project-config/+/910031 | 22:19 |
| clarkb | fungi: for the manual reprepro mirror cleanup steps they seem to leave behind indexes. For example https://mirror.bhs1.ovh.opendev.org/ubuntu-ports/dists/xenial/ | 22:19 |
| clarkb | fungi: should we test cleanup of those indexes by hand after letting reprepro cleaer out the packages? | 22:20 |
| clarkb | and I guess if that works we can update our docuemtnation | 22:20 |
| fungi | yeah, at least see if it ends up recreating them | 22:21 |
| clarkb | ok /me scribbles a note | 22:22 |
| fungi | they're isolated to their own directories so shouldn't be hard to rm -rf | 22:22 |
| clarkb | fungi: for the openafs on arm64 problem how much will I hate myself if I try to report an issue to debian? I really don't want to use the reportbug tool but I can probably do so in a container | 22:24 |
| fungi | the expectation is that reportbug can be run on the problem where you're experiencing the bug, and then either sent directly or copied into a separate e-mail message to the bts | 22:27 |
| clarkb | fungi: would it do anything useful in this case though? | 22:27 |
| fungi | though in this case we're hitting the bug in ubuntu rather than debian, right? | 22:27 |
| clarkb | no this is debian | 22:27 |
| fungi | nodepool builder container image, right | 22:28 |
| fungi | running it from inside the container would be a pain, agreed | 22:28 |
| clarkb | its actually for wheel cache builds I think | 22:28 |
| fungi | oh, test node then | 22:28 |
| clarkb | we have a test case for openafs in system-config to try and catch tehse issues | 22:28 |
| clarkb | whcih is where I saw it, but I think where we hit this in production is wheel cache builds on bookworm for arm | 22:28 |
| clarkb | it is possible ubuntu has the same issue we just haven't updated to a set of kernel and openafs versions that break | 22:29 |
| fungi | anyway, you don't actually have to use reportbug at all, opening a bug against the openafs dkms module package about the build failure and pointing to the upstream commit that fixes it and saying we're encountering that issue on bookworm arm64 ought to suffice | 22:30 |
| clarkb | ah ok that makes it easier | 22:32 |
| clarkb | I'll do that once upstream merges the fix (or a fix) | 22:32 |
| clarkb | buster appears to be removed from nodepool | 22:33 |
| clarkb | looks like I should merge the mirror removal change with the lock for debian mirror updates already held | 22:34 |
| fungi | packaging seems to be maintained in https://salsa.debian.org/debian/openafs so a merge request there would also be an option i expect, though i'd still recommend a bug report either way | 22:34 |
| clarkb | I'll hold off on approving the mirror cleanup until tomorrow when I can give it the attention it likely needs over many hours (I expect that to not be fast but maybe I'll be surprised) | 22:35 |
| fungi | probably everything except the vos release will be fast | 22:36 |
| clarkb | fungi: an MR for that would look like adding a patch file and changelog info to https://salsa.debian.org/debian/openafs/-/tree/master/debian/patches?ref_type=heads to add the patch? | 22:36 |
| fungi | though vos release may also be fast since it shouldn't need to transfer a bunch of new file content, just deletions | 22:37 |
| clarkb | hrm it isn't clear what branch that should be applied to | 22:37 |
| clarkb | probably more than one but which ones :) | 22:37 |
| clarkb | (there is no bookworm branch) | 22:38 |
| fungi | clarkb: yeah, and i didn't see a branch for bookworm when i looked, so they may be handling stable updates separately/by hand anyway | 22:38 |
| clarkb | I guess we can send a bug email, then do an MR against master then ask for it to be backported or something along those lines | 22:39 |
| clarkb | noonedeadpunk: we could also try to help ansible improve their performance. Though i think there hasn't been a lto of interest in that perviously because the forking and task engine code is not super easy to understand or refactor | 22:46 |
| fungi | clarkb: in case it wasn't obvious in the test added for the hook script permissions change, that's before the change which embeds a copy of the hook script so all hook script installation is still remote at that stage | 22:52 |
| clarkb | fungi: yup it jumped out to me because you changed the name between patchsets | 22:52 |
| clarkb | but then when I reviewed the child change realized it call got resolved there | 22:52 |
| fungi | yeah, did that just to avoid extra churn in the second change | 22:52 |
| fungi | so that it would be more obvious from the test diff what the second change is altering behavior-wise | 22:53 |
| clarkb | fungi: fwiw I stopped short of installing the proposed git-review update and testing it in my own workflow but can do that if iti would be helpful (the test cases seemed pretty decent though) | 23:05 |
| fungi | i ran it extensively on my machine when developing it | 23:06 |
| fungi | since it was easy to do so | 23:06 |
| fungi | and yes, the tests were designed around the gotchas i encountered when doing so (permissions, encoding, etc) | 23:07 |
| fungi | also, it saves several seconds on git review -s invocation based on my observations | 23:09 |
| fungi | would probably be good to do a sweep for other pending git-review changes and see if there's more we'd want to incorporate into the next release | 23:10 |
| *** Guest500 is now known as diablo_rojo_phone | 23:26 | |
| clarkb | ++ | 23:33 |
| clarkb | there was one .d/ dir and two hash files for buster builds between nb01 and nb02. I've deleted them all. nb04 was cleaned up completely via nodepool itself | 23:36 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!