Monday, 2022-05-09

ianwi'm assuming https://launchpad.net/~openstack-ci-core/+archive/ubuntu/bubblewrap is no longer in use too00:11
fungithat's my assumption as well00:25
opendevreviewIan Wienand proposed openstack/project-config master: Add opendev/infra-vhd-util-deb repository  https://review.opendev.org/c/openstack/project-config/+/84105400:59
*** ysandeep|out is now known as ysandeep|rover04:58
opendevreviewMerged openstack/diskimage-builder master: Ensure cloud-init is configured to generated host keys  https://review.opendev.org/c/openstack/diskimage-builder/+/84082505:24
opendevreviewSandeep Yadav proposed openstack/diskimage-builder master: [DNM] test patch  https://review.opendev.org/c/openstack/diskimage-builder/+/84106505:46
*** soniya29 is now known as soniya29|ruck06:13
*** soniya29 is now known as soniya29|ruck06:58
*** soniya29 is now known as soniya29|ruck07:19
*** ysandeep|rover is now known as ysandeep|rover|lunch07:20
*** jpena|off is now known as jpena07:30
*** soniya29 is now known as soniya29|ruck07:38
*** soniya29|ruck is now known as soniya29|ruck|lunch07:57
*** soniya29|ruck|lunch is now known as soniya29|ruck08:23
*** ysandeep|rover|lunch is now known as ysandeep|rover08:23
dirkhi all, I am trying to push a new signed tag for renderspec, but I get this error message: remote: You need 'Create Tag' rights to push a normal tag. remote: User: dmllr09:48
dirkI was hoping I would have create tag rights via https://review.opendev.org/admin/groups/6fa146e80fe07f4d94f5c1ee34b4e1b590f7bb63,members but that doesn't seem to be enough. what else is missing?09:49
*** rlandy|out is now known as rlandy10:24
*** dviroel_ is now known as dviroel\11:14
*** dviroel\ is now known as dviroel11:14
fungidirk: the message says "normal tag" which implies at least one tag you're trying to push isn't actually signed. are you following the instructions here? https://docs.opendev.org/opendev/infra-manual/latest/drivers.html#tagging-a-release11:40
*** ysandeep|rover is now known as ysandeep|rover|break11:57
dirkfungi: I did fillow, but it is a ssh tag11:57
fungiwhat do you mean a ssh tag?11:58
fungido you mean you're using a signed push instead of signing the tag object? if so, that's not the same thing11:58
dirkfungi: https://calebhearth.com/sign-git-with-ssh11:59
fungidirk: right, not the same. you need to use git tag -s12:00
dirkI guess I need to create a gpg key and retry12:00
fungiyes, please see the instructions i linked above12:00
fungisigned pushes are signing the activity of pushing objects, not the objects themselves (it's distinct, since you could be pushing objects on behalf of another committer)12:01
fungiyou need to sign the tag object, which is what git tag -s does12:01
dirkfungi: I did ran git tag -s12:20
*** pojadhav is now known as pojadhav|break12:22
fungidirk: and you're explicitly pushing the tag you created, specifying it by name in the push command?12:36
*** ysandeep|rover|break is now known as ysandeep|rover12:57
dirkfungi: https://paste.openstack.org/show/b6wQqO9AE5T6fpo2mF9H/13:05
dirkhttps://paste.openstack.org/show/bcxHi01YANJ8DUzonvjp/ is the tag13:07
dirkit looks like gerrit is not patched to handle ssh signatures, https://github.com/GerritCodeReview/gerrit/blob/master/java/com/google/gerrit/server/restapi/project/CreateTag.java#L11013:07
fungiahh, yes, i've never heard of ssh signatures, which is why i thought you were talking about signed pushes13:08
fungiit needs an openpgp signature13:09
dirkits a relatively recent feature13:09
fungigerrit, being implemented in java, doesn't use cgit but rather jgit, which tends to lag behind cgit feature-wise13:10
fungiand gerrit usually doesn't bundle the most recent jgit version, we're usually not on the very latest gerrit release anyway13:10
fungiso git features can take years to trickle into our gerrit deployment, unfortunately13:11
fungibut thanks for educating me on git's ssh signature format, i'll need to do some more reading on that13:13
*** pojadhav|break is now known as pojadhav13:57
*** soniya29 is now known as soniya29|ruck14:06
dirkfungi: thank you for the help, openpgp signature worked14:38
*** pojadhav is now known as pojadhav|afk14:54
*** soniya29|ruck is now known as soniya29|ruck|dinner14:54
Clark[m]I suspect it would work if you allowed normal tag pushes. Gerrit doesn't recognize the ssh tag as a gpg tag which is correct but there isn't a specific ACL for ssh tags14:58
Clark[m]And the signing is embedded in the objects so jgit doesn't need to know much about them. That said I could be wrong and pushing such a tag could be problematic14:59
*** dviroel is now known as dviroel|lunch15:10
clarkbinfra-root does 15:30 UTC (about 10 minutes from now) still work to chat about CD stuff?15:19
fungiwfm, yep. thanks!15:19
*** soniya29|ruck|dinner is now known as soniya29|ruck15:36
*** marios is now known as marios|out15:50
*** ysandeep|rover is now known as ysandeep|out16:09
*** soniya29|ruck is now known as soniya29|out16:12
*** dviroel|lunch is now known as dviroel16:24
clarkbfungi: any reason to not approve https://review.opendev.org/c/opendev/system-config/+/840955 at this point? I expect to be around today and can monitor16:35
clarkbsorry meant to describe the  change. It is the etherpad upgrade16:35
funginope! i've approved it now16:35
clarkbthanks!16:36
clarkbGerrit has pushed their final release candidate before the 3.6 release16:45
clarkbI think we should still target 3.5 as there is a history of new gerrit releases needing time to bake16:45
clarkbBut we could potentially be on 3.6 this year before 3.7 happens? That is exciting16:46
clarkbcorvus: if you get a chance can you check https://review.opendev.org/c/opendev/system-config/+/840529 this is an update to where we pull keycloak docker images and that will cause a service upgrade I think?16:47
clarkbcorvus: I suspect you are the primary use of keycloak currently so wanted your input on that16:47
corvusclarkb: i think it's experimental until someone else wants to start using it, so i'm totally fine merging that and seeing what happens17:00
corvus+317:00
clarkbwmf17:00
clarkber wfm17:00
opendevreviewMerged opendev/system-config master: Update etherpad to 1.8.18  https://review.opendev.org/c/opendev/system-config/+/84095517:13
clarkbI don't think we auto restart the container for etherpad. I'll work on that once automation pulls it down17:13
*** jpena is now known as jpena|off17:17
opendevreviewMerged opendev/system-config master: Pull keycloak from quay.io  https://review.opendev.org/c/opendev/system-config/+/84052917:25
clarkbI was wrong it restarted itself jsut now17:42
clarkband it is unhappy :/17:43
fungihuh...17:44
fungiDatabase is not configured with charset utf8mb4 -- This may lead to crashes when certain characters ar17:45
fungie pasted in pads17:45
clarkbya then later it complains about logging level not being defined?17:45
fungiwhich seems like it could be a secondary error17:46
fungialso i thought we explicitly configured the backing db for 4-byte utf-817:47
fungiis it misdetecting it, or hitting the wrong db?17:47
fungivery odd17:47
clarkbthat first one almost seems like a warning and ya I thought we configured it for that. But we can double check17:47
clarkbalso why doesn't it break in CI? I even held the node and it seemed to work17:48
fungishow table status says it's utf8mb4_bin17:52
fungimaybe it checks for utf8mb4 rather than utf8mb4_bin?17:53
clarkbya I'm pulling up source code now to see if logging that error is what raises the level issue17:54
clarkbits possible that is the problem?17:54
fungiwe seem to set utf8mb4_bin for collation_server, but utf8mb4 for default-character-set, character-set-server, and in the etherpad settings.json17:54
fungimaybe in ci jobs we're creating the table as utf8mb4 not utf8mb4_bin17:54
fungido we want to roll back the container, or in-place migrate the table after dumping it?17:56
clarkbfungi: in our json.settings we set the db charset to utf8mb4 not utf8mb4_bin. I wonder if that is the issue17:56
fungii guess that's not hard to try changing by hand first17:57
clarkbwell can we convert between the two? eg utf8mb4_bin -> utf8mb4? I don't know17:57
clarkbbut also I'm not sure if the mixed settinsg you found represent a problem either way17:57
fungiwe may need to dump and load17:57
fungiyeah, i honestly have no idea17:57
fungii mainly wanted to double-check that it wasn't somehow back to the 3-byte default width17:57
clarkbI think before we do anything we need to better understand why this error is happening17:59
fungiyes17:59
clarkbalso rolling back may not be straigtforward if we already pruned the image on docker hub? we could try a revert and rebuilt I guess17:59
fungiand whether that's the actual error causing it to crash, or unrelated18:00
clarkbbut ya I'm going to dig around ad see if I can figure out why it is complaining when we seem to set it that way18:00
clarkb++18:00
fungiand yeah, looks like we don't still have the old container image on the server18:00
clarkbthe db error checks DEFAULT_CHARACTER_SET_NAME against charset18:02
clarkband it queries that for our db schema (not table level)18:02
clarkband that code hasn't changed in a long time18:03
fungiand we haven't touched that config ourselves for 2 years18:04
fungiit got added to our settings.json and my.cnf by 718536 two years ago18:05
clarkbok I strongly suspect reverting the etherpad change will fix this18:06
fungioh?18:06
clarkbone of the changes 1.8.18 made was updating euberdb2 to 2.2.4 which is a super new change18:06
clarkb1.8.17 used euberdb2 1.4.1818:06
fungioh, so maybe it's misreading the schema18:07
clarkband these errors are all originating in euberdb218:07
clarkbfungi: https://github.com/ether/ueberDB/blob/v2.2.4/databases/mysql_db.js#L68-L79 thats the first error18:10
clarkbthat exists in 1.4.18 though18:10
clarkboh but the logging error originates in https://github.com/ether/ueberDB/blob/v1.4.18/databases/mysql_db.js#L42-L52 which is called above18:11
clarkbso anyway I do suspect that rolling back has a good chance of working18:12
fungiso maybe in trying to log the encoding mismatch it's been complaining about for who knows how long, it tripped a bug in logging code not normally traversed and perhaps poorly-tested18:12
clarkbya thats what I'm suspecting18:13
clarkbhowever in testing we don't seem to have this problem https://zuul.opendev.org/t/openstack/build/94b0c05140cd48e0a6fb4d4a7562b7b0/log/etherpad01.opendev.org/docker/etherpad-docker_etherpad_1.txt18:15
clarkbin testing it shows connection errors to the database while the database is coming up. And that all looks similar to what we see here except for hte logging failure18:16
clarkbthen once the db is up it starts and is happy18:16
clarkbin production the db was never stopped and should be up18:16
clarkbI'm going to try and look at the actual db now18:18
clarkbfungi: etherpad-lite is utf8                       | utf8_bin18:22
clarkbaccording to the schemata which is what it is checking18:22
clarkbfungi: where did you see the utf8 settings before that seemed to match up?18:23
clarkbI think it may work in production because we let etherpad lite create the db freshly which sets the values it wants18:24
clarkbhowever these are not new checks from what I can tell so I'm still confused as to why it is failing now18:24
clarkboh I see18:25
clarkbfungi: https://github.com/ether/ueberDB/blob/v2.2.4/databases/mysql_db.js#L106-L113 I think we may actually be failing there18:25
fungii looked at show table status18:26
fungithe database may be utf8 but the table is utf8mb418:27
clarkbya I think that may be what is going on. I'm going to try this select to get the migration level18:27
clarkbI think what is happening is that it is trying to log the charset for us after that error. in the process of logging the charset it is trying to lookup the migration level which is set later18:31
clarkbthat errors and it short circuits there18:31
clarkbif we set the charsets on the schema properly it would in theory get by all this. But then our migration level may be wrong. However, I think migration level may already be correct because it would've set it in the past?18:31
fungiwell, s/properly/like etherpad wants/ anyway18:32
fungiit's set properly (the database is still default, but we've set the table to utf8mb4)18:33
clarkbright18:33
clarkbhow do yu show it on a table? is that in the information schema too?18:33
clarkbah yup18:34
fungii checked show table status which lists extended information for all tables18:34
clarkbmy suspicion now is that what broke is logging and that we likely always had these issues previously18:36
clarkbbut before it was basically a warning. now with broken logging its fatal18:37
fungibecause it breaks trying to log the warning18:37
fungimakes sense18:37
clarkbgiven that I think what we can do is docker-compose down the etherpad service but not the db. mysqldump the db just in case. Update the schema default utf8 settings for etherpad-lite and then try starting the service again18:38
fungiagreed, that shouldn't require any migration since it's just modifying the db default18:38
fungialso it will at that point match what we're testing, at least in theory18:39
clarkboh! the sqlalter only affects the store table not the etherpad-lite schema so that explains why it wasn't auot updating18:40
clarkbfungi: I'll go ahead and down the service and run a db backup?18:40
fungiyes, thanks!18:41
clarkbfungi: we want default charset to be 'utf8mb4' and default collation to be 'utf8mb4_bin' if I read this properly18:41
clarkbfungi: maybe you can sort out how to modify the db to do that while I do backups?18:41
fungiright, that's what i see in the my.cnf anyway18:41
fungilooking into it now18:42
clarkbI ran `sudo docker-compose stop etherpad` so that we can check docker-compose logs against that if necessary (down would delete it and no more logs)18:43
clarkbI'm going to give it a few minutes to make sure it isn't going to auto restart somehow then start backups18:43
clarkbits been 2 minutes and it hasn't started. I'll run the db backup now18:45
clarkbfungi: `ALTER DATABASE etherpad-lite CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;` ?18:46
clarkbthis backup is not fast. About a tenth of the way complete18:47
fungihttps://mariadb.com/kb/en/setting-character-sets-and-collations/ other than quoting the encodings18:48
fungiyes18:49
fungiALTER DATABASE etherpad-lite CHARACTER SET 'utf8mb4' COLLATE 'utf8mb4_bin';18:49
fungier, and the =18:50
fungiALTER DATABASE etherpad-lite CHARACTER SET = 'utf8mb4' COLLATE = 'utf8mb4_bin';18:50
fungiclarkb: ^ so like that18:51
fungiat least that's what i gather from the mariadb kb18:51
NeilHanlonthat looks right to me18:51
clarkbnow I'm trying to figure out why logging is broken :/18:52
NeilHanlon:(18:52
clarkbin theory it works enouh for our testing to g et by though18:52
fungican it be just that one logging call is wrong somehow, maybe missed in a transition to a new call structure at some point?18:55
fungibecause it's logging other stuff18:55
clarkbhttps://github.com/ether/ueberDB/blob/v2.2.4/databases/mysql_db.js#L78 is where we break and I think result[0] is taken as the log level since it is logger.log and not logger.error as per the statement above18:57
clarkbso ya it may just be this is specific to that specific doe18:57
clarkbs/doe/code/18:57
clarkbhttps://github.com/log4js-node/log4js-node/blob/v0.6.38/lib/logger.js#L53-L55 is where it seems to explode18:58
fungiand logging a particular error condition is likely to not have code coverage for testing18:58
clarkbya makes sense18:59
clarkbabout 2/3s done with the backup18:59
clarkbfungi: ok db backup is done. Did you want to run the alter on the etherpad-lite db schema?19:10
fungialter test?19:11
fungioh the alter19:11
fungiyeah, i can, just a sec19:11
fungiit doesn't like that syntax19:13
fungiERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '-lite CHARACTER SET = utf8mb4 COLLATE = utf8mb4_bin' at line 119:13
clarkbI don't think you want the ='s19:13
fungii guess it doesn't expect the token there to have a hyphen in it, so maybe that's not where the db name goes19:13
fungiwell, the alter commands in the mariadb kb included the =19:14
clarkboh huh ya maybe you ahve to quote the db name?19:14
clarkbdoes it use backticks for that?19:14
fungii did try quoting the db name as well19:14
fungioh, mackticks19:14
fungibackticks19:14
clarkbya backticks19:14
fungiyeah, that seems to have done it19:15
fungiftr, complete command line was:19:15
fungisudo docker-compose -f /etc/etherpad-docker/docker-compose.yaml exec -T mariadb bash -c '/usr/bin/mysql -uroot -p"$MYSQL_ROOT_PASSWORD" -e "ALTER DATABASE \`etherpad-lite\` CHARACTER SET = 'utf8mb4' COLLATE = 'utf8mb4_bin'" etherpad-lite'19:15
clarkbI see it reflected in the information schema too. Should I start the servicen ow?19:15
fungiyep!19:15
clarkbok doing that19:15
fungioh, hah, i also just noticed that my quoting of the encodings was probably eaten by the shell, but obviously unnecessary19:16
clarkb[2022-05-09 19:16:18.763] [INFO] server - Etherpad is running19:16
clarkbUp 43 seconds (healthy)19:16
clarkbnow to check the actual etherpads are happy19:17
clarkbhttps://etherpad.opendev.org/p/isitbroken19:17
fungi#status log Updated database-wide default encoding for Etherpad server to utf8mb4, as setting it only on the table is insufficient to satisfy its safety checks19:18
opendevstatusfungi: finished logging19:18
fungiyeah, looks like it's working as desired19:18
clarkbya I've got multiple browsers pulling up and editing that pad. I think that was it.19:19
clarkbI'll file a bug upstream after lunch19:19
fungiyes, looks to me like it's working as desired19:19
fungithanks for figuring that out!19:20
clarkbthat was far more exciting than expected, but at least running it down was productive19:21
clarkbI'm going t ogo eat lunch now and will file that bug when I get back19:21
Clark[m]As a sanity check those values are only defaults if unspecified when creating tables right? And since etherpad only has a single table with the correct values already the change here is basically nil for us?19:26
Clark[m]That was my understanding and want to make sure others agree19:26
Clark[m]Ya docs say it affects create table, load data, and stored routines that don't have values specified19:29
Clark[m]I think those are non issues here19:29
clarkbI wonder if https://github.com/ether/ueberDB/commit/7a352387eb43dc111a2547e19343134cedb4b622 is to blame19:40
fungiclarkb: for added amusement, checking the default character set for the db is kinda silly, since that doesn't mean the character set for the table etherpad is actually writing to is correct19:42
clarkbya I think its a asnity check for those other operations but I can make note of that in the issue too19:43
fungione of the things to come out of the python 3.11.0b1 release this weekend is that pep 594 marks a bunch of things in stdlib deprecated for removal in 3.13, and one of those things is the cgi module which lots of projects have been relying on for http header paramaterization, including pip. so i spent a chunk of time this weekend proposing a fix to replace it with email.message (which has a19:50
fungisimilar mime parser), and that entailed not only familiarizing myself with pip's testsuite but even moreso reminding myself how to use github. spoiler: it's still frustrating19:50
fungiat least my pr is passing all 19 of pip's ci jobs now, including coercing things to unambiguous data types in order to appease the typechecking19:51
fungistill need to propose a similar fix for use of cgi in Cython.Tempita._tempita, crypt in passlib.utils, and some places in my own code where i'm using telnetlib in its testsuite19:53
fungisort of sad to see telnetlib on the chopping block, since the thing i'm testing with it is an actual rfc-compliant telnet server19:55
clarkbhttps://github.com/ether/ueberDB/issues/266 posted19:59
fungithanks!20:12
fungimordred: in the spi board meeting (#spi channel right now) they're talking about whether to renew the drizzle trademark or let it expire, just fyi (no idea if you know anyone who cares)20:27
mordredfungi: I don't think anyone cares. (I mean, I *care* but nobody has worked on the project in long enough that it makes sense for SPI to renew20:30
fungiyeah, i figured20:30
mordred\o/ I have participated in a discussion20:36
fungiindeed. i was hesitant to speak up during the board meeting as i'm not a board member, but i figure feedback from someone who can represent the drizzle project would probably count as solicited feedback20:41
*** dviroel is now known as dviroel|afk20:43
ianwclarkb/fungi/mordred even: if you have a sec for https://review.opendev.org/c/openstack/project-config/+/841054 that will add a repo to build vhd-util and we can use the same infrastructure as the openafs-debs21:18
ianwi think that's really the two ppa's in active use and it will be good to have it zuulified21:18
clarkb++ I have to do a school run now and have some stuff on the backlog already but will try to get to it21:18
ianwnp, that's just adding the repo, nothing interesting yet ... i still have to do that bit :)21:19
ianwfungi: do we have a specific "infra-core" acl i missed?21:49
fungii don't think so. i was considering just checksumming the acl directory and seeing which ones were identical21:56
opendevreviewdasm proposed opendev/elastic-recheck rdo: Enable podman-compose  https://review.opendev.org/c/opendev/elastic-recheck/+/84117221:58
fungithat was the main point of the normalization script years ago: figuring out which acls we could deduplicate21:58
fungilooks like the infra-openafs-deb, zone-opendev.org and zone-zuul-ci.org share the same content checksums right now, so not as many as i'd guessed22:01
clarkbeach of the puppet repos got specific acls iirc22:01
fungialso those are the only duplicated acls in the opendev/ namespace22:01
fungiyep22:01
clarkband then system-config and project-config don't match22:02
clarkbwe can probably make a number of repos match though22:02
fungiso not a huge savings to deduplicate the other three at the moment22:02
fungimaybe if we standardize our acls more in the future there would be more reason22:02
ianwyeah there's some others like afsmon that seems to have it's own group but doesn't really need it22:11
clarkbI'm going to put the PPA building stuff on the meeting agenda just so that we can get a quick tldr22:12
fungii'd love to find time to scan gerrit for unused groups, but...22:12
*** rlandy is now known as rlandy|bbl22:13
opendevreviewMerged openstack/project-config master: Add opendev/infra-vhd-util-deb repository  https://review.opendev.org/c/openstack/project-config/+/84105422:13
*** cloudnull8 is now known as cloudnull23:04
opendevreviewIan Wienand proposed opendev/infra-vhd-util-deb focal: Initial import  https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/84117423:22
opendevreviewIan Wienand proposed opendev/infra-vhd-util-deb focal: Initial import of debian build files  https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/84117523:25
clarkbianw: in ^ we have a bionic.patch and a focal.patch. Should we be handling that with branches or is the build system applying them based on the name and target and we don't need branches for this case?23:31
ianwclarkb: yeah, i was planning on applying those patches as separate changes to the actual source tree, and removing from debian/patches23:31
ianwthat's clearer as to what's going on, now the source is in git23:32
clarkbwe would have to carry the entire source tree if we did that though?23:33
clarkbI guess some deb pacakges do that and some don't?23:33
ianwyeah, given that this is a very bespoke tool now, and our efforts at upstreaming bits have not succeeded for the past ... 10 years ... i think this is a case of keeping our special source tree is the most practical approach23:36
clarkbthat is a good point23:36
ianwi've forgotten all the details, but xen removed big bits of it (blktap2, something something)23:38
clarkbthe etehrpad service hasn't restarted since we applied the schema default charset and collation update23:56
opendevreviewIan Wienand proposed opendev/infra-vhd-util-deb focal: Initial import  https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/84117423:59
opendevreviewIan Wienand proposed opendev/infra-vhd-util-deb focal: Initial import of debian build files  https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/84117523:59
opendevreviewIan Wienand proposed opendev/infra-vhd-util-deb focal: [wip] test vhd-util build  https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/84117923:59

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!