| ianw | fungi: yeah, i don't have any particular insight over your testing. i'm happy to babysit a restart | 00:01 |
|---|---|---|
| fungi | happy to do a restart myself too, just wanted to make sure i wasn't overlooking anything obvious | 00:01 |
| fungi | and clearly it's not crippling us as it's been broken since tuesday and nobody reported it until saturday | 00:02 |
| fungi | or wednesday/sunday au time | 00:02 |
| fungi | i can restart it now if you're cool with that | 00:03 |
| ianw | yep i am | 00:04 |
| fungi | ianw: okay, doing it now | 00:05 |
| *** cloudnull has quit IRC | 00:05 | |
| fungi | it's starting back up now | 00:05 |
| *** cloudnull has joined #opendev | 00:06 | |
| fungi | ianw: also i saw there were some backup errors e-mailed to our shared root mail address | 00:06 |
| fungi | gerrit webui seems to be back again | 00:07 |
| fungi | and `docker-compose exec gerrit lslocks -u` includes /var/gerrit/index/accounts_0011/write.lock again | 00:07 |
| fungi | if brinzhang happens back by, they should be able to try generating an http password again | 00:08 |
| fungi | i'll mention it over in #openstack-nova since that's where it seems to have been initially mentioned | 00:09 |
| fungi | #status log Restarted the Gerrit container on review.o.o to address a recurrence of https://bugs.chromium.org/p/gerrit/issues/detail?id=13726 | 00:10 |
| openstackstatus | fungi: finished logging | 00:10 |
| *** brinzhang has joined #opendev | 00:43 | |
| *** tosky has quit IRC | 00:53 | |
| *** LowKey has quit IRC | 00:56 | |
| *** LowKey has joined #opendev | 00:56 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] gerrit : add mariadb_container option https://review.opendev.org/c/opendev/system-config/+/775961 | 01:26 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: gerrit: download latest mysql connector https://review.opendev.org/c/opendev/system-config/+/776857 | 01:26 |
| *** LowKey has quit IRC | 02:14 | |
| *** fdegir has quit IRC | 02:58 | |
| openstackgerrit | Brin Zhang proposed openstack/project-config master: Create inspur/venus-dashboard repository https://review.opendev.org/c/openstack/project-config/+/776864 | 03:01 |
| *** ykarel has joined #opendev | 03:14 | |
| *** lbragstad_ has joined #opendev | 03:41 | |
| *** lbragstad has quit IRC | 03:43 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] gerrit : add mariadb_container option https://review.opendev.org/c/opendev/system-config/+/775961 | 04:54 |
| *** ysandeep|away is now known as ysandeep|ruck | 05:25 | |
| *** zoharm has joined #opendev | 05:27 | |
| brinzhang | frickler, infra-root： generate password for HTTP Credentials works fine now, thanks^ | 05:47 |
| *** dtantsur|afk is now known as dtantsur | 06:14 | |
| *** whoami-rajat has joined #opendev | 06:23 | |
| *** dtantsur is now known as dtantsur|brb | 07:05 | |
| *** hashar has joined #opendev | 07:09 | |
| *** lpetrut has joined #opendev | 07:17 | |
| *** slaweq has joined #opendev | 07:17 | |
| *** guillaumec has quit IRC | 07:34 | |
| *** guillaumec has joined #opendev | 07:34 | |
| *** eolivare has joined #opendev | 07:38 | |
| *** ralonsoh has joined #opendev | 07:38 | |
| *** zoharm has quit IRC | 07:40 | |
| *** zoharm has joined #opendev | 07:40 | |
| *** hashar has quit IRC | 07:52 | |
| *** hashar has joined #opendev | 07:54 | |
| *** ysandeep|ruck is now known as ysandeep|lunch | 08:08 | |
| *** andrewbonney has joined #opendev | 08:10 | |
| *** ykarel_ has joined #opendev | 08:30 | |
| *** ykarel has quit IRC | 08:33 | |
| *** ykarel_ is now known as ykarel | 08:33 | |
| *** rpittau|afk is now known as rpittau | 08:34 | |
| *** tosky has joined #opendev | 08:43 | |
| *** jpena|off is now known as jpena | 08:58 | |
| *** fressi has joined #opendev | 08:59 | |
| *** ysandeep|lunch is now known as ysandeep|ruck | 09:03 | |
| *** ykarel has quit IRC | 09:23 | |
| *** ykarel has joined #opendev | 09:38 | |
| *** DSpider has joined #opendev | 09:59 | |
| *** slaweq has quit IRC | 10:11 | |
| *** slaweq has joined #opendev | 10:15 | |
| *** sshnaidm__ is now known as sshnaidm | 10:22 | |
| *** ykarel has quit IRC | 10:36 | |
| *** jhesketh has quit IRC | 10:37 | |
| *** jhesketh has joined #opendev | 10:37 | |
| *** ykarel has joined #opendev | 10:42 | |
| *** tosky_ has joined #opendev | 10:53 | |
| *** tosky has quit IRC | 10:54 | |
| *** tosky_ is now known as tosky | 10:54 | |
| *** slaweq_ has joined #opendev | 11:27 | |
| *** dviroel has joined #opendev | 11:30 | |
| *** slaweq has quit IRC | 11:31 | |
| openstackgerrit | Sorin Sbârnea proposed zuul/zuul-jobs master: Upgrade ansible-lint to 5.0 https://review.opendev.org/c/zuul/zuul-jobs/+/773245 | 11:40 |
| *** dtantsur|brb is now known as dtantsur | 11:50 | |
| *** hashar is now known as hasharLunch | 11:59 | |
| openstackgerrit | Guillaume Chauvel proposed opendev/system-config master: Increase autogenerated comment width to avoid line wrap https://review.opendev.org/c/opendev/system-config/+/771445 | 12:01 |
| openstackgerrit | Guillaume Chauvel proposed opendev/system-config master: [DNM] test comment width: review without autogenerated tag https://review.opendev.org/c/opendev/system-config/+/771798 | 12:01 |
| *** jpena is now known as jpena|lunch | 12:31 | |
| fungi | brinzhang: thanks for confirming! | 12:42 |
| *** iurygregory_ has joined #opendev | 13:14 | |
| *** iurygregory has quit IRC | 13:15 | |
| *** lbragstad_ is now known as lbragstad | 13:16 | |
| *** tosky has quit IRC | 13:18 | |
| *** iurygregory_ is now known as iurygregory | 13:24 | |
| *** tosky has joined #opendev | 13:24 | |
| *** jpena|lunch is now known as jpena | 13:25 | |
| *** lbragstad has quit IRC | 13:35 | |
| *** lbragstad has joined #opendev | 13:37 | |
| *** ykarel has quit IRC | 13:56 | |
| *** hasharLunch is now known as hashar | 14:06 | |
| openstackgerrit | Guillaume Chauvel proposed opendev/system-config master: Increase autogenerated comment width to avoid line wrap https://review.opendev.org/c/opendev/system-config/+/771445 | 14:14 |
| openstackgerrit | Guillaume Chauvel proposed opendev/system-config master: [DNM] test comment width: review without autogenerated tag https://review.opendev.org/c/opendev/system-config/+/771798 | 14:14 |
| *** tosky has quit IRC | 14:29 | |
| *** tosky has joined #opendev | 14:33 | |
| openstackgerrit | Jeremy Stanley proposed opendev/git-review master: Test/assert Python 3.9 support https://review.opendev.org/c/opendev/git-review/+/772589 | 14:48 |
| *** brinzhang_ has joined #opendev | 14:52 | |
| *** brinzhang_ has quit IRC | 14:53 | |
| *** brinzhang_ has joined #opendev | 14:53 | |
| *** brinzhang has quit IRC | 14:54 | |
| *** tosky has quit IRC | 15:15 | |
| *** tosky has joined #opendev | 15:15 | |
| *** tosky_ has joined #opendev | 15:18 | |
| *** tosky is now known as Guest43629 | 15:19 | |
| *** tosky_ is now known as tosky | 15:19 | |
| *** Guest43629 has quit IRC | 15:21 | |
| *** fressi has left #opendev | 15:22 | |
| *** lpetrut has quit IRC | 15:29 | |
| *** redrobot has quit IRC | 15:30 | |
| *** redrobot has joined #opendev | 15:30 | |
| *** brinzhang_ has quit IRC | 15:32 | |
| *** brinzhang_ has joined #opendev | 15:33 | |
| *** ysandeep|ruck is now known as ysandeep|dinner | 15:42 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Try to make gatling-git work with our test gerrit https://review.opendev.org/c/opendev/system-config/+/775883 | 16:05 |
| *** whoami-rajat has quit IRC | 16:19 | |
| *** ysandeep|dinner is now known as ysandeep|away | 16:21 | |
| *** LowKey has joined #opendev | 16:26 | |
| openstackgerrit | Guillaume Chauvel proposed opendev/system-config master: Increase autogenerated comment width to avoid line wrap https://review.opendev.org/c/opendev/system-config/+/771445 | 16:36 |
| openstackgerrit | Guillaume Chauvel proposed opendev/system-config master: [DNM] test comment width: review without autogenerated tag https://review.opendev.org/c/opendev/system-config/+/771798 | 16:36 |
| *** dtantsur is now known as dtantsur|afk | 16:38 | |
| *** mlavalle has joined #opendev | 17:10 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Try to make gatling-git work with our test gerrit https://review.opendev.org/c/opendev/system-config/+/775883 | 17:19 |
| *** fbo is now known as fbo|off | 17:23 | |
| clarkb | guillaumec: note if you are writing that plugin the copyright should be for you (or possibly your employer) but not the OSF | 17:26 |
| clarkb | I had assumed this was something android was already doing hence the request for attribution given the previous copyright line | 17:26 |
| *** sgw has left #opendev | 17:26 | |
| *** zoharm has quit IRC | 17:34 | |
| *** rpittau is now known as rpittau|afk | 17:49 | |
| *** sgw has joined #opendev | 17:54 | |
| *** jpena is now known as jpena|off | 17:55 | |
| *** hashar has quit IRC | 18:07 | |
| clarkb | now that the zuul release is largely done, any reason to not try and build new ze/zm instances on focal? All of the currenty instances are z(e|m)XY.openstack.org. I think I can make z(e|m)XY.opendev.org and once they are up and running remove teh corresponding .openstack.org host | 18:09 |
| clarkb | the nodepool launchers and and zk servers are also on my list but a bit more complicated as they are more stateful | 18:09 |
| clarkb | for the launchers I think we may need to put the old on in emergency and disable it by hand, then land .opendev.org addition change and when ahppy remove the old .openstack.org launcher | 18:10 |
| *** hamalq has joined #opendev | 18:10 | |
| clarkb | for zk I think we can rotate servers out and add new ones in without downtime. The gotcha there is going to be the configs that tell zuul and nodepool and zk itself where the zk servers are | 18:10 |
| clarkb | not sure what that will look like yet | 18:11 |
| guillaumec | clarkb, "ze/zm instances on focal", perhaps this: https://review.opendev.org/c/opendev/gear/+/741288 | 18:11 |
| clarkb | guillaumec: I think we'll be ok since gear runs in a container on the scheduler (whcih means upadting the scheduler too should be fine), but that is a good thing to be aware of | 18:14 |
| *** mlavalle has quit IRC | 18:21 | |
| *** _mlavalle_1 has joined #opendev | 18:21 | |
| clarkb | I'll start with a zm01.opendev.org since that should have the smallest impact if something like gear is unhappy | 18:21 |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: Revert "Revert "Update upload-logs roles to support endpoint override"" https://review.opendev.org/c/zuul/zuul-jobs/+/776677 | 18:22 |
| fungi | yeah, seems like we should be all set to start doing system upgrades | 18:26 |
| *** sboyron has joined #opendev | 18:33 | |
| *** ralonsoh has quit IRC | 18:35 | |
| *** sboyron has quit IRC | 18:37 | |
| openstackgerrit | Guillaume Chauvel proposed opendev/system-config master: [DNM] test comment width: review without autogenerated tag https://review.opendev.org/c/opendev/system-config/+/771798 | 18:43 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Add an nl01.opendev.org config https://review.opendev.org/c/openstack/project-config/+/776979 | 18:51 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Update test platform for zm and ze servers https://review.opendev.org/c/opendev/system-config/+/776980 | 18:54 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Update nodepool launchers to focal in testing https://review.opendev.org/c/opendev/system-config/+/776981 | 18:54 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Update zookeeper servers in testing to focal https://review.opendev.org/c/opendev/system-config/+/776982 | 18:54 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Update zuul01 to focal in testing https://review.opendev.org/c/opendev/system-config/+/776983 | 18:54 |
| clarkb | there are enough moving parts in ^ that I'm sure I will get something wrong. | 18:54 |
| clarkb | I'll also WIP https://review.opendev.org/c/opendev/system-config/+/776980 to prevent it from merging until we are ready | 18:54 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add new zm01.opendev.org server https://review.opendev.org/c/opendev/system-config/+/776984 | 18:59 |
| openstackgerrit | Clark Boylan proposed opendev/zone-opendev.org master: Add zm01.opendev.org to DNS https://review.opendev.org/c/opendev/zone-opendev.org/+/776985 | 19:05 |
| clarkb | I think these two changes are ready for review unless we want to wait for test results from https://review.opendev.org/c/opendev/system-config/+/776980 furst | 19:05 |
| clarkb | one thing I'm noticing is that there is a bit of config intertwining particularly for the gearman server specification that will be fun to untangle when we get around to updating zuul01 | 19:06 |
| clarkb | anyway one step at a time, we'll start with the mergers and executors | 19:07 |
| clarkb | and with zm01.opendev.org above :) | 19:08 |
| fungi | i'm really stumped on https://review.opendev.org/772589 | 19:15 |
| fungi | i think something about the gerrit setup in the git-review functional jobs must be breaking the dev mode default creds | 19:16 |
| fungi | i've even tried outright replacing the test_user/test_pass with admin/secret and not trying to create a new user, but when i try to post to a/accounts/self/sshkeys to write the ssh public key as admin/secret (like zuul's quickstart job does), i still get a 401 unauthorized | 19:17 |
| fungi | and if i comment out all the test cleanup so that the gerrit instance is left running after a test, i can't authenticate to its rest api as admin/secret either, so i don't think it's anything as simple as i've called requests wrong | 19:18 |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: Revert "Revert "Update upload-logs roles to support endpoint override"" https://review.opendev.org/c/zuul/zuul-jobs/+/776677 | 19:21 |
| openstackgerrit | Guillaume Chauvel proposed opendev/system-config master: [DNM] test comment width: review without autogenerated tag https://review.opendev.org/c/opendev/system-config/+/771798 | 19:50 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/776980/1 got a +1 from zuul so I think we should be good to proceed with https://review.opendev.org/c/opendev/system-config/+/776984 and https://review.opendev.org/c/opendev/zone-opendev.org/+/776985 | 20:02 |
| *** slaweq_ has quit IRC | 20:07 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Try to make gatling-git work with our test gerrit https://review.opendev.org/c/opendev/system-config/+/775883 | 20:09 |
| clarkb | guillaumec: for the gear tls thing I wonder if ssl.create_default_context would be better? | 20:23 |
| clarkb | or just construct a mask for TLS 1, 1.1 and 1.2? | 20:29 |
| clarkb | hrm it doesn't appear to be a mask | 20:31 |
| clarkb | 1 is value 3, 1.1 is value 4 and 1.2 is value 5 | 20:32 |
| openstackgerrit | Guillaume Chauvel proposed opendev/gear master: Create SSL context using PROTOCOL_TLS: highest supported version https://review.opendev.org/c/opendev/gear/+/741288 | 20:34 |
| guillaumec | jsut rebased to recheck :) , no idea atm | 20:34 |
| *** andrewbonney has quit IRC | 20:39 | |
| clarkb | poking around at it I think create_default_context may be the closest thing to what we're looking for without just requiring tlsv2. However, I suppose that may also be an option | 20:41 |
| clarkb | maybe with a fallback eg if ssl.PROTOCOL_TLS is defined use that else use ssl.PROTOCOL_TLSv1_2 | 20:41 |
| clarkb | fungi: corvus: ianw: have time for https://review.opendev.org/c/opendev/system-config/+/776984 and https://review.opendev.org/c/opendev/zone-opendev.org/+/776985 ? I can keep an eye on that and then also cleanup zm01.openstack.org assuming zm01.opendev.org joins the cluster happily | 20:43 |
| fungi | yeah, i can take a peek | 20:48 |
| *** LowKey has quit IRC | 20:51 | |
| *** LowKey has joined #opendev | 20:51 | |
| clarkb | then I expect I'll try to get through all of the mergers then look at the executors next. The executors are a bit more complciated as their disk layout is not quitedefault, but other than that I expect it to be very similar to the mergers | 20:54 |
| fungi | went ahead and approved those, they should be entirely safe on their own | 20:55 |
| clarkb | thanks | 20:55 |
| ianw | o/ | 20:59 |
| ianw | i can help out with some upgrades if time is getting away | 20:59 |
| clarkb | hasn't gotten away yet :) I'll let you know if that chagnes later today | 20:59 |
| clarkb | at this point the server is up and just need to land those changes, have them run config management against the server then see what it looks like after the fact | 21:00 |
| *** eolivare has quit IRC | 21:01 | |
| ianw | fungi: unless you've started with DEVELOPMENT_BECOME_ANY_ACCOUNT i don't think you'll have the admin user | 21:03 |
| ianw | i guess https://review.opendev.org/c/opendev/git-review/+/772589/17/git_review/tests/__init__.py used to "fake" this by directly inserting a user into the db | 21:03 |
| openstackgerrit | Merged opendev/zone-opendev.org master: Add zm01.opendev.org to DNS https://review.opendev.org/c/opendev/zone-opendev.org/+/776985 | 21:03 |
| fungi | ianw: yeah, it does set auth.type=DEVELOPMENT_BECOME_ANY_ACCOUNT in the config for each "site" but i've also tried to go farther and add --dev to the gerrit init command line for when it sets up the "golden site" as well as using sed to inline replace the OPENID with DEVELOPMENT_BECOME_ANY_ACCOUNT in its default config too | 21:07 |
| fungi | i'm assuming none of that is sufficient though | 21:07 |
| fungi | because it still seemed to make no difference in the end | 21:07 |
| *** slaweq_ has joined #opendev | 21:09 | |
| ianw | fungi: are we talking about line 162 there? | 21:09 |
| fungi | yeah, i've tried adding --dev to that | 21:11 |
| fungi | which in theory should be happening when the database deson't exist yet | 21:12 |
| openstackgerrit | Merged opendev/system-config master: Add new zm01.opendev.org server https://review.opendev.org/c/opendev/system-config/+/776984 | 21:12 |
| ianw | fungi: you'll definitely need that i think -- that is what i found with https://opendev.org/opendev/system-config/commit/d1694d4c98eabc5fda7ea4a38334fdffa6be2cc9 | 21:12 |
| fungi | so the check in gerrit-pgm/src/main/java/com/google/gerrit/pgm/init/InitAdminUser.java for existing users list being empty should still succeed | 21:12 |
| ianw | i suspect either the --dev or the DEVELOPMENT_BECOME_ANY_ACCOUNT somehow isn't being applied | 21:12 |
| fungi | git_review/tests/utils.py is what adds DEVELOPMENT_BECOME_ANY_ACCOUNT to the config and has been there since ages, but apparently wasn't doing anything | 21:13 |
| ianw | on line 165, it seems to have an extra "download-commands" on the end of the init? | 21:15 |
| openstackgerrit | Jeremy Stanley proposed opendev/git-review master: Test/assert Python 3.9 support https://review.opendev.org/c/opendev/git-review/+/772589 | 21:15 |
| fungi | ianw: that's ^ what i've been trying locally | 21:16 |
| fungi | looks like download-commands was added by https://review.openstack.org/93951 in 2014 | 21:17 |
| fungi | i can remove it, the commit message seems to imply it was needed for plugin stuff | 21:17 |
| ianw | yeah, i can't see that it takes any arguments, i wonder if it's making your init silently fail? | 21:18 |
| fungi | i'm trying locally with it stripped out. if that was it i'm gonna scream | 21:19 |
| fungi | s/scream/start drinking early/ | 21:19 |
| fungi | Stdout: fatal: Option "--install-plugin" takes an operand | 21:19 |
| fungi | i guess that's saying install the download-commands plugin? | 21:19 |
| ianw | oh, ok, the argument before it | 21:20 |
| ianw | still, i don't think you need it | 21:20 |
| fungi | i'll drop --install-plugin too in that case | 21:20 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Try to make gatling-git work with our test gerrit https://review.opendev.org/c/opendev/system-config/+/775883 | 21:20 |
| clarkb | eventually I'll figure out the java ssh stuff | 21:21 |
| fungi | mmm... i still get 401/Unauthorized | 21:21 |
| ianw | fungi: umm, is it setting DEVELOPMENT_... *after* init? | 21:23 |
| fungi | ianw: that was a belt and suspenders hail mary | 21:24 |
| fungi | the init includes --dev which should have the same effect | 21:24 |
| fungi | at least based on my reading of the docs | 21:25 |
| ianw | umm, i seem to have a strong memory that it does not | 21:25 |
| ianw | you need both | 21:25 |
| ianw | i think line 166 needs to move above the init | 21:25 |
| fungi | will that file exist before the init is run? | 21:27 |
| fungi | we make the etc dir a few lines earlier so we can precreate the hostkey in it | 21:28 |
| fungi | the gerrit init seems to unpack the tree which includes the default config, then i stream edit that unpacked config just for added security since gerrit is started via the unpacked gerrit.sh script later which doesn't have any command-line option to tell it to start in --dev mode | 21:31 |
| ianw | hrm, i guess it's sufficiently different to the setup of our container that ... i don't know :) | 21:32 |
| mordred | ianw: the upstream gerrit containers are quite a bit different than ours | 21:37 |
| mordred | they do a bunch of things in a container start script that my OCD found annoying and wanted to do at build time :) | 21:38 |
| ianw | mordred: apropos nothing to do with that ... i hit an issue connecting gerrit to mariadb with key lengths fixed with https://gerrit-review.googlesource.com/c/gerrit/+/297522 ... anyway given your skills in db optimisation might be of interest | 21:40 |
| ianw | as far as i can tell, all the upstream gerrit CI has failed for reasons not related to the change ... | 21:41 |
| mordred | ianw: the patch looks good. out of curiosity, how did that come up? | 21:42 |
| ianw | mordred: i'm looking at connecting us up to a container ... for several of the reasons presented in https://review.opendev.org/c/opendev/system-config/+/775961 :) | 21:42 |
| ianw | i've run some prelim memory stats on that too, importing the production tables it sits @ | 21:43 |
| ianw | df82e25bbb8c gerrit-compose_mariadb_1 0.09% 462.6MiB / 7.61GiB | 21:43 |
| ianw | similar after dumping the whole db. i'm not sure if maybe it sets its limits % wise based on total memory though, so putting it in a 64gb machine *may* have other affects | 21:44 |
| mordred | ianw: ++ cool. well - you mention it in the commit message, but this is likely one of the few cases where just using h2 would be fine - the data is "neat" but if it all DIAF nobody would care | 21:44 |
| clarkb | mordred: we were explicitly told to not use h2 | 21:44 |
| mordred | but also - just spinning up a local mariadb container seems great! | 21:44 |
| clarkb | because the performance is terrible I think it impacts user web browsing as it wants to update that for each file you look at | 21:44 |
| clarkb | I'm still very skeptical a local mariadb will work for us given the memory situation, but this sort of investigating can help us evaluate that | 21:45 |
| mordred | nod. I mean - gerrit also uses h2 for several of the caches | 21:45 |
| mordred | so I've been considering it a reasonable choice for stuff that has cache-level data importance in my brainhole | 21:46 |
| mordred | but I don't have strong opinions - mostly that that table is super annoying :P) | 21:46 |
| clarkb | ya its definitely not the most important data | 21:46 |
| clarkb | but luca specifically warned against it for performance reasons | 21:46 |
| mordred | nod | 21:47 |
| mordred | paying attention to luca is important | 21:47 |
| mordred | so - the table as it stands now is 460M ? | 21:47 |
| ianw | no, but that's the container memory stats after importing it and dumping it | 21:48 |
| clarkb | I think the raw data is closer to 5GB | 21:48 |
| clarkb | but I haven't looked at it recently | 21:48 |
| mordred | oh - gross | 21:48 |
| ianw | which i figure has read/written the whole thing, so probably reflects about worst case | 21:48 |
| ianw | (sorry, gotta run kids to school, back later) | 21:48 |
| clarkb | I think it tends to have very hot data and very cold data too | 21:49 |
| ianw | df82e25bbb8c gerrit-compose_mariadb_1 0.09% 67.32MiB / 7.61GiB | 21:49 |
| clarkb | because its going to be what people are actively reviewing at the moment | 21:49 |
| ianw | that's the quiescent container just started | 21:49 |
| ianw | smaller than i thought | 21:49 |
| clarkb | not the sort of db where you'd expect the whole thing to need to be in memory at all times | 21:50 |
| mordred | yeah. and in innodb, it'll really just be the hot pages loaded | 21:53 |
| mordred | mostly poking there to think about innodb buffer pool size - in this case it could be pretty small I think | 21:55 |
| ianw | mordred: in the usual yak shaving way, this all actually started from https://bugs.launchpad.net/ubuntu/+source/mysql-5.7/+bug/1914695 where an update stopped making mysqldump --all-databases stop working against our very ancient instance ... you might know somebody who might care about that? | 22:04 |
| openstack | Launchpad bug 1914695 in mysql-5.7 (Ubuntu) "mysqldump --all-databases not dumping any databases with 5.7.33" [Undecided,Fix released] | 22:04 |
| ianw | clarkb: changing the base job exposes us very widely to docker limit issues @ https://review.opendev.org/c/opendev/system-config/+/775051 ... | 22:07 |
| ianw | we're almost at the point there where the unreliability of it makes it impractical to merge reliably | 22:09 |
| clarkb | ianw:ya | 22:12 |
| clarkb | ianw: in the latest run though it failed to stop the container for some reason | 22:14 |
| clarkb | and the nested ara doesn't seem to have any std(out | 22:14 |
| clarkb | er std(out|err) to explain why | 22:14 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Try to make gatling-git work with our test gerrit https://review.opendev.org/c/opendev/system-config/+/775883 | 22:29 |
| *** slaweq_ has quit IRC | 22:40 | |
| ianw | clarkb: it looks like for graphite it couldn't load a certificate https://zuul.opendev.org/t/openstack/build/3094763cc5294a17beb6b03464120f24/log/graphite02.opendev.org/docker/graphite-docker_graphite_1.txt | 22:52 |
| clarkb | ianw: fyi waiting zuul to update zm01.opendev.org and service-borg-backup failed. It is unhappy about ansible_ssh_host_key_ed25519_public being undefined | 22:53 |
| ianw | hrm, is that a bad match? we shouldn't be running backup roles on zm hosts? | 22:54 |
| ianw | bad group match i mean | 22:54 |
| clarkb | ianw: it wasn't trying to run on zm01, but the zm01 change triggered all the jobs because the inventory updated | 22:54 |
| clarkb | it was running against hte proper hosts I think | 22:55 |
| ianw | oh right, ok. yeah something up, will look | 22:55 |
| clarkb | I wonder if that is a fact loading problem from one host to another | 22:55 |
| clarkb | like maybe we need a play in the playbook that just looks up facts fo the backup hosts first | 22:56 |
| clarkb | before doing anything on the other hsots | 22:56 |
| ianw | to the graphite job issue; it doesn't appear to collect acme.sh logs so i can't quite tell what went wrong. i really need to work on running a container with the fake acme thing for testing purposes | 22:57 |
| guillaumec | ianw, is it possible to pull an image from intermediate registry ? I have different behavior for system-config-run-review-3.2 and system-config-run-review-3.3. I tried to get the console log but it doesn't seem possible. So I would like to debut run the docker image myself | 22:59 |
| clarkb | guillaumec: it is the images are listed in the job artifacts list | 23:00 |
| clarkb | for the build I think | 23:00 |
| ianw | guillaumec: yes it should be an artifact, e.g. https://zuul.opendev.org/t/openstack/build/dd8a7627904045bdae8296a01cd05945/artifacts | 23:00 |
| guillaumec | ah nice, never clicked on that tab yet :) good to know | 23:01 |
| ianw | i was thinking it would be cool to make the zuul summary tab, for each job be a "clickdown" arrow (i don't know what the real term for that is) that then queries zuul and shows the job artifacts | 23:02 |
| ianw | if it was async and only queried when clicked, it wouldn't be too much load on the zuul web side | 23:02 |
| ianw | it's a little above my current polymer/javascript/gerrit-plugin skillset though :) | 23:03 |
| ianw | guillaumec: although it's some work to get it setup to the point that it has users and projects and changes. if you make a failing job with your change, i can arrange to put it on hold for you. that way zuul does all that setup | 23:05 |
| fungi | ianw: not sure if you saw me mention yesterday, but gitea01 and translate01 have been sending backup failure messages to us | 23:12 |
| fungi | i haven't had time to dig into logs, the e-mails don't really provide detail | 23:13 |
| clarkb | zm01.opendev.org has finished ansibling. Things look about right, but we don't auto start the service. I will start it now. Then if it looks happy stop the merger on zm01.openstack.org | 23:13 |
| clarkb | it appears to have started happily and is awaiting jobs | 23:14 |
| clarkb | I guess we have 7 other dedicated mergers so I should be fine to stop zm01.openstack.org now | 23:15 |
| clarkb | I'll do that | 23:15 |
| clarkb | thats done. Assuming everything continues to look happy I'll plan to try and do the other 7 tomorrow | 23:16 |
| clarkb | probably just do them all in one batch? they are small servers so doing them like that shouldn't be a huge deal | 23:16 |
| fungi | do we need to open gearman access for it in the scheduler's iptables rules? | 23:17 |
| clarkb | we shouldn't, that is all handled by the iptables ansibel role | 23:17 |
| clarkb | 2021-02-22 23:14:19,982 DEBUG zuul.BaseMergeServer: Registering 4 jobs <- that implies to connected to gearman just fine | 23:17 |
| clarkb | *it connected to | 23:17 |
| fungi | ahh, great | 23:18 |
| clarkb | updating the zuul scheduler itself becomes problematic for this reason though | 23:18 |
| clarkb | as everything needs to connect to it | 23:18 |
| clarkb | however, I think we may be able to get away with it since we'll give it a dns alias for the old openstack.org name. | 23:18 |
| clarkb | whcih means we can redeploy it, switch things over in dns, restart other services, they connect with the old name, then update services to use the new name | 23:19 |
| clarkb | I've been puzzling through some of that today and definitely don't have all the answers yet, but it should be doable | 23:19 |
| clarkb | now I just need to see zm01.opendev.org handle some jobs | 23:19 |
| ianw | fungi: ok, will look into | 23:28 |
| ianw | i'm just pulling up the backup logs for the other issue now | 23:28 |
| clarkb | I am having the hardest time convincing jsch to ssh to gerrit properly | 23:38 |
| clarkb | it says a bunch of "closing immediately" on the client side | 23:39 |
| openstackgerrit | Guillaume Chauvel proposed opendev/system-config master: [DNM] test comment width: review without autogenerated tag https://review.opendev.org/c/opendev/system-config/+/771798 | 23:40 |
| clarkb | now it is almost like 29418 isn't listening | 23:41 |
| openstackgerrit | Guillaume Chauvel proposed opendev/system-config master: [DNM] test comment width: review without autogenerated tag https://review.opendev.org/c/opendev/system-config/+/771798 | 23:41 |
| clarkb | ya got a bunch of connection refused | 23:42 |
| * clarkb waits for the 3.2 run to finish to check its logs | 23:42 | |
| *** DSpider has quit IRC | 23:43 | |
| fungi | local container-to-container networking? | 23:43 |
| fungi | or host-to-container? | 23:43 |
| clarkb | fungi: possibly, though I've split the job up to run the http pass first then do ssh (since it was hard to read the logs when they were all mixed together. And http was fine | 23:43 |
| clarkb | also I'm using host networking to avoid these problems | 23:43 |
| clarkb | and it should all be connections to localhost so the firewall shouldn't be getting in the way | 23:44 |
| fungi | ahh, okay, so probably not | 23:44 |
| clarkb | zm01.opendev.org has found some jobs \o/ it looks fine too | 23:46 |
| clarkb | we dno' | 23:47 |
| clarkb | er | 23:47 |
| clarkb | we don't need to preclone repos like nova do we? | 23:47 |
| clarkb | we fixed that timeout problem right? | 23:47 |
| fungi | we doubled the timeout to, like, 600 | 23:49 |
| guillaumec | ianw, understood why, gerrit has merged https://gerrit-review.googlesource.com/c/gerrit/+/296851 (80ch to 120ch) which is stable-3.2 and it's not in 3.3 atm, and sample reviews were not longer than 120 characters :) | 23:49 |
| clarkb | in the 3.2 case most pushes failed, but most clones and pulls succeeded | 23:51 |
| clarkb | that at least is progress | 23:51 |
| clarkb | I wonder if the http pass is polluting the repo and that is causing: Ref master already exists - null | 23:52 |
| clarkb | I guess I can try it with another psuh specific repo | 23:52 |
| * clarkb writes that up | 23:52 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Try to make gatling-git work with our test gerrit https://review.opendev.org/c/opendev/system-config/+/775883 | 23:54 |
| ianw | service-borg-backup.yaml.log.2021-02-22T22:47:02Z failed with 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'ansible_ssh_host_key_ed25519_public', but service-borg-backup.yaml.log.2021-02-22T07:10:24Z did not | 23:58 |
| ianw | i.e. sometimes it seems ansible is not populating the hostvars of the backup servers with the ssh_host_key info | 23:58 |
| clarkb | ianw: ya my previous comments were assuming it was something along those lines. I think that this has to do with how ansible loads facts for other hosts | 23:59 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!