| tonyb | Looks like gitea08 is fine now? | 00:33 |
|---|---|---|
| fungi | last i checked | 00:41 |
| tonyb | \o/ | 00:42 |
| tonyb | Was it the VM or some other "bad actor" ? | 00:42 |
| fungi | though more generally, it seems like some traffic through zayo london and atlanta may be having trouble getting to vexxhost sjc1 where the gitea servers are | 00:42 |
| tonyb | are they all in the same DC? | 00:44 |
| * tonyb doesn't go through London ;P https://paste.opendev.org/show/b8t6OfqPu1wzc374ikiR/ | 00:46 | |
| fungi | the gitea servers? yeah all in the same vexxhost cloud region | 00:48 |
| fungi | the gerrit server is in a different vexxhost region (ca-ymq-1) | 00:48 |
| ianw | all the giteas have also been rebooted in the last hour or so, just incase | 00:48 |
| fungi | oh, i missed that | 00:48 |
| ianw | fungi: heh, that was the idea -- i downed them in the lb and updated docker and rebooted | 00:49 |
| fungi | no new containers though | 00:49 |
| ianw | i'll quickly do the lb later when it's quiet | 00:49 |
| fungi | still doesn't have the donor logos | 00:49 |
| fungi | but we can wait for something else to merge to trigger another image pull on them | 00:49 |
| ianw | oh, sorry i didn't pull, it was just getting the new daemon | 00:49 |
| fungi | no worries, it's not pressing | 00:49 |
| fungi | was just curious to see them in production if they actually were | 00:50 |
| ianw | i have the up down scripted, i can run it again with a pull | 00:53 |
| tonyb | Okay, FWIW the problems I was seeing were that origin/https/gitea? were slow but gerrit/ssh were fine ... and for a while origin was behind gerrit (for >15mins) in terms of visible commits | 00:53 |
| fungi | that could be consistent with the slow transfer speeds and very high latency i was seeing through some parts of the zayo backbone | 00:59 |
| tonyb | Possibly (not that I use/transit Zayo?), it's all fixed now so. yay | 01:00 |
| fungi | i'm seeing fairly decent bandwith between gerrit and the vexxhost sjc1 network at the moment though, just testing http transfer of a test file | 01:01 |
| tonyb | Yeah it's all good for me right now | 01:02 |
| ianw | fungi: ok, the logos are there now :) | 01:04 |
| fungi | thanks! | 01:07 |
| fungi | https://opendev.org/#cloud-donors | 01:17 |
| tonyb | nice | 01:18 |
| tonyb | I have a workaround but I'm seeing gitea08 as slow again | 02:15 |
| tonyb | for i in 1 8 ; do time curl -s https://gitea0$i.opendev.org:3081/openstack/governance>/dev/null; done | 02:16 |
| tonyb | real0m0.919s | 02:16 |
| tonyb | real3m28.998s | 02:16 |
| opendevreview | Ian Wienand proposed opendev/system-config master: gerrit: increase ssh channel debugging https://review.opendev.org/c/opendev/system-config/+/873214 | 02:18 |
| opendevreview | Ian Wienand proposed opendev/system-config master: gerrit: increase ssh channel debugging https://review.opendev.org/c/opendev/system-config/+/873214 | 02:54 |
| ianw | load average: 1.16, 2.13, 22.38 | 03:02 |
| ianw | it's busy but not crazy | 03:02 |
| ianw | the first half of the executors have updated docker and rebooted | 03:09 |
| *** yadnesh|away is now known as yadnesh | 04:04 | |
| ianw | clarkb: looks like some types changed which breaks the master build with the ssh debug patch (https://gerrit.googlesource.com/gerrit/+/4f7ef7d590ba99342948fe6c26d24e4c3cd7003d%5E%21/#F0) | 05:27 |
| ianw | since i'd only propose we keep this on for a few days, until we catch an error or two, i'd propose we just leave the master build as broken for that period. if upstream wants to take it, obviously we can port it properly in gerrit and problem solved | 05:28 |
| ianw | (i'm talking about https://zuul.opendev.org/t/openstack/build/a5636f778eea4fd18fff6b80af17b884/console) | 05:33 |
| *** jpena|off is now known as jpena | 08:27 | |
| *** ysandeep is now known as ysandeep|afk | 09:52 | |
| *** ysandeep|afk is now known as ysandeep | 10:53 | |
| opendevreview | yatin proposed openstack/project-config master: Cache Cirros 0.6.1 images https://review.opendev.org/c/openstack/project-config/+/873244 | 11:36 |
| *** ysandeep is now known as ysandeep|afk | 11:55 | |
| *** ysandeep|afk is now known as ysandeep | 12:31 | |
| *** ysandeep is now known as ysandeep|afk | 13:21 | |
| *** dasm|offline is now known as dasm|rover | 13:47 | |
| opendevreview | Ron Stone proposed openstack/project-config master: Update StarlingX docs promote job for R8 release https://review.opendev.org/c/openstack/project-config/+/873266 | 14:22 |
| fungi | slittle1_: if you're good with ron's https://review.opendev.org/873266 i'll go ahead and approve it | 14:53 |
| fungi | slittle1: ^ (i guess you're in here twice at the moment) | 14:54 |
| *** yadnesh is now known as yadnesh|away | 14:58 | |
| *** ysandeep|afk is now known as ysandeep|PTO | 15:25 | |
| *** dasm|rover is now known as dasm|bbiab | 15:25 | |
| clarkb | ianw: ya we can break the master gerrit builds. I need to review your change. I'll try to get to that this morning. There was a zuul bugfix too | 15:44 |
| fungi | i should have waited another day to do my python recompiles. new 3.10, 3.11 and 3.12 prerelease last night | 16:13 |
| * fungi sighs | 16:13 | |
| *** dasm|bbiab is now known as dasm|rover | 16:28 | |
| clarkb | fungi: at least in the winter the extra heat output isn't so bad | 17:18 |
| fungi | too true | 17:21 |
| *** jpena is now known as jpena|off | 17:33 | |
| opendevreview | Ade Lee proposed zuul/zuul-jobs master: Add ubuntu to enable-fips role https://review.opendev.org/c/zuul/zuul-jobs/+/866881 | 18:08 |
| clarkb | I think one of our swift job log endpoint targets is having a sad resulting in POST_FAILURES for jobs. THe first one I've tracked down was for rax dfw | 19:01 |
| clarkb | trying to look at a few more to see if it is more widespread than that | 19:01 |
| opendevreview | Jeremy Stanley proposed zuul/zuul-jobs master: Add ubuntu to enable-fips role https://review.opendev.org/c/zuul/zuul-jobs/+/866881 | 19:02 |
| clarkb | now up to two failing in rax dfw | 19:02 |
| clarkb | let me check one more and it isn't also in another region I'll push a change for that region | 19:02 |
| fungi | thanks | 19:04 |
| fungi | standing by to approve. i guess we may need to bypass zuul and submit it directly in gerrit? | 19:04 |
| clarkb | third one is in rax_iad | 19:04 |
| clarkb | so maybe this is rackspace wide | 19:04 |
| fungi | just do all three | 19:04 |
| clarkb | ack incoming | 19:04 |
| opendevreview | Clark Boylan proposed opendev/base-jobs master: Temporarily disable rax swift log uploads https://review.opendev.org/c/opendev/base-jobs/+/873320 | 19:06 |
| clarkb | there is a decent chance that will need to be force merged but we can try to do it normally first | 19:06 |
| clarkb | fungi: ^ | 19:08 |
| fungi | clarkb: is commenting safe there? i thought ansible got cranky if you had a ' in a comment | 19:09 |
| fungi | but maybe it's only if the ' is mismatched? | 19:10 |
| clarkb | fungi: it was only if unmatched | 19:10 |
| clarkb | so this should be fine | 19:10 |
| fungi | i'll just bypass zuul in that case, thanks | 19:11 |
| opendevreview | Merged opendev/base-jobs master: Temporarily disable rax swift log uploads https://review.opendev.org/c/opendev/base-jobs/+/873320 | 19:11 |
| opendevreview | Merged zuul/zuul-jobs master: Add ubuntu to enable-fips role https://review.opendev.org/c/zuul/zuul-jobs/+/866881 | 19:11 |
| clarkb | I'm not seeing anything on their cloud status page that would indicate this is a known problem yet | 19:14 |
| fungi | #status log Bypassed testing to merge urgent change 873320 temporarily disabling uploads to one of our log storage providers which is exhibiting problems | 19:14 |
| opendevstatus | fungi: finished logging | 19:14 |
| clarkb | don't see email either so this is probably the leading edge of whatever it is and hopefull we'll have better indication of what is going on in a bit | 19:15 |
| clarkb | fungi: also 873320 threaded the needle huh? | 19:16 |
| clarkb | er sorry I mean 866881 | 19:16 |
| fungi | apparently | 19:17 |
| fungi | i've approved 872222 now since it was waiting on that | 19:17 |
| fungi | and then we can recheck and hopefully approve 872223 | 19:17 |
| fungi | and then start trying to exercise it and find out what else is still broken | 19:18 |
| opendevreview | Merged openstack/project-config master: Add base openstack FIPS job https://review.opendev.org/c/openstack/project-config/+/872222 | 19:22 |
| opendevreview | Clark Boylan proposed opendev/base-jobs master: Update base-test to only upload to rax https://review.opendev.org/c/opendev/base-jobs/+/873321 | 19:23 |
| clarkb | fungi: ^ landing that willallow us to check if rax is working via the base-test job more easily | 19:23 |
| ianw | ok zuul01/02 are the last two on the docker upgrade list | 19:38 |
| fungi | other than the list servers which i haven't finished yet | 19:39 |
| ianw | oh right; yep and lists01 -- the others aren't dockerised right? | 19:39 |
| fungi | right | 19:40 |
| ianw | fungi: if down/up is reliable i'm happy to do it -- just didn't want to step on anything in progress | 19:40 |
| fungi | i think it probably is, i believe the failure i hit on the held node was unrelated to the docker package upgrade and instead something incomplete with the domain filtering config | 19:42 |
| ianw | ok, i'll try ... | 19:43 |
| opendevreview | Merged opendev/base-jobs master: Update base-test to only upload to rax https://review.opendev.org/c/opendev/base-jobs/+/873321 | 19:43 |
| fungi | i guess go for it, worst case i jump into troubleshooting a config problem with the listserv but brief outages there aren't a disaster | 19:43 |
| fungi | thanks! | 19:44 |
| ianw | doing a quick reboot as it pulled a new kernel, it hadn't been up that long though | 19:46 |
| fungi | cool | 19:46 |
| clarkb | re zuul01/02 those should be straightforward as well | 19:47 |
| clarkb | https://review.opendev.org/c/zuul/zuul-jobs/+/680178 has been recheck to gather an intiail set of info on the rax failures | 19:48 |
| clarkb | really its just going to say "is this still happening or not" and thenwe can debug from there. | 19:48 |
| ianw | ok; https://lists.opendev.org/mailman3/lists/ seems up, the containers don't have anything crazy in logs | 19:49 |
| fungi | thanks! | 19:52 |
| ianw | it does say | 19:53 |
| ianw | "Your models in app(s): 'django_mailman3', 'hyperkitty', 'postorius' have changes that are not yet reflected in a migration, and so won't be applied." | 19:53 |
| ianw | do we need to manually apply migrations? | 19:53 |
| fungi | i thought i saw the compose running it at each start | 19:54 |
| ianw | hrm it does give conflicting information | 19:55 |
| ianw | Operations to perform: | 19:55 |
| ianw | Apply all migrations: account, admin, auth, contenttypes, django_mailman3, django_q, hyperkitty, postorius, sessions, sites, socialaccount | 19:55 |
| ianw | Running migrations: | 19:55 |
| ianw | No migrations to apply. | 19:55 |
| ianw | Your models in app(s): 'django_mailman3', 'hyperkitty', 'postorius' have changes that are not yet reflected in a migration, and so won't be applied. | 19:55 |
| ianw | that's the full output | 19:55 |
| ianw | i guess it's looking for a migration.py file to apply, and can't find one, but seems to know the models might need it | 19:56 |
| ianw | playing taxi, bib | 19:57 |
| ianw | so the entrypoint does run migrate -> https://github.com/maxking/docker-mailman/blob/main/web/docker-entrypoint.sh#L126 | 20:52 |
| ianw | but not makemigrations afaics. it does seem that is something that needs to be done as new versions are included | 20:53 |
| clarkb | the dnm base-test change ran 3 rax ord uploads and one each for dfw and iad. All were successful. I've rechecked again to gather more data, but its possible this blip was hsort lived? | 20:54 |
| fungi | ianw: interesting, we haven't actually upgraded mailman services since the initial deployment, so i wonder what we're lacking | 20:54 |
| ianw | fungi: yeah, i'm thinking run makemigrations and see what it spits out | 20:54 |
| clarkb | ianw: note we're on a fork, but one that tries to be close to upstream | 20:55 |
| clarkb | er of the docker images I mean | 20:55 |
| ianw | here's what it thinks is different | 20:56 |
| ianw | https://paste.opendev.org/show/bWnwl4DPOryqpGVNmbml/ | 20:56 |
| clarkb | wait so the application doesn't actually ship with proper migrations? I find django db stuff very confusing | 20:58 |
| ianw | yeah, i'm wondering how much of /usr/lib/python3.10/site-packages/django_mailman3/migrations/ is in the upstream container | 21:00 |
| ianw | i think that all these are changing an "id" field | 21:04 |
| ianw | perhaps to a models.BigAutoField | 21:05 |
| ianw | # Explicitly set default auto field type to avoid migrations in Django 3.2+ | 21:06 |
| ianw | default_auto_field = 'django.db.models.AutoField' | 21:06 |
| ianw | ^ other projects set that | 21:06 |
| clarkb | oh maybe because I set the value in our settings file to make a warning about it go away | 21:06 |
| clarkb | ? | 21:06 |
| clarkb | though I thought i set it to the default | 21:06 |
| ianw | clarkb: hrm, i don't see anything related to this in our settings files | 21:08 |
| clarkb | let me see if I can find it. | 21:08 |
| clarkb | ianw: opendev/system-config/playbooks/roles/mailman3/files/web-settings_local.py:DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField' | 21:09 |
| ianw | oh right, AUTO_FIELD not auto_field :) | 21:09 |
| clarkb | in particular the warning we got seemed to say big auto filed was the default so I set it to that iirc. But maybe it isn't the default and normal size is and now it wants to migrate us to the big size? | 21:10 |
| ianw | it looks like it changed in django 3.2+ | 21:13 |
| clarkb | ianw: not to distract from django, but looking at your java change to gerrit the change looks good. Thomas had a suggestion for an additional logging point though | 21:23 |
| clarkb | that said, I'm looking at the log files from our system-config-run 3.6 job and not seeing where those log lines end up | 21:25 |
| clarkb | might be a good idea to ensure we're writing those logs in the test job so we can confirm they are working before pushing to prod? | 21:26 |
| ianw | clarkb: so we need to turn up the logging flag before that will show | 21:27 |
| clarkb | aha | 21:27 |
| clarkb | ianw: seems like we should do that in CI anyway? would proably be useful in general there | 21:27 |
| ianw | yes, i can look at it. either via config, or you can do it on the live system with the gerrit log-level (i think, smoething like that) | 21:27 |
| clarkb | and ya no real concern from me to temporarily break the master builds. I think we may have done in the past anyway | 21:29 |
| opendevreview | Ian Wienand proposed opendev/system-config master: mailman: set web auto field size to "AutoField" https://review.opendev.org/c/opendev/system-config/+/873337 | 21:31 |
| ianw | now I've run makemigrations -- i didn't realise it would write to /usr. i want to be careful to *not* apply that. so I think i'll down the containers, which should delete them, and then up so they come up with only the migrations in the actual container image | 21:32 |
| ianw | i don't want it to somehow restart and apply the migration to upgrade all the id's | 21:33 |
| ianw | doing school run v2 ... bib | 21:33 |
| clarkb | makes sense especially if we're switching to the current value | 21:34 |
| clarkb | on the second pass of my dnm log check change all 5 uploaded to dfw | 22:16 |
| clarkb | I'm going to push up reverts now I guess and we can decide if we think this is sufficient checking | 22:16 |
| *** dasm|rover is now known as dasm|off | 22:18 | |
| opendevreview | Clark Boylan proposed opendev/base-jobs master: Revert "Temporarily disable rax swift log uploads" https://review.opendev.org/c/opendev/base-jobs/+/873339 | 22:18 |
| ianw | lgtm; we can always turn it off again | 22:22 |
| opendevreview | Merged opendev/base-jobs master: Revert "Temporarily disable rax swift log uploads" https://review.opendev.org/c/opendev/base-jobs/+/873339 | 22:29 |
| ianw | i made a merge request about the key type -> https://gitlab.com/mailman/django-mailman3/-/merge_requests/189 ... see what happpens | 22:31 |
| ianw | i've restarted mailman containers again. so it still has that warning, but we know what's going on now | 22:32 |
| ianw | i see that update on the gerrit change, i'll take a look | 22:39 |
| opendevreview | Ian Wienand proposed opendev/system-config master: gerrit: increase ssh channel debugging https://review.opendev.org/c/opendev/system-config/+/873214 | 23:29 |
| ianw | ^ that should set the logging flag. i couldn't see an api way to do that? | 23:30 |
| ianw | i mean http api, i guess the cmd-line is an api of sorts | 23:30 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!