| *** diablo_rojo_phone has joined #opendev | 00:01 | |
| fungi | we could move the more static parts of that site to afs too | 00:01 |
|---|---|---|
| fungi | serving from /afs/openstack.org/projects/eavesdrop/... (maybe via symlink) would make that a much simpler transition | 00:02 |
| fungi | then all the yaml2ical bits could just turn into typical docs/content publication jobs | 00:02 |
| fungi | for that matter, the bot could also switch to writing its logs into afs, i suppose, if we wanted to get extra fancy, but serving a mix of local files and afs from that vhost would be trivial anyway | 00:03 |
| *** open10k8s has quit IRC | 00:06 | |
| *** open10k8s has joined #opendev | 00:07 | |
| *** open10k8s has quit IRC | 00:17 | |
| *** open10k8s_ has joined #opendev | 00:18 | |
| *** whoami-rajat has quit IRC | 00:20 | |
| ianw | i don't know if i'd like to take on the bot as AFS writer | 00:26 |
| ianw | but the yaml2ical bits would probably be quite helpful to just serve out of afs | 00:26 |
| ianw | we could just have eavesdrop show it's logs as irclogs.opendev.org, serve eavesdrop.openstack/dev.org from AFS and then redirect eavesdrop.openstack.org/irclogs -> irclogs.opendev.org | 00:26 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Create ircbot container https://review.opendev.org/c/opendev/system-config/+/793704 | 00:28 |
| opendevreview | Ian Wienand proposed opendev/system-config master: limnoria setup on eavesdrop https://review.opendev.org/c/opendev/system-config/+/793719 | 00:28 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Disable puppet runs on eavesdrop https://review.opendev.org/c/opendev/system-config/+/794051 | 00:28 |
| *** seongsoocho has joined #opendev | 00:31 | |
| fungi | ianw: yep, either use separate domain names, or make the eavesdrop server a read-only afs client to get the static site content, whichever is easier | 00:36 |
| opendevreview | Ian Wienand proposed opendev/irc-meetings master: Create irc-meetings-ical-tox-publish job https://review.opendev.org/c/opendev/irc-meetings/+/794052 | 01:01 |
| *** open10k8s_ has quit IRC | 01:07 | |
| *** open10k8s has joined #opendev | 01:08 | |
| fungi | also it'll be a great opportunity to switch to an opendev.org dns name | 01:10 |
| opendevreview | Ian Wienand proposed opendev/irc-meetings master: Create irc-meetings-ical-tox-publish job https://review.opendev.org/c/opendev/irc-meetings/+/794052 | 01:19 |
| *** ysandeep has joined #opendev | 01:24 | |
| opendevreview | Ian Wienand proposed openstack/project-config master: Add promote-irc-meetings-ical https://review.opendev.org/c/openstack/project-config/+/794075 | 01:30 |
| opendevreview | Ian Wienand proposed opendev/irc-meetings master: Add static site promote job https://review.opendev.org/c/opendev/irc-meetings/+/794076 | 01:33 |
| opendevreview | Merged opendev/irc-meetings master: Create irc-meetings-ical-tox-publish job https://review.opendev.org/c/opendev/irc-meetings/+/794052 | 01:35 |
| *** timburke has quit IRC | 01:38 | |
| *** timburke has joined #opendev | 01:54 | |
| opendevreview | Merged openstack/project-config master: Add promote-irc-meetings-ical https://review.opendev.org/c/openstack/project-config/+/794075 | 02:04 |
| *** boistordu has joined #opendev | 02:11 | |
| *** ysandeep is now known as ysandeep|afk | 02:11 | |
| *** boistordu_ex has quit IRC | 02:17 | |
| opendevreview | Merged opendev/irc-meetings master: Add static site promote job https://review.opendev.org/c/opendev/irc-meetings/+/794076 | 02:51 |
| opendevreview | Ian Wienand proposed opendev/zone-opendev.org master: Add meetings.opendev.org https://review.opendev.org/c/opendev/zone-opendev.org/+/794085 | 02:56 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Serve meetings.opendev.org https://review.opendev.org/c/opendev/system-config/+/794086 | 02:57 |
| *** brtknr has quit IRC | 03:00 | |
| ianw | fungi: ^ so that is publishing (https://static.opendev.org/project/meetings.opendev.org/). i just wanted to get it working, so i'll take advice if you want to change the name | 03:00 |
| *** brtknr has joined #opendev | 03:00 | |
| ianw | i'm thinking for production we move eavesdrop.openstack.org to static, and redirect everything to meetings.opendev.org; except /irclogs which we proxy from eavesdrop01.openstack.org until we implement the new server | 03:07 |
| fungi | that all sounds fine to me, no problem with the name either | 03:15 |
| fungi | i'll hopefully have some time to look it over when i wake up | 03:16 |
| fungi | thanks for getting it going! | 03:16 |
| ianw | see you tomorrow! | 03:17 |
| ianw | another week of lockdown, home learning, etc. here :( | 03:17 |
| fungi | sorry to hear that, hope it improves soon | 03:21 |
| ianw | i'm at least booked for a vaccine shot, but not till 15th | 03:22 |
| fungi | that's sounding promising | 03:23 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Add static eavesdrop.openstack.org site https://review.opendev.org/c/opendev/system-config/+/794089 | 03:34 |
| opendevreview | Ian Wienand proposed openstack/project-config master: Remove publish-irc-meetings job https://review.opendev.org/c/openstack/project-config/+/794091 | 03:46 |
| *** ykarel|away has joined #opendev | 03:51 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: Serve meetings.opendev.org https://review.opendev.org/c/opendev/system-config/+/794086 | 03:56 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Add static eavesdrop.openstack.org site https://review.opendev.org/c/opendev/system-config/+/794089 | 03:56 |
| *** ykarel|away is now known as ykarel | 04:04 | |
| ianw | fungi / clarkb : to summarise. the irc-meetings content currently served at http://eavesdrop.openstack.org is now available and published by irc-meetings at /afs/openstack.org/project/meetings.opendev.org (https://review.opendev.org/c/opendev/irc-meetings/+/794076) | 04:10 |
| ianw | i'm proposing we publish that at meetings.opendev.org (https://review.opendev.org/c/opendev/system-config/+/794086) | 04:12 |
| ianw | then, we make eavesdrop.openstack.org redirect to meetings.opendev.org, but with /irclogs proxied to eavesdrop01.openstack.org (https://review.opendev.org/c/opendev/system-config/+/794089) | 04:13 |
| ianw | then, in WIP, we run meetbot on a new server out of a container under limnoria. we probably publish channels logs under irclogs.opendev.org or something like that. we can then redirect eavesdrop.openstack.org/irclogs there | 04:14 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Add static eavesdrop.openstack.org site https://review.opendev.org/c/opendev/system-config/+/794089 | 04:18 |
| opendevreview | Merged opendev/system-config master: Run system-config-run-lists when testinfra changes https://review.opendev.org/c/opendev/system-config/+/794037 | 04:38 |
| *** ysandeep|afk is now known as ysandeep|ruck | 04:57 | |
| *** marios has joined #opendev | 05:04 | |
| ysandeep|ruck | #opendev limestone mirror is again affecting us. | 05:50 |
| ysandeep|ruck | - Curl error (28): Timeout was reached for http://mirror.regionone.limestone.opendev.org:8080/rdo/centos8/component/tripleo/12/5a/125a6fb9942bdcc10a31537386c5b3956bb7d694_85b09f68/repodata/repomd.xml [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds] | 05:50 |
| ysandeep|ruck | example: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_0fe/793806/2/check/tripleo-ci-centos-8-content-provider/0fe430c/logs/undercloud/home/zuul/workspace/logs/container-builds/9cdb8cb2-3374-4e68-8b8c-5e650865c089/base/os/designate-base/designate-sink/designate-sink-build.log | 05:50 |
| opendevreview | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/794136 | 06:08 |
| *** ralonsoh has joined #opendev | 06:10 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: Add static eavesdrop.openstack.org site https://review.opendev.org/c/opendev/system-config/+/794089 | 06:14 |
| *** bhagyashris has joined #opendev | 06:16 | |
| ianw | hrm, i can't really see any errors on the mirror node, i'm logged in | 06:16 |
| ianw | load average: 14.05, 8.42, 7.65 | 06:18 |
| ianw | i would say that's quite high | 06:18 |
| ianw | there's nothing using CPU | 06:18 |
| *** timburke has quit IRC | 06:20 | |
| ianw | 104857600 bytes (105 MB, 100 MiB) copied, 0.257419 s, 407 MB/s | 06:27 |
| ianw | writing a file to disk, but now it's stuck trying to rm it | 06:28 |
| *** mrunge_ has joined #opendev | 06:31 | |
| ianw | now i can't ping it at all, although the console shows it as up | 06:33 |
| ianw | (nova console) | 06:33 |
| *** mrunge has quit IRC | 06:35 | |
| ianw | it's come back up but afs is dead. i bet it's the cache. i'll clear and reboot | 06:38 |
| ianw | i bet this was the cause of the retries i mentioned in #zuul | 06:40 |
| ysandeep|ruck | ianw: thank you for looking into it | 06:43 |
| ianw | ok it looks like something to do with the attached storage volumes. performance to the root disk is fine, but operations on /var/cache/openafs are taking an eternity | 06:46 |
| *** dklyle has quit IRC | 06:49 | |
| opendevreview | Ian Wienand proposed openstack/project-config master: Disable limestone due to mirror issues https://review.opendev.org/c/openstack/project-config/+/794142 | 06:51 |
| opendevreview | Ian Wienand proposed openstack/project-config master: Revert "Disable limestone due to mirror issues" https://review.opendev.org/c/openstack/project-config/+/794143 | 06:51 |
| ianw | i'm out of time to look, it's pretty easy to replicate with any operations on /var/cache/openafs | 06:53 |
| ianw | #status log disabled limestone due to mirror issues, appears to be slow operations on cache volume | 06:54 |
| opendevstatus | ianw: finished logging | 06:54 |
| ianw | i've sent a msg to logan- as the listed contact | 07:00 |
| opendevreview | Merged zuul/zuul-jobs master: Handle no-sudo in stage-output https://review.opendev.org/c/zuul/zuul-jobs/+/793829 | 07:03 |
| opendevreview | Merged zuul/zuul-jobs master: Tidy up file matcher for bindep jobs https://review.opendev.org/c/zuul/zuul-jobs/+/793834 | 07:04 |
| *** hashar has joined #opendev | 07:06 | |
| ysandeep|ruck | ianw: >> log disabled limestone - we should not hit the issues anymore, right? Can we post rechecks? | 07:07 |
| ysandeep|ruck | ignore.. i see https://review.opendev.org/c/openstack/project-config/+/794142 is in gate | 07:08 |
| ysandeep|ruck | ianw++ thanks! | 07:11 |
| ianw | yeah it will have to merge and then deploy | 07:12 |
| opendevreview | Merged openstack/project-config master: Disable limestone due to mirror issues https://review.opendev.org/c/openstack/project-config/+/794142 | 07:12 |
| ianw | i just manually edited it in anyway, so should be good | 07:13 |
| ysandeep|ruck | ack o/ | 07:15 |
| *** andrewbonney has joined #opendev | 07:19 | |
| *** bhagyashris has quit IRC | 07:23 | |
| *** bhagyashris has joined #opendev | 07:23 | |
| *** amoralej|off is now known as amoralej | 07:24 | |
| *** bhagyashris_ has joined #opendev | 07:38 | |
| *** ysandeep|ruck is now known as ysandeep|lunch | 07:41 | |
| *** bhagyashris has quit IRC | 07:44 | |
| *** kinpaa12389 has joined #opendev | 07:56 | |
| *** jpena|off is now known as jpena | 07:58 | |
| *** kinpaa12389 has quit IRC | 08:01 | |
| *** ykarel_ has joined #opendev | 08:07 | |
| *** ykarel has quit IRC | 08:13 | |
| *** lucasagomes has joined #opendev | 08:14 | |
| *** ykarel_ is now known as ykarel | 08:16 | |
| *** lucasagomes has quit IRC | 08:19 | |
| *** lucasagomes has joined #opendev | 08:26 | |
| *** lucasagomes has quit IRC | 08:27 | |
| *** ykarel is now known as ykarel|lunch | 08:38 | |
| *** ykarel_ has joined #opendev | 08:47 | |
| *** ykarel|lunch has quit IRC | 08:47 | |
| *** lucasagomes has joined #opendev | 08:56 | |
| *** lucasagomes has quit IRC | 08:58 | |
| *** ysandeep|lunch is now known as ysandeep | 09:04 | |
| *** ysandeep is now known as ysandeep|ruck | 09:04 | |
| *** mgoddard_ has joined #opendev | 09:13 | |
| *** ykarel_ has quit IRC | 09:15 | |
| *** mgoddard_ is now known as mgoddard | 09:18 | |
| opendevreview | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/794136 | 09:24 |
| *** odyssey4me has joined #opendev | 09:25 | |
| opendevreview | Matthias Runge proposed openstack/project-config master: Re-add publish jobs after panko deprecation https://review.opendev.org/c/openstack/project-config/+/793891 | 09:27 |
| *** lucasagomes has joined #opendev | 09:34 | |
| *** lucasagomes has quit IRC | 09:37 | |
| ianw | i think we might be getting to too many certificates on static.o.o now. 794089 seems to be failing with one cert or another not being provisioned. i think we can probably consolidate into one cert, i think it's up to 100 names? | 09:41 |
| *** lucasagomes has joined #opendev | 09:42 | |
| *** lucasagomes has quit IRC | 09:44 | |
| *** hashar has quit IRC | 09:53 | |
| *** boistordu_old has joined #opendev | 10:14 | |
| *** boistordu has quit IRC | 10:14 | |
| *** ralonsoh has quit IRC | 10:17 | |
| *** ralonsoh has joined #opendev | 10:17 | |
| opendevreview | Merged opendev/irc-meetings master: Schedule an extra Nova team meeting https://review.opendev.org/c/opendev/irc-meetings/+/794010 | 10:19 |
| *** ykarel_ has joined #opendev | 10:20 | |
| *** ykarel_ is now known as ykarel | 10:21 | |
| *** lucasagomes has joined #opendev | 10:32 | |
| *** lucasagomes has quit IRC | 10:47 | |
| *** whoami-rajat has joined #opendev | 10:49 | |
| *** lucasagomes has joined #opendev | 10:51 | |
| *** artom has quit IRC | 10:57 | |
| *** icey has quit IRC | 10:59 | |
| *** artom has joined #opendev | 10:59 | |
| mnaser | ianw: i misread that as 'now that we've hit 794089 certs, things are failing for too many certs' :P | 11:19 |
| *** jpena is now known as jpena|lunch | 11:33 | |
| *** icey has joined #opendev | 11:33 | |
| *** arxcruz|rover has joined #opendev | 11:35 | |
| *** lucasagomes has quit IRC | 11:35 | |
| *** lucasagomes has joined #opendev | 11:37 | |
| *** lucasagomes has quit IRC | 11:55 | |
| *** lucasagomes has joined #opendev | 12:01 | |
| *** hashar has joined #opendev | 12:05 | |
| *** bhagyashris_ is now known as bhagyashris | 12:09 | |
| *** amoralej is now known as amoralej|lunch | 12:12 | |
| *** jpena|lunch is now known as jpena | 12:32 | |
| *** amoralej|lunch is now known as amoralej | 13:11 | |
| *** ykarel has quit IRC | 13:30 | |
| *** ykarel has joined #opendev | 13:30 | |
| *** lucasagomes has quit IRC | 13:38 | |
| *** lucasagomes has joined #opendev | 13:48 | |
| *** tosky has joined #opendev | 13:54 | |
| *** zbr has quit IRC | 13:55 | |
| *** zbr has joined #opendev | 13:55 | |
| opendevreview | Merged openstack/project-config master: Remove x/gearman-plugin from infrastructure https://review.opendev.org/c/openstack/project-config/+/793459 | 14:17 |
| hashar | \o/ | 14:24 |
| hashar | corvus: mordred: in case you are around, the nodepool instance for Gerrit CI are somehow idle/stall and Lucas lack rights to delete them or look at the logs. | 14:26 |
| hashar | the nodes are all idle https://ci.gerritcodereview.com/t/gerrit/nodes and of course nothing can run ;] | 14:27 |
| hashar | the thread on their Slack is https://app.slack.com/client/TE1EA8WG3/CE2V1QU1L/thread/CE2V1QU1L-1622216537.029700 | 14:27 |
| clarkb | ianw: 100 names per cert sounds about right | 14:28 |
| corvus | hashar: thanks, i mean, i'm not going to notice it if they just talk about it in slack :( | 14:30 |
| hashar | corvus: maybe the person that raised it wasn't aware of who is maintaining it or where to raise it | 14:35 |
| hashar | I commented there a few days ago and only figured out I could ping you here :-\ | 14:35 |
| corvus | hashar: well, it looked like nasser pinged me on slack, but that's not useful if i'm not logged in | 14:35 |
| hashar | indeed :\ | 14:36 |
| *** opendevreview has quit IRC | 14:38 | |
| *** opendevreview has joined #opendev | 14:42 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM check mailman setup on bionic https://review.opendev.org/c/opendev/system-config/+/794239 | 14:42 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM check mailman against focal https://review.opendev.org/c/opendev/system-config/+/794240 | 14:42 |
| clarkb | I'm starting to collect notes at https://etherpad.opendev.org/p/listserv-inplace-upgrade-testing-2021 and realized that I should ensure that config management works against our targets as well | 14:43 |
| *** ykarel is now known as ykarel|away | 14:44 | |
| *** dklyle has joined #opendev | 14:53 | |
| opendevreview | Roman Dobosz proposed ttygroup/gertty master: Bump python version. https://review.opendev.org/c/ttygroup/gertty/+/794241 | 14:54 |
| *** david-lyle has joined #opendev | 14:57 | |
| *** dklyle has quit IRC | 14:57 | |
| *** david-lyle is now known as dklyle | 15:00 | |
| *** ykarel|away has quit IRC | 15:11 | |
| *** timburke has joined #opendev | 15:32 | |
| fungi | looks like the +o channelmode could sub in for the +O access flag we relied on from freenode so that meetbot could go back to updating channel topics: https://www.oftc.net/ChannelModes/ | 15:40 |
| fungi | though at this point the limnoria based implementation switching to coordinating topic changes through chanserv would be cleaner | 15:41 |
| *** david-lyle has joined #opendev | 15:45 | |
| *** dklyle has quit IRC | 15:45 | |
| *** ysandeep|ruck is now known as ysandeep|dinner | 15:57 | |
| fungi | running some errands and i'll also be working from the car while i'm waiting for christine to get out of an appointment, so my responsiveness may be spotty for the next few hours | 16:00 |
| *** marios has quit IRC | 16:00 | |
| *** ykarel|away has joined #opendev | 16:11 | |
| *** lucasagomes has quit IRC | 16:13 | |
| *** cloudnull has quit IRC | 16:13 | |
| *** cloudnull has joined #opendev | 16:15 | |
| opendevreview | Sankar Palanivel proposed zuul/zuul-jobs master: TEST: Do not merge https://review.opendev.org/c/zuul/zuul-jobs/+/794268 | 16:22 |
| *** bhagyashris_ has joined #opendev | 16:26 | |
| *** timburke has quit IRC | 16:32 | |
| *** david-lyle is now known as dklyle | 16:33 | |
| *** david-lyle has joined #opendev | 16:33 | |
| *** bhagyashris has quit IRC | 16:34 | |
| *** ykarel|away has quit IRC | 16:36 | |
| *** tosky has quit IRC | 16:48 | |
| *** david-lyle has quit IRC | 16:48 | |
| *** bhagyashris_ is now known as bhagyashris | 16:50 | |
| *** ysandeep|dinner is now known as ysandeep | 16:50 | |
| *** ysandeep is now known as ysandeep|ruck | 16:50 | |
| fungi | anybody happen to know whether the remove command works here on oftc? i'm not seeing it listed in the `/quote help` output, but then again that doesn't list it on freenode either and our docs claim it works there | 16:53 |
| johnsom | Shout out to all of those involved in the IRC move. From a user perspective it went pretty smooth. | 16:54 |
| fungi | thanks for the vote of confidence, johnsom! | 16:54 |
| fungi | we're still in progress updating documentation, processes and policies, as well as working through some less urgent changes to our bots | 16:54 |
| fungi | but yeah it went pretty well | 16:55 |
| *** jpena is now known as jpena|off | 17:03 | |
| *** ykarel|away has joined #opendev | 17:04 | |
| *** fungicide has joined #opendev | 17:07 | |
| *** ChanServ sets mode: +o fungi | 17:08 | |
| *** ChanServ sets mode: -o fungi | 17:10 | |
| *** erbarr has joined #opendev | 17:10 | |
| fungi | yeah, the server-side remove command doesn't appear to be supported, so i'll clean that up in the docs | 17:10 |
| *** open10k8s has quit IRC | 17:16 | |
| *** fungicide has quit IRC | 17:16 | |
| *** ralonsoh has quit IRC | 17:17 | |
| *** open10k8s has joined #opendev | 17:17 | |
| *** ykarel|away has quit IRC | 17:21 | |
| *** timburke has joined #opendev | 17:24 | |
| *** amoralej is now known as amoralej|off | 17:29 | |
| fungi | oof, gerrit server's load average is up around/over 40 | 17:34 |
| fungi | may be memory pressure, kswapd is fairly active, though iowait isn't sky high or anything | 17:35 |
| *** ysandeep|ruck is now known as ysandeep | 17:36 | |
| fungi | i only noticed because trying to submit a stack of irc doc changes for system-config took >5min and then returned an internal server error | 17:38 |
| *** ysandeep is now known as ysandeep|away | 17:39 | |
| *** ysandeep|away has quit IRC | 17:39 | |
| fungi | though apparently it did push them and gerritbot just never received an event | 17:40 |
| *** open10k8s_ has joined #opendev | 17:40 | |
| fungi | zuul didn't either, looks like | 17:43 |
| fungi | load is slowly dropping | 17:43 |
| *** open10k8s has quit IRC | 17:46 | |
| *** andrewbonney has quit IRC | 17:49 | |
| *** dklyle has quit IRC | 17:49 | |
| clarkb | we need to get back to review02. I think the sql db container stuff is still the next step there | 17:49 |
| *** dklyle has joined #opendev | 17:49 | |
| clarkb | looks like ansible mailman passes testing on bionic and focal. That means my next step is manually testing the upgrade on the held test servers. I'm going to dig into that shortly | 17:50 |
| fungi | load has climbed back up a bit on review01, kswapd seems to be consuming most of a cpu | 17:50 |
| clarkb | fungi: looks like deep stacks of nova and glance changes fwiw | 17:51 |
| fungi | yeah, though also dockerd, containerd, apache and unbound processes are using a fair amount of memory | 17:51 |
| clarkb | huh I wouldn't expect most of those to need much memory | 17:52 |
| clarkb | fungi: is it vmm or resident? | 17:52 |
| fungi | top reports the virtual memory usage for dockerd is 2286020 | 17:52 |
| fungi | apache processes and containerd are in a similar range there | 17:53 |
| clarkb | vmm shouldn't matter too much right? it is resident that is important | 17:53 |
| fungi | for resident, dockerd is only using 20552 | 17:53 |
| fungi | but if it needs anything from the vmm it needs to page that in, right? | 17:53 |
| clarkb | yes, pages would be allocated for that memory if they get used | 17:54 |
| fungi | going dark again, need to drive around some more, will be on again once i get back to the house | 17:54 |
| corvus | regarding the ping from hashar earlier; i have now set up a bridge from acmegating.com's matrix homeserver to gerritcodereview's slack. if you use both, let me know and i can invite you to the bridge room. | 18:33 |
| corvus | (er by use both, i mean any matrix service; homeserver doesn't matter; @:matrix.org accounts are fine) | 18:33 |
| corvus | (mostly mentioned the homeser bit as information about what extra things you can do when you run a homeserver) | 18:34 |
| hashar | nice corvus! | 18:34 |
| mordred | in fact, I am now connected to the gerrit slack rooms via corvus' homeserver from my account on my homeserver | 18:35 |
| corvus | matrix: putting the 'relay' back in internet relay chat | 18:36 |
| *** lowercase has joined #opendev | 18:44 | |
| lowercase | opendev website down? | 18:45 |
| mordred | lowercase: it's up for me? | 18:45 |
| lowercase | Its just spinning for me, and my servers aren't able to reach it | 18:45 |
| lowercase | let me check from a proxy | 18:45 |
| lowercase | works on my phone | 18:46 |
| fungi | maybe one of the backends is overloaded, checking them all | 18:46 |
| lowercase | I'm in the middle of an upgrade, so im hitting the openvdev.org/openstack/nova right now | 18:47 |
| fungi | yeah, gitea08 looks like it's struggling | 18:51 |
| fungi | lowercase: you aren't using openstack-ansible by any chance, are you? | 18:51 |
| lowercase | I am using openstack-ansible. | 18:52 |
| fungi | if so, we've already seen at least two sites hit an osa bug of some kind which causes all their servers to clone directly from opendev.org rather than from the local cache | 18:52 |
| lowercase | I can say, in my logs i see: Command errored out with exit status 128: git clone -q https://opendev.org/openstack/nova | 18:52 |
| lowercase | which is why I am here, figuered there was an outage. | 18:52 |
| fungi | i'm willing to bet that if we compare client ip addresses in our logs, you're overloading the gitea08 backend right now with too many machines trying to clone it at the same time | 18:52 |
| lowercase | kk, I COULD replicate your git repositories into my local git repos and set a have it pull updates. | 18:53 |
| fungi | i recommend stopping it (so we don't have to block the client addresses which are currently taking the service offline) and talk to folks in #openstack-ansible to troubleshoot the behavior | 18:53 |
| fungi | from what i've been told, openstack-ansible is supposed to maintain a central cache which all the servers pull from | 18:54 |
| lowercase | That would be ideal. | 18:54 |
| fungi | and supposedly if all the servers are cloning from us independently, then something has happened to that central cache | 18:55 |
| mordred | yeah- even without the risk of overloading -such a cache would be much more efficient/faster | 18:55 |
| clarkb | side note: you can inspect the cert for your connection to determine which backend you are hitting. We add appropriate altnames | 18:55 |
| fungi | that is, if you can get it to connect at all ;) | 18:55 |
| clarkb | ya | 18:55 |
| lowercase | i just pm'd you my ip | 18:55 |
| clarkb | I've successfully upgraded the test lists.k.i to bionic for some value of successfully :) | 18:55 |
| clarkb | This feels useful because it is allowing me to check the simpler things but I'm noticing that our test nodes are sufficiently different from the real nodes that I Think I should plan to do the snapshot upgrade tests too | 18:56 |
| mordred | clarkb: did a meteor hit your home as a result? if not, I consider it success ;) | 18:56 |
| clarkb | mordred: ha, no so I win! | 18:56 |
| clarkb | I'll be doing bionic -> focal upgrade testing after eating some lunch | 18:56 |
| lowercase | clarkb: i just did a dozen servers bionic -> focal, would you like my version of a bash oneliner to do it? | 18:58 |
| clarkb | lowercase: no thats ok. we'll want to walk through it anyway and make sure our specific server setup is happy with it | 18:59 |
| *** hashar has quit IRC | 19:20 | |
| *** tosky has joined #opendev | 19:23 | |
| fungi | jrosser: i've lost track of what the results were for ad hoc testing of fabricated VERSION_INFO values in /etc/ci/mirror_info.sh a couple of weeks ago. did it solve the problem, and/or has the problem been solved in other ways since? | 19:26 |
| jrosser | hi | 19:27 |
| jrosser | well, we have a WIP patch for osa which currently contains this https://review.opendev.org/c/openstack/openstack-ansible/+/783606/22/scripts/bootstrap-ansible.sh | 19:28 |
| jrosser | and thats sufficient to get things moving beyond failing at the missing version stuff | 19:29 |
| fungi | okay, i had notes that if https://review.opendev.org/791176 fixed it via depends-on with a dnm change to set base-test as a parent, then we could safely merge the wip revert of that at https://review.opendev.org/791177 and propose a similar edit to the mirror-info role | 19:30 |
| jrosser | we've not got as far as merging a complete job for bullseye yet | 19:39 |
| jrosser | so you could take 783606 and make a new DNM change from it without the bootstrap-ansible adjustment, and if it can be made to use base-test we should get a result | 19:40 |
| fungi | yeah, i'll give that a shot. thanks | 19:40 |
| jrosser | which should be a one liner here https://github.com/openstack/openstack-ansible/blob/master/zuul.d/jobs.yaml#L18 | 19:40 |
| jrosser | lowercase: if you could join us in #openstack-ansible and help work out what is going on it would be massively helpful | 19:43 |
| jrosser | it's kind of late for me now today but EU tz tomorrow is do-able | 19:44 |
| jrosser | lowercase: did you upgrade your control plane to focal before doing a bunch of computes? | 19:47 |
| *** slaweq has quit IRC | 20:34 | |
| *** dhellmann has quit IRC | 20:58 | |
| clarkb | infra-root ok I managed to get a test lists.k.i all the way to focal without too much trouble https://etherpad.opendev.org/p/listserv-inplace-upgrade-testing-2021 if you have a chance can you ssh into that node and look at it to see if anything seems terribly wrong? the ip address is on the etherpad | 21:03 |
| *** dhellmann has joined #opendev | 21:03 | |
| clarkb | I need to pop out again, but will try to do the same with the test lists.o.o shortly as that server is a bit different | 21:03 |
| *** david-lyle has joined #opendev | 21:18 | |
| *** dklyle has quit IRC | 21:18 | |
| *** amoralej|off has quit IRC | 21:22 | |
| opendevreview | Ade Lee proposed zuul/zuul-jobs master: Add role to enable FIPS on a node https://review.opendev.org/c/zuul/zuul-jobs/+/788778 | 21:30 |
| *** amoralej|off has joined #opendev | 21:38 | |
| *** amoralej|off has quit IRC | 22:12 | |
| ianw | did we get any feedback on the limestone mirror? | 22:16 |
| clarkb | ianw: I haven't seen any (I also missed the original disabling) | 22:17 |
| opendevreview | Ade Lee proposed zuul/zuul-jobs master: Add role to enable FIPS on a node https://review.opendev.org/c/zuul/zuul-jobs/+/788778 | 22:18 |
| ianw | my very scientific testing of "ls of empty directory on mounted volume doesn't take > 1 minute" suggests it has maybe been fixed. i'll reboot it and see if comes back to life | 22:23 |
| fungi | i have a feeling it's cinder over iscsi, and whatever network iscsi is riding across is getting clogged from time to time | 22:25 |
| ianw | A start job is running for OpenAFS client (1min 27s / 1min 50s) on the boot console isn't too great | 22:27 |
| corvus | i feel like at this point "A start job is running" is a euphemism for "get out your rescue disk" | 22:28 |
| ianw | http://mirror.regionone.limestone.opendev.org:8080/rdo/centos8/component/tripleo/12/5a/125a6fb9942bdcc10a31537386c5b3956bb7d694_85b09f68/repodata/repomd.xml came back immediately | 22:28 |
| ianw | i think it's back to being ok; i'd say we can re-enable, it's not doing anything funny like it was last night | 22:29 |
| fungi | i suppose that could also explain the reports we've been getting about read timeouts hitting the proxy paths | 22:30 |
| fungi | if apache can't get a timely read from its cachedir | 22:31 |
| fungi | i was thinking it was likely to be network connectivity between the mirror and pypi/fastly, but didn't consider that there could be local filesystem problems | 22:32 |
| ianw | i've approved the revert and will keep an eye for retrying jobs etc. | 22:34 |
| ianw | in typical fashion i got distracted and more worried about the format of the retry message than the underlying issue :) (https://review.opendev.org/c/zuul/zuul/+/794081) | 22:35 |
| opendevreview | Merged openstack/project-config master: Revert "Disable limestone due to mirror issues" https://review.opendev.org/c/openstack/project-config/+/794143 | 22:41 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Add static eavesdrop.openstack.org site https://review.opendev.org/c/opendev/system-config/+/794089 | 22:41 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Add static eavesdrop.openstack.org site https://review.opendev.org/c/opendev/system-config/+/794089 | 23:19 |
| *** lowercase has quit IRC | 23:23 | |
| *** tosky has quit IRC | 23:25 | |
| ianw | i'm going to go ahead with the meeting.opendev.org site as i've had positive responses on that name from fungi and clarkb | 23:26 |
| ianw | i think i've got rid of the typos in https://review.opendev.org/c/opendev/system-config/+/794089 which turns eavesdrop.openstack.org into a static site. it would be useful if anyone could think of anything else that needs redirects there | 23:27 |
| clarkb | ianw: the yaml2ical stuff goes there too right? but thats the static site stuff I think | 23:28 |
| ianw | clarkb: yeah, that's all @ https://static.opendev.org/project/meetings.opendev.org/ | 23:28 |
| ianw | the things to proxy to eavesdrop01 i can see are /irclogs and /meetings | 23:29 |
| opendevreview | Merged opendev/zone-opendev.org master: Add meetings.opendev.org https://review.opendev.org/c/opendev/zone-opendev.org/+/794085 | 23:29 |
| fungi | ianw: sounds great | 23:29 |
| fungi | sorry i've been spread too thin to review | 23:30 |
| ianw | no worries. i just feel like we get eavesdrop stable on ansible infrastructure, and then it seems pretty clear we'll be focusing on other integrations for bots etc. into the future | 23:30 |
| fungi | yeah, wfm | 23:31 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!