| fungi | noonedeadpunk: i've registered #openstack-freezer and added our opendevaccess bot account to the acl for it now, sorry for the wait | 00:48 |
|---|---|---|
| opendevreview | Merged openstack/project-config master: Add #openstack-freezer to accessbot https://review.opendev.org/c/openstack/project-config/+/912767 | 01:06 |
| opendevreview | Merged openstack/project-config master: Add Freezer to gerritbot https://review.opendev.org/c/openstack/project-config/+/912716 | 01:25 |
| *** liuxie is now known as liushy | 02:31 | |
| johnsom | Not sure if it's just me, but I can't reach review.opendev.org at the moment. Zuul opens, but not gerrit. | 02:56 |
| tonyb | Same here | 02:57 |
| johnsom | It seems like traffic is being dropped at Cogent | 02:57 |
| tonyb | Yup | 02:57 |
| johnsom | Ok, not alone. Likely not an opendev issue either. Probably a bad BGP change somewere... | 02:57 |
| tkajinam | I came here to report the problem and learned that I'm not alone :-) | 02:59 |
| tonyb | I'm doing more research but yes it looks like some ip/BGP connectivity | 03:04 |
| johnsom | https://www.irccloud.com/pastebin/nh78jMOX/ | 03:05 |
| johnsom | The internet is telling me to stop working at 8pm. lol | 03:07 |
| tonyb | ns04.opendev.org is in the same cloud/region is "up" | 03:08 |
| tonyb | https://bgp.he.net/AS33028#_prefixes | 03:10 |
| tonyb | there is a lot of red in there | 03:10 |
| tonyb | and it seems to cover off the ip block | 03:11 |
| tonyb | ... in which resides review.opendev.org | 03:11 |
| tonyb | #status log OpenDev is experiencing connectivity issues to several key services including review.opendev.org. Admins are monitoring. | 03:29 |
| opendevstatus | tonyb: finished logging | 03:29 |
| clarkb | server list/show indicate the server is shutdown | 03:30 |
| clarkb | I don't see a reason for that. We can try starting it back up again, but I'm not sure I'm in a spot where I can check that it is happy on its way back up | 03:31 |
| clarkb | I don't think this is a network issue | 03:32 |
| clarkb | according to my history on brideg I had to start review on December 11th | 03:32 |
| clarkb | looks like it stopped just before 0300 UTC according to catci | 03:34 |
| johnsom | Yeah, that is when my session started failing | 03:34 |
| clarkb | 2024-03-26T02:47:33Z is the updated timestamp which aligns with that. I have no idea why it was shutdown | 03:34 |
| clarkb | If it doesn't mysteriously change its state back to working again in the next 5 minutes I'll issue the manual start | 03:35 |
| johnsom | It seems odd that the traceroute would go out to lunch at Cogent. Maybe they advertise the IPs all the way up... | 03:36 |
| clarkb | I feel like everyone was confused last time this happened too | 03:40 |
| clarkb | I issued a manual start request | 03:40 |
| clarkb | the server is up. Containers didn't auto start (I think because they didn't shut down cleanly on the shutdown?) | 03:43 |
| clarkb | I'm trying to look around for anything obviously wrong before I try starting the containers | 03:43 |
| clarkb | I'm not seeing anything I guess I'll give this a shot... | 03:46 |
| johnsom | https://www.irccloud.com/pastebin/G04XeRXq/ | 03:47 |
| tonyb | I need to go grab the kids but I can for sure restart it when I get back | 03:47 |
| johnsom | Looks up now | 03:48 |
| clarkb | I forgot to move the replication waiting queue aside so we have a bunch of errors related to that (known problem and mostly harmless) | 03:48 |
| tonyb | clarkb: Thanks | 03:48 |
| clarkb | doing a quick skim of the startup log I think those are the only errors I see so far | 03:49 |
| clarkb | I can load a change of mine | 03:50 |
| clarkb | still waiting for diffs to be happy ( a known startup caching thing) | 03:50 |
| clarkb | I see diffs now (on 914119) | 03:51 |
| clarkb | I was able to push a new patchset to that change too. Gerritbot seems to need a restart | 03:54 |
| clarkb | I've restarted gerritbot. I want to say it takes its time joined channels to avoid getting kicked on a restart | 03:55 |
| clarkb | #status log review02.opendev.org was in a shutdown state for nearly an hour. Manually starting the instance then manually restarting containers appears to have restored services | 03:56 |
| opendevstatus | clarkb: finished logging | 03:56 |
| clarkb | I think https://review.opendev.org/c/openstack/manila/+/914168 didn't get indexed properly as it was pushed near the shutdown. It doesn't show up in project:openstack/manila change listings as a result. We can trigger a reindex for openstack/manila but that doesnt' seem critical so I'm going to stop poking and will look tomorrow | 04:00 |
| clarkb | guilhermesp: mnaser ^ fyi not sure if this points to a problem in the cloud. I think we've largely recovered at this point but you may want to check things are operating as expected? Instance is review02.opendev.org | 04:07 |
| mnaser | clarkb: will check out tmrw in AM but i suspect the VM might have somehow oom’d on the host side and got killed.. | 04:59 |
| mnaser | at least that’s usually the running theory when the vm is just stopped | 04:59 |
| Clark[m] | mnaser: I didn't see any evidence of oomkiller in syslog | 05:37 |
| Clark[m] | Oh on the host side. I wouldn't see it on my side then | 05:37 |
| *** mmalchuk_ is now known as mmalchuk | 06:49 | |
| noonedeadpunk | fungi: oh, thanks a ton! | 08:46 |
| noonedeadpunk | though I don't see bots joining the channel? | 08:47 |
| opendevreview | Merged zuul/zuul-jobs master: Make prepare-workspace-git fail faster. https://review.opendev.org/c/zuul/zuul-jobs/+/910582 | 09:04 |
| opendevreview | Slawek Kaplonski proposed openstack/project-config master: Add reference to the project-team-guide in the fail ci msg https://review.opendev.org/c/openstack/project-config/+/914189 | 10:17 |
| opendevreview | Merged openstack/project-config master: Notify in IRC regarding patches to OSA unmaintained branches https://review.opendev.org/c/openstack/project-config/+/912717 | 12:08 |
| clarkb | there is a gitea 1.21.10 now | 14:51 |
| clarkb | its liek a game of tag between us and them | 14:51 |
| clarkb | I have an early meeting, then I'll get a meeting agenda sent out then look into another gitea upgrade change | 14:51 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update etherpad to v2.0.1 https://review.opendev.org/c/opendev/system-config/+/914119 | 14:57 |
| clarkb | infra-root https://review.opendev.org/c/opendev/system-config/+/914109 should be a quick easy review to add the linaro cert renewal process to our docs | 16:10 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update gitea to v1.21.10 https://review.opendev.org/c/opendev/system-config/+/914292 | 16:43 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM force etherpad failure to hold node https://review.opendev.org/c/opendev/system-config/+/840972 | 16:53 |
| clarkb | I've put an autohold in place for ^ | 16:54 |
| clarkb | JayF: you have an autohold for ironic-standalone is that still needed or can I clean it up? | 16:54 |
| JayF | it can be cleaned up, my bad for leaving it hanging so long | 16:55 |
| clarkb | no worries I'll do that now | 16:55 |
| slittle | please set me up as first core of starlingx-app-rook-ceph-core | 17:27 |
| clarkb | slittle: you're already in there | 17:29 |
| slittle | so I am. Thanks | 17:30 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add docs for linaro cloud cert renewal process https://review.opendev.org/c/opendev/system-config/+/914109 | 17:34 |
| clarkb | etherpad 2.0.1 doesn't seem to have a working plugin for the text format anymore | 17:40 |
| clarkb | 104.239.142.129 is the held node if you want to see that yourself | 17:41 |
| clarkb | seems like plugins may not be working at all which isn't super surprising I guess given that ws the major change they made | 17:41 |
| fungi | has there been any indication of plugins getting brushed up some since the release? maybe we just need to wait for the folks getting them back into shape? | 17:43 |
| clarkb | clarkb-test is the pad I used if anyone wants to look at the existing content I made | 17:43 |
| clarkb | fungi: maybe. I think the main thing that changed is how plugins are installed, but maybe the hook points updated too | 17:43 |
| fungi | oh, makes sense | 17:44 |
| fungi | so they might not need a ton of work | 17:44 |
| clarkb | it is possible we're just not installing the plugin correctly anymore. I ported over the dockerfile from upstream that should accomodate that but maybe its broken there too. Or the plugin is no longer compatible. I marked the chagne wip until we know more | 17:44 |
| clarkb | I'll have to dig into it more later. Now about to enter the next block of meetings | 17:51 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update etherpad to v2.0.1 https://review.opendev.org/c/opendev/system-config/+/914119 | 18:54 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM force etherpad failure to hold node https://review.opendev.org/c/opendev/system-config/+/840972 | 18:54 |
| clarkb | the TC meeting ended early. I think maybe the problem has/had to do with ARG definitions in the multi stage build | 18:55 |
| clarkb | I'm going to refresh the autohold | 18:55 |
| clarkb | I ended up jumping into the held node and exec'd the plugin install step. From that I realized that the build didn't ever seem to run it implying the arg var was empty | 18:58 |
| clarkb | I realized this because it emits output that didn't show up in the log of the build | 18:58 |
| fungi | aha | 18:59 |
| opendevreview | Merged opendev/system-config master: Cleanup opensuse mirroring configs entirely https://review.opendev.org/c/opendev/system-config/+/913454 | 20:05 |
| clarkb | best I can tell ^ applied properly. The cronjob is gone and the logrotate config appears to be gone as well | 21:15 |
| clarkb | https://zuul.opendev.org/t/openstack/build/1c26a1ff53404595aafacdc907c3b694/log/job-output.txt#1281 this time the etherpad image build seems to have installed the headings plugin. Now to pull up the held node and see if it works | 21:16 |
| clarkb | I'm still getting the weird chrome disconnects that I think we've narrowed down to ssl problems with the test nodes. But FF seems to work and the plugin is working | 21:20 |
| clarkb | 213.32.79.214 is the held node and clarkb-test is my test pad. | 21:21 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Enable nodepool delete after upload option https://review.opendev.org/c/openstack/project-config/+/914412 | 22:12 |
| *** dmellado6 is now known as dmellado | 22:26 | |
| *** dmellado4 is now known as dmellado | 22:52 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!