| *** tosky has quit IRC | 00:12 | |
| *** sshnaidm is now known as sshnaidm|afk | 00:22 | |
| *** Alex_Gaynor has quit IRC | 00:46 | |
| *** Alex_Gaynor has joined #opendev | 00:47 | |
| *** elod_pto has quit IRC | 00:57 | |
| *** tbarron has quit IRC | 00:57 | |
| *** valleedelisle has quit IRC | 00:57 | |
| *** mhu has quit IRC | 00:57 | |
| *** stephenfin has quit IRC | 00:57 | |
| *** openstackgerrit has quit IRC | 01:00 | |
| *** DSpider has quit IRC | 01:10 | |
| *** elod_pto has joined #opendev | 01:17 | |
| *** tbarron has joined #opendev | 01:17 | |
| *** valleedelisle has joined #opendev | 01:17 | |
| *** mhu has joined #opendev | 01:17 | |
| *** stephenfin has joined #opendev | 01:17 | |
| *** auristor has quit IRC | 01:27 | |
| *** auristor has joined #opendev | 01:32 | |
| *** tkajinam has quit IRC | 02:35 | |
| *** tkajinam has joined #opendev | 02:35 | |
| *** brinzhang has joined #opendev | 02:41 | |
| *** brinzhang_ has joined #opendev | 04:12 | |
| *** brinzhang has quit IRC | 04:15 | |
| *** ianw is now known as ianw_pto | 04:22 | |
| *** ykarel has joined #opendev | 04:44 | |
| *** bhagyashris is now known as bhagyashri|ruck | 05:27 | |
| *** bhagyashri|ruck is now known as bhagyashris|ruck | 05:28 | |
| *** marios has joined #opendev | 06:04 | |
| *** hamalq has joined #opendev | 06:50 | |
| *** hamalq has quit IRC | 06:55 | |
| *** ykarel has quit IRC | 07:21 | |
| *** slaweq has joined #opendev | 07:45 | |
| *** ykarel has joined #opendev | 07:57 | |
| *** ralonsoh has joined #opendev | 08:02 | |
| *** marios has quit IRC | 08:03 | |
| *** rpittau|afk is now known as rpittau | 08:07 | |
| *** ykarel is now known as ykarel|lunch | 08:25 | |
| *** ykarel|lunch has quit IRC | 08:25 | |
| *** cloudnull has quit IRC | 08:53 | |
| *** cloudnull6 has joined #opendev | 08:53 | |
| *** lpetrut has joined #opendev | 08:54 | |
| *** ykarel has joined #opendev | 09:03 | |
| *** ykarel has quit IRC | 09:10 | |
| *** hashar has joined #opendev | 09:18 | |
| *** hamalq has joined #opendev | 09:28 | |
| *** ykarel has joined #opendev | 09:29 | |
| *** hamalq_ has joined #opendev | 09:29 | |
| *** ykarel_ has joined #opendev | 09:33 | |
| *** hamalq has quit IRC | 09:33 | |
| *** fabv has joined #opendev | 09:34 | |
| *** hamalq_ has quit IRC | 09:34 | |
| *** ykarel has quit IRC | 09:36 | |
| *** ykarel_ is now known as ykarel | 09:36 | |
| fabv | hello, could someone review this https://review.opendev.org/c/openstack/project-config/+/767369 please? | 09:36 |
|---|---|---|
| *** marios has joined #opendev | 09:49 | |
| *** tosky has joined #opendev | 10:03 | |
| *** brinzhang_ has quit IRC | 10:16 | |
| *** brinzhang_ has joined #opendev | 10:17 | |
| *** JayF has quit IRC | 10:34 | |
| *** JayF has joined #opendev | 10:38 | |
| *** ykarel_ has joined #opendev | 10:55 | |
| *** ykarel has quit IRC | 10:58 | |
| *** sshnaidm|afk is now known as sshnaidm | 11:17 | |
| *** hamalq has joined #opendev | 11:30 | |
| *** dviroel has joined #opendev | 11:32 | |
| *** hamalq has quit IRC | 11:35 | |
| *** DSpider has joined #opendev | 11:35 | |
| *** ykarel_ is now known as ykarel | 11:42 | |
| *** larainema has quit IRC | 11:46 | |
| *** dtantsur|afk is now known as dtantsur | 11:47 | |
| *** hamalq has joined #opendev | 11:58 | |
| *** fabv has quit IRC | 12:00 | |
| *** hamalq has quit IRC | 12:03 | |
| *** iurygregory has joined #opendev | 13:16 | |
| danpawlik | fungi: hey, is it possible to merge that before next year: https://review.opendev.org/c/opendev/system-config/+/767396 ? | 13:25 |
| fungi | danpawlik: i'll need to check the available quota on that volume but can take a look | 13:27 |
| danpawlik | fungi thank you | 13:27 |
| fungi | https://grafana.opendev.org/d/HxrNXt2Gk/afs?orgId=1 indicates mirror.fedora has plenty of room, excellent | 13:28 |
| *** marios has quit IRC | 13:41 | |
| *** ykarel_ has joined #opendev | 13:52 | |
| *** brinzhang_ has quit IRC | 13:54 | |
| *** ykarel has quit IRC | 13:54 | |
| *** brinzhang_ has joined #opendev | 13:55 | |
| *** ykarel_ is now known as ykarel | 13:55 | |
| *** cloudnull6 is now known as kecarter | 13:58 | |
| *** hamalq has joined #opendev | 13:59 | |
| *** kecarter is now known as cloudnull | 14:03 | |
| *** hamalq has quit IRC | 14:04 | |
| *** marios has joined #opendev | 14:39 | |
| *** lpetrut has quit IRC | 14:51 | |
| frickler | #status log restarted gerritbot on eavesdrop.o.o which seemed to have lost connections | 14:59 |
| openstackstatus | frickler: finished logging | 14:59 |
| fungi | yeah, a couple hours ago i pushed a change which should have been reported in two channels but only one got notified, which was sort of strange | 15:30 |
| fungi | thanks for the restart! | 15:30 |
| *** ykarel has quit IRC | 15:36 | |
| fungi | looks like lists.o.o is broken. apache returns nothing, can't ssh into it... i thought there were surprisingly few ml posts when i woke up, guess they're all queued by senders. will work on getting the server back in shape now | 15:48 |
| corvus | fungi: ow; lmk if you need a hand | 15:50 |
| fungi | thanks, just checking the emergency console in rax first | 15:52 |
| clarkb | I'm slowly catching up too | 15:53 |
| fungi | nothing on the console, and no response to carriage return. will try to use the ctrl+alt+del button it provides | 15:53 |
| fungi | if this doesn't work, i'll reboot via api | 15:53 |
| fungi | yeah, nothing. soft reboot triggered via api now... | 15:55 |
| fungi | doesn't seem to have taken effect. will do --hard | 15:58 |
| clarkb | (that is "normal") | 15:58 |
| *** hamalq has joined #opendev | 16:00 | |
| fungi | yeah | 16:02 |
| *** hamalq has quit IRC | 16:04 | |
| fungi | lists.o.o seems to be online again for ~7 minutes now | 16:07 |
| fungi | taking a look to make sure it seems sane | 16:07 |
| fungi | i'm able to process the moderation queues | 16:08 |
| fungi | messages are coming through from openstack-discuss already | 16:12 |
| clarkb | I can confirm I see messages arriving | 16:13 |
| fungi | judging from cacti the server died between 04:05 and 04:10 utc, i think | 16:13 |
| fungi | unfortunately i stopped being able to use the graph zoom in cacti some months back (maybe when the server was rebuilt? maybe something with my browser) | 16:14 |
| clarkb | I can confirm that as well | 16:15 |
| fungi | #status log lists.openstack.org rebooted at 16:00z to remedy complete hang around 04:10z today | 16:15 |
| openstackstatus | fungi: finished logging | 16:15 |
| fungi | anyway, i think that's back on track | 16:16 |
| fungi | unrelated, rackspace opened a support ticket on saturday to inform us of a block storage outage impacting nb01.opendev.org/main02 around 05:45 utc that day... i haven't found time to dig into it yet | 16:18 |
| clarkb | I can probably take a look at the nodepool builder a bti later today just to check it is happy | 16:19 |
| fungi | dmesg indicates i/o errors on xvdc starting at 2020-12-19 02:56:06 immediately following hung task timeouts at 02:41:10 | 16:22 |
| fungi | amd /opt seems to now be mounted read-only | 16:23 |
| clarkb | fungi: rebooting should be fine for those servers (rather than remounting) | 16:23 |
| clarkb | we get a fsck automatically if we do that? | 16:23 |
| clarkb | (though maybe a fsck is not desireable?) | 16:24 |
| fungi | yeah, i'm thinking it will since the fs will be marked dirty | 16:24 |
| fungi | we could disable fsck at boot for /opt in fstab, but would also need to disable nodepool services on it if we want it to come up idle so we can manually fsck or mkfs it | 16:25 |
| fungi | in my opinion we should just reboot and let the chips fall where they may | 16:26 |
| clarkb | wfm | 16:26 |
| fungi | doing that now | 16:26 |
| fungi | Dec 21 16:27:29 nb01 systemd-fsck[578]: main01: recovering journal | 16:29 |
| fungi | Dec 21 16:27:29 nb01 systemd-fsck[578]: main01: clean, 379798/67108864 files, 146014364/268434432 blocks | 16:29 |
| fungi | Dec 21 16:27:29 nb01 systemd[1]: Mounted /opt. | 16:30 |
| fungi | so should be good to go | 16:30 |
| fungi | #status log rebooted nb01.o.o to recover from a block device outage around 2020-12-19 03:00z | 16:31 |
| openstackstatus | fungi: finished logging | 16:31 |
| *** hamalq has joined #opendev | 17:00 | |
| *** marios has quit IRC | 17:01 | |
| *** ralonsoh has quit IRC | 17:05 | |
| *** ralonsoh has joined #opendev | 17:05 | |
| *** openstackgerrit has joined #opendev | 17:12 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add configuration to make logs public https://review.opendev.org/c/zuul/zuul-jobs/+/764483 | 17:12 |
| *** rpittau is now known as rpittau|afk | 17:13 | |
| *** hamalq has quit IRC | 17:14 | |
| *** hamalq has joined #opendev | 17:14 | |
| openstackgerrit | Merged ttygroup/gertty master: Handle gerrit 3.x style internal urls https://review.opendev.org/c/ttygroup/gertty/+/767928 | 17:16 |
| openstackgerrit | Merged ttygroup/gertty master: Add WIP support https://review.opendev.org/c/ttygroup/gertty/+/767934 | 17:16 |
| *** hamalq has quit IRC | 17:17 | |
| *** hamalq has joined #opendev | 17:17 | |
| *** ralonsoh_ has joined #opendev | 17:24 | |
| *** ralonsoh has quit IRC | 17:25 | |
| *** fdegir has quit IRC | 17:25 | |
| *** fdegir has joined #opendev | 17:25 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: upload-artifactory: no_log upload task https://review.opendev.org/c/zuul/zuul-jobs/+/768111 | 17:33 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: upload-artifactory: no_log upload task https://review.opendev.org/c/zuul/zuul-jobs/+/768111 | 17:33 |
| *** stzsch has quit IRC | 17:35 | |
| *** tristanC has quit IRC | 17:44 | |
| *** tristanC has joined #opendev | 17:46 | |
| *** hashar has quit IRC | 18:00 | |
| *** ralonsoh_ has quit IRC | 18:01 | |
| *** ysandeep is now known as ysandeep|away | 18:05 | |
| *** dtantsur is now known as dtantsur|afk | 18:15 | |
| *** stzsch has joined #opendev | 18:25 | |
| *** yoctozepto has quit IRC | 19:55 | |
| *** yoctozepto has joined #opendev | 19:55 | |
| *** tkajinam_ has joined #opendev | 21:31 | |
| *** ShadowJonathan has quit IRC | 21:32 | |
| *** TheJulia has quit IRC | 21:33 | |
| *** ShadowJonathan has joined #opendev | 21:34 | |
| *** hamalq has quit IRC | 21:34 | |
| *** TheJulia has joined #opendev | 21:34 | |
| *** tkajinam has quit IRC | 21:34 | |
| *** hamalq has joined #opendev | 21:34 | |
| *** rpittau|afk has quit IRC | 23:21 | |
| *** johnsom has quit IRC | 23:21 | |
| *** TheJulia has quit IRC | 23:22 | |
| *** tosky has quit IRC | 23:41 | |
| *** redrobot has quit IRC | 23:43 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!