| *** rosmaita has quit IRC | 03:57 | |
| dmsimard | fyi: cloned volumes mounted on a temporary VM, doing fsck on the log LVM but FWIW it mounted cleanly /dev/mapper/main-logs 14T 7.6T 5.7T 58% /mnt | 05:05 |
|---|---|---|
| dmsimard | I'll check back for the results of the fsck tomorrow | 05:05 |
| dmsimard | fsck is finished, nothing in lost+found | 13:42 |
| dmsimard | There's 7.6TB to rsync and we're currently using only 246GB. I'll disable the log maintenance script temporarily and put static.o.o in the emergency file | 13:47 |
| dmsimard | rsync started, made sure to carry over timestamps/permissions/etc so that logs are phased out appropriately when we resume the cron | 14:06 |
| dmsimard | stepping away | 14:07 |
| dmsimard | Realized I should've started the transfer on the private network instead of using the public IP -- restarted the rsync with the private IP of the restore vm instead. | 14:37 |
| -openstackstatus- NOTICE: Jobs are currently not running and are staying queued in Zuul pending the completion of a maintenance at our cloud provider. Jobs will resume once this maintenance has been completed. | 15:03 | |
| pabelanger | dmsimard: how long did fsck take? | 15:08 |
| dmsimard | pabelanger: the timestamped loop I did outran the screen buffer size so I don't know for usre | 15:10 |
| dmsimard | it took less than 10 hours? best guess | 15:10 |
| dmsimard | or less than 8, actually | 15:10 |
| -openstackstatus- NOTICE: Zuul has been restarted due to an outage in our cloud provider. Changes already in queues have been restored, but changes uploaded and approved since 12:30 UTC may need to be rechecked or reapproved. | 17:20 | |
| *** rosmaita has joined #openstack-infra-incident | 19:29 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!