How I restored a virtual machine disk image

A miracle happened the other day. In a bad way. :( For some unknown reason, a virtual machine image suddenly became corrupted. The machine was billed, the information was critical, and as luck would have it, it didn’t have a current backup at the time. The virtual server was running and suddenly started acting up for no apparent reason. After rebooting, the console prompted me to run fsck, which I foolishly followed, ignoring the need to make a backup first. Five minutes later, I was nervously kicking myself, as the procedure had damaged the partition so badly that it was no longer readable or repairable by any tools. Why, exactly, was there no backup? Because the other day I’d migrated this virtual machine from another platform, which in turn had to be destroyed to create backup storage there. Who would have thought such a problem would arise right in the middle of nowhere? Murphy’s Law, classic… After some brainstorming, I decided to try restoring the LVM partition on the old storage I’d already cleared. Sometimes laziness can be very helpful… Even though I’d cleared the storage, due to the weekend, I hadn’t had time to write anything to it, so my chances of recovering the data were practically 100%.

step-by-step:

So, I needed to restore the entire LVM structure in order to dig out the LV partition, which contained the coveted information, albeit no longer entirely relevant. This is done as follows: cd /etc/lvm/backup/ In this directory, LVM2 carefully stores information about the pv (physical volume) and vg (volume group) that were created on the disks. If anything remains there, it’s good. Let’s try to restore: vgcfgrestore -f /etc/lvm/backup/st1 st1 Activate lvmdiskscan lvs -a -o +devices See what happens root@st1:~# vgs VG #PV #LV #SN Attr VSize VFree VolGroup 1 2 0 wz--n- 9.51g 0 st1 1 16 0 wz--n- 5.46t 5.46t Great. The group is in place. But the partitions haven’t been restored yet. Let’s go to the archive directory where all the logical drive manipulations are stored: root@st1:~# cd /etc/lvm/archive/ root@st1:/etc/lvm/archive# ls st1_00251-221876201.vg st1_00290-2082643790.vg
st1_00252-24104864.vg st1_00291-180822193.vg
.... If you open one of them, you can see the configuration of the drives that existed in the group at that time. We select the most viable one and test it: root@st1:/etc/lvm/archive# vgcfgrestore prim --test -f /etc/lvm/archive/st1_00275-669926956.vg TEST MODE: Metadata will NOT be updated and volumes will not be (de)activated. Restored volume group st1 [quads id=2] Everything seems OK? Then we do the same, but in production mode: root@st1:/etc/lvm/archive# vgcfgrestore prim -f /etc/lvm/archive/st1_00275-669926956.vg After the group is restored from the archive, we need to scan it and activate the found logical partitions root@st1:/etc/lvm/archive# lvscan
inactive '/dev/st1/volume-4b748459-83f2-4b3a-9298-fe02764785b5' [11.00 GiB] inherit inactive '/dev/st1/shared' [500.00 GiB] inherit inactive '/dev/st1/volume-41597caa-007a-4880-8c03-f8f6aa59953e' [31.00 GiB] inherit inactive '/dev/st1/volume-8826f883-42ac-46cf-a90c-2c9407c51a77' [31.00 GiB] inherit inactive '/dev/st1/volume-a0d819ee-c08f-40c8-a789-7bef1ac14ffb' [300.00 GiB] inherit The found partitions are not yet active. We activate the desired ones and map the contents: lvchange -a y /dev/st1/volume-8826f883-42ac-46cf-a90c-2c9407c51a77 kpartx -a /dev/st1/volume-8826f883-42ac-46cf-a90c-2c9407c51a77 Now we go to /dev/mapper and see the partitions that are inside the LVM we are interested in. root@st1:/dev/mapper# ls |grep st1-volume--8826f883 st1-volume--8826f883--42ac--46cf--a90c--2c9407c51a77 st1-volume--8826f883--42ac--46cf--a90c--2c9407c51a77p1 st1-volume--8826f883--42ac--46cf--a90c--2c9407c51a77p2 st1-volume--8826f883--42ac--46cf--a90c--2c9407c51a77p5 Now we can simply mount the desired partition. But I had slightly different goals. I needed to somehow recover the current drive using the old one, since the information remaining there was very important. After making copies of both partitions, I began experimenting with recovery. The problem was that all the superblocks on the new partition were destroyed, and I couldn’t recover anything even using the backups. Then a completely idiotic idea occurred to me! I decided to merge a piece of the backup disk (or rather, its partition) with a partition of the broken one to reinsert the superblock, which, in theory, would offer a chance of recovery. Since the disks are structurally identical, this looked quite promising. Create a new LVM partition and copy the broken disk to it. lvcreate -L 32G -n billrectest st1
dd if=/root/billpart2.img of=/dev/st1/billrectest
Then proceed with the operation: dd if=/dev/mapper/st1-volume--8826f883--42ac--46cf--a90c--2c9407c51a77p1 of=/dev/st1/billrectest bs=51200 count=1 And so, we’ve ripped 50 kilobytes from the beginning of the partition. (I tried smaller pieces, but they didn’t work.) Of course, after this «vaccination,» the disk won’t mount, as it’s still damaged. Repairing the disk… fsck.ext4 -y /dev/st1/billrectest Of course, there’s no guarantee, but I was lucky. Even though half the file system went into lost+found mode and 2.5% was completely lost, the database somehow survived! Eventually, I transferred the relevant data to a backup and started the server. It was 6 a.m…. Relieved, I opened a bottle of beer, drank it down, and went to bed. ;)