pusa, a server I administer at uni, suffered a massive data accident on Wednesday. When I went to see why it didn't come up from a reboot on Friday, I found out the initrd hadn't been able to mount /. Weird...

Luckily, the two new disks were already installed in the host and waiting for me to finish the migration to the RAID1 and the new Linux-VServer setup, but unfortunately I've been way too busy and it was too late for some of our data. A fsck of /dev/hda1 resulted on large portions of the data going to /lost+found. Discovering this made me feel like a great fool after not having dd'd the device before doing this (a dry-run of fsck had not reported anything useful). I found out some of the lost data in random directories, but in general lots were missing, and others made no sense:

/oldpusa/etc: gzip compressed data, was "libpng.txt", from Unix, last modified: Wed Dec 20 00:58:51 2006, max compression

I hoped for my PostgreSQL stuff being intact, so after dd'ing /dev/hda5, I fsck'd the image. The result was an empty filesystem, and a lost+found full of stuff. I can't find a directory with stuff that resembles postgresql data at all. I did find a directory with a PG_VERSION file in it, but the rest of the files in it (around 100) had numeric names and little more. If anyone thinks I might be able to rebuild my /var/lib/postgresql from this, I'll be infinitely grateful.

Anyway, I haven't written to the corrupted after I fucked up the root partition. I'm very interested in knowing what could cause corruption on all partitions, making them unmountable, but still recognisable by fsck, even if the result is not good at all. Maybe a corrupted partition table? If so, what does the Dear Lazyweb recommend me to try out? I suspect the first portion of all partitions were damaged, but maybe just that. Some “partition table shift”, which makes the filesystems lose the first superblock (trying other superblocks didn't work either)? Suggestions is very welcome by comment or email, and detail on what tools and how to use to try out things, better. My backup of PostgreSQL is not so recent, and recovering some SmartList data would also be great.

As for the mandatory “where are your backups”, the answer is basically we had no resources to store them until very recently, and when we finally got the disks I've had no time until now to set it up entirely, so some bits (db, lists, web) were still not running off the new drives. The luckiest people have been the MUD owners, who have had no data loss at all, as they were living entirely on /dev/md0. Losing MUD data probably means getting angry calls at 4AM or so. :)