It was caused by a silly mistake but it totally destroyed my database. I wasn’t entirely unprepared for the situation and I did have backups. However, I hadn’t bothered to get familiar with the restoration process to recover data from my backup and I was now rummaging around in unfamiliar parts of the Linode Manager.
As mentioned in my review of Linode’s VPS services, I’ve paid for the Linode Backup service but I’d never bothered to look into it. I assumed it would be a one-click process and that would be that.
My assumption about Linode Backup wasn’t exactly wrong, but I was a bit optimistic regarding how easy it would be to use. I’d imagine the process would be something like this:
- Login to the Linode Manager
- Select my virtual server and go to Backups
- Locate the most recent backup before the disaster, and click “Restore”
So far, I was on track, but from there things got a bit hazier.
Backup Details screen in the Linode Manager
Where do you go from the Backup Details screen? What happens next? There were no instructions, and Linode’s documentation of its backup service is a bit lackluster. I guess they think it’s very intuitive as they built it. I, however, was left guessing as to what to do next.
The only information I’d to go on was the “not enough free space” info message. I guessed that I’d have to delete the virtual disks that were attached to my virtual server to free up virtual space so I’d enough room to restore my virtual backups. I wasn’t sure if this was the right course of action and I was a bit worried that my backups were associated with the drives rather than the server, and removing the drives would delete the drive backups. As I also had a separate local backup of everything too, so I decided to just go for it even without knowing what would happen.
After deleting my Swap image and virtual disk, I headed back to the Backups manager, select the most recent backup, and I now had the option to restore a copy of my backed up disks to the server. After about 90 seconds, I could resize my backup drive and grow it to fill my server’s storage capacity and I was ready to boot from the recovered virtual drive.
I think there should be a “restore and replace” option in the Linode Manager. I’m surely not the only customer who would have wanted this option.
For most Linode customers, this would have been where my backup recovery story ended. Unfortunately for me, I’m a stubborn man of principles and convictions.
Rather than relying on the stripped-down Linode’s Unified Linux Kernel, I’d choose to restore and use the default Fedora Linux kernel instead. (Linode disables Security-Enhanced Linux (SELinux) in their kernels, and I prefer to leave the security feature enabled.) If I’d used the default kernel, I would have been able to boot up instantly and be on my merry way. With a custom kernel and boot loader, I’d to make further adjustments before my server could be fully recovered.
To boot from the custom kernel (arguably the default kernel, as I used an unmodified Fedora Linux kernel), I’d setup GRUB2. Unfortunately for me, the recovered backup drives had completely different UUIDs, meaning GRUB didn’t recognize the drives. I messed about with Linode’s various recovery systems and environments for one hour before admitting defeat. I swapped my server configuration back to using the default Linode kernel from the Linode Manager, and it booted up in seconds. After two minutes of disk and journaling integrity checks, my server was back up and running.
From within the new working environment, I was quickly above to adjust GRUB2’s expectations for disk IDs and reboot into the default kernel again.
I was a bit upset that Linode had not replicated the UUIDs for the backup disks as both the original and recovered disks were virtual disks. There are clearly use cases where it would be beneficial with unique disk IDs and use cases where identical disk IDs would be beneficial. The gist of it is that it’s easier to recover from my scenario than to deal with two disks with the same ID in other scenarios.
I definitely wish there would have been more clear instructions provided in the Backup Details screen. I wasted a good hour trying to locate better documentation on what it all meant. It’s unnecessarily vague and sparse on details.
The lesson I learned from this experience? To always make myself familiar with the full backup restoration process before I start relying on any backup solution. It’s the sensible thing to do anyway.
PS: If you’re looking to sign up for Linode, you can get a better deal with the link and coupon code that you can find at the bottom of my Linode review.