How to Prevent Data Loss Due to Database Failure

July 16, 2015

When we were about a week away from launching the new IEG site, catastrophe struck. Our development server was wiped clean and replaced with the production server. With the exception of the IEG website, it didn’t matter – the rest of our sites were not being populated at the moment on the development server. Although we have an incredible system of backups for each of our live production servers and a separate backup for all our themes via Git, our development server missed that boat.

The feeling of utter terror that washed over our entire department was palpable. Instead of being enveloped by this terror, we immediately did everything we could to pick up the pieces. We were able to restore about 80% of the content thanks to drafts saved in Google Docs, assorted emails, and content written in Basecamp comments. All that was missing was, arguably, the most important content: the portfolio piece descriptions. With the help of my colleagues, we were able to re-write these descriptions and still launch within days of our intended date. Perhaps our content was even better with their contributions.

This data loss taught us a very valuable lesson: anywhere development is happening and content is being written should have multiple layers of backup. Here are a few ways to think about backups for your business:

Backup Your Backups

First things first – don’t ever put yourself in the situation we were in and have backups on every level, live and production servers. We didn’t have the first level of backup for our development server but now we have two, a fail safe for catastrophe. Our system now automatically backs up every night, without the batting of an eye.

I know this sounds simple and self-explanatory, but so often backups are put off until you experience the loss of data. Don’t let your website collapse before you become prepared. Save yourself the panic and the despair by having a backup and a backup of the backup already set up.

Don’t Write Longform Content in WordPress

As frustrating as it can be to write in one place and then re-format in another, I am eternally thankful that most of our content was written in Google Drive and not in WordPress. It’s true, Google Drive could also fail, but it’s highly unlikely that both your website and Google would fail at the same time. We were able to restore 80% of the content because it was written in Google Docs, stored on Basecamp (our project management tool of choice), and in local copies on various team member’s computers.

WordPress has the capability of storing and retrieving revisions of a document. Be sure to turn that capability on for posts and custom post types in order to maximize the chance of recovering data.

Plan for the Worst

Have a plan in place in case the worst happens. You may have content saved in the cloud, but that doesn’t mean you’re completely protected. If there’s a catastrophic failure at the data center hosting your content, will you be out of luck? Will you have lost years of content? Know what the most vital pieces of information are, the information that would be near impossible to recover, and have a plan B and plan C. For example, if you’re collecting member data or a historical archive of images, make sure that data is repeatable and in multiple locations. It may seem like overkill at the moment, but future disaster survivors will thank you for thinking ahead.

For the most part, data centers do this preparation work for you. They often ask for you to pay extra for that security, but isn’t it worth it? You may not think so when you’re looking at your bill, but I promise it is.

We may have lost a lot of data, but it was an invaluable lesson. I’m eternally grateful that it was my project with a flexible deadline and with content stored across the web. Had it been a client of ours or a site with a larger amount of data, heads would have rolled.

For now, all of our heads get to stay in place.