OVH datacenter fire - backup is only good if it covers all possibilitiesHaving spent a number of years working as in Information Technology Disaster Recovery (ITDR) subject matter expert (SME), I can assure you that the topic of backup is about as unsexy as it gets. I do have plenty of horror stories to tell. It feels like I spend a lot of time delivering bad news, and regardless of bedside manners, no one likes to hear bad news.

World Backup Day is 31 March — today at time of publication. Considering the OVH data center fire on 10 March 2021, it has really brought home some of the points I make in this editorial.

This is a long editorial, but it points out all the reasons why you should pay attention to your data.

If this is tl;dr, then head on down to the Western Digital Home Backup Solutions section.

 

Backup, Schmbackup! Who cares it’ll be fine!

Let me preface this by saying I hate the term backup or data backup (disregarding the poll below). To me it implies a one way street – data goes from live or production, into a secure backup data store where it is magically protected. Personally I prefer to say, data protection. It is more holistic, covering data at all layers and stages of usage.

As I stated in the first paragraph, I am considered an ITDR SME. So often I run into people, even IT professionals who confuse data protection as ITDR. In fact so many people have no concept of what differentiates the two.

 

Data Protection vs ITDR

Data protection and ITDR are not interchangeable terms. In really simplistic terms, data protection is a subset of ITDR. Just a component into the bigger plan of IT Disaster Recovery, which incorporates processes, resources, recovery targets.

I was going to say, to be an ITDR SME can command a decent amount of money for the knowledge. Reality is that it is only valuable to those who value their information. A full ITDR process is a very involved process that requires deep consultations with each part of a business, working out their core requirements, processes and then breaking the complex into something that can be worked with.

At the end of the day, an ITDR SME has an intimate understanding of the business, their risk profiles, linkages, and present a roadmap to mitigate the risk profile. In essence we are translators for what business to continue functioning wants versus what business wants to spend to meet that goal.

ITDR itself is just one component in the broader Business Disaster Recovery Plan. At the heart of it, without data protection, the rest is pretty much irrelevant.

 

True story #1 – The SAN Corruption

A not-for-profit organisation was told their backup strategy is not robust enough to recover from a significant event. They also required a formal ITDR assessment to determine the overall risk profile to the organisation, their IT infrastructure and corporate data. It was a long and torturous sales cycle spanning many sales quarters but the board eventually decided that an ITDR assessment would be prudent and signed the purchase order on a Friday.

That weekend the NFP corporate office was the target of an arson attack. It was just a week before Christmas and it was all hands on deck. My company sent a number of staff interstate to the client’s temporary headquarters to start the rebuild process. It was about 3 months before we could say the client was considered fully operational from a business perspective. The entire core infrastructure was built on loan equipment from a major hardware vendor.

SAN Disk Array for data backup

My role was to implement their new data backup solution. Like any data backup solutions that I implement, I test it to death before I sign off on it. It was just as well because we were about to sign off on the interim infrastructure when I got a call at 3.40pm on a Monday afternoon.

I still remember the day, gorgeous Melbourne late summer, sun shining through the large windows of the office. The call came through that their SQL server had gone down. My SQL guy wasn’t around, in fact not many people were, so I said “I will have a look”. I did not sleep until 1.30pm Wednesday.

What started as a small isolated event turned out to be a SAN (Storage Area Network) wide corruption affecting every piece of data stored on the network. And the reason why we could recover the network as soon as we sorted out the hardware issue? Rock solid data protection implementation.

 

True story #2 – Mobile Device Management Fail

MDM PoliciesA corporate policy update locked out an iPhone. I got a panicked call asking for help to get back in, there were years of precious photos of my friend’s son that did not exist anywhere else.

As a favour I got in touch with their overseas helpdesk to explain the situation. I stressed that they cannot initiate a remote wipe to regain access that way.

An hour later I get a very distressed call that the phone is sitting on the “Hello, Hola, Buongiorno” screen. His computer is not recognising it as a known device at all.

 

What is the point of these two stories?

Truth be known data protection is rarely viewed as an investment by any company, let alone individuals. And ITDR? That’s a complete waste of money, isn’t it? Well, so is a car, health, a home and content insurance when you don’t need to make a claim on it. The point is it is considered a waste of money until it is not.

Unfortunately, a lot of integrators largely view it in the same light. Data protection is an afterthought, a bolt on that increases the cost of a proposal or solution. Therefore it is a line item that is generally there to tick a box with the best looking price point, rather than a solution that meets the undefined needs of a requirement.

 

Why do I say undefined?

How often do you think of data protection and what it really entails? I can say, when I ask this question to clients and colleagues, the answer invariably would be some form of getting a copy of production data and put it somewhere in case something goes wrong.

Simple isn’t it? Make a copy of everything that is important and precious to you or your business, in some regular kind of schedule. Voila! We have data backup. Note: I am specifically using the term data backup.

 

What does real data backup really include?

  1. Taking point-in-time cop(ies) of your precious data and storing it on media that is separate from your “production copy”.
    That means if it is on your computer, a copy is taken off the hard drive and stored onto something else – be it magnetic tape, an USB key, external hard drive, NAS, cloud storage.
    If it is on your phone, then it is backed up to computer hard disk or cloud storage.
    The key here is copying the data to a storage medium that is NOT the primary source.
  2. Set a schedule on how often this copying of data occurs.
    Traditionally backup occurs once a day overnight. This would leave about 24 hours of exposure to potential data loss.
    Most businesses would run numerous point-in-time “snapshots” of their data during core business hours. This reduces that 24 hours risk exposure to something less.
    On smartphones, some apps such as Whatsapp allow you to set backup schedule to run nightly, with the option of syncing to cloud storage.
    Personally I have an app that performs weekly backups of my SMS with a cloud storage option.
  3. Set and forget.
    Now that we have our separate media, a backup schedule, all is well right? Well this is where it leads me to the next story.

 

True Story #3 – Just the One Tape

Magnetic Tape - tried but true backup methods... when used correctlyAh, this was the heady days of Windows NT and Windows 95, and “Create a shortcut” was a brand new feature. A staff member had created a shortcut to a critical business file, and then deleted the file itself because you know, it’s on her desktop now.

It was not my client and I did not set up the infrastructure, but at 9pm my phone rang and I’m begged for a favour. Being the sucker for punishment that I am, I jumped into the car and duly rocked up to the site. I thought it was going to be something simple and quick and I would score a free drink afterwards.

When I arrived, I asked for the backup tapes and was told it is in the drive. Alarm bells are going off in my head here and I asked for the rest of them. Well apparently they back up to the same tape constantly and they only ever owned one tape. No one has ever checked to see if backup works. To add insult to injury, the tape was damaged from, well, basically being overuse.

Suffice to say they had to recreate that file from scratch, as well as implement tape rotations amongst other basic procedures. (Yes, it was a very short site visit.)

 

What is the take away here?

Your data backup is only ever as good as your vigilance on the process. Set and forget is wonderful, and will absolutely, always guaranteed to bite you on your butt at the most inconvenient moment.

Even if you ask IT professionals, how often do you really check the backup status? You may look at the green tick on the screen and say that’s good. But do you ever compare actual backup dataset size to size on original disk? Do the dataset sizes even come close?

What about data integrity? Sure you can set the verify after backup flag. It takes as long as the backup itself to do but you can get a nice green tick at the end for that, too. But do you go and randomly restore a bunch of files to know that you can when you want to?

You see, taking a copy of the information and putting it elsewhere is the “backup”. The bringing back of the information from the elsewhere back to where you can use it is the “restore”. Together, that is “data protection”.

And that is a part of the backup conversation that is rarely ever aired. The two go hand in hand; there is no backup without restore. Without both running in tandem, there is no data protection.

 

How many copies do I need? Backup the backups

How long is a piece of string? My view is, if you have your primary copy and your backup copy in the same geographic location, then you have nothing.

Brutal statement but consider this. Burglary, house fire, flood, acts of God. It’s gone in one fell swoop.

So how many copies do you need and where? That is the million dollar question and the answer comes down to:

  • How much information is there?
  • What is the value that information to you?
  • How paranoid are you? *twitch*
  • Do you have a data jurisdiction requirement? That is, does that data have to be physically residing in a specific region in the world?

 

I like the approach that is akin to dressing children for cold weather – layered.

  1. I have a local copy for anything that is “operational”. These include frequently used work files.
  2. I have a copy of all my photos on a NAS. This is replicated into the cloud, specifically Amazon Glacier. (I have other copies but *waves hand like Obi-Wan*, not publicising it is a layer of protection on its own.)
    Why Glacier? At the time I devised my strategy, it was the lowest cost per GB solution for my increasing collection of photos which is over 1TB in size now. Amazon Glacier is synced directly from my NAS to their data centre, with the throughput limited only by my local bandwidth.
    Glacier is also my last bastion if I lose everything physically. I can trickle the restore from their server free of charge and will take a long time. Or I can pay for a fast restore for things I absolutely need right away.
  3. By the same token, I utilise popular cloud productivity tools such as Dropbox, Google Drive etc, all of which are supported by my Synology NAS so I have multiple layers of protection.

 

Why the emphasis on off-site storage in data protection strategies?

Because I am paranoid. Many moons ago long before cloud storage, I had archival grade DVD-ROMs for backing up my photos which I had planned to send for storage with a friend in America. It means trusting him to look after my data, which I do (I just didn’t trust his ex-wives which was just as well).

Having an off-site copy removes the single point of failure of keeping everything in the one physical location.

Unless you have been living under a rock, you would have heard of ransomware. A particularly nasty, indiscriminate approach to … everything. In a nutshell, ransomware encrypts all the information on your computer, and tries to propagate through all your connected devices to do the same.

Generally you are given 72 hours to pay an “unlocking fee”. When that 72 hours expires the “fee” goes up. There is no guarantee that you will recover your files even if you pay, but if you have no other choices, then it’s worth a gamble.

An off-site copy means it is “air-gapped” from your normal environment. Sure that means the data is older but it sure beats not having anything at all.

 

True Story #4 – Ransomware

RansomwareOne evening I received a frantic call from a client. One of the managers opened the wrong email and copped a ransomware attack. I did not even finish listening to it when I told the caller to get into the office and isolate his server ASAP.

 

Luckily the manager was working remotely at home, so the ransomware attack was limited to his computer. Naturally, he had important files on his machine that he needed. Luckily I had layers of redundancy and backups in place that were not publicised. After some very careful isolation of the infected machine, I managed to retrieve all the files with only 8 hours of work lost and came out looking like a hero.

 

Are these all scare tactics like “Y2k bug”?

Y2k bug was not a scare tactic. I was eyeballs deep with my clients in preparing for it. A lot of hard work by a lot of people to ensure that it all seems like a hoax.

 

True Story #5 – Our DR Plan is Good Enough.

Data protection is essential for a rainy day... back up in a variety of waysMy boss and I had been working to close a deal with a client to assess their ITDR plan and mitigate the known risks. We knew it had major flaws but the client was unconvinced.

I remember distinctly being told, “We have a disaster recovery plan and we are happy with it.” Famous last words, eh?

One particularly bad winter storm in Melbourne resulted in widespread damage and flooding. I happened to be standing next to my boss when the call came through. The client’s IT Manager is standing knee deep in water in their data centre, and their ITDR plan activation failed. They wanted our help urgently.

Unfortunately, our answer: nothing we can do.

There is no edit-undo when an incident strikes. Your only mitigation is to be always prepared for the eventuality. Like insurance. Like a fire plan. It only works if you invest the time, effort and money into it first.

 

Backing up into the cloud: the one stop shop

It is there. It is convenient. Some of it is free, some are cheap. The question is, is it there forever? I have seen so many come and go. Forever plans that have a different definition of forever than in the Oxford dictionary.

With the OVH data center fire impacting some 3.6 million websites on 464,000 distinct domains and potentially no recovery of data possible, it is all eggs in one basket. There is a flow on effect on other sites, ironically it is also affecting cybercriminals.

No doubt such events at a data center are one in a million event, but it only takes that *one*.

Mind you, aside from the obvious, something went catastrophically wrong with how the data center handles a fire event. Personally I would be most curious to see the final report, although, it would never be made publicly available.

 

True Story #6 – I back up to the Cloud. It’s fine…

Cloud Computing and backup

This one is actually with my brother. The MSP his business server was hosted on ran into “difficulties”. Well, the difficulties turned out to be a rogue ex-employee who crippled the company infrastructure and also destroyed the backup data whilst he was at it.

Luckily for him, he was granted about 24 hours to access his server. Between his little brother and his colleague, we pulled a copy of his server out of there pronto.

This one earned us a bottle of Ardbeg Alligator and two matching Ardbeg glasses. Slainte!

The lesson is, what cloud gives, cloud can take. There are attempts to breach accounts going on constantly. Spear phishing, social engineering, brute force, even attempts by ex-loved ones.

Storing all your data in the cloud is no safer than having all your eggs in one basket. One lax mistake and the bad actor who gets access can wipe it all out.

 

So what happens now? How can you employ data protection strategies?

Another suitcase in another hall. No, sorry Madonna, that isn’t going to fix it. Now that you have made it this far: what can you do?

As I mentioned above, there are plenty of data protection solutions out there. No one of them is one size fits all because we all value our data differently. Generally speaking for the average user, you can probably afford to lose a few days of messages and photos. Sure, it hurts, but it beats losing it all.

Layering your data protection for all the aforementioned reasons would beat a single point of failure.

When it comes to mobile devices, we are in an age where backup is (largely) available in the background.

For Android devices, you can turn on Photos backup as well as system backup to protect your information. The information is stored with Google, and may incur a cost depending how much your data is.

For iOS there is iCloud, which again has a cost component if you need to have a complete backup of your phone.

 

Western Digital Home Backup Solutions

What you need depends on your usage, and WD breaks it down into three broad categories:

  • Single device at home
  • On-the-go and need portability
  • Whole house with multiple devices

 

WD My Book - a single device data protection deviceSingle device: I mostly use my one home computer

A standalone, AC powered hard drive with an USB interface will give you a level of protection for your data. Depending on what you use (Windows / Mac), look for something that can work with your computer. Most brands will have some sort of software bundled in to help the backup process.

The Western Digital My Book Desktop comes with 256-bit AES hardware encryption, bundled with everything you need to get started right away. It supports both Windows and Apple Time Machine.

It also comes with WD Discovery to help preserve your online life, allowing you to import content from popular social media and cloud storage services, like Facebook, Dropbox and Google Drive.

The WD My Book Desktop is available in capacities between 4 and 18 TB in 2 TB increments.

 

WD Passport UltraOn-the-go: I need portability with my data protection!

Small, lighter and portable than the WD MyBook Desktop, the My Passport Ultra is for people that need their data to travel with them.

Although it is geared for Windows users, Mac users can download the NTFS driver for macOS to enable compatibility. It also comes with 256-bit AES hardware encryption, and WD Discovery for that all important preservation of your online life.

The WD My Passport Ultra drive is available in 1, 2, 4 or 5 TB sizes.

 

WD My Cloud Home - whole of house data protectionWhole house data protection: protection for everyone

The centralised unit for everyone, the My Cloud Home is the personal cloud storage device that plugs directly into your Wi-Fi router. WD shies away from calling it a NAS (Network Attached Storage); rather, it is a simple centralised solution to backup your files in one location.

For those technically inclined, the My Cloud Home has two drives in a mirror configuration – meaning your data is written on two physically separated disks for redundancy. Both Windows and Mac are supported.

As a personal cloud storage in your home, there are no on-going subscription fees. Your data is accessible from anywhere (with an internet connection) using the My Cloud Home mobile and desktop app or MyCloud.com.

Each user you grant access to have individual accounts with their own private space, where they can manage, access and share their files as they wish.

The WD My Cloud Home is available in 2, 3, 4, 6 and 8TB capacities.

 

Western Digital is not the only solution out there, but they do have a product that will fit many situations. Remember whatever you do, consider the importance of your data to you. I refer back to the six true stories that I wrote in this article – it may be worth investing in multiple strategies. You may opt for a local copy, as well as a cloud copy akin to what I do in a nutshell.