So I came across this interesting link on how a Canadian company had been using a Disaster Recovery vendor that was unable to execute their contract during the massive East Coast power failure in August of 2003. I won’t put the details in this blog since both the original DR vendor and the newly selected DR vendor are, in my opinion, outstanding and reputable firms, as is the customer in question. In my mind, this is not about failure at all, but about often overlooked details in planning a good DR strategy. In this case, the DR vendor was unfortunately on the same power grid. Someone, somewhere, should have caught this nasty detail.

So this story is in the subtleties, the “gotcha’s”, and how a $5 O-ring can cause a disaster. My favorite of these stories was a West Coast bank that, after our big 1989 earthquake, cheered because their backup generators kicked in flawlessly on the subsequent power failure, and operations continued. For a while.

It seems their generators were on the roof of the datacenter but their diesel supply was in the basement and the pumps used to bring the fuel to the roof were, you guessed it, electrical. Thirty minutes after the cheering, blackout occurred.

There’s a moral in here somewhere, that it’s the little and overlooked details will ultimately bring you down. Those of you serious about Disaster Recovery should look again to see how prepared you really are for the event we all hope never occurs.

Comments RSS