Data Center Advisors

The trends are clear and spectacular.

We create, collect and store an immense amount of data and information at an exponential rate.

IDC research has shown that in 2009 there were 800 exabytesof data (that’s 800 million terabytes), by 2020, they forecast this number to be over 35,000.  It’s hard for me to wrap my head around such a large number, but “Data, data everywhere”, an article published last year by The Economist, put the scale and magnitude of this increase in perspective.  The piece noted that Wal-Mart “handles more than one million customer transactions every hour, feeding databases more than an estimated2.5 petabytes—the equivalent of 167 times the books in America’s Library of Congress.”

Do we really need all this data?

Well, I think we do.  (A separate post on Information Overload is sure to follow if we can think of something new to add and thereby add to the noise…er… I mean information.)

First off, I’m in awe of the totally cool and amazing things people are doing with information that was once not even a consideration. In a recent article on Feedback Loops, Wired magazine reported some fascinating work done by Shwetak Patel, who translated “the cacophony of electromagnetic interference into the symphony of signals given off by specific appliances and devices and lights.” Through a single device in a single outlet and a stack of algorithms, he could tell if someone left the blender on, and thereby possibly get assistance where needed.   Who would have thought that your house has such unique and useful electronic fingerprints?

And then even the most seemingly useless information—such as Twitter status updates on anything and everything and nothing—is proving to be valuable. Last year researchers at Indiana University and the University of Manchester submitted a paper that claimed and illustrated at a high level that the, “moods expressed in Twitter feeds can accurately predict some changes in the Dow Jones Industrial Average three or four days before they occur”. If this seems far-fetched, well I’ll have you know that the Library of Congress is now saving your tweets. Yes, our tweets are now part of the data stream that is considered a historical timeline, as reported by the New York Times last year. Even though this information previously had questionable relevance, there is obviously an expectation of current importance and future potential.

(Am I a total geek for hoping that someone will dub in “Every Bit is Sacred” over a somewhat well known Monty Python tune?)

When I start thinking about the idea of saving all possible data, the image of my garage suddenly pops into my mind.

No matter how many new and valuable things I want to pack into it, I only have so much space.  And that leads me back to that The Economist article on “Data, data everywhere”, because they also reported on IDC studies indicating the amount of digital information being created already exceeds the amount of available storage, as illustrated in this chart.

Now this really starts to look like a classic hockey-stick problem.

In a Washington Post article earlier this year they cited a University of Southern California study regarding the massive overtaking of digital data over analog data, which this chart really illustrates.  The piece quotes one of the authors of the study, Martin Hilbert, who states that “Humans generate enough data – from TV and radio broadcasts, telephone conversations and, of course, Internet traffic – to fill our 276 exabyte storage capacity every eight weeks.”  And that’s JUST the human generated data!

So what does this mean now?  What happens when this starts to look like 0/1? Better technology and quantum mechanics to the rescue?

I don’t know.  But I do know the only way I fit more stuff in my garage is to organize, repack and prioritize what stays. Knowing there will come a time when you, your organization or your clients have more digital information than space available, how are you planning to prioritize?  Are you taking any steps now?

Want to read more about our Cloud Roadmap? Visit our bit.ly bundle here: http://bitly.com/pCt5Gk