The Art of Information >> Blog Archive >> Storage for the 31st Century?

Storage for the 31st Century?

by Pete Gerr on March 2, 2010

In 2010 alone, we will create 5-10 times the amount of information EVER created. What’s more, the pace of digital information creation is accelerating. Whether using Moore’s, Metcalf’s or any other law you might like to measure against - the “up-and-to-the-right curve” of information growth is arcing more and more vertical - outpacing all historical comparisons. The question certainly on the minds of HDS enterprise-class customers, those in small-to-medium-sized shops, and increasingly of prosumers with multi-TB personal archives like myself is how are we going to store, retain, and archive all this information long-term?

However much actual information industry analysts, pundits, or informationists like myself predict is being created annually, the important point to remember, I believe, is that we’re just getting started. Even though I content that “Information is the currency of the new economy”, only 20-25% of the world’s population even use the internet - there is unfathomable growth coming in the future. Perhaps as few as 10% use “information-rich” sites like Facebook, Flickr, or YouTube, or have a digital camera or camcorder that create and store massive amounts of digital data.

While countless industries - retail, media & entertainment, real estate for example - have been remade over the past 15 years by disruptive business models fueled by the growth and expansion of “always on” internet connectivity, these same industries are still learning to deal with the challenge of maintaining large digital archives of valuable information assets. Some industries, like healthcare, are still just beginning to be transformed and digitized.

The challenge? Hard drive (HDD) technology has remained virtually unchanged for decades - though HDD media continues to become more dense and affordable, we’re still packing 1s and 0s onto spinning platters of aluminum alloy or glass and reading or writing them onto that surface with a mechanical arm. While Hitachi’s own Ultrastar A7K2000 drive can now store the equivalent of about 1,000,000 3 1/2″ floppies in a 3.5″ form factor, access speeds, and reliability of the media have improved, but have struggled to keep pace with the explosion in capacity (that said, no modern media types have).

What’s more, some modern digital storage media types continue to become obsolete or impractical - we’ve seen the extinction of several media types in just the past 10-15 years. The 5 1/4″ floppy disk ceased to be shipped with new systems in the late 1980s, and the 3 1/2″ disk that supplanted the 5 1/4″ met its demise about 10 years later. CDs have become pretty much disposable media and we’re stretching the boundaries of dual-layered Blu-ray DVDs. So what’s next?

Hitachi Data Systems, and its parent company, Hitachi Ltd. have been at the forefront of digital recording media and archival systems since the mid-1950s. Hitachi’s Global Storage Technologies (HGST) continues to invest heavily to bring digital storage media improvements in density, performance, and reliability to market. Innovations such as patterned media and thermally-assisted recording are but two of the novel methods being explored by GST to address the data explosion, which promise to revolutionize long-term information preservation. In fact, Hitachi recently reported a breakthrough using the later method which could lead to a 5-fold increase in hard drive capacity; this improvement could vastly reduce the amount of floorspace, and by extension the heating and cooling required, to maintain a large digital archive.

Think about electronic medical records - basically storing and maintaining a person’s entire medical history, from birth to death, and even beyond, electronically (thanks to compliance regulations), immutably and still have it available at a moment’s notice, for instance if the person was brought into a hospital while on vacation with a serious illness. The average human lifespan is 70-80 years (it might be 100 or 150 years soon, but that’s another topic), hard drives might last 5-10 years, but major changes in recording or media technology necessitate a data migration every 20 or 30 years, or 3-5 major migrations over the life of an average patient. Electronic medical records are just one example - I have a HD-capable video recorder in my pocket right now that consumes about 30MB for each minute of video I shoot - that’s about 2GB per hour.

Clearly media alone cannot solve the problem of long-term archival; storing the data is sort of the easy part - how do you FIND that single email, photo, or x-ray you’re looking for in a digital haystack filled with billions or trillions of records?

Leading digital archival solutions like the Hitachi Content Platform, combine the most advanced hard drive technologies available with intelligent software to provide a secure, cost-effective, intelligent, yet easy-to-implement solution to meet corporate governance, compliance, or simply preservation needs.

HDS and Hitachi continue to explore options for extending the useful life of both digital information and the media on which it’s stored. The possibilities are endless, but so too, is the amount of new digital information being created every day, every hour, every minute. I’ll continue to track innovative developments in this space both within Hitachi and HDS and elsewhere.

Is your company struggling with or already come to grips with its digital archiving needs? Would like to hear from you and what you think the future holds.

If you’re interested in learning more: