I would like to hear your opinions as to the information in the chart below (my timing might be a little off, but you should be able to get the gist of this post). Fortunately or unfortunately, I’ve been in the technology industry for a long time. I have participated in and implemented optical technology both professionally, and as a “prosumer” (prosumer in this case is a very knowledgeable consumer, or professional consumer). Actually, most of you have as well whether you know it or not.

Let me explain what I am trying to describe in this chart at a high level. The first commercially available Compact Discs (CD) hit the market in the early 1980s. There were eventually two formats released which allowed higher capacity going from 650MB per disc to 700MB per disc. The first commercial use was for music distribution and competed with tapes (cassettes and 8-tracks) and vinyl records. Tapes and records were analog recording technologies while CDs were digitally recorded. The important takeaway here is, digitally recorded music means music was stored as digital data and converted to analog sound. Soon thereafter, CDs were used as a vehicle for other content distribution like audio books, video, documents and multimedia documents, games and software. Today, in 2012 (about 30 years later), CDs are still in use for content distribution, consumer storage, software games and data distribution, etc. During this time, the prices dropped dramatically for both the blank media and the drive devices for reading and writing, yet the quality and features continued to increase. I like to use my example of a used CD I bought at a swap meet for a $1.00 that didn’t work 12 years ago on the drives back then (I knew it wouldn’t work due to the scratches, but “Dark Side of the Moon” for a buck??), but works today on a modern drive. I’m not saying this always works, but in this case, it did.

Around the early to mid 1990s (roughly 12 years later), the follow up to CDs was announced, Digital Versatile Discs, DVD. This time, multiple layers for recording data was incorporated with 4.7GB and 8.5GB capacities. Primary market focus was video distribution competing with VHS videotape and possibly LaserDisc (though no real threat on that front). The follow on markets for DVD included, game distribution, software distribution, music, consumer and enterprise recording for content distribution, backup and archiving, and other multimedia documents. Today, DVD is still the main media distribution vehicle for software, games, consumers, etc. We are currently in the “overlap” era between Blu-Ray (BD) technology and DVD. The overlap era between movies on VHS tape and movies on DVD lasted about 2 to 3 years, and my guess is the overlap era between DVD and BD will be much longer. The cost of DVD blank media and drive devices for reading and writing, have also dramatically dropped like CD technology. One important note here, every DVD drive device today, reader, writers, rewriters, supports both DVD and CD media.

Today, Blu-ray Disc (BD) is the new media introduced around 2006/2007 and became the supported standard around 2007. Blu-ray is still in the overlap era with DVD for movie distribution and as the other content distribution vehicle, but also in the mix are online streaming services, application downloads and “cloud”. In this regard, I’m only going to discuss the BD technology. BD capacities when announced were 25GB and 50GB (dual layer) and famously competed with Hi-Definition DVD (HD-DVD). This consumer-based, public battle was not the replacement of a main staple media like CD’s taking over audio tapes and vinyl records for music, or DVD replacing VHS video, but more like the VHS versus Betamax battle for the video tape standard back in the 1980s. The new BDXL Blu-ray format now supports 100GB and 128GB capacities using both additional recording layers (3 and 4 layers) and a new recording format to increase the per layer density. Projections are that this many layer approach for the Blu-ray technology will continue for some time to increase the capacity density of the media and drive down the bit cost per disc.

Ok, now that the history and background is laid out the way I want it for this article, let’s talk about longevity. Today, if you Google “buy a Blu-ray drive”, you will see internal drive devices for around $60 with a modern SATA interfaces. If you read closely, these drive devices are capable of reading and writing media for CD, DVD, and BD (BDXL support is still new and a little more expensive for now). For a few dollars more, you can even have the drive device write custom labels for you directly on the disc. Think about that, a brand new, mass-produced, commodity modern device manufactured to support a 30-year-old media technology for around $60. The device itself is greatly different from the original CD device from 30 years ago with newer technology, enhanced features, new interface(s), faster, smaller and denser packaging, etc. The oldest CD in my collection that I burned is from March 1995, and this disc still works today in my new MacBookPro (of course, this CD isn’t stored in the bottom of a drawer somewhere), and my first music CD from the 1980s still plays today in these new devices.

In my chart, I tried to illustrate the industry standard interfaces of the time of the introduction of the technology such as SCSI and IDE, and so forth (avoiding the exotic stuff). This shows the evolution of the technology world advancing forward with faster and more economical interfaces, improved manufacturing efficiencies, and so forth, but here’s my take: with the many markets that use these technologies, especially the consumer and high volume markets, the support for legacy media isn’t an option. My guess is, it could be harder to rewrite the firmware to drop support for these media formats than it would be to just keep including it going forward. Bottom-line is 3 generations of media supported with read/write capability today spanning about 30 years with improvements and added features, all with reduced pricing.

Now let discuss tape, specifically Linear Tape-Open (LTO), and more specifically tape for long-term data preservation. From the Wikipedia entry for LTO, a modern LTO tape drive, for example LTO5,

• Can read the current generation tape cartridge (n), LTO5, and the two prior generation tape cartridges (n-2), that is, LTO4 and LTO3 written by those tape’s associated tape drives in their native capacities and format
• Can write a current generation tape cartridge (n), LTO5 drive writes to an LTO5 tape cartridge, and the prior generation tape cartridge (n-1), LTO5 drive writes to an LTO4 tape cartridge at its native capacity and format
• This would apply to all tape generations. For example, an LTO4 tape drive can read LTO4, LTO3 and LTO2 tape cartridges, and can write an LTO4 and LTO3 tape cartridge

Also, according to the LTO generation table below from the same Wikipedia page, LTO dates back to 2000 with new generations of LTO standards introduced on average about every 30 months (2 ½ years).

The next generation LTO release, LTO6, is rumored to be out by the end of this year, 2012. This means that read support for LTO3, released in 2005, and write support for LTO4, released in 2007, will be dropped at that time, or at least for the LTO6 tape drives. This is the 6^th generation of this technology in 12 years, so this standard moves very fast, maybe too fast. For long-term data preservation requirements, the idea of technology and media being obsolete so quickly drives Operational Costs (OPEX) higher over the lifetime of the data due to technology and media migration costs, which in many cases is, forever.

Granted, the standard optical storage track (notice at the bottom of my chart the carcasses of defeated technologies throughout this optical storage history) is currently only in its 3^rd generation, so there’s no telling what will be supported in the next generation. There are some holographic technologies that promise backwards compatibility, but maybe not for all generations and with a loss of significant advances like capacity, and there are some holographic technologies that might be disruptive to the compatibility track. Then there’s the notion that the market could split for consumer and distribution based holographic technology and support, and a holographic archiving and enterprise class. The trick will be to maintain the price curves using common manufacturing and parts.

Currently, Blu-ray has its pros and cons. Technically, the BDXL format is denser than the current LTO5 tape cartridge when measured in gigabytes per cubic inch, data stored is randomly accessed which is great for non-streaming use cases, has a highly rated durability factor (to be discussed at another blog), and has media longevity options that will last 30 years with standard quality media, and 50 to 100 years for special certified quality media. The technology evolution is slow, but this is not necessarily a negative situation when long-term data preservation requirements need a stable, enduring technology. Stable technology here does not mean static. The technology advances and improves, which applies to the older media as well, but the requirement to support the media going forward has several market segments driving the support requirements for stability while also driving down cost. Stated another way, you cannot upset billions of users and consumers in a highly competitive market, and stay in business long.

Tape is faster for streaming uses cases like backup restores, but is not suited for randomly accessing its stored data. In fact, randomly seeking within a tape shortens its lifespan, as does every tape load operation. It will be interesting to see what the longer-term effect will be with the support for LTFS (Linear Tape File System) for archiving and data repository systems that have a slightly active requirement to them. However, with the short and accelerated compatibility matrix, this may not be an issue, as the underlying technology will require higher tape migration frequencies in order to remain supported and compatible.

What are your thoughts and experiences on this subject? Do you agree with my chart? What’s the oldest CD or DVD you’ve burned that still works? Do you have any predictions going forward?

A highlighted list of these new and enhanced features include:

Improved Operational Efficiency to Lower Costs

• Lower costs for large scale unstructured data storage and reduce overall energy consumption with HCP support of spin-down disk in Hitachi Unified Storage (HUS)
• Eliminate downtime with nondisruptive, online hardware & software upgrades
• Proactively address any bottlenecks or hardware issues before they impact SLAs with improved component and performance monitoring and email alerting

Greater Scalability and Reliability

• Improve economies of scale, reduce costs and maximize utilization with support for thousands of tenants and tens of thousands of namespaces per system
• Ensure service availability with advanced replication and failover capabilities
• Meet vaulting requirements with support for tape-based copies of objects

Robust Security and Control from Edge and Core

• Reduce risk and control access to content with new object access control lists
• Support corporate security policies with active directory integration
• Identify sets of related objects for information, action and automation with custom metadata search across system and custom metadata

HCP is already one of the densest data storage platforms in the industry scaling from a few terabytes up to tens of petabytes in a single system, easily and seamlessly. In fact, it’s this scalability and multi-tenant support that has transformed HCP into the premiere object storage platform for the massive scale-out of unstructured data management in the industry. HCP embodies our content cloud approach allowing organizations to store and manage billions of data objects while providing intelligence layers and policies to help index and search the data independently of the application that created it, expand and scale to match or exceed to the unstructured data growth rate organizations are experiencing, and protect data in the most cost efficient manner and of your choosing.

While I’m flaunting and listing the individual features and enhancements of the new release of HCP as isolated capabilities, the overall total result is an unstructured data storage platform that is flexible and agile, that scales to meet any demand, and which provides upper layer capabilities embedded directly in the platform. This advanced combination of feature sets and reliability, without the traditional management complexity, is unique in the industry.

Actually, the flexibility of HCP may cause a slight retro-effect. HCP and its predecessor, HCAP (Hitachi Content Archiving Platform), has always been the platform of choice for compliance based archiving. That is, customers that HAVE TO archive data because of compliance laws and regulations of their respective industries. HCP can be configured to erase and/or digitally shred data when it is legally time to, and keep data safe and immutable in the meantime. For the data archiving use cases of HCP, there are many customers using HCP for long-term data archiving, but many more customers use HCP for shorter-term archiving based on regulation requirements and retention timeframes.

So, this is a series of articles on archiving data, specifically, long-term data archiving. Now, when configured with the new HUS storage platform of  products, HCP is more cost effective and environmentally friendly with the support of disk drive spin-down for overall power savings over the life of data. Data stored in HCP can be stored on HUS storage platform that can spin its disk drives down consuming less power and generating less heat, thereby saving on datacenter cooling power as well. For data that needs to be stored for a long period of time, this new feature can have a significant positive impact on the Total Cost of Ownership (TCO) over the lifetime of the data where power and cooling costs add up to a significant operational cost over time.

At one point in time, Massive Arrays of Idle Disks (MAID) was once a technology, coined by Copan Systems, captured the imagination of many, but failed to fulfill the promise. Since HCP maintains and stores metadata and data differently, the requirement to continually reference storage for even the slightest metadata reference is negated. So while metadata can be searched, queried and referenced actively in HCP, the actual data does not have to be active or accessed. In an active archive or data repository used for research or as a library of information, searching through indexes, metadata queries, and custom metadata searches and queries will be the majority of the activity in these types of systems. The actual retrieval of data based on search requests, will most likely be the last task performed.

HCP lays down a critical foundation for future ways of designing cost efficient data systems for large active data archives, research libraries and other long-term data repositories. While HCP lists an impressive number of significant enhancements to simplify the complex task of managing massive-scale data repositories, disk drive spin-down support is my personal favorite. With environmental impact and energy costs on the forefront of many people’s minds these days, this exciting feature allows the current explosion in global data growth to also have a positive global effect.

With HCP and HDI we are bringing well over 130 new features and capabilities to the cloud-enabled object storage market. While documenting a log-like post of every detail would be downright boring I believe referencing the top 5 HCP capabilities and top 2 HDI capabilities will expose the stellar work completed by our HCP and HDI R&D teams.

HCP’s top 5 new capabilities are:

1. Energy efficient content storage via spin down disk on HUS (Note: this will be covered in depth by Ken Wood)
2. A radical increase of the number of tenants and namespaces
3.  Improved authentication and authorization through Microsoft AD support and S3-like or S3 inspired object level ACLs
4. Native metadata search via the enhancements to the existing Metadata Query Engine (MQE)
5. HCP packaged in VMWare to support limited use cases like custom application development, evaluation, PoCs, etc.

HDI’s top 2 new capabilities are:

1. File pinning at the edge to ensure fast, local access to designated files
2. Improved VMWare appliances supporting HA and easier installation

HCP code named Godzilla – Scaling done right

As HCP has evolved, the need for extreme scale has become paramount. Extreme scale means different things to the different types of users. For example application authors are likely more concerned about how to best extract solid performance, object density, etc; whereas service providers are more interested in things like extreme tenancy, operational efficiencies and stellar manageability. For HCP we’ve focused on the needs of the application and service provider by expanding key attributes of the system, improving user performance/usability, and zeroing in on the details needed to improve online upgrades. I’m using the image at the beginning of this section to illustrate that when we build features and capabilities into HCP we look at all aspects. Specifically with HCP plumping up the number of supported tenants and namespaces to 1000 and 10,000, respectively, we needed to ensure the administrator can gracefully interact with an extreme number of tenants. Further, tenants themselves are now more power packed through the additions of more protocols (i.e. SMTP, NFS, CIFS and WebDAV), a new external authentication method (i.e. Microsoft Active Directory), and S3 inspired ACLs — a great illustration of a feature with strong manageability. Object level ACLs can be individually controlled by an application or user, but application or service providers can establish overrides at the namespace or tenant level which are then “pushed down” to all objects below. This kind of activity is critical for use cases where application and service providers may want to quickly remove or restrict access to an entire tenant when something untoward has happened such as a security breach. Again we’ve made this feature scalable both for other applications and systems using HCP and for the administrators.

You’ll see this same theme applied to nearly all of the features our monster Godzilla brings to the table with the most recent release. Whether it is a seamless online upgrade or a VMWare version available for an application developer or for special evaluation, I believe you’ll see we’ve done scaling right!

HDI code named Emerald — Access done right

While many in-development or modern applications can take advantage of the cloud or object lingua franca, REST over HTTP, many existing applications and certainly human beings cannot. Further, we know that many users are interested in relieving pressures on WANs and pushing out content to remote sites both of which are key facets of wide area collaboration. Today I’m proud to talk about two key new HDI capabilities that add to its growing arsenal of capabilities: file pinning and availability/reliability improvements to our VMWare appliance model. Firstly, file pinning allows an administrator to define which files will remain in HDI’s cache. From a user’s perspective pinning allows popular or hot content to sustain higher performance because in a WAN style deployment the entry point to the content is intentionally close to the user. Complimenting file pinning is an HA cluster version of our popular Virtual Machine Appliance (VMA) model of HDI. HDI’s HA VMA allows administrators to quickly and easily deploy and re-deploy a highly available HDI infrastructure in a remote site. When taken together, these two capabilities allow users to rely on the power of HDI, afford administrators simple administration meeting users needs, and allow companies to deploy infrastructures leading them down the path towards wide area collaboration.

Our keen attention to detail in our gem of a product, HDI, helps our users do access right so they can begin to meet objectives like reducing pressure on WAN pipes and move forward in generating new information over distance.

Why is this relevant and why I’m excited?

The last points on HDI are really why I’m excited: efficient usage of WAN resources to share and engender insightful information that matters. Intimate interaction with our global customers produced the spark that headed us down this specific path for HDI and HCP. Here is an anonymous quote from an EMEA customer which crystalized our goal:

“I want to be able to find and share information from both London and Hong Kong so that we can create collaborative projects to analyze markets on a global basis.” (EMEA Financial Services Customer)

Helping our customers and users get at and manage their information without limitations is really what jazzes me and all of the teams at Hitachi and HDS. I hope you can see that we are moving intentionally down this path, and I really appreciate all of the efforts by the teams so far to get us to the super competitive position we are in today.

1. Those that HAVE TO archive their data,
2. And those that WANT TO archive their data.

Of the many companies I’ve talked to about archiving of their digital data, these two types standout. Sure, there are a few industry segments that HAVE TO archive, and have for a very long time, like healthcare. Typically however, the data archived in healthcare is active data, and while compliance mandates the retention of healthcare records for the life of the patient (plus some number of years beyond), the records are actually used and provide long-term benefits to the overall well-being of patients. Plus, much of this data is redacted of personal information and preserved for research purposes. So, I typically classify healthcare as the WANT TO type.

Another industry segment that potentially has a duality between HAVE TO archive and WANT TO archive is manufacturing and engineering. The original designs, test data, simulation results, etc., typically are kept for a very long time by choice, mostly because researching prior designs or results, or reusing old test data can yield many benefits in newer designs. However, many sub-segments of this industry also have compliance laws or regulations to adhere to. For example, design data in the aerospace industry or medical device manufacturers (including components of the devices) must be kept for years beyond the end-of life of an airframe or device. So again, of the two types, I classify manufacturing and engineering as a WANT TO industry.

Now, for the easy to classify types. The industries that I label as the WANT TO types are typically those that generate or acquire content. Movie and animation studios, publishers, video game studios, marketing firms, recording studios, research and academics, national archives and possibly law firms fall into this category. The content that is generated typically has a large investment in generating this data, and in some cases can provide a source of future revenue. In other cases, the capability of the state-of-the-art doesn’t allow for sufficient processing today.

Take a movie studio for example. New distribution formats (VHS to DVD to Blu-ray), “never before seen footage”, director’s cut editions, 20-year anniversary editions, collector’s editions, and so on and so on are ways that original content is reused and retrieved from archives. These assets can have a business requirement to maintain this content forever, or at least for the life of the company. In fact, there are cases where the archives of a failed company are worth more than the other assets of a company itself during acquisition or takeover talks.

There’s also the government of countries that want to preserve the history and culture of a country. Besides trying to convert everything to a digital format and digitally archiving data, the long-term language used by these organizations takes on an ominous tone: “Data preservation for the life of the republic”. The retention requirements during these types of discussions are always fun and challenging to participate in. Some organizations have requirements for storage media that is electromagnetic pulse (EMP) proof, or the ability to survive a disaster. Again, quite a grim tone is set when this comes up.

There’s also a different mindset in managing these data repositories and archives. This is their primary storage. Capital expenditure (CAPEX) takes a backseat to operational expenditure (OPEX), but the data stills needs to be accessible in a timely manner. For most government related meetings about archiving data, 25 year planning is considered short-term planning. The operational cost of managing data through many technology migrations and possibly data format migrations are painstakingly planned. The space, power, facilities and maintenance costs are the overwhelming expenditures “over the life of the republic”. The WANT TO archive types consider archiving as the primary mission.

On the other hand, those organizations that HAVE TO archive, usually do so because they are mandated through laws and regulations in order to do business. They must be compliant and prove it in many cases. Now, don’t confuse the HAVE TO types with a WANT TO types for these cases. It’s true that companies don’t want to pay a hefty fine due to not having this data available, squarely landing them in the WANT TO category, but for violation avoidance reasons. Given a choice, most would delete unwanted data. In fact, the compliance archiving mentality is to be able to prove that you have the data for the required retention period, but you can also delete the data as soon as the retention period expires and they need to prove that the data doesn’t exist any more.

These organizations are concerned with primarily CAPEX with some concern for OPEX, but only in light discussions. These environments rarely prepare beyond the 5 year total cost of ownership (TCO) plan, as it is a very long-term endeavor. In fact, it has been stated to me that no CIO looks beyond 5 years in a TCO study with which I counter: “Maybe in these industries. You should talk to Hollywood”.

So, it comes down to compliance archiving, which is typically seen as a HAVE TO archive activity where the data is rarely accessed, if at all, but must be there just in case an auditor or regulator wants to see it. This data is usually short term and uses language like expiration dates and data expunging.

At the same time, long-term data preservation activities are typically a WANT TO archive mission, where data is reused, researched and retained forever. The planning for these types of systems uses its own type of language, such as 100-year archive, longevity and durability. With the forecasted growth of data we create as humans and the amount of machine-generated data being created, (and with the “keep everything” mindset, mostly due to our current inability to do everything that needs to be done with the data) the discussions of Exabyte –1,000,000,000,000,000,000 bytes or 10^18 bytes –archives have been in some serious discussions recently.
So, what’s your archive type? Is your organization a WANT TO archive or a HAVE TO archive? How busy or active is your archive? I would love to hear about your organization.

As a footnote, Hitachi Content Platform (HCP) is a great solution for compliance archiving. The ability to immutably retain data and digitally shred data (when allowed) is a great way to manage these compliance-based archives. In fact, HCP is the archiving platform used by many organizations for long-term archiving projects with no plans for deletion. This is sort of a paradox to this article, but goes to show the versatility of the HCP solution.

Do you remember them? If you go back that far in personal computing land you should recall what an external FPU or math co-processor is. Here’s the Wikipedia definition for context, which I find personally very interesting for this post:

A floating-point unit (FPU, colloquially a math coprocessor) is a part of a computer system specially designed to carry out operations on floating point numbers. Typical operations are addition, subtraction, multiplication, division, and square root. Some systems (particularly older, microcode-based architectures) can also perform various transcendental functions such as exponential or trigonometric calculations, though in most modern processors these are done with software library routines. In most modern general purpose computer architectures, one or more FPUs are integrated with the CPU; however many embedded processors, especially older designs, do not have hardware support for floating-point operations. In the past, some systems have implemented floating point via a coprocessor rather than as an integrated unit; in the microcomputer era, this was generally a single integrated circuit, while in older systems it could be an entire circuit board or a cabinet. Not all computer architectures have a hardware FPU. In the absence of an FPU, many FPU functions can be emulated, which saves the added hardware cost of an FPU but is significantly slower. Emulation can be implemented on any of several levels: in the CPU as microcode, as an operating system function, or in user space code. (source: http://en.wikipedia.org/wiki/Math_coprocessor )

If you’ve noticed the bold and colored sentence in the selected text above, it points to the fact that most modern processors have replaced math co-processors with embedded Floating Point Units and software libraries. So what has happened is that a previous cottage industry, which provided ASICs functioning alongside a CPU, have disappeared.

However, that hasn’t stopped new technologies from cropping up in the area of numerical processing. A type that has become extraordinarily popular for graphics and vector processing of late are GPUs. For specific numerical and highly parallel tasks GPUs with standard x86 CPUs have arrived on the scene and become popular for increasing compute capability while decreasing physical system footprint. Generalizing a bit, what I see is the sedimentary hypothesis in action: separate HW function lives for a while, but eventually, when functioning as the microprocessor, libraries and compliers become good enough that the need for a separate HW goes away. Repeat cycle!

Now let’s take a look at what Intel has been doing with their microprocessor family around embedded applications such as storage. Specifically, if you read some of Intel’s product briefs on their microprocessors for embedded applications and you’re a storage vendor, you might think that hell has finally frozen over.

Intel has been implementing embedded application functionality into their Xeon processor line adding in a veritable alphabet soup of TLAs. Here are but a few of the capabilities:

• Internal support for RAID 0, 1, 5, and 10
• Integrated SAS and PCIe
• Support for AES, Hashing, Chunking and Compression
• Non-transparent bridging
• Various virtualization assists

There’s also the assertion from Intel that software RAID stacks with Intel microprocessor assists are on par with ASICs that support RAID offload from a standard microprocessor.

My response: Okay, this is nothing more than the sedimentary hypothesis in action, and eventually Intel’s Xeon SoC for embedded systems will solve some, but not all, storage problems. Furthermore, new whitespace problems will emerge in the storage market, and guess what? Intel won’t have that capability on or near their processor for a while — just like we see with math co-processors being sucked into the micro process and GPUs in a Phoenix-like way, rising from the math co-processor ashes. So from ASIC to microprocessor and back again!

Any ideas for what the white space could be? Drop me a line or comment here if you have any suggestions. Otherwise, tune in soon to read some ideas in a future post.

Here goes example number one: NAS virtualization.

You may recall past companies and products in this space. Those that come immediately to mind include Rainfinity, Acopia, and StorageX, with only Acopia ARX really existing at F5 as a standalone NAS virtualization product. All the others have either been acquired or have gone out of business (at least as far as I know). As there are no longer being highlighted via a standalone application or appliance begs the question: Is NAS virtualization a viable technology?

You bet, and you can see it in action within two Hitachi products, except not as separate appliances: Notably, you’ll find NAS virtualization in the Hitachi Data Ingestor (HDI) and the Hitachi NAS Platform (HNAS).

Our first incarnation was done in 2007/2008 by applying engineering talent from HDS to the then standalone BlueArc. (Here’s a shout out to Simon, Paul, and Phil…welcome back!) It showed up as a feature called eXternal Volume Link (XVL) and was controlled through a basic interface on the native element manager or through full content and indexing via Hitachi Data Discovery Suite (HDDS). XVL can talk to any NFSv3 server as well as using REST over HTTP to talk to Hitachi Content Platform (HCP). So what we did was to put NAS Virtualization as a feature into the storage infrastructure four years ago.

The second incarnation is within HDI and was first implemented as a connection to HCP using REST over HTTP. It is and was designed as a cloud on-ramp for remote locations to connect to stellar Hitachi Private cloud/object storage infrastructure. Most recently with the updated version of HDI we are now able to also virtualize via the CIFS protocol to consolidate existing NAS and Windows Filers into a Hitachi Private Cloud infrastructure. The setup of HDI for this purpose, just like XVL, is as an inline file system virtualizer which can take over shares from the target filers or file servers and allow users to smartly drain these older systems into the cloud.

In both instances you can see that in-band/inline NAS or file system virtualization is no longer a standalone product like F5 ARX or any of the other legacy technologies. In fact the NAS virtualization feature has transformed from a standalone application or appliance to features in the storage infrastructure. Digging a little deeper, two more key questions are: Why did we do this and why in this way?

Well to answer the first one, our customers asked us to. Here is a customer quote from 2006/2007. (Now, I will add that at the time this customer was the “poster child” for Acopia and since there is no statute of limitations on protecting customer names, I’ve removed the customer name from the quote.)

“Acopia was our only choice at the time, but if it was incorporated into a NAS product we’d throw out their [ARX] product in a second.”

Wow! This is still, and was back then, a very clear driver to do what we did. As to why we implemented XVL and HDI file system and NAS virtualization the way we did, that is pretty simple. When we looked at our existing portfolio we already had what was becoming a blockbuster success in the form of in-band block storage virtualization in the form of the original USP. This system had the data movement engine within the storage controller sporting a basic control point on the native element manager and an advanced control mechanism in an out-of-band controller called Tiered Storage Manager at the time. As a result we made the determination that to help our customers as they wanted to add NAS to their portfolio, we’d follow a similar approach with the hope of making adoption easier.

If this isn’t a data point screaming that the sedimentary hypothesis of technology is true then I don’t know what else is. However, this is only one data point and more are needed, and for that you’ll have to wait until the next post.

Organic sedimentary rocks are formed under varying degrees of pressure and temperature over long periods of time. More pressure and an increase in temperature will form different types of organic sedimentary rocks. When organic material is broken down it becomes peat. Peat is the first step in the organic sedimentary rock process. As more earth accumulates over the peat and causes the peat to come under greater pressure and a higher temperature, then lignite is formed, another type of organic sedimentary rock. After the lignite is formed it begins to undergo a similar process as the peat. More pressure is applied to the lignite and the temperature becomes hotter resulting in the formation of bituminous coal. Bituminous coal then becomes anthracite coal as its temperature and pressure increases. Coal is created under swampy conditions that are not commonly found in our era because it needs higher sea levels to help it form. (Source: eHow.com on Organic Sedimentary Rock)

Obviously what precedes the generation of organic sedimentary rock is a vibrant active ecosystem filled with fauna and flora—both of which can die initiating the process of rock formation. I see technology in much the same way; basically it goes like this:

1. Application – correlates to the vibrant and active ecosystem, but eventually every application or at least some parts of an application “die”, begetting.
2. Middleware/Feature-ware – matches the peat stage of organic sedimentary rock formation and occurs when what were once vibrant applications or several application components transform into a middleware stack or a set of capabilities within an existing middleware stack, and with time and market pressure produce.
3. OS-ware/Infrastructure-ware – is rather like ignite or bituminous coal happening when middleware and feature-ware end up as either features or components  in either the OS or within the infrastructure (e.g. Storage, network or compute), and finally with additional market innovations result in.

Microprocessors, ASICs, ASSPs or FPGAs – realize the equivalent of anthracite coal and are comprised of accelerators, full/partial offloads of capabilities into silicon or assembly-like instructions executing on FPGAs. (Note that complete implementations may never find their way into silicon; however when algorithms arrive on silicon often extreme performance boosts and power consumption reductions are major benefits.) This is the general “hypothesis” that I’ve been referring to, and I think there may even be more sub-cycles within each layer. For example, multimedia functions (e.g. graphics and audio) used to be merely a set of software running on a general purpose processors. Then, over time, the GPUs and other accelerators have arisen, taking moving a large part of this function onto silicon. Now, given even more time, there is a processor from the SoC model to further compress things like GPUs onto a single multi-type many-core processor produced by the likes of Intel or AMD. Another example is in the DBMS world where there are a plethora of open source alternatives to Oracle and NO-SQL systems whose core is available for free. I believe that this shows in the middleware layer there is healthy market pressure/competition resulting in a wide selection of offerings.

A conclusion, and an inappropriate one, is that because of the large number of DBMS technologies, especially with the focus on open source, this market is officially commoditizing.

I have a couple of other posts up my sleeve with some real world examples coming soon. Until then, what do you think? Am I on to something? Can we transform the hypothesis into a theory?

I would say that the next tool in the arsenal of any Big Data question is Search!

However, the big “S” Search that I’m talking about is before an analytic query across data residing in a data mart, Key Value Store, Columnar Data Store, or any other NO-SQL (not only-SQL) system. Since in the era of the big bang of Data the super majority of data is potentially exabytes in scale and structured, unstructured and semi-strucured in type, I argue that this pre-Search may indeed be the most important of all.

In his post, Philip Russom talks about this very point: an early step in the overall analytic process, he calls “Discovery Analytics,” which is prior to the institutionalization phase requiring formal ETL placing the data into a DWH or NO-SQL store. This is not dissimilar to early phases in eDiscovery, which include a kind of raw search across mounds of content. Results from this search are then passed to a case management tool for further refinement and analysis. This Discovery Analytic process, to use Philip’s term, identifies the insightful diamonds in the rough which can literally transform, refine, revolutionize, or save an enterprise. Without this phase we are left with no seed to initiate a longer term or deep and recurring analytic process—the kind that Mr. Russom dubs as being institutionalized.

My worry is that the industry is largely leaving behind Search or Discovery Analytics in the general discussions surrounding Big Data. Instead there appears to be fascination with NO-SQL data stores, feeding Hadoop, releasing your own version of a Hadoop, evolving BI tools to handle Big Data, etc. Perhaps this is due to the fact that Search is not trendy enough to warrant hype and excitement, but I suppose if we modify the name to “Discovery Analytics” things could change.

Rest assured that worrying about Search within the enterprise can yield real and tangible results beyond Big Data. In fact, at least Forrester states, as of 2009 information workers spend almost a half a day a week merely finding things inside of an enterprise. To me, this means if the enterprises and vendors who provide to the enterprise focus on Search as Discovery Analytics, we could improve the lives of everyday users and put in the rebar needed to pave the path towards managing Big Data.

Furthermore, I think that an added and positive consequence of focusing on search is the real potential to start the democratization of the Data Scientist. In my humble opinion this could not happen soon enough so that the role is prevented from being entrenched in an almost ivory tower-esque way throughout the industry.

Here’s to a Big-Data-verse for the people, of the people, and by the people.

• The Grand Design By: Stephen Hawking (@Prof_S_Hawking)
  • I’m a huge Stephen Hawking fan and have read (more than once) every book he has published—which will explain the next book pick.
• The Illustrated – A Brief History of Time & The Universe in a Nutshell (double book release) By: Stephen Hawking (@Prof_S_Hawking)
  • Having read the original versions of these books, this superbly illustrated release is packed with high quality, glossy pictures compared to the original books. This is more of a collector’s edition and of course, when reading a Hawking’s book, a quality picture is with worth a billion-billion (Carl Sagan reference) words. The best part of this book is I bought it at an “everything must go” blowout sale as the Borders in my neighborhood was shutting down.
• Holographic Data Storage – from Theory to Practical Systems By: Kevin Curtis; Lisa Dhar; Adrian Hill; William Wilson; Mark Ayres
  • Since I was researching some optical storage technologies for Hitachi, this book came highly recommended and from an interesting angle. Customers were asking about the Hitachi references within the book, so I bought it. It has been extremely helpful for me to understand this evolving area of technology which I believe will be game changing in the future.
• CUDA by Example – An introduction to General-Purpose GPU Programming By: Jason Sanders; Edward Kandrot (@ekandrot)
  • Another part of my research for hardware accelerated applications and their uses in enterprise applications.
• HTML5 – Step by Step By: Faithe Wempen M.A.
  • Mainly purchased this as an HTML5 reference book for an internal project I am working on.
• Adobe Dreamweaver CS5 with PHP: Training Source Code By: David Powers
  • Same project support as above.
• HTML5 – 24–Hour Trainer By: Joseph W. Lowery; Mark Fletcher
  • Again, same project support as the above.

Also, here are some miscellaneous books I picked up at a clearance table. If you’re like me, you can’t pass up one of those 70% off clearance deals to fortify your technical library. And since I do a lot of video and audio editing, I also needed these for some personal projects.

• Producing Great Sounds for Digital Video By: Jay Rose
• Audio/Video Protocol Handbook By: Jerry Whitaker
• Gigahertz and Terahertz – Technologies for Broadcast Communications By: Terry Edwards

What are your top book recommendations from 2011?

In the United States, Commercially available Off-The-Shelf (COTS) is a Federal Acquisition Regulation (FAR) term defining a non-developmental item (NDI) of supply that is both commercial and sold in substantial quantities in the commercial marketplace, and that can be procured or utilized under government contract in the same precise form as available to the general public. For example, technology related items, such as computer software, hardware systems or free software with commercial support, and construction materials qualify, but bulk cargo, such as agricultural or petroleum products, do not. (source: Commercial off-the-shelf – Wikipedia, the free encyclopedia)

My colleague Ken Wood talks about commodity in a post several months ago, Soybean Is A Commodity, where he muses on what is and is not a commodity. His summary is that basically the output and the resulting measures of many of the devices and systems that are produced in the ICT field are a commodity. However, the systems and devices themselves aren’t.

He says, “It is my opinion that there is a misunderstanding and confusion in the IT industry between ‘commodity goods’ and ‘consumer products’ when it comes to technology. I can’t pinpoint the exact origin of why or how these two concepts seem to have become synonyms for each other, especially in the IT industry, but there is a difference between commodity goods and consumer products.”

Personally, I think that this stems from confusion about COTS and commodity, and may have crept into the ICT vocabulary just like “NIC Card” and “Transparent to the Application,” see my previous post where I ranted on the topic of language misuse in ICT. While occasional misuse is relatively harmless, I believe that the misapplication of commodity has resulted in inappropriate thinking about the costs of technology. Let’s explore this last point for a bit.

Let’s assume that hard disk drives and the resulting capacity were a commodity, if so they are a strange beast I would have to say. In fact they may even represent a kind of unique commodity which follow the law of supply and demand, but have planned cost erosion. The cost erosion is associated to predictions by industry patterns like Moore’s law, and is somewhere between 20%-30% per year. Imagine that—a commodity with a predictable annual per unit price decline. I’m sure that the mathematical wizards on Wall Street would love to have something with the level of predictability experienced in both consumer and enterprise capacity production/purchases. Sure tragedies like those in Thailand and restrictions of rare Earth metals can cause disruptions in supply that has the potential to increase costs if demand is not damped or constrained, but through the innovative human potential and given enough time even these unfortunate events have little impact. So is storage capacity a commodity? I would say that unless the definition of a commodity has changed, both the capacity measures and the actual devices aren’t a commodity. They are rather COTS devices with commodity properties.

With that in mind, what about CPUs, memories, and more importantly advanced systems that aggregate and combine COTS in unique ways to release innovation? In my opinion the clear answer is no. Storage, servers, networking, etc. are not commodity, but surely can be COTS. Obvious questions are: Why is this important? Why does it matter? I see this is important because of the potential for COTS to contain innovations unique to a particular technology supplier. This matters to any consumer of ICT because these innovations may actually be a better match to your business, and potentially even the entire market as a whole.

This last statement “entire market as a whole” is interesting because I see that the fundamental tectonic plates of the technology industry are shifting to favor more OPEX-friendly technologies. It also means that as a consumer, you may be willing to pay more for innovation in the short term especially if the technology delivers innovation you can leverage and amplify, or it reduces your OPEX such that you can reinvest money elsewhere. So where do we see this occurring in the industry now? Well, I would say that the trend to deliver complete IT stacks as I’ve discussed in The Rack is the New Server, the Data Center is the New Rack, is an example where CAPEX may be a bit higher but the potential for savings on the backend through staff reallocation, reduced maintenance costs, and assured configurations may make the slightly higher CAPEX worth it.

So the next time you hear an IT professional say something like, “That’s just a commodity technology…” stop them and correct the usage of commodity with COTS. By keeping this terminology misuse in the ICT industry it serves to devalue innovations that vendors add, users can take advantage of, and that creative companies can leverage to engender new innovation on top. I personally fear that without a grass roots effort to make a change here as an industry we are going to be increasingly satisfied with mediocre offerings and products.

I’d rather see more shoot for the moon and focus on “the insanely great” instead of settling for the mediocre.