Return to year 2000 and consolidation
by Hu Yoshida on Sep 9, 2006
According to the latest projections from IDC, we will reach a significant milestone this year. That milestone would be the recovery of the external disk storage industry to the revenue level of CY2000, when we went over $18B USD. In CY2000 the industry shipped about 200PB of external disk storage. This year the storage industry will ship about ten times that much capacity and will finally attain the same revenue level we had then. Welcome to the new economics.
At the rate that customers are installing storage today, it may be hard to realize that we have been in a down turn since the peak in CY2000. Our memories are short. Prior to CY2000 we had the Y2K and Dot Com booms. Y2K came and went and the Dot Coms went bust in 2001. IT shops started to consolidate and work off the over buy of the previous years. SAN’s got their start about then because they were seen as a way to consolidate the many islands of storage into shared storage pools. The storage industry spent millions of dollars on SAN infrastructure which did not include a byte of new storage.
In 2000 HDS introduced the Lightning Storage Array with a switched architecture which enabled users to consolidate 6 to 8 bus based storage arrays onto one Lightning frame with no loss of connectivity or bandwidth. Although consolidation involved a rip and replace, the average frame size was about 2 to 3TB, so the consolidation to 12 to 18TB frames was manageable. HDS saw its market share percentage increase from the low teens to the high 30′s and we were able to post double digit revenue growth during this 5 year downturn in the storage market.
In the last few years, an improving economy, the increasing need for data protection, and regulations like SOX have been driving increasing demands for data storage. However, since IT budgets have generally remained flat, many IT shops spent their money on buying more storage instead of hiring people to manage the storage they already had. As a result the growth rates for external disk storage have increased to about 60% over the last two years, while utilization of storage has been dropping to about 30%. Analysts like Jon Toigo have been warning that storage is now over subscribed and under utilized for some time. The feed back I have been getting from customer visits confirms this trend. The CIO of a large financial company with over 2PB of storage said that his storage was only 20% utilized, and over 70% of it was expensive tier 1 storage.
He summed up his pain in four statements:
He is buying too much storage
He is paying too much for that storage
His storage is under utilized
He is running out of room and power in his data centers.
With this over subscription of storage, and with looming economic uncertainties, increases in energy prices, rising interest rates, wars, global warming, housing market bust; we are setting our selves up for another phase of consolidation. Most analysts are projecting a straight line 60% CAGR through the rest of this decade. I am beginning to have some doubts. If we believe that storage is under utilized at 30%, than it is time to consolidate, drive that utilization up, and get CAGR down to more reasonable growth rates.
If we go through another major phase of consolidation, we need to take some precautions. One thing that would help is storage virtualization. In 2001 and 2002 when we did consolidation, it was a rip and replace, which was very disruptive. But that was manageable since storage frames in 2000 only had about 2 or 3 TB and a handful of applications that had to be scheduled for consolidation. Today storage frames have 20 to 30 TB and many more applications and users per frame. Also given the real time nature of business today, consolidation will be orders of magnitude more disruptive.
The only way to do consolidation today is through the use of virtualization to migrate and consolidate frames with minimal or no disruption to the application and their users. Storage virtualization can add another dimension to consolidation through the non disruptive movement of data to different cost tiers of storage. Not all the copies of a tier 1 application’s data need to reside on tier 1 storage, and it doesn’t need to remain on tier 1 storage if the access rate or availability requirements of that data decrease over time.
Storage virtualization and the ability to move data non-disruptively to the appropriate cost tier of storage will enable IT shops to bring the growth of storage under control. Storage vendors can continue to be successful in terms of revenue growth even if capacity growth rates decline. The new economics is not about selling storage capacity but in selling storage solutions which enable better utilization of what we already have. An example would be a storage control unit with no internal storage capacity, that can attach existing storage units and virtualize them into a single storage pool behind the latest functionality of a tier 1 control unit. Just like SAN’s were key to the last consolidation phase, storage virtualization will be key to success in the next consolidation phase.
Comments (2 )
This is a great analysis that I agree with, how do you propose software vendors are dealt with though. For data sources that do not lend themselves to integrating with virtualized storage, customers are forced into more complex storage migrations and management. Vendors like Microsoft need to be convinced that their applications need to support and perform on virtualized or tiered arrays, as without this joint support most customers are rightly frustrated with storage vendors for putting them in this situation.
All of this makes sense and tracks with what we have seen since 2001. You mention the issue of utilization, that’s pretty obvious. One of the greater issues, probably only for larger shops, is the issue where internal storage clients not following the evolution of HBAs. This has forced us to continually demand support for old (circa 1999, 2000 and beyond) HBA technology from our vendors. Of course, this has been a good litmus test to see who our partners are and identify the vendors.
On the subject of virtualization, yes there are some wonders to be found in virtualization but we still face the issue of the many volume managers that exist on each OS. We clear the issue of lease expirations and right typing, but not growing and shrinking of storage as appropriate (yes, we actually shrink allocations).