What’s wrong with this picture?
by Hu Yoshida on Feb 20, 2007
After spending the week visiting customers in Belgium and Luxemburg with Dany DeBudt, I went to one of my favorite places, in Brussels, the Grand Place, to do some shopping and have dinner. In one of our conversations, Dany suggested that the next time I visited the Grand Place; I should stand back and take a careful look at the famous Town Hall which towers over the plaza. On Saturday afternoon, as I sat on the steps of the museum and gazed across the plaza at the Town Hall I began to understand what Dany was referring to. Previously I had been so awed by the baroque beauty of the steeple, the statues, and the arches and windows of the Town Hall that I never noticed what I now saw. As I stood back, I began to notice that the main door was not centered with the steeple, the windows and arches on the right side were different than those on the other side, and the building on the right side appeared to end abruptly as though the builders just ran out of time.Whenever I see architecture, it triggers my thoughts about storage, in particular, storage virtualization architecture.Shortly after the successful introduction of SANs in the 1990’s, storage architects were all excited about going to the next level and developing storage virtualization in the SAN network. Startup companies and venture capitalist that had made billions of dollars on the SAN market were jumping at the opportunity to create another new market. The early picture of network based storage virtualization was very appealing. A virtualization engine sitting in the SAN could control where all the data in a SAN was placed and use meta data or mapping tables to keep track of the application view of data across the entire pool of SAN storage. This would eliminate the need for intelligence in a storage controller or intelligent host software to manage storage, since all the storage services and management functions could run in the network. Disk arrays could be commoditized, fewer storage administrators would be needed, and storage could be optimized to 90% utilization.
The community of industry and financial analysts also bought into storage virtualization in the SAN and fueled discussions around in-band versus out of band approaches to virtualization. There were a number of papers published in the early 2000s when this concept got a boost from the intelligent switch vendors who saw an opportunity to extend their market through virtualization. But this market never really took off. Seven years later, many of the startups are out of business or bought out by bigger companies only to quietly disappear. A few managed to survive by changing their business focus to data migration, or virtual tape, or file serving.
Why did storage virtualization in the SAN not survive the initial burst of enthusiasm? IT users took a closer look at SAN based storage virtualization and did not buy into it. Mainly because it could not deliver all the storage services and management functions that were originally promised. Yes they could provide a limited ability to pool storage across heterogeneous storage arrays, but they could not provide enterprise class storage services like remote replication of consistency groups, availability and scalability for thousands of host connections, and non disruptive movement of large amounts of data across tiers of storage, etc. In order to provide an enterprise level of storage services the virtualization engine had to have the resources of an enterprise storage controller, 128 processors sharing a large, dynamic, global cache, thousands of host connections, logical partitioning, switch architecture, and non-disruptive data mobility functions.
Virtualization in the storage controller is now available and accepted in the market, serving enterprise applications in leading Fortune 500 companies. Storage virtualization in the SAN is still being talked about.
One vendor claims to have over 2000 of their SAN virtualization appliance installed over the course of 5 years. While they do storage pooling, their lack of storage services has limited their use with enterprise applications.
Much like the architecture of the Town Hall in the Grand Place, storage virtualization in the SAN, sounds very appealing until you stand back and take a closer look. Storage virtualization needs to be more than the pooling of storage; it must also provide enterprise class storage services.
Can’t agree more. We have evaluated HDS USP, EMC Invista, IBM SVC and NetApp V-series for block level storage virtualization. We choose USP, becuase only USP has the enterprise capabilities that don’t force vendor lock-in. We have 200 TB of EMC CX, AMS500, and EMC DMX storage behind the USP and it has worked quite well for the last one year. In the pre-virtualization days it was a nightmare to provison heterogeneous storage with all the multi-pathing software incompabilities and driver requirements. Not any more. Storage IO performance issues ? Use TSM to migrate LUNs seamlessly to 15 K RPM drives and we are done !
We have built a very effective Tiered storage environmnet that has saved us millions of dollars. Now, we don’t do the time consuming study of IO profile of application and try to do capacity planning etc. By default every server will go to Tier 2 storage behind USP and if the DBA’s don’t like the IO performance we move them to Tier 1 storage in few hours. Since we act as internal storage service provider to business units, our customers really like this model. Previously every storage provision was done to Tier1 storage because of the FUD about performance and nightmare of data migration from Tier2 to Tier1.