Data Center Transformation Part 5: Leveraging Dynamic Provisioning with storage virtualization
by Hu Yoshida on Aug 9, 2010
After my last post on Data Center Transformation Part 4: Dynamic Provisioning, where I talked about the benefits of Hitachi Dynamic Provisioning, Lucas Mearian from Computerworld wrote an article “A waste of Space: Bulk of drive capacity still underutilized – Most companies can reclaim as much as 60% of their storage Capacity with Monitoring and thin provisioning tool”
In this article, he observed that the low utilization of storage capacity is still a rampant problem even though tools like thin provisioning are available today to monitor usage and thin provision storage. While a recent survey from TheInfoPro showed as many as 50% of Fortune 1000 companies were using thin provisioning or were planning to do so, other analysts like Forrester’s Andrew Reichman contend that companies are not using these tools and utilization of storage is still between 20 to 40 percent. While he did say that thin provisioning was not being used for thin provisioning he did observe that it was being used for striping performance which is a result of writing data in chunks across many disk drives.
According to this article thin provisioning can reclaim up to 60 percent. So why aren’t companies jumping at this especially in the current economic environment? The article goes on to speculate on some of the reasons for this lack of adoption of thin provisioning. Hitachi Data Systems is having a great deal of success with our Dynamic Provisioning which includes the ability to thin provision storage. In this post I will look at some of the inhibitors to the adoption of thin provisioning and how the Hitachi USP V/VM can leverage Dynamic Provisioning with storage virtualization to solve thee problems.
It takes time to acquire more storage
Data Centers need to keep their businesses running and no one wants to be caught short because of a lack of disk capacity. Since these disk frames are getting larger with more applications per frame, they need a longer lead time to add or upgrade storage. A lead time buffer is required to satisfy their growth requirements while they go through the next acquisition and device migration cycle which may take 6 to 9 months. Since storage capacity is cheap and getting cheaper it seems easier to just buy more capacity today than wait until you need it. Unfortunately, all this extra unused capacity means higher total cost since the operational costs of that capacity will far exceed the acquisition cost.
Hitachi recognizes the need to have a lead time buffer. However, unlike other thin provisioning solutions, the Hitachi USP V/VM can leverage Dynamic Provisioning with storage virtualization to minimize this lead time. The USP V/VM can minimize that buffer by pooling different silos of storage capacity into a virtualized pool of capacity. If you had 3 separate 100TB frames at 20% utilization, each frame would have 80TB of buffer capacity which cannot be shared with other frames. If we virtualize the three frames behind a USP V, we could consolidate the excess capacity into a shared pool of 240TB. Then, may be, we can reduce the total buffer pool to 80 TB to be shared across the three frames. If you need more capacity and you need it quickly, you can plug in another bank of drives or roll in another frame and attach it behind the USP V where it can join the Dynamic Provisioning pool without disruption to the application. With Storage virtualization you can add storage quickly when you need it and take advantage of the price erosion of storage capacity.
The risks of over commitment
While this technology is called thin provisioning, provisioning less than what the allocation calls for, it can also be used to over provision or over commit your usable capacity which could cause problems under peak loading conditions. By over commitment, we commit to some number of users that we will support their full allocation request. Let’s say there are 10 users who asked for 5TB apiece. So we commit a total of 50TB, when we only have 30TB of physical capacity, since we assume that everyone is over stating their allocation requirements. Thin provisioning software does have thresholds to warn and or limit new provisioning requests when the thresholds are reached. But there is the danger that a spike in demand will hit these limits. Again, Hitachi’s combination of storage virtualization and dynamic provisioning can quickly add additional resources to the dynamic provisioning pool. Virtualization can also non disruptively migrate lower priority volumes out of the pool temporarily if a higher priority volume needs more capacity.
Some of our Dynamic Provisioning customers choose not to overcommit the physical storage in the pool because of the risk of running out of storage during peak periods. Also some file systems and data bases are not thin friendly. They may write formatting information across the initial allocation which negates the potential for thin provisioning. Some file systems start out thin friendly but get bloated very quickly as files are deleted and created. The storage system has no way of knowing if the file system has freed up some space unless the file system notifies the storage through an API or special SCSI command.
If a user asks for an allocation of 10TB, 10TB of physical capacity is reserved in the pool even if only a fraction of the capacity is actually used. While there are no savings on capacity, these customers use Dynamic Provisioning for its many other benefits. The first benefit is the ability to dynamically provision new storage in minutes. When Oracle ASM wants to expand a data base, Dynamic Provisioning can provision that in minutes. No need to create RAID groups, carve out LUNs, format, concatenate, etc, which could take hours. The second major benefit is the performance improvement that comes from wide striping the data across the width of the pool. Just by going from 8 spindles to 32 spindles, tests have shown a 700% improvement in throughput for a random workload. As Andrew Reichman observed, the customers he contacted were using thin provisioning for srtiping performance and not for recovering unused capacity. The third benefit is that even though we reserve enough capacity for the total allocation, the provisioning is still done a page at a time, so any moves, copies, replications, migrations only work on the provisioned pages which can greatly reduce the time and operational costs for these activities. However, that being said, a number of our customers like Qualcomm are using the thin provisioning feature of Dynamic Provisioning.
While some thin provisioning vendors claim that you can drive utilization up to 80 or 90 percent. I personally do not recommend that if your application is critical and your business is dynamic. You need a contingency buffer for peak demand. I think 60 to 65% is a conservative objective.
The requirement to replace existing infrastructure
Gartner is referenced in this article as noting another reason for not embracing thin provisioning is the requirement to replace existing infrastructure which is difficult to justify in a recession. Adam Couture of Gartner is quoted as saying:
“if your array wasn’t built to take advantage of thin provisioning, there’s no way you can retrofit it.”
He is not aware of Hitachi Dynamic Provisioning and storage virtualization in the USP V/VM. Hitachi Dynamic Provisioning does not require replacing existing infrastructure as long as it can be virtualized behind a USP V/VM. Once it is virtualized behind a USP V/VM, the LUNs on external storage can be formatted into a dynamic provisioning pool, and from there it can have all the capability of Dynamic Provisioning. Once the storage is connected behind the USP V/VM, we can move the “fat” external volume into a dynamic provisioning pool creating a page at a time. When we are finished we check the pages for zero formats and return those pages back to the pool as empty pages, leaving the volume in a thin state. One of our large customers reclaimed 40% of existing capacity just by virtualizing behind a USP V and moving the volumes into a dynamic provisioning pool.
Thin provisioning storage can not scale
Another inhibitor that was called out was the fact that many thin provisioning systems did not scale and required costly rip and replace. Some vendors required a large system to install before you can get thin provisioning and other vendors started small but remained small which required disruptive replacement as requirements grew. Hitachi can start with a USP VM which is a modular USP. If there is existing capacity already available, the USP VM can be installed without any internal disks. Another approach is to start with an entry model of the USP V. A third option is to use the ASM 2000 for Dynamic Provisioning within the AMS 2000. It does not virtualize external storage but can scale up to 2048 host connections and 4096 LUNs in a modular architecture.
Unlike other thin provisioning system that only do thin provisioning; Hitachi’s Dynamic Provisioning can be combined with the Storage virtualization of the USP V/VM and leverage all the other capabilities of an enterprise controller like migration, tiering, and distance replication. This eliminates the inhibitors that were mentioned above. Considering that Dynamic Provisioning is a relatively new product, it’s adoption rate is impressively high and climbing. The ability to reclaim unused capacity simply by attaching existing capacity to a USP V/VM and moving the volume into a dynamic provisioning pool can provide immediate payback on your investment. There are many immediate benefits to implementing Dynamic Provisioning and storage virtualization, and it does not require you to rip out your existing storage investments. Thin provisioning is only one of the benefits of Dynamic Provisioning. Not all files are thin provision friendly, but all files can benefit from dynamic provisioning in a matter of minutes from a pool of preformatted pages and automatic wide striping for high performance.
[...] I pointed out the benefits of combining dynamic provisioning with storage virtualization in part 5 and explained how we use this as our base for our “one platform all data” approach to storage [...]