Is the future of storage scale up or scale out?
by Hu Yoshida on Nov 9, 2009
What do we mean by scale up versus scale out? If you go to Wikipedia you will find a section under the definition for scalability for “Scale vertically vs. horizontally”.
Here is what it says:
It then goes on to describe the tradeoffs as follows:
In the server world the direction is clearly towards scale up with future Intel Nehalem processors and virtual servers/hypervisors running hundreds of instances on a single node. This will put an increasing load on the network and network vendors like CISCO and Brocade are also scaling up to 8Gps FC and 10Gps Ethernet. It stands to reason that the storage systems will also need to scale up to support the increasing workloads of scale up servers and networks.
Hitachi Data Systems’ USP V is a storage system that can scale up to 128 processors which are tightly coupled around a large 512 GB data cache and 28 GB of separate control memory. It can support 224 physical FC ports, each of which can support 1024 virtual ports. Through virtualization of external storage it can scale to 247 PB of internal and external storage and support in-system copies, dynamic tiering, distance replication, and dynamic thin provisioning. It can support over 4 million aggregate IOPs.
The rest of the storage market appears to be going for a scale out approach, with a clustering of modular, two controller, storage systems that are limited in connectivity, cache and capacity.
For example, EMC has taken the scale out approach with their VMax storage. While they can loosely cluster 16 VMax engines together, each VMax engine has only two processors (directors) each with 64 GB of cache and 8 FC ports. Currently the maximum raw capacity with 1TB drives is 360 TB per engine. Since EMC does not provide any performance numbers, it is difficult to make comparisons on performance. That is the maximum scale up capability of a VMax cluster.
As indicated in the preceding Wikipedia comment on the tradesoffs between scale up and scale out, the scale out approach to storage may be cheaper for applications that fit that paradigm, but with the virtualization of servers and the consolidation of networks a scale up storage system that can virtualize and aggregate storage services will almost always be less expensive that a cluster of modular, dual processor storage systems.
In the recent EMC, CISCO, VMware coalition announcement they announced that they will have proprietary bundles for 300 to 800, 800 to 3000, and 3000 to 6000 virtual machines. While CISCO and VMware may be able to scale up to those numbers of virtual machines, it will be a challenge to see how VMax meets the I/O demands of these virtual machines with a scale out design.
Comments (9 )
Hu, surely the problem is that while you *could* ADDRESS 247PB of storage, the reality of attaching such a huge amount of capacity under a single entity (not only from a reliability point of view) is that you couldn’t possibly actually use it.
128 processors, spread over 247,000 TB of storage is about 1 processor for 2,000 TB – even at 1TB SATA drives, giving 100 iops, that would need each processor the ability to sustain 200,000 iops.
Now correct me if I’m wrong, but isn’t 200,000 iops what SPC-1 you got for the TOTAL 128 processors, not just 1…
Scale out is definitely more, well, scalable, when you can add as many processing units as you need. Otherwise you end up with monolithic islands and many more objects to manage.
While you claim its more complex to scale out – its more complex to implement from a storage vendors point of view, but at the end of the day, having a single cluster to monitor and manage is much easier than many individual monoliths.
why should you not have both? If you don’t stay monolithic, you can scale out while also scaling the single entities or nodes up! Development of commodity hardware components proceeds continuously. In regular cycles you would release a new hardware model with better performance and extended features compared with the models before and give your customers the ability to integrate these new models seamlessly into their existing clusters or – by sticking more with the scale-up approach – replace older entities without any interruption. You can have both, scale up and scale out!
Disclaimer: All products are fine, don’t let my post influence your buying decision!
I tend to agree with Sebastian. Why can’t we have both. It’s clear from the computing wars that scale out beat out SMP and folks like Sun/HP/IBM are still suffering from the onslaught of cheap commodity servers powered by Linux overrunning their once profitable power server groups.
How much bandwidth and IOPS do you think a monolithic controller frame can handle eventually. Isn’t it more practical and economical to do the hybrid approach where you scale up as scale out as required and dictated by your application workloads.
We’ve already seen monolithic SAN arrays being crushed under the loads of virtual machines which essentially show up as random IO workloads.
Interesting case points is Google. How can they serve millions of users cost effectively? Are they scaling up or out. Maybe both!
Barry, the 247 PB is a maximum adressable capacity. Would any one put that much capacity behind a USP V? Propably not, unless they were storing content archive data that is not very active. Remember, this is a storage virtualization system that can move data between tiers of storage without disruption to the application. Active data can be migrated to the internal storage on the USP V while inactive storage can be migrated to external lower cost storage. If the external storage is our AMS 2000 modular storage, it can also be powered down during times of inactivity.
You can start with 32 processors and 48 FC ports and scale up to 128 processors with 224 FC ports, and by the way each FC port can be virtualized to 1024 virtual ports with their own address space. When we add these processors and FC ports they can be assigned to separate servers or they can be assigned to the same server if that server can support that many connections. Since these servers access data through the same global cache, there is no additional management required to distribute the workload across servers and or manage locking across the cluster of servers each with their own individual cache.
How many nodes and FC ports can you scale out on an SVC?
In response to Sebastian and Cameron, this is a solution that can scale up as well as scale out. Scale up is required to support the crushing workloads of virtual servers. Remember all those virtual volumes are mapped to one VMFS through the physical ports on the VMware server. Random workloads are best handled by Flash or a large pool of dynamically (thin) provisioned storage where pages are striped across the width of a pool composed of many disk drives and accessed through a very large 512 GB Cache. Scaling out across a cluster of dual controller storage arrays with limited, non-shared, cache and a limited number of disks on a shared arbitrated loop would be more restricted even if you could disribute the workload across the cluster.
If you want to know how google processes 20 PB a day. This is hosted on large clusters, equivalent to 2000+ nodes. The data is stored on each of these machines using a distributed file system. One example is Hadoop’s HDFS. The data is replicated so that no single failure causes unavailability or data loss.The data can be queried using parellel data processing models like MapReduce, Hive, Pig etc. These software frameworks help people with some experience on java or python programming, write code for queries.
Hu, I’m surprised to see you say that the USP V “scales up.” I suppose if you look at it a black box, it would seem that way. However, by adding CHAs you are effectively adding more “front end” processing nodes, which is a scale out approach. One could even make the argument that the USP V’s architecture is grid based, with each CHA being a gateway node into the grid.
It all comes down to how the mind defines and uses each term. So there’s no sense in getting lost in the terms “scale out” vs “scale up.” Bottom line to me is that the USP V gives you true symmetric multipathing and unprecedented scalability.
[...] a storage system that could scale up as well as scale out as I noted in my previous post on Scale up or Scale out. Their next question was about the need for cache when I/O loads are random, since random I/Os are [...]
[...] data. The internal switch provided the ability to scale up to meet increasing server demands. (see my previous post on scale up versus scale out) Besides scaling up it enables the storage system to dynamically scale out by partitioning the [...]
[...] Ken Wood and Michael Hay have been using the term Cartesian scaling to describe what I call Scale up and Scale out. Cartesian scaling is the ability to scale in two directions, up and out. I have resisted using [...]