Controllers: Internal versus External Switching
by Hu Yoshida on Sep 15, 2009
In 2000, Hitachi introduced the Lightning 9900 storage system. This was the first enterprise storage system with an internal switch which enabled the switching of paths between front end port processors, cache modules, and back end disk processors. Prior to the introduction of the switch, the front and back end processors had to contend with each other over a pair of shared buses for access to the cache. This ability to switch paths around hot spot and failures and eliminate path contention to the cache, enabled the Lightning 9900 to scale I/O processing beyond any other storage unit available at that time. The Lightning 9900 could replace 6 to 8 competitive storage frames and still provide better performance at a lower price. Hitachi has continued to evolve this architecture with virtualization of external storage and dynamic provisioning which virtualizes capacity and creates a virtual pool of storage across internal and external storage systems.
Nine years later we are seeing other storage vendors introduce switching with their storage systems. However, the switching that they are introducing is external switching, where the switch is used to cluster storage controllers and not components internal to a storage controller as in the USP V. A cluster of storage controllers does not enable the components of a storage controller to work with components from other controllers. External switching does not allow you to scale out by adding the controller components that you need or balance the workload across components in the controller. Instead of enabling the controller to balance the workload internally, external switching requires additional management for workload distribution and resource balancing across controllers.
These two diagrams shows the difference between internal switching as we have in the USP V on the left, and external switching as EMC has in the VMAX on the right.
On the USP V, workload can come in over multiple front end (FE) port processors, and access a common cache image of its data over any available switch path, and then be downloaded over multiple backend processors over other switch paths. If additional cache is needed, it simply acquires it. If more storage capacity is needed, additional capacity is provisioned automatically from a Dynamic Provisioning pool of virtual capacity that can span internal and external storage arrays.
I am certainly not an expert on VMAX, so I would appreciate any clarification that any body can provide. It seems to me that with VMAX, each VMAX engine acts like a single dual processor, storage controller with separate cache modules. Each processor in a VMAX engine must process the front end ports, manage the cache, and process the back end ports to the disk. The switch in VMAX may allow a processor to connect to the cache in another engine, but overhead would be added to keep track of which processor is working with which cache table. Since the processor in the VMAX engine handles both the front end ports and the backend storage ports, if data is staged in the cache of another engine it has to be downloaded back to the original VMAX engine. If you want to add storage capacity that exceeds the capacity of one VMAX engine, you will need to add a whole VMAX engine, even if you do not need the additional two processors and additional cache modules. Performance should scale as you add VMAX engines as long as you distribute the workload evenly across the multiple engines. Multiple VMAX engines can not work on the same workload unless the workload is distributed by the application. This means additional operational costs to balance the distribution of workload, provisioning, masking, and zoning.
Another concern with external switching is availability. With internal switching I can lose one or more front end processors, cache module, or back end processor, and the USP V will continue to run with no data loss or noticeable performance degradation. All the processors in the USP V can access the same global cache image so we can recover over an alternate path through another processor. With VMAX, if you lose a processor in a VMAX engine you lose half your processing power and half your cache in that engine. Although there are two processors per engine, as soon as you lose one processor you are exposed to data loss if the other processor fails so it would be prudent to stop and fix the failed processor before continuing to process data. That means an outage. Since each VMAX engine has its own cache, an alternate path through another engine would require that engine’s cache to have a mirror copy of the primary engine’s cache. Since the caches are owned by different engines, I would guess that the alternate path would be an active/passive standby path and not an active/active path as we have in the USP V with it’s one global cache. That is a lot of overhead for the switch and eats into the total available cache in the system.
Clustering is not bad. In fact HDS can cluster two USP V’s together for high availability and for non-disruptive migration to the next generation of USP V when that becomes available. While each USP V is highly available with no single point of failure, there is always the chance that a sprinkler goes off or the site is damaged in some way that the USP V is not accessible. This clustering can be done across Fibre Channel distances so if one site is damaged a USP V in another site could take over without disruption to the applications. By contrast the VMAX engines all sit in one frame connected within short copper distance to a Rapid I/O switch. If a sprinkler goes off, all these engines are lost.
As I said before, I am not an expert on VMAX architecture, so I apologize for any misunderstandings on my part and would appreciate any clarification that anyone could share without violating any agreements with their vendor.
[...] have written many posts on scaling and the unique value we have brought to the market, as has my colleague Hu. Many of my discussions have centered on our application of differentiation in a variety of [...]