Hu's Blog

In my previous post on Redefining Unified Storage – Hardware Differences Part 1, I described how the active/active controllers in Hitachi Unified Storage (HUS) set a new standard in dual controller storage systems by enabling enterprise class load balancing for performance, QoS, and availability.  In this post I will describe the main difference in the hardware architecture of the HUS file configurations, which sets it apart from other Unified Storage systems and sets a new standard for file processing.

Hybrid Core Architecture

HUS file modules have a Hybrid Core Architecture, which sets it apart from all other Unified Storage systems. HUS File Modules use a combination of multicore processors and FPGAs (Field Programmable Gate Arrays) to optimize and separate data movement and management processes that would normally contend for system resources. This is very similar to the hybrid architecture of VSP where we use a pool of multi-core Intel Processors for management functions and customized ASICs for the front and backend data movers. The File Modules in HUS have the same architecture as our HNAS products, which Hitachi Data Systems acquired through the purchase of BlueArc last year. The HUS FM-M1 is equivalent to the HNAS 3080 and the HUS FM-M2 is equivalent to the HNAS 3090.  The figure below shows the results of this hybrid architecture in terms of scalability in IOPs and capacity, which far surpasses anything else on the market.

2-Node 40K1 IOPS per Node 4PB Usable Capacity

4-Node 95K1 IOPS per Node 8PB Usable Capacity

Differences between FPGAs and CPUs

The following chart shows the difference between FPGAs and CPUs. FPGAs are designed to deliver massively parallel processing, which allows data to move through them at extremely high speeds. Multi-core CPUs are limited to a single channel per core (two if hyper threaded) and process data in a serial fashion, using arbitration to share the I/O bus among various processes. However, CPUs are much more flexible in the variety and complexity of tasks that they can accomplish. There are four FPGAs that are used in the HUS file module.

¹Using SPECsfs_2008 NFS benchmark tests (www.spec.org).

Multi-Core CPUs

HUS file modules take full advantage of the innovations in multi-core processors. High-speed data movement is a highly repeatable task that can be best executed in FPGAs, but higher level functions such as protocol session handling, packet decoding, and error/exception handling need a flexible processor to handle these computations quickly and efficiently. These two processing elements are integrated seamlessly within the operating and file system structure of the HUS file module. One core of the CPU is dedicated to working directly with the FPGAS, while the remaining cores are dedicated to systems management.

Field Programmable Gate Arrays

Where traditional computers require arbitration between processes, an FPGA architecture with integrated file system (SiliconFS) allows data to be transferred between logical blocks in a parallel fashion, ensuring no conflicts or bottlenecks in the data path.  This is different from conventional file systems where all I/O must navigate through shared busses and shared memory, which can cause significant performance degradation. When conventional file systems get saturated, performance drops dramatically due to retries. With the use of FPGAs in this hybrid architecture, performance remains at a constant level at the point of saturation.

In the HUS file modules there are four FPGAs, which operate in parallel. The following is a description of specific FPGAs used.

Network Interface FPGA

This FPGA is responsible for all Ethernet I/O functions through the physical, data link, network, transport, and session layers of the Open Systems Interconnection (OSI) basic Reference model. It handles standard Ethernet, and jumbo Ethernet frames up to 9000 bytes, ARP, IP protocol and routing, and TCP and UDP protocols. It also provides contention free TCP offload through the use of a dedicated fast path connection to the server. Each of the LAN ports connect directly to this FPGA through a high speed data path providing full duplex, dedicated bandwidth to each port.

Data Movement FPGA

The Data Movement FPGA is responsible for all data and control traffic routing throughout the server. It interfaces with all the major processing elements within the server as well as connecting to companion servers within a cluster. It controls all data movement and sits directly on the fast path pipeline between the Network Interface FPGA and the Disk Interface FPGA. Two 10 GbE cluster interconnect ports are controlled by this FPGA to provide redundant paths to all servers in the cluster.  This FPGA ensures that copies of all data requests are stored in NVRAM so that an acknowledgement can be sent immediately instead of waiting until the data is stored further down the pipeline in sector cache or on disk. NVRAM ensures that there is a valid copy of data to restore the file system to a consistent state if a reboot is required.

Disk Interface FPGA

Disk Interface FPGA is responsible for connectivity to physical disk or SSD on the backend. It can connect directly to HUS FC ports through a high-speed data path providing full duplex, dedicated bandwidth to each port. These ports provide redundant high-speed connectivity to the active-active controllers of HUS.  The DI FPGA logically organizes RAID storage into a virtualized pool of storage, striping data across any number of drives for performance. The DI also controls the Sector Cache memory, a large block of memory that stores data writes which are waiting to be committed to disk. Once data is stored in the Sector Cache it is immediately available for reads. After the data is written to disk a copy remains in the Sector Cache for subsequent read requests until the space is needed for incoming data.

SiliconFS FPGA

This FPGA is responsible for the Object-based file system structure, metadata management, and for executing advanced functions such as data management and data protection. By moving these functions into FPGA, HUS achieves an unprecedented amount of parallelism at the file system level that provides consistently high performance and stretch beyond the typical file system size and scalability limitations. This object-based file system will be discussed further in a subsequent post.

Summary

The packaging of HUS file modules with HUS block storage provides a significant reduction in cost and simplification of configuration and management over using a separate HNAS gateway with modular block storage.  The hybrid approach, using multicore CPUs for management and FPGAs for data flow, provides unparalleled scalability in performance and capacity for HUS. In my next post I will talk about the uniqueness of the object-based file system provided by the SiliconFS FPGA.