The reason that big data is getting such attention today is the greater variety of new data types that are available. This data is being generated by many new sources, like click streams, smart meters, smart phones, RFID, NFC (Near Field Communications), etc. Almost every new piece of equipment has sensors built in that can transmit information. In some verticals the increasing us of tools creates many different types of data. A stay in the hospital may involve x-rays, CT scans, MRIs, sonograms, electrocardiograms, PET scans and other monitoring tools for patient care, pharmacy and billing. This variety of data provides more information to solve a particular problem or provide better service. The trick is to capture these different types of data in a way that makes it possible to easily correlate the information that is contained in the data.

In order to harness the power of this big data variety, we must be able to virtualize the data from the application that created it so that it can be used with other applications. How do you virtualize the data from the application? If you simply separate the data from the application, it is just a bunch of bits unless you put that data into a container with the metadata that describes the data content and the policies that govern it. Once you do this, you have created an object, which is self describing and is not dependent on the application that created it. There are many ways to create an object. Here are some examples of how objects are created and stored within Hitachi Content Platform (HCP).

HCP is a multi-tenant content or object store that can store a variety of data using standard protocols like NFS or HTTP in the same virtualized storage pool that can scale to petabytes and billions of objects. HCP can store content from different data owners together in the same object pool, but assign each data owner its own secure tenant space so that other unauthorized users can not access it. However, with the right permissions, a user can do a content aware search across all the modalities of data. A good example of this is in healthcare with the use of Hitachi Clinical Repository (HCR).

Here you see a variety of data sources or modalities within the hospital. Hitachi Data Ingestors (HDI) at remote clinics can ingest data into a central HCP where they are kept as separate tenants.  An authorized patient care provider can access all the clinical tests and evaluations done on a patient to coordinate his care. There may be others who do not need to see the patient’s personal data, but may be authorized to do an analysis on the effectiveness of procedures or medication.  All this variety of data can be managed by HCP so that authorized people can see all the data that is relevant to a certain task.

A content platform like the HCP that can ingest a variety of data objects is a key enabler for big data variety. Having all the different data objects in the same repository makes it easier to correlate the different varieties of data. Storing the metadata with the data makes it possible to do a content aware query or search against the metadata without having to access the actual data.  If each variety of data were stored in its own repository, you would have to process or query each repository separately or manually merge the data from each repository, which may be impractical if you are also concerned with the volume and velocity of big data.

The display consisted of photographs and placards, which described the establishment of the first IBM storage lab and the development of the 305 RAMAC. RAMAC consisted of 50 disk platters, which were 24 inches in diameter. At first they were mounted on a horizontal spindle, then later they were mounted on a vertical spindle to make it easier to add disks. Each disk had a 5 inch recording bandwidth, with 20 tracks per inch which could store 100,000 characters (7 bit) per disk. With 50 disks, this totaled 5,000,000 characters per 305 RAMAC. The rotation time was 50 ms while the maximum seek time was 0.6 sec.

While I was fascinated by this display, and taking notes since my camera had to be checked in at security, everyone else in the waiting room paid no attention since they had other concerns or problems (you don’t go to Superior court unless you have a serious problem). I doubt if anyone else in the crowded room was even born in the 1950’s and most were born in other countries, all were struggling to find their place in society.

Since I am currently writing a blog series on Big Data, it was interesting for me to be standing in the place where it all began 60 years ago. Many things have changed since then, but it is good to remember what progress can be made when we are focused. IBM was focused on creating a lab for the sole purpose of developing a random access storage device and they accomplished it in a matter of a few years, from 1952 when the first specs were written until 1955 when the RAMAC was announced. A few years later a young President set a national goal to have a man on the moon by the end of the 1960’s, and that was accomplished.

As I looked around the waiting room of the old IBM Lab that is now a waiting area for Superior Court I wondered what we could accomplish if we could focus on solving the problems that brings so many of these young people to court. I know other young people at my work who are energized and focused on solving problems outside of their private lives. Hopefully the information systems we work on can lead to social innovations, which can help the young people around me in this room.

This velocity has to do with the speed at which we can input and output big data. This becomes particularly important as more and more of the data is machine generated and the size of data objects explode with multimedia and multi-dimensional objects.

The performance characteristics of big data will vary. Some will be sequential streaming with large block sizes; others will be short bursts with very random block sizes, with everything else in between.

Since everything is eventually stored on a block device, you need to start with a very fast block device. The new Hitachi Unified Storage (HUS) system is an extremely fast storage system for high performance applications and makes an excellent backend for file, content and virtualization systems which manage the flow of data into HUS. HUS has up to sixteen 8Gbps FC front end ports, load balanced cross two high performance controllers with 32MB of cache. On the back end it has 32 SAS (Serial Attached SCSI) 6Gbps Serial Attached SCSI links to high performance SSD and HDD media. HUS has three times the sequential performance of its high performance predecessor, AMS. HUS can scale to 960 drives, for nearly 3PB with current 3TB disk technologies.

For greater scalability, enterprise Virtual Storage Platform (VSP) can scale to 255PB across internally and externally virtualized storage. With 192 x 8Gbps FC ports, 1TB of shared cache, and 6Gbps SAS Flash and disk drives, VSP has the performance to serve multiple big data streams.

Most of the unstructured data will come in over file systems, primarily NFS for sensor type data. The key to high performance in any file system is parallelism, but there is a difference in how Hitachi NAS (HNAS) does this compared to other parallel file systems on the market. Historically, multiple data stream architectures, especially shared memory implementations require synchronization via a host operating system for memory coherence. This dependence limits overall performance and scalability. HNAS file system is implemented in FPGAs, which control and enable specific functions. Two key features of this implementation, which contribute to high performance and scalability are offloading and pipelining.

Offloading allows HNAS file system to independently process metadata and simultaneously move data to and from the hosts and disks. File system operations, which do not require hardware acceleration through the FPGAs, are separated and sent to a metadata processor module while operations in the data path are handled by a pipeline of FPGAs. Each file system path has dedicated memory. There is no single bus and therefore no points of contention for memory access.

Pipelining is achieved when multiple file system operations are simultaneously overlapped in their execution sequence and data pipelining is achieved by routing data operations to independent sets of FPGAs. The operations are independent and have neither shared memory nor message passing dependencies.

Unstructured data can also be objects, which are generated and accessed over RESTful interfaces like HTTP. Hitachi Content Platform (HCP) is a high performance platform for capturing this type of data. HCP can scale to 80 federated nodes for ingestion of object or content data. HCP like HNAS are gateways that connect through high speed FC to VSP or HUS high performance block storage. HNAS, Hitachi Data Ingestor and HUS can ingest NFS or CIFS files into HCP as objects. HCP provides high-speed ingestion, search and access through its clusters of storage and search nodes.

A key part of big data output is the ability to quickly find what you want amongst billions of objects or files. With billions of objects it would be very tedious to find an object in a database using a table scan or crawling a directory in a traditional file system. This is where an object store like HCP or an object based file system like HNAS can provide “velocity” in accessing information. Objects are stored with metadata which describe the content of the data along with the policies that govern it. Instead of a table scan or directory crawl, objects can be accessed quickly through a query of the metadata. iTunes is a good example of an object store.

Velocity is a key enabler for big data input and output as well as search. Hitachi provides high velocity building blocks, VSP, HUS, HNAS, and HCP for big data whether it is block, file or object.

“Big” is defined by most dictionaries as “considerable size, number, quantity, magnitude or extent”. Big is a relative term. For instance big may be 2TB when considered for an “in memory database” like SAP HANA, or it could be exabytes for search engines like Google. Big is also a rapidly moving target. When Hitachi announced USP storage virtualization platform in 2004, with the capability of managing 32PB of internally and externally attached storage, most people thought it was over the top. Today most enterprise customers have over a petabyte of storage and they can install it in less space than they required 5 or 10 years ago. We can install 3PB in Hitachi Unified Storage (HUS) with 3TB disk drives in the width of two data center floor tiles and VSP, which we announced in 2010, can manage 255PB of internal and external storage. 255PB is a quarter of an exabyte!

An exabyte used to be considered a futuristic capacity, but search engine companies already have exabytes of storage. Some cloud companies are looking at file sharing or backup services for home data. Since many homes are storing a TB of data, it only takes a million subscribers to have an exabyte of data.

So the first requirement for big data volumes is scalable capacity, way beyond what is available today since the demand for data storage is accelerating. Where terabytes used to be the norm and petabytes seemed beyond the horizon just a few short years ago, we are now in the world of petabytes with exabytes around the corner. Many companies have 5 year planning cycles. In the past, they would plan for a doubling of capacity. Now they need to plan for an increase of an order of magnitude and plan it so that they can grow it nondisruptively. This requires storage virtualization.

VSP and HUS can scale capacity in the petabytes, and with virtualization in VSP we can create a pool of storage capacity approaching a quarter of an exabyte. But what about file and content data? How will they be able to scale since more and more of the growth will come from unstructured data?

The difference with unstructured data is that it is accessed through internet protocols and stored in file or content platforms. Most file and content systems are limited to terabytes where they need to scale to petabytes today and into exabytes tomorrow. Since this data is unstructured it must be searched and accessed as files or objects. Traditional file systems from UNIX or Linux store information about a file, directory or other file system object in an inode. The inode is not the data itself but the metadata that describes the data in terms of ownership, access mode, file size, timestamps, file pointers and file type, for example. When a traditional file system is created there is a finite upper limit on the total number of inodes, which then limits the maximum number of files, directories or other objects the file system can hold. HNAS and HCP use object based file systems, which enable them to scale to petabytes and billions of files or objects. HNAS and HCP are gateways, which sit on top of VSP or HUS so that they can leverage the scalability of the block storage and while enjoying the benefits of a common management platform, Hitachi Command Suite. HNAS and HCP are architected for big data for file and content

In addition to scalability, big data volumes must be able to scale nondisruptively and migrate across technology generations. Movement of data must be kept at a minimum and done in the background. Big data should be copied only once for availability and changes should be versioned instead of backing up the whole big data volume with every change. Big data is too big to be backed up.

Across the Hitachi family we can move and tier data in the background. We can add capacity to VSP or HUS block pools, HNAS file systems or HCP tenants, and automatically rebalance the data across the new capacity. Older file systems and block storage devices do not allow for dynamic expansion. In order to use new capacity, data in these older systems has to be unloaded from the old block or file system and reloaded onto the new capacity. This is totally impractical with the volumes associated with big data today.

Big data volumes also have to be resilient. There can’t be any single point of failure, which would require the rebuilding of a big data volume. With block systems we have redundancies built throughout VSP and HUS. We also need to have the same resiliency for HNAS and HCP nodes. These nodes have to be stateless so they can be easily replaced if any node fails.

The volume of big data is not just about the amount of capacity but also includes technologies to eliminate the traditional methods of storing files, moving volumes, backup and replication. Big data also requires the integration of file, block and content under a common Hitachi Command Suite management platform.

In a 2001 research reportand related conference presentations, then META Group (now Gartner) analyst, Doug Laney, defined data growth challenges (and opportunities) as being three-dimensional, i.e. increasing volume (amount of data), velocity (speed of data in/out), and variety (range of data types, sources). Gartner continues to use this model for describing big data.

IDC uses these same parameters of volume, velocity and variety, and also adds value as a fourth dimension (source: IDC http://www.idc.com/getdoc.jsp?containerId=IDC_P23177)

The objective of big data is to provide more strategic value from the massive and varied data being produced today and ultimately produce even greater value to the organization or to society as a whole.

The idea of big data has been around for some time. The most notable examples of big data practitioners are Amazon and Google who mine click streams for ad placement. Other examples are Walmart and 7-Eleven for stocking and promotion. These companies have distinguished themselves from their competition through the value of big data.

Last month Time magazine quoted Ray Kurzwell, a well known futurist as saying “A kid in Africa with a smart phone has access to more information than the President of the U.S. 15 years ago.”
If a child in Africa has access to more information today than President Clinton did in 1997, how much more information is available to businesses today? And are they getting value from that information?

It is no longer enough to be a storage company. As Hitachi Data Systems, we need to be in the business of helping our customers collect, consolidate, consume and analyze data for to create actionable information.

In the next few posts I will be discussing how we support the attributes of big data: volume, velocity, variety and value. I would also like to get feedback on what others are doing in this area.

As the name implies, this will be a unified storage platform, meaning that this will support multiple protocols like FC, iSCSI and Ethernet (NFS and CIFS) and manage multiple data types like block, file and object in a single framework. This platform incorporates the high performance FPGA-based file architecture that we acquired from BlueArc last year. This architecture is based on a single clustered namespace and an object-based file system that enables fast searches, fast replication, and automated tiering and movement of files.

In addition to the unified file and object data support, what has been creating the current buzz and excitement is the increased capacity, performance, connectivity and features of this product all at a reduced price/GB, as well as the fact that HUS (and the entire HDS hardware product portfolio) will be supported by Hitachi Command Suite software.

The first thing one notices is how the density has been improved to reduce datacenter floorspace consumption.  The HUS 110 and 130 block modules are slimmed down to 2U trays each with a choice of 24 small form factors (SFF) 2.5 inch drives or 12 large form factor (LFF) 3.5 inch drives.  The largest HUS model 150 block module is housed in a 3U drawer which does not have any internal disks. Instead of drives, the 150 block module can house different types of replaceable front-end port modules.  Each model has an option to add a file module for NFS, CIFS and FTP file sharing access.  The file modules can be installed in single node or clustered configurations.  Customers can add expansion trays for additional capacity.  These have been re-designed to provide better density.

The difference between the 3 models, 110, 130, and 150 is in the scalability of cache, front end ports, back end ports, clustered file modules, and number of drives that it can support.  The HUS 150 with a maximum of 20 high density expansion trays can provide 960 drives across two frames or the width of two data center floor tiles. With 3TB large capacity SAS drives, this is 2.8PB of storage system with 32GB of cache, 16 FC front end 8 GB/sec ports and 32 back-end 6GB/sec SAS links. This is ‘Big Data” in the eyes of most customers.

HUS will be one of the first enterprise storage systems to utilize the new Multi Level Cell (MLC) 200 and 400GB SSDs, which will lower the cost per capacity from previous Single Level Cell (SLC) technology, making SSD affordable for a wider range of applications.

In my prior post, I noted that the historic price erosion of disk media will level off at a much slower rate than we have enjoyed over the past 50 years; and that the price increases that were caused by supply shortages in the beginning of this year will remain with us even after the supply chain has been restored in order to recover the costs of rebuilding that supply chain.  While HUS uses the same SAS drive types, this HUS product will be able to provide significant price savings through improvements in the design of the packaging and newer technology components. In addition, more of the software will be included in the Base Operating System (BOS) of HUS. Contact your Hitachi sales partner or integrator to see how you can increase your capacity, performance and functionality at a lower cost.

In short, the improvement in performance comes from faster front and back end ports, faster ASICs and processors, and an improvement in memory management. The introduction of metadata enhancements will enable an increase in snapshots to 1024, and improve the performance of memory dependent features like CoW snapshots, full capacity replication and dynamic tiering. The common management tools in Hitachi Command Suite also support it. HUS is a perfect repository for our Hitachi NAS and HCP products, as well as external tiered storage for VSP.

If you are thinking of moving to storage virtualization with the VSP, there is a campaign to encourage users to virtualize external storage. It is called Switch It On. This campaign reduces the software costs associated with virtualizing external storage on VSP and includes a “Virtualization Starter Kit” with services and education for users to hook up and utilize external storage. This campaign in conjunction with the storage efficient HUS is an easy cost effective way to start your storage virtualization journey to realizing a sustainable storage environment in the face of escalating data growth and complexity. Check with your Hitachi storage provider for the latest VSP release codes that support HUS as external storage.

This is only the beginning and there is much more to come.

I estimated there to be potentially a billion new users in developing countries. Industry sources say that there are 5.9 billion mobile subscribers in 2012. If the ratio was one person per phone, that would be 87% of the world’s population! I recently read that mobile subscriptions in China alone just surpassed the one billion mark. But these mobile users now appear to be the tip of the iceberg if we look to 2020.

There is a research company in the UK, Machina Research who advises on the M2M and mobile broadband market. They recently published a report, which had some very surprising statistics on what they call “connected devices” which they define as all devices that are used for transmitting or receiving packet data telecommunications via any wide area or local area network. Cell phones, PCs and tablets may be the least of these devices. Here are three key predictions for connected devices:

• Overall connected devices will treble over the next 10 years from 9 billion in 2011 to 24 billion in 2020.
• The lion’s share will be machine-to-machine connections
• The growth in PC/laptop, tablet, and handset data usage will result in a massive increase in data. Machina Research forecasts that global mobile data traffic will increase from 4 exabytes in 2011 to 42 exabytes in 2020 with 60% coming from PC/laptop connections and 37% from handsets.

You can see the full report here.

We are already seeing an increase in machine-to-machine connections, like those that monitor every Boeing 787 flight for efficiency and safety, surveillance monitors for security, and sensors for weather monitoring. Aside from these industrial applications of connected devices, we are seeing more and more in our personal lives. Using this definition of a connected device, try counting up the number of connected devices that you use. You may be surprised.

I started to count up the number of connected devices that my wife and I share in our home. We both have cellphones and laptops, we have two landline phones which we converted to Verizon wireless, one iPad, an ePrinter, a Direct TV set top box, and a smart meter from PG&E for a total of ten connected devices. In the future we are told that our cars will be connected, as well as our refrigerators, toilets, solar panels, treadmill, heart monitors, etc.

This all means a lot of data that needs storage and a lot of information to be analyzed to enhance our lives.

Today Hitachi and Hitachi Data Systems announces the new Hitachi Compute Blade 500, which incorporates the new Intel Xeon processor E5-2600. The Hitachi Compute Blade 500 is a high-density 6U rack mount blade server system with a chassis that supports up to 8 server blades with associated I/O and management hardware. It can also scale deep through the use of native logical partition (LPAR) technology implemented in firmware. Hitachi supplies the only x86 servers that can provide LPARs similar to those available in more expensive UNIX and mainframe systems for high availability and scalability. The Hitachi Compute Blade 500 supports the co-existence of its native LPAR capability with different software hypervisors in the same chassis.

Another unique feature of the Hitachi Compute Blade 500 is the introduction of a state-of-the-art embedded 10Gbps Ethernet Fabric switch module, designed to increase scalability and enhance virtual machine (VM) mobility, further simplifying management and reducing operational overhead of virtualized blade server environments. This new switch—which we OEM from Brocade—has non-blocking, cut-through architecture for performance, low latency and dual speed (1Gbps/10Gbps) capable external ports. The Brocade VDX 6746 Switch enables Hitachi CB500 to support flexible connectivity options for cloud architectures, including lossless Ethernet fabrics.  VDX 6746 provides CB500 customers the choice to enhance their hierarchical network architectures, deploy flatter scale-out fabrics or converge networks when deploying virtualization and cloud IT infrastructures. It also supports the convergence of SAN and LAN with conventional Ethernet, FCoE and the new DCB (Data Center Bridging) protocol.

The performance, availability and scalability of the Hitachi Compute Blade 500 with the Intel Xeon processor E5-2600 and embedded 10Gig Ethernet switch make it the perfect platform for tier 1 virtual server applications.

First was Seagate’s demonstration of HAMR (Heat Assisted Magnetic Recording), which surpasses the 1TB per-square-inch magnetic recording barrier. This breaks the previous recording barrier of 620GB per-square-inch for perpendicular recording, which is the current magnetic recording technology.

Seagate has achieved a milestone 1TB per-square-inch storage density using heat-assisted magnetic recording (HAMR) technology

Seagate’s press release describes the technology as follows:

HAMR technology will significantly extend the capacity of modern magnetic disc drives that use magnetic heads to read and write digital data onto spinning discs. If the storage density (the number of data bits stored on a given disc surface) continues its phenomenal growth rate, within the next five-to-ten years the data bits will become so small that they may become magnetically unstable due to a phenomenon known as superparamagnetism. The solution is to use a more stable medium, however today’s magnetic heads are unable to write data on such media. HAMR solves this problem by heating the medium with a laser-generated beam at the precise spot where data bits are being recorded. When heated, the medium becomes easier to write, and the rapid subsequent cooling stabilizes the written data. The result of this heat-assisted recording is a dramatic increase in the recorded density that can be achieved.

With this technology, Seagate predicts that they can achieve 6TB 3.5 inch disks and 2TB 2.5 inch disks later this decade, with 60TB disks available in the following 10 years. Note that there was no specific timeline for this new technology and no indication of pricing for these new HDDs. From the picture, HAMR looks like it will require some retooling of the manufacturing process, and that ultimately means cost.

Last week, HGST announced a 4TB 3.5 inch hard disk drive, using perpendicular recording, by going to 5 platters instead of the 4 which are in most 3.5 inch HDDs. This means more heads and platters–which also means more costs.

Tom Coughlin, who publishes the report HDD Capital Equipment and Technology, posted a piece on this in Forbes.

He says that the increases in HDD areal densities have slowed down to 20-25% annually, which is down from 40% or higher in past years, and one way for vendors to increase storage capacity is to add more heads and platters—like HGST is doing for 3.5 inch HDDs, and Western Digital is doing for 2.5 inch HDDs. (HGST is owned by Western Digital, but due to Federal Trade Commission requirements, they must operate as two separate brands.)

All of this indicates that the steep price erosion that we have enjoyed due to increasing areal densities of magnetic recording for the past 50 years is over, and we have to find ways to be more efficient if we are to keep our storage costs sustainable. To read more around this, take a look at previous HDS posts on storage efficiencies.

One analyst noted that there were some layoffs as companies tried to reduce costs, however he argues that reducing costs does not enable you to grow and create jobs unless it is tied to a strategy for increasing efficiency, which helps your business grow.

In IT there is tremendous pressure to reduce costs as the volume of data and new applications continue to explode in the face of a slow recovery. It is tempting to go for the lowest acquisition cost to get immediate relief, particularly in the area of storage costs. If you take the view that storage is a commodity, then all storage is the same and you go for the lowest price.

We believe that storage is strategic and has an impact on the efficiency of the whole data center, including servers, hypervisors, networks, applications, facilities and operations. This impact increases as we see applications and hypervisors, like VMware, offload more functions to storage so that the applications can be more efficient.

Site recovery manager is a good example. VMware can manage site recovery all on its own, by sending the VM disks to another VMware server. But when this is done, the VM disks must be moved over the LAN/WAN, consuming network bandwidth and processor cycles, and the recovery time and recovery point will take hours. This may be fine for non-critical applications, but how much would this outage cost for critical ones?

If down time is important, then VMware can download the data movement to a storage system through the use of VAAI and let the storage system move the data at FC speeds without impact to the CPU cycles and LAN/WAN. There will be extra costs for the replication function in the storage and FC connection, but this may be less important than the recovery time for the application and in the end be more efficient.

David Merrill talks more about efficiencies rather than costs. Visit his archives on capacity efficiency here: http://blogs.hds.com/david/category/capacityefficiency

Or, for all other posts on maximizing storage and capacity efficiencies, check these out: http://blogs.hds.com/capacity-efficiency.php