As part of the Information Forums in Mainz and London, I’ll be moderating panel discussions on big data with HDS executives, which should be both entertaining and enlightening. We will cut through the hype and discuss practical big data applications today while taking a look to the future and how the scale of data retention will evolve to create unique business value.

By attending, you will gain meaningful insights and tactical tips for evolving your datacenter to gain the most efficiency and reduce expenses. In addition, you will also be among the first to hear about innovative solutions and strategies such as:

Maximize Your IT Advantage – Discover how to accelerate performance, control your data growth, and transform your IT environment to be more intelligent, efficient, and cloud-ready.

Cloud, Your Way – Learn how to improve agility and flexibility through various private, public and hybrid cloud solutions provided by HDS and our partners.

Capitalize on Big Data – See examples of innovative, big data solutions today that transform how you capture and capitalize on data.

Click here to register for the HDS Information Forum in Mainz, Germany

Click here to register for the HDS Information Forum in London, UK

But corporate entities at branch offices also require storage to get their work done.  Their IT needs will typically be smaller, but their data needs will also continue to grow.  To mitigate this situation, Hitachi Data Ingestor can be installed at these locations as a cache for frequently used data, making it locally available with the vast majority of the data stored at the core HCP and accessed via links.

We complete our content cloud with the incorporation of the new HCP Anywhere (announced today), our enterprise-ready, safe, secure file synchronization and sharing solution.  HCP Anywhere is an enabler to mobile and remote employees, allowing them to collaborate and share files efficiently and securely, eliminating the need for them to use consumer clouds that can compromise security.

Finally, the picture is made complete by offering customers the means to offload data and content to a hosted cloud with offsite hosted cloud solutions that allow customers to reduce or eliminate additional capital costs and better control operational costs.

This tightly integrated portfolio, coupled with our new delivery model and partner programs enables our customers to deploy their own clouds, in their own ways allowing them to focus on how to take advantage of the transformation from traditional IT to cloud rather than trying to figure out how to get from where they are now to where they are going next.

An exciting new solution is Hitachi Content Platform (HCP) Anywhere-  the industry’s 1st integrated file sync and share designed for the enterprise and built by a single, well respected vendor. This integration-by-design puts HCP Anywhere ahead of the competition in security, ease of implementation, keeping IT in control. HCP Anywhere is:

• Secure: HCP Anywhere is an on-premise, private cloud solution that keeps data within the control and governance of corporate IT.
• Simple: Users download the HCP Anywhere client application from a portal or via the iTunes store, self-register their devices, and they are ready to go since it works just like other folders on their device.
• Smart: HCP Anywhere uses the backup, compression, single instancing, spin-down disk support, and metadata capabilities of HCP to create a highly efficient content sharing environment.

A new version of Hitachi Content Platform (HCP) plays an important and foundational role in the HDS cloud portfolio.  With this announcement, HDS expands  the robust capabilities of HCP as a cloud storage platform, featuring the most advanced metadata management in the market today.   A more feature-rich metadata capability means HCP can better help automate operations,  providing a strong foundation for a big data repository and helping find the right datasets for deeper analytics.

Also featured are new hosted cloud services from HDS.  These include Hitachi Cloud Services, an extension of the enterprise into a secure, robust off-premise hosted cloud managed by HDS, and the Hitachi Cloud Service Provider Program, a partner provided public cloud offering built on HDS cloud infrastructure and solutions.

With today’s announcement, HDS delivers another milestone on its strategy for the cloud with solutions that first and foremost addresses what customers need.  Our new and enhanced enabling technologies and cloud solutions help you:

• Choose the best possible solutions to address your needs, at your own pace, and in a way that makes sense for your business
• Solve the challenges you face today – and gives you a variety of new options to help manage the explosive growth of unstructured content and achieve the cost and flexibility benefits of the cloud by reducing CAPEX and OPEX
• Leverage cloud and file sharing technologies in ways that were never before possible. With these new offerings, you can be prepared for what’s coming next. (big data, bring your own device, next-generation file services, secure clouds, distributed IT, metadata-driven automation, management and analytics)

This of course leaves them with the following alternatives:

1. Work with a provider who can manage a private cloud that resides within the customer’s data center, built on assets owned by the provider
2. Choose a public cloud that charges for services delivered based on usage, allocates resources on demand, and supports a service level agreement (SLA) consistent with the needs of the customer

So how does one decide? Because an organization’s data and content are critical to deriving insight used to make sound business decisions, they are strategic assets to be seen and used only by those authorized to do so. For this reason, any cloud approach must ensure adequate data protection. At the same time, access to data and content, while carefully controlled, must be unfettered to those who need to find, use, and share it. Then effective collaboration and innovation can take place. Finally, the cloud must support the policies and processes needed to meet whatever compliance and regulatory requirements that apply to the data involved.

It goes without saying that performance and reliability are also important considerations. In the case of private clouds, the robustness of the hardware, software, and networking elements that make up the cloud environment are fundamental to determining if these criteria are met. This is true for public clouds as well, of course, although in this case they are reflected at a higher level in the commitment the provider makes through its SLA.

One could argue that there is something approximating a tradeoff between reaping the cost and resource benefits of the cloud and meeting security, access control, performance, and reliability requirements. The more that is owned and/or managed by a provider, the more control one could perceive they are relinquishing. That introduces risk – or, at the very least, the perception of risk.

The solution to this, of course, is to work with a provider that can assume the role of a “trusted advisor.” One that not only possesses the experience and knowledge to build, host, and manage quality cloud solutions, but one that has demonstrated the ability to deliver them securely. And one that is willing to back its performance and security claims with a strong SLA. The foundation of whatever they host has to be a set of proven platforms and technologies that can meet these demands.

In other words –put your strategic data in the hands of a provider who can prove they are worthy of it.

At the heart of this growing problem are data retention policies, which haven’t evolved much over the last 10 years. While we have more dynamic backup and archive (or data management) software, the cost of storage hasn’t dropped as quickly as growing consumption. Chris mentioned it wasn’t uncommon for his clients to have 60 to 100% annual data growth which is fueling concerns about the cost of data retention. Martin’s post specifically speaks to the total cost of a petabyte-sized tape archive. While the upfront cost of tape media is attractive, the realistic cost and time of maintaining a tape archive when you consider media migration and management is daunting. From Martin’s post:

It currently takes 88 days to migrate a petabyte of data from LTO5-to-LTO6; this assumes 24×7, no drive issues, no media issues and a pair of drives to migrate the data. You will also be loading about 500 tapes and unloading about 500 tapes. You can cut this time by putting in more drives but your costs will soon start escalate as SAN ports, servers and periphery infrastructure mounts up.

Further complicating this equation for CIOs is the promise of big data analytics and the challenge of extracting business value out of this retained data. As a standalone archive, tape doesn’t inherently lend itself to frequent access and data mining.

This challenge is larger than the medium used to store data. Today, data retention policies are often developed around retention costs and legal compliance and less on strategic business value. Copy management of backup and disaster recovery data also continues to compound the problem as does growing machine-to-machine (M2M) data. For many companies, petabyte-sized data retention is becoming the norm, not the exception.

Hitachi is no stranger to this challenge. As a $96 Billion dollar global conglomerate with business units that include Information and Telecommunication Systems, Electronic Systems and Equipment, Social Infrastructure and Industrial Systems, Automotive Systems,  Construction Machinery and Financial Services…this challenge is personal.

All of this points to the need for data retention architectures which span hardware, software and people.

Metadata and Object Storage

As more data is generated, cost effective, accessible long term data retention is going to be dependent on the optimization of storage resources and good utilization of storage metadata. Greg Schulz summed this up recently in FedTech:

For some applications, an important attribute of storage solutions and services are their metadata capabilities. This includes the ability to support flexible and user-defined metadata.

Another enabling capability is policy management, which can use metadata for implementing or driving functions such as how long to retain data, when and how to securely dispose of it, and where to keep it (along with application-related information). This adds some flexible structure to unstructured data without the limits or constraints associated with structured data management.

This is where object storage can play a significant role. By leveraging application metadata that describes the content and its data retention policies, it can enable automation of those policies, freeing up storage and human resources, as well as managing the problem of unnecessary duplication of data.

Storage Tiering

While the concept of storage tiering has been around awhile, I’m surprised how infrequently it is used in many data centers. If implemented properly, storage tiering provides dramatic optimization of high performance (and more expensive) types of storage while intelligently leveraging lower cost storage based on performance, capacity and reliability characteristics. Both structured and unstructured data can be highly optimized in most environments. Database applications, in particular, can benefit significantly from tiering. Frequently referenced data, such as control tables and indexes can reside on the highest performance storage tier while less frequently accessed reference pages can be moved to a lower tier of storage. I believe we’ll see a much broader adoption of automatic (or dynamic) tiering as budgets force IT administrators to gain better storage utilization rates.

Primary Storage Deduplication

Data deduplication has been broadly adopted for backup data, but adoption of primary storage deduplication has not. One of the biggest barriers to primary storage deduplication is the very real concern of performance degradation. Like so many storage technologies, primary storage deduplication has evolved to have little or no impact to performance. Like tiering, deduplication processes can be automated. Administrators no longer need to experiment with scheduling and tuning deduplication processes.

Conclusion

These aren’t the only technologies which can be used to optimize storage utilization, but they should be among the options you consider as you look at managing your growing data requirements. The challenge of large data is very real and data retention policies will need to evolve as well. Intelligent data management, which includes developing compliant data expiration practices, will also need to develop as a matter of discipline.

• can expand or shrink with the demand fluctuation,
• can heal on its own,
• can be programmed,
• is available 24/7 anywhere, any device

You’re probably seeing an overarching theme across HDS around enabling our next generation platforms to intelligently react with information in the platforms. As our CEO Jack puts it we Innovate with Information.  Software Defined Data Center, Cloud Infrastructure, Cloud Management are some of the current aspects of the IT industry that help make these intelligent platforms real.

Hitachi Data Systems has always innovated and contributed to society in a sustainable way.  Our recent launch of Hitachi UCP solutions is a great example of this effort. The work done with Hitachi UCP Pro for VMware vSphere not only allows our customers to leverage their investments in VMware infrastructures, but also arms them with REST API to programmatically talk to UCP. Look for a subsequent post from me on this topic.

As we innovate and enable customers to be ready for their future needs, we value the role our technology and channel partners play and we continue to work with them as we add more intelligence into our platforms. A great example for this is the recent launch of UCP Select with Cisco UCS servers, bringing Cisco’s server technology to our customers and empower our channel to add additional value.

From an IT platform point of view there are clearly some major movements happening in the industry– such as OpenStack, flash technologies, unified storage, unified fabric, and big data, and HDS will continue to make contributions in all of those areas. I have the privilege of shaping some of these efforts by playing the new role as CTO of Intelligent Platforms. I am honored and humbled by this new role and I hope to use this blog as a forum to engage with you all and to bring some technology insights into various efforts as we evolve. Speaking of OpenStack – I’ll also be writing about some of the near term activities we are doing in this area. Stay tuned and until then, here is another perspective on the Intelligent Platforms from our good friend Krishnan Subramanian.

• The industry’s first converged platform offering 100% parity across RESTful API, CLI and GUI
• A truly unified, “single pane of glass” end-to-end infrastructure orchestration solution with a low
  learning curve, due to our unique  UCP Director software, which integrates directly into VMware
  vCenter
• The only converged infrastructure that can leverage a customer’s existing storage, either by connecting to a customer’s existing Hitachi Virtual Storage Platform (VSP); or, by virtualizing a customer’s third party storage arrays using Hitachi VSP (more than 100 third party storage arrays from different vendors have already been certified – see supported storage here)

Hitachi UCP Pro for VMware vSphere was designed and developed from the ground up as a highly integrated solution with industry-leading elements. Unlike competing converged solutions, UCP Pro for VMware vSphere offers customers:

• Delivery in weeks not months.
• Lower OPEX.
• Future-proofing investment
• End to end infrastructure orchestration

And things are only getting better. With the recent release of an updated version of UCP Pro for VMware vSphere, we’ve responded to customer asks with several key features:

We are expanding our storage to include lower cost and capacity entry points with Hitachi Unified Storage (HUS) 150 and HUS VM storage systems. HUS 150 provides industry-leading price / performance and high density per rack for maximum space efficiency, while HUS VM provides high density, with our industry-leading storage virtualization capabilities. Customers can also leverage existing VSPs to easily incorporate this solution into their data center and again, lower the price to entry.

UCP Director now offers deeper integration with the vCenter Web Client. Of particular interest to customers with cross-platform support, vCenter Web Client can run on any standard web browser, and still provide the features and functionality of our “thick” server-based version. It even extends additional features such as inventory lists, related objects and portlets, offering a more comprehensive view of the data center to the vCenter administrator. Our web client is perfectly aligned with VMware’s strategy of developing web-based client versions for all their major applications and management tools.

The latest version of UCP Pro for VMware vSphere expands on the strengths of the previous version, offering customers even greater flexibility, cost savings and price / performance from their converged infrastructure solutions.

Interested in learning more?  Check out these quick videos to see how easily you can provision and manage your servers, storage and network in your VMware environment.

• How to provision a new ESXi server with 1 click using Hitachi UCP Pro for VMware vSphere
• How to provision storage to a VMware host using Hitachi UCP Pro for VMware vSphere
• Expanding the size of a VMware data store using Hitachi UCP Pro for VMware vSphere
• How to configure VLANs using Hitachi Unified Compute Platform Pro for VMware vSphere

Let’s start simply with a document saved on a user’s laptop (1 copy). Then the backup application makes a copy (1 copy). Let’s assume that this is then emailed to 5 reviewers. While the mail server will likely single-instance the attachment (1 copy), the 5 reviewers save local copies to their laptops (5 copies). We’ll give the benefit of the doubt and assume that the laptop backup app can single instance the backup copies. Still, we’re already up to 8 copies.

Now a few days go by and mailboxes are filling up, so the 5 reviewers move the document to their local email archives. Since the creator hasn’t archived it, it’s still in the mail system and 5 more copies have been created. Then the backup application copies the reviewers’ archive files which cannot be single instanced. Now we have another 10 copies, taking us to 18 total and neither the creator nor the reviewers have sent it to others (whose devices also get backed up), posted it to websites (which also get backed up), loaded it into content management systems (also backed up). We also haven’t touched on the replication and network traffic.

Imagine this scenario playing out across hundreds or thousands of users. Now imagine that the file isn’t some 10K spreadsheet or 100K document, but a 10MB presentation or a 100MB video. This is the kind of inefficiency that is running rampant in traditional IT environments.

#CloudAnywhere radically changes the equation by sharing access to files instead of sending copies in the thousands. Because all files are stored in a common, backup-free repository there can be just one well protected and properly replicated copy of a file that is available to all the right people, wherever they may be, with whatever device they have. Should users or applications need a local copy, it can be pinned to a user device or a file server and when that level of access is no longer required the file returns to being just a link to the latest version in the ‘content core,’ Hitachi Content Platform at the corporate data center serving as the centerpoint for all corporate data and content. More very soon…

In the spirit of exposing hidden but meaty things, I do want to reference two visionary gems.

• HDS joins PARC in the Emerging Networks Consortium – Being as this year is the 40th anniversary of the Ethernet I’m particularly proud of this effort. The guys at PARC are beyond smart and have had a tremendous impact on our industry. Again we are working now to make sure that in the years ahead our customers are going to get the best and we are well poised to meet key partners, like Brocade and Cisco, to realize a future with clear benefits.

• Hitachi MRI Scanners Ranked #1 for 3rd Consecutive Year – While at-a-glance this may not be relevant to HDS customers today, a hidden fact with stellar MRI capabilities is that for Hitachi to do well here we have to be super smart about image processing. That is because sharper images save lives, and this creates a pocket of capability we can leverage through open innovation with our customers.

These visionary beefy nuggets aside, I think that there is another big bun that Hitachi’s “Wagyu beef burger” more than covers: pervasive data center-wide software defined storage services. Truthfully, our Hitachi Data Ingestor (HDI) virtualizes our killer private cloud storage offering (Hitachi Content Platform, HCP) and addresses Hitachi, EMC, NetApp or DAS capacity because it can be delivered as a virtual machine appliance. Further, using our UCP Director software our users can orchestrate Hitachi’s current and future (that’s a really big hint)cutting-edge software defined offerings on award-winning Hitachi infrastructure.

Speaking of Hitachi infrastructure, this is another area we’ve been innovating in for years. We innovated separation of the control and data planes within the block storage infrastructure layer and Hitachi has subsequently cascaded this approach in our Hitachi NAS Platform (HNAS) and HCP offerings. Moreover, a deeper look at our Hitachi Virtual Storage Platform (VSP) and its little brother Hitachi Unified Storage VM (HUS-VM) reveals a Hardware Abstraction Layer (HAL) ensuring a high degree of flexibility in our choice of physical infrastructure — hmm…I wonder where that is going.

So this week if you’re spellbound by a ringleader’s sizzling showmanship while he announces the latest hijinks of the Bourne clown troupe masquerading as a fire brigade to “save the storage world as we know it,” pause and ask “where’s the beef?” If you ask this question, and I believe you will, I have an answer for you and it is coming later this month.

Given that I commented on the Magic Quadrant for General Purpose Disk Arrays with a blog, I thought it best to give the server business some equal attention. When the storage-focused and blade-focused Magic Quadrants are reviewed, you’ll see that there is a lot of commonality in how positively Hitachi is positioned. It’s great to see our strategy becoming understood by the analyst community.

There is one main difference between the Magic Quadrants, and it’s one that multi-tasking readers rush to – the picture. Of course in this case the picture is the “Quadrant” that highlights the placement of each vendor. In the blade server space, HDS is shown as being in the Niche category, while in the general purpose disk arrays we are positioned as a Leader.

So the question is, in today’s world does the old adage still hold true? Does a picture still equate to a thousand words?

As Nick pointed out, the HDS position in the “Niche” quadrant is actually quite logical given Gartner’s definitions for placement. We are successfully helping our enterprise customers, but are not one of the “big guy” blade server vendors that is covering every possible use case across the globe; our approach is far more targeted.

Since Gartner has continued to move the position of Hitachi upward (better Execution) and rightward (better Vision) with each of the last three Magic Quadrants, I take that as a clear sign that they see that our strategy of focusing on Hitachi Unified Compute Platform and enterprise-level compute innovation as one that is working for us and our customers.

For a bit of “Niche” category perspective, we are positioned similarly to Oracle with respect to our “Ability to Execute” and well ahead of them in terms of “Completeness of Vision.”

So while the positioning makes sense, the “thousand words” that the Quadrant picture represents doesn’t feel as illustrative as the 327 words that Gartner writes about HDS and our server capability.

Of the noteworthy comments, Gartner states that “Hitachi innovates strongly around blade aggregation and highly integrated virtualization” and “…Hitachi’s blade servers are highly popular among the vendor’s installed base…”

Gartner goes on to mention how their client feedback on Hitachi Unified Compute Platform, our primary go-to-market with our Compute Blades, is “very positive.”   They also mention that our Compute Blade hardware is “a well-proven platform, with a strong Japanese installed base.”

To be fair, Gartner also mentions “Cautions” about a few HDS server areas, as they do with all vendors.  For HDS, Gartner’s commentary focused on a storage-oriented sales and marketing focus, that we are “relatively unknown” as a server vendor outside of Japan, and a “limited” channel presence for servers.  Relative to the largest blade server vendors, those seem fair.

Of course, none of the “Caution” points suggest we cannot help our customers, as demonstrated by our continuous growth, both in the market and as represented in the Gartner “picture” of the world.

Is it possible then for 327 be greater than 1,000? I’ve never been a mathematician, but to me Gartner’s 327 words are far more valuable to understanding how HDS can help you with your converged infrastructure and computing needs than the thousand words presented by that picture.

Hopefully everyone will read them.

Because, while the math doesn’t seem to work, we continue to get customer feedback on our converged infrastructure offerings that “the sum is far greater than the parts,” so maybe it’s a comeback of the “new math” we heard so much about a while ago.