Techno Musings

A quick recap of part 1 where we talked about migration to external file storage devices coordinated by HDDS with the migration engine inside of HCAP, and support of Netapp as a search target.  Here there are two more items to discuss.

The two capabilities  in the previous post provide a great adoption patterns for NetApp storage: simple NFS storage targets which are already depreciated or to solve a namespace sprawl problem.  The third major feature of HDDS V1.2 that I want to bring up is complex user mapping.  Basically as we looked at enterprise LDAP and AD infrastructures we found that it was highly possible for users have many credentials across many directory instances and when interacting with a system that is driven by search they should be able to link their individual user accounts together.  An additional use case that we can support is if a physical person has left a company and a search needs to happen that includes their credentials, our approach allows the “retired user’s” credentials to be linked to an existing employee and searches can be performed as the “retired user”.  This is a big deal as it solves one of the chief worries that corporations are dealing with today: how to find content as users who are no longer working at a company.  As when a discovery order comes down under the likes of the FRCP all information assets which are governed by a records retention policy are up for grabs and just because an employee is no longer at the company doesn’t mean that their files and records aren’t subject to search.

Fourth, is a patent pending technology that we call “index-stubbing” internally.  Namely as content from HNAS is migrated to externally attached systems there is a stub that remains on the HNAS system within the file system.  This stub contains all of the requisite meta-data like the NTFS ACL information, POSIX metadata, etc.  Knowing that our users would not allow us to index the content twice, HDDS V1.2 implements our index-stubbing technology which is a little hard to grok.  Basically if you can imagine an HDDS instance connected to HNAS and HCAP-Search connected to an HCAP system.  Then basically on the HNAS instance, only a stub exists in the index containing the key metadata including the ACL information while HCAP-Search contains only relevant metadata and the full content.  This way our users are paying for extra indexing capacity, but only the minimum they need.

Why is this important and how does it relate to content services?

As I pointed out here, there is an inherent flaw in traditional content management and the industry needs to enter into a full rethink of these platforms.  More specifically, key questions that are being asked by users are around building an ROI for unstructured information, constructing an unstructured data warehouses, or defining levels of risk for unstructured data assets, and reuse/repurposing of these assets and the associated security models around them.  So what HDDS in concert with file storage devices provides is the foundation needed to begin the realization of a next generation scalable content services platform which can begin answering the questions mentioned above at scale.  Already we have customers like NARA who are using HCAP and HCAP-Search to build a dense scalable content services platform on top the aggregate system, in short Hitachi’s advanced customers are beginning to flirt with solving this problem.  If you reread the post referenced in this paragraph you might be able to extrapolate how the HDDS plus file repository architecture can be used to start the rethink…