Data Center Advisors

Horizontal-UVM – Step 1: Setup and Copy-less Data Distribution

So it’s been about a month since my last post, sorry about that, but I’ve been working on several exciting projects that has me quite booked up as far as time goes. One of these projects I can actually share with you. I shared this concept last week during the version 0.9 Blogger Day at HDS headquarters with several storage industry bloggers from around the world. This concept is Horizontal Universal Volume Manager and this actually isn’t a “concept” as much as it is now a proof-of-concept, proven. That is, this is based on GA products and features but configured in such a way that is different. Actually, in talking with some folks, it always seemed to make sense that UVM could do this, but no one’s try it.

I’ve been working with the help of Gordon Bews, from our Advanced Technical Consultants Group, in one of HDS’ innovation labs, on a method of loosely scaling the Universal Volume Manager in the USP-V and USP-VM systems (which would also work with the USP and NSC55). This Horizontal-UVM allows the USP-V(M) controllers to horizontally scale-out using Fibre Channel links as a bidirectional trunk that essentially scales up bandwidth as needed by adding additional links and out by adding additional USP controllers. Basically, cross virtualizing internal LUNs between controllers with UVM and presenting them to the SAN and/or servers. This cross LUN presentation could be used as an active LUN locally and a read LUN from a remote location within MAN distances (Fibre Channel link distances). The use case that I will to address with this cross-linking method is copy-less data distribution scenario which also addresses a one writer many reader situations as well.

In this post I describe the initial steps to set up the first Horizontal-UVM configuration and what you could do with it. Refer to this figure for these step-by-step instructions. For these instructions, I’m using Storage Navigator to set this up. My description here will be at a high level as I don’t want to copy-n-paste the individual instructions from the manual however, I do encourage you to refer to the manual for the detailed descriptions of these steps.

Step A, make some LUNs.

To begin with, configure two USP-VMs systems (what I happen to have lying around in the innovation lab) with some number of internal LUNs and assign them to FC ports using LUN Manager. I’m going to use the term “physical LUNs” to describe “internal LUNs” to make a more pointed distinction between real LUNs and virtual LUNs. In this description, I’m using 4 physical LUNs per USP-VM and set them up with multiple paths for the Linux servers. I’m also using HDP for my LUNs so I’m only using a small percentage of capacity to start with. Everything we configure now starts as an HDP provisioned LUN and only in exception cases do we configure traditionally provisioned LUNs.

Step B1, map some LUNs.

From USP-V(M)-Local, set up a pair of ports to be your external link, switch them to be initiators versus targets, as though you were setting LUNs up to another set of ports to another host server. So now you should have a set of 4 physical LUNs with a pair of FC paths to a Linux server as target ports and a pair of FC ports set up as an initiator with these 4 LUNs assigned to them.

Step B2, repeat.

Now, set this same configuration up on USP-V(M)-Remote. Again, refer to the figure for your reference of location.

Step C1, device discovery from USP-V(M)-Local.

Go into the Universal Volume Manager and have USP-V(M)-Local discover USP-V(M)-Remote’s LUNs. This should happen fairly quickly and painlessly. Configure these LUNs as LDEVs and address them in one of the Logical Control Units in order to give them a controller address. The assigned addresses in a LCU can be done automatically or manually. You can now configure these LUNs as eLUNs (external LUNs) and map them to a FC port. In this case, map them to the same port that the Linux server is connected to that you set up in Step A. For this description, I have “Cache Disabled” since these are USP-VM and the local caches are very fast as it is.

Step C2, repeat device discovery from USP-V(M)-Remote.

Now do the same thing from USP-V(M)-Remote. Go into the Universal Volume Manager and have USP-V(M)-Remote discover USP-V(M)-Local’s LUNs.

Step D1, map the new eLUNs to the Linux server on USP-V(M)-Local.

Go into LUN Manager on USP-V(M)-Local and map the new eLUNs from USP-V(M)-Remote to the same port as the initial physical LUN (internal LUNs) from Step A. These LUNs will be noted with a “#” symbol appended to them.

Step D2, map the new eLUNs to the Linux server on USP-V(M)-Remote.

Go into LUN Manager on UPS-V(M)-Remote and map the new eLUN from USP-V(M)-Local to the same port as the initial physical LUN (internal LUNs) from Step A.

OK, so by this time, you should have 4 physical LUNs (internal LUNs) on each USP-VM for a total of 8 LUNs. There should also be 4 virtualized LUNs on each USP-VM. So each Linux server should “see” eight LUNs, assuming a multipath setup to these LUNs is present. Now you should be able to write data to USP-V(M)-Local physical LUNs and have the remote Linux server on USP-V(M)-Remote access this data via its virtual LUNs. I like to call this the reflection of the local LUNs at the remote location. Another set of terminology we are playing with are ghost LUNs or “Casper LUNs” because they’re so friendly.

A scenario for using this configuration goes as follows;

• 1. A research department in a remote research facility writes to a file system on the local USP-VM LUNs. This is a large data set that needs to be processed at one of the corporation’s datacenters where an HPC Cluster resides which is 50 miles away.
• 2. A. This local file system can then be unmounted from the research facility ad mounted by the research facility,
• 2. B. OR the remote site can mount the ghost LUNs as read-only.
• 3. The HPC Cluster then reads the data set in a batch job and processes the data over the course of a few days.
• 4. The resulting data set is actually written to the physical LUNs on the USP-VM as a normal read/write mounted file system.
• 5. The remote research facility can read the ghost LUNs on the local USP-VM by either
• 6. A. unmounting the remote USP-VM file system and mounting the local USP-VM file system
• 6. B. OR, mounting the local USP-VM file system read-only.

Obviously, the read-only mounting or unmounting and unmounting of a file system is a bit of scripting that needs to be added to the batch jobs, but this whole scenario is accomplished without copying data over the MAN distance SAN. Data is moved over the MAN-SAN, but a resident copy, even temporary isn’t performed.

Some of the next steps in this capability will include adding other software features into the mix to solve a greater range of use cases or by fanning out the access of a physical LUN to multiple USP-VMs in a core/edges model for data distribution. Stay tuned as I will continue to post updates as I experiment and test these use cases and the various configurations. Also, if you can think of a use case that includes the basic connections described in this article or with the addition of a USP-V storage software package to solve a use case or a genuine customer problem, please let me know. I will try to validate it and put it in a future post and give you the credit for suggesting it.