The Mathematics of Storage Performance and XIV
by Claus Mikkelsen on Jan 31, 2009
So my last blog was on IBM’s XIV product, and I must say it did create some controversy. I got a few comments along the lines of “really?” and a few that disputed some of the various facts. I even got a call from an old friend of mine who is now with XIV (or IBM or whatever). But I’m reasonably comfortable with what I said.
Now we have “thestorageanarchist” chiming in and among other things he brings up the notion of “Hitachi Math” as a yardstick to compare what XIV is doing with their performance claims. I had to chuckle at that one.
What is at root here is that we (HDS) periodically claim that we can do millions of IOPS (4.5, to be exact, or close to ballpark-ish). But before I go further into this, I should explain what the number is.
Firstly, there is a test that all us vendors do (from an engineering perspective). And if you delve into each vendor’s website you’ll generally find this number of max-IOPS. It is a simple test of reading the same 512-byte block from cache over and over again. No disk drive is ever harmed in this test. Really. But we don’t just do it for fun; it has its purpose. We model in advance how fast the front end can perform and this test just validates that model. It also gives us information in how to architect the array (bandwidth, processors, etc.). It’s kinda like putting a car on the rack to see how fast you can make the wheels spin.
So it’s not just a number to get bragging rights. But like many things engineering, stuff does make it into marketing material. And it actually does give you a relative and very rough comparison of how an array might scale. As in a 1-million IOPS array might just scale better than one with half the IOPS. But not necessarily, which is why we’ve been slogging through TPC and SPC benchmarks for years, since they are more representative of real workloads. So “Hitachi Math” or not, it’s pretty standard. We publish ours because it’s higher than anyone else’s. Do you blame us?
So what about SPC benchmarks? Yes, they are valid and do give you a much better understanding of array performance, and so that you don’t have to slop through a 467-page document, the final SPC-1 number for the USP-V is 200,245.73 IOPS and if someone can explain to me what a .73 IO is I’ll be eternally grateful. “Hitachi Math” at its finest!! Or how about calling it “EMC Math” since they won’t join the SPC. So all of their numbers are theoretical!! I’m still waiting for an explanation (other than the obvious one) as to why EMC won’t join. But I digress……
What is new here is learning that XIV might be spoofing iometer to obtain false results. It’s certainly not a new technique and I’ve seen a number of network-based virtualization companies a few years ago that were doing the same thing. Not cool at all, if it’s true. I hope IBM reads the referenced blog to either (1) refute it with detail, and/or (2) stop the practice. It really makes us all look bad. Although you have to admit it’s pretty clever!
So, yes, the max-IOPS number is real and can be validated and if someone wants to do that, I would suggest you have way too much time on your hands. In this job market that’s not advisable. But what’s really real is what performance you get on your own workloads. We all know that, and if you don’t know that then get out of the way of people that do know that. That’s all that matters and the SPC-1 benchmarks are guidelines to that.
Anyway, it should be interesting to see how this story plays out. As a vendor I actually hope the XIV story is not true….but stay tuned!
Comments (2 )
Good to have someone in Hitachi that sort of understands what they are posting on their blog.
However, not all vendors publish 512byte read cache hits, or even super read cache hits – and we (IBM) certainly don’t make such a song and dance about it. As for the highest number, well I’d have to disagree there. But then we don’t publish it cause it is kinda meaningless – I agree it does show how good the top end of the box is. But there is so much more, like adding thin provisioning – sorry Dynamic Provisioning. Do Hitachi plan to join us and submit a thin provisioned SPC-1 benchmark? 274k was last years line in the sand – just wait for this years…
I pointed several people at BarryB’s blog when I read it and have it on good authority that this has not been the practice when running XIV benchmarks. If you happen to read a block that has not yet been allocated (as with any thin provisioned solution) then the software will use the equivalent of /dev/zero. However, with XIV having proportionally very large cache capacity compared with most storage controllers. i.e. each 16 SATA drives have their own dedicated 8GB of cache, it means XIV can be very ‘prolific’ with its pre-fetch, and has lots of space to cache random writes. Thus giving much higher performance than would be expected from 16 SATA drives.
Just like USP and SVC wide striping, you can get more from less in a virtual world.
Thanks for the notice, Claus!
But that post of mine that you referenced is hardly “chiming in” – you might want to review some of my earlier posts to get a more complete picture of the various “hidden secrets” of XIV: