Techno Musings

Where is the huge inflection in solid state adoption that was hyped up?  The lack of extreme adoption is perhaps due to Apple consuming the World’s supply of flash technologies for iPod, iPhone, and soon iPad, but this is just mere speculation.  (Note there is a good article at Apple Insider that says Apple may ultimately be responsible for an increase in solid state pricing.)   However, what about the glut of flash storage that EMC has in inventory?  What does this mean?  I’ve heard some people state that this is due to the downturn in the economy and that the primary buyers of super high performance storage are well no longer buying.  However, all of this is merely speculation, already we know top tier storage providers, including Hitachi, now include solid state storage in their arrays as an option.  Personally, I think that in order to answer this question we have to first begin thinking about what workloads within an application are best suited to flash storage. To start there are a raft of articles on how MySpace is using flash technology here is one and an excerpt from said article.

“Consequently, interest moved to the backburner for about 18 months. In late 2008, MySpace talked to Fusion- io and Intel, which had introduced its new x25 Flash storage systems, about how Flash could improve its operation. Initially, the expectation was that MySpace could plug the technology in and it would speed up its applications, however, they found that performance improvements were workload dependent. “With a slight change in an application, workloads could see performance fluctuations of 10 to 1 or more,” said Buckingham.”

If you read the entire article, which I suggest you do, you will find that MySpace tuned a portion of their application stack to best use the flash technology.  By doing this they were able to remove many servers while still keeping the same level of performance, helping them reduce their overall application TCO.  One thing that I want to point out is that their interest in flash storage occurred first in 2007, but the price was too high to contemplate doing something.  With the constant erosion pressure on storage media MySpace was able to reconsider in 2009 — let’s hold onto this thought for later on in my post.

Over at the ANZ Search Storage site is a top 10 listing of where flash storage might be best applied.  Of all of the discussions about why flash has not caught on at large, I would argue that what MySpace has done is key to understanding the situation: tuning of the IT infrastructure, processes and expectations to use flash storage is a work in progress.  The simple fact of the matter is that developers of IT systems and the consumers in the organizations are still figuring out the best usage profile of flash storage.  Let’s face it the POSIX file system approach to storing data on mass storage has assumed spinning rust media for decades.  Not only that but IT organizations at large also are contemplating things in $/GB with $/IOP being something that a few organizations are flirting with.  Both of these examples mean two things: 1.)making the financial case for flash storage when $/GB is the prevailing way to look at mass storage is hard, and 2.) merely changing the media type within a server or storage device and crossing your fingers will not provide a magical boost in TCO or user satisfaction.  The MySpace example tells me that the reality of using flash technologies is still more art than science.  IT systems providers need to spend time learning about how to best apply and use flash storage and the IT organizations themselves need to learn how to justify the purchases of performance dense media including the adoption of new metrics.

With all of that said we have to start somewhere, and that somewhere is in the storage systems provided by top tier storage vendors — and even some that don’t need tiers because tiers are for crying and not storage.  Already as users you can find good places to make use of flash storage.  For instance if we look at the ZFS file system you will find that some portions of the file system can be moved onto flash storage for read acceleration with the remaining staying on regular spinning rust.  The key to this is knowing your storage I/O workload from the storage device perspective — note this last bit is important because I know of customers who were assuming that because there were thousands of small sequential transactions at the application layer the storage should have a small block sequential I/O profile, which is completely wrong.   A pretty good presentation on optimizing Postgres on Solaris/ZFS for solid state usage can be found here, I suggest you take a gander as it is pretty interesting.  Another company, RethinkDB, is developing a new storage engine for MySQL that is flash friendly and tries to remove the assumptions of spinning rust within the DBMS.  I’m sure that there are many other examples that can be cited, but this is enough for now to get my message across: namely the industry is just now becoming flash ready at large and it is the platform providers, the application developers, the users, and the user’s organizations that need to adjust for real mainstream adoption to occur.

Now back to the title is a pun about solid state adoption from the perspective of a summer time swimming pool game: Marco Polo.  I mention this because sometimes the market can be fickle and is always difficult to predict.  The very thing that could cause a spike in flash storage prices, Apple’s insatiable consumption, may ultimately be the thing that pushes flash adoption into the mainstream.  This may not have to do with getting the IT stacks and their wrapper organizations all ready for flash storage, but instead with pricing.  If pricing of flash storage meets or beats the pricing of SAS and FC technologies you can bet that this will further fuel the adoption.  Then all of the old organizational inertia will fall away because the $/GB metrics will make sense and in fact be more stilted towards flash — for instance why by the SAS device when it is more costly to operate and provides less performance.  (I suppose that this kind of organizational lore is similar to the misunderstanding that encryption is security and security is encryption.)

Also I am interested in any configurations, thoughts or general musings about SSDs.  So if you have any please speak up!

NOTE: The reference to the Storage Wrapin’ Show video is something that Barry Burke has on his blog and I reference as well.  A hearty thank you to Barry for finding this gem as it is worth a chuckle or two.