Race to the Bottom: Can Moore’s Law Keep up with Big Data?
by David Merrill on Jul 26, 2013
Moore’s law has been a stable predictor of density and price in the IT world for many decades. Initially used to describe transistor density as a function of time, it has been loosely applied to the price of IT, and for our purposes today the price of storage. Except for 2012 (with Tsunamis and flooding) we have enjoyed storage price erosion in the range of 20-25% per year for many, many years. Storage price erosion is a function of areal density and technology improvements, not necessary transistor density. The chart below (from IIEEE Transactions on Magnetics Vol. 48 May 2012) can give you a rough idea of the future for areal density of NAND, HDD and tape.
There are several technologies that will contribute to the improved areal density including:
- Shingled Magnetic Recording
- Bit Patterned Recording (BPR)
- Thermally-assisted Recording (TAR)
- BPR + TAR
The good news is that media density will be improving by 20%+ year on year, and we could correlate that to a continued price erosion for data storage. See below for projections on price per GB for different media types.
If you have followed my blog over the years, you know that I put more focus on total cost as opposed to price (price is only 12-15% of storage total cost), but in this analysis I want to focus on future prices. With price erosion roughly 20% per year matched with data growth at 20% per year, one could argue that CAPEX budgets would be roughly flat year on year, in that the increase capacity cost would be covered by price erosion. This simplified view is not really accurate, but for this discussion it demonstrates the parity of price erosion and organic capital growth.
Enter-now hyper growth with big data. A customer that I have worked with recently talked about rapid capacity increases that exceed 100% per year, even 400% in a year or 18 months. With this kind of growth rate, there will be a significant increase in CAPEX needed that cannot benefit from a20% price erosion. Dr. Howard Rubin has written on this topic, and I refer you to his concepts around the tipping point and where “the geometric growth rate of computing demand — technology intensity in the context of business and our personal lives — will drive computing costs past the point at which Moore’s Law will keep the costs manageable.”
This tipping point could not have come at a worse time, economically speaking. Companies do not have the stomach for large IT investments, even with all the analytic benefits of harvesting revenue and business value from big data. Seeing the requirement for significant IT investments in servers and particularly storage for moderate-to-long-term retention of data, there will be a pull back due to sticker shock. Hyper growth due to big data, without the accompanying price erosion in the midst of economic uncertainty does not make for a good recipe.
One of the solutions to this conundrum will be to get away from the CAPEX (purchase, lease, depreciate) tradition and consider consumption-based ownership models. Too many people confuse consumption options with a cloud delivery model, and they are two different methods. Many cloud delivery options do come with consumption or utility pricing, but my recommendation is to simply replace traditional ownership models with a consumption model. The characteristics of a consumption model are:
- You pay for what you use, when and if you actually use it
- Monthly rates can rise and fall based on utilization
- The storage (or other IT) assets are in your facility, behind your firewall
- You can still manage the assets if you want, but purchasing a remote management service gets into other parts of the total cost model
- The control of the assets and local management can be tight, secure and structured as if you owned the assets
- You no longer look at or care about end-of-asset-life events. The service provider should be responsible for upgrades, transformation and migrations
- The monthly bill can be passed directly to the organization consuming the resource.
- Even if you do not have a formal storage service catalog or chargeback system in place, this consumption approach brings that wrapped into the offering
- Since the utility rate is an operational expense, there can be benefits (and drawbacks) for not having these book assets around, and not burdening the procurement group with rapid quote-negotiate-purchase cycles in a hyper-growth period of time
All in all, the good news of price erosion mixed with the bad news that Moore’s law cannot keep up with hyper-growth scenarios leads to consideration of options that can meet the business and technical needs of the organization. Consumption methods are one option to consider that will help provide balance and alternatives in an era where business uncertainty should not derail technological advancements.