The Storage Muse

For new applications I think that we are seeing, largely because of AWS, that moving away from clustered or traditional file systems makes perfect sense.  At scale companies like Facebook, Yahoo, Google, etc. are finding that new approaches are mandatory.  However, I caution against saying that that file systems are dead.  There are a number of reasons for this and the most profound, in my opinion, is that we find artifacts of the file system in everyday objects like a Sony Playstation 3 when it displays photos on a photo CD in the form of folders.  This comes from file system interfaces for users, from developers to office workers, being well understood and tuned to the language of the various end users.  After all the verbs open, read, write, close, etc. on files are actions that make sense not only for a developer using a POSIX style file system API, but also for a secretary filing an expense report.  What this means is that the user experience for the file system cannot go away, while the implementations might shift over time.  However,  as I pointed out in my previous post on the new Google File System and clustered file systems, there has been a lot of money spent on getting file systems to be the end-all-be-all for every workload across all scaling sizes and geographic distance.  What we have a found out, as an industry, that this logic does does not work, except of course that NetApp believes AWFL, err WAFL solves all of this problems already.  Cloud issues like distributed lock management across many nodes, or support for millions of files, or single namespace across multiple geographic sites mean that we cannot just take file systems that are venerable today and apply them directly to these kinds of cloud problems.  Instead I would suggest that existing file systems such as xfs or btrfs and typical file system experiences are merely ingredients within an overall solution at cloud scale.  For instance I might have a solution which uses Hadoop on top, but then on each individual node xfs is utilized so that the storage of the file objects is highly optimized for features like thin provisioning.  Saying that the day of file systems are over is a bit too soon in my opinion.