Big Data – It’s not the size of your source but the size of the insight that really matters
by Michael Hay on Nov 27, 2012
Over the past couple of weeks I’ve heard some great quotes from acquaintances and colleagues. Ron Lee from my team recently attended a CIO breakfast in Asia Pacific (APAC) where he heard the following from a CIO: “I’m not worried about my Big Data; I’m really worried about my Little Data.” Also, in a recent interaction with a Hitachi partner we explored the broad usage of smaller data sets to help in organizational and interior architectural design. There, one of the two data scientists we met with (he holds a Ph.D. in Psychology/Cognitive Science) had the following to say: “For years I’ve been developing islands of specialty in vast seas of ignorance.”
These discussions got me thinking…
I think that CIO in APAC is on to something. Big Data is not just about mining big data sources – big insight can also come from smaller data sets that just haven’t been tapped yet. The comments from the data scientist about democratization and extreme usability also resonated with me. The real prize here is making insight and analysis as ubiquitous and easy to use and collaborate around as, say, email.
So though the de facto definitions for Big Data seem to revolve around the 3 Vs –volume, variety and velocity–I think there is another angle. Here is my proposed definition of what Big Data will mean in the future.
Big Data of the Future – At scale agile processes realized by multidiscipline teams leveraging a variety of data categories and types flowed through various technologies, including provisions for security and privacy. The end result – timely discovery of sparks of insights leading to valuable innovation and knowledge.
Putting your business under the microscope
At Hitachi we are already starting to realize this vision through a number of really innovative projects – one of these is called the Business Microscope. You may have already seen our announcements about how it is used to deliver new insight in retail stores and call centers. Check out my previous blog post for more information. These efforts are powered by “the continuous measurement of human behaviors(1)” using a variety of sensors connected to and near human beings in their environment.
Traffic flow at a retail store goes under the microscope
In the retail example the technology was utilized in a ”home center” store. Over a period of 10 days they used the Business Microscope to track and analyze customer service activity, the standing locations of employees throughout the store floor and the impact on customer flow. By analyzing the results and appropriately repositioning shop floor staff, they were able to achieve a 15% average increase in sales per customer.
That’s a big insight for the retailer from a relatively small amount of previously untapped data!
I’ve included some images here of the traffic patterns and you can see clearly the impact of a few subtle changes that would have been unrealized without this type of analysis.
So, certainly, the output and resulting recommendations point to valuable insight and knowledge: repositioning a sales team to the right location for 16 seconds or more results in a 15% increase in average sales per customer.
In my mind this certainly matches a key part of our proposed Big Data definition in that there was a multidiscipline team behind it. In this instance the team comprised a management consulting firm, Hitachi’s own Ph.D. grade data scientists, along with customers, sales clerks, and temporarily employed staff to collect the data.
There was also a wide variety of different data sources involved in the analysis and different analytic engines to process all the point of sale and sensor data before we arrived at the visualization shown above. This was not as clear cut a process as traditional database transactional style analytics.
What about security and privacy issues of data? In our retail example we were very cognizant of those. Our data scientists K. Yano and N. Moriwaki along with the other partners involved in the study employed techniques like data anonymization, network and file encryption and limit collection duration to ease privacy fears, and to ensure compliance with security and privacy policies at the customer’s site.
Surprising insight when call center sales performance also goes under the microscope
We leveraged the Business Microscope on a second project this July with MOSHI MOSHI HOTLINE, Inc. in Japan to identify what impacts call center sales performance. The results of the analysis were pretty enlightening and dare I say surprising too. Two call centers participated in the study with 51 telemarketers at one location and 79 at the other. A variety of data was collected from face-to-face interactions between employees and supervisors to sensors monitoring body movement of the employees as they went about their day. The goal was to determine what factors directly correlated with the ”order receipt rate” of the telemarketers. If one were to second guess it would be fair to assume that skill level was the primary factor influencing sales success but the analysis showed that the degree of activity during breaks had the biggest impact on sales. Surprise!
Armed with this new insight, the team worked to test their theory by picking a group of telemarketers of the same age and testing sales results over a 3 week period when they were on breaks together versus separately. The result – when they took breaks together their activity increased and so did their sales–by approximately 13%.
So size really doesn’t matter
This is another great example of how obscure patterns in small amounts of data can lead to big insights and big business impact. Both these examples dealt with relatively small scale projects, with limited investment and small data sets in the 10s of gigabytes . Of course because Hitachi has been doing experiments like this with the Business Microscope for years the overall amount of data analyzed runs closer to the billions of data items across a wide number of projects but the point here is that Big Data doesn’t have to mean big project, big volumes of data, or big cost. The big refers to insight and sometimes it’s the little things that make all the difference.
1. K. Yano et al., “Measurement of Human Behavior: Creating a Society for Discovering Opportunities,” April, 2009
[...] Engineer at the Disk Array Systems Division (nice short job title there). In his post “Big Data – It’s not the size of your source but the size of the insight that really matters” (another short and snappy title), Michael makes the point that Big Data doesn’t [...]