Unlocking Big Data in Social Technologies

The display at EMC Corp.’s booth at this week’s Oracle OpenWorld show in San Francisco featured a famous quote uttered by a British entrepreneur in 2006: “Data is the new oil.”

The quote was being bandied about to promote an ambitious global project called “The Human Face of Big Data,” an effort commissioned by EMC, and sponsored by the likes of Cisco Systems and VMware, that aims to use crowdsourcing to get a handle on humanity’s increasing need to generate and crunch data. For example, a widely distributed smartphone application collected data, between Sept. 25 and Oct. 2, that indicated that the reason people can’t find a cab when it rains in Singapore is that drivers looking to avoid having their pay withheld for accidents simply pull over to wait out rainstorms. They don’t pick up new fares.

While such findings may not hold much value for the average IT executive, the implications of big data certainly do. And although the news from OpenWorld centered on Oracle’s slew of new cloud services and a new platform that socially enables all of the company’s applications, big data was clearly the dominant theme.

Oracle CEO Larry Ellison’s anticipated keynote address, which was entitled “The Oracle Cloud: Where Social is Built In,” focused instead on how the company’s venerable database and analytics technologies can crunch the big data inherent in social network streams.

Ellison began his keynote touting Oracle’s cloud — which now features new services such as planning and budgeting, financial reporting, and data and insight — as having the broadest set of applications in the industry. He then quickly introduced Oracle’s new social platform, which he characterized as being far preferable to stand-alone social applications.

But what he clearly wanted to demonstrate was the kind of insight that can be gleaned from social data when the right analytical tools are used. Specifically, he showed the packed hall how two products — Oracle’s Exadata database and its Exalytics in-memory analytics appliance — were used to analyze nearly 5 billion Twitter posts to determine what celebrity would be the best spokesperson to promote a new Lexus sedan.

Ellison made it clear that Twitter data, in particular, consists of much more than the posts themselves — it includes timestamps, geotags, device types, and more, and the data is of both the structured and unstructured variety. In the end, Oracle ended up analyzing 27 billion relationships, nearly a billion retweets and hashtags, 2.8 billion mentions and another 1.3 billion replies.

And as Ellison pointed out, the conclusion itself — that gold-medal Olympic gymnast Gabby Douglas was the best fit to promote the new Lexus — wasn’t nearly as significant as the process by which that conclusion was reached, which included drilling down into the data to find out whose posts most frequently mentioned cars, for instance.

“This was a very simple question that required an enormous amount of data processing to get the data,” Ellison said. “This is something we would have had to guess at before.”

Now that sophisticated data crunching tools from Oracle, EMC and the like are making it possible to extract the value of big data, companies have no choice but to try and use that data to change their business.

“Otherwise,” said EMC CEO Joe Tucci during a morning OpenWorld keynote Oct. 2, “they’ll be out of business.”