Order From Chaos: How Big Data Will Change the World

Order From Chaos: How Big Data Will Change the World

Harnessing the exponential surge in data creates big opportunities.

Thanks to Purefunds Big Data ETF (BDAT) for helping us put this together.

IBM estimates that each day, 2.5 quintillion bytes of data are created or replicated. That’s the equivalent of a million hard drives filling up with data every hour.

The current volume of data created is substantial: it is so much that 90% of the world’s data has been created in the last two years. However, the amount of information today pales in comparison to what our future holds, as the rate at which data is created is accelerating exponentially.

It’s for this reason that The Economist estimates that there will be roughly 7x more data in 2020 than there was in 2014.

Where Does Big Data Come From?

Big data comes from both internal and external sources. Internally, millions of old documents and records are scanned and archived by businesses. Most of the time, no detailed analytics have ever been run on this information. Externally, the public web offers millions of data sets published for public consumption by government, economic, census, and other sources.

There’s also a broad spectrum of data that exists that can be a part of both of these categories: social media posts, documents, emails, business applications, machine log data, media, and sensor data can all be collected, processed, and analyzed. To get a sense of the extent of this information, here’s what is created every hour just from social media and email: 72 hours of video uploaded to Youtube, 4 million search inquiries on Google, 200 million emails sent, 2.5 million shares on Facebook, and 300,000 tweets made.

Big Data = Big Opportunities for Business

With proper analysis, Big Data can lead to new understandings of consumer behaviour, better management decisions, new innovations, and improved risk management. However, there are big challenges in making use of so much information.

• Too much data creates an information overload.
• Organizing and storing all of this data can be problematic.
• Companies don’t know how to use all of this data to create insight.

To organize and make sense of it all, data scientists use the three V’s to describe Big Data.

Volume is the scale at which data is created, and includes the massive amounts of information derived from phones, internet users, machine logs, and internet of things.

Velocity is the analysis of streaming data: for example, modern cars have 100 sensors that monitor different systems in real-time.

Variety is the different forms of data, and it reflects the fact that data comes in all shapes and forms. Finding a way to harmonize multiple types of data can be quite a challenge. Research finds that organizations spend up to 80% of their time modelling and preparing data, rather than actually gaining insight.

Let’s see how companies have been able to use Big Data to create opportunity.

Case Studies of Big Data

Macy’s adjusts pricing in near-real time for 73 million items based on demand and inventory.

American Express developed predictive models that analyze historical transactions and 115 variables to forecast the loyalty of customers. Using this data, they can see if customers may be potentially closing their accounts in the near future. Launching a pilot program in Australia, the company can now identify 24% of accounts in the country that will close in the next four months.

Walmart built a new search engine for their website that includes semantic data relying on text analysis, machine learning, and even synonym mining to create better search results. Online shoppers have been more likely to complete purchases as a result by 10% to 15%, increasing revenue by billions.

Los Angeles and Santa Cruz police departments have used an algorithm that is typically used to predict earthquakes, now using it to look at crime data. The software can predict where crimes are likely to occur down to 500 square feet. In areas the software is being used, there has been a 33% reduction in burglaries and a 21% reduction in violent crimes.

Big Market

Today’s data centers occupy the land to equivalent to almost 6,000 football fields. By 2020, the amount of digital information is expected to increase exponentially to more than 7x of what it is today.

In healthcare alone, Big Data is expected to eventually save $300 billion per year in healthcare analytics. Retailers may increase margins up to 60% through Big Data analytics.

“Information is the oil of the 21st century, and analytics is the combusion engine.” – Peter Sondergaard, Gartner Research.