Order From Chaos: How Big Data Will Change the World
Harnessing the exponential surge in data creates big opportunities.
Thanks to Purefunds Big Data ETF (BDAT) for helping us put this together.
IBM estimates that each day, 2.5 quintillion bytes of data are created or replicated. That’s the equivalent of a million hard drives filling up with data every hour.
The current volume of data created is substantial: it is so much that 90% of the world’s data has been created in the last two years. However, the amount of information today pales in comparison to what our future holds, as the rate at which data is created is accelerating exponentially.
It’s for this reason that The Economist estimates that there will be roughly 7x more data in 2020 than there was in 2014.
Where Does Big Data Come From?
Big data comes from both internal and external sources. Internally, millions of old documents and records are scanned and archived by businesses. Most of the time, no detailed analytics have ever been run on this information. Externally, the public web offers millions of data sets published for public consumption by government, economic, census, and other sources.
There’s also a broad spectrum of data that exists that can be a part of both of these categories: social media posts, documents, emails, business applications, machine log data, media, and sensor data can all be collected, processed, and analyzed. To get a sense of the extent of this information, here’s what is created every hour just from social media and email: 72 hours of video uploaded to Youtube, 4 million search inquiries on Google, 200 million emails sent, 2.5 million shares on Facebook, and 300,000 tweets made.
Big Data = Big Opportunities for Business
With proper analysis, Big Data can lead to new understandings of consumer behaviour, better management decisions, new innovations, and improved risk management. However, there are big challenges in making use of so much information.
- Too much data creates an information overload.
- Organizing and storing all of this data can be problematic.
- Companies don’t know how to use all of this data to create insight.
To organize and make sense of it all, data scientists use the three V’s to describe Big Data.
Volume is the scale at which data is created, and includes the massive amounts of information derived from phones, internet users, machine logs, and internet of things.
Velocity is the analysis of streaming data: for example, modern cars have 100 sensors that monitor different systems in real-time.
Variety is the different forms of data, and it reflects the fact that data comes in all shapes and forms. Finding a way to harmonize multiple types of data can be quite a challenge. Research finds that organizations spend up to 80% of their time modelling and preparing data, rather than actually gaining insight.
Let’s see how companies have been able to use Big Data to create opportunity.
Case Studies of Big Data
Macy’s adjusts pricing in near-real time for 73 million items based on demand and inventory.
American Express developed predictive models that analyze historical transactions and 115 variables to forecast the loyalty of customers. Using this data, they can see if customers may be potentially closing their accounts in the near future. Launching a pilot program in Australia, the company can now identify 24% of accounts in the country that will close in the next four months.
Walmart built a new search engine for their website that includes semantic data relying on text analysis, machine learning, and even synonym mining to create better search results. Online shoppers have been more likely to complete purchases as a result by 10% to 15%, increasing revenue by billions.
Los Angeles and Santa Cruz police departments have used an algorithm that is typically used to predict earthquakes, now using it to look at crime data. The software can predict where crimes are likely to occur down to 500 square feet. In areas the software is being used, there has been a 33% reduction in burglaries and a 21% reduction in violent crimes.
Today’s data centers occupy the land to equivalent to almost 6,000 football fields. By 2020, the amount of digital information is expected to increase exponentially to more than 7x of what it is today.
In healthcare alone, Big Data is expected to eventually save $300 billion per year in healthcare analytics. Retailers may increase margins up to 60% through Big Data analytics.
“Information is the oil of the 21st century, and analytics is the combusion engine.” – Peter Sondergaard, Gartner Research.
The Game of Life: Visualizing China’s Social Credit System
This infographic explores how China’s proposed social credit system will monitor and surveil citizens, and how it’ll be used to reward or punish them.
The Game of Life: Visualizing China’s Social Credit System
In an attempt to imbue trust, China has announced a plan to implement a national ranking system for its citizens and companies. Currently in pilot mode, the new system will be rolled out in 2020, and go through numerous iterations before becoming official.
While the system may be a useful tool for China to manage its growing 1.4 billion population, it has triggered global concerns around the ethics of big data, and whether the system is a breach of fundamental human rights.
Today’s infographic looks at how China’s proposed social credit system could work, and what the implications might be.
The Government is Always Watching
Currently, the pilot system varies from place to place, whereas the new system is envisioned as a unified system. Although the pilot program may be more of an experiment than a precursor, it gives a good indication of what to expect.
In the pilot system, each citizen is assigned 1,000 points and is consistently monitored and rated on how they behave. Points are earned through good deeds, and lost for bad behavior. Users increase points by donating blood or money, praising the government on social media, and helping the poor. Rewards for such behavior can range from getting a promotion at work fast-tracked, to receiving priority status for children’s school admissions.
In contrast, not visiting one’s aging parents regularly, spreading rumors on the internet, and cheating in online games are considered antisocial behaviors. Punishments include public shaming, exclusion from booking flights or train tickets, and restricted access to public services.
Big Data Goes Right to the Source
The perpetual surveillance that comes with the new system is expected to draw on huge amounts of data from a variety of traditional and digital sources.
Police officers have used AI-powered smart glasses and drones to effectively monitor citizens. Footage from these devices showing antisocial behavior can be broadcast to the public to shame the offenders, and deter others from behaving similarly.
For more serious offenders, some cities in China force people to repay debts by switching the person’s ringtone without their permission. The ringtone begins with the sound of a police siren, followed by a message such as:
“The person you are calling has been listed as a discredited person by the local court. Please urge this person to fulfill his or her legal obligations.”
Two of the largest companies in China, Tencent and Alibaba, were enlisted by the People’s Bank of China to play an important role in the credit system, raising the issue of third-party data security. WeChat—China’s largest social media platform, owned by Tencent—tracked behavior and ranked users accordingly, while displaying their location in real-time.
Following data concerns, these tech companies—and six others—were not awarded any licenses by the government. However, social media giants are still involved in orchestrating the public shaming of citizens who misbehave.
The Digital Dang’an
The social credit system may not be an entirely new initiative in China. The dang’an (English: record) is a paper file containing an individual’s school reports, information on physical characteristics, employment records, and photographs.
These dossiers, which were first used in the Maoist years, helped the government in maintaining control of its citizens. This gathering of citizen’s data for China’s social credit system may in fact be seen as a revival of the principle of dang’an in the digital era, with the system providing a powerful tool to monitor citizens whose data is more difficult to capture.
Is the System Working?
In 2018, people with a low score were prohibited from buying plane tickets almost 18 million times, while high-speed train ticket transactions were blocked 5.5 million times. A further 128 people were prohibited from leaving China, due to unpaid taxes.
The system could have major implications for foreign business practices—as preference could be given to companies already ranked in the system. Companies with higher scores will be rewarded with incentives which include lower tax rates and better credit conditions, with their behavior being judged in areas such as:
- Paid taxes
- Customs regulation
- Environmental protection
Despite the complexities of gathering vast amounts of data, the system is certainly making an impact. While there are benefits to having a standardized scoring system, and encouraging positive behavior—will it be worth the social cost of gamifying human life?
How Many Music Streams Does it Take to Earn a Dollar?
Streaming has breathed new life into the music business, but as new data shows, these services pay out wildly different rates per stream.
How Many Music Streams Does it Take to Earn a Dollar?
A decade ago, the music industry was headed for a protracted fade-out.
The disruptive effects of peer-to-peer file sharing had slashed music revenues in half, casting serious doubts over the future of the industry.
Ringtones provided a brief earnings bump, but it was the growing popularity of premium streaming services that proved to be the savior of record labels and artists. For the first time since the mid-90s, the music industry saw back-to-back years of growth, and revenues grew a brisk 12% in 2018 – nearly reaching $10 billion. In short, people showed they were still willing to pay for music.
Although most forecasts show streaming services like Spotify and Apple Music contributing an increasingly large share of revenue going forward, recent data from The Trichordist reveals that these services pay out wildly different rates per stream.
Note: Due to the lack of publicly available data, calculating payouts from streaming services is not an exact science. This data set is based on revenue from an indie label with a ~150 album catalogue generating over 115 million streams.
Full Stream Ahead
One would expect streaming services to have fairly similar payout rates every time a track is played, but this is not the case. In reality, the streaming rates of major players in the market – which have very similar catalogs – are all over the map. Below is a full breakdown of how many streams it takes to earn a dollar on various platforms:
|Streaming service||Avg. payout per stream||# of streams to earn one dollar||# of streams to earn minimum wage*|
|Google Play Music||$0.00676||147||217,751|
*U.S. monthly minimum wage of $1,472 **Premium tier
Napster, once public enemy number one in the music business, has some of the most generous streaming rates in the industry. On the downside, the brand currently has a market share of less than 1%, so getting a high volume of plays on an album isn’t likely to happen for most artists.
On the flip side of the equation, YouTube has the highest number of plays per song, but the lowest payout per stream by far. It takes almost 1,500 plays to earn a single dollar on the Google-owned video platform.
Spotify, which is now the biggest player in the streaming market, is on the mid-to-low end of the compensation spectrum.
The Payment Pipeline
How do companies like Spotify calculate the amount paid out to license holders? Here’s a look at their payout process:
As this chart reveals, dollars earned from streaming still don’t tell the full story of how much artists receive at the end of the line. This amount is influenced by whether or not the performer has a record deal, and if other contributors have a stake in the recorded work.
The Pressure is Heating Up
When Spotify was a scrappy startup providing a much needed revenue stream to the music industry, labels were temporarily willing to accept lower streaming rates.
But now that Spotify is a public company, and tech giants like Apple and Amazon are in the picture, a growing chorus of industry players will likely dial up the pressure to increase compensation rates.
Markets8 months ago
The Jeff Bezos Empire in One Giant Chart
Maps11 months ago
Mercator Misconceptions: Clever Map Shows the True Size of Countries
Advertising7 months ago
Meet Generation Z: The Newest Member to the Workforce
Misc10 months ago
24 Cognitive Biases That Are Warping Your Perception of Reality
Advertising6 months ago
How the Tech Giants Make Their Billions
Technology9 months ago
The 20 Internet Giants That Rule the Web
Chart of the Week8 months ago
Chart: The World’s Largest 10 Economies in 2030
Environment7 months ago
The World’s 25 Largest Lakes, Side by Side