Order From Chaos: How Big Data Will Change the World
Harnessing the exponential surge in data creates big opportunities.
Thanks to Purefunds Big Data ETF (BDAT) for helping us put this together.
IBM estimates that each day, 2.5 quintillion bytes of data are created or replicated. That’s the equivalent of a million hard drives filling up with data every hour.
The current volume of data created is substantial: it is so much that 90% of the world’s data has been created in the last two years. However, the amount of information today pales in comparison to what our future holds, as the rate at which data is created is accelerating exponentially.
It’s for this reason that The Economist estimates that there will be roughly 7x more data in 2020 than there was in 2014.
Where Does Big Data Come From?
Big data comes from both internal and external sources. Internally, millions of old documents and records are scanned and archived by businesses. Most of the time, no detailed analytics have ever been run on this information. Externally, the public web offers millions of data sets published for public consumption by government, economic, census, and other sources.
There’s also a broad spectrum of data that exists that can be a part of both of these categories: social media posts, documents, emails, business applications, machine log data, media, and sensor data can all be collected, processed, and analyzed. To get a sense of the extent of this information, here’s what is created every hour just from social media and email: 72 hours of video uploaded to Youtube, 4 million search inquiries on Google, 200 million emails sent, 2.5 million shares on Facebook, and 300,000 tweets made.
Big Data = Big Opportunities for Business
With proper analysis, Big Data can lead to new understandings of consumer behaviour, better management decisions, new innovations, and improved risk management. However, there are big challenges in making use of so much information.
- Too much data creates an information overload.
- Organizing and storing all of this data can be problematic.
- Companies don’t know how to use all of this data to create insight.
To organize and make sense of it all, data scientists use the three V’s to describe Big Data.
Volume is the scale at which data is created, and includes the massive amounts of information derived from phones, internet users, machine logs, and internet of things.
Velocity is the analysis of streaming data: for example, modern cars have 100 sensors that monitor different systems in real-time.
Variety is the different forms of data, and it reflects the fact that data comes in all shapes and forms. Finding a way to harmonize multiple types of data can be quite a challenge. Research finds that organizations spend up to 80% of their time modelling and preparing data, rather than actually gaining insight.
Let’s see how companies have been able to use Big Data to create opportunity.
Case Studies of Big Data
Macy’s adjusts pricing in near-real time for 73 million items based on demand and inventory.
American Express developed predictive models that analyze historical transactions and 115 variables to forecast the loyalty of customers. Using this data, they can see if customers may be potentially closing their accounts in the near future. Launching a pilot program in Australia, the company can now identify 24% of accounts in the country that will close in the next four months.
Walmart built a new search engine for their website that includes semantic data relying on text analysis, machine learning, and even synonym mining to create better search results. Online shoppers have been more likely to complete purchases as a result by 10% to 15%, increasing revenue by billions.
Los Angeles and Santa Cruz police departments have used an algorithm that is typically used to predict earthquakes, now using it to look at crime data. The software can predict where crimes are likely to occur down to 500 square feet. In areas the software is being used, there has been a 33% reduction in burglaries and a 21% reduction in violent crimes.
Today’s data centers occupy the land to equivalent to almost 6,000 football fields. By 2020, the amount of digital information is expected to increase exponentially to more than 7x of what it is today.
In healthcare alone, Big Data is expected to eventually save $300 billion per year in healthcare analytics. Retailers may increase margins up to 60% through Big Data analytics.
“Information is the oil of the 21st century, and analytics is the combusion engine.” – Peter Sondergaard, Gartner Research.
The 20 Internet Giants That Rule the Web
A lot has changed since Yahoo and AOL were the homepages of choice. This visualization looks at the largest internet giants in the U.S. since 1998.
The 20 Internet Giants That Rule the Web (1998-Today)
With each passing year, an increasingly large segment of the population no longer remembers images loading a single pixel row at a time, the earsplitting sound of a 56k modem, or the early domination of web portals.
Many of the top websites in 1998 were news aggregators or search portals, which are easy concepts to understand. Today, brand touch-points are often spread out between devices (e.g. mobile apps vs. desktop) and a myriad of services and sub-brands (e.g. Facebook’s constellation of apps). As a result, the world’s biggest websites are complex, interconnected web properties.
The visualization above, which primarily uses data from ComScore’s U.S. Multi-Platform Properties ranking, looks at which of the internet giants have evolved to stay on top, and which have faded into internet lore.
America Moves Online
For millions of curious people the late ’90s, the iconic AOL compact disc was the key that opened the door to the World Wide Web. At its peak, an estimated 35 million people accessed the internet using AOL, and the company rode the Dotcom bubble to dizzying heights, reaching a valuation of $222 billion dollars in 1999.
AOL’s brand may not carry the caché it once did, but the brand never completely faded into obscurity. The company continually evolved, finally merging with Yahoo after Verizon acquired both of the legendary online brands. Verizon had high hopes for the company—called Oath—to evolve into a “third option” for advertisers and users who were fed up with Google and Facebook.
Sadly, those ambitions did not materialize as planned. In 2019, Oath was renamed Verizon Media, and was eventually sold once again in 2021.
A City of Gifs and Web Logs
As internet usage began to reach critical mass, web hosts such as AngelFire and GeoCities made it easy for people to create a new home on the Web.
GeoCities, in particular, made a huge impact on the early internet, hosting millions of websites and giving people a way to actually participate in creating online content. If it were a physical community of “home” pages, it would’ve been the third largest city in America, after Los Angeles.
This early online community was at risk of being erased permanently when GeoCities was finally shuttered by Yahoo in 2009, but luckily, the nonprofit Internet Archive took special efforts to create a thorough record of GeoCities-hosted pages.
From A to Z
In December of 1998, long before Amazon became the well-oiled retail machine we know today, the company was in the midst of a massive holiday season crunch.
In the real world, employees were pulling long hours and even sleeping in cars to keep the goods flowing, while online, Amazon.com had become one of the biggest sites on the internet as people began to get comfortable with the idea of purchasing goods online. Demand surged as the company began to expand their offering beyond books.
Amazon.com has grown to be the most successful merchant on the Internet.
– New York Times (1998)
Digital Magazine Rack
Meredith will be an unfamiliar brand to many people looking at today’s top 20 list. While Meredith may not be a household name, the company controlled many of the country’s most popular magazine brands (People, AllRecipes, Martha Stewart, Health, etc.) including their sizable digital footprints. The company also owned a slew of local television networks around the United States.
After its acquisition of Time Inc. in 2017, Meredith became the largest magazine publisher in the world. Since then, however, Meredith has divested many of its most valuable assets (Time, Sports Illustrated, Fortune). In December 2021, Meredith merged with IAC’s Dotdash.
When people have burning questions, they increasingly turn to the internet for answers, but the diversity of sources for those answers is shrinking.
Even as recently as 2013, we can see that About.com, Ask.com, and Answers.com were still among the biggest websites in America. Today though, Google appears to have cemented its status as a universal wellspring of answers.
As smart speakers and voice assistants continue penetrate the market and influence search behavior, Google is unlikely to face any near-term competition from any company not already in the top 20 list.
New Kids on the Block
Social media has long since outgrown its fad stage and is now a common digital thread connecting people across the world. While Facebook rapidly jumped into the top 20 by 2007, other social media infused brands took longer to grow into internet giants.
By 2018, Twitter, Snapchat, and Facebook’s umbrella of platforms were all in the top 20, and you can see a more detailed and up-to-date breakdown of the social media universe here.
A Tangled Web
Today’s internet giants have evolved far beyond their ancestors from two decades ago. Many of the companies in the top 20 run numerous platforms and content streams, and more often than not, they are not household names.
A few, such as Mediavine and CafeMedia, are services that manage ads. Others manage content distribution, such as music, or manage a constellation of smaller media properties, as is the case with Hearst.
Lastly, there are still the tech giants. Remarkably, three of the top five web properties were in the top 20 list in 1998. In the fast-paced digital ecosystem, that’s some remarkable staying power.
This article was inspired by an earlier work by Philip Bump, published in the Washington Post.
Visualizing the Power of the World’s Supercomputers
Supercomputers are some of the most advanced machines humans have ever created. See how they stack up in this infographic.
Visualizing the Power of the World’s Supercomputers
A supercomputer is a machine that is built to handle billions, if not trillions of calculations at once. Each supercomputer is actually made up of many individual computers (known as nodes) that work together in parallel.
A common metric for measuring the performance of these machines is flops, or floating point operations per second.
In this visualization, we’ve used November 2021 data from TOP500 to visualize the computing power of the world’s top five supercomputers. For added context, a number of modern consumer devices were included in the comparison.
Ranking by Teraflops
Because supercomputers can achieve over one quadrillion flops, and consumer devices are much less powerful, we’ve used teraflops as our comparison metric.
1 teraflop = 1,000,000,000,000 (1 trillion) flops.
|#1||🇯🇵 Supercomputer Fugaku||Supercomputer||537,212|
|#4||🇨🇳 Sunway Taihulight||Supercomputer||125,436|
|n/a||Nvidia Titan RTX||Consumer device||130|
|n/a||Nvidia GeForce RTX 3090||Consumer device||36|
|n/a||Xbox Series X||Consumer device||12|
|n/a||Tesla Model S (2021)||Consumer device||10|
Supercomputer Fugaku was completed in March 2021, and is officially the world’s most powerful supercomputer. It’s used for various applications, including weather simulations and innovative drug discovery.
Sunway Taihulight is officially China’s top supercomputer and fourth most powerful in the world. That said, some experts believe that the country is already operating two much more powerful systems, based on data from anonymous sources.
As you can see, the most advanced consumer devices do not come close to supercomputing power. For example, it would take the combined power of 4,000 Nvidia Titan RTX graphics cards (the most powerful consumer card available) to measure up to the Fugaku.
One of China’s unrevealed supercomputers is supposedly named Oceanlite, and is a successor to Sunway Taihulight. It’s believed to have reached 1.3 exaflops, or 1.3 quintillion flops. The following table makes it easier to follow all of these big numbers.
In the U.S., rival chipmakers AMD and Intel have both won contracts from the U.S. Department of Energy to build exascale supercomputers. On the AMD side, there’s Frontier and El Capitan, while on the Intel side, there’s Aurora.
Also involved in the EL Capitan project is Hewlett Packard Enterprise (HPE), which claims the supercomputer will be able to reach 2 exaflops upon its completion in 2023. All of this power will be used to support several exciting endeavors:
- Enable advanced simulation and modeling to support the U.S. nuclear stockpile and ensure its reliability and security.
- Accelerate cancer drug discovery from six years to one year through a partnership with pharmaceutical company, GlaxoSmithKline
- Understand the dynamic and mutations of RAS proteins that are linked to 30% of human cancers
Altogether, exascale computing represents the ability to conduct complex analysis in a matter of seconds, rather than hours. This could unlock an even faster pace of innovation.
Money4 weeks ago
Visualizing the $94 Trillion World Economy in One Chart
Misc1 week ago
From Greek to Latin: Visualizing the Evolution of the Alphabet
Best of3 weeks ago
Our Top 21 Visualizations of 2021
Markets2 weeks ago
Prediction Consensus: What the Experts See Coming in 2022
Technology2 weeks ago
Companies Gone Public in 2021: Visualizing IPO Valuations
Misc3 weeks ago
Mapped: Top Trending Searches of 2021 in Every U.S. State
Green3 weeks ago
Mapped: 30 Years of Deforestation and Forest Growth, by Country
Technology15 hours ago
The 20 Internet Giants That Rule the Web