Connect with us

Technology

Charted: The Exponential Growth in AI Computation

Published

on

Click to view this graphic in a higher-resolution.

A time series chart showing the creation of machine learning systems on the x-axis and the amount of AI computation they used on the y-axis measured in FLOPs.

Charted: The Exponential Growth in AI Computation

Electronic computers had barely been around for a decade in the 1940s, before experiments with AI began. Now we have AI models that can write poetry and generate images from textual prompts. But what’s led to such exponential growth in such a short time?

This chart from Our World in Data tracks the history of AI through the amount of computation power used to train an AI model, using data from Epoch AI.

The Three Eras of AI Computation

In the 1950s, American mathematician Claude Shannon trained a robotic mouse called Theseus to navigate a maze and remember its course—the first apparent artificial learning of any kind.

Theseus was built on 40 floating point operations (FLOPs), a unit of measurement used to count the number of basic arithmetic operations (addition, subtraction, multiplication, or division) that a computer or processor can perform in one second.

ℹ️ FLOPs are often used as a metric to measure the computational performance of computer hardware. The higher the FLOP count, the higher computation, the more powerful the system.

Computation power, availability of training data, and algorithms are the three main ingredients to AI progress. And for the first few decades of AI advances, compute, which is the computational power needed to train an AI model, grew according to Moore’s Law.

PeriodEraCompute Doubling
1950–2010Pre-Deep Learning18–24 months
2010–2016Deep Learning5–7 months
2016–2022Large-scale models11 months

Source: “Compute Trends Across Three Eras of Machine Learning” by Sevilla et. al, 2022.

However, at the start of the Deep Learning Era, heralded by AlexNet (an image recognition AI) in 2012, that doubling timeframe shortened considerably to six months, as researchers invested more in computation and processors.

With the emergence of AlphaGo in 2015—a computer program that beat a human professional Go player—researchers have identified a third era: that of the large-scale AI models whose computation needs dwarf all previous AI systems.

Predicting AI Computation Progress

Looking back at the only the last decade itself, compute has grown so tremendously it’s difficult to comprehend.

For example, the compute used to train Minerva, an AI which can solve complex math problems, is nearly 6 million times that which was used to train AlexNet 10 years ago.

Here’s a list of important AI models through history and the amount of compute used to train them.

AIYearFLOPs
Theseus195040
Perceptron Mark I1957–58695,000
Neocognitron1980228 million
NetTalk198781 billion
TD-Gammon199218 trillion
NPLM20031.1 petaFLOPs
AlexNet2012470 petaFLOPs
AlphaGo20161.9 million petaFLOPs
GPT-32020314 million petaFLOPs
Minerva20222.7 billion petaFLOPs

Note: One petaFLOP = one quadrillion FLOPs. Source: “Compute Trends Across Three Eras of Machine Learning” by Sevilla et. al, 2022.

The result of this growth in computation, along with the availability of massive data sets and better algorithms, has yielded a lot of AI progress in seemingly very little time. Now AI doesn’t just match, but also beats human performance in many areas.

It’s difficult to say if the same pace of computation growth will be maintained. Large-scale models require increasingly more compute power to train, and if computation doesn’t continue to ramp up it could slow down progress. Exhausting all the data currently available for training AI models could also impede the development and implementation of new models.

However with all the funding poured into AI recently, perhaps more breakthroughs are around the corner—like matching the computation power of the human brain.

Where Does This Data Come From?

Source: “Compute Trends Across Three Eras of Machine Learning” by Sevilla et. al, 2022.

Note: The time estimated to for computation to double can vary depending on different research attempts, including Amodei and Hernandez (2018) and Lyzhov (2021). This article is based on our source’s findings. Please see their full paper for further details. Furthermore, the authors are cognizant of the framing concerns with deeming an AI model “regular-sized” or “large-sized” and said further research is needed in the area.

Methodology: The authors of the paper used two methods to determine the amount of compute used to train AI Models: counting the number of operations and tracking GPU time. Both approaches have drawbacks, namely: a lack of transparency with training processes and severe complexity as ML models grow.

green check mark icon

This article was published as a part of Visual Capitalist's Creator Program, which features data-driven visuals from some of our favorite Creators around the world.

Click for Comments

Brands

How Tech Logos Have Evolved Over Time

From complete overhauls to more subtle tweaks, these tech logos have had quite a journey. Featuring: Google, Apple, and more.

Published

on

A cropped chart with the evolution of prominent tech companies’ logos over time.

How Tech Logos Have Evolved Over Time

This was originally posted on our Voronoi app. Download the app for free on iOS or Android and discover incredible data-driven charts from a variety of trusted sources.

One would be hard-pressed to find a company that has never changed its logo. Granted, some brands—like Rolex, IBM, and Coca-Cola—tend to just have more minimalistic updates. But other companies undergo an entire identity change, thus necessitating a full overhaul.

In this graphic, we visualized the evolution of prominent tech companies’ logos over time. All of these brands ranked highly in a Q1 2024 YouGov study of America’s most famous tech brands. The logo changes are sourced from 1000logos.net.

How Many Times Has Google Changed Its Logo?

Google and Facebook share a 98% fame rating according to YouGov. But while Facebook’s rise was captured in The Social Network (2010), Google’s history tends to be a little less lionized in popular culture.

For example, Google was initially called “Backrub” because it analyzed “back links” to understand how important a website was. Since its founding, Google has undergone eight logo changes, finally settling on its current one in 2015.

CompanyNumber of
Logo Changes
Google8
HP8
Amazon6
Microsoft6
Samsung6
Apple5*

Note: *Includes color changes. Source: 1000Logos.net

Another fun origin story is Microsoft, which started off as Traf-O-Data, a traffic counter reading company that generated reports for traffic engineers. By 1975, the company was renamed. But it wasn’t until 2012 that Microsoft put the iconic Windows logo—still the most popular desktop operating system—alongside its name.

And then there’s Samsung, which started as a grocery trading store in 1938. Its pivot to electronics started in the 1970s with black and white television sets. For 55 years, the company kept some form of stars from its first logo, until 1993, when the iconic encircled blue Samsung logo debuted.

Finally, Apple’s first logo in 1976 featured Isaac Newton reading under a tree—moments before an apple fell on his head. Two years later, the iconic bitten apple logo would be designed at Steve Jobs’ behest, and it would take another two decades for it to go monochrome.

Continue Reading
HIVE Digital Technologies

Subscribe

Popular