AI
How Smart is ChatGPT?
Visualizing ChatGPT’s Performance in Human Exams
ChatGPT, a language model developed by OpenAI, has become incredibly popular over the past year due to its ability to generate human-like responses in a wide range of circumstances.
In fact, ChatGPT has become so competent, that students are now using it to help them with their homework. This has prompted several U.S. school districts to block devices from accessing the model while on their networks.
So, how smart is ChatGPT?
In a technical report released on March 27, 2023, OpenAI provided a comprehensive brief on its most recent model, known as GPT-4. Included in this report were a set of exam results, which we’ve visualized in the graphic above.
GPT-4 vs. GPT-3.5
To benchmark the capabilities of ChatGPT, OpenAI simulated test runs of various professional and academic exams. This includes SATs, the bar examination, and various advanced placement (AP) finals.
Performance was measured in percentiles, which were based on the most recently available score distributions for test takers of each exam type.
Percentile scoring is a way of ranking one’s performance relative to the performance of others. For instance, if you placed in the 60th percentile on a test, this means that you scored higher than 60% of test-takers.
The following table lists the results that we visualized in the graphic.
Category | Exam | GPT-4 Percentile | GPT-3.5 Percentile |
---|---|---|---|
Law | Uniform Bar Exam | 90 | 10 |
Law | LSAT | 88 | 40 |
SAT | Evidence-based Reading & Writing | 93 | 87 |
SAT | Math | 89 | 70 |
Graduate Record Examination (GRE) | Quantitative | 80 | 25 |
Graduate Record Examination (GRE) | Verbal | 99 | 63 |
Graduate Record Examination (GRE) | Writing | 54 | 54 |
Advanced Placement (AP) | Biology | 85 | 62 |
Advanced Placement (AP) | Calculus | 43 | 0 |
Advanced Placement (AP) | Chemistry | 71 | 22 |
Advanced Placement (AP) | Physics 2 | 66 | 30 |
Advanced Placement (AP) | Psychology | 83 | 83 |
Advanced Placement (AP) | Statistics | 85 | 40 |
Advanced Placement (AP) | English Language | 14 | 14 |
Advanced Placement (AP) | English Literature | 8 | 8 |
Competitive Programming | Codeforces Rating | <5 | <5 |
The scores reported above are for GPT-4 with visual inputs enabled. Please see OpenAI’s technical report for more comprehensive results.
As we can see, GPT-4 (released in March 2023) is much more capable than GPT-3.5 (released March 2022) in the majority of these exams. It was, however, unable to improve in AP English and in competitive programming.
Regarding AP English (and other exams where written responses were required), ChatGPT’s submissions were graded by “1-2 qualified third-party contractors with relevant work experience grading those essays”. While ChatGPT is certainly capable of producing adequate essays, it may have struggled to comprehend the exam’s prompts.
For competitive programming, GPT attempted 10 Codeforces contests 100 times each. Codeforces hosts competitive programming contests where participants must solve complex problems. GPT-4’s average Codeforces rating is 392 (below the 5th percentile), while its highest on a single contest was around 1,300. Referencing the Codeforces ratings page, the top-scoring user is jiangly from China with a rating of 3,841.
What’s Changed With GPT-4?
Here are some areas where GPT-4 has improved the user experience over GPT-3.5.
Internet Access and Plugins
A limiting factor with GPT-3.5 was that it didn’t have access to the internet and was only trained on data up to June 2021.
With GPT-4, users will have access to various plugins that empower ChatGPT to access the internet, provide more up to date responses, and complete a wider range of tasks. This includes third-party plugins from services such as Expedia which will enable ChatGPT to book an entire vacation for you.
Visual Inputs
While GPT-3.5 could only accept text inputs, GPT-4 has the ability to also analyze images. Users will be able to ask ChatGPT to describe a photo, analyze a chart, or even explain a meme.
Greater Context Length
Lastly, GPT-4 is able to handle much larger amounts of text and keep conversations going for longer. For reference, GPT-3.5 had a max request value of 4,096 tokens, which is equivalent to roughly 3,000 words. GPT-4 has two variants, one with 8,192 tokens (6,000 words) and another with 32,768 tokens (24,000 words).
![]() |
Interested in learning more about the impact artificial intelligence is having on the world of work? VC+ members have access to this special dispatch as well as our entire archive of VC+ content. Find out more. |
Technology
Visualizing the Top U.S. States for AI Jobs
Nearly 800,000 AI jobs were posted in the U.S. throughout 2022. View this graphic to see a breakdown by state.

Visualizing the Top U.S. States for AI Jobs
Much ink has been spilled over fears that artificial intelligence (AI) will eliminate jobs in the economy. While some of those fears may be well-founded, red-hot interest in AI innovation is creating new jobs as well.
This graphic visualizes data from Lightcast, a labor market analytics firm, which shows how many AI-related jobs were posted in each state throughout 2022.
In total there were 795,624 AI jobs posted throughout the year, of which 469,925 (59%) were in the top 10. The full tally is included in the table below.
Rank | State | Number of job postings | % of total |
---|---|---|---|
1 | California | 142,154 | 17.9% |
2 | Texas | 66,624 | 8.4% |
3 | New York | 43,899 | 5.5% |
4 | Massachusetts | 34,603 | 4.3% |
5 | Virginia | 34,221 | 4.3% |
6 | Florida | 33,585 | 4.2% |
7 | Illinois | 31,569 | 4.0% |
8 | Washington | 31,284 | 3.9% |
9 | Georgia | 26,620 | 3.3% |
10 | Michigan | 25,366 | 3.2% |
11 | North Carolina | 23,854 | 3.0% |
12 | New Jersey | 23,447 | 2.9% |
13 | Colorado | 20,421 | 2.6% |
14 | Pennsylvania | 20,397 | 2.6% |
15 | Arizona | 19,514 | 2.5% |
16 | Ohio | 19,208 | 2.4% |
17 | Maryland | 16,769 | 2.1% |
18 | Minnesota | 11,808 | 1.5% |
19 | Tennessee | 11,173 | 1.4% |
20 | Missouri | 10,990 | 1.4% |
21 | Oregon | 10,811 | 1.4% |
22 | Washington, D.C. | 9,606 | 1.2% |
23 | Indiana | 9,247 | 1.2% |
24 | Connecticut | 8,960 | 1.1% |
25 | Wisconsin | 8,879 | 1.1% |
26 | Alabama | 7,866 | 1.0% |
27 | Kansas | 7,683 | 1.0% |
28 | Arkansas | 7,247 | 0.9% |
29 | Utah | 6,885 | 0.9% |
30 | Nevada | 6,813 | 0.9% |
31 | Idaho | 6,109 | 0.8% |
32 | Oklahoma | 5,719 | 0.7% |
33 | Iowa | 5,670 | 0.7% |
34 | South Carolina | 4,928 | 0.6% |
35 | Louisiana | 4,806 | 0.6% |
36 | Kentucky | 4,536 | 0.6% |
37 | Nebraska | 4,032 | 0.5% |
38 | Delaware | 3,503 | 0.4% |
39 | New Mexico | 3,357 | 0.4% |
40 | Rhode Island | 2,965 | 0.4% |
41 | New Hampshire | 2,719 | 0.3% |
42 | Hawaii | 2,550 | 0.3% |
43 | Mississippi | 2,548 | 0.3% |
44 | Maine | 2,227 | 0.3% |
45 | South Dakota | 2,195 | 0.3% |
46 | Vermont | 1,571 | 0.2% |
47 | North Dakota | 1,227 | 0.2% |
48 | Alaska | 970 | 0.1% |
49 | West Virginia | 887 | 0.1% |
50 | Montana | 833 | 0.1% |
51 | Wyoming | 769 | 0.1% |
The following chart adds some context to these numbers. It shows how the percentage of AI job postings in some of the top states has changed since 2010.
We can see that California quickly became the primary destination for AI jobs in the early 2010s, presumably as Silicon Valley companies began developing the technology.
California’s share has since declined, with a significant number of jobs seemingly moving to Texas. In fact, many tech companies are relocating to Texas to avoid California’s relatively higher taxes and cost of living.
The 10 Most In-Demand Specialized Skills
Lightcast also captured the top 10 specialized skills that were required for AI-related jobs. These are listed in the table below.
Skill | Frequency (number of postings) | Frequency (% of postings) |
---|---|---|
Python | 296,662 | 37% |
Computer Science | 260,333 | 33% |
SQL | 185,807 | 23% |
Data Analysis | 159,801 | 20% |
Data Science | 157,855 | 20% |
Amazon Web Services | 155,615 | 19% |
Agile Methodology | 152,965 | 19% |
Automation | 138,791 | 17% |
Java | 133,856 | 17% |
Software Engineering | 133,286 | 17% |
If you’re interested in a career that focuses on AI, becoming proficient in Python is likely to be a good first step.
-
Urbanization3 weeks ago
Ranked: The World’s Biggest Steel Producers, by Country
-
Markets3 days ago
Mapped: The State of Economic Freedom in 2023
-
Announcement2 weeks ago
Join Us For Data Creator Con 2023
-
Datastream2 days ago
The 10 Longest Range EVs for 2023
-
Money2 weeks ago
Charted: Public Trust in the Federal Reserve
-
Real Estate2 hours ago
Mapped: The Growth in House Prices by Country
-
Technology2 weeks ago
Ranked: The World’s Top 25 Websites in 2023
-
Cities4 weeks ago
Ranked: Top 10 Cities Where International Travelers Spend the Most