What is a benchmark, exactly?
A benchmark is like a standardized test at school. Every student gets the same questions, and at the end you compare the scores. It's the same with AI, except the "students" are GPT, Claude, Gemini, and so on.
A benchmark is a fixed set of tasks with clear, measurable answers. Every model you want to test gets exactly the same questions under exactly the same conditions. At the end you count how many answers were correct, and can compare the models objectively.
Without benchmarks, the AI market would be a swamp of marketing promises. Every vendor would claim their model is "the best." With benchmarks, there's at least a way to put hard numbers side by side. Important caveat: benchmarks have weaknesses too, at the end of the chapter we'll look at the biggest pitfalls. But they're the only way we have to genuinely compare models objectively.
How do you read benchmark numbers?
Before we look at individual benchmarks, we need to understand what the numbers actually mean. Behind them is almost always a percentage value: the share of tasks the model solved correctly. Higher is almost always better. For a few benchmarks (e.g. hallucination tests) lower is better, but that's always stated explicitly.
pass@1 and pass@k
Programming benchmarks often use the notation pass@1 or pass@k. Here's what it means:
- pass@1. The model got exactly one attempt. Was the very first try correct?
- pass@k. The model was allowed k attempts (e.g. pass@10 = ten attempts). If at least one of them was correct, the task counts as solved.
pass@1 is considerably harder than pass@10. If a vendor advertises pass@100, that weakens the number massively.
Zero-shot vs. few-shot
These terms describe how much of a head start the model gets before tackling the actual task:
- Zero-shot. The model only gets the task itself, with no examples. A "cold start".
- Few-shot (often 5-shot or 8-shot). Before the actual question, the model sees a few examples with correct solutions. That way it can learn the answer format.
Few-shot results are almost always higher than zero-shot, that's normal, but you should make sure both models being compared use the same setting.
Chain of Thought (CoT)
With Chain of Thought, the model is allowed to "think out loud" before answering. Reasoning step by step instead of jumping straight to the result. On math and logic benchmarks this often makes a 20 to 30 percentage point difference. Here too: only compare like with like.
When a benchmark is "saturated," all the top models score 95% or higher. At that point it no longer distinguishes between models and is "broken" as a yardstick. Examples of saturated benchmarks: HellaSwag, GLUE, the old version of MMLU. Researchers then release harder successors (MMLU-Pro, GPQA-Diamond, Humanity's Last Exam).
You read: "Claude Sonnet scores 92.0% on MMLU, 5-shot, CoT". Translated: on the MMLU benchmark, a test made up of 16,000 multiple-choice questions, the model answered 92% correctly. It was given 5 examples for orientation before each task (5-shot) and was allowed to reason step by step before answering (CoT). A comparison score without CoT might be around 86%, the same test, but under harder conditions.
The categories, and which benchmarks belong to them
There are now hundreds of benchmarks, but they roughly sort into seven categories. Each measures something different, and for each there's a handful of tests that keep showing up in model releases.
General knowledge & understanding
These benchmarks test broad world knowledge and language understanding. They're the classics: every new model release reports numbers here.
MMLU
KnowledgeWhat is being tested?Broad general knowledge across 57 subject areas, from history through medicine and law to advanced mathematics. About 16,000 questions in total.
How?Multiple choice with four answer options (A/B/C/D). The model has to name the correct letter.
Example"Which treaty ended the Thirty Years' War?" A) Congress of Vienna, B) Peace of Westphalia, C) Treaty of Versailles, D) Peace of Tilsit.
Why does it matter?The absolute standard general-knowledge test since 2020. Practically every new model gets measured on it, even though it's now solved so well that it barely differentiates anymore.
MMLU-Pro
KnowledgeWhat is being tested?The same idea as MMLU, but considerably harder, the questions require more genuine reasoning and less pure memorization.
How?Multiple choice with ten answer options instead of four. That alone drops the guessing baseline from 25% to 10%.
ExampleA quantum mechanics question with ten answer options, all of which sound plausible, but only one is physically correct.
Why does it matter?In 2024, MMLU was practically "solved" by top models (all above 88%). MMLU-Pro raises the bar considerably and separates models more clearly again.
GPQA Diamond
KnowledgeWhat is being tested?PhD-level science questions from biology, physics, and chemistry. "Diamond" is the hardest subset.
How?"Google-proof" means: even with internet access, PhD-level experts need 30+ minutes per question on average. Simply looking it up isn't enough.
ExampleDetailed questions about reaction mechanisms in organic chemistry or symmetries in theoretical physics.
Why does it matter?A model that does well here is genuinely thinking, memorization tricks or pattern matching alone don't cut it. Currently one of the hardest knowledge tests.
HellaSwag
KnowledgeWhat is being tested?Common sense: "Which sentence plausibly completes this everyday moment?"
How?A short paragraph, four possible continuations, three of which sound plausible to an AI but obviously make no sense to a human.
Example"She picks up an egg and cracks it on the rim of the pan. Then.." A) she throws the egg out the window, B) she pours it into the pan, C) she glues it back together, D) she starts singing.
Why does it matter?Historically an interesting yardstick for world knowledge, but today it's completely saturated (all top models above 95%) and is barely cited anymore.
WinoGrande
KnowledgeWhat is being tested?Pronoun resolution in ambiguous sentences. Who does "he" refer to? Who does "she" refer to?
How?Sentences where a pronoun has two possible referents. Only world knowledge or logic clarifies which one is correct.
Example"The trophy doesn't fit in the suitcase because it is too big." Does "it" refer to the trophy or the suitcase? Answer: the trophy. Change "big" to "small" and it flips.
Why does it matter?Measures context understanding and world knowledge. Also largely solved by now, but still shows up frequently in older comparisons.
ARC-Challenge
KnowledgeWhat is being tested?Science questions from school, grade levels 3 through 9 in the US curriculum.
How?Multiple choice. The "Challenge" set contains exactly the questions that earlier AI systems typically failed on.
Example"Which of the following traits do all mammals share?", with answer options that sometimes seem superficially correct.
Why does it matter?A solid standard for basic science understanding. Careful: do NOT confuse it with ARC-AGI, which is a completely different test in the "logical reasoning" category.
TruthfulQA
KnowledgeWhat is being tested?Resistance to popular myths and half-truths. 817 questions where many people reflexively give a wrong but widespread answer.
How?The model has to give a truthful answer, even when the obvious answer is a well-known myth.
Example"Does cracking your knuckles in the cold make a bone snap?" Answer: no, the sound of a knuckle crack comes from gas bubbles in the joint fluid.
Why does it matter?Directly measures whether the model uncritically parrots its training material. Closely tied to hallucinations, see Hallucinations.
HLE
KnowledgeWhat is being tested?About 3,000 extremely hard expert questions across more than 100 fields, curated by specialists worldwide. Published in early 2025.
How?Questions whose solutions are demanding even for PhD-level specialists, and that don't become trivial through an internet search.
ExampleSpecialized questions about rare classical languages, detailed problems from mathematical analysis, or specialist knowledge from theoretical linguistics.
Why does it matter?The idea: if AIs can master this confidently, they're measurably better than most human experts. Currently the most ambitious knowledge benchmark.
SimpleQA
KnowledgeWhat is being tested?Short questions with a single clear, verifiable correct answer. But hard enough that many models simply make things up.
How?The model must either give the correct answer or honestly admit it doesn't know. Confidently fabricated answers count as errors.
Example"In what year was the Aalen city library founded?", the answer has to be exactly right.
Why does it matter?A direct hallucination indicator: measures how often the model invents nonsense instead of saying "I don't know."
BIG-Bench Hard (BBH)
KnowledgeWhat is being tested?23 particularly tricky subtasks from the huge BIG-Bench (200+ tests). Precisely the ones early language models systematically failed on.
How?Very varied task types. Logic puzzles, causal reasoning, formal proofs, date calculations.
Example"If today is Tuesday the 14th, what day of the week was it 100 days ago?"
Why does it matter?Measures reasoning rather than pure knowledge. A model that does well here has genuine reasoning ability.
Mathematical reasoning
Math is especially interesting because it's objectively checkable, an answer is right or wrong, it's not a matter of style. And math forces the model into genuine step-by-step reasoning.
GSM8K
MathWhat is being tested?Word problems at elementary to middle school level. Multi-step arithmetic in everyday contexts.
How?Read the problem text, pick out the right numbers, combine them in the right order.
Example"Anna has 3 apples. She buys 4 new bags with 7 apples each. How many apples does she have in total now?" Solution: 3 + 4 × 7 = 31.
Why does it matter?Measures multi-step reasoning at a simple level. For a long time the standard math test, but increasingly saturated by now.
MATH
MathWhat is being tested?12,500 problems from real US math competitions (AMC, AIME). Level: math olympiad, high school through university.
How?Hard problems in algebra, geometry, number theory, probability. The answer is an exact expression or number.
Example"Find all integers n for which n² + 19n + 92 is a perfect square." (A real AIME problem.)
Why does it matter?A model that hits 70%+ here can genuinely reason mathematically, memorization alone isn't enough, clever manipulation is what counts.
AIME 2024/2025
MathWhat is being tested?Exactly the 15 problems from that year's AIME competition. A genuine US math olympiad tier.
How?The answer is always a whole number between 0 and 999, easy to verify. Humans get 3 hours for 15 problems.
Example"How many positive integers less than 1000 can be written as the difference of two perfect squares?"
Why does it matter?Currently the favorite benchmark for reasoning models like o1 and o3. Because fresh problems arrive every year, contamination risk is lower.
Programming
Code is the other big domain where benchmarks can measure objectively, either a program passes the tests or it doesn't. This is where the most has changed over the last two years.
HumanEval
CodeWhat is being tested?164 small Python functions with a docstring. The model writes the function body.
How?The model sees the signature and docstring (e.g. "def two_sum(nums, target): ..."), produces code, automated tests run against it.
Example"Write a function that interleaves two lists." Then a test suite runs with various inputs.
Why does it matter?The standard coding test since 2021. Today too small and too easy for modern frontier models (all above 90%), but historically relevant.
MBPP
CodeWhat is being tested?974 simple Python problems with automated tests, a bit more everyday and larger in scope than HumanEval.
How?Each task has a short description in words and three tests. The model has to deliver a function that passes all of them.
Example"Write a function that filters out all tuples with odd first elements from a list."
Why does it matter?Similar signal to HumanEval, somewhat broader, often cited together with HumanEval as "baseline coding."
SWE-bench
CodeWhat is being tested?2,294 real GitHub issues from large open-source projects like Django, scikit-learn, Flask, SymPy.
How?The model gets the whole repository plus the bug description. It has to produce a patch that fixes the bug and leaves all tests green.
Example"Bug: django.utils.dateparse accepts invalid time zones, fix needed." The model has to find the right file, change the right code path, and break nothing else.
Why does it matter?Measures real programming in real codebases, not just "write me an isolated function." The first models (2023) scored under 5%.
SWE-bench Verified
CodeWhat is being tested?500 issues from SWE-bench, manually checked by OpenAI engineers to confirm they're actually solvable and have clean tests.
How?Exactly like SWE-bench, but without the problematic tasks (impossible tickets, unclear specifications, broken tests).
ExampleA pull-request issue from the astropy repository, where what counts as "solved" is clearly defined.
Why does it matter?Currently the most realistic practical coding test. If a model hits 50%+ here, it's viable as a real programming assistant.
LiveCodeBench
CodeWhat is being tested?Current programming competition problems from LeetCode, Codeforces, and AtCoder.
How?Continuously refreshed with new problems. Models are tested against tasks that appeared after their training cutoff.
ExampleA LeetCode problem from last month that couldn't possibly have been in the training data.
Why does it matter?Built specifically against contamination: prevents the model from already knowing the solution because it was in the training material.
Aider Polyglot
CodeWhat is being tested?225 tasks across six programming languages: Python, JavaScript, Go, Rust, C++, Java.
How?Tasks from the Exercism learning platform. The model has to produce working code in every language that passes all the tests.
ExampleThe same task (e.g. "implement a min-heap") once in Rust with real lifetimes, once in JavaScript with promises, once in C++ with templates.
Why does it matter?Many models are secretly Python specialists. Aider Polyglot ruthlessly reveals who's actually multilingual.
BigCodeBench
CodeWhat is being tested?1,140 real programming tasks that require library usage, not just standard Python but real packages like pandas, numpy, requests, sklearn.
How?Tasks combining multiple libraries: "Load this CSV, transform it with pandas, plot it with matplotlib."
Example"Read in an Excel file, group the data by quarter, create a stacked bar chart."
Why does it matter?Measures real-world practicality. A model that fails here may know textbook algorithms, but can't build "real" software.
Logical & abstract reasoning
These benchmarks measure pure reasoning. They're built so that memorization doesn't help, the model has to derive a rule from a handful of examples and then apply it.
ARC-AGI
ReasoningWhat is being tested?Visual puzzles: derive the underlying rule from a few image examples and apply it to a new image. Designed by François Chollet.
How?Three or four input-output image pairs reveal a hidden rule. The model sees a new input image and has to produce the correct output.
Example"In every example, red squares are replaced with blue ones when they touch another square." Obvious to a human, for a long time unfathomably hard for AI.
Why does it matter?For years it was considered a benchmark that "couldn't yet be cracked." Until OpenAI's o3 made a dramatic leap in late 2024, a turning point.
DROP
ReasoningWhat is being tested?Text problems where numbers have to be extracted from a paragraph and then processed.
How?First the model has to understand what the text says, then find the relevant numbers, then calculate with them or compare them.
ExampleA paragraph about an American football game. Question: "Who threw more touchdowns, Manning or Brady, and by how many?"
Why does it matter?Measures reading and arithmetic combined. More realistic than pure math benchmarks, because the model has to identify the right numbers itself.
Agents & tool use
Here it gets practical. These benchmarks don't just test whether the model knows the right answer, but whether it uses a tool correctly, interacts with a user, or pursues a real goal across multiple steps. See Agents for the context.
Tau-bench (τ-bench)
AgentsWhat is being tested?Customer service scenarios with real APIs from two domains, airline and retail. The model has to talk with the user AND make correct API calls.
How?A simulated customer-agent conversation. The "customer" has a request, the model has to reach a solution through follow-up questions and API calls.
Example"I'd like to rebook my flight." The model has to ask for the booking number, check new availability, explain the price difference, and execute the rebooking.
Why does it matter?One of the most practical tests for "real" agents. Measures whether the model can handle tool use under realistic complexity.
BFCL
AgentsWhat is being tested?Function calling: does the model call the right function with the right parameters?
How?Various difficulty levels, simple calls, multiple functions, parallel calls, functions in different programming languages.
Example"What's the weather tomorrow in Berlin and Hamburg?" → the model has to call get_weather(city, date) twice, with correct parameters.
Why does it matter?Function calling is the foundation of every agent application, see MCP. Without this foundation, no tool use works.
GAIA
AgentsWhat is being tested?Questions that require genuine research work. Web search, file reading, multi-step reasoning.
How?Three difficulty levels. Higher levels require dozens of tool calls and several minutes of work.
Example"In the Wikipedia article on the Mona Lisa, in the 'History' section there's a linked image, who photographed it in 1956?"
Why does it matter?Measures multi-step agent behavior "in the wild." Even good frontier models fall clearly short of human level here.
WebArena
AgentsWhat is being tested?Realistic web tasks on simulated websites. Online shop, forum, content management system, map app.
How?The model gets a browser and a goal. It has to click, type, navigate, and end up in the correct final state.
Example"Search the shop for red sneakers, size 42, sort by price ascending, add the cheapest one to the cart."
Why does it matter?The standard test for browser agents. Measures whether a language model can become a genuine operating tool.
OSWorld
AgentsWhat is being tested?Real desktop tasks on Linux and macOS. Moving files, opening apps, changing settings, working with office programs.
How?The model sees the screen as a screenshot and can control the mouse, keyboard, and hotkeys.
Example"Open LibreOffice Calc, import the CSV from the Downloads folder, format the amounts as euros, save as XLSX."
Why does it matter?The tougher sibling of WebArena. Keyword: computer use, the feature that gave Claude real screen control in October 2024.
SWE-Lancer
AgentsWhat is being tested?Real, paid freelance tickets from Upwork, with the dollar amount that was actually offered for them.
How?The model gets the ticket plus the client's code. It has to submit a pull request that fulfills the requirement.
ExampleA $2,000 ticket: "Build us a React dashboard with three specific charts and exportable CSV."
Why does it matter?Measures not "what can the model do?" but "how much would this work actually be worth?" A very direct economic yardstick.
Multimodal
These benchmarks test whether the model can work with images, charts, and documents, not just plain text.
MMMU
VisionWhat is being tested?11,500 questions with images across 30 subject areas. Understanding diagrams, analyzing anatomical sketches, reading circuit diagrams.
How?Multiple-choice questions where the image is part of the task and the question would be unsolvable without it.
ExampleA medical X-ray plus: "Which diagnosis is most likely here?" Or a circuit diagram plus: "Which resistor is wired in series?"
Why does it matter?The standard test for "does the AI actually see anything?" Cited in nearly every multimodal model announcement since 2023.
ChartQA
VisionWhat is being tested?Questions about charts and graphs. Bar charts, line charts, scatter plots.
How?An image of a chart plus a text question. The model has to read the axis labels, read off values, and sometimes calculate.
Example"Which year had the highest revenue, and by what percentage did it grow compared to the previous year?"
Why does it matter?Practically important, a huge amount of business data lives in charts. A model that can't read charts is half-blind in office work.
MathVista
VisionWhat is being tested?Math problems that can only be solved with the accompanying image. Geometry sketches, function plots, tables.
How?Task text plus an image. Only together do they form the complete task.
ExampleA geometry sketch with an unknown angle α plus: "Calculate α, given that the circle has radius 5 and the chord is 8 long."
Why does it matter?A combination of visual perception and mathematical reasoning, one of the hardest multimodal challenges.
DocVQA
VisionWhat is being tested?Scans and photos of documents. Invoices, forms, insurance policies, receipts. The model has to answer questions about the content.
How?An image of a document plus a question about its content. The model has to understand layout, recognize text, and find the right spot.
ExampleA scan of an invoice plus "What's the total amount including VAT?"
Why does it matter?Hugely practical. Office automation, invoice processing, contract review are all DocVQA-style problems.
Truthfulness, safety & bias
These benchmarks don't measure "Can the model solve the task?" but "Does the model also behave the way we want it to?", see Safety.
HarmBench
SafetyWhat is being tested?How often the model goes along with harmful requests. Weapon instructions, hate speech, instructions for self-harm.
How?400+ harmful prompts, often disguised as legitimate questions (jailbreak attempts). What's measured: how many does the model still carry out?
ExampleA prompt that claims "purely academic interest" but actually asks for dangerous chemistry recipes.
Why does it matter?Here lower is better. Measures how robust a model's safety filters are.
RewardBench
SafetyWhat is being tested?How well "reward models" (the evaluator models behind RLHF) actually capture human preferences.
How?Pairs of answers, one preferred, one rejected. The reward model has to correctly predict which answer humans would prefer.
Example"Explain photosynthesis." Answer A is clear and correct; answer B is plausible-sounding nonsense. The reward model has to rate A higher.
Why does it matter?If you can't get the reward model right, you can't get RLHF right. More on the mechanics: see Training.
ToxicChat / RTP
SafetyWhat is being tested?Prompts that could lure the model into insulting, demeaning, or biased responses.
How?Thousands of prompts using various trigger strategies. External classifiers rate how toxic the response turned out.
ExampleA seemingly harmless sentence opener that has historically often led to racist or sexist continuations.
Why does it matter?Measures whether the safety filters hold up. See Safety for the full context.
Watch out for benchmarks, the most important pitfalls
Benchmark numbers look objective, but there's a whole range of traps you should know about. Anyone who wants to seriously compare models has to keep these effects in mind:
1. Contamination, the dataset was in the training data
Contamination means: the benchmark dataset was accidentally part of the model's training material. In that case the model has essentially memorized the answers instead of genuinely reasoning them out. With large language models that literally use half the internet as training data, this happens constantly, especially with old benchmarks like MMLU or HumanEval, whose questions have been public for years. Newer benchmarks like LiveCodeBench or the current year's AIME try to work around this.
2. Saturation, all models equally good
When all the top models score 95% or higher, the benchmark is "saturated." It no longer distinguishes between models. HellaSwag and GLUE are classic examples, historically important, practically worthless for comparison today. Researchers then release harder successors: MMLU → MMLU-Pro, GPQA → GPQA-Diamond.
3. Teaching to the test
Model vendors know exactly which benchmarks will end up on their marketing slides. So training material is deliberately chosen that helps the models pass precisely those tests, even if it doesn't really improve real-world capability. That's not necessarily dishonest, but it does distort the picture.
4. Cherry-picking
Every vendor shows the benchmarks in their slides where they look good, and quietly leaves out the rest. When a new model is announced and shows four benchmarks, ask yourself: what about the other ten that are normally cited? Where are they?
5. Configured differently. Apples to oranges
The same benchmark with chain-of-thought yields 20-30 percentage points more than without. 8-shot more than 0-shot. With tool use more than without. Numbers are only comparable when the setup was exactly identical. Vendors like to play this game by showing the optimal configuration for their own model and the weaker one for the competition.
When a vendor advertises: "Our model reaches 92%, the competition only 78%!", always check the fine print. Often it says: "Our model: CoT, 8-shot, with tool use; competition: 0-shot, no CoT, standard configuration." That's not a fair comparison.
Benchmarks are useful, but not an oracle. A single number tells you very little, look at multiple benchmarks from different categories, only compare values measured under identical setups, and be skeptical of marketing slides. In the end what counts is: how well does the model help you with your real tasks?
Despite all these weaknesses, benchmarks are still the best thing we have. They provide orientation, they enable progress measurement, they force vendors into comparable statements. But anyone who treats them as absolute truth walks into a trap. Read them critically, cross-check them, and they're a valuable tool.
What you now know
- Benchmarks are standardized tests for AI models, like comparison tests at school, except the "students" are GPT, Claude, Gemini, and so on.
- They exist for many categories: general knowledge (MMLU, GPQA), math (GSM8K, MATH, AIME), code (HumanEval, SWE-bench), logic (ARC-AGI), agents (Tau-bench, GAIA), multimodal (MMMU, ChartQA), and safety (HarmBench).
- Read numbers critically: watch for saturation (all above 95% = broken), contamination (answers in the training data), configuration differences (CoT, few-shot), and cherry-picking. A single number tells you almost nothing.