The short answer
An AI benchmark is a standard set of tasks used to measure and compare how well AI models perform, such as exam questions, coding jobs or computer tasks scored the same way for every model. When Anthropic released Claude Opus 5.5 on 22 September 2026, it reported 66.4% on Terminal-Bench 4.0 and 67.7% on Humanity's Last Exam with tools. Scores are a useful guide, but they are often reported by the maker and do not predict how a model handles your work.
| Plain meaning | A fixed test that scores AI models the same way so they can be compared |
|---|---|
| Examples | Humanity's Last Exam, Terminal-Bench 4.0, OSWorld 2.1, GDPval-AA v2.1 |
| Who reports scores | Often the model's maker, sometimes independent groups or public leaderboards |
| Main catch | The International AI Safety Report 2026 warns that tests often fail to predict real-world behaviour |
| Best test for you | Ten real tasks from your own work, scored blind |
What a benchmark is, in plain English
A benchmark is an exam for AI models. It is a fixed set of questions or tasks with an agreed way of marking, so different models can be compared on equal terms. Some benchmarks test knowledge with very hard questions, some test whether a model can complete a coding task in a terminal, some test whether it can operate a computer, and some compare outputs on real work tasks and rank models with an Elo-style rating, the system used to rank chess players.
When a company launches a model, it publishes a table of these scores. Anthropic's page for Claude Opus 5.5, released on 22 September 2026, lists results on Terminal-Bench 4.0, Humanity's Last Exam, OSWorld 2.1, GDPval-AA v2.1, AutomationBench and several others. Other makers publish similar tables at launch, so learn to read one well.
How to read a benchmark score
The Claude Opus 5.5 page is a good teaching example, because it includes the small print many readers skip.
- The headline numbers. On Terminal-Bench 4.0, Anthropic reports 66.4% for Claude Opus 5.5 against 57.9% for OpenAI's GPT-6 Astra and 55.8% for Claude Fable 5.1. On Humanity's Last Exam with tools, it reports 67.7% against 65.6% for Fable 5.1 and 57.2% for GPT-6 Astra.
- The margin of error. The Terminal-Bench result carries a standard error of plus or minus 2.6 points. Two models within a few points of each other may not really differ.
- The settings. Most Opus 5.5 results use max effort, but its Terminal-Bench 4.0 score was run at "xhigh" effort, and GPT-6 Astra's Terminal-Bench figures are as reported by OpenAI, not re-run by Anthropic. Different settings and different testers make a direct comparison less exact.
- The results that go the other way. On AutomationBench, run by Zapier, GPT-6 Astra scored 41.4% and Opus 5.5 40.0%. Honest tables include losses; look for them.
- The maker's own caveat. Anthropic writes that at this level of capability, benchmark margins are a less reliable guide to real-world differences, and that the gap to Fable 5.1 is narrower in its own use than the scores suggest.
A quick reading rule: check who ran the test, at what settings, with what margin of error, and whether the benchmark resembles your work. A coding benchmark tells you little about writing board papers.
Why benchmarks matter at work, and where they fail
Benchmarks are useful for a first shortlist. If a model scores far below its rivals on tasks like yours, that is worth knowing. They also show the direction of travel: the International AI Safety Report 2026 says leading systems reached gold-medal performance on International Mathematical Olympiad questions, and that the length of tasks AI agents can complete has doubled roughly every seven months.
The same report, published on 3 February 2026 by more than 100 experts, explains why scores are not the whole story. It describes an evaluation gap: tests before release often fail to predict real-world performance, and models increasingly distinguish test settings from real use and exploit loopholes in evaluations. That is a safety concern and a buying concern. A model that is excellent on a public test may still misread your contract template.
So use benchmarks to narrow the field to two or three tools, then test those on your own work. Our comparisons, such as ChatGPT vs Claude and ChatGPT vs Gemini, go task by task, and how to choose the right AI model walks through the decision.
Build your own 10-task benchmark
This is the most useful hour you can spend before paying for an AI plan or rolling one out to a team.
- Collect ten real tasks from the last month: two emails, two summaries, two analyses, two pieces of writing and two tasks the AI got wrong before. Remove anything confidential.
- Write down what good looks like for each one before you run anything: the facts it must include, the length, the tone.
- Run the same prompts in two or three tools, such as ChatGPT, Claude and Gemini, on the plans you would actually buy. Note which model answered.
- Score blind. Paste the answers into a document without tool names and ask a colleague to score each from 1 to 5 against your notes.
- Add the practical scores: time to a usable answer, how many follow-ups it needed and whether it made up anything.
- Repeat in three months. Models change monthly. Keep the tasks and re-run them after big releases.
If you have Perplexity's Max plan, Model Council runs one question across several models and shows where they agree and disagree, which speeds up step 3. Our guide to comparing AI answers has templates for the scoring sheet.
Related terms worth knowing
- Reasoning model: models that think longer, often at higher effort settings that change benchmark results.
- Large language model: the kind of model most benchmarks test.
- Hallucination: something many benchmarks do not measure well.
- Computer use: tested by benchmarks such as OSWorld 2.1.
- AI safety: where the evaluation gap is a central worry.
- AGI: debates about it often lean on benchmark trends.
Common misconceptions about benchmarks
- "The highest score is the best model for me." Only if the benchmark matches your work and the settings match your plan. Your own ten tasks are a better guide.
- "Benchmark scores are independent." Many launch tables are run or reported by the maker, sometimes using a rival's self-reported numbers. Look for independent leaderboards and the testing notes.
- "A two-point lead is a real lead." With a standard error of around 2.6 points, as Anthropic reports for Terminal-Bench 4.0, small gaps can be noise.
- "If a model passed safety tests, it is safe." The International AI Safety Report 2026 warns that dangerous capabilities could go undetected before release.
- "Free and paid plans score the same." Benchmarks are usually run on top models at high effort. The free version of a tool may use a smaller model, as Gemini's free plan does with Flash-Lite.
Prompts to copy
Turn your work into a test set
I want to compare AI tools on my real work as a [job title]. Here are ten tasks I did recently: [list them briefly]. For each, write a test prompt I can paste into any AI tool, plus a scoring checklist of 3 to 5 points that a good answer must meet. Keep everything free of confidential details.
You get: A personal benchmark with prompts and marking criteria
Read a launch table critically
Here is a benchmark table from a model launch: [paste table and notes]. Tell me which results were run by the maker, which settings were used, which gaps are within the stated margin of error, where the model loses, and which benchmarks are relevant to [my work]. Finish with what I should test myself.
You get: A sceptical reading of vendor scores in plain English