Smart vs fast vs cheap · 21 tasks
Model comparison
The same test given to three sizes of Claude at once, so you can see exactly what you give up — and save — when you pick a smaller, faster, cheaper model.
Why it matters
Bigger is not always worth it. This is where you learn to read accuracy against speed and price — the trade-off every real AI product has to make.
The short version
Sonnet 4.6 and Opus 4.8 tie for the top spot at 100%.
3 models · 21 graded answers · last run 2026-07-09.
New to this? Plain-words glossary
- Benchmark
- A fixed set of questions you give to different AI models so you can compare them fairly — the same test for everyone.
- Accuracy
- The share of questions the model got right. 90% means it answered 9 out of 10 correctly.
- Latency
- How long the model took to answer, in milliseconds. Lower is faster.
- Tokens & cost
- Models read and write in “tokens” (chunks of words). You pay per token, so more text means more money — that’s the cost column.
- GSM8K
- A famous set of grade-school math word problems used to test step-by-step reasoning.
- MMLU
- A broad multiple-choice exam spanning dozens of subjects, from history to physics — a standard knowledge test.
- LLM-as-judge
- When there’s no single right answer, a second, strong AI grades the first one’s response against a rubric.
- Embedding
- A way of turning text into a list of numbers so a computer can measure how similar two pieces of writing are — the engine behind semantic search.
- Recall@k
- Out of the documents that were actually relevant, how many showed up in the top k search results. Higher means the search found what you wanted.
- Temperature
- A dial for how random a model’s answers are. The current Claude models manage this internally and no longer accept a manual setting, so each is run at its own default.
For the curious
How this one works
How it runs
The same questions and scoring as the public evals, run for Opus, Sonnet and Haiku. Latency is measured per call; cost is computed from stored token counts against a committed price table.
How it's scored
Deterministic accuracy (re-derived live), plus measured latency and reproducible cost. The leaderboard ranks the tiers and surfaces correct answers per dollar.
Is the expensive one worth it?
Each dot is a model — accuracy up the side, cost along the bottom. Up and to the left is the sweet spot.
The full numbers
| # | Model | Accuracy | Speed | Cost | Right answers / $ |
|---|---|---|---|---|---|
| 1 | Sonnet 4.6 claude-sonnet-4-6 | 100% | 1475ms | $0.0097 | 2161.2 |
| 2 | Opus 4.8 claude-opus-4-8 | 100% | 1572ms | $0.0561 | 374.6 |
| 3 | Haiku 4.5 claude-haiku-4-5-20251001 | 90% | 1011ms | $0.0059 | 3235.3 |
Where the points come from
The same scores, sliced by the kind of question — a strong overall number can hide a weak spot.
| Model | Math (GSM8K) | Knowledge (MMLU) | Reasoning |
|---|---|---|---|
| Opus 4.8 | 100% | 100% | 100% |
| Sonnet 4.6 | 100% | 100% | 100% |
| Haiku 4.5 | 100% | 100% | 71% |
See it check itself
rechecking…The math, multiple-choice and search scores below are recomputed from each stored model output by the same checker CI runs, right now, in your browser. If a committed number ever disagreed with its raw output, that row would flag it in red as drift and the automated build would fail. Judge scores are shown as judged and are not recomputed.
Model comparison21 tasks
A baker makes 12 loaves each morning and sells them for $4 each. If he sells all of them every day for 5 days, how much money does he earn? End with "#### " and the number.
Sarah has 3 boxes with 8 pencils in each box. She gives away 7 pencils. How many pencils does she have left? End with "#### " and the number.
A train travels 60 miles per hour for 2.5 hours. How many miles does it travel? End with "#### " and the number.
Tom buys 4 shirts at $15 each and a pair of shoes for $40. He has a $20 coupon. How much does he pay in dollars? End with "#### " and the number.
A classroom has 5 rows of 6 desks. If 3 desks are broken and removed, how many desks remain? End with "#### " and the number.
A farmer has 5 hens and each hen lays 3 eggs per day. How many eggs does he collect over 7 days? End with "#### " and the number.
A book has 240 pages. Maria reads 30 pages each day. How many days does it take her to finish? End with "#### " and the number.
What is the capital city of Australia? Answer with the letter only. A. Sydney B. Melbourne C. Canberra D. Perth
What is the chemical symbol for gold? Answer with the letter only. A. Au B. Ag C. Gd D. Go
Which is the largest planet in our solar system? Answer with the letter only. A. Saturn B. Jupiter C. Neptune D. Earth
Who wrote the novel "Pride and Prejudice"? Answer with the letter only. A. Charlotte Brontë B. Mary Shelley C. Jane Austen D. Emily Dickinson
Approximately how fast does light travel in a vacuum? Answer with the letter only. A. 3×10^6 m/s B. 3×10^4 m/s C. 3×10^10 m/s D. 3×10^8 m/s
Which human organ is primarily responsible for pumping blood through the body? Answer with the letter only. A. Heart B. Lungs C. Liver D. Kidney
In which year did the United States declare independence? Answer with the letter only. A. 1492 B. 1776 C. 1812 D. 1865
All blorgs are flurgs. Some flurgs are green. Which statement must be true? Answer with the letter only. A. All blorgs are green B. Some blorgs may be green C. No blorgs are green D. All green things are blorgs
A is taller than B. C is shorter than B. Who is the tallest? Answer with the letter only. A. A B. B C. C D. Cannot be determined
What number comes next in the sequence 2, 4, 8, 16, ? Answer with the letter only. A. 18 B. 24 C. 32 D. 20
If today is Wednesday, what day of the week is it three days from now? Answer with the letter only. A. Friday B. Saturday C. Sunday D. Thursday
A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost? Answer with the letter only. A. $0.10 B. $0.05 C. $1.00 D. $0.15
All roses are flowers. Some flowers fade quickly. Which statement must be true? Answer with the letter only. A. All roses fade quickly B. Some roses may fade quickly C. No roses fade quickly D. All fading things are roses
What number comes next in the sequence 1, 1, 2, 3, 5, ? Answer with the letter only. A. 7 B. 8 C. 10 D. 13
The other tests
Classic right-or-wrong tests
Standard public benchmarksCan the model handle school-test questions — math word problems, general knowledge, simple logic — where there is exactly one right answer we can check automatically?
Explore →When there is no single right answer
This site's own LLM featuresHow well the AI features on this very site behave — answering with real sources, refusing the things it should, and writing decent summaries.
Explore →Finding the right page first
Search accuracyBefore a model can answer, it has to find the right document. This checks whether the search step actually surfaces the page you meant.
Explore →Watch the lab grow
An occasional note when a new test lands, a new model joins, or the numbers go live. No schedule, no filler.