Methodology

This page explains exactly how every label on this site is decided. If a rule isn’t written here, it isn’t a rule.

The five signal labels

They are checked in this order, and the first one that applies is the one you see. A gap has to clear every earlier test before its size is even looked at. Setup-dependent appears twice in the flow: variant differences (tools, prompts) are caught before the provenance check, harness differences after it, and reasoning-effort differences after that — either way the label is the same. Added 2026-10-02: a run at Max effort and a run at High effort are not the same measurement, and a site that quotes them as a gap is quoting its own convenience.

  • Tainted — the benchmark itself is saturated or has known contamination issues. We don’t trust rankings on it regardless of the gap size.
  • Setup-dependent — the two scores come from different setups: a different variant (with tools vs without), caught before the provenance check, or a different evaluation harness, or a different reasoning effort tier, caught after it. Not a fair comparison; the “winner” can flip depending on setup. One limit worth stating: the effort check only fires when both rows state their tier. On Humanity’s Last Exam, 22 of 33 independent rows state one; the other eleven publish none, and a comparison involving one of those is not tier-checked.
  • Unverified — either score is the vendor’s own claim rather than an independent run. One vendor-reported side is enough: a lab has every incentive to publish a flattering number and none to publish an unflattering one, so the risk runs one way. That applies to the close calls too — a tie between a measurement and a claim isn’t a tie yet.
  • Tie — both scores are independent runs, and the difference is smaller than the benchmark’s meaningful gap (see below). Statistically indistinguishable. Ignore the ranking.
  • Real gap — both scores are independent runs and the difference clears the benchmark’s noise band. This is the only label that says one model is actually ahead.

What “meaningful gap” means

Every benchmark has a meaningful_gap value: roughly, the smallest score difference that isn’t explained by run-to-run noise given the benchmark’s sample size. It comes from one formula applied identically to all 14: 100 ÷ √n, rounded up to one decimal — two standard errors on a pass/fail test set of n items. A 2,500-item benchmark earns a 2.0-point band; an 89-item one needs 10.6. It is not a precise confidence interval, and no benchmark gets a discount for having a good reputation — it is one published number per benchmark that you can recompute yourself from the item count in the table below.

Benchmark trust grades

We grade each benchmark A through D on how much weight its scores deserve: contamination risk, sample size, and how much of the field self-reports versus gets independently verified.

BenchmarkCategoryGradeItemsStatus
HLEReasoningB2500current
Terminal-Bench 2.1AgenticB89current
DeepSWECodingB113current
Toolathlon-VerifiedAgenticB108current
Agents' Last ExamAgenticA1000current
SWE-bench VerifiedCodingD500saturated
GPQA DiamondReasoningD198saturated
LiveCodeBenchCodingD1055saturated
ARC-AGI-2ReasoningB120current
LiveBenchCompositeA1436current
HMMTReasoningC33contaminated
AnalystAgentAgenticB80current
Terminal-Bench 4.0AgenticB66current
OSWorld 2.0AgenticC108current

A = designed to resist contamination (e.g. a living benchmark refreshed on a schedule). B = reliable, mostly vendor-reported, no major known issues. C = usable but small sample size or nearing saturation — treat close results with extra skepticism. D = saturated or known contamination — rankings here are treated as unreliable regardless of gap size.

What the categories mean

Benchmarks are filed by execution form — what one run of the benchmark asks the model to do — not by subject. That is why Terminal-Bench sits under Agentic (a run is multi-step terminal work) even though leaderboards that group by subject file it under coding. Categories open only once two tracked benchmarks belong to them, so competition mathematics — one benchmark — currently sits under Reasoning rather than a category of its own.

Why we never publish a single composite score

Combining several benchmarks into one weighted number requires picking which version of a multi-setting benchmark counts — and that choice alone can flip the ranking. On Humanity’s Last Exam, DeepSeek V4 Pro trails Claude Opus 4.8 without tools and leads with tools. A composite score would have to silently pick one of those numbers, dress a scaffolding choice up as model quality, and call it precision it doesn’t have.

Instead, we publish tiers and per-benchmark verdicts, not rankings with false precision. A tier board — grouping models that are statistically indistinguishable on trust-graded benchmarks, with zero weighting — is in progress; this page will document its exact rule before it goes live.

What “Previous version” means

A model is marked Previous version when a newer model of the same product line and the same price tier has shipped — Grok 4.6 once Grok 4.7 arrived, GPT-5.6 Sol once GPT-6 Sol did. It is a statement about the lineup, not about the model: the older model keeps its page, its prices and every score it was measured with, and most are still sold.

This is our own rule, and deliberately not a vendor’s. A vendor’s lifecycle table answers a different question — whether the API endpoint stays online — and the two disagree in both directions. Anthropic lists Opus 5 and Sonnet 5 as Active with retirement dates in 2027 while its own model pages call them legacy and point at their replacements; xAI and OpenAI publish nothing at all about Grok 4.6 or GPT-6 Sol, both of which have a newer version on sale beside them. Where a vendor has published something about a specific model’s retirement, we record it on that model’s own page.

The rule is computed from the catalog rather than set by hand, which is why it cannot drift: a new model added tomorrow marks its predecessor with no one remembering to. It is also narrow on purpose. Line AND tier, never line alone — GPT-6 Luna stays current because GPT-6.1 Sol is newer but is a different model at a different price, and dropping it would delete the cheapest option in that family.

Data sourcing

Every score on this site carries a source link and a self-reported/independent tag. If we can’t find a real, checkable source, the field says “Not sourced yet” — we don’t estimate or fill gaps with a plausible-looking number.

The rules on this page are the how. For the why — what this site is for, who runs it, how corrections are handled and how it is funded — read about The Model Gap. Every number here is open: download the full dataset and check any of it yourself.