GPQA Diamond is saturated — here is what that costs our own table

August 24, 2026

On August 20, 2026, we did something most GPQA leaderboards don't: we downgraded our own GPQA Diamond leaderboard to trust grade D, marked it saturated, and turned off ranking for every pair of models that share the benchmark. Not just the close calls. Every pair, all 91 of them.

That is not a hedge. It is a rule enforced in code, and this is the receipt.

The data behind the D

We track 14 models on GPQA Diamond, all independently scored — no vendor self-reports made the cut here. Sorted high to low, the spread runs from Gemini 3.1 Pro Preview at 95.45% down to GLM-5.2 at 85.61%: a 9.84-point range across the whole field, with a population standard deviation of about 2.58 points.

Compare that to the benchmark's own noise floor. GPQA Diamond has 198 items, and two standard errors on a set that size works out to 7.2 points — the meaningful gap below which a difference is statistically indistinguishable from run-to-run variance. Thirteen of our 14 tracked models sit within that 7.2-point band of the leader. The one exception, GLM-5.2, isn't isolated either: its closest tracked neighbor by score, Claude Sonnet 5, sits just 3.28 points above it, safely inside the band. The pair that comes nearest to actually crossing the threshold against GLM-5.2 without doing so — DeepSeek V4 Pro and Claude Opus 4.8 — land 6.81 points ahead, just 0.39 points short. Every model we track has at least one other tracked model within noise distance of it — a close top, and no clear bottom.

What "saturated" actually costs

Before the downgrade, our own signal logic — the same compareScores() function that grades every pairwise matchup on the site — would have called 8 of these 91 pairs a confirmed Real gap, 70 a Tie, and 13 Setup-dependent (all 13 involve GLM-5.3, whose tracked GPQA score runs on a different evaluation harness than the other 13 models). Zero would have come back Unverified, because every score here is independently sourced.

Those 8 "real" gaps weren't spread across the field — they were one shape repeated eight times: every model in the top eight beating GLM-5.2 by more than 7.2 points. (One of those eight is Claude Fable 5's 93.18% — a number that carries its own caveat on our benchmark page: it counts refusal-triggered fallback answers as correct, and scoring refusals as failures instead drops it to 55.56%. Even the numbers feeding our own "real" column aren't uniformly clean.) Two others, DeepSeek V4 Pro and Claude Opus 4.8, also clear GLM-5.2 by a wide absolute margin (6.81 points) but land just short of the threshold, so under the old grading they'd have read as ties instead.

After the downgrade, all 91 pairs read Tainted. Not because we recalculated and found the gaps smaller — the lifecycle check in our code runs before the gap is even computed. For a saturated benchmark, compareScores() returns Tainted on its first line; the point difference between two models never gets touched. Eight pairs that used to clear our own bar for "real" now carry the identical label as a pair separated by 0.00 points.

Why a ceiling does this to a benchmark

Once most of a field clusters within a couple of points of the maximum score, a benchmark stops discriminating between models and starts measuring who got lucky on the hardest handful of questions. Epoch AI's "GPQA Diamond: What's left?" put numbers on that risk: it flagged the weakest-performing 40 of the 198 questions for review, went deep on the six most extreme cases — items where every model tested scored under 5% — and judged more than a third of that handful flawed, which it scales up to a benchmark-wide error estimate near 8%. Epoch's own conclusion is more measured than "the benchmark is broken": it still counts at least 90% of GPQA Diamond as sound. But a one-in-twelve error rate sitting on top of a field already bunched within single digits of a 100% ceiling is exactly the combination that turns a ranking into a coin flip.

What the outside world shows, as of today

Checking the live leaderboard rather than trusting our own capture: as of today, August 24, 2026, vals.ai's GPQA Diamond tracker lists 135 models, 24 of them scoring 90% or higher — the same 24-model count we cited from our August 17 snapshot of 133 models, just two more added to the tail since then. The leader hasn't moved: Gemini 3.1 Pro Preview, still 95.45%. vals.ai's own read on that density is blunt — a high score, in their words, "no longer meaningfully distinguishes frontier models."

Contamination evidence isn't static either, though it isn't a clean upward line. The n-gram audit we cite on our benchmark page (Xu et al., EMNLP 2025) checked GPQA's original 448-question set against six Common Crawl snapshots and found the contaminated-item count bouncing between 3 and 12 across them — not a steady climb, but net higher at the end than at the start: 4 flagged items in the earliest snapshot tested (CC-2025-05), dipping as low as 3 partway through, then 12 in the most recent one (CC-2025-26). A benchmark whose contamination count can double or triple between snapshots, in either direction, is its own argument for a D — separate from the ceiling problem.

One thing worth being precise about: this isn't just a polite ask. GPQA's dataset is gated on Hugging Face — downloading it requires clicking through an agreement stating "you agree to NOT reveal examples from this dataset in plain text or images online, to reduce the risk of leakage into foundation model training corpora." That's a real condition of access, even though it isn't a copyright license in the legal sense. We're not quoting a test item here either way.

The mechanism, not a one-off call

None of this is a manual override applied to one page. compareScores() checks a benchmark's lifecycle before it looks at anything else — before matching variants, before checking whether either score is vendor-reported, before comparing harnesses, before any gap is measured against the 7.2-point threshold. A saturated or contaminated benchmark exits the function on that first check, every time, for every pair. The full decision order, and the other four labels a comparison can land on, is laid out on our methodology page.

So what do you read instead

Not "GLM-5.2 is bad at science questions" — an 85.61% on a Google-proof PhD-level exam is a real result, and the distance to the field's leader sits inside what repeat runs alone would produce. If you need to compare two models' scientific reasoning today, GPQA Diamond isn't where that comparison should happen; benchmarks with a higher trust grade still separate models cleanly. If you want the underlying numbers yourself, sources and dates included, they're open in our data export.

Sources: 14 tracked GPQA Diamond scores and their provenance from our own scores data, cross-checked against the vals.ai GPQA Diamond leaderboard (checked August 24, 2026). Question-quality analysis from Epoch AI, "GPQA Diamond: What's left?" (May 30, 2025). Contamination figures from Xu et al., "Infini-gram mini: Exact n-gram Search at the Internet Scale with FM-Index" (EMNLP 2025, Table 7), verified against the paper's full text. Dataset access terms from the GPQA dataset card on Hugging Face. The rule behind every label in bold above is written out in the methodology.