Self-reported vs. independent AI benchmarks: how big is the gap?

August 17, 2026

Here's a number we didn't expect to be this stark until we counted it ourselves: of the 42 benchmark scores on this site as of August 17, 2026, 29 — 69% — are vendor-reported, with no independent lab having rerun them. Only 13 have a source we didn't get from the company that made the model.

That's not this site failing to do its job. That's the industry norm, reflected honestly. Most benchmark numbers you read anywhere — on a launch blog post, a leaderboard, a comparison chart — trace back to the vendor grading its own homework.

How big can that gap actually get?

We didn't have to go looking for a headline case — we found one in our own table. Z.ai's benchmark card for GLM-5.2 lists a self-reported 91.2% on GPQA Diamond. vals.ai's independent run of the same model on the same benchmark measured 85.61% — a 5.6-point gap. It isn't the widest one in our table either; every vendor claim we've since replaced is recorded alongside the independent number that replaced it in our public scores file.

That gap doesn't mean anyone lied. Different harnesses, prompt formats, and grading scripts can each move a score several points with nobody doing anything dishonest. But it's also not the only kind of gap out there: a separate 2026 study by UC Berkeley's Responsible Data Infrastructure group found 45 confirmed benchmark-gaming exploits spanning 16 distinct attack types across 13 benchmarks — from weak test assertions to submissions that can read the answer key because they run in the same process as the grader. Some self-reported numbers drift for innocent reasons. Some don't. From the outside, a launch chart doesn't tell you which one you're looking at.

Why the gap exists

A few reasons, roughly in order of how often they show up:

Nobody else grades the test. When a lab publishes "our model scores X on benchmark Y," the default is that the lab ran the eval, on infrastructure it controls, with a harness it wrote. No outside party touched it. That's not necessarily dishonest — it's just unverified.

Harnesses disagree with each other, even on the same model. Researchers at Hugging Face once found conflicting MMLU scores for the identical LLaMA weights, traced entirely to differences in how two evaluation harnesses parsed and scored the same answers — no code change, no retraining, just a different scoring script. More than two dozen harnesses exist for common benchmarks, and a nominally identical test can swing by up to 20 percentage points depending which one graded it.

The incentive runs one direction. A lab choosing its own eval setup, its own prompt format, its own reasoning-effort setting, and its own benchmark subset is not doing anything against the rules. But every one of those choices can nudge a score upward, and there's no symmetric pressure pushing it back down before publication.

What we do about it on this site

This isn't a call to distrust every number you see — most self-reported scores are directionally honest, and outright fabrication is rare. It's a call to know which kind of number you're looking at before you act on it.

Every score on The Model Gap carries a self-reported or independent tag, visible right next to the number. When two models' scores get compared, our Signal Labels fold that distinction directly into the verdict: a big gap between two self-reported numbers gets flagged Unverified rather than Real gap — because a large gap and a confirmed gap are two different claims, and conflating them is exactly how a 5.6-point overstatement survives long enough to make it into a launch chart.

To see the labels applied to a real release end-to-end, read our DeepSeek V4 Pro verdict — the launch chart that started this site.

We also grade the benchmarks themselves — see the Trust Grades breakdown — because a self-reported score on a well-audited, hard-to-game benchmark is worth more than an independently-run score on a benchmark riddled with contamination.

None of this makes self-reported numbers worthless. It makes them exactly as trustworthy as they are: a claim, not a fact, until someone else checks.

Update, later the same day

After publishing this, we did the obvious thing: went looking for independent runs to replace our own vendor-reported rows. By the end of the day the site's ratio had flipped — 91 scores, 13 self-reported (14%), 78 independent (86%). Most of the replacements came from vals.ai and Artificial Analysis, who run these benchmarks themselves.

Two things worth taking from that. First, the independent numbers mostly existed all along — aggregators just don't bother swapping them in. Second, the swaps were not cosmetic: the GLM-5.2 gap above surfaced during that pass, and it was not the only one. The gap this article is about isn't hypothetical. It was sitting in our own table.

Sources: GLM-5.2 GPQA Diamond scores from Z.ai's benchmark card and vals.ai's independent run. UC Berkeley RDI, "Trustworthy Benchmarks" on benchmark-gaming exploits. Site-wide self-reported/independent ratio computed from this site's own data/scores.json on 2026-08-17.