Self-reported vs. independently verified: how big is the gap?

August 17, 2026

Here's a number we didn't expect to be this stark until we counted it ourselves: of the 42 benchmark scores on this site as of August 17, 2026, 29 — 69% — are vendor-reported, with no independent lab having rerun them. Only 13 have a source we didn't get from the company that made the model.

That's not this site failing to do its job. That's the industry norm, reflected honestly. Most benchmark numbers you read anywhere — on a launch blog post, a leaderboard, a comparison chart — trace back to the vendor grading its own homework.

How big can that gap actually get?

The most-cited case in 2026 is Moonshot AI's Kimi K2. The model's own benchmark card listed a score of roughly 50% on Humanity's Last Exam. When outside testers ran the same benchmark themselves, they measured 29.4% — a 21-point gap, not a rounding error. UC Berkeley's Responsible Data Infrastructure group documented this and similar cases across 13 benchmarks, finding 45 confirmed exploits spanning 16 distinct attack types — from weak test assertions to submissions that can read the answer key because they run in the same process as the grader.

We haven't independently verified the Kimi K2 numbers ourselves — we're reporting what Berkeley's researchers reported. That distinction matters, and it's exactly the distinction this article is about.

Why the gap exists

A few reasons, roughly in order of how often they show up:

Nobody else grades the test. When a lab publishes "our model scores X on benchmark Y," the default is that the lab ran the eval, on infrastructure it controls, with a harness it wrote. No outside party touched it. That's not necessarily dishonest — it's just unverified.

Harnesses disagree with each other, even on the same model. Researchers at Hugging Face once found conflicting MMLU scores for the identical LLaMA weights, traced entirely to differences in how two evaluation harnesses parsed and scored the same answers — no code change, no retraining, just a different scoring script. More than two dozen harnesses exist for common benchmarks, and a nominally identical test can swing by up to 20 percentage points depending which one graded it.

The incentive runs one direction. A lab choosing its own eval setup, its own prompt format, its own reasoning-effort setting, and its own benchmark subset is not doing anything against the rules. But every one of those choices can nudge a score upward, and there's no symmetric pressure pushing it back down before publication.

What we do about it on this site

This isn't a call to distrust every number you see — most self-reported scores are directionally honest, and outright fabrication is rare. It's a call to know which kind of number you're looking at before you act on it.

Every score on The Model Gap carries a self-reported or independent tag, visible right next to the number. When two models' scores get compared, our Signal Labels fold that distinction directly into the verdict: a big gap between two self-reported numbers gets flagged ⚠️ Unverified rather than ✅ Real gap — because a large gap and a confirmed gap are two different claims, and conflating them is exactly how a 21-point overstatement survives long enough to make it into a launch chart.

We also grade the benchmarks themselves — see the Trust Grades breakdown — because a self-reported score on a well-audited, hard-to-game benchmark is worth more than an independently-run score on a benchmark riddled with contamination.

None of this makes self-reported numbers worthless. It makes them exactly as trustworthy as they are: a claim, not a fact, until someone else checks.

Update, later the same day

After publishing this, we did the obvious thing: went looking for independent runs to replace our own vendor-reported rows. By the end of the day the site's ratio had flipped — 69 scores, 20 self-reported (29%), 49 independent (71%). Most of the replacements came from vals.ai and Artificial Analysis, who run these benchmarks themselves.

Two things worth taking from that. First, the independent numbers mostly existed all along — aggregators just don't bother swapping them in. Second, the swaps were not cosmetic: one model's self-reported GPQA Diamond score was 5.6 points above the independent run of the same benchmark. The gap this article is about isn't hypothetical. It was sitting in our own table.

Sources: UC Berkeley RDI, "Trustworthy Benchmarks". Site-wide self-reported/independent ratio computed from this site's own data/scores.json on 2026-08-17.