Toolathlon-Verified
Realistic multi-step chores that require an agent to juggle many real software tools over a long session, not just answer questions.
What does a Toolathlon-Verified task look like?
A single task gives the agent one natural-language goal embedded in an already-populated, live multi-application environment — it is a chore to carry out end-to-end via MCP tool calls, not a question to answer in text. Reaching the goal typically takes on the order of 20 sequential tool-calling turns spanning more than one application (e.g. an inbox plus a calendar, or a spreadsheet plus a database). The benchmark's own team has published short summaries of a few illustrative tasks of this shape: one has the agent pull student submissions out of an email inbox and enter the corresponding grades into a course on Canvas (the LMS); another has the agent gather several quarters of institutional-ownership data, adjust the figures for a stock split, and populate a specific spreadsheet template with only the holdings that are common across quarters. In both, correctness is judged purely by whether the live application(s) end up in the correct final state — not by anything the agent says.
How many points on Toolathlon-Verified is real?
On Toolathlon-Verified, a gap smaller than 9.7 points is treated as noise — with 108 items, run-to-run variance alone can produce a difference that size. The floor itself is derived, not asserted: 100 ÷ √108 ≈ 9.7, two standard errors on Toolathlon-Verified at this size.
Grade B: No major known issues on Toolathlon-Verified, and the results here hold up — but with 108 items, close calls still deserve an independent check.
9 independent Toolathlon-Verified scores — dots inside one dashed band are closer than Toolathlon-Verified’s 9.7-point noise band, statistically indistinguishable. 11 vendor-reported Toolathlon-Verified scores are not plotted — see the table below.
Can you trust Toolathlon-Verified? Caveats
Toolathlon-Verified is actively discriminating between frontier models — a gap of at least 9.7 points here is worth taking seriously, though it only counts as a confirmed real gap once both scores come from independent runs.
108 tasks across 32 MCP servers / 600+ tools (per the ICLR 2026 paper, arXiv:2510.25726, the underlying app count is 32 real applications — e.g. Google Calendar, Notion, WooCommerce, Kubernetes, BigQuery — wired to 604 tools total). Each task hands the agent one multi-step chore that needs roughly 20 agent turns on average across several already-populated live applications to finish. Scoring is execution-based, not an LLM judge: every task ships a deterministic per-task verification script that checks whether the final state of the live application(s) matches a ground-truth end state, so the published number is a Pass@1 success rate — the percentage of the 108 tasks whose end-state check passed on a single attempt (the toolathlon.xyz leaderboard also tracks Pass@3 and Pass^3 for repeated-attempt scoring, and reports per-task average turns/tool-calls as a separate efficiency metric). There is no multiple-choice element in this benchmark and therefore no chance floor. Toolathlon is built and maintained by the HKUST NLP group (lead author Junlong Li, advisor Junxian He).
The "Verified" revision hardened the grading and isolated task state after the original Toolathlon shipped some scoring bugs. This is now a named, dated fix rather than a vague claim: Toolathlon-Verified was released 2026-06-30 (per the GitHub repo's release note and the team's own "Introducing Toolathlon-Verified" post on toolathlon.xyz).
What changed between the two versions
That post reports the two versions cover the identical 108 tasks — none added, removed, or renamed — but that 83 of the 108 task packages had net changes across prompts, initial states, ground truth, or evaluators, produced over a repair-and-review span of 339 commits (309 non-merge, 30 merge). It documents specific fixes in both directions: e.g. an evaluator in one task called a logging-lookup helper with positional arguments so a stray parameter shifted into the wrong slot, causing the checker to query the wrong log target and reject otherwise-correct runs; separately, a Python comparison helper in another task returned a (bool, message) tuple that the caller truth-tested directly — since non-empty tuples are always truthy in Python, this let some incorrect answers silently pass.
The team’s own account
Lead author Junlong Li announced the release on X on 2026-07-03, stating the team "corrected tasks, align[ed] graders, isolat[ed] state, harden[ed] the infrastructure, and more — so failures reflect model capabilities" rather than benchmark bugs, while flagging that "Toolathlon-Verified is not perfect." Advisor Junxian He's companion post named the underlying problem the fix addressed: "some live tasks are drifting from the original groundtruth, some tasks are found to have flaws on the design." The revision resets the leaderboard as a new official score series — pre-2026-06-30 Toolathlon scores are not comparable to Toolathlon-Verified numbers (the toolathlon.xyz leaderboard itself states this explicitly).
Reading the leaderboard badges
The toolathlon.xyz leaderboard marks independently-run results with an "evaluated by us" checkmark badge; unbadged numbers are vendor self-reported — the same independent/self-reported split already tracked in this benchmark's score list, e.g. DeepSeek V4 Flash's independently-confirmed +19.8-point jump between its pre-0731 and 0731 minor versions (50.9 to 70.7, both badge-verified on the official board) is evidence the badge distinction is meaningful and not just a labeling formality.
Sources · 7
- Toolathlon GitHub repo (hkust-nlp/Toolathlon) — README, Toolathlon-Verified release date, authorship
- The Tool Decathlon paper, arXiv:2510.25726 (ICLR 2026) — 32 apps/604 tools, ~20 turns/task, execution-based evaluation via dedicated verification scripts
- Toolathlon official leaderboard — badge system ("independently evaluated by us"), Pass@1/Pass@3/Pass^3 metrics, Verified release date and "new official score series" language
- "Introducing Toolathlon" official blog post (Oct 28, 2025) — per-task verification script description, officially published NVIDIA-ownership and Canvas-grading example tasks
- "Introducing Toolathlon-Verified" official blog post (Jun 30, 2026) — 83/108 task packages changed, 339-commit repair range, named bug-fix examples, badge/score-series explanation
- Junlong Li (@lockonlvange) announcing Toolathlon-Verified on X, 2026-07-03
- Junxian He (@junxian_he) on X explaining why Toolathlon-Verified was needed
Is the current lead on Toolathlon-Verified real?
UnverifiedA 2.2-point gap on Toolathlon-Verified looks close to a tie, but Claude Opus 5's score here is the vendor's own claim — a tie between a measurement and a claim isn't a tie yet.
Written up in full, for models on this Toolathlon-Verified board: Gemini 3.1 Pro Preview vs Kimi K3 · Claude Opus 4.8 vs DeepSeek V4 Pro (0813) · Claude Opus 4.8 vs Gemini 3.1 Pro Preview — every benchmark each pair shares, not just Toolathlon-Verified.
Toolathlon-Verified leaderboard: current scores
Toolathlon-Verified’s 20 tracked scores were each verified against their source between 2026-06-30 and 2026-10-25.
| Model | Toolathlon-Verified score |
|---|---|
| GLM-5.3-Flash | 78.4independent |
| Kimi K3 | 76.5independent |
| Claude Opus 4.8 | 76.2independent |
| Muse Spark 1.2 | 75.9independent |
| DeepSeek V4 Pro (0813) | 74.4independent |
| Claude Sonnet 5 | 71.6independent |
| DeepSeek V4 Flash (0731) | 70.7independent |
| Gemini 3.1 Pro Preview | 61.1independent |
| GLM-5.2 | 59.9independent |
| Claude Opus 5 | 80.6self-reported |
| Claude Sonnet 5.5 | 77.8self-reported |
| Claude Opus 5.5 | 77.8self-reported |
| Claude Fable 5.1 | 77.8self-reported |
| MiMo-V2.6-Pro | 76.9self-reported |
| Step 5 Preview | 74.1self-reported |
| Tencent Hy4 preview | 74.1self-reported |
| MiMo-V2.6-Flash | 73.6self-reported |
| Qwen3.8-Flash-Next | 73.5self-reported |
| GLM-5.3 | 73.0self-reported |
| Qwen3.8-Max | 72.5self-reported |
Toolathlon-Verified FAQ
What does Toolathlon-Verified measure?
Realistic multi-step chores that require an agent to juggle many real software tools over a long session, not just answer questions. Toolathlon-Verified has 108 items.
How big does a gap on Toolathlon-Verified have to be to mean anything?
At least 9.7 points on Toolathlon-Verified. Across Toolathlon-Verified's 108 items, a smaller gap sits inside the spread that repeat runs produce on their own, so we call it a tie — provided both scores are independent runs; if either side is a vendor claim, the row is labelled unverified instead.
Can you trust Toolathlon-Verified scores?
We grade it B and list it as current — it still tells frontier models apart across 108 Toolathlon-Verified items. A gap of at least 9.7 points clears Toolathlon-Verified's own noise band, but it only counts as a real gap once both scores come from independent runs.
What is the difference between Toolathlon and Toolathlon-Verified?
Toolathlon-Verified, released 2026-06-30, hardened the grading and isolated task state after the original Toolathlon shipped some scoring bugs. It is a named, dated fix rather than a vague "improved" claim — the release note and the team's own announcement both carry the date.
Who built Toolathlon?
The HKUST NLP group — lead author Junlong Li, advisor Junxian He — published at ICLR 2026 (arXiv:2510.25726). The 108 tasks run across 32 MCP servers wiring up 32 real applications (Google Calendar, Notion, WooCommerce, Kubernetes, BigQuery among them) to 604 tools.
How is Toolathlon scored?
Execution-based Pass@1: each task ships a deterministic verification script that checks whether the final state of the live application matches a ground-truth end state, so the published number is the share of the 108 tasks whose end-state check passed on a single attempt. No judge model scores the run, and there is no multiple-choice element, hence no guess floor; the official leaderboard also tracks Pass@3 and repeated-attempt scoring separately.
Benchmarks related to Toolathlon-Verified
- Agents' Last Exam — Professional work, trust grade A
- Terminal-Bench 2.1 — Terminal ops, trust grade B