Version history

Each release retires items overtaken by a newer training cutoff, admits strictly post-cutoff replacements and re-scores the board. A published release is never amended. The next one replaces it, so figures from an earlier version are not comparable to a later one.

v2.1

10 November 2026

• Claude Fable 5.2, released on 9 November 2026, joins the respondent panel. It ran in Claude Code at xhigh reasoning effort like the other Claude models and answered 72 of the 80 tasks.
• Its training data ends before July 2026, so all 80 tasks stay in the corpus.
• With a new model on the board, every task was judged again by Claude Opus 5 and GPT-5.6 Sol, and the v2.1 scores replace those of v2.0.

v2.0

4 November 2026

• The complete v2.0 release has been open source on GitHub since 6 November 2026. It holds the tasks with their first public traces and abstracts, the model responses, the blind judge prompts and judge responses, the scores, token counts and costs, and the code.
• The new corpus has 80 findings across the five fields, each with its first public trace (paper, preprint, talk, press release or other disclosure) in July 2026 or later. No item carries over from v1.x, so v2.0 scores are not comparable with earlier releases.
• The respondent panel is new, with every model at xhigh reasoning effort. Claude Opus 5.5, Claude Fable 5.1, Claude Sonnet 5.5 and Claude Haiku 5.5 ran in Claude Code, and GPT-6.1 Sol, GPT-6 Astra and GPT-6.1 Terra ran in Codex.
• Claude Opus 5 and GPT-5.6 Sol form the new judging panel, both at xhigh reasoning effort. Neither is a respondent.
• The leaderboard chart now plots each model's score against the cost of the whole task set, as subscription usage on a $200 plan or as API price.
• A new Hard score covers the 12 items rated most paradigm-breaking and is shown next to the score over all 80 items.
• Safety filters stopped some items for Claude Opus 5.5 (70 of 80 answered) and Claude Fable 5.1 (66 of 80). Every model is scored over the items it answered, and per-field figures are shown only where a model answered at least 7 items in the field.

v1.1

15 July 2026

• GPT-5.6 Sol joins the panel, run in Codex at maximum reasoning effort.
• Claude Fable 5 joins in two runs, original and latest. Its safety classifier refuses every life-sciences task, most of Earth & Planetary, and roughly half of Chemistry and Materials. Fable answered 45% of the corpus in the original run and 39% in the latest, so it is scored only over the items it attempted.
• Claude Sonnet 5 replaces Claude Sonnet 4.6 on the board.
• The contamination floor moved forward to GPT-5.6 Sol's training cutoff of 16 February 2026. Every item whose first public trace falls on or before that date was retired, and all respondents were re-scored on the cleaned corpus.
• The Anthropic judge seat moved from Opus 4.5 to Opus 4.6, whose 1M-token context fits the longer judge prompts produced by the larger roster. GPT-5.4 keeps the second seat.
• Opus 4.6 leaves the respondent board, since no respondent may also judge.

v1.0

May 2026

• This was the first public release, with six respondents (Claude Opus 4.8, Opus 4.7, Opus 4.6, Claude Sonnet 4.6, Gemini 3.1 Pro and GPT-5.5) scored by a two-judge cross-lab panel of Opus 4.5 and GPT-5.4.
• The corpus was admitted under the multi-source contamination audit and the per-model cutoff grid search, and each item was scored on Reasoning × Outcome.
• The paper documents the full methodology, the contamination-audit protocol and the results. Read the paper