Version History

Each release retires items overtaken by a newer training cutoff, admits strictly post-cutoff replacements, and re-scores the board. Releases are superseded rather than amended, so figures from an earlier version are not comparable to a later one.

v1.1

15 July 2026

• GPT-5.6 Sol joins the panel, run in Codex at maximum reasoning effort.
• Claude Fable 5 joins in two runs, original and latest. Its safety classifier declines a large share of the corpus. It refuses every life-sciences task, most of Earth & Planetary, and roughly half of Chemistry and Materials, and the latest run refuses more than the original. Fable answered 45% of the corpus in the original run and 39% in the latest, so it is scored only over the items it attempted.
• Claude Sonnet 5 replaces Claude Sonnet 4.6 on the board.
• The contamination floor moved forward to GPT-5.6 Sol’s training cutoff of 16 February 2026. Every item whose first public trace falls on or before that date was retired, and all respondents were re-scored on the cleaned corpus.
• The Anthropic judge seat moved from Opus 4.5 to Opus 4.6, whose 1M-token context absorbs the longer judge prompts produced by the larger roster. GPT-5.4 retains the second seat.
• Opus 4.6 leaves the respondent board, since no respondent may also judge.

v1.0

May 2026

• Initial public release. Six respondents, Claude Opus 4.8, Opus 4.7, Opus 4.6, Claude Sonnet 4.6, Gemini 3.1 Pro, and GPT-5.5, scored by a two-judge cross-lab panel of Opus 4.5 and GPT-5.4.
• Corpus admitted under the multi-source contamination audit and the per-model cutoff grid search, with per-item scoring on Reasoning × Outcome.
• Full methodology, contamination-audit protocol, and results are documented in the paper. Read the paper