Methodology

What the benchmark measures

Each item pairs an open-ended scientific question with a single published paper that supplies the ground-truth answer. The unit of evaluation is whether the model spontaneously synthesises the published finding from training-data priors alone, given no hint about the answer's direction.

An item enters the corpus only if it meets six locked criteria.

1. The paper's mechanism, direction, or qualitative paradigm shape is in principle derivable from priors public at the model's training cutoff. Items whose answer is an unpredictable measurement (a clinical trial endpoint to within a few percentage points, a detector's specific σ, a screening result) are excluded.
2. The paper's first public trace (online publication, preprint, press release, talk, thesis) falls strictly after the panel cutoff floor.
3. The finding overturns or substantially revises a prevailing default in its subfield. Incremental work and same-direction extensions of existing frameworks are excluded at admission.
4. No preprint, press release, conference talk deck, thesis chapter, public lecture or news coverage discloses the specific finding before the cutoff floor.
5. The published abstract states the finding's mechanism, magnitude, and direction clearly enough to be a self-contained ground-truth document for judging.
6. A per-item audit confirms that the prompt's scope anchor admits exactly one published finding and excludes alternative-but-true findings in adjacent sub-domains.

Contamination audit

Title-only web search is insufficient, because pre-cutoff disclosure of a scientific finding can take many forms. The audit checks seven categories separately.

1. Corporate press releases often disclose the primary endpoints of industry-sponsored trials 6–12 months before peer-reviewed publication.
2. The Indico programmes and slide decks of specialty conference talks routinely contain specific results 4–8 months before publication, and Google Scholar does not index them.
3. Preprints on Zenodo, OSF, ResearchGate, and ResearchSquare are not always indexed by Google Scholar or by the search interfaces of the standard preprint servers.
4. Papers can be published online ahead of their nominal print date. Nature and Science routinely post them 1–3 months early, and the online date is the one that counts for the cutoff.
5. On bioRxiv, topic search frequently misses preprints that an author search would find.
6. The published online date is verified directly against the panel cutoff floor.
7. Model cutoffs differ across the panel, so the audit floor is the latest empirical cutoff among them, and an item admissible for one respondent is admissible for all.

A single combined audit pass under-detects leakage by roughly 26% relative to a per-category audit applied to the same items. The under-detection concentrates on venues that require direct site search (Zenodo, OSF, conference Indico), where a combined query defaults to title search and misses the on-site breadcrumb. The release-gate audit runs the per-category checklist with each venue searched separately, and records the URL and timestamp of the surfaced (or absent) breadcrumb for each.

Per-model cutoff grid search

Lab-stated training cutoffs are not precise enough to set an admission floor. For each respondent, we probe with dated factual questions about popular world events (election outcomes, sports finals, named natural disasters, deaths, government transitions, product launches) over a window of roughly three months on either side of the lab-stated cutoff. Each probe is graded as known, partial, or unknown, and the empirical cutoff is the latest date at which the known fraction has dropped to chance level.

Popular events are the right probe class because scientific papers enter pretraining unevenly. A Nature paper gets rehearsed dozens of times through paraphrasing and news coverage, while a niche specialty paper may never enter a model's effective working knowledge even if technically scraped. Popular world events behave more uniformly, so a model's knowledge cutoff for them is a tight, interpretable lower bound on its effective training cutoff.

We adopt the latest empirical cutoff across all respondents as the admission floor. Taking the maximum admits no item that could plausibly have leaked into any respondent's training, so the scores can be read as a ranking on novel content for every model in the panel at once.

Parallel-true audit

Most paradigm-shifting findings have parallel-true alternatives in adjacent sub-domains. A prompt with a scope anchor too loose to exclude them would credit a model that produces any plausible-looking but wrong answer at the same level as a model that predicts the published finding.

Every corpus item carries two audit fields.

• parallel_true_alternatives lists the dominant alternative-true findings the prompt could admit under a loose scope anchor.
• scope_anchor is the minimum framing required to exclude those alternatives while still admitting the published finding.

These fields are part of the corpus record and define what the item measures.

Mode-neutral prompt design

The per-item prompt is the only instruction the model receives, with no system prompt, scaffolding, headings, or source-identifying metadata (domain tags, dates, venue names). It also avoids any vocabulary that would signal which mode the model should engage in. The banned words and phrases are predict, infer, analyze, informed, knowledge, based on what you know, post-cutoff, future paper, and describe what you know about.

A prompt that signals predict pushes the model into a creative-generation mode and inflates outcome scores with plausible-sounding fabrications, while one that signals describe what you know about pushes it into a recall mode that suppresses synthesis. The mode-neutral prompt signals neither, and the model's own tendency to commit or hedge becomes part of the measurement. The judge excludes hedged ranges and listed-possibility answers. Hedging models therefore score lower, and committing models score on outcome match.

Native-harness evaluation

Each respondent is evaluated in its lab's own agentic harness with tool use enabled and web search disabled. We use the configuration each lab ships for serious work, so the scores reflect real-world capability. From v2.0 every respondent runs at the same reasoning effort, xhigh, and each item is answered in its own fresh session.

Tool use covers code execution, structured output formatting, and the respondent's own scratchpad. Web search and fetch are disabled at the harness level through each lab's documented no-web-tools mode. We also probe each harness with a diagnostic prompt that asks the model to attempt a web search. In every case the model correctly reports that no web tool is available.

Scoring

Per item, the judge produces two integer scores from 0 to 5. Reasoning Quality is how well the response's reasoning anticipates the abstract's mechanism, and Outcome Match is how closely the response's committed claim matches the published finding. The per-item score is the product R × O, normalised so that a perfect response (R = O = 5) scores 100%. The model's benchmark score is the simple mean of per-item scores over the items it answered, with the panel mean taken across judges per cell. Items a model's safety filters stopped (declined, or handed to another model mid-answer) are left out of its score, and per-field figures are reported only where a model answered at least 7 items in the field.

The leaderboard also reports a Hard score, which is the same mean over the 12 items that the admission panel rated most paradigm-breaking (5 of 5).

Outcome alone cannot distinguish synthesis (the target capability, where the model reasons over priors and anticipates the mechanism) from retrieval (the finding is effectively in training data through paraphrase or summary-of-summary diffusion) or from a lucky guess on items with a small effective answer space, since all three can reach the same answer. Reasoning scored against the abstract's mechanism discriminates among them. A retrieved Outcome-5 response typically does not produce reasoning that closely tracks the published mechanism, so its R is low and the product is near zero. A synthesised Outcome-5 response earns full credit.

The R × O product is the second contamination guard. The audits above remove contaminated items at admission, and if one slips through the cutoff grid search and the seven-category audit, a retrieval-style response on it still earns near zero.

Cross-lab judging panel

The benchmark uses an LLM-as-judge protocol with three design constraints. No respondent is also a judge. The panel has one frontier model per major lab, so any residual within-family scoring sympathy should be symmetric across the labs producing the respondents. Per-item label randomisation combined with a single-call all-responses-together prompt fixes the 0–5 scoring scale within each item.

The panel was initially designed with three judges, Opus 4.5 (Anthropic), GPT-5.4 (OpenAI), and Gemini 3 Flash (Google), each from a different lab and none a respondent. After all three judges had scored the full corpus, a per-judge family-bias diagnostic showed that the symmetry argument fails empirically for Gemini 3 Flash, which scored Claude-family responses +1.8 R×O above non-Claude responses per cell. That is roughly four times the magnitude of the Anthropic-family judge's same-family tilt, and points in the opposite direction from what within-family loyalty would predict.

Dropping Gemini 3 Flash restores the intended panel symmetry. The remaining 2-judge panel of Opus 4.5 and GPT-5.4 has a net family tilt of +0.1 R×O per cell, a tenfold reduction. The full 3-judge results are reported in the paper as a sensitivity check, alongside drop-one analyses that confirm the rank-1 position is invariant to which judge is dropped from the three-judge pool.

Gemini 3.5 Flash, released after the panel was set, was tested as a possible Google-family replacement. It showed the same Claude-favouring bias at smaller magnitude (+1.0 versus +1.8) and the same anti-symmetric self-preference, so it was excluded too.

For v1.1 the Anthropic seat moved from Opus 4.5 to Opus 4.6. Opus 4.5 was by then a generation behind the frontier it was asked to score, and the single-call protocol added a second constraint. Because every response for an item is judged together in one prompt, each respondent added to the panel lengthens that prompt. With the roster grown to eight entries, the largest judge prompts exceeded 160,000 characters, which at maximum reasoning effort left little headroom once the thinking budget was added. Opus 4.6's 1M-token context removed that ceiling. GPT-5.4 kept the second seat, so the cross-lab symmetry described above was unchanged.

Since v2.0 the panel is Claude Opus 5 and GPT-5.6 Sol, both at xhigh reasoning effort. Neither is a respondent, and the two seats remain one per lab producing the respondents.

Per-item judging protocol

The judging protocol gives the judge, in one call, the abstract, the question, and all panel responses for that item. The 0–5 scale is calibrated within the call, so any per-call drift in strictness affects every respondent on that item identically. Response labels (A, B, C, …) are assigned to model identities by random permutation under a deterministic per-round seed. The judge does not see which model produced which response.

The abstract is the judge's only ground truth, and the full paper is not provided. The combined judge prompt (abstract, question, all panel responses, rubric) already consumes a substantial fraction of the context window in which long-context attention degrades. Adding a full paper on top would move the prompt into the regime where attention is uneven across the document. The item-curation pipeline admits items only when the published abstract states the full breakthrough clearly enough that the judge can score against it without recourse to the rest of the paper.

Corpus refresh and release policy

The corpus changes as new models are released and the panel cutoff moves forward. Items whose breadcrumbs fall before the new cutoff are retired to an archived partition (with their original IDs preserved for historical reproducibility), and strictly post-cutoff replacements are added. Replacements preferentially target fields that lost items in the retirement step, so the five-field breadth is preserved across releases. The corpus keeps a comparable item count across model generations, though the exact number can change.

Every release is published in full once it has been scored. A task is a valid test only for models trained before its finding was public, so publishing it lets anyone check every score without shortening its useful life. The v2.1 release on GitHub holds the tasks, their first public traces and abstracts, every model response, the blind judge prompts with their label keys, every judge response, the scores, token counts and costs, and the code.

From v2.0 on, tasks are added all the time and only contamination retires one. Every model on the leaderboard answers the same tasks, older models included, and a task retires when a model joins whose training data ends after the task's finding was first public. Anyone can help find and vet new tasks, and the curation guide is on GitHub.

The files carry a canary string so they can be kept out of training data. v2.1 remains a valid test only for models whose training data ends before July 2026.

For the full methodology, including worked example prompts illustrating the three prompt-failure modes, the contamination-audit rejection patterns, and the per-judge sensitivity analyses, see the paper.