When models disagree with the answer key—and are right

The premise

Disagreement is not always evidence of failure

A conventional benchmark asks whether a model agrees with a reference answer. The expanded UniAIBench dataset exposed a second possibility: systematic, independently reasoned disagreement can be evidence that the reference itself needs review.

This does not make models authorities over the benchmark. It makes their responses potential audit signals—signals that remain provisional until they are checked mathematically and resolved through explicit human adjudication.

Case one

The Catalan constant case

An Analysis III exercise required the evaluation of

The official solution reported an incorrect final value. Claude Fable 5 and GPT-5.6 Sol independently derived

where G is Catalan's constant. The alternative was subsequently confirmed analytically and by numerical quadrature.

Data & methodResponse text, versioned adjudication and score history: Atlas A3-26-P3-E2 ↗.

The Claude Fable 5 and GPT-5.6 Sol responses each moved from 93.5 to 100 without changing their answer text. This was not a difference in notation: both reached the same non-trivial result against the reference and were proven correct. See the Catalan constant Atlas record (A3-26-P3-E2) ↗.

When independent, coherent derivations converge against a precise reference value, the correct response is an audit—not an automatic penalty.

Decision rule

When disagreement becomes an audit signal

Agreement between models is not proof: systems can reproduce the same misconception or draw on similar training distributions. It becomes useful evidence only when several safeguards are present.

1. Models from independent providers reach the same alternative result.
2. Their derivations are mathematically coherent, rather than mere matching answers.
3. The disagreement concerns a precise and independently verifiable step.
4. Symbolic, numerical or expert checks support the alternative.
5. The models did not have access to the reference solution during generation.

These conditions suggest an extension to evaluation pipelines: in addition to measuring agreement with the key, a benchmark can identify clusters of independent responses that disagree with it in the same structured way.

Boundary. A cluster raises the priority of review; it does not decide the review.

Case two

When a model proves more than the reference

A related but conceptually different case arose in a nonlinear differential-equation problem. The course solution established global existence only for α = 1. GPT-5.6 Sol instead proved it for α > 1.

The response introduced the coercive energy

and combined finite-interval estimates, Grönwall's inequality and a continuation argument. Human adjudication verified the proof and classified it as ALT_STRONGER: the reference was not wrong, but the model had established a strictly more complete conclusion.

Data & methodResponse-level evidence and adjudication outcome: Atlas A3-26-P4-E3 ↗. The human-review procedure is documented in the benchmark paper, §6.2 ↗.

Fable and Gemini fell from 100 to 90 because the corrected rubric required the broader α > 1 conclusion: their answers matched the narrower official reference but did not establish that extension. The response text did not deteriorate; the evaluation authority became more complete.

The Atlas record (A3-26-P4-E3) ↗ contains the case evidence. The human-authority protocol is discussed in Section 6.2 of the benchmark paper ↗.

A response can disagree with a reference because its reasoning is weaker—or because it has proved something the reference never considered.

Quality control

From model evaluation to examination auditing

Across the complete workflow, the benchmark identified several distinct defects in the examination material:

1. An incorrect value involving Catalan's constant — Atlas record A3-26-P3-E2 ↗.
2. A missing √2 factor in a Taylor expansion — Atlas record A3-26-P2A-E1 ↗.
3. An incorrect fundamental matrix — Atlas record A3-26-P2A-E3 ↗.
4. Incomplete conclusions about nonlinear differential equations — Atlas record A3-26-P4-E3 ↗.
5. An incorrect physical unit — Atlas record F2-26-P1-E1 ↗.
6. An erroneous impedance formula — Atlas record F2-26-P1-E2 ↗.
7. Incorrect numerical values in a physics solution — Atlas record F2-26-P3-E1 ↗.

Not every discrepancy was first discovered by a model, and no model output was accepted without independent verification. The methodological result is narrower and more useful: model responses can supply diagnostic evidence about the material used to evaluate them.

Why it matters

Ground-truth errors are not neutral noise

A faulty reference can systematically penalize models that derive the correct result independently while rewarding answers that reproduce an incomplete or incorrect key. That risks measuring conformity to the reference rather than reasoning quality.

The problem is not unique to this benchmark. MMLU-Redux documented question and ground-truth errors and showed that reannotation changes measured performance. Other work found that ensembles of language models can flag mislabeled examples and that correcting confirmed label errors increases reported capability. BetterBench places such quality controls within a broader framework for benchmark design.

Models should not autonomously rewrite ground truth. Every discrepancy still requires mathematical verification and an explicit human decision. The broader lesson is not that AI is more reliable than experts, but that several independent solutions can create an additional layer of quality control.

A carefully designed benchmark can evaluate models while simultaneously revealing weaknesses in the material used to evaluate them.