Same mathematics, different answer

Natural counterfactual

The A/B exams are a built-in robustness test

A model that understands a problem's mathematical structure should preserve its reasoning when variables are renamed, coordinates are exchanged or geometry is reflected. The 2026 Analysis III exams provide an unusually clean test: Form A and Form B contain exercise pairs connected by explicit mathematical symmetries.

The six pairs include the transformation gB(x,y) = −gA(y,x), exchanges of x ↔ y, a sign change y ↦ −y, reflection of a polar curve across the y-axis, a coordinate permutation applied to a domain, rotation axis, solid and vector field, and the cyclic permutation (xB,yB,zB) = (zA,xA,yA) with the corresponding exchange of parameters.

Part 3(b) of the first exam changes the differential equation and is not strictly equivalent. It is included in the full exercise-score A/B means but excluded from the 143 mapped-criterion invariance analysis. All mapped components preserve the underlying mathematics.

Aggregate scores

Nearly identical means conceal substantial instability

The aggregate difference appears negligible because improvements and regressions across models cancel. Mean absolute A/B differences tell a different story.

Descriptive limit. Each configuration–exercise variant has one paired run. The results can reveal observed representation sensitivity but cannot separate it from run-level stochastic variation.

Pair-by-pair robustness map

Data & methodDownload the A/B pair and criterion-invariance CSV ↗. Transformations, exclusions and aggregation are documented in the benchmark method ↗.

Reading the map. Mean |Δ| averages each configuration's absolute A/B score change, so opposite reversals cannot cancel. The 143 equivalent criteria are used separately to test whether criterion credit is preserved.

“Eight lower-performing configurations” means those whose 60-exercise benchmark mean is below 90: GPT-4.1, Gemma 4 31B IT, Mistral Medium 3.5, Grok 4.3, Mistral Small 4, Claude Sonnet 4.5, DeepSeek V4 Flash no-thinking and Claude Haiku 4.5. Their average variation is 10.81 points per exercise, compared with 1.29 points for the remaining eight configurations.

Data & methodConfiguration means, absolute differences and group-level invariance are available in the derived A/B CSV ↗; underlying exercise scores are in the score CSV ↗.

A pass/fail switch occurs when a model receives less than 50% credit on only one side of an equivalent criterion. In this panel, the observed responses from the eight lower-performing configurations are not only less accurate; they are also less invariant across the tested changes of coordinates and symbolic roles.

Negative invariance

Similar scores can mean being consistently wrong

Coordinate stationarity

In the first exercise of the first exam, A and B exchange the roles of x and y. Claude Sonnet and GPT-4.1 reproduce almost the same misconception in both versions: they exchange the conditions for a point to be stationary with respect to the two coordinates. The error transforms together with the problem, indicating stable conceptual confusion rather than sensitivity to wording.

Singular initial data

The first part of Exercise 3 is mapped by y ↦ −y. Gemma uses the integrated equation instead of testing the original implicit equation; Claude Sonnet applies an invalid integration procedure; Claude Haiku fails to establish incompatibility. Each failure persists after the sign change.

A perfectly symmetric failure

In both variants, Haiku computes the characteristic polynomial incorrectly, obtains the wrong spectrum and omits the constant solution branch in the parameter classification. This is nearly perfect error symmetry: the model is robust under the permutation, but robustly wrong.

Representation fragility

Equivalent problems can trigger different competence

GPT-4.1: a 36.58-point collapse

On Exercise 3 of the second exam, GPT-4.1 scores 93.16 on Tema A and 56.58 on Tema B. In A it constructs the matrix exponential correctly and makes only an incomplete case distinction. In B it also introduces incorrect signs when converting a complex eigenvector into real oscillating solutions, so the final matrix no longer satisfies the defining differential system.

Claude Sonnet: the opposite inversion

On the same pair, Sonnet moves from 77.89 on A to 98.95 on B. In A it overlooks the constant solution valid for every α = β; in B it explicitly separates constant and non-constant cases and solves both correctly.

Gemma: a systematic Form A bias

Gemma averages 94.74 on Form A and 81.76 on Form B, a −12.97-point difference. Its largest regression occurs in Exercise 2 of the second exam: 98.00 falls to 70.50 after coordinates are exchanged. It then miscomputes both centroid moments and misanalyses the local geometry governing integrability of 1/x.

The most unstable lower-performing-model pair is Exercise 3 of the second exam, with a 15.48-point mean absolute difference, followed by Exercise 2 at 14.67. Robustness decreases when a symmetry must be preserved across several dependent representations at once.

Interpretation

Three behaviours must be separated

Only the first represents genuine robust competence. A simple comparison of average A and B scores can conceal both stable misconceptions and large individual reversals.

The pattern is consistent with GSM-Symbolic ↗, which finds performance changes under surface modifications; PutnamGAP ↗, which reports degradation under mathematically equivalent transformations including variable renaming; and MATH-Perturb ↗, which shows that familiar procedures may be reused without checking whether they remain appropriate.