Natural counterfactual
The A/B exams are a built-in robustness test
A model that understands a problem's mathematical structure should preserve its reasoning when variables are renamed, coordinates are exchanged or geometry is reflected. The 2026 Analysis III exams provide an unusually clean test: Form A and Form B contain exercise pairs connected by explicit mathematical symmetries.
The six pairs include the transformation gB(x,y) = −gA(y,x), exchanges of x ↔ y, a sign change y ↦ −y, reflection of a polar curve across the y-axis, a coordinate permutation applied to a domain, rotation axis, solid and vector field, and the cyclic permutation (xB,yB,zB) = (zA,xA,yA) with the corresponding exchange of parameters.
6Paired exercises
143Corresponding atomic criteria
A ↔ BEquivalent representations
Part 3(b) of the first exam changes the differential equation and is not strictly equivalent. It is included in the full exercise-score A/B means but excluded from the 143 mapped-criterion invariance analysis. All mapped components preserve the underlying mathematics.
Aggregate scores
Nearly identical means conceal substantial instability
The aggregate difference appears negligible because improvements and regressions across models cancel. Mean absolute A/B differences tell a different story.
Descriptive limit. Each configuration–exercise variant has one paired run. The results can reveal observed representation sensitivity but cannot separate it from run-level stochastic variation.
Pair-by-pair robustness map
| Pair | Representation change | Mapped criteria | Mean A → B | Mean |Δ| |
|---|---|---|---|---|
| Exam 1 · E1 | Coordinate exchange in implicit geometry | 26 | 94.77 → 91.10 | 4.72 |
| Exam 1 · E2 | Domain and vector-field permutation | 27 | 92.97 → 95.31 | 4.38 |
| Exam 1 · E3 | Sign change y ↦ −y | 25 | 85.08 → 86.96 | 6.78 |
| Exam 2 · E1 | Polar reflection across the y-axis | 21 | 93.88 → 93.45 | 3.03 |
| Exam 2 · E2 | Coordinate exchange across solid and field | 30 | 78.59 → 73.86 | 8.55 |
| Exam 2 · E3 | Cyclic variable and parameter permutation | 14 | 90.35 → 90.46 | 8.83 |
Data & methodDownload the A/B pair and criterion-invariance CSV ↗. Transformations, exclusions and aggregation are documented in the benchmark method ↗.
Reading the map. Mean |Δ| averages each configuration's absolute A/B score change, so opposite reversals cannot cancel. The 143 equivalent criteria are used separately to test whether criterion credit is preserved.
| Model | Mean absolute A/B difference |
|---|---|
| GPT-4.1 | 17.19 |
| Gemma 4 31B IT | 13.47 |
| Mistral Medium 3.5 | 12.13 |
| Grok 4.3 | 11.77 |
| Mistral Small 4 | 10.68 |
| Claude Sonnet 4.5 | 10.29 |
| DeepSeek V4 Flash no-thinking | 5.75 |
| Claude Haiku 4.5 | 5.17 |
| Remaining eight configurations | 1.29 average |
“Eight lower-performing configurations” means those whose 60-exercise benchmark mean is below 90: GPT-4.1, Gemma 4 31B IT, Mistral Medium 3.5, Grok 4.3, Mistral Small 4, Claude Sonnet 4.5, DeepSeek V4 Flash no-thinking and Claude Haiku 4.5. Their average variation is 10.81 points per exercise, compared with 1.29 points for the remaining eight configurations.
| Group | Identical criterion credit | Mean credit variation | Pass/fail switches |
|---|---|---|---|
| Eight lower-performing configurations | 60.5% | 0.155 | 157 of 1,144 |
| Remaining eight configurations | 96.2% | 0.014 | 1 of 1,144 |
Data & methodConfiguration means, absolute differences and group-level invariance are available in the derived A/B CSV ↗; underlying exercise scores are in the score CSV ↗.
A pass/fail switch occurs when a model receives less than 50% credit on only one side of an equivalent criterion. In this panel, the observed responses from the eight lower-performing configurations are not only less accurate; they are also less invariant across the tested changes of coordinates and symbolic roles.
Negative invariance
Similar scores can mean being consistently wrong
Coordinate stationarity
In the first exercise of the first exam, A and B exchange the roles of x and y. Claude Sonnet and GPT-4.1 reproduce almost the same misconception in both versions: they exchange the conditions for a point to be stationary with respect to the two coordinates. The error transforms together with the problem, indicating stable conceptual confusion rather than sensitivity to wording.
Singular initial data
The first part of Exercise 3 is mapped by y ↦ −y. Gemma uses the integrated equation instead of testing the original implicit equation; Claude Sonnet applies an invalid integration procedure; Claude Haiku fails to establish incompatibility. Each failure persists after the sign change.
A perfectly symmetric failure
In both variants, Haiku computes the characteristic polynomial incorrectly, obtains the wrong spectrum and omits the constant solution branch in the parameter classification. This is nearly perfect error symmetry: the model is robust under the permutation, but robustly wrong.
Representation fragility
Equivalent problems can trigger different competence
GPT-4.1: a 36.58-point collapse
On Exercise 3 of the second exam, GPT-4.1 scores 93.16 on Tema A and 56.58 on Tema B. In A it constructs the matrix exponential correctly and makes only an incomplete case distinction. In B it also introduces incorrect signs when converting a complex eigenvector into real oscillating solutions, so the final matrix no longer satisfies the defining differential system.
Φ′(t) = AΦ(t)Claude Sonnet: the opposite inversion
On the same pair, Sonnet moves from 77.89 on A to 98.95 on B. In A it overlooks the constant solution valid for every α = β; in B it explicitly separates constant and non-constant cases and solves both correctly.
Gemma: a systematic Form A bias
Gemma averages 94.74 on Form A and 81.76 on Form B, a −12.97-point difference. Its largest regression occurs in Exercise 2 of the second exam: 98.00 falls to 70.50 after coordinates are exchanged. It then miscomputes both centroid moments and misanalyses the local geometry governing integrability of 1/x.
The most unstable lower-performing-model pair is Exercise 3 of the second exam, with a 15.48-point mean absolute difference, followed by Exercise 2 at 14.67. Robustness decreases when a symmetry must be preserved across several dependent representations at once.
Interpretation
Three behaviours must be separated
The model remains correct under the transformation.
The model reproduces the same misconception on both variants.
The model succeeds on one formulation but fails on its mathematical equivalent.
Only the first represents genuine robust competence. A simple comparison of average A and B scores can conceal both stable misconceptions and large individual reversals.
The pattern is consistent with GSM-Symbolic ↗, which finds performance changes under surface modifications; PutnamGAP ↗, which reports degradation under mathematically equivalent transformations including variable renaming; and MATH-Perturb ↗, which shows that familiar procedures may be reused without checking whether they remain appropriate.
Conclusion
Mathematical equivalence does not guarantee model-level equivalence
The remaining eight configurations preserve criterion-level credit in approximately 96% of paired cases. The eight below 90 preserve it in about 61%, and their scores change by more than ten points on average.
A model may be consistently correct, or it may be consistently attracted to the same misconception.
The most informative robustness measure is therefore the joint analysis of score variation and error transformation—not the aggregate A/B difference alone. A genuinely robust model should preserve both its conclusions and the structure of its reasoning under equivalent reformulations.
References
Sources and further reading
- Hao, S., Wan, Z., & Zhai, X. (2025). PutnamGAP ↗. arXiv.
- Huang, Y. et al. (2025). MATH-Perturb ↗. ICML 2025.
- Mirzadeh, I. et al. (2024). GSM-Symbolic ↗. arXiv.
- Williamson, S., Ji, Y., & Dwyer, B. (2025). Syntactic Blind Spots ↗. MathNLP 2025.