When different AI models make the same mistake

Failure overlap

Shared errors are the rule, not the exception

The benchmark's errors are far from randomly distributed. They repeatedly concentrate around the same structural bottlenecks: geometric reconstruction, circuit topology, global reasoning, missing solution branches and sign propagation.

The current analysis covers all 16 model-mode configurations and 60 exercises in the matched panel, corresponding to 1,600 distinct rubric locations. Repeated runs are first averaged within each configuration–exercise pair. A rubric criterion is then classified as substantially failed when the resulting credit is below 50%.

Under this definition, the panel contains 1,995 model–criterion failures. Of these, 1,686 occur on criteria also failed by a model from another provider.

The result remains strong when the analysis is restricted to the eight configurations whose current 60-exercise benchmark average is below 90.

Data & methodDownload the shared-failure summary CSV ↗ and the 1,600-location criterion-deficit CSV ↗. Thresholding and within-configuration repetition averaging follow the method ↗.

The automated analysis detects the same failed rubric criterion, not necessarily the same logical error. Manual examination is required before describing the underlying misconception as shared.

Observed versus independent

Models fail together more often than expected

For each configuration pair, the observed number of joint failures can be compared with the number expected if their errors were independent.

The expected count is computed from each model's marginal failure rate across the same 1,600 criterion locations. These models do not merely make many mistakes: they make them disproportionately often at the same points. Kim et al. (2025) ↗ report a related pattern across more than 350 language models, finding substantial error correlation even across different architectures and providers.

Model limit. The independence baseline uses marginal failure rates but does not control for criterion difficulty. Difficult criteria can increase overlap even without a shared mechanism.

Complete pairwise overlap

Data & methodDownload the pairwise comparison CSV ↗ and the complete overlap-matrix CSV ↗. Expected counts use each pair's marginal failure rates across the same criterion locations.

Manual error analysis

Five recurring shared failure modes

Accuracy groups

More errors do not necessarily mean more correlated errors

Less accurate models make the patterns easier to observe because they generate more substantial criterion failures. Yet the rarer failures of stronger models are even more concentrated on shared rubric locations.

Data & methodGroup totals are in the summary CSV ↗; the underlying criterion locations are in the criterion-deficit CSV ↗.

The 98.1% denominator is the 261 substantial failures made by configurations with an overall mean at or above 90; 256 occur at locations failed by another provider. This smaller denominator requires caution and still measures shared locations, not identical reasoning.

Weaker models reach shared failed criteria more frequently. Stronger models reach them less often, but their rarer failures cluster around the same exceptionally difficult rubric locations.

The most important convergences occur before most calculations begin: the wrong geometric object or circuit topology is constructed; a local result replaces a global classification; a branch, boundary or physical contribution is omitted; a singular constraint is lost; or an incorrect sign is propagated through the full solution.

Operational implication

Provider diversity is not automatically reasoning diversity

Agreement between models is often treated as independent confirmation. These results show why that assumption can be unsafe. Models from different providers may agree because they are correct, but they may also share the same failure attractor.

A useful benchmark should therefore measure not only accuracy, but also which criteria models fail together, whether they share the same root misconception, how overlap changes across providers and whether equivalent reformulations preserve or remove the error.