Mistral joins UniAIBench: accuracy, efficiency and trade-offs

Benchmark update

Two Mistral models, one common test

UniAIBench added Mistral Medium 3.5 and Mistral Small 4 to its exercise-by-exercise ranking. Both answered 33 Mathematical Analysis III and 27 General Physics II exercises under the same high-reasoning configuration, with no explicit output-token ceiling, and were judged with the same approved and frozen T1–T8 rubrics as the other models.

Correctness

High medians, lower overall positions

The site’s default Balanced preset weights correctness at 50%, speed at 25% and cost at 25%, excluding token consumption. Its composite score determines the ranking; the rubric statistics below describe correctness separately.

Both correctness medians exceed 91 and each model earns full marks on 14 exercises, but lower-scoring answers reduce their means. Small has the higher median; Medium has the higher mean and less variability.

Frontier comparison

What the balanced ranking captures

Mistral positions Medium 3.5 as frontier-class and Small 4 as its lighter hybrid reasoning model. The selected configurations below retain their positions in the full 17-configuration Balanced ranking, alongside their mean rubric scores.

DeepSeek V4.1 Flash leads Balanced. Small ranks above Medium because its lower cost and shorter response time offset its slightly lower correctness. The next section isolates answer quality with Accuracy only.

Data & methodCheck the preset scores in the Balanced ranking ↗.

Data & methodDownload the historical Mistral comparison CSV ↗, the overall-ranking CSV ↗ or inspect the 60-exercise score CSV ↗. The historical Mistral summary preserves the earlier 16-configuration rankings; the tables here use the current 17-configuration panel.

An open-weight comparison

Closer to Gemma than to the leading group

Switching to Accuracy only gives correctness 100% of the weight and excludes cost, latency and token consumption. The eight selected configurations below retain their positions and normalized preset scores from the full ranking, with the original mean rubric scores in a separate column. Gemma 4 31B IT is an open-weight model available to download for free: Small’s mean rubric score is almost identical to Gemma’s, while Medium leads it by just 1.49 rubric points.

On the original rubric scale, Sol leads Medium and Small by 11.64 and 13.15 points; DeepSeek V4.1 Flash leads them by 10.15 and 11.66 points. Even o3, immediately above Medium in Accuracy only, scores 92.75 on the rubric—5.68 points above Medium and 7.18 above Small. Both Mistral models remain below the leading frontier group.

Free access to Gemma’s weights does not mean free inference: local hardware or a hosted API can still incur costs.

Data & methodCheck the preset scores in the Accuracy only ranking ↗ and the mean rubric scores in the ranking snapshot CSV ↗. Gemma’s downloadable weights are described in the official model overview ↗.

Resource profile

Similar accuracy, very different costs

Resources explain the different Balanced positions. The table below reports separate ranks for token use, speed and cost, not composite scores.

Medium leads Small by 1.51 mean points. Across the 60 direct comparisons, Medium wins 23 exercises, Small 25 and 12 are ties. The approximate 95% paired bootstrap interval for the mean difference is −1.68 to +5.07 points and includes zero: the observed accuracy advantage is not statistically clear. The resource differences are clearer.

Both models are among the heaviest consumers of billed output tokens: two configurations use more than Medium, and three use more than Small. Mistral’s metadata does not separate reasoning from the visible answer, so these counts do not establish answer length alone. Their latency is closer to the middle of the field. Medium is the third-most expensive configuration, while Small is the fourth-cheapest.

Small offers the stronger price–performance balance on these exercises. Medium’s clearest advantages are its higher minimum score and lower variability.

Data & methodResource and historical cost values come from the run-derived token-and-cost CSV ↗.

Observed issues

Where the score is lost

The following cases illustrate interpretation errors and missing checks that affect substantial parts of a solution, rather than isolated arithmetic slips.

Two cases show the same issue in both models; the other two show large reversals between Medium and Small. The more expensive model is therefore not consistently better on individual exercises, and both can make conceptual errors.

Data & methodRead the answers and judgments in the linked Atlas records: A3-25-P1-E3 ↗, F2-25-P3-E3 ↗, A3-26-P2B-E2 ↗ and A3-25-P5-E3 ↗.