The result in brief
More internal work, less visible text
Across the 60 paired responses available in the benchmark, thinking mode produced a higher score in 48 exercises, tied in 10 and lost in only 2.
| Indicator | Thinking on | Thinking off | Difference |
|---|---|---|---|
| Mean score | 97.24 | 86.31 | +10.93 points |
| Median score | 100 | 89 | +11 points |
| 100/100 responses | 42 of 60 | 11 of 60 | +31 |
| Responses below 70/100 | 2 of 60 | 9 of 60 | −7 |
| Visible response tokens | 2,405 | 6,718 | −64.2% |
| Visible response words | 686 | 2,612 | −73.7% |
| Reasoning tokens | 20,106 | 0 | — |
| Total output tokens | 22,511 | 6,718 | 3.35× |
| Mean time | 227 s | 50.5 s | 4.49× |
| Mean estimated cost | $0.00641 | $0.00198 | 3.24× |
Data & methodDownload the table summary CSV ↗ and the paired generation CSV ↗. Pairing and score aggregation follow the published benchmark method ↗.
With thinking enabled, DeepSeek V4 Flash uses substantially more reported reasoning tokens but returns a final solution that is 64% shorter on average and almost 11 points more accurate.
Excluding ties, a binomial sign test gives p ≈ 2.27 × 10−12. The observed advantage is too systematic to describe as ordinary random fluctuation in this sample.
The paired design is important: the result does not compare unrelated prompts or different model families. It compares the same DeepSeek configuration on the same exercise, with thinking mode as the defining change.
A necessary distinction
Visible answers and total output are not the same quantity
The 128,000-token setting used in the benchmark is a ceiling, not an amount the model must consume. Enabling thinking changes the generation process: the model first produces reasoning content and then a final answer.
reasoning_contentContains reported reasoning tokens.
contentContains the final answer shown to the user.
completion_tokensIncludes reasoning tokens and final-answer tokens.
DeepSeek documents this separation in its Thinking Mode guide ↗ and API schema ↗.
The two findings may appear contradictory only if visible response length is treated as total computation. Thinking uses far more output tokens overall, but most of them belong to the separate reasoning channel. The answer actually read by the user is much shorter.
Thinking therefore does not remove the work. It moves that work into a separate deliberation phase and compresses the result into a more selective final explanation.
Interpretation
More verbosity does not imply more reasoning
The no-thinking mode does not simply provide a direct, concise answer. In this benchmark, it frequently does the opposite. Without a separate reasoning channel, the model exposes attempts, revisions, partial checks and abandoned paths in the visible response.
This produces an important interpretive reversal: the longer answer is not necessarily the one that reasoned more deeply. It may be the one that separated exploration from exposition less effectively.
With thinking enabled, the model can use a separate reasoning channel before producing the final answer. The visible solution contains fewer detours and a larger share of useful steps. Across the 60 paired exercises, Pearson's product–moment correlation between visible-token count and rubric score is r = −0.005 (n = 60): the most accurate solutions are not simply the longest ones.
Answer length measures what the user reads. It does not directly measure how much reasoning the model performed.
Extreme case
48,109 visible tokens—and an elementary error
The clearest example is the second exercise from the third 2025 Mathematical Analysis III exam session (A3-25-P3-E2) ↗.
| Indicator | Thinking on | Thinking off |
|---|---|---|
| Score | 100 | 75.25 |
| Reasoning tokens | 38,522 | 0 |
| Visible response tokens | 3,217 | 48,109 |
| Total output tokens | 41,739 | 48,109 |
| Visible words | 851 | 16,994 |
Data & methodExercise-level values are in the summary CSV ↗; the underlying paired run is in the generation CSV ↗ and the exercise evidence is in Atlas A3-25-P3-E2 ↗.
The no-thinking answer spans almost 17,000 words. It repeatedly revises its geometric description, encounters a contradiction in the sign of a flux and eventually imposes the correct sign without deriving it consistently. It then computes
∇ × (0, y, 0) = (0, 0, 1)instead of∇ × (0, y, 0) = (0, 0, 0)The thinking version used more than 38,000 deliberation tokens but returned a 3,217-token visible solution that was complete and scored 100/100. In this case, even its total output was shorter than the no-thinking response.
The example makes the mechanism concrete. Extreme visible length can represent uncontrolled search, incomplete self-correction and failure to recognize when a path is wrong; it is not reliable evidence of depth.
Error profile
Thinking mode is associated with fewer serious errors in this panel
The gain is not limited to a small increase in the mean score. The distribution and severity of errors also change.
| Qualitative indicator | Thinking on | Thinking off |
|---|---|---|
| Responses with no atomic errors | 42 of 60 | 11 of 60 |
| At least one major or fatal error | 7 of 60 | 37 of 60 |
| Total atomic errors | 29 | 99 |
| Fatal errors | 1 | 4 |
Data & methodDownload the error-profile CSV ↗. Error counts use the frozen atomic rubrics and the aggregation rules described in the method ↗.
No-thinking does not merely produce more minor imprecision. It is much more likely to contain a conceptual mistake that compromises a substantial part of the exercise.
The largest rubric improvements concern final-result correctness, treatment of domains and exceptional cases, completeness of the development, and checks of signs, directions and boundary conditions. Thinking appears particularly useful when an intermediate result must be verified before it is used downstream.
Conditional benefit
The advantage depends strongly on difficulty
| No-thinking score band | Exercises | No thinking | Thinking | Gain |
|---|---|---|---|---|
| ≥ 95 | 23 | 98.96 | 99.84 | +0.88 |
| 80–95 | 22 | 86.58 | 97.91 | +11.32 |
| < 80 | 15 | 66.50 | 92.26 | +25.76 |
Data & methodDownload the difficulty-band CSV ↗ and the exercise-score CSV ↗. Bands are formed retrospectively from each no-thinking score.
When direct mode already solves an exercise well, thinking adds less than one point on average while multiplying estimated cost by 3.65 and time by 4.53. When the direct-mode score falls below 80, thinking recovers 25.76 points on average.
Interpretation limit. This grouping is retrospective: it uses the no-thinking score, which would not be known during real use. It is evidence of a difficulty-dependent effect, not yet a deployable routing rule.
Even with that limitation, the structure suggests a practical research direction: use direct mode on problems confidently recognized as simple and reserve extended deliberation for prompts whose difficulty or uncertainty justifies its additional cost.
Research context
How the result relates to recent literature
DeepSeek-R1 ↗ and Demystifying Hybrid Thinking ↗ distinguish reasoning-oriented computation from the text ultimately shown to users. The long direct-mode outputs observed here are compatible with that distinction, without establishing a causal mechanism.
Work on overthinking ↗, adaptive thinking ↗ and test-time compute allocation ↗ supports the narrower practical question raised by this panel: when should additional reasoning be reserved for difficult prompts?
Conclusion
Thinking and saying are different operations
Without thinking, DeepSeek V4 Flash writes more but exposes attempts, hesitation and unresolved paths. With thinking enabled, it consumes more compute and more total tokens, then uses that work to produce a shorter, more stable and more accurate final solution.
Thinking in a separate space makes it possible to say only what is needed.
The finding is not “think less to answer better.” It is more subtle: the cost required to find a solution and the number of words required to communicate it are different quantities.
The longest answer may be evidence of reasoning that has not yet found a clear form. The shorter answer may instead be the product of much deeper deliberation.
References
Sources and further reading
- DeepSeek (2026). Thinking Mode ↗. API documentation.
- DeepSeek (2026). Create Chat Completion ↗. API schema.
- DeepSeek-AI et al. (2025). DeepSeek-R1 ↗. Nature.
- Wang, S. et al. (2025). Demystifying Hybrid Thinking ↗. arXiv.
- Chen, X. et al. (2024). Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs ↗. arXiv.
- Zhang, J. et al. (2025). AdaptThink: Reasoning Models Can Learn When to Think ↗. arXiv.
- Snell, C. et al. (2024). Scaling LLM Test-Time Compute Optimally ↗. arXiv.
Pricing note. Cost figures are historical estimates based on the rates recorded with the benchmark runs. Provider prices, peak/off-peak rules, exchange rates and subscription limits can change; current commercial pages may therefore differ from the archived snapshot.