REVIEW 4 major objections 5 minor 22 references
Soft Guidance Starts to Outperform CoT Prompting as LLMs Improve
T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Zero-shot generation beats few-shot chain-of-thought on GSM8K for reasoning-tuned models.
desk verdict The Mathstral numbers are strong and worth knowing, but the paper misreads its own Table 1 for Qwen—the real few-shot ceiling is 87.6%, not 83.4%—so the zero-shot advantage there nearly vanishes. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the guidance-distraction tradeoff: CoT prompting supplies useful structure but also demands style imitation of human-written examples, adherence to answer formats such as '#### <number>', and contextual integration of unrelated examples; for models with native reasoning skills these costs dominate. Methodologically, the paper isolates the format effect by using randomly selected in-context examples, by separating rule-based from prompt-based answer extraction, and by comparing dataset examples against model-generated examples (Self-CoT). The distinction between reasoning accuracy and formatting compliance is what turns the raw score differences into evidence abo
What would settle it
Rerun the GSM8K comparison using the officially reported few-shot CoT baselines or retrieval- or complexity-selected examples and see whether they close the gap to the zero-shot scores; if they do, the guidance-distraction effect is an artifact of random example selection.
Extended reading notes
Core claim
The central discovery is that for models already trained to produce multi-step rationales, the format burden of CoT prompts outweighs their guidance. Concretely: Mathstral-7B rises from at most 74.2% under the best few-shot dataset CoT variant (or 77.1% officially reported) to 83.8% with zero-shot free-form generation; Qwen2.5-7B-Instruct rises from at most 83.4% to 88.6%; the general-purpose Llama-3.1-8B-Instruct also reaches its best results under zero-shot CoT. Model-generated exemplars help, but for Mathstral they still do not beat the zero-shot baseline. The authors name this the guidance-distraction tradeoff and conclude that few-shot CoT with dataset examples is no longer a reliable b
Load-bearing premise
The central comparison assumes that randomly selected few-shot examples fairly represent standard few-shot CoT evaluation, so the measured gap may largely reflect example quality rather than the CoT format if curated examples would score higher.
Editorial extensions
If this is right
- Few-shot CoT with dataset examples systematically underrates reasoning-specialized models: Mathstral's best DS-CoT result is 74.2% versus 83.8% zero-shot, and Qwen's is 83.4% versus 88.6% zero-shot.
- Zero-shot CoT ('Let's think step by step') still helps but gives only marginal gains for reasoning-optimized models (e.g., 86.1% vs 83.8% for Mathstral).
- Even model-generated CoT examples fail to beat the zero-shot baseline for the most reasoning-specialized model tested, suggesting the distraction is tied to enforcing an external format rather than example content alone.
- Standard few-shot CoT should no longer be the default evaluation baseline for reasoning models; zero-shot evaluation provides a more faithful and more competitive measure of capability.
- Performance gains attributed to new methods may be inflated when those methods are compared against a degraded few-shot CoT baseline.
Reading between the lines
- The trend suggests that as models are trained more heavily on reasoning traces, the optimal evaluation protocol may shift further toward unconstrained generation, with the 'Let's think step by step' prefix eventually becoming a mild handicap rather than an aid.
- The guidance-distraction logic should generalize beyond math to other domains where models natively produce structured rationales, such as symbolic logic or code; a direct test using existing benchmarks would be inexpensive.
- A practical extension is to adopt self-generated free-form exemplars plus prompt-based answer extraction as a stronger default baseline in future model reports, which would change how much of the reported gains of new prompting methods are real.
- The paper's random-example setup implies that curated example selection may still rescue few-shot CoT; testing this by comparing retrieval-selected examples against zero-shot generation would clarify whether the core effect is format or example quality.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper evaluates three mid-sized instruction-tuned LLMs (Mathstral-7B, Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct) on GSM8K under several prompting regimes: few-shot CoT with dataset examples, few-shot CoT with model-generated examples, and zero-shot baselines (free-form and "Let's think step by step"). The authors report that reasoning-specialized models perform better with zero-shot free-form or zero-shot CoT than with few-shot CoT using randomly selected dataset examples, and they attribute this to a 'guidance-distraction tradeoff': CoT prompts impose style and formatting burdens that become counterproductive as models learn to reason natively. They advocate replacing few-shot CoT with unconstrained zero-shot evaluation as the default baseline.
Significance. If the central empirical claim is robust, the paper would be a useful corrective to default few-shot CoT baselines in model evaluation. The study has several methodological strengths: it isolates the effect of prompt format by using randomly selected examples rather than retrieval-based selection; it separates reasoning quality from answer-format compliance via two extraction methods; it reports greedy decoding with 10 seeds; and it includes an ablation of a cleaning pipeline for self-generated demonstrations. The paper also candidly identifies its own limitations (single benchmark, mid-sized models, random selection). However, the central claim is currently not supported for one of the two reasoning-adjacent models because the text contradicts its own table, and the reported averages lack confidence intervals or significance tests. The headline contribution therefore rests heavily on Mathstral alone, and the paper's own internal inconsistency must be resolved before the conclusion can be accepted.
major comments (4)
- [§5.1, Table 1, Abstract] The paper repeatedly states that DS-CoT variants achieve no more than 83.4% for Qwen (Abstract; §5.1) and contrasts this with 88.6% zero-shot. This is contradicted by Table 1: DS-CoT+Prompt at 4-shot reaches 87.6%, and DS-CoT+Instr+Prompt at 2-shot reaches 84.0%. The 83.4% figure corresponds to only the 1-shot DS-CoT+Instr+Prompt configuration. Taking the best few-shot DS-CoT result (87.6%), the zero-shot advantage is 1.0 percentage point, well within sampling noise for 1,319 GSM8K test items. This does not support the claim that zero-shot 'significantly outperforms' few-shot CoT for Qwen. The abstract, contributions, and Section 5.1 must be corrected to use the best few-shot result, or the Qwen results must be presented with appropriate uncertainty.
- [§4.2, Table 1] The evaluation methodology defines answer extraction only for the CoT prompting variants (rule-based and prompt-based), but never specifies how the zero-shot free-form baseline is scored. Since the zero-shot baseline is the central comparator, the extraction procedure for that condition must be described. If a prompt-based extraction step is used, it is an additional prompt interaction and may not be a 'free-form' generation; if a rule is used, the rule must be stated because free-form outputs need not contain '#### <number>'.
- [§4.2, Table 1] All numbers are averages over 10 seeds with no standard deviations, confidence intervals, or pairwise significance tests. For large gaps such as Mathstral 83.8 vs. 74.2 the conclusion is likely stable, but for Qwen the best few-shot result (87.6) versus zero-shot (88.6) is a 1.0-point difference. Without variance estimates or a paired test, the paper cannot support 'significantly outperforms' for Qwen. Reporting the standard error of the 10-seed average and, ideally, per-seed paired differences would address this concern directly.
- [§1, §5.1] The comparison to 'officially reported results' is not controlled. The paper uses randomly selected examples throughout, whereas official baselines (e.g., Mathstral's 77.1%) typically use a fixed or tuned set of demonstrations and may use different decoding protocols. The paper's own footnote 1 says the official Mathstral accuracy could not be reproduced, and §5.3 concedes that advanced example selection can still improve performance. The claim that zero-shot 'significantly surpasses officially reported results' is therefore confounded by example-selection strategy and evaluation protocol. The internal random-example comparisons are cleaner and should be made the primary claim; the official-report comparison should be framed as suggestive rather than definitive.
minor comments (5)
- [Throughout] Typos and grammatical slips: 'Baslines' in Table 1 header; 'few-show' in §2; 'autonated' in §4.2; 'a sources of interference' in §3; 'more advance' in §1; 'this conclusion aligns' capitalization in §5.3. These should be corrected.
- [Table 1] The column header 'Self-CoT free form + 0-shot free form' is confusing: it appears to list two Self-CoT variants but the 'free form' label is duplicated/modified. The header should clearly separate 'Self-CoT (free-form)' and 'Self-CoT+0shot'.
- [§6] The 'guidance-distraction tradeoff' is presented as the explanation for the results, but the paper does not directly measure distraction, style adaptation, or formatting burden. It is a plausible post-hoc interpretation rather than a testable mechanism. This is not fatal to the empirical comparison, but the wording should remain explicitly interpretive.
- [§5.2] The paper distinguishes between benchmark performance and rationale-generation proficiency, but does not provide a direct measure of rationale quality. The claim that Mathstral is 'likely superior in rationale generation' would benefit from a quantitative assessment (e.g., human or LLM-judged rationale quality on a sample).
- [§4.2] The description of the post-processing pipeline for self-generated examples says 'two-stage autonated cleaning' but the details of the 'simple automated check' for incomplete reasoning are not given. Since Table 3 reports the effect of this cleaning, the check should be specified for reproducibility.
Circularity Check
No circularity: the paper reports direct empirical comparisons; the guidance-distraction tradeoff is a post-hoc interpretation, not a fitted parameter or self-referential derivation.
full rationale
The paper contains no equation-level derivation chain to reduce. Its central claims are direct accuracy measurements on GSM8K under different prompting formats, and the 'guidance-distraction tradeoff' is introduced after the results as an explanatory hypothesis, not as a construct defined in terms of the outcome. No parameter is fitted to data and no 'prediction' is derived from the tradeoff; removing the tradeoff language leaves the measured accuracies intact. The paper does not rely on load-bearing self-citations: the cited prior work (Wei et al., Kojima et al., Zhang et al., etc.) is external and not from the present authors, and no uniqueness theorem or ansatz is imported by citation. The limitations flagged by the paper itself—random example selection, single benchmark, and inability to reproduce Mathstral's official reported number—are validity concerns, not circularity. Likewise, the skeptic's observation that Section 5.1 reports Qwen DS-CoT as 'no more than 83.4%' while Table 1 shows 87.6% for DS-CoT+Prompt at 4-shot is a factual/reporting inconsistency that weakens the empirical conclusion, but it does not make any result equivalent to its inputs by construction. Overall, this is an empirical measurement paper, and no circular step is present.
Assumptions & free parameters
assumptions (3)
- domain assumption Randomly selected few-shot examples are representative of 'standard' few-shot CoT baselines.
- domain assumption The answer extraction methods measure reasoning accuracy without systematic bias.
- domain assumption Averages over 10 greedy-decoding runs provide a stable estimate of performance.
Cite this review
Pith. "Pith review of Soft Guidance Starts to Outperform CoT Prompting as LLMs Improve." pith.science (2026). https://pith.science/paper/X52YDGE3
@misc{pith2026260803550,
author = {Pith},
title = {Pith review of: Soft Guidance Starts to Outperform CoT Prompting as LLMs Improve},
year = {2026},
howpublished = {\url{https://pith.science/paper/X52YDGE3}},
note = {Machine review of arXiv:2608.03550}
}
abstract
Chain-of-Thought (CoT) prompting remains the standard baseline for evaluating models' reasoning abilities. Originally, this technique was introduced to elicit step-by-step reasoning from large language models (LLMs), which would otherwise tend to directly output the final answer. However, many modern LLMs produce CoT-style responses \textit{natively} when presented with reasoning tasks, which made us revisit the effectiveness of standard CoT prompting. We evaluate several modern mid-sized language models on a math problem-solving task and find that models specialized for reasoning achieve better performance in a simple zero-shot setting than when using few-shot CoT examples - significantly surpassing officially reported results at no additional cost (e.g., from $\sim$77\% to $\sim$84\% for Mathstral on GSM8K). For the tested general-purpose model, a zero-shot CoT prompt is also sufficient to outperform a few-shot CoT baseline. We attribute this to a `guidance-distraction' tradeoff: standard CoT prompting also demands style adaptation, formatting compliance, and potentially undesired contextualization, which can distract models from the core reasoning task. Our findings suggest that using standard CoT prompting increasingly acts as a source of distraction as models grow stronger.
Figures
Reference graph
Works this paper leans on
-
[1]
Advances in neural information processing systems , volume=
Chain-of-thought prompting elicits reasoning in large language models , author=. Advances in neural information processing systems , volume=
-
[2]
arXiv preprint arXiv:2404.14963 , year=
Achieving> 97\ author=. arXiv preprint arXiv:2404.14963 , year=
-
[3]
arXiv preprint arXiv:2305.04091 , year=
Plan-and-solve prompting: improving zero-shot chain-of-thought reasoning by large language models (2023) , author=. arXiv preprint arXiv:2305.04091 , year=
arXiv 2023
-
[4]
arXiv preprint arXiv:2211.12588 , year=
Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks , author=. arXiv preprint arXiv:2211.12588 , year=
-
[5]
arXiv preprint arXiv:2110.14168 , year=
Training verifiers to solve math word problems , author=. arXiv preprint arXiv:2110.14168 , year=
-
[6]
Advances in neural information processing systems , volume=
Large language models are zero-shot reasoners , author=. Advances in neural information processing systems , volume=
-
[7]
AlignedCoT: Prompting Large Language Models via Native-Speaking Demonstrations
AlignedCoT: Prompting Large Language Models via Native-Speaking Demonstrations , author=. arXiv preprint arXiv:2311.13538 , year=
-
[8]
arXiv preprint arXiv:2403.20046 , year=
Can LLMs Learn from Previous Mistakes? Investigating LLMs' Errors to Boost for Reasoning , author=. arXiv preprint arXiv:2403.20046 , year=
Show all 22 references
-
[9]
arXiv preprint arXiv:2103.03874 , year=
Measuring mathematical problem solving with the math dataset , author=. arXiv preprint arXiv:2103.03874 , year=
-
[10]
5 technical report , author=
Qwen2. 5 technical report , author=. arXiv preprint arXiv:2412.15115 , year=
-
[11]
2024 , howpublished =
Mistral AI , title =. 2024 , howpublished =
2024
-
[12]
2024 , howpublished =
Meta AI , title =. 2024 , howpublished =
2024
-
[13]
arXiv preprint arXiv:2210.03493 , year=
Automatic chain of thought prompting in large language models , author=. arXiv preprint arXiv:2210.03493 , year=
-
[14]
Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd workshop on knowledge extraction and integration for deep learning architectures , pages=
What makes good in-context examples for GPT-3? , author=. Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd workshop on knowledge extraction and integration for deep learning architectures , pages=
2022
-
[15]
Proceedings of the 2022 conference of the North American chapter of the association for computational linguistics: human language technologies , pages=
Learning to retrieve prompts for in-context learning , author=. Proceedings of the 2022 conference of the North American chapter of the association for computational linguistics: human language technologies , pages=
2022
-
[16]
arXiv preprint arXiv:2205.10625 , year=
Least-to-most prompting enables complex reasoning in large language models , author=. arXiv preprint arXiv:2205.10625 , year=
-
[17]
ACL , year=
Language models are better in-context learners when matching example types , author=. ACL , year=
-
[18]
arXiv preprint arXiv:2210.00720 , year=
Complexity-based prompting for multi-step reasoning , author=. arXiv preprint arXiv:2210.00720 , year=
-
[19]
International Conference on Machine Learning , pages=
Compositional exemplars for in-context learning , author=. International Conference on Machine Learning , pages=. 2023 , organization=
2023
-
[20]
Advances in neural information processing systems , volume=
Transformers as statisticians: Provable in-context learning with in-context algorithm selection , author=. Advances in neural information processing systems , volume=
-
[21]
arXiv preprint arXiv:2302.12246 , year=
Active prompting with chain-of-thought for large language models , author=. arXiv preprint arXiv:2302.12246 , year=
-
[22]
arXiv preprint arXiv:2506.14641 , year=
Revisiting chain-of-thought prompting: Zero-shot can be stronger than few-shot , author=. arXiv preprint arXiv:2506.14641 , year=
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.