{"id":"ced38947-2a5e-4b0f-8cde-60336366f2b6","arxiv_id":"2608.03550","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On GSM8K, zero-shot free-form generation beats few-shot CoT prompting for Mathstral, Qwen2.5, and Llama-3.1, suggesting standard CoT baselines can underestimate modern reasoning models.","lead":"This paper tests three mid-sized large language models on math word problems and finds that letting them answer in their native style, with no few-shot examples, beats standard chain-of-thought prompting. The authors argue that as models improve at reasoning on their own, CoT prompts become a distraction, and recommend zero-shot evaluation as a fairer baseline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 contradicts the claimed Qwen few-shot ceiling: best DS-CoT+Prompt is 87.6%, not 83.4%, cutting the zero-shot advantage to ~1 point.","rationale":"The reader's weakest_assumption concerns random vs curated examples, which is a valid external-validity concern. I find a more direct internal-validity problem: the paper's cited Qwen few-shot ceiling (83.4%) is contradicted by its own Table 1. Recomputing the maximum gives 87.6%, reducing the claimed gap from 5.2 to 1.0 points. This does not overturn the Mathstral result (74.2% vs 83.8%), so the overall verdict should remain CONDITIONAL, but the paper must correct its Qwen reporting and clarify the answer-extraction protocol for zero-shot baselines before the broad claim is accepted. The concern is concrete and testable by recomputation.","tokens_in":9392,"tokens_out":12730,"duration_ms":122080,"concrete_test":"Recompute the maximum accuracy over all DS-CoT and DS-CoT+Instr variants with prompt-based extraction for Qwen2.5-7B-Instruct from Table 1, specifically checking whether the 83.4% figure cited in §5.1 and the contributions is the maximum or a cherry-picked subconfiguration (DS-CoT+Instr+Prompt 1-shot). If the max is 87.6% (DS-CoT+Prompt 4-shot), then the claimed gap of 5.2 points reduces to 1.0 point, and the central claim for Qwen is not supported. Also verify the same for Mathstral: the cited 74.2% should be the max across all four dataset-based variants.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline comparison for Qwen is internally inconsistent with its own Table 1. Section 5.1 states 'DS-CoT variants achieve no more than 83.4%, while zero-shot prompting yields 88.6%,' and the contributions repeat 'from ≤83.4% to 88.6%.' However, Table 1 shows DS-CoT+Prompt (without +Instr) reaching 87.6% at 4-shot for Qwen, and DS-CoT+Instr+Prompt reaching 84.0% at 2-shot. The 83.4% figure corresponds to the 1-shot DS-CoT+Instr+Prompt configuration, which is not the maximum. When the best few-shot result (87.6%) is compared to zero-shot free-form (88.6%), the advantage is only 1.0 percentage point—well within the expected sampling noise for 1,319 GSM8K test items. Moreover, the paper itself argues in §5.4 that rule-based extraction confounds formatting compliance, yet the low 83.4% appears to be drawn from a variant with answer-format instructions and prompt-based extraction at an unfavorable shot count; it is not a robust best-case baseline. This materially weakens the claim that zero-shot free-form 'significantly outperforms' few-shot CoT for Qwen, and leaves Mathstral as the sole strong support for the central claim. The issue is not about example selection strategy; it is a factual inconsistency in the paper's reporting of its own results.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates three mid-sized instruction-tuned LLMs (Mathstral-7B, Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct) on GSM8K under several prompting regimes: few-shot CoT with dataset examples, few-shot CoT with model-generated examples, and zero-shot baselines (free-form and \"Let's think step by step\"). The authors report that reasoning-specialized models perform better with zero-shot free-form or zero-shot CoT than with few-shot CoT using randomly selected dataset examples, and they attribute this to a 'guidance-distraction tradeoff': CoT prompts impose style and formatting burdens that become counterproductive as models learn to reason natively. They advocate replacing few-shot CoT with unconstrained zero-shot evaluation as the default baseline.","tokens_in":9766,"tokens_out":4299,"duration_ms":48690,"significance":"If the central empirical claim is robust, the paper would be a useful corrective to default few-shot CoT baselines in model evaluation. The study has several methodological strengths: it isolates the effect of prompt format by using randomly selected examples rather than retrieval-based selection; it separates reasoning quality from answer-format compliance via two extraction methods; it reports greedy decoding with 10 seeds; and it includes an ablation of a cleaning pipeline for self-generated demonstrations. The paper also candidly identifies its own limitations (single benchmark, mid-sized models, random selection). However, the central claim is currently not supported for one of the two reasoning-adjacent models because the text contradicts its own table, and the reported averages lack confidence intervals or significance tests. The headline contribution therefore rests heavily on Mathstral alone, and the paper's own internal inconsistency must be resolved before the conclusion can be accepted.","major_comments":[{"comment":"The paper repeatedly states that DS-CoT variants achieve no more than 83.4% for Qwen (Abstract; §5.1) and contrasts this with 88.6% zero-shot. This is contradicted by Table 1: DS-CoT+Prompt at 4-shot reaches 87.6%, and DS-CoT+Instr+Prompt at 2-shot reaches 84.0%. The 83.4% figure corresponds to only the 1-shot DS-CoT+Instr+Prompt configuration. Taking the best few-shot DS-CoT result (87.6%), the zero-shot advantage is 1.0 percentage point, well within sampling noise for 1,319 GSM8K test items. This does not support the claim that zero-shot 'significantly outperforms' few-shot CoT for Qwen. The abstract, contributions, and Section 5.1 must be corrected to use the best few-shot result, or the Qwen results must be presented with appropriate uncertainty.","section":"§5.1, Table 1, Abstract"},{"comment":"The evaluation methodology defines answer extraction only for the CoT prompting variants (rule-based and prompt-based), but never specifies how the zero-shot free-form baseline is scored. Since the zero-shot baseline is the central comparator, the extraction procedure for that condition must be described. If a prompt-based extraction step is used, it is an additional prompt interaction and may not be a 'free-form' generation; if a rule is used, the rule must be stated because free-form outputs need not contain '#### <number>'.","section":"§4.2, Table 1"},{"comment":"All numbers are averages over 10 seeds with no standard deviations, confidence intervals, or pairwise significance tests. For large gaps such as Mathstral 83.8 vs. 74.2 the conclusion is likely stable, but for Qwen the best few-shot result (87.6) versus zero-shot (88.6) is a 1.0-point difference. Without variance estimates or a paired test, the paper cannot support 'significantly outperforms' for Qwen. Reporting the standard error of the 10-seed average and, ideally, per-seed paired differences would address this concern directly.","section":"§4.2, Table 1"},{"comment":"The comparison to 'officially reported results' is not controlled. The paper uses randomly selected examples throughout, whereas official baselines (e.g., Mathstral's 77.1%) typically use a fixed or tuned set of demonstrations and may use different decoding protocols. The paper's own footnote 1 says the official Mathstral accuracy could not be reproduced, and §5.3 concedes that advanced example selection can still improve performance. The claim that zero-shot 'significantly surpasses officially reported results' is therefore confounded by example-selection strategy and evaluation protocol. The internal random-example comparisons are cleaner and should be made the primary claim; the official-report comparison should be framed as suggestive rather than definitive.","section":"§1, §5.1"}],"minor_comments":[{"comment":"Typos and grammatical slips: 'Baslines' in Table 1 header; 'few-show' in §2; 'autonated' in §4.2; 'a sources of interference' in §3; 'more advance' in §1; 'this conclusion aligns' capitalization in §5.3. These should be corrected.","section":"Throughout"},{"comment":"The column header 'Self-CoT free form + 0-shot free form' is confusing: it appears to list two Self-CoT variants but the 'free form' label is duplicated/modified. The header should clearly separate 'Self-CoT (free-form)' and 'Self-CoT+0shot'.","section":"Table 1"},{"comment":"The 'guidance-distraction tradeoff' is presented as the explanation for the results, but the paper does not directly measure distraction, style adaptation, or formatting burden. It is a plausible post-hoc interpretation rather than a testable mechanism. This is not fatal to the empirical comparison, but the wording should remain explicitly interpretive.","section":"§6"},{"comment":"The paper distinguishes between benchmark performance and rationale-generation proficiency, but does not provide a direct measure of rationale quality. The claim that Mathstral is 'likely superior in rationale generation' would benefit from a quantitative assessment (e.g., human or LLM-judged rationale quality on a sample).","section":"§5.2"},{"comment":"The description of the post-processing pipeline for self-generated examples says 'two-stage autonated cleaning' but the details of the 'simple automated check' for incomplete reasoning are not given. Since Table 3 reports the effect of this cleaning, the check should be specified for reproducibility.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and the central idea is interesting, but the internal inconsistency between the Qwen results in Table 1 and the claims in §5.1/Abstract is a load-bearing issue that must be fixed. The authors should also add uncertainty quantification and clarify the zero-shot extraction protocol. I would not recommend rejection on the merits of the core comparison for Mathstral, but the current version does not support the broad claim as stated. If the authors correct the Qwen reporting and temper the official-baseline comparison, the paper could become acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the Mathstral result is real and worth knowing, but the paper misreads its own Table 1 for Qwen, and that error weakens the headline more than the authors seem to realize.\n\nWhat's good: the paper runs a clean controlled comparison of prompt formats—dataset examples vs self-generated, with and without answer-format instructions, and with two extraction methods—using random example selection specifically to isolate format effects. That's the right design for the question. The Mathstral numbers are dramatic: best few-shot CoT with dataset examples is 74.2%, while free-form zero-shot hits 83.8%. The guidance-distraction tradeoff is a plausible explanation, and they honestly cite concurrent work by Cheng et al. reporting the same headline conclusion.\n\nNow the soft spots. The Qwen claim does not survive a close read of Table 1. Section 5.1 and the contributions say DS-CoT variants achieve at most 83.4%, but the table shows DS-CoT+Prompt (without the +Instr instruction) reaching 87.6% at 4-shot. So the actual best few-shot result for Qwen is 1.0 point below zero-shot free-form, not 5.2 points. That's within the kind of sampling noise you'd expect from 1,319 test items, especially with no error bars or significance tests reported. The gap for Mathstral is ~10 points, so it carries the paper; Qwen should not be billed as a second example.\n\nTwo more issues. The zero-shot free-form answer extraction method is never described—no rule-based or prompt-based method is specified for the baseline that the whole claim rests on. And the paper deliberately uses random example selection, which is fine for isolating format, but then it compares against officially reported few-shot baselines that used tuned prompts. The 77.1% vs 83.8% comparison for Mathstral is real, but the paper's inability to reproduce 77.1% deserves more discussion.\n\nOverall: the central claim is probably right for reasoning-specialized models, and Mathstral is strong evidence. But the Qwen misreport and the missing extraction details mean the paper needs revision before its numbers can be trusted. A serious referee would catch these. I'd send it out, but with a request for corrected reporting, error bars, and a clear extraction protocol.","headline":"The Mathstral numbers are strong and worth knowing, but the paper misreads its own Table 1 for Qwen—the real few-shot ceiling is 87.6%, not 83.4%—so the zero-shot advantage there nearly vanishes.","tokens_in":10196,"tokens_out":4418,"would_cite":false,"duration_ms":44509,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Zero-shot generation beats few-shot chain-of-thought on GSM8K for reasoning-tuned models.","keywords":["Chain-of-Thought prompting","zero-shot prompting","guidance-distraction tradeoff","GSM8K","math reasoning","in-context learning","evaluation baselines","reasoning-specialized LLMs"],"falsifier":"Rerun the GSM8K comparison using the officially reported few-shot CoT baselines or retrieval- or complexity-selected examples and see whether they close the gap to the zero-shot scores; if they do, the guidance-distraction effect is an artifact of random example selection.","tokens_in":9347,"feed_emoji":"🧮","tokens_out":6744,"duration_ms":65100,"temperature":0.7,"pith_summary":"The paper argues that few-shot chain-of-thought prompting, once essential for eliciting step-by-step reasoning, has become a net distraction for modern reasoning-specialized LLMs. On GSM8K, zero-shot free-form generation beats the best few-shot dataset CoT variants by roughly 9-10 points for Mathstral and about 5 points for Qwen. The proposed cause is a guidance-distraction tradeoff: CoT examples impose style adaptation, formatting compliance, and unneeded contextualization that compete with the core math reasoning. If true, this matters because few-shot CoT remains the standard evaluation baseline, meaning official benchmarks may underestimate current models and overstate the gains of new methods. The paper recommends replacing the default baseline with minimally constrained zero-shot evaluation.","feed_headline":"Zero-shot beats few-shot chain-of-thought for math reasoning models","feed_subtitle":"For reasoning-specialized models, standard CoT examples act as a distraction, new GSM8K results show.","key_machinery":"The load-bearing mechanism is the guidance-distraction tradeoff: CoT prompting supplies useful structure but also demands style imitation of human-written examples, adherence to answer formats such as '#### <number>', and contextual integration of unrelated examples; for models with native reasoning skills these costs dominate. Methodologically, the paper isolates the format effect by using randomly selected in-context examples, by separating rule-based from prompt-based answer extraction, and by comparing dataset examples against model-generated examples (Self-CoT). The distinction between reasoning accuracy and formatting compliance is what turns the raw score differences into evidence abo","core_discovery":"The central discovery is that for models already trained to produce multi-step rationales, the format burden of CoT prompts outweighs their guidance. Concretely: Mathstral-7B rises from at most 74.2% under the best few-shot dataset CoT variant (or 77.1% officially reported) to 83.8% with zero-shot free-form generation; Qwen2.5-7B-Instruct rises from at most 83.4% to 88.6%; the general-purpose Llama-3.1-8B-Instruct also reaches its best results under zero-shot CoT. Model-generated exemplars help, but for Mathstral they still do not beat the zero-shot baseline. The authors name this the guidance-distraction tradeoff and conclude that few-shot CoT with dataset examples is no longer a reliable b","pith_inferences":["The trend suggests that as models are trained more heavily on reasoning traces, the optimal evaluation protocol may shift further toward unconstrained generation, with the 'Let's think step by step' prefix eventually becoming a mild handicap rather than an aid.","The guidance-distraction logic should generalize beyond math to other domains where models natively produce structured rationales, such as symbolic logic or code; a direct test using existing benchmarks would be inexpensive.","A practical extension is to adopt self-generated free-form exemplars plus prompt-based answer extraction as a stronger default baseline in future model reports, which would change how much of the reported gains of new prompting methods are real.","The paper's random-example setup implies that curated example selection may still rescue few-shot CoT; testing this by comparing retrieval-selected examples against zero-shot generation would clarify whether the core effect is format or example quality."],"forward_implications":["Few-shot CoT with dataset examples systematically underrates reasoning-specialized models: Mathstral's best DS-CoT result is 74.2% versus 83.8% zero-shot, and Qwen's is 83.4% versus 88.6% zero-shot.","Zero-shot CoT ('Let's think step by step') still helps but gives only marginal gains for reasoning-optimized models (e.g., 86.1% vs 83.8% for Mathstral).","Even model-generated CoT examples fail to beat the zero-shot baseline for the most reasoning-specialized model tested, suggesting the distraction is tied to enforcing an external format rather than example content alone.","Standard few-shot CoT should no longer be the default evaluation baseline for reasoning models; zero-shot evaluation provides a more faithful and more competitive measure of capability.","Performance gains attributed to new methods may be inflated when those methods are compared against a degraded few-shot CoT baseline."],"supporting_citations":[{"why":"Introduces the few-shot CoT prompting format that the paper re-evaluates and finds distracting.","marker":"Wei et al. (2022)"},{"why":"Introduces zero-shot CoT prompting, the 'Let's think step by step' baseline the paper compares against.","marker":"Kojima et al. (2022)"},{"why":"Provides the GSM8K dataset, its training examples, test split, and answer format used in all experiments.","marker":"Cobbe et al. (2021)"},{"why":"Defines Mathstral-7B and the official 77.1% few-shot CoT baseline that zero-shot free-form surpasses.","marker":"(AI, 2024b)"},{"why":"Defines Qwen2.5-7B-Instruct, the general-purpose model with reported reasoning strengths used in the study.","marker":"Yang et al. (2024)"},{"why":"Defines Llama-3.1-8B-Instruct, the general-purpose model used as the non-reasoning-specialized contrast.","marker":"(AI, 2024a)"},{"why":"Supplies Auto-CoT, the model-generated example format and style-alignment intuition the paper tests as Self-CoT.","marker":"Zhang et al. (2022)"},{"why":"Supplies Active-Prompt, an advanced example-selection CoT method the paper positions against its random-selection setup.","marker":"Diao et al. (2023)"},{"why":"Concurrent study agreeing that zero-shot prompting outperforms few-shot prompting for modern LLMs on math problems.","marker":"Cheng et al. (2025)"}],"fun_headline_variants":["Zero-shot math reasoning tops few-shot CoT","CoT examples distract math models, zero-shot wins","Reasoning-specialized LLMs prefer zero-shot over CoT","Few-shot CoT now slows math reasoning models"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The central comparison assumes that randomly selected few-shot examples fairly represent standard few-shot CoT evaluation, so the measured gap may largely reflect example quality rather than the CoT format if curated examples would score higher.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot math reasoning tops few-shot CoT","CoT examples distract math models, zero-shot wins","Reasoning-specialized LLMs prefer zero-shot over CoT","Few-shot CoT now slows math reasoning models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000623,"raw_usage":{"total_tokens":2742,"prompt_tokens":783,"completion_tokens":1959,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":1895}},"tokens_in":527,"tokens_out":1959,"duration_ms":15927,"temperature":1.0,"reasoning_tokens":1895,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T16:56:55.184926+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the GSM8K comparison using the officially reported few-shot CoT baselines or retrieval- or complexity-selected examples and see whether they close the gap to the zero-shot scores; if they do, the guidance-distraction effect is an artifact of random example selection.","supporting_citations":[],"review_version":1}