{"id":"db15ddc5-db49-4383-b104-1620da364046","arxiv_id":"2506.07142","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"On the GPQA Diamond benchmark, chain-of-thought prompting gives only small average accuracy gains for non-reasoning models, little or no gain for reasoning models, and large increases in response time and tokens.","lead":"This report tests whether asking ChatGPT-style models to think step by step actually improves answers on hard PhD-level multiple-choice questions. It finds the benefit is small and uneven: it raises average accuracy for non-reasoning models, but can add errors on easy questions and costs much more time and tokens.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The main evidence that CoT improves non-reasoning models compares CoT to a 'Direct' prompt that explicitly forbids thinking; under its own 'Default' baseline, 4 of 5 models show no significant average gain, so the claim that CoT is 'generally useful' rests on a handicapped control.","rationale":"The paper is transparent about the Default comparison and about its limitations, and the internal inconsistency in the Table S2/S3 notes about trial thresholds is secondary. The baseline issue is the load-bearing one because the abstract and conclusion advance a general usefulness claim while the evidence under Default is mostly null. The concern does not destroy the paper's more modest claim that CoT is not a universal accuracy booster, which is well supported by the reasoning-model results and the cost data. It does mean the central 'generally useful' claim needs either a control that does not forbid thinking or a reframing as a comparison against a deliberately suppressed baseline. Since the reader already arrived at CONDITIONAL for essentially this reason, the conditional verdict stands without further adjustment.","tokens_in":12128,"tokens_out":6093,"duration_ms":67265,"concrete_test":"Re-run the GPQA Diamond protocol on the same five non-reasoning models with an additional 'Answer only' condition using 'Provide only the correct answer' without 'without any explanation or thinking', while keeping the formatting constraint used in Direct and Step-by-Step. Compare Step-by-Step versus this new baseline. If the average RD shrinks toward the small and mostly null Default-versus-CoT values in Table S4, the suppression instruction is the active ingredient and the 'generally useful' framing must be downgraded. As a secondary check, code Default responses for reasoning tokens before the answer and regress the CoT-minus-Default average RD on that rate across models; if no monotonic relationship appears, remove the 'particularly when the model does not already engage in step-by-step processing' qualifier.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Location: 'How we Prompt the AI' and Tables S4/S5. The Direct condition is 'Answer directly without any explanation or thinking. Just provide the answer.' The Step-by-Step condition only adds 'Think step by step.' So the statistically significant RD values in Table S3 measure CoT against a baseline that is instructed to suppress the very default reasoning that recent non-reasoning models produce (Table S1). The paper includes a Default condition that removes both the formatting constraint and the no-thinking instruction. In Table S4, CoT versus Default is significant for Average only for Gemini Flash 2.0 (RD = 0.062, p < .001) and GPT-4o (RD = 0.069, p = 0.003); Sonnet 3.5 (RD = -0.019, p = 0.189) and GPT-4o-mini (RD = 0.004, p = 0.728) are null. Thus the abstract's 'generally improves average performance' is driven by the Direct comparison, not by the natural usage baseline. Additionally, the qualifier 'particularly if the model does not inherently engage in step-by-step processing by default' is never operationalized: no per-model measure of spontaneous reasoning is reported, making the subset claim post-hoc. The Discussion's statement that 'answer only' prompting is 'likely to harm' non-reasoning models is also not isolated, because the Direct prompt conflates answer-only with 'without thinking.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript reports a large-scale benchmarking study of simple Chain-of-Thought (CoT) prompting on the 198-question GPQA Diamond dataset, covering five non-reasoning models (Sonnet 3.5, Gemini 2.0 Flash, GPT-4o-mini, GPT-4o, Gemini Pro 1.5) and three reasoning models (o3-mini, o4-mini, Gemini Flash 2.5), with 25 trials per question and condition (4,950 runs per condition per model). The authors compare three prompt variants — “Answer directly without any explanation or thinking” (Direct), “Think step by step” (CoT), and no suffix with no formatting constraint (Default) — across four metrics (average proportion correct plus per-question 100%, 90%, and 51% thresholds). Paired bootstrap permutation tests show that CoT raises average accuracy relative to Direct for most non-reasoning models while often decreasing the 100%-correct rate, increases latency by 35–600% for these models, and yields only marginal accuracy changes (RD ≈ 0.03 in either direction) for reasoning models at substantially higher latency. The paper concludes that CoT is a conditional, cost-sensitive technique whose value depends on model type, baseline behavior, and the accuracy metric, and that many recent models already perform CoT-like reasoning unprompted.","tokens_in":12445,"tokens_out":16549,"duration_ms":142219,"significance":"If the empirical patterns hold, this report makes a useful practical contribution: falsifiable, parameter-free measurements of a widely adopted prompting technique on a standardized benchmark, with per-model effect sizes, confidence intervals, and cost data practitioners can act on. The manuscript’s strengths include direct measurement against an external benchmark with no fitted parameters, paired bootstrap permutation tests with 5,000 replicates, transparent reporting of both beneficial and harmful effects (including declines on the 100%-correct metric), explicit acknowledgement of limitations (single benchmark, limited model set, simple prompt variants), and a robustness check in Figure S3 showing that more sophisticated CoT prompt variants yield negligible differences. The finding that recent models self-generate CoT-like reasoning by default, and the quantification of latency and token costs of explicit CoT prompting for reasoning models, are the most novel and practically relevant elements.","major_comments":[{"comment":"The main CoT-versus-baseline comparison is built on a prompt that actively suppresses default model behavior, conflating two distinct questions. The Direct condition instructs the model to “Answer directly without any explanation or thinking. Just provide the answer,” and the Step-by-Step condition only appends “Think step by step,” so the statistically significant average gains in Table S3 (Sonnet 3.5 RD = 0.117, Gemini Flash 2.0 RD = 0.135, both p < .001) measure CoT against a baseline told not to think rather than against natural usage. The manuscript’s own Default condition, which removes both the formatting constraint and the no-thinking instruction, shows significant average CoT gains for only two of the four models reported in Table S4 (Gemini Flash 2.0 RD = 0.062, p < .001; GPT-4o RD = 0.069, p = 0.003), with null results for Sonnet 3.5 (RD = -0.019, p = 0.189) and GPT-4o-mini (RD = 0.004, p = 0.728), and Gemini Pro 1.5 is missing from both the text and Table S4. Consequently, the abstract’s claim that CoT “generally improves average performance” for non-reasoning models, and the Discussion’s claim that “answer only” prompting is “likely to harm” such models (which cannot be separated from the “without thinking” instruction), are stronger than the evidence supports. The Discussion’s assertion that models were “fairly consistent in their responses to CoT” is also difficult to square with the heterogeneous 100%-metric RDs in Table S3, which range from +0.101 (Sonnet 3.5) to -0.172 (Gemini Pro 1.5). Please reframe the principal claims as baseline-dependent, report the Default condition for all five non-reasoning models, and lead the practical recommendations with the Default comparisons if the intended reference point is ordinary chatbot use.","section":"How we Prompt the AI; Figure 2; Tables S3–S4"},{"comment":"The qualifier “particularly if the model does not inherently engage in step-by-step processing by default” is never operationalized. The only evidence for spontaneous CoT reasoning is a single worked example in Table S1 (GPT-4o) plus a general statement in “How we Prompt the AI”; no per-model measure of default reasoning behavior (such as the fraction of Default-condition responses containing reasoning tokens before the answer, or blind coding of a sample of Default outputs) is reported. Without such a measure, the subset claim that CoT helps most when models do not already reason step by step is post-hoc and unfalsifiable within this report. Please add a quantitative per-model measure of spontaneous reasoning and test whether CoT gains correlate with it, or explicitly label the claim as a hypothesis for future work.","section":"Abstract; Discussion and Conclusion"},{"comment":"The trial-count and threshold documentation is internally inconsistent and needs reconciliation. Table S2 compares “100% Correct” at 25 trials (all 25/25) with “100% Correct” at 100 trials (all 100/100), which are different estimands by construction; the observed difference (0.185 [0.162, 0.207] versus 0.147 [0.136, 0.162]) therefore cannot support the note’s conclusion that 25 trials provide “estimation precision and statistical power comparable to 100 trials.” Additionally, the notes to Tables S3, S4, and S5 state that “Questions were considered 'perfect' at ≥100 successes,” which is impossible under the stated 25-trial design and contradicts the definitions in “How we Benchmark the AI” (thresholds of 25/25, 23/25, and 13/25). Because the reported 100%-metric risk differences (e.g., Gemini Flash 2.0 RD = -0.131 and Gemini Pro 1.5 RD = -0.172 in Table S3) are load-bearing for the claim that CoT trades average gains for increased answer variability, please state explicitly which thresholds produced each reported value and correct the table notes accordingly.","section":"Tables S2–S5"},{"comment":"The manuscript performs roughly 48 hypothesis tests across Tables S3–S5 (five non-reasoning models in Table S3, four in Table S4, three reasoning models in Table S5, each across four metrics) and reports raw p-values with no multiple-comparison correction or false-discovery-rate control. Several claims in the Results section rest on p-values that would not survive a per-table Bonferroni correction: GPT-4o at 51% (p = 0.042), Gemini Pro 1.5 Average (p = 0.014), o3-mini Average (p = 0.024, Table S5), and Flash 2.5 Average (p = 0.005). The headline effects (Sonnet 3.5 and Gemini Flash 2.0 Average, both p < .001) are robust, so the direction of the main findings is probably unaffected, but the text should apply a correction, report q-values, or explicitly state which conclusions depend on uncorrected marginal p-values.","section":"Tables S3–S5; Results"}],"minor_comments":[{"comment":"The figure legends report N = 197 per condition (and 196 for Flash 2.5), while the main text states that the GPQA Diamond set comprises 198 questions and that each condition involves 4,950 runs per model; please reconcile these numbers or state that one question was excluded from the timing analyses.","section":"Figures S1–S2"},{"comment":"Please define the abbreviation “RD” (risk difference) at its first use in the Results section, and fix the small textual errors in the first results paragraph (“GPT-4o’s saw” should be “GPT-4o saw,” and the passage “Google-proof)” contains a stray closing parenthesis).","section":"Results"},{"comment":"The example responses in Table S1 are truncated mid-sentence (for instance, the Step-by-Step column breaks off during the option analysis), and the Default column’s final answer is given only as “C.” despite the preceding hedged discussion; please provide complete or clearly marked excerpts so readers can verify the qualitative claim about default CoT behavior.","section":"Table S1"},{"comment":"The “Think step by step” instruction is the zero-shot CoT phrasing popularized by Kojima et al. (2022), whereas the cited Wei et al. (2022) work is few-shot CoT; please cite Kojima et al. as well, and also cite the source for the “Answer directly without any explanation or thinking” instruction if it follows an established protocol.","section":"References"},{"comment":"The sentence stating that PASS@N and CONSENSUS are “not useful” for benchmarking real-world applications is a strong normative claim presented without references or supporting analysis; please add a brief justification or soften the wording.","section":"How we Benchmark the AI"}],"recommendation":"major_revision","confidential_remarks":"This is a preprint-style report from a series rather than a self-contained methods paper, and the incremental contribution over the existing CoT literature is modest; however, the transparency of the effect-size reporting and the cost and latency data give it practical value for an applied readership. The main barrier to publication is the mismatch between the abstract’s “generally improves average performance” claim and the baseline-dependence of the evidence, together with the missing Default-condition data for Gemini Pro 1.5 and the trial-count documentation errors in the supplementary tables; these need to be fixed before I would endorse acceptance. The journal should also consider whether the report format and the extensive self-citation of Report 1 for the rating measures fit its standards for novelty disclosure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a solid measurement report that gives the field a useful data point—explicit CoT prompting adds little for current reasoning models and only small, model-dependent gains for non-reasoning models—but the abstract overstates the case by leaning on a baseline that tells the model not to think. The paper's own Default condition mostly fails to show significant gains.\n\nWhat is actually new: repeated trials (25 per question) on 2025 frontier models, with bootstrap permutation tests and threshold-based reliability metrics. The inclusion of a Default condition (no prompt, no formatting constraint) is a genuine strength, because it partially controls for the fact that many non-reasoning models already produce CoT-like traces unprompted. The cost and latency findings are directly relevant to practitioners. No fitted parameters drive the conclusions; every effect is a direct measurement against a fixed benchmark. The citation pattern is fine—the self-citation to Report 1 is not load-bearing for the CoT results.\n\nThe soft spots are real but fixable. First, the main CoT-versus-baseline comparison uses a Direct prompt that says \"Answer directly without any explanation or thinking.\" That suppresses the model's default reasoning, so part of the measured CoT gain may come from handicapping the control, not from CoT itself. The paper's own Table S4 shows that against a natural Default baseline, only Gemini Flash 2.0 and GPT-4o have significant average gains; Sonnet 3.5 and GPT-4o-mini are null. The abstract's \"generally improves average performance\" is therefore too strong. Second, the Discussion's claim that \"answer only\" prompting is likely to harm non-reasoning models is not isolated, because Direct conflates answer-only with \"without thinking.\" Third, Tables S3–S5 say questions were considered perfect at ≥100 successes, but the design uses 25 trials; that is an internal inconsistency that will confuse readers. Fourth, raw data and code are not released, which limits reproducibility checks.\n\nNone of these flaws sink the core finding. The measurements are what they are, and the qualitative conclusion—CoT is a conditional, cost-sensitive technique, not a universal accuracy booster—holds up. The audience is applied users and researchers who want current-model evidence on prompt effects, not readers looking for a new mechanism.\n\nRecommendation: yes, send it to peer review, but as a short applied benchmarking report, not as a conceptual advance. The authors should be asked to re-center the abstract on the Default comparison, correct the table notes, and release data or at least per-question scores. With those changes, the paper would be a clean, citeable data point.","headline":"Useful measurements of CoT on 2025 models, but the headline gain is mostly an artifact of a no-thinking baseline; the Default condition tells the more honest story.","tokens_in":12939,"tokens_out":2412,"would_cite":true,"duration_ms":27851,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Chain-of-thought prompting, the 'think step by step' technique, is not a universal accuracy booster: on a PhD-level benchmark it yields small and inconsistent gains for ordinary models, often cuts perfect-accuracy scores, and gives…","keywords":["chain-of-thought prompting","LLM evaluation","GPQA Diamond","prompt engineering","reasoning models","response latency","answer variability","cost-aware prompting"],"falsifier":"Run the same protocol with an unconstrained baseline that does not forbid reasoning, such as 'Answer the question' with no formatting constraint, and compare it against 'Think step by step'; if the average gain over Direct shrinks to near zero while the latency penalty remains, the reported CoT benefit is largely an artifact of the Direct prompt suppressing default reasoning.","tokens_in":11934,"feed_emoji":"🧠","tokens_out":7534,"duration_ms":74303,"temperature":0.7,"pith_summary":"This report tries to establish that Chain-of-Thought prompting (asking a language model to 'think step by step' before answering) is not a universal way to make AI more accurate. Testing five ordinary models and three models with built-in reasoning on the 198-question GPQA Diamond benchmark, with 25 trials per question and prompt, the paper finds that a simple CoT prompt modestly raises average accuracy for ordinary models but also increases answer variability, sometimes turning previously correct answers into wrong ones. For dedicated reasoning models, the same prompt produces at best marginal accuracy gains while substantially increasing response time and token use. If correct, the practical conclusion is that CoT should be chosen conditionally, weighing accuracy gains against latency, cost, and reliability needs.","feed_headline":"CoT prompting is not a universal AI accuracy booster","feed_subtitle":"New tests on a PhD-level benchmark show step-by-step asks add time and tokens with only modest, conditional accuracy gains.","key_machinery":"The carrying instrument is a three-way prompt contrast repeated on every question: Direct ('Answer directly without any explanation or thinking'), Step-by-step ('Think step by step'), and Default (no suffix, no formatting constraint). Each of the 198 GPQA Diamond questions is run 25 times under each condition, giving 4,950 runs per prompt per model, and scored four ways: average accuracy across all trials, and per-question thresholds requiring 25/25, 23/25, or 13/25 correct. This setup lets the authors separate average gains from reliability losses, since a model can improve on average while losing perfect-accuracy questions. The comparison against Default is what reveals that many models already perform CoT internally, which is why explicit prompting adds little.","core_discovery":"The paper's central claim is that explicit CoT prompting has decreasing and conditional value. For non-reasoning models, a 'Think step by step' prompt generally improved average accuracy over the 'answer directly' condition, with significant gains for Gemini 2.0 Flash (0.135) and Claude Sonnet 3.5 (0.117), but the strict 25/25 perfect-accuracy metric often declined, sharply for Gemini 2.0 Flash (-0.131) and Gemini Pro 1.5 (-0.172). When compared to an unprompted default in which the model chooses how to answer, CoT gains were much smaller and usually insignificant, because many models already reason step by step unprompted. For reasoning models (o3-mini, o4-mini, Gemini 2.5 Flash), explicit CoT changed average accuracy by no more than about plus or minus 0.03, with two models gaining slightly and one losing slightly, while response time rose 20 to 80 percent, or 10 to 20 seconds.","pith_inferences":["The Direct-versus-CoT comparison likely overstates CoT's benefit: the Direct prompt forbids the model's default reasoning, so a neutral prompt that neither asks nor forbids thinking would be a fairer baseline and would probably shrink the measured gains.","For high-volume applications, the 35 to 600 percent latency increase for non-reasoning models and 20 to 80 percent for reasoning models suggests a cost-aware strategy: reserve CoT for hard questions or use short reasoning budgets rather than applying it uniformly.","If reliability is the goal, repeated sampling with majority voting may address CoT's increased variability more directly than a single CoT answer, although this report does not test that combination.","Since generic CoT prompt variants produced nearly identical results, further gains are more likely to come from task-specific reasoning scaffolds or model-specific tuning than from rewording the step-by-step instruction."],"forward_implications":["For non-reasoning models, 'think step by step' can lift average accuracy but may lower the chance of a perfect 25/25 score, so users with zero-tolerance tasks should not assume CoT helps.","For reasoning models, explicit CoT adds little or no accuracy while increasing response time by 20 to 80 percent, so the added latency is often not justified.","Many current non-reasoning models reason step by step even when not asked, meaning an explicit CoT prompt frequently duplicates behavior the model would exhibit anyway.","Prompting a model to 'answer directly without explanation' can suppress useful internal reasoning and may hurt performance on hard questions.","Choosing whether to use CoT should depend on the model, the difficulty of the task, the accuracy threshold required, tolerance for variability, and acceptable latency and cost."],"supporting_citations":[{"why":"Defines Chain-of-Thought prompting, the intervention whose value the paper re-evaluates.","marker":"Wei et al. 2022"},{"why":"Supplies the GPQA Diamond dataset and the expert and non-expert human baselines used to frame task difficulty.","marker":"Rein et al. 2024"},{"why":"Provides the statistical rationale for using 25 trials per question and for reporting confidence intervals in LLM evaluations.","marker":"Miller 2024"},{"why":"Supplies the rating metrics (100 percent, 90 percent, 51 percent, average) reused in this report.","marker":"Meincke et al. 2025"}],"fun_headline_variants":["CoT prompting: modest gains, big token bills","Step-by-step prompts rarely lift reasoning accuracy","For reasoning models, CoT adds time, not accuracy","Chain-of-thought's edge shrinks with reasoning models","CoT's token cost outpaces its accuracy payoff"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that telling a model to 'answer directly without any explanation or thinking' is a neutral baseline; if that instruction suppresses reasoning the model would otherwise do, the CoT gains may measure the cost of the baseline rather than a benefit of CoT.","fun_headline_variants_meta":{"raw":{"variants":["CoT prompting: modest gains, big token bills","Step-by-step prompts rarely lift reasoning accuracy","For reasoning models, CoT adds time, not accuracy","Chain-of-thought's edge shrinks with reasoning models","CoT's token cost outpaces its accuracy payoff"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000851,"raw_usage":{"total_tokens":3736,"prompt_tokens":1016,"completion_tokens":2720,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":2644}},"tokens_in":632,"tokens_out":2720,"duration_ms":21193,"temperature":1.0,"reasoning_tokens":2644,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:41:21.842650+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same protocol with an unconstrained baseline that does not forbid reasoning, such as 'Answer the question' with no formatting constraint, and compare it against 'Think step by step'; if the average gain over Direct shrinks to near zero while the latency penalty remains, the reported CoT benefit is largely an artifact of the Direct prompt suppressing default reasoning.","supporting_citations":[],"review_version":1}