{"id":"2377cea8-c664-4020-88ee-84c85c6917ed","arxiv_id":"2505.10981","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Under majority voting, simple Chain-of-Thought prompting outperforms complex prompting strategies as test-time sampling scales.","lead":"This paper compares eight prompting strategies under majority-vote scaling across six language models and finds that plain Chain-of-Thought consistently overtakes more elaborate prompting methods as sampling grows. It offers a probability-based formula to predict which strategy will win at large sampling budgets, along with two oracle-based ways to improve scaling performance.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CoT dominance is stated as universal but the paper's own Table 4 shows SBP as P*_N on LLaMA-3-8B/GSM8K for all N≥20; the central claim should be qualified to 'simple strategies usually dominate' until per-dataset tallies support it.","rationale":"The reader's CONDITIONAL verdict is based on the unjustified normal approximation in Eq. 5-7 and Theorem 4. That concern is real and testable: the max of multinomial counts is not Gaussian, and its dependence on x1 is ignored. But the more immediately load-bearing problem for the paper's central claim is the gap between the universal wording and the paper's own reported exceptions. The paper itself flags that only 80% of results conform and that Table 4 is an exception; a single concrete counterexample is enough to refute the phrase 'without exception.' Repairing the normal approximation would not fix this overclaim. I therefore keep the CONDITIONAL verdict but would require the authors to either restrict the claim to aggregate results or provide per-dataset tallies showing CoT is the eventual argmax in the large-N limit. The proposed re-run on the known counterexample settles which revision is needed. This is not a rejection: the broad empirical trend and the released code are real contributions; the issue is the precision of the central claim, not an integrity problem.","tokens_in":29080,"tokens_out":13965,"duration_ms":150698,"concrete_test":"Using the released code, recompute the actual majority-vote accuracy of all P1 strategies on LLaMA-3-8B-Instruct/GSM8K at N=20, 50, 100, and, if feasible, N=200, and identify the true argmax P*_N at each N. If SBP remains the argmax at the largest feasible N, the Section 3 'without exception' claim is falsified and the Abstract should be revised to say 'simple prompting strategies usually dominate.' If CoT overtakes SBP before N=200, then Table 4 is a finite-N artifact and the universal asymptotic claim can be tested further by extending to larger N or by checking whether the Eq. 10 predictor at those N matches.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption for the title claim is that CoT is the eventual winner under majority-vote scaling. Section 3 states this as 'CoT dominates without exception on all models when largely scaling,' but the paper's own evidence contradicts it. The Limitations section says the finding 'does not always hold for every LLM on every dataset, e.g., Table 4,' and only about 80% of results conform. Table 4 is a concrete counterexample: for LLaMA-3-8B-Instruct on GSM8K, the true best strategy P*_N is SBP for every N from 20 through 1000, including the largest N predicted by the paper's own Eq. 10. Appendix C further states that DiP also performs best on several datasets and that 'almost 83% of results satisfy that CoT or DiP performs best.' Consequently, the theoretical analysis in Section 4, including Theorem 4 and the normal approximation, explains why a strategy with a larger correct-answer gap can overtake, but it does not establish CoT-specific universality. The empirical support is an aggregate trend, not a universal asymptotic ordering, so the strong headline in Section 3 and the Abstract overstates what is shown. The 'eventually' is also only demonstrated up to N≈100 for actual sampling; extrapolation to very large N rests on the unvalidated predictor.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the test-time scaling of eight prompting strategies under majority voting, claiming that simple Chain-of-Thought (CoT) eventually outperforms more complex strategies as the number of samples N grows, despite underperforming at small N. The claim is supported by experiments on 6 LLMs, 8 prompting strategies, and 6 benchmarks, with additional results on GPQA and AIME. The authors propose a probabilistic explanation (Theorems 1–4) based on the proportions of easy/moderate/hard questions and the uniformity of incorrect-answer distributions, introduce an O(1) formula (Eq. 10) to predict majority-vote accuracy and select the best strategy P*_N, and present two oracle-based ways to improve scaling performance (adaptive difficulty-based scaling and dynamic strategy selection). The paper's Limitations section explicitly states that the CoT-dominance finding holds for only about 80% of results and does not hold for every LLM/dataset.","tokens_in":29346,"tokens_out":6175,"duration_ms":63209,"significance":"If the central claim were fully established, the paper would be a valuable contribution: it addresses a practical question (which prompting strategy to use when scaling test-time compute), provides a broad empirical matrix under equivalent cost budgets, and offers a cheap predictor for majority-vote scaling. The paper ships code and includes closed and open models. Theorems 1–3, which characterize the limiting behavior of majority voting for easy/moderate/hard questions, are elementary and valid within the multinomial model. The core weakness is that the headline universality claim is contradicted by the paper's own tables, and the proposed predictor and Theorem 4 rely on an unproven normal approximation for the maximum of multinomial counts. These issues are fixable with a more cautious framing and additional finite-N validation, so the contribution is potentially significant but currently overclaimed.","major_comments":[{"comment":"The statement in Section 3 that 'CoT dominates without exception on all models when largely scaling' is directly contradicted by Table 4, which shows that for LLaMA-3-8B-Instruct on GSM8K the true best strategy P*_N is SBP for every N from 20 to 1000. Appendix C further states that 'almost 83% of results satisfy that CoT or DiP performs best,' and the Limitations section concedes that the finding 'does not always hold for every LLM on every dataset.' These statements are inconsistent with the abstract's claim of 'consistently show' and with the title's strong framing. The central claim should be reformulated as an aggregate empirical trend over the tested matrix, with per-dataset tallies and explicit caveats, rather than a universal asymptotic ordering.","section":"Section 3, Table 4, and Limitations"},{"comment":"The O(1) predictor and the proof of Theorem 4 approximate M = max(x2,...,xm) by a normal distribution N(N p_max, N p_max(1-p_max)). This approximation is not justified: the maximum of multinomial counts for fixed m is not asymptotically normal in the relevant tail regime, and the derivation also ignores the dependence between M and x1 when forming Z = x1 - M. No finite-N bound or empirical justification for the approximation is provided beyond a single dataset (LLaMA-3-8B-Instruct on GSM8K). In addition, the proof of Theorem 4 contains an inequality-direction error: it derives Pr(x_i',1 > M'_i) > Pr(x_i,1 > M_i) but then concludes Pr(a1|Pi;N) > Pr(a1|Pi';N), which is the opposite of the theorem's statement; the text should conclude with '<'. Because the predictor is used to select P*_N and the paper claims correctness for arbitrary N, this gap is load-bearing.","section":"Section 5, Eq. (6)–(10), and Appendix B (Theorem 4)"},{"comment":"The experimental validation of the O(1) predictor is limited to one model-dataset pair (LLaMA-3-8B-Instruct on GSM8K) and uses only 40 samples to estimate the per-answer probabilities p_i,j. The paper reports no error bars, confidence intervals, or variance across the five runs, despite averaging over five runs elsewhere. The claim that 'when N≥10, the error is already less than 1%' is based on this single pair, and the stability of the selected P*_N with respect to the noise in the 40-sample estimate is not quantified. To support the practical claim of predicting scaling performance and selecting the best strategy, the validation should be extended to multiple model-dataset pairs, and the sensitivity of P*_N to the probability estimates should be reported (e.g., via bootstrap).","section":"Section 5, 'Experiment' paragraph and Table 4"},{"comment":"The abstract and introduction state that the authors 'introduce two ways to significantly improve the scaling performance,' but the practical versions of these methods do not deliver the advertised gains: Section 6.1 reports that 'Adaptive' performance is 'almost equal to the usual scaling performance,' and Section 6.2 reports that 'Dynamic' performance is 'almost equal to CoT.' The substantial improvements come only from oracle upper bounds, which the paper acknowledges. This framing overstates the practical contribution and should be revised to clearly distinguish an upper-bound analysis from deployable methods, both in the abstract and in the contributions list.","section":"Section 6 and Abstract"}],"minor_comments":[{"comment":"In Table 1, the L2M row for Gemini-1.5-Flash lists Hard as 89.8%, which makes the row sum to more than 100%; it should likely be 8.9% to match the Easy and Moderate entries.","section":"Table 1"},{"comment":"The caption reads 'GQPA' but the benchmark is GPQA; please correct the typo.","section":"Figure 37 caption"},{"comment":"Beyond the inequality-direction issue noted above, the proof says 'the distribution of Mi can be approximated by Mi ~ N(N p_i,n, N p_i,1(1-p_i,1))' but uses p_i,q in the variance formula; the notation should be consistent.","section":"Appendix B, Theorem 4 proof"},{"comment":"The paper gives slightly different percentages for the same claim ('about 80%' in Section 3, '80% of the results' in Limitations, and 'almost 83%' in Appendix C); these numbers should be reconciled with a precise per-model and per-dataset count.","section":"Section 3 and Limitations"},{"comment":"The statement that the prediction error 'decreases until 0% as scaling' is not generally guaranteed when p_i,j are estimated from a finite sample; the variance of the estimator should be discussed or the claim qualified.","section":"Section 5, Eq. (10)"}],"recommendation":"major_revision","confidential_remarks":"The empirical aggregate trend is plausible and the paper is within scope, but the title and abstract overclaim universality that the paper's own data contradict. The main correctness risk is the unproven normal approximation for the maximum of multinomial counts, which underlies both Theorem 4 and the O(1) predictor; if a rigorous finite-N justification cannot be provided, the predictor should be reframed as a heuristic with stronger empirical validation. The oracle-based improvements should also be labeled as upper bounds throughout, including the abstract."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I'll keep this short. The thing to know: the empirical sweep is genuinely useful, but the headline is stronger than the evidence. Section 3 says CoT dominates without exception on all models when largely scaling, yet their own Table 4 shows SBP as best on LLaMA-3-8B-Instruct on GSM8K for every N from 20 to 1000, and the Limitations section concedes only about 80% of results fit. The abstract and intro repeat the strong version. That mismatch should be fixed before this is citable as stated.\n\nWhat's new and good: the 6 LLMs x 8 prompting strategies x 6 benchmarks comparison under majority voting, with both sampling-time and cost budgets, is a useful reference. The cost-controlled comparison is fairer than most prior work. The two explanatory factors — CoT having a better easy/hard question mix and a flatter wrong-answer distribution — are intuitive and partly quantified via KL divergence and the Theorem 4 condition counts. The O(1) predictor is a nice practical idea: estimate answer probabilities from a small sample, then use a normal approximation to extrapolate to larger N. The oracle-based improvements in Section 6 are clearly labeled upper bounds; no one should mistake them for deployed methods.\n\nSoft spots, in order of importance. First, the universality claim. It is not just a wording issue: the predictor itself selects SBP on LLaMA/GSM8K, so the paper's own tool disagrees with the title claim on that cell. Second, the theory. Theorem 3's \"general declining trend\" is not a theorem, and the proof relies on limit statements without monotonicity. More seriously, Eq. (6) approximates the maximum of the wrong-answer counts by a normal with mean N p_max and variance N p_max(1-p_max). The max of multinomial counts is not normally distributed that way; the approximation is unproven and could misrank strategies in finite samples. The predictor is validated on one model/dataset pair, so we don't know how often the approximation holds. Third, missing error bars. Five runs are averaged; we need at least ranges or standard deviations, especially because some adjacent strategies are within a point. Fourth, the extended GPQA results show non-monotone aggregate accuracy, which is fine theoretically but further undercuts the simple \"CoT eventually dominates\" narrative.\n\nWho this is for: anyone working on test-time scaling or budget-aware inference. It deserves a serious referee. I'd send it out, with a request to qualify the central claim, tighten or weaken the theorems, and add uncertainty quantification. As it stands I wouldn't cite the headline; I'd cite the comparison matrix and the predictor once those are fixed.","headline":"Useful empirical sweep and a plausible predictor, but the CoT-dominance headline is overstated and the probability theory is not tight enough to support it as stated.","tokens_in":29888,"tokens_out":1954,"would_cite":true,"duration_ms":20634,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that under majority-voting scaling, simple Chain-of-Thought eventually dominates all eight tested prompting strategies, and provides a closed-form formula to predict the winner.","keywords":["chain-of-thought prompting","majority voting","test-time scaling","self-consistency","prompting strategies","probability theory","scaling law","LLM reasoning"],"falsifier":"Record the actual majority-voting accuracy for each prompting strategy at $N$ up to 1000 on a benchmark such as GSM8K, and also compute the exact multinomial majority probability from the empirically estimated answer distributions; if the exact values and the observed values diverge materially from Eq. (10) for large $N$, the normal-max approximation and any strategy ranking based on it are falsified.","tokens_in":28854,"feed_emoji":"🧠","tokens_out":6489,"duration_ms":61689,"temperature":0.7,"pith_summary":"The paper sets out to show that under majority voting, the simplest reasoning prompt—plain chain-of-thought—eventually beats every more elaborate prompting strategy once the number of samples is large, even when those strategies start with higher single-sample accuracy. It argues this is not an accident of the benchmarks: the answer counts follow a multinomial distribution, and the asymptotic winner is determined by whether a question is easy or hard for a strategy and by how flat the wrong-answer distribution is. The paper also claims an O(1) formula that predicts each strategy's scaling curve and selects the best strategy for any sampling budget using only a small sample of answers. If true, this means a large share of test-time scaling gains can be had with simple prompting, and expensive large-N evaluations are unnecessary for strategy selection.","feed_headline":"As sampling grows, simple Chain-of-Thought wins","feed_subtitle":"A closed-form formula predicts the winner, so teams can skip expensive large-N runs.","key_machinery":"The load-bearing object is the multinomial model of sampled answers, $X_i \\sim \\mathrm{Mult}(N, p_{i,1},\\dots,p_{i,m})$, with majority voting selecting the most frequent answer. On top of it, Eq. (10) is a closed-form normal approximation: $\\Pr(\\text{correct}) \\approx 1 - \\Phi\\left(-(p_1 - p_{\\max})\\big/\\sqrt{(p_1(1-p_1) + p_{\\max}(1-p_{\\max}))/N}\\right)$, where $p_1$ is the correct-answer probability and $p_{\\max}$ is the largest wrong-answer probability. This formula is the paper's test-time scaling law: it turns any strategy's answer distribution into a predicted scaling curve and a predicted best strategy $P^*_N$. The theorems about easy, moderate, and hard questions supply the qualitative mechanism, while Eq. (10) supplies the quantitative tool.","core_discovery":"The central claim is that, for majority voting under a fixed sampling-time or compute budget, the ordering of prompting strategies is not fixed by pass@1 accuracy. For each question, the sampled answers of a strategy are modeled as multinomial draws; a question is 'easy' when the correct answer is the unique most probable answer, 'moderate' when it ties for the most probable, and 'hard' otherwise. Theorems 1–3 state that majority-voting accuracy tends to 1 on easy questions, to $1/|S|$ on moderate ties, and to 0 on hard questions, so the dataset composition of easy versus hard questions governs the scaling limit. Theorem 4 gives conditions under which two strategies cross: one with lower single-sample accuracy but a flatter wrong-answer distribution overtakes another with higher initial accuracy as $N$ grows. Empirically, CoT has the fewest hard questions and the most uniform wrong-answer distribution across six LLMs and six benchmarks, so it dominates at large $N$.","pith_inferences":["The normal-max approximation in Eq. (10) makes the predicted ordering most fragile on questions where $p_1$ and $p_{\\max}$ are close, because the crossing point $N_0$ is largest there; a dataset deliberately composed of near-tie questions should delay or reverse CoT's dominance, and testing that composition would isolate whether the approximation or the empirical difficulty split drives the result","The same multinomial argument applies to other aggregation rules beyond majority voting, such as weighted voting or verifier reranking: any selector extracting an answer from multinomial samples is governed by the gap $p_1 - p_{\\max}$ and the easy-versus-hard split, so the 'simple prompt wins at scale' finding may extend to other test-time scaling mechanisms.","The dynamic-strategy oracle results suggest a practical calibration route: with a small labeled sample, one could learn a per-question difficulty-and-strategy classifier that approximates the oracle $P^*_N$, turning the theoretical upper bound into an achievable inference-time policy."],"forward_implications":["At sufficiently large $N$, choosing a prompting strategy reduces to comparing each prompt's answer distribution; the strategy with the best combination of high correct-answer probability and flat wrong-answer distribution wins, so no full large-N run is needed.","The O(1) formula predicts $P^*_N$ using only about 40 samples to estimate each strategy's answer probabilities, and on the tested setup the error falls below 1% once $N \\geq 10$, so it can serve as a cheap test-time scaling law.","Because majority voting converges to 0 on hard questions, spending more samples on hard questions is wasteful; accuracy can be improved by scaling only easy or moderate questions.","Because the best strategy differs per question, dynamically choosing the optimal prompt per question—even at $N=1$—can exceed the best fixed strategy's infinite-sample accuracy.","Combining per-question difficulty-aware scaling with per-question strategy selection gives large upper-bound accuracy gains on the reported benchmarks, including GSM8K, MATH-500, GPQA, and AIME."],"supporting_citations":[{"why":"Establishes chain-of-thought prompting, the simple strategy whose scaling dominance is the paper's subject.","marker":"Wei et al. 2022"},{"why":"Defines self-consistency and majority voting, the scaling procedure under study.","marker":"Wang et al. 2023b"},{"why":"Provides the zero-shot 'Let's think step by step' prompt used for CoT.","marker":"Kojima et al. 2022"},{"why":"Positions test-time compute scaling and gives the fixed-prompt analysis this paper contrasts with majority-voting strategy comparison.","marker":"Snell et al. 2025"},{"why":"Supplies the precedent of a scaling law that motivates predicting performance from small low-cost experiments.","marker":"Kaplan et al. 2020"},{"why":"GSM8K is the primary benchmark for the scaling, prediction, and improvement experiments.","marker":"Cobbe et al. 2021"},{"why":"MATH-500 is used as a math benchmark for evaluation and for the adaptive and dynamic improvement results.","marker":"Hendrycks et al. 2021b"},{"why":"Step-Back Prompting is one of the compared complex strategies whose initial advantage is overtaken at scale.","marker":"Zheng et al. 2024"},{"why":"Tree-of-Thoughts is one of the iterative reasoning strategies included in the comparison set.","marker":"Yao et al. 2023"}],"fun_headline_variants":["Simple CoT overtakes complex prompts as sampling grows","Probability theory explains why simple CoT wins at scale","Formula predicts prompt winner for large sample counts","CoT dominates complex prompting in majority voting","Majority-voting limit: simple Chain-of-Thought wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole prediction and overtaking argument assumes that each strategy's answer probabilities are fixed across samples and that the largest count among wrong answers is approximately normal with mean and variance set by the second-best answer, an approximation the paper uses without proof.","fun_headline_variants_meta":{"raw":{"variants":["Simple CoT overtakes complex prompts as sampling grows","Probability theory explains why simple CoT wins at scale","Formula predicts prompt winner for large sample counts","CoT dominates complex prompting in majority voting","Majority-voting limit: simple Chain-of-Thought wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000181,"raw_usage":{"total_tokens":1308,"prompt_tokens":946,"completion_tokens":362,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":288}},"tokens_in":562,"tokens_out":362,"duration_ms":3582,"temperature":1.0,"reasoning_tokens":288,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:59:27.753725+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record the actual majority-voting accuracy for each prompting strategy at $N$ up to 1000 on a benchmark such as GSM8K, and also compute the exact multinomial majority probability from the empirically estimated answer distributions; if the exact values and the observed values diverge materially from Eq. (10) for large $N$, the normal-max approximation and any strategy ranking based on it are falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes chain-of-thought prompting, the simple strategy whose scaling dominance is the paper's subject."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the zero-shot 'Let's think step by step' prompt used for CoT."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Positions test-time compute scaling and gives the fixed-prompt analysis this paper contrasts with majority-voting strategy comparison."}],"review_version":1}