{"id":"2c39f0e4-1315-474e-a710-9962481b6a3c","arxiv_id":"2608.09351","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"For mid-tier LLMs, generating paraphrased versions of an input and majority voting over answers converts inference budget into accuracy more efficiently than self-consistency sampling.","lead":"Researchers tested whether rewriting a question in several ways before asking an AI model buys more accuracy per dollar than asking the exact same question many times. Across six benchmarks, semantic rephrasing beat self-consistency on five of six tasks and delivered roughly 1.8x more accuracy per dollar.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's 'matched compute' claim is tested at matched k, not equal total LLM calls; the missing equal-budget comparison could shrink or reverse the 1.8X efficiency advantage.","rationale":"I read the paper as a careful empirical study: the held-out k selection, bootstrap confidence intervals, and explicit limitations are real strengths, and the authors deserve credit for spelling out the extra rephrasing cost and for testing multiple model sizes. The central claim, however, is an efficiency claim, and efficiency is only meaningful at a fixed budget. The paper's own Section 3.5 converts 'matched compute' into 'matched k', and the cost analysis does not supply the equal-total-call frontier. The single-family external-validity issue identified by the reader is genuine, but it is secondary: even if the result holds for Claude, the efficiency comparison is not yet demonstrated at equal total calls. The proposed test settles this without new data beyond rerunning the existing pipeline. I therefore keep the reader's CONDITIONAL verdict; this pass sharpens the condition but does not change it.","tokens_in":12518,"tokens_out":8139,"duration_ms":87761,"concrete_test":"Run a direct equal-total-call comparison on Math500 and MMLU. For each total budget B in {3,5,7,9}: semantic TTA = one rephrasing call plus B-1 answer calls, majority vote over B-1 answers; self-consistency = B answer calls on the original input, majority vote. Use the same T=0.75, same answer template, same held-out k-selection rule, and price both methods with on-demand token costs. If self-consistency at B calls matches or exceeds semantic TTA at the same B, the 'matched compute' claim and the 1.8X per-dollar figure fail; if semantic TTA still dominates across all B, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing gap is that the paper's headline comparison is not made at the quantity named in the title. The abstract and Section 1 promise a 'matched-compute' comparison, but Section 3.5 defines the comparison as 'at matched k': semantic TTA spends one rephrasing call plus k answer calls, while self-consistency spends k answer calls. The cost analysis in Section 5.2 and Figure 5 normalizes by dollars or calls using each method's held-out optimal k, but it does not report accuracy at equal total inference-call budgets. Figure 4's claim that semantic TTA achieves the highest accuracy 'at every cost level' is only evaluated at k in {2,4,6}; the total budgets of the two methods differ by one call at every point. Since self-consistency continues to improve with k (Figure 6), the missing comparison - semantic TTA with k answers plus one rephrasing call versus self-consistency with k+1 answers - is exactly where the advertised '1.8X more accuracy per dollar' could shrink or reverse. The central claim is therefore not internally contradicted, but it is not directly tested at the compute budget the title names; the per-dollar figure aggregates over different k rather than comparing frontiers at equal total calls.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies Test-Time Augmentation (TTA) for LLMs, where predictions are aggregated across transformed versions of the input, and compares input-side diversity (semantic rephrasing, lexical perturbations, visual transformations) against output-side diversity (self-consistency) on six benchmarks (MMLU, MMMLU, MMMU, HLE, Math500, IMDB). The main claim, stated in the abstract and Section 6, is that for current mid-tier LLMs, varying the input converts inference compute into accuracy more efficiently than varying the reasoning path alone, with semantic rephrasing delivering roughly 1.8× more accuracy per dollar and outperforming self-consistency on five of six tasks. The paper also reports ablations on the number of augmentations, multi-modal strategies, and base model scaling, and it releases an implementation on GitHub.","tokens_in":12754,"tokens_out":6980,"duration_ms":65891,"significance":"The question is practically important: if correct, semantic TTA is a training-free, input-side complement to self-consistency that improves accuracy per inference dollar on discrete-answer benchmarks. The study has several methodological strengths: k is selected on a held-out sample (Section 4.2), the main comparisons include paired t-tests and a paired bootstrap (Section 5.1, Appendix C), and the implementation is public. The scaling analysis across three Claude model sizes (Section 5.5) gives a sensible account of where the method is most useful. The paper's evidence is, however, restricted to one proprietary model family, and the headline 'matched compute' comparison is not actually made at equal total inference budgets.","major_comments":[{"comment":"The paper's title and abstract promise a 'matched-compute' comparison, but the actual comparison is 'matched k': semantic TTA spends one rephrasing call plus k answer calls, while self-consistency spends k answer calls (Section 3.5). The cost-effectiveness ratios in Figure 5 are computed at each method's own held-out optimal k, so the reported 1.8× 'accuracy per dollar' is not a comparison of the two methods at equal total LLM calls or equal dollar budgets. Because Figure 6 shows self-consistency still improving up to k=10 on Math500, the missing comparison—semantic TTA with k answers plus one rephrasing call versus self-consistency with k+1 answers, and the corresponding equal-dollar comparison—is exactly where the advertised advantage could shrink or reverse. This is a load-bearing gap for the central efficiency claim; it can be closed by reporting accuracy at equal total calls and at equal total cost for each dataset.","section":"§3.5, §5.2, Figs. 4–5"},{"comment":"The abstract and conclusion generalize the finding to 'current mid-tier LLMs,' but all experiments use a single proprietary family (Claude 4.5 Haiku, Sonnet, Opus). The limitation in Section 6 acknowledges that behavior on open-weight or other families 'remains to be verified,' yet the conclusion still states the general claim. Since the paper's central claim is about a class of models rather than one family, this is an unsupported extrapolation; either add at least one additional model family on a subset of datasets or restrict the claims to the Claude family and adjust the title and abstract accordingly.","section":"§5.5, §6"}],"minor_comments":[{"comment":"The sentence 'Semantic TTA can be seen as a strict extension of this procedure' is not strictly accurate, since semantic TTA adds an extra rephrasing call and changes the input distribution; a more precise phrasing would be 'an extension that adds input-side diversity.'","section":"§3.5"},{"comment":"The paired bootstrap pools 2,400 per-question gains as independent observations, but per-question gains within a dataset are correlated; reporting a dataset-level or cluster-robust confidence interval would strengthen the significance claim.","section":"§5.1, Appendix C"},{"comment":"The HLE row shows '62' for both lexical TTA and self-consistency, which appears to be a table alignment issue; the formatting should be corrected to make the per-method optimal k values clear.","section":"Table 2"},{"comment":"The sentence 'All methods incur computational cost proportional to k' is imprecise for semantic TTA, which also incurs a fixed rephrasing cost; the next sentence says this, so a small clarification would avoid the seeming contradiction.","section":"§4.3"},{"comment":"The rephrasing prompt explicitly preserves answer choices for multiple-choice items, but for open-ended questions it could more clearly instruct the model to preserve all numeric constraints and units; this is a minor clarity issue.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"This is a workshop paper with a clear, practical message. The main gap is fixable: the authors should add the equal-total-call and equal-dollar comparisons that the title promises. If the results hold under that comparison, the paper would be a solid contribution; as is, the headline claim is not directly tested. The paper's framing of 'current mid-tier LLMs' from a single proprietary family is a scope concern for the journal's readership, but it is explicitly acknowledged in the limitations and can be addressed by rewording or additional experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear Colleague,\n\nThis paper is a genuine, useful empirical comparison: input-side test-time augmentation (semantic rephrasing, lexical perturbations, visual transforms) against output-side self-consistency, across six benchmarks, with careful attention to cost. It is not a new technique, but it is the first systematic matched-budget comparison across diverse tasks, and the experimental design is better than most: held-out k selection, paired bootstrap significance testing, a consistent CoT template, ablations across model sizes, and an unusually honest limitations section. The main empirical finding, that semantic rephrasing beats self-consistency on five of six tasks and looks more cost-effective for mid-tier models, is plausible and worth taking seriously.\n\nThe soft spots, in proportion. First and most important: the 'matched compute' claim in the title and abstract is not what is tested. The body compares methods at matched k, not matched total LLM calls. Semantic TTA spends one extra rephrasing call, so at k=4 it uses five calls versus self-consistency's four. The paper never reports the equal-total-budget comparison, e.g., semantic TTA with k=4 versus self-consistency with k=5. Given that Figure 6 shows self-consistency continuing to improve with k and nearly catching semantic TTA at k=10 on Math500, that missing comparison could shrink or reverse the headline 1.8x efficiency advantage. This is a real gap, not a nit.\n\nSecond, the evidence is one proprietary model family (Claude Haiku, Sonnet, Opus). The limitations acknowledge this, but the conclusion is stated as a property of 'current mid-tier LLMs,' which overreaches. Third, 'Pareto-dominates' is loose: the MMLU result drops at k=6, so the dominance is not strict. Fourth, the evaluation subsets (400 examples per dataset) are not released, which limits independent reproducibility, though the code and prompts are available.\n\nOverall: this deserves a serious referee. The core experiment is well designed and the result is practically relevant. The main revision is to either add the equal-total-call comparison or soften the matched-compute framing. I would take it as a major-revision paper, not a rejection.","headline":"A well-designed empirical study of input-side vs output-side test-time scaling, but the headline 'matched compute' claim is not tested at equal total calls; needs a revision.","tokens_in":13307,"tokens_out":3737,"would_cite":true,"duration_ms":35207,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"At a fixed inference budget, rephrasing the input question and voting over the answers yields more accuracy per dollar than sampling multiple reasoning paths from the same prompt.","keywords":["test-time augmentation","self-consistency","chain-of-thought prompting","inference-time scaling","semantic rephrasing","majority voting","cost-accuracy trade-off","LLM robustness"],"falsifier":"Repeat the paper's matched-compute protocol, chain-of-thought at temperature 0.75, k in {2,4,6}, majority voting with k selected on a held-out sample, on a mid-tier open-weight model from a different family across the same six benchmarks; the central claim is falsified if semantic TTA does not beat self-consistency on five of six tasks, or if its accuracy-per-dollar advantage falls well below 1.8x.","tokens_in":12267,"feed_emoji":"🔄","tokens_out":15320,"duration_ms":138597,"temperature":0.7,"pith_summary":"Test-time scaling usually spends a fixed inference budget on the output side: prompt the model once and sample many reasoning paths, as in self-consistency. This paper asks the symmetric question of whether the same budget is better spent on the input side, generating several rephrasings of each question and voting over their answers. The central claim is that for current mid-tier LLMs it is: semantic rephrasing yields roughly 1.8x more accuracy per dollar than self-consistency and outperforms it on five of six benchmarks, with smaller but real gains from cheap lexical typos and mild image transforms. If correct, input-side augmentation is a training-free, model-agnostic way to push a fixed base model further along the cost-accuracy frontier, especially when a stronger model is unavailable or too expensive.","feed_headline":"Rephrasing the question beats resampling the answer at equal compute","feed_subtitle":"Rephrasing each question and voting over answers gives 1.8x more accuracy per dollar than resampling the same prompt.","key_machinery":"The engine of the argument is the Test-Time Augmentation (TTA) pipeline: generate k transformed versions of the input, obtain one prediction per version, and aggregate by majority voting, breaking ties at random. Against it is the output-side baseline self-consistency, which samples k chain-of-thought paths from one unchanged prompt at temperature 0.75. Semantic TTA is the version that carries the claim; it uses a single LLM call to produce k meaning-preserving paraphrases, with the rephrasing prompt instructed to keep answer choices, formatting, and image references intact, and answers each with the same chain-of-thought template at temperature 0.75. Because every semantic-TTA answer already has the same output-side diversity as self-consistency, any measured advantage is attributed to the input variation itself.","core_discovery":"On a matched inference budget, varying the input converts compute into accuracy more efficiently than varying the reasoning path alone. Every method in the comparison answers the same questions with chain-of-thought prompting at the same temperature, and the budget is the number of answer calls; self-consistency spends all calls on one unchanged prompt, while semantic TTA spends one extra call to generate paraphrases and then answers each. Across six benchmark suites, semantic TTA's majority vote beat self-consistency's on five of six tasks, gained 1.8 percentage points over single-call CoT on average, and was statistically significant in paired tests; the authors describe it as Pareto-dominating self-consistency because it wins on both accuracy and accuracy per dollar, roughly 1.8x. Lexical TTA (character-level typos) produced smaller but significant gains, visual TTA helped on the multimodal benchmark, and gains shrank as the base model grew stronger, so the paper scopes its conclusion to current mid-tier models and positions TTA as a compute-efficiency tool rather than a substitute for a model upgrade.","pith_inferences":["If the mechanism is variance reduction over surface form, then the benefit should scale with a model's paraphrase flip rate; a benchmark-by-benchmark correlation between TTA gain and paraphrase sensitivity would test this mechanism more directly than the paper's aggregate significance test.","The paper couples the rephraser and the answerer as the same model, so the observed advantage might partly reflect correlation between paraphrase errors and answer errors; using a different or stronger model for rephrasing could widen or narrow the 1.8x gap.","The cost comparison uses token prices, so under latency or parallelism constraints, where the rephrasing call is either hidden or serialized, the relative advantage of TTA over self-consistency would differ from the reported ratio.","MMMLU is scored as one accuracy number across 14 languages, but paraphrase quality varies by language; a per-language breakdown could reveal that input-side diversity helps mainly high-resource languages, which is not tested in the paper."],"forward_implications":["At a fixed number of answer calls, semantic TTA should beat self-consistency on discrete-answer benchmarks where predictions are sensitive to phrasing, since it adds input diversity on top of the same output diversity.","The accuracy-per-dollar advantage is roughly 1.8x for mid-tier models, but it shrinks as the base model improves (2.75pp on Haiku, 0.25pp on Opus), so the method targets the regime where a stronger model is unavailable or too expensive.","Semantic TTA peaks near k=4, while self-consistency keeps improving up to k=10, making TTA the better use of a small-to-moderate inference budget and making the ordering budget-dependent.","Lexical TTA is a nearly free fallback that still beats single-call CoT, while visual TTA gives modest gains on multimodal questions; combining text and image augmentation hurts accuracy, so text-only augmentation is the recommended multimodal strategy.","Because the gain is measured against a self-consistency baseline that already includes repeated sampling, the result implies input-side and output-side diversity can be combined, with tuning their balance identified as future work."],"supporting_citations":[{"why":"Defines the self-consistency baseline, repeated chain-of-thought sampling from one prompt with majority voting, against which input-side TTA is compared at matched compute.","marker":"Wang et al., 2023"},{"why":"Provides chain-of-thought prompting, the single-call baseline and the answer-generation template used by every method in the comparison.","marker":"Wei et al., 2022"},{"why":"Introduces test-time augmentation and majority aggregation for visual inputs, the framework the paper adapts to LLM text and image inputs.","marker":"Shanmugam et al., 2020"},{"why":"Documents LLM sensitivity to prompt phrasing, which motivates the hypothesis that aggregating over rephrasings reduces prediction variance.","marker":"Seleznyov et al., 2025"},{"why":"Provides the robustness evidence and perturbation-rate guidance (5% per word) that lexical TTA relies on.","marker":"Agrawal et al., 2025"},{"why":"Shows paraphrasing math problems diversifies reasoning, the task-specific precursor that semantic TTA generalizes across six datasets.","marker":"Zhou et al., 2024"},{"why":"Identifies the Claude 4.5 model family used in all experiments, fixing the scope of the claim to current mid-tier LLMs.","marker":"Anthropic, 2025"}],"fun_headline_variants":["Rephrasing inputs beats resampling outputs for LLM accuracy per dollar","Input-side diversity wins: 1.8x more accuracy per compute dollar","Vary the question, not just the answer path: cheaper LLM gains","Test-time augmentation beats self-consistency at matched compute"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion rests on the assumption that the Claude model family speaks for other current mid-tier LLMs, so the advantage measured on Haiku, Sonnet, and Opus is a property of mid-tier models generally and not an artifact of this one family or of using the same model to rephrase and answer.","fun_headline_variants_meta":{"raw":{"variants":["Rephrasing inputs beats resampling outputs for LLM accuracy per dollar","Input-side diversity wins: 1.8x more accuracy per compute dollar","Vary the question, not just the answer path: cheaper LLM gains","Test-time augmentation beats self-consistency at matched compute"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000273,"raw_usage":{"total_tokens":1690,"prompt_tokens":1055,"completion_tokens":635,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":671,"completion_tokens_details":{"reasoning_tokens":557}},"tokens_in":671,"tokens_out":635,"duration_ms":9140,"temperature":1.0,"reasoning_tokens":557,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:55:25.234489+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the paper's matched-compute protocol, chain-of-thought at temperature 0.75, k in {2,4,6}, majority voting with k selected on a held-out sample, on a mid-tier open-weight model from a different family across the same six benchmarks; the central claim is falsified if semantic TTA does not beat self-consistency on five of six tasks, or if its accuracy-per-dollar advantage falls well below 1.8x.","supporting_citations":[],"review_version":1}