Pith. sign in

REVIEW 2 major objections 5 minor 36 references

Test-Time Augmentation for LLMs: When Input Diversity Beats Output Diversity at Matched Compute

T0 review · 2 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read At a fixed inference budget, rephrasing the input question and voting over the answers yields more accuracy per dollar than sampling multiple reasoning paths from the same prompt.

desk verdict A well-designed empirical study of input-side vs output-side test-time scaling, but the headline 'matched compute' claim is not tested at equal total calls; needs a revision. read the letter →

arxiv 2608.09351 v1 pith:46LFMSDK submitted 2026-08-10 cs.LG cs.AIstat.ML

Nikita Kozodoi , Zainab Afolabi , Jack Butler This is my paper · ORCID
classification cs.LGcs.AIstat.ML
keywords test-timeaugmentationself-consistencychain-of-thoughtpromptinginference-timescalingsemanticrephrasingmajorityvotingcost-accuracytrade-offLLMrobustness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Test-time scaling usually spends a fixed inference budget on the output side: prompt the model once and sample many reasoning paths, as in self-consistency. This paper asks the symmetric question of whether the same budget is better spent on the input side, generating several rephrasings of each question and voting over their answers. The central claim is that for current mid-tier LLMs it is: semantic rephrasing yields roughly 1.8x more accuracy per dollar than self-consistency and outperforms it on five of six benchmarks, with smaller but real gains from cheap lexical typos and mild image transforms. If correct, input-side augmentation is a training-free, model-agnostic way to push a fixed base model further along the cost-accuracy frontier, especially when a stronger model is unavailable or too expensive.

What carries the argument

The engine of the argument is the Test-Time Augmentation (TTA) pipeline: generate k transformed versions of the input, obtain one prediction per version, and aggregate by majority voting, breaking ties at random. Against it is the output-side baseline self-consistency, which samples k chain-of-thought paths from one unchanged prompt at temperature 0.75. Semantic TTA is the version that carries the claim; it uses a single LLM call to produce k meaning-preserving paraphrases, with the rephrasing prompt instructed to keep answer choices, formatting, and image references intact, and answers each with the same chain-of-thought template at temperature 0.75. Because every semantic-TTA answer already has the same output-side diversity as self-consistency, any measured advantage is attributed to the input variation itself.

What would settle it

Repeat the paper's matched-compute protocol, chain-of-thought at temperature 0.75, k in {2,4,6}, majority voting with k selected on a held-out sample, on a mid-tier open-weight model from a different family across the same six benchmarks; the central claim is falsified if semantic TTA does not beat self-consistency on five of six tasks, or if its accuracy-per-dollar advantage falls well below 1.8x.

Watch

Extended reading notes

Core claim

On a matched inference budget, varying the input converts compute into accuracy more efficiently than varying the reasoning path alone. Every method in the comparison answers the same questions with chain-of-thought prompting at the same temperature, and the budget is the number of answer calls; self-consistency spends all calls on one unchanged prompt, while semantic TTA spends one extra call to generate paraphrases and then answers each. Across six benchmark suites, semantic TTA's majority vote beat self-consistency's on five of six tasks, gained 1.8 percentage points over single-call CoT on average, and was statistically significant in paired tests; the authors describe it as Pareto-dominating self-consistency because it wins on both accuracy and accuracy per dollar, roughly 1.8x. Lexical TTA (character-level typos) produced smaller but significant gains, visual TTA helped on the multimodal benchmark, and gains shrank as the base model grew stronger, so the paper scopes its conclusion to current mid-tier models and positions TTA as a compute-efficiency tool rather than a substitute for a model upgrade.

Load-bearing premise

The conclusion rests on the assumption that the Claude model family speaks for other current mid-tier LLMs, so the advantage measured on Haiku, Sonnet, and Opus is a property of mid-tier models generally and not an artifact of this one family or of using the same model to rephrase and answer.

Editorial extensions

If this is right

  • At a fixed number of answer calls, semantic TTA should beat self-consistency on discrete-answer benchmarks where predictions are sensitive to phrasing, since it adds input diversity on top of the same output diversity.
  • The accuracy-per-dollar advantage is roughly 1.8x for mid-tier models, but it shrinks as the base model improves (2.75pp on Haiku, 0.25pp on Opus), so the method targets the regime where a stronger model is unavailable or too expensive.
  • Semantic TTA peaks near k=4, while self-consistency keeps improving up to k=10, making TTA the better use of a small-to-moderate inference budget and making the ordering budget-dependent.
  • Lexical TTA is a nearly free fallback that still beats single-call CoT, while visual TTA gives modest gains on multimodal questions; combining text and image augmentation hurts accuracy, so text-only augmentation is the recommended multimodal strategy.
  • Because the gain is measured against a self-consistency baseline that already includes repeated sampling, the result implies input-side and output-side diversity can be combined, with tuning their balance identified as future work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism is variance reduction over surface form, then the benefit should scale with a model's paraphrase flip rate; a benchmark-by-benchmark correlation between TTA gain and paraphrase sensitivity would test this mechanism more directly than the paper's aggregate significance test.
  • The paper couples the rephraser and the answerer as the same model, so the observed advantage might partly reflect correlation between paraphrase errors and answer errors; using a different or stronger model for rephrasing could widen or narrow the 1.8x gap.
  • The cost comparison uses token prices, so under latency or parallelism constraints, where the rephrasing call is either hidden or serialized, the relative advantage of TTA over self-consistency would differ from the reported ratio.
  • MMMLU is scored as one accuracy number across 14 languages, but paraphrase quality varies by language; a per-language breakdown could reveal that input-side diversity helps mainly high-resource languages, which is not tested in the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper studies Test-Time Augmentation (TTA) for LLMs, where predictions are aggregated across transformed versions of the input, and compares input-side diversity (semantic rephrasing, lexical perturbations, visual transformations) against output-side diversity (self-consistency) on six benchmarks (MMLU, MMMLU, MMMU, HLE, Math500, IMDB). The main claim, stated in the abstract and Section 6, is that for current mid-tier LLMs, varying the input converts inference compute into accuracy more efficiently than varying the reasoning path alone, with semantic rephrasing delivering roughly 1.8× more accuracy per dollar and outperforming self-consistency on five of six tasks. The paper also reports ablations on the number of augmentations, multi-modal strategies, and base model scaling, and it releases an implementation on GitHub.

Significance. The question is practically important: if correct, semantic TTA is a training-free, input-side complement to self-consistency that improves accuracy per inference dollar on discrete-answer benchmarks. The study has several methodological strengths: k is selected on a held-out sample (Section 4.2), the main comparisons include paired t-tests and a paired bootstrap (Section 5.1, Appendix C), and the implementation is public. The scaling analysis across three Claude model sizes (Section 5.5) gives a sensible account of where the method is most useful. The paper's evidence is, however, restricted to one proprietary model family, and the headline 'matched compute' comparison is not actually made at equal total inference budgets.

major comments (2)
  1. [§3.5, §5.2, Figs. 4–5] The paper's title and abstract promise a 'matched-compute' comparison, but the actual comparison is 'matched k': semantic TTA spends one rephrasing call plus k answer calls, while self-consistency spends k answer calls (Section 3.5). The cost-effectiveness ratios in Figure 5 are computed at each method's own held-out optimal k, so the reported 1.8× 'accuracy per dollar' is not a comparison of the two methods at equal total LLM calls or equal dollar budgets. Because Figure 6 shows self-consistency still improving up to k=10 on Math500, the missing comparison—semantic TTA with k answers plus one rephrasing call versus self-consistency with k+1 answers, and the corresponding equal-dollar comparison—is exactly where the advertised advantage could shrink or reverse. This is a load-bearing gap for the central efficiency claim; it can be closed by reporting accuracy at equal total calls and at equal total cost for each dataset.
  2. [§5.5, §6] The abstract and conclusion generalize the finding to 'current mid-tier LLMs,' but all experiments use a single proprietary family (Claude 4.5 Haiku, Sonnet, Opus). The limitation in Section 6 acknowledges that behavior on open-weight or other families 'remains to be verified,' yet the conclusion still states the general claim. Since the paper's central claim is about a class of models rather than one family, this is an unsupported extrapolation; either add at least one additional model family on a subset of datasets or restrict the claims to the Claude family and adjust the title and abstract accordingly.
minor comments (5)
  1. [§3.5] The sentence 'Semantic TTA can be seen as a strict extension of this procedure' is not strictly accurate, since semantic TTA adds an extra rephrasing call and changes the input distribution; a more precise phrasing would be 'an extension that adds input-side diversity.'
  2. [§5.1, Appendix C] The paired bootstrap pools 2,400 per-question gains as independent observations, but per-question gains within a dataset are correlated; reporting a dataset-level or cluster-robust confidence interval would strengthen the significance claim.
  3. [Table 2] The HLE row shows '62' for both lexical TTA and self-consistency, which appears to be a table alignment issue; the formatting should be corrected to make the per-method optimal k values clear.
  4. [§4.3] The sentence 'All methods incur computational cost proportional to k' is imprecise for semantic TTA, which also incurs a fixed rephrasing cost; the next sentence says this, so a small clarification would avoid the seeming contradiction.
  5. [Appendix A] The rephrasing prompt explicitly preserves answer choices for multiple-choice items, but for open-ended questions it could more clearly instruct the model to preserve all numeric constraints and units; this is a minor clarity issue.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: all headline comparisons are measured against external benchmarks, with k selected on held-out data; the self-citation in Related Work is not load-bearing.

full rationale

The paper's central claims are empirical measurements rather than derivations. Semantic TTA is defined by a concrete pipeline (one rephrasing LLM call producing k variants, k answer calls, majority voting) and is compared against self-consistency on six external benchmarks; no reported accuracy or cost figure is defined in terms of the conclusion that input diversity is more efficient. The 1.8X per-dollar figure is computed from measured accuracy gains and token costs, not fitted to produce the result. The only self-citation (Butler et al. 2025) appears in Related Work and supports no load-bearing premise. The use of the same model for rephrasing and answering is a potential confound that the limitations section explicitly acknowledges, but a confound is not circularity. The mismatch between the title's 'matched compute' and the actual 'matched k' comparison is an interpretational weakness about the compute frontier, not a reduction of the result to its inputs by construction. Accordingly, no circular step meets the evidence bar of quoting an equation or definition that makes the output equal to the input.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper's central claim rests on a small set of empirical preconditions: that LLMs are phrasing-sensitive, that the rephraser preserves meaning, and that the Claude family is representative of current mid-tier LLMs. The augmentations k and perturbation strengths are hand-set or selected on a held-out sample, and the evaluation subset is not fully specified. No invented entities are introduced.

free parameters (5)
  • Augmentation count k = k in {2,4,6} per dataset and method; optimal values in Table 2 (e.g., semantic TTA average k=4.33)
    Selected per method and dataset on a held-out sample via grid search; affects reported accuracy and all cost-effectiveness ratios.
  • Lexical perturbation probability = 5% per word, max 10 perturbations
    Hand-set based on prior work on character-level noise; controls lexical TTA input diversity and its gains.
  • Visual transformation ranges = rotation +/-3 degrees, brightness/contrast +/-5%
    Hand-set to keep images recognizable; controls visual TTA results on MMMU.
  • Sampling temperature = T=0.75
    Set for answer sampling; affects diversity of both self-consistency and TTA answer paths.
  • Evaluation subset of 400 examples per dataset = not specified (only 'fixed random seed' mentioned)
    The exact 400 questions per dataset are not released, so the reported accuracies and bootstrap intervals cannot be reproduced exactly.
assumptions (4)
  • domain assumption LLM predictions are sensitive to surface phrasing; aggregating across phrasings reduces variance.
    Cited from literature (Seleznyov et al., Chatziveroglou et al., Agrawal et al.) in Section 2; not re-derived or re-measured independently.
  • domain assumption The rephraser preserves the meaning and correct answer while varying surface form.
    Prompt instructs this (Appendix A.1), but no automatic or human validation of paraphrase quality is reported.
  • domain assumption Majority voting with random tie-breaking is a valid aggregation for all six tasks.
    Equation (1) is standard for discrete answers, but random tie-breaking adds noise at even k and can affect non-monotonic accuracy curves.
  • ad hoc to paper Claude 4.5 Haiku, Sonnet, and Opus represent current mid-tier and larger LLMs.
    Only one proprietary model family is tested; generalization to other families is assumed and flagged as a limitation in Section 5.5.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Test-Time Augmentation for LLMs: When Input Diversity Beats Output Diversity at Matched Compute." pith.science (2026). https://pith.science/paper/46LFMSDK

@misc{pith2026260809351,
  author       = {Pith},
  title        = {Pith review of: Test-Time Augmentation for LLMs: When Input Diversity Beats Output Diversity at Matched Compute},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/46LFMSDK}},
  note         = {Machine review of arXiv:2608.09351}
}
read the original abstract

Test-time scaling improves LLM accuracy but multiplies inference cost, making the accuracy gained per unit of compute the metric that matters in deployment. Self-consistency is one of the established approaches, which spends this budget entirely on the output side by sampling repeated reasoning paths. We study Test-Time Augmentation (TTA), which extends self-consistency by also perturbing the input, aggregating predictions across transformed versions of the input, and ask whether input-side diversity converts compute into accuracy more efficiently than output-side diversity. We perform a systematic, matched-compute comparison: we evaluate three simple input-side strategies (semantic rephrasing, lexical perturbations, and visual transformations) across six datasets covering general and multilingual knowledge, mathematical reasoning, multi-modal question answering, and sentiment classification, against chain-of-thought prompting and self-consistency. Semantic rephrasing delivers consistent and statistically significant accuracy gains while Pareto-dominating self-consistency on cost-effectiveness, delivering roughly 1.8X more accuracy per dollar and outperforming it on five of six tasks. We further analyze the number of augmentations, multi-modal strategies, and base model scaling, finding that TTA is most cost-effective for mid-tier models where a stronger model is unavailable or too expensive. Our findings indicate that for current mid-tier LLMs, varying the input converts inference compute into accuracy more efficiently than varying the reasoning path alone. The TTA implementation is available at https://github.com/aws-samples/sample-genai-reflection-for-bedrock.

Figures

Figures reproduced from arXiv: 2608.09351 by the authors.

Figure 1
Figure 1. TTA framework. The input question and image are augmented into [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Accuracy improvement over the CoT prompting baseline across benchmarks (in [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Pairwise statistical significance of accuracy gains across datasets. Semantic TTA [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Cost-accuracy frontiers (Math500, MMLU). [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Cost-effectiveness of each method, measured as accuracy gain relative to additional [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: shows extended experiments on Math500 with k up to 10. Accuracy is non￾monotonic in k: small drops at certain k reflect stochastic paraphrase generation and tie￾breaking dynamics, particularly for odd vs. even k. These fluctuations are small (within 1 pp). Semantic TTA…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 8 canonical work pages

  1. [1]

    Enhancing llm robustness to perturbed instructions: An empirical study.arXiv preprint arXiv:2504.02733,

    Aryan Agrawal, Lisa Alazraki, Shahin Honarvar, and Marek Rei. Enhancing llm robustness to perturbed instructions: An empirical study.arXiv preprint arXiv:2504.02733,

    arXiv

  2. [3]

    The surprising effectiveness of test-time training for few-shot learning.arXiv preprint arXiv:2411.07279,

    Ekin Akyürek, Mehul Damani, Adam Zweiger, Linlu Qiu, Han Guo, Jyothish Pari, Yoon Kim, and Jacob Andreas. The surprising effectiveness of test-time training for few-shot learning.arXiv preprint arXiv:2411.07279,

    arXiv

  3. [5]

    Yaping Chai, Haoran Xie, and Joe S. Qin. Text data augmentation for large language models: A comprehensive survey of methods, challenges, and opportunities.arXiv preprint arXiv:2501.18845,

    arXiv

  4. [6]

    Exploring llm reasoning through controlled prompt variations.arXiv preprint arXiv:2504.02111,

    Giannis Chatziveroglou, Richard Yun, and Maura Kelleher. Exploring llm reasoning through controlled prompt variations.arXiv preprint arXiv:2504.02111,

    arXiv

  5. [7]

    Teaching large language models to self-debug.arXiv preprint arXiv:2304.05128,

    Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug.arXiv preprint arXiv:2304.05128,

    arXiv

  6. [9]

    No language left behind: Scaling human-centered machine translation.arXiv preprint arXiv:2207.04672,

    Marta R Costa-Jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. No language left behind: Scaling human-centered machine translation.arXiv preprint arXiv:2207.04672,

    arXiv

  7. [11]

    Frustratingly easy test-time adaptation of vision-language models.arXiv preprint arXiv:2405.18330,

    Matteo Farina, Gianni Franchi, Giovanni Iacca, Massimiliano Mancini, and Elisa Ricci. Frustratingly easy test-time adaptation of vision-language models.arXiv preprint arXiv:2405.18330,

    arXiv

  8. [12]

    M-Ped: Multi-Prompt Ensemble Decoding for Large Language Models

    Jiaxin Guo, Daimeng Wei, Yuanchang Luo, Shimin Tao, Hengchao Shang, Zongyao Li, Shaojun Li, Jinlong Yang, Zhanglin Wu, Zhiqiang Rao, and Hao Yang. M-Ped: Multi- prompt ensemble decoding for large language models.arXiv preprint arXiv:2412.18299,

    work page Pith review arXiv

Show all 36 references
  1. [13]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300,

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300,

    2009 arXiv

  2. [15]

    Efficiently learning at test-time: Active fine-tuning of llms.arXiv preprint arXiv:2410.08020,

    11 Published at the COLM 2026 Workshop on Efficient Reasoning Jonas Hübotter, Sascha Bongni, Ido Hakimi, and Andreas Krause. Efficiently learning at test-time: Active fine-tuning of llms.arXiv preprint arXiv:2410.08020,

    2026 arXiv

  3. [16]

    Calibrating language models via augmented prompt ensembles

    Mingjian Jiang, Yangjun Ruan, Sicong Huang, Saifei Liao, Silviu Pitis, Roger Grosse, and Jimmy Ba. Calibrating language models via augmented prompt ensembles. InICML 2023 Workshop on Deployment Challenges for Generative AI,

    2023

  4. [17]

    Test-time aug- mentation for factual probing.arXiv preprint arXiv:2310.17121,

    Go Kamoda, Benjamin Heinzerling, Keisuke Sakaguchi, and Kentaro Inui. Test-time aug- mentation for factual probing.arXiv preprint arXiv:2310.17121,

    arXiv

  5. [18]

    Improved text classification via test-time augmentation.arXiv preprint arXiv:2206.13607,

    Helen Lu, Divya Shanmugam, Harini Suresh, and John Guttag. Improved text classification via test-time augmentation.arXiv preprint arXiv:2206.13607,

    arXiv

  6. [19]

    Reft: Reasoning with reinforced fine-tuning.arXiv preprint arXiv:2401.08967,

    Trung Quoc Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning.arXiv preprint arXiv:2401.08967,

    arXiv

  7. [21]

    Long Phan et al

    Dataset available athttps://huggingface.co/datasets/openai/MMMLU. Long Phan et al. Humanity’s last exam.arXiv preprint arXiv:2501.14249,

    arXiv

  8. [22]

    Zhang, Andrew Wang, and Jimmy Ba

    Silviu Pitis, Michael R. Zhang, Andrew Wang, and Jimmy Ba. Boosted prompt ensembles for large language models.arXiv preprint arXiv:2304.05970,

    arXiv

  9. [23]

    When punctuation matters: A large-scale comparison of prompt robustness methods for llms.arXiv preprint arXiv:2508.11383,

    Mikhail Seleznyov, Mikhail Chaichuk, Gleb Ershov, Alexander Panchenko, Elena Tutubalina, and Oleg Somov. When punctuation matters: A large-scale comparison of prompt robustness methods for llms.arXiv preprint arXiv:2508.11383,

    arXiv

  10. [24]

    Rewarding progress: Scaling automated process verifiers for llm reasoning.arXiv preprint arXiv:2410.08146,

    Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for llm reasoning.arXiv preprint arXiv:2410.08146,

    arXiv

  11. [25]

    Better aggregation in test-time augmentation.arXiv preprint arXiv:2011.11156,

    Divya Shanmugam, Davis Blalock, Guha Balakrishnan, and John Guttag. Better aggregation in test-time augmentation.arXiv preprint arXiv:2011.11156,

    2011 arXiv

  12. [27]

    On the self-verification limitations of large language models on reasoning and planning tasks.arXiv preprint arXiv:2402.08115,

    Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. On the self-verification limitations of large language models on reasoning and planning tasks.arXiv preprint arXiv:2402.08115,

    arXiv

  13. [28]

    Confidence improves self-consistency in llms.arXiv preprint arXiv:2502.06233,

    12 Published at the COLM 2026 Workshop on Efficient Reasoning Amir Taubenfeld, Tom Sheffer, Eran Ofek, Amir Feder, Ariel Goldstein, Zorik Gekhman, and Gal Yona. Confidence improves self-consistency in llms.arXiv preprint arXiv:2502.06233,

    2026

  14. [29]

    Can large language models really improve by self-critiquing their own plans?arXiv preprint arXiv:2310.08118,

    Karthik Valmeekam, Matthew Marquez, and Subbarao Kambhampati. Can large language models really improve by self-critiquing their own plans?arXiv preprint arXiv:2310.08118,

    arXiv

  15. [30]

    Paraphrase types elicit prompt engineering capabilities.arXiv preprint arXiv:2406.19898,

    Jan Philip Wahle, Terry Ruas, Yang Xu, and Bela Gipp. Paraphrase types elicit prompt engineering capabilities.arXiv preprint arXiv:2406.19898,

  16. [31]

    Paraphrase and aggregate with large lan- guage models for minimizing intent classification errors.arXiv preprint arXiv:2406.17163,

    Vikas Yadav, Zheng Tang, and Vijay Srinivasan. Paraphrase and aggregate with large lan- guage models for minimizing intent classification errors.arXiv preprint arXiv:2406.17163,

    arXiv

  17. [32]

    Your language model may think too rigidly: Achieving reasoning consistency with symmetry-enhanced training.arXiv preprint arXiv:2502.17800,

    Yihang Yao, Zhepeng Cen, Miao Li, William Han, Yuyou Zhang, Emerson Liu, Zuxin Liu, Chuang Gan, and Ding Zhao. Your language model may think too rigidly: Achieving reasoning consistency with symmetry-enhanced training.arXiv preprint arXiv:2502.17800,

    arXiv

  18. [33]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.arXiv preprint arXiv:2311.16502,

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.arXiv preprint arXiv:2311.16502,

    arXiv

  19. [34]

    PREFER: Prompt ensemble learning via feedback-reflect-refine.arXiv preprint arXiv:2308.12033,

    Chenrui Zhang, Lin Liu, Jinpeng Wang, Chuyuan Wang, Xiao Sun, Hongyu Wang, and Mingchen Cai. PREFER: Prompt ensemble learning via feedback-reflect-refine.arXiv preprint arXiv:2308.12033,

    arXiv

  20. [35]

    Paraphrase and solve: Exploring and exploiting the impact of surface form on mathematical reasoning in large language models.arXiv preprint arXiv:2404.11500,

    Yue Zhou, Yada Zhu, Diego Antognini, Yoon Kim, and Yang Zhang. Paraphrase and solve: Exploring and exploiting the impact of surface form on mathematical reasoning in large language models.arXiv preprint arXiv:2404.11500,

    arXiv

  21. [36]

    output as array

    13 Published at the COLM 2026 Workshop on Efficient Reasoning A Prompt Templates A.1 Answering and Rephrasing Prompts Section A.1 presents the answering and rephrasing prompts. The answering prompt is used by both the baseline and all TTA methods. The rephrasing prompt is used...

    2026

  22. [2017]

    Dynamic sentiment analysis with local large language models using majority voting: A study on factors affecting restaurant evaluation.arXiv preprint arXiv:2407.13069,

    Junichiro Niimi. Dynamic sentiment analysis with local large language models using majority voting: A study on factors affecting restaurant evaluation.arXiv preprint arXiv:2407.13069,

    arXiv

  23. [2020]

    Scaling llm test-time compute opti- mally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314,

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute opti- mally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314,

    arXiv

  24. [2021]

    Mirror-consistency: Harnessing inconsistency in majority voting.arXiv preprint arXiv:2410.10857,

    Siyuan Huang, Zhiyuan Ma, Jintao Du, Changhua Meng, Weiqiang Wang, and Zhouhan Lin. Mirror-consistency: Harnessing inconsistency in majority voting.arXiv preprint arXiv:2410.10857,

  25. [2022]

    Rephrase and respond: Let large language models ask better questions for themselves.arXiv preprint arXiv:2311.04205,

    Yihe Deng, Weitong Zhang, Zixiang Chen, and Quanquan Gu. Rephrase and respond: Let large language models ask better questions for themselves.arXiv preprint arXiv:2311.04205,

    arXiv

  26. [2023]

    RoParQ: Paraphrase-aware alignment of large language models towards robustness to paraphrased questions.arXiv preprint arXiv:2511.21568,

    Minjoon Choi. RoParQ: Paraphrase-aware alignment of large language models towards robustness to paraphrased questions.arXiv preprint arXiv:2511.21568,

  27. [2024]

    Finding the sweet spot: Trading quality, cost, and speed during inference-time llm reflection.arXiv preprint arXiv:2510.20653,

    Jack Butler, Nikita Kozodoi, Zainab Afolabi, Brian Tyacke, and Gaiar Baimuratov. Finding the sweet spot: Trading quality, cost, and speed during inference-time llm reflection.arXiv preprint arXiv:2510.20653,

  28. [2025]

    Beyond major- ity voting: LLM aggregation by leveraging higher-order information.arXiv preprint arXiv:2510.01499,

    Rui Ai, Yuqi Pan, David Simchi-Levi, Milind Tambe, and Haifeng Xu. Beyond major- ity voting: LLM aggregation by leveraging higher-order information.arXiv preprint arXiv:2510.01499,

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.