Pith. sign in

REVIEW 4 major objections 5 minor 41 references

Incorporating Token Usage into Prompting Strategy Evaluation

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that prompting strategies should be judged by tokens per accuracy point, not accuracy alone, and presents evidence that extra tokens buy drastically shrinking accuracy gains.

desk verdict A genuinely useful token-efficiency metric wrapped in an over-interpreted headline number; the qualitative diminishing-returns pattern is real and worth citing. read the letter →

arxiv 2505.14880 v1 pith:5KQUCBRK submitted 2025-05-20 cs.CL

Chris Sypherd , Sergei Petrov , Sonny George , Vaishak Belle This is my paper
classification cs.CL
keywords promptingstrategyevaluationtokenusageBig-O_tokCostchain-of-thoughtself-consistencyLLMefficiencydiminishingreturns
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that evaluating prompting strategies by benchmark accuracy alone is misleading, because strategies differ enormously in how many tokens they consume. It proposes two tools for efficiency-aware evaluation: Big-$O_{tok}$, an asymptotic class for the token usage of a prompting strategy, and Token Cost, the number of tokens spent per percentage point of accuracy. Across three open-weight models (8B, 14B, 32B) and three benchmarks (BBH, GSM8K, MMLU), average Token Cost grows from 5.0 $t/p$ for the least accurate strategy to 119.4 $t/p$ for the most accurate one — a more than twenty-fold drop in efficiency — and the marginal cost of moving from Fewshot CoT to self-consistency with ten chains is 6,701.8 $t/p$. If the claim is right, accuracy-only leaderboards systematically hide the cost of chasing the last few accuracy points.

What carries the argument

Two objects carry the argument. Big-$O_{tok}$ is a token-usage analogue of Big-O time complexity: it assigns each prompting strategy an asymptotic class as a function of its strategy variables — $O(1)$ for Vanilla IO and Zeroshot CoT, $O(k)$ for Vanilla Fewshot and Fewshot CoT (with $k$ exemplars), and $O(pk)$ for Chain-of-Thought Self-Consistency (with $p$ sampled chains). Token Cost (TC) is the empirical ratio of tokens consumed to accuracy achieved, measured in tokens per percentage point ($t/p$), with average TC = tokens/accuracy and marginal TC = $\Delta$tokens$/\Delta$accuracy between two strategies. Big-$O_{tok}$ predicts the relative token-usage ratios observed in the experiments, and TC places each strategy on the accuracy-versus-tokens return curve, which the authors fit to a $\log(\log(x))$ trend.

What would settle it

Re-run the five prompting strategies on GSM8K with five independent seeds per strategy and recompute average and marginal Token Cost; if the 6,701.8 $t/p$ marginal step from Fewshot CoT to CoT-SC10 does not reproduce, or if extending self-consistency to twenty chains keeps marginal cost near the 65.3 $t/p$ early-step level, the claim of drastically diminishing performance returns would fail.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that increased token usage leads to drastically diminishing performance returns: across all models and benchmarks, the accuracy-versus-token curves plateau in the shape of $\log(\log(x))$, so that each additional accuracy point costs far more tokens than the one before. Concretely, the average Token Cost of the lowest-accuracy strategy is 5.0 $t/p$ while that of the highest-accuracy strategy is 119.4 $t/p$, a more than twenty-fold efficiency decline. The marginal Token Cost between Vanilla IO and Fewshot CoT is 65.3 $t/p$, whereas the marginal Token Cost between Fewshot CoT and CoT-SC10 is 6,701.8 $t/p$ — more than two orders of magnitude less efficient. The authors take this as evidence that accuracy alone is an inadequate measure of prompting strategy success and that reporting efficiency, via Token Cost and the Big-$O_{tok}$ classes, is more practical for real-world utility.

Load-bearing premise

The whole comparison rests on token counts from a single run of each strategy-model-benchmark pair, after deleting empty outputs and outputs longer than four standard deviations (up to 8.59 percent of pairs in one cell); if the deleted outputs were mostly from the token-hungry strategies, the reported twenty-fold efficiency decline could be exaggerated or reversed.

Editorial extensions

If this is right

  • Prompting-strategy comparisons should report tokens alongside accuracy, because a strategy that gains a few points at twenty times the token cost is not automatically preferable.
  • Big-$O_{tok}$ lets practitioners predict relative token usage before running experiments — for instance, self-consistency grows linearly in the number of sampled chains — so efficiency can be designed in rather than measured afterwards.
  • In cost-sensitive applications, the data favour the cheap strategies (Vanilla IO, Zeroshot CoT) with average Token Cost below 10 $t/p$, reserving multi-chain strategies for settings where the last few points justify about 6,700 tokens each.
  • The fewshot ablation shows the same diminishing-returns shape, so the claim also applies to choosing how many exemplars to include, not just which strategy family to use.
  • Token Cost provides a single number to budget against when selecting prompting strategy variables such as the number of exemplars or the number of sampled chains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the return curve is as flat as these numbers suggest, production systems that keep adding reasoning or sampling tokens (longer chains, larger ensembles, reasoning models) are likely operating on the plateau, and an explicit per-task Token Cost budget would make that waste visible before deployment.
  • The authors counted input and output tokens together, but commercial APIs typically price output tokens several times higher, so re-weighting tokens by price would steepen the diminishing-returns curve and strengthen the practical case for efficiency-aware evaluation.
  • A direct test of the framework's boundary would apply Big-$O_{tok}$ and Token Cost to strategies the paper did not test, such as self-consistency with far more chains or structurally different multi-hop strategies; the authors concede these may break the trend, which would mark the edge of diminishing returns rather than refute the metrics.
  • Since each strategy-model-benchmark cell was executed once, the reported 20-fold and 100-fold figures carry no confidence intervals, making multi-seed replication the immediate next step for turning Token Cost into a decision tool.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper makes a case for incorporating token usage into the evaluation of prompting strategies. It proposes Big-O_tok, an asymptotic token-complexity framework for prompting strategies, and Token Cost (TC), an empirical ratio of tokens consumed per percentage point of benchmark accuracy. Using three open-weight LLMs, three benchmarks, and six prompting strategies, the authors report that average TC ranges from 5.0 t/p for the lowest-accuracy strategy to 119.4 t/p for the highest-accuracy strategy, and that marginal TC between Fewshot CoT and CoT-SC10 is 6701.8 t/p, a claimed drop in efficiency of more than two orders of magnitude. They also compare theoretical token-usage ratios derived from Big-O_tok with observed ratios and interpret the agreement as validation of the framework. The paper is transparent about its scope, provides reproducibility artifacts, and explicitly discusses limitations.

Significance. If the quantitative claims held, the paper would be a useful contribution to efficiency-aware LLM evaluation: it gives simple, interpretable metrics (Big-O_tok and TC), a small but carefully scoped empirical study, and openly available results and configurations. The central qualitative message, that accuracy alone is a misleading criterion when token cost is ignored, is plausible and broadly consistent with the reported data. However, the headline numerical claims about 'drastic' diminishing returns and 'two orders of magnitude' efficiency loss are not statistically supported by the current analysis, and one measurement assumption (single-run token counts with exclusion of outliers) is load-bearing for the reported TC values. These issues are fixable, but they require additional analysis or substantially hedged conclusions.

major comments (4)
  1. [Section 4, marginal TC calculation] The claim that marginal TC between Fewshot CoT and CoT-SC10 is 6701.8 t/p, and the associated 'two orders of magnitude' statement, is not statistically grounded. The marginal TC is an average of ratios whose denominators are accuracy differences; for several experiments those differences are within the reported standard errors. For example, Table 6 shows Qwen 2.5 32B on GSM8K at 89.4% (SE 0.85) for Fewshot CoT and 89.8% (SE 0.83) for CoT-SC10, a 0.4-point difference that produces a marginal TC near 17,000 t/p even though the accuracies are not statistically distinguishable. The paper should report confidence intervals for marginal TC (e.g., via bootstrap over samples or over evaluation runs), or should soften the wording from 'drastically diminishing returns' to evidence compatible with the actual precision of the measurements.
  2. [Appendix C.1 and Table 5] All token-usage statistics come from a single execution per configuration, and outputs that are empty or more than four standard deviations from the mean are excluded before computing token counts. The exclusion rates are not negligible: Table 5 reports 8.59% of IO pairs excluded for Llama 3.1 8B on BBH under Vanilla Fewshot, and several other cells exceed 2%. Because the central TC values and the 20x efficiency decline are computed from these counts, the paper needs to show that the results are stable under this exclusion rule and across repeated runs. At minimum, a sensitivity analysis varying the exclusion threshold, or a small number of repeated runs for one or two configurations, is needed to establish that the average TC values are not artifacts of a single run.
  3. [Figure 1 and Section 4] The statement that 'across all benchmarks and models, our experiments demonstrate consistent trend lines (of the form y = log(log(x)))' overstates the regularity in the data. The token-accuracy relationship is not monotone in token usage: for Llama 3.1 8B on BBH, Vanilla Fewshot uses 646 total tokens at 35.4% accuracy, whereas Zeroshot CoT uses only 530 tokens at 63.2% accuracy (Table 6). Since token usage is the horizontal variable in Figure 1, a single increasing trend curve cannot represent such points. The authors should either plot per-strategy and per-model data without imposing a global functional form, or explicitly acknowledge that the log(log(x)) curve is a schematic summary rather than a fitted model with reported goodness of fit.
  4. [Section 4 and Table 2] The agreement between theoretical and observed token-usage ratios in Table 2 is presented as validation of Big-O_tok, but it is better described as a consistency check. The theoretical ratios are computed by substituting the exact values of k and p that were used to construct the prompts (e.g., p=5 and k=8 for GSM8K), so the agreement largely reflects the known construction of the inputs rather than an independent prediction about output token behavior. To make the validation claim substantive, the authors could test Big-O_tok out of sample, for example by varying k and p and checking whether observed token usage scales accordingly, or by predicting a ratio for a configuration not used in deriving the theoretical values.
minor comments (5)
  1. [Limitations] There is a duplicated phrase: 'We do not seek to seek to solve issues of prompting strategy efficiency' should read 'We do not seek to solve issues of prompting strategy efficiency.'
  2. [Appendix C.2] 'An potentially insightful future work' should be 'A potentially insightful future work.'
  3. [Figure 1] The trend lines in Figure 1 are described as log(log(x)), but no fitting procedure, parameter values, or goodness-of-fit statistics are reported; please clarify whether these are fitted curves or illustrative sketches.
  4. [Table 2] The caption for Table 2 does not specify whether the observed ratios use total tokens, input tokens, or output tokens, and it does not state that the counts exclude empty and >4SD outputs; this should be stated for reproducibility.
  5. [Section 3.1 and Limitations] The Big-O_tok analysis treats chain-of-thought instructions as constant overhead, but the empirical data show that output reasoning length can dominate token counts; the limitation paragraph on input/output tokens is helpful, but the point should be made earlier so that the reader does not expect O(1) to describe actual output token behavior.

Circularity Check

1 steps flagged · score 3.0 of 10

Big-Otok 'validation' is a self-consistency check, but the core diminishing-returns finding is empirically independent.

  1. self definitional [Section 4, Table 2 and surrounding paragraph]
    "To substantiate our Big-Otok analyses, we use the observed token usages from our experiments to calculate the relative token usage ratios between prompting strategies. We derive theoretical estimates of those ratios from our Big-Otok functions by substituting in the values from our experiments for the variables in Big-Otok (e.g., p= 5 and k= 5 for CoT-SC5)."

    The theoretical ratios in Table 2 are computed by substituting the same design variables that define the strategies (k exemplars, p sampled chains) into the O(1), O(k), and O(pk) labels from Table 1; for example, the CoT-SC10/CoT-SC5 ratio is (10k)/(5k)=2 by construction. The observed ratios are measured on runs that execute exactly those k and p counts, so the agreement for such ratios largely restates the experimental design rather than testing the framework. The paper accepts imperfect matches and requires only broad alignment with the three complexity classes, so the claim that the experiments 'validate the Big-Otok analyses' is a consistency check, not an external confirmation.

full rationale

The paper's main empirical contribution, Token Cost and the diminishing-returns curve, is not circular: TC is a definitional summary of independently measured tokens and accuracies, and the 5.0 versus 119.4 t/p and 65.3 versus 6701.8 t/p comparisons are arithmetic on those measurements. The partial circularity is confined to the Big-Otok validation: the theoretical token ratios are derived from the same k and p counts that define the strategies being measured, so they cannot serve as an independent test; the paper even concedes imperfect matches and only claims class-level alignment. I found no load-bearing self-citations; the cited prompting strategies, benchmarks, and evaluation harness are external. The unstable marginal-TC denominators flagged by a skeptical reader are a statistical robustness concern, not a circularity, and do not raise the score. On balance, the circularity is real but minor and isolated to the theoretical-validation claim, so I assign 3/10.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The paper's claims rest on standard big-O semantics, a characters/4 token approximation, and the assumption that prompt overhead is constant and output length scales with exemplars/chains. No parameters are fitted to data; the 4-standard-deviation exclusion threshold and fewshot counts are hand-chosen experimental settings.

free parameters (1)
  • 4 standard deviation exclusion threshold = 4
    Hand-chosen threshold for removing output outliers from token usage statistics (Appendix C.1). The percentage of excluded pairs varies by model and strategy, up to 8.59% for Llama 3.1 8B BBH Vanilla Fewshot, so this choice could affect reported token counts.
assumptions (5)
  • standard math Big-O notation and its asymptotic semantics (Knuth 1976)
    Used to define Big-O_tok in Section 3.1.
  • domain assumption Token count is proportional to text length, estimated as num characters / 4
    Token counts are estimated by the characters/4 heuristic (Section 4, footnote 10), which may diverge from true tokenizer counts across models.
  • domain assumption Additive prompt adjustments (e.g., 'Think step by step') contribute constant token overhead
    Big-O_tok treats additive input/output adjustments as constants (Section 3.1, Appendix A), which is reasonable asymptotically but affects low-variable regimes.
  • domain assumption Output token usage scales with the number of exemplars and sampled chains for fewshot and CoT-SC
    The O(k) and O(pk) classifications assume model output length is roughly proportional to the number of prompt exemplars/chains, not dominated by fixed template overhead (Section 3.1, Table 1).
  • domain assumption Accuracy differences between strategies are attributable to the prompting strategy rather than answer extraction or model idiosyncrasy
    The paper relies on LM Evaluation Harness extraction and notes 'certain inaccuracies in answer extraction' (Limitations).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Incorporating Token Usage into Prompting Strategy Evaluation." pith.science (2026). https://pith.science/paper/5KQUCBRK

@misc{pith2026250514880,
  author       = {Pith},
  title        = {Pith review of: Incorporating Token Usage into Prompting Strategy Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5KQUCBRK}},
  note         = {Machine review of arXiv:2505.14880}
}
abstract

In recent years, large language models have demonstrated remarkable performance across diverse tasks. However, their task effectiveness is heavily dependent on the prompting strategy used to elicit output, which can vary widely in both performance and token usage. While task performance is often used to determine prompting strategy success, we argue that efficiency--balancing performance and token usage--can be a more practical metric for real-world utility. To enable this, we propose Big-$O_{tok}$, a theoretical framework for describing the token usage growth of prompting strategies, and analyze Token Cost, an empirical measure of tokens per performance. We apply these to several common prompting strategies and find that increased token usage leads to drastically diminishing performance returns. Our results validate the Big-$O_{tok}$ analyses and reinforce the need for efficiency-aware evaluations.

Figures

Figures reproduced from arXiv: 2505.14880 by the authors.

Figure 1
Figure 1. Accuracy vs. token usage plots with standard error bars for various prompting strategies, models, and [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Sample derivations of Big-Otok. The textual descriptions in each figure are drawn from the following sources: (c) (Kojima et al., 2022); (d) (Brown et al., 2020); (e) (Wei et al., 2022); (f) (Wang et al., 2023b). Note that for (d), the fewshot examples are equivalent to the MVIO and we make the assumption that the LLM follows that pattern. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Accuracy and total token usage for the ablation study on the number of fewshot exemplars on the GSM8K [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Accuracy and total token usage information for Qwen 2.5 14B and Qwen 2.5 32B from the empirical [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 8 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Lee, Haonan Li, Charles Lovering, Niklas Muennighoff, Ellie Pavlick, Jason Phang, Aviya Skowron, Samson Tan, Xiangru Tang, Kevin A

    Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Jeffrey Hsu, Mimansa Jaiswal, Wilson Y. Lee, Haonan Li, Charles Lovering, Niklas Muennighoff, Ellie Pavlick,...

    arXiv 2024

  4. [4]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

  5. [5]

    Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Lorenzo Baraldi, Marcella Cornia, and Rita Cucchiara. 2024. https://doi.org/10.18653/v1/2024.findings-acl.807 The revolution of multimodal large language models: A survey . In Findings of the Association for Computational Linguistics: ACL 2024, pages 13590...

    doi:10.18653/v1/2024.findings-acl.807 2024

  6. [6]

    Lingjiao Chen, Matei Zaharia, and James Zou. 2023. https://arxiv.org/abs/2305.05176 Frugalgpt: How to use large language models while reducing cost and improving performance . Preprint, arXiv:2305.05176

    arXiv 2023

  7. [7]

    Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. 2024. https://doi.org/10.18653/v1/2024.acl-long.65 Navigate through enigmatic labyrinth a survey of chain of thought reasoning: Advances, frontiers and future . In Proceedings of the 62nd Annual Meeting of the Association for Comput...

    doi:10.18653/v1/2024.acl-long.65 2024

  8. [8]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . Preprint, arXiv:2110.14168

    arXiv 2021

Show all 41 references
  1. [9]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

    2025 arXiv

  2. [10]

    Payal Dhar. 2020. The carbon impact of artificial intelligence

    2020

  3. [11]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zha...

    2024 arXiv

  4. [12]

    Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. 2024. Open llm leaderboard v2. https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard

    2024

  5. [13]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2023 doi

  6. [14]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR)

    2021

  7. [15]

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.825 LLML ingua: Compressing prompts for accelerated inference of large language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural...

    2023 doi

  8. [16]

    JoongHoon Kim, Sangmin Lee, Seung Hun Han, Saeran Park, Jiyoon Lee, Kiyoon Jeong, and Pilsung Kang. 2023. https://doi.org/10.18653/v1/2023.eval4nlp-1.14 Which is better? exploring prompting strategy for LLM -based metrics . In Proceedings of the 4th Workshop on Evaluation and ...

    2023 doi

  9. [17]

    Donald E. Knuth. 1976. https://doi.org/10.1145/1008328.1008329 Big omicron and big omega and big theta . SIGACT News, 8(2):18–24

    1976

  10. [18]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. https://openreview.net/forum?id=e2TBb5y0yFf Large language models are zero-shot reasoners . In Advances in Neural Information Processing Systems

    2022

  11. [19]

    Seyed Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. 2025. https://openreview.net/forum?id=AjXkRZIvjB GSM -symbolic: Understanding the limitations of mathematical reasoning in large language models . In The Thirteenth Intern...

    2025

  12. [20]

    Jesse Mu, Xiang Lisa Li, and Noah Goodman. 2023. https://openreview.net/forum?id=2DtxPCL3T5 Learning to compress prompts with gist tokens . In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  13. [21]

    Sania Nayab, Giulio Rossolini, Giorgio Buttazzo, Nicolamaria Manes, and Fabrizio Giacomelli. 2024. https://arxiv.org/abs/2407.19825 Concise thoughts: Impact of output length on llm reasoning and cost . Preprint, arXiv:2407.19825

    2024 arXiv

  14. [22]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025 arXiv

  15. [23]

    Matthew Renze and Erhan Guven. 2024. https://arxiv.org/abs/2401.05618 The benefits of a concise chain of thought on problem-solving in large language models . Preprint, arXiv:2401.05618

    2024 arXiv

  16. [24]

    Abel Salinas and Fred Morstatter. 2024. https://doi.org/10.18653/v1/2024.findings-acl.275 The butterfly effect of altering prompts: How small changes and jailbreaks affect large language model performance . In Findings of the Association for Computational Linguistics: ACL 2024...

    2024 doi

  17. [25]

    Smith, and Oren Etzioni

    Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. 2020. https://doi.org/10.1145/3381831 Green ai . Commun. ACM, 63(12):54–63

    2020 doi

  18. [26]

    Bilgehan Sel, Ahmad Tawaha, Vanshaj Khattar, Ruoxi Jia, and Ming Jin. 2024. https://proceedings.mlr.press/v235/sel24a.html Algorithm of thoughts: Enhancing exploration of ideas in large language models . In Proceedings of the 41st International Conference on Machine Learning, ...

    2024

  19. [27]

    Sonish Sivarajkumar, Mark Kelley, Alyssa Samolyk-Mazzanti, Shyam Visweswaran, and Yanshan Wang. 2024. https://doi.org/10.2196/55318 An empirical evaluation of prompting strategies for large language models in zero-shot clinical natural language processing: Algorithm developmen...

    2024 doi

  20. [28]

    Brown, Adam Santoro, Aditya Gupta, Adri \`a Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adri \`a Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, ...

    2023

  21. [29]

    Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. 2023. https://doi.org/10.18653/v1/2023.findings-acl.824 Challenging BIG -bench tasks and whether chain-of-thought c...

    2023 doi

  22. [30]

    Shubham Vatsal and Harsh Dubey. 2024. https://arxiv.org/abs/2407.12994 A survey of prompt engineering methods in large language models for different nlp tasks . Preprint, arXiv:2407.12994

    2024 arXiv

  23. [31]

    Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, Mosharaf Chowdhury, and Mi Zhang. 2024. https://openreview.net/forum?id=bsCCJHbO8A Efficient large language models: A survey . Transactions on Machine Learning Res...

    2024

  24. [32]

    Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023 a . https://doi.org/10.18653/v1/2023.acl-long.147 Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models . In Proceedings of the 61st Annua...

    2023 doi

  25. [33]

    Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023 b . https://openreview.net/forum?id=1PL1NIMMrw Self-consistency improves chain of thought reasoning in language models . In The Eleventh International Confer...

    2023

  26. [34]

    Chi, Quoc V Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. https://openreview.net/forum?id=_VjQlMeSB_J Chain of thought prompting elicits reasoning in large language models . In Advances in Neural Information Proc...

    2022

  27. [35]

    Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with chatgpt. In Proceedings of the 30th Conference on Pattern Languages o...

    2023

  28. [36]

    Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga, Jinshi Huang, Charles Bai, et al. 2022. Sustainable ai: Environmental implications, challenges and opportunities. Proceedings of Machine Learning and Systems, 4:795--813

    2022

  29. [37]

    Rossi, Ruiyi Zhang, Subrata Mitra, Dimitris N

    Junda Wu, Zhehao Zhang, Yu Xia, Xintong Li, Zhaoyang Xia, Aaron Chang, Tong Yu, Sungchul Kim, Ryan A. Rossi, Ruiyi Zhang, Subrata Mitra, Dimitris N. Metaxas, Lina Yao, Jingbo Shang, and Julian McAuley. 2024. https://arxiv.org/abs/2409.15310 Visual prompting in multimodal large...

    2024 arXiv

  30. [38]

    Griffiths, Yuan Cao, and Karthik R Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. 2023. https://openreview.net/forum?id=5Xc1ecxO1h Tree of thoughts: Deliberate problem solving with large language models . In Thirty-seventh Conference on Neural Informat...

    2023

  31. [39]

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2024. https://doi.org/10.1093/nsr/nwae403 A survey on multimodal large language models . National Science Review, 11(12):nwae403

    2024 doi

  32. [40]

    Qingchen Yu, Zifan Zheng, Shichao Song, Zhiyu Li, Feiyu Xiong, Bo Tang, and Ding Chen. 2025. https://openreview.net/forum?id=7UqQJUKaLM xfinder: Large language models as automated evaluators for reliable evaluation . In The Thirteenth International Conference on Learning Repre...

    2025

  33. [41]

    Denny Zhou, Nathanael Sch \"a rli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. 2023. https://openreview.net/forum?id=WZH7099tgfM Least-to-most prompting enables complex reasoning in large language mode...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.