Pith. sign in

REVIEW 3 major objections 5 minor 48 references

Enhancing Zero-shot Chain of Thought Prompting via Uncertainty-Guided Strategy Selection

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Using the model's own uncertainty to choose chain-of-thought examples matches hand-crafted demonstrations without the human effort.

desk verdict Useful uncertainty-based demonstration selection, but the label-free strategy chooser is confounded by including demonstration questions in the uncertainty estimate. read the letter →

arxiv 2412.00353 v2 pith:3KVUH3QN submitted 2024-11-30 cs.CL cs.AI

Shanu Kumar , Saish Mendke , Karody Lubna Abdul Rahman , Santosh Kurasa , Parag Agrawal , Sandipan Dandapat This is my paper · ORCID
classification cs.CLcs.AI
keywords zero-shotchain-of-thoughtuncertaintyestimationdemonstrationselectionpredictiveentropyperturbation-basedlargelanguagemodelsreasoningbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes ZEUS, a method for building chain-of-thought demonstrations for large language models from unlabeled questions alone. ZEUS estimates each question's difficulty by sampling many rationale-answer pairs under three perturbations — temperature variation, different trigger phrases, and rephrased questions — then collapsing them into a predictive-entropy score. It selects questions whose entropy falls in a chosen band, builds demonstrations by clustering the selected questions, and at inference time chooses the band whose average temperature-perturbation uncertainty is lowest. The authors claim this uncertainty-guided selection matches or beats zero-shot, few-shot, Zero-Shot-CoT, Manual-CoT, and Auto-CoT across GSM8K, Logical Fallacy, StrategyQA, and EPR on five language models, without manual rationales or access to model parameters. If true, this removes the main human cost of chain-of-thought prompting while preserving most of its accuracy gain.

What carries the argument

The load-bearing object is the perturbation-based predictive entropy: a question's uncertainty $u_j$ computed from unique answers generated across temperature sampling, trigger-phrase variation, and question rephrasing. Low entropy means the model converges to the same answer under all perturbations; high entropy means its reasoning is unstable, and the paper treats that instability as a proxy for the question's difficulty or usefulness as a demonstration. The second piece of machinery is the strategy table: seven uncertainty bands (Trivial, Very Easy, Easy, Moderate, Challenging, Hard, Very Hard) defined by thresholds on $\mu$ and $\sigma$ of the unlabeled set. The entropy estimate carries the selection argument, while a separate, well-calibrated temperature-perturbation uncertainty is used at the end to choose which band to deploy without labels.

What would settle it

Run all seven ZEUS strategies on a held-out reasoning dataset and a model family beyond the five tested, computing each strategy's average Temp-Perb uncertainty on the unlabeled set and its actual accuracy on the test set. The central claim would be undercut if the lowest-uncertainty strategy is not among the top performing strategies, or if ZEUS's confidence-vs-accuracy slope is not closer to 1 than the temperature-only estimator's slope.

Watch

Extended reading notes

Core claim

The central discovery is that the model's own predictive entropy under perturbations is a usable, label-free signal for choosing which examples to put in a chain-of-thought prompt. For each unlabeled question $q_j$, ZEUS generates 15 responses: ten from five trigger phrases at temperature 1 (two per phrase) and five from GPT-4o-rephrased versions of $q_j$ at temperature 0. Unique answers are scored by their consistency across this pool, and the uncertainty is the predictive entropy $u_j=-\sum_c p(y^c_j|q_j)\log p(y^c_j|q_j)$. The paper reports that the regression slope of confidence against accuracy for this score is close to the ideal value of 1, while temperature-only perturbation is either under-sensitive or over-sensitive depending on the dataset. Seven selection strategies are then defined as bands on the unlabeled distribution using the mean $\mu$ and standard deviation $\sigma$; demonstrations are formed by clustering the selected questions and taking the cluster-centroid representative, following the Auto-CoT pipeline. The recommended variant, ZEUS (LU), picks the strategy with the lowest average Temp-Perb uncertainty and reaches accuracy close to the oracle-best strategy ZEUS (HA).

Load-bearing premise

The load-bearing premise is that the average temperature-perturbation uncertainty (Temp-Perb) of the unlabeled set, measured for each candidate demonstration strategy, is inversely correlated with that strategy's test accuracy, so the lowest-uncertainty strategy is the best one; on a new task or model where this correlation weakens, ZEUS (LU) could select a suboptimal strategy.

Editorial extensions

If this is right

  • On GSM8K, ZEUS (LU) outperforms all five baselines for every one of the five models tested.
  • On Logical Fallacy, ZEUS (LU) beats the baselines for four of the five models; the exception is GPT3-XL, where Few-Shot remains higher.
  • On StrategyQA and EPR, ZEUS (LU) matches or exceeds zero-shot, Zero-Shot-CoT, and Auto-CoT for most models, with Manual-CoT still best on StrategyQA for several models.
  • The optimal difficulty band is model-dependent: advanced models (GPT-4o, Phi3, GPT-3.5) do best with Hard or Challenging demonstrations, while simpler models (Mistral, GPT3-XL) do best with Trivial or Easy ones.
  • The recommended ZEUS (LU) requires no human annotations, no rationales, and no model parameter access, only unlabeled questions and the ability to sample from the model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same selection logic could be applied to other discrete prompt choices, such as demonstration count, clustering granularity, trigger phrase, or answer-format instructions, treating each as a strategy with its own average uncertainty.
  • The confidence-versus-accuracy slope used as a sensitivity diagnostic could become a general, label-free way to compare uncertainty estimators on new tasks, independent of whether ZEUS itself is used.
  • The advanced-versus-simple model split is described from five specific models and is likely a capability effect, so as model generations improve, the optimal-strategy mapping should be expected to shift rather than stay fixed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ZEUS, a three-stage zero-shot CoT prompting method: (1) estimate per-question uncertainty by perturbing decoding temperature, trigger phrases, and question rephrasing; (2) define seven selection strategies that pick questions in different uncertainty bands derived from the mean and standard deviation of the unlabeled set; (3) construct demonstrations by k-means clustering within the chosen band, following Auto-CoT. The method is evaluated on GSM8K, Fallacy, StrategyQA, and EPR with five LLMs. The main reported findings are that ZEUS's uncertainty estimates have higher 'sensitivity' than temperature perturbation alone, that the best-performing strategy varies by model and task, and that the strategy selected by lowest average Temp-Perb uncertainty (ZEUS(LU)) closely matches the oracle best strategy (ZEUS(HA)) without needing labels.

Significance. If the ZEUS(LU) match with ZEUS(HA) were established on a clean signal, the paper would provide a practical label-free way to choose demonstration difficulty for CoT prompting, with the welcome property of requiring no model-internal access. Strengths include the breadth of the evaluation (four datasets, five LLMs), the clearly specified three-stage pipeline, and the release of code and datasets. However, the central selection signal is confounded by a self-copying artifact (Major Comment 1), and the headline comparisons lack variance reporting, so the significance of the core claim is currently not established.

major comments (3)
  1. [§5.4, Figure 6, Table 3] The ZEUS(LU) selection rule is computed by averaging Temp-Perb uncertainty over the same unlabeled set Q from which each demonstration D_s is drawn. Since D_s already contains the question, rationale, and answer for every selected q, any such q in Q will have near-zero uncertainty when scored with D_s in the prompt. Strategies that deliberately select high-uncertainty questions (Hard, Very Hard) therefore remove the largest contributions to the average, mechanically lowering the average uncertainty relative to Trivial/Easy strategies, independent of any effect on unseen questions. This can manufacture the inverse accuracy-uncertainty trend in Figure 6 and the LU≈HA match in Table 3. The paper does not report recomputing the average on Q \ D_s or on a held-out subset. To support the label-free optimal-selection claim, the authors should either exclude demonstration questions when computing the selection signal or use a separate validation subset, and show LU≈HA under that corrected signal.
  2. [§5.5, Table 3] No variance or significance measures accompany the reported accuracies, despite the paper stating in §4 that performance is averaged over three runs. Several key differences are small (e.g., GSM8K GPT4o: 95.8 vs 94.8; EPR Phi3: 76.2 vs 75.3), and on StrategyQA Manual-CoT beats ZEUS(LU) for four of five models. Without standard deviations, confidence intervals, or paired tests, the claim that ZEUS 'consistently outperforms' baselines is not statistically supported. The authors should report variability across runs and perform paired comparisons against Auto-CoT and Manual-CoT.
  3. [Abstract, §6] The abstract states ZEUS 'consistently outperforms existing CoT strategies across four challenging reasoning benchmarks,' but Table 3 shows Manual-CoT has the highest StrategyQA accuracy for GPT4o (tie), Phi3, Mistral, and GPT3.5. The body acknowledges this in §5.5 ('Manual-CoT achieves the highest accuracy for most models'). The abstract and conclusion should be revised to describe the task- and model-dependent results accurately, or the claim should be restricted to the datasets and settings where it holds.
minor comments (5)
  1. [§3.1, Eq. (1)] The confidence score in Eq. (1) is written as a sum over n responses, but the total number of generated responses per question is n × t × v; the normalization should be made consistent and the notation clarified.
  2. [§4] The paper says temperature is set to 0 for deterministic outputs and results are averaged over three runs; please clarify what varies across the three runs (for example, random seed or sampling of the unlabeled set).
  3. [§4, Table 3] Manual-CoT results for GPT3.5 and GPT3-XL on the Fallacy dataset are marked with dashes; please explain why these baselines are missing.
  4. [§3.1, §4] The rephrasing perturbation is generated with GPT4o for all evaluated models; this makes the method dependent on a strong external model, and for the GPT4o rows the uncertainty estimator and the evaluated model coincide. This dependency should be disclosed and its effect discussed.
  5. [§5.2, Figure 5] The sensitivity slopes are reported without error bars or confidence intervals, so the claim that ZEUS is 'closest to ideal' is not quantified; please add uncertainty estimates to the regression coefficients.

Circularity Check

0 steps flagged · score 2.0 of 10

No derivation-level circularity; ZEUS(LU) is label-free and benchmark-validated, with only a minor non-load-bearing self-citation.

full rationale

The central claim is an empirical comparison: ZEUS(LU) selects a demonstration strategy by minimizing Temp-Perb uncertainty averaged over the unlabeled set Q, and this selection is evaluated on held-out test accuracy against external baselines. No test labels enter the LU selection rule, so the result is not a fitted parameter renamed as a prediction. The skeptic concern that demonstration questions q in D_s are included in the averaged uncertainty is quantitatively negligible because D_s contains only k=8 (or 6) questions whereas Q contains thousands; self-copying at most k answers cannot drive the inverse correlation in Figure 6. The 'lowest uncertainty' rule was adopted after observing the accuracy-uncertainty trend on the same four datasets; this is post-hoc selection on a validation set rather than a circular derivation, and the Limitation section explicitly discloses the need for exhaustive strategy exploration. The only self-citation is Kumar et al. (2022) for predictive entropy and active learning; PE is a standard formula and the citation is not load-bearing for the claimed result. The derivation chain is therefore self-contained, and the paper's own limitation statements do not reveal any circular step.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The method relies on a set of hand-chosen hyperparameters (number of demonstrations, strategy thresholds, perturbation count, rephrasing model) and on domain assumptions about uncertainty, rationale quality, and clustering. No new physical or conceptual entities are introduced.

free parameters (4)
  • Number of demonstrations k = 8 (GSM8K, Fallacy, EPR), 6 (StrategyQA)
    Set to match Auto-CoT conventions; the number of examples in each constructed demonstration directly affects performance and is not derived from the uncertainty model.
  • Uncertainty band boundaries for seven strategies = 0, mu-sigma, mu, mu+sigma, infinity combinations
    Table 1 defines Trivial, Very Easy, Easy, Moderate, Challenging, Hard, Very Hard using arbitrary cutoffs based on mean mu and standard deviation sigma of unlabeled uncertainty; the central comparison depends on these hand-chosen ranges.
  • Number of perturbation responses per question = 15 (10 via temperature/triggers, 5 via rephrasing)
    The implementation generates 15 responses per question; the entropy estimate depends on this sample size, chosen by the authors.
  • Rephrasing model = GPT4o
    All questions are rephrased by GPT4o regardless of the target LLM, introducing an external dependency that shapes the uncertainty estimates.
assumptions (5)
  • domain assumption Answer diversity under perturbations is a valid proxy for question-level uncertainty and informativeness for demonstration selection.
    Section 3.1 assumes that inconsistent answers across temperatures, trigger phrases, and rephrasings indicate model uncertainty, and that such uncertainty identifies useful demonstrations.
  • domain assumption The strategy with the lowest Temp-Perb uncertainty on the unlabeled set yields the highest accuracy.
    Section 5.4 and Figure 6 show an inverse correlation on the four datasets; ZEUS (LU) relies on this correlation holding, yet it is validated only on the same benchmarks used for evaluation.
  • domain assumption Zero-Shot-CoT generated rationales are accurate enough to serve as demonstrations.
    Demonstration construction in Section 3.3 uses Zero-Shot-CoT rationales without human verification, following Auto-CoT; errors in rationales may propagate.
  • domain assumption k-Means++ clustering on Sentence-BERT embeddings yields diverse, representative demonstration questions.
    Section 3.3 inherits Auto-CoT's clustering assumption; diversity is meant to mitigate poor rationales, but this is not separately validated.
  • domain assumption GPT4o rephrasing preserves the intent of the original question.
    Footnote 3 states rephrasing ensures the intent does not change; the uncertainty estimate for rephrasing depends on this.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Zero-shot Chain of Thought Prompting via Uncertainty-Guided Strategy Selection." pith.science (2026). https://pith.science/paper/3KVUH3QN

@misc{pith2026241200353,
  author       = {Pith},
  title        = {Pith review of: Enhancing Zero-shot Chain of Thought Prompting via Uncertainty-Guided Strategy Selection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3KVUH3QN}},
  note         = {Machine review of arXiv:2412.00353}
}
read the original abstract

Chain-of-thought (CoT) prompting has significantly enhanced the capability of large language models (LLMs) by structuring their reasoning processes. However, existing methods face critical limitations: handcrafted demonstrations require extensive human expertise, while trigger phrases are prone to inaccuracies. In this paper, we propose the Zero-shot Uncertainty-based Selection (ZEUS) method, a novel approach that improves CoT prompting by utilizing uncertainty estimates to select effective demonstrations without needing access to model parameters. Unlike traditional methods, ZEUS offers high sensitivity in distinguishing between helpful and ineffective questions, ensuring more precise and reliable selection. Our extensive evaluation shows that ZEUS consistently outperforms existing CoT strategies across four challenging reasoning benchmarks, demonstrating its robustness and scalability.

Figures

Figures reproduced from arXiv: 2412.00353 by the authors.

Figure 1
Figure 1. Overview of ZEUS: Uncertainty for a question qj is calculated using a pool of answers generated using various prompts, including trigger phrases, non-zero temperature-based decoding, and rephrasing of qj . Subse￾quently, questions with uncertainty within a certain range are selected and used for constructing demonstrations. results in more confident and consistent responses (Koehn, 2009). According to Wang et al. (2… view at source ↗
Figure 2
Figure 2. Mean and standard deviation of uncertainty values as error graph -specific statistics across models. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Probability density function of uncertainty [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Normalized values of accuracy for various selection strategies using multiple LLMs. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Sensitivity coefficient of confidence score [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Accuracy vs Temp-Perb Uncertainty trend across all selection strategies for GPT4o. allows us to categorize them into two broad groups: advanced models (GPT-4o, Phi3, GPT-3.5) and simpler models (Mistral, GPT-3 XL). This classi￾fication is based on observed performance …
Figure 7
Figure 7. Figure 7: Probability density function of uncertainty estimates of our method using [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Probability density function of uncertainty estimates of our method using [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Probability density function of uncertainty estimates of our method using [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Probability density function of uncertainty estimates of our method using [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Probability density function of uncertainty estimates of our method using [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Accuracy vs Temp-Perb Uncertainty trend across all selection strategies forGPT4o [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Accuracy vs Temp-Perb Uncertainty trend across all selection strategies forPhi3 [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Accuracy vs Temp-Perb Uncertainty trend across all selection strategies forMistral [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: Accuracy vs Temp-Perb Uncertainty trend across all selection strategies forGPT3.5 [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]
Figure 16
Figure 16. Figure 16: Accuracy vs Temp-Perb Uncertainty trend across all selection strategies forGPT3-XL [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 13 canonical work pages

  1. [1]

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

    arXiv 2024

  2. [2]

    David Arthur and Sergei Vassilvitskii. 2007. K-means++ the advantages of careful seeding. In Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms, pages 1027--1035

    work page 2007

  3. [3]

    Markus Bayer and Christian Reuter. 2024. Activellm: Large language model-based active learning for textual few-shot scenarios. arXiv preprint arXiv:2405.10808

    arXiv 2024

  4. [4]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

  5. [5]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

  6. [6]

    Shizhe Diao, Pengcheng Wang, Yong Lin, Rui Pan, Xiang Liu, and Tong Zhang. 2024. https://aclanthology.org/2024.acl-long.73 Active prompting with chain-of-thought for large language models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1330--1350, Bangkok, Thailand. Association fo...

    work page 2024

  7. [7]

    Shizhe Diao, Pengcheng Wang, Yong Lin, and Tong Zhang. 2023. Active prompting with chain-of-thought for large language models. arXiv preprint arXiv:2302.12246

    arXiv 2023

  8. [8]

    Guhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang. 2024. Towards revealing the mystery behind chain of thought: a theoretical perspective. Advances in Neural Information Processing Systems, 36

    2024

Show all 48 references
  1. [9]

    Yifan Fu, Xingquan Zhu, and Bin Li. 2013. A survey on instance selection for active learning. Knowledge and information systems, 35(2):249--283

    2013

  2. [10]

    Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Tnternational Conference on Machine Learning, pages 1050--1059

    2016

  3. [11]

    Xiang Gao, Jiaxin Zhang, Lalla Mouatadid, and Kamalika Das. 2024. Spuq: Perturbation-based uncertainty quantification for large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Paper...

    2024

  4. [12]

    Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. https://doi.org/10.1162/tacl_a_00370 Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies . Transactions of the Association for Computational Lingu...

    2021 doi

  5. [13]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. https://proceedings.mlr.press/v70/guo17a.html On calibration of modern neural networks . In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Resea...

    2017

  6. [14]

    Dan Hendrycks and Kevin Gimpel. 2016. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations

    2016

  7. [15]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  8. [16]

    Zhijing Jin, Abhinav Lalwani, Tejas Vaidhya, Xiaoyu Shen, Yiwen Ding, Zhiheng Lyu, Mrinmaya Sachan, Rada Mihalcea, and Bernhard Schoelkopf. 2022. Logical fallacy detection. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 7180--7198

    2022

  9. [17]

    Philipp Koehn. 2009. Statistical machine translation. Cambridge University Press

    2009

  10. [18]

    Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/8bb0d291acd4acf06ef112099c16f326-Paper-Conference.pdf Large language models are zero-shot reasoners . In Advances in Neural Inf...

    2022

  11. [19]

    Aobo Kong, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, Xin Zhou, Enzhi Wang, and Xiaohang Dong. 2024. https://doi.org/10.18653/v1/2024.naacl-long.228 Better zero-shot reasoning with role-play prompting . In Proceedings of the 2024 Conference of the North American C...

    2024 doi

  12. [20]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664

    2023 arXiv

  13. [21]

    Shanu Kumar, Sandipan Dandapat, and Monojit Choudhury. 2022. ” diversity and uncertainty in moderation” are the key to data selection for multilingual few-shot transfer. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1042--1055

    2022

  14. [22]

    Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. https://proceedings.neurips.cc/paper_files/paper/2017/file/9ef2ed4b7fd2c810847ffa5fa85bce38-Paper.pdf Simple and scalable predictive uncertainty estimation using deep ensembles . In Advances in Neural Info...

    2017

  15. [23]

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110

    2022 arXiv

  16. [24]

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. 2023. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305.19118

    2023 arXiv

  17. [25]

    OpenAI. 2024. Introducing gpt-4o. https://openai.com/index/hello-gpt-4o/. Accessed: 2024-09-16

    2024

  18. [26]

    Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. 2019. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information pr...

    2019

  19. [27]

    Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446

    2021 arXiv

  20. [28]

    Nils Reimers and Iryna Gurevych. 2019. https://doi.org/10.18653/v1/D19-1410 Sentence- BERT : Sentence embeddings using S iamese BERT -networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference...

    2019 doi

  21. [29]

    Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of nlp models with checklist. arXiv preprint arXiv:2005.04118

    2020 arXiv

  22. [30]

    Guy Rotman and Roi Reichart. 2022. Multi-task active learning for pre-trained transformer-based models. Transactions of the Association for Computational Linguistics, 10:1209--1228

    2022

  23. [31]

    Burr Settles and Mark Craven. 2008. An analysis of active learning strategies for sequence labeling tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 1070--1079

    2008

  24. [32]

    Kashun Shum, Shizhe Diao, and Tong Zhang. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.811 Automatic prompt augmentation and selection with chain-of-thought from labeled data . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12113--1213...

    2023 doi

  25. [33]

    Damien Sileo and Antoine Lernould. 2023. Mindgames: Targeting theory of mind in large language models with dynamic epistemic modal logic. arXiv preprint arXiv:2305.03353

    2023 arXiv

  26. [34]

    Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239

    2022 arXiv

  27. [35]

    Christian Tomani, Kamalika Chaudhuri, Ivan Evtimov, Daniel Cremers, and Mark Ibrahim. 2024. Uncertainty-based abstention in llms improves safety and reduces hallucinations. arXiv preprint arXiv:2404.10960

    2024 arXiv

  28. [36]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  29. [37]

    Joost Van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. 2020. https://proceedings.mlr.press/v119/van-amersfoort20a.html Uncertainty estimation using a single deep deterministic neural network . In Proceedings of the 37th International Conference on Machine Learning, vo...

    2020

  30. [38]

    Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Akim Tsvigun, Daniil Vasilev, Rui Xing, Abdelrahman Boda Sadallah, Lyudmila Rvanova, Sergey Petrakov, Alexander Panchenko, et al. 2024. Benchmarking uncertainty quantification methods for large language models with lm-polygr...

    2024 arXiv

  31. [39]

    Xingchen Wan, Ruoxi Sun, Hanjun Dai, Sercan Arik, and Tomas Pfister. 2023. https://doi.org/10.18653/v1/2023.findings-acl.216 Better zero-shot reasoning with self-adaptive prompting . In Findings of the Association for Computational Linguistics: ACL 2023, pages 3493--3514, Toro...

    2023 doi

  32. [40]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations

    2022

  33. [41]

    Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. 2023. Unleashing cognitive synergy in large language models: A task-solving agent through multi-persona selfcollaboration. arXiv preprint arXiv:2307.05300, 1(2):3

    2023 arXiv

  34. [42]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824--24837

    2022

  35. [43]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601

    2023 arXiv

  36. [44]

    Zhangyue Yin, Qiushi Sun, Cheng Chang, Qipeng Guo, Junqi Dai, Xuanjing Huang, and Xipeng Qiu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.936 Exchange-of-thought: Enhancing large language model capabilities through cross-model communication . In Proceedings of the 2023 C...

    2023 doi

  37. [45]

    Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic chain of thought prompting in large language models. In The Eleventh International Conference on Learning Representations

    2022

  38. [46]

    Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Yongfeng Huang, Ruyi Gan, Jiaxing Zhang, and Yujiu Yang. 2023. https://doi.org/10.18653/v1/2023.acl-long.245 Solving math word problems via cooperative reasoning induced language models . In Proceedings of the 61st Annual Meeti...

    2023 doi

  39. [47]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  40. [48]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.