Pith. sign in

REVIEW 4 major objections 5 minor 48 references

Self-Training Large Language Models with Confident Reasoning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning an LLM to prefer reasoning paths it itself judges true, rather than answers it repeats most often, improves accuracy and reduces reasoning errors on six benchmarks.

desk verdict A practical self-training recipe with real promise, but the key experiment that would separate reasoning confidence from other changes is missing. read the letter →

arxiv 2505.17454 v1 pith:3PXB3R2Z submitted 2025-05-23 cs.LG cs.CL

Hyosoon Jang , Yunhui Jang , Sungjae Lee , Jungseul Ok , Sungsoo Ahn This is my paper
classification cs.LGcs.CL
keywords reasoning-levelconfidenceself-trainingP(True)directpreferenceoptimizationchain-of-thoughtself-consistencycalibrationLLMreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

To improve chain-of-thought reasoning without human labels, the paper proposes training an LLM to prefer its own reasoning paths that the model itself judges to be correct. Existing confidence-based self-training relies on majority-voted answers, which can reward a wrong chain of thought that happens to end in a correct answer. The paper's method, CORE-PO, measures reasoning-level confidence with P(True) and fine-tunes the model with direct preference optimization to favor high-confidence reasoning. The authors claim this improves answer accuracy and reduces reasoning errors on four in-distribution and two out-of-distribution benchmarks, beating answer-confidence-based self-training.

What carries the argument

The load-bearing object is the reasoning-level confidence score $C_\theta(r|x)$, defined as $P(\text{True})$: the probability that the LLM returns “true” when prompted to judge whether its reasoning path $r$ is correct for question $x$. It comes in two variants, a monolithic one that asks for a single verdict on the whole chain and a statement-wise one that averages per-step verdicts, and it is multiplied by the answer-level confidence $C_\theta(a|x,r)$ to form the preference reward. That reward feeds an online direct preference optimization update that raises the likelihood of high-confidence reasoning–answer sequences and lowers the likelihood of low-confidence ones, which is the mechanism that transfers the model's self-assessment into a better policy.

What would settle it

On held-out questions where the base model's P(True) is known to be overconfident, measure P(True) and an external judge's reasoning-accuracy score before and after CORE-PO training: if P(True) rises while reasoning accuracy stays flat or falls, the training signal is being gamed rather than improved. A cleaner control is to run the same DPO loop with the P(True) scores randomly shuffled or reversed; answer accuracy that still improves under that control would show the gain does not come from reasoning-confidence information.

Watch

Extended reading notes

Core claim

Stated on the paper's own terms, the discovery is that answer-level confidence, estimated by majority voting over sampled answers, is an unreliable proxy for reasoning quality: a chain of thought can be wrong even when its final answer is confidently repeated. Reasoning-level confidence, obtained by asking the model whether its reasoning is true, tracks reasoning accuracy more closely and separates sound chains of thought from faulty ones. When the model is fine-tuned to prefer reasoning–answer pairs with high combined confidence $C_\theta(r|x)C_\theta(a|x,r)$ using online direct preference optimization, it produces fewer erroneous reasoning steps and more accurate answers than models trained to prefer high-confidence answers alone, both on in-distribution tasks and on held-out tasks from a different domain.

Load-bearing premise

The method rests on the model's P(True) self-evaluation being a trustworthy, non-gameable measure of reasoning quality; if that confidence is miscalibrated or becomes inflated during training, the model can be trained to prefer incorrect reasoning that merely sounds confident.

Editorial extensions

If this is right

  • Self-training with only unlabeled questions can improve chain-of-thought reasoning when the training signal is reasoning quality rather than answer agreement.
  • The accuracy gains appear on out-of-distribution tasks such as code execution and arithmetic puzzles, suggesting the preference signal teaches a transferable disposition toward careful reasoning.
  • Even in the supervised setting where ground-truth answers are available, adding reasoning-level confidence to an answer-accuracy reward improves both answer accuracy and reasoning accuracy.
  • Because both the monolithic and the statement-wise P(True) variants improve over the base model, the benefit is attributed to the reasoning-confidence principle itself, not to one particular confidence implementation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step the paper leaves implicit is monitoring calibration after DPO: if P(True) rises faster than externally judged reasoning accuracy, the preference signal eventually becomes self-confirming, and a calibration penalty or held-out truthfulness filter would be needed.
  • Statement-wise P(True) is effectively a parameter-free process reward; distilling it into a learned process reward model and using it for tree search at inference is a direct extension that could combine CORE-PO's signal with search-time scaling.
  • Because the paper's gains are largest on multiple-choice benchmarks, where wrong reasoning can reach the right answer by chance, the method should transfer especially well to tasks with many plausible distractors, such as medical or legal multiple-choice QA.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes CORE-PO, a self-training method for LLM reasoning that uses reasoning-level confidence, estimated via P(True), as the preference signal in DPO instead of answer-level majority-vote confidence. The authors motivate the approach with observational evidence that answer-level confidence can be high even when the reasoning path is incorrect, and that reasoning-level confidence correlates better with reasoning accuracy. They report accuracy improvements over SR-PO and SC-PO on four in-distribution benchmarks (GSM8K, ARC-Challenge, GPQA, MATH) and two out-of-distribution benchmarks (CRUXEval, Game of 24), using Llama3.1-8B-Instruct and Qwen2.5-7B-Instruct. They also present ablations comparing monolithic versus statement-wise P(True) and a ground-truth fine-tuning variant that adds reasoning confidence to answer-accuracy rewards.

Significance. If the central claim holds, the paper makes a useful contribution by identifying a limitation of answer-level confidence in self-training and by providing a simple, DPO-based recipe that uses reasoning-level confidence. The empirical coverage is reasonable: two base models, four in-distribution and two out-of-distribution benchmarks, plus ablations for the confidence estimator and a ground-truth fine-tuning variant. The paper is also candid in its Limitations section about overconfidence risk, language scope, and model scale. However, the main evidence does not yet isolate the proposed reasoning-level confidence signal from other changes in the training protocol, and the reliance on P(True) without calibration checks leaves the mechanism under-supported.

major comments (4)
  1. [Section 3.2, Eq. (1); Tables 1 and 2; Appendix B.4] The comparison between CORE-PO and SC-PO changes two variables at once: the confidence estimator (P(True) product vs. majority-vote answer confidence) and the DPO protocol (online, N=5, no score-gap threshold, fixed reference model vs. offline, N=8, score-gap >=3, updated reference model). The base-model rows in Tables 1 and 2 show that P(True|r,a) inference-time selection already outperforms SC selection on several benchmarks (e.g., Qwen GSM8K 93.2 vs. 92.0; GPQA 34.1 vs. 33.7), so part of the reported gain may come from the inference-time selector rather than from training on reasoning-level confidence. The only ablation that adds reasoning confidence (Table 6) is in a ground-truth fine-tuning setting, not in the self-training setting. Please add a controlled comparison that isolates the confidence estimator from the training protocol, for example SC-PO with answer-level P(True) or CORE-PO with majority-vote answer confidence.
  2. [Appendix B.4] The authors select the checkpoint with the highest accuracy on the ARC-Challenge validation split, and ARC-Challenge is also one of the four in-distribution evaluation benchmarks in Tables 1 and 2. This selection procedure can inflate the reported ARC-Challenge results and any other numbers reported for the selected checkpoint. Please report results for a checkpoint selected on a validation set disjoint from all evaluated benchmarks, or report results across all checkpoints.
  3. [Tables 1, 2, and 4] All results are from a single run with no error bars, standard deviations, or significance tests. Several reported differences are small (e.g., GSM8K 93.0 vs. 93.5 for Qwen; ARC-Challenge 92.0 vs. 92.8), and without variance estimates it is unclear whether these differences are reliable. Please provide multiple seeds and report means and standard deviations, or otherwise justify that the differences are not within run-to-run noise.
  4. [Section 3.2; Table 3; Limitations] The method uses P(True) both as the training signal and as the inference-time selection score, and Table 3 shows that confidence increases after training. However, the paper does not report calibration or precision-recall analysis of P(True) before and after training, and the Limitations section explicitly acknowledges that the confidence measure 'can suffer from overconfidence.' Without such analysis, the observed confidence increase could reflect miscalibrated inflation rather than improved reasoning, and the reported Reasoning Acc. gains under P(True|r,a) selection may be partly a selection artifact. Please report calibration curves or reliability diagrams for P(True) and show that the post-training confidence increase coincides with improved external reasoning quality rather than only with higher self-assigned scores.
minor comments (5)
  1. [Table 2] The column header 'QPGA ext' is a typo and should read 'GPQA ext'.
  2. [Appendix B.3] CRUXEval is attributed to 'Guo et al. (2024)' in the data-statistics text, but the reference list entry is by Gu et al. (2024); please correct the citation.
  3. [Table 4 and Section 4.1] The benchmark name is written inconsistently as 'Cruxout', 'CRUXout', and 'CRUXEval'; please use one consistent name.
  4. [Figure 2] The x-axis label 'Top-N%' is undefined; please specify what N denotes and how the curves are computed.
  5. [Appendix B.4] The learning-rate search is reported as {1e-6, 5e-6} but the final chosen learning rate for each model is not stated; please report the selected value.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: central claims are evaluated against external benchmarks and an external reasoning oracle; the self-training loop is an acknowledged robustness risk, not a derivational circle.

full rationale

No circularity in the derivation sense. The paper's central empirical claim—that DPO training toward higher P(True)-based reasoning confidence improves final-answer accuracy and reduces reasoning errors—is checked against external benchmarks (GSM8K, ARC-Challenge, GPQA, MATH, CRUXEval, Game-of-24) and an external oracle (o4-mini) for reasoning correctness, not against the training objective itself. The preference score in Sec. 3.2 is optimized by DPO, but the reported accuracy numbers are ground-truth accuracies; no reported result is defined in terms of the quantity being optimized. The motivation (Observations 1-2) is an empirical correlation measured with an external judge, not an assumption equivalent to the conclusion. The acknowledged limitation—P(True) confidence may be miscalibrated and can reinforce incorrect reasoning—is a robustness or correctness risk of self-training, not a circular derivation: the method would fail if the signal were uninformative, but its success is not guaranteed by construction. No load-bearing self-citation, imported uniqueness theorem, or ansatz smuggled by citation appears; P(True) is adopted from Kadavath et al. (2022) and used as a stated external measure. The paper is self-contained against external benchmarks, so the appropriate finding is no circularity.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical quantities or entities beyond the already-known P(True) confidence measure. The load-bearing assumptions are about the reliability of self-evaluation and the external oracle used to measure reasoning quality. Hyperparameters such as beta, sample counts, and prompt example counts are hand-chosen and not matched across baselines.

free parameters (6)
  • DPO beta = 0.1
    Hand-chosen DPO temperature that controls the strength of preference optimization; central results depend on this hyperparameter.
  • Number of sampled outputs per question for training = 5 (CORE-PO), 8 (SC-PO baseline)
    Hand-chosen; confidence estimates and preference-pair construction depend on the number of samples, and the comparison is not matched across methods.
  • Number of random reasoning examples in P(True) prompt = 4
    Hand-chosen; the P(True) confidence prompt includes M random reasoning examples for calibration, and the value is arbitrary.
  • LoRA rank and alpha = rank 128, alpha 256
    Hand-chosen low-rank adaptation hyperparameters for fine-tuning.
  • Learning rate
    Searched over {1e-6, 5e-6} and selected by ARC-Challenge validation accuracy; the final value is not reported and ARC-Challenge is also an evaluation benchmark.
  • Sampling temperature and top-p = training T=1.0, top-p=0.9; inference T=0.7, top-p=0.9
    Hand-chosen generation hyperparameters that affect the diversity of sampled reasoning paths and confidence estimates.
assumptions (4)
  • domain assumption P(True) self-evaluation probability correlates with reasoning correctness and remains non-gameable under DPO training.
    Section 3.2 uses P(True) as the reward signal; the Limitations section explicitly warns about overconfidence and calibration issues.
  • domain assumption Majority voting over sampled answers is a valid estimate of answer-level confidence.
    Section 2.2 adopts self-consistency scores as answer confidence, following Wang et al. 2023.
  • domain assumption The external model o4-mini can correctly judge whether a reasoning path is correct.
    Section 4.2 and Appendix B.2 use o4-mini as an oracle for reasoning-level accuracy; no human validation of these judgments is provided.
  • standard math Online DPO with pairs generated by the current model improves expected preference in the intended direction.
    Section 3.2 relies on the standard DPO objective from Guo et al. 2024.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Self-Training Large Language Models with Confident Reasoning." pith.science (2026). https://pith.science/paper/3PXB3R2Z

@misc{pith2026250517454,
  author       = {Pith},
  title        = {Pith review of: Self-Training Large Language Models with Confident Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3PXB3R2Z}},
  note         = {Machine review of arXiv:2505.17454}
}
read the original abstract

Large language models (LLMs) have shown impressive performance by generating reasoning paths before final answers, but learning such a reasoning path requires costly human supervision. To address this issue, recent studies have explored self-training methods that improve reasoning capabilities using pseudo-labels generated by the LLMs themselves. Among these, confidence-based self-training fine-tunes LLMs to prefer reasoning paths with high-confidence answers, where confidence is estimated via majority voting. However, such methods exclusively focus on the quality of the final answer and may ignore the quality of the reasoning paths, as even an incorrect reasoning path leads to a correct answer by chance. Instead, we advocate the use of reasoning-level confidence to identify high-quality reasoning paths for self-training, supported by our empirical observations. We then propose a new self-training method, CORE-PO, that fine-tunes LLMs to prefer high-COnfidence REasoning paths through Policy Optimization. Our experiments show that CORE-PO improves the accuracy of outputs on four in-distribution and two out-of-distribution benchmarks, compared to existing self-training methods.

Figures

Figures reproduced from arXiv: 2505.17454 by the authors.

Figure 1
Figure 1. Limitations in existing confidence-based self-training methods. Existing self-training methods prefer reasoning paths associated with a high-confidence answer (a), estimated via majority voting. However, they fail to capture the errors in their third reasoning path, even though the answer is correct. As a result, they can degrade the reasoning capabilities of the LLM, e.g., preferring “(b)-(d) are boiling points” ca… view at source ↗
Figure 3
Figure 3. Overview of CORE-PO. The LLM Mθ generates multiple outputs, each consisting of a reasoning and an answer s = [r, a] for a given question. Next, we measure the reasoning-level confidence Cθ(r|x) = P(True) for each reasoning path. Then, we fine-tune the LLM to prefer high-confidence reasoning paths. answer induced from this reasoning path. In this paper, we are particularly interested in the model’s confidence in its … view at source ↗
Figure 2
Figure 2. Answer vs. reasoning accuracies. We use Llama3.1-8B-Instruct (Meta AI, 2024). Reasoning-level accuracy coincides more closely with reasoning-level confidence than with answer-level confidence. To support our claim, we conduct an observa￾tional experiment using multiple-choice questions in the GPQA dataset (Rein et al., 2023). We gener￾ate reasoning paths and assign two types of confi￾dence scores to each path: one b… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 15 canonical work pages

  1. [1]

    Evan Becker and Stefano Soatto. 2024. https://arxiv.org/abs/2406.03441 Cycles of thought: Measuring llm confidence through stable explanations . Preprint, arXiv:2406.03441

    arXiv 2024

  2. [2]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877--1901

    work page 2020

  3. [3]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    arXiv 2018

  4. [4]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . Preprint, arXiv:2110.14168

    arXiv 2021

  5. [5]

    DeepSeek-AI. 2025. https://arxiv.org/abs/2501.12948 Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning . Preprint, arXiv:2501.12948

    arXiv 2025

  6. [6]

    Alex Gu, Baptiste Roziere, Hugh James Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida Wang. 2024. https://proceedings.mlr.press/v235/gu24c.html CRUXE val: A benchmark for code reasoning, understanding and execution . In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, p...

    work page 2024

  7. [7]

    Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Marc Sun, and Benjamin Bossan. 2022. Accelerate: Training and inference at scale made simple, efficient and adaptable. https://github.com/huggingface/accelerate

    2022

  8. [8]

    Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, and 1 others. 2024. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792

    arXiv 2024

Show all 48 references
  1. [9]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874

    2021 arXiv

  2. [10]

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. https://openreview.net/forum?id=rygGQyrFvH The curious case of neural text degeneration . In International Conference on Learning Representations

    2020

  3. [11]

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lo RA : Low-rank adaptation of large language models . In International Conference on Learning Representations

    2022

  4. [12]

    Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.67 Large language models can self-improve . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages...

    2023 doi

  5. [13]

    Fangkai Jiao, Chengwei Qin, Zhengyuan Liu, Nancy Chen, and Shafiq Joty. 2024. Learning planning-based reasoning by trajectories collection and process reward synthesizing. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 334--350

    2024

  6. [14]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav For...

    2022 arXiv

  7. [15]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199--22213

    2022

  8. [16]

    Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations

  9. [17]

    Abhishek Kumar, Robert Morabito, Sanzhar Umbet, Jad Kabbara, and Ali Emami. 2024. https://doi.org/10.18653/v1/2024.acl-long.20 Confidence under the hood: An investigation into the confidence-probability alignment in large language models . In Proceedings of the 62nd Annual Mee...

    2024 doi

  10. [18]

    Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, and 1 others. 2023. Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702

    2023 arXiv

  11. [19]

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. https://openreview.net/forum?id=v8L0pN6EOi Let's verify step by step . In International Conference on Learning Representations (ICLR)

    2024

  12. [20]

    Nick Lile. 2025. Game of 24: A benchmark for arithmetic reasoning. https://huggingface.co/datasets/nlile/24-game. Accessed: 2025-05-07

    2025

  13. [21]

    Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.578 Contextualized sequence likelihood: Enhanced confidence scores for natural language generation . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Pr...

    2024 doi

  14. [22]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023 a . https://open...

    2023

  15. [23]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, and 1 others. 2023 b . Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534--46594

    2023

  16. [24]

    Meta AI . 2024. Introducing llama 3.1: Our most capable models to date. https://ai.meta.com/blog/meta-llama-3-1/. Accessed: 2025-05-19

    2024

  17. [25]

    OpenAI. 2025. Introducing openai o3 and o4-mini. https://openai.com/index/introducing-o3-and-o4-mini/. Accessed: 2025-05-15

    2025

  18. [26]

    Archiki Prasad, Weizhe Yuan, Richard Yuanzhe Pang, Jing Xu, Maryam Fazel-Zarandi, Mohit Bansal, Sainbayar Sukhbaatar, Jason Weston, and Jane Yu. 2024. Self-consistency preference optimization. arXiv preprint arXiv:2411.04109

    2024 arXiv

  19. [27]

    Qwen Team . 2024. Qwen2.5: A party of foundation models. https://huggingface.co/Qwen/Qwen2.5-7B-Instruct. Accessed: 2025-05-07

    2024

  20. [28]

    Leonardo Ranaldi and Andr \`e Freitas. 2024. Self-refine instruction-tuning for aligning reasoning in language models. arXiv preprint arXiv:2405.00402

    2024 arXiv

  21. [29]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2023. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022

    2023 arXiv

  22. [30]

    Amir Taubenfeld, Tom Sheffer, Eran Ofek, Amir Feder, Ariel Goldstein, Zorik Gekhman, and Gal Yona. 2025. Confidence improves self-consistency in llms. arXiv preprint arXiv:2502.06233

    2025

  23. [31]

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.330 Just ask for calibration: Strategies for eliciting calibrated confidence scores from language mod...

    2023 doi

  24. [32]

    Luong Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. 2024. https://doi.org/10.18653/v1/2024.acl-long.410 R e FT : Reasoning with reinforced fine-tuning . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1...

    2024 doi

  25. [33]

    Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. 2020. Trl: Transformer reinforcement learning. https://github.com/huggingface/trl

    2020

  26. [34]

    Guangya Wan, Yuqi Wu, Jie Chen, and Sheng Li. 2025. https://aclanthology.org/2025.naacl-long.184/ Reasoning aware self-consistency: Leveraging reasoning paths for efficient LLM sampling . In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Assoc...

    2025

  27. [35]

    Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. https://openreview.net/forum?id=1PL1NIMMrw Self-consistency improves chain of thought reasoning in language models . In The Eleventh International Conferenc...

    2023

  28. [36]

    Chi, Quoc V Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. https://openreview.net/forum?id=_VjQlMeSB_J Chain of thought prompting elicits reasoning in large language models . In Advances in Neural Information Proc...

    2022

  29. [37]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and ...

    2020

  30. [38]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809--11822

    2023

  31. [39]

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. http://dblp.uni-trier.de/db/conf/icml/icml2024.html\#YuanPCLSXW24 Self-rewarding language models . In Proceedings of the 41st International Conference on Machine L...

    2024

  32. [40]

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. https://openreview.net/forum?id=_3ELRdg2sgI ST ar: Bootstrapping reasoning with reasoning . In Advances in Neural Information Processing Systems

    2022

  33. [41]

    Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. 2024 a . https://openreview.net/forum?id=8rcFOqEud5 Re ST - MCTS *: LLM self-training via process reward guided tree search . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  34. [42]

    Xiaoying Zhang, Baolin Peng, Ye Tian, Jingyan Zhou, Lifeng Jin, Linfeng Song, Haitao Mi, and Helen Meng. 2024 b . https://doi.org/10.18653/v1/2024.acl-long.107 Self-alignment for factuality: Mitigating hallucinations in LLM s via self-evaluation . In Proceedings of the 62nd An...

    2024 doi

  35. [43]

    Xuan Zhang, Chao Du, Tianyu Pang, Qian Liu, Wei Gao, and Min Lin. 2024 c . https://openreview.net/forum?id=2cczgOfMP4 Chain of preference optimization: Improving chain-of-thought reasoning in LLM s . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  36. [44]

    Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023. https://openreview.net/forum?id=5NTt8GFjUHkr Automatic chain of thought prompting in large language models . In The Eleventh International Conference on Learning Representations

    2023

  37. [45]

    Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. 2025. https://arxiv.org/abs/2505.03335 Absolute zero: Reinforced self-play reasoning with zero data . Preprint, arXiv:2505.03335

    2025 arXiv

  38. [46]

    Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuekai Zhu, Biqing Qi, Youbang Sun, Ganqu Cui, Ning Ding, and Bowen Zhou. 2025. https://arxiv.org/abs/2504.16084 Ttrl: Test-time reinforcement learning . Preprint, arXiv:2504.16084

    2025 arXiv

  39. [47]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  40. [48]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.