Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

Post-Training Large Language Models via Reinforcement Learning from Self-Feedback

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A post-training loop that ranks a model's own chain-of-thought answers by their confidence can improve both calibration and step-by-step reasoning without any external labels.

desk verdict RLSF is a plausible label-free post-training trick with honest limitations, but the calibration claim is overstated and the experiments need error bars and a control that destroys the confidence signal. read the letter →

arxiv 2507.21931 v1 pith:PRXD77GH submitted 2025-07-29 cs.CL cs.AI

Carel van Niekerk , Renato Vukovic , Benjamin Matthias Ruppik , Hsien-chin Lin , Milica Gašić This is my paper
classification cs.CLcs.AI
keywords reinforcementlearningfromself-feedbackconfidencecalibrationexpectederrorchain-of-thoughtdecodingpreferenceoptimizationintrinsicrewardarithmeticreasoningmultiple-choicequestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reinforcement Learning from Self-Feedback (RLSF) is a post-training stage that uses a frozen LLM's own answer confidence as the only reward signal. For each prompt, the model generates several chain-of-thought completions, ranks them by a span-level confidence score, and turns that ranking into preferences for reward-model training followed by policy optimization. The paper claims that this loop simultaneously restores well-behaved calibration and strengthens step-by-step reasoning, improving accuracy on arithmetic word problems and multiple-choice question answering without human labels, gold answers, or externally curated rewards. If true, it means a model can improve its own reliability from nothing but its own uncertainty.

What carries the argument

The load-bearing object is the probability-disparity confidence score of Eq. (1): for each position in the answer span, it averages the difference between the model's top token probability and its second-best token probability. A high disparity means the model commits to the answer token with little competition, which the paper interprets as calibrated confidence. This score is computed on the answer span extracted from each chain-of-thought hypothesis, and hypotheses are then ordered by disparity to produce synthetic preferences. Those preferences can train a Bradley-Terry reward model for PPO or be plugged directly into DPO. The rest of the pipeline is standard RLHF machinery: the reward model initializes the critic, and PPO optimizes the policy with a KL penalty.

What would settle it

Give a base model a synthetic calibration corruption that makes it confidently wrong on most of its high-confidence answers, run RLSF on its own preferences, and compare expected calibration error and accuracy against the uncorrupted base model; a rise in ECE or a drop in accuracy after RLSF would falsify the claim that self-generated confidence preferences restore calibration.

Watch

Extended reading notes

Core claim

The central claim is that the gap between the top token probability and the second-best token probability over an answer span, computed as the probability disparity in Eq. (1), is a usable intrinsic reward for post-training. Starting from a prompt, the method decodes $K$ candidate chains of thought from the top-$K$ first tokens, appends "So the answer is" to locate each answer span, scores each span by probability disparity, and builds a preference ranking. That ranking trains a Bradley-Terry reward model, which then guides PPO, or directly updates the policy via DPO. The paper reports that on MultiArith, GSM8K, CommonsenseQA, and ARC Easy, the resulting policy is better calibrated and not worse in accuracy, and that the resulting reward model ranks answers on RewardBench competitively using only prompts. A secondary claim is that the PPO variant dominates the DPO variant, indicating that online reinforcement learning matters for incorporating intrinsic motivation.

Load-bearing premise

The method assumes that in a well-calibrated model the confidence of an answer tracks the presence of reasoning and therefore the accuracy of that answer.

Editorial extensions

If this is right

  • Post-training can be made label-free for calibration and reasoning: the preference signal is entirely self-generated, so RLSF can be inserted as an extra stage without human annotation.
  • Because the reward model is trained from prompts alone, RLSF doubles as a method for extracting an evaluation signal from a frozen LLM, giving a cheaper route to reward models on new task families.
  • The gap between PPO and DPO results suggests that online reward optimization, not just supervised preference fitting, is doing the work in intrinsic-reward post-training.
  • Inference cost is unaffected by RLSF; the expensive chain-of-thought decoding happens only during training, so the trained model stays cheap to deploy.
  • The XSTest result shows that the same mechanism can amplify an existing safety bias when applied to safety-related data, so RLSF is not bias-neutral in all settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable extension the paper does not run: apply RLSF to a model whose calibration is deliberately corrupted, such as by fine-tuning it to be overconfident on wrong answers, and measure whether the self-feedback loop repairs or worsens the corruption.
  • The confidence-as-reward mechanism could transfer to long-horizon agent tasks by scoring each subgoal completion's answer span; the paper notes intrinsic rewards can replace or augment extrinsic ones, but does not demonstrate this.
  • Because the method depends on locating answer spans, it is currently restricted to tasks with extractable final answers; a learned span proposer would generalize it to open-ended generation, at the cost of reintroducing external machinery.
  • A broader implication is that calibration itself can be treated as a training signal, not just an evaluation metric; if that holds, then expected-calibration-error improvements are not merely diagnostic but causal to reasoning gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces RLSF, a post-training stage in which an LLM generates K chain-of-thought candidates, scores each by the probability disparity of its answer span (Eq. 1), ranks the candidates to build synthetic preferences, trains a Bradley-Terry reward model, and optimizes the original policy with PPO or DPO. Experiments on MultiArith, GSM8K, CommonsenseQA, ARC Easy, and RewardBench compare accuracy and Expected Calibration Error against greedy decoding, CoT decoding, and a URM-based RLHF baseline. The paper claims that RLSF improves calibration and accuracy without external labels, and that the induced reward model is competitive on RewardBench.

Significance. If the central claims hold, RLSF would be a label-free post-training step that uses a model's own confidence as an intrinsic reward, which is relevant to both calibration and reasoning research. The paper has concrete strengths: it evaluates on two task families, reports hyperparameters in Appendix B, includes a RewardBench evaluation of the learned reward model, and attempts a bias analysis with XSTest and AlpacaEval. However, the headline calibration claim is not consistently supported by the reported tables, and the mechanism depends on an unmeasured confidence-correctness correlation in miscalibrated base models. The idea is potentially useful, but the current evidence is insufficient to establish the claims as stated.

major comments (4)
  1. [§5.2.1, Table 2] The abstract and Section 1 claim that RLSF improves calibration on all examined tasks, but the Gemma 2 rows of Table 2 contradict this: RLSF(PPO) greedy has ECE 7.81 on MultiArith versus 7.43 for the greedy base model, and 12.54 on GSM8K versus 12.24. The text's statement that RLSF achieves better calibration than most methods except CoT decoding does not hold against these greedy baselines. Please clarify the intended comparison protocol and either qualify the calibration claim or identify the configurations that support it.
  2. [§3, Eq. (1)] The entire preference signal is the disparity c in Eq. (1), and Section 3 explicitly conditions on 'if the model is well calibrated.' The models used are described and shown to be miscalibrated (e.g., Phi-2 with ECE 25.12 in Table 3), yet the paper never measures the correlation between c and answer correctness on the K=10 candidate pools from which preferences are built. Without this measurement, or a control that destroys the confidence ordering (e.g., random or inverted preferences), the reported calibration improvements cannot be attributed to confidence-ranked self-feedback rather than to CoT-style data exposure, reward-model initialization, or PPO regularization. The acknowledgment in Section 8 that 'some basic level of reasoning needs to be observed' does not fill this gap.
  3. [Tables 2-5, §5.2] All results are single-run point estimates with no standard errors or significance tests. Many claimed improvements are small relative to plausible run-to-run variation: for example, Gemma 2 accuracy on MultiArith is 98.83 for RLSF(PPO) versus 98.12 for greedy, and on ARC Easy the RLSF(PPO) ECE of 5.12 is worse than the CoT(10) baseline ECE of 3.03. Please report multiple seeds or confidence intervals, at least for the headline accuracy and calibration numbers, so that the comparative claims can be assessed.
  4. [§7, Table 2] The conclusion that 'RLSF combined with PPO is superior to DPO' is too broad. On GSM8K with QWEN 2.5 7B, RLSF(DPO) has ECE 38.15 versus RLSF(PPO) 41.92, so DPO gives better calibration while PPO gives better accuracy. The superiority claim should be metric- and task-specific.
minor comments (5)
  1. [Abstract] The word 'warrents' should be 'warrants'.
  2. [Table 3 caption] The caption says 'Gemma 2 9B IT' while Section 4.2 describes Gemma 2 as the 2 billion parameter instruction-tuned variant; also 'Gemma 2Greedy' is missing a space. Please correct the model size and typography.
  3. [§5.1, Table 1] Calling an accuracy of 81.43 (Gemma 2 RLSF) 'competitive' against URM's 97.00 may overstate the result; please provide a clearer basis for the comparison, such as the fact that RLSF uses only prompts.
  4. [§3.1, Eq. (1)] The phrase 'K-top probability tokens' should be 'top-K tokens,' and the maximum over 'w ≠ arg max π' should define how ties are handled.
  5. [Appendix B] The hyperparameters are useful, but no code release or random seed information is provided; adding these would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the self-referential training signal is explicit and the central claims are evaluated against external labels.

full rationale

The derivation chain is not circular under the provided definitions. The preference signal in Eq. (1) is the base model's span-level probability disparity, and the paper explicitly frames the method as a self-bootstrap: 'uses the model's own confidence as an intrinsic reward' and 'rank the traces accordingly.' The central empirical claims—calibration (ECE) and accuracy on MultiArith, GSM8K, CommonsenseQA, ARC Easy, plus RewardBench accuracy—are measured against external labels (gold answers and human preferences) that are not used to construct the preferences. There is no fitted parameter that is later renamed as a prediction; the learned reward model is externally validated on RewardBench human preference pairs. The load-bearing citation is to Wang and Zhou (2024) for the conditional observation that confidence correlates with accuracy in well-calibrated models; that is an external premise, not a self-citation, and the paper's own experiments can in principle falsify it. Self-citations (Heck et al. 2020; Wesselmann et al. 2018) are peripheral and not load-bearing. Whether the premise actually holds for the miscalibrated base models is a correctness/validity concern, not circularity. The paper's own mixed ECE results (e.g., Gemma 2 RLSF-PPO ECE 7.81 vs. base 7.43 on MultiArith) show that the claimed calibration improvements are not forced by construction. The self-referential nature of the training signal is the method itself, not a hidden equivalence between input and output.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The method relies on a small number of hand-chosen hyperparameters and on the domain assumption that the model's own confidence is a meaningful proxy for answer quality. No new physical or architectural entities are introduced. The central claims rest on the reliability of the self-generated preference signal and on the external benchmarks used for evaluation.

free parameters (8)
  • K (number of CoT candidates) = 10
    Chosen in Section 4.4; no ablation reported, directly controls preference dataset size and diversity.
  • PPO discount factor gamma = 0.98
    Selected after comparing with gamma=1.0 in Appendix C (Table 10); single run, no significance test.
  • DPO beta = 0.2
    Used for all RLSF-DPO and AfD runs (Table 8); no ablation reported.
  • PPO KL coefficient beta = 0.05
    Table 7; hand-set, standard or chosen for stability.
  • Exploration temperature = 0.7
    Table 7; affects PPO exploration and response diversity.
  • GAE lambda = 0.95
    Table 7; hand-set for advantage estimation.
  • Label smoothing for DPO = 0.01
    Table 8; hand-set, mentioned as beneficial for calibration.
  • Reward linear rescaling = [-1, 1]
    Section 4.4; raw rewards rescaled to [-1,1] before reward modeling.
assumptions (5)
  • domain assumption Answer confidence correlates with reasoning quality and correctness.
    Stated in the Introduction and Section 3, based on Wang and Zhou (2024); this is the core premise that makes self-feedback useful. If the base model is confidently wrong, self-feedback can reinforce errors.
  • ad hoc to paper Answer spans can be reliably extracted by appending "So the answer is" and string matching.
    Section 3.1; if the model does not produce this phrase or the span match fails, the confidence calculation is undefined or noisy. The paper lists span identification as a limitation.
  • domain assumption The base model already exhibits some basic reasoning ability.
    Acknowledged in Limitations: RLSF requires some reasoning to be observable in the data before optimization, otherwise there is nothing to bootstrap from.
  • domain assumption ECE on answer tokens or choice tokens is a valid calibration measure.
    Standard calibration metric, but its validity depends on correctly identifying the answer span, which the paper acknowledges is not always reliable.
  • ad hoc to paper Preference data from K=10 CoT candidates is sufficient for reward learning.
    Section 4.4 fixes K=10 without an ablation; the quality and coverage of preferences depend on this choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Post-Training Large Language Models via Reinforcement Learning from Self-Feedback." pith.science (2026). https://pith.science/paper/PRXD77GH

@misc{pith2026250721931,
  author       = {Pith},
  title        = {Pith review of: Post-Training Large Language Models via Reinforcement Learning from Self-Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PRXD77GH}},
  note         = {Machine review of arXiv:2507.21931}
}
read the original abstract

Large Language Models (LLMs) often produce plausible but poorly-calibrated answers, limiting their reliability on reasoning-intensive tasks. We present Reinforcement Learning from Self-Feedback (RLSF), a post-training stage that uses the model's own confidence as an intrinsic reward, mimicking how humans learn in the absence of external feedback. After a frozen LLM generates several chain-of-thought solutions, we define and compute the confidence of each final answer span and rank the traces accordingly. These synthetic preferences are then used to fine-tune the policy with standard preference optimization, similar to RLHF yet requiring no human labels, gold answers, or externally curated rewards. RLSF simultaneously (i) refines the model's probability estimates -- restoring well-behaved calibration -- and (ii) strengthens step-by-step reasoning, yielding improved performance on arithmetic reasoning and multiple-choice question answering. By turning a model's own uncertainty into useful self-feedback, RLSF affirms reinforcement learning on intrinsic model behaviour as a principled and data-efficient component of the LLM post-training pipeline and warrents further research in intrinsic rewards for LLM post-training.

Figures

Figures reproduced from arXiv: 2507.21931 by the authors.

Figure 1
Figure 1. An overview of the RLSF pipeline. across extended chains of reasoning is frequently not maintained. It has also been observed that the output of an LLM is largely uncalibrated – its confidence is not predictive of its accuracy, particularly after rein￾forcement learning from human feedback (RLHF) is applied (Bai et al., 2022). Such miscalibration re￾sults in overconfidence during complex reasoning tasks (OpenAI et a… view at source ↗
Figure 2
Figure 2. Example responses from the RLSF fine-tuned Gemma 2 model. The response words are colored based on [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Example of calibration improvement via RLSF. Both responses incorrectly answer “A” instead of the correct answer “E”, but the post-RLSF model produces a better explanation and expresses lower con￾fidence. fixed instruction regime. We use the outputs of the base model as the reference and compare them to the responses generated by the post-RLSF model. GPT 4o 2024-05-13 is employed as the annotating LLM. Training Data… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Towards AI epidemiology: a measurement standardisation framework for prospective risk detection

    cs.AI 2025-12 conditional novelty 5.0 of 10

    A Logia grammar is introduced that turns expert–AI interactions into eight standardised fields, enabling population-level surveillance of AI output failures without access to model internals.

  2. From Meta-Thought to Execution: Cognitively Aligned Post-Training for Generalizable and Reliable LLM Reasoning

    cs.AI 2026-01 conditional novelty 4.0 of 10

    Post-training LLMs first on abstract number-free reasoning plans (CoMT), then with confidence-weighted rewards (CCRL), raises math accuracy by ~2-5 points over standard CoT-SFT+RL across four models.

  3. Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

    cs.CL 2025-09 conditional novelty 3.0 of 10

    A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.

Reference graph

Works this paper leans on

58 extracted references · 28 canonical work pages · cited by 3 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Stephen Adams, Tyler Cody, and Peter A. Beling. 2022. https://doi.org/10.1007/s10462-021-10108-x A survey of inverse reinforcement learning . Artif. Intell. Rev., 55(6):4307–4346

    doi:10.1007/s10462-021-10108-x 2022

  4. [4]

    Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, K...

    arXiv doi:10.48550/arxiv.2212.08073 2022

  5. [5]

    Andrew G. Barto. 2013. Intrinsic motivation and reinforcement learning. In Gianluca Baldassarre and Marco Mirolli, editors, Intrinsically motivated learning in natural and artificial systems, pages 17--47. Springer Science+Business Media

    work page 2013

  6. [6]

    Ralph Allan Bradley and Milton E. Terry. 1952. https://doi.org/10.2307/2334029 Rank Analysis of Incomplete Block Designs : I . The Method of Paired Comparisons . Biometrika, 39(3/4):324--345. Publisher: [Oxford University Press, Biometrika Trust]

    doi:10.2307/2334029 1952

  7. [7]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

  8. [8]

    Guan-Lin Chao and Ian Lane. 2019. https://doi.org/10.21437/Interspeech.2019-1355 BERT - DST : Scalable End -to- End Dialogue State Tracking with Bidirectional Encoder Representations from Transformer . In Interspeech 2019, pages 1468--1472. ISCA

    work page doi:10.21437/interspeech.2019-1355 2019

Show all 58 references
  1. [9]

    Nuttapong Chentanez, Andrew Barto, and Satinder Singh. 2004. https://proceedings.neurips.cc/paper_files/paper/2004/file/4be5a36cbaca8ab9d2066debfe4e65c1-Paper.pdf Intrinsically motivated reinforcement learning . In Advances in Neural Information Processing Systems, volume 17. ...

    2004

  2. [10]

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. https://papers.nips.cc/paper_files/paper/2017/hash/d5e2c0adad503c91f91df240d0cd4e49-Abstract.html Deep Reinforcement Learning from Human Preferences . In Advances in Neural Information ...

    2017

  3. [11]

    Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinso...

    2024

  4. [12]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. https://doi.org/10.48550/arXiv.1803.05457 Think you have Solved Question Answering ? Try ARC , the AI2 Reasoning Challenge . arXiv preprint. ArXiv:1803.05457 [cs]

    2018 doi arXiv

  5. [13]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://doi.org/10.48550/arXiv.2110.14168 Training Verifiers to Solve Math Word Pr...

    2021 doi arXiv

  6. [14]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

    2025 arXiv

  7. [15]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

    2025 doi arXiv

  8. [16]

    Nicolai Dorka. 2024. https://doi.org/10.48550/arXiv.2409.10164 Quantile Regression for Distributional Reward Models in RLHF . arXiv preprint. ArXiv:2409.10164 [cs]

    2024 doi arXiv

  9. [17]

    Jacob Eisenstein. 2019. https://mitpress.mit.edu/books/introduction-natural-language-processing Introduction to N atural L anguage P rocessing . MIT Press

    2019

  10. [18]

    Jinlan Fu, Xuanjing Huang, and Pengfei Liu. 2021. https://doi.org/10.18653/v1/2021.acl-long.558 SpanNER : Named Entity Re -/ Recognition as Span Prediction . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International J...

    2021 doi

  11. [19]

    Michael Heck, Carel van Niekerk, Nurul Lubis, Christian Geishauser, Hsien-Chin Lin, Marco Moresi, and Milica Gasic. 2020. https://doi.org/10.18653/v1/2020.sigdial-1.4 TripPy : A Triple Copy Strategy for Value Independent Neural Dialog State Tracking . In Proceedings of the 21t...

    2020 doi

  12. [20]

    Liangjie Huang, Dawei Li, Huan Liu, and Lu Cheng. 2025. https://doi.org/10.48550/arXiv.2504.02902 Beyond Accuracy : The Role of Calibration in Self - Improving Large Language Models . arXiv preprint. ArXiv:2504.02902 [cs] version: 1

    2025 doi arXiv

  13. [21]

    Shijia Huang, Jianqiao Zhao, Yanyang Li, and Liwei Wang. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.570 Learning Preference Model for LLMs via Automatic Preference Data Generation . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processin...

    2023 doi

  14. [22]

    Alyssa Hughes. 2023. https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/ Phi-2: The surprising power of small language models

    2023

  15. [23]

    Subbarao Kambhampati. 2024. https://doi.org/10.1111/nyas.15125 Can large language models reason and plan? Annals of the New York Academy of Sciences, 1534(1):15--18. \_eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/nyas.15125

    2024 doi

  16. [24]

    Sanyam Kapoor, Nate Gruver, Manley Roberts, Arka Pal, Samuel Dooley, Micah Goldblum, and Andrew Wilson. 2024. https://aclanthology.org/2024.uncertainlp-1.1/ Calibration- Tuning : Teaching Large Language Models to Know What They Don `t Know . In Proceedings of the 1st Workshop ...

    2024

  17. [25]

    Martin Klissarov, Pierluca D'Oro, Shagun Sodhani, Roberta Raileanu, Pierre-Luc Bacon, Pascal Vincent, Amy Zhang, and Mikael Henaff. 2024. https://openreview.net/forum?id=tmBKIecDE9 Motif: Intrinsic motivation from artificial intelligence feedback . In The Twelfth International...

    2024

  18. [26]

    Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/hash/8bb0d291acd4acf06ef112099c16f326-Abstract-Conference.html Large Language Models are Zero - Shot Reasoners . In Advances in Neur...

    2022

  19. [27]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2022. https://openreview.net/forum?id=VD-AYtP0dve Semantic Uncertainty : Linguistic Invariances for Uncertainty Estimation in Natural Language Generation

    2022

  20. [28]

    Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, Phillip H. S. Torr, Fahad Shahbaz Khan, and Salman Khan. 2025. https://arxiv.org/abs/2502.21321 LLM Post-Training: A Deep Dive into Reasoning Large Language Models...

    2025 arXiv

  21. [29]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luc...

    2025 arXiv

  22. [30]

    Smith, and Hannaneh Hajishirzi

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. 2024. https://arxiv.org/abs/2403.13787 RewardBench: Evaluating Reward Models for Language Mo...

    2024 arXiv

  23. [31]

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Ren Lu, Thomas Mesnard, Johan Ferret, Colton Bishop, Ethan Hall, Victor Carbune, and Abhinav Rastogi. 2023. https://openreview.net/forum?id=AAxIs3D2ZZ RLAIF : Scaling Reinforcement Learning from Human Feedback with AI Feedback

    2023

  24. [32]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. AlpacaEval: An Automatic Evaluator of Instruction-following Models

    2023

  25. [33]

    Xingzhou Lou, Dong Yan, Wei Shen, Yuzi Yan, Jian Xie, and Junge Zhang. 2025. https://doi.org/10.48550/arXiv.2410.00847 Uncertainty-aware Reward Model : Teaching Reward Models to Know What is Unknown . arXiv preprint. ArXiv:2410.00847 [cs]

    2025 doi arXiv

  26. [34]

    Machado and Michael Bowling

    Marlos C. Machado and Michael Bowling. 2016. Learning purposeful behaviour in the absence of rewards. In ICML Workshop on Abstraction in Reinforcement Learning

    2016

  27. [35]

    OpenAI. 2025. https://openai.com/index/o3-o4-mini-system-card/ OpenAI o3 System Card

    2025

  28. [36]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 doi arXiv

  29. [37]

    OpenAI, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, ...

    2024 arXiv

  30. [38]

    Pierre-Yves Oudeyer, Frdric Kaplan, and Verena V. Hafner. 2007. https://doi.org/10.1109/TEVC.2006.890271 Intrinsic motivation systems for autonomous mental development . IEEE Transactions on Evolutionary Computation, 11(2):265--286

    2007

  31. [39]

    Christiano, Jan Leike, and Ryan Lowe

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, ...

    2022

  32. [40]

    Lena Esther Ptasczynski, Isa Steinecker, Philipp Sterzer, and Matthias Guggenmos. 2022. https://doi.org/10.1371/journal.pcbi.1010580 The value of confidence: Confidence prediction errors drive value-based learning in the absence of external feedback . PLOS Computational Biolog...

    2022 doi

  33. [41]

    Manning, Stefano Ermon, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html Direct Preference Optimization : Your Language Mod...

    2023

  34. [42]

    Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jero...

    2024 doi arXiv

  35. [43]

    Paul R \"o ttger, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2024. https://doi.org/10.18653/v1/2024.naacl-long.301 XST est: A test suite for identifying exaggerated safety behaviours in large language models . In Proceedings of the 2024 Co...

    2024 doi

  36. [44]

    Abulhair Saparov and He He. 2022. https://openreview.net/forum?id=qFVVBzXxR2V Language Models Are Greedy Reasoners : A Systematic Formal Analysis of Chain -of- Thought

    2022

  37. [45]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. https://doi.org/10.48550/arXiv.1707.06347 Proximal Policy Optimization Algorithms . arXiv preprint. ArXiv:1707.06347 [cs]

    2017 doi arXiv

  38. [46]

    Paul Stangel, David Bani-Harouni, Chantal Pellegrini, Ege \"O zsoy, Kamilia Zaripova, Matthias Keicher, and Nassir Navab. 2025. Rewarding Doubt: A Reinforcement Learning Approach to Confidence Calibration of Large Language Models . arXiv preprint arXiv:2503.02623

    2025

  39. [47]

    Hao Sun and Mihaela van der Schaar. 2025. https://doi.org/10.48550/arXiv.2405.15624 Inverse- RLignment : Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learning . arXiv preprint. ArXiv:2405.15624 [cs]

    2025 doi arXiv

  40. [48]

    Sutton, David McAllester, Satinder Singh, and Yishay Mansour

    Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. 1999. Policy gradient methods for reinforcement learning with function approximation. In Proceedings of the 13th International Conference on Neural Information Processing Systems, NIPS'99, page 1057–1063,...

    1999

  41. [49]

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. https://doi.org/10.18653/v1/N19-1421 CommonsenseQA : A Question Answering Challenge Targeting Commonsense Knowledge . In Proceedings of the 2019 Conference of the North American Chapter of the Associatio...

    2019 doi

  42. [50]

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.330 Just Ask for Calibration : Strategies for Eliciting Calibrated Confidence Scores from Language Mo...

    2023 doi

  43. [51]

    Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. 2020. Trl: Transformer reinforcement learning. https://github.com/huggingface/trl

    2020

  44. [52]

    Xuezhi Wang and Denny Zhou. 2024. https://doi.org/10.48550/arXiv.2402.10200 Chain-of- Thought Reasoning Without Prompting . arXiv preprint. ArXiv:2402.10200 [cs]

    2024 doi arXiv

  45. [53]

    Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html Chain-of- Thought Prompting Elicits Rea...

    2022

  46. [54]

    Paula Wesselmann, Yen-Chen Wu, and Milica Ga s i c . 2018. Curiosity-driven reinforcement learning for dialogue management. In International Conference on Acoustics, Speech, and Signal Processing

    2018

  47. [55]

    Yuxin Xiao, Paul Pu Liang, Umang Bhatt, Willie Neiswanger, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2022. https://doi.org/10.18653/v1/2022.findings-emnlp.538 Uncertainty Quantification with Pre -trained Language Models : A Large - Scale Empirical Analysis . In Finding...

    2022 doi

  48. [56]

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. https://papers.nips.cc/paper_files/paper/2022/hash/639a9a172c044fbb64175b5fad42e9a5-Abstract-Conference.html STaR : Bootstrapping Reasoning With Reasoning . In Advances in Neural Information Processing Systems , volum...

    2022

  49. [57]

    Yifei Zhou, Andrea Zanette, Jiayi Pan, Aviral Kumar, and Sergey Levine. 2024. https://openreview.net/forum?id=sT5wIGq7BV ArCHer : Training Language Model Agents via Hierarchical Multi - Turn RL

    2024

  50. [58]

    Enwei Zhu, Yiyang Liu, and Jinpeng Li. 2023. https://doi.org/10.18653/v1/2023.findings-acl.672 Deep Span Representations for Named Entity Recognition . In Findings of the Association for Computational Linguistics : ACL 2023 , pages 10565--10582, Toronto, Canada. Association fo...

    2023 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.