Pith. sign in

REVIEW 3 major objections 5 minor 5 cited by

Ignore the KL Penalty! Boosting Exploration on Critical Tokens to Enhance RL Fine-Tuning

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Down-weighting the KL penalty on uncertain tokens improves RL fine-tuning

desk verdict A clever but confounded KL-penalty tweak; the main comparison doesn't isolate targeted weighting from a globally weaker penalty, so the headline claim needs a matched-penalty control. read the letter →

arxiv 2502.06533 v1 pith:W7J5BAKA submitted 2025-02-10 cs.CL cs.LG

Jean Vassoyan , Nathanaël Beau , Roman Plaud This is my paper · ORCID
classification cs.CLcs.LG
keywords reinforcementlearninglanguagemodelsKLpenaltycriticaltokensexplorationarithmeticreasoningout-of-distributiongeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reinforcement-learning fine-tuning of language models usually counts every departure from the pre-trained model as a cost, through a single KL penalty applied to all tokens. This paper claims the penalty should be relaxed exactly on the few 'critical tokens' where the old model is uncertain and a wrong choice ruins the entire answer. It proposes a prioritized KL penalty in which each state-action term is weighted by the old policy's normalized negentropy raised to a power $\beta$, so low-certainty states are freed to explore while confident states stay pinned to pre-training. In a controlled addition task, this change substantially improves RL exploration efficiency and final accuracy when the model is asked to add numbers longer than any seen in pre-training.

What carries the argument

The load-bearing object is the per-state certainty estimate $\hat J_{\theta_{old}}(s)$, the normalized negentropy of the frozen pre-trained policy: it equals 1 when the old model is certain and 0 when it is maximally uncertain, and it is cheap to compute because it uses only the old policy's output probabilities. Raising it to the power $\beta$ and multiplying each term of the KL penalty makes the penalty vanish on uncertain states while leaving it nearly intact on confident states. This turns the qualitative observation about critical tokens into a training signal: the policy is allowed to explore precisely where the pre-trained model has no strong opinion, and is held close to pre-training everywhere else.

What would settle it

Construct a distribution shift in which the model's critical mistakes are made with high confidence—for example, a memorized copying rule that fails only on longer numbers—so that $\hat J_{\theta_{old}}(s)$ stays near 1 on the decisive tokens. The prioritized penalty should then behave like the standard KL penalty and give no accuracy gain; observing a gain would show the mechanism is not what the paper claims, while observing no gain would show the certainty proxy is limited to low-confidence error modes.

Watch

Extended reading notes

Core claim

The paper's central discovery is that success in RL fine-tuning on an out-of-distribution arithmetic task hinges on a small set of 'critical tokens'—tokens that are decisive for the final answer and on which the pre-trained model is markedly less certain than elsewhere—and that the standard uniform KL penalty suppresses exactly the exploration needed to fix them. To exploit this, the paper introduces the prioritized KL penalty $\tilde L_{KL} = \mathbb{E}_{s,a \sim \pi_\theta}\left[\hat J_{\theta_{old}}(s)^{\beta} \log\frac{\pi_\theta(a|s)}{\pi_{\theta_{old}}(a|s)}\right]$, with certainty estimate $\hat J_{\theta_{old}}(s) = (H_{\max} - H(\pi_{\theta_{old}}(\cdot|s)))/H_{\max}$, the normalized negentropy of the frozen pre-trained policy at state $s$. This weight is close to 1 where the old model is confident and close to 0 where it is uncertain, so the KL leash is loosened only at the decision points that matter. In experiments with an 85M-parameter GPT-2 fine-tuned by A2C to add numbers one or two digits beyond its training range, the prioritized penalty gives higher accuracy and more persistent correction of critical tokens than the standard penalty, with the gain robust across a wide range of $\beta$.

Load-bearing premise

The method assumes that the frozen pre-trained model's token-level uncertainty is a reliable guide to where exploration is needed, and that loosening the KL penalty in low-certainty states improves exploration rather than merely allowing the policy to drift.

Editorial extensions

If this is right

  • RL fine-tuning on sparse-reward generation tasks can be made more efficient without changing the policy class, the reward, or the RL algorithm; only the KL penalty weights are modified.
  • In the tested addition setting, the prioritized penalty keeps the probability of correct predictions on critical tokens high and stable, while the standard-KL model periodically reverts to its pre-trained mistakes.
  • The benefit holds for $\beta$ values from roughly 10 to 500, peaks near $\beta = 500$, and degrades at $\beta = 1000$ before collapsing at $\beta = 10000$, so the method does not require exact tuning of the exponent.
  • More extensive pre-training improves out-of-distribution accuracy on longer numbers, but those stronger models tend to plateau during RL; the prioritized penalty is demonstrated to help in the $N=7$ to $N+1=8$ shift.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the same entropy-weighting idea should transfer to any reasoning task with a few decisive 'choice points', such as multi-step math or planning, where old-policy entropy can flag the steps that determine success.
  • Editorial extension: because the modification only changes a loss term, it should combine with larger models, other policy-gradient algorithms, and techniques like reward shaping; the paper tests only A2C on GPT-2.
  • Editorial extension: the working range of $\beta$ is likely task- and calibration-dependent; the paper notes its pre-trained model is extremely confident except on critical tokens, so a differently calibrated model may need a different exponent.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies RL fine-tuning of a small GPT-2 model on an arithmetic addition task with a scratchpad, after pre-training on shorter digit lengths. It identifies a small set of 'critical tokens' where the pre-trained model is uncertain and where errors are decisive. The authors propose a modification to the standard KL penalty, replacing it with a state-dependent weight bJ(s)^beta that down-weights the KL term when the pre-trained model's normalized negentropy is low. Experiments show improved accuracy on N+1 digit addition for the prioritized penalty compared to the standard KL, and a beta sweep (10-500) shows the advantage is robust, while beta=10000 collapses. The paper concludes that the prioritized penalty boosts exploration on critical tokens and improves RL fine-tuning efficiency.

Significance. If the central claim holds, the proposed modification is simple, computationally cheap, and plausibly transferable to other RL fine-tuning settings. The study is carried out in a controlled environment with a small model, which allows careful analysis of exploration dynamics. The public code and detailed appendices are strengths. However, the evidence for the mechanism is weakened by a confound: the prioritized KL also lowers the overall KL penalty, and the statistical support is limited. The critical-token analysis is descriptive but does not directly establish that the improvement is due to targeting these tokens.

major comments (3)
  1. [Section 4, Eq. (1) and Section 5.3] The prioritized KL penalty uses weights bJ(s)^beta that are less than 1 for any non-maximally certain state. With alpha fixed at 5 for both objectives, the prioritized objective also reduces the total KL penalty, not just its allocation across tokens. The observed improvement could be due to weaker overall KL regularization. To isolate the 'critical tokens' mechanism, the authors should run a control with the standard KL penalty whose coefficient alpha is scaled so that the mean effective penalty matches that of the prioritized KL (e.g., using the empirical mean of bJ^beta over states). The beta sweep in Figure 6 does not control for this, as varying beta changes both the targeting profile and the mean weight. This is the central load-bearing point and should be addressed before the mechanistic claim can be accepted.
  2. [Figure 4 (Section 5.3)] The headline result is shown as single learning curves without error bars or multiple seeds, while the beta sweep in Figure 6 reports 'important error margins' and only 'slightly outperforms' for beta=10-500. The text says 'significantly outperformed' but this is not supported by the displayed data. Please provide seed-averaged learning curves with confidence intervals for the main comparison, along with final-accuracy statistics. Without this, the 'substantially enhances exploration efficiency' claim is not quantitatively established.
  3. [Section 3.3 and Section 5.3 (bottom of Figure 4, Figure 7)] The paper does not directly measure whether the prioritized KL increases exploration on critical tokens. The descriptive probabilities of correct predictions on critical tokens are suggestive, but they do not distinguish between a model that explores more on critical tokens and one that explores more everywhere. Please report a direct metric of exploration on critical states (e.g., entropy of the policy at critical tokens, or the diversity of actions attempted) for both methods. This would strengthen the mechanistic interpretation beyond the post hoc analysis.
minor comments (5)
  1. [Figure 4] Consider adding error bars or shaded regions to the learning curves to display variance across seeds.
  2. [Appendix C, Figure 6] The text says 'important error margins'; please provide the numerical means and standard deviations, and state whether the differences between prioritized and standard KL are statistically significant (e.g., via confidence interval comparison or a significance test).
  3. [Appendix D, Table 3] The KL penalty coefficient alpha=5 used for the prioritized KL experiment differs from alpha=10 used in the pre-training comparison (Table 2). Please clarify why this choice was made and confirm that the same alpha was used for both standard and prioritized KL in the main comparison.
  4. [Section 3.3 and Appendix B] The claim that an error at a critical token leads to a wrong final answer 'most likely' is qualitative; please provide a quantitative estimate, such as the percentage of generations where a critical-token error co-occurs with an incorrect final result.
  5. [Title and Abstract] The title 'Ignore the KL Penalty!' is catchy but may be misleading; the paper proposes a modified KL penalty rather than ignoring it. Consider a more precise phrasing.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor self-definitional step in the critical-token analysis; central RL comparison is an external benchmark and is not circular.

  1. self definitional [Section 3.3 (Critical tokens) and Table 1]
    "Within the output generated by a language model, a "critical token" is a token that satisfies both of these criteria: • it is decisive for the rest of the answer: if the model is wrong about this token, the final answer will most likely be wrong (the model fails to correct itself); • the pre-trained model shows substantially more uncertainty on these tokens than on the rest of the output."

    High uncertainty on these tokens is one of the two criteria used to define "critical tokens", so Table 1's finding of a large certainty gap on critical tokens is guaranteed by the selection rule rather than discovered independently. Likewise, "decisive" is defined as a wrong token making the final answer wrong, and the paper reports that "whenever these tokens are generated incorrectly, the model inevitably produces the wrong answer" as a finding, restating the definition. This is a mild self-definitional step, but it is not load-bearing for the main result: the prioritized-KL improvement is evaluated on end-task accuracy against a standard-KL baseline, not on the critical-token statistics.

full rationale

The paper's central contribution is a heuristic modification of the KL penalty (Eq. 1) weighted by normalized negentropy (Eq. 2), compared against a standard KL penalty on an external accuracy benchmark (Section 5.3, Appendix C). That comparison is not circular: the accuracy numbers are measured on held-out additions, the critical-token probabilities in Figure 4 are measurements of the trained policy, and the robustness of the beta sweep (10-500, Appendix C) shows the headline does not reduce to a single fitted value. There is no self-citation chain, no imported uniqueness theorem, and no fitted parameter renamed as a prediction. The only circular step is in the descriptive critical-token analysis: "critical token" is defined partly by high pre-trained-model uncertainty (and by decisiveness), so Table 1 and the associated claims about uncertainty and wrong-token consequences largely restate the definition. This does not drive the empirical RL result, so the overall circularity score is low (2). A separate concern, noted by the skeptic, is that with the same coefficient alpha the prioritized penalty also lowers the average KL weight, confounding "targeting critical tokens" with a global reduction in regularization strength; that is a correctness/experimental-design risk, not a circularity, because the comparison is still an external measurement rather than an identity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The paper's empirical method relies on standard RL machinery plus a new certainty-weighting heuristic. The free parameters are beta and alpha; beta is selected on the evaluation task. The main ad hoc assumption is the negentropy certainty proxy. No physical entities are introduced.

free parameters (2)
  • beta (KL certainty exponent) = 150 (main experiment), swept over 10-500
    Hyperparameter controlling the influence of certainty weighting; chosen based on performance on the target task (Appendix C).
  • alpha (KL penalty coefficient) = 5 (main experiment), 10 (pretraining comparison)
    Standard KL coefficient, set by hand; affects the balance between RL loss and KL.
assumptions (4)
  • domain assumption The KL divergence penalty is the appropriate regularizer for RL fine-tuning of language models (Ziegler et al., 2020).
    This is the standard practice the paper builds on; the paper does not justify it beyond citing prior work.
  • ad hoc to paper Normalized negentropy bJ(s) = (Hmax - H(pi_old))/Hmax is a valid certainty measure for the pretrained model.
    Introduced in Eq. 2 with the caveat that it ignores model uncertainty; the paper acknowledges this is an approximation.
  • domain assumption A2C with the listed hyperparameters is an effective optimizer for this task.
    The paper chooses A2C for simplicity but does not compare with PPO or other algorithms, so the conclusions may depend on this choice.
  • domain assumption The scratchpad format from Lee et al. (2024) is a suitable action representation for addition.
    The paper adopts this format without re-validation; performance could differ with another scratchpad.
invented entities (1)
  • critical tokens independent evidence
    purpose: Describe tokens that are both decisive for the final answer and have high uncertainty in the pretrained model; used to justify the certainty-weighted KL penalty.
    The paper provides quantitative evidence of lower certainty on these tokens (Table 1) and failure analysis (Appendix B), so they are measurable rather than purely hypothetical.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Ignore the KL Penalty! Boosting Exploration on Critical Tokens to Enhance RL Fine-Tuning." pith.science (2026). https://pith.science/paper/W7J5BAKA

@misc{pith2026250206533,
  author       = {Pith},
  title        = {Pith review of: Ignore the KL Penalty! Boosting Exploration on Critical Tokens to Enhance RL Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W7J5BAKA}},
  note         = {Machine review of arXiv:2502.06533}
}
read the original abstract

The ability to achieve long-term goals is a key challenge in the current development of large language models (LLMs). To address this, pre-trained LLMs can be fine-tuned with reinforcement learning (RL) to explore solutions that optimize a given goal. However, exploration with LLMs is difficult, as a balance has to be struck between discovering new solutions and staying close enough to the pre-trained model, so as not to degrade basic capabilities. This is typically controlled with a Kullback-Leibler (KL) penalty. In this paper, we investigate the exploration dynamics of a small language model on a simple arithmetic task. We show how varying degrees of pre-training influence exploration and demonstrate the importance of "critical tokens" which have a dramatic impact on the final outcome. Consequently, we introduce a simple modification to the KL penalty that favors exploration on critical tokens, increasing the efficiency of the RL fine-tuning stage.

Figures

Figures reproduced from arXiv: 2502.06533 by the authors.

Figure 1
Figure 1. Illustration of the addition task with scratch [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Model accuracy on addition tasks for models [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Top: Learning curves of a model fine-tuned [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Output examples for addition tasks on N + 1 digit lengths (the model is faced with numbers one notch longer than those encountered in pre-training). Each generated token is colored according to its certainty. A green color is a maximal certainty, while a red color is a…
Figure 6
Figure 6. Figure 6: Fine-tuning results with various values of [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Evolution of the right prediction probability on multiple critical tokens, during the RL fine-tuning on [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Embarrassingly Simple Self-Distillation Improves Code Generation

    cs.CL 2026-04 conditional novelty 7.0 of 10

    Simple self-distillation—fine-tuning a code model on its own temperature-sampled, truncated outputs—raises LiveCodeBench pass@1 substantially without verifiers, teachers, or RL.

  2. Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization

    cs.CL 2025-10 conditional novelty 6.0 of 10

    LLM attention maps reveal a preplan-and-anchor pattern, and reweighting RL credit toward the flagged tokens improves math/QA reasoning.

  3. Self-Reflective Generation at Test Time

    cs.CL 2025-10 conditional novelty 5.0 of 10

    SRGen improves LLM math reasoning by detecting high-entropy tokens and injecting a small corrected vector into the hidden state at those points during decoding, without training.

  4. Discovering Algorithms with Computational Language Processing

    cs.AI 2025-07 conditional novelty 5.0 of 10

    A machine learning framework called CLP discovers, improves, and tailors algorithms by chaining computational tokens with MCTS and RL, with strong results on the Quadratic Assignment Problem and quantum search.

  5. CogniSQL-R1-Zero: Lightweight Reinforced Reasoning for Efficient SQL Generation

    cs.AI 2025-07 reject novelty 3.0 of 10

    A 7B model trained with GRPO and a sparse execution-correctness reward reaches 59.97% execution accuracy on BIRD dev, though the evaluation protocol and baseline numbers contain inconsistencies.

Reference graph

Works this paper leans on

38 extracted references · 16 canonical work pages · cited by 5 Pith papers

  1. [1]

    Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. 2022. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 378(6624):1067--1074

    work page 2022

  2. [2]

    Filippos Bellos, Yayuan Li, Wuao Liu, and Jason Corso. 2024. https://aclanthology.org/2024.scalellm-1.3 Can large language models reason about goal-oriented tasks? In Proceedings of the First edition of the Workshop on the Scaling Behavior of Large Language Models (SCALE-LLM 2024), pages 24--34, St. Julian ' s, Malta. Association for Computational Linguistics

    work page 2024

  3. [3]

    Leon Brillouin. 1953. The negentropy principle of information. Journal of Applied Physics, 24(9):1152--1163

    work page 1953

  4. [4]

    Meng Cao, Lei Shu, Lei Yu, Yun Zhu, Nevan Wichers, Yinxiao Liu, and Lei Meng. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.515 Enhancing reinforcement learning with dense rewards from language model critic . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9119--9138, Miami, Florida, USA. Association fo...

    doi:10.18653/v1/2024.emnlp-main.515 2024

  5. [5]

    Thomas Carta, Cl \' e ment Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre - Yves Oudeyer. 2023. https://proceedings.mlr.press/v202/carta23a.html Grounding large language models in interactive environments with online reinforcement learning . In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, US...

    work page 2023

  6. [6]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, and et al. 2021. https://arxiv.org/abs/2107.03374 Evaluating large language models trained on code . Preprint, arXiv:2107.03374

    arXiv 2021

  7. [7]

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. https://proceedings.neurips.cc/paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf Deep reinforcement learning from human preferences . In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc

    work page 2017

  8. [8]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . Preprint, arXiv:2110.14168

    arXiv 2021

Show all 38 references
  1. [9]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  2. [10]

    Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. 2024. https://arxiv.org/abs/2403.04642 Teaching large language models to reason with reinforcement learni...

    2024 arXiv

  3. [11]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/be83ab3ecd0db773eb2dc1b0a17836a1-Abstract-round2.html Measuring mathematical problem so...

    2021

  4. [12]

    Hyeonbin Hwang, Doyoung Kim, Seungone Kim, Seonghyeon Ye, and Minjoon Seo. 2024. https://aclanthology.org/2024.findings-emnlp.78 Self-explore: Enhancing mathematical reasoning in language models with fine-grained rewards . In Findings of the Association for Computational Lingu...

    2024

  5. [13]

    Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/4e85362c02172c0c6567ce593122d31c-Abstract-Conference.html The impact of positional encoding on length generalization in tran...

    2023

  6. [14]

    Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/8636419dea1aa9fbd25fc4248e702da4-Paper-Conference.pdf Coderl: Mastering code generation through pretrained models and deep rei...

    2022

  7. [15]

    Lee, Kangwook Lee, and Dimitris Papailiopoulos

    Nayoung Lee, Kartik Sreenivasan, Jason D. Lee, Kangwook Lee, and Dimitris Papailiopoulos. 2024. https://openreview.net/forum?id=dsUB4bst9S Teaching arithmetic to small transformers . In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austri...

    2024

  8. [16]

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, and et al. 2022. https://doi.org/10.1126/science.abq1158 Competition-level code generation with alphacode . Science, 378(6624):1092–1097

    2022 doi

  9. [17]

    Tiedong Liu and Bryan Kian Hsiang Low. 2023. https://doi.org/10.48550/ARXIV.2305.14201 Goat: Fine-tuned llama outperforms GPT-4 on arithmetic tasks . CoRR, abs/2305.14201

    2023 doi arXiv

  10. [18]

    Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, Jonas Geiping, Avi Schwarzschild, and Tom Goldstein

    Sean McLeish, Arpit Bansal, Alex Stein, Neel Jain, John Kirchenbauer, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, Jonas Geiping, Avi Schwarzschild, and Tom Goldstein. 2024. Transformers can do arithmetic with the right embeddings. In Advances in Neural Information ...

    2024

  11. [19]

    Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. https://proceedings.mlr.press/v48/mniha16.html Asynchronous methods for deep reinforcement learning . In Proceedings of The 33rd Intern...

    2016

  12. [20]

    OpenAI . 2023. Gpt-4: Generative pre-trained transformer 4. https://openai.com. Accessed: 2024-02-06

    2023

  13. [21]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf Language models are unsupervised multitask learners . OpenAI. Accessed: 2024-11-15

    2019

  14. [22]

    John Schulman. 2020. Approximating kl divergence, 2020. URL http://joschu. net/blog/kl-approx. html

    2020

  15. [23]

    Ruoqi Shen, Sébastien Bubeck, Ronen Eldan, Yin Tat Lee, Yuanzhi Li, and Yi Zhang. 2023. https://arxiv.org/abs/2311.14737 Positional description matters for transformers arithmetic . Preprint, arXiv:2311.14737

    2023 arXiv

  16. [24]

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. https://proceedings.neurips.cc/paper_files/paper/2020/file/1f89885d556929e98d3ef9b86448f951-Paper.pdf Learning to summarize with human fee...

    2020

  17. [25]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, and et al. 2023. https://doi.org/10.48550/ARXIV.2307.09288 Llama 2: Open foundation and fine-tuned chat models . CoRR, abs/2307.09288

    2023 doi arXiv

  18. [26]

    Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019. https://doi.org/10.18653/v1/D19-1534 Do NLP models know numbers? probing numeracy in embeddings . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th ...

    2019 doi

  19. [27]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. http://papers.nips.cc/paper\_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html Chain-of-thought prompting elicits reasoning...

    2022

  20. [28]

    Jiayi Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Duburcq, Minghao Zhang, Yi Su, Hang Su, and Jun Zhu. 2022. http://jmlr.org/papers/v23/21-1127.html Tianshou: A highly modularized deep reinforcement learning library . Journal of Machine Learning Research, 23(267):1--6

    2022

  21. [29]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020

  22. [30]

    Changnan Xiao and Bing Liu. 2023. https://arxiv.org/abs/2311.16173 Conditions for length generalization in learning reasoning skills . Preprint, arXiv:2311.16173

    2023 arXiv

  23. [31]

    Hausknecht, and Karthik Narasimhan

    Shunyu Yao, Rohan Rao, Matthew J. Hausknecht, and Karthik Narasimhan. 2020. https://doi.org/10.18653/V1/2020.EMNLP-MAIN.704 Keep CALM and explore: Language models for action generation in text-based games . In Proceedings of the 2020 Conference on Empirical Methods in Natural ...

    2020 doi

  24. [32]

    Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang. 2023. https://doi.org/10.48550/ARXIV.2304.02015 How well do large language models perform in arithmetic tasks? CoRR, abs/2304.02015

    2023 doi arXiv

  25. [33]

    Eric Zelikman, Georges Raif Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah Goodman. 2024. https://openreview.net/forum?id=oRXPiSOGH9 Quiet- ST ar: Language models can teach themselves to think before speaking . In First Conference on Language Modeling

    2024

  26. [34]

    Hugh Zhang and David C. Parkes. 2023. https://arxiv.org/abs/2309.08589 Chain-of-thought reasoning is a policy improvement operator . Preprint, arXiv:2309.08589

    2023 arXiv

  27. [35]

    Yongchao Zhou, Uri Alon, Xinyun Chen, Xuezhi Wang, Rishabh Agarwal, and Denny Zhou. 2024. https://openreview.net/forum?id=DWkWIh3vFJ Transformers can achieve length generalization but not robustly . In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models

    2024

  28. [36]

    Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B

    Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2020. https://arxiv.org/abs/1909.08593 Fine-tuning language models from human preferences . Preprint, arXiv:1909.08593

    2020 arXiv

  29. [37]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  30. [38]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.