Pith. sign in

REVIEW 4 major objections 7 minor 4 cited by

SCAR: Shapley Credit Assignment for More Efficient RLHF

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SCAR claims that replacing RLHF's single terminal reward with Shapley-value-based dense rewards leaves the optimal policy unchanged and makes training converge faster and score higher.

desk verdict Useful and clearly motivated method, but the empirical claims rest on an unvalidated characteristic function and the invariance theorem is a definitional identity; still worth a careful referee. read the letter →

arxiv 2505.20417 v1 pith:ZW7LRDGO submitted 2025-05-26 cs.AI

Meng Cao , Shuyuan Zhang , Xiao-Wen Chang , Doina Precup This is my paper
classification cs.AI
keywords reinforcementlearningfromhumanfeedbackShapleyvaluecreditassignmentdenserewardlargelanguagemodelsOwenshapingpolicyinvariance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to fix the sparse-reward problem in Reinforcement Learning from Human Feedback (RLHF): the reward model gives one score for a whole generated response, and the language model gets no signal about which choices caused that score. SCAR splits the terminal score among tokens or spans by computing Shapley values, treating each text unit as a player in a cooperative game. The paper proves that this reshaped reward leaves the optimal policy unchanged, because the dense Shapley rewards sum to exactly the original terminal reward. Empirically, across sentiment control, text summarization, and instruction tuning, SCAR is reported to converge faster and reach higher final reward scores than sparse RLHF and attention-based dense reward baselines, without extra human annotations or critic models. If this holds, RLHF can be made more sample-efficient with a principled credit-assignment rule.

What carries the argument

The central object is the Shapley value (a player's average marginal contribution over all possible orders of adding players) of a text unit under the characteristic function $v(S) = r_\phi(x, y_S)$, where $y_S$ is the partial sequence formed by concatenating the units in coalition $S$ in their original order, with non-coalition positions blanked. The load-bearing identity is that, for every complete trajectory, $\sum_{t=1}^T R_t(\alpha) = \sum_{t=1}^T R_t^{\mathrm{orig}}$: the efficiency property of Shapley values makes the dense Shapley rewards sum to the terminal reward $r_\phi(x,y)$, and the convex-combination parameter $\alpha$ cancels, so the total episode return is unchanged. The paper presents this as a potential-based reward shaping argument, and it is what carries Theorem 3.1. To make computation tractable, players are adaptively grouped into tokens, syntactic spans, or sentences, and Shapley values are approximated by Owen values, lowering the number of reward-model queries from exponential $O(2^N)$ to quadratic $O(N^2)$.

What would settle it

Run PPO on a short task with at most six spans per response under three rewards: exact Shapley dense rewards, Owen-approximated SCAR rewards, and sparse terminal rewards. The central claim fails if the Owen-approximated policy trails the exact-Shapley policy materially, or if neither dense variant beats sparse RLHF on final reward, or if blanking out a phrase that changes human preference does not change the reward model's score.

Watch

Extended reading notes

Core claim

SCAR treats the tokens or spans of a generated response as players in a cooperative game, defines the value of a coalition as the reward model's score for the partial sequence obtained by keeping just those players, and assigns each player its Shapley value. The total of these values equals the original sequence-level score, so the dense reward changes the total return of every complete trajectory by exactly zero; Theorem 3.1 concludes that the optimal policy is the same under the sparse and the dense rewards. Empirically the paper reports that SCAR converges faster and reaches higher final reward scores than sparse RLHF, uniform reward distribution, and attention-based credit assignment across sentiment control, text summarization, and instruction tuning, and that its outputs win more pairwise LLM-as-judge comparisons.

Load-bearing premise

The method depends on the reward model giving meaningful scores when chunks of the text are blanked out, and on the fast Owen-value approximation staying close enough to the true Shapley credits that the dense rewards are faithful.

Editorial extensions

If this is right

  • Training with SCAR's dense rewards is equivalent to training with the original sparse reward in terms of the set of optimal policies, so any alignment objective that is valid for RLHF remains valid under SCAR.
  • Across sentiment control, text summarization, and instruction tuning, SCAR reaches higher reward-model scores than sparse RLHF, uniform distribution, and attention-based credit assignment, and its outputs win more pairwise LLM-judge comparisons.
  • Because SCAR requires no fine-grained human annotations and no auxiliary critic model, it can be dropped into existing RLHF pipelines at the cost of extra reward-model queries.
  • Adaptive segmentation controls that cost: span-level and sentence-level players cut Shapley computation from exponential to quadratic complexity, and in the summarization experiments span-level SCAR matched token-level SCAR at roughly one seventh of the GPU time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The policy-invariance theorem uses only the efficiency axiom of Shapley values, so the same guarantee would hold for any redistribution rule whose dense credits sum to the terminal reward; the empirical comparison between SCAR and ABC is really a comparison of which credit rule is more informative, not which is more theoretically justified.
  • A natural stress test is to compare exact Shapley rewards with the Owen approximation on short sequences; if the approximation materially changes the dense rewards or the final policy, the paper's theoretical guarantee applies to the exact rule, not necessarily to the implemented algorithm.
  • For reward models that score only complete answers, the characteristic function $v(S)=r_\phi(x,y_S)$ should be redefined, e.g., by sampling the missing units from the current policy or by infilling a learned placeholder; otherwise the credit values cease to reflect true marginal contributions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes SCAR (Shapley Credit Assignment Rewards), a method for RLHF that redistributes the sparse terminal reward from a learned reward model into dense per-token or per-span rewards using Shapley values. The reward model is queried on partial sequences formed by concatenating the units present in a coalition, with absent units replaced by empty spaces. The authors claim that this dense reward preserves the optimal policy (Theorem 3.1) and demonstrate empirical improvements in convergence speed and final reward on sentiment control, text summarization, and instruction tuning compared to standard sparse RLHF and attention-based dense reward baselines.

Significance. If the method works as claimed, it offers a principled alternative to attention-based dense reward in RLHF, and the paper correctly identifies the credit-assignment challenge in sparse-reward settings. The manuscript is clearly written, provides detailed hyperparameters, and covers three diverse tasks. However, the theoretical guarantee is a definitional identity that follows from the efficiency axiom of Shapley values and does not specifically validate the choice of characteristic function or the approximation scheme. The empirical advantage rests on an unvalidated assumption about the reward model's ability to score gap-filled partial sequences, and the connection between the exact theory and the implemented Owen-value approximation is not established. These load-bearing issues require substantial additional evidence.

major comments (4)
  1. [Section 3.2, Eq. (3)] The characteristic function v(S) = r_phi(x, y_S) is defined using partial sequences in which absent units are replaced by empty spaces. The paper itself notes that the ideal definition would be an expectation over completions from the current policy, but it does not test whether the gap-filled proxy is an accurate approximation. Since the empirical gains are attributed to principled Shapley credit assignment, the manuscript must provide evidence that reward models trained on complete sequences produce meaningful scores on such partial, gap-filled inputs (e.g., correlation with leave-one-out ablations or human judgments). Without such evidence, the dense rewards are an arbitrary zero-sum decomposition of the terminal reward, and the observed improvements could be a generic dense-reward effect rather than a consequence of Shapley credit assignment.
  2. [Theorem 3.1 and Appendix A] The proof shows that the total undiscounted return is identical for the original and shaped rewards because the Shapley values sum to the terminal reward. This is true for any allocation satisfying efficiency; it does not rely on potential-based reward shaping, and indeed the shaping term F_t = alpha*Rshap_t - alpha*I(t=T)*r_phi(x,y) depends on the whole trajectory and cannot be written as gamma*Phi(s') - Phi(s). The theorem is correct for exact Shapley values, but it is a definitional identity and provides no inductive support for the specific choice of Shapley values. More importantly, the implemented algorithm (Section 3.4) uses Owen-value approximations via the SHAP package with no error analysis or statement of whether the approximation satisfies efficiency. If the approximations do not satisfy efficiency exactly, the total per-episode reward is not conserved and the policy-invariance guarantee does not transfer to the actual algorithm.
  3. [Section 3.4] The claim that the Owen-value approximation reduces computational complexity from exponential, O(2^N), to quadratic in N is not supported by a precise description of the approximation scheme. The Owen value with a coalition structure still requires sums over subsets of groups and over subsets within groups; the number of characteristic-function evaluations depends on the partition and the number of samples used by SHAP. The paper does not specify the number of samples, the error tolerance, or how the hierarchical structure B is obtained from constituency parsing. This makes both the computational-efficiency claim and the accuracy of the resulting dense rewards untestable and potentially incorrect.
  4. [Table 1] The SCAR result on IMDB reports a standard deviation of 0.00 across 5 seeds, and the sparse RLHF result on HH-RLHF also reports 0.00. Such zero variance is surprising and needs explanation. If the evaluation is deterministic after convergence (e.g., greedy decoding) and all seeds converge to the same policy, state this explicitly; if the numbers are rounded or the standard deviation is computed incorrectly, provide the correct values. Without clarification, these entries cast doubt on the reliability of the reported variance estimates.
minor comments (7)
  1. [Section 1] In the Introduction, "We then employs the Shapley value" should be "We then employ the Shapley value."
  2. [Section 2.2] "Attention Based Credit (ABC) [9] proposed using the reward model's internal attention weights" should be reworded for grammatical correctness, e.g., "Attention Based Credit (ABC) [9] proposes using..."
  3. [Section 3.2] The text says "satisfies desirable axioms such asefficiency" - this is a typo for "such as efficiency."
  4. [Section 4.1] "Instruction Turning" should be "Instruction Tuning."
  5. [Section 4.2] "the sparse RLHFC-tuned model" should be "the sparse RLHF-tuned model."
  6. [Section 3.4] The description of the Owen-value computation is too vague. Please specify the hierarchical structure B, how it is derived from the constituency parse, and how the partition into groups is defined, so that the complexity and approximation claims can be checked.
  7. [Appendix A] The proof uses both M and N to denote the number of units; please make the notation consistent. Also, v(N) should be v(P) in the displayed equation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the theoretical guarantee is a valid consequence of Shapley efficiency and the empirical claims are tested against independent baselines.

full rationale

The paper's derivation chain is self-contained. Theorem 3.1 is proved by showing that, for every episode, the SCAR total return equals the original sparse return (Appendix A), which follows from the efficiency property of Shapley values (Eq. 5 and the line 'due to the efficiency property'). This is a transparent mathematical proof rather than a circular inference: the dense rewards are not fitted to the optimal policy or to the empirical outcomes, and the theorem does not assume the conclusion it establishes. The central empirical claim—faster convergence and higher final reward versus sparse RLHF, ABC, and Uniform—is evaluated with held-out reward-model scores and LLM-judge win rates, so it is not a restatement of the method's definition. The gap-filled characteristic function v(S) = r_phi(x, y_S) is an unvalidated modeling assumption acknowledged in Section 5, but an approximation or limitation is not a circular reduction. Self-citations (e.g., [8]) appear only as related work and carry no load-bearing argument; Shapley uniqueness and Owen-value results are cited to external classical sources ([45], [39]). No fitted parameter is renamed as a prediction, and no load-bearing premise depends on a self-citation chain.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new physical or architectural entities are introduced; the method rests on standard Shapley/Owen game theory, a domain assumption about partial-sequence scoring, and an unverified approximation claim.

free parameters (3)
  • alpha (Shapley reward weight) = 0.8 (sentiment), 1.0 (summarization), 0.8 (instruction tuning)
    Chosen per task; interpolates between dense Shapley reward and sparse terminal reward; no sensitivity analysis is reported.
  • segmentation granularity = token (IMDB), span (TL;DR), sentence (HH-RLHF)
    Selected per task to control player count N and approximation cost; a modeling choice that affects credit assignment quality.
  • Owen value approximation sample count = not reported
    The number of coalition samples or other SHAP parameters is not given, leaving an unquantified approximation error in the dense rewards.
assumptions (5)
  • standard math Shapley value axioms: efficiency, symmetry, linearity, null player
    Used in Section 3.2 to justify fair credit allocation; efficiency is the key property that makes sum Rshap = r_phi and drives the policy-invariance argument.
  • standard math Potential-Based Reward Shaping preserves optimal policies
    Invoked in Appendix A to prove Theorem 3.1, though the proof actually uses total-return identity rather than a state-based potential function.
  • domain assumption Reward model scores are meaningful on partial sequences with empty-space gaps
    Defines v(S) in Eq. (3); the paper notes in Section 5 that this may fail for rule-based or final-answer-only reward models.
  • domain assumption Syntactic constituents are appropriate credit-assignment units
    Span-level segmentation via constituency parsing in Section 3.4 assumes grouping tokens into phrases preserves the credit information needed for effective dense rewards.
  • ad hoc to paper Owen-value approximation via SHAP is accurate and reduces cost to O(N^2)
    Stated in Section 3.4 without proof or error analysis; for a general neural reward-model characteristic function, exact Owen value computation is not generally polynomial.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SCAR: Shapley Credit Assignment for More Efficient RLHF." pith.science (2026). https://pith.science/paper/ZW7LRDGO

@misc{pith2026250520417,
  author       = {Pith},
  title        = {Pith review of: SCAR: Shapley Credit Assignment for More Efficient RLHF},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZW7LRDGO}},
  note         = {Machine review of arXiv:2505.20417}
}
read the original abstract

Reinforcement Learning from Human Feedback (RLHF) is a widely used technique for aligning Large Language Models (LLMs) with human preferences, yet it often suffers from sparse reward signals, making effective credit assignment challenging. In typical setups, the reward model provides a single scalar score for an entire generated sequence, offering little insight into which token or span-level decisions were responsible for the outcome. To address this, we propose Shapley Credit Assignment Rewards (SCAR), a novel method that leverages Shapley values in cooperative game theory. SCAR distributes the total sequence-level reward among constituent tokens or text spans based on their principled marginal contributions. This creates dense reward signals, crucially, without necessitating the training of auxiliary critique models or recourse to fine-grained human annotations at intermediate generation stages. Unlike prior dense reward methods, SCAR offers a game-theoretic foundation for fair credit attribution. Theoretically, we demonstrate that SCAR preserves the original optimal policy, and empirically, across diverse tasks including sentiment control, text summarization, and instruction tuning, we show that SCAR converges significantly faster and achieves higher final reward scores compared to standard RLHF and attention-based dense reward baselines. Our findings suggest that SCAR provides a more effective and theoretically sound method for credit assignment in RLHF, leading to more efficient alignment of LLMs.

Figures

Figures reproduced from arXiv: 2505.20417 by the authors.

Figure 1
Figure 1. Comparison of reward distribution strategies for an example generated sequence. Sparse [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Average reward per timestep during RLHF training for sentiment control (left), text [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Reward-KL tradeoff on the sentiment control (IMDB) task. The y-axis represents the average [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Rewards/GPU hours curves on the TL;DR dataset. We sampled one run from each method. [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Comparison between token-level and span-level SCAR on the text summarization (TL;DR) [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Element-Aware Group Learning for E-Commerce Image Generation

    cs.CV 2026-08 conditional novelty 6.0 of 10

    EAGLE-GRPO uses kernel ridge regression to split GRPO image rewards across structured prompt elements, and reports better e-commerce image generation results than standard GRPO.

  2. S2T-RLHF: Hierarchical Credit Assignment for Stable Preference-Based RLHF

    cs.AI 2026-05 conditional novelty 6.0 of 10

    S2T-RLHF splits each response-level RLHF reward into sentence shares and then token shares, via bargaining and Dirichlet weighting, yielding steadier training with competitive preference alignment.

  3. Who Gets the Reward & Who Gets the Blame? Evaluation-Aligned Training Signals for Multi-LLM Agents

    cs.MA 2025-11 conditional novelty 5.0 of 10

    A game-theoretic framework converts global success/failure evaluations of multi-LLM agents into signed, credit-conserving per-message training signals.

  4. Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontiers

    cs.AI 2025-02 conditional novelty 5.0 of 10

    A taxonomy-based survey of bidirectional game theory and LLM research, spanning evaluation, alignment, economic competition, and LLM-driven game solving.

Reference graph

Works this paper leans on

67 extracted references · 41 canonical work pages · cited by 4 Pith papers

  1. [1]

    Cooperative games with coalition structures.International Journal of game theory, 3:217–237, 1974

    Robert J Aumann and Jacques H Dreze. Cooperative games with coalition structures.International Journal of game theory, 3:217–237, 1974

    work page 1974

  2. [2]

    An actor-critic algorithm for sequence prediction

    Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. An actor-critic algorithm for sequence prediction. In International Conference on Learning Representations, 2017. URL https://openreview.net/ forum?id=SJDaqqveg

    work page 2017

  3. [3]

    Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    arXiv 2022

  4. [4]

    Unifying count-based exploration and intrinsic motivation

    Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors,Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL https://proceedings.neurips.cc/paper_...

    work page 2016

  5. [5]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. InInternational Conference on Machine Learning, pages 2397–2430. PMLR, 2023

    2023

  6. [6]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    work page 1901

  7. [7]

    Ask the right questions: Active question reformulation with reinforcement learning

    Christian Buck, Jannis Bulian, Massimiliano Ciaramita, Wojciech Gajewski, Andrea Gesmundo, Neil Houlsby, and Wei Wang. Ask the right questions: Active question reformulation with reinforcement learning. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=S1CChZ-CZ

    work page 2018

  8. [8]

    Enhancing reinforcement learning with dense rewards from language model critic

    Meng Cao, Lei Shu, Lei Yu, Yun Zhu, Nevan Wichers, Yinxiao Liu, and Lei Meng. Enhancing reinforcement learning with dense rewards from language model critic. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9119–9138, Miami, Florida, USA, November

    work page 2024

Show all 67 references
  1. [9]

    Dense reward for free in reinforcement learning from human feedback

    Alex James Chan, Hao Sun, Samuel Holt, and Mihaela van der Schaar. Dense reward for free in reinforcement learning from human feedback. InForty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=eyxVRMrZ4m

    2024

  2. [10]

    Deep reinforcementlearningfromhumanpreferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcementlearningfromhumanpreferences. In I.Guyon, U. VonLuxburg, S.Bengio, H.Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors,Advances in Neural Information Processing S...

    2017

  3. [11]

    Raft: Reward ranked finetuning for generative foundation model alignment.arXiv preprint arXiv:2304.06767, 2023

    Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment.arXiv preprint arXiv:2304.06767, 2023

    2023 arXiv

  4. [12]

    Openllama: An open reproduction of llama, May 2023

    Xinyang Geng and Hao Liu. Openllama: An open reproduction of llama, May 2023. URL https://github.com/openlm-research/open_llama

    2023

  5. [13]

    Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space

    Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors,Proceedings of the 2022 Conference on Empirical Methods in Natur...

    2022

  6. [14]

    Tokenshap: Interpreting large language models with monte carlo shapley value estimation.arXiv preprint arXiv:2407.10114, 2024

    Roni Goldshmidt and Miriam Horovicz. Tokenshap: Interpreting large language models with monte carlo shapley value estimation.arXiv preprint arXiv:2407.10114, 2024

    2024 arXiv

  7. [15]

    A duality approach for regret minimization in average-award ergodic markov decision processes

    Hao Gong and Mengdi Wang. A duality approach for regret minimization in average-award ergodic markov decision processes. In Alexandre M. Bayen, Ali Jadbabaie, George Pappas, Pablo A. Parrilo, Benjamin Recht, Claire Tomlin, and Melanie Zeilinger, editors,Proceedings of the 2nd ...

    2020

  8. [16]

    Dealing with sparse rewards in reinforcement learning

    Joshua Hare. Dealing with sparse rewards in reinforcement learning. arXiv preprint arXiv:1910.09281, 2019

    1910 arXiv

  9. [17]

    Shed: Shapley-based automated dataset refinement for instruction fine-tuning.arXiv preprint arXiv:2405.00705, 2024

    Yexiao He, Ziyao Wang, Zheyu Shen, Guoheng Sun, Yucong Dai, Yongkai Wu, Hongyi Wang, and Ang Li. Shed: Shapley-based automated dataset refinement for instruction fine-tuning.arXiv preprint arXiv:2405.00705, 2024

    2024 arXiv

  10. [18]

    Deep reinforcement learning that matters

    Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. InProceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligenc...

    2018

  11. [19]

    The n+ implementation details of RLHF with PPO: A case study on TL;DR summariza- tion

    Shengyi Huang, Michael Noukhovitch, Arian Hosseini, Kashif Rasul, Weixun Wang, and Lewis Tunstall. The n+ implementation details of RLHF with PPO: A case study on TL;DR summariza- tion. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum? id=kHO2ZTa8e3. 11

    2024

  12. [20]

    Sarthak Jain and Byron C. Wallace. Attention is not Explanation. In Jill Burstein, Christy Doran, and Thamar Solorio, editors,Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume ...

    2019 doi

  13. [21]

    Motif: Intrinsic motivation from artificial intelligence feedback

    Martin Klissarov, Pierluca D’Oro, Shagun Sodhani, Roberta Raileanu, Pierre-Luc Bacon, Pascal Vincent, Amy Zhang, and Mikael Henaff. Motif: Intrinsic motivation from artificial intelligence feedback. In The Twelfth International Conference on Learning Representations, 2024. URL...

    2024

  14. [22]

    Learning explainable dense reward shapes via bayesian optimization.arXiv preprint arXiv:2504.16272, 2025

    Ryan Koo, Ian Yang, Vipul Raheja, Mingyi Hong, Kwang-Sung Jun, and Dongyeop Kang. Learning explainable dense reward shapes via bayesian optimization.arXiv preprint arXiv:2504.16272, 2025

    2025 arXiv

  15. [23]

    Learning to solve the credit assignment problem.arXiv preprint arXiv:1906.00889, 2019

    Benjamin James Lansdell, Prashanth Ravi Prakash, and Konrad Paul Kording. Learning to solve the credit assignment problem.arXiv preprint arXiv:1906.00889, 2019

    1906 arXiv

  16. [24]

    Rlaif: Scaling reinforcement learning from human feedback with ai feedback

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Ren Lu, Thomas Mesnard, Johan Ferret, Colton Bishop, Ethan Hall, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. 2023

    2023

  17. [25]

    Deep reinforcement learning for dialogue generation

    Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep reinforcement learning for dialogue generation. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, ...

    2016 doi

  18. [26]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 5 2023

    2023

  19. [27]

    Let’s verify step by step.arXiv preprint arXiv:2305.20050, 2023

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step.arXiv preprint arXiv:2305.20050, 2023

    2023 arXiv

  20. [28]

    Prompt valuation based on shapley values.arXiv preprint arXiv:2312.15395, 2023

    Hanxi Liu, Xiaokai Mao, Haocheng Xia, Jian Lou, and Jinfei Liu. Prompt valuation based on shapley values.arXiv preprint arXiv:2312.15395, 2023

    2023 arXiv

  21. [29]

    A unified approach to interpreting model predic- tions

    Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predic- tions. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 4765–4774. Curran Assoc...

    2017

  22. [30]

    Maas, Raymond E

    Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Dekang Lin, Yuji Matsumoto, and Rada Mihalcea, editors,Proceedings of the 49th Annual Meeting of the Association for Computational L...

    2011

  23. [31]

    Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz

    Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank.Computational Linguistics, 19(2):313–330, 1993. URL https://aclanthology.org/J93-2004/

    1993

  24. [32]

    Locating and editing factual associations in GPT

    Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URLhttps://openreview. net/for...

    2022

  25. [33]

    Explaining large language models decisions using shapley values.arXiv preprint arXiv:2404.01332, 2024

    Behnam Mohammadi. Explaining large language models decisions using shapley values.arXiv preprint arXiv:2404.01332, 2024. 12

    2024 arXiv

  26. [34]

    Policy invariance under reward transformations: Theory and application to reward shaping

    Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. InIcml, volume 99, pages 278–287. Citeseer, 1999

    1999

  27. [35]

    Ng, Daishi Harada, and Stuart J

    Andrew Y. Ng, Daishi Harada, and Stuart J. Russell. Policy invariance under reward transforma- tions: Theory and application to reward shaping. InProceedings of the Sixteenth International Conference on Machine Learning, ICML ’99, page 278–287, San Francisco, CA, USA, 1999. Mo...

    1999

  28. [36]

    Reward augmented maximum likelihood for neural struc- tured prediction

    Mohammad Norouzi, Samy Bengio, zhifeng Chen, Navdeep Jaitly, Mike Schuster, Yonghui Wu, and Dale Schuurmans. Reward augmented maximum likelihood for neural struc- tured prediction. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Inform...

    2016

  29. [37]

    Bellemare, Aäron van den Oord, and Rémi Munos

    Georg Ostrovski, Marc G. Bellemare, Aäron van den Oord, and Rémi Munos. Count-based exploration with neural density models. In Doina Precup and Yee Whye Teh, editors,Proceedings of the 34th International Conference on Machine Learning, volume 70 ofProceedings of Machine Learni...

    2017

  30. [38]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Chris- tiano, Jan Leike,...

    2022

  31. [39]

    Values of games with a priori unions

    Guilliermo Owen. Values of games with a priori unions. InMathematical economics and game theory: Essays in honor of Oskar Morgenstern, pages 76–88. Springer, 1977

    1977

  32. [40]

    Efros, and Trevor Darrell

    Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Doina Precup and Yee Whye Teh, editors,Proceedings of the 34th International Conference on Machine Learning, volume 70 ofProceedings of Machine Le...

    2017

  33. [41]

    Language models are unsupervised multitask learners

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019

    2019

  34. [42]

    Susskind, and Etai Littwin

    Noam Razin, Hattie Zhou, Omid Saremi, Vimal Thilak, Arwen Bradley, Preetum Nakkiran, Joshua M. Susskind, and Etai Littwin. Vanishing gradients in reinforcement finetuning of language models. In The Twelfth International Conference on Learning Representations, 2024. URL https:/...

    2024

  35. [43]

    Framework of automatic text summarization using rein- forcement learning

    Seonggi Ryang and Takeshi Abekawa. Framework of automatic text summarization using rein- forcement learning. In Jun’ichi Tsujii, James Henderson, and Marius Paşca, editors,Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computa-...

    2012

  36. [44]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  37. [45]

    Stochastic games

    Lloyd S Shapley. Stochastic games. Proceedings of the national academy of sciences, 39(10): 1095–1100, 1953. 13

    1953

  38. [46]

    Learning to summarize with human feed- back

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feed- back. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Ad- vances in Neural In...

    2020

  39. [47]

    Efficient shapley value-based non-uniform pruning of large language models

    Chuan Sun, Han Yu, and Lizhen Cui. Efficient shapley value-based non-uniform pruning of large language models. arXiv preprint arXiv:2505.01731, 2025

    2025 arXiv

  40. [48]

    Temporal credit assignment in reinforcement learning

    Richard Stuart Sutton. Temporal credit assignment in reinforcement learning. University of Massachusetts Amherst, 1984

    1984

  41. [49]

    #exploration: A study of count-based exploration for deep reinforcement learning

    Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. #exploration: A study of count-based exploration for deep reinforcement learning. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S....

    2017

  42. [50]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  43. [51]

    TL;DR: Mining Reddit to learn automatic summarization

    Michael V"olske, Martin Potthast, Shahbaz Syed, and Benno Stein. TL;DR: Mining Reddit to learn automatic summarization. InProceedings of the Workshop on New Frontiers in Summarization, pages 59–63, Copenhagen, Denmark, September 2017. Association for Computational Linguistics....

    2017 doi

  44. [52]

    Attention is not not explanation

    Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors,Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural La...

    2019 doi

  45. [53]

    Fine-grained human feedback gives better rewards for language model training

    Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S....

    2023

  46. [54]

    Fine-grained human feedback gives better rewards for language model training.arXiv preprint arXiv:2306.01693, 2023

    Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training.arXiv preprint arXiv:2306.01693, 2023

    2023 arXiv

  47. [55]

    TLCR: Token-level continuous reward for fine-grained reinforcement learning from human feedback

    Eunseop Yoon, Hee Suk Yoon, SooHwan Eom, Gunsoo Han, Daniel Nam, Daejin Jo, Kyoung-Woon On, Mark Hasegawa-Johnson, Sungwoong Kim, and Chang Yoo. TLCR: Token-level continuous reward for fine-grained reinforcement learning from human feedback. In Lun-Wei Ku, Andre Martins, and V...

    2024 doi

  48. [56]

    Towards sample efficient reinforcement learning

    Yang Yu. Towards sample efficient reinforcement learning. InIJCAI, pages 5739–5743, 2018

    2018

  49. [57]

    Gonzalez, and Ion Stoica

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. InThirty-seventh Conference on Neural Inform...

    2023

  50. [58]

    Summary A

    Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gra- dient methods. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors,Advances in Neural Information Processing Systems, volume 31. Curran As- socia...

    2018

  51. [61]

    Accuracy & Faithfulness: • Does the summary accurately represent the main points of the original document? • Does it avoid introducing new information or misinterpreting facts from the document (hallucinations)?

  52. [62]

    Coverage & Comprehensiveness: • Does the summary cover the most important information and key takeaways from the original document? • Are there any critical omissions of essential information?

  53. [63]

    Conciseness & Succinctness: • Is the summary brief and to the point, avoiding unnecessary jargon, redundancy, or overly verbose phrasing, while still capturing essential information? • Is it significantly shorter than the original document, as a good summary should be?

  54. [64]

    Clarity & Readability: • Is the summary well-written, grammatically correct, easy to understand, and fluent? • Is the language clear and precise?

  55. [65]

    Coherence: 19 • Do the sentences in the summary flow logically? Does it make sense as a standalone piece of text? • Is there a logical structure to the summary? Input: Original Document: {original_document} Summary A: {summary_A} Summary B: {summary_B} Instructions for your response:

  56. [66]

    Regarding Accuracy, Summary A does X well, while Summary B struggles with Y

    Reasoning: • First, briefly state your understanding of the main purpose or key points of theOriginal Document. • Then, provide a step-by-step comparative analysis of Summary A and Summary B based on the criteria listed above (Accuracy, Coverage, Conciseness, Clarity, Coherenc...

  57. [67]

    Overall Decision:

    Overall Decision: • After your detailed reasoning, clearly state which summary you believe is better overall and why, making a holistic judgment. If they are of very comparable quality, or if one excels in some areas while the other excels in others making a clear choice diffi...

  58. [2022]

    doi: 10.18653/v1/2022.emnlp-main.3

    Association for Computational Linguistics. doi: 10.18653/v1/2022.emnlp-main.3. URL https://aclanthology.org/2022.emnlp-main.3/

    2022 doi

  59. [2024]

    doi: 10.18653/v1/2024.emnlp-main.515

    Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.515. URL https://aclanthology.org/2024.emnlp-main.515/

    2024 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.