Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Learning Explainable Dense Reward Shapes via Bayesian Optimization

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that per-token reward shaping from SHAP/LIME explanations, with weights tuned by Bayesian optimization, preserves the original RLHF objective while speeding up training and improving downstream benchmark performance.

desk verdict A plausible new dense-reward recipe for RLHF, but the policy-invariance proof doesn't match the implemented reward and the experiments are too noisy to carry the claim. read the letter →

arxiv 2504.16272 v1 pith:TIMPT5RJ submitted 2025-04-22 cs.LG

Ryan Koo , Ian Yang , Vipul Raheja , Mingyi Hong , Kwang-Sung Jun , Dongyeop Kang This is my paper · ORCID
classification cs.LG
keywords RLHFrewardshapingtoken-levelcreditassignmentadditivefeatureattributionSHAPLIMEBayesianoptimizationpolicyinvariance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the sparse, sequence-level reward used in reinforcement learning from human feedback can be replaced by a dense, token-level reward without changing what the policy optimizes. The proposed shaping takes per-token credit scores from additive feature attribution methods such as SHAP and LIME, combines them with learned weights, and broadcasts the scalar reward across the sequence. The weights are selected in an outer Bayesian-optimization loop while PPO trains the policy in the inner loop. The authors argue that any shaping built from an additive feature attribution function is a potential-based reward shaping, so the optimal policy of the original reward is preserved, and report that the BO-tuned shapes improve training speed, value-function stability, and downstream benchmark performance. If right, this gives RLHF a principled way to give language models fine-grained per-token feedback instead of a single final-token surrogate.

What carries the argument

The load-bearing object is the additive feature attribution model, $g(z') = \phi_0 + \sum_{i=1}^{M}\phi_i z'_i$, a locally fitted linear model whose coefficients $\phi_i$ are per-token credit scores for the reward model's prediction. The paper combines these scores with a constant into $E=[E_{\mathrm{SHAP}},\mathbf{1}]$ and shapes the reward as $r'(s,a)=W^\top E \cdot r(s,a)$, with $W$ drawn from a Bayesian-optimization loop (a Gaussian-process surrogate with a log noisy expected improvement acquisition function) that maximizes the validation reward of the PPO-trained policy. The proof machinery is potential-based reward shaping: defining $\Phi(s)=w_2\sum_{i=1}^{s}\phi_i$, the added term $w_2\phi_{s'}$ is claimed to equal $\Phi(s')-\Phi(s)$, which is the condition under which the optimal policy is unchanged.

What would settle it

Run the shaped reward of Eq. 4 on a small MDP whose optimal policy is known, solve both the shaped and unshaped MDPs exhaustively, and check whether the optimal policies coincide for all sampled weight vectors; a single weight vector where they differ falsifies policy invariance.

Watch

Extended reading notes

Core claim

On its own terms, the paper's claim is that reward shaping via additive feature attribution is both safe and useful for RLHF. The shaped reward is written as $r'(s,a) = W^\top E \cdot r(s,a)$ with $E = [E_{\mathrm{SHAP}}, \mathbf{1}]$, so the scalar reward $r(s,a)$ is multiplied by a convex combination of token-level explanation scores and the constant 1; in practice the explanation vector is softmax-normalized and the scalar reward is broadcast over the sequence. The paper proves (Appendix B) that this family is potential-based by setting $\Phi(s) = w_2\sum_{i=1}^{s}\phi_i(R,x)$, so that the shaping term equals $\Phi(s')-\Phi(s)$, and invokes the classic result that potential-based shaping preserves the optimal policy. Empirically, the authors show that unblended SHAP/LIME rewards stabilize PPO value loss, and that BO-tuned combinations such as SHAP+LIME achieve higher validation reward and better AlpacaEval-2 and MT-Bench win rates than sparse RLHF and attention-based dense reward baselines.

Load-bearing premise

The policy-invariance proof applies to a reward built by adding a potential-based bonus, but the method's actual reward is built by multiplying the sparse reward by a weighted blend of token scores; if those two constructions are not equivalent, the guarantee does not cover the trained method.

Editorial extensions

If this is right

  • Practitioners can add per-token feedback from any additive feature attribution method without re-deriving the task objective, because the shaping is claimed to preserve the original optimal policy.
  • Combining several token-importance signals (SHAP, LIME, attention) and tuning their convex weights via Bayesian optimization can outperform any single dense-reward signal within a modest trial budget.
  • Dense explanation rewards make PPO value learning more stable, shown by lower value-head loss, which can reduce the variance of RLHF training runs.
  • Because the BO outer loop is a hyperparameter search over reward shapes, the final policy is only as good as the validation reward used to select weights; adding more signal sources requires more trials before performance is guaranteed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test is to compare BO-selected weights with an exhaustive grid over $w_1,w_2$ on a short-sequence task; if a grid matches BO, the gains come from tuning the weights rather than from the acquisition function.
  • The same additive-attribution recipe could be applied to process-supervised reasoning tasks: per-token SHAP scores of a verifier would give step-level credit for chain-of-thought, and the BO outer loop could tune the balance between step-level and final rewards.
  • If the invariance claim is repaired to cover the multiplicative form, the framework generalizes to any token-importance signal, such as saliency maps, gradient norms, or learned critics, provided the shaping can be written as a potential difference.
  • The static weight vector could become prompt-conditional, turning the outer loop into a meta-learning problem, a direction the authors note in the conclusion.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a reward-shaping method for RLHF in which per-token feature attributions (SHAP, LIME, and attention) from the reward model are combined into a dense token-level reward, with combination weights selected by Bayesian optimization in an outer loop while PPO trains in the inner loop. The authors claim that additive feature attribution functions preserve the optimal policy via potential-based reward shaping, and report improved reward scores and win rates on HH-RLHF and Ultrafeedback, including evaluations on AlpacaEval-2 and MTBench.

Significance. If the theoretical claim were established, the paper would provide a principled way to densify RLHF rewards without changing the intended optimal policy, and the combination of explainability with Bayesian optimization is a reasonable research direction. The release of code and the use of independent benchmarks such as AlpacaEval-2 and MTBench are strengths. However, the policy-invariance proof as stated does not apply to the implemented reward transformation, and the empirical evaluation rests on single runs with a very small training budget, so the central claims are not yet supported.

major comments (3)
  1. [Eq. (4); Appendix B] The central policy-invariance proof does not apply to the reward actually used. Eq. (4) defines the shaped reward multiplicatively as r'(s,a) = (w1 E_SHAP(s,a) + w2) r(s,a), and the implementation described in Section 3.1 softmax-normalizes the per-token scores before broadcasting the scalar reward. Appendix B instead proves invariance for an additive potential-based shaping R' = R + F with F = gamma Phi(s') - Phi(s), identifying Phi(s) = w2 sum_{i=1}^s phi_i. No step connects the multiplicative/softmax form to R + F. Eq. (7) also assumes that cumulative attribution up to state s' approximates the prefix reward, which is not implied by SHAP local accuracy; local accuracy only fixes the sum over all tokens. The correct weaker argument--that softmax normalization makes the per-token rewards sum to the original scalar reward for every trajectory, so the finite-horizon total return is unchanged--is not the argument given and does not justify Eq. (4) as written. Without normalization, the total shaped return is (w1 sum_i phi_i + w2 M) R, which is not generally invariant. This is a load-bearing gap because policy invariance is the paper's main theoretical contribution.
  2. [Section 4, Table 1] The empirical evaluation does not support the strength of the empirical claims. Table 1 reports a single run per method with no multiple seeds or error bars, and the BO training budget is only 25 trials with 80 PPO samples per trial. Under this noise level, differences such as BO-SHAP-LIME (6.58) versus BO-SHAP-Attn (6.47) versus LIME* (5.86) on HH-RLHF cannot be distinguished from random variation. The paper itself acknowledges that the d=4 combination degrades performance (Section 4.2), which undercuts the claim that BO 'balances' token-level rewards. Rerunning with several seeds and reporting means with standard deviations or confidence intervals for the headline numbers is necessary before the empirical conclusions can be assessed.
  3. [Sections 3.2, 4.1, 4.2] The outer BO loop is optimized on average validation reward from the same reward model whose test-split predictions form the 'Score' column of Table 1. This creates selection pressure toward reward-model overfitting, and the independent benchmarks are the only protection against circularity. The paper should report the gap between validation and test reward for the BO-selected weights and should consider using a separate reward model or a preference-based metric for the outer objective. As written, the 'Score' improvements are not a clean measure of alignment quality.
minor comments (5)
  1. [Appendix A, Example 1] The numerical values used in the example are inconsistent with the table: f({1,3}) is given as 0.6 but the table lists 1.3, and f({2,3}) is given as 0.7 but the table lists 1.7. This makes the example's arithmetic unreliable.
  2. [Section 3.1, Eq. (4)] Please define the dimensions of E and W and clarify whether r(s,a) is a scalar or a per-token vector; the notation W^T E · r is ambiguous as written.
  3. [Section 3.1, Section 4.1] The paper states that sampled weights satisfy sum_i w_i = 1, but the manual baseline sets w = 0.8 for a single dense reward type; please clarify whether the manual baseline also includes a constant term that sums to one.
  4. [Figure 4] The caption says the shading represents the standard error (95% confidence interval), but the paper does not report the number of runs or seeds used to compute this quantity; please clarify.
  5. [Minor wording] There are several typographical issues: 'backpropogate' should be 'backpropagate' in Section 4.2, 'LL AMA' appears in Section 4, and 'Also inspects' in the Related Work paragraph should have consistent capitalization.

Circularity Check

1 steps flagged · score 6.0 of 10

The policy-invariance proof in Appendix B defines the potential after assuming cumulative SHAP equals the reward, making the invariance true by construction rather than derived from additive attribution.

  1. self definitional [Appendix B, Eqs. 7-9 (with Eq. 4 in Section 3.1)]
    "We also have by locality from Eq. 1 that g(x′)≈ Rϕ(s, a, s′) or: ∑_{i=1}^{s′} φ_i(f, x)≈ R(s, a, s′) (7) ... Then, by substituting Eq. 4 for F with γ=1, we have: Φ(s′)−Φ(s) = w2φ_{s′}(R, x) (8) Then defining Φ(s) = w2∑_{i=1}^s φ_i(R, x) we have: Φ(s′)−Φ(s) = w2∑_{i=1}^{s′}φ_i(R, x)−w2∑_{i=1}^s φ_i(R, x) = w2φ_{s′}(R, x) (9)"

    The proof needs to show that the shaped reward is R+F with F=γΦ(s′)−Φ(s). Instead, Eq. 7 simply asserts that the scalar reward R(s,a,s′) equals the cumulative SHAP sum up to prefix s′, which is not a consequence of additive local accuracy (local accuracy only fixes the sum over all tokens). Defining Φ as w2 times that cumulative sum then makes F=Φ(s′)−Φ(s)=w2φ_{s′} true by algebra; the potential-based structure is manufactured after the fact. The proof never derives Eq. 7 from SHAP/LIME properties, and it never connects this additive R+F form to the implemented multiplicative r′=(w1 E_SHAP + w2)r of Eq. 4. Thus the policy-invariance conclusion is assumed in Eq. 7 and made true by construction, rather than derived from the stated first principles.

full rationale

The central theoretical claim — that additive feature attribution methods preserve the optimal policy via potential-based shaping — is circular at a specific point: Appendix B assumes that cumulative SHAP values equal the prefix reward (Eq. 7), then defines the potential as that cumulative sum, so the resulting F is a potential difference by definition. This identification is not implied by the local-accuracy property of SHAP/LIME, and the proof does not bridge the additive R+F form to the multiplicative reward shape actually used in Eq. 4. The rest of the paper is largely self-contained: the BO weights are optimized against validation reward, but the headline empirical results are also checked on independent benchmarks (AlpacaEval-2, MTBench), and the few self-citations (Hayati et al., De Langis & Kang, Kim et al.) are not load-bearing. The score of 6 reflects that the theoretical guarantee partially reduces to an assumed equality plus a definition, while the empirical contribution retains independent content.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central mechanism introduces BO-learned weights as free parameters that are fit to the validation reward. The theoretical policy-invariance claim rests on an unproven identification of Eq. 4 with potential-based shaping (Appendix B). No new entities are postulated.

free parameters (2)
  • BO weights w (one per explainability source, e.g., w_SHAP, w_LIME, w_Attn, w_constant) = Learned per experiment; Figure 5 shows trajectories, final values not tabulated
    Optimized by Bayesian optimization to maximize validation reward; these weights directly determine the token-level reward shape.
  • Manual weight w=0.8 = 0.8
    Used for the non-BO SHAP* and LIME* baselines, following the weighting scheme in Chan et al. (2024); not optimized.
assumptions (4)
  • domain assumption The local linear model g(z') in additive feature attribution (Eq. 1) approximates the reward model's prediction f(h_x(z')) well enough for token-level credit assignment.
    The method relies on SHAP/LIME attributions being informative about token contributions; the paper itself notes these methods are noisy (Section 4.2).
  • ad hoc to paper The shaping function in Eq. 4 is an additive potential-based reward shaping function R' = R + F, so the optimal policy is invariant under Ng et al. (1999).
    Appendix B assumes this identification, but Eq. 4 is multiplicative (r' = (w1 * E + w2) * r). The proof analyzes a different additive form and does not bridge this gap.
  • domain assumption The softmax normalization of token scores preserves the total per-episode reward sum, so the total return for every trajectory is unchanged.
    This is implied by the description in Section 3.1 but never stated explicitly; it is the simplest route to policy invariance.
  • domain assumption The PPO training with 80 samples per BO trial adequately reflects the quality of the reward shape for BO's objective.
    The lower-level optimization in each BO trial is extremely short; the paper acknowledges computational limits and treats the objective as noisy, but still assumes signal.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning Explainable Dense Reward Shapes via Bayesian Optimization." pith.science (2026). https://pith.science/paper/TIMPT5RJ

@misc{pith2026250416272,
  author       = {Pith},
  title        = {Pith review of: Learning Explainable Dense Reward Shapes via Bayesian Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TIMPT5RJ}},
  note         = {Machine review of arXiv:2504.16272}
}
read the original abstract

Current reinforcement learning from human feedback (RLHF) pipelines for large language model (LLM) alignment typically assign scalar rewards to sequences, using the final token as a surrogate indicator for the quality of the entire sequence. However, this leads to sparse feedback and suboptimal token-level credit assignment. In this work, we frame reward shaping as an optimization problem focused on token-level credit assignment. We propose a reward-shaping function leveraging explainability methods such as SHAP and LIME to estimate per-token rewards from the reward model. To learn parameters of this shaping function, we employ a bilevel optimization framework that integrates Bayesian Optimization and policy training to handle noise from the token reward estimates. Our experiments show that achieving a better balance of token-level reward attribution leads to performance improvements over baselines on downstream tasks and finds an optimal policy faster during training. Furthermore, we show theoretically that explainability methods that are feature additive attribution functions maintain the optimal policy as the original reward.

Figures

Figures reproduced from arXiv: 2504.16272 by the authors.

Figure 1
Figure 1. Overview of the bilevel optimization setup to find the best reward shape and the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Redistribution sequence of the scalar reward prediction over the explanation [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. (Left) The mean training reward per timestep with increasing BO dimensionality. (Middle) The mean validation reward over each BO trial. The highlighted dots indicate the best validation reward received at trial n. (Right) The average dense reward attribution over each trial for SHAP + ATTN. The highlighted row indicates the shape in trial 17 that received the highest validation reward. first five trials to build a p… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Helpfulness. (Left) The average training reward per timestep. (Right) The average value head loss per timestep. The shading represents the standard error (95% confidence interval) as training progresses. We first analyze the impact of explanation-based rewards in opti￾…
Figure 5
Figure 5. Figure 5: (Top) The weight transition between trials for SHAPley scores. (Bottom) The weight transition between trials for LIME scores. The black boxes indicate the ”best weights” sampled by the BO model. Credit assignment with explainability helps exploration Before applying an…
Figure 6
Figure 6. Figure 6: (Ultrafeedback) The top left represents the baseline per-token reward without shaping. The color of each token represents the reward received, with a darker color representing a higher proportion of the reward assigned. A more uniform coloring indicates a more uniform …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SCAR: Shapley Credit Assignment for More Efficient RLHF

    cs.AI 2025-05 conditional novelty 5.0 of 10

    SCAR redistributes the terminal RLHF reward to tokens and spans via Shapley values, preserving the total return while improving training efficiency and final reward across three LLM alignment tasks.

Reference graph

Works this paper leans on

62 extracted references · 17 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Searching for optimal solutions with LLM s via bayesian optimization

    Dhruv Agarwal, Manoj Ghuhan Arivazhagan, Rajarshi Das, Sandesh Swamy, Sopan Khosla, and Rashmi Gangadharaiah. Searching for optimal solutions with LLM s via bayesian optimization. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=aVfDrl7xDV

    work page 2025

  3. [3]

    Unexpected improvements to expected improvement for bayesian optimization, 2025

    Sebastian Ament, Samuel Daulton, David Eriksson, Maximilian Balandat, and Eytan Bakshy. Unexpected improvements to expected improvement for bayesian optimization, 2025. URL https://arxiv.org/abs/2310.20708

    arXiv 2025

  4. [4]

    Bayesian optimization with llm-based acquisition functions for natural language preference elicitation

    David Austin, Anton Korikov, Armin Toroghi, and Scott Sanner. Bayesian optimization with llm-based acquisition functions for natural language preference elicitation. In Proceedings of the 18th ACM Conference on Recommender Systems, RecSys '24, pp.\ 74–83, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400705052. doi:10.1145/3640457...

    arXiv 2024

  5. [5]

    Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, ...

    arXiv 2022

  6. [6]

    Ae: A domain-agnostic platform for adaptive experimentation

    Eytan Bakshy, Lili Dworkin, Brian Karrer, Konstantin Kashin, Ben Letham, Ashwin Murthy, and Shaun Singh. Ae: A domain-agnostic platform for adaptive experimentation. In NeurIPS Systems for ML Workshop, 2018. URL http://learningsys.org/nips18/assets/papers/87CameraReadySubmissionAE

    work page 2018

  7. [7]

    Mechanistic interpretability for ai safety--a review

    Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety--a review. arXiv preprint arXiv:2404.14082, 2024

    arXiv 2024

  8. [9]

    Enhancing reinforcement learning with dense rewards from language model critic

    Meng Cao, Lei Shu, Lei Yu, Yun Zhu, Nevan Wichers, Yinxiao Liu, and Lei Meng. Enhancing reinforcement learning with dense rewards from language model critic. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.\ 9119--9138, Miami, Florida, USA, November 202...

    doi:10.18653/v1/2024.emnlp-main.515 2024

Show all 62 references
  1. [10]

    Chan, Hao Sun, Samuel Holt, and Mihaela van der Schaar

    Alex J. Chan, Hao Sun, Samuel Holt, and Mihaela van der Schaar. Dense reward for free in reinforcement learning from human feedback, 2024. URL https://arxiv.org/abs/2402.00782

    2024 arXiv

  2. [11]

    Rlhf deciphered: A critical analysis of reinforcement learning from human feedback for llms, 2024

    Shreyas Chaudhari, Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, Karthik Narasimhan, Ameet Deshpande, and Bruno Castro da Silva. Rlhf deciphered: A critical analysis of reinforcement learning from human feedback for llms, 2024. URL https://arxiv.org/abs...

    2024 arXiv

  3. [12]

    I nstruct Z ero: Efficient instruction optimization for black-box large language models

    Lichang Chen, Jiuhai Chen, Tom Goldstein, Heng Huang, and Tianyi Zhou. I nstruct Z ero: Efficient instruction optimization for black-box large language models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Ber...

    2024

  4. [13]

    Improving large language models via fine-grained reinforcement learning with minimum editing constraint

    Zhipeng Chen, Kun Zhou, Xin Zhao, Junchen Wan, Fuzheng Zhang, Di Zhang, and Ji-Rong Wen. Improving large language models via fine-grained reinforcement learning with minimum editing constraint. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Associatio...

    2024 doi

  5. [14]

    Ultrafeedback: Boosting language models with scaled ai feedback, 2024

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with scaled ai feedback, 2024. URL https://arxiv.org/abs/2310.01377

    2024 arXiv

  6. [15]

    Osborne, Enlu Zhou, and Eytan Bakshy

    Samuel Daulton, Sait Cakmak, Maximilian Balandat, Michael A. Osborne, Enlu Zhou, and Eytan Bakshy. Robust multi-objective bayesian optimization under input noise, 2022. URL https://arxiv.org/abs/2202.07549

    2022 arXiv

  7. [16]

    A comparative study on textual saliency of styles from eye tracking, annotations, and language models

    Karin De Langis and Dongyeop Kang. A comparative study on textual saliency of styles from eye tracking, annotations, and language models. arXiv preprint arXiv:2212.09873, 2022

    2022 arXiv

  8. [17]

    Raft: Reward ranked finetuning for generative foundation model alignment, 2023

    Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment, 2023. URL https://arxiv.org/abs/2304.06767

    2023 arXiv

  9. [18]

    Hashimoto

    Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators, 2025. URL https://arxiv.org/abs/2404.04475

    2025 arXiv

  10. [19]

    Implementation matters in deep policy gradients: A case study on ppo and trpo, 2020

    Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on ppo and trpo, 2020. URL https://arxiv.org/abs/2005.12729

    2020 arXiv

  11. [20]

    Fröhlich, Edgar D

    Lukas P. Fröhlich, Edgar D. Klenske, Julia Vinogradska, Christian Daniel, and Melanie N. Zeilinger. Noisy-input entropy search for efficient robust bayesian optimization, 2020. URL https://arxiv.org/abs/2002.02820

    2020 arXiv

  12. [21]

    Reward shaping to mitigate reward hacking in rlhf, 2025

    Jiayi Fu, Xuandong Zhao, Chengyuan Yao, Heng Wang, Qi Han, and Yanghua Xiao. Reward shaping to mitigate reward hacking in rlhf, 2025. URL https://arxiv.org/abs/2502.18770

    2025 arXiv

  13. [22]

    Scaling laws for reward model overoptimization, 2022

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization, 2022. URL https://arxiv.org/abs/2210.10760

    2022 arXiv

  14. [23]

    B ayesian calibration of win rate estimation with LLM evaluators

    Yicheng Gao, Gonghan Xu, Zhe Wang, and Arman Cohan. B ayesian calibration of win rate estimation with LLM evaluators. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.\ 4757-...

    2024 doi

  15. [24]

    Beyond imitation: Leveraging fine-grained quality signals for alignment

    Geyang Guo, Ranchi Zhao, Tianyi Tang, Wayne Xin Zhao, and Ji-Rong Wen. Beyond imitation: Leveraging fine-grained quality signals for alignment. arXiv preprint arXiv:2311.04072, 2023

    2023 arXiv

  16. [25]

    Kakade, and Sergey Levine

    Abhishek Gupta, Aldo Pacchiano, Yuexiang Zhai, Sham M. Kakade, and Sergey Levine. Unpacking reward shaping: Understanding the benefits of reward engineering on sample complexity, 2022. URL https://arxiv.org/abs/2210.09579

    2022 arXiv

  17. [26]

    Sathiya Keerthi

    Aman Gupta, Shao Tang, Qingquan Song, Sirou Zhu, Jiwoo Hong, Ankan Saha, Viral Gupta, Noah Lee, Eunki Kim, Siyu Zhu, Parag Agrawal, Natesh Pillai, and S. Sathiya Keerthi. Alphapo -- reward shape matters for llm alignment, 2025. URL https://arxiv.org/abs/2501.03884

    2025 arXiv

  18. [27]

    Does bert learn as humans perceive? understanding linguistic styles through lexica

    Shirley Anugrah Hayati, Dongyeop Kang, and Lyle Ungar. Does bert learn as humans perceive? understanding linguistic styles through lexica. arXiv preprint arXiv:2109.02738, 2021

    2021 arXiv

  19. [28]

    Learning to utilize shaping rewards: a new approach of reward shaping

    Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang, Yingfeng Chen, Jianye Hao, Feng Wu, and Changjie Fan. Learning to utilize shaping rewards: a new approach of reward shaping. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '...

    2020

  20. [29]

    Training language models to generate text with citations via fine-grained rewards

    Chengyu Huang, Zeqiu Wu, Yushi Hu, and Wenya Wang. Training language models to generate text with citations via fine-grained rewards. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistic...

    2024 doi

  21. [30]

    Sarthak Jain and Byron C. Wallace. Attention is not explanation, 2019. URL https://arxiv.org/abs/1902.10186

    2019 arXiv

  22. [31]

    Align to structure: Aligning large language models with structural information, 2025

    Zae Myung Kim, Anand Ramachandran, Farideh Tavazoee, Joo-Kyung Kim, Oleg Rokhlenko, and Dongyeop Kang. Align to structure: Aligning large language models with structural information, 2025

    2025

  23. [32]

    Agustinus Kristiadi, Felix Strieth-Kalthoff, Marta Skreta, Pascal Poupart, Al\' a n Aspuru-Guzik, and Geoff Pleiss. A sober look at llms for material discovery: are they actually good for bayesian optimization over molecules? In Proceedings of the 41st International Conference...

    2024

  24. [33]

    Dvornek, Yufeng Gu, Pamela Ventola, and James S

    Xiaoxiao Li, Yuan Zhou, Nicha C. Dvornek, Yufeng Gu, Pamela Ventola, and James S. Duncan. Efficient shapley explanation for features importance estimation under uncertainty. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2020: 23rd International Confere...

    2020 doi

  25. [34]

    Let's verify step by step, 2023

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step, 2023. URL https://arxiv.org/abs/2305.20050

    2023 arXiv

  26. [35]

    Checkpoint merging via bayesian optimization in llm pretraining

    Deyuan Liu, Zecheng Wang, Bingning Wang, Weipeng Chen, Chunshan Li, Zhiying Tu, Dianhui Chu, Bo Li, and Dianbo Sui. Checkpoint merging via bayesian optimization in llm pretraining. CoRR, 2024

    2024

  27. [36]

    Choosing the sample size of a computer experiment: A practical guide

    Jason L Loeppky, Jerome Sacks, and William J Welch. Choosing the sample size of a computer experiment: A practical guide. Technometrics, 54 0 (4): 0 435--446, 2012

    2012

  28. [37]

    A unified approach to interpreting model predictions

    Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2...

    2017

  29. [38]

    López and Martha Saboyá

    S. López and Martha Saboyá. On the relationship between shapley and owen values. Central European Journal of Operations Research, 17: 0 415--423, 12 2009. doi:10.1007/s10100-009-0100-8

    2009 doi

  30. [39]

    Ng, Daishi Harada, and Stuart J

    Andrew Y. Ng, Daishi Harada, and Stuart J. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the Sixteenth International Conference on Machine Learning, ICML '99, pp.\ 278–287, San Francisco, CA, USA, 1999. Mor...

    1999

  31. [40]

    Optimizing instructions and demonstrations for multi-stage language model programs

    Krista Opsahl-Ong, Michael J Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab. Optimizing instructions and demonstrations for multi-stage language model programs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of th...

    2024 doi

  32. [41]

    Token-level proximal policy optimization for query generation

    Yichen Ouyang, Lu Wang, Fangkai Yang, Pu Zhao, Chenghua Huang, Jianfeng Liu, Bochen Pang, Yaming Yang, Yuefeng Zhan, Hao Sun, et al. Token-level proximal policy optimization for query generation. arXiv preprint arXiv:2411.00722, 2024

    2024 arXiv

  33. [42]

    From r to q^* : Your language model is secretly a q-function, 2024

    Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to q^* : Your language model is secretly a q-function, 2024. URL https://arxiv.org/abs/2404.12358

    2024 arXiv

  34. [43]

    Vanishing gradients in reinforcement finetuning of language models, 2024

    Noam Razin, Hattie Zhou, Omid Saremi, Vimal Thilak, Arwen Bradley, Preetum Nakkiran, Joshua Susskind, and Etai Littwin. Vanishing gradients in reinforcement finetuning of language models, 2024. URL https://arxiv.org/abs/2310.20703

    2024 arXiv

  35. [44]

    why should i trust you?

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?": Explaining the predictions of any classifier, 2016. URL https://arxiv.org/abs/1602.04938

    2016 arXiv

  36. [45]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347

    2017 arXiv

  37. [46]

    Principled penalty-based methods for bilevel reinforcement learning and rlhf, 2024

    Han Shen, Zhuoran Yang, and Tianyi Chen. Principled penalty-based methods for bilevel reinforcement learning and rlhf, 2024. URL https://arxiv.org/abs/2402.06886

    2024 arXiv

  38. [47]

    Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms, 2012. URL https://arxiv.org/abs/1206.2944

    2012 arXiv

  39. [48]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018. URL http://incompleteideas.net/book/the-book-2nd.html

    2018

  40. [49]

    The llama 3 herd of models, 2024

    Meta Llama Team. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    2024 arXiv

  41. [50]

    Solving math word problems with process- and outcome-based feedback, 2022

    Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback, 2022. URL https://arxiv.org/abs/2211.14275

    2022 arXiv

  42. [51]

    Trl: Transformer reinforcement learning

    Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. https://github.com/huggingface/trl, 2020

    2020

  43. [52]

    Smith, Mari Ostendorf, and Hannaneh Hajishirzi

    Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training, 2023. URL https://arxiv.org/abs/2306.01693

    2023 arXiv

  44. [53]

    Text2reward: Reward shaping with language models for reinforcement learning, 2024

    Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Text2reward: Reward shaping with language models for reinforcement learning, 2024. URL https://arxiv.org/abs/2309.11489

    2024 arXiv

  45. [54]

    Bayesian reward models for llm alignment

    Adam X Yang, Maxime Robeyns, Thomas Coste, Zhengyan Shi, Jun Wang, Haitham Bou Ammar, and Laurence Aitchison. Bayesian reward models for llm alignment. In ICML 2024 Workshop on Structured Probabilistic Inference \ & \ Generative Modeling

    2024

  46. [55]

    Tlcr: Token-level continuous reward for fine-grained reinforcement learning from human feedback

    Eunseop Yoon, Hee Suk Yoon, SooHwan Eom, Gunsoo Han, Daniel Nam, Daejin Jo, Kyoung-Woon On, Mark Hasegawa-Johnson, Sungwoong Kim, and Chang Yoo. Tlcr: Token-level continuous reward for fine-grained reinforcement learning from human feedback. In Findings of the Association for ...

    2024

  47. [56]

    Token-level direct preference optimization

    Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token-level direct preference optimization. In Proceedings of the 41st International Conference on Machine Learning, pp.\ 58348--58365, 2024

    2024

  48. [57]

    An introduction to bi-level optimization: Foundations and applications in signal processing and machine learning, 2023

    Yihua Zhang, Prashant Khanduri, Ioannis Tsaknakis, Yuguang Yao, Mingyi Hong, and Sijia Liu. An introduction to bi-level optimization: Foundations and applications in signal processing and machine learning, 2023. URL https://arxiv.org/abs/2308.00788

    2023 arXiv

  49. [58]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023 a . URL https://arxiv.org/abs/2306.05685

    2023 arXiv

  50. [59]

    Secrets of rlhf in large language models part i: Ppo, 2023 b

    Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, Limao Xiong, Lu Chen, Zhiheng Xi, Nuo Xu, Wenbin Lai, Minghao Zhu, Cheng Chang, Zhangyue Yin, Rongxiang Weng, Wensen Cheng, Haoran Huang, Tianxiang Sun, Hang Yan, T...

    2023 arXiv

  51. [60]

    Dpo meets ppo: Reinforced token optimization for rlhf, 2025

    Han Zhong, Zikang Shan, Guhao Feng, Wei Xiong, Xinle Cheng, Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf, 2025. URL https://arxiv.org/abs/2404.18922

    2025 arXiv

  52. [61]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  53. [62]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  54. [63]

    reward shaping

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2025

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.