Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

When Can Proxies Improve the Sample Complexity of Preference Learning?

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Proxy data can slash the sample cost of true preference learning.

desk verdict Genuinely interesting factorization theorem, but the main sample-complexity claim rests on a wrong algebraic inversion and an invalid Stage 2 loss. read the letter →

arxiv 2412.16475 v1 pith:VC4JEIZL submitted 2024-12-21 cs.LG cs.AIstat.ML

Yuchen Zhu , Daniel Augusto de Souza , Zhengyan Shi , Mengyue Yang , Pasquale Minervini , Alexander D'Amour , Matt J. Kusner This is my paper · ORCID
classification cs.LGcs.AIstat.ML
keywords preferencelearningrewardhackingproxyrewardssamplecomplexitydirectoptimizationlow-dimensionaladapterlargelanguagemodelsover-optimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks when cheap, abundant proxy preference data—say, ratings from a student doctor—can provably reduce how many expensive true/expert preference samples are needed to learn a ground-truth policy. It identifies four sufficient conditions on the proxy relative to the true policy: shared level sets, inclusion of the true policy's outputs in the proxy's outputs, a low-dimensional encoding of those outputs, and Lipschitz similarity between the two policies. Under these conditions the true policy equals the proxy's decoder and encoder wrapped around a low-dimensional adapter map on a $D$-simplex (Theorem 3). Consequently the number of true preference samples needed scales with the small intrinsic dimension $D$ rather than the high embedding dimension $D'$ (Theorems 5 and 6). If right, this turns reward hacking from a pure risk into a design question: collect proxy feedback that satisfies the conditions, reuse its learned structure, and fine-tune only a small adapter on expert data.

What carries the argument

The machinery is a factored parametrization of policies: any policy $\pi$ is written as $\phi \circ \Theta \circ \tau$, where $\tau$ maps prompts into a $D$-simplex $\Delta^D$, $\Theta$ is a linear map from the simplex to a convex polytope $V$, and $\phi$ is a bi-Lipschitz decoder from $V$ to response distributions. The key identity (Theorem 3) states that if Conditions 1–4 hold, every decomposition of the proxy policy as $\tilde{\pi} = \tilde{\phi} \circ \tilde{\Theta} \tilde{\tau}^{\circ}$ can be reused to express the true policy as $\pi^{\dagger} = \tilde{\phi} \circ \tilde{\Theta} \bar{\pi}^{\dagger} \circ \tilde{\tau}^{\circ}$, with $\bar{\pi}^{\dagger} : \Delta^D \to \Delta^D$ Lipschitz. The adapter $\bar{\pi}^{\dagger}$ is the only component that needs to be learned from true expert preference data, and its low dimension $D$ is what makes the sample-complexity gain possible.

What would settle it

One concrete falsifier is a real or synthetic preference dataset containing two prompts that the proxy maps to the same response distribution but the true policy maps to different distributions; such level-set disagreement violates Condition 1, and if it is widespread the identity $\pi^{\dagger} = \tilde{\phi} \circ \tilde{\Theta} \bar{\pi}^{\dagger} \circ \tilde{\tau}^{\circ}$ cannot hold.

Watch

Extended reading notes

Core claim

The central claim is that proxy feedback can improve sample complexity exactly when the proxy policy and the true policy are the same map up to a low-dimensional interchange: they must partition prompts identically (Condition 1), the true policy's responses must be expressible by the proxy's response decoder (Condition 2), that response space must admit a bi-Lipschitz coordinate chart of dimension $D$ (Condition 3), and the re-assignment from proxy responses to true responses must be Lipschitz (Condition 4). Theorem 3 shows that under these conditions the true policy decomposes as $\pi^{\dagger} = \tilde{\phi} \circ \tilde{\Theta} \bar{\pi}^{\dagger} \circ \tilde{\tau}^{\circ}$, where $\tilde{\phi}$, $\tilde{\Theta}$, $\tilde{\tau}^{\circ}$ are the proxy policy's decoder, linear map, and encoder, and $\bar{\pi}^{\dagger}$ is a Lipschitz adapter from $\Delta^D$ to $\Delta^D$. This decomposition is what carries the sample-complexity gain: the adapter is a map between $D$-simplices, so the covering-number bound, and hence the number of true preference labels needed for generalization, depends on $D$ rather than on the dimension $D'$ of the prompt embedding. The paper's proof adapts a standard generalization-error route via covering numbers, giving a bound for the adapter hypothesis class (Theorem 5) and a comparison bound for the same class learned without proxy structure (Theorem 6).

Load-bearing premise

The sample-complexity guarantee assumes the first stage has already identified the true proxy encoder, linear map, and decoder exactly; the paper gives no bound on how much proxy data that identification needs or on how Stage-1 estimation error inflates the Stage-2 guarantee.

Editorial extensions

If this is right

  • Under the two-stage scheme, the number of true preference samples needed for $\epsilon$-generalization with probability $1-\omega$ scales like $\Omega\!\left(\frac{D}{\epsilon^2}\left(\frac{C\sqrt{D}}{\epsilon}\right)^D \log(C/\epsilon) - \log\omega\right)$, with the intrinsic dimension $D$ in the exponent (Theorem 5).
  • Learning the same true policy from scratch, without the proxy structure, requires a bound with the embedding dimension $D'$ in the exponent, where $D' \gg D$ (Theorem 6).
  • The decomposition gives a concrete LLM parametrization: train the encoder–linear–decoder triplet on abundant proxy preferences, then fine-tune only a low-dimensional adapter on the small expert dataset while keeping the proxy components frozen.
  • The four conditions function as data-collection guidance: recruit proxy raters who group prompts the same way experts do, can in principle express every expert response, and stay within a Lipschitz distance of expert judgment.
  • Because the adapter lives on $\Delta^D$, the architecture can be realized by adapting existing encoder–decoder language models without retraining the pretrained components on expert data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves the Stage 1 guarantee open: its sample-complexity theorem assumes the true $\tilde{\phi}$, $\tilde{\Theta}$, $\tilde{\tau}^{\circ}$ have already been recovered. A natural extension would bound the proxy-sample size needed for Stage 1 and show how Stage-1 estimation error inflates the Stage-2 $D$-scaling; until then, the practical regime is one where proxy data are so abundant that
  • The sufficient conditions suggest a testable screening procedure before training: collect paired proxy/expert ratings on a probe set and estimate how often the proxy's level-set partition disagrees with the expert's; high disagreement would predict that the $D$-scaling guarantee will not hold.
  • If the $D$-scaling result holds, it also gives a diagnostic for reward hacking: the norm $\|\bar{\pi}^{\dagger}(p) - p\|$ measures how far the proxy policy is from the true policy at each latent point, and large jumps would flag regions where proxy data should be distrusted.
  • Combining the adapter view with active learning is a natural next step: choose which expert preferences to query by maximizing the reduction in the adapter's Lipschitz uncertainty, which could reduce the constant in front of the $D$-scaling bound.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies when abundant proxy preference data can provably reduce the number of true preference samples needed to learn a target policy. It introduces four sufficient conditions (shared level sets, image inclusion, finite-dimensional encoding, and Lipschitz similarity) and proves a factorization theorem: under these conditions the true policy can be written as the proxy decoder and encoder composed with a low-dimensional Lipschitz adapter. The authors then propose a two-stage procedure (learn the proxy components with proxy data, learn the adapter with true data) and state sample-complexity bounds, Theorems 5 and 6, claiming that the true-preference sample complexity scales with the small intrinsic dimension D rather than the large embedding dimension D′, yielding a 'superexponential' improvement.

Significance. If the claimed results were correct, this would be a constructive contribution to RLHF theory: it gives explicit sufficient conditions under which proxy data can be safely reused, and a concrete parametrisation for LLMs that realizes the improvement. The paper is clearly organized, the problem is well motivated, and the factorization idea (Conditions 1-4 leading to a shared decoder/encoder with a low-dimensional adapter) is attractive. However, the central sample-complexity statements are not supported by the proofs: the algebra in Theorems 5 and 6 is wrong, and the claimed superexponential improvement disappears after correction. The paper also does not analyze the sample complexity of the first stage, so the total two-stage guarantee is incomplete. These issues are load-bearing rather than cosmetic.

major comments (3)
  1. [Section 5, Theorem 5, Appendix C.1, Eqs. (142)-(144)] The inversion of the concentration bound is incorrect. Eq. (143) is ω = exp(-nε²/(8C²)) A^D B^D with A = 96Lφ‖Θ‖p√D/ε and B = 96Lφ‖Θ‖p L̄π√D/ε. Taking logarithms and solving for n gives n = (8C²/ε²)(D log A + D log B - log ω), i.e. n = O(D/ε² log(D/ε) + ε^{-2} log(1/ω)). The factor B^D cannot be pulled out of the logarithm; it contributes only D log B. Therefore Eq. (144), which states n = Ω(D/ε² B^D log A - log ω), is not a consequence of the preceding equations. The same algebraic error appears in the proof of Theorem 6 in Eqs. (154)-(155). Because Theorems 5 and 6 are the statements that carry the central sample-complexity claim, the paper's advertised superexponential improvement over D′ is unsupported as written. After correction, the comparison would be between roughly O(D log D/ε²) and O(D′ log D′/ε²), a much weaker statement than the one made in Section 3.
  2. [Section 5, first paragraph] The analysis assumes 'given that we have identified the true φ~, Θ~ and τ~° from the proxy dataset in the first stage.' The paper provides no bound on the number of proxy samples needed to identify these components, nor an analysis of how approximation error in Stage 1 propagates into the Stage 2 bound. This is an extra, unquantified condition beyond Conditions 1-4. Since the central claim is about the total sample complexity of learning π† with proxy data, the stated improvement is not established for the full two-stage procedure. The paper should either prove a Stage 1 sample-complexity bound or explicitly frame the results as conditional on exact identification.
  3. [Section 4.3, Eq. (13)] The Stage 2 objective is not well defined as written. The adapter ¯π†θ is defined as a map ΔD → ΔD, but Eq. (13) evaluates ¯π†θ(yw,i | xi) and ¯π†θ(yl,i | xi), treating it as a policy over completions. The DPO ratio should be based on the full model π†θ = φ~θ ∘ Θ~ ¯π†θ ∘ τ~°θ, i.e. β log π†θ(yw,i|xi)/πref(yw,i|xi) minus the corresponding term for yl,i. As written, the loss cannot be computed from the stated model, so the algorithm cannot be implemented without further clarification.
minor comments (5)
  1. [Appendix A, Proposition 1] The proof claims PY = ⋃_{d=1}^∞ Δd, but this union contains only distributions with finite support. Since PY contains distributions with infinite support, the equality is false. The conclusion that the topological dimension is infinite may still be true, but the given proof does not establish it.
  2. [Appendix C.1] The constants E(p,D) and E′(p,D′) are used in the covering-number bounds but are never defined, and the assertion E(p,D) ≤ 1 for simplices is stated without proof. The reader cannot verify the constants in Theorems 5 and 6.
  3. [Theorem 6] The no-proxy baseline class is defined with Lipschitz constant Lφ‖Θ‖pL̄π, but these constants come from the proxy decomposition. The paper does not explain why a learner without proxy data would know or satisfy this constant, so the comparison is not fully grounded.
  4. [Section 2, Eq. (7)] The implicit reward is written as r(x,y) = β log π(yw | x)/πref(yw | x), but the argument on the left is y, not yw. This appears to be a typo.
  5. [Section 7] The statement 'to the best of our knowledge, we are the first to consider this setting' is difficult to verify and could be softened, especially since the related-work section does not discuss the possibility of prior theoretical treatments of proxy-data sample-complexity gains in preference learning.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the factorization and sample-complexity claims are conditional theorems derived from explicitly stated sufficient conditions, not fits or self-citations.

full rationale

The derivation chain is self-contained and conditional. Theorem 3 proves π† = φ~∘Θ~π̄†∘τ~° from Conditions 1-4; Condition 4 (π†∘π~^{-1} Lipschitz) is admittedly close in content to the low-dimensional Lipschitz adapter, but the paper states it as an assumption and uses the theorem to transfer it to coordinates, so the conclusion is not being passed off as an independent prediction. Theorems 5 and 6 then bound the covering number of the explicitly defined hypothesis classes Π(φ~,Θ~,τ~°,L_π̄) and Π̊(L_φ‖Θ~‖_p L_π̄); because the classes are defined with the structure whose benefit they measure, the comparison is a mathematical consequence rather than a fitted input renamed as a prediction. No parameters are fitted to data and no empirical benchmark is claimed. The paper explicitly flags its own limitation that Stage 1 is assumed to recover φ~,Θ~,τ~° exactly (Sec. 5) and that empirical evaluation is ongoing (Sec. 7); these are gaps in scope, not circularity. The only self-citation involving an author (Yang et al. 2024, co-authored by Shi) appears in related work and is not load-bearing. An algebraic slip in the inversion of Eq. (143) to obtain Eq. (144) is a correctness concern, not a circularity, and does not affect this verdict.

Assumptions & free parameters 2 free parameters · 8 assumptions · 0 invented entities

The central result is a theorem from the four stated conditions; the ledger records the domain assumptions and the unstated perfect-recovery assumption. There are no fitted parameters, but D and Lπ̄ are assumed structural constants that the improvement depends on.

free parameters (2)
  • D (intrinsic dimension of proxy policy image)
    Assumed in Condition 3 and controls the sample-complexity bound in Theorem 5; no procedure is given to estimate it from data.
  • Lπ̄ (Lipschitz constant of the adapter)
    Appears in the sample-complexity bound (Theorem 5) and in the hypothesis class definition; assumed from Condition 4, not estimated.
assumptions (8)
  • domain assumption Preference data are generated by the Bradley-Terry model with a scalar reward (Eqs 1-3).
    Used throughout to define true and proxy datasets and DPO losses; standard in RLHF but a modeling choice.
  • standard math The optimal policy takes the KL-regularized form π ∝ πref exp(r/β) (Eq 5) and DPO recovers it (Eq 6).
    Borrowed from Rafailov et al. (2023); the paper relies on this equivalence.
  • domain assumption Condition 1: π†(x1)=π†(x2) iff π~(x1)=π~(x2).
    Makes π†∘π~^{-1} well-defined (Lemma 2); exact equality of prompt partitions.
  • domain assumption Condition 2: π†(X) ⊆ π~(X).
    Lets the same decoder φ~ decode the true policy; requires proxy to be able to express all expert responses.
  • domain assumption Condition 3: π~(X) lies in the image of a bi-Lipschitz map from a bounded convex polytope V⊂R^D.
    Gives the low-dimensional manifold structure used in the sample-complexity bounds.
  • domain assumption Condition 4: π†∘π~^{-1} restricted to π~(X) is Lipschitz.
    Ensures the adapter is Lipschitz and controls the covering-number bound.
  • ad hoc to paper The proxy components φ~, Θ~, τ~° are identified exactly from the proxy dataset before Stage 2 analysis.
    Section 5 assumes this without a sample-complexity bound for Stage 1; the central improvement depends on it.
  • standard math Standard covering-number and Hoeffding concentration inequalities apply to the preference loss.
    Used in Proposition 7 and Theorem 5; no proof of the concentration bound beyond citation and adaptation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When Can Proxies Improve the Sample Complexity of Preference Learning?." pith.science (2026). https://pith.science/paper/VC4JEIZL

@misc{pith2026241216475,
  author       = {Pith},
  title        = {Pith review of: When Can Proxies Improve the Sample Complexity of Preference Learning?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VC4JEIZL}},
  note         = {Machine review of arXiv:2412.16475}
}
read the original abstract

We address the problem of reward hacking, where maximising a proxy reward does not necessarily increase the true reward. This is a key concern for Large Language Models (LLMs), as they are often fine-tuned on human preferences that may not accurately reflect a true objective. Existing work uses various tricks such as regularisation, tweaks to the reward model, and reward hacking detectors, to limit the influence that such proxy preferences have on a model. Luckily, in many contexts such as medicine, education, and law, a sparse amount of expert data is often available. In these cases, it is often unclear whether the addition of proxy data can improve policy learning. We outline a set of sufficient conditions on proxy feedback that, if satisfied, indicate that proxy data can provably improve the sample complexity of learning the ground truth policy. These conditions can inform the data collection process for specific tasks. The result implies a parameterisation for LLMs that achieves this improved sample complexity. We detail how one can adapt existing architectures to yield this improved sample complexity.

Figures

Figures reproduced from arXiv: 2412.16475 by the authors.

Figure 1
Figure 1. Medical question answering. (Illustrative purpose only. Not medical advice.) Patients [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustrations of conditions 1-3. Left, middle, right: Condition 1, 2, 3, respectively. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Shallow Preference Signals: Large Language Model Aligns Even Better with Truncated Data?

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Preference signals in LLM alignment are concentrated in early response tokens, so models trained on data truncated to the first half perform as well as or better than those trained on full responses.

Reference graph

Works this paper leans on

43 extracted references · 19 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Accuracy of chatgpt, google bard, and microsoft bing for simplifying radiology reports

    Kanhai S Amin, Melissa A Davis, Rushabh Doshi, Andrew H Haims, Pavan Khosla, and Howard P Forman. Accuracy of chatgpt, google bard, and microsoft bing for simplifying radiology reports. Radiology, 309 0 (2): 0 e232561, 2023

    work page 2023

  4. [4]

    The evolved radio and its implications for modelling the evolution of novel sensors

    Jon Bird and Paul Layzell. The evolved radio and its implications for modelling the evolution of novel sensors. In Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No. 02TH8600), volume 2, pp.\ 1836--1841. IEEE, 2002

    work page 2002

  5. [5]

    Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 1952

    1952

  6. [6]

    Open problems and fundamental limitations of reinforcement learning from human feedback

    Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, J \'e r \'e my Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Tong Wang, Samuel Marks, Charbel-Raphael Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J Mi...

    work page 2023

  7. [7]

    Maxmin-rlhf: Towards equitable alignment of large language models with diverse human preferences

    Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Singh Bedi, and Mengdi Wang. Maxmin-rlhf: Towards equitable alignment of large language models with diverse human preferences. arXiv preprint arXiv:2402.08925, 2024

    arXiv 2024

  8. [8]

    ODIN : Disentangled reward mitigates hacking in RLHF

    Lichang Chen, Chen Zhu, Jiuhai Chen, Davit Soselia, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. ODIN : Disentangled reward mitigates hacking in RLHF . In Forty-first International Conference on Machine Learning, 2024

    work page 2024

Show all 43 references
  1. [9]

    Meta discovery: Learning to discover novel classes given very limited data

    Haoang Chi, Feng Liu, Wenjing Yang, Long Lan, Tongliang Liu, Bo Han, Gang Niu, Mingyuan Zhou, and Masashi Sugiyama. Meta discovery: Learning to discover novel classes given very limited data. In International Conference on Learning Representations, 2022. URL https://openreview...

    2022

  2. [10]

    Faulty reward functions in the wild, 2016

    Jack Clark and Dario Amodei. Faulty reward functions in the wild, 2016. URL https://openai.com/index/faulty-reward-functions/

    2016

  3. [11]

    Reward model ensembles help mitigate overoptimization

    Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=dcjtMYkpXx

    2024

  4. [12]

    The expertise problem: Learning from specialized feedback

    Oliver Daniels-Koch and Rachel Freedman. The expertise problem: Learning from specialized feedback. arXiv preprint arXiv:2211.06519, 2022

    2022 arXiv

  5. [13]

    Group symmetry in pac learning

    Bryn Elesedy. Group symmetry in pac learning. In ICLR 2022 workshop on geometrical and topological representation learning, 2022

    2022

  6. [14]

    Active teacher selection for reinforcement learning from human feedback

    Rachel Freedman, Justin Svegliato, Kyle Wray, and Stuart Russell. Active teacher selection for reinforcement learning from human feedback. arXiv preprint arXiv:2310.15288, 2023

    2023 arXiv

  7. [15]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, pp.\ 10835--10866. PMLR, 2023

    2023

  8. [16]

    Glass floor colleges reject top applicants, accepting only the students likely to enroll, 2001

    Daniel Golden. Glass floor colleges reject top applicants, accepting only the students likely to enroll, 2001

    2001

  9. [17]

    Lee, Wen Sun, Akshay Krishnamurthy, and Dylan J

    Audrey Huang, Wenhao Zhan, Tengyang Xie, Jason D. Lee, Wen Sun, Akshay Krishnamurthy, and Dylan J. Foster. Correcting the mythos of kl-regularization: Direct alignment without overoptimization via chi-squared preference optimization, 2024. URL https://arxiv.org/abs/2407.13399

    2024 arXiv

  10. [18]

    Ai alignment: A comprehensive survey

    Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. Ai alignment: A comprehensive survey. arXiv preprint arXiv:2310.19852, 2023

    2023 arXiv

  11. [19]

    Reward (mis) design for autonomous driving

    W Bradley Knox, Alessandro Allievi, Holger Banzhaf, Felix Schmitt, and Peter Stone. Reward (mis) design for autonomous driving. Artificial Intelligence, 316: 0 103829, 2023

    2023

  12. [20]

    Mitigating reward hacking via information-theoretic reward modeling

    Yuchun Miao, Sen Zhang, Liang Ding, Rong Bao, Lefei Zhang, and Dacheng Tao. Mitigating reward hacking via information-theoretic reward modeling. arXiv preprint arXiv:2402.09345, 2024

    2024 arXiv

  13. [21]

    Mohri, A

    M. Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations of Machine Learning. Adaptive Computation and Machine Learning series. MIT Press, 2012. ISBN 9780262018258. URL https://books.google.co.uk/books?id=maz6AQAAQBAJ

    2012

  14. [22]

    Overcoming exploration in reinforcement learning with demonstrations

    Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference on robotics and automation (ICRA), pp.\ 6292--6299. IEEE, 2018

    2018

  15. [23]

    The effects of reward misspecification: Mapping and mitigating misaligned models

    Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. arXiv preprint arXiv:2201.03544, 2022

    2022 arXiv

  16. [24]

    A deep reinforced model for abstractive summarization

    Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations, 2018

    2018

  17. [25]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. CoRR, abs/2103.00020...

    2021 arXiv

  18. [26]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openr...

    2023

  19. [27]

    Scaling laws for reward model overoptimization in direct alignment algorithms

    Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms. arXiv preprint arXiv:2406.02900, 2024. URL https://arxiv.org/abs/2406.02900

    2024 arXiv

  20. [28]

    Learning complex dexterous manipulation with deep reinforcement learning and demonstrations

    Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:1709.10087, 2017

    2017 arXiv

  21. [29]

    Countering reward over-optimization in llm with demonstration-guided reinforcement learning

    Mathieu Rita, Florian Strub, Rahma Chaabouni, Paul Michel, Emmanuel Dupoux, and Olivier Pietquin. Countering reward over-optimization in llm with demonstration-guided reinforcement learning. arXiv preprint arXiv:2404.19409, 2024

    2024 arXiv

  22. [30]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  23. [31]

    Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback

    Wei Shen, Rui Zheng, Wenyu Zhan, Jun Zhao, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang. Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023

    2023

  24. [32]

    A long way to go: Investigating length correlations in rlhf

    Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716, 2023

    2023 arXiv

  25. [33]

    Defining and characterizing reward gaming

    Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35: 0 9460--9471, 2022

    2022

  26. [34]

    Causal confusion and reward misidentification in preference-based reward learning

    Jeremy Tien, Jerry Zhi-Yang He, Zackory Erickson, Anca D Dragan, and Daniel S Brown. Causal confusion and reward misidentification in preference-based reward learning. arXiv preprint arXiv:2204.06601, 2022

    2022 arXiv

  27. [35]

    Relatively rational: Learning utilities and rationalities jointly from pairwise preferences

    Taku Yamagata, Tobias Oberkofler, Timo Kaufmann, Viktor Bengs, Eyke H \"u llermeier, and Raul Santos-Rodriguez. Relatively rational: Learning utilities and rationalities jointly from pairwise preferences. In ICML 2024 Workshop on Models of Human Feedback for AI Alignment, 2024

    2024

  28. [36]

    Yang, Maxime Robeyns, Thomas Coste, Zhengyan Shi, Jun Wang, Haitham Bou-Ammar, and Laurence Aitchison

    Adam X. Yang, Maxime Robeyns, Thomas Coste, Zhengyan Shi, Jun Wang, Haitham Bou-Ammar, and Laurence Aitchison. Bayesian reward models for llm alignment, 2024. URL https://arxiv.org/abs/2402.13210

    2024 arXiv

  29. [37]

    Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Co...

    2024

  30. [38]

    Larger and more instructable language models become less reliable

    Lexin Zhou, Wout Schellaert, Fernando Martínez-Plumed, Yael Moros-Daval, Cèsar Ferri, and José Hernández-Orallo. Larger and more instructable language models become less reliable. Nature, 634 0 (8032), Sep 2024. ISSN 1476-4687. doi:10.1038/s41586-024-07930-y. URL http://dx.doi...

    2024 doi

  31. [39]

    Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf

    Banghua Zhu, Michael I Jordan, and Jiantao Jiao. Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf. arXiv preprint arXiv:2401.16335, 2024

    2024 arXiv

  32. [40]

    Consequences of misaligned ai

    Simon Zhuang and Dylan Hadfield-Menell. Consequences of misaligned ai. Advances in Neural Information Processing Systems, 33: 0 15763--15773, 2020

    2020

  33. [41]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  34. [42]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  35. [43]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.