Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

Best-of-N through the Smoothing Lens: KL Divergence and Regret Analysis

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A temperature-smoothed version of Best-of-N sampling can keep alignment gains under imperfect proxy rewards, where hard Best-of-N degrades.

desk verdict Useful new KL and regret bounds for Soft Best-of-N, but the headline claim that smoothing beats BoN under proxy rewards rests on an invalid comparison condition and should not be taken at face value. read the letter →

arxiv 2507.05913 v1 pith:K5M3J6QP submitted 2025-07-08 stat.ML cs.LG

Gholamali Aminian , Idan Shenfeld , Amir R. Asadi , Ahmad Beirami , Youssef Mroueh This is my paper
classification stat.MLcs.LG MSC 62B1068T50
keywords softbest-of-nrewardoveroptimizationhackingproxymodelKLdivergenceregretboundinference-timealignmenttemperaturesmoothing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Best-of-N alignment samples N responses and keeps the one a reward model scores highest, but when that model is a learned approximation of the true reward, more samples can mean more reward hacking. This paper studies Soft Best-of-N (SBoN), which instead samples the final response from the N candidates with probability proportional to $\exp(\beta \hat{r}(x,y))$, interpolating between reference sampling at $\beta=0$ and hard Best-of-N as $\beta\to\infty$. The paper proves finite-sample upper bounds on the KL divergence between the SBoN policy and the reference policy and on the regret gap to the optimal true-reward policy, with the proxy error entering through a temperature-dependent quantity $\varepsilon_{\beta,r}(x)$. In the overoptimization regime $\varepsilon_{\beta,r}(x)>0$, an intermediate temperature makes the SBoN regret bound no worse than the BoN bound, so smoothing becomes a principled hedge against a bad proxy. The empirical section shows this hedge in action: with a weak reward model, hard Best-of-N worsens as $N$ grows while SBoN does not.

What carries the argument

The engine is the closed-form SBoN policy with its normalizer $Z_{N,\beta}$ averaging over the other $N-1$ sampled rewards. Two summary quantities carry the argument: the tilted estimation error $\varepsilon_{\beta,r}(x)=\beta^{-1}\log\mathbb{E}_{y\sim\pi_{\mathrm{ref}}}[\exp(\beta(r^\star-\hat{r})^2)]$, which interpolates between mean-squared error at $\beta=0$ and squared sup-norm error as $\beta\to\infty$, and the coverage constants $C_{\beta,r,\mathrm{ref}}(x)$ and $C_{\infty,r,\mathrm{ref}}(x)$, which measure how much reference-policy mass sits on high-reward responses. The regret proof splits the gap into the distance from the deterministic optimal policy to the tilted policy, the distance from the tilted policy to the SBoN policy under the proxy, and the estimation-error term controlled by $\varepsilon$ and the coverage constants. The BoN-versus-SBoN comparison is decided by the auxiliary function $g(\beta)=\beta(\varepsilon_{\infty,r}(x)-\varepsilon_{\beta,r}(x))$ and the condition (23) involving the coverage ratio.

What would settle it

Enumerate a small finite output space (say two responses) with known $\pi_{\mathrm{ref}}$, bounded rewards, $N=2$, and a small proxy perturbation, and compute the exact KL divergence between $\pi_{r^\star}^{(N,\beta)}$ and $\pi_{\hat{r}}^{(N,\beta)}$; if it exceeds the right-hand side of Lemma 4.2, the central misspecification bound is false. Directly, one can numerically evaluate $\sup |\delta f/\delta r|$ in the same setup and compare it with Lemma D.11's $N=2$ bound $4\beta e^{2\beta R_{\max}}$.

Watch

Extended reading notes

Core claim

The central discovery is that SBoN has a closed-form policy, $\pi_{\hat{r}}^{(N,\beta)}(y|x)=\pi_{\mathrm{ref}}(y|x)\exp(\beta\hat{r}(x,y))/Z_{N,\beta}$, and that this form is enough to carry finite-$N$ guarantees under a misspecified proxy reward. Lemma 4.1 bounds $\mathrm{KL}(\pi_{r^\star}^{(N,\beta)}\|\pi_{\mathrm{ref}})$ by $\log\big(N/(1+(N-1)e^{-\beta R_{\max}})\big)$, recovering about $\log N$ in the BoN limit. Lemma 4.2 bounds the KL between SBoN policies built on the true and proxy rewards in terms of $\beta$, $N$, the reward range, and the tilted estimation error $\varepsilon_{\beta,r}(x)$; Theorem 4.3 converts the two lemmas into an upper bound on the expected true-reward improvement over the reference. Theorem 5.2 bounds the regret of SBoN, decomposing the gap into a proxy-error term times coverage constants and a KL/coverage term, and Theorem 5.3 obtains the BoN bound by taking $\beta\to\infty$. Remark 5.6 then states the paper's headline comparison: whenever the proxy is imperfect for every $\beta$ and condition (23) holds, the SBoN regret bound is tighter than the BoN regret bound at an intermediate $\beta$, which is the formal sense in which smoothing mitigates reward hacking.

Load-bearing premise

The load-bearing premise is the sensitivity estimate in Lemma D.11, $|\delta f/\delta r| \le N^2\beta e^{2\beta R_{\max}}/(N-1)^2$, on the SBoN normalizer; Lemma 4.2 and hence the regret comparisons inherit it, and the appendix derivation of this bound is not fully justified and the expression diverges at $N=1$, so the quantitative claims collapse if the estimate is wrong.

Editorial extensions

If this is right

  • For a fixed $N$, an intermediate temperature $\beta$ can give a smaller regret upper bound than hard Best-of-N whenever the proxy reward is imperfect and condition (23) holds, providing a concrete knob for reward-hacking mitigation.
  • Because the KL bound grows logarithmically in $N$ and saturates in $\beta$, SBoN interpolates between reference sampling ($\beta=0$) and BoN-like selection ($\beta\to\infty$) without the divergence exceeding $\log N$.
  • The regret bound depends on the reference policy's mass on both true-optimal and proxy-optimal responses through $C_{\infty,r^\star,\mathrm{ref}}$ and $C_{\infty,\hat{r},\mathrm{ref}}$, so reference-policy quality and proxy quality jointly determine alignment.
  • As $\beta\to\infty$, the SBoN regret bound converges to the BoN bound, so the BoN guarantee is recovered as a limit of the smooth family rather than proved separately.
  • When the proxy is perfect ($\varepsilon_{\beta,r}(x)=0$), the estimation term vanishes and larger $\beta$ is preferred, consistent with the known near-optimality of BoN under the true reward.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A prompt-dependent temperature is a natural next step: since $\varepsilon_{\beta,r}(x)$ varies with prompt and proxy, one could lower $\beta$ where an uncertainty estimate says the proxy is unreliable and raise it where the proxy is trusted.
  • The same smoothing-lens argument transfers to other inference-time compute schemes, such as repeated sampling with an external verifier or rejection sampling: replacing hard accept/reject decisions with softmax weights introduces the same $\varepsilon$-versus-KL tradeoff.
  • Condition (23) makes a testable quantitative prediction: the temperature maximizing $g(\beta)=\beta(\varepsilon_{\infty,r}(x)-\varepsilon_{\beta,r}(x))$ should approximate the empirical optimum at which true-reward performance under a weak proxy stops degrading; a small ground-truth audit could check this.
  • Because every bound is finite-sample, the displayed regret upper bound could itself be used as a tuning criterion: estimate $\varepsilon_{\beta,r}(x)$ and the coverage constants, then choose $\beta$ to minimize the bound before running an expensive judge-based evaluation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper studies Soft Best-of-N (SBoN), a smoothed, temperature-controlled variant of Best-of-N (BoN), under a proxy reward model. It derives finite-sample upper bounds on the KL divergence between the SBoN policy and the reference policy (Lemma 4.1), between the SBoN policies under true and proxy rewards (Lemma 4.2), on expected true-reward improvement (Theorem 4.3), and on regret gaps for SBoN and BoN (Theorems 5.2 and 5.3, with BoN obtained as the β→∞ limit). The central comparative claim is Remark 5.6: under overoptimization (εβ,r(x)>0), an intermediate temperature β makes the SBoN regret upper bound no worse than the BoN regret upper bound, so smoothing mitigates reward hacking. The paper also reports experiments with strong and weak reward models suggesting that SBoN degrades less than BoN for large N when the proxy reward is weak.

Significance. If correct, the KL bound in Lemma 4.1 is a clean interpolation between the reference policy (β=0) and BoN (β→∞), and the coverage-dependent regret bounds in Theorems 5.2 and 5.3 would give practitioners a principled way to choose β in the presence of proxy reward error. I credit the paper for making the dependence on N, β, the tilted estimation error εβ,r, and the coverage constants C∞,r explicit, and for providing an empirical demonstration of the qualitative smoothing effect. However, the headline theoretical comparison — that smoothing provably mitigates overoptimization — rests on Remark 5.6, and the condition asserted there is not derived from (21) and (22); in fact, condition (23) is not sufficient for the claimed comparison. The main theoretical contribution is therefore not established as stated. The remaining lemmas are not sufficient to carry the paper's central claim, and the empirical results, while suggestive, are one-figure evidence rather than a substitute for a correct theorem.

major comments (2)
  1. [Remark 5.6, Eq. (23)] The claimed implication from condition (23) to (21)≤(22) is false. Let S∞ = sqrt(C∞,r̂,ref(x)) + sqrt(C∞,r⋆,ref(x)). Subtracting (22) from (21) gives B_S(β) − B_B = S∞(sqrt(εβ,r(x)) − sqrt(ε∞,r(x))) + log(C∞,r⋆,ref(x))/β. Hence B_S(β)≤B_B is equivalent to log C∞,r⋆,ref(x) ≤ β S∞ (sqrt(ε∞,r(x))−sqrt(εβ,r(x))). Condition (23) instead compares log C∞,r⋆,ref(x)/(sqrt(Cβ,r̂,ref(x))+sqrt(C∞,r⋆,ref(x))) with g(β⋆)=β⋆(ε∞,r(x)−εβ⋆,r(x)); this is not equivalent to the correct inequality, because it replaces the difference of square roots by the difference of the ε's and replaces S∞ by a β-dependent denominator. The assertion g(0)=g(∞)=0 is also generally false: for a finite output space with reference mass p on the reward maximizer, β(ε∞,r−εβ,r) tends to −log p as β→∞, which need not be 0. Concretely, take Y={a,b}, πref uniform, r̂(a)=1, r̂(b)=0, r⋆(a)=1, r⋆(b)=2, Rmax=2, β=10. Then ε∞,r=4, εβ,r≈3.9307, C∞,r̂=C∞,r⋆=2, Cβ,r̂≈2, so the left side of (23) is log2/(2√2)≈0.245 and g(10)≈0.693, so (23) holds. Yet (21)−(22)=2√2(√3.9307−2)+log2/10≈0.020>0, so the SBoN bound is not tighter. The comparison step in Remark 5.6 must be reworked, and the current statement of the paper's central overoptimization claim is not valid.
  2. [Lemma 4.2 and Lemma D.11] The sensitivity bound in Lemma D.11 is load-bearing: Lemma 4.2 controls the proxy-error term in Theorem 4.3 and, through the regret theorems, the comparison in Remark 5.6. As written, the proof of Lemma D.11 is not verifiable. The first displayed inequality in the proof (Eq. (44)) asserts a combinatorial identity for the functional derivative of f(r)=log E[1/(exp(βr)+Σ exp(βR_i))] without derivation, and the object being differentiated is not clearly matched to Definition D.1/D.2 (the derivative with respect to the reward at a single y vs. a uniform shift of the reward function). The resulting bound diverges at N=1 even though the KL divergence in Lemma 4.2 is exactly 0 for N=1. Because Lemma 4.2 is used to quantify the estimation error in Theorem 4.3, the paper needs either a complete, rigorous proof of Lemma D.11 (or a replacement bound) before the quantitative regret statements can be accepted.
minor comments (5)
  1. [Remark 5.7] The last sentence says 'the bound in (24) is tighter than the bound in (24)'; this should compare the SBoN and BoN displays in (24). Since the SBoN bound contains the extra nonnegative term log(C∞,r⋆,ref(x))/β, it is looser, not tighter.
  2. [Eq. (5), Section 3.2] The displayed definition of Z_{N,β} appears to depend on y through the expectation, so it is not the normalizing constant for (4); please clarify the formula, including the missing factor 1/N and the y-dependence.
  3. [Theorem 4.3] The statement in the main text has a factor 1/2 inside the second minimum of the upper bound, while the appendix statement of the same theorem omits that factor; the two statements should be reconciled.
  4. [Appendix F] The numbering is inconsistent: Section 5 states Theorem 5.2 (SBoN optimal gap) and Theorem 5.3 (BoN optimal gap), but the appendix labels the SBoN regret bound as Theorem 5.3 and then gives another Theorem 5.3 for BoN; please renumber the theorems consistently.
  5. [Section 6] The sentence 'We also studied the behavior of our upper bound on the KL divergence between the SBoN policy and the reference policy, Lemma 4.2' should refer to Lemma 4.1, since Lemma 4.2 bounds the KL between SBoN under true and proxy rewards.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the bounds follow from stated assumptions and prior closed forms, and the only self-citation is a standard, non-load-bearing sensitivity identity.

full rationale

The paper's central claims are derived, not assumed. Lemma 4.1's KL bound follows from the explicit SBoN closed form (cited to Claudio et al. 2025) plus bounded rewards, and it is used only as an upper bound, not as the quantity it claims to bound. Lemma 4.2 and Theorem 4.3 relate KL and reward improvement to the independently defined tilted estimation error εβ,r(x); the sensitivity bound in Lemma D.11 is a direct calculation with N, β and Rmax, and even if that calculation is incomplete or divergent at N=1, that is a correctness risk, not a circular reduction. The regret theorems 5.2 and 5.3 are upper bounds assembled from Pinsker's inequality, coverage constants Cβ,r,ref, and the KL estimates; no fitted parameter is later renamed as a prediction. Remark 5.6's condition (23) is posed as a sufficient condition for comparing two upper bounds; whether it is actually sufficient is a mathematical question (the skeptic's two-point counterexample targets exactly this implication), not a definitional equivalence. The only self-citation is Lemma D.4, attributed to Aminian et al. 2025, which states a standard softmax-policy sensitivity identity and is not load-bearing for the main theorems; other cited KL bounds, e.g., Beirami et al. 2024 and Mroueh 2024, are independent prior results. No step reduces by construction to its own inputs, so the derivation is self-contained and the modest score reflects only the presence of a minor non-load-bearing self-citation.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new entities, no fitted constants, and no ad hoc parameters. The bounds are expressed in terms of the reward bound R_max, the sample count N, the temperature β, and the tilted estimation error ε_{β,r}(x), all of which are inputs or measurable quantities from the setting. The analysis relies on the prior closed-form SBoN policy (Claudio et al. 2025) and standard inequalities.

assumptions (5)
  • domain assumption True and proxy rewards are non-negative and bounded by R_max (Assumption 3.1).
    Used in Lemma 4.1, Lemma 4.2, Lemma D.11 and all regret bounds to control exponentials and apply the min(1, .) cap on TV distances.
  • domain assumption The tilted average of squared reward estimation error ε_{β,r}(x) = (1/β) log E[exp(β(r* − r̂)²)] is finite (Assumption 3.2).
    Needed so that the KL and regret bounds involving ε_{β,r}(x) are finite; stated in Section 3.3.
  • domain assumption The proxy reward attains its maximum R_max on at least one response (Assumption 3.3).
    Used to define C_{∞,r̂,ref}(x) and to show the coverage constants are finite; the paper argues this holds trivially for finite output spaces.
  • domain assumption Closed-form marginal policy of SBoN, π^{(N,β)}_r(y|x) proportional to π_ref(y|x) exp(β r(y,x)) times N E[1/(exp(β r(y,x)) + Σ_{i=1}^{N-1} exp(β r(Y_i,x)))] (Claudio et al., 2025, Lemma 1).
    The starting point for Lemma 4.1 and the KL proofs; imported without proof from prior work.
  • standard math Standard inequalities: Pinsker, Cauchy-Schwarz, Jensen, mean value theorem, and monotonicity of tilted means.
    Used throughout the appendix proofs; not proved in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Best-of-N through the Smoothing Lens: KL Divergence and Regret Analysis." pith.science (2026). https://pith.science/paper/K5M3J6QP

@misc{pith2026250705913,
  author       = {Pith},
  title        = {Pith review of: Best-of-N through the Smoothing Lens: KL Divergence and Regret Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K5M3J6QP}},
  note         = {Machine review of arXiv:2507.05913}
}
abstract

A simple yet effective method for inference-time alignment of generative models is Best-of-$N$ (BoN), where $N$ outcomes are sampled from a reference policy, evaluated using a proxy reward model, and the highest-scoring one is selected. While prior work argues that BoN is almost optimal in reward vs KL tradeoffs, the effectiveness of BoN depends critically on the quality of the proxy reward model used for selection. For this purpose, we study BoN through a smooth version known as Soft Best-of-N (SBoN) and develop a theoretical framework to address this gap. We analyze the scaling behaviour of BoN by providing bounds on the KL divergence between the SBoN policy and the reference policy, offering insights into how performance varies with the number of samples. We also study the regret gap, i.e., the gap between the expected true reward under the optimal policy and the SBoN policy. Our theoretical and empirical findings show that smoothing helps SBoN mitigate reward overoptimization, especially when the quality of the proxy reward is low.

Figures

Figures reproduced from arXiv: 2507.05913 by the authors.

Figure 1
Figure 1. Connections of different policies under reward [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Soft Best-of-N experiment using a strong reward model (Left) and a weak one (Right). When the performance of [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. True KL divergence vs upper bound in Lemma [PITH_FULL_IMAGE:figures/full_fig_p023_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Certifiable Safe RLHF: Semantic Grounding and Fixed Penalty Constraint Optimization for Safer LLM Alignment

    cs.LG 2025-10 reject novelty 5.0 of 10

    A fixed ReLU penalty and a semantically labeled cost model are proposed to make RLHF safer, but the 'certifiable' guarantee is not fully supported.

Reference graph

Works this paper leans on

64 extracted references · 28 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Variational best-of-n alignment

    Afra Amini, Tim Vieira, Elliott Ash, and Ryan Cotterell. Variational best-of-n alignment. In International Conference on Learning Representations (ICLR), 2025

    work page 2025

  4. [4]

    Theoretical analysis of kl-regularized rlhf with multiple reference models

    Gholamali Aminian, Amir R Asadi, Idan Shenfeld, and Youssef Mroueh. Theoretical analysis of kl-regularized rlhf with multiple reference models. arXiv preprint arXiv:2502.01203, 2025

    arXiv 2025

  5. [5]

    Concrete problems in ai safety

    Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man \'e . Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016

    arXiv 2016

  6. [6]

    Infalign: Inference-aware language model alignment

    Ananth Balashankar, Ziteng Sun, Jonathan Berant, Jacob Eisenstein, Michael Collins, Adrian Hutter, Jong Lee, Chirag Nagpal, Flavien Prost, Aradhana Sinha, Ananda Theertha Suresh, and Ahmad Beirami. Infalign: Inference-aware language model alignment. International Conference on Machine Learning (ICML), 2025

    work page 2025

  7. [7]

    Theoretical guarantees on the best-of-n alignment policy

    Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander D'Amour, Jacob Eisenstein, Chirag Nagpal, and Ananda Theertha Suresh. Theoretical guarantees on the best-of-n alignment policy. International Conference on Machine Learning (ICML), 2024

    work page 2024

  8. [8]

    Q-learning for risk-sensitive control

    Vivek S Borkar. Q-learning for risk-sensitive control. Mathematics of Operations Research, 2002

    work page 2002

Show all 64 references
  1. [9]

    Large language monkeys: Scaling inference compute with repeated sampling

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher R \'e , and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024

    2024 arXiv

  2. [10]

    A short note on an inequality between kl and tv

    Cl \'e ment L Canonne. A short note on an inequality between kl and tv. arXiv preprint arXiv:2202.07198, 2022

    2022 arXiv

  3. [11]

    The master equation and the convergence problem in mean field games:(ams-201)

    Pierre Cardaliaguet, Fran c ois Delarue, Jean-Michel Lasry, and Pierre-Louis Lions. The master equation and the convergence problem in mean field games:(ams-201). Princeton University Press, 2019

    2019

  4. [12]

    Open problems and fundamental limitations of reinforcement learning from human feedback

    Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, J \'e r \'e my Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint...

    2023 arXiv

  5. [13]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  6. [14]

    Soft best-of-n sampling for model alignment

    Mayrink Verdun Claudio, Oesterling Alex, Lakkaraju Himabindu, and P Calmon Flavio. Soft best-of-n sampling for model alignment. URL http://github.com/alex-oesterling/soft-best-of-n/blob/main/soft\_bon.pdf, 2025

    2025

  7. [15]

    Reward model ensembles help mitigate overoptimization

    Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. In The Twelfth International Conference on Learning Representations

  8. [16]

    Safe rlhf: Safe reinforcement learning from human feedback

    Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310.12773, 2023

    2023 arXiv

  9. [17]

    Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking

    Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alexander Nicholas D'Amour, Krishnamurthy Dj Dvijotham, Adam Fisch, Katherine A Heller, Stephen Robert Pfohl, Deepak Ramachandran, et al. Helping or herding? reward model ensembles mitigate but do not eliminate rew...

  10. [18]

    Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking

    Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D'Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:231...

    2023 arXiv

  11. [19]

    A framework for few-shot language model evaluation

    L Gao, J Tow, B Abbasi, S Biderman, S Black, A DiPofi, C Foster, L Golding, J Hsu, A Le Noac’h, et al. A framework for few-shot language model evaluation. URL https://zenodo. org/records/10256836, 7, 2023 a

    2023

  12. [20]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835--10866. PMLR, 2023 b

    2023

  13. [21]

    Guided speculative inference for efficient test-time alignment of llms

    Jonathan Geuter, Youssef Mroueh, and David Alvarez-Melis. Guided speculative inference for efficient test-time alignment of llms. arXiv preprint arXiv:2506.04118, 2025

    2025 arXiv

  14. [22]

    Bonbon alignment for large language models and the sweetness of best-of-n sampling

    Lin Gui, Cristina G \^a rbacea, and Victor Veitch. Bonbon alignment for large language models and the sweetness of best-of-n sampling. arXiv preprint arXiv:2406.00832, 2024

    2024 arXiv

  15. [23]

    Statistical theory of extreme values and some practical applications

    Emil Julius Gumbel. Statistical theory of extreme values and some practical applications. Nat. Bur. Standards Appl. Math. Ser. 33, 1954

    1954

  16. [24]

    Hilton, P

    J. Hilton, P. Clark, et al. Measuring goodhart’s law: Towards an evaluation framework for open-ended generative models. https://openai.com/index/measuring-goodharts-law, 2022. Accessed: 2025-01-30

    2022

  17. [25]

    Risk-sensitive markov decision processes

    Ronald A Howard and James E Matheson. Risk-sensitive markov decision processes. Management science, 1972

    1972

  18. [26]

    Is best-of-n the best of them? coverage, scaling, and optimality in inference-time alignment

    Audrey Huang, Adam Block, Qinghua Liu, Nan Jiang, Dylan J Foster, and Akshay Krishnamurthy. Is best-of-n the best of them? coverage, scaling, and optimality in inference-time alignment. arXiv preprint arXiv:2503.21878, 2025

    2025 arXiv

  19. [27]

    Best-of-n jailbreaking

    John Hughes, Sara Price, Aengus Lynch, Rylan Schaeffer, Fazl Barez, Sanmi Koyejo, Henry Sleight, Erik Jones, Ethan Perez, and Mrinank Sharma. Best-of-n jailbreaking. arXiv preprint arXiv:2412.03556, 2024

    2024 arXiv

  20. [28]

    Evaluation of best-of-n sampling strategies for language model alignment

    Yuki Ichihara, Yuu Jinnai, Tetsuro Morimura, Kenshi Abe, Kaito Ariu, Mitsuki Sakamoto, and Eiji Uchibe. Evaluation of best-of-n sampling strategies for language model alignment. Transactions on Machine Learning Research

  21. [29]

    Regularized best-of-n sampling to mitigate reward hacking for language model alignment

    Yuu Jinnai, Tetsuro Morimura, Kaito Ariu, and Kenshi Abe. Regularized best-of-n sampling to mitigate reward hacking for language model alignment. In ICML 2024 Workshop on Models of Human Feedback for AI Alignment, 2024

    2024

  22. [30]

    Inference-time reward hacking in large language models

    Hadi Khalaf, Claudio Mayrink Verdun, Alex Oesterling, Himabindu Lakkaraju, and Flavio du Pin Calmon. Inference-time reward hacking in large language models. arXiv preprint arXiv:2506.19248, 2025

    2025

  23. [31]

    Args: Alignment as reward-guided search

    Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. Args: Alignment as reward-guided search. arXiv preprint arXiv:2402.01694, 2024

    2024 arXiv

  24. [32]

    On reinforcement learning and distribution matching for fine-tuning language models with no catastrophic forgetting

    Tomasz Korbak, Hady Elsahar, Germ \'a n Kruszewski, and Marc Dymetman. On reinforcement learning and distribution matching for fine-tuning language models with no catastrophic forgetting. Advances in Neural Information Processing Systems, 35: 0 16203--16220, 2022 a

    2022

  25. [33]

    Rl with kl penalties is better viewed as bayesian inference

    Tomasz Korbak, Ethan Perez, and Christopher L Buckley. Rl with kl penalties is better viewed as bayesian inference. arXiv preprint arXiv:2205.11275, 2022 b

    2022 arXiv

  26. [34]

    A new penalty function method for constrained minimization

    Barry W Kort and Dimitri P Bertsekas. A new penalty function method for constrained minimization. In IEEE Conference on Decision and Control and 11th Symposium on Adaptive Processes, 1972

    1972

  27. [35]

    Unveiling safety vulnerabilities of large language models

    George Kour, Marcel Zalmanovici, Naama Zwerdling, Esther Goldbraich, Ora Nova Fandina, Ateret Anaby-Tavor, Orna Raz, and Eitan Farchi. Unveiling safety vulnerabilities of large language models. arXiv preprint arXiv:2311.04124, 2023

    2023 arXiv

  28. [36]

    On tilted losses in machine learning: Theory and applications

    Tian Li, Ahmad Beirami, Maziar Sanjabi, and Virginia Smith. On tilted losses in machine learning: Theory and applications. Journal of Machine Learning Research, 24 0 (142): 0 1--79, 2023

    2023

  29. [37]

    Deep learning theory review: An optimal control and dynamical systems perspective

    Guan-Horng Liu and Evangelos A Theodorou. Deep learning theory review: An optimal control and dynamical systems perspective. arXiv preprint arXiv:1908.10920, 2019

    1908 arXiv

  30. [38]

    Information theoretic guarantees for policy alignment in large language models

    Youssef Mroueh. Information theoretic guarantees for policy alignment in large language models. arXiv preprint arXiv:2406.05883, 2024

    2024 arXiv

  31. [39]

    Controlled decoding from language models

    Sidharth Mudgal, Jong Lee, Harish Ganapathy, Yaguang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, et al. Controlled decoding from language models. In International Conference on Machine Learning, pages 36486--36503. PMLR, 2024

    2024

  32. [40]

    Webgpt: Browser-assisted question-answering with human feedback

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021

    2021 arXiv

  33. [41]

    2 olmo 2 furious

    Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656, 2024

    2024 arXiv

  34. [42]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 2...

    2022

  35. [43]

    On solving large-scale finite minimax problems using exponential smoothing

    EY Pee and Johannes O Royset. On solving large-scale finite minimax problems using exponential smoothing. Journal of Optimization Theory and Applications, 2011

    2011

  36. [44]

    Information theory: From coding to learning, 2022

    Yury Polyanskiy and Yihong Wu. Information theory: From coding to learning, 2022

    2022

  37. [45]

    Treebon: Enhancing inference-time alignment with speculative tree-search and best-of-n sampling

    Jiahao Qiu, Yifu Lu, Yifan Zeng, Jiacheng Guo, Jiayi Geng, Huazheng Wang, Kaixuan Huang, Yue Wu, and Mengdi Wang. Treebon: Enhancing inference-time alignment with speculative tree-search and best-of-n sampling. arXiv preprint arXiv:2410.16033, 2024

    arXiv

  38. [46]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023

    2023 arXiv

  39. [47]

    Bond: Aligning llms with best-of-n distillation

    Pier Giuseppe Sessa, Robert Dadashi, L \'e onard Hussenot, Johan Ferret, Nino Vieillard, Alexandre Ram \'e , Bobak Shariari, Sarah Perrin, Abe Friesen, Geoffrey Cideron, et al. Bond: Aligning llms with best-of-n distillation. arXiv preprint arXiv:2407.14622, 2024

    2024 arXiv

  40. [48]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024

    2024 arXiv

  41. [49]

    The importance of online data: Understanding preference fine-tuning via coverage

    Yuda Song, Gokul Swamy, Aarti Singh, Drew Bagnell, and Wen Sun. The importance of online data: Understanding preference fine-tuning via coverage. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  42. [50]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33: 0 3008--3021, 2020

    2020

  43. [51]

    Inference scaling f-laws: The limits of llm resampling with imperfect verifiers

    Benedikt Stroebl, Sayash Kapoor, and Arvind Narayanan. Inference scaling f-laws: The limits of llm resampling with imperfect verifiers. arXiv preprint arXiv:2411.17501, 2024

    2024

  44. [52]

    Fast best-of-n decoding via speculative rejection

    Hanshi Sun, Momin Haider, Ruiqi Zhang, Huitao Yang, Jiahao Qiu, Ming Yin, Mengdi Wang, Peter Bartlett, and Andrea Zanette. Fast best-of-n decoding via speculative rejection. arXiv preprint arXiv:2410.20290, 2024

    2024 arXiv

  45. [53]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  46. [54]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  47. [55]

    Interpretable preferences via multi-objective reward modeling and mixture-of-experts

    Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In EMNLP, 2024

    2024

  48. [56]

    Robust variable selection with exponential squared loss

    Xueqin Wang, Yunlu Jiang, Mian Huang, and Heping Zhang. Robust variable selection with exponential squared loss. Journal of the American Statistical Association, 2013

    2013

  49. [57]

    Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint

    Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In Forty-first International Conference on Machine Learning, 2024

    2024

  50. [58]

    Asymptotics of language model alignment

    Joy Qiping Yang, Salman Salamatian, Ziteng Sun, Ananda Theertha Suresh, and Ahmad Beirami. Asymptotics of language model alignment. In 2024 IEEE International Symposium on Information Theory (ISIT), pages 2027--2032. IEEE, 2024

    2024

  51. [59]

    Convergence of the inexact langevin algorithm and score-based generative models in kl divergence

    Kaylee Yingxi Yang and Andre Wibisono. Convergence of the inexact langevin algorithm and score-based generative models in kl divergence. arXiv preprint arXiv:2211.01512, 2022

    2022

  52. [60]

    A theoretical analysis of nash learning from human feedback under general kl-regularized preference

    Chenlu Ye, Wei Xiong, Yuheng Zhang, Nan Jiang, and Tong Zhang. A theoretical analysis of nash learning from human feedback under general kl-regularized preference. arXiv preprint arXiv:2402.07314, 2024

    2024 arXiv

  53. [61]

    Provable offline preference-based reinforcement learning

    Wenhao Zhan, Masatoshi Uehara, Nathan Kallus, Jason D Lee, and Wen Sun. Provable offline preference-based reinforcement learning. arXiv preprint arXiv:2305.14816, 2023

    2023 arXiv

  54. [62]

    Sharp analysis for kl-regularized contextual bandits and rlhf

    Heyang Zhao, Chenlu Ye, Quanquan Gu, and Tong Zhang. Sharp analysis for kl-regularized contextual bandits and rlhf. arXiv preprint arXiv:2411.04625, 2024

    2024 arXiv

  55. [63]

    Calibrating sequence likelihood improves conditional language generation

    Yao Zhao, Mikhail Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J Liu. Calibrating sequence likelihood improves conditional language generation. In The Eleventh International Conference on Learning Representations, 2022

    2022

  56. [64]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36: 0 46595--46623, 2023

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.