Pith. sign in

REVIEW 3 major objections 5 minor 114 references

Dualizing a KL-regularized safety constraint reduces safe decoding to one calibrated dual variable that defines a plug-in reward for frozen language models.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 07:04 UTC pith:UZPANBDB

load-bearing objection Clean dual-calibration layer that turns Safe RLHF into a drop-in score for frozen decoders; theory holds, BoN is the real win, sequence-level exactness is a mild overclaim. the 3 major comments →

arxiv 2607.02781 v1 pith:UZPANBDB submitted 2026-07-02 cs.LG cs.AIcs.CL

Safe Inference-Time Alignment via Lagrangian Reward Augmentation

Yaswanth Chittepu , Ativ Joshi , Sohini Chintala , Scott Niekum This is my paper
classification cs.LG cs.AIcs.CL
keywords inference-time alignmentsafe RLHFLagrangian dualityBest-of-Nreward augmentationlanguage model safetyconstrained decodingdual calibration
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Inference-time alignment steers a frozen language model during decoding with auxiliary scores, avoiding the cost of weight updates. Existing methods still force a hand-tuned scalar trade-off when helpfulness and safety compete. This paper shows that the standard constrained Safe RLHF objective dualizes to a one-dimensional convex problem over a single nonnegative multiplier λ. Estimating that multiplier on a small calibration set produces an augmented reward r − λc that can be dropped into existing sequence-level and token-level decoders. For Best-of-N the calibrated λ solves the expected-cost constrained problem; for token-level reward-guided search it supplies a principled heuristic from the same dual. Experiments report a better helpfulness–harmlessness tradeoff, with Best-of-N approaching finetuning-based safe-alignment baselines without updating the base model.

Core claim

Starting from a KL-regularized objective with a reward model and a cost model under an expected-cost budget, Lagrangian duality reduces the policy optimization to minimizing a convex dual function g(λ) over one nonnegative scalar. The minimizer λ* defines the augmented reward r − λ*c, which can be estimated from a small calibration set of prompts and reference samples and then used as the scoring signal inside existing inference-time methods. For sequence-level samplers such as Best-of-N this construction matches the solution of the expected-cost constrained problem; for token-level methods it yields a dual-calibrated heuristic rather than an exact constrained-policy guarantee.

What carries the argument

Lagrangian Reward Augmentation (LARA): the dual objective g(λ) = β E[log E exp((r − λc)/β)] + λτ, whose derivative is the budget minus expected cost under the Gibbs-tilted policy; binary search for the root of that derivative on a compact interval yields λ̂, after which decoding uses the score r − λ̂c.

Load-bearing premise

The real decoder, especially Best-of-N under the calibrated score, must come close enough to the ideal Gibbs-tilted policy that the dual variable found on the calibration set still meets the population safety budget in deployment.

What would settle it

On held-out prompts, plot expected cost and reward of Best-of-N under the calibrated λ*; if cost still exceeds the target budget while any smaller λ that meets the budget does not, or if every λ that meets the budget collapses helpfulness below the unconstrained baseline, the claimed optimality of the dual calibration fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A safety budget can be enforced at decode time by calibrating one scalar instead of retraining the model.
  • Best-of-N with the calibrated score can approach the helpfulness–harmlessness tradeoff of finetuning-based safe alignment methods.
  • The same dual variable can be reused across sequence-level reranking and token-level reward-guided decoders.
  • Changing the safety budget later requires only re-running the one-dimensional calibration, not a new alignment stage.
  • Finite-sample bounds on the dual estimator quantify how large the calibration set and number of samples per prompt need to be.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • When calibration and deployment prompts differ, the paper’s bootstrap-upper-quantile heuristic for λ is the practical safeguard; systematic cost under-runs or over-runs on a shifted test set would reveal how much that heuristic is doing the real work.
  • Because the exact expected-cost guarantee is sequence-level, hybrids that sample full responses and only lightly bias tokens may inherit stronger constraint satisfaction than pure token-level reward-guided search.
  • The same dual reduction applies to any soft scalar constraint whose dual gradient remains monotone (length, style, domain fidelity), not only harmlessness.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Lagrangian Reward Augmentation (LARA), which transfers the Safe RLHF constrained objective to inference time without weight updates. Dualizing the KL-regularized expected-cost constraint reduces the problem to a one-dimensional convex minimization over a nonnegative multiplier λ; the resulting dual-calibrated score r−λc is then used as a drop-in reward for existing sequence-level (Best-of-N) and token-level (ARGS, PARGS, CD-Q) decoders. Theorem 1 gives a closed-form dual objective and the associated Gibbs policy; Lemma 2 establishes convexity and monotonicity of the dual gradient; Theorem 3 provides a finite-sample concentration bound for the empirical dual minimizer under bounded rewards/costs and strong convexity on a compact interval. Experiments on Llama-3.2-3B and Qwen-2.5-3B with BeaverTails show that BoN under the calibrated score improves the helpfulness–harmlessness tradeoff relative to λ=0 and over-penalized baselines, approaching SafeDPO while remaining below full Safe-RLHF.

Significance. If the construction holds, LARA supplies a principled alternative to hand-tuned safety penalties for frozen-model decoding, which is practically valuable when retraining is expensive or safety budgets change. The dual reduction itself is clean and standard, but packaging it as a decoder-agnostic calibration layer with an explicit finite-sample guarantee and an honest sequence-level vs. token-level distinction is a useful contribution. Strengths include the closed-form dual (Theorem 1), the monotonicity argument that yields efficient binary search (Lemma 2), the finite-sample bound (Theorem 3), and the empirical check that BoN cost crosses τ near the calibrated λ (Figure 1). The result is incremental relative to Safe RLHF / SafeDPO rather than foundational, but it is a concrete and usable step for inference-time safe alignment.

major comments (3)
  1. [Abstract / §3.1 / Theorem 1] Abstract and §3.1 claim that for sequence-level samplers such as Best-of-N the calibrated dual variable “corresponds to the solution of the expected-cost constrained problem” and that the exact expected-cost interpretation of λ⋆ “applies directly” to BoN. Theorem 1 and Lemma 2 characterize the Gibbs policy π⋆_λ, not BoN selection under r−λc among N draws from π_ref. BoN is only an approximate realization of that policy (cf. Beirami et al., cited but not invoked quantitatively). The claim should be restated as approximate/empirical, or the paper should supply a precise BoN–Gibbs error bound under which the population dual minimizer remains meaningful for the actual decoder used in §4.
  2. [§4.1 / Appendix E.2 / Lemma 2] Appendix E.2 states that the deployed λ is the 97.5th percentile of a bootstrap distribution over the dual minimizer, adopted as a “principle of pessimism.” Section 4.1 and RQ1 then present this λ∗ as sitting at the constraint boundary and as the optimality point of the dual problem. A conservative upper quantile systematically overshoots the dual root, so the reported λ is not the dual minimizer whose complementary-slackness property is proved in Lemma 2. Either report the pure dual estimate in the main experiments and treat the bootstrap quantile as an ablation, or revise the optimality language in §4.1 to match the conservative procedure actually used.
  3. [§4.2 / Figure 2b] Under GPT-4o-mini evaluation (Figure 2b), BoN helpfulness and harmlessness are largely insensitive to λ ∈ {0, λ∗, λ_max}, in contrast to the model-based curves in Figure 2a and the clear boundary behavior in Figure 1. If human/GPT judgment does not track the calibrated trade-off, the practical case for dual calibration (as opposed to any fixed moderate penalty) is weakened for the strongest inference-time method. The paper should discuss this discrepancy and, if possible, report whether constraint satisfaction under the cost model still holds for the GPT-preferred generations, or qualify the claim that LARA improves the tradeoff under human-aligned evaluation.
minor comments (5)
  1. [§4 / Appendix E] The choice of β (KL coefficient) is not stated in the main experimental section; only τ is given per model. Please report β and the search upper bound Λ used for calibration.
  2. [Figure 1] Figure 1 caption says “expected cost threshold τ” but does not mark the calibrated λ∗ on the horizontal axis; adding that marker would make the boundary claim immediate.
  3. [§4.2] Token-level methods (ARGS, PARGS, CD-Q) underperform SFT on helpfulness across λ; a short discussion of why dual calibration does not rescue them (prefix–sequence reward mismatch, weight w, top-k) would help readers decide when LARA is worth applying.
  4. [Throughout] Typos / style: “LAgrangian” capitalization in the method title is inconsistent; “harmfullness” appears in a figure caption (Figure 2); “finetuning-based” vs “fine-tuning” is mixed.
  5. [Appendix E.1] SafeDPO and CD-Q implementations are reimplemented without public reference code; a brief validation against reported numbers from Kim et al. / Mudgal et al. (where available) would increase confidence in the baseline comparisons.

Circularity Check

0 steps flagged

No significant circularity: dual reduction and calibration are self-contained; minor overlapping-author background citations are not load-bearing.

full rationale

The central derivation starts from the standard KL-regularized constrained Safe RLHF primal (Eq. 3), dualizes it under a Slater condition, and obtains a closed-form dual objective g(λ) and Gibbs policy π*λ (Theorem 1) via the usual variational identity for KL-regularized reward maximization; the proof in Appendix B is self-contained and does not presuppose the target λ* or the experimental tradeoff. Lemma 2 then shows g' equals the budget residual and g'' ≥ 0, so calibration reduces to a one-dimensional root search that is estimated by Monte Carlo on a held-out calibration set (Eqs. 14–16, Algorithm 1). The finite-sample guarantee (Theorem 3) is a standard concentration + strong-convexity localization argument. Experiments (Figure 1) check that BoN cost crosses τ near the calibrated λ*, which is an empirical validation rather than a definitional identity. Citations to Safe RLHF, HC-RLHF, RAD, and SafeDPO supply background and baselines; they are not used to force uniqueness of the dual construction or to redefine the objective in terms of a fitted quantity. The paper itself already qualifies that only sequence-level samplers inherit an exact expected-cost interpretation while token-level methods receive a heuristic, so the main claim is not circularly overstated. Score 1 reflects only the ordinary presence of overlapping-author background citations that do not carry the derivation.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 1 invented entities

The central claim rests on standard convex dualization plus several modeling and regularity assumptions that are either domain-standard for Safe RLHF or introduced to make the one-dimensional search and finite-sample bound work. Free parameters are the usual RLHF knobs plus the safety budget and the conservative bootstrap quantile. No new physical entities are postulated; LARA is a method, not an ontology.

free parameters (5)
  • safety budget τ = −4 (Llama), 0.8 (Qwen)
    Chosen by hand per base model (τ=−4 for Llama, τ=0.8 for Qwen) to match SFT cost levels; directly determines the target constraint and therefore λ*.
  • KL coefficient β
    Standard RLHF temperature; enters the dual objective and the Gibbs tilt; not re-derived.
  • search upper bound Λ
    Compact interval I=[0,Λ] is required for strong convexity and binary search; chosen by the practitioner.
  • calibration N, K and bootstrap 97.5th percentile = N=2000, K=20, 97.5th percentile
    N=2000 prompts, K=20 samples, and the upper quantile of 10k bootstrap λ* are design choices that set the deployed dual variable; the quantile rule is an unanalyzed robustness heuristic.
  • BoN N and RGTG weight w / top-k = N=20, w=2, k=50
    Decoder hyperparameters (N=20, w=2, top-k=50, temperature 1.2) affect how closely the realized policy matches the theoretical tilt.
axioms (5)
  • domain assumption Slater condition: there exists a feasible policy with expected cost strictly below τ, so strong duality holds.
    Invoked in §3.1 to equate primal and dual; standard for constrained RL but not automatically true for every cost model and budget.
  • domain assumption Reward and cost models are bounded: |r|≤R, |c|≤C for all (x,y).
    Used throughout the dual analysis and for the finite-sample constants a,b,BK in Theorem 3.
  • ad hoc to paper g is μ-strongly convex on the compact interval I=[0,Λ] (variance of cost under π*λ bounded away from zero).
    Required for uniqueness and the localization argument in Theorem 3; the paper restricts to compact I precisely because this need not hold on [0,∞).
  • standard math Policies are absolutely continuous w.r.t. π_ref so conditional KL is well-defined.
    Standard support condition for KL-regularized policy optimization (§3.1).
  • domain assumption Bradley-Terry preference models yield usable reward and cost models from BeaverTails labels.
    Inherited from Safe RLHF / SafeDPO experimental protocol; quality of r and c is load-bearing for any claim about real harmlessness.
invented entities (1)
  • LARA dual-calibrated augmented reward r_λ = r − λc independent evidence
    purpose: Serves as the single drop-in scoring signal that transfers the Safe RLHF trade-off to any inference-time decoder.
    Not a new physical object; it is the Lagrangian net reward already present in constrained RL. Independent evidence is the dual derivation itself plus empirical trade-off plots.

pith-pipeline@v1.1.0-grok45 · 24409 in / 3755 out tokens · 41810 ms · 2026-07-12T07:04:27.302964+00:00 · methodology

0 comments
read the original abstract

Inference-time alignment steers a frozen language model during decoding using auxiliary reward signals, avoiding the cost of repeated weight updates. However, existing inference-time alignment methods typically optimize a single scalar score, so explicit safety constraints must either be ignored or encoded through manually tuned penalties. We propose Lagrangian Reward Augmentation (LARA), a general inference-time alignment framework under safety constraints. Starting from a KL-regularized constrained objective with a reward model and a cost model, LARA dualizes the constraint and reduces the optimization problem to a one-dimensional convex problem over a nonnegative dual variable. Estimated on a small calibration set, this dual variable defines an augmented reward that can be used as a drop-in scoring signal within existing inference-time alignment methods. For sequence-level sampling methods, such as Best-of-N reranking, the calibrated dual variable corresponds to the solution of the expected-cost constrained problem. For token-level reward-guided decoding methods, the same construction yields a principled dual-calibrated heuristic rather than an exact constrained-policy guarantee. We evaluate LARA on both sequence-level and token-level inference-time alignment methods, and find that LARA improves the helpfulness-harmlessness tradeoff, with Best-of-N achieving the best performance among inference-time methods, approaching finetuning-based direct alignment baselines.

Figures

Figures reproduced from arXiv: 2607.02781 by Ativ Joshi, Scott Niekum, Sohini Chintala, Yaswanth Chittepu.

Figure 1
Figure 1. Figure 1: Expected helpfulness E[r] and expected cost E[c] of the BoN policy as a function of λ, for the Llama base model on the left and Qwen base model on the right. The dashed vertical line indicates the expected cost threshold τ. first finetuned on the Alpaca dataset (Taori et al., 2023) to obtain the SFT policy, which serves as the reference policy for all methods. The reward model and cost model are trained us… view at source ↗
Figure 2
Figure 2. Figure 2: Helpfulness vs Harmfullness for LLama3.2-3b under both model evaluation (left) [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Helpfulness vs Harmfulness for Qwen2.5-3b under both model evaluation (left) [PITH_FULL_IMAGE:figures/full_fig_p022_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

114 extracted references · 30 linked inside Pith

  1. [2]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. T he method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952

    1952

  2. [3]

    Reinforcement learning from human feedback with high-confidence safety guarantees

    Yaswanth Chittepu, Blossom Metevier, Will Schwarzer, Austin Hoag, Scott Niekum, and Philip S Thomas. Reinforcement learning from human feedback with high-confidence safety guarantees. In Reinforcement Learning Conference, 2025

    2025

  3. [5]

    Brown, Miljan Martic, Shane Legg, and Dario Amodei

    Paul Francis Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. ArXiv, abs/1706.03741, 2017. URL https://api.semanticscholar.org/CorpusID:4787508

    Pith/arXiv arXiv 2017

  4. [9]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, 2022. URL https://api.semanticscholar.org/CorpusID:252992904

    2022

  5. [10]

    Realtoxicityprompts: Evaluating neural toxic degeneration in language models

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. In Findings of the association for computational linguistics: EMNLP 2020, pp.\ 3356--3369, 2020

    2020

  6. [11]

    The llama 3 herd of models, 2024

    Aaron Grattafiori et al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    Pith/arXiv arXiv 2024

  7. [12]

    Value augmented sampling for language model alignment and personalization, 2024

    Seungwook Han, Idan Shenfeld, Akash Srivastava, Yoon Kim, and Pulkit Agrawal. Value augmented sampling for language model alignment and personalization, 2024. URL https://arxiv.org/abs/2405.06639

    Pith/arXiv arXiv 2024

  8. [13]

    Beavertails: Towards improved safety alignment of llm via a human-preference dataset

    Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Systems, 36: 0 24678--24704, 2023

    2023

  9. [14]

    u chemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan G \

    Enkelejda Kasneci, Kathrin Se ler, Stefan K \"u chemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan G \"u nnemann, Eyke H \"u llermeier, et al. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual differences, 103: 0 102274, 2023

    2023

  10. [15]

    Gpt-4 passes the bar exam

    Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. Gpt-4 passes the bar exam. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 382 0 (2270), 2024

    2024

  11. [18]

    Performance of chatgpt on usmle: potential for ai-assisted medical education using large language models

    Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepa \ n o, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al. Performance of chatgpt on usmle: potential for ai-assisted medical education using large language models. PLoS digital health, 2 0 (2): 0 e0000198, 2023

    2023

  12. [19]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023. URL https://arxiv.org/abs/2309.06180

    Pith/arXiv arXiv 2023

  13. [20]

    Foundation models for generalist medical artificial intelligence

    Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar. Foundation models for generalist medical artificial intelligence. Nature, 616 0 (7956): 0 259--265, 2023

    2023

  14. [22]

    Webgpt: Browser-assisted question-answering with human feedback, 2022

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question-answering with human feedback, 2022. URL https://a...

    Pith/arXiv arXiv 2022

  15. [23]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022

    2022

  16. [24]

    Computational optimal transport, 2020

    Gabriel Peyré and Marco Cuturi. Computational optimal transport, 2020. URL https://arxiv.org/abs/1803.00567

    Pith/arXiv arXiv 2020

  17. [25]

    Qwen2.5 technical report, 2025

    Qwen et al. Qwen2.5 technical report, 2025. URL https://arxiv.org/abs/2412.15115

    Pith/arXiv arXiv 2025

  18. [26]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36: 0 53728--53741, 2023

    2023

  19. [27]

    Scaling laws for reward model overoptimization in direct alignment algorithms

    Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sushil Sikchi, Joey Hejna, Brad Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms. Advances in Neural Information Processing Systems, 37: 0 126207--126242, 2024

    2024

  20. [30]

    Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano

    Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022. URL https://arxiv.org/abs/2009.01325

    Pith/arXiv arXiv 2022

  21. [31]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford A lpaca: An instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca, 2023

    2023

  22. [32]

    Preventing undesirable behavior of intelligent machines

    Philip S Thomas, Bruno Castro da Silva, Andrew G Barto, Stephen Giguere, Yuriy Brun, and Emma Brunskill. Preventing undesirable behavior of intelligent machines. Science, 366 0 (6468): 0 999--1004, 2019

    2019

  23. [33]

    Asymptotic statistics, volume 3

    Aad W Van der Vaart. Asymptotic statistics, volume 3. Cambridge university press, 2000

    2000

  24. [35]

    Fudge: Controlled text generation with future discriminators

    Kevin Yang and Dan Klein. Fudge: Controlled text generation with future discriminators. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.\ 3511--3535, 2021

    2021

  25. [36]

    A large language model for electronic health records

    Xi Yang, Aokun Chen, Nima PourNejatian, Hoo Chang Shin, Kaleb E Smith, Christopher Parisien, Colin Compas, Cheryl Martin, Anthony B Costa, Mona G Flores, et al. A large language model for electronic health records. NPJ digital medicine, 5 0 (1): 0 194, 2022

    2022

  26. [38]

    arXiv preprint arXiv:2310.17022 , year=

    Controlled decoding from language models , author=. arXiv preprint arXiv:2310.17022 , year=

    Pith/arXiv arXiv

  27. [39]

    arXiv preprint arXiv:2401.01879 , year=

    Theoretical guarantees on the best-of-n alignment policy , author=. arXiv preprint arXiv:2401.01879 , year=

    Pith/arXiv arXiv

  28. [40]

    arXiv preprint arXiv:2402.01694 , year=

    Args: Alignment as reward-guided search , author=. arXiv preprint arXiv:2402.01694 , year=

    Pith/arXiv arXiv

  29. [41]

    Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    FUDGE: Controlled text generation with future discriminators , author=. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    2021

  30. [42]

    arXiv preprint arXiv:2406.07780 , year=

    A critical look at tokenwise reward-guided text generation , author=. arXiv preprint arXiv:2406.07780 , year=

    arXiv

  31. [43]

    Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model

    Deng, Haikang and Raffel, Colin. Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. doi:10.18653/v1/2023.emnlp-main.721

    doi:10.18653/v1/2023.emnlp-main.721 2023

  32. [44]

    arXiv preprint arXiv:2502.04517 , year=

    Towards cost-effective reward guided text generation , author=. arXiv preprint arXiv:2502.04517 , year=

    Pith/arXiv arXiv

  33. [45]

    arXiv preprint arXiv:2310.12773 , year=

    Safe rlhf: Safe reinforcement learning from human feedback , author=. arXiv preprint arXiv:2310.12773 , year=

    Pith/arXiv arXiv

  34. [46]

    Advances in neural information processing systems , volume=

    Direct preference optimization: Your language model is secretly a reward model , author=. Advances in neural information processing systems , volume=

  35. [47]

    arXiv preprint arXiv:1910.00177 , year=

    Advantage-weighted regression: Simple and scalable off-policy reinforcement learning , author=. arXiv preprint arXiv:1910.00177 , year=

    Pith/arXiv arXiv 1910

  36. [48]

    Advances in Neural Information Processing Systems , volume=

    Scaling laws for reward model overoptimization in direct alignment algorithms , author=. Advances in Neural Information Processing Systems , volume=

  37. [49]

    arXiv preprint arXiv:2505.20065 , year=

    Safedpo: A simple approach to direct preference optimization with enhanced safety , author=. arXiv preprint arXiv:2505.20065 , year=

    arXiv

  38. [50]

    Advances in Neural Information Processing Systems , volume=

    Beavertails: Towards improved safety alignment of llm via a human-preference dataset , author=. Advances in Neural Information Processing Systems , volume=

  39. [51]

    Attention is All you Need , url =

    Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, ukasz and Polosukhin, Illia , booktitle =. Attention is All you Need , url =

  40. [52]

    ArXiv , year=

    Deep Reinforcement Learning from Human Preferences , author=. ArXiv , year=

  41. [53]

    Constitutional

    Yuntao Bai and others , journal=. Constitutional. 2022 , volume=

    2022

  42. [54]

    ArXiv , year=

    Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog , author=. ArXiv , year=

  43. [55]

    2022 , eprint=

    Learning to summarize from human feedback , author=. 2022 , eprint=

    2022

  44. [56]

    2017 , eprint=

    Proximal Policy Optimization Algorithms , author=. 2017 , eprint=

    2017

  45. [57]

    International Conference on Machine Learning , year=

    Scaling Laws for Reward Model Overoptimization , author=. International Conference on Machine Learning , year=

  46. [58]

    2004 , publisher=

    Convex optimization , author=. 2004 , publisher=

    2004

  47. [59]

    University of Pennsylvania Philadelphia, PA , volume=

    Fundamentals of optimization theory with applications to machine learning , author=. University of Pennsylvania Philadelphia, PA , volume=

  48. [60]

    2025 , eprint=

    Qwen2.5 Technical Report , author=. 2025 , eprint=

    2025

  49. [61]

    Xing and Hao Zhang and Joseph E

    Lianmin Zheng and Wei-Lin Chiang and Ying Sheng and Siyuan Zhuang and Zhanghao Wu and Yonghao Zhuang and Zi Lin and Zhuohan Li and Dacheng Li and Eric P. Xing and Hao Zhang and Joseph E. Gonzalez and Ion Stoica , year=. Judging. 2306.05685 , archivePrefix=

    Pith/arXiv arXiv

  50. [62]

    2021 , publisher=

    Constrained Markov decision processes , author=. 2021 , publisher=

    2021

  51. [63]

    Rank analysis of incomplete block designs: I

    Bradley, Ralph Allan and Terry, Milton E , journal=. Rank analysis of incomplete block designs: I. 1952 , publisher=

    1952

  52. [64]

    Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in

    Arash Ahmadian and Chris Cremer and Matthias Gallé and Marzieh Fadaee and Julia Kreutzer and Olivier Pietquin and Ahmet Üstün and Sara Hooker , year=. Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in. 2402.14740 , archivePrefix=

    Pith/arXiv arXiv

  53. [65]

    DeepRLStructPred@ICLR , year=

    Buy 4 REINFORCE Samples, Get a Baseline for Free! , author=. DeepRLStructPred@ICLR , year=

  54. [66]

    ArXiv , year=

    Proximal Policy Optimization Algorithms , author=. ArXiv , year=

  55. [67]

    Machine learning , volume=

    Simple statistical gradient-following algorithms for connectionist reinforcement learning , author=. Machine learning , volume=. 1992 , publisher=

    1992

  56. [68]

    2024 , eprint=

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models , author=. 2024 , eprint=

    2024

  57. [69]

    Reinforcement Learning Conference , year=

    Reinforcement learning from human feedback with high-confidence safety guarantees , author=. Reinforcement Learning Conference , year=

  58. [70]

    Science , volume=

    Preventing undesirable behavior of intelligent machines , author=. Science , volume=. 2019 , publisher=

    2019

  59. [71]

    2020 , eprint=

    Computational Optimal Transport , author=. 2020 , eprint=

    2020

  60. [72]

    2016 , publisher=

    Linear programming and extensions , author=. 2016 , publisher=

    2016

  61. [73]

    International conference on machine learning , pages=

    Computational optimal transport: Complexity by accelerated gradient descent is better than by Sinkhorn’s algorithm , author=. International conference on machine learning , pages=. 2018 , organization=

    2018

  62. [74]

    Advances in neural information processing systems , volume=

    Sinkhorn distances: Lightspeed computation of optimal transport , author=. Advances in neural information processing systems , volume=

  63. [75]

    James Bradbury and Roy Frostig and Peter Hawkins and Matthew James Johnson and Chris Leary and Dougal Maclaurin and George Necula and Adam Paszke and Jake Vander

  64. [76]

    Advances in Neural Information Processing Systems 32 , pages =

    PyTorch: An Imperative Style, High-Performance Deep Learning Library , author =. Advances in Neural Information Processing Systems 32 , pages =. 2019 , publisher =

    2019

  65. [77]

    2015 , url=

    Optimal Transport for Applied Mathematicians: Calculus of Variations, PDEs, and Modeling , author=. 2015 , url=

    2015

  66. [78]

    Advances in Neural Information Processing Systems , volume=

    Distributional preference alignment of llms via optimal transport , author=. Advances in Neural Information Processing Systems , volume=

  67. [79]

    Mathematical finance , volume=

    Coherent measures of risk , author=. Mathematical finance , volume=. 1999 , publisher=

    1999

  68. [80]

    Journal of banking & finance , volume=

    Spectral measures of risk: A coherent representation of subjective risk aversion , author=. Journal of banking & finance , volume=. 2002 , publisher=

    2002

  69. [81]

    Advances in mathematical economics , pages=

    On law invariant coherent risk measures , author=. Advances in mathematical economics , pages=. 2001 , publisher=

    2001

  70. [82]

    2024 , eprint=

    Tilted Quantile Gradient Updates for Quantile-Constrained Reinforcement Learning , author=. 2024 , eprint=

    2024

  71. [83]

    NPJ digital medicine , volume=

    A large language model for electronic health records , author=. NPJ digital medicine , volume=. 2022 , publisher=

    2022

  72. [84]

    Nature , volume=

    Foundation models for generalist medical artificial intelligence , author=. Nature , volume=. 2023 , publisher=

    2023

  73. [85]

    Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences , volume=

    Gpt-4 passes the bar exam , author=. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences , volume=. 2024 , publisher=

    2024

  74. [86]

    Learning and individual differences , volume=

    ChatGPT for good? On opportunities and challenges of large language models for education , author=. Learning and individual differences , volume=. 2023 , publisher=

    2023

  75. [87]

    PLoS digital health , volume=

    Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models , author=. PLoS digital health , volume=. 2023 , publisher=

    2023

  76. [88]

    Findings of the association for computational linguistics: EMNLP 2020 , pages=

    Realtoxicityprompts: Evaluating neural toxic degeneration in language models , author=. Findings of the association for computational linguistics: EMNLP 2020 , pages=

    2020

  77. [89]

    arXiv preprint arXiv:2112.04359 , year=

    Ethical and social risks of harm from language models , author=. arXiv preprint arXiv:2112.04359 , year=

    Pith/arXiv arXiv

  78. [90]

    arXiv preprint arXiv:2209.07858 , year=

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned , author=. arXiv preprint arXiv:2209.07858 , year=

    Pith/arXiv arXiv

  79. [91]

    arXiv preprint arXiv:2209.14375 , year=

    Improving alignment of dialogue agents via targeted human judgements , author=. arXiv preprint arXiv:2209.14375 , year=

    Pith/arXiv arXiv

  80. [92]

    Advances in neural information processing systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in neural information processing systems , volume=

Showing first 80 references.