REVIEW 5 major objections 4 minor 58 references
Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approach
T0 review · 5 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read IRO converges to the optimal aligned policy for a frozen LLM by iteratively training and applying small value functions, matching Best-of-N quality with exponentially fewer test-time tokens.
desk verdict IRO is a promising empirical method, but the theory is disconnected from the implemented decoder; the paper deserves serious review with revisions to close that gap. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the iterated reweighting identity $\pi_t(a\mid s)\propto \pi_{\mathrm{base}}(a\mid s)\exp\left(\sum_{i=0}^{t-1}\hat V_{\pi_i}(s,a)/\beta_i\right)$, which is the closed-form solution of a KL-constrained surrogate objective in the style of trust-region policy optimization and turns policy improvement into value-function learning. Each step regresses a lightweight value model to the terminal rewards of completions drawn from the current reweighted policy; at test time, the same value models score chunks during a value-guided beam search, a diversity-first rule keeps redundant candidates from crowding out diverse high-value prefixes, and the reward model picks the final continuation. This identity is what connects a purely weight-free decoding procedure to the theory of policy iteration.
What would settle it
On a synthetic token-level MDP with a known optimal policy and value function, run Algorithm 2 exactly as implemented and count how often the returned continuation is optimal; compare this with the predicted probability $(U/|\mathcal{A}|)^{H/L}$ and with Best-of-N's success rate at $N=U^{H/L}$. If the implemented scorer, which omits the base-model likelihood term, succeeds materially below the idealized reweighted policy, the exponential-efficiency claim is tied to the update in Eq. (10) rather than to the deployed decoder.
Extended reading notes
Core claim
The paper's central claim is that the optimal aligned policy over a frozen base model has the explicit form $\pi_t(a\mid s)\propto \pi_{\mathrm{base}}(a\mid s)\exp\left(\sum_{i=0}^{t-1}\hat V_{\pi_i}(s,a)/\beta_i\right)$, and that IRO realizes this form by alternating regression of a value function to Monte Carlo returns with reweighted guided decoding. Theorem 1 states that, with bounded value-function classes and a concentrability assumption, choosing $\beta_t=\sqrt{t+1}/\omega$ makes the gap to the optimal policy shrink as $O(T^{-1/2})$ plus a least-squares estimation term, so IRO is a policy-iteration method. Proposition 1 states that if the accumulated value functions are close to the optimal value function $V^*$, then IRO with beam width $U$ has success probability $(U/|\mathcal{A}|)^{H/L}$; matching Best-of-N forces $U=N^{1/H}$, giving token-cost and query-cost ratios $(BK)^{H/L-1}$ and $(L/HI)(BK)^{H/L-1}$, respectively. The paper reads these bounds as: Best-of-N quality can be obtained with exponentially fewer tokens at test time.
Load-bearing premise
The load-bearing premise is that the small learned value functions approximate the optimal value function closely enough to steer the reweighted policy, and that the deployed scorer—which omits the base model's own likelihood term—still behaves like the theoretical reweighted policy; if either gap is substantial, the convergence and exponential-efficiency guarantees do not transfer to the implemented decoder.
Editorial extensions
If this is right
- A frozen or deployed model can be re-aligned to a new reward signal without weight access, since only small value models are trained by regression on sampled completions.
- Alignment improves across iterations: later value functions correct biases left by earlier ones, rather than relying on a single imperfect guidance pass.
- At equal success probability, the token cost of IRO relative to Best-of-N scales like $(BK)^{H/L-1}$, so the advantage grows as the generation horizon lengthens.
- Small value models can guide much larger base models (1B guiding 6.9B and 7B guiding 70B in the paper), which is what makes the frozen-model setting practical.
Reading between the lines
- Because the theorems are proven for the full reweighted policy in Eq. (10) while the deployed decoder scores candidates with only the value-function sum, a direct ablation that re-adds the base model's own likelihood term would isolate whether the implemented decoder is the algorithm being analyzed.
- The same reweighting identity suggests an online variant: continue training value functions on reward feedback collected during deployment, turning IRO into a continually adapting alignment loop for a model whose weights remain hidden.
- The diversity-first principle implies that candidate redundancy, not just value accuracy, limits greedy guided search; a controlled test of clustering-aware selection with the same value functions would quantify that contribution separately from the iterative value updates.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Iterative Reweight-then-Optimize (IRO), a test-time alignment method that leaves the base LLM frozen and instead trains a sequence of lightweight value functions, each regressed on Monte Carlo returns from data generated by the previous reweighted policy. At test time, these value functions guide a beam-search-style decoder over the frozen base policy, and the final response is selected by an outcome reward model. The authors claim that IRO is a form of policy iteration converging to the optimal policy (Theorem 1) and that, under idealized conditions, it matches Best-of-N performance with exponentially fewer tokens and reward queries (Proposition 1). Experiments on TL;DR summarization and AlpacaEval 2.0 show consistent win-rate gains over BoN, ARGS, and weak-to-strong search when using 1B or 7B value models to guide 1B, 6.9B, 8B, or 70B base models.
Significance. If the theoretical claims held for the implemented algorithm, the paper would make a substantial contribution: it would provide the first test-time alignment method that provably performs policy iteration on a frozen model, with a quantitative efficiency argument against Best-of-N, and it would open the door to API-only reinforcement fine-tuning. The empirical results are extensive and encouraging, including weak-to-strong generalization and ablations over chunk length, data size, and search budget, and the authors provide a code link. However, the paper's central theoretical statements are currently proved for a softmax policy that the implemented decoder does not run, and the efficiency proposition relies on assumptions about uniform candidate sampling and exact value-function approximation that are not satisfied by the deployment. The convergence and exponential-efficiency claims therefore do not yet cover the algorithm whose results are reported.
major comments (5)
- [§4.1, Theorem 1] Theorem 1 is stated and proved for the token-level softmax policy in Eq. (10) and Eq. (33), namely π̂_t(a|s) ∝ π̂_{t-1}(a|s) exp(V̂_{π̂_{t-1}}(s,a)/β_{t-1}) ∝ π_base(a|s) exp(Σ_i V̂_{π̂_i}(s,a)/β_i). This update explicitly contains the log-policy term log π_base. The implemented decoder in Algorithm 2, however, scores each chunk by V(y_j) = Σ_i (1/β_i) V̂_i(x,y_j) with no log π_base term and then deterministically keeps the top-K beams, rather than sampling from the softmax policy. Appendix G.1.5 confirms that the log π_t term is deliberately omitted during scoring. Consequently, Theorem 1's convergence and regret bounds do not apply to the deployed search procedure, and the reported empirical gains are not protected by the paper's main theoretical guarantee.
- [§4.1, Assumption 2] Assumption 2 posits concentrability of π_base to π* (d^{π*}_h/d^{π_base}_h ≤ C_ST), but the value functions in Eq. (9) are fitted on data collected under the sequence of policies π̂_t. The proof of Theorem 1 needs a distribution-mismatch bound between d^{π̂_t}_h and d^{π_base}_h (or between d^{π̂_t}_h and d^{π*}_h) to control the ϵ2(t) term in Eq. (34). In Eq. (44) the bound is written as C_ST E_{s∼d^{π_1}_h}(...), which is not justified by Assumption 2 as stated, since π_1 (or π̂_t) is not π_base. As written, the estimation-error term in the theorem is not rigorously controlled.
- [§4.2, Proposition 1] Proposition 1's cost ratios in Eq. (14) are derived under two modeling assumptions that the implementation does not satisfy. First, the proof in Appendix B.2 assumes that at each chunk step the algorithm samples U candidates uniformly from the full action space A^L (see Eq. (24)), while Algorithm 2 samples successors from the base policy π_base and uses beam selection, not uniform proposals. Second, the proof computes BoN's success probability as N/|A|^H in Eq. (21), which holds only for uniform action draws; for a non-uniform base policy the probability of sampling the optimal trajectory under Best-of-N depends on π_base's probability mass on τ*, and the ratio in Eq. (14) does not follow. The exponential-efficiency claim therefore has not been established for the actual candidate-generation process.
- [§4.2, Proposition 1] Proposition 1 takes as input the condition that the weighted sum of the I learned value functions approximates V*, i.e. Σ_{i=0}^{I-1} V_i/β_i ≈ V*. This is precisely the object that the training procedure (Eq. (9)) is supposed to produce, so the exponential-efficiency claim is conditional on the training having already succeeded, rather than derived from the stated value-learning objective with its finite-sample error. Remark 4.1 concedes that with a small number of iterations the value functions may not approximate V* well, and that reweighting over a sampled subset adds approximation error. The proposition's conclusion therefore holds only under an idealized condition that the paper does not show is met by the empirical setup.
- [§3, Algorithm 2] Algorithm 2's pseudocode describes only value-based beam selection (line 6) and does not include the diversity-first principle introduced in Remark 3.1 or implemented in the experiments. Since the ablation in Fig. 7 shows that diversity-first is important for the reported gains, the pseudocode does not fully specify the evaluated algorithm. The paper should either add the clustering/diversity step to Algorithm 2 or state in the pseudocode that diversity-first is applied as a separate search heuristic.
minor comments (4)
- [§4.1, Eq. (10)] The index range in Eq. (10) is written as 't = 0, 2, ···', which should presumably be 't = 0, 1, 2, ···'.
- [§3, Algorithm 1] Algorithm 1 returns 'π̂_T constructed according to (3)', but (3) defines the optimal value function V*, not a policy; the reference should presumably be to Eq. (10).
- [Appendix G.1.5] The statement 'we omit the term log π_t(a_h|s_h) when scoring the candidates, as it is already revealed during candidate generation' is misleading: in beam search, the log-probability of a candidate is not a constant across candidates and affects their relative ranking, so omitting it changes the induced policy. This point should be discussed explicitly.
- [Appendix H] The heading 'Example 1 in the TL;DR Taks' contains a typo: 'Taks' should be 'Task'.
Circularity Check
No circular derivation: the TRPO-style policy iteration proof is self-contained; the value-only decoder gap is a soundness issue, not a circularity.
full rationale
The paper's main convergence claim (Theorem 1) is proved for the softmax reweighting policy (10)/(33), using standard performance-difference and least-squares lemmas from prior external work ([9], [44]) and a self-contained Lagrangian derivation in Appendix B.1; the one self-citation ([52]) is not load-bearing because the closed-form update is re-derived in the appendix. Proposition 1 is an explicitly conditional cost comparison: it assumes Σ(1/β_i)V_i≈V* and uniform proposals, and derives the token/query ratios (14) from that assumption; because the assumption is stated as an 'ideal situation' (Remark 4.1) rather than claimed to be guaranteed, the proposition is a conditional analysis, not a result whose conclusion is identical to its input. The most serious issue found is an implementation/theory mismatch: Algorithm 2 scores chunks by value-function sum only and deliberately omits log π_base (Appendix G.1.5), so Theorem 1's guarantees are not shown to apply to the deployed decoder, and Proposition 1's uniform-proposal assumption is not matched by sampling from π_base. This is a correctness/soundness gap that should be weighed separately; it is not a circular reduction in which an equation is equivalent to its own input by construction.
Assumptions & free parameters
free parameters (5)
- beta_t schedule =
beta1=1, beta2=2, beta3=2.5 on Alpaca; beta=1 on TL;DR
- chunk length L =
16
- beam width K and successors B =
K=4, B=4
- value function training schedule =
LR 3e-6, batch size 256 or 512, 10 epochs
- sampling hyperparameters for candidate generation =
TL;DR: T=0.7, top-k=50, top-p=1.0; Alpaca: T=0.6, top-k=50, top-p=0.9
assumptions (6)
- domain assumption LLM generation is a finite-horizon MDP with deterministic transitions and terminal-only reward
- standard math The TRPO KL-constrained surrogate in Eq. (6) is a valid approximation of the performance difference
- domain assumption Assumption 1: true Q functions lie in a bounded function class F
- domain assumption Assumption 2: the base policy covers the optimal policy with finite concentrability C_ST
- ad hoc to paper The weighted sum of I learned value functions approximates the true optimal value function V*
- ad hoc to paper Algorithm 2's value-only beam scoring behaves like the theoretical reweighted policy in Eq. (10)
Cite this review
Pith. "Pith review of Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approach." pith.science (2026). https://pith.science/paper/JDYNQUMR
@misc{pith2026250617828,
author = {Pith},
title = {Pith review of: Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approach},
year = {2026},
howpublished = {\url{https://pith.science/paper/JDYNQUMR}},
note = {Machine review of arXiv:2506.17828}
}
read the original abstract
Aligning large language models (LLMs) with human preferences usually requires fine-tuning methods such as RLHF and DPO. These methods directly optimize the model parameters, so they cannot be used in test-time to improve model performance, nor are they applicable when the model weights are not accessible. In contrast, test-time methods sidestep weight updates by leveraging reward functions to guide and improve output quality. However, they incur high inference costs, and their one-shot guidance is often based on imperfect reward or value functions, leading to suboptimal outputs. In this work, we present a method named Iterative Reweight-then-Optimize (IRO), a reinforcement learning (RL) framework that performs RL-style alignment of the (frozen) base model without touching its parameters. During training, each iteration (i) samples candidates from the base model, (ii) resamples using current value functions, and (iii) trains a new lightweight value function that guides the next decoding pass. At test time, the value functions are used to guide the base model generation via a search-based optimization process. Notably, users can apply IRO to align a model on their own dataset, similar to OpenAI's reinforcement fine-tuning (RFT), but without requiring access to the model weights.
Figures
Figures from the paper (9 more)
Reference graph
Works this paper leans on
-
[1]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023
arXiv 2023
-
[2]
Reinforcement learning: Theory and algorithms
Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, 32, 2019
work page 2019
-
[3]
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms.arXiv preprint arXiv:2402.14740, 2024
arXiv 2024
-
[4]
A general theoretical paradigm to understand learning from human preferences
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences. arXiv preprint arXiv:2310.12036, 2023
-
[5]
Qwen technical report.arXiv preprint arXiv:2309.16609, 2023
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609, 2023
arXiv 2023
-
[6]
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022
arXiv 2022
-
[7]
A survey of monte carlo tree search methods.IEEE Transactions on Computational Intelligence and AI in games, 4(1):1–43, 2012
Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods.IEEE Transactions on Computational Intelligence and AI in games, 4(1):1–43, 2012
2012
-
[8]
Transfer q star: Principled decoding for llm alignment
Souradip Chakraborty, Soumya Suvra Ghosal, Ming Yin, Dinesh Manocha, Mengdi Wang, Amrit Singh Bedi, and Furong Huang. Transfer q star: Principled decoding for llm alignment. arXiv preprint arXiv:2405.20495, 2024
arXiv 2024
Show all 58 references
-
[9]
Dataset reset policy optimization for rlhf.arXiv preprint arXiv:2404.08495, 2024
Jonathan D Chang, Wenhao Zhan, Owen Oertell, Kianté Brantley, Dipendra Misra, Jason D Lee, and Wen Sun. Dataset reset policy optimization for rlhf.arXiv preprint arXiv:2404.08495, 2024
2024 arXiv
-
[10]
Stop summation: Min-form credit assignment is all process reward model needs for reasoning.arXiv preprint arXiv:2504.15275, 2025
Jie Cheng, Ruixi Qiao, Lijun Li, Chao Guo, Junle Wang, Gang Xiong, Yisheng Lv, and Fei-Yue Wang. Stop summation: Min-form credit assignment is all process reward model needs for reasoning.arXiv preprint arXiv:2504.15275, 2025
2025
-
[11]
Ultrafeedback: Boosting language models with high-quality feedback
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. 2023
2023
-
[12]
Raft: Reward ranked finetuning for generative foundation model alignment
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767, 2023
2023 arXiv
-
[13]
Rlhf workflow: From reward modeling to online rlhf, 2024
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf, 2024
2024
-
[14]
Length-controlled alpacaeval: A simple way to debias automatic evaluators.arXiv preprint arXiv:2404.04475, 2024
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators.arXiv preprint arXiv:2404.04475, 2024
2024 arXiv
-
[15]
Alpacafarm: A simulation framework for methods that learn from human feedback.Advances in Neural Information Processing Systems, 36:30039–30069, 2023
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback.Advances in Neural Information Processing Systems, 36:3...
2023
-
[16]
Stop regressing: Training value functions via classification for scalable deep rl.arXiv preprint arXiv:2403.03950, 2024
Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taïga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, et al. Stop regressing: Training value functions via classification for scalable deep rl.arXiv preprint arXiv:2403.03950, 2024
2024 arXiv
-
[17]
Scaling laws for reward model overoptimization
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835–10866. PMLR, 2023
2023
-
[18]
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025
2025 arXiv
-
[19]
Value augmented sampling for language model alignment and personalization.arXiv preprint arXiv:2405.06639, 2024
Seungwook Han, Idan Shenfeld, Akash Srivastava, Yoon Kim, and Pulkit Agrawal. Value augmented sampling for language model alignment and personalization.arXiv preprint arXiv:2405.06639, 2024
2024 arXiv
-
[20]
Inverse preference learning: Preference-based rl without a reward function
Joey Hejna and Dorsa Sadigh. Inverse preference learning: Preference-based rl without a reward function. Advances in Neural Information Processing Systems, 36, 2024
2024
-
[21]
Deal: Decoding-time alignment for large language models
James Y Huang, Sailik Sengupta, Daniele Bonadiman, Yi-an Lai, Arshit Gupta, Nikolaos Pappas, Saab Mansour, Katrin Kirchhoff, and Dan Roth. Deal: Decoding-time alignment for large language models. arXiv preprint arXiv:2402.06147, 2024
2024
-
[22]
The n+ implementation details of rlhf with ppo: A case study on tl; dr summarization.arXiv preprint arXiv:2403.17031, 2024
Shengyi Huang, Michael Noukhovitch, Arian Hosseini, Kashif Rasul, Weixun Wang, and Lewis Tunstall. The n+ implementation details of rlhf with ppo: A case study on tl; dr summarization.arXiv preprint arXiv:2403.17031, 2024
2024 arXiv
-
[23]
Llm-blender: Ensembling large language models with pairwise ranking and generative fusion.arXiv preprint arXiv:2306.02561, 2023
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion.arXiv preprint arXiv:2306.02561, 2023
2023 arXiv
-
[24]
A natural policy gradient.Advances in neural information processing systems, 14, 2001
Sham M Kakade. A natural policy gradient.Advances in neural information processing systems, 14, 2001
2001
-
[25]
Args: Alignment as reward-guided search.arXiv preprint arXiv:2402.01694, 2024
Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. Args: Alignment as reward-guided search.arXiv preprint arXiv:2402.01694, 2024
2024 arXiv
-
[26]
Aligning large language models with representation editing: A control perspective
Lingkai Kong, Haorui Wang, Wenhao Mu, Yuanqi Du, Yuchen Zhuang, Yifei Zhou, Yue Song, Rongzhi Zhang, Kai Wang, and Chao Zhang. Aligning large language models with representation editing: A control perspective. arXiv preprint arXiv:2406.05954, 2024
2024 arXiv
-
[27]
Optimization issues in kl-constrained approximate policy iteration.arXiv preprint arXiv:2102.06234, 2021
Nevena Lazić, Botao Hao, Yasin Abbasi-Yadkori, Dale Schuurmans, and Csaba Szepesvári. Optimization issues in kl-constrained approximate policy iteration.arXiv preprint arXiv:2102.06234, 2021
2021 arXiv
-
[28]
Cascade reward sampling for efficient decoding-time alignment.arXiv preprint arXiv:2406.16306, 2024
Bolian Li, Yifan Wang, Anamika Lochab, Ananth Grama, and Ruqi Zhang. Cascade reward sampling for efficient decoding-time alignment.arXiv preprint arXiv:2406.16306, 2024
2024 arXiv
-
[29]
Alpacaeval: An automatic evaluator of instruction-following models, 2023
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models, 2023
2023
-
[30]
Let’s verify step by step
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InThe Twelfth International Conference on Learning Representations, 2023
2023
-
[31]
Inference-time language model alignment via integrated value guidance.arXiv preprint arXiv:2409.17819, 2024
Zhixuan Liu, Zhanhui Zhou, Yuanfu Wang, Chao Yang, and Yu Qiao. Inference-time language model alignment via integrated value guidance.arXiv preprint arXiv:2409.17819, 2024
2024 arXiv
-
[32]
Controlled decoding from language models
Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, et al. Controlled decoding from language models. arXiv preprint arXiv:2310.17022, 2023. 15
-
[33]
Webgpt: Browser-assisted question-answering with human feedback.arXiv preprint arXiv:2112.09332, 2021
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback.arXiv preprint arXiv:2112.09332, 2021
2021 arXiv
-
[34]
Training language models to follow instructions with human feedback.Advances in Neural Information Processing Systems, 35:27730–27744, 2022
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in Neural Information Processing Systems, 35:27730...
2022
-
[35]
Treebon: Enhancing inference-time alignment with speculative tree-search and best-of-n sampling
Jiahao Qiu, Yifu Lu, Yifan Zeng, Jiacheng Guo, Jiayi Geng, Huazheng Wang, Kaixuan Huang, Yue Wu, and Mengdi Wang. Treebon: Enhancing inference-time alignment with speculative tree-search and best-of-n sampling. arXiv preprint arXiv:2410.16033, 2024
-
[36]
Fromr to q∗: Your language model is secretly a q-function.arXiv preprint arXiv:2404.12358, 2024
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. Fromr to q∗: Your language model is secretly a q-function.arXiv preprint arXiv:2404.12358, 2024
2024 arXiv
-
[37]
Direct preference optimization: Your language model is secretly a reward model.Advances in Neural Information Processing Systems, 36, 2024
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model.Advances in Neural Information Processing Systems, 36, 2024
2024
-
[38]
Trust region policy optimization.arXiv preprint arXiv:1502.05477, 2015
John Schulman. Trust region policy optimization.arXiv preprint arXiv:1502.05477, 2015
2015 arXiv
-
[39]
High-dimensional continuous control using generalized advantage estimation.arXiv preprint arXiv:1506.02438, 2015
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation.arXiv preprint arXiv:1506.02438, 2015
2015 arXiv
-
[40]
Proximal policy optimization algorithms
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017
2017 arXiv
-
[41]
Adaptive trust region policy optimization: Global convergence and faster rates for regularized mdps
Lior Shani, Yonathan Efroni, and Shie Mannor. Adaptive trust region policy optimization: Global convergence and faster rates for regularized mdps. InProceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5668–5675, 2020
2020
-
[42]
Deepseekmath: Pushing the limits of mathematical reasoning in open language models
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024
2024 arXiv
-
[43]
Scaling llm test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024
2024 arXiv
-
[44]
Hybrid rl: Using both offline and online data can make rl efficient.arXiv preprint arXiv:2210.06718, 2022
Yuda Song, Yifei Zhou, Ayush Sekhari, J Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid rl: Using both offline and online data can make rl efficient.arXiv preprint arXiv:2210.06718, 2022
2022 arXiv
-
[45]
Learning to summarize with human feedback.Advances in Neural Information Processing Systems, 33:3008–3021, 2020
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback.Advances in Neural Information Processing Systems, 33:3008–3021, 2020
2020
-
[46]
On the convergence rates of policy gradient methods.Journal of Machine Learning Research, 23(282):1–36, 2022
Lin Xiao. On the convergence rates of policy gradient methods.Journal of Machine Learning Research, 23(282):1–36, 2022
2022
-
[47]
Lillicrap, Kenji Kawaguchi, and Michael Shieh
Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P. Lillicrap, Kenji Kawaguchi, and Michael Shieh. Monte carlo tree search boosts reasoning via iterative preference learning.arXiv preprint arXiv:2405.00451, 2024
2024 arXiv
-
[48]
Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl- constraint
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl- constraint. In Forty-first International Conference on Machine Learning, 2024. 16
2024
-
[49]
Is dpo superior to ppo for llm alignment? a comprehensive study.arXiv preprint arXiv:2404.10719, 2024
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for llm alignment? a comprehensive study.arXiv preprint arXiv:2404.10719, 2024
2024 arXiv
-
[50]
Genarm: Reward guided generation with autoregressive reward model for test-time alignment
Yuancheng Xu, Udari Madhushani Sehwag, Alec Koppel, Sicheng Zhu, Bang An, Furong Huang, and Sumitra Ganesh. Genarm: Reward guided generation with autoregressive reward model for test-time alignment. arXiv preprint arXiv:2410.08193, 2024
-
[51]
Fudge: Controlled text generation with future discriminators.arXiv preprint arXiv:2104.05218, 2021
Kevin Yang and Dan Klein. Fudge: Controlled text generation with future discriminators.arXiv preprint arXiv:2104.05218, 2021
2021 arXiv
-
[52]
Llm alignment through successive policy re-weighting (spr)
Xinnan Zhang, Siliang Zeng, Jiaxiang Li, Kaixiang Lin, and Mingyi Hong. Llm alignment through successive policy re-weighting (spr). InNeurIPS 2024 Workshop on Fine-Tuning in Modern Machine Learning: Principles and Scalability, 2024
2024
-
[53]
Weak-to-strong search: Align large language models via searching over small language models.arXiv preprint arXiv:2405.19262, 2024
Zhanhui Zhou, Zhixuan Liu, Jie Liu, Zhichen Dong, Chao Yang, and Yu Qiao. Weak-to-strong search: Align large language models via searching over small language models.arXiv preprint arXiv:2405.19262, 2024
2024 arXiv
-
[54]
Starling-7b: Improving llm helpfulness & harmlessness with rlaif.2023, 2023
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b: Improving llm helpfulness & harmlessness with rlaif.2023, 2023. 17 A Related work A.1 Reinforcement learning with Human Feedback Reinforcement learning from human feedback (RLHF) [45, 34] is a wide...
2023
-
[55]
However, optimizing these algorithms for peak performance requires substantial effort and resources, which are often beyond the reach of the open-source community
and GRPO [42] have also demonstrated effectiveness in advanced foundation language models. However, optimizing these algorithms for peak performance requires substantial effort and resources, which are often beyond the reach of the open-source community. A.2 Training-time Alig...
-
[56]
or outcome reward function [25]. Based on the type of model used to guide the frozen LLM, we categorize related work into two main approaches: (1) outcome reward guidance, and (2) value function and PRM-based guidance. A.3.1 Outcome Reward Guidance Based on ORM, a widely used ...
-
[57]
Ctoken: The total number of token costs, representing the number of tokens needed by the algorithm
-
[58]
A"or "B"to indicate your choice. Your response should use the format: Comparison: <one-sentence comparison and explanation> Preferred: <
Cquery: The query cost, representing the number of calls to an external reward or value function oracle. Specifically, BoN sampling requiresN reward model evaluations to select the optimal trajectory with the highest reward score, while IRO requires querying allI value functio...
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.