REVIEW 5 major objections 5 minor 40 references
Mutual-Taught for Co-adapting Policy and Reward Models
T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Policy and reward model co-adapt without new human labels: pre/post-update responses form the reward model's next preference pairs, lifting an 8B policy to 54.1% win rate on AlpacaEval-2 and its reward model to 87.0 RewardBench.
desk verdict Solid empirical paper on co-updating policy and reward models; the theory is decorative and the circularity concern is real but not fatal. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the M-step pseudo-preference pair $(y_t, y_{t-1})$: for each prompt, the updated policy's response is treated as preferred over the previous policy's response, and those pairs are fit with the Bradley-Terry objective. This is what turns policy movement into a calibration signal for the reward model, letting the reward model track the evolving policy distribution instead of staying frozen on its original training data. Two stabilizers make the loop usable: the E-step selects the policy checkpoint with the highest measured win rate over the previous policy on a fixed validation set, and the M-step keeps only pseudo-pairs whose reward margin exceeds a variance-aware threshold. The paper's analysis of data types adds a third component: mixing the pseudo-pairs with the original reward-model-labeled preference data prevents the reward model from forgetting its initial distribution while still absorbing the new distributional signal.
What would settle it
Run Mutual-Taught with a deliberately impaired E-step—for instance, flip the preference labels inside the DPO update so the policy is trained to prefer the worse response—while keeping the M-step pseudo-label rule unchanged. If the reward model still improves on RewardBench, the M-step labels are not the causal driver; if the reward model degrades, the method's gains depend on the E-step actually making the policy better.
Extended reading notes
Core claim
Mutual-Taught's central claim is that the preference signal needed to keep a reward model aligned with a moving policy can be extracted from the policy's own change. After a direct preference optimization (DPO) update takes the policy from $\pi_{t-1}$ to $\pi_t$ under reward model $r_{t-1}$, the M-step samples $y_t \sim \pi_t(\cdot|x)$ and $y_{t-1} \sim \pi_{t-1}(\cdot|x)$ for the same prompt and declares $y_t$ preferred over $y_{t-1}$, then fits $r_t$ by maximizing the Bradley-Terry likelihood on those pairs. The paper argues this EM-style alternation, protected by checkpoint selection in the E-step and variance-aware margin filtering in the M-step, makes the sequence $(\pi_t, r_t)$ improve monotonically toward the latent optimal preference distribution. Empirically the claim is that two iterations of this loop are enough to lift the 8B policy Llama-3-8B-Instruct to 54.1 LC win rate on AlpacaEval-2 and 38.4 on Arena-Hard, while the 8B reward model FsfairX-Llama3-RM reaches 87.0 on RewardBench, at parity with GPT-4o-2024-08-06.
Load-bearing premise
The load-bearing premise is that each E-step policy update makes the new response genuinely better, so that labeling it preferred over the old response is a true preference; the paper's convergence argument in the appendix assumes this—along with a perfect estimate of the latent optimal responses—rather than proving it, and if the premise fails, the reward model is trained on false comparisons and the loop can amplify its own error.
Editorial extensions
If this is right
- Iterative preference-optimization methods that leave the reward model frozen (iterative DPO, SPPO, Meta-Rewarding) are leaving the documented gains on the table, since Mutual-Taught beats them on both AlpacaEval-2 and Arena-Hard while using less policy training data.
- The same dataset can support repeated rounds: when the previous round's policy and reward models are used only to generate next-round training data and training restarts from the base models, both models keep improving rather than overfitting.
- A reward model trained inside the loop transfers to other policies: one DPO pass on Mistral-7B-Instruct-v0.2 using the iterated reward model raises AlpacaEval-2 LC win rate from 42.0 with the base reward model to 46.8.
- The two-stage stabilization is load-bearing, not decorative: ablations show removing model selection or data filtering individually lowers both AlpacaEval-2 and RewardBench results, and the reward-model gains concentrate in dimensions where the policy is already strong, notably a 9.3-point RewardBench reasoning jump.
Reading between the lines
- If the policy-distribution signal is the real driver, the same M-step construction should work when the E-step uses other preference-optimization objectives, such as PPO, IPO, or SimPO; the paper only demonstrates DPO, so this is a testable extension rather than a reported result.
- The convergence theorem assumes the fitted policy estimates the latent optimal preference distribution without bias, an assumption the paper asserts rather than proves; in practice the variance-aware filter is likely doing substantial work to compensate for biased pseudo-labels, and its role deserves direct measurement.
- Because the loop needs no human labels, an obvious deployment is sustained adaptation on a fixed prompt set in specialized domains, with the reward model being recalibrated in the background as the policy improves.
- The method's dependence on a strong initial reward model is under-tested: the paper notes the base reward model already scores 84.7 on RewardBench, so the gains might shrink or reverse with a weaker starting reward model, and that boundary is worth probing deliberately.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Mutual-Taught, an iterative procedure that co-trains a policy model and a reward model without additional human labels. In the E-step, the policy is updated with DPO on responses sampled from the current policy and ranked by the current reward model, with a validation-based checkpoint selection. In the M-step, the reward model is trained on pseudo-preference pairs formed by responses generated before and after the policy update, labeled as the post-update response being preferred, with a margin-based filter. Experiments with Llama-3-8B-Instruct and FsfairX-Llama3-RM-v0.1 on UltraFeedback report an LC win rate of 54.1 on AlpacaEval-2 and 38.4 on Arena-Hard, and a RewardBench score of 87.0, matching GPT-4o-2024-08-06. Additional experiments examine data types, two-stage stabilization ablations, an extra training round, and generalization to Mistral-7B.
Significance. The empirical contribution is substantial if the results hold: a fully self-supervised loop that improves both policy and reward model, with large gains over strong iterative baselines and a generalization check to a different policy. The paper also provides useful ablations (data type, filtering, model selection) and releases code. However, the formal EM framing is not well-posed, the convergence theorem is conditional on an assumption equivalent to the desired property, and the self-labeling loop creates a co-adaptation risk that the current LLM-judge benchmarks do not eliminate. These issues do not necessarily invalidate the empirical findings, but they need to be addressed before the paper can be accepted.
major comments (5)
- [§4.2, Eq. (3)] The objective in Eq. (3) is not well-posed as stated: maximizing E_{x∼D,y∼π∗(·|x)}[r(y;x)] over r is unbounded unless r is constrained or regularized, and the E-step in Eq. (4) is a KL-regularized DPO update rather than the argmax of this objective, so the formal link between the stated objective and the algorithm is missing. Please replace Eq. (3) with a well-defined objective (e.g., a regularized expected-reward objective plus a reward-model likelihood term) and derive the E- and M-steps from it, or explicitly state that Eq. (3) is only an informal motivation.
- [Appendix D, Theorem 1] Theorem 1 assumes 'unbiased estimation in pseudo-labeling: E[π̂(y|x)] = π∗(y|x)' (Assumption 2), which is precisely the property the method is supposed to obtain; it is asserted rather than derived and is unlikely to hold for a finite-capacity policy trained against a biased reward model. The proof sketch also does not establish the two inequalities R(πt,rt) ≥ R(πt,rt−1) and R(πt,rt−1) ≥ R(πt−1,rt−1); the M-step inequality is asserted after the filtering step and the E-step inequality is asserted from model selection, but neither follows from the stated assumptions. As written, the convergence theorem does not apply to the actual algorithm; either provide a rigorous proof under verifiable conditions or remove the convergence claim and present the EM analogy as intuition.
- [§4.2, Eq. (5); §4.3, Eq. (8)] The M-step labels y_t ≻ y_{t−1} solely because π_t was optimized against r_{t−1}; this is not independent evidence of preference. The filtering rule in Eq. (8) then keeps pairs for which r_{t−1} is confident, so the updated reward model is trained to reinforce r_{t−1}'s existing judgments. This creates a real circularity risk: the policy can overfit to r_{t−1}, and the reward model can co-adapt to the policy rather than to human preferences. The reported benchmarks use GPT-4 judges (AlpacaEval-2, Arena-Hard) and RewardBench, which may share stylistic biases with the RM, so they do not rule out co-adaptation. Please add an evaluation against an independent human-preference ground truth (or an analysis showing the RM's errors decrease on held-out human judgments) and discuss what evidence would falsify the co-adaptation hypothesis.
- [Tables 1–3 and Figures 3–7] All benchmark results are single-run point estimates without variance or confidence intervals. Several key differences are small (e.g., RewardBench 85.8 vs 87.0 in Table 2; Table 4 average 67.76 vs 67.85; Figure 7 differences of ±0.1 points), so 'consistent improvement' is not statistically supported. Please report multiple seeds or otherwise quantify uncertainty (e.g., bootstrap over judge decisions or evaluation-set resampling) for at least the main tables.
- [Appendix H, Table 5] The early-stopping threshold τ is a free parameter whose value (60%) is chosen with knowledge of the first-iteration win rate (63.5%) and then tested on the same data in Table 5. This makes the early-stopping demonstration vulnerable to selection bias. Please specify how τ is set a priori or provide a sensitivity analysis over τ.
minor comments (5)
- [§4.1–§4.2] The term 'EM' is used loosely: no latent-variable posterior is ever computed, and Eq. (3)–(5) are not derived from a likelihood with latent variables. Please consistently say 'EM-inspired' or provide an actual EM derivation.
- [Figures 4, 6, and 7] These figures report only differences relative to a baseline without absolute values or error bars, which makes the magnitude of the effects difficult to assess; please include the underlying numbers or confidence intervals.
- [Table 5] The table is missing the final benchmark score of the early-stopped model relative to the round-3 model; please state explicitly which model is selected and report its AlpacaEval-2 LC win rate.
- [Table 4] The claim that Mutual-Taught is 'particularly beneficial' on HellaSwag and TruthfulQA is not supported by the average (67.85 vs 67.76) and is accompanied by drops on GSM8K and MMLU; please soften this claim or add analysis of the trade-off.
- [Abstract and throughout] There are formatting issues such as 'we proposeMutual-Taught' in the abstract (missing space) and inconsistent capitalization/hyphenation of 'Mutual-Taught' versus 'Mutual-Taught'; please proofread.
Circularity Check
Reward-model pseudo-labels and model selection are defined by the previous reward model, and the convergence theorem assumes the pseudo-labels are already unbiased.
-
fitted input called prediction
[Section 4.2, Eq. (5)]
"Since πt is optimized with respect to rt−1, we treat yt as the preferred response relative to yt−1. We then construct pseudo-preference pairs (yt, yt−1) and update rt−1 by maximizing the Bradley-Terry log-likelihood: rt = argmax r Ex∼DR [log Pr(yt ≻ yt−1|x)] (5)"
The 'preferred' label in Eq. (5) is not an independent annotation. It is defined by the fact that Eq. (4) optimized πt against rt−1, so yt is the response that rt−1 already ranks highly relative to yt−1. The M-step then fits rt to these self-generated labels, and Eq. (8) filters to pairs where rt−1 is confident. Consequently, the reward model's 'improvement' is, by construction, a regression toward its predecessor's preferences; any bias in rt−1 is propagated rather than corrected. The external benchmarks prevent this from being a pure tautology, but the internal training signal is not independent.
-
self definitional
[Section 4.3, Eqs. (6)-(7)]
"The win rate for each checkpoint is computed as: wkt = 1/|DMS| Σ_{x∈D_MS} I(ykt ≻ yt−1|x) (6) ... I(ykt≻yt−1|x) = (1 if rt−1(ykt;x)>rt−1(yt−1;x), 0 otherwise). Only the checkpoint that demonstrates maximum improvement over the previous policy is selected..."
Policy 'improvement' in the E-step is certified by the same reward model rt−1 that produced the DPO training signal in Eq. (4). A checkpoint is selected only if rt−1 scores its outputs higher than the previous policy's outputs. The pseudo-pairs for the M-step are then built from this selected policy and labeled as preferred. Thus the entire loop defines 'better' as 'better according to rt−1'; the reward model update cannot correct an error in rt−1's notion of quality, because that notion is the selection criterion.
1 more flagged steps
-
self definitional
[Appendix D.2, Theorem 1]
"Under the assumptions that: 1. Exact optimization in E-step and M-step. 2. Unbiased estimation in pseudo-labeling: E[π̂(y|x)] = π∗(y|x). The Mutual-Taught sequence satisfies: R(πt,rt)≥R(πt−1,rt−1) ∀t≥0..."
The theorem's second assumption states that the policy used to generate pseudo-labels is already unbiased relative to the true optimal distribution π∗. That unbiasedness is precisely the property the Mutual-Taught loop is supposed to produce; assuming it makes the monotonic-improvement conclusion true by assumption, not by the E-step/M-step dynamics. The proof sketch asserts 'This data filtering strategy ensures Cov(π̂)→Cov(π∗)' without deriving it, so the claimed convergence guarantee is conditional on the target result rather than established by the algorithm.
full rationale
The paper's headline empirical results are genuinely external: AlpacaEval-2, Arena-Hard, and RewardBench are independent benchmarks, and no load-bearing self-citation chain was found. However, the internal derivation chain is partially circular. The M-step (Eq. 5) constructs pseudo-labels by declaring the DPO-updated response preferred solely because it was optimized against rt−1; model selection (Eqs. 6–7) uses rt−1 as the judge of 'improvement'; and data filtering (Eq. 8) keeps pairs rt−1 already endorses. The new reward model is therefore fit to reproduce its own predecessor's preferences, so the 'reward model improvement' signal is not independent of the input reward model. The convergence theorem (App. D.2) makes this circularity explicit: it assumes E[π̂]=π∗, i.e., that the pseudo-labeling distribution is already unbiased, which is exactly the property the iterative procedure is supposed to establish. Because external benchmarks provide some independent grounding, the paper is not wholly circular, but the claimed theoretical guarantee and the internal self-training signal reduce by construction.
Assumptions & free parameters
free parameters (2)
- early-stop threshold tau =
0.60 (60%)
- margin threshold epsilon_t =
sqrt(V_x[r_{t-1}(y_{t-1};x)]) (std of RM scores)
assumptions (5)
- standard math Bradley-Terry model of preferences (Eq. 1)
- domain assumption There exists a latent optimal response distribution pi*(y|x) representing true human preferences
- ad hoc to paper The DPO update in the E-step (Eq. 4), using r_{t-1} to rank on-policy samples, genuinely moves the policy closer to pi*
- ad hoc to paper Unbiased pseudo-labeling: E[pi-hat(y|x)] = pi*(y|x) (Appendix D, Theorem 1 assumption 2)
- standard math Alternating maximization of the non-convex objective (Eq. 3) converges under regularity conditions
Cite this review
Pith. "Pith review of Mutual-Taught for Co-adapting Policy and Reward Models." pith.science (2026). https://pith.science/paper/BXHZNXDF
@misc{pith2026250606292,
author = {Pith},
title = {Pith review of: Mutual-Taught for Co-adapting Policy and Reward Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/BXHZNXDF}},
note = {Machine review of arXiv:2506.06292}
}
read the original abstract
During the preference optimization of large language models (LLMs), distribution shifts may arise between newly generated model samples and the data used to train the reward model (RM). This shift reduces the efficacy of the RM, which in turn negatively impacts the performance of the policy model (PM). To address this challenge, we propose Mutual-Taught, a self-training method that iteratively improves both the PM and RM without requiring additional human annotation. Our approach mirrors the expectation-maximization (EM) algorithm. In the E-step, the PM is updated using feedback from the current RM, guiding the PM toward a better approximation of the latent optimal preference distribution. In the M-step, we update the RM by constructing training data from the outputs of the PM before and after the E-step update. This process ensures that the RM adapts to the evolving policy distribution. Experimental results demonstrate that this iterative approach leads to consistent improvements in both models. Specifically, our 8B policy model, LLaMA-3-8B-Instruct-MT, achieves a length-controlled win rate of 54.1\% on AlpacaEval-2, while our 8B reward model, FsfairX-LLaMA3-RM-MT, performs on par with GPT-4o-2024-08-06 on RewardBench.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics
work page 2024
-
[2]
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. 2023. Open LLM leaderboard
work page 2023
-
[3]
Ralph Allan Bradley and Milton E. Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39:324
work page 1952
-
[4]
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. 2024. Self-play fine-tuning converts weak language models to strong language models. In International Conference on Machine Learning
work page 2024
-
[5]
Pengyu Cheng, Yifan Yang, Jian Li, Yong Dai, Tianhao Hu, Peixin Cao, Nan Du, and Xiaolong Li. 2024. Adversarial preference optimization: Enhancing your alignment via rm-llm game. In Findings of the Association for Computational Linguistics ACL 2024, pages 3705--3716
work page 2024
-
[6]
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168
arXiv 2021
-
[7]
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2024. UltraFeedback : Boosting language models with high-quality feedback. In International Conference on Machine Learning
work page 2024
-
[8]
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. 2024. RLHF workflow: From reward modeling to online RLHF . Transactions on Machine Learning Research
work page 2024
Show all 40 references
-
[9]
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783
2024 arXiv
-
[10]
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. KTO : Model alignment as prospect theoretic optimization. In International Conference on Machine Learning
2024
-
[11]
Leo Gao, John Schulman, and Jacob Hilton. 2023. Scaling laws for reward model overoptimization. In International Conference on Machine Learning
2023
-
[12]
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations
2021
-
[13]
Tony Huang, Jack Chu, and Fangyun Wei. 2022. Unsupervised prompt learning for vision-language models. arXiv preprint arXiv:2204.03649
2022 arXiv
-
[14]
AQ Jiang, A Sablayrolles, A Mensch, C Bamford, DS Chaplot, D de las Casas, F Bressand, G Lengyel, G Lample, L Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825
2023 arXiv
-
[15]
o pf, Yannic Kilcher, Dimitri von R \
Andreas K \"o pf, Yannic Kilcher, Dimitri von R \"u tte Sotiris Anagnostidis Zhi Rui Tam, et al. 2023. Openassistant conversations - democratizing large language model alignment. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track
2023
-
[16]
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. 2024. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787
2024 arXiv
-
[17]
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. 2024. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939
2024 arXiv
-
[18]
Hashimoto
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval
2023
-
[19]
Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA : Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214--3252
2022
-
[20]
Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Sim PO : Simple preference optimization with a reference-free reward. In Advances in Neural Information Processing Systems
2024
-
[21]
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems
2022
-
[22]
Alizée Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. 2024. West-of-n: Synthetic preferences for self-improving reward models. arXiv preprint arXiv:2401.12086
2024 arXiv
-
[23]
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems
2023
-
[24]
Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. 2024. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.03715
2024 arXiv
-
[25]
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347
2017 arXiv
-
[26]
Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, et al. 2024. Beyond human data: Scaling self-training for problem-solving with language models. Transactions on Machine Learning Research
2024
-
[27]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288
2023 arXiv
-
[28]
Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. 2024. Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge. arXiv preprint arXiv:2407.19594
2024 arXiv
-
[29]
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. 2025. Self-play preference optimization for language model alignment. In The Thirteenth International Conference on Learning Representations
2025
-
[30]
Tengyang Xie, Dylan J Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Hassan Awadallah, and Alexander Rakhlin. 2025. Exploratory preference optimization: Provably sample-efficient exploration in RLHF with general function approximation. In The Thirteenth International Confer...
2025
-
[31]
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. 2024. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In International Conference on Machine Learning
2024
-
[32]
Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. 2023. Some things are more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312.16682
2023 arXiv
-
[33]
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. 2024. Self-rewarding language models. In International Conference on Machine Learning
2024
-
[34]
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. H ella S wag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791--4800
2019
-
[35]
Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. 2024 a . Re ST - MCTS *: LLM self-training via process reward guided tree search. In Advances in Neural Information Processing Systems
2024
-
[36]
Shenao Zhang, Donghan Yu, Hiteshi Sharma, Ziyi Yang, Shuohang Wang, Hany Hassan, and Zhaoran Wang. 2024 b . Self-exploring language models: Active preference elicitation for online alignment. arXiv preprint arXiv:2405.19332
2024 arXiv
-
[37]
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems
2023
-
[38]
Rui Zheng, Wei Shen, Yuan Hua, Wenbin Lai, Shihan Dou, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Haoran Huang, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. Improving generalization of alignment with human preferences through group invariant learning. In The Twelfth International Conf...
2024
-
[39]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[40]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.