{"id":"65713708-87d5-4251-b875-78b8ba9b3f94","arxiv_id":"2506.06292","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A method that alternately fine-tunes a policy model with DPO and retrains its reward model on pseudo-preferences drawn from the policy's pre- and post-update outputs, reporting gains on AlpacaEval-2 and RewardBench.","lead":"This paper presents a self-training loop that alternates between improving a language model and improving its reward model, using the model's own before-and-after outputs as training data. It reports large gains on chat benchmarks and a reward model that scores comparably to GPT-4o, without new human preference labels.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The M-step's pseudo-labels (Eq. 5) assume y_t is better than y_{t-1} because the E-step improved the policy under r_{t-1}, but the convergence proof (App.","rationale":"The reader's weakest-assumption correctly identifies the pseudo-label bias as the key risk. I agree with that assessment: the M-step's label construction (Sec. 4.2, Eq. 5) is the linchpin of the method because it replaces human annotations with a comparison derived from the current reward model's own influence on the policy. The convergence theorem in App. D.2 does not save the argument: it assumes unbiased pseudo-labels, which is exactly the property that the self-training loop does not ensure. In fact, the theorem's assumption E[π̂(y|x)] = π∗(y|x) is stronger than what the method's empirical checkpoints provide, and the proof sketch contains unsupported steps (e.g., the data-filtering step is claimed to ensure Cov(π̂)→Cov(π∗) without proof). My stress-test does not refute the empirical results, which are plausible and well-documented with code. But the central claim—that Mutual-Taught improves both models without human annotation—would be on much stronger footing if the pseudo-labels' reliability were demonstrated. Therefore I recommend keeping the reader's CONDITIONAL verdict and adding the concrete validation test described above. If that test shows pseudo-labels are near-random, the verdict should move toward REJECT because the mechanism would be unsupported.","tokens_in":16026,"tokens_out":6324,"duration_ms":63712,"concrete_test":"Use the released code/checkpoints to sample 500 pseudo-pairs (y_t, y_{t-1}) from a mid-training M-step (e.g., around iteration 2). Have independent human annotators (or a panel of GPT-4o judges with a different prompt template not used in training) label which response is better. Compute the accuracy of the pseudo-labels against this oracle. If accuracy is not significantly above 50% (or if the pairs where Δr > ε_t are no more accurate than random), the M-step is training on uninformative/biased labels, and the empirical gains likely stem from length/style artifacts or judge bias rather than genuine preference learning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism relies on self-generated preferences: Eq. (5) in Sec. 4.2 optimizes the reward model against pairs (y_t, y_{t-1}) labeled by the assertion that the DPO-updated policy π_t produces better responses than π_{t-1} under the same reward model r_{t-1}. This is a circular pseudo-label: r_{t-1} guided the DPO update (Eq. 4), so y_t is chosen to satisfy r_{t-1}; using that same r_{t-1} to certify y_t ≻ y_{t-1} is not independent verification. The convergence theorem (App. D.2) explicitly assumes 'unbiased estimation in pseudo-labeling: E[π̂(y|x)] = π∗(y|x)', which would make the pseudo-labels unbiased. But this assumption is asserted, not derived, and is unlikely to hold because π̂ is a limited-capacity model trained against a biased reward model. If the assumption fails, the M-step can reinforce the RM's initial biases instead of correcting them; the filtering rule (Sec. 4.3, Eq. 8) keeps pairs where r_{t-1} is confident, concentrating on the RM's existing beliefs. The result could be a co-adapted policy-RM pair that scores well on each other but not on true human preferences. The reported benchmark improvements (AlpacaEval-2, RewardBench) do not rule this out: both use LLM judges that may share the RM's stylistic biases. This is the load-bearing concern because it targets the mechanism by which the method claims to improve without human annotation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Mutual-Taught, an iterative procedure that co-trains a policy model and a reward model without additional human labels. In the E-step, the policy is updated with DPO on responses sampled from the current policy and ranked by the current reward model, with a validation-based checkpoint selection. In the M-step, the reward model is trained on pseudo-preference pairs formed by responses generated before and after the policy update, labeled as the post-update response being preferred, with a margin-based filter. Experiments with Llama-3-8B-Instruct and FsfairX-Llama3-RM-v0.1 on UltraFeedback report an LC win rate of 54.1 on AlpacaEval-2 and 38.4 on Arena-Hard, and a RewardBench score of 87.0, matching GPT-4o-2024-08-06. Additional experiments examine data types, two-stage stabilization ablations, an extra training round, and generalization to Mistral-7B.","tokens_in":16353,"tokens_out":7572,"duration_ms":72488,"significance":"The empirical contribution is substantial if the results hold: a fully self-supervised loop that improves both policy and reward model, with large gains over strong iterative baselines and a generalization check to a different policy. The paper also provides useful ablations (data type, filtering, model selection) and releases code. However, the formal EM framing is not well-posed, the convergence theorem is conditional on an assumption equivalent to the desired property, and the self-labeling loop creates a co-adaptation risk that the current LLM-judge benchmarks do not eliminate. These issues do not necessarily invalidate the empirical findings, but they need to be addressed before the paper can be accepted.","major_comments":[{"comment":"The objective in Eq. (3) is not well-posed as stated: maximizing E_{x∼D,y∼π∗(·|x)}[r(y;x)] over r is unbounded unless r is constrained or regularized, and the E-step in Eq. (4) is a KL-regularized DPO update rather than the argmax of this objective, so the formal link between the stated objective and the algorithm is missing. Please replace Eq. (3) with a well-defined objective (e.g., a regularized expected-reward objective plus a reward-model likelihood term) and derive the E- and M-steps from it, or explicitly state that Eq. (3) is only an informal motivation.","section":"§4.2, Eq. (3)"},{"comment":"Theorem 1 assumes 'unbiased estimation in pseudo-labeling: E[π̂(y|x)] = π∗(y|x)' (Assumption 2), which is precisely the property the method is supposed to obtain; it is asserted rather than derived and is unlikely to hold for a finite-capacity policy trained against a biased reward model. The proof sketch also does not establish the two inequalities R(πt,rt) ≥ R(πt,rt−1) and R(πt,rt−1) ≥ R(πt−1,rt−1); the M-step inequality is asserted after the filtering step and the E-step inequality is asserted from model selection, but neither follows from the stated assumptions. As written, the convergence theorem does not apply to the actual algorithm; either provide a rigorous proof under verifiable conditions or remove the convergence claim and present the EM analogy as intuition.","section":"Appendix D, Theorem 1"},{"comment":"The M-step labels y_t ≻ y_{t−1} solely because π_t was optimized against r_{t−1}; this is not independent evidence of preference. The filtering rule in Eq. (8) then keeps pairs for which r_{t−1} is confident, so the updated reward model is trained to reinforce r_{t−1}'s existing judgments. This creates a real circularity risk: the policy can overfit to r_{t−1}, and the reward model can co-adapt to the policy rather than to human preferences. The reported benchmarks use GPT-4 judges (AlpacaEval-2, Arena-Hard) and RewardBench, which may share stylistic biases with the RM, so they do not rule out co-adaptation. Please add an evaluation against an independent human-preference ground truth (or an analysis showing the RM's errors decrease on held-out human judgments) and discuss what evidence would falsify the co-adaptation hypothesis.","section":"§4.2, Eq. (5); §4.3, Eq. (8)"},{"comment":"All benchmark results are single-run point estimates without variance or confidence intervals. Several key differences are small (e.g., RewardBench 85.8 vs 87.0 in Table 2; Table 4 average 67.76 vs 67.85; Figure 7 differences of ±0.1 points), so 'consistent improvement' is not statistically supported. Please report multiple seeds or otherwise quantify uncertainty (e.g., bootstrap over judge decisions or evaluation-set resampling) for at least the main tables.","section":"Tables 1–3 and Figures 3–7"},{"comment":"The early-stopping threshold τ is a free parameter whose value (60%) is chosen with knowledge of the first-iteration win rate (63.5%) and then tested on the same data in Table 5. This makes the early-stopping demonstration vulnerable to selection bias. Please specify how τ is set a priori or provide a sensitivity analysis over τ.","section":"Appendix H, Table 5"}],"minor_comments":[{"comment":"The term 'EM' is used loosely: no latent-variable posterior is ever computed, and Eq. (3)–(5) are not derived from a likelihood with latent variables. Please consistently say 'EM-inspired' or provide an actual EM derivation.","section":"§4.1–§4.2"},{"comment":"These figures report only differences relative to a baseline without absolute values or error bars, which makes the magnitude of the effects difficult to assess; please include the underlying numbers or confidence intervals.","section":"Figures 4, 6, and 7"},{"comment":"The table is missing the final benchmark score of the early-stopped model relative to the round-3 model; please state explicitly which model is selected and report its AlpacaEval-2 LC win rate.","section":"Table 5"},{"comment":"The claim that Mutual-Taught is 'particularly beneficial' on HellaSwag and TruthfulQA is not supported by the average (67.85 vs 67.76) and is accompanied by drops on GSM8K and MMLU; please soften this claim or add analysis of the trade-off.","section":"Table 4"},{"comment":"There are formatting issues such as 'we proposeMutual-Taught' in the abstract (missing space) and inconsistent capitalization/hyphenation of 'Mutual-Taught' versus 'Mutual-Taught'; please proofread.","section":"Abstract and throughout"}],"recommendation":"major_revision","confidential_remarks":"Dear Editor, the paper's central empirical finding is plausible and potentially interesting, but the theoretical section and the pseudo-labeling loop need substantial work. The convergence theorem is essentially conditional on the conclusion, and the M-step labels are generated by the same reward model used for filtering, so the paper currently does not rule out co-adaptation to RM biases. I would encourage a major revision focused on independent validation (held-out human preference data, multiple seeds, honest treatment of the EM analogy) rather than rejection, because the reported gains are large and the ablations are informative."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version. Mutual-Taught is an iterative alignment method where you alternate DPO policy updates with reward model retraining on pseudo-comparisons between the policy's pre- and post-update outputs. The paper's headline numbers are real: the 8B policy reaches 54.1 LC win rate on AlpacaEval-2 and the 8B RM hits 87.0 on RewardBench, matching GPT-4o. The RM improvement also transfers to a different policy (Mistral-7B), which is the kind of out-of-distribution check that makes the result credible.\n\nWhat is actually new is the specific construction: pseudo-pairs formed by comparing the policy before and after a DPO step, filtered by a variance-aware margin, combined with checkpoint selection by win rate against the previous policy. The prior work (ReSTEM, West-of-N, Meta-Rewarding) does not do this exact combination, and the ablations are reasonably thorough: the paper shows what happens when you use self-training data alone vs. policy-comparison data alone, and it tests different filtering rules.\n\nNow the soft spots. The theoretical framing is the weakest part. Equation 3 is not a well-posed optimization: if the expectation is over the true optimal distribution π*, as written, then varying π does not change the objective; the EM analogy is loose. Theorem 1 in Appendix D assumes 'unbiased estimation in pseudo-labeling' — that the fitted policy is unbiased for π* — and that assumption is asserted, not derived. The M-step improvement R(π_t, r_t) ≥ R(π_t, r_{t-1}) is also not actually established. So the convergence claim should be heavily qualified or removed.\n\nThe circularity concern from the stress test is real but not, in my reading, disqualifying. Yes, y_t is chosen to please r_{t-1}, and then r_{t-1} scores the pair; the loop could reinforce reward model biases. But the filtering keeps pairs where the post-update response scores higher, and the RewardBench OOD improvement plus the Mistral transfer suggest the RM is learning something beyond its own priors. Still, the paper should report confidence intervals — every benchmark number is a point estimate, and differences of a few points on RewardBench could be noise. The τ=60% threshold is also selected based on observed win rates, which is a bit post hoc.\n\nBottom line: this is a useful empirical paper, and the empirical claims are plausible. It deserves a serious referee. I'd recommend sending it to review, and asking for revision on the theory (or removing the theorem) and for variance estimates.","headline":"Solid empirical paper on co-updating policy and reward models; the theory is decorative and the circularity concern is real but not fatal.","tokens_in":16912,"tokens_out":3154,"would_cite":true,"duration_ms":31368,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Policy and reward model co-adapt without new human labels: pre/post-update responses form the reward model's next preference pairs, lifting an 8B policy to 54.1% win rate on AlpacaEval-2 and its reward model to 87.0 RewardBench.","keywords":["Mutual-Taught","preference optimization","reward model","policy model","distribution shift","self-training","expectation-maximization","reward hacking"],"falsifier":"Run Mutual-Taught with a deliberately impaired E-step—for instance, flip the preference labels inside the DPO update so the policy is trained to prefer the worse response—while keeping the M-step pseudo-label rule unchanged. If the reward model still improves on RewardBench, the M-step labels are not the causal driver; if the reward model degrades, the method's gains depend on the E-step actually making the policy better.","tokens_in":15767,"feed_emoji":"🤖","tokens_out":16086,"duration_ms":129048,"temperature":0.7,"pith_summary":"The paper tries to establish that the reward model's drift away from a shifting policy distribution can be fixed automatically, without collecting new human preference labels. Its method, Mutual-Taught, alternates two updates: the policy is fine-tuned against the current reward model (E-step), and then the reward model is retrained on pseudo-preference pairs formed by responses sampled from the policy before and after that update, with the post-update response treated as preferred (M-step). This makes the policy's own movement supply the signal that recalibrates the reward model, so the two models move toward the same notion of what a good response looks like. On two UltraFeedback-driven iterations, the policy climbs from 23.1 to 54.1 length-controlled win rate on AlpacaEval-2 and the reward model from 84.7 to 87.0 on RewardBench, matching GPT-4o-2024-08-06. A sympathetic reader should care because the result suggests iterative preference optimization can keep working without a growing budget of human annotations or external judge calls.","feed_headline":"Self-training loop lifts 8B model to 54.1% win rate","feed_subtitle":"The same loop also lifts its reward model to GPT-4o-level RewardBench scores, with no new human labels.","key_machinery":"The load-bearing machinery is the M-step pseudo-preference pair $(y_t, y_{t-1})$: for each prompt, the updated policy's response is treated as preferred over the previous policy's response, and those pairs are fit with the Bradley-Terry objective. This is what turns policy movement into a calibration signal for the reward model, letting the reward model track the evolving policy distribution instead of staying frozen on its original training data. Two stabilizers make the loop usable: the E-step selects the policy checkpoint with the highest measured win rate over the previous policy on a fixed validation set, and the M-step keeps only pseudo-pairs whose reward margin exceeds a variance-aware threshold. The paper's analysis of data types adds a third component: mixing the pseudo-pairs with the original reward-model-labeled preference data prevents the reward model from forgetting its initial distribution while still absorbing the new distributional signal.","core_discovery":"Mutual-Taught's central claim is that the preference signal needed to keep a reward model aligned with a moving policy can be extracted from the policy's own change. After a direct preference optimization (DPO) update takes the policy from $\\pi_{t-1}$ to $\\pi_t$ under reward model $r_{t-1}$, the M-step samples $y_t \\sim \\pi_t(\\cdot|x)$ and $y_{t-1} \\sim \\pi_{t-1}(\\cdot|x)$ for the same prompt and declares $y_t$ preferred over $y_{t-1}$, then fits $r_t$ by maximizing the Bradley-Terry likelihood on those pairs. The paper argues this EM-style alternation, protected by checkpoint selection in the E-step and variance-aware margin filtering in the M-step, makes the sequence $(\\pi_t, r_t)$ improve monotonically toward the latent optimal preference distribution. Empirically the claim is that two iterations of this loop are enough to lift the 8B policy Llama-3-8B-Instruct to 54.1 LC win rate on AlpacaEval-2 and 38.4 on Arena-Hard, while the 8B reward model FsfairX-Llama3-RM reaches 87.0 on RewardBench, at parity with GPT-4o-2024-08-06.","pith_inferences":["If the policy-distribution signal is the real driver, the same M-step construction should work when the E-step uses other preference-optimization objectives, such as PPO, IPO, or SimPO; the paper only demonstrates DPO, so this is a testable extension rather than a reported result.","The convergence theorem assumes the fitted policy estimates the latent optimal preference distribution without bias, an assumption the paper asserts rather than proves; in practice the variance-aware filter is likely doing substantial work to compensate for biased pseudo-labels, and its role deserves direct measurement.","Because the loop needs no human labels, an obvious deployment is sustained adaptation on a fixed prompt set in specialized domains, with the reward model being recalibrated in the background as the policy improves.","The method's dependence on a strong initial reward model is under-tested: the paper notes the base reward model already scores 84.7 on RewardBench, so the gains might shrink or reverse with a weaker starting reward model, and that boundary is worth probing deliberately."],"forward_implications":["Iterative preference-optimization methods that leave the reward model frozen (iterative DPO, SPPO, Meta-Rewarding) are leaving the documented gains on the table, since Mutual-Taught beats them on both AlpacaEval-2 and Arena-Hard while using less policy training data.","The same dataset can support repeated rounds: when the previous round's policy and reward models are used only to generate next-round training data and training restarts from the base models, both models keep improving rather than overfitting.","A reward model trained inside the loop transfers to other policies: one DPO pass on Mistral-7B-Instruct-v0.2 using the iterated reward model raises AlpacaEval-2 LC win rate from 42.0 with the base reward model to 46.8.","The two-stage stabilization is load-bearing, not decorative: ablations show removing model selection or data filtering individually lowers both AlpacaEval-2 and RewardBench results, and the reward-model gains concentrate in dimensions where the policy is already strong, notably a 9.3-point RewardBench reasoning jump."],"supporting_citations":[{"why":"Supplies the DPO objective used as the E-step policy update.","marker":"Rafailov et al., 2023"},{"why":"Supplies the pairwise preference model used to fit the reward model in the M-step.","marker":"Bradley and Terry, 1952"},{"why":"Provides the base reward model FsfairX-Llama3-RM and the iterative preference-learning setting this work extends.","marker":"Xiong et al., 2024"},{"why":"Provides the UltraFeedback prompt set and its preference labels, used for both policy and reward training.","marker":"Cui et al., 2024"},{"why":"Provides the Llama-3-8B-Instruct base policy model.","marker":"Dubey et al., 2024"},{"why":"Contributes the variance-aware threshold and the self-training idea that the M-step filtering adapts.","marker":"Pace et al., 2024"},{"why":"Defines AlpacaEval-2 and its length-controlled win-rate metric used for policy evaluation.","marker":"Li et al., 2023"},{"why":"Defines RewardBench, the benchmark used to compare the iterated reward model with GPT-4o-2024-08-06.","marker":"Lambert et al., 2024"},{"why":"Meta-Rewarding is the main LLM-as-a-judge baseline against which Mutual-Taught's no-external-judge design is compared.","marker":"Wu et al., 2024"}],"fun_headline_variants":["Mutual-taught loop lifts policy and reward without labels","One self-training loop updates both policy and reward","8B policy reaches 54.1% win rate via mutual-taught","Co-adaptive loop: policy and reward improve together","Self-training co-adapts policy and reward to GPT-4o level"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that each E-step policy update makes the new response genuinely better, so that labeling it preferred over the old response is a true preference; the paper's convergence argument in the appendix assumes this—along with a perfect estimate of the latent optimal responses—rather than proving it, and if the premise fails, the reward model is trained on false comparisons and the loop can amplify its own error.","fun_headline_variants_meta":{"raw":{"variants":["Mutual-taught loop lifts policy and reward without labels","One self-training loop updates both policy and reward","8B policy reaches 54.1% win rate via mutual-taught","Co-adaptive loop: policy and reward improve together","Self-training co-adapts policy and reward to GPT-4o level"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000341,"raw_usage":{"total_tokens":1943,"prompt_tokens":1076,"completion_tokens":867,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":692,"completion_tokens_details":{"reasoning_tokens":781}},"tokens_in":692,"tokens_out":867,"duration_ms":8179,"temperature":1.0,"reasoning_tokens":781,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:46:42.704512+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Mutual-Taught with a deliberately impaired E-step—for instance, flip the preference labels inside the DPO update so the policy is trained to prefer the worse response—while keeping the M-step pseudo-label rule unchanged. If the reward model still improves on RewardBench, the M-step labels are not the causal driver; if the reward model degrades, the method's gains depend on the E-step actually making the policy better.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the DPO objective used as the E-step policy update."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the pairwise preference model used to fit the reward model in the M-step."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the UltraFeedback prompt set and its preference labels, used for both policy and reward training."}],"review_version":1}