{"id":"5cecffb1-e24c-4f05-be99-e051701205f0","arxiv_id":"2505.02363","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Mixing on-policy and off-policy preference data in equal proportions (SIMPLEMIX) improves DPO alignment over either source alone, with on-policy data best for reasoning tasks and off-policy data best for open-ended tasks.","lead":"This paper finds that AI models learn best from their own answers on math and coding, but from other models' answers on creative writing and recommendations. Simply mixing the two types of training data, half and half, improves the final model's quality on standard instruction-following benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central complementarity claim depends on a nonstandard definition of on-policy data as frozen-SFT samples; the paper never tests true current-policy sampling, so the headline finding may not extend to the on-policy setting it claims to address.","rationale":"The paper's strongest claim—that on-policy and off-policy data are complementary and a simple 50/50 mix is a robust default—stands or falls on what 'on-policy' means. Section 2 defines on-policy as sampling from πSFT or πθ, a departure from RL convention acknowledged in footnote 1. All experiments (Sections 3–5, Table 1) use only frozen πSFT samples, generated once before training; the policy is never used as the sampling distribution during training. Thus the data mix is really SFT-data plus off-policy dataset, both static. The observed complementarity and superiority over HyPO/DPO-Mix-P may reflect the quality or diversity of SFT-generated responses rather than any property of current-policy data. This is not an internal inconsistency, but it limits the external validity of the central claim as communicated in the abstract and introduction. The reader's weakest assumption captures exactly this issue, so I agree. Additional support for a conditional verdict comes from the paper's own Section 7, which concedes incomplete hyperparameter search and judge-benchmark vulnerabilities, and from the fact that the abstract's 6.03% and 3.05% figures reproduce only for the Tulu model in Table 2, contrary to the 'across diverse tasks and benchmarks' phrasing. The proposed concrete test—replacing the frozen-SFT sampling with current-policy sampling while holding everything else fixed—would settle whether the complementarity pattern persists. Until that is run, the method is a useful, clearly-described empirical recipe, but the 'on-policy' interpretation of the results is not established. Therefore the reader's CONDITIONAL verdict is appropriate and unchanged.","tokens_in":27009,"tokens_out":7788,"duration_ms":82784,"concrete_test":"Run SIMPLEMIX with the on-policy component sampled from the current policy πθ at each training step (or from a πθ/πSFT interpolation as in DPO-Mix-P) instead of frozen πSFT, holding total data budget, epochs, and all hyperparameters fixed, on Llama-3.1-8B-Instruct and Tulu-3-8B-SFT with UltraFeedback. Re-measure the Alpaca Eval 2.0 category win rates (math, coding, creative writing, recommendation). If the math/coding advantage of the on-policy component disappears or reverses, the complementarity result depends on the frozen-SFT definition and the abstract's 'on-policy' claims need substantial revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2 and footnote 1 define on-policy data as responses sampled from either πSFT or πθ, acknowledging this departs from the RL convention where on-policy means samples from the current policy πθ. Throughout Sections 3–5, the 'on-policy' component is always sampled from the frozen SFT model before training and never from the evolving policy, and DPO trains for one epoch on these fixed pairs. As a result, SIMPLEMIX actually mixes two static data sources—SFT-generated pairs and an external dataset—both of which are off-policy with respect to the final model. The observed complementarity in Section 3.2 (math/coding gains from the SFT component) and the margin over HyPO and DPO-Mix-P in Section 4.2, both of which use current-policy sampling, may therefore depend on this nonstandard definition. If true current-policy sampling behaves differently, the central claim that on-policy and off-policy data are complementary, and the implied recommendation that a 50/50 SFT/off-policy mixture is a robust default, would not generalize beyond the frozen-SFT case. The paper's own footnote flags the definitional departure but does not test whether the findings survive under the standard notion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the interplay between on-policy and off-policy preference data in language model alignment. It defines on-policy data as responses sampled from either the SFT model or the current policy, and off-policy data as external preference datasets. Through DPO experiments on Llama-3.1-8B-Instruct and Llama-3.1-Tulu-3-8B-SFT with UltraFeedback and HelpSteer2, the paper reports that on-policy data is more effective for math and coding while off-policy data is more effective for open-ended tasks such as creative writing and recommendations. It then proposes SIMPLEMIX, which samples preference pairs equally from the SFT model and the off-policy dataset, and reports that this simple mixture outperforms single-source DPO and more complex hybrid methods (HyPO, DPO-Mix-P) on Alpaca Eval 2.0 and an eight-benchmark average. The paper also investigates diversity, mixing ratios, and off-policy data filtering.","tokens_in":27246,"tokens_out":5992,"duration_ms":66745,"significance":"If the claims hold, SIMPLEMIX provides a simple and effective default for preference optimization, and the task-dependent complementarity result is a practically useful guideline for data curation. The paper's strengths include controlled comparisons that vary only the data source while fixing the algorithm, the use of two SFT base models and two preference datasets, evaluation with both LM-as-a-judge and reference-based benchmarks, and a length-control analysis that rules out a trivial explanation of the reasoning-task gains. The central empirical finding—that a 50/50 mixture of SFT-generated and externally sourced preference pairs improves alignment—is consistent across the four experimental settings reported in Table 2. However, the headline 'on-policy' terminology rests on a nonstandard definition, and the task-complementarity analysis is based on very small per-category prompt subsets, so the scope of the conclusions is narrower than the abstract suggests.","major_comments":[{"comment":"The paper defines on-policy data as responses sampled from either pi_SFT or pi_theta and acknowledges in footnote 1 that this departs from the standard RL convention where on-policy means samples from the current policy pi_theta. Throughout the experiments, the 'on-policy' component is always sampled from the frozen SFT model before training, so both data sources are actually off-policy with respect to the final model. Because the central complementarity claim and the comparison against HyPO and DPO-Mix-P (both of which use current-policy sampling) are framed in terms of 'on-policy' data, the findings may not transfer to the standard online setting. The paper should either rename this data source (e.g., 'SFT-generated data') throughout the abstract, main text, and conclusions, or add a condition that samples from the evolving policy to show that the results hold under the standard notion.","section":"Section 2 (Definition of on-policy) and Sections 3–5"},{"comment":"The per-category win rates underlying the complementarity claim are computed on very small prompt subsets: Appendix B shows that math and coding constitute only 4.35% and 7.08% of the 805 Alpaca Eval 2.0 prompts, corresponding to roughly 35 and 57 prompts, respectively. The paper reports differences of +5.72, +7.02, -2.85, and -3.29 percentage points for these categories, but Figure 2 shows no confidence intervals or significance tests. These differences may be within sampling noise. The authors should report bootstrap confidence intervals or significance tests for each per-category comparison, as they already do in Figure 3 for the length-stratified analysis.","section":"Section 3.2, Figure 2, and Appendix B"},{"comment":"The 0.5 mixing ratio and p=0.4 filtering threshold are selected by maximizing the length-controlled win rate on Alpaca Eval 2.0, which is the same benchmark used to report the headline improvement in Table 2. This selection on the evaluation set can inflate the reported gains and makes the '6.03% average improvement' claim difficult to interpret. The authors should disclose this selection procedure explicitly in the main text and provide evidence that the 0.5 ratio is not an artifact of tuning to this single benchmark, for example by evaluating on a held-out set of preference prompts or by reporting results for a fixed, pre-specified ratio.","section":"Sections 5.2–5.3, Figures 4–6, and Table 2"}],"minor_comments":[{"comment":"The text reads '6.03 over' where a percent sign appears to be missing; it should be '6.03%.' The same issue occurs in the introduction.","section":"Abstract"},{"comment":"The paper treats meta-llama/Llama-3.1-8B-Instruct as pi_SFT, but this model is an instruction-tuned chat model, not a pure SFT checkpoint. This should be justified, since the DPO reference model is the starting model and the 'on-policy' samples are drawn from this same model.","section":"Section 3.1"},{"comment":"The column 'Avg Reward' in Table 3 is not defined in the text or caption; the reader cannot tell whether it is the average reward of the generated training responses or of the final model. Please clarify in the caption.","section":"Section 5.1, Table 3"},{"comment":"The caption contains the typo 'Perfomance' instead of 'Performance.'","section":"Figure 4 caption"},{"comment":"The benchmark name 'Winogrande' is misspelled as 'Winograde' in multiple places (Appendix D and the table captions).","section":"Appendix D and Tables 4–5"},{"comment":"In the definition of on-policy data, the notation D_on = {(x,y) | y ~ pi_theta or pi_SFT} would be clearer if it indicated which variable is the model and which is the policy; consider writing 'pi_theta' and 'pi_SFT' with explicit subscripts on the sampling distribution.","section":"Section 2, Set notation"}],"recommendation":"major_revision","confidential_remarks":"The main practical contribution—that mixing SFT-generated and external preference pairs at 50/50 improves alignment—is plausible and consistently supported across the reported settings. However, the manuscript's central 'on-policy' claim is more limited than the abstract suggests because all experiments use frozen-SFT samples rather than current-policy samples, and the complementarity analysis is based on very small per-category subsets. These issues are addressable with additional experiments or stronger framing, so I recommend major revision rather than rejection. I would also ask the authors to be more transparent about hyperparameter selection on the evaluation benchmark."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: SIMPLEMIX is a useful, controlled empirical study showing that mixing preference pairs sampled from the frozen SFT model with an existing offline dataset improves DPO; around 50/50 works well. But the 'on-policy' label is doing real work. By the RL definition the field usually uses, both sources are off-policy with respect to the final model, and the paper never tests sampling from the evolving policy. The comparison with HyPO and DPO-Mix-P is therefore less impressive than it looks.\n\nWhat's genuinely new: the task-dependent breakdown. Keeping the algorithm and data budget fixed, the authors show SFT-sampled pairs help math and coding, offline data help creative writing and recommendation, and a balanced mix does best on both. That's a clean, reproducible characterization, and the ablations on sampling temperature, diversity prompting, and off-policy filtering are thoughtful. The reward-distribution plot explaining why high-temperature sampling hurts quality is a nice touch.\n\nThe soft spots, in order of importance. First, the definitional issue above—it's acknowledged in footnote 1, but the abstract and conclusion still claim complementarity of on- and off-policy data, which overstates what's tested. Second, the headline numbers don't match Table 2. Averaging the per-row gains against on- and off-policy DPO gives about 4.5 points, not 6.03; against HyPO/DPO-Mix-P about 2.4, not 3.05. The authors need to correct the abstract. Third, the task-complementarity result rests on small subsets—roughly 35 math and 57 coding prompts—and Figure 2 has no error bars. Fourth, the 0.5 ratio and p=0.4 filter are chosen by looking at Alpaca Eval 2.0, so the headline configuration is tuned on the eval benchmark. This is partly mitigated because the main result also holds on the average of 8 reference-based benchmarks.\n\nNone of these is fatal. The core claim—mixing helps—survives the controlled comparisons, and the paper's Section 7 is honest about hyperparameter and judge-benchmark limitations. The citation pattern is fair and the math is standard DPO with no apparent errors.\n\nWho's this for? People working on preference optimization and data curation. It's a pragmatic recipe worth trying, and the task-type breakdown is a useful guide. My recommendation: send it to referees, but require a fix to the abstract numbers, confidence intervals or at least a caveat on the task-level results, and a serious attempt to either reframe the language or test actual current-policy sampling. With those changes, this becomes a solid contribution rather than a slightly overclaimed one.","headline":"The mixing recipe works and is well-controlled, but the 'on-policy' label is misleading—this is really SFT-sample plus offline data—and the abstract's numbers need fixing.","tokens_in":27837,"tokens_out":4312,"would_cite":true,"duration_ms":44983,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On-policy and off-policy preference data help different tasks, and SIMPLEMIX—an equal 50/50 mixture of the two—consistently outperforms using either source alone in DPO-based alignment.","keywords":["preference optimization","direct preference optimization","on-policy data","off-policy data","data mixing","language model alignment","task-dependent evaluation"],"falsifier":"Run SIMPLEMIX with on-policy data sampled from the current policy $\\pi_\\theta$ instead of the frozen $\\pi_{\\mathrm{SFT}}$, keeping all other settings identical; if the 50/50 mixture then no longer beats both pure on- and pure off-policy DPO on math and coding, or if open-ended tasks stop favoring off-policy data, the paper's complementarity claim as stated would be falsified.","tokens_in":26745,"feed_emoji":"📊","tokens_out":12027,"duration_ms":109527,"temperature":0.7,"pith_summary":"This paper attempts to settle a live dispute in language-model alignment: is preference data sampled from the model being trained (on-policy) strictly better than data collected from other models (off-policy), or is the advantage task-dependent? By varying only the data source while keeping the DPO algorithm fixed, the authors find that on-policy data brings most of its gains on objectively verifiable tasks such as math and coding, while off-policy data does better on open-ended tasks such as creative writing and personal recommendations. From this they propose SIMPLEMIX, which samples preference pairs with equal probability from the supervised-fine-tuned starting model $\\pi_{\\mathrm{SFT}}$ and from an off-policy dataset, then trains with the standard DPO loss. Across two base models and two preference datasets, the 50/50 mixture beats pure on-policy and pure off-policy DPO by an average of 6.03% on Alpaca Eval 2.0 and beats the more complex hybrid methods HyPO and DPO-Mix-P by an average of 3.05%.","feed_headline":"50/50 on- and off-policy mix beats pure DPO","feed_subtitle":"SimpleMIX lifts Alpaca Eval 2.0 win rate by 6.03% on average over on- or off-policy DPO alone.","key_machinery":"The mechanism that carries the argument is SIMPLEMIX's data distribution: for each example, the winning and losing responses are drawn with equal probability either from on-policy generations of the supervised-fine-tuned model $\\pi_{\\mathrm{SFT}}$ or from the off-policy dataset $D_{\\mathrm{off}}$, and the pair is trained with the DPO loss. This makes SIMPLEMIX a 'hard' version of the DPO-Mix-P sampler, which draws from an interpolation between the current policy and the SFT policy. The other load-bearing piece is the paper's task-type hypothesis: on-policy data helps verifiable reasoning tasks while off-policy data helps open-ended tasks, and the paper shows in §3 that this difference is not explained by response length.","core_discovery":"The central claim is that on-policy and off-policy preference data are complementary in direct preference optimization (DPO), the standard algorithm that increases the model's log-probability of chosen responses relative to rejected ones. In controlled experiments that change only the data source, on-policy data—responses sampled from the frozen supervised-fine-tuned model and labeled by a reward model—improves performance most on tasks with objective answers, such as mathematical reasoning and coding, while off-policy data from existing collections like UltraFeedback and HelpSteer2 improves performance most on open-ended tasks like creative writing and recommendations. Because the two sources help different task types, the authors propose SIMPLEMIX: mix the two data sources with equal probability inside the same DPO objective, so the training distribution is $D = D_{\\mathrm{on}} \\cup D_{\\mathrm{off}}$. This simple mixture outperforms both pure sources and the more complex hybrid methods HyPO and DPO-Mix-P, on both LM-as-a-judge evaluation (Alpaca Eval 2.0 length-controlled win rate) and reference-based benchmarks.","pith_inferences":["If the field's notion of on-policy were tied to the evolving policy, the 50/50 optimum might shift; a direct comparison of frozen-SFT sampling versus live-policy sampling would show whether SIMPLEMIX's simplicity carries over to true online DPO.","The task-dependent split suggests a finer-grained rule than a global 50/50 mix: tune the on/off sampling ratio per prompt category, which could improve on the flat mixture reported here.","The same mixing idea could be tested on other preference objectives such as IPO or KTO, since the mechanism is about where the preference pairs come from, not the specific loss.","Because off-policy quality filtering helps, a natural extension is to filter on-policy pairs with a held-out reward model as well, which would make the reported margin over hybrid baselines less sensitive to judge alignment."],"forward_implications":["A balanced 50/50 mix of on- and off-policy preference data is a strong default for DPO-style alignment, beating pure on-policy, pure off-policy, and the more complex hybrid samplers tested across two base models and two datasets.","Data-origin decisions should be task-dependent: on-policy sampling earns its extra compute on verifiable reasoning tasks, while abundant off-policy data suffices or wins on open-ended tasks.","Off-policy data is not just a diversity booster; in the right mixture it improves the final model, and filtering it by reward-model quality pushes the improvement further, for example from 29.41 to 31.62 length-controlled win rate on Llama-3.1-8B-Instruct.","The on-policy advantage on math and coding is not a length artifact, since the gain persists across generation-length bins, so practitioners can attribute it to task type rather than output length."],"supporting_citations":[{"why":"Supplies the DPO loss that all compared methods optimize and the off-policy DPO baseline.","marker":"(Rafailov et al., 2024)"},{"why":"UltraFeedback dataset; the main source of off-policy preference pairs in the experiments.","marker":"(Cui et al., 2024)"},{"why":"HelpSteer2 dataset; the second off-policy source used to test generality.","marker":"(Wang et al., 2024)"},{"why":"Provides the Tulu-3 SFT model and the on-policy definition the paper adopts.","marker":"(Lambert et al., 2024a)"},{"why":"HyPO baseline; the more complex hybrid method SIMPLEMIX is compared against.","marker":"(Song et al., 2024b)"},{"why":"DPO-Mix-P baseline; another complex sampler mixing current and SFT policies.","marker":"(Shi et al., 2024)"},{"why":"Llama-3.1 models used as the second SFT base for all experiments.","marker":"(Dubey et al., 2024)"},{"why":"Alpaca Eval 2.0 with length-controlled win rate, the primary LM-as-a-judge metric.","marker":"(Dubois et al., 2024)"}],"fun_headline_variants":["Mix on- and off-policy data, get 6% better DPO","On-policy for reasoning, off-policy for open-ended, mix wins","SIMPLEMIX: equal mix of data sources beats complex hybrids","Simple 50/50 data mix outperforms sophisticated preference methods","Complementary DPO data: on-policy + off-policy = better alignment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim rests on defining on-policy data as responses sampled once from the frozen supervised-fine-tuned starting model rather than from the model being updated during training; if on-policy is understood as sampling from the current evolving policy, the reported complementarity and the margins over HyPO and DPO-Mix-P may not carry over.","fun_headline_variants_meta":{"raw":{"variants":["Mix on- and off-policy data, get 6% better DPO","On-policy for reasoning, off-policy for open-ended, mix wins","SIMPLEMIX: equal mix of data sources beats complex hybrids","Simple 50/50 data mix outperforms sophisticated preference methods","Complementary DPO data: on-policy + off-policy = better alignment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000315,"raw_usage":{"total_tokens":1823,"prompt_tokens":1023,"completion_tokens":800,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":639,"completion_tokens_details":{"reasoning_tokens":701}},"tokens_in":639,"tokens_out":800,"duration_ms":7909,"temperature":1.0,"reasoning_tokens":701,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:54:45.420416+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SIMPLEMIX with on-policy data sampled from the current policy $\\pi_\\theta$ instead of the frozen $\\pi_{\\mathrm{SFT}}$, keeping all other settings identical; if the 50/50 mixture then no longer beats both pure on- and pure off-policy DPO on math and coding, or if open-ended tasks stop favoring off-policy data, the paper's complementarity claim as stated would be falsified.","supporting_citations":[],"review_version":1}