{"id":"a32d13dc-a4d0-47e7-9e98-b0dee66bd47e","arxiv_id":"2505.11245","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Training a diffusion model on reversed preference pairs and using it as the negative CFG predictor improves human-preference alignment over standard preference-tuned models.","lead":"This paper proposes Negative Preference Optimization (NPO), which trains a second diffusion model to generate images people dislike, then uses that bad model as the negative branch of classifier-free guidance during generation. The result is a plug-in that improves standard preference-aligned text-to-image and text-to-video models on automated quality scores.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper does not rule out that the NPO branch works as a generic degraded predictor rather than a prompt-specific negative-preference model; its own corrupted-data ablation (Table 7) and the c/c′ mismatch in Eq. 9 leave the central mechanism underdetermined.","rationale":"The reader's weakest assumption correctly identifies that a reversed-preference model may be degenerate rather than semantically negative-aligned. My stress-test sharpens this into a concrete, testable mechanism: the paper never specifies whether c′ in Eq. 9 is the empty unconditional embedding or the original prompt, and a prompt-conditioned negative model evaluated at an empty conditioning cannot know what to avoid. The appendix's Table 7 provides internal evidence that a generically corrupted negative image (Gaussian blur) yields essentially identical scores to the original NPO, which is exactly what one would expect if the improvement comes from contrasting against a lower-quality branch rather than from learned prompt-specific avoidance. The Autoguidance comparison shows not every worse model works, so the direction of the effect is not vacuous, but it does not isolate the preference-reversal signal. The reason I keep the reader's CONDITIONAL verdict rather than moving to a stronger one is that the paper's broad empirical success, the scaled-DPO comparison, and the fact that some degraded branches (Autoguidance) fail mean the claim is plausible and testable; the missing piece is an ablation that removes the preference-reversal information while preserving branch quality. If that ablation matches NPO, the central conceptual claim would be substantially weakened, but the plug-and-play recipe might still be useful. The concrete test proposed here would settle this: if randomly labeled or weight-noised branches reproduce NPO's win rates, then 'negative preference optimization' as a distinct mechanism is unsupported; if they do not, the mechanism is confirmed. I also agree with the reader's secondary concerns about missing error bars and hyperparameter selection on validation, but these are not the load-bearing issue; the semantic content of the negative branch is.","tokens_in":19292,"tokens_out":6301,"duration_ms":68903,"concrete_test":"Re-run the NPO inference pipeline (same prompts, seeds, CFG settings, α/β) with four alternative negative branches: (a) base model θ, (b) a model trained with randomly shuffled pair labels (no preference signal), (c) θ with small Gaussian weight noise, and (d) the blurred-negative-image model from Table 7. Compute all four reported metrics and a larger pre-registered human preference study (at least 100 participants) on the Pick-a-pic validation set. If (a)-(c) match NPO's win rates, or if (d) matches NPO on every metric, the reversed-preference training is not the active ingredient and the 'negative preference optimization' claim reduces to CFG contrast with a degraded branch. If (a)-(c) fail while NPO and (d) succeed, the result is about any low-quality branch, not negative-preference semantics. If (d) fails and NPO wins, the semantic negative-preference account is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2's inference rule (Eq. 9) feeds the negative-preference model θneg the 'negative-conditional' input c′, while θneg is trained on prompt-conditioned pairs (Eq. 8, c). If c′ is the usual empty CFG conditioning, the branch cannot encode what is bad for the prompt, so any gain must come from generic properties of the branch. The appendix's own comparison (Table 7) is consistent with this: NPO trained with Gaussian-blurred negative images—a purely generic degradation—matches or slightly beats original NPO on all four metrics (Aesthetic 5.7676 vs 5.7621, HPS 27.63 vs 27.60, ImageReward 0.3222 vs 0.3102, PickScore 21.60 vs 21.58). That is the signature of a 'worse model' contrast effect, not of learned negative preference. The motivating weight-merge argument (Eq. 4-6) also only shows that a weaker model helps; the Autoguidance comparisons show that some weaker models hurt, but no condition isolates preference-reversal as the cause. Hence the broad plug-and-play gains may reflect CFG extrapolation and sharpening rather than the proposed negative-preference mechanism. This is not an internal inconsistency, but it is a load-bearing gap because the paper's central claim is that training on reversed preference pairs is what improves alignment.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Diffusion-NPO proposes to train an additional diffusion model that is aligned against human preferences by reversing preference pair order (Eq. 8) or negating the reward (Eq. 7), and then uses this model as the negative/unconditional branch in classifier-free guidance (Eq. 9). The paper evaluates the method as a plug-and-play enhancement for SD1.5, SDXL, and VideoCrafter2 across three preference optimization families (Diffusion-DPO, Diffusion-SPO, and VADER), reporting consistent improvements on PickScore, HPSv2, ImageReward, and LAION-Aesthetic, along with a 15-person user study. The core claim is that any existing preference optimization method can be converted into a negative-preference method at negligible extra cost, and that the resulting negative branch of CFG improves alignment with human preferences.","tokens_in":19543,"tokens_out":6859,"duration_ms":63036,"significance":"The proposed recipe is simple, training-data-free, and broadly applicable; the empirical coverage across three method families and two modalities is a real strength, and the release of code supports reproducibility. The paper also includes comparisons with training-free CFG-strengthening baselines such as Autoguidance and SEG. However, the evidence does not currently establish that reversing preference pairs is the mechanism behind the gains: the manuscript's own ablation (Table 7) shows that a model trained on generic Gaussian-blurred negative images performs equally well, and the inference rule (Eq. 9) evaluates the NPO branch on conditioning input c' that is not specified and may differ from the training conditioning c. If the mechanism is merely 'a worse version of the model yields better CFG contrast,' the contribution overlaps heavily with Autoguidance (Karras et al., 2024). The reported gains, if confirmed with significance tests, would still be practically useful, but the central interpretational claim needs substantial additional evidence.","major_comments":[{"comment":"The negative branch is evaluated at conditioning input c', but the NPO model is trained on paired data (x0, x1, c) with the same prompt c. The paper never states what c' is at inference or whether the NPO model receives the prompt at all. If c' is the standard empty/unconditional conditioning, the NPO model is evaluated out of its training distribution and cannot implement prompt-specific avoidance; if c' = c, the rule reduces to contrasting two models on the same prompt, which is precisely the Autoguidance setup and does not require any notion of negative preference. The authors should specify c' in their experiments and include a condition that varies c' (empty vs. prompt) to determine which setting is responsible for the gains.","section":"§3.2, Eq. (9) and §3.1, Eq. (8)"},{"comment":"Replacing the reversed-preference NPO training data with purely generic Gaussian-blurred negative images yields essentially the same or slightly better scores (Aesthetic 5.7676 vs. 5.7621, HPS 27.63 vs. 27.60, ImageReward 0.3222 vs. 0.3102, PickScore 21.60 vs. 21.58). Under the paper's hypothesis, the negative-preference content of the training pairs is the causal ingredient; under the alternative hypothesis, any sufficiently degraded predictor improves CFG contrast. The near-identical performance of the corruption-trained model is a direct test of the mechanism and currently supports the alternative. The authors need to add a condition that controls the training data while keeping the backbone, inference rule, and effective guidance strength fixed, and show that reversed-preference NPO outperforms a model trained to be generically worse (e.g., on blurred images) by a margin that cannot be explained by guidance-strength calibration.","section":"Appendix II.3, Table 7"},{"comment":"The weight-merge motivating example establishes only that a model with a weakened positive offset η improves the negative branch; it does not establish that the NPO offset δ contains a component orthogonal to η (δ⊥) that is responsible for the gains. The appendix acknowledges δ⊥ conceptually but provides no measurement of it. The scaling-DPO comparisons in Appendix II.1 show that DPO trained 10x longer is worse than DPO+NPO, but NPO also alters the negative branch; a proper control would be to compare against DPO(k=1)+NPO with δ replaced by -aη (or by a corrupt-trained offset) and show that the reversed-preference training contributes beyond the scale and direction of the existing offset.","section":"§2, Eqs. (4)–(6) and Appendix II.1"},{"comment":"All quantitative comparisons report means and win ratios without variance, confidence intervals, or significance tests. Several headline differences are small (e.g., Table 2, SDXL vs. SDXL+NPO: PickScore 22.06 vs. 22.32, HPSv2 27.89 vs. 28.11; Table 1, Diff.-SPO vs. Diff.-SPO+NPO: HPSv2 26.85 vs. 27.09), and win ratios are reported without error bars despite being computed over a finite prompt set. The authors should report paired significance tests (e.g., bootstrap or Wilcoxon) and the number of prompts/samples per cell so the reader can judge whether the consistency claim is supported.","section":"Tables 1–3 and Fig. 7"}],"minor_comments":[{"comment":"The caption says 'WinRate represents the percentage of cases where the method outperforms the original NPO,' which is an unusual baseline; please clarify the comparison protocol and state the number of prompts used.","section":"Appendix II.3, Table 7"},{"comment":"The transform RNPO = 1 - R assumes R ∈ [0,1], but the paper applies it to HPSv2/ImageReward-style rewards without stating how non-bounded rewards are rescaled in practice; please specify the exact scaling used in the experiments.","section":"§3.1, Eq. (7)"},{"comment":"There are several typos: 'Fo methods' (page 6), 'emplogy' (page 5), 'pptimization' in the Appendix heading II.3, 'f the deviation' (page 10), and 'Shanghai Jiang Tong University' in the affiliation should be 'Shanghai Jiao Tong University'.","section":"Throughout"},{"comment":"The text says NPO requires 'no new training strategies or datasets,' while Appendix II.3 explores modified corruptions of the dataset; please reconcile the wording, e.g., 'no new datasets are required for the core method, though data curation can further improve it.'","section":"Abstract and Fig. 2"},{"comment":"The relationship to Autoguidance (Karras et al., 2024) is discussed only in an appendix; given that the paper's own mechanism question turns on the distinction between 'negative preference' and 'a bad version of itself,' this comparison should be moved to the main text and expanded with the c'/c analysis requested above.","section":"Appendix II.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript carries the header 'Published as a conference paper at ICLR 2025' on the arXiv version. If this submission is to a journal, the editor should verify whether it is a substantially extended version of the ICLR paper and whether the venue allows such dual submission. The code repository link is mentioned; the provided code could help resolve the c'/c ambiguity in Eq. (9), so the authors should be strongly encouraged to specify the exact inference conditioning used in all reported experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a useful empirical package, but the central mechanism is underdetermined. Training a second model on reversed preferences and using it as the CFG negative branch is a genuinely new move, and the evaluation is broad: DPO, SPO, reward backprop, SD1.5, SDXL, video. The consistent wins across those settings argue that the trick works. The appendix's scaling-DPO comparison is the right control and it strengthens the case that NPO is not just more of the same.\n\nThe soft spots are real, and one is load-bearing. Eq. 9 feeds the negative model a negative-conditional input c' (typically the empty prompt) while θneg was trained on prompt-conditioned pairs. So the branch is never shown to encode anything prompt-specific. The appendix's Table 7 makes that worse: swapping in Gaussian-blurred negative images matches or slightly beats the original NPO on every metric. That is the signature of a generic 'worse model' contrast effect, not of learned negative preference. The motivating weight-merge argument only shows that a weaker model can help; it does not isolate preference reversal as the cause. The Autoguidance comparison is a useful baseline, but it doesn't close that gap either.\n\nAlso, Tables 1–3 report means and win ratios without variances or significance tests, and several headline hyperparameters are selected on validation. That is not fatal in a systems paper, but it limits how much weight the quantitative story can carry. The limitation statement about doubled storage is honest but minor, especially with LoRA.\n\nWho is this for? Anyone working on preference alignment in diffusion models. The recipe is simple and likely to be tried regardless of the theoretical story because it costs little. The paper deserves a serious referee and a fair hearing. The most useful revision would be a direct test of the mechanism: train the negative branch without preference reversal on the same data, and compare prompting it with c versus c' at inference. Until that exists, I would treat the gains as empirically real but the explanation as open.","headline":"The NPO trick is cheap and the gains look consistent across models and methods, but the paper never rules out that its negative branch works simply because it is a worse model.","tokens_in":20110,"tokens_out":2543,"would_cite":true,"duration_ms":25747,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training a model to generate bad images makes diffusion models better","keywords":["negative preference optimization","classifier-free guidance","diffusion models","preference alignment","text-to-image","text-to-video","direct preference optimization","plug-and-play inference"],"falsifier":"Run a controlled test where NPO is replaced by a model trained on randomized preference labels (noise instead of reversed preferences) at the same regularization strength; if the random-negative model yields the same score gains as genuine NPO on the Pick-a-pic benchmark, then the reported improvements do not depend on learning true negative preferences.","tokens_in":1646,"feed_emoji":"🎨","tokens_out":2625,"duration_ms":42368,"temperature":0.7,"pith_summary":"The paper argues that preference-aligned diffusion models are held back by a neglected half of classifier-free guidance: the negative or unconditional branch. Existing alignment methods teach a model what humans like, but not what they dislike, so the guidance contrast is weak. The authors propose Negative Preference Optimization (NPO), which trains a second model on reversed preference pairs or a negated reward, then uses that model as the negative-conditional predictor during CFG. This requires no new datasets, reward models, or training strategies, only a sign flip in the existing objective. A sympathetic reading takes the central claim to be that NPO consistently improves preference scores across SD1.5, SDXL, video diffusion, and already preference-optimized models, acting as a plug-and-play inference enhancement. The paper also notes that NPO doubles the stored weight offsets, though LoRA training keeps the extra cost small.","feed_headline":"Teach diffusion what to avoid and scores rise","feed_subtitle":"Negative Preference Optimization flips standard training pairs to build a contrast branch that improves image and video generation quality.","key_machinery":"The central object is the negative weight offset δ, trained by negative preference optimization, combined with the classifier-free guidance formula ε^ω = (ω+1)ε_{θ_pos} - ω ε_{θ_neg}. Preference alignment produces a positive offset η from the base weight θ; NPO produces a negative offset δ. The authors recommend composing the negative model as θ_neg = θ + αη + βδ, keeping a controlled amount of the positive direction to avoid divergence. This weight-offset picture explains why the method works: CFG is only as strong as the contrast between the conditional and negative-conditional predictions, and a deliberately trained negative branch sharpens that contrast.","core_discovery":"The central discovery is that reversing the preference order in any existing diffusion preference optimization method yields a model whose CFG guidance sharply increases alignment with human preferences. Concretely, training on swapped pairs r_NPO = (x1, x0, c), or equivalently maximizing 1 - R(x,c) instead of R(x,c), produces a weight offset δ that points away from desirable outputs. When this offset is added to the base model to form θ_neg, and CFG is run as ε = (ω+1)ε_{θ_pos} - ωε_{θ_neg} with θ_pos the positively aligned weights, the negative branch serves as a learned 'anti-model'. The paper thereby converts a training-time oversight into a training-free benefit: without new data or algorithms, any DR, RL, or DPO pipeline can be dual-trained into a positive and a negative model, and the contrast between them improves detail, color, and composition.","pith_inferences":["NPO's benefit may stem from implicit unlearning: the negative branch learns to suppress features that reward models over-weight, effectively acting as an anti-reward regularizer that reduces over-saturated or 'reward-hacked' outputs.","The same pair-reversal trick could apply to any alignment signal beyond visual preference, such as safety or style constraints, suggesting NPO as a general recipe for building contrastive guidance branches.","Because preference optimization is often trained with LoRA, the two-offset storage overhead is small, which lowers the practical barrier to retraining existing pipelines with this dual-branch scheme."],"forward_implications":["Any existing preference optimization method (differentiable reward, RL, or DPO) can be converted to a negative-preference variant by reversing the image pair order or negating the reward, with no new datasets or training strategies.","NPO functions as a plug-and-play inference enhancement: it raises PickScore, HPSv2, ImageReward, and LAION-Aesthetic scores on Stable Diffusion 1.5, SDXL, DreamShaper, and preference-optimized variants of each.","The method transfers to text-to-video generation, where a VADER model plus NPO(HPSv2) improves both in-domain aesthetics and out-of-domain human-prompt benchmarks.","NPO's improvements persist relative to simply training DPO for ten times longer, indicating the negative offset captures something that scaling the positive direction does not.","The weight offsets can be flexibly scaled (α, β) to trade off between sharpness and naturalness, giving users a direct dial on the strength of negative guidance."],"supporting_citations":[{"why":"Supplies the Diffusion-DPO baseline whose preference pairs are reversed to train NPO.","marker":"Wallace et al., 2024"},{"why":"Supplies the Diffusion-SPO baseline that NPO is tested against and combined with.","marker":"Liang et al., 2024b"},{"why":"Supplies the VADER reward-gradient method for text-to-video, which NPO augments with a negative branch.","marker":"Prabhudesai et al., 2024"},{"why":"Provides the Pick-a-pic dataset and PickScore metric used for preference pairs and evaluation.","marker":"Kirstain et al., 2023"},{"why":"Defines classifier-free guidance, the contrast mechanism NPO refines.","marker":"Ho & Salimans, 2022"},{"why":"Provides HPSv2, the human preference score used to measure alignment.","marker":"Wu et al., 2023"},{"why":"Contributes ImageReward as a reward and evaluation metric.","marker":"Xu et al., 2024"},{"why":"Presents Autoguidance, a training-free CFG-strengthening baseline compared against.","marker":"Karras et al., 2024"}],"fun_headline_variants":["Anti-model for diffusion: flip preferences, boost alignment","Diffusion's negative branch: better guidance by contrast","Flipping preference pairs trains a diffusion anti-model","Negative preference optimization sharpens CFG guidance","Train an anti-diffusion to guide positive outputs"],"cache_read_input_tokens":22144,"weakest_assumption_plain":"The approach assumes that a model trained on reversed preference pairs or a negated reward learns to produce genuinely bad outputs, not merely degenerate or miscalibrated ones, so that the negative branch in classifier-free guidance encodes a useful avoidance direction.","fun_headline_variants_meta":{"raw":{"variants":["Anti-model for diffusion: flip preferences, boost alignment","Diffusion's negative branch: better guidance by contrast","Flipping preference pairs trains a diffusion anti-model","Negative preference optimization sharpens CFG guidance","Train an anti-diffusion to guide positive outputs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000228,"raw_usage":{"total_tokens":1454,"prompt_tokens":903,"completion_tokens":551,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":477}},"tokens_in":519,"tokens_out":551,"duration_ms":5345,"temperature":1.0,"reasoning_tokens":477,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:54:49.771777+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled test where NPO is replaced by a model trained on randomized preference labels (noise instead of reversed preferences) at the same regularization strength; if the random-negative model yields the same score gains as genuine NPO on the Pick-a-pic benchmark, then the reported improvements do not depend on learning true negative preferences.","supporting_citations":[],"review_version":1}