Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that replacing binary preference labels with a reward-model-smoothed distribution and estimating sampling trajectories with ReNoise inversion makes text-to-image diffusion alignment both more accurate and far cheaper…

desk verdict Worth a referee's time, but the headline numbers should not be trusted until the derivation gap is fixed and the evaluation is cleaned up. read the letter →

arxiv 2506.02698 v2 pith:KDJNRE6J submitted 2025-06-03 cs.CV

Yunhong Lu , Qichao Wang , Hengyuan Cao , Xiaoyin Xu , Min Zhang This is my paper
classification cs.CV
keywords text-to-imagediffusionmodelsDirectPreferenceOptimizationhumanalignmentsmootheddistributionReNoiseinversionDDIMPickScorerewardmodeltrainingefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that two simple modifications make Direct Preference Optimization (DPO) much better for text-to-image diffusion models: replacing binary win/lose labels with a smoothed preference distribution, and estimating the sampling trajectory with ReNoise inversion instead of random forward noise. The smoothed labels come from a reward model, so no new human annotation is needed, and the loss is designed to shrink toward zero when the two images are judged nearly equal, which is meant to stop over-optimization. The inversion step is meant to fix the misalignment between the objective being optimized and the actual sampling process. If the method is right, preference alignment of diffusion models becomes both more faithful to varied human taste and much cheaper: the paper reports HPSv2.1 gains of +4.08 over SDXL and +4.7 over SD1.5, with training time reduced to 15.5% of Diffusion-DPO for SDXL and 20.2% for SD1.5.

What carries the argument

The load-bearing object is the smoothed preference distribution of Eq. (8), a weighted geometric mixture of winner and loser densities with normalization constants $Z_w$ and $Z_l$ that the derivation treats as constant and cancels. It produces the key identity Eq. (9), in which the entire effect of smoothing is a per-pair scaling factor $(2\alpha-\gamma)\beta$ on the standard DPO log-ratio; because $\alpha/\gamma$ is the reward-model softmax, pairs that are nearly tied get a coefficient near zero. The second mechanism is the Renoise-inversion estimator of Eq. (14): after a few DDIM inversion steps, a single ReNoise step rebuilds the latent, and Eqs. (15)-(16) compute the preference loss from the residual between the predicted noise and the latent's actual noise. Together these two mechanisms are what make the loss adaptive and the gradient signal aligned with the model's own generation trajectory.

What would settle it

Re-run SmPO training on a subset of Pick-a-Pic with $Z_w$ and $Z_l$ computed instead of omitted for both the trained and reference models, and compare the resulting loss and images with the paper's closed-form version; if the gains disappear or the two losses diverge significantly, the smoothing mechanism's derivation is not the actual source of improvement. A second check is to train with true human soft votes instead of PickScore labels; if gains vanish, the benefit comes from the reward model rather than from modeling human variability.

Watch

Extended reading notes

Core claim

The central discovery is that both the preference model and the trajectory estimator in diffusion DPO are too crude, and each has a straightforward fix. For preferences, the paper replaces the binary Bradley-Terry label with a smoothed distribution of the form $p(x_w)^\alpha p(x_l)^{\gamma-\alpha}$ (and symmetrically for the loser), where $\alpha/\gamma$ is set to the softmax of normalized reward-model scores; substituting this into the DPO ratio collapses the loss to $\log\sigma\big((2\alpha-\gamma)\beta\,(\log \frac{p_\theta(x_w)}{p_{\mathrm{ref}}(x_w)} - \log \frac{p_\theta(x_l)}{p_{\mathrm{ref}}(x_l)})\big)$, so that close preferences produce near-zero gradients. For trajectory estimation, the paper replaces random forward-process noise with a few steps of DDIM inversion plus a single ReNoise step, yielding latents highly correlated with the real image and a score function based on the noise residual at those latents. On HPDv2 and Parti-Prompts, the resulting SmPO models beat Diffusion-DPO, MaPO, KTO, SFT, and the base models on nearly all reward-evaluator metrics, with the largest gains on HPSv2.1, and the ablation table shows each component (inversion, ReNoise, smoothing) contributes.

Load-bearing premise

The simplified loss in Eq. (9) is exact only if the normalization constants $Z_w$ and $Z_l$ of the smoothed preference distribution are the same for the trained model and the reference model; the paper approximates them as constant and drops them, and if those constants actually differ per pair, the real optimized loss is not the clean $(2\alpha-\gamma)\beta$ rescaling.

Editorial extensions

If this is right

  • SmPO-SDXL reaches a median HPSv2.1 score of 32.53 on HPDv2, up from 28.45 for base SDXL, while SmPO-SD1.5 reaches 29.31, up from 23.61, and wins a majority of comparisons against each baseline on most metrics.
  • Training cost drops to 150.8 GPU hours for SDXL and 41.3 for SD1.5, versus 976.0 and 204.8 for Diffusion-DPO, and to about 3.9% of the cost of Diffusion-KTO on SD1.5.
  • Ablations show the gains are additive: replacing random noise with DDIM inversion, then adding the ReNoise correction, then adding smoothed preference labels each raises all five evaluation scores on SD1.5.
  • The aligned model transfers to conditional generation: using ControlNet with Canny and depth maps, the SmPO-aligned SDXL keeps its quality gains without additional training.
  • PickScore serves both as the pseudo-label source and as the best-performing training signal in the reward-model ablation, so the method needs no new human annotations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the smoothing derivation is right, the $(2\alpha-\gamma)\beta$ factor acts as an automatic per-pair regularization weight: near-ties get near-zero gradients, which suggests a general recipe against over-optimization in any DPO-style alignment, not just diffusion.
  • The ReNoise trajectory estimator should reduce gradient variance compared with random-noise Diffusion-DPO; a direct test would be to measure per-batch gradient variance or effective sample size under the two estimators.
  • Because the smoothed labels come from PickScore, the method's 'human variability' is really the reward model's variability; a testable extension is to replace PickScore with pooled soft human votes and check whether the HPSv2.1 gains survive.
  • The paper leaves implicit that the method should combine well with online data collection, since the inversion estimator does not depend on the dataset's image source; collecting new pairs from the current model and re-labeling them with the reward model would give an online version of SmPO.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes SmPO-Diffusion, a fine-tuning method for text-to-image diffusion models that replaces binary preference labels with soft labels derived from a reward model (PickScore) via a smoothed preference distribution, and that uses ReNoise inversion to estimate the diffusion trajectory distribution inside the DPO objective. Experiments on SD1.5 and SDXL claim state-of-the-art performance across five reward metrics on HPDv2 and Parti-Prompts, together with large reductions in training GPU hours compared with Diffusion-DPO, MaPO, and Diffusion-KTO.

Significance. If the derivation were correct, the smoothed-preference formulation would be a valuable step toward modeling preference variability in DPO, and the ReNoise-inversion estimator is a plausible efficiency improvement. The reported empirical gains are large, and the paper includes ablations, a user study, and computational-cost comparisons that are informative. However, the central theoretical derivation is invalid: the normalization constants in Eq. (8) do not cancel as claimed, and the advertised property that the loss approaches zero when preferences are similar is mathematically false. As a result, the paper's main conceptual contribution is not established, and the implemented loss is at best a heuristic whose relationship to the 'smoothed preference distribution' is unproven.

major comments (4)
  1. [Section 4.1, Eq. (8)-(9) and Appendix B, Eq. (17)] The normalization constants Z_w^p(c) and Z_l^p(c) in Eq. (8) are per-prompt partition functions that depend on the model distribution p (either p_theta or p_ref). In the DPO ratio, they enter as log(Z_w^{ref} Z_l^{theta} / (Z_w^{theta} Z_l^{ref})), which does not vanish. Appendix B cancels these terms without justification and even assigns Z_w to both the winner and loser slots, which is inconsistent with Eq. (8). Since Z_w^theta(c) = sum over image pairs for prompt c of p_theta(x_w)^alpha p_theta(x_l)^{gamma-alpha} (and similarly for Z_l), the omitted terms depend on theta and on alpha and gamma, so they change the gradient and the optimum of the loss. Consequently Eq. (9) is not a valid derivation of the smoothed objective, and the loss in Eq. (15) is not the objective that Eq. (9) claims to optimize.
  2. [Abstract, Section 1, and Section 4.1] The paper states that the loss function 'approaches zero when preferences are similar.' For a perfect tie, alpha/gamma = 0.5, so 2*alpha - gamma = 0, the logit in Eq. (9) is zero, and -log sigma(0) = log 2, not zero. Even if the normalization residual is ignored, the loss cannot approach zero at indifference because sigma(0) = 0.5. This is a central advertised property of the method and is false as stated.
  3. [Section 5.1 and Table 2] The training labels in Eq. (12) are computed with PickScore, and PickScore is also one of the five evaluators in Table 2. The PickScore scores and win-rates are therefore partially measuring the model's optimization of its own training signal, which is a form of reward overfitting and weakens the 'state-of-the-art' claim. The HPSv2.1, ImageReward, Aesthetic, and CLIP results are less affected, but the authors should either exclude PickScore from the evaluator set or clearly qualify the PickScore results as reflecting the training reward rather than an independent preference estimate.
  4. [Section 4.2 and Eq. (15)] The abstract and Section 4.2 describe the objective as a 'numerical upper bound estimation' for the diffusion DPO objective, but no upper bound is derived or proved. Replacing the forward noising process with ReNoise inversion yields a biased estimator of the expectation in Eq. (10), and the paper does not analyze the sign or size of this bias. If the paper is intended as an empirical method, the upper-bound framing should be removed or substantiated with a proof.
minor comments (6)
  1. [Table 2 caption] The caption does not mention that PickScore, HPSv2.1, and CLIP scores are multiplied by 100, although Appendix D explains this; please add a note to the caption so the table is self-contained.
  2. [Section 5.1] The learning-rate specification '2000/β 2.048−8' is unreadable; please write the formula explicitly, for example as lr = 2.048e-8 * 2000 / beta.
  3. [Section 2] The heading 'Text-to-Imgae Generative Models' contains a typo; it should be 'Text-to-Image Generative Models'.
  4. [Section 4.2] There is a duplicated word in the sentence after Eq. (16): 'denoising denoising xl0' should read 'denoising xl0'.
  5. [Appendix F] The text says 'Table 8 presents the quantitative comparison against SDXL baselines' and then repeats 'Table 8 presents' for the SD1.5 comparison; the second reference should be to Table 9.
  6. [Section 4.1, Eq. (8)] The notation for the normalization constants is inconsistent between Eq. (8), where Z_l^p(c) appears in the loser density, and Appendix B, where Z_w appears in both slots; please clarify the indexing.

Circularity Check

1 steps flagged · score 3.0 of 10

Moderate benchmark circularity: PickScore is both the soft-label source (Eq. 12) and one of the five evaluators (Table 2), but the central claim is independently supported by external metrics and the loss derivation is not an input-output equivalence.

  1. fitted input called prediction [Section 4.1 (Eq. 12) and Section 5.1 / Table 2]
    "For various preference data pairs, the weight-to-sensitivity ratio α/γ is modeled as: α/γ(xw0,xl0) = exp(r′(xw0,c))/(exp(r′(xw0,c))+exp(r′(xl0,c))) ... In our work, we employ PickScore (Kirstain et al., 2023) as the reward model. ... To comprehensively assess quality of image generation, we use five reward evaluators: CLIP ... PickScore (Kirstain et al., 2023) and HPSv2.1 ... for simulating human preferences."

    The same PickScore reward model that generates the soft labels α/γ in Eq. 12 (and thereby determines the training weight (2α−γ)β in Eq. 15) is also one of the five evaluators in Table 2. The PickScore win-rate reported in Table 2 is therefore not an independent test of the method: it measures the model against the very scorer used to create its training targets. The remaining four metrics (HPSv2.1, ImageReward, Aesthetic, CLIP) are external to the training signal, so the paper's overall SOTA claim remains independently grounded; only the PickScore column is partially circular.

full rationale

The central derivation is not circular: Eq. 9 does not reduce to Eq. 8 by construction, because the per-prompt normalizers Z_w and Z_l from Eq. 8 do not cancel in the DPO ratio unless one assumes they are equal across models and prompts; the paper approximates them as constants and omits them, and Appendix B's Eq. 17 cancels them by using Z_w for both winner and loser factors. This is an omitted-proof/algebraic correctness gap, not an input-output equivalence, so it does not count as circularity under the stated rules. The method's non-circular core is the reweighted DPO objective with ReNoise-inversion trajectory estimation, and the reported gains on external metrics (HPSv2.1, ImageReward, Aesthetic, CLIP) provide independent empirical support. The only genuine circularity is the PickScore overlap: PickScore supplies the soft labels and also appears as an evaluator, so the PickScore win-rate in Table 2 is a training-target overlap rather than an independent validation. This weakens that single metric but does not undermine the overall conclusion, which is why the score is moderate rather than high. Additionally, the paper's claim that the loss approaches zero when preferences are similar is not supported by Eq. 9 (as (2α−γ)→0 the loss tends to log 2, not 0); this is a mathematical error to be weighed as correctness risk, not circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method depends on four hand-chosen hyperparameters (gamma, beta, inversion steps, CFG), and on three background assumptions: the cancelation of the smoothed-distribution normalizers, the calibration of PickScore softmax values as human preference probabilities, and the validity of the inherited Diffusion-DPO upper bound. No new physical or model entities are introduced.

free parameters (4)
  • gamma (sensitivity factor) = 10 (SD1.5)
    Fixed exponent in Eq. (8) balancing winner/loser terms; selected by HPDv2 test-set evaluation (Table 5).
  • beta (DPO regularization) = 2000 (SD1.5), 5000 (SDXL)
    RLHF KL regularization coefficient in Eq. (15); tuned on HPDv2 test set (Table 5).
  • Max DDIM inversion steps = 9
    Number of steps in Eq. (13); 19 steps are better but costlier, 9 chosen as trade-off on HPDv2 (Table 4).
  • CFG scale during inversion = 1
    Classifier-free guidance weight used in the inversion equations; CFG=1 optimal on HPDv2 (Table 6).
assumptions (4)
  • ad hoc to paper The normalization constants Z_w and Z_l in Eq. (8) are parameter-independent and cancel in the DPO ratio.
    Section 4.1: 'the normalization term is approximated as a constant and omitted'; required for the closed-form loss (Eq. 9).
  • domain assumption The reward model PickScore, after min-max normalization and softmax (Eqs. 11-12), yields calibrated probabilities of human preference.
    The soft labels alpha/gamma are set to this softmax; no calibration or consistency check against human votes is provided.
  • domain assumption The Diffusion-DPO upper bound (Wallace et al., 2023) remains valid for the smoothed objective in Eq. (10).
    The paper inherits the upper bound without reproof; the ReNoise-based estimator is plugged into that bound.
  • standard math The Bradley-Terry preference model and DPO-to-RL equivalence are accepted as given.
    Standard results from Rafailov et al. (2024) used as the foundation of the loss in Eq. (4).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences." pith.science (2026). https://pith.science/paper/KDJNRE6J

@misc{pith2026250602698,
  author       = {Pith},
  title        = {Pith review of: Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KDJNRE6J}},
  note         = {Machine review of arXiv:2506.02698}
}
read the original abstract

Direct Preference Optimization (DPO) aligns text-to-image (T2I) generation models with human preferences using pairwise preference data. Although substantial resources are expended in collecting and labeling datasets, a critical aspect is often neglected: \textit{preferences vary across individuals and should be represented with more granularity.} To address this, we propose SmPO-Diffusion, a novel method for modeling preference distributions to improve the DPO objective, along with a numerical upper bound estimation for the diffusion optimization objective. First, we introduce a smoothed preference distribution to replace the original binary distribution. We employ a reward model to simulate human preferences and apply preference likelihood averaging to improve the DPO loss, such that the loss function approaches zero when preferences are similar. Furthermore, we utilize an inversion technique to simulate the trajectory preference distribution of the diffusion model, enabling more accurate alignment with the optimization objective. Our approach effectively mitigates issues of excessive optimization and objective misalignment present in existing methods through straightforward modifications. Our SmPO-Diffusion achieves state-of-the-art performance in preference evaluation, outperforming baselines across metrics with lower training costs. The project page is https://jaydenlyh.github.io/SmPO-project-page/.

Figures

Figures reproduced from arXiv: 2506.02698 by the authors.

Figure 1
Figure 1. Enhancement over Vanilla SDXL. We introduce SmPO-Diffusion, designed to address the variability of human preferences, as an adaptive and efficient approach to aligning diffusion models with human preferences. The images in the first row are generated by the SDXL model, while the second row presents images generated by the SDXL model fine-tuned using SmPO-Diffusion (denoted as SmPO-SDXL). The results demonstrate that… view at source ↗
Figure 2
Figure 2. Overview of SmPO-Diffusion. We present two steps of SmPO-Diffusion: (1) Smoothed preference modeling. We calculate smoothed preference labels for all image pairs in the dataset. Unlike traditional methods such as expert ratings and human voting, we employ a reward model (PickScore) to assign scores to preference pairs. These reward scores are subsequently normalized and processed using a Softmax function to derive t… view at source ↗
Figure 3
Figure 3. Qualitative comparison. We provide a qualitative comparison of SmPO-Diffusion and other different preference optimization methods (Refiner, Diffusion-DPO, MaPO) for SDXL. The results indicate that our model generates the highest quality images, including aspects such as llumination and spatial composition, text rendering, visual appeal, and others. dataset. To comprehensively assess quality of image gener￾ation, we … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Conditional generation comparison. SmPO-SDXL produces images with the highest quality (cf. Section 5.3). Computational cost. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: User Studies. We present the comparative user evaluation results of SmPO-SDXL against Base-SDXL, DPO￾SDXL, and MaPO-SDXL. The first row displays the user study results for text alignment, the second row for visual attractiveness, and the third row for overall user pref…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Inversion-DPO: Precise and Efficient Post-Training for Diffusion Models

    cs.CV 2025-07 reject novelty 5.0 of 10

    Inversion-DPO uses DDIM inversion to convert winning and losing images into noise trajectories, yielding a simpler DPO loss for diffusion model alignment that trains faster and improves text-to-image and compositional...

Reference graph

Works this paper leans on

57 extracted references · 9 canonical work pages · cited by 1 Pith paper

  1. [1]

    Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms.arXiv preprint arXiv:2402.14740,

    Ahmadian, A., Cremer, C., Gall´e, M., Fadaee, M., Kreutzer, J., Pietquin, O., ¨Ust¨un, A., and Hooker, S. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms.arXiv preprint arXiv:2402.14740,

    arXiv

  2. [4]

    Training diffusion models with reinforcement learning

    Black, K., Janner, M., Du, Y ., Kostrikov, I., and Levine, S. Training diffusion models with reinforcement learning. ArXiv, abs/2305.13301,

    arXiv

  3. [5]

    Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023a

    Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y ., English, Z., V oleti, V ., Letts, A., et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023a. Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S. W., Fidler, S., and Kreis, K. Align you...

    arXiv

  4. [6]

    URL https://openai.com/research/ video-generation-models-as-world\ -simulators. 9 Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences Cao, H., Feng, Y ., Gong, B., Tian, Y ., Lu, Y ., Liu, C., and Wang, B. Dimension-reduction attack! video generative models are experts on controllable image sy...

    arXiv

  5. [9]

    S., Wang, J., Wang, R., Zhang, P., Vandenhende, S., Wang, X., Dubey, A., Yu, M., Kadian, A., Radenovic, F., Mahajan, D

    Dai, X., Hou, J., Ma, C.-Y ., Tsai, S. S., Wang, J., Wang, R., Zhang, P., Vandenhende, S., Wang, X., Dubey, A., Yu, M., Kadian, A., Radenovic, F., Mahajan, D. K., Li, K., Zhao, Y ., Petrovic, V ., Singh, M. K., Motwani, S., Wen, Y ., Song, Y .-Z., Sumbaly, R., Ramanathan, V ., He, Z., Vajda, P., and Parikh, D. Emu: Enhancing image generation models using ...

    arXiv

  6. [10]

    Raft: Re- ward ranked finetuning for generative foundation model alignment.arXiv preprint arXiv:2304.06767,

    Dong, H., Xiong, W., Goyal, D., Zhang, Y ., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, T. Raft: Re- ward ranked finetuning for generative foundation model alignment.arXiv preprint arXiv:2304.06767,

    arXiv

  7. [11]

    Scaling rectified flow transformers for high-resolution image synthesis

    Esser, P., Kulal, S., Blattmann, A., Entezari, R., Muller, J., Saini, H., Levi, Y ., Lorenz, D., Sauer, A., Boesel, F., Podell, D., Dockhorn, T., English, Z., Lacey, K., Good- win, A., Marek, Y ., and Rombach, R. Scaling rectified flow transformers for high-resolution image synthesis. ArXiv, abs/2403.03206,

    arXiv

  8. [12]

    Kto: Model alignment as prospect theoretic optimization.arXiv preprint arXiv:2402.01306,

    Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model alignment as prospect theoretic optimization.arXiv preprint arXiv:2402.01306,

    arXiv

Show all 57 references
  1. [13]

    Dpok: Reinforcement learning for fine-tuning text-to- image diffusion models.ArXiv, abs/2305.16381,

    Fan, Y ., Watkins, O., Du, Y ., Liu, H., Ryu, M., Boutilier, C., Abbeel, P., Ghavamzadeh, M., Lee, K., and Lee, K. Dpok: Reinforcement learning for fine-tuning text-to- image diffusion models.ArXiv, abs/2305.16381,

    arXiv

  2. [14]

    S., Matsuo, Y ., Faust, A., Zen, H., and Gur, I

    Furuta, H., Lee, K.-H., Gu, S. S., Matsuo, Y ., Faust, A., Zen, H., and Gur, I. Geometric-averaged preference optimization for soft preference labels.arXiv preprint arXiv:2409.06691,

    arXiv

  3. [15]

    Gadre, S. Y ., Ilharco, G., Fang, A., Hayase, J., Smyrnis, G., Nguyen, T., Marten, R., Wortsman, M., Ghosh, D., Zhang, J., Orgad, E., Entezari, R., Daras, G., Pratt, S., Ramanujan, V ., Bitton, Y ., Marathe, K., Mussmann, S., Vencu, R., Cherti, M., Krishna, R., Koh, P. W., Sau...

    arXiv

  4. [16]

    Renoise: Real image inversion through iterative noising.ArXiv, abs/2403.14602,

    Garibi, D., Patashnik, O., V oynov, A., Averbuch-Elor, H., and Cohen-Or, D. Renoise: Real image inversion through iterative noising.ArXiv, abs/2403.14602,

    arXiv

  5. [17]

    Diffusion- rpo: Aligning diffusion models through relative prefer- ence optimization.ArXiv, abs/2406.06382,

    Gu, Y ., Wang, Z., Yin, Y ., Xie, Y ., and Zhou, M. Diffusion- rpo: Aligning diffusion models through relative prefer- ence optimization.ArXiv, abs/2406.06382,

    arXiv

  6. [18]

    Prompt-to-prompt im- age editing with cross attention control.arXiv preprint arXiv:2208.01626,

    Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y ., and Cohen-Or, D. Prompt-to-prompt im- age editing with cross attention control.arXiv preprint arXiv:2208.01626,

    arXiv

  7. [20]

    Denoising diffusion proba- bilistic models.ArXiv, abs/2006.11239,

    Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models.ArXiv, abs/2006.11239,

    2006 arXiv

  8. [22]

    A style-based generator architecture for generative adversarial networks.2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp

    Karras, T., Laine, S., and Aila, T. A style-based generator architecture for generative adversarial networks.2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4396–4405,

    2019

  9. [24]

    R., Mesnard, T., Ferret, J., Bishop, C., Hall, E., Carbune, V ., and Rastogi, A

    Lee, H., Phatale, S., Mansoor, H., Lu, K. R., Mesnard, T., Ferret, J., Bishop, C., Hall, E., Carbune, V ., and Rastogi, A. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. 2023a. Lee, K., Liu, H., Ryu, M., Watkins, O., Du, Y ., Boutilier, C., Abbeel,...

    2024 arXiv

  10. [25]

    Lipman, Y ., Chen, R. T. Q., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling.ArXiv, abs/2210.02747,

    arXiv

  11. [26]

    Alignment of diffusion models: Fundamentals, challenges, and future.ArXiv, abs/2409.07253,

    Liu, B., Shao, S., Li, B., Bai, L., Xu, Z., Xiong, H., Kwok, J., Helal, A., and Xie, Z. Alignment of diffusion models: Fundamentals, challenges, and future.ArXiv, abs/2409.07253,

  12. [27]

    J., and Liu, J

    Liu, T., Zhao, Y ., Joshi, R., Khalman, M., Saleh, M., Liu, P. J., and Liu, J. Statistical rejection sam- pling improves preference optimization.arXiv preprint arXiv:2309.06657,

    arXiv

  13. [28]

    Flow straight and fast: Learning to generate and transfer data with rectified flow

    Liu, X., Gong, C., and Liu, Q. Flow straight and fast: Learning to generate and transfer data with rectified flow. ArXiv, abs/2209.03003,

    arXiv

  14. [29]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101,

    Loshchilov, I. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101,

    arXiv

  15. [30]

    Spo: Multi-dimensional preference sequential alignment with implicit reward modeling.ArXiv, abs/2405.12739,

    Lou, X., Zhang, J., Xie, J., Liu, L., Yan, D., and Huang, K. Spo: Multi-dimensional preference sequential alignment with implicit reward modeling.ArXiv, abs/2405.12739,

    arXiv

  16. [31]

    Sdedit: Guided image synthesis and edit- ing with stochastic differential equations.arXiv preprint arXiv:2108.01073,

    Meng, C., He, Y ., Song, Y ., Song, J., Wu, J., Zhu, J.-Y ., and Ermon, S. Sdedit: Guided image synthesis and edit- ing with stochastic differential equations.arXiv preprint arXiv:2108.01073,

    arXiv

  17. [32]

    and Dhariwal, P

    Nichol, A. and Dhariwal, P. Improved denoising diffusion probabilistic models.ArXiv, abs/2102.09672,

    arXiv

  18. [33]

    Peebles, W. S. and Xie, S. Scalable diffusion models with transformers.2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4172–4182,

    2023

  19. [34]

    Sdxl: Im- proving latent diffusion models for high-resolution image synthesis.ArXiv, abs/2307.01952,

    Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Muller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffusion models for high-resolution image synthesis.ArXiv, abs/2307.01952,

    arXiv

  20. [35]

    T., and Mildenhall, B

    Poole, B., Jain, A., Barron, J. T., and Mildenhall, B. Dream- fusion: Text-to-3d using 2d diffusion.arXiv preprint arXiv:2209.14988,

    arXiv

  21. [36]

    Aligning text-to-image diffusion models with reward backpropagation.ArXiv, abs/2310.03739,

    Prabhudesai, M., Goyal, A., Pathak, D., and Fragkiadaki, K. Aligning text-to-image diffusion models with reward backpropagation.ArXiv, abs/2310.03739,

    arXiv

  22. [37]

    Video diffusion alignment via reward gradients.arXiv preprint arXiv:2407.08737,

    11 Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences Prabhudesai, M., Mendonca, R., Qin, Z., Fragkiadaki, K., and Pathak, D. Video diffusion alignment via reward gradients.arXiv preprint arXiv:2407.08737,

    arXiv

  23. [38]

    High-resolution image synthesis with la- tent diffusion models.2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with la- tent diffusion models.2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10674–10685,

    2022

  24. [40]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    arXiv

  25. [41]

    Denoising diffusion implicit models.ArXiv, abs/2010.02502, 2020a

    Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models.ArXiv, abs/2010.02502, 2020a. Song, Y . and Ermon, S. Generative modeling by estimating gradients of the data distribution.Advances in neural information processing systems, 32,

    2010 arXiv

  26. [43]

    R., and Naik, N

    Wallace, B., Dang, M., Rafailov, R., Zhou, L., Lou, A., Purushwalkam, S., Ermon, S., Xiong, C., Joty, S. R., and Naik, N. Diffusion model alignment using direct preference optimization.2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8228–8238,

    2024

  27. [44]

    K., Ramnath, K., Chaud- huri, S., Mehrotra, S., Zhu, Z., Mao, X.-B., Asur, S., and Cheng, N

    Wang, Z., Bi, B., Pentyala, S. K., Ramnath, K., Chaud- huri, S., Mehrotra, S., Zhu, Z., Mao, X.-B., Asur, S., and Cheng, N. A comprehensive survey of llm align- ment techniques: Rlhf, rlaif, ppo, dpo and more.ArXiv, abs/2407.16216,

    arXiv

  28. [45]

    Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis

    Wu, X., Hao, Y ., Sun, K., Chen, Y ., Zhu, F., Zhao, R., and Li, H. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. ArXiv, abs/2306.09341,

    arXiv

  29. [46]

    Multimodal large language model is a human-aligned annotator for text-to-image generation.ArXiv, abs/2404.15100,

    Wu, X., Huang, S., and Wei, F. Multimodal large language model is a human-aligned annotator for text-to-image generation.ArXiv, abs/2404.15100,

    arXiv

  30. [47]

    Imagereward: Learning and evaluating human preferences for text-to-image generation.ArXiv, abs/2304.05977,

    Xu, J., Liu, X., Wu, Y ., Tong, Y ., Li, Q., Ding, M., Tang, J., and Dong, Y . Imagereward: Learning and evaluating human preferences for text-to-image generation.ArXiv, abs/2304.05977,

    arXiv

  31. [48]

    Using human feedback to fine- tune diffusion models without any reward model.2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp

    Yang, K., Tao, J., Lyu, J., Ge, C., Chen, J., Li, Q., Shen, W., Zhu, X., and Li, X. Using human feedback to fine- tune diffusion models without any reward model.2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8941–8951,

    2024

  32. [49]

    A dense reward view on aligning text-to-image diffusion with preference.ArXiv, abs/2402.08265,

    Yang, S., Chen, T., and Zhou, M. A dense reward view on aligning text-to-image diffusion with preference.ArXiv, abs/2402.08265,

    arXiv

  33. [50]

    Y ., Luong, T., Baid, G., Wang, Z., Va- sudevan, V ., Ku, A., Yang, Y ., Ayan, B

    12 Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences Yu, J., Xu, Y ., Koh, J. Y ., Luong, T., Baid, G., Wang, Z., Va- sudevan, V ., Ku, A., Yang, Y ., Ayan, B. K., et al. Scaling autoregressive models for content...

    arXiv

  34. [51]

    Rrhf: Rank responses to align language mod- els with human feedback without tears.arXiv preprint arXiv:2304.05302,

    Yuan, Z., Yuan, H., Tan, C., Wang, W., Huang, S., and Huang, F. Rrhf: Rank responses to align language mod- els with human feedback without tears.arXiv preprint arXiv:2304.05302,

    arXiv

  35. [52]

    Zhang, C., Zhang, C., Zheng, S., Zhang, M., Qamar, M., Bae, S.-H., and Kweon, I. S. A survey on audio diffusion models: Text to speech synthesis and enhancement in generative ai.arXiv preprint arXiv:2303.13336, 2023a. Zhang, L., Rao, A., and Agrawala, M. Adding conditional con...

    2024 arXiv

  36. [53]

    and score-based matching (Song & Ermon, 2019; Song et al., 2020b), each providing unique theoretical frameworks and computational strategies for the generation process. In recent years, diffusion models and their variants, including Rectified Flow (Liu et al., 2022), have rise...

    2019

  37. [54]

    is a pivotal technique for aligning large language models (LLMs) with human preferences (Ouyang et al., 2022). This methodology typically involves two main steps: first, training a reward model to approximate human preferences, and second, applying reinforcement learning to op...

    2024

  38. [55]

    (2023) applies rejection sampling optimization to gather preference data from the optimal policy

    uses ranking loss for alignment, and Liu et al. (2023) applies rejection sampling optimization to gather preference data from the optimal policy. Direct Preference Optimization (DPO) (Rafailov et al.,

    2023

  39. [56]

    This method utilizes an image reward model to improve video quality while reducing fine-tuning costs through partial DDIM sampling

    enhances text-to-video diffusion models by incorporating reward fine-tuning supplemented with human feedback. This method utilizes an image reward model to improve video quality while reducing fine-tuning costs through partial DDIM sampling. Similarly, Prabhudesai et al. (2024...

    2024

  40. [57]

    For tasks involving depth maps and Canny edges, the controlnet conditioning scales are set to 0.5 and 0.3, respectively, with CFG fixed at 5 for both

    values for SD1.5 and SDXL are set to 7.5 and 5, respectively, following widely accepted standards. For tasks involving depth maps and Canny edges, the controlnet conditioning scales are set to 0.5 and 0.3, respectively, with CFG fixed at 5 for both. The random seeds for all co...

  41. [2017]

    Clark, K., Vicol, P., Swersky, K., and Fleet, D. J. Directly fine-tuning diffusion models on differentiable rewards. ArXiv, abs/2309.17400,

    arXiv

  42. [2018]

    Pick-a-pic: An open dataset of user preferences for text-to-image generation.ArXiv, abs/2305.01569,

    10 Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences Kirstain, Y ., Polyak, A., Singer, U., Matiana, S., Penna, J., and Levy, O. Pick-a-pic: An open dataset of user preferences for text-to-image generation.ArXiv,...

    arXiv

  43. [2019]

    N., Kingma, D

    Song, Y ., Sohl-Dickstein, J. N., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations.ArXiv, abs/2011.13456, 2020b. Sutton, R. S. and Barto, A. G.Reinforcement learning: An introduction. MIT press,

    2011 arXiv

  44. [2020]

    Reference-free monolithic preference optimization with odds ratio.arXiv e-prints, pp

    Hong, J., Lee, N., and Thorne, J. Reference-free monolithic preference optimization with odds ratio.arXiv e-prints, pp. arXiv–2403, 2024a. Hong, J., Paul, S., Lee, N., Rasul, K., Thorne, J., and Jeong, J. Margin-aware preference optimization for aligning dif- fusion models wit...

  45. [2021]

    Laion-5b: An open large-scale dataset for training next generation image-text models.ArXiv, abs/2210.08402,

    Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., Schramowski, P., Kundurthy, S., Crow- son, K., Schmidt, L., Kaczmarczyk, R., and Jitsev, J. Laion-5b: An open large-scale dataset for training next g...

    arXiv

  46. [2022]

    and Salimans, T

    Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598,

    arXiv

  47. [2023]

    Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models.arXiv preprint arXiv:2405.04233,

    Bao, F., Xiang, C., Yue, G., He, G., Zhu, H., Zheng, K., Zhao, M., Liu, S., Wang, Y ., and Zhu, J. Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models.arXiv preprint arXiv:2405.04233,

    arXiv

  48. [2024]

    Albergo, M. S. and Vanden-Eijnden, E. Building nor- malizing flows with stochastic interpolants.ArXiv, abs/2209.15571,

    arXiv

  49. [2025]

    Textdiffuser: Diffusion models as text painters.ArXiv, abs/2305.10855, 2023a

    Chen, J., Huang, Y ., Lv, T., Cui, L., Chen, Q., and Wei, F. Textdiffuser: Diffusion models as text painters.ArXiv, abs/2305.10855, 2023a. Chen, J., Yu, J., Ge, C., Yao, L., Xie, E., Wu, Y ., Wang, Z., Kwok, J., Luo, P., Lu, H., et al. Pixart-\α: Fast training of diffusion tra...

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.