Pith. sign in

REVIEW 21 cited by

Aligning Text-to-Image Diffusion Models with Reward Backpropagation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.03739 v5 pith:YSHUYZCM submitted 2023-10-05 cs.CV cs.AIcs.LGcs.RO

Mihir Prabhudesai , Anirudh Goyal , Deepak Pathak , Katerina Fragkiadaki This is my paper · ORCID
classification cs.CVcs.AIcs.LGcs.RO
keywords modelsdiffusionrewardalignproptext-to-imagebackpropagationdownstreamfunctions
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Text-to-image diffusion models have recently emerged at the forefront of image generation, powered by very large-scale unsupervised or weakly supervised text-to-image training datasets. Due to their unsupervised training, controlling their behavior in downstream tasks, such as maximizing human-perceived image quality, image-text alignment, or ethical image generation, is difficult. Recent works finetune diffusion models to downstream reward functions using vanilla reinforcement learning, notorious for the high variance of the gradient estimators. In this paper, we propose AlignProp, a method that aligns diffusion models to downstream reward functions using end-to-end backpropagation of the reward gradient through the denoising process. While naive implementation of such backpropagation would require prohibitive memory resources for storing the partial derivatives of modern text-to-image models, AlignProp finetunes low-rank adapter weight modules and uses gradient checkpointing, to render its memory usage viable. We test AlignProp in finetuning diffusion models to various objectives, such as image-text semantic alignment, aesthetics, compressibility and controllability of the number of objects present, as well as their combinations. We show AlignProp achieves higher rewards in fewer training steps than alternatives, while being conceptually simpler, making it a straightforward choice for optimizing diffusion models for differentiable reward functions of interest. Code and Visualization results are available at https://align-prop.github.io/.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 21 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AssetDropper: Asset Extraction via Diffusion Models with Reward-Driven Optimization

    cs.CV 2025-06 conditional novelty 7.0 of 10

    AssetDropper introduces a task-specific diffusion model, a 212k-pair synthetic dataset, and a generative reward model to extract standardized assets from reference images.

  2. Sample and Computationally Efficient Continuous-Time Reinforcement Learning with General Function Approximation

    cs.LG 2025-05 conditional novelty 7.0 of 10

    PURE achieves an Õ(√(d_R+d_F)/√N) suboptimality gap, up to horizon factors, in continuous-time RL with general function approximation, and adds low-switching and low-rollout variants.

  3. Sample-Adaptive Latent Rewards for Uncertainty-Guided Diffusion Post-Training

    cs.CV 2026-08 conditional novelty 6.0 of 10

    SURE learns sample-adaptive variance in a latent reward model and uses that variance to weight dense post-training feedback, improving image and video diffusion alignment in reported experiments.

  4. WorldCycle: Self-Verifiable Reinforcement Learning for Long-Horizon Video World Models

    cs.AI 2026-08 conditional novelty 6.0 of 10

    WorldCycle post-trains interactive video world models with reinforcement learning rewards for spatial closure and temporal consistency on reversible action cycles, reducing long-horizon drift and improving composite-a...

  5. DanceOPD: On-Policy Generative Field Distillation

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    Hard-routed, single low-noise on-policy velocity matching composes conflicting image-generation capabilities into one flow student better than joint training, merging, or dense OPD baselines.

  6. Plug-and-Play Guidance for Discrete Diffusion Models via Gradient-Informed Logit Correction

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Introduces GILC, a training-free plug-and-play guidance framework for discrete diffusion models that uses Jacobian-free logit correction to achieve SOTA results on DNA, protein, and molecular generation tasks.

  7. Direct Diffusion Score Preference Optimization via Stepwise Contrastive Policy-Pair Supervision

    cs.CV 2025-12 conditional novelty 6.0 of 10

    Diffusion image models can be aligned without human labels by supervising every denoising step with score targets from original versus degraded prompts.

  8. Test-Time Alignment of Text-to-Image Diffusion Models via Null-Text Embedding Optimisation

    cs.CV 2025-11 conditional novelty 6.0 of 10

    Optimizing the null-text embedding in classifier-free guidance aligns diffusion outputs to a target reward while preserving cross-reward quality.

  9. A novel method and dataset for depth-guided image deblurring from smartphone Lidar

    eess.IV 2025-09 conditional novelty 6.0 of 10

    Lidar depth-guided deblurring via a zero-shot diffusion method, evaluated on a new 45-scene dataset, achieves the best perceptual quality (LPIPS).

  10. ShortFT: Diffusion Model Alignment via Shortcut-based Fine-Tuning

    cs.CV 2025-07 conditional novelty 6.0 of 10

    ShortFT fine-tunes Stable Diffusion by backpropagating reward gradients through a distilled few-step shortcut denoising chain, improving alignment scores over DRaFT-LV and DRTune.

  11. Diffusion Tree Sampling: Scalable inference-time alignment of diffusion models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Diffusion Tree Sampling is a Monte Carlo tree search over denoising trajectories that propagates terminal rewards backward to sample from reward-aligned distributions, showing up to 10x compute savings on tested benchmarks.

  12. Enhancing Diffusion-based Unrestricted Adversarial Attacks via Adversary Preferences Alignment

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Decoupling visual similarity and attack success into separate diffusion-model alignment stages yields unrestricted adversarial images with state-of-the-art black-box transferability.

  13. Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planning

    cs.LG 2025-06 conditional novelty 6.0 of 10

    LoMAP projects each guided diffusion sample onto a PCA subspace of nearby offline trajectories, reducing infeasible plans and improving returns in Maze2D, MuJoCo locomotion, and AntMaze.

  14. Text2Stereo: Repurposing Stable Diffusion for Stereo Generation with Consistency Rewards

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Text2Stereo adapts Stable Diffusion to generate wide-baseline stereo image pairs from text by fine-tuning with LoRA and a disparity-correlation consistency reward.

  15. DiffusionReward: Enhancing Blind Face Restoration through Reward Feedback Learning

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A reward-feedback fine-tuning framework trains a face reward model and uses its gradient plus structural and regularization losses to improve diffusion face restoration models.

  16. Dichotomous Diffusion Policy Optimization

    cs.LG 2025-12 conditional novelty 5.0 of 10

    DIPOLE decomposes a KL-regularized RL objective into a pair of sigmoid-weighted diffusion policies whose score combination (CFG-like) yields stable and controllable policy improvement.

  17. Instant Preference Alignment for Text-to-Image Diffusion Models

    cs.CV 2025-08 conditional novelty 5.0 of 10

    An MLLM-driven, training-free pipeline extracts preference keywords from a reference image and modulates diffusion cross-attention at global and regional levels for instant, multi-round preference-aligned image generation.

  18. ImageReFL: Balancing Quality and Diversity in Human-Aligned Diffusion Models

    cs.CV 2025-05 conditional novelty 5.0 of 10

    ImageReFL combines base-model early diffusion steps with a real-image-based fine-tuning objective to improve the quality-diversity trade-off in reward-aligned text-to-image generation.

  19. $I^2G$: Generating Instructional Illustrations via Text-Conditioned Diffusion

    cs.CL 2025-05 reject novelty 5.0 of 10

    A pairwise-conditioned diffusion model generates instructional illustrations from procedural text and is finetuned with a text-image alignment reward.

  20. Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences

    cs.CV 2025-06 conditional novelty 4.0 of 10

    SmPO-Diffusion improves diffusion-model preference alignment with reward-model soft labels and ReNoise inversion, reporting higher human-preference scores and up to 26x lower training cost than Diffusion-KTO.

  21. Rhetorical Text-to-Image Generation via Two-layer Diffusion Policy Optimization

    cs.CV 2025-05 reject novelty 4.0 of 10

    Rhet2Pix combines staged LLM prompt decomposition with a discounted PPO fine-tuning scheme for Stable Diffusion, claiming strong rhetorical text-to-image generation, but the quantitative evidence is circular and undefined.

Pith tools