Pith. sign in

REVIEW 4 major objections 6 minor 58 references

Fake it till You Make it: Reward Modeling as Discriminative Prediction

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a reward model for visual generation can be learned as a binary discriminator between a few hundred unpaired, unlabeled target samples and the generator's own outputs, achieving preference-alignment results…

desk verdict Useful method with a real circularity in its headline FID evaluation; worth reviewing after the evaluation is fixed. read the letter →

arxiv 2506.13846 v2 pith:GGBKEZHR submitted 2025-06-16 cs.CV cs.AIcs.LG

Runtao Liu , Jiahao Zhan , Yingqing He , Chen Wei , Alan Yuille , Qifeng Chen This is my paper
classification cs.CVcs.AIcs.LG
keywords rewardmodelingpreferencealignmenttext-to-imagegenerationdiffusionmodelsdirectoptimizationadversarialtrainingsampleselectionvideo
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

GAN-RM is a reward-modeling framework that turns preference learning into a binary discrimination task. Instead of collecting human preference annotations or designing quality dimensions, the method takes a few hundred unpaired target samples (called Preference Proxy Data) and trains a classifier on a CLIP vision backbone to tell these samples apart from the generator's outputs. The paper shows this reward signal can drive Best-of-N sample selection, supervised fine-tuning, and direct preference optimization, bringing image FID, PickScore, ImageReward, and HPS on SD1.5 and SDXL to roughly the level of DiffusionDPO, which uses one million human preference labels. It also applies the same recipe to safety alignment and video generation, and reports that iterative rounds of retraining the discriminator keep improving the model. The value of the claim is that it suggests human preference alignment can be bootstrapped from a tiny, unlabeled set of representative samples.

What carries the argument

GAN-RM is a binary reward classifier built from the CLIP vision encoder with a trainable Reward Projection Layer, a normalized multi-layer perceptron, trained by binary cross-entropy to distinguish the positive class (Preference Proxy Data, e.g., 500 JourneyDB images) from the negative class (raw generator outputs). Rank-based Bootstrapping then generates additional pseudo-labeled samples from the current model, with the top-scoring outputs receiving soft labels that decay exponentially with rank, and these are added to the training set. The same classifier is updated each round against the current generator's outputs, creating a loop that simultaneously refines the reward model and the policy.

What would settle it

Train GAN-RM on a deliberately unrepresentative proxy set, for example 500 images of a single artistic style, and measure whether downstream FID and human-preference scores collapse; the paper itself notes in its limitations that low-quality proxy data degrades performance, so a controlled scale-up of that degradation would test the representativeness premise directly.

Watch

Extended reading notes

Core claim

The central discovery is that the reward function for aligning a visual generator with human preference can be approximated by a decision boundary between a small set of representative 'Preference Proxy Data' and the generator's ordinary outputs. The paper argues that a binary classifier trained on this boundary, GAN-RM, learns latent human preferences encoded in the proxy samples, with no manual annotation of preferences or explicit engineering of quality dimensions. Empirically the paper reports that, using only 0.5K proxy images, this classifier produces reward scores that correlate with ImageReward, HPS, PickScore, and FID for Best-of-N selection, and that SFT and DPO fine-tuning on those scores yields models comparable to or better than DiffusionDPO trained on the 1M-pair Pick-a-Pic dataset. The same mechanism is claimed to generalize to safety alignment (reducing inappropriate content probability) and to video generation (improving FVD and VBench with 500 Artgrid videos).

Load-bearing premise

The load-bearing premise is that a few hundred unpaired, unlabeled proxy samples genuinely represent the human preferences the user cares about across the whole prompt distribution; if they are not representative, the reward model will push the generator toward that narrow sample set rather than toward true preference.

Editorial extensions

If this is right

  • Reward modeling for visual generators no longer requires human preference annotation or hand-designed quality dimensions; a few hundred unpaired target samples suffice.
  • Test-time scaling via Best-of-N becomes practical without any fine-tuning: generate K candidates and select by GAN-RM score, which the paper shows correlates with preference metrics.
  • Post-training can be driven by the discriminator's scores, and multi-round training can update both the generator and the reward model simultaneously, avoiding the single-round limitation of fixed preference datasets.
  • The same discrimination recipe works across image quality, safety alignment, and video generation, indicating a general mechanism rather than a per-domain pipeline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the reward signal is defined by proximity to the proxy set, the method's success should hinge on how representative those few hundred samples are; an extension would be to select proxy samples that explicitly span the prompt and style distribution, or to monitor the discriminator's calibration on held-out human judgments.
  • The bootstrapping loop creates a feedback between the generator and the reward model; an inference is that this could be exploited by reward hacking, so one could test whether the discriminator's confidence grows while an independent human-preference metric stagnates.
  • The same 'discriminate proxy data from generator output' recipe could be applied to other generative domains (audio, 3D assets, long-form video), where small curated sets of desirable outputs are available, although the paper only demonstrates images and short clips.
  • If the proxy data contains harmful or biased images, the reward model will steer the generator toward those biased characteristics; a practical safeguard would be to audit the proxy set before training, rather than relying on the discriminator to discover bias on its own.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes GAN-RM, a reward model trained as a binary discriminator between a small set of unpaired 'Preference Proxy Data' (a few hundred samples) and outputs of the generative model, with a rank-based bootstrapping step that augments the training set using the model's own scored samples. The reward model is then used for Best-of-N sample selection, SFT, and DPO post-training on SD1.5, SDXL, and VideoCrafter2, with additional experiments on safety alignment and video generation. The central claim is that with only 0.5K unpaired proxy samples GAN-RM achieves performance comparable to or better than DiffusionDPO, which uses 1M human preference labels, across FID, ImageReward, PickScore, HPS, and CLIPScore, plus a user study showing a 74.4% preference for the DPO-tuned model.

Significance. If validated, the paper would make a useful contribution: it offers a simple, low-annotation alternative to large preference datasets for reward modeling in visual generation, and it demonstrates the idea across image quality, image safety, and video generation, including multi-round DPO. The method is clearly described and the experimental scope is broad. However, the headline FID evaluation is compromised by direct overlap between the proxy training distribution and the FID reference distribution, which is a load-bearing issue for the paper's central efficiency claim. The non-FID metrics and the user study provide partial support, but the current presentation does not disentangle genuine preference learning from distribution matching toward the proxy dataset.

major comments (4)
  1. [Sec. 4.1-4.2, Table 1] The headline FID results are compromised by dataset overlap: Sec. 4.1 states that the 500 Preference Proxy images are sampled from JourneyDB, and Sec. 4.2 states that FID is computed on JourneyDB. Since GAN-RM is trained to assign high scores to JourneyDB images and low scores to generator outputs, Best-of-N selection and DPO/SFT training move the generated distribution toward JourneyDB; reporting FID against JourneyDB therefore measures proximity to the training distribution rather than independent human preference. The comparison with DiffusionDPO is unfair in this respect because DiffusionDPO's Pickapic training data is unrelated to the FID reference. I request that FID be evaluated on a reference dataset disjoint from the proxy source (as the safety experiment does with COCO), or that FID be removed from the headline claims and the paper rely on ImageReward, PickScore, HPS, and the user study, which are not subject to this artifact.
  2. [Sec. 3.3, Algorithm 1, Table 5] The multi-round DPO experiment in Table 5 reports only FID, which is exactly the metric affected by the JourneyDB circularity. As rounds progress, GAN-RM is retrained on the same 500 JourneyDB positives and the generator is pulled toward them, so FID improvement across rounds is expected even without learning generalizable human preferences. The appendix Table 9 does include ImageReward, PickScore, and HPS for SD1.5; these non-FID metrics should be reported in the main multi-round table, and the same metrics should also be provided for SDXL before the multi-round claim is accepted.
  3. [Sec. 3.1, Eq. (1)] The rank-based bootstrapping pseudo-labels y = exp(-alpha * r) are generated from GAN-RM's own scores and then used as training targets for the same model. This is a self-training loop whose reliability is asserted rather than demonstrated; the Table 4 ablation shows that bootstrap beats naive training on downstream metrics, but it does not show that the pseudo-labels track human preference rather than the model's own confidence. Please report sensitivity to alpha and M and include an experiment where pseudo-labeled positives are checked against a small human-annotated validation set.
  4. [Table 1] All comparisons in Table 1 appear to be single runs with no variance or significance information; FID differences between Ours-SFT/DPO and DiffusionDPO are sometimes small, and SDXL DiffusionDPO's FID is actually worse than the base model. Please report means and standard deviations over at least three seeds and specify the exact DiffusionDPO training recipe and evaluation protocol used for the numbers in Table 1.
minor comments (6)
  1. [Throughout] There are several typos and grammatical slips: 'Naiive' should be 'Naive', 'Boostrap' should be 'Bootstrap', 'ultilized' should be 'utilized', 'inpropriate' should be 'inappropriate', and 'Or method like RAFT' should be 'Other methods like RAFT'.
  2. [Table 4] The CLIPScore column in Table 4 lists values around 0.063-0.064, which are inconsistent with the CLIPScore values around 0.70 in Table 1 for the same settings; this appears to be a decimal-place error and should be corrected.
  3. [Appendix C] The sentence 'As a supplement to Tab. 9' is confusing because it appears in the paragraph describing Table 9 itself; the intended cross-reference is probably to Table 5 in the main text.
  4. [Figure 2] Figure 2 shows averages over prompts without error bars or confidence intervals; adding per-rank variability would strengthen the claim of a monotone correlation between GAN-RM rank and the four metrics.
  5. [User study, Sec. 4.2] The user study reports a 74.4% winning rate and calls it 'statistically significant' but provides no test statistic, p-value, or confidence interval; with 14 evaluators and 50 prompts, a binomial confidence interval should be reported.
  6. [Table 2] The 'IP' metric in Table 2 is reported without units or a definition of how the inappropriate probability is aggregated, and the values (e.g., 421 vs. 34) are hard to interpret; please clarify the computation and range of this metric.

Circularity Check

1 steps flagged · score 6.0 of 10

FID gains in Table 1 are partially circular because the reward model is trained on 500 JourneyDB images and then evaluated by FID computed on the same JourneyDB dataset.

  1. fitted input called prediction [Section 4.1 (Implementation details) and Section 4.2 (Performance), Tables 1, 7, 8]
    "For the image quality setting, we selected 500 images from JourneyDB [43] as our target images to train the reward model. ... For the image quality evaluation, the FID metric is computed on the JourneyDB dataset [43]."

    GAN-RM is trained with binary cross-entropy to label the 500 JourneyDB Preference Proxy images as positive and generated samples as negative (Sec. 3.1-3.2). Best-of-N selection and SFT/DPO then move generated outputs toward that positive class. FID is computed on JourneyDB, the same dataset from which the proxy was sampled, so a policy that merely mimics the proxy distribution mechanically lowers FID, independent of whether it captures human preference. This makes the headline FID comparison against DiffusionDPO partly circular; the external metrics (ImageReward, HPS, PickScore, CLIP) and the user study provide independent evidence.

full rationale

The paper's core method is a GAN-style discriminator: it separates a few hundred Preference Proxy samples (label 1) from generator outputs (label 0), and the reward is the sigmoid confidence. There is no hidden derivation that reduces to its inputs beyond the standard adversarial objective, and no load-bearing self-citation chain: the cited prior work by the authors provides datasets and baselines rather than the uniqueness of the method. The principal circularity is in the image-quality evaluation. Section 4.1 states that the 500 target images are selected from JourneyDB, and Section 4.2 states that FID is computed on JourneyDB. Since FID measures closeness to the target distribution, and the reward model is trained to label exactly that target distribution as positive, the FID improvements in Table 1 (and Tables 7-8) are to a significant degree forced by construction. This does not invalidate the paper because ImageReward, HPS, PickScore, CLIPScore, and the 74.4% user-study win rate are computed with respect to external human-preference references and are not contaminated by the JourneyDB overlap. The central efficiency claim is therefore supported by independent metrics, but the FID-based head-to-head with DiffusionDPO should be re-measured against a reference disjoint from the Preference Proxy Data before being taken as evidence.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the representativeness of the preference proxy data, the sufficiency of CLIP features for quality/safety, and the reliability of self-generated pseudo-labels. The free parameters alpha and M are not fully specified or analyzed.

free parameters (2)
  • alpha
    Used in the pseudo-label formula y = exp(-alpha * rank). Controls the decay of confidence with rank. The paper does not report the value or provide sensitivity analysis.
  • M
    Number of pseudo-positive and pseudo-negative samples selected in the rank-based bootstrapping step. Not specified in the main text, affecting the bootstrapping dataset composition.
assumptions (3)
  • domain assumption Preference Proxy Data is representative of human preferences.
    The entire method assumes that a small set of unlabeled, hand-picked samples (e.g., from JourneyDB) encodes latent human preferences, and that closeness to this set in CLIP feature space corresponds to human preference. The limitations section admits degradation if proxy data quality is poor.
  • domain assumption CLIP vision features are sufficient to distinguish high-quality from low-quality and safe from unsafe samples.
    The reward model is a linear/MLP head on CLIP features; if CLIP features do not capture the relevant quality/safety dimensions, the discriminator cannot learn them. No dedicated analysis of feature sufficiency is provided.
  • ad hoc to paper Pseudo-labels from the reward model's own ranking are reliable enough to improve training.
    Rank-based bootstrapping selects top and bottom samples from the current reward model and labels them as positive/negative with decaying confidence; this is a self-training assumption that the reward model's errors are not systematic.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fake it till You Make it: Reward Modeling as Discriminative Prediction." pith.science (2026). https://pith.science/paper/GGBKEZHR

@misc{pith2026250613846,
  author       = {Pith},
  title        = {Pith review of: Fake it till You Make it: Reward Modeling as Discriminative Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GGBKEZHR}},
  note         = {Machine review of arXiv:2506.13846}
}
read the original abstract

An effective reward model plays a pivotal role in reinforcement learning for post-training enhancement of visual generative models. However, current approaches of reward modeling suffer from implementation complexity due to their reliance on extensive human-annotated preference data or meticulously engineered quality dimensions that are often incomplete and engineering-intensive. Inspired by adversarial training in generative adversarial networks (GANs), this paper proposes GAN-RM, an efficient reward modeling framework that eliminates manual preference annotation and explicit quality dimension engineering. Our method trains the reward model through discrimination between a small set of representative, unpaired target samples(denoted as Preference Proxy Data) and model-generated ordinary outputs, requiring only a few hundred target samples. Comprehensive experiments demonstrate our GAN-RM's effectiveness across multiple key applications including test-time scaling implemented as Best-of-N sample filtering, post-training approaches like Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO). Code and data will be released at https://github.com/Visualignment/GAN-RM.

Figures

Figures reproduced from arXiv: 2506.13846 by the authors.

Figure 1
Figure 1. Illustration of the GAN-RM framework in the [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. This figure illustrates the distribution of FID, PickScore, ImageReward, and HPS for [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 4
Figure 4. These results underscore the robustness and generalizability of GAN-RM across diverse [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (8 more)
Figure 3
Figure 3. Figure 3: Qualitative results. This figure compares the generation results of different strategies based on GAN-RM. The image quality generated by our method is significantly improved compared to the original models SD1.5 and SDXL in terms of text alignment and aesthetics. Model…
Figure 4
Figure 4. Figure 4: Qualitative results under the safety alignment setting. We train GAN-RM using safe [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Additional quality results of SD1.5. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Additional quality results of SDXL. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Additional safety results of SD1.5. Our method improves the safety of the output while [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Additional safety results of SDXL. Index Layer Output size (1) CLIP embedded tokens 1 x 50 x 768 (2) CLS token of (1) 1 x 768 (3) Mean(CLIP embedded tokens, dim=1) 1 x 768 (4) Concatenate (2) and (3) 1 x 1536 (5) Linear (1536 → 2304) 1 x 2304 (6) ReLU 1 x 2304 (7) Batc…
Figure 9
Figure 9. Figure 9: Qualitative results of video generation for VC2. Left: original VC2 results; Right: [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: User study interface example. Each set contains two images generated for the same [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

58 extracted references · 6 canonical work pages

  1. [1]

    https://artlist.io/stock-footage/,

    Artgrid: Specializes in royalty-free digital content. https://artlist.io/stock-footage/,

    work page

  2. [2]

    Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    arXiv 2023

  3. [3]

    Videocrafter2: Overcoming data limitations for high-quality video diffusion models

    Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7310–7320, 2024

    2024

  4. [4]

    Directly fine-tuning diffusion models on differentiable rewards.arXiv preprint arXiv:2309.17400, 2023

    Kevin Clark, Paul Vicol, Kevin Swersky, and David J Fleet. Directly fine-tuning diffusion models on differentiable rewards.arXiv preprint arXiv:2309.17400, 2023

    arXiv 2023

  5. [5]

    Raft: Reward ranked finetuning for generative foundation model alignment.arXiv preprint arXiv:2304.06767, 2023

    Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment.arXiv preprint arXiv:2304.06767, 2023

    arXiv 2023

  6. [6]

    Raft: Reward ranked finetuning for generative foundation model alignment, 2023

    Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment, 2023

    2023

  7. [7]

    Unified autoregressive visual generation and under- standing with continuous tokens.arXiv preprint arXiv:2503.13436, 2025

    Lijie Fan, Luming Tang, Siyang Qin, Tianhong Li, Xuan Yang, Siyuan Qiao, Andreas Steiner, Chen Sun, Yuanzhen Li, Tao Zhu, et al. Unified autoregressive visual generation and under- standing with continuous tokens.arXiv preprint arXiv:2503.13436, 2025

    arXiv 2025

  8. [8]

    Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models.Advances in Neural Information Processing Systems, 36:79858–79885, 2023

    Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models.Advances in Neural Information Processing Systems, 36:79858–79885, 2023

    2023

Show all 58 references
  1. [9]

    An image is worth one word: Personalizing text-to-image generation using textual inversion.arXiv preprint arXiv:2208.01618, 2022

    Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion.arXiv preprint arXiv:2208.01618, 2022

    2022 arXiv

  2. [10]

    Generative adversarial networks.Communications of the ACM, 63(11):139–144, 2020

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks.Communications of the ACM, 63(11):139–144, 2020

    2020

  3. [11]

    Videoscore: Building automatic metrics to simulate fine-grained human feedback for video generation.arXiv preprint arXiv:2406.15252, 2024

    Xuan He, Dongfu Jiang, Ge Zhang, Max Ku, Achint Soni, Sherman Siu, Haonan Chen, Abhranil Chandra, Ziyan Jiang, Aaran Arulraj, et al. Videoscore: Building automatic metrics to simulate fine-grained human feedback for video generation.arXiv preprint arXiv:2406.15252, 2024. 10

    2024 arXiv

  4. [12]

    Llms meet multimodal generation and editing: A survey.arXiv preprint arXiv:2405.19334, 2024

    Yingqing He, Zhaoyang Liu, Jingye Chen, Zeyue Tian, Hongyu Liu, Xiaowei Chi, Runtao Liu, Ruibin Yuan, Yazhou Xing, Wenhai Wang, et al. Llms meet multimodal generation and editing: A survey.arXiv preprint arXiv:2405.19334, 2024

    2024 arXiv

  5. [13]

    Latent video diffusion models for high-fidelity long video generation.arXiv preprint arXiv:2211.13221, 2022

    Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation.arXiv preprint arXiv:2211.13221, 2022

    2022 arXiv

  6. [14]

    Clipscore: A reference-free evaluation metric for image captioning.arXiv preprint arXiv:2104.08718, 2021

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning.arXiv preprint arXiv:2104.08718, 2021

    2021 arXiv

  7. [15]

    Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  8. [16]

    Cascaded diffusion models for high fidelity image generation.Journal of Machine Learning Research, 23(47):1–33, 2022

    Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation.Journal of Machine Learning Research, 23(47):1–33, 2022

    2022

  9. [17]

    Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    2022 arXiv

  10. [18]

    Vbench: Comprehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. InProceedings of the IEEE/CVF Conference on Computer Vision and Patt...

    2024

  11. [19]

    Pick-a-pic: An open dataset of user preferences for text-to-image generation.Advances in Neural Information Processing Systems, 36:36652–36663, 2023

    Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation.Advances in Neural Information Processing Systems, 36:36652–36663, 2023

    2023

  12. [20]

    Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    2024 arXiv

  13. [21]

    Aligning text-to-image models using human feedback.arXiv preprint arXiv:2302.12192, 2023

    Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback.arXiv preprint arXiv:2302.12192, 2023

    2023 arXiv

  14. [22]

    Rich human feedback for text-to-image generation

    Youwei Liang, Junfeng He, Gang Li, Peizhao Li, Arseniy Klimovskiy, Nicholas Carolan, Jiao Sun, Jordi Pont-Tuset, Sarah Young, Feng Yang, et al. Rich human feedback for text-to-image generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition...

    2024

  15. [23]

    Step-aware preference optimization: Aligning preference with denoising performance at each step.arXiv preprint arXiv:2406.04314, 2(3), 2024

    Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang, Ji Li, and Liang Zheng. Step-aware preference optimization: Aligning preference with denoising performance at each step.arXiv preprint arXiv:2406.04314, 2(3), 2024

    2024 arXiv

  16. [24]

    Safetydpo: Scalable safety alignment for text-to-image generation, 2024

    Runtao Liu, Chen I Chieh, Jindong Gu, Jipeng Zhang, Renjie Pi, Qifeng Chen, Philip Torr, Ashkan Khakzar, and Fabio Pizzati. Safetydpo: Scalable safety alignment for text-to-image generation, 2024

    2024

  17. [25]

    Videodpo: Omni-preference alignment for video diffusion generation.arXiv preprint arXiv:2412.14167, 2024

    Runtao Liu, Haoyu Wu, Zheng Ziqiang, Chen Wei, Yingqing He, Renjie Pi, and Qifeng Chen. Videodpo: Omni-preference alignment for video diffusion generation.arXiv preprint arXiv:2412.14167, 2024

    2024 arXiv

  18. [26]

    Multimodal pragmatic jailbreak on text-to-image models, 2024

    Tong Liu, Zhixin Lai, Gengyuan Zhang, Philip Torr, Vera Demberg, V olker Tresp, and Jindong Gu. Multimodal pragmatic jailbreak on text-to-image models, 2024

    2024

  19. [27]

    Evalcrafter: Benchmarking and evaluating large video generation models

    Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and evaluating large video generation models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2024

  20. [28]

    Unitok: A unified tokenizer for visual generation and understanding.arXiv preprint arXiv:2502.20321, 2025

    Chuofan Ma, Yi Jiang, Junfeng Wu, Jihan Yang, Xin Yu, Zehuan Yuan, Bingyue Peng, and Xiaojuan Qi. Unitok: A unified tokenizer for visual generation and understanding.arXiv preprint arXiv:2502.20321, 2025. 11

    2025

  21. [29]

    Inference-time scaling for diffusion models beyond scaling denoising steps.arXiv preprint arXiv:2501.09732, 2025

    Nanye Ma, Shangyuan Tong, Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang, Yandong Li, Tommi Jaakkola, Xuhui Jia, et al. Inference-time scaling for diffusion models beyond scaling denoising steps.arXiv preprint arXiv:2501.09732, 2025

    2025 arXiv

  22. [30]

    I think, therefore i diffuse: Enabling multimodal in-context reasoning in diffusion models.arXiv preprint arXiv:2502.10458, 2025

    Zhenxing Mi, Kuan-Chieh Wang, Guocheng Qian, Hanrong Ye, Runtao Liu, Sergey Tulyakov, Kfir Aberman, and Dan Xu. I think, therefore i diffuse: Enabling multimodal in-context reasoning in diffusion models.arXiv preprint arXiv:2502.10458, 2025

    2025 arXiv

  23. [31]

    Webgpt: Browser-assisted question-answering with human feedback.arXiv preprint arXiv:2112.09332, 2021

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback.arXiv preprint arXiv:2112.09332, 2021

    2021 arXiv

  24. [32]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730...

    2022

  25. [33]

    Strengthening multimodal large language model with bootstrapped preference optimization

    Renjie Pi, Tianyang Han, Wei Xiong, Jipeng Zhang, Runtao Liu, Rui Pan, and Tong Zhang. Strengthening multimodal large language model with bootstrapped preference optimization. In European Conference on Computer Vision, pages 382–398. Springer, 2024

    2024

  26. [34]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023

    2023

  27. [35]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    2023

  28. [36]

    Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 1(2):3, 2022

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 1(2):3, 2022

    2022 arXiv

  29. [37]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, June 2022

    2022

  30. [38]

    Image super-resolution via iterative refinement.IEEE transactions on pattern analysis and machine intelligence, 45(4):4713–4726, 2022

    Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement.IEEE transactions on pattern analysis and machine intelligence, 45(4):4713–4726, 2022

    2022

  31. [39]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  32. [40]

    A general framework for inference-time scaling and steering of diffusion models.arXiv preprint arXiv:2501.06848, 2025

    Raghav Singhal, Zachary Horvitz, Ryan Teehan, Mengye Ren, Zhou Yu, Kathleen McKeown, and Rajesh Ranganath. A general framework for inference-time scaling and steering of diffusion models.arXiv preprint arXiv:2501.06848, 2025

    2025 arXiv

  33. [41]

    Deep unsuper- vised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. InInternational conference on machine learning, pages 2256–2265. pmlr, 2015

    2015

  34. [42]

    Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020

    2010 arXiv

  35. [43]

    Journeydb: A benchmark for generative image understanding, 2023

    Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, Jifeng Dai, Yu Qiao, Limin Wang, and Hongsheng Li. Journeydb: A benchmark for generative image understanding, 2023

    2023

  36. [44]

    Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024. 12

    2024 arXiv

  37. [45]

    Towards accurate generative models of video: A new metric and challenges, 2019

    Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric and challenges, 2019

    2019

  38. [46]

    Diffusion model alignment using direct preference optimization

    Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. InProceedings of the IEEE/CVF Conference on Computer Vision and ...

    2024

  39. [47]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    2025 arXiv

  40. [48]

    Vidprom: A million-scale real prompt-gallery dataset for text-to- video diffusion models.arXiv preprint arXiv:2403.06098, 2024

    Wenhao Wang and Yi Yang. Vidprom: A million-scale real prompt-gallery dataset for text-to- video diffusion models.arXiv preprint arXiv:2403.06098, 2024

    2024 arXiv

  41. [49]

    Internvid: A large-scale video-text dataset for multimodal understanding and generation.arXiv preprint arXiv:2307.06942, 2023

    Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation.arXiv preprint arXiv:2307.06942, 2023

    2023 arXiv

  42. [50]

    Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis.arXiv preprint arXiv:2306.09341, 2023

    Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis.arXiv preprint arXiv:2306.09341, 2023

    2023 arXiv

  43. [51]

    Visionreward: Fine-grained multi-dimensional human preference learning for image and video generation.arXiv preprint arXiv:2412.21059, 2024

    Jiazheng Xu, Yu Huang, Jiale Cheng, Yuanming Yang, Jiajun Xu, Yuan Wang, Wenbo Duan, Shen Yang, Qunlin Jin, Shurun Li, et al. Visionreward: Fine-grained multi-dimensional human preference learning for image and video generation.arXiv preprint arXiv:2412.21059, 2024

    2024 arXiv

  44. [52]

    Imagereward: Learning and evaluating human preferences for text-to-image generation

    Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:15903–15935, 2023

    2023

  45. [53]

    Using human feedback to fine-tune diffusion models without any reward model

    Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8941–8951, 2024

    2024

  46. [54]

    Denoising diffusion step-aware models.arXiv preprint arXiv:2310.03337, 2023

    Shuai Yang, Yukang Chen, Luozhou Wang, Shu Liu, and Yingcong Chen. Denoising diffusion step-aware models.arXiv preprint arXiv:2310.03337, 2023

    2023 arXiv

  47. [55]

    Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    2024 arXiv

  48. [56]

    The unreason- able effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as a perceptual metric. InCVPR, 2018

    2018

  49. [57]

    "fuck the police

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. A Limitations and Broader Impacts. Limitations.Despite its ef...

    1909 arXiv

  50. [2024]

    [Online; accessed 15-Aug-2024]

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.