Pith. sign in

REVIEW 3 major objections 6 minor 17 references

A frozen diffusion transformer's intermediate noisy latents already encode final-image quality; small learnable register tokens extract dense reward gradients that enable cheaper training-time alignment and inference-time guidance.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Prepending learnable register tokens to a frozen DiT yields dense latent reward estimates, enabling faster on-policy distillation and training-free guided sampling that improve preference alignment.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Novel register mechanism on a frozen DiT makes latent rewards cheap and dense; the core gap is that the register is validated on forward-noised latents, not the reverse trajectories it steers. the 3 major comments →

arxiv 2608.03929 v1 pith:35Z74LHN submitted 2026-08-04 cs.LG cs.CV

Latent Reward Registers for Diffusion Preference Alignment

Yuanshen Guan , Zipeng Feng , Zhiwei Xiong , Peiqin Sun This is my paper
classification cs.LG cs.CV
keywords latent reward modelsdiffusion preference alignmentregister tokensdense rewardreward-guided samplingflow matchingtext-to-image generationtemporal credit assignment
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that the temporal credit-assignment problem in diffusion preference alignment — the difficulty of assigning a single terminal quality score back to individual denoising steps — can be sidestepped rather than solved. The central claim is that a frozen Diffusion Transformer's internal representations already carry enough evidence about final-image quality that a small set of learnable, position-free register tokens can predict the terminal preference directly from any intermediate noisy latent, without touching the generator's hidden states or velocity field. From that dense, differentiable reward signal the authors derive two alignment strategies: a training-time method (RG-OPD) that distills one-step reward-tilted targets along on-policy trajectories and claims up to a 33x reduction in GPU hours relative to online RL baselines, and an inference-time method (RGS) that steers sampling with magnitude-matched reward gradients and reports improvements in both reward and perceptual metrics. If correct, alignment would no longer require backpropagating through full denoising chains or collecting large rollout batches, and preference objectives could be composed and swapped at test time without retraining. The load-bearing premise — that register accuracy transfers from forward-noised clean latents to the reverse-sampler states encountered in actual generation — is asserted but not directly validated.

Core claim

Central claim: a frozen diffusion transformer's internal representations already encode enough about final output quality that expected terminal preference can be read from intermediate noisy latents. Latent Reward Registers implement this as learnable, position-free tokens prepended to the frozen backbone's input; they accumulate evidence through a side-stream reusing frozen key/value projections without altering hidden states or the velocity field. Fused with pooled backbone features, reward-specific heads are trained by a noise-aware pairwise-ranking objective that preserves endpoint rankings. At u=0.8 the registers lead latent reward models; RG-OPD beats online RL baselines with 33x fewe

What carries the argument

Load-bearing object: the Latent Reward Register — K=32 learnable, position-free tokens prepended to a frozen DiT's input. Over the first L blocks they attend to native image/text tokens using the backbone's frozen key, value, output and noise-gating modules plus one trainable query projection, and skip the feed-forward networks, so the velocity field stays invariant. Reward-specific heads (HPS, ImageReward, PickScore) read fused register states, trained with a noise-aware pairwise-ranking loss in which paired images share noise level and the same Gaussian draw, preserving endpoint-reward rankings rather than calibrated values. Both alignment strategies consume the same artifact — the score g

Load-bearing premise

Everything rests on the assumption that a reward register trained on deliberately noised clean images stays accurate on the different kind of noisy states the generator actually visits while sampling — states that already carry the register's own corrections.

What would settle it

Run the frozen register on actual reverse-sampling states, with and without prior RGS corrections, at training noise levels; compare its pairwise preference accuracy and gradient-reward correlation against forward-noised pairs at the same noise level. If accuracy on sampling states approaches chance while forward-noised accuracy holds, the dense reward signal is a training-distribution artifact — and a magnitude-matched random-direction control would then show whether the reward direction itself (rather than the extra displacement) drives the RG-OPD and RGS gains.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Training-time alignment bypasses both full-chain backpropagation (which tends to over-optimize reward) and rollout-level policy gradients (which have high variance): RG-OPD's detached one-step targets reach Flow-GRPO-equivalent HPSv3 levels 14x-33x faster in GPU hours on both SD3-Medium and FLUX.1-dev.
  • Preference alignment can be applied at inference time without retraining: RGS improves target HPSv3 and ImageReward scores while holding MUSIQ and CLIP-IQA at or above the CFG baseline, and equal-RMS multi-head aggregation balances competing objectives.
  • Because the generator stays frozen and the reward correction is matched in magnitude to the solver step's own displacement, the same register works with higher-order ODE integrators (Adams-Bashforth, midpoint, Heun, RK4) without per-solver retuning.
  • The register is a standalone latent reward model: at u=0.8 it ranks above prior latent reward baselines (LRM, PA VRM, Diffusion Probe, DiNa-LRM) on most of four preference benchmarks, and its rank agreement with endpoint models stays stable (Spearman 0.76-0.83) across noise levels.
  • The high-noise band carries most of the signal, so the three-band guidance schedule (alpha 0.30 early, 0.05-0.10 mid, zero in the low-noise tail) gives users a direct trade-off between reward gain and fine-detail preservation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the register is trained on forward-noised clean latents that share a noise draw, yet RG-OPD and RGS apply it to reverse-sampler states carrying accumulated corrections; a cheap check is to measure register rank accuracy and gradient-reward correlation on buffers of actual sampling states per noise level, with and without prior corrections.
  • Editorial extension: if early latents are this informative about terminal preference, the same readout could drive compute allocation decisions - adaptively stopping or refining trajectories whose predicted reward has saturated - a use the paper does not explore.
  • Editorial extension: because the mechanism is agnostic to what the endpoint reward measures, a natural test is training the same register architecture on an objective verifier (factual consistency, structural constraints) and checking whether dense latent rewards accelerate convergence there as they do for human preferences.
  • Editorial extension: the paper isolates whether direction matters only against a CFG-direction control; a stronger control would replace the reward gradient with a magnitude-matched random or perturbed unit vector, testing whether the guidance signal itself - not just the extra displacement - drives the gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Latent Reward Registers, a side-stream of learnable, position-free register tokens inserted into a frozen Diffusion Transformer (DiT). The registers are trained with a pairwise ranking objective to predict terminal image-preference scores directly from intermediate noisy latents, yielding a dense, differentiable reward field. This field is then used in two alignment strategies: RG-OPD, which distills reward-tilted one-step targets along on-policy student trajectories, and RGS, which applies magnitude-matched reward-gradient corrections at inference time on a frozen generator. Experiments report pairwise preference accuracy at noise level u=0.8 on four benchmarks, training-time alignment results on SD3-Medium and FLUX.1-dev versus reward-backpropagation and online RL baselines, and inference-time results versus DNO and Demon, together with ablations, efficiency analyses, and a multi-solver ODE study.

Significance. If the central claim holds—that a frozen DiT's intermediate representations already contain enough terminal-preference evidence for a lightweight register to yield reliable dense reward gradients—this would be a valuable contribution to diffusion-model alignment. The paper targets a real credit-assignment problem, introduces a non-intrusive mechanism that preserves the generative dynamics, and demonstrates both training-time and inference-time applications. It also ships code, gives detailed cost measurements (Table 10), and includes a high-order-ODE-solver robustness analysis (Table 11), all of which are strengths. However, the load-bearing empirical support is weakened by a train/deploy distribution shift, a partially circular evaluation loop, and the absence of uncertainty quantification. The core claim is plausible and important, but the current evidence does not fully establish it.

major comments (3)
  1. [Sec. 3.1, Eq. (4), Eq. (7)] The register is trained and validated on paired clean latents forward-noised as z_u=(1-u)z_clean+uε with a shared Gaussian realization (Sec. 3.1). RG-OPD and RGS, however, evaluate and differentiate the register score at states produced by the student or guided reverse sampler: finite-step Euler/CFG integration plus accumulated reward corrections (Eq. 4, Eq. 7). These states are not draws from the forward-corruption marginal used in training. The paper never directly validates register ranking accuracy or gradient reliability on this deployment distribution. Appendix E.4 / Figure 11 tracks register-head z-scores along the guided trajectory, but that is self-evaluation by the very function whose reliability is in question; it cannot establish that a high register score at z_i corresponds to high terminal preference. Please add a direct validation on actual deployment states—for example, c
  2. [Tables 1–3 and Fig. 6] The central empirical claims—register achieves the highest latent-model accuracy, RG-OPD outperforms online RL, RGS 'strictly enhances both alignment and perceptual metrics'—are supported only by point estimates. Differences such as Table 1 average accuracy 68.32 (register) versus 66.47 (DiNa-LRM), or Table 3 MUSIQ 73.2 versus 73.0, could be within evaluation noise. Without confidence intervals, hypothesis tests, or per-seed/per-prompt breakdowns, these claims are not fully supported. Moreover, 'strictly enhancing both alignment and perceptual metrics' is contradicted by the RGS+IR row in Table 3 on SD3-Medium, where MUSIQ decreases (73.2→73.0) and CIQA is unchanged (0.694). Please report uncertainty and qualify such claims accordingly.
  3. [Sec. 3.1, Sec. 4.3, Table 3] The evaluation loop is partly closed. The register heads are trained to reproduce the rankings of endpoint reward models (HPSv3, ImageReward, and, in the design ablation, PickScore), and the main RGS/OPD results are then evaluated largely on those same endpoint reward models. Gains on HPSv3/ImageReward therefore conflate true preference improvement with optimization of a differentiable surrogate trained by the same teacher. The no-reference quality metrics (MUSIQ, CLIP-IQA) are a useful independent check, but they are not preference metrics. The paper should evaluate RG-OPD and RGS on held-out preference benchmarks that were not used as optimization objectives—for example HPDv2 or GenAI-Bench—and, if feasible, on a small human-study subset.
minor comments (6)
  1. [Abstract and Sec. 4.3] The phrase 'strictly enhancing both alignment and perceptual metrics' is too strong given Table 3; see major comment. Suggest rewording to 'improving target rewards while preserving or improving perceptual quality in most configurations.'
  2. [Sec. 4.4, Fig. 6] The efficiency claim '14.0×–33.2× faster' is based on a 256-prompt validation subset, while Table 2 uses 400/800 prompt–seed sets. Please clarify that these are different evaluation sets in the main text, not only in the analysis paragraph.
  3. [Appendix E.4, Fig. 11] The legend uses λ for guidance strength while the method section uses α; this notation inconsistency should be fixed.
  4. [Table 11] The FID values are reported relative to a reference set (Euler CFG for CFG rows, same-solver CFG for RGS rows). This is unusual and potentially misleading; the definition should be stated explicitly in the main text or table caption.
  5. [Sec. 4.1, Table 1] LRM-SD1.5 and LRM-SDXL use different backbones from the SD3-Medium latent baselines. The text says 'fair comparison' but should more explicitly state that these rows are cross-backbone references, not matched comparisons.
  6. [Figure 2] The overview figure is extremely dense and difficult to parse. Consider simplifying or splitting into separate panels for the architecture and the attention/module details.

Circularity Check

2 steps flagged

The register is trained on endpoint reward models and then evaluated on those same models; Appendix E.4's reward-progression plot measures the very objective RGS optimizes.

specific steps
  1. fitted input called prediction [Sec. 3.1 (Reward prediction and learning); Tables 2 and 3]
    "The learned score preserves the rankings induced by the endpoint reward models rather than estimating a calibrated reward value."

    The register is trained to reproduce the rankings of endpoint reward models (HPSv3, ImageReward) via the pairwise ranking objective. RG-OPD (Eq. 4) and RGS (Eq. 7) then use gradients of this learned score to maximize those same endpoint reward models, and Tables 2 and 3 evaluate on HPSv3 and ImageReward as 'optimization targets.' Thus a large part of the reported alignment gain is a closed loop: the surrogate is optimized with gradients from its own teacher, and the metric is the teacher itself. The gain is partially forced by construction, though not entirely, because the gradient of a ranking-trained surrogate need not perfectly align with the teacher's gradient off-distribution.

  2. self definitional [Appendix E.4, Figure 11]
    "Figure 11 tracks the register-head scores for HPSv3, PickScore, and ImageReward through the denoising process. The correction raises the scores by 2–3.3 standard deviations relative to the CFG baseline mean."

    RGS updates latents by Eq. 7: z_{i+1} = mu_ref_i + alpha_i * RMS(delta z_ref_i) * unitRMS(g_i), where g_i = grad_z r_phi(z_i) is the gradient of the register score. Therefore the register score is the exact objective being greedily increased at each step; its increase is a first-order Taylor consequence of the update rule, not independent evidence of terminal-preference improvement. The figure is a self-evaluation by the very function whose reliability is in question.

full rationale

The paper's central mechanism—a frozen-DiT register readout trained with a pairwise ranking loss against endpoint reward models, then used as a dense reward for training-time distillation and inference-time guidance—is not circular in its construction. The register's pairwise accuracy is also evaluated on human-preference benchmarks (HPDv2, HPDv3, GenAI-Bench) that are not the training teachers, and the CFG-direction control (Appendix E.1) isolates the learned gradient direction from mere magnitude effects. However, two evaluation loops are partially closed. First, RG-OPD and RGS are evaluated largely on HPSv3 and ImageReward, which are the same endpoint reward models used to supervise the register; improvement on those metrics is partly a fit-transfer sanity check rather than an independent demonstration of preference alignment. Second, Appendix E.4's 'reward along the trajectory' analysis uses the register's own score as evidence of alignment, but that score is the objective being optimized by Eq. 7, so its increase is expected by construction. These issues do not invalidate the central claim—there is independent human-preference and perceptual-quality evidence—but they indicate partial circularity in the evaluation chain, warranting a score of 4.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 1 invented entities

The central mechanism rests on empirical premises rather than derivation. The main free parameters are architectural choices and the reward-correction strength schedules. The register is a learned surrogate of endpoint reward scores, so its validity depends on the quality of those teachers and on distributional overlap between forward-noised training states and reverse-sampling states.

free parameters (6)
  • Number of register tokens K = 32
    Hand-chosen; ablations in Table 4 show component-level changes affect accuracy, but K itself is not swept.
  • Register-equipped blocks L = 12 (SD3), 8 (FLUX)
    Chosen per backbone; the authors argue persistent aggregation matters, but no selection criterion is given.
  • RG-OPD reward tilt schedule alpha_i = SD3: {0.20,0.40,0.50} for sigma>0.2, 0 otherwise; FLUX: 0.80
    Controls the magnitude of the reward correction in the distillation target; differs by backbone and was tuned.
  • RGS guidance schedule alpha_early/alpha_mid = 0.30 early, 0.05 (SD3) or 0.10 (FLUX) mid, 0 low-noise
    Three-band schedule chosen post hoc to avoid distorting fine details; performance depends on these values.
  • Feature-snapshot blocks for fusion = {4,8,12} (SD3), {9,19,28} (FLUX)
    Selected for the fusion readout; the ablation shows layers[4,8] and layers[4] have small effect, indicating this choice is not heavily load-bearing.
  • Sampling steps and CFG scale = 42 steps / CFG 4.5 (SD3), 40 steps / CFG 3.5 (FLUX)
    Fixed evaluation configuration; all RGS and baseline numbers are measured under these schedules.
axioms (5)
  • domain assumption Frozen DiT representations encode enough terminal-preference evidence from intermediate noisy latents
    Stated as the central insight in Sec. 1; supported only by the paper's own noisy-accuracy benchmarks, not by an external derivation.
  • domain assumption Endpoint reward models HPSv3 and ImageReward provide valid preference labels for training the register
    The register is trained to reproduce endpoint-model rankings in Sec. 3.1; if the endpoint labels are biased, the register inherits the bias.
  • domain assumption Register gradients applied to reverse-sampling latents point toward higher-quality images and remain near the natural image manifold
    Both RGS (Eq. 7) and RG-OPD (Eq. 4) rely on this; the paper validates end-of-trajectory metrics but not gradient correctness on reverse-trajectory states.
  • domain assumption Paired forward-noising with shared Gaussian noise and noise level controls pairwise confounds
    Assumed in the Thurstone pairwise training objective in Sec. 3.1; if not, the reported accuracy overstates true ranking ability.
  • standard math Standard flow-matching ODE discretization is faithful for the frozen generator
    Relies on FlowMatch Euler and higher-order integrators from Lipman et al. and Hairer et al.; standard but not re-derived here.
invented entities (1)
  • Latent reward register tokens independent evidence
    purpose: Persistent global readouts appended to frozen DiT blocks to aggregate terminal-preference evidence from noisy latents.
    The released weights and the external preference benchmarks give a testable handle, but this is a trained network component, not a physical entity with an independent falsifiable prediction.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Latent Reward Registers for Diffusion Preference Alignment." pith.science (2026). https://pith.science/paper/35Z74LHN

@misc{pith2026260803929,
  author       = {Pith},
  title        = {Pith review of: Latent Reward Registers for Diffusion Preference Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/35Z74LHN}},
  note         = {Machine review of arXiv:2608.03929}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Aligning diffusion models with human preferences usually relies on a sparse terminal reward evaluated on the final generated samples, presenting a severe temporal credit-assignment challenge across the multi-step denoising process. We propose Latent Reward Registers, a mechanism that estimates terminal preference directly from intermediate noisy latents by prepending learnable, position-free register tokens to the input sequence of a frozen Diffusion Transformer (DiT). This independent readout mechanism extracts latent reward evidence without altering the generator's hidden states or velocity field. The resulting dense, differentiable reward signal throughout the full denoising process facilitates two alignment strategies. For training, Reward-Gradient On-Policy Distillation (RG-OPD) distills reward-guided updates along on-policy trajectories, bypassing the computationally expensive rollouts of standard policy gradients. For inference, Reward-Guided Sampling (RGS) steers trajectories via magnitude-matched reward gradients without parameter updates. Empirically, at high noise levels (u = 0.8), the registers reach the highest pairwise accuracy among the evaluated latent reward models. Furthermore, RG-OPD outperforms online reinforcement learning baselines while reducing GPU hours by up to 33x, and RGS establishes a new state-of-the-art among training-free methods, strictly enhancing both alignment and perceptual metrics. Code and weights are available at https://github.com/Guanys-dar/latent-reward-register

Figures

Figures reproduced from arXiv: 2608.03929 by Peiqin Sun, Yuanshen Guan, Zhiwei Xiong, Zipeng Feng.

Figure 1
Figure 1. Figure 1: Latent Reward Registers vs. existing paradigms. Unlike conventional alignment meth￾ods that rely on delayed, sparse feedback from a clean image (Left) or require intrusive fine-tuning of the generator (Middle), our approach (Right) appends learnable registers to a frozen DiT. This en￾ables the direct estimation of step-wise rewards from intermediate noisy latents (zt). Consequently, we derive dense, differ… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the Reward Register mechanism. Learnable registers establish a non￾intrusive auxiliary stream through a frozen DiT. These registers derive representations from frozen image and text features without altering the original token sequence, thereby strictly preserving the original velocity field. A lightweight fusion module then combines the final register states with in￾termediate backbone represe… view at source ↗
Figure 3
Figure 3. Figure 3: Preference accuracy across noise levels. Pairwise preference accuracy of latent reward models scored at noise level u [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative comparison of training-time alignment. We compare the visual results of our RG-OPD against baseline methods, including CFG, ReFL, AlignProp, Flow-GRPO, and Diffusion-NFT. The generated samples demonstrate that our approach achieves better prompt align￾ment and higher perceptual quality without introducing visible artifacts. OPD preserves prompt-relevant content and perceptual quality without th… view at source ↗
Figure 5
Figure 5. Figure 5: Comparison of training-free sampling methods. CFG, DNO, Demon, and our single￾and multi-head RGS variants use matched prompts and seeds within each backbone. 4.4 ANALYSIS Register design. The persistent register state is the main driver of accuracy. Removing the register causes the largest performance drop, while the skip-FFN design and the trainable query projection each add complementary gains ( [PITH_F… view at source ↗
Figure 6
Figure 6. Figure 6: Training efficiency of RG-OPD. We compare the HPSv3 optimization trajectories of RG-OPD and Flow-GRPO against aggregate GPU hours on SD3-Medium and FLUX.1-dev. The curves indicate that RG-OPD requires fewer GPU hours to reach equivalent reward thresholds. This acceleration comes from the dense, step-wise gradients extracted from the latent reward registers. benchmarks in [PITH_FULL_IMAGE:figures/full_fig_… view at source ↗
Figure 7
Figure 7. Figure 7: Effect of reward tilt (rt) during RG-OPD training. Training dynamics are compared between rt=0.20 and rt=0.40. The results show that a larger reward scale accelerates convergence and yields a higher peak performance. Moreover, the upward trend of the internal latent reward (a) remains consistent with the endpoint HPSv2 evaluation (b), indicating alignment between the intermediate proxy and the terminal obj… view at source ↗
Figure 8
Figure 8. Figure 8: Multi-head RGS. Multi-head reward gradients achieve a balanced performance across different reward metrics, compared to single-head RGS under identical settings. isters, a non-intrusive auxiliary stream that predicts endpoint rewards directly from intermediate noisy latents. The resulting dense reward field supports two alignment strategies. Reward-Gradient On-Policy Distillation (RG-OPD) utilizes step-wis… view at source ↗
Figure 9
Figure 9. Figure 9: Rank agreement between register heads and endpoint reward models. The figure displays the joint probability density of ranking percentiles derived from register-head evaluations on noised latents, contrasted with corresponding percentiles obtained from endpoint models on fully decoded images. The parameter ρ denotes the Spearman rank correlation coefficient, and the diago￾nal line represents theoretically … view at source ↗
Figure 10
Figure 10. Figure 10: Extended qualitative comparison of training-time alignment. We provide additional visual examples comparing our RG-OPD framework against baseline methods across a diverse set of text prompts. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Reward progression during generation under uniform guidance. Register-head eval￾uations are plotted as z-scores, normalized against the per-step distribution of the unguided CFG baseline. Curves denote different guidance strengths α (labeled as λ in the legend), with shaded regions indicating variance across diverse text prompts. E.3 RGS ACROSS SAMPLING BUDGETS AND STRENGTHS [PITH_FULL_IMAGE:figures/full… view at source ↗
Figure 12
Figure 12. Figure 12: Effect of sampling budgets and guidance strengths. Panels evaluate distinct metrics against the total number of generation steps. We compare the standard CFG baseline with RGS at various uniform guidance scales α (labeled as λ). Target rewards scale positively with guidance in￾tensity, while no-reference perceptual quality consistently outperforms the CFG baseline, plateauing at higher guidance levels. E.… view at source ↗
Figure 13
Figure 13. Figure 13: Qualitative impact of guidance strength variation. This progression illustrates gen￾erated samples subject to monotonically increasing guidance strengths (α), synthesized using an identical text prompt and initial latent seed. The visual sequence demonstrates the progressive in￾stantiation and refinement of preference-aligned details in correlation with the applied guidance magnitude. register mechanism w… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

17 extracted references · 4 canonical work pages · 1 internal anchor

  1. [1]

    Training diffusion models with reinforcement learning.arXiv preprint arXiv:2305.13301,

    Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning.arXiv preprint arXiv:2305.13301,

    Pith/arXiv arXiv

  2. [6]

    Pick-a-pic: An open dataset of user preferences for text-to-image generation.arXiv preprint arXiv:2305.01569,

    Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation.arXiv preprint arXiv:2305.01569,

    Pith/arXiv arXiv

  3. [7]

    Aligning text-to-image models using human feedback.arXiv preprint arXiv:2302.12192,

    Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback.arXiv preprint arXiv:2302.12192,

    Pith/arXiv arXiv

  4. [8]

    GenAI-Bench: Evaluating and improv- ing compositional text-to-visual generation.arXiv preprint arXiv:2406.13743,

    Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Tiffany Ling, Xide Xia, Pengchuan Zhang, Graham Neubig, and Deva Ramanan. GenAI-Bench: Evaluating and improv- ing compositional text-to-visual generation.arXiv preprint arXiv:2406.13743,

    Pith/arXiv arXiv

  5. [9]

    Beyond VLM-based rewards: Diffusion-native latent reward modeling.arXiv preprint arXiv:2602.11146,

    Gongye Liu, Bo Yang, Yida Zhi, Zhizhou Zhong, Lei Ke, Didan Deng, Han Gao, Yongxiang Huang, Kaihao Zhang, Hongbo Fu, and Wenhan Luo. Beyond VLM-based rewards: Diffusion-native latent reward modeling.arXiv preprint arXiv:2602.11146,

    Pith/arXiv arXiv

  6. [11]

    Video generation models are good latent reward models.arXiv preprint arXiv:2511.21541,

    Xiaoyue Mi, Wenqing Yu, Jiesong Lian, Shibo Jie, Ruizhe Zhong, Zijun Liu, Guozhen Zhang, Zixiang Zhou, Zhiyong Xu, Yuan Zhou, Qinglin Lu, and Fan Tang. Video generation models are good latent reward models.arXiv preprint arXiv:2511.21541,

    arXiv

  7. [12]

    Human preference score v2: A solid benchmark for evaluating human preferences of text-to- image synthesis.arXiv preprint arXiv:2306.09341,

    Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to- image synthesis.arXiv preprint arXiv:2306.09341,

    Pith/arXiv arXiv

  8. [14]

    DanceGRPO: Unleashing GRPO on visual generation

    Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, et al. DanceGRPO: Unleashing GRPO on visual generation. arXiv preprint arXiv:2505.07818,

    Pith/arXiv arXiv

  9. [15]

    A dense reward view on aligning text-to-image diffusion with preference.arXiv preprint arXiv:2402.08265,

    Shentao Yang, Tianqi Chen, and Mingyuan Zhou. A dense reward view on aligning text-to-image diffusion with preference.arXiv preprint arXiv:2402.08265,

    Pith/arXiv arXiv

  10. [16]

    top-k bottom-k =0.82 HPS ideal ranking top-k bottom-k =0.80 ImageReward top-k bottom-k =0.78 PickScore 0.00 0.25 0.50 0.75 1.00 u = 0.5 Register Ranking Pct

    Variant ImageReward HPDv2 HPDv3 GenAI-Bench Avg +HPS62.12 81.1873.3061.75 69.59 +ImageReward63.13 78.84 65.8663.4267.81 +Raw score-sum63.46 81.2072.36 62.9670.00 0.00 0.25 0.50 0.75 1.00 u = 0.2 Register Ranking Pct. top-k bottom-k =0.82 HPS ideal ranking top-k bottom-k =0.80 ImageReward top-k bottom-k =0.78 PickScore 0.00 0.25 0.50 0.75 1.00 u = 0.5 Regi...

    work page arXiv 2072

  11. [17]

    Each method runs in its own process and is averaged over ten runs after ten warm-up runs. In the standard deployment the register shares the frozen denoiser and text encoders of the sampler, so the memory overhead is limited to the147.6M trainable param- eters of the read path and reward head: peak memory rises only from16.9GiB to17.6GiB versus unguided C...

    work page 2023

  12. [1993]

    Jonathan Ho and Tim Salimans

    doi: 10.1007/978-3-540-78862-1. Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598,

    Pith/arXiv arXiv

  13. [2021]

    Stitched Value Model for Diffusion Alignment

    Hyojun Go, Hyungjin Chung, Prune Truong, Goutam Bhat, Li Mi, Zhaochong An, Zixiang Zhao, Dominik Narnhofer, Serge Belongie, Federico Tombari, and Konrad Schindler. Stitched value model for diffusion alignment.arXiv preprint arXiv:2605.19804,

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [2023]

    Directly fine-tuning diffusion models on differentiable rewards

    Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. InInternational Conference on Learning Representations, volume 2024, pp. 4793–4822,

    work page 2024

  15. [2024]

    Imagereward: Learning and evaluating human preferences for text-to-image generation

    Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. arXiv preprint arXiv:2304.05977,

    Pith/arXiv arXiv

  16. [2025]

    HPSv3: Towards wide-spectrum human preference score.arXiv preprint arXiv:2508.03789,

    Yuhang Ma, Xiaoshi Wu, Keqiang Sun, and Hongsheng Li. HPSv3: Towards wide-spectrum human preference score.arXiv preprint arXiv:2508.03789,

    Pith/arXiv arXiv

  17. [2026]

    Timoth´ee Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski

    URLhttps://arxiv.org/abs/2602.23783. Timoth´ee Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. InInternational Conference on Learning Representations,

    arXiv

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.