Pith. sign in

REVIEW 4 major objections 8 minor 2 cited by

Rethinking Reward Signals in Video GRPO: When Scores Become Targets

T0 review · 4 major / 8 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GRPO for video generation should use rewards that co-evolve with the generator; fixed rewards saturate and encourage hacking.

desk verdict The abstract and body describe different papers, and the saturation signal in Eq. (8)-(9) is mathematically inverted, so the central mechanism likely does the opposite of what is claimed; the empirical gains are small and unvalidated. read the letter →

arxiv 2511.19356 v4 pith:D75O2IKG submitted 2025-11-24 cs.CV

Rui Li , Yuanzhi Liang , Ziqi Ni , Haibin Huang , Chi Zhang , Xuelong Li This is my paper
classification cs.CV
keywords GRPOvideogenerationrewardhackingself-pacedlearningsaturationVBenchtext-to-videoalignmentreinforcement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Self-Paced GRPO is a modification of Group Relative Policy Optimization for video generation in which the reward signal itself changes during training. Instead of a fixed reward model that saturates as the generator improves, the method continuously reweights a mixture of reward terms — visual quality, temporal consistency, and text alignment — based on group-level statistics. A high group mean and high Hoyer sparsity indicate that a reward component has become saturated, so its weight is reduced and training emphasis shifts to more demanding criteria. The authors report that this co-evolving reward curriculum improves VBench scores across multiple video backbones and reduces reward hacking. If correct, it establishes that reward adaptation is a viable alternative to static reward shaping in reinforcement learning for generative models.

What carries the argument

The Co-Evolving Reward Mechanism (CERM): a soft, competence-dependent weighting function over a set of reward terms, where the weight of term j is exp(α g_j(c, r_j)) normalized over terms. The contribution function g_j combines a sigmoid of the group-mean reward relative to a threshold (i.e., h(mean_rj - tau_j)) with a term comparing Hoyer sparsity at successive steps (β[S_Hoyer(r_{j-1}) - S_Hoyer(r_j)]). High mean and high sparsity are interpreted as saturation; CERM then downweights that term and shifts supervision to finer-grained criteria.

What would settle it

Apply the saturation detector to prompt groups where human preference scores are available. If groups flagged as saturated show no divergence between rising proxy reward and flat or falling human preference, the detector is not identifying reward hacking, and the mechanism's justification collapses.

Watch

Extended reading notes

Core claim

The paper's central claim is that the reward model in GRPO-based video generation should not be treated as a fixed evaluator. It identifies two failure modes of static rewards: shortcut-driven optimization under composite objectives and reward saturation within prompt groups. To counter these, it introduces the Co-Evolving Reward Mechanism (CERM), which computes a competence-weighted mixture of reward terms. Competence is inferred from the mean reward and the Hoyer sparsity of each term's scores across a group; high values trigger a gradual shift of weight toward more advanced reward terms. The authors instantiate this with three reward stages and show that the resulting Self-Paced GRPO yiel

Load-bearing premise

The load-bearing premise is that high group-mean reward plus high Hoyer sparsity marks a reward component as saturated and hackable; if sparsity instead reflects a prompt that genuinely cannot discriminate quality, the curriculum will downweight useful rewards.

Editorial extensions

If this is right

  • Static-reward GRPO will plateau or regress once the generator surpasses the reward model's discriminative capacity; adaptive rewards prolong effective training.
  • The three-stage reward ordering — visual quality, temporal smoothness, text alignment — provides a natural curriculum that yields balanced improvements, whereas joint training of all three rewards conflicts.
  • The Hoyer-sparsity signal offers a cheap, training-free way to detect reward saturation within prompt groups.
  • The method transfers across model scales (1.3B, 14B) and backbones, suggesting the reward-adaptation principle is general rather than model-specific.
  • Downweighting saturated reward components reduces the incentive for the generator to overfit to a single reward model, mitigating reward hacking.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same competence-aware reweighting could be applied to other generative RL settings (images, audio, long-form video), where static reward models also saturate.
  • A sharper detector of reward hacking — for instance, divergence between proxy reward and an independent human-preference judge — could replace the Hoyer sparsity heuristic and likely yield more principled curricula.
  • The threshold parameters tau_j are currently calibrated from separate pre-training runs; an online, self-tuning version would remove this manual step and is a testable extension.
  • Because the method only reweights precomputed reward terms, it can be combined with any GRPO implementation, making it a plug-in rather than a wholesale retraining scheme.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper proposes Self-Paced GRPO (also referred to as TaRoS in the abstract) for post-training video generation models with group-relative policy optimization. The method introduces a competence-aware reweighting of three reward aspects — visual quality, temporal smoothness, and text alignment — based on group-level reward statistics (mean and Hoyer sparsity). The authors claim this co-evolving reward mechanism alleviates reward saturation, reduces category bias, and yields consistent improvements on VBench over static-reward GRPO baselines across Wan2.1-1.3B, Wan2.1-14B, and HunyuanVideo. Additional ablations compare one-stage, two-stage, and joint-training configurations, and a bias analysis compares the coefficient of variation of VideoAlign and Qwen2.5-VL reward scores across semantic clusters.

Significance. The problem addressed is timely and practically relevant: static reward models in RL fine-tuning of video diffusion models can saturate and be exploited, and adapting reward supervision to generator competence is an attractive idea. If the proposed adaptive curriculum were mathematically sound and convincingly validated, it could be a useful contribution to the growing literature on GRPO for generative video. The paper includes multi-backbone experiments, a joint-training ablation, and qualitative comparisons. However, the main control signal is mis-specified in the paper, and the experimental evidence is not sufficiently controlled or statistically grounded. As presented, the central claim cannot be accepted.

major comments (4)
  1. [Sec. 4.2, Eqs. (8)-(9)] The Hoyer sparsity index is misread in the text. For positive reward vectors in [1,10], Hoyer = (sqrt(G) - ||r||1/||r||2)/(sqrt(G)-1). When all scores are equal, ||r||1/||r||2 = sqrt(G) and Hoyer = 0; when scores are heterogeneous, Hoyer > 0. Thus high sparsity indicates discriminating, not saturated, scores. The text claims the opposite. As a result, g_j = h(mean - tau_j) + beta[S(r_{j-1}) - S(r_j)] routes weight toward low-sparsity (saturated) terms and away from high-sparsity (informative) terms — the opposite of the stated intent. Since this control signal is the mechanism on which the central claim rests, this is a load-bearing error, not a typo. Correcting the sign or changing the definition of sparsity would require re-running the experiments.
  2. [Sec. 5.3, Table 1] The headline improvements are small point estimates without error bars or significance tests. On Wan2.1-1.3B, the total score goes from 79.58 (Wan base) to 80.22 (Ours); on the 14B model the gain is 81.46 to 82.09. These differences are within what could be run-to-run noise for RL fine-tuning of diffusion models. Moreover, the main baseline DanceGRPO uses a different reward model (VideoAlign) than the proposed method (Qwen2.5-VL), so the comparison conflates algorithm with reward-model choice. The joint-training ablation in Table 3 partly controls for reward model, but it reports different metrics (VideoAlign VQ/MQ/TA, LAION) and still lacks variance or repeated runs. The claim of 'consistent improvements across backbones' is not statistically supported.
  3. [Sec. 5.3, Figs. 4-5; Supp. B] The bias analysis is circular. The paper uses Qwen2.5-VL-72B to score real videos and then interprets its low cross-cluster coefficient of variation as evidence that this reward model is unbiased. A low coefficient of variation could equally result from a compressed scoring scale or from consistent but systematically incorrect scores across all categories. No human-rated quality labels or independent, validated evaluator are used as ground truth. The finding that VideoAlign has higher CV than Qwen2.5-VL shows only that the two models disagree in their score dispersion; it does not establish which model is more accurate or less biased.
  4. [Sec. 4.3, Sec. 5.2, Supp. A] The 'co-evolving' or 'self-paced' component is significantly weaker than claimed. The stage thresholds tau_j are set to 0.7 times the reward improvements observed from separate stage-only training runs, and the three stages are fixed prompt templates. This is a pre-scheduled curriculum with hand-set hyperparameters (tau, alpha, beta, 0.7 multiplier), not an online self-paced adaptation. No sensitivity analysis is provided for these free parameters, so it is unclear how robust the reported results are to their choice.
minor comments (8)
  1. [Title/Abstract] The abstract introduces 'TaRoS, a Target-Robust Reward Signaling framework', but the title and body describe 'Self-Paced GRPO'. The names and framing should be reconciled; as written the paper appears to be a mix of two drafts.
  2. [Eq. (3)] The cluster-level coefficient of variation defined in Section 3 is not used in the method or experiments (Figs. 4-5 report a different procedure). Either remove it from the preliminaries or clarify its role.
  3. [Eq. (5)] The competence c is mentioned in the text around Eq. (5) but does not appear in the equation. The definition of c and how it is computed from reward statistics should be stated explicitly.
  4. [Table 1] 'pretrainde Wan14B' is a typo for 'pretrained Wan14B'. Also, the 14B baseline is not described clearly; state whether it is the original Wan2.1-14B without any RL post-training.
  5. [Table 3] The table formatting for one-stage and two-stage rows is distorted ('One stage3.4810.3096'). Please fix the spacing and notation.
  6. [References] Reference [23] is cited as 'VideoAlign', but the entry is for 'LIV: Language-image representations and rewards for robotic control'. The actual VideoAlign paper appears to be missing from the bibliography. Please correct the citation.
  7. [Supp. A] The supplementary text mentions 'the VideoGen-RewardBench benchmark' without providing a reference or citation. This needs to be added.
  8. [Figures 11-13] The qualitative comparisons are presented without any selection protocol. State how the examples were chosen to avoid the appearance of cherry-picking.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central VBench results are externally benchmarked; flagged issues are self-evaluative evidence or correctness concerns, not by-construction reductions.

full rationale

The core effectiveness claim is supported by VBench, an external benchmark whose metrics are not used to fit the reward weights or thresholds. The adaptive weights w_j in Eqs. (5)-(6) and (9) are computed from the same reward statistics used in the objective; this is feedback control, not a tautological prediction. The bias analysis in Sec. 5.3/Fig. 4 is self-evaluative—Qwen2.5-VL-72B scores are used to assert Qwen's content-invariance via low cross-category coefficient of variation—so it is weak independent evidence, but it is not a derivation that reduces to its input by construction. The Hoyer index argument in Sec. 4.2 (Eq. 8) has an internal sign inconsistency: for positive scores, high Hoyer means heterogeneous scores, not saturation, which is a correctness risk for the claimed control signal, but it is not circularity. Threshold calibration in the supplement (0.7x observed reward increase from pilot runs) is hyperparameter fitting, not a predicted result. No load-bearing self-citations or imported uniqueness theorems are present. The self-referential reward-curve statement in the supplement (reward increase 'indicating effectiveness') is not load-bearing because the main claims are externally evaluated. Overall, the derivation chain does not equate its conclusions to its inputs.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

The method depends on several domain assumptions about reward-model behavior and uses fitted thresholds and unreported weighting hyperparameters. No new physical entities are introduced beyond the latent competence variable.

free parameters (3)
  • Stage thresholds τ_I, τ_II = 0.75/0.75 (Wan1.3B); 0.75/0.73 (Wan14B); 0.70/0.68 (Hunyuan)
    Calibrated empirically: supplement says each stage was trained for 50 steps and thresholds set to 0.7× the observed reward increase.
  • Softmax sharpness α and sparsity weight β = not reported
    Eq. (6) and Eq. (9); control how quickly components are reweighted and how strongly sparsity modulates the gate, but no values are given.
  • 0.7 threshold multiplier = 0.7
    Ad hoc scaling factor in threshold calibration from supplementary material.
assumptions (5)
  • domain assumption Reward scores from Qwen2.5-VL are proxies for true video quality and human preference
    Underlies the entire optimization; asserted but not validated against human ratings.
  • domain assumption Qwen2.5-VL-72B is content-invariant (low bias)
    Supported only by coefficient-of-variation analysis in Figs. 4-5 using the same model's scores; no external ground-truth quality labels.
  • domain assumption Increasing prompt complexity in stages yields harder, more informative supervision
    Supplement Fig. 7 shows mean scores decrease across stages, but lower mean is not evidence of improved discriminative power.
  • domain assumption Hoyer sparsity / high mean reward indicates reward saturation
    Eqs. 8-9; assumption that low within-group score variance implies saturation rather than convergence or prompt insensitivity.
  • standard math Standard GRPO and SDE sampling assumptions
    Uses known GRPO equations and SDE sampling; not the source of novelty.
invented entities (1)
  • Generator competence c
    purpose: Latent variable used to modulate reward component weights in Eqs. (5)-(9).
    c is invoked in the formulation but never formally defined or measured independently; it is operationalized through group reward statistics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Reward Signals in Video GRPO: When Scores Become Targets." pith.science (2026). https://pith.science/paper/D75O2IKG

@misc{pith2026251119356,
  author       = {Pith},
  title        = {Pith review of: Rethinking Reward Signals in Video GRPO: When Scores Become Targets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D75O2IKG}},
  note         = {Machine review of arXiv:2511.19356}
}
read the original abstract

Group Relative Policy Optimization (GRPO) enables stable and preference-oriented updates via group-wise comparisons for post-training video generation. However, GRPO directly optimizes reward-induced advantages. Under sustained optimization, the reward score can lose fidelity as a proxy for true video quality, consistent with the phenomenon described by Goodhart's Law. This leads to two recurring issues: (i) shortcut-driven optimization under composite objectives and (ii) reward saturation within prompt groups. To address these issues, we introduce TaRoS, a Target-Robust Reward Signaling framework for Video generation GRPO. TaRoS leverages component level performance assessment together with intra-group sparsity to organize multi-aspect rewards towards optimization objectives. In addition, it adaptively downweights components that exhibit saturation, thereby preserving effective optimization directions and mitigating redundancy. This maintains meaningful optimization directions and preserves within-group ranking separation, thereby preventing reward hacking and leading to more reliable policy updates. Extensive experiments show consistent improvements in visual fidelity, motion coherence, and text-video alignment over strong baselines.

Figures

Figures reproduced from arXiv: 2511.19356 by the authors.

Figure 1
Figure 1. As video quality improves, scores of static reward mod [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of reward model bias. VideoAlign shows [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Pipeline of the proposed Self-Paced GRPO. The generator produces candidate outputs, which are evaluated by a progressive [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Reward preference analysis. We partition 2,000 real [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Ablation study: we further grouped 2,000 iStock [ [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Qualitative comparison between Wan2.1-T2V-14B and our fine-tuned model. Top: baseline Wan2.1-T2V-14B results. Bottom: [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Self-paced Reward Model: Different stages of VLMs [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Bias of Other VLMs on Real-World videos. The results [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Bias of Other VLMs on Generated Videos. The results [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 11
Figure 11. Figure 11: Examples of generated results with fewer structural inconsistencies. [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: Examples of generated results with better visual quality. [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: Examples of generated results with better real-world alignment and text alignment. [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 14
Figure 14. Figure 14: Detailed illustration of input template of reward model. [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Physics-Grounded Fluid Video Generation with a Simulation Dataset and Dual-Stream Optical-Flow Supervision

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Adding a dual-stream optical-flow decoder and a simulation+real fluid dataset to a frozen video diffusion model improves the physical plausibility of generated pours and splashes.

  2. Twins: Learn to Predict Unified Representations with Focal Loss

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Channel-wise concatenation of SigLIP2 and Flux VAE features into one token, trained with a focal-style flow-matching loss, yields a unified representation with 1.59 gFID on ImageNet 256 and VAE-level reconstruction.

Reference graph

Works this paper leans on

36 extracted references · 17 linked inside Pith · cited by 2 Pith papers

  1. [1]

    V-jepa 2: Self- supervised video models enable understanding, prediction and planning.arXiv preprint arXiv:2506.09985, 2025

    Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, et al. V-jepa 2: Self- supervised video models enable understanding, prediction and planning.arXiv preprint arXiv:2506.09985, 2025. 2

    arXiv 2025

  2. [2]

    Boosting the generalization and reasoning of vision language models with curriculum reinforcement learning.arXiv preprint arXiv:2503.07065, 2025

    Huilin Deng, Ding Zou, Rui Ma, Hongchen Luo, Yang Cao, and Yu Kang. Boosting the generalization and reasoning of vision language models with curriculum reinforcement learning.arXiv preprint arXiv:2503.07065, 2025. 5, 1

    arXiv 2025

  3. [3]

    Reinforcement learning in continuous time and space.Neural computation, 12(1):219–245, 2000

    Kenji Doya. Reinforcement learning in continuous time and space.Neural computation, 12(1):219–245, 2000. 2

    2000

  4. [4]

    Inflvg: Reinforce inference-time con- sistent long video generation with grpo.arXiv preprint arXiv:2505.17574, 2025

    Xueji Fang, Liyuan Ma, Zhiyang Chen, Mingyuan Zhou, and Guo-jun Qi. Inflvg: Reinforce inference-time con- sistent long video generation with grpo.arXiv preprint arXiv:2505.17574, 2025. 1, 2

    arXiv 2025

  5. [5]

    You only look at one sequence: Rethinking transformer in vision through object detection.Advances in Neural Information Processing Systems, 34:26183–26197, 2021

    Yuxin Fang, Bencheng Liao, Xinggang Wang, Jiemin Fang, Jiyang Qi, Rui Wu, Jianwei Niu, and Wenyu Liu. You only look at one sequence: Rethinking transformer in vision through object detection.Advances in Neural Information Processing Systems, 34:26183–26197, 2021. 2

    2021

  6. [6]

    Im- proving dynamic object interactions in text-to-video gener- ation with ai feedback.arXiv preprint arXiv:2412.02617,

    Hiroki Furuta, Heiga Zen, Dale Schuurmans, Aleksandra Faust, Yutaka Matsuo, Percy Liang, and Sherry Yang. Im- proving dynamic object interactions in text-to-video gener- ation with ai feedback.arXiv preprint arXiv:2412.02617,

    arXiv

  7. [7]

    Hierar- chical process memory: memory as an integral component of information processing.Trends in cognitive sciences, 19 (6):304–313, 2015

    Uri Hasson, Janice Chen, and Christopher J Honey. Hierar- chical process memory: memory as an integral component of information processing.Trends in cognitive sciences, 19 (6):304–313, 2015. 2

    2015

  8. [8]

    Tempflow-grpo: When timing matters for grpo in flow models.arXiv preprint arXiv:2508.04324, 2025

    Xiaoxuan He, Siming Fu, Yuke Zhao, Wanli Li, Jian Yang, Dacheng Yin, Fengyun Rao, and Bo Zhang. Tempflow-grpo: When timing matters for grpo in flow models.arXiv preprint arXiv:2508.04324, 2025. 1

    arXiv 2025

Show all 36 references
  1. [9]

    Non-negative matrix factorization with sparseness constraints.Journal of machine learning re- search, 5(Nov):1457–1469, 2004

    Patrik O Hoyer. Non-negative matrix factorization with sparseness constraints.Journal of machine learning re- search, 5(Nov):1457–1469, 2004. 3

    2004

  2. [10]

    A reinforcement learning-based automatic video editing method using pre-trained vision-language model

    Panwen Hu, Nan Xiao, Feifei Li, Yongquan Chen, and Rui Huang. A reinforcement learning-based automatic video editing method using pre-trained vision-language model. In Proceedings of the 31st ACM International Conference on Multimedia, pages 6441–6450, 2023. 2

    2023

  3. [11]

    Vbench: Comprehensive bench- mark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive bench- mark suite for video generative models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pa...

    2024

  4. [12]

    istock video dataset.https : / / www

    iStock. istock video dataset.https : / / www . istockphoto.com/videos, 2025. Accessed: Oct. 17,

    2025

  5. [13]

    Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024. 5

    2024 arXiv

  6. [14]

    Early language acquisition: cracking the speech code.Nature reviews neuroscience, 5(11):831–843,

    Patricia K Kuhl. Early language acquisition: cracking the speech code.Nature reviews neuroscience, 5(11):831–843,

  7. [15]

    T2v- turbo-v2: Enhancing video generation model post-training through data, reward, and conditional guidance design.arXiv preprint arXiv:2410.05677, 2024

    Jiachen Li, Qian Long, Jian Zheng, Xiaofeng Gao, Robinson Piramuthu, Wenhu Chen, and William Yang Wang. T2v- turbo-v2: Enhancing video generation model post-training through data, reward, and conditional guidance design.arXiv preprint arXiv:2410.05677, 2024. 1

    2024

  8. [16]

    Mixgrpo: Unlocking flow- based grpo efficiency with mixed ode-sde.arXiv preprint arXiv:2507.21802, 2025

    Junzhe Li, Yutao Cui, Tao Huang, Yinping Ma, Chun Fan, Miles Yang, and Zhao Zhong. Mixgrpo: Unlocking flow- based grpo efficiency with mixed ode-sde.arXiv preprint arXiv:2507.21802, 2025. 1

    2025 arXiv

  9. [17]

    Spatial-then-temporal self-supervised learning for video correspondence

    Rui Li and Dong Liu. Spatial-then-temporal self-supervised learning for video correspondence. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2279–2288, 2023. 2

    2023

  10. [18]

    Vila: On pre-training for vi- sual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Moham- mad Shoeybi, and Song Han. Vila: On pre-training for vi- sual language models. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 26689–26699, 2024. 2

    2024

  11. [19]

    Flow-grpo: Training flow matching models via on- line rl.arXiv preprint arXiv:2505.05470, 2025

    Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via on- line rl.arXiv preprint arXiv:2505.05470, 2025. 1, 2

    2025 arXiv

  12. [20]

    Improving video generation with human feedback.arXiv preprint arXiv:2501.13918, 2025

    Jie Liu, Gongye Liu, Jiajun Liang, Ziyang Yuan, Xiaokun Liu, Mingwu Zheng, Xiele Wu, Qiulin Wang, Menghan Xia, Xintao Wang, et al. Improving video generation with human feedback.arXiv preprint arXiv:2501.13918, 2025. 1, 5, 7

    2025 arXiv

  13. [21]

    When the future becomes the past: Taming temporal correspondence for self-supervised video represen- tation learning

    Yang Liu, Qianqian Xu, Peisong Wen, Siran Dai, and Qing- ming Huang. When the future becomes the past: Taming temporal correspondence for self-supervised video represen- tation learning. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 24033–24044...

    2025

  14. [22]

    Enhance-a-video: Better generated video for free.arXiv preprint arXiv:2502.07508, 2025

    Yang Luo, Xuanlei Zhao, Mengzhao Chen, Kaipeng Zhang, Wenqi Shao, Kai Wang, Zhangyang Wang, and Yang You. Enhance-a-video: Better generated video for free.arXiv preprint arXiv:2502.07508, 2025. 2

    2025 arXiv

  15. [23]

    Liv: Language-image represen- tations and rewards for robotic control

    Yecheng Jason Ma, Vikash Kumar, Amy Zhang, Osbert Bas- tani, and Dinesh Jayaraman. Liv: Language-image represen- tations and rewards for robotic control. InInternational Con- ference on Machine Learning, pages 23301–23320. PMLR,

  16. [24]

    Video diffusion align- ment via reward gradients.arXiv preprint arXiv:2407.08737,

    Mihir Prabhudesai, Russell Mendonca, Zheyang Qin, Kate- rina Fragkiadaki, and Deepak Pathak. Video diffusion align- ment via reward gradients.arXiv preprint arXiv:2407.08737,

    arXiv

  17. [25]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, p...

    2021

  18. [26]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learn- ing with a unified text-to-text transformer.arXiv preprint arXiv:1910.10683, 2020. 5

    1910 arXiv

  19. [27]

    Laion-5b: An open large-scale dataset for training next generation image-text models.Advances in neural in- formation processing systems, 35:25278–25294, 2022

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models.Advances in neural in- fo...

    2022

  20. [28]

    Predictive reward signal of dopamine neu- rons.Journal of neurophysiology, 1998

    Wolfram Schultz. Predictive reward signal of dopamine neu- rons.Journal of neurophysiology, 1998. 2

    1998

  21. [29]

    Deepseekmath: Pushing the limits of math- ematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of math- ematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024. 2

    2024 arXiv

  22. [30]

    Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.Advances in neural information processing systems, 35:10078–10093, 2022

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.Advances in neural information processing systems, 35:10078–10093, 2022. 2

    2022

  23. [31]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianx- iao Yang, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025. 5

    2025 arXiv

  24. [32]

    Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis.arXiv preprint arXiv:2306.09341,

    Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis.arXiv preprint arXiv:2306.09341,

    arXiv

  25. [33]

    Visionreward: Fine-grained multi-dimensional human preference learning for image and video generation

    Jiazheng Xu, Yu Huang, Jiale Cheng, Yuanming Yang, Jiajun Xu, Yuan Wang, Wenbo Duan, Shen Yang, Qunlin Jin, Shu- run Li, et al. Visionreward: Fine-grained multi-dimensional human preference learning for image and video generation. arXiv preprint arXiv:2412.21059, 2024. 1

    2024 arXiv

  26. [34]

    Dancegrpo: Unleashing grpo on visual generation.arXiv preprint arXiv:2505.07818, 2025

    Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, et al. Dancegrpo: Unleashing grpo on visual generation.arXiv preprint arXiv:2505.07818, 2025. 1, 2

    2025 arXiv

  27. [35]

    Self-rewarding large vision-language models for opti- mizing prompts in text-to-image generation.arXiv preprint arXiv:2505.16763, 2025

    Hongji Yang, Yucheng Zhou, Wencheng Han, and Jianbing Shen. Self-rewarding large vision-language models for opti- mizing prompts in text-to-image generation.arXiv preprint arXiv:2505.16763, 2025. 5, 1

    2025

  28. [36]

    {text_prompt}

    Xiaomeng Yang, Zhiyu Tan, and Hao Li. Ipo: Iterative preference optimization for text-to-video generation.arXiv preprint arXiv:2502.02088, 2025. 2 Growing with the Generator: Self-paced GRPO for Video Generation Supplementary Material Due to space limitations in the main paper...

    2025

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.