Pith. sign in

REVIEW 5 major objections 5 minor 58 references

What You See Is What Matters: A Novel Visual and Physics-Based Metric for Evaluating Video Generation Quality

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read VAMP is a reference-free video quality metric that scores generated videos on visual appearance consistency and motion plausibility, and the paper argues it tracks human perception better than FVD, IS, and CLIP score.

desk verdict The appearance branch and corrupted-video dataset are useful, but the motion score doesn't measure physical plausibility and no human study backs the alignment claim, so the central contribution fails. read the letter →

arxiv 2411.13609 v2 pith:QQRSBY2M submitted 2024-11-20 cs.CV

Zihan Wang , Songlin Li , Lingyan Hao , Xinyu Hu , Bowen Song This is my paper · ORCID
classification cs.CV
keywords videogenerationevaluationreference-freequalitymetricvisualappearanceconsistencymotionplausibilitycolor-shape-texturesimilarityFréchetDistanceSAM2objecttrackingEarthMover's
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces VAMP, a reference-free metric that judges generated video quality the way a viewer might: by whether object appearance stays consistent from frame to frame and whether object motion looks smooth and physically plausible. The authors argue that existing metrics such as FVD, IS, and CLIP score operate on latent features and miss these human-visible cues. VAMP combines a color, shape, and texture appearance score with a motion score based on centroid velocity and acceleration stability. Two experiments, one on corrupted real videos and one on videos from four generation models, are used to argue that VAMP tracks degradation severity and model ranking in line with human judgment, and that it can replace FVD when no reference videos are available.

What carries the argument

VAMP is the weighted sum of an appearance score and a motion score, $S_{\mathrm{VAMP}} = \alpha S_{\mathrm{appearance}} + \beta S_{\mathrm{motion}}$ with $\alpha + \beta = 1$. The appearance score averages color similarity (Earth Mover's Distance between consecutive color histograms), shape similarity (symmetric Hausdorff distance on object contours), and texture similarity (cosine similarity of GLCM-derived features). The motion score combines $S_{\mathrm{vel}} = \exp(-\mathrm{std}(v)/\mathrm{mean}(v))$ and $S_{\mathrm{acc}} = \exp(-\mathrm{Var}(a))$, computed from the centroids of objects tracked across frames. Object masks and tracks come from SAM2, and the whole pipeline is reference-free because it never compares to ground-truth videos.

What would settle it

Take a video of an object moving at constant velocity with no acceleration (e.g., a ball hovering and sliding uniformly) and a second video of the same object undergoing realistic acceleration; if human raters judge the first as clearly less realistic while VAMP-M gives it a higher or equal motion score, the claim that VAMP measures physical plausibility is refuted. A complementary check is to feed a completely static video, where mean($v$)=0 makes the velocity-consistency term undefined, and observe what score the metric assigns.

Watch

Extended reading notes

Core claim

The central claim is that video quality can be decomposed into two measurable, reference-free dimensions: visual appearance consistency (color, shape, texture) and motion plausibility (smooth velocity and acceleration of tracked object centroids), and that their weighted sum, VAMP, captures both visual fidelity and temporal coherence better than latent-space metrics. The paper shows that VAMP scores decrease monotonically with corruption severity across five corruption types, while CLIP score and IS stay nearly flat, and that the resulting model ranking (Pika above VideoCrafter2 above ModelScope above Text2Video-Zero) matches what the authors' human evaluation suggests. It concludes that VAMP is an interpretable, reference-free alternative to FVD.

Load-bearing premise

The physical-plausibility score assumes that natural motion is well captured by smoothness of object-centroid velocity and acceleration, so a perfectly smooth but physically impossible motion (like a floating object moving at constant speed) would still score as perfectly plausible.

Editorial extensions

If this is right

  • If VAMP is adopted, video generation models can be ranked without collecting a corpus of real reference videos, lowering evaluation cost and enabling use in deployment settings where ground truth is unavailable.
  • Because VAMP reports separate appearance and motion subscores, practitioners can diagnose whether a model fails visually or dynamically rather than receiving a single opaque FVD number.
  • The monotonic response to corruption severity makes VAMP usable as a perceptual robustness probe: a model whose outputs degrade under brightness, noise, blur, or occlusions will show that degradation directly in the score.
  • The component weights of VAMP can be re-tuned per application, as the authors demonstrate with weight-sensitivity analyses, letting a user emphasize motion realism or visual fidelity as needed.
  • The SAM2-based variant, despite its higher compute cost, is the version the authors recommend because it yields stronger correlations with corruption level than the lightweight SIFT-based variant.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct testable extension is to show VAMP videos with physically implausible but perfectly smooth motion (a floating object gliding at constant velocity) and check whether the motion score stays high; the paper's definition suggests it would, which would weaken the 'physical plausibility' interpretation.
  • Because SAM2 segmentation failures are assigned a score of zero in the supplement, VAMP's ranking of low-quality videos may partly reflect tracker failure rather than video content; measuring how often this rule fires across datasets would clarify the metric's signal source.
  • The metric is best suited to videos with a small number of trackable objects; a natural extension is to test VAMP on dense crowds, heavy occlusion, or strong camera motion, where centroid-based smoothness may become less diagnostic.
  • The paper's corrupted-video validation uses UCF101 human-action clips; extending the same corruption-severity protocol to other domains (scenery, vehicles, object-centric synthetic scenes) would show whether VAMP's monotonic response generalizes beyond human actions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes VAMP, a reference-free video quality metric combining an appearance score (color, shape, and texture consistency across frames) with a motion score (velocity and acceleration consistency of tracked object centroids). The authors evaluate sensitivity by corrupting UCF101 videos at five severity levels and compare scores of four text-to-video generation models. They argue that VAMP aligns with human perception, can serve as a reference-free alternative to FVD, and captures physical plausibility. The abstract promises a comparison with human evaluators' rankings, but no human study appears in the manuscript. The motion score is defined from scalar centroid speeds and speed changes rather than vector accelerations or physical constraints, so constant-step-length trajectories receive perfect motion scores. Validation relies on hand-tuned weights evaluated on the same corruption data, and SAM2 failures are scored as zero without reporting failure rates. The supplementary material reports weight sensitivity and SIFT/SAM2 efficiency comparisons.

Significance. If the central claims held, the metric would be practically useful: it is reference-free, interpretable, and decomposes video quality into appearance and motion, and the corrupted-video stress test with five corruption types and severity levels is a sensible evaluation idea. The supplementary material also deserves credit for reporting weight sensitivity and the SIFT/SAM2 efficiency trade-off. However, the construct validity of the motion score is not established: as written, it measures a scalar smoothness heuristic rather than physical plausibility. The absence of the promised human evaluation leaves the alignment-with-human-perception claim unverified, and the weight tuning on the same corruption data makes the validation partly circular. If these issues were addressed, the metric might serve as a perceptual smoothness baseline, but the current evidence does not support the advertised claims.

major comments (5)
  1. [§3.2.2, Eqs. (9)–(12)] The motion score cannot measure physical plausibility because v_t in Eq. (9) is the scalar centroid speed and a_t in Eq. (11) is the change in scalar speed; no direction information enters S_vel or S_acc. A constant-speed zigzag, circular trajectory, or an object floating sideways at constant velocity yields std(v)=0 and Var(a)=0, giving S_vel=S_acc=1. With the supplement's motion weight of 0.7, this degenerate component dominates VAMP, so the advertised physics-based evaluation is not delivered.
  2. [§3.2.2, Eq. (10)] S_vel divides by mean(v), so for static videos mean(v)=0 and the score is undefined unless special-cased, in which case it becomes trivially perfect. Static or near-static generated videos are common, yet the paper neither defines the behavior for this case nor reports how such videos were handled in the experiments.
  3. [Abstract and §4.2] The abstract claims comparison with human evaluators' rankings, but §4.2 and Table 2 report only scores for Text2Video-Zero, ModelScope, VideoCrafter2, and Pika. No human study, no correlation with human ratings, and no inter-observer protocol are presented anywhere in the manuscript or supplement, so the claim that VAMP aligns with human perception is unsupported.
  4. [Supplement A and §5] The validation is not independent of the design choices: the supplement reports the weight tuple (0.069, 0.138, 0.092, 0.7), and the Limitations section admits 'extensive parameter tuning,' which appears to be performed on the same corrupted-video data used to demonstrate sensitivity. Table 1 and Figure 4 therefore show correlation with the tuning objective, not independent evidence. Additionally, Supplement A assigns a score of 0 to every video where SAM2 produces no masks; without reporting how often this occurs, this policy can dominate the reported scores by construction.
  5. [Supplement B.5, Table 3] The SIFT sampling variant, presented as an alternative in the paper, produces non-monotonic or weakly correlated VAMP scores for several corruption types, for example Impulse Noise VAMP-A values 0.918, 0.918, 0.756, 0.891, 0.887 and Black Shapes VAMP-A values 0.900, 0.888, 0.884, 0.869, 0.861; the supplement itself states that these correlations are 'not as ideal.' This limits the generality of the metric across its own proposed sampling choices and is not reflected in the main text's conclusions.
minor comments (5)
  1. [§3.2.1, Eqs. (4)–(5)] Equation (4) already applies 1/(1+max(...)) to obtain S_Hausdorff, and Eq. (5) applies 1/(1+S_Hausdorff) again; this double normalization is likely not intended and should be corrected or explained.
  2. [§3.2.2, Eq. (13)] The text preceding Eq. (13) refers to w_depth as a weight for depth consistency, but no depth consistency component appears in the equation, and Eq. (14) only normalizes w_vel + w_acc.
  3. [Table 1] Several VAMP rows are not strictly monotonic in corruption severity, for example VAMP-A for Gaussian noise goes 0.694, 0.661, 0.663, yet the paper reports only qualitative trends and provides no correlation coefficient, confidence interval, or significance test.
  4. [Figure 2] The numerical annotations in Figure 2, such as '0.867' and '0.575+0.721', are unexplained and should be defined in the caption or in the text.
  5. [Throughout] There are several typographical and formatting issues, including 'with and smooth transitions' in the Introduction, the inconsistent spacing of 'V AMP' versus 'VAMP', and a repeated self-citation to the VidProM reference [44] in the conclusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: VAMP's equations are self-contained; tuning and physics concerns are validity issues, not derivation-circularity.

full rationale

VAMP is an operational metric, not a fitted model. Its appearance sub-scores are standard distance/similarity measures (EMD, Hausdorff, GLCM-cosine), and its motion sub-scores are defined directly by Eqs. (9)-(14); there is no hidden parameter fit to the evaluation target and then reported as a prediction. The only tunable quantities are component weights, given as fixed tuples in the supplement, and the Limitations section admits 'extensive parameter tuning' that 'may not generalize well.' That is a legitimate generalizability and validation concern, but the paper does not state that the weights were optimized on the UCF101 corruption set or that the reported corruption correlations are held-out predictions, so it does not reduce to the fitted-input-called-prediction pattern. The SAM2 zero-scoring rule in the supplement ('we assigned a score of 0 to the corresponding videos') is an external-tracker assumption, not a self-referential input. The serious concern that Svel and Sacc measure scalar speed smoothness rather than physical plausibility (constant-speed non-physical trajectories can receive Svel=Sacc=1) is a construct-validity and correctness objection: the equations do not secretly contain the conclusion that they measure physics, so it is not a circular derivation. No load-bearing self-citation or imported uniqueness theorem appears; references [5] and [48] are prior-work citations unrelated to the metric's derivation. The abstract's claim of comparison to 'human evaluators' rankings' is not substantiated in the described experiments, but that is missing evidence, not circularity. Overall, the derivation chain is self-contained and no significant circularity is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

VAMP introduces no new physical entities. The metric's outputs depend on several hand-tuned weights and on an unvalidated assumption that smooth centroid motion equals physical plausibility. These parameters and assumptions are the real load-bearing content beyond the off-the-shelf image distances.

free parameters (3)
  • alpha (appearance weight) and beta (motion weight) = alpha=0.3, beta=0.7 (SAM2-based main results)
    In Eq. (15), the VAMP score is a weighted sum of appearance and motion scores. The supplement reports the weight tuple (0.069, 0.138, 0.092, 0.7) for color, shape, texture, and motion, corresponding to alpha=0.3, beta=0.7. These are hand-selected, and the sensitivity analysis in the supplement shows that rankings of generative models change with alpha.
  • Appearance sub-weights (w_color, w_shape, w_texture) = 0.069, 0.138, 0.092 (before renormalization)
    Eq. (7) requires w_color + w_shape + w_texture = 1, but the reported tuple sums to 0.299 when combined with the motion weight 0.7, so the implementation either renormalizes or violates the equation. These weights are manually tuned and affect the appearance score.
  • Zero score for SAM2 failure = 0
    Supplementary Section A states that videos where SAM2 fails to produce segmentation masks are assigned a VAMP score of 0, a manual rule that conflates tracking failure with low video quality.
assumptions (4)
  • ad hoc to paper Physical plausibility is equivalent to low variance of centroid speed and low variance of speed changes.
    This is the core modeling choice in Section 3.2.2 (Eqs. 10 and 12). It is not derived from physics; constant-acceleration motion (e.g., free fall) yields perfect scores, and static videos yield undefined scores.
  • domain assumption Appearance quality is fully captured by color, shape, and texture consistency of tracked objects.
    Section 3.2.1 selects these three attributes based on human visual system literature [12], but semantic correctness, lighting, and global scene coherence are ignored.
  • domain assumption SAM2 segmentation and tracking reliably isolates corresponding objects in generated and corrupted videos.
    The entire pipeline depends on SAM2 masks; the paper's response to tracker failure is to zero the score, implying failures mark bad videos (Supplement Section A).
  • domain assumption Corruption severity levels 1-5 are a monotonic proxy for human-perceived video quality.
    Section 4.1 corrupts UCF101 videos but collects no human ratings; the experiment assumes worsening appearance and motion under corruption is equivalent to perceptual quality loss.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What You See Is What Matters: A Novel Visual and Physics-Based Metric for Evaluating Video Generation Quality." pith.science (2026). https://pith.science/paper/QQRSBY2M

@misc{pith2026241113609,
  author       = {Pith},
  title        = {Pith review of: What You See Is What Matters: A Novel Visual and Physics-Based Metric for Evaluating Video Generation Quality},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QQRSBY2M}},
  note         = {Machine review of arXiv:2411.13609}
}
read the original abstract

As video generation models advance rapidly, assessing the quality of generated videos has become increasingly critical. Existing metrics, such as Fr\'echet Video Distance (FVD), Inception Score (IS), and ClipSim, measure quality primarily in latent space rather than from a human visual perspective, often overlooking key aspects like appearance and motion consistency to physical laws. In this paper, we propose a novel metric, VAMP (Visual Appearance and Motion Plausibility), that evaluates both the visual appearance and physical plausibility of generated videos. VAMP is composed of two main components: an appearance score, which assesses color, shape, and texture consistency across frames, and a motion score, which evaluates the realism of object movements. We validate VAMP through two experiments: corrupted video evaluation and generated video evaluation. In the corrupted video evaluation, we introduce various types of corruptions into real videos and measure the correlation between corruption severity and VAMP scores. In the generated video evaluation, we use state-of-the-art models to generate videos from carefully designed prompts and compare VAMP's performance to human evaluators' rankings. Our results demonstrate that VAMP effectively captures both visual fidelity and temporal consistency, offering a more comprehensive evaluation of video quality than traditional methods.

Figures

Figures reproduced from arXiv: 2411.13609 by the authors.

Figure 1
Figure 1. Comparison of Embedding-Based Metrics and Vi￾sual Physics-Based Metrics. The figure illustrates the evalua￾tion pipelines for generated video quality. The top section rep￾resents embedding-based metrics, which rely on extracting fea￾tures from generated videos and comparing them in a latent space against reference videos. The bottom section introduces a human￾visual-system-inspired (HVS) evaluation framework, which … view at source ↗
Figure 2
Figure 2. Pipeline for the VAMP Score Calculation. This diagram illustrates the process for calculating the VAMP score. Starting with video input, point sampling and SAM masking are applied to identify and track objects across frames. The pipeline computes the Appearance Score, incorporating metrics for color similarity , shape similarity, and texture similarity. Concurrently, the Motion Score is calculated by evaluating velo… view at source ↗
Figure 3
Figure 3. Evaluation of video quality using the VAMP metric across different levels of corruptions and generative models. This figure [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Normalized changes in metric values across corruption types and levels. Each heatmap represents the impact of a specific [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Heatmap of Percentage Changes in VAMP Scores from Level 0 Across Corruption Types Using SIFT Sampling. This figure [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Sensitivity of VAMP Scores to Appearance Weights ( [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Impact of α on VAMP Scores Across Different Video Generation Models. The heatmap compares the VAMP scores for different generative models (VC2, MS, and Pika) as a function of α, the weight of the appearance score [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 28 canonical work pages

  1. [1]

    Multidiffusion: Fusing diffusion paths for controlled image generation

    Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. 2023. 2

    2023

  2. [2]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 2

    arXiv 2023

  3. [3]

    Large scale gan training for high fidelity natural image synthesis

    Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096, 2018. 2

    arXiv 2018

  4. [4]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators

  5. [5]

    Learning from imperfect demonstrations via adversarial confidence transfer

    Zhangjie Cao, Zihan Wang, and Dorsa Sadigh. Learning from imperfect demonstrations via adversarial confidence transfer. In 2022 International Conference on Robotics and Automation (ICRA), pages 441–447. IEEE, 2022. 2

    work page 2022

  6. [6]

    Videocrafter2: Overcoming data limitations for high-quality video diffusion models, 2024

    Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models, 2024. 7

    work page 2024

  7. [7]

    Measuring the quality of text-to-video model out- puts: Metrics and dataset

    Iya Chivileva, Philip Lynch, Tomas E Ward, and Alan F Smeaton. Measuring the quality of text-to-video model out- puts: Metrics and dataset. arXiv preprint arXiv:2309.08009,

    arXiv

  8. [8]

    One-shot imitation learning

    Yan Duan, Marcin Andrychowicz, Bradly Stadie, Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wo- jciech Zaremba. One-shot imitation learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 1087–1098, 2017. 2

    work page 2017

Show all 58 references
  1. [9]

    Scaling rectified flow trans- formers for high-resolution image synthesis, 2024

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yan- nik Marek, and Robin Rombach. Scaling rectified flow tr...

    2024

  2. [10]

    A density-based algorithm for discovering clusters in large spatial databases with noise

    Martin Ester, Hans-Peter Kriegel, Jorg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd, pages 226–231,

  3. [11]

    Exploring the interplay between video generation and world models in autonomous driving: A survey, 2024

    Ao Fu, Yi Zhou, Tao Zhou, Yi Yang, Bojun Gao, Qun Li, Guobin Wu, and Ling Shao. Exploring the interplay between video generation and world models in autonomous driving: A survey, 2024. 1

    2024

  4. [12]

    Contributions of shape, texture, and color in visual recognition

    Yunhao Ge, Yao Xiao, Zhi Xu, Xingrui Wang, and Lau- rent Itti. Contributions of shape, texture, and color in visual recognition. In European Conference on Computer Vision , pages 369–386. Springer, 2022. 3

    2022

  5. [13]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014. 2

    2014

  6. [14]

    A review on generative adversarial networks: Algorithms, theory, and applications

    Jie Gui, Zhenan Sun, Yonggang Wen, Dacheng Tao, and Jieping Ye. A review on generative adversarial networks: Algorithms, theory, and applications. IEEE Transactions on Knowledge and Data Engineering, 35(10):1041–1060, 2023. 2

    2023

  7. [15]

    Videoscore: Building auto- matic metrics to simulate fine-grained human feedback for video generation, 2024

    Xuan He, Dongfu Jiang, Ge Zhang, Max Ku, Achint Soni, Sherman Siu, Haonan Chen, Abhranil Chandra, Ziyan Jiang, Aaran Arulraj, Kai Wang, Quy Duc Do, Yuansheng Ni, Bo- han Lyu, Yaswanth Narsupalli, Rongqi Fan, Zhiheng Lyu, Yuchen Lin, and Wenhu Chen. Videoscore: Building auto- m...

    2024

  8. [16]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020. 2

    2020

  9. [17]

    Imagen video: High definition video generation with diffusion mod- els

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion mod- els. arXiv preprint arXiv:2210.02303, 2022. 2

    arXiv

  10. [18]

    Video dif- fusion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video dif- fusion models. Advances in Neural Information Processing Systems, 35:8633–8646, 2022. 1, 2

    2022

  11. [19]

    Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhut- dinov, and Eric P. Xing. Toward controlled generation of text. In Proceedings of the 34th International Conference on Machine Learning (ICML), pages 1587–1596, 2017. 2

    2017

  12. [20]

    Vbench: Comprehensive bench- mark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive bench- mark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and P...

    2024

  13. [21]

    Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional adver- sarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1125–1134, 2017. 2

    2017

  14. [22]

    A style-based generator architecture for generative adversarial networks

    Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4401–4410, 2019. 2

    2019

  15. [23]

    Alias- free generative adversarial networks

    Tero Karras, Miika Aittala, Samuli Laine, Erik H ¨ark¨onen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias- free generative adversarial networks. arXiv preprint arXiv:2106.12423, 2021. 1

    2021 arXiv

  16. [24]

    Text2video-zero: Text-to- image diffusion models are zero-shot video generators, 2023

    Levon Khachatryan, Andranik Movsisyan, Vahram Tade- vosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to- image diffusion models are zero-shot video generators, 2023. 7

    2023

  17. [25]

    Make-a-video: Text-to-video generation with- out text-video data

    Yoonjin Lee, Minho Kim, Inhan Kim, Sanghun Lee, and Jaegul Kang. Make-a-video: Text-to-video generation with- out text-video data. arXiv preprint arXiv:2303.07637, 2023. 2

    2023 arXiv

  18. [26]

    Video transformer network

    Xinyi Li, Huanyu Wang, and Yi Yang. Video transformer network. arXiv preprint arXiv:2102.00719, 2022. 2 9

    2022 arXiv

  19. [27]

    Sora generates videos with stunning geometrical consistency

    Xuanyi Li, Daquan Zhou, Chenxu Zhang, Shaodong Wei, Qibin Hou, and Ming-Ming Cheng. Sora generates videos with stunning geometrical consistency. arXiv preprint arXiv:2402.17403, 2024. 3

    2024 arXiv

  20. [28]

    Evalcrafter: Benchmarking and eval- uating large video generation models

    Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and eval- uating large video generation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern R...

    2024

  21. [29]

    Videofusion: Decomposed diffusion mod- els for high-quality video generation

    Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion mod- els for high-quality video generation. arXiv preprint arXiv:2303.08320, 2023. 2

    2023 arXiv

  22. [30]

    Pika: Text to video ai, 2024

    Pika Labs. Pika: Text to video ai, 2024. Accessed: 2024-11-

    2024

  23. [31]

    Worldsimbench: Towards video generation models as world simulators, 2024

    Yiran Qin, Zhelun Shi, Jiwen Yu, Xijun Wang, Enshen Zhou, Lijun Li, Zhenfei Yin, Xihui Liu, Lu Sheng, Jing Shao, Lei Bai, Wanli Ouyang, and Ruimao Zhang. Worldsimbench: Towards video generation models as world simulators, 2024. 1

    2024

  24. [32]

    Un- supervised representation learning with deep convolu- tional generative adversarial networks

    Alec Radford, Luke Metz, and Soumith Chintala. Un- supervised representation learning with deep convolu- tional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015. 2

    2015 arXiv

  25. [33]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. arXiv preprint arXiv:2103.00020, 2021. 1, 3

    2021 arXiv

  26. [34]

    Hierarchical text-conditional image gen- eration with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gen- eration with clip latents. arXiv preprint arXiv:2204.06125,

    arXiv

  27. [35]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junt- ing Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Doll´ar, and Christoph Feic...

    2024 arXiv

  28. [36]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 2

    2022

  29. [37]

    The earth mover’s distance as a metric for image retrieval

    Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. The earth mover’s distance as a metric for image retrieval. Inter- national journal of computer vision, 40:99–121, 2000. 3

    2000

  30. [38]

    Improved techniques for training gans

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016. 1

    2016

  31. [39]

    Ucf101: A dataset of 101 human actions classes from videos in the wild

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. In CRCV-TR-12-01, pages 1–8. University of Central Florida, Center for Research in Computer Vision,

  32. [40]

    Mocogan: Decomposing motion and content for video generation

    Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. IEEE Conference on Computer Vision and Pattern Recognition, 2018. 2

    2018

  33. [41]

    To- wards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. To- wards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018. 1, 3

    2018 arXiv

  34. [42]

    Generating videos with scene dynamics

    Carl V ondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. InAdvances in neu- ral information processing systems, pages 613–621, 2016. 2

    2016

  35. [43]

    Modelscope text-to-video technical report, 2023

    Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report, 2023. 7

    2023

  36. [44]

    Vidprom: A million-scale real prompt-gallery dataset for text-to-video diffusion models,

    Wenhao Wang and Yi Yang. Vidprom: A million-scale real prompt-gallery dataset for text-to-video diffusion models,

  37. [45]

    Drivedreamer: Towards real-world- driven world models for autonomous driving

    Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jia- gang Zhu, and Jiwen Lu. Drivedreamer: Towards real-world- driven world models for autonomous driving. arXiv preprint arXiv:2309.09777, 2023. 1

    2023 arXiv

  38. [46]

    Driving into the future: Multiview visual forecasting and planning with world model for au- tonomous driving

    Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for au- tonomous driving. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pa...

    2024

  39. [47]

    Video quality as- sessment based on structural distortion measurement

    Zhou Wang, Ligang Lu, and Alan C Bovik. Video quality as- sessment based on structural distortion measurement. Signal processing: Image communication, 19(2):121–132, 2004. 3

    2004

  40. [48]

    Weakly supervised correspondence learning

    Zihan Wang, Zhangjie Cao, Yilun Hao, and Dorsa Sadigh. Weakly supervised correspondence learning. In 2022 Inter- national Conference on Robotics and Automation (ICRA) , pages 469–476. IEEE, 2022. 2

    2022

  41. [49]

    Unleashing large-scale video generative pre- training for visual robot manipulation

    Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre- training for visual robot manipulation. arXiv preprint arXiv:2312.13139, 2023. 1

    2023 arXiv

  42. [50]

    A survey on video dif- fusion models

    Zhen Xing, Qijun Feng, Haoran Chen, Qi Dai, Han Hu, Hang Xu, Zuxuan Wu, and Yu-Gang Jiang. A survey on video dif- fusion models. ACM Computing Surveys, 57(2):1–42, 2024. 2

    2024

  43. [51]

    Videogpt: Video generation using vq-vae and transformers

    Wilson Xu, Aditya Agrawal, and David Dohan. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.10157, 2022. 2

    2022 arXiv

  44. [52]

    Dif- fusion probabilistic modeling for video generation

    Ruihan Yang, Prakhar Srivastava, and Stephan Mandt. Dif- fusion probabilistic modeling for video generation. Entropy, 25(10):1469, 2023. 2

    2023

  45. [53]

    Seqgan: Sequence generative adversarial nets with policy gradient

    Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In Proceedings of the 31st AAAI Conference on Artificial Intel- ligence (AAAI), pages 2852–2858, 2017. 2

    2017

  46. [54]

    Physics-based human 10 motion estimation and synthesis from videos

    Peizhuo Yu, Kang Xie, Ying-Cong Wang, Yuting Zhang, Xi- aoxiao Liu, Bo Yang, and Qiang Fu. Physics-based human 10 motion estimation and synthesis from videos. arXiv preprint arXiv:2109.09913, 2022. 2

    2022

  47. [55]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 2

    2023

  48. [56]

    A survey of diffusion based image generation models: Issues and their solutions

    Tianyi Zhang, Zheng Wang, Jing Huang, Mohiud- din Muhammad Tasnim, and Wei Shi. A survey of diffusion based image generation models: Issues and their solutions. arXiv preprint arXiv:2308.13142, 2023. 2

    2023 arXiv

  49. [57]

    Magicvideo: Efficient video generation with latent diffusion models, 2023

    Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models, 2023. 2

    2023

  50. [58]

    Cocon: Cooperative-contrastive learning

    Xueting Zhou, Bo Kang, Ziwei Liu, Ali Borji, and Chen Change Loy. Cocon: Cooperative-contrastive learning. IEEE Conference on Computer Vision and Pattern Recogni- tion, 2021. 2 11 What Y ou See Is What Matters: A Novel Visual and Physics-Based Metric for Evaluating Video Gener...

    2021

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.