{"id":"df073fad-1dea-4eed-8774-fdceef79a6bd","arxiv_id":"2411.13609","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"VAMP scores generated videos by combining per-object color, shape, and texture consistency with centroid velocity and acceleration smoothness, but its physics-based and human-aligned claims are not supported by the reported experiments.","lead":"This paper proposes VAMP, a reference-free metric for generated video quality that combines appearance scores for color, shape, and texture with a motion score based on tracked object trajectories. The authors claim VAMP aligns with human perception better than existing latent-space metrics, but the paper omits the human evaluation promised in its abstract and equates physical plausibility with temporal smoothness.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The motion score in §3.2.2 cannot measure physical plausibility: Eqs. (9)–(12) discard direction, so constant-speed zigzag, circular, and levitating motion all receive perfect Smotion; the central claim collapses.","rationale":"Reader REJECT is supported. The paper's strongest claim is that VAMP is a visual and physics-based metric that aligns with human perception; for that to hold, the motion score must reward physically plausible dynamics and penalize implausible ones. The equations do the opposite: by collapsing velocity to a scalar centroid distance and acceleration to a scalar speed difference, all direction information is discarded, so constant-speed zigzags, circular motion, and levitation are indistinguishable from natural motion. The static case is a second edge case that is undefined or trivially perfect. Since motion weight is 0.7 in the supplement, this flaw dominates the final score. Secondary issues (inverted shape score in Eq. 5, absent human-evaluator data, hand-tuned weights) reinforce the rejection but are not needed for it. I would keep the reader's verdict: the central contribution is not validated as a physics-based metric.","tokens_in":16587,"tokens_out":7352,"duration_ms":122531,"concrete_test":"One check settles it: take a clean synthetic video of a single object on a static background and compare three trajectories: (A) natural projectile with gravity and ground contact; (B) same object moving along a zigzag or circular path with constant step length; (C) object sliding in a straight line at constant speed with no support. Run VAMP-M on all three plus a fully static version. Under the paper's construct, (A) must score highest; under Eqs. (9)–(12), (B) and (C) will both score near 1 and static may also score 1 or be undefined. Also recompute a_t with vector velocities: if vector-based scores differ from the scalar-based scores, the scalar formulation is the culprit.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing flaw is in §3.2.2. Eq. (9) defines v_t = ||c_{t+1} - c_t||, a scalar centroid speed, and Eq. (11) defines a_t = v_{t+1} - v_t, the change in that scalar speed rather than a vector acceleration. No direction information enters the motion score. Consequently, Eq. (10) gives S_vel = exp(0) = 1 for any trajectory with constant step length, regardless of whether the object zigzags, circles, or otherwise changes direction; equal step lengths make std(v) = 0. Eq. (12) gives S_acc = 1 for the same trajectories because Var(a) = 0. A rigid object sliding sideways through the air at constant speed, or an object jittering with constant-magnitude jumps, receives a perfect Smotion. A static object makes mean(v) = 0 in Eq. (10), producing an undefined score (or a trivially perfect one if special-cased). The supplement assigns motion a weight of 0.7 in the final tuple, so this component dominates VAMP. The metric therefore measures a degenerate kind of temporal smoothness, not physical plausibility, which is exactly the construct advertised in the title and abstract. This is an internal failure, not a disagreement with external consensus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VAMP, a reference-free video quality metric combining an appearance score (color, shape, and texture consistency across frames) with a motion score (velocity and acceleration consistency of tracked object centroids). The authors evaluate sensitivity by corrupting UCF101 videos at five severity levels and compare scores of four text-to-video generation models. They argue that VAMP aligns with human perception, can serve as a reference-free alternative to FVD, and captures physical plausibility. The abstract promises a comparison with human evaluators' rankings, but no human study appears in the manuscript. The motion score is defined from scalar centroid speeds and speed changes rather than vector accelerations or physical constraints, so constant-step-length trajectories receive perfect motion scores. Validation relies on hand-tuned weights evaluated on the same corruption data, and SAM2 failures are scored as zero without reporting failure rates. The supplementary material reports weight sensitivity and SIFT/SAM2 efficiency comparisons.","tokens_in":16873,"tokens_out":5022,"duration_ms":52011,"significance":"If the central claims held, the metric would be practically useful: it is reference-free, interpretable, and decomposes video quality into appearance and motion, and the corrupted-video stress test with five corruption types and severity levels is a sensible evaluation idea. The supplementary material also deserves credit for reporting weight sensitivity and the SIFT/SAM2 efficiency trade-off. However, the construct validity of the motion score is not established: as written, it measures a scalar smoothness heuristic rather than physical plausibility. The absence of the promised human evaluation leaves the alignment-with-human-perception claim unverified, and the weight tuning on the same corruption data makes the validation partly circular. If these issues were addressed, the metric might serve as a perceptual smoothness baseline, but the current evidence does not support the advertised claims.","major_comments":[{"comment":"The motion score cannot measure physical plausibility because v_t in Eq. (9) is the scalar centroid speed and a_t in Eq. (11) is the change in scalar speed; no direction information enters S_vel or S_acc. A constant-speed zigzag, circular trajectory, or an object floating sideways at constant velocity yields std(v)=0 and Var(a)=0, giving S_vel=S_acc=1. With the supplement's motion weight of 0.7, this degenerate component dominates VAMP, so the advertised physics-based evaluation is not delivered.","section":"§3.2.2, Eqs. (9)–(12)"},{"comment":"S_vel divides by mean(v), so for static videos mean(v)=0 and the score is undefined unless special-cased, in which case it becomes trivially perfect. Static or near-static generated videos are common, yet the paper neither defines the behavior for this case nor reports how such videos were handled in the experiments.","section":"§3.2.2, Eq. (10)"},{"comment":"The abstract claims comparison with human evaluators' rankings, but §4.2 and Table 2 report only scores for Text2Video-Zero, ModelScope, VideoCrafter2, and Pika. No human study, no correlation with human ratings, and no inter-observer protocol are presented anywhere in the manuscript or supplement, so the claim that VAMP aligns with human perception is unsupported.","section":"Abstract and §4.2"},{"comment":"The validation is not independent of the design choices: the supplement reports the weight tuple (0.069, 0.138, 0.092, 0.7), and the Limitations section admits 'extensive parameter tuning,' which appears to be performed on the same corrupted-video data used to demonstrate sensitivity. Table 1 and Figure 4 therefore show correlation with the tuning objective, not independent evidence. Additionally, Supplement A assigns a score of 0 to every video where SAM2 produces no masks; without reporting how often this occurs, this policy can dominate the reported scores by construction.","section":"Supplement A and §5"},{"comment":"The SIFT sampling variant, presented as an alternative in the paper, produces non-monotonic or weakly correlated VAMP scores for several corruption types, for example Impulse Noise VAMP-A values 0.918, 0.918, 0.756, 0.891, 0.887 and Black Shapes VAMP-A values 0.900, 0.888, 0.884, 0.869, 0.861; the supplement itself states that these correlations are 'not as ideal.' This limits the generality of the metric across its own proposed sampling choices and is not reflected in the main text's conclusions.","section":"Supplement B.5, Table 3"}],"minor_comments":[{"comment":"Equation (4) already applies 1/(1+max(...)) to obtain S_Hausdorff, and Eq. (5) applies 1/(1+S_Hausdorff) again; this double normalization is likely not intended and should be corrected or explained.","section":"§3.2.1, Eqs. (4)–(5)"},{"comment":"The text preceding Eq. (13) refers to w_depth as a weight for depth consistency, but no depth consistency component appears in the equation, and Eq. (14) only normalizes w_vel + w_acc.","section":"§3.2.2, Eq. (13)"},{"comment":"Several VAMP rows are not strictly monotonic in corruption severity, for example VAMP-A for Gaussian noise goes 0.694, 0.661, 0.663, yet the paper reports only qualitative trends and provides no correlation coefficient, confidence interval, or significance test.","section":"Table 1"},{"comment":"The numerical annotations in Figure 2, such as '0.867' and '0.575+0.721', are unexplained and should be defined in the caption or in the text.","section":"Figure 2"},{"comment":"There are several typographical and formatting issues, including 'with and smooth transitions' in the Introduction, the inconsistent spacing of 'V AMP' versus 'VAMP', and a repeated self-citation to the VidProM reference [44] in the conclusion.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The paper is not ready for publication in its current form. The motion-score design makes 'physics-based' an unearned label, the abstract's human-comparison claim is unsubstantiated, and the weight tuning on the same corruption data weakens the validation. I recommend rejection rather than major revision because the core metric would need to be redefined and the validation protocol rebuilt; a conference-length revision cannot fix these issues locally."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nVAMP combines an appearance score (color EMD, shape Hausdorff, texture GLCM over SAM2 masks) with a motion score (consistency of centroid speeds and speed changes). The appearance part is a reasonable reference-free heuristic, and the corrupted-video dataset on UCF101 with five corruption types and five severity levels is a reusable asset. The SIFT vs SAM2 efficiency comparison is also honest.\n\nThe soft spots are serious. The motion score is not physics-based. Equations (9)–(12) use scalar centroid speed and changes in that scalar, so direction information is discarded. A constant-speed zigzag, circle, or levitating object gets Smotion = 1. A static object makes mean(v) = 0, so the score is undefined. Since motion is weighted at 0.7, this breaks the central claim in the title and abstract. The supplement scores SAM2 tracking failures as zero, so the metric partly measures tracker reliability, not video quality. The paper's conclusion says VAMP aligns with human evaluations, but no human study appears in the paper.\n\nThere is also a concrete mathematical error: Eq. (5) defines Sshape = 1/(1+SHausdorff), where SHausdorff is already a reciprocal from Eq. (4). That double reciprocal makes larger Hausdorff distances produce higher shape similarity, which is backwards. The hand-tuned weight tuple (0.069, 0.138, 0.092, 0.7) and the sensitivity analysis do not fix the lack of an independent validation set.\n\nThe paper is not ready for acceptance. The central \"physical plausibility\" claim collapses under inspection, and there is no human evaluation to back the alignment claim. A thorough revision would need to redefine the motion score using vector acceleration and add a real human study. I would not cite this in its current form, but I would send it to peer review because the topic is timely and the appearance branch plus dataset could be salvaged. Reviewers will need to guide the authors toward a more defensible motion metric.","headline":"The appearance branch and corrupted-video dataset are useful, but the motion score doesn't measure physical plausibility and no human study backs the alignment claim, so the central contribution fails.","tokens_in":17418,"tokens_out":4661,"would_cite":false,"duration_ms":47957,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VAMP is a reference-free video quality metric that scores generated videos on visual appearance consistency and motion plausibility, and the paper argues it tracks human perception better than FVD, IS, and CLIP score.","keywords":["video generation evaluation","reference-free video quality metric","visual appearance consistency","motion plausibility","color-shape-texture similarity","Fréchet Video Distance","SAM2 object tracking","Earth Mover's Distance"],"falsifier":"Take a video of an object moving at constant velocity with no acceleration (e.g., a ball hovering and sliding uniformly) and a second video of the same object undergoing realistic acceleration; if human raters judge the first as clearly less realistic while VAMP-M gives it a higher or equal motion score, the claim that VAMP measures physical plausibility is refuted. A complementary check is to feed a completely static video, where mean($v$)=0 makes the velocity-consistency term undefined, and observe what score the metric assigns.","tokens_in":16362,"feed_emoji":"🎥","tokens_out":6083,"duration_ms":56247,"temperature":0.7,"pith_summary":"This paper introduces VAMP, a reference-free metric that judges generated video quality the way a viewer might: by whether object appearance stays consistent from frame to frame and whether object motion looks smooth and physically plausible. The authors argue that existing metrics such as FVD, IS, and CLIP score operate on latent features and miss these human-visible cues. VAMP combines a color, shape, and texture appearance score with a motion score based on centroid velocity and acceleration stability. Two experiments, one on corrupted real videos and one on videos from four generation models, are used to argue that VAMP tracks degradation severity and model ranking in line with human judgment, and that it can replace FVD when no reference videos are available.","feed_headline":"VAMP metric scores generated video quality without reference videos","feed_subtitle":"It pairs color-shape-texture consistency with motion smoothness to match human judgment where FVD needs ground truth.","key_machinery":"VAMP is the weighted sum of an appearance score and a motion score, $S_{\\mathrm{VAMP}} = \\alpha S_{\\mathrm{appearance}} + \\beta S_{\\mathrm{motion}}$ with $\\alpha + \\beta = 1$. The appearance score averages color similarity (Earth Mover's Distance between consecutive color histograms), shape similarity (symmetric Hausdorff distance on object contours), and texture similarity (cosine similarity of GLCM-derived features). The motion score combines $S_{\\mathrm{vel}} = \\exp(-\\mathrm{std}(v)/\\mathrm{mean}(v))$ and $S_{\\mathrm{acc}} = \\exp(-\\mathrm{Var}(a))$, computed from the centroids of objects tracked across frames. Object masks and tracks come from SAM2, and the whole pipeline is reference-free because it never compares to ground-truth videos.","core_discovery":"The central claim is that video quality can be decomposed into two measurable, reference-free dimensions: visual appearance consistency (color, shape, texture) and motion plausibility (smooth velocity and acceleration of tracked object centroids), and that their weighted sum, VAMP, captures both visual fidelity and temporal coherence better than latent-space metrics. The paper shows that VAMP scores decrease monotonically with corruption severity across five corruption types, while CLIP score and IS stay nearly flat, and that the resulting model ranking (Pika above VideoCrafter2 above ModelScope above Text2Video-Zero) matches what the authors' human evaluation suggests. It concludes that VAMP is an interpretable, reference-free alternative to FVD.","pith_inferences":["A direct testable extension is to show VAMP videos with physically implausible but perfectly smooth motion (a floating object gliding at constant velocity) and check whether the motion score stays high; the paper's definition suggests it would, which would weaken the 'physical plausibility' interpretation.","Because SAM2 segmentation failures are assigned a score of zero in the supplement, VAMP's ranking of low-quality videos may partly reflect tracker failure rather than video content; measuring how often this rule fires across datasets would clarify the metric's signal source.","The metric is best suited to videos with a small number of trackable objects; a natural extension is to test VAMP on dense crowds, heavy occlusion, or strong camera motion, where centroid-based smoothness may become less diagnostic.","The paper's corrupted-video validation uses UCF101 human-action clips; extending the same corruption-severity protocol to other domains (scenery, vehicles, object-centric synthetic scenes) would show whether VAMP's monotonic response generalizes beyond human actions."],"forward_implications":["If VAMP is adopted, video generation models can be ranked without collecting a corpus of real reference videos, lowering evaluation cost and enabling use in deployment settings where ground truth is unavailable.","Because VAMP reports separate appearance and motion subscores, practitioners can diagnose whether a model fails visually or dynamically rather than receiving a single opaque FVD number.","The monotonic response to corruption severity makes VAMP usable as a perceptual robustness probe: a model whose outputs degrade under brightness, noise, blur, or occlusions will show that degradation directly in the score.","The component weights of VAMP can be re-tuned per application, as the authors demonstrate with weight-sensitivity analyses, letting a user emphasize motion realism or visual fidelity as needed.","The SAM2-based variant, despite its higher compute cost, is the version the authors recommend because it yields stronger correlations with corruption level than the lightweight SIFT-based variant."],"supporting_citations":[{"why":"Defines FVD, the reference-based metric that VAMP positions itself against as a reference-free alternative.","marker":"[41]"},{"why":"Supplies the SAM2 segmentation and tracking that produce the object masks and centroids used by both appearance and motion scores.","marker":"[35]"},{"why":"Provides the Earth Mover's Distance (Wasserstein) used to quantify color similarity between consecutive frames.","marker":"[37]"},{"why":"UCF101 is the real-video dataset on which the corruption-severity experiments are run.","marker":"[39]"},{"why":"VidProM is the source of the 3,000 prompts used to generate videos from the four tested generation models.","marker":"[44]"},{"why":"VideoCrafter2 is one of the four generation models whose outputs are ranked by VAMP.","marker":"[6]"},{"why":"Text2Video-Zero is one of the four generation models whose outputs are ranked by VAMP.","marker":"[24]"},{"why":"ModelScope is one of the four generation models whose outputs are ranked by VAMP.","marker":"[43]"}],"fun_headline_variants":["VAMP: reference-free video quality metric","VAMP: two-part metric for visual and motion realism","No ground truth: VAMP scores video quality like humans","VAMP: physics-aware video evaluation without reference videos","VAMP: interpretable video metric that outperforms FVD"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The physical-plausibility score assumes that natural motion is well captured by smoothness of object-centroid velocity and acceleration, so a perfectly smooth but physically impossible motion (like a floating object moving at constant speed) would still score as perfectly plausible.","fun_headline_variants_meta":{"raw":{"variants":["VAMP: reference-free video quality metric","VAMP: two-part metric for visual and motion realism","No ground truth: VAMP scores video quality like humans","VAMP: physics-aware video evaluation without reference videos","VAMP: interpretable video metric that outperforms FVD"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000181,"raw_usage":{"total_tokens":1289,"prompt_tokens":907,"completion_tokens":382,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":304}},"tokens_in":523,"tokens_out":382,"duration_ms":4940,"temperature":1.0,"reasoning_tokens":304,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:00:26.309173+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a video of an object moving at constant velocity with no acceleration (e.g., a ball hovering and sliding uniformly) and a second video of the same object undergoing realistic acceleration; if human raters judge the first as clearly less realistic while VAMP-M gives it a higher or equal motion score, the claim that VAMP measures physical plausibility is refuted. A complementary check is to feed a completely static video, where mean($v$)=0 makes the velocity-consistency term undefined, and observe what score the metric assigns.","supporting_citations":[{"cited_title":"The earth mover’s distance as a metric for image retrieval","cited_arxiv_id":null,"evidence_quote":"Provides the Earth Mover's Distance (Wasserstein) used to quantify color similarity between consecutive frames."},{"cited_title":"Ucf101: A dataset of 101 human actions classes from videos in the wild","cited_arxiv_id":null,"evidence_quote":"UCF101 is the real-video dataset on which the corruption-severity experiments are run."},{"cited_title":"Vidprom: A million-scale real prompt-gallery dataset for text-to-video diffusion models,","cited_arxiv_id":null,"evidence_quote":"VidProM is the source of the 3,000 prompts used to generate videos from the four tested generation models."},{"cited_title":"Videocrafter2: Overcoming data limitations for high-quality video diffusion models, 2024","cited_arxiv_id":null,"evidence_quote":"VideoCrafter2 is one of the four generation models whose outputs are ranked by VAMP."},{"cited_title":"Text2video-zero: Text-to- image diffusion models are zero-shot video generators, 2023","cited_arxiv_id":null,"evidence_quote":"Text2Video-Zero is one of the four generation models whose outputs are ranked by VAMP."},{"cited_title":"Modelscope text-to-video technical report, 2023","cited_arxiv_id":null,"evidence_quote":"ModelScope is one of the four generation models whose outputs are ranked by VAMP."}],"review_version":1}