{"id":"2593a90b-3101-4bb1-a5e4-e378642bf848","arxiv_id":"2502.05503","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"PhyCoBench and PhyCoPredictor provide a new benchmark and a learned optical-flow-guided evaluator for physical coherence of text-to-video models, but the evaluator's agreement with humans is modest (Kendall tau 0.34).","lead":"Researchers built a benchmark of 120 prompts covering seven physics categories (gravity, collision, vibration, friction, rotation, projectile motion, fluid dynamics) and an automated evaluator that predicts motion and scores generated videos against it. The evaluator's rankings match human judgments only moderately, so it is a promising tool, not yet a definitive one.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The automated score likely measures predictability and appearance match to the training distribution, not physical coherence; no per-category or ablation evidence supports the claimed alignment.","rationale":"The paper's strongest claim is comparative: PhyCoPredictor's rankings correlate better with human rankings than DynamiCrafter or VideoPhy. The reported Kendall's tau-b (0.3367) is modest, and the paper provides no confidence intervals or significance tests; more importantly, the score itself (Eq. 4) is a hand-weighted blend of optical-flow and pixel-space prediction errors. The human evaluation instructions (Table 7) explicitly exclude appearance and style, yet the video-MSE term is exactly an appearance error. Because the predictor is trained on the same datasets that inspired the benchmark prompts (UCF101, PennAction, HAA500, Physics101; Sec. 3.1 and 4.2.3), the automated score could be rewarding videos that match the predictor's learned motion and appearance priors, not physical laws. Category-specific correlation is absent; if the overall correlation is driven by categories such as gravity or collision that dominate the training data and collapses on fluid dynamics (a category with essentially no training videos), the central claim would fail. The proposed check would settle this by isolating the contribution of appearance error and testing robustness to the arbitrary weight. This in no way impugns the authors; the concern is purely about the evidence supporting the interpretive leap from prediction-error to physical-coherence score. The reader's conditional verdict already captures this risk, so no change is recommended.","tokens_in":13871,"tokens_out":6956,"duration_ms":70746,"concrete_test":"Using the released PhyCoBench data, recompute Kendall's tau-b between human rankings and automated scores separately for each of the seven prompt categories, and also for the score with the video-MSE weight set to 0, 1, 2, and 4 (re-optimizing ranking ties as in Sec. B.2). If the combined score's advantage over the DynamiCrafter baseline disappears under any weight in [0.5, 4] or is concentrated in categories whose motion types dominate the training data (gravity, collision, rotation) rather than fluid dynamics, the claim that the metric measures physical coherence is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that PhyCoPredictor aligns most closely with human evaluation (Sec. 4.4.1) rests on the untested premise that prediction error equals physical incoherence. Eq. 4 scores a video by 1/(MSE_flow + 2*MSE_video), where MSE_video is a pixel-space reconstruction error. Human evaluators are explicitly told to ignore style, appearance, and camera motion (Table 7), but the video MSE is dominated by appearance differences, and the flow MSE depends on the predictor's motion prior. The predictor was trained on UCF101, PennAction, HAA500, Physics101, and filtered OpenVid (Sec. 4.2.3, Table 6); the benchmark prompts were themselves created from those same datasets (Sec. 3.1). Thus the metric may rank videos by how typical they are of the predictor's training distribution rather than by physical correctness. A physically correct but out-of-distribution video (e.g., fluid dynamics, which is absent from the action datasets) will have large MSE and be scored as incoherent. No per-category correlation, no ablation of the flow vs. video terms, and no uncertainty quantification are reported, so the overall Kendall's tau-b of 0.3367 could be driven by a few categories or by the arbitrary 2x weight on video MSE. If that is the case, the claim that this is a physical-coherence evaluator is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PhyCoBench, a benchmark of 120 text prompts spanning seven categories of physically observable motion (gravity, collision, vibration, friction, fluid dynamics, projectile motion, rotation), together with human rankings of videos generated by four text-to-video models. It also proposes PhyCoPredictor, a two-stage latent diffusion evaluator that predicts future optical flow from the first frame and prompt, then uses that flow to predict future video frames. The automatic score in Eq. (4) combines optical-flow MSE and video MSE, and the authors report that the resulting model ranking correlates with human ranking with Kendall's tau-b of 0.3367, outperforming DynamiCrafter and VideoPhy. The paper claims that PhyCoPredictor 'currently aligns most closely with human evaluation' and can effectively evaluate physical coherence.","tokens_in":14086,"tokens_out":2828,"duration_ms":31952,"significance":"If the central claim is established, the paper would make a useful contribution: it provides a compact, category-structured prompt set for physical coherence, releases generated videos and human rankings, and proposes a reference-based automated evaluator that is more transparent than VLM-based scoring. The authors also release the benchmark, predictor, and dataset, which supports reproducibility. However, the current evidence for the central claim is modest: the headline Kendall's tau-b of 0.3367 is a single aggregate number with no uncertainty quantification, and the automated metric's construct validity -- that prediction error in this specific model equals physical incoherence -- is not directly tested. The overlap between the prompt sources and the predictor's training data (UCF101, PennAction, HAA500, Physics101, filtered OpenVid) further raises the possibility that the metric measures distributional typicality rather than physical correctness. The core idea is defensible, but the validation is not yet load-bearing.","major_comments":[{"comment":"The human evaluation procedure is under-specified: the paper does not state how many evaluators participated, whether all 120 prompts were ranked by the same set of evaluators, how disagreements were resolved, or what inter-annotator agreement was. Since every correlation coefficient in Table 2 is computed against this manual ranking, the target variable itself needs reliability evidence. Please report the number of annotators, the aggregation rule for ties, and an agreement measure such as pairwise agreement or Krippendorff's alpha; without this, the ranking standard is unvalidated.","section":"§3.2 and §B.1 (Manual Evaluation)"},{"comment":"The scoring metric contains two ad-hoc free choices: the relative weight of video MSE versus flow MSE (the factor of 2) and the number of sampled frames N. No sensitivity analysis or ablation is provided. Because the flow and video MSEs are in different units (2-channel displacement versus 3-channel pixel values), the summation is not scale-invariant, and the aggregate tau of 0.3367 could be dominated by either term or by the arbitrary weight. Please report per-category Kendall/Spearman correlations, flow-only and video-only scores, and a sweep over the weight (e.g., 0, 0.5, 1, 2, 5) and over N to show that the conclusion is not an artifact of these choices.","section":"§4.3, Eq. (4)"},{"comment":"The paper's premise is that a video is physically coherent exactly when its future frames and optical flow are predictable by PhyCoPredictor. This premise is not tested. The predictor is trained on UCF101, PennAction, HAA500, Physics101, and filtered OpenVid, and the benchmark prompts were themselves created from the same sources, so the metric may reward videos that resemble the training distribution rather than videos that obey physics. A physically correct but out-of-distribution video (for example, fluid dynamics, which is absent from the action datasets) would receive a large MSE and a low score. Please add control experiments: (i) physically plausible videos from categories or visual styles outside the training distribution, (ii) physically implausible but appearance-predictable videos, and (iii) comparison with human labels at the video level rather than only the model-ranking level. These tests would directly address whether the score measures physical coherence rather than predictability.","section":"§4.3 and §A.2 (Construct validity of prediction error as physical incoherence)"},{"comment":"The central quantitative claim rests on a single Kendall's tau-b of 0.3367 with no confidence interval, no significance test, and no comparison of whether the difference from the baseline correlations (-0.2438 and 0.0147) is statistically reliable. With 120 prompts and tied rankings, a permutation or bootstrap test is straightforward. Please report p-values or bootstrap confidence intervals for each correlation and for the difference between methods; otherwise the claim that PhyCoPredictor 'aligns most closely' is not statistically supported.","section":"Table 2 / §4.4.1"}],"minor_comments":[{"comment":"The sentence 'the Latent Adapter... downsamples the feature dimension of zf from 2 to 4' appears to be a typo: zf is described as having 4 channels and the optical flow has 2 channels, so the direction of the dimension change should be clarified.","section":"§4.2.1"},{"comment":"Please clarify whether the video loss Lvideo is computed in pixel space or latent space, and specify how the predicted video \\(v\\) is decoded and compared with the original video.","section":"§4.2.2, Eq. (3)"},{"comment":"The notation MSE(f, \\(f\\)) and MSE(v, \\(v\\)) is introduced without defining the sampling over N frames or the dimensionality of the inputs. Please define all symbols and state how N is chosen.","section":"§4.3, Eq. (4)"},{"comment":"The proportion of prompts across the seven categories is shown only graphically; please report the exact counts in the caption or text, especially because the categories are not equally sized.","section":"Figure 2"},{"comment":"There are several typographical issues (e.g., 'to to train' in §2.1, 'V AE' in §4.2.1, inconsistent spacing in references). A careful proofread would improve readability.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central claim is plausible but the validation is currently too thin for the strength of the conclusion. The most important concern is construct validity: the automated score may largely reflect predictability and distributional match to the training data, especially since the prompts and training data share sources. The authors should be encouraged to add the proposed control experiments and uncertainty quantification rather than only expanding the benchmark. The paper is within scope for the journal, but the current evidence does not yet support acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know about this paper: it ships a genuinely new benchmark asset, PhyCoBench, with 120 prompts across seven physical categories, plus a cascade predictor (optical flow diffusion then flow-guided frame prediction) that is new as a combination. The empirical comparison is informative: PhyCoPredictor's ranking of four T2V models matches human rankings better than DynamiCrafter or VideoPhy, with Kendall's tau 0.3367. That is a real positive result, even if modest.\n\nThe paper is honest about its pipeline: the predictor is not trained on human judgments; it is an external proxy, so the validation has independent grounding. That is worth crediting. The visual examples (falling leaves, rolling stone) do show plausible motion trajectories. The release of prompts, generated videos, and evaluator code is a concrete contribution.\n\nWhere the paper is soft is in the strength of the claim. \"Aligns most closely with human evaluation\" is true among the three compared options, but tau of 0.3367 is not strong evidence that the metric can reliably rank videos on physical coherence. There are no confidence intervals, no significance tests, and no inter-annotator agreement for the manual rankings; the number of evaluators is not stated. The score in Eq. 4 puts a 2x weight on video MSE with no justification, and that video MSE is a pixel-space reconstruction error, which is heavily influenced by appearance, style, and camera motion—exactly the things the human evaluators were told to ignore. So the metric may be rewarding videos that look like the predictor's training distribution rather than videos that obey physics. The predictor was trained on UCF101, PennAction, HAA500, Physics101, and filtered OpenVid, and the benchmark prompts were derived from those same datasets. That does not make the result circular—the human rankings are external—but it does mean the metric is in-distribution for these prompts, and we don't know how it would behave for out-of-distribution physical scenes like fluid dynamics, which are absent from the action datasets.\n\nThere is no per-category correlation and no ablation of the flow versus video terms, so we can't tell whether the overall correlation is driven by a couple of categories or by the arbitrary weighting. That is the main gap. The paper should be read as an early-stage proof of concept, not as a definitive evaluator.\n\nWho is this for? People working on video generation evaluation, especially automated metrics for physical plausibility. It deserves a serious referee; the benchmark is useful even if the metric needs more validation. I'd encourage the authors to add uncertainty quantification, a per-category breakdown, and an ablation on the score weights before publication.\n\nRecommendation: send to peer review with a request for major revision.","headline":"Useful new benchmark for physical coherence in T2V evaluation, but the automated metric's evidence is thinner than the abstract suggests.","tokens_in":14647,"tokens_out":2492,"would_cite":true,"duration_ms":23684,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Optical-flow-guided frame prediction scores video physics closest to human rankings.","keywords":["physical coherence","video generation benchmark","text-to-video evaluation","optical flow guidance","frame prediction","latent diffusion model","video anomaly detection","human evaluation alignment"],"falsifier":"Take a set of videos that are all physically correct but differ in rendering style, texture, or object category, run PhyCoPredictor on them, and check whether scores stay flat. If scores vary substantially with appearance, or if human rankings on a fresh prompt set disagree with the model at near-zero Kendall correlation, the claim that the model evaluates physical coherence rather than distribution fit is falsified.","tokens_in":13614,"feed_emoji":"🎬","tokens_out":5990,"duration_ms":52058,"temperature":0.7,"pith_summary":"The paper introduces PhyCoBench, a benchmark of 120 prompts spanning seven physical principles, and argues that physical coherence of text-to-video output should be judged automatically by comparing a generated video against frames and optical flow predicted from its first frame and prompt. The proposed evaluator, PhyCoPredictor, is a cascade of two latent diffusion models: the first forecasts future optical flow, and the second uses that flow to guide future-frame prediction. On the benchmark, rankings produced by PhyCoPredictor agree with human rankings better than an image-to-video predictor fine-tuned on the same data and better than a video-language-model baseline. The paper's central claim is that predictability under its model is a usable proxy for human judgment of physical coherence, giving video generation a benchmark dimension that most current metrics ignore.","feed_headline":"Flow-guided model ranks AI video physics closest to human judgment","feed_subtitle":"PhyCoBench's flow-guided predictor beats existing video-physics metrics in matching human rankings.","key_machinery":"The central object is PhyCoPredictor, a two-stage latent diffusion frame-prediction model. Its Latent Flow Diffusion Module (a 3D U-Net trained from scratch) takes the first frame and text prompt and predicts future optical flow in latent space; its Latent Video Diffusion Module (initialized from DynamiCrafter) takes the predicted flow, first frame, and prompt and generates future frames. The automatic evaluation score is $\\mathrm{score} = 1/\\mathrm{MSE}(f,\\hat{f}) + 2\\,\\mathrm{MSE}(v,\\hat{v})$, where higher score means judged more physically coherent. The machinery translates physical coherence into frame-predictability: a video that cannot be anticipated from its opening frame and prompt is scored as physically incoherent.","core_discovery":"The central claim is that physical coherence can be scored by comparing a generated video with the frames and optical flow that PhyCoPredictor predicts from the first frame and prompt. PhyCoPredictor is a cascade of two latent diffusion modules trained on dynamic-scene video data; at evaluation, its predicted flow and video are compared with the generated video's computed flow and frames through the score $1/\\mathrm{MSE}(f,\\hat{f}) + 2\\,\\mathrm{MSE}(v,\\hat{v})$, where $f,\\hat{f}$ are original and predicted optical flow and $v,\\hat{v}$ are original and predicted video frames. On the 120-prompt PhyCoBench, model rankings from this score correlate with manual rankings at Kendall's Tau-b 0.3367 and Spearman's 0.3751, while the flow-free DynamiCrafter baseline and the VideoPhy baseline produce negative or near-zero values. The paper concludes that PhyCoPredictor currently aligns most closely with human evaluation among the compared approaches.","pith_inferences":["A natural extension is to use the PhyCoPredictor score as a reward or ranking loss when training text-to-video models, treating low predictability as a differentiable penalty.","The premise that predictability equals physical coherence could be tested directly by holding physical correctness fixed while varying appearance: if scores move with style, the metric is partly measuring distribution fit rather than physics.","Because the predictor is trained on human-action and physics-experiment datasets, it may rank videos with unfamiliar objects or stylistic renderings as less coherent even when their trajectories obey physics; evaluating on out-of-distribution prompts would measure this bias."],"forward_implications":["A practical automatic metric now exists for ranking text-to-video models on physical coherence without per-video human labeling.","Researchers can use PhyCoBench's 120 prompts and seven physical categories to compare future text-to-video models against a fixed reference set.","Optical flow guidance, rather than raw appearance, is what lets the predictor track motion trajectories, suggesting motion-conditioned training improves downstream physical plausibility.","If the correlation holds on other model sets, PhyCoPredictor can serve as a cheap signal for iterating on video generators, replacing expensive manual evaluation.","The negative Kendall value for the flow-free DynamiCrafter baseline indicates that without such guidance, frame-prediction error alone misranks physical coherence."],"supporting_citations":[{"why":"DynamiCrafter is the image-to-video latent diffusion framework PhyCoPredictor's video module is built on, and the flow-free baseline fine-tuned on the same data for comparison.","marker":"[43]"},{"why":"VideoPhy is the video-language-model benchmark baseline that PhyCoPredictor must beat in correlating with human judgment.","marker":"[3]"},{"why":"FlowFormer++ computes the optical flow used as training supervision and as the reference flow of generated videos in the evaluation score.","marker":"[35]"},{"why":"OpenVid supplies the large pool of captioned videos that are filtered for dynamic scenes to train the latent flow diffusion module from scratch.","marker":"[29]"},{"why":"UCF101 contributes action videos to the Motion Data used to fine-tune both modules on dynamic human actions.","marker":"[37]"},{"why":"PennAction contributes human motion videos to the Motion Data used in training PhyCoPredictor.","marker":"[48]"},{"why":"HAA500 contributes curated human-centric action videos to the Motion Data used in training PhyCoPredictor.","marker":"[9]"},{"why":"Physics101 supplies physical experiment videos used in training, directly teaching the predictor about falling, collision, and other physical scenarios.","marker":"[41]"}],"fun_headline_variants":["AI video physics checker matches human judgment","New benchmark scores video physics like a human","Flow-guided video physics benchmark outperforms baselines","PhyCoBench: AI video physics scoring closer to humans","Video generation physics benchmark aligns with human ratings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a generated video is physically coherent exactly when its future frames and optical flow are predictable by PhyCoPredictor; if a physically correct video looks or moves differently than the predictor's training distribution, its MSE will be high and it will be wrongly scored as incoherent.","fun_headline_variants_meta":{"raw":{"variants":["AI video physics checker matches human judgment","New benchmark scores video physics like a human","Flow-guided video physics benchmark outperforms baselines","PhyCoBench: AI video physics scoring closer to humans","Video generation physics benchmark aligns with human ratings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000176,"raw_usage":{"total_tokens":1293,"prompt_tokens":953,"completion_tokens":340,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":270}},"tokens_in":569,"tokens_out":340,"duration_ms":3852,"temperature":1.0,"reasoning_tokens":270,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T19:04:05.280217+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of videos that are all physically correct but differ in rendering style, texture, or object category, run PhyCoPredictor on them, and check whether scores stay flat. If scores vary substantially with appearance, or if human rankings on a fresh prompt set disagree with the model at near-zero Kendall correlation, the claim that the model evaluates physical coherence rather than distribution fit is falsified.","supporting_citations":[{"cited_title":"Dynamicrafter: Animating open-domain images with video diffusion priors","cited_arxiv_id":null,"evidence_quote":"DynamiCrafter is the image-to-video latent diffusion framework PhyCoPredictor's video module is built on, and the flow-free baseline fine-tuned on the same data for comparison."},{"cited_title":"Flowformer++: Masked cost volume autoen- coding for pretraining optical flow estimation","cited_arxiv_id":null,"evidence_quote":"FlowFormer++ computes the optical flow used as training supervision and as the reference flow of generated videos in the evaluation score."},{"cited_title":"From actemes to action: A strongly-supervised repre- sentation for detailed action understanding","cited_arxiv_id":null,"evidence_quote":"PennAction contributes human motion videos to the Motion Data used in training PhyCoPredictor."},{"cited_title":"Haa500: Human-centric atomic action dataset with curated videos","cited_arxiv_id":null,"evidence_quote":"HAA500 contributes curated human-centric action videos to the Motion Data used in training PhyCoPredictor."},{"cited_title":"Physics 101: Learning phys- ical object properties from unlabeled videos","cited_arxiv_id":null,"evidence_quote":"Physics101 supplies physical experiment videos used in training, directly teaching the predictor about falling, collision, and other physical scenarios."}],"review_version":1}