{"id":"ae5c5c78-94df-406d-9aae-b90c6b65b436","arxiv_id":"2412.00526","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CLIP-based semantic embeddings, with a fine-tuned variant, detect AI-generated human-motion video with high accuracy (up to 99.2% video-level) and generalize to unseen generators.","lead":"This paper tests whether AI-made videos of humans can be spotted by looking at the semantic meaning of frames instead of small pixel errors. It reports high accuracy on a new dataset and shows the method partly survives video compression and transfers to new AI models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's 'robust to resolution' claim is contradicted by Table 3: video accuracy falls to 72.8% at 25% and 51.0% at 10% resolution, and only a generic CLIP linear SVM was tested, not the headline 99.2% model.","rationale":"The reader's weakest assumption targets dataset representativeness; my concern is complementary and more internal: the paper's own Table 3 contradicts the advertised robustness to resolution. This is a correctness risk to the abstract's central claim, but it does not invalidate the paper's core contribution as a benchmark and a detection method at full resolution. The appropriate remedy is a conditional accept requiring the authors to (1) test the best-performing model under resolution and compression laundering, (2) report error bars, and (3) temper the abstract's robustness phrasing. Since the reader's verdict was already CONDITIONAL, my analysis reinforces that decision rather than changing it.","tokens_in":12570,"tokens_out":9224,"duration_ms":95113,"concrete_test":"On the DeepAction test split, evaluate the FT-CLIP + poly SVM and the FT-CLIP frame-to-prompt (P1) on videos downscaled to 25% and 10% of original resolution, using the same five action-level splits and reporting macro-averaged video accuracy with 95% confidence intervals. If video accuracy at 25% falls below roughly 80% or at 10% approaches chance, the resolution-robustness claim fails. Additionally, rerun Table 3 with action-level splits (as done for FT-CLIP) to determine whether the current random-split protocol inflates the reported accuracies through prompt leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim states the method is 'robust' to laundering 'including resolution and compression attacks.' Yet the paper's own robustness evaluation (Table 3) shows a two-class linear SVM on CLIP embeddings degrading from 96.9% video accuracy at full resolution to 72.8% at 25% and 51.0% at 10% resolution — the latter is chance. This directly undercuts the robustness claim and the authors' rationale that semantic embeddings are resilient to laundering; they themselves attribute the loss to 'small-scale features.' Compounding this, Table 3 only evaluates the generic CLIP embedding with a linear kernel, not the FT-CLIP + poly SVM or the frame-to-prompt FT-CLIP that achieve the abstract's headline numbers. There is therefore no evidence that the best-performing method is resolution-robust, while the one method tested contradicts the claim. Because 'robust' is a key qualifier in the central claim, this is a load-bearing internal inconsistency, not merely a missing experiment.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DeepAction, a new open-sourced dataset of 3,100 AI-generated video clips produced by seven text-to-video models together with 100 real Pexels videos matched on 100 human-action prompts. The authors evaluate four multi-modal embedding models (CLIP, SigLIP, JinaCLIP, and a custom fine-tuned CLIP) with linear, RBF, and polynomial SVM classifiers for two-class real/fake detection and eight-class source attribution, as well as a 'frame-to-prompt' cosine-similarity classifier that requires no classifier training. The paper reports video-level two-class accuracies up to 99.2%, an average leave-one-out generalization accuracy of 95.2% for the CLIP linear SVM, robustness to compression, and mixed results on resolution, non-human motion, talking-head deepfakes, and CGI.","tokens_in":12833,"tokens_out":7768,"duration_ms":69074,"significance":"If the reported results hold, the paper demonstrates that simple linear classifiers on multi-modal semantic embeddings provide a strong baseline for detecting AI-generated human-motion video, and the release of DeepAction is a useful benchmark for the community. Strengths include the action-level held-out split, macro-averaged evaluation over five repetitions, leave-one-out model generalization analysis, and evaluation on external datasets (DeepSpeak, GTA-Human, and Sora/Runway non-human videos). However, several load-bearing claims are currently overstated, particularly the abstract's robustness claim, the 'unsupervised' description of frame-to-prompt classification, and the non-human generalization result, which lacks a real-class control.","major_comments":[{"comment":"The abstract's claim that the method is 'robust' to laundering 'including resolution and compression attacks' is not supported by the paper's own evaluation. Table 3 shows that the only resolution-robustness experiment, a two-class linear SVM on CLIP embeddings, drops from 96.9% video accuracy at full resolution to 72.8% at 25% resolution and 51.0% at 10% resolution, the latter at chance for balanced classes. Furthermore, Table 3 tests only the generic CLIP embedding and does not report corresponding results for the FT-CLIP embedding or the polynomial-kernel SVM that produce the headline 99.2% video accuracy in Table 1, so there is no evidence that the best-performing method is resolution-robust. Please add robustness results for the best-performing configuration or revise the abstract and contribution claims to specify that the robustness to resolution holds only for the CLIP linear-SVM variant and only down to 50% resolution.","section":"Abstract and §5.1 (Table 3)"},{"comment":"The frame-to-prompt method is repeatedly described as requiring 'no explicit training' and as an unsupervised forensic technique, but its best performance in Table 2 is achieved with FT-CLIP, which was fine-tuned on the same DeepAction dataset using real/fake labels and captions like 'a REAL image' versus 'a FAKE image' (Appendix A.4). The 'no training' statement is therefore true only for the cosine-similarity decision rule, not for the underlying representation. Please state this explicitly, and report the frame-to-prompt results for the off-the-shelf CLIP, SigLIP, and JinaCLIP embeddings as the unsupervised baseline.","section":"§4.2 and Table 2"},{"comment":"The DeepAction real class consists of 100 Pexels videos, while the fake class consists of 3,100 AI-generated clips, and the paper reports macro-averaged accuracy over five repetitions without any variance estimate. Because the video-level accuracies for the real class are based on only 100 videos, claims of >95% video accuracy should be accompanied by confidence intervals or per-repetition/per-class results. In addition, because the real videos are matched to the AI prompts only on coarse action category, differences in camera motion, lighting, and compression between Pexels footage and AI-generated clips may inflate the reported accuracies relative to in-the-wild performance; this limitation should be acknowledged in the generalization discussion.","section":"§3 and §5"},{"comment":"The non-human-motion generalization experiment is missing a real-class control. Section 5.3 reports that the model 'correctly classified 97.1%' of 100 Sora/Runway videos as AI-generated, but all of these videos are AI-generated, so this number is a true-positive rate on the fake class, not an accuracy. Without a matched set of real non-human videos, the result cannot distinguish a genuinely semantic detector from a fake-biased classifier, and it does not support the claim that the model 'has not learned something specific to AI-generated human motion.' Please report a balanced accuracy or per-class recall/precision on a test set that includes real non-human videos from a comparable distribution.","section":"§5.3"}],"minor_comments":[{"comment":"The sentence 'the P1 paired prompt authentic image vs. AI-generated image' appears to mislabel the prompt pair: P1 is 'real photo vs. fake photo,' while the authentic/AI-generated pair is P3. Please correct the cross-reference.","section":"§5 (Table 2 discussion)"},{"comment":"The text states that the best frame-to-prompt result 'underperforms by only 6.6 percentage points' relative to 99.2%; the difference between 99.2% and the best frame-to-prompt video accuracy of 96.2% is 3.0 percentage points. Please correct this arithmetic.","section":"§5 (Table 2 discussion)"},{"comment":"The table header describes compression as 'compression ratio measured as bits per second,' which conflates bitrate with compression ratio. Please use 'bitrate' and clarify that lower values correspond to stronger compression.","section":"§5.1 (Table 3)"},{"comment":"The comparison to prior work mixes metrics: AUC from [Vahdati et al.] is compared with accuracy from this paper. Please report the same metric or explicitly caution that the numbers are not directly comparable.","section":"§5.6"},{"comment":"The PCA projections in Figures 3 and 4 do not report the amount of variance explained by the displayed two principal components; please add this information so the separations can be interpreted quantitatively.","section":"Figures 3 and 4"},{"comment":"The paper states that the code will not be open-sourced, which makes the reported preprocessing, fine-tuning, and SVM setups harder to reproduce; releasing the extraction and classification code would strengthen the reproducibility of the benchmark.","section":"Ethical Statement"}],"recommendation":"major_revision","confidential_remarks":"The DeepAction dataset and the systematic comparison of four CLIP-like embeddings with simple classifiers are useful contributions. The main issues are overclaiming in the abstract and in the 'unsupervised' and 'non-human generalization' narratives, all of which are fixable with additional experiments or careful rewording. The resolution-robustness contradiction in Table 3 in particular must be addressed before the paper can be considered for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a legitimate empirical contribution—a new 3,100-clip dataset of text-to-video human motion from seven generators plus matched real footage, a clean comparison of four CLIP-family embeddings, and the best evidence yet that semantic embeddings generalize across unseen generators. The core supervised detection is solid. But the abstract oversells resolution robustness, and the 'unsupervised' frame-to-prompt method is not as clean as advertised. Neither problem sinks the paper; both need fixing before publication.\n\nWhat's new: DeepAction itself is the main gift. Evaluating on action-level held-out splits with macro-averaged accuracy, five repetitions, and leave-one-out across generators is a step beyond the image-only CLIP deepfake papers. The FT-CLIP variant reaching 99.2% video accuracy on the in-distribution test is credible, and the leave-one-out 91.9–96.7% range is the strongest result in the paper. The compression robustness test is also genuinely good—92.3% at 10% bitrate.\n\nSoft spots:\n- The resolution-robustness claim in the abstract does not match what is tested. Table 3 only uses the generic CLIP embedding with a linear SVM; at 25% resolution video accuracy drops to 72.8%, and at 10% it is at chance. The authors' best FT-CLIP model is never evaluated under resolution reduction. The stress-test note is right: as written, 'robust to ... resolution attacks' is unsupported and partly contradicted by the paper's own table. This is fixable by testing the best method and by softening the claim.\n- The frame-to-prompt method is described as requiring no explicit training, but the only version that works (95.2% frame accuracy) uses FT-CLIP, fine-tuned on the same DeepAction dataset. With off-the-shelf CLIP, frame-to-prompt is at chance. Call it 'prompt-based classification with a fine-tuned encoder' or report the fine-tuned variant honestly as supervised in the embedding.\n- No code is released. The dataset is public, which is good; for a method paper, releasing the SVM training pipeline or at least embeddings would materially help replication.\n- Evaluation is careful but small on the real side: 100 Pexels videos. No error bars or confidence intervals on the five repetitions. I'd want those before relying on the 99.2% number.\n\nThe central argument—that semantic embeddings separate real from AI-generated human motion and generalize to new generators reasonably well—holds up. The robustness and no-training claims are overstated, not fabricated.\n\nThis paper deserves a serious referee. Send it out, but expect a major revision on framing. I'd bring it to reading group and would cite the dataset.","headline":"Solid dataset and cross-generator results for CLIP-based video deepfake detection, but the abstract overstates resolution robustness and the 'unsupervised' frame-to-prompt claim needs an honest rewrite.","tokens_in":13341,"tokens_out":4081,"would_cite":true,"duration_ms":78222,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Semantic embeddings that align video frames with text can tell real from AI-generated human motion with up to 99.2% video-level accuracy.","keywords":["AI-generated video detection","deepfake detection","CLIP embeddings","media forensics","text-to-video","human motion","semantic embeddings","DeepAction dataset"],"falsifier":"Train the two-class classifier on DeepAction, then test it on a fresh collection of real videos filmed with varied cameras, lighting, and compression levels, and on clips from a newly released text-to-video model not among the seven generators. If video-level accuracy for the real videos falls well below the reported 98.5% or for the unseen model below roughly 90%, the claimed robustness and generalization would be contradicted.","tokens_in":12390,"feed_emoji":"🎥","tokens_out":9626,"duration_ms":81920,"temperature":0.7,"pith_summary":"This paper claims that the semantic meaning of a video frame, as captured by multi-modal embeddings that align images with text, is enough to tell real footage of people from AI-generated footage of people. The authors build DeepAction, a dataset of 3,100 AI clips from seven text-to-video generators plus 100 matching real clips, and show that simple classifiers on these embeddings reach video-level accuracy as high as 99.2%. The signal survives heavy resolution and compression laundering, and it transfers to generators the classifier never saw, averaging 95.2% accuracy in leave-one-out tests. Even a training-free check that compares a frame to the text prompts 'real image' and 'fake image' reaches 96.2% video accuracy. If this holds, a lightweight semantic detector could be deployed quickly and keep working as generators improve.","feed_headline":"Semantic embeddings catch AI-made human video at 99.2%","feed_subtitle":"A CLIP-based detector stays accurate under heavy compression and generalizes to generators it never trained on.","key_machinery":"The load-bearing mechanism is the multi-modal semantic embedding: a neural network (CLIP or its variants SigLIP and JinaCLIP) that maps an image and a text caption into a shared vector space, so a video frame can be compared directly to words. The authors add a fine-tuned CLIP (FT-CLIP) trained on real/fake frame pairs captioned as 'a REAL image of {action}' versus 'a FAKE image of {action}', which sharpens the real/fake axis. Classification is deliberately simple: a support-vector machine over per-frame embeddings (two-class 'real vs fake' or eight-class 'which generator'), with a video labeled fake when most frames are fake, plus a training-free 'frame-to-prompt' classifier that assigns a frame to whichever of two text prompts ('authentic image' vs 'AI-generated image') has the higher cosine similarity. The DeepAction dataset, 3,100 AI clips from seven generators matched to 100 real stock clips on the action prompt, is the testbed that lets the authors measure both accuracy and generalization.","core_discovery":"The central discovery is that the multi-modal embedding space carries a dependable real-versus-fake axis for generated video, even though the embeddings were never trained for forensics. On the DeepAction dataset, a fine-tuned CLIP model followed by a polynomial-kernel SVM reaches 99.2% video-level accuracy, and a fully unsupervised cosine-similarity comparison to the text prompts 'real image' and 'fake image' reaches 96.2%. The separation survives dropping resolution to 25% and bitrate to 10%, and it transfers to a generator withheld from training with an average video accuracy of 95.2%. The same signal also flags 97.1% of non-human AI-generated videos as fake, while failing on talking-head deepfakes (55.8%) and CGI (39.1%). The authors read this as evidence that semantic embeddings capture a general, content-agnostic signature of whole-scene generative synthesis rather than a pixel-level artifact.","pith_inferences":["The authors do not say this, but the frame-to-prompt result implies the text prompts themselves define a real/fake axis in the embedding space; one testable extension is to use other prompt pairs (for example 'natural motion' versus 'synthetic motion') and see whether accuracy changes.","Because the method works on non-human content, the underlying signal may be a shared property of diffusion or latent-video generation rather than of human bodies; an extension would be to test the same embeddings on AI-generated audio spectrograms or synthetic 3D scenes.","The authors leave open whether adversarial perturbations in embedding space can fool the classifier; a concrete extension is to run white-box attacks on the SVM decision boundary to measure how much perturbation flips a fake video to 'real'.","Since the real clips come from a single stock-video source, deploying the detector on user-generated video with different cameras, motion blur, and stabilization would test whether the 98.5% real-video accuracy holds beyond clean Pexels footage."],"forward_implications":["Semantic-embedding classification keeps video-level accuracy above 90% even when the bitrate is cut to 10% of the original, so the detector can survive typical social-media re-encoding.","In leave-one-out tests, a classifier trained on six generators detects the seventh at an average video accuracy of 95.2%, so the detector is not tied to the quirks of one synthesis model.","The training-free frame-to-prompt variant reaches 96.2% video accuracy, so a usable detector could be deployed without collecting labeled real/fake pairs.","The same model flags 97.1% of AI-generated videos without humans as fake, suggesting the embedding captures a general sign of generative synthesis rather than a human-motion-specific artifact.","The detector does not transfer to talking-head deepfakes (55.8%) or CGI (39.1%), so it is a detector of full-body generated video, not a universal synthetic-video detector."],"supporting_citations":[{"why":"supplies the CLIP model whose embeddings are the paper's core representation.","marker":"[Radford et al., 2021]"},{"why":"provides the fine-tuning methodology used to build the best-performing FT-CLIP embedding.","marker":"[Khan and Dang-Nguyen, 2024]"},{"why":"is the low-level video detector whose poor generalization the paper contrasts with its own.","marker":"[Vahdati et al., 2024]"},{"why":"showed CLIP embeddings distinguish AI-generated images with a linear classifier, the direct precursor for the video approach.","marker":"[Cozzolino et al., 2024]"},{"why":"is the multimodal-LLM detector the paper cites as the semantic-reasoning baseline.","marker":"[Jia et al., 2024]"},{"why":"supplies the 100 real videos that define the real class in DeepAction.","marker":"[Pexels, 2024]"},{"why":"supplies the DeepSpeak talking-head deepfakes used to test non-generalization.","marker":"[Barrington et al., 2024]"},{"why":"supplies the GTA-Human CGI videos used to show the detector does not transfer to CGI.","marker":"[Cai et al., 2021]"}],"fun_headline_variants":["CLIP embeddings spot AI human motion at 99.2% accuracy","Semantic embeddings detect AI video even after heavy compression","Robust CLIP-based detector for AI-generated human video","AI video forensics embeddings generalize to unseen generators"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 100 real stock videos and 3,100 AI clips from seven generators are matched only on the action prompt, so the reported accuracy assumes these two collections represent how real and AI-generated human-motion videos actually look in the wild.","fun_headline_variants_meta":{"raw":{"variants":["CLIP embeddings spot AI human motion at 99.2% accuracy","Semantic embeddings detect AI video even after heavy compression","Robust CLIP-based detector for AI-generated human video","AI video forensics embeddings generalize to unseen generators"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001114,"raw_usage":{"total_tokens":4595,"prompt_tokens":859,"completion_tokens":3736,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":475,"completion_tokens_details":{"reasoning_tokens":3669}},"tokens_in":475,"tokens_out":3736,"duration_ms":26041,"temperature":1.0,"reasoning_tokens":3669,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:16:31.361210+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the two-class classifier on DeepAction, then test it on a fresh collection of real videos filmed with varied cameras, lighting, and compression levels, and on clips from a newly released text-to-video model not among the seven generators. If video-level accuracy for the real videos falls well below the reported 98.5% or for the unseen model below roughly 90%, the claimed robustness and generalization would be contradicted.","supporting_citations":[],"review_version":1}