{"id":"f039f6fa-8da4-4fec-8161-fa9a28e8148a","arxiv_id":"2608.06865","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A multi-agent forensic system with four specialized observation agents and a judge outperforms single-model MLLMs, including GPT and Gemini, on an out-of-domain deepfake video benchmark the authors built.","lead":"Researchers built a 100,000-video deepfake dataset with fine-grained text labels and a multi-agent AI system where four specialist agents (texture, lighting, motion, physics) report evidence and a judge agent decides real or fake. On unseen generators, the small open-source system beat larger closed-source models like GPT and Gemini on the benchmark's reported F1 and accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline F1 gain is confounded with training: the multi-agent system is fine-tuned on 68.9K labeled videos while GPT/Gemini baselines are zero-shot, so multi-agent reasoning may not be the cause.","rationale":"The reader's weakest assumption about label-conditioned annotation is legitimate and targets the validity of the supervision signal. I regard the comparison asymmetry as more load-bearing for the headline claim because the paper's own Table 5 shows that training, not architecture, accounts for most of the gap: the training-free multi-agent system is not ahead of the best closed-source baselines, and only after SFT+GRPO does it reach 53.28 F1. The claimed 'despite being composed entirely of small open-source MLLMs' phrasing invites the reader to compare model families, but the comparison conflates model family with training status. This is an addressable experimental gap rather than evidence of error: a fine-tuned single Qwen2.5-VL-7B control with identical supervision would settle it. Because the fix is straightforward and the current evidence does not eliminate the alternative explanation, the appropriate verdict remains CONDITIONAL.","tokens_in":28424,"tokens_out":8411,"duration_ms":74905,"concrete_test":"Fine-tune a single Qwen2.5-VL-7B using the exact SFT+GRPO recipe (LoRA, learning rate 1e-4 / 5e-6, one epoch, eight GRPO samples) on the same 68,906-video training split, providing the same sampled frames and the same per-dimension observation and explanation labels through a single holistic prompt (or the Multi-turn-All baseline with frames in all turns). Evaluate on the identical 7,636-video OOD split. If this equally trained single-model F1 is within a small margin of 53.28, the multi-agent structure is not the source of the headline gain; if it is substantially lower, the decomposition claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is that the headline comparison does not hold training status constant. In Table 2, 'Ours (w/ Video)' is a Qwen2.5-VL-7B multi-agent system trained with SFT+GRPO on 68,906 videos of FaceVid-Forensics-100K (Training Pipeline section), while GPT-4o, GPT-5-mini, and Gemini-2.5-Pro are evaluated zero-shot through their APIs, and Skyra and VideoVeritas use released checkpoints. The small vision models are retrained on the dataset, but they are not MLLMs and do not test the claim that 'small open-source MLLMs' beat closed-source MLLMs. The paper's own Table 5 shows the training-free multi-agent system reaches 42.29 F1 with video on Qwen2.5-VL-7B, below several zero-shot baselines, while the trained system reaches 53.28 F1. Consequently, the 5.83-point F1 gain over Gemini-2.5-Pro may come from supervised fine-tuning on 68.9K labeled videos rather than from four-perspective decomposition and judge reconciliation. Without an equally trained single-MLLM control, the central attribution claim is underdetermined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FaceVid-Forensics-100K, a 100,000-video deepfake dataset spanning 33 synthesis methods, with automatically generated textual observations in four forensic dimensions (texture, lighting, motion, physics) and verdict-consistent explanations. It also proposes a multi-agent forensic reasoning framework in which four specialized MLLM agents independently analyze these dimensions and a judge agent reconciles their outputs into a binary prediction and explanation. The main empirical claim is that, on a generator-disjoint out-of-distribution test set of 7,636 videos, the multi-agent system trained with SFT and GRPO on Qwen2.5-VL-7B achieves 69.87% accuracy, 81.82% recall, and 53.28% F1, outperforming all baselines including closed-source GPT and Gemini models evaluated zero-shot. The paper additionally reports ablations on the contribution of each agent, training stages, reasoning strategies, model combinations, and model scale.","tokens_in":28685,"tokens_out":7177,"duration_ms":61557,"significance":"If the results hold, the dataset is a substantial resource: 100K videos, 33 synthesis methods, generator-disjoint OOD evaluation, and per-dimension textual annotations with released prompts are valuable for the deepfake-detection community. The multi-agent decomposition is clearly described, and the ablation design is thoughtful, including training-free comparisons, model-scale experiments, and heterogeneous agent/judge combinations. The paper also ships detailed prompt templates and training dynamics in the appendix, which supports reproducibility. However, the headline comparison against closed-source MLLMs does not hold training status constant, the annotation pipeline is conditioned on ground-truth labels, and the OOD evaluation lacks a majority-class baseline and error bars. These issues leave the central attribution claim underdetermined and the reported margins potentially unquantified, so the paper needs substantial revision before its main claims can be accepted.","major_comments":[{"comment":"The central attribution claim is underdetermined because the comparison does not hold training status constant. In Table 2, GPT-4o, GPT-5-mini, and Gemini-2.5-Pro are evaluated zero-shot through their APIs, and the forensics-tuned MLLMs use released checkpoints, while 'Ours (w/ Video)' is Qwen2.5-VL-7B trained with SFT+GRPO on 68,906 videos of FaceVid-Forensics-100K. Table 5 shows that the training-free multi-agent system on the same backbone reaches only 33.53 F1 without video and 42.29 F1 with video, both below Gemini-2.5-Pro's 47.45 F1. The reported 5.83-point F1 gain may therefore come from supervised fine-tuning on the large labeled dataset rather than from the four-perspective decomposition and judge reconciliation. Please add an equally trained single-MLLM control (same backbone, same SFT+GRPO protocol, same data, but without the multi-agent decomposition) and an equally trained non-agent reasoning baseline, and report these controls in the main comparison table.","section":"Experiments, Tables 2 and 5"},{"comment":"The textual supervision used to train the observation agents is generated with access to ground-truth labels, so the agents may learn language-label priors rather than transferable visual forensic cues. The aggregation prompt in Listing 7 instructs DeepSeek-V4 Pro to 'use the ground-truth label solely as internal guidance to resolve conflicts' between models, and the explanation-label section states that aggregation is 'strictly conditioned on the ground-truth video label' with reverse inference when all annotators are wrong. Although the output text is required not to mention the label, the resulting observations and explanations are still functions of the label. The OOD detection metric is not formally circular because the test labels are independent, but the stronger claim that the agents acquire visually grounded, transferable forensic knowledge is not established by the present experiments. Please add an ablation that trains agents on annotations produced without ground-truth access, and test whether observation text alone (without the judge) predicts the label on OOD data at above-chance rates while controlling for the base MLLM.","section":"Label Generation, Listing 7"},{"comment":"The explanation-quality evaluation is conditioned on the ground-truth answer and a reference rationale, which inflates the reported quality scores. In Listing 8, both evaluator judges receive the ground-truth answer, the reference rationale, the model answer, and the model rationale, and the rubric explicitly assigns low scores to rationales that contradict the ground-truth label. This makes the explanation scores a measure of agreement with label-conditioned references rather than an independent assessment of forensic quality. The claim that the framework produces high-quality explanations is therefore overstated. Please report a blind evaluation in which judges do not receive the ground-truth label or reference rationale, or provide a human study on a subset of the OOD explanations.","section":"Appendix B.4, Listing 8"},{"comment":"The OOD evaluation is missing a majority-class baseline and error bars, which undermines the strength of the reported margins. The OOD test set contains 5,716 real and 1,920 fake videos, a 75.1% real base rate; predicting 'real' for every video yields 74.9% accuracy, which is higher than the reported 69.87% accuracy of the full system, and an undefined F1 (or 0 under the standard convention when no fakes are predicted). The paper should report majority-class accuracy and F1, per-class precision/recall, and a confusion matrix. In addition, all results are single runs without confidence intervals or significance tests, so the 5.83 F1 gain over Gemini-2.5-Pro and the 2.27 F1 gain from adding video frames to the judge cannot be distinguished from run-to-run variation. Please report multiple seeds or bootstrap confidence intervals for the main results.","section":"Experiments, Table 2 and Evaluation Protocols"}],"minor_comments":[{"comment":"The phrase 'ranks first across all reported metrics' is inaccurate on the OOD set because the reported accuracy of 69.87% is below the 74.9% always-real accuracy, and the in-domain results in Table 8 show small vision models outperforming the framework; please qualify this claim.","section":"Abstract and Section 4.2"},{"comment":"The OOD split contains only entire-face synthesis (EFS) generators, with no face-swap or face-reenactment examples, so the generalization claim should be scoped to entire-face synthesis rather than deepfake video detection in general.","section":"Figure 2 and Appendix A, Table 6"},{"comment":"The collection counts are typeset without separators ('442,000,6,780,000,3,000, and5,000') and should be corrected for readability and verifiability.","section":"Section 3.1, Collection and Processing"},{"comment":"The user prompt in Listing 7 passes 'This video is {ground_truth}' to the aggregator while the system prompt forbids mentioning the label in the output; this tension is the source of the label-conditioning issue and should be acknowledged explicitly in the dataset description.","section":"Listing 7"},{"comment":"The paper provides the project page URL but does not state whether the dataset and trained models will be released; please clarify data and code availability, since the dataset is a core contribution.","section":"Project page and reproducibility"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The benchmark is a real contribution, and the multi-agent framework is well constructed, but the headline claim—that small open-source MLLMs beat GPT/Gemini because of multi-agent reasoning—is not supported by the experiments as reported. The comparison trains one side and zero-shots the other.\n\nCredit where due: FaceVid-Forensics-100K is genuinely useful. 100K videos, 33 synthesis methods, a generator-disjoint OOD split, and four-dimensional textual labels. The annotation pipeline is thoughtful, though not without issues. The ablations in Tables 3–5 are internally consistent, and the per-generator results in Table 7 show the improvement is not driven by one generator. The four-perspective-plus-judge design is a sensible way to attack the problem, and Figure 8's training dynamics are honestly reported.\n\nNow the soft spots, in order of severity.\n\nOne: training status is confounded. The system is SFT+GRPO on 68.9K labeled videos; GPT-4o, GPT-5-mini, Gemini, Skyra, and VideoVeritas are zero-shot or off-the-shelf. Table 5 shows the training-free multi-agent system reaches 42.29 F1 with video on Qwen2.5-VL-7B, below several zero-shot baselines. So the 5.83-point gain over Gemini-2.5-Pro may be mostly from supervised fine-tuning, not from four-perspective decomposition. Without an equally trained single-MLLM control, the central attribution claim is underdetermined. This is the load-bearing issue.\n\nTwo: the textual supervision has a circularity smell. Listing 7 and the Label Generation section say the aggregator is \"guided by the ground-truth video label\" and performs \"reverse inference\" when all annotators are wrong. That means observation and explanation labels are conditioned on the answer. The detection evaluation itself is not circular—test labels are independent—but the explanation-quality results (Table 9) and the claim that agents learn transferable visual cues are weakened. The agents may be learning label-conditioned rationalizations rather than independent forensic evidence.\n\nThree: evaluation hygiene. No error bars, no repeated runs, no majority-class baseline, and the OOD set is 75/25 real/fake. These are minor relative to the training confound, but they should be fixed. Also, no dataset or code release; for a benchmark paper, that should be a condition.\n\nBottom line: the paper deserves a serious referee and likely conditional acceptance, assuming the authors add a trained single-MLLM control, release the data, and rerun with variance estimates. The benchmark is valuable regardless of how the attribution question resolves.","headline":"The benchmark is a real contribution, but the headline F1 gain is confounded with training: the multi-agent system is fine-tuned on 68.9K videos while closed-source baselines are zero-shot.","tokens_in":29225,"tokens_out":2291,"would_cite":true,"duration_ms":19124,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Decomposing deepfake video detection into four specialist agent perspectives plus a judge produces better cross-generator accuracy than holistic single-model judgment.","keywords":["deepfake video detection","multi-agent reasoning","multimodal large language models","out-of-distribution generalization","forensic video analysis","FaceVid-Forensics-100K","explainable forgery detection"],"falsifier":"Train the four observation agents on observation labels produced without telling the aggregator the ground-truth label, then evaluate on the same 20 unseen generators; if F1 drops to the level of the no-annotation baselines, the original labels were encoding the answer rather than transferable visual cues.","tokens_in":28206,"feed_emoji":"🎥","tokens_out":5776,"duration_ms":44352,"temperature":0.7,"pith_summary":"This paper argues that single deepfake detectors fail on new AI video generators because they judge the whole video at once, so subtle evidence spread across different cues gets drowned out. It proposes splitting the examination into four independent expert perspectives—texture, lighting, motion, and physics—and letting a judge model reconcile the four reports into a verdict and an explanation. To train such agents it introduces FaceVid-Forensics-100K, a dataset of 100,000 face videos from 33 synthesis methods with fine-grained text annotations for each of the four perspectives. On videos from 20 unseen generators, the multi-agent system built from small open-source models reports higher accuracy, recall, and F1 than every single-model baseline, including closed-source GPT and Gemini systems. If the finding holds, decomposition into specialist perspectives is a practical route to cross-generator deepfake detection without depending on the largest proprietary models.","feed_headline":"Four-agent team outperforms GPT and Gemini at deepfake detection","feed_subtitle":"Splitting forgery checks into texture, lighting, motion, and physics lifts F1 to 53.28% on unseen generators.","key_machinery":"The central mechanism is the decomposition of deepfake detection into four independent forensic perspectives—texture, lighting, motion, and physics—each handled by its own observation agent that reports cues without committing to a verdict, followed by a judge agent that reconciles the reports. The supporting machinery is FaceVid-Forensics-100K, whose automatically generated textual labels split observations into those same four dimensions, so each agent can be supervised on a single class of evidence. A separate aggregation step (five annotator MLLMs synthesized by a text-only aggregator, DeepSeek-V4 Pro) produces the observation and explanation labels, and GRPO refines the judge's decisions after supervised fine-tuning. The judge-with-video variant shows that the visual frames still add information on top of the text reports, while the text-only judge remains competitive.","core_discovery":"On the paper's out-of-domain test set—5,716 real and 1,920 fake videos from 20 generation methods held out of training—the full framework reports 69.87% accuracy, 81.82% recall, and 53.28% F1, compared with 47.45% F1 for the strongest single-model baseline, Gemini-2.5-Pro. The system consists of four observation agents, each a small open-source multimodal large language model (Qwen2.5-VL-7B) directed to one forensic dimension, plus a judge agent trained with supervised fine-tuning and group relative policy optimization. The judge receives the four textual reports and optionally the sampled frames, and produces a binary verdict with a supporting explanation. The paper interprets the result as evidence that explicit multi-perspective collaborative reasoning generalizes across unseen generators better than holistic judgment by a single MLLM, and that the benchmark's dimension-specific textual supervision is what makes the specialized agents learn transferable cues.","pith_inferences":["The headline result should be read with the supervision caveat: because the annotation pipeline conditions on ground-truth labels, the metric gap may partly reflect agents learning label-consistent language patterns rather than purely visual transferable cues; a blind-label variant would settle how much.","The same four-perspective decomposition could transfer to other forgery domains—audio deepfakes, image manipulation localization, or document forgery—where holistic judgment also tends to miss weak distributed evidence.","The OOD benchmark itself could be reused as a testbed for whether explanation quality tracks detection accuracy; the paper's own explanation scores show different judge models rank the system differently, so explanation gains are less settled than detection gains.","If label-conditioned supervision is indeed the key ingredient, then scaling the dataset's generator coverage may matter more than scaling model size, since the paper shows modest gains from 3B to 32B parameters compared with the jump from single-model to multi-agent reasoning."],"forward_implications":["Small open-source multi-agent systems can outperform closed-source single models for out-of-domain deepfake video detection, lowering the capability threshold for deployment.","The four-perspective decomposition is robust across generators: the framework ranks first or second on 11 of the 20 held-out generators after GRPO with video input.","Adding direct video access to the judge improves F1 from 51.01% to 53.28%, so textual expert reports are useful but not a full substitute for visual evidence.","The framework is modular: mixing different MLLMs for observation and judge roles remains competitive, meaning components can be upgraded independently.","Training the judge with GRPO after SFT raises both accuracy and F1, indicating that decision-policy refinement adds generalization beyond learning to produce evidence."],"supporting_citations":[{"why":"Supplies the Qwen2.5-VL-7B backbone used for all agents and several baselines.","marker":"Bai et al. 2025"},{"why":"Skyra sets the forensic-tuned MLLM baseline and the explanation-evaluation setup the paper follows.","marker":"Li et al. 2026b"},{"why":"VidGuard-R1 supplies the explanation-quality evaluation protocol with two independent judges and a 1–10 rubric.","marker":"Park et al. 2026"},{"why":"VideoVeritas serves as an annotator in label generation and as a forensic-tuned baseline.","marker":"Tan et al. 2026"},{"why":"DeepSeek-V4 Pro is the text-only aggregator that synthesizes observation and explanation labels.","marker":"DeepSeek-AI et al. 2026"},{"why":"Gemini 3.5 Flash acts as an annotator and as a closed-source baseline the framework must beat.","marker":"Google DeepMind 2026"},{"why":"GPT-4o acts as an annotator and as a closed-source baseline.","marker":"OpenAI 2024"},{"why":"AIGVDBench supplies most of the real videos and many generated videos in the dataset.","marker":"Ma et al. 2026a"},{"why":"GenVidBench supplies additional real and generated videos for the benchmark.","marker":"Ni et al. 2026"},{"why":"AltFreezing is used to prioritize harder-to-distinguish fake videos during dataset curation.","marker":"Wang et al. 2023b"}],"fun_headline_variants":["Four small models outsmart GPT and Gemini on deepfake videos","Texture, lighting, motion, physics: multi-agent deepfake detector wins","Open-source agents beat top closed MLLMs at unseen deepfakes","Multi-agent reasoning generalizes on deepfakes; single models lag","New 100K video benchmark and agent team top deepfake detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automatically generated text labels describe real visual evidence in the video, and are not just paraphrases of the known real-or-fake answer that the aggregator was allowed to see when resolving conflicts.","fun_headline_variants_meta":{"raw":{"variants":["Four small models outsmart GPT and Gemini on deepfake videos","Texture, lighting, motion, physics: multi-agent deepfake detector wins","Open-source agents beat top closed MLLMs at unseen deepfakes","Multi-agent reasoning generalizes on deepfakes; single models lag","New 100K video benchmark and agent team top deepfake detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001047,"raw_usage":{"total_tokens":4445,"prompt_tokens":1038,"completion_tokens":3407,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":654,"completion_tokens_details":{"reasoning_tokens":3315}},"tokens_in":654,"tokens_out":3407,"duration_ms":22195,"temperature":1.0,"reasoning_tokens":3315,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:28:38.843292+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the four observation agents on observation labels produced without telling the aggregator the ground-truth label, then evaluate on the same 20 unseen generators; if F1 drops to the level of the no-annotation baselines, the original labels were encoding the answer rather than transferable visual cues.","supporting_citations":[{"cited_title":"VidGuard-R1:","cited_arxiv_id":null,"evidence_quote":"VidGuard-R1 supplies the explanation-quality evaluation protocol with two independent judges and a 1–10 rubric."},{"cited_title":"L ong V ideo A gent: Multi-Agent Reasoning with Long Videos","cited_arxiv_id":null,"evidence_quote":"VideoVeritas serves as an annotator in label generation and as a forensic-tuned baseline."}],"review_version":2}