{"id":"41274a8c-9f0c-40ac-82d2-5143dd7e8149","arxiv_id":"2501.08514","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new dataset (FakeVE) and a benchmark model (MRGT) for generating natural-language explanations of why multimodal news videos are fake.","lead":"This paper introduces a new task and dataset for explaining why news videos are fake, with 2,672 annotated posts covering four manipulation types. It also presents a benchmark model (MRGT) that generates these explanations and compares it with large multimodal models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central risk is that FakeVE inherits veracity labels from FMNV and FakeTT without independent fact-checking; if those labels are noisy, all gold explanations and benchmark conclusions are contaminated.","rationale":"The reader's weakest assumption is precisely the one that matters most: the FakeVE gold labels are inherited from FMNV and FakeTT without independent fact-checking. I examined the dataset construction in Section 3.1 and found no procedure for verifying that a post is actually false beyond having annotators watch the video and read the title. That is insufficient for a task whose output is an explanation of falsity: a misleading title on a true video, or a video presenting an unverified but plausible claim, can produce a high-quality explanation of 'inconsistency' while the post is not actually fake. The paper's central contribution is the dataset, so a systematic label-quality problem would invalidate not only the dataset's intended use but also the training signal for MRGT and the G-Eval based quality comparisons. I found no evidence that the paper resolves this, and the absence of a dataset release mechanism makes the issue unfalsifiable from the paper alone. Because this is the same concern the reader identified, my independent read does not change the verdict: the paper should be accepted only conditionally, pending external validation of the inherited labels or a clear statement that the labels are provisional. I do not see a stronger technical flaw in the modeling contribution itself; the model claim is subordinate to the dataset claim and would need to be re-evaluated after the dataset audit.","tokens_in":14154,"tokens_out":2627,"duration_ms":28829,"concrete_test":"Sample 100 FakeVE posts stratified by source dataset (FMNV vs. FakeTT) and by annotated aspect. For each post, have two independent fact-checkers retrieve primary sources—news reports, official statements, and fact-checking databases such as Snopes or PolitiFact—and determine whether the post is genuinely false. Compute the positive predictive value (PPV) of the inherited veracity labels with a 95% confidence interval. If the lower bound falls below 0.90, the gold explanations cannot be trusted as a benchmark and the dataset construction must be revised to include external verification. As a secondary check, recompute the Table 2 comparison between MRGT and the best baseline on only the verified-fake subset to see whether the reported ranking persists when the label noise is removed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"FakeVE's 2,672 items are the union of all fake posts from FMNV (1,500) and FakeTT (1,172), and Section 3.1 describes no re-verification of those labels. The annotation scheme instructs annotators to watch the video and check the title before writing an explanation, but never asks them to consult external sources or to establish independently that the post is false. The reported Cohen's kappa of 0.865 measures agreement among annotators on the aspect/explanation, not agreement with an external ground truth. Consequently, any source-label false positives—such as true posts with exaggerated titles, or opinion content that contradicts no verifiable fact—propagate directly into FakeVE's gold explanations. The claim that FakeVE 'can definitively explain four real-life fake news video aspects' is therefore unsupported: the explanations are only as valid as the inherited veracity labels. The paper also says the code and dataset are released, but provides no access mechanism, which prevents the community from auditing the labels. This is not a stylistic issue; it is the load-bearing assumption for every result in Sections 3.3, 5.3, and 5.4, because both the evaluation of annotation quality and the supervision signal for MRGT depend on the reliability of these labels.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Fake News Video Explanation (FNVE) task, in which a multimodal news post (title, video frames, audio transcript) is used to generate a natural-language explanation of why the news video is false. The authors construct FakeVE, a dataset of 2,672 fake news video posts collected from the existing FMNV and FakeTT datasets, and manually annotate each post with an explanation and one of four error aspects (Contextual Dishonesty, Splice Tampering, Synthetic Voiceover, Contrived Absurdity). They report annotation statistics, analyze aspect distributions, and propose a Multimodal Relation Graph Transformer (MRGT) that combines a Transformer encoder, a graph convolutional network over a multimodal relation graph, and a BART decoder as a benchmark model. Experiments compare MRGT against several MLLM and fine-tuned baselines, with additional ablation studies on modality and graph contributions.","tokens_in":14418,"tokens_out":3471,"duration_ms":35866,"significance":"If the dataset labels are reliable, FakeVE would be a valuable first resource for the new task of natural-language explanation of fake news videos, going beyond binary veracity classification. The paper is clearly useful to the community in proposing a concrete task formulation, a detailed annotation protocol with good inter-annotator agreement (reported Cohen's kappa 0.865), and a reproducible baseline architecture. The exploratory analysis of aspect distribution and the qualitative case studies are helpful for understanding the task. However, the significance is conditional on two load-bearing assumptions: that the inherited veracity labels from FMNV and FakeTT are correct without independent re-verification, and that the LLM-based G-Eval scores used to validate annotation quality are themselves trustworthy. The absence of a dataset release mechanism also limits the community's ability to audit the resource.","major_comments":[{"comment":"The veracity labels of all 2,672 posts are inherited from FMNV and FakeTT without independent re-verification. The annotation instructions ask annotators to watch the video and check the title, but do not ask them to consult external sources or to independently determine whether the post is actually false. If any source-label false positives exist (e.g., true posts with exaggerated titles, opinion content, or satire), the gold explanations will be invalid, and both the annotation-quality evaluation in Section 3.3 and the supervised training signal for MRGT in Sections 5.3 and 5.4 will be contaminated. Please add a label re-verification step (e.g., fact-checking a random sample against external sources, or a two-stage annotation in which annotators first confirm falsity) and report the resulting agreement or disagreement rates.","section":"Section 3.1"},{"comment":"The evaluation of annotation quality and of generated explanations relies entirely on G-Eval, an LLM-based referenceless scorer, with no human validation of G-Eval's ratings on this specific domain and no error bars or statistical tests. Figure 4 reports scores such as 4.61 for annotators, but without knowing whether G-Eval agrees with human judgments for this task, the claim that 'manual annotation performs well' is not supported. Similarly, the model comparisons in Section 5.5 use the same unvalidated G-Eval protocol. Please add a human evaluation on a sample of explanations, report inter-judge agreement, and include variance or significance tests for the claim that MRGT has a 'significant advantage.'","section":"Section 3.3.1 and Figure 4"},{"comment":"The construction of the multimodal relation graph is not fully specified. The three rules in Section 4.2 state that 'semantically related text nodes' are linked via dependency tree relations, that edges are added between video frames and title tokens with 'highest modal similarity measured by cosine similarity,' and that 'visual nodes' are linked by adding an edge between video frame and audio transcript. It is unclear which tokens count as nodes, how many edges are added per rule, what threshold is used for cosine similarity, and how dependency relations apply to video-frame nodes. Without an exact adjacency-matrix definition, MRGT is not reproducible, and the source of the improvement attributed to the graph in Table 3 cannot be independently assessed. Please provide the full formal construction of A and include the release of code that implements it.","section":"Section 4.2, Eq. (2)"},{"comment":"The paper states 'we release our code and dataset,' but no repository URL, DOI, or other access mechanism is provided anywhere in the manuscript. Since the paper's central contribution is a new dataset, the community cannot audit the inherited labels, the annotation protocol, or the benchmark results without access to the data. Please include a working anonymous link or an availability statement in the camera-ready version.","section":"Section 1, Contribution bullet"},{"comment":"Table 2 reports single-run scores on a test set of only 267 posts, and the differences between MRGT and the best baseline are small on several metrics (e.g., BLEU-4 8.21 vs. 6.70 for GPT-4o, METEOR 91.62 vs. 86.68 for AMFM). No confidence intervals, multiple seeds, or significance tests are reported. The abstract's claim that MRGT 'has a significant advantage' therefore needs statistical support, especially given that the test set is small and the baselines include zero-shot MLLMs that were not fine-tuned.","section":"Section 5.2 and Table 2"}],"minor_comments":[{"comment":"The paper reports 'Cohen's Kappa coefficient (Fleiss, 1971)' but the Fleiss reference is for Fleiss' kappa for multiple raters; please clarify whether Cohen's kappa or Fleiss' kappa was computed and over how many annotators.","section":"Section 3.1"},{"comment":"There are inconsistencies in the abbreviations: Figure 2(a) uses 'CDS' while the text uses 'CD,' and 'the minimum number of cas' should be 'CA.'","section":"Figure 2 and Section 3.2"},{"comment":"The baseline names are inconsistent: Section 5.3 refers to 'PTSN' while Section 5.1 and Table 2 list 'NCT,' and the model is written as both 'HAA V' and 'HAAV.'","section":"Sections 5.1 and 5.3"},{"comment":"The term 'spurious explanation' appears several times (e.g., 'generate spurious explanations,' 'spurious inference') and should be replaced with 'generated explanation' or 'target explanation.' These appear to be translation artifacts.","section":"Section 4.3"},{"comment":"The prompt template P is described with placeholders but the actual prompt sent to the MLLMs is not shown in full; please provide the complete template and the formatting used for the audio transcript and video frames.","section":"Section 3.3.2"},{"comment":"There are several remaining typos and awkward phrasings, including 'Pering a news video' (Section 3.1), 'utuber' (Section 3.1), and 'violates basic common sense or logical errors' (Section 3.2). A careful proofreading pass is recommended.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The dataset-inheritance issue is the key risk. If the authors can show that their annotation process effectively re-verified the veracity labels on a sample, or if they provide evidence that FMNV and FakeTT labels have high precision on the subset used, the paper could be publishable after revision. The lack of a data release link is also a concrete blocker for a dataset-oriented paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the actual contribution is FakeVE, the first dataset that pairs fake news videos with natural-language explanations and a four-way error taxonomy. The MRGT model is a reasonable first baseline, not the main event.\n\nWhat the paper does well: the annotation setup is serious—four degree-holding annotators, a pilot round, Cohen's kappa of 0.865, and two independent annotators per post with the first author adjudicating disagreements. The four aspects (contextual dishonesty, splice tampering, synthetic voiceover, contrived absurdity) are a sensible organizing device, and the distribution analysis in Section 3.2 gives a useful sense of what these videos look like. The ablation study actually demonstrates that all three modalities and the graph help, which is real evidence for the model's design.\n\nThe soft spots are real but not fatal. The main one is that FakeVE takes veracity labels wholesale from FMNV and FakeTT. The annotators were told to watch the video and check the title, but there is no independent fact-checking against outside sources. If a source label is wrong—say, a true post with a misleading title—the gold explanation is wrong too. The phrase 'can definitively explain' in the abstract is too strong. The paper should either re-verify a sample against external sources or at least state the inherited nature of the labels as a limitation.\n\nTwo smaller issues. First, the paper says the code and dataset are released but gives no access mechanism; that needs a URL. Second, the evaluation relies on G-Eval, an LLM judge, and gives no error bars or human evaluation of model outputs. That is minor for a first baseline but should be acknowledged. Also, Section 5.3 references a baseline 'PTSN' that doesn't appear in Table 2; that's a typo but it confuses the comparison.\n\nWho this is for: researchers building explainable fake news detection systems. It is a starting point, not yet a community benchmark. With the label-verification issue addressed and the dataset actually accessible, it could become one.\n\nRecommendation: send it to peer review. The task and dataset are genuinely novel and the annotation quality is above average. Require the authors to address the inherited-label concern and provide the dataset access.","headline":"A useful first dataset and task for explaining fake news videos, but the gold labels inherit veracity from source sets without re-verification; fix that and the access issue before treating it as a benchmark.","tokens_in":14909,"tokens_out":3333,"would_cite":true,"duration_ms":32205,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces fake news video explanation as a new task, with a 2,672-post dataset and a graph-based model that outperforms existing baselines.","keywords":["fake news video explanation","FakeVE dataset","natural language explanation","multimodal misinformation","Multimodal Relation Graph Transformer","video veracity","explainable AI","misinformation detection"],"falsifier":"A fact-checking audit of a random sample of FakeVE posts against independent external sources would settle whether the gold explanations are reliable; a human preference study comparing MRGT explanations with GPT-4o's on accuracy and usefulness would settle whether the model's claimed advantage is real.","tokens_in":13960,"feed_emoji":"🎬","tokens_out":5486,"duration_ms":51062,"temperature":0.7,"pith_summary":"The paper proposes a new task, fake news video explanation (FNVE): given a news post with a title, video frames, and audio transcript, a system should generate a natural language explanation of why the post is false, rather than merely classifying it. To support this task the authors built FakeVE, a dataset of 2,672 fake news video posts with human-written explanations covering four aspects of falsity: contextual dishonesty, splice tampering, synthetic voiceover, and contrived absurdity. They also present a Multimodal Relation Graph Transformer (MRGT) that builds a graph over title, frame, and transcript nodes and uses graph propagation plus a BART decoder to generate explanations. Their experiments report that MRGT outperforms existing multimodal language models and fine-tuned baselines on standard generation metrics, and that manual annotations score higher than model explanations on LLM-based quality evaluation. If correct, this establishes the first benchmark for explainable fake news video analysis and a reusable resource for training verification systems.","feed_headline":"New dataset trains AI to explain why news videos are fake","feed_subtitle":"FakeVE adds 2,672 human-written explanations, and a graph model beats existing multimodal baselines in tests.","key_machinery":"The central objects are the FakeVE dataset and the MRGT architecture. The dataset supplies 2,672 posts labeled with one of four falsity aspects and a human reference explanation; the four-aspect taxonomy is the annotation scheme that organizes what counts as an explanation. The MRGT carries the argument: it concatenates title, sampled video frames, and audio transcript tokens, encodes them with a Transformer, builds a multimodal relation graph whose edges link semantically related text tokens, frames matched to title tokens by cosine similarity, and frames to transcripts, then applies a graph convolutional network with a residual connection before a BART decoder generates the explanation autoregressively.","core_discovery":"The paper's central claim is that the falsity of a multimodal news video can be captured in a short natural language explanation, and that this explanation can be generated automatically from the title, video frames, and audio transcript. This claim rests on FakeVE, which the authors present as the first dataset to definitively explain four real-life fake news video aspects, with 2,672 posts inherited from existing fake news video datasets and explanations written by trained annotators. The paper further claims that MRGT, by representing cross-modal relations as a graph and propagating information through a graph convolutional network before decoding with BART, has a significant advantage in capturing the multimodal feature association of video and generating logically coherent explanatory text, supported by higher BLEU, ROUGE, METEOR and Sentence-BERT scores than all compared baselines and by ablations showing the graph and each modality contribute. It also claims that the human annotations are high quality, with inter-annotator agreement of 0.865 and higher G-Eval scores than GPT-4o and Qwen2-VL on persuasiveness, informativeness, soundness, and readability.","pith_inferences":["The four-aspect taxonomy could be extended to other languages and platforms, but the annotation protocol would need external fact-checking to verify the source labels.","The paper's evaluation relies on automatic n-gram metrics and G-Eval; a human preference study on explanation accuracy would test whether MRGT's higher scores correspond to more useful explanations.","A natural testable extension would be to use the explanations as supervision for detecting the specific manipulation type, not just generating text.","Because FakeVE inherits its veracity labels from FMNV and FakeTT, the dataset's quality is bounded by the accuracy of those source labels; a fact-check audit could quantify that bound."],"forward_implications":["FNVE becomes an available benchmark task: future work can train and compare explanation generators on FakeVE using standard metrics.","The four-aspect taxonomy gives content moderation systems a concrete vocabulary for why a video is false, not just that it is false.","The dataset's explanations can be used as fine-grained evidence chains to train models to recognize specific tampering techniques such as splice tampering and synthetic voiceover.","The reported results imply that combining a relation graph with a pretrained encoder-decoder improves explanation coherence enough to beat strong multimodal language models on this task."],"supporting_citations":[{"why":"Supplies the FMNV news video posts from which part of FakeVE is drawn.","marker":"Wang et al., 2025"},{"why":"Supplies the FakeTT TikTok posts from which the rest of FakeVE is drawn.","marker":"Bu et al., 2024"},{"why":"Provides the BART pretrained encoder-decoder used as the backbone of MRGT.","marker":"Lewis et al., 2020"},{"why":"Provides G-Eval, the referenceless LLM-based method used to assess explanation quality.","marker":"Liu et al., 2023c"},{"why":"Provides GPT-4o, one of the multimodal large language model baselines compared against MRGT.","marker":"Hurst et al., 2024"},{"why":"Provides Qwen2-VL, another multimodal large language model baseline used in the evaluation.","marker":"Wang et al., 2024b"},{"why":"Supplies the inter-annotator agreement coefficient used to validate the consistency of FakeVE annotations.","marker":"Fleiss, 1971"},{"why":"Supplies the residual connection and Transformer architecture components adopted in MRGT.","marker":"Vaswani et al., 2017"}],"fun_headline_variants":["FakeVE dataset teaches AI to explain fake video reasons","Graph model justifies fake news video verdicts clearly","AI system reveals the why behind fake news videos","New dataset fuels natural language fake video explanations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every one of the 2,672 posts inherited from FMNV and FakeTT is genuinely fake, and that an annotator watching the video and reading the title can identify the falsity without external fact-checking.","fun_headline_variants_meta":{"raw":{"variants":["FakeVE dataset teaches AI to explain fake video reasons","Graph model justifies fake news video verdicts clearly","AI system reveals the why behind fake news videos","New dataset fuels natural language fake video explanations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00049,"raw_usage":{"total_tokens":2426,"prompt_tokens":977,"completion_tokens":1449,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":1389}},"tokens_in":593,"tokens_out":1449,"duration_ms":11105,"temperature":1.0,"reasoning_tokens":1389,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:24:00.639997+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A fact-checking audit of a random sample of FakeVE posts against independent external sources would settle whether the gold explanations are reliable; a human preference study comparing MRGT explanations with GPT-4o's on accuracy and usefulness would settle whether the model's claimed advantage is real.","supporting_citations":[{"cited_title":"Measuring nominal scale agreement among many raters","cited_arxiv_id":null,"evidence_quote":"Supplies the inter-annotator agreement coefficient used to validate the consistency of FakeVE annotations."}],"review_version":1}