{"id":"d5e6b887-d0d2-42c2-af50-fed5a0aafea9","arxiv_id":"2509.08008","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"GroundLie360 is a 2,044-video Snopes-derived benchmark with fine-grained annotations localizing six types of multimodal misinformation; the VLM prompting baseline FakeMark shows the task remains difficult.","lead":"This paper introduces a new task and dataset, GroundLie360, for localizing misinformation within videos across text, speech, and visuals, along with a VLM-based baseline called FakeMark. The resource supports fine-grained grounding, not just binary fake/real labels, for explainable misinformation detection.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth label generation is underspecified: Snopes rating-to-binary mapping and annotator agreement for veracity/type are missing, so benchmark validity is unverified.","rationale":"The reader's weakest assumption focuses on missing inter-annotator agreement for binary and type labels. I agree that this is a serious gap, but I also identify a more specific and potentially more damaging issue: the automatic labeling from Snopes ratings is underspecified. Snopes uses non-binary ratings, and the paper does not explain how they are converted to Real/Fake. This is not merely a missing statistic but a potential source of systematic label error. For example, if 'Mixture' or 'Unproven' videos are treated as Fake, many labels would be questionable even with perfect annotator agreement. The paper also contains an apparent contradiction between automatic labeling (Section 3.1) and human annotation with third-expert resolution (Section 3.2), which highlights the ambiguity. This concern is load-bearing because the benchmark's only value is its ground truth; if that is unreliable, all baseline results and difficulty claims are meaningless. I judge the reader's conditional verdict to be appropriate: acceptance should require release of the dataset, the Snopes-to-binary mapping, and label reliability statistics. My additional concern strengthens the case for conditionality but does not change the verdict, so I select UNCHANGED. The concrete test—re-annotation of a random subset combined with an audit of the binarization rule—would determine whether the concern lands.","tokens_in":11641,"tokens_out":4117,"duration_ms":49016,"concrete_test":"Request from the authors the per-video original Snopes rating, the exact rule mapping Snopes ratings to binary veracity, and per-annotator decisions for binary and type labels. Independently re-annotate a random sample of 100 videos with two fresh annotators using the stated guidelines and the linked Snopes article, then compute Cohen's kappa for binary veracity and for each fake type. If kappa for binary is below 0.6 or for any type is below 0.5, the label reliability assumption fails. Alternatively, if the provided mapping shows that 'Mixture' or 'Unproven' items were labeled Fake, the binarization is unjustified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that GroundLie360 is the first reliable real-world benchmark for grounding multimodal misinformation depends entirely on the trustworthiness of its labels. Section 3.1 states that Level 1 binary veracity is 'automatically labeled as Real or Fake based on fact-check ratings (e.g., from Snopes),' yet Section 3.2 says two annotators independently annotate each sample and a third expert resolves disagreements on 'binary veracity and fake types.' These statements are in tension: if the binary label is automatic, what do annotators disagree about? More importantly, Snopes ratings are not binary; they include categories such as 'Mixture,' 'Unproven,' 'Mostly true,' and 'Outdated.' The paper never specifies how these ratings are collapsed into Real/Fake, nor does it report inter-annotator agreement (e.g., Cohen's kappa) for binary veracity or fake-type labels—only IoU for grounding spans is reported (Section 3.2). Since every evaluation in Section 5 is computed against these labels, an arbitrary binarization or unstable type labels would directly invalidate the benchmark's conclusions, including the claim that the task is 'challenging' because FakeMark performs poorly. Until the exact labeling protocol and label reliability statistics are released, the dataset's validity cannot be assessed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper defines a new task, Grounding Multimodal Misinformation (GroundMM), and introduces GroundLie360, a dataset of 2,044 Snopes-derived real-world videos with three annotation levels: binary veracity, six fake types, and fine-grained grounding of text, speech, temporal edits, and visual regions. The authors also propose FakeMark, a training-free VLM/LLM pipeline that performs binary classification, multi-label fake-type classification, and multimodal grounding. Experiments report that the task is challenging: FakeMark achieves macro F1 29.23 for sub-type classification and low grounding F1 across most modalities, which the authors interpret as evidence that the benchmark is non-trivial.","tokens_in":11947,"tokens_out":6412,"duration_ms":80997,"significance":"If the dataset is released with reliable labels, this is a useful contribution. Existing DGM4 datasets are limited to image-text pairs, and video deepfake datasets typically target a single manipulation type; a real-world video benchmark with token-level, temporal, and spatial grounding would fill a real gap. The paper is also honest about the low performance of its baseline, which is appropriate for a new benchmark. However, the value of the contribution is currently unverifiable: the dataset is not accessible, the label-construction protocol is underspecified, and inter-annotator reliability is reported only for grounding spans. These issues directly affect the benchmark's central claim of being a reliable first benchmark for grounding video misinformation.","major_comments":[{"comment":"Ground-truth label construction is underspecified in a load-bearing way. §3.1 says Level 1 is 'automatically labeled as Real or Fake based on fact-check ratings (e.g., from Snopes)', but §3.2 says two annotators independently annotate each sample and a third expert resolves disagreements on 'binary veracity and fake types'. If the L1 label is automatic, it is unclear what annotators disagree about; if they are validating the automatic mapping, that procedure is not described. Moreover, Snopes rulings are not binary (e.g., 'Mixture', 'Unproven', 'Mostly true'), and no mapping from those ratings to Real/Fake is given. The only inter-annotator statistic reported is IoU on grounding spans (0.74/0.84/0.98); no agreement measure is reported for L1 or L2. Since all results in §5 are computed against these labels, an arbitrary binarization or unstable type labels would directly affect the benchm","section":"§3.1–3.2"},{"comment":"The evaluation protocol is not fully specified, making the reported numbers non-reproducible. First, the Problem Definition in §4 defines the fake-type vector c only when y=1; it is not stated whether sub-type metrics are computed only for fake instances, whether a 'Real' prediction contributes zeros to all sub-types, or how multi-label predictions are thresholded. Second, token-level textual grounding lacks a matching criterion (exact span, partial overlap, or token F1 independent of position). Third, frame-level temporal grounding lacks a tolerance for matching predicted transition indices to ground-truth edited timestamps. These definitions are essential for a benchmark; otherwise other researchers cannot apply the metric consistently.","section":"§5.1, Tables 4–5"},{"comment":"The dataset is the central deliverable, yet the manuscript only states it 'will be released' at a GitHub URL and provides no sample annotations, annotation guidelines, or exact per-type counts in the paper or supplement. I could not audit the claimed 2,044 videos, the 995T/1049F split, or the grounding distributions from the text alone. For a dataset/benchmark paper, accessibility and reproducibility of the dataset are load-bearing. Please make the data and annotations available for review, or include a detailed datasheet/supplement with statistics, sample entries, and the annotation instrument.","section":"Abstract, §3, §6"},{"comment":"The zero-shot claim is weakened by possible pretraining contamination and by an incomparable evaluation setting. FakeMark uses GPT-4o-mini and InternVL-8B, both trained on large public web corpora; GroundLie360 is derived from Snopes pages spanning 2016–2024, so the model may have memorized the fact-check outcomes for some samples. The paper does not test on a subset with a post-training cutoff or otherwise assess contamination. In addition, Table 3 reports FakeMark on the entire dataset but SVFEND and FakingRecipe on a 70/15/15 split, so the rows are not directly comparable. Please report results on the same test partition and address contamination, for example by evaluating on a recently fact-checked held-out set or by ablating with evidence retrieval disabled.","section":"§5.2"}],"minor_comments":[{"comment":"The statistical-analysis text contains confusing figure references. The sentence 'As shown in Figure 1 2' appears garbled, and the discussion of text vs. video segment counts assigns Figure 3(j) to text and Figure 3(k) to video, while the caption defines (i) as per-sample fake text segments and (j) as per-sample fake video segments. Please correct the cross-references.","section":"§3.3"},{"comment":"The ASR/transcription pipeline is not described. Since false-speech grounding is a core modality of the benchmark, please specify which ASR system was used, its language coverage, and whether transcript errors were corrected during annotation.","section":"§4"},{"comment":"The table header is dense and abbreviations such as 'M.A.', 'Annotation', and 'Range' are ambiguous. A legend or table notes section would improve readability.","section":"Table 1"},{"comment":"GPT-4o-mini is cited via the GPT-4 Technical Report [1]; if GPT-4o-mini is used, cite its model documentation or API specification for reproducibility.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"I would make final acceptance conditional on the dataset and annotation materials being made available to reviewers, and on the authors addressing the Snopes rating-to-binary mapping and L1/L2 agreement. The first-benchmark claim is plausible but currently rests on unverifiable annotation quality."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I think this paper is genuinely useful. It defines GroundMM, extends DGM4 from image-text to real-world video, and ships a dataset with a sensible three-level annotation scheme: binary veracity, six fake types, and grounding across title, speech, and video. The grounding annotations are unusually fine-grained, including temporal spans, bounding boxes, and token spans, and the construction from Snopes gives an external evidence base. The baseline FakeMark is a reasonable first attempt, and I credit the authors for reporting low scores without dressing them up. The scene-segmentation-plus-SAM2 pipeline is a sensible design for temporal and spatial grounding.\n\nThe soft spots are real, though. The label generation protocol is not specified precisely enough. Section 3.1 says L1 is 'automatically labeled as Real or Fake based on fact-check ratings (e.g., from Snopes),' but Section 3.2 says two annotators independently annotate each sample and disagreements on binary veracity and fake types are resolved by a third expert. Those two descriptions need to be reconciled: do annotators override the automatic label, or is the automatic label a starting point? More importantly, Snopes ratings are not binary – there are 'Mixture,' 'Mostly true,' 'Unproven' – and the paper never says how those are collapsed into Real/Fake. The paper reports IoU for grounding spans but no inter-annotator agreement (e.g., Cohen's kappa) for the veracity and type labels. Since every baseline result is computed against those labels, that missing reliability data is a load-bearing gap.\n\nThe dataset and code are also not released yet. That is common for in-press conference papers, but it means the central deliverable cannot be inspected right now. The 'first comprehensive benchmark' claim is currently plausible – DGM4 is indeed limited to synthetic image-text pairs – but it rests on those unverified labels. The paper also has a few minor figure-reference slips (Section 3.3 references Figure 1 for counts that look like they should be in Figure 3), but those are cosmetic.\n\nWho should read this: anyone building video misinformation datasets or grounding systems. It does not need to be right in every detail to be a useful benchmark, but it does need the label protocol and agreement numbers to be credible. I'd send it to review with a request for that information, and be willing to cite it once it's out.","headline":"Worth engaging: a new grounded-video-misinformation benchmark that needs its labeling protocol opened up before the 'first' claim can be fully trusted.","tokens_in":12413,"tokens_out":2867,"would_cite":true,"duration_ms":30191,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces GroundLie360, the first real-world benchmark that localizes falsehoods in misinformation videos across title, speech, and visuals.","keywords":["multimodal misinformation","grounding","video fact-checking","vision-language model","fake news dataset","temporal grounding","cross-modal inconsistency","benchmark"],"falsifier":"Re-annotate a stratified sample of the 2,044 videos with an independent panel and report Cohen's kappa for veracity and for each of the six fake types; if kappa falls well below the levels implied by the annotation pipeline, or if FakeMark's ranking changes when evaluated on the adjudicated subset, the core claim of a reliable grounding benchmark gives way.","tokens_in":11581,"feed_emoji":"🎯","tokens_out":5221,"duration_ms":56972,"temperature":0.7,"pith_summary":"The paper's goal is to make multimodal misinformation detection explainable by benchmarking not just whether a video is fake, but where the fake is. It defines the Grounding Multimodal Misinformation (GroundMM) task and introduces GroundLie360, a dataset of 2,044 real-world fact-checked videos with labels for six misinformation types and fine-grained spans: false text in the title or transcript, edited timestamps, and bounding boxes in frames. The authors also build FakeMark, a question-driven vision-language-model pipeline that classifies and grounds in one pass. If the dataset is sound, it gives researchers a common yardstick for systems that must point to the false element while saying why it is false, not just flag a video. The paper's experiments show the task is hard: current general-purpose models lag well behind specialized binary detectors and grounding scores remain low.","feed_headline":"First benchmark pinpoints fake video segments across modalities","feed_subtitle":"True/false labels come with spans marking exactly which title, speech, or video segment is false.","key_machinery":"The load-bearing object is the three-level annotation schema: Level 1 binary veracity from Snopes ratings, Level 2 six fake types (false title, false speech, temporal edit, CGI, contradictory content, unsupported content), Level 3 grounding (text token spans, frame-index intervals, bounding boxes). On the method side, FakeMark's four prompt-based analyses (text, video temporal, video spatial, cross-modal) produce analyses that a binary classifier, multilabel classifier, and four localizers consume; scene segmentation (TransNetV2) and object tracking (SAM 2) anchor temporal and visual grounding.","core_discovery":"On its own terms, the central discovery is that grounding video misinformation can be turned into a measurable benchmark. GroundLie360 covers 2,044 target videos (995 real, 1,049 fake) drawn from 1,466 Snopes-investigated events, annotated at three levels: binary veracity, six fake types, and grounding. Each video can carry multiple type labels, and annotations locate false spans in text, speech, temporal edit timestamps, and CGI regions. The paper argues this is the first comprehensive benchmark for grounding video misinformation, extending the earlier DGM4 text-image manipulation task to real video posts spanning visual, auditory, and textual modalities.","pith_inferences":["The dataset's Snopes-derived evidence could be repurposed as retrieval ground truth for end-to-end fact-checking models, though the paper itself only uses it to create labels.","If fake types co-occur as reported, a natural next benchmark would isolate error propagation: evaluate grounding conditioned on oracle type labels to separate 'failure to detect' from 'failure to localize' — the paper's case study hints at this but does not quantify it.","The annotation subjectivity problem suggests a testable stress test: re-annotate a subset with an independent panel and measure type-label agreement; the paper does not report this, so the benchmark's ceiling is not yet known.","Because most fake video segments are short and near the start, systems optimized for early-frame evidence may get disproportionate gains; this is an empirical prediction, not a paper claim."],"forward_implications":["Researchers get a public benchmark that measures localization quality, so future systems can be compared on whether they identify the exact false title, false utterance, edit point, or CGI region.","The six-type taxonomy lets downstream work study how misinformation types co-occur; the paper reports most samples contain at least two fake types, so multi-label evaluation becomes standard.","Because grounding is tied to upstream binary and type classification, pipeline evaluation makes error propagation visible; fixing early classification is a precondition for localization.","The low baseline scores (e.g., temporal-edit grounding F1 of 2.37) define clear headroom that motivates specialized grounding architectures."],"supporting_citations":[{"why":"Supplies a prior short-video fake news dataset and baseline method derived from fact-checking platforms, used for comparison.","marker":"[3]"},{"why":"Establishes the practice of building a dataset from fact-checking website content, the data-provenance route GroundLie360 follows.","marker":"[23]"},{"why":"FakeSV is the prior short-video benchmark with only video-level binary labels, used as a comparison point.","marker":"[10]"},{"why":"TRUE provides explainable fact-checking annotations without fine-grained grounding, a key contrast for the new benchmark.","marker":"[8]"},{"why":"FakeNVE offers explanation-based annotations but lacks precise localization, used as a comparison in the dataset table.","marker":"[5]"},{"why":"DGM4 defines the prior detecting-and-grounding task on text-image pairs that GroundMM extends to real video with speech and temporal grounding.","marker":"[15, 16]"},{"why":"TransNetV2 supplies scene segmentation that anchors temporal-edit and CGI grounding to scene transitions.","marker":"[17]"},{"why":"SAM 2 provides object tracking used to refine bounding-box and temporal spans in CGI localization.","marker":"[13]"}],"fun_headline_variants":["New benchmark localizes fake video segments across modalities","First benchmark to ground multimodal misinformation spans","Real-world video benchmark pinpoints false moments in text, speech, imagery","GroundLie360: new dataset for grounding fake video claims","Multimodal benchmark pinpoints misleading spans in videos"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The ground truth rests on Snopes ratings plus annotator judgment, and the paper reports agreement only for grounding spans (IoU), not for the binary veracity or fake-type labels; if those labels are unstable, every benchmark score measured against them is unstable.","fun_headline_variants_meta":{"raw":{"variants":["New benchmark localizes fake video segments across modalities","First benchmark to ground multimodal misinformation spans","Real-world video benchmark pinpoints false moments in text, speech, imagery","GroundLie360: new dataset for grounding fake video claims","Multimodal benchmark pinpoints misleading spans in videos"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000871,"raw_usage":{"total_tokens":3562,"prompt_tokens":652,"completion_tokens":2910,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":396,"completion_tokens_details":{"reasoning_tokens":2833}},"tokens_in":396,"tokens_out":2910,"duration_ms":22464,"temperature":1.0,"reasoning_tokens":2833,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T23:28:24.423786+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a stratified sample of the 2,044 videos with an independent panel and report Cohen's kappa for veracity and for each of the six fake types; if kappa falls well below the levels implied by the annotation pipeline, or if FakeMark's ranking changes when evaluated on the adjudicated subset, the core claim of a reliable grounding benchmark gives way.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"TRUE provides explainable fact-checking annotations without fine-grained grounding, a key contrast for the new benchmark."}],"review_version":1}