{"id":"56c53afd-922b-4770-9633-34c17ba006e9","arxiv_id":"2411.10032","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"VMID combines speech-to-text, OCR subtitle extraction, and video captioning into a single text prompt for a LoRA-tuned LLM, and reports 90.93% accuracy on the FakeSV short-video misinformation benchmark.","lead":"This paper describes VMID, a system that converts short videos into text from subtitles, audio, and visual descriptions, then asks a large language model to classify the video as real, fake, or debunking. The authors report a large accuracy gain over prior baselines on the FakeSV dataset, but the paper omits implementation details, code, and error bars, making the result difficult to verify.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported FakeSV gains may reflect LLM memorization of video fact-check outcomes rather than multimodal fusion; no contamination check is reported.","rationale":"The reader's weakest assumption is exactly the load-bearing concern: the evaluation assumes the LLM's pretrained parametric knowledge does not contain the ground-truth labels of FakeSV test videos. The paper itself highlights this risk in the case study of Fig. 5(b), where correct detection is attributed to the LLM's external knowledge rather than to video content. Because FakeSV consists of real-world videos whose veracity has been fact-checked, contamination through pretraining is a concrete, plausible mechanism, not a vague methodological worry. If the zero-shot contamination test shows high accuracy without any fine-tuning or with metadata-only prompts, then the 90.93% result measures recall of pretraining data, not the effectiveness of multimodal fusion, and the paper's central claim collapses. If the test instead shows near-chance zero-shot accuracy and a large drop with stripped prompts, the concern would be resolved, but the paper still lacks code, data splits, and implementation details needed for independent verification. Therefore the reader's REJECT verdict is appropriate and no verdict adjustment is needed. The concrete test proposed here would decisively separate the contamination explanation from the multimodal-fusion explanation, and the authors should be required to report it before any acceptance decision.","tokens_in":16405,"tokens_out":4078,"duration_ms":46846,"concrete_test":"Run a zero-shot contamination check on the exact FakeSV test prompts before any LoRA fine-tuning. For each test video, feed the base Qwen2.5 and GLM4 models the same integrated prompt used by VMID, with the classification instruction but no fine-tuning, and measure accuracy and macro F1. Then repeat with a stripped prompt containing only the video title and metadata, omitting subtitles, Whisper transcripts, and CogVLM2 summaries. If zero-shot accuracy is already near 90%, or if title/metadata-only accuracy is near 90%, the reported gain is explained by memorized fact-check knowledge rather than multimodal fusion. Additionally, evaluate on FakeSV videos whose events postdate each LLM's pretraining cutoff; a large accuracy drop on those videos would confirm contamination.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Central claim: VMID's multimodal fusion yields 90.93% accuracy and 90.89% macro F1 on FakeSV, roughly 10 to 19 absolute points above non-LLM baselines. This claim requires that the LoRA-tuned LLM's predictions are driven by the extracted video modalities, not by ground-truth answers already present in pretraining. The paper never checks this. Section V-D and Fig. 5(b) explicitly credit 'the knowledge base of the large language model' for a correct detection and contrast VMID with SV-FEND, which lacks 'external knowledge support.' Because FakeSV is built from real short-video posts about real events, web-scale pretraining corpora for Qwen2.5 or GLM4 very plausibly contain the same videos' fact-check status, debunking articles, or rumor discussions. If so, the model can answer from parametric memory of the label, making the multimodal prompt and LoRA tuning incidental. The 9.87 to 19.6 point gap over SV-FEND, a model without parametric memory, is exactly what such contamination would look like. No code, data splits, or baseline reproduction details are provided, and Section III-B states the method is described 'at a high level, without delving into specific implementation details,' so the result cannot be independently checked. The central claim therefore rests on an unverified assumption that the test labels are unknown to the base LLM.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VMID, a pipeline that extracts subtitle text (VSE), audio transcripts (Whisper), visual descriptions (CogVLM2), and metadata from short videos, concatenates these into a single natural-language prompt, and feeds the prompt to a LoRA-tuned LLM (Qwen2.5, GLM4, InternLM2.5, or Baichuan) for three-way classification as real, fake, or debunking. The central empirical claim is that on the FakeSV dataset VMID achieves 90.93% accuracy and 90.89% F1, outperforming SV-FEND (81.05%) and other baselines by roughly 9.87 to 19.6 absolute percentage points (Table I), with stable results across LLM backbones (Table II). The paper also presents two case studies, one correct and one incorrect prediction, and argues that VMID demonstrates the value of multimodal fusion and LLM knowledge for short-video misinformation detection.","tokens_in":16619,"tokens_out":5909,"duration_ms":61252,"significance":"Short-video misinformation detection is a timely and practically important problem, and the idea of normalizing heterogeneous modalities into a single text prompt for an LLM is a reasonable engineering contribution. If the reported gains over SV-FEND and SVRPM are real and attributable to the VMID fusion pipeline, the result would be practically useful. The paper's strengths are the breadth of baselines considered and the consistent performance across four LLM backbones in Table II. However, the empirical evaluation as written does not establish the claim: there is no contamination analysis, no error bars or significance tests, no train/test split details, no code release, and the one success case explicitly credits the LLM's parametric knowledge rather than the video content. The significance of the contribution therefore cannot be assessed until these gaps are closed.","major_comments":[{"comment":"The main result is reported as a single run without any description of the train/test split, the number of videos, the LoRA hyperparameters (rank, learning rate, number of epochs, batch size), or actual values for the parameters that appear as placeholders in Equations (8) and (10) (“segment duration seconds”, “filter threshold”). Section III-B explicitly says the method is described “without delving into specific implementation details,” but the paper's second contribution claims a full implementation. Without these details, with no error bars or significance tests, and with no code or data release, the 90.93% accuracy cannot be independently verified. Please provide the complete configuration, release the code, and report mean and standard deviation over multiple runs.","section":"Section V-C, Table I; Section IV"},{"comment":"The success case in Fig. 5(b) credits “the knowledge base of the large language model” for the correct detection and contrasts VMID with SV-FEND, which lacks “external knowledge support.” Because FakeSV is built from real short-video posts with known fact-check outcomes, the web-scale pretraining corpora of Qwen2.5 or GLM4 may contain those very outcomes. The paper reports no contamination check, so the large improvement over non-LLM baselines could reflect parametric memorization of test labels rather than VMID's multimodal fusion. This is load-bearing: please add a leakage analysis, for example zero-shot evaluation of the base LLM with and without the multimodal prompt, ablations withholding each modality, and tests on videos from after the LLM's knowledge cutoff.","section":"Section V-D, Fig. 5(b); Section III-B5"},{"comment":"The caption of Table I says the results are “on our short video rumor dataset and the FakeSV dataset,” but the text only describes FakeSV; it is unclear whether the baselines were retrained on the same split or whether the numbers were taken from the original papers. Table I also reports no variance, and the text in Section V-C.2 refers to “The FakeSV model” when it apparently means SV-FEND. Please clarify the evaluation protocol, report per-class performance (real/fake/debunking) in addition to the aggregate metrics, and state explicitly which baselines were reproduced and under what settings.","section":"Section V-C, Table I"},{"comment":"The central claim is that VMID “successfully integrates multimodal features,” but no ablation removes audio, visual, subtitle, or metadata components, and there is no comparison to a text-only LLM prompt on the same data. The reported accuracy could largely come from the title/subtitle text or from the LLM's prior, rather than from multimodal fusion. The two case studies, one success and one failure, are anecdotal and do not substitute for a quantitative ablation. Please add an ablation table and a text-only baseline to substantiate the multimodal-fusion attribution.","section":"Section V, overall; Section V-D"}],"minor_comments":[{"comment":"The section begins with an orphaned sentence, “Through this backtracking analysis, peak attention aligns with the item most pertinent to the query,” which appears to be leftover text from another source and should be removed.","section":"Section V-B"},{"comment":"The SwiGLU formula is written as \\(SwiGLU(x) = x \\odot \\sigma(W_1 x + b_1) + W_2 x + b_2\\), which does not match the standard SwiGLU definition; please correct the equation.","section":"Section III-B3"},{"comment":"The Mel spectrogram formula is incomplete: the summation index \\(k\\) and the term \\(h_m(t,f)\\) are not defined with proper short-time Fourier transform frame indexing and Mel filterbank specifications.","section":"Section III-B2, Eq. (3)"},{"comment":"In Equation (5), both \\(score(seq)\\) and the set \\(C\\) of candidate decoding sequences are left undefined; please define them for clarity.","section":"Section III-B2, Eq. (5)"},{"comment":"The text says “we learn the weight coefficients \\(\\alpha\\) and \\(\\beta\\)” for modality importance, but these coefficients are never defined or used in Equations (6) or (7); please remove or operationalize them.","section":"Section III-B6"},{"comment":"The caption says videos are differentiated by yellow, gray, and blue backgrounds, but this color coding is difficult to read in the dense table and is inaccessible in grayscale printing; consider adding explicit labels for each row.","section":"Figure 3 and its caption"},{"comment":"Reference [1] lists “Association for Computing Machinery” as the publisher for the AAAI FakeSV paper; the publisher and venue should be corrected.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript would benefit from a substantially stricter evaluation section. I would ask the editor to require a contamination analysis before considering acceptance, because the current case study explicitly attributes a correct prediction to the LLM's parametric knowledge; if the base LLM has seen the FakeSV fact-check outcomes, the headline numbers would measure memorization rather than detection. The authors should also be required to either release code or provide complete hyperparameter and split details, since the paper currently claims a full implementation without making it available."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this paper wires Whisper, subtitle OCR, CogVLM2 visual descriptions, and video metadata into a single text prompt for a LoRA-tuned LLM, and reports 90.93% accuracy on FakeSV, about 10 points above SV-FEND. That number is plausible but not verified. There is no code, no hyperparameter table, no train/test split description, no error bars, and the method section explicitly says implementation details are omitted. The core empirical claim cannot be independently checked.\n\nCredit where due: the application is new, the system is sensibly assembled, and the authors show an ablation across four LLMs plus a failure case, which is more honest than most papers in this area. The idea of flattening all modalities into text for an LLM is not novel, but its application to short-video misinformation is a legitimate extension, and the reported margins over older baselines are large.\n\nThe soft spots are serious. Most important is the contamination risk. FakeSV is built from real videos about real events, and the paper's own case study credits the LLM's pretrained knowledge base for a correct detection. If Qwen2.5 or GLM4 saw the fact-check outcomes or debunking articles for those videos during pretraining, the model can answer from memory, making the multimodal prompt incidental. The 10-point gap over SV-FEND, which has no such memory, is exactly what contamination would look like. No memorization check is reported. The authors do not discuss this at all.\n\nSecond, reproducibility is close to zero. The contribution list promises a full implementation, but nothing is released. The evaluation is a single table with no variance. There are also copy-paste artifacts and small factual errors, like referring to FakeSV as a model and a stray sentence about backtracking analysis in the metrics section. These are not fatal, but they signal a rushed manuscript.\n\nMy take: the architecture is sensible and the direction is worth publishing if the numbers hold up, but this version does not establish that. A serious referee should send it back for code, exact splits, and a contamination check, e.g., comparing against a text-only prompt or shuffled-modality baselines. For your own work, I would not cite the numbers yet.","headline":"A plausible multimodal pipeline for short-video misinformation detection whose headline numbers are undermined by missing reproducibility and a very real label-memorization risk.","tokens_in":17219,"tokens_out":2306,"would_cite":false,"duration_ms":27426,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single multimodal prompt lets a tuned LLM detect fake short videos at 90.9% accuracy.","keywords":["multimodal fusion","short video misinformation","large language model","LoRA fine-tuning","FakeSV dataset","fake news detection","audio transcription","visual-language model"],"falsifier":"Retest VMID on videos released after the knowledge cutoff of the underlying LLM, or ablate the video-derived text (subtitles, audio, vision) and prompt with only title and metadata; if accuracy stays near 90 percent, the gain is recalled memorized fact-checks rather than multimodal detection.","tokens_in":16150,"feed_emoji":"🎬","tokens_out":5059,"duration_ms":45681,"temperature":0.7,"pith_summary":"The paper claims that misinformation in short videos can be detected reliably by converting every modality—subtitles, speech, visual content, and metadata—into one text prompt and feeding it to a large language model fine-tuned with LoRA. On the FakeSV benchmark this VMID framework reaches 90.93% accuracy and 90.89% macro F1, beating the strongest baseline by roughly ten percentage points. The finding matters because current video fake-news detectors rely on one modality or simple fusion and struggle with the mismatch between what is said, shown, and written in a video. If the claim holds, a single LLM can act as a general-purpose cross-modal reasoner for video misinformation without task-specific fusion architectures.","feed_headline":"Tuned LLM catches fake short videos with 90.9% accuracy","feed_subtitle":"Merging subtitles, speech, visual descriptions, and metadata into one prompt beats the best baseline by nearly 10 points.","key_machinery":"The load-bearing object is the integrated prompt: outputs from a subtitle extractor (VSE), a speech recognizer (Whisper), and a visual-language model (CogVLM2) are concatenated with video metadata and fed to a LoRA-tuned LLM, which performs cross-modal attention over the combined text. The paper's argument is that converting all modalities into a shared textual space lets the LLM treat cross-modal consistency as a text-reasoning problem, so no specialized fusion network or modality-alignment training is needed. LoRA fine-tuning keeps the pretrained knowledge intact while adapting the model to the three-way classification of fake, real, and debunking content.","core_discovery":"VMID establishes that a large language model, given a unified textual transcript of a video's subtitle stream, spoken audio, keyframe descriptions, and social metadata, can classify short videos as fake, real, or debunking more accurately than dedicated multimodal architectures. The paper reports 90.93% accuracy and 90.89% macro F1 on FakeSV, outperforming SV-FEND (81.05%) and other baselines by 9.87–19.6 absolute percentage points, and shows that the approach remains strong across different backbone LLMs (Qwen2.5, GLM4, InternLM2.5, Baichuan). A case study credits the LLM's external knowledge for catching a mislabeled police drill video, indicating the model leverages both video content and pretrained world knowledge.","pith_inferences":["The reported margin may shrink on videos published after the LLM's training cutoff; a fair test would withhold post-cutoff examples or block parametric fact recall, since the case study shows the model using memorized knowledge.","The framework's dependence on the quality of the extractors means that Whisper transcription errors or CogVLM2 misdescriptions propagate directly into the prompt; an end-to-end noisy-input robustness study would clarify how much of the gain is genuine fusion versus extractor quality.","The same prompting strategy could be applied to longer-form video or to multimodal misinformation beyond news (health claims, product scams), reusing the identical pipeline with different metadata fields."],"forward_implications":["Fake-news detection for short video platforms can be implemented as a single LLM call plus three off-the-shelf extractors, replacing task-specific multimodal architectures.","Because the prompt is plain text, the same framework transfers to new video genres or languages by swapping the extractors, with no redesign of the fusion layer.","The large gains over SV-FEND and SVRPM suggest that cross-modal reasoning within an LLM captures inconsistencies that modality-tampering detectors miss.","Accuracy above 90% on FakeSV implies that the three-way distinction among fake, real, and debunking content is learnable from video-derived text alone when social context is included."],"supporting_citations":[{"why":"Supplies the FakeSV benchmark and the SV-FEND baseline that define the comparison.","marker":"[1]"},{"why":"Whisper provides the audio-to-text transcription module used for the speech modality.","marker":"[14]"},{"why":"CogVLM2 generates textual descriptions of keyframes for the visual modality.","marker":"[15]"},{"why":"VSE extracts timestamp-aligned subtitle text that feeds the unified prompt.","marker":"[16]"},{"why":"SVRPM is the modality-tampering baseline that VMID outperforms in the experiments.","marker":"[17]"},{"why":"Qwen2.5 is the base LLM for the main VMID configuration and LoRA fine-tuning.","marker":"[26]"},{"why":"GLM4 is the LLM variant that reaches the highest reported accuracy (91.20%), used to show architecture robustness.","marker":"[24]"}],"fun_headline_variants":["LLM fusion beats dedicated models on fake video detection","90.9% accuracy: multimodal LLM outsmarts fake short videos","Short video fake news? LLM with fused text scores 90.9%","VMID: one prompt fuses video streams to catch misinformation","Fake video detection leap: LLM beats SV-FEND by 10 points"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results assume the LLM's pretrained knowledge does not already contain the ground-truth labels of the test videos; if the fact-check outcomes were in the training data, the reported accuracy would reflect memorization rather than detection.","fun_headline_variants_meta":{"raw":{"variants":["LLM fusion beats dedicated models on fake video detection","90.9% accuracy: multimodal LLM outsmarts fake short videos","Short video fake news? LLM with fused text scores 90.9%","VMID: one prompt fuses video streams to catch misinformation","Fake video detection leap: LLM beats SV-FEND by 10 points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1453,"prompt_tokens":974,"completion_tokens":479,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":384}},"tokens_in":590,"tokens_out":479,"duration_ms":5182,"temperature":1.0,"reasoning_tokens":384,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:03:00.516732+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retest VMID on videos released after the knowledge cutoff of the underlying LLM, or ablate the video-derived text (subtitles, audio, vision) and prompt with only title and metadata; if accuracy stays near 90 percent, the gain is recalled memorized fact-checks rather than multimodal detection.","supporting_citations":[{"cited_title":"Fakesv: A multimodal benchmark with rich social context for fake news detection on short video platforms,","cited_arxiv_id":null,"evidence_quote":"Supplies the FakeSV benchmark and the SV-FEND baseline that define the comparison."},{"cited_title":"Robust speech recognition via large-scale weak supervision,","cited_arxiv_id":null,"evidence_quote":"Whisper provides the audio-to-text transcription module used for the speech modality."},{"cited_title":"Video Subtitle Extractor,","cited_arxiv_id":null,"evidence_quote":"VSE extracts timestamp-aligned subtitle text that feeds the unified prompt."},{"cited_title":"Interpretable short video rumor detection based on modality tampering,","cited_arxiv_id":null,"evidence_quote":"SVRPM is the modality-tampering baseline that VMID outperforms in the experiments."},{"cited_title":"Qwen2.5,","cited_arxiv_id":null,"evidence_quote":"Qwen2.5 is the base LLM for the main VMID configuration and LoRA fine-tuning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GLM4 is the LLM variant that reaches the highest reported accuracy (91.20%), used to show architecture robustness."}],"review_version":1}