{"id":"55853454-3938-428b-a85b-3aa4eb5d2482","arxiv_id":"2504.21165","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Manicod combines live web retrieval with an LLM to detect zero-day manipulated news, reporting F1 0.856 on a new dataset of 4,270 manipulated headlines and large gains over existing benchmarks.","lead":"Manicod is a fake-news detector that searches the web for live reporting and asks a large language model to check whether a headline contradicts the facts. The paper also introduces a new dataset of 4,270 manipulated headlines built from 2,500 real news stories from 2024.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported F1 of 0.856 and the up-to-1.9x benchmark gains are not established because the prompt was tuned on the evaluation cases and the benchmark baselines were not re-run on the filtered label sets.","rationale":"The reader's explicit weakest_assumption is that all retrieved Internet knowledge is true, but their rationale centers on uncontrolled evaluation. My load-bearing concern is the same broad evaluation problem: the reported F1 and benchmark gains are not valid estimates because the prompt was tuned on the test distribution, the negation test cases were generated by the same model family used for detection, and the benchmark comparisons filter labels without re-running baselines on the same filtered tasks. I do not focus the primary attack on the 'retrieved knowledge is true' assumption because the paper acknowledges that limitation in Section 7 and it is a common caveat for retrieval-augmented fact-checking; the evaluation flaws are more decisive for the paper's quantitative claims. Since the reader already recommends REJECT and my analysis supports that judgment, no verdict adjustment is needed. A controlled re-evaluation with a held-out prompt selection and identical baseline settings could legitimately change the verdict if the reported results survive, but as presented the central empirical claims are not supported.","tokens_in":20694,"tokens_out":8009,"duration_ms":87832,"concrete_test":"Re-run the full evaluation with a pre-registered protocol: split the proposed dataset into a prompt-development split (e.g., 80%) and a held-out evaluation split (20%), tune the prompt only on the development split, freeze it, and report F1 separately for X, Xn, and Xc on the held-out split, with the negation set regenerated by a different model or by human writers. If the held-out F1 drops substantially below 0.856 or the per-category accuracies change materially, the reported headline result is an artifact of prompt tuning and same-model generation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5 reports F1=0.856 on the proposed dataset, but the evaluation protocol does not support this as a predictive estimate. The prompt template in Figure 3 was produced by a 'continuous tuning process' (Section 3) in which the authors repeatedly instructed Llama 3.1 with 'hard' cases until predictions were stable and balanced; no held-out split is described, so the final prompt is fit to the test distribution and the reported F1 is an upper-bound rather than a generalization estimate. The negation test set Xn is also generated by Llama 3.1, the same model family used for detection, creating a same-generator dependency that the manual review does not remove. For the comparative claim, Table 3 filters out half-true and barely-true labels from LIAR-RAW and RAWFC but compares against published F1 numbers computed on the unfiltered tasks; baselines were not re-run on the identical filtered subsets or with the same evidence augmentation. The up-to-1.9x improvement therefore cannot be attributed to the method rather than to the changed evaluation setup. These are correctness risks in the central claim, not merely missing ablations.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MANICOD, a two-phase pipeline for detecting manipulated news content about recent ('zero-day') events. In Phase 1, the user's claim is sent to a search engine via SerpAPI, the top-k pages are crawled and chunked, and the chunks are embedded with nomic-embed-text into a ChromaDB vector store. In Phase 2, the retrieved chunks are fed as RAG context to Llama 3.1 (8b), which produces a binary veracity decision (True/False) and a natural-language explanation. To evaluate, the authors construct a dataset of 4,270 manipulated headlines derived from 2,500 real news headlines collected from RSS feeds over 20 days in 2024, using sentiment reversal and context alteration. They report an overall F1 of 0.856 on this dataset, an ablation showing large gains over Llama 3.1 and GPT-4o-mini without retrieval, and comparisons on COVID-Scientific, FEVER, LIAR-RAW, and RAWFC where they claim up to 1.9x F1 improvements over prior work.","tokens_in":20898,"tokens_out":6196,"duration_ms":63119,"significance":"If the reported results were trustworthy, MANICOD would be a useful contribution: it targets a real gap (verifying claims about events newer than a model's training cutoff), it is fully automated and explainable, and the ablation in Table 1 does suggest that retrieval helps. The authors also deserve credit for a thoughtfully constructed dataset, for reporting non-conclusive LLM outputs conservatively, and for a candid Limitations section that acknowledges prompt sensitivity and the assumption that retrieved knowledge is true. However, the headline F1 and the benchmark comparisons are not established by the evaluation as described. The prompt was tuned on difficult evaluation cases, the negation test set was generated by the same model family used for detection, and the multi-class benchmarks were filtered and evidence-augmented without re-running the baselines. These are load-bearing threats to the central claims, not merely missing ablations.","major_comments":[{"comment":"The prompt template is selected by a 'continuous tuning process' in which the authors 'repeatedly instruct LLMs with hard cases that tend to result in different predictions until we find a prompt that is stable and yields a reasonable performance.' No held-out split or nested validation is described. The reported F1 of 0.856 is therefore an optimistic in-sample estimate: the prompt is effectively fit to the evaluation distribution. To support the zero-day generalization claim, the prompt must be frozen before any test-set interactions, or the evaluation must use a separate validation split for prompt development and a held-out test split for reporting.","section":"Section 3 and Section 5.1 (Figure 3)"},{"comment":"The negation set Xn is generated by Llama 3.1, the same model family used as the detector. The manual review filters out inapplicable negations (as in C.2), but it cannot remove systematic stylistic or lexical regularities introduced by the generator; the detector may be recognizing artifacts of Llama 3.1's negation style rather than the semantic property of manipulation. This concern is partially acknowledged in C.3. The accuracy gap between Xn (93.9%) and X (65.7%) is consistent with such an artifact effect. I request a cross-generator test set (e.g., negations by a different model or human-authored perturbations) or, at minimum, an analysis of which features drive the Xn decisions.","section":"Section 4 and Section 5.1"},{"comment":"The comparison on LIAR-RAW and RAWFC is not apples-to-apples. The authors exclude half-true and barely-true labels and directly augment the built-in evidence for MANICOD, yet they compare against published F1 numbers from prior work that used the full label sets and their own evidence pipelines. Baselines were not re-run on the same filtered label subsets or with the same evidence augmentation. Consequently, the reported up-to-1.9x improvement cannot be attributed to the method rather than to the changed evaluation setup. Please re-run all baselines under an identical protocol, or restrict the comparison to published results on exactly the same label set and evidence condition.","section":"Section 5.3 (Table 3)"},{"comment":"The paper states 'To simplify our study, we assume the knowledge retrieved from the Internet is true.' This assumption is load-bearing for zero-day detection, because a False decision is an assertion of contradiction between the claim and the retrieved context. If the retrieved webpages are wrong, outdated, or mutually conflicting (as the authors note can happen), the decision is invalid. The paper does not provide a sensitivity analysis for noisy or contradictory retrieval results. At minimum, the evaluation should quantify how often the top-k retrieved documents agree with the eventual ground truth, and the claims should be scoped accordingly.","section":"Section 3 and Section 7"},{"comment":"The confusion matrix as printed is internally inconsistent with the dataset description in Section 4. The entries imply 4,973 actual positive (fake) examples and 1,797 actual negative (truth) examples, whereas the dataset contains 4,270 manipulated and 2,500 truthful claims. The reported precision (0.795), recall (0.926), and F1 (0.856) cannot be reproduced from the table as printed without an unexplained relabeling of rows or columns. Please correct the table or the surrounding text so the metrics are reproducible.","section":"Appendix F (confusion matrix)"}],"minor_comments":[{"comment":"The phrase 'threading the solidarity and harmony of society' appears to be a typo for 'threatening the solidarity and harmony of society.'","section":"Section 1"},{"comment":"The Snopes reference is titled 'the definitive face-checking site'; this should be 'fact-checking site.'","section":"References and Section 5.3"},{"comment":"The abstract and Section 5.3 both say 'up to 1.9x' improvement, but the cited LIAR-RAW numbers (0.915 vs 0.314) correspond to a factor of about 2.9. Please reconcile the arithmetic.","section":"Section 5.3"},{"comment":"The manual dataset review is described as involving three team members, but no inter-annotator agreement statistic or the number of discarded examples is reported. Adding this information would improve confidence in dataset quality.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The topic is timely and the pipeline is sensible, but as submitted the headline F1 and the benchmark gains are not supported by the evaluation protocol. I do not think this is an outright reject: the flaws are correctable. A serious revision would need a held-out prompt-selection procedure, a cross-generator or human-written manipulation test, and a fair re-run of baselines under identical filtered/evidence conditions. Without those, the central claims remain unestablished."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline numbers are not supported, but the dataset and the retrieval ablation are worth something. The stress-test concerns check out on reading: the reported F1 of 0.856 is an upper bound, not a generalization estimate.\n\nWhat is actually new is the dataset: 4,270 manipulated headlines derived from 2,500 real headlines collected over 20 days in 2024, with two manipulation types (sentiment reversal and context alteration) and manual review. That is a reusable artifact. The system itself — search-engine retrieval, vectorization with RAG, and Llama 3.1 inference with a natural-language explanation — is a sensible combination, and the ablation in Table 1 shows retrieval matters: without it, Llama 3.1 and GPT-4o-mini both stay below 70% accuracy on the manipulated subsets, while the full system reaches 91–94%. That is a real signal.\n\nThe soft spots are load-bearing. The prompt template was “continuously tuned” on “hard” cases until predictions were “stable and balanced”; no held-out split is described, so the prompt is fit to the test distribution. The negation set was generated by the same Llama 3.1 model family used for detection, creating a same-generator dependency. And in Table 3, the authors filter out half-true and barely-true labels from LIAR-RAW and RAWFC but compare against published F1 scores computed on the unfiltered tasks; baselines are not re-run on the filtered subsets. So the “up to 1.9x” improvement cannot be attributed to the method. These are not minor ablations — they affect the correctness of the central claims.\n\nThe paper is honest about its assumptions, notably that retrieved knowledge is true, which is a real limit for zero-day events but at least stated plainly. The authors also list genuine limitations: prompt sensitivity, LLM reasoning errors, unstable ground truth. That is to their credit.\n\nWho this is for: people working on claim verification or RAG-based detection. The dataset may be useful, but I would not build on it without a careful look at the actual files. For peer review, I would send it out — it deserves a serious referee — but I would expect major revisions, specifically a held-out prompt-selection protocol, re-run baselines on the identical filtered tasks, and a sample of the dataset checked by external annotators. As it stands, the central numbers are not established.","headline":"Useful dataset and a sensible RAG pipeline, but the headline F1 and the 1.9x gains are not established because the prompt was tuned on the test set and baselines were not re-run on the filtered tasks.","tokens_in":21473,"tokens_out":3399,"would_cite":false,"duration_ms":32970,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Manicod detects zero-day manipulated news by grounding LLM claims in live web retrieval, reaching F1 0.856 without fine-tuning.","keywords":["manipulated content detection","zero-day fake news","retrieval-augmented generation","large language models","fact-checking","claim verification","news headline dataset","knowledge-grounded inference"],"falsifier":"Serve Manicod a true recent headline whose live search results contain a plausible but incorrect figure, such as a casualty count lower than the real one. If the system labels the true headline 'False' because of that contradiction, the retrieved-context-is-truth assumption, not the inference, is carrying the decision. A complementary test: feed a deliberately manipulated headline with only unrelated web results; a genuinely grounded detector should not produce a contradiction-based 'False' verdict.","tokens_in":20455,"feed_emoji":"🔎","tokens_out":8194,"duration_ms":77316,"temperature":0.7,"pith_summary":"Manicod is a tool for detecting manipulated content: fake news built from a true recent event by a small edit, such as changing a casualty number or negating a headline's sentiment. The paper's central claim is that such zero-day manipulated content, which cannot be known from a model's training cutoff, becomes detectable when the model is fed live web context about the event. The system searches the claim, vectorizes the top three pages with retrieval-augmented generation, and asks an open LLM to find contradictions, altered facts, or numerical exaggerations, then to explain its 'True' or 'False' verdict. On a new dataset of 4,270 manipulated headlines derived from 2,500 real 2024 news headlines, the tool reaches F1 0.856, and it beats existing fact-checkers by up to 1.9x F1 on four public benchmarks. If correct, this points to grounding rather than ever-larger training data as the way to keep up with fast-moving misinformation.","feed_headline":"Live web context catches same-day fake news at F1 0.856","feed_subtitle":"Manicod grounds claims in live search results, beating trained fact-checkers by up to 1.9x F1.","key_machinery":"The load-bearing mechanism is the two-phase grounding loop. In the first phase, the input claim is sent to a mainstream search engine, the top $k=3$ URLs are crawled, and the raw text is chunked, embedded, and stored in a vector database so a small pre-trained LLM can be given relevant context despite token limits. In the second phase, a carefully engineered prompt tells the LLM to reason first and decide last: answer 'False' only when it finds a contradiction, a factual mistake, or altered context such as number, person, location, or date, and answer 'True' when no evidence of manipulation exists; the output must end in exactly the word 'True' or 'False' and include a textual explanation. The dataset-generation pipeline is part of the same machinery: the same LLM produces sentiment reversals and extracts key context slots, while human reviewers replace those slots so the altered headlines are factually false. For context-altered fake news, a correct detection is required to name both the original context and its replacement, making the explanation auditable.","core_discovery":"The paper argues that the veracity of recent-event claims is not inferable from parameter knowledge alone; what makes a claim decidable is external, temporally current context. Manicod operationalizes this by retrieving live search results for the claim, vectorizing the top documents into a retrieval store, and prompting an LLM to reason over the retrieved chunks and output an explanation ending in exactly 'True' or 'False'. The decision rule is deliberately asymmetric: the model must answer 'False' when it finds any contradiction, factual mistake, or inconsistent context such as number, quantity, person, or location, and must answer 'True' when it cannot find evidence of manipulation. The ablation supports the central claim: without live retrieval, Llama 3.1 and GPT-4o-mini score much lower on the same recent-news claims, such as 43.4% and 57.9% accuracy on the true headlines versus 65.7% with retrieval. The paper also contributes a dataset of 2,500 true headlines plus 4,270 simulated manipulations produced by sentiment reversal and context alteration, with human review ensuring the altered headlines genuinely contradict ground truth.","pith_inferences":["A direct extension the paper leaves implicit: the retriever-in-the-loop design should transfer to any claim domain whose ground truth changes faster than model releases, such as election counts, disease advisories, or financial statements, as long as authoritative pages are searchable.","The reported asymmetry (93.9% accuracy on sentiment reversals, 91.0% on context alterations, but 65.7% on true headlines) suggests the detector's practical failure mode is false alarms on true but rapidly updated news; one testable fix would be a confidence floor or an 'unverifiable' output when retrieved pages disagree.","Since the verdict 'False' requires a contradiction in retrieved context, an attacker who can manipulate the top search results for a narrow event could in principle flip verdicts; measuring this would require a red-team study where the context database is deliberately poisoned and accuracy is re-measured.","The authors' observation that exact prompt wording changes behavior implies the reported F1 is partly a property of the prompt, not only of retrieval; an extension would evaluate the same pipeline across prompt variants and LLMs to separate grounding gains from prompt-tuning gains."],"forward_implications":["Zero-day manipulated content is detectable without any fine-tuning or task-specific training: the same pre-trained LLM becomes a veracity judge the moment current context is supplied.","The ablation implies that for recent events, intrinsic knowledge alone is not enough: direct Llama 3.1 got 43.4% accuracy on true 2024 headlines, and direct GPT-4o-mini got 57.9%, versus 65.7% with retrieval.","The pipeline generalizes beyond manipulated news to fabricated and rumor-style claims, beating prior systems on COVID-Scientific, FEVER, RAWFC, and LIAR-RAW with up to 1.9x F1 improvement.","An auditable output is part of the detection contract: for context-altered fake news, a correct detection must name the original context and its replacement, not merely say 'False'.","The released dataset of 4,270 simulated manipulations, built from 2,500 real 2024 headlines, gives future detectors a common benchmark for zero-day fake news."],"supporting_citations":[{"why":"Supplies the best existing perplexity-based LLM fact-checking baseline that Manicod compares against on COVID-Scientific and FEVER.","marker":"Lee et al., 2021"},{"why":"Provides the FEVER benchmark dataset used for binary fact-checking evaluation.","marker":"Thorne et al., 2018"},{"why":"Provides the COVID-Scientific fact-checking benchmark with WHO/CDC-labeled truths and myths.","marker":"Google, 2021"},{"why":"Defines knowledge-grounded reasoning by LLMs with user-supplied evidence, the approach Manicod automates and an explicit baseline.","marker":"Wang and Shu, 2023"},{"why":"Baseline defense-based explainable fake-news detection that partitions evidence into competing parties before LLM reasoning.","marker":"Wang et al., 2024"},{"why":"Supplies the CofCED method and RAWFC claim-verification benchmark used in the multi-class comparison.","marker":"Yang et al., 2022"},{"why":"Supplies LIAR-RAW, the six-label PolitiFact claim dataset used in the multi-class comparison.","marker":"Wang, 2017"},{"why":"Supplies dEFEND, an explainable fake-news detection baseline in the multi-class comparison.","marker":"Shu et al., 2019"},{"why":"Prior retrieval-augmented fake-news detection that requires fine-tuning or closed models; Manicod's contribution is removing that requirement.","marker":"Niu et al., 2024"},{"why":"Cited to justify retrieval augmentation as a way to reduce LLM hallucination in structured outputs.","marker":"Béchard and Ayala, 2024"}],"fun_headline_variants":["Zero-day fake news caught with live search, F1 0.856","Manicod: live web context outdoes trained fact-checkers by 1.9x","LLM plus live retrieval detects same-day manipulation, F1 0.856","For recent claims, live context beats memory: 0.856 F1","Study: RAG with live search catches manipulated news at F1 0.856"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is the paper's explicit assumption that the knowledge retrieved from the Internet is true: if the top search results are wrong, outdated, or irrelevant, then the 'contradiction' the LLM cites is not evidence that the claim is manipulated, and a 'False' verdict is invalid.","fun_headline_variants_meta":{"raw":{"variants":["Zero-day fake news caught with live search, F1 0.856","Manicod: live web context outdoes trained fact-checkers by 1.9x","LLM plus live retrieval detects same-day manipulation, F1 0.856","For recent claims, live context beats memory: 0.856 F1","Study: RAG with live search catches manipulated news at F1 0.856"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000296,"raw_usage":{"total_tokens":1743,"prompt_tokens":995,"completion_tokens":748,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":640}},"tokens_in":611,"tokens_out":748,"duration_ms":7034,"temperature":1.0,"reasoning_tokens":640,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:11:56.262494+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Serve Manicod a true recent headline whose live search results contain a plausible but incorrect figure, such as a casualty count lower than the real one. If the system labels the true headline 'False' because of that contradiction, the retrieved-context-is-truth assumption, not the inference, is carrying the decision. A complementary test: feed a deliberately manipulated headline with only unrelated web results; a genuinely grounded detector should not produce a contradiction-based 'False' verdict.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the COVID-Scientific fact-checking benchmark with WHO/CDC-labeled truths and myths."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Baseline defense-based explainable fake-news detection that partitions evidence into competing parties before LLM reasoning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior retrieval-augmented fake-news detection that requires fine-tuning or closed models; Manicod's contribution is removing that requirement."}],"review_version":1}