{"id":"f2618228-90ad-4f65-ad72-35b48a11bc07","arxiv_id":"2607.23739","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Claude 3.5 Sonnet with dynamic few-shot multi-modal prompts detects misleading YouTube thumbnails at 93.8% accuracy on a balanced cross-country dataset, outperforming CHECKER and open-weight VLMs.","lead":"A multi-modal LLM pipeline flags misleading YouTube thumbnails by comparing the image to subtitles and auto-generated video descriptions. On a new 2,843-video, eight-country dataset, Claude 3.5 Sonnet reaches 93.8% accuracy and beats a prior supervised detector without task-specific training.","discovery_kind":"new_application","skeptic_critique":{"model":"moonshotai/kimi-k3","headline":"The 93.8% is a development-set number: prompts were tuned on, few-shot examples retrieved from, and ambiguous cases filtered out of the same 2,843 videos used for evaluation — there is no held-out split anywhere in the paper.","rationale":"The reader's weakest_assumption centered on description circularity (Claude-written descriptions feeding a Claude classifier) and mentioned agreement-filtered ground truth in the rationale. I agree those are real, and the authors themselves concede the self-reinforcement risk in §7. But I judge the more load-bearing issue to be one level up: the entire evaluation is in-sample. Even with perfectly faithful, independently generated descriptions, a 93.8% obtained after prompt tuning on the evaluation set, with labeled retrieval neighbors drawn from the evaluation set, and with ~11% of the hardest (disagreement) cases excluded, does not estimate deployment accuracy. This subsumes part of the reader's ground-truth concern and adds the retrieval/tuning contamination dimension the reader did not foreground, hence \"partial.\" I do not recommend REJECT: the dataset contribution, cross-model consistency (six models, sensible ordering, per-country and per-category breakdowns), released artifacts, and honest limitations section make this a usable systems paper once the evaluation is redone properly. The CHECKER comparison should be downgraded from \"surpassed on all metrics\" to \"statistically indistinguishable on a 197-video set\" unless a bootstrap says otherwise. These are exactly the conditions the reader attached, so the verdict stays CONDITIONAL.","tokens_in":21111,"tokens_out":2369,"duration_ms":116721,"concrete_test":"Freeze all prompts and the dynamic few-shot pipeline as-is. Split the 2,843 videos by country (e.g., leave-two-countries-out) or random 80/20, restricting the SBERT retrieval pool and all prompt tuning to the training portion only. Have a third adjudicator label the ~357 excluded disagreement cases and include them in the test set. Evaluate Claude 3.5 Sonnet once on this untouched test set, and separately re-run the CHECKER comparison with 1,000-sample bootstrap CIs on per-video F1. If held-out accuracy drops below ~88%, or the CHECKER CI straddles zero, the headline numbers are development-set inflated and must be re-reported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim (\"93.8% accuracy... exceeds CHECKER's best F1\") rests on an evaluation protocol with no held-out data. Three compounding issues: (1) §2.4 describes iterative prompt engineering and example construction, and the dynamic few-shot pool is drawn from the same dataset being classified — every test instance is evaluated with labeled neighbors retrieved from the evaluation set itself (transductive evaluation), and prompt choices were refined by observing accuracy on this same data. (2) §2.2 drops all videos where the two annotators disagreed (3,200 → 2,843, minus availability losses), so the model is scored only on unambiguous cases; real-world accuracy on contested thumbnails — precisely the deployment-relevant slice — is unknown and almost certainly lower. The 71.6% human baseline under identical inputs, versus 93.8% for Claude, is itself a red flag that the labels encode information not recoverable from the model's inputs, or that easy-case filtering dominates. (3) The CHECKER comparison (§5) reports F1 0.7227 vs 0.7153 on only 197 videos — a 0.007 gap with no significance test, which on n=197 is far inside sampling noise. The paper is candid about several of these in §7, but the abstract and contribution 5 headline the unqualified numbers. None of this suggests the qualitative finding (frontier LLMs beat open VLMs and match supervised baselines) is wrong — model ordering across six systems is consistent — but the specific headline figure and the claimed CHECKER win are not established by the reported protocol.","agreement_with_reader":"partial"},"referee_report":{"model":"moonshotai/kimi-k3","summary":"The paper studies whether multimodal LLMs can detect misleading YouTube thumbnails. The authors build a dataset of 2,843 videos from eight countries (1,359 misleading, 1,484 not), annotated by two trained annotators with only full-agreement items retained. They construct a classification pipeline that feeds each model the thumbnail image, subtitles, and a machine-generated video-to-text description, and evaluate four proprietary LLMs (Claude 3.5 Sonnet, GPT-4o, GPT-4o Mini, Gemini 1.5 Flash) and two open-weight VLMs (LLaVA-1.5, Qwen2.5-VL-7B) under zero-shot, fixed few-shot, and dynamic (SBERT-retrieved) few-shot prompting. Claude 3.5 Sonnet with dynamic few-shot prompting achieves 93.8% accuracy and is compared against the supervised CHECKER pipeline on CHECKER's 197-video test set (F1 0.7227 vs 0.7153). The paper also includes modality ablations, per-category and per-country breakdowns, a small human baseline (71.6% accuracy), and a deployment cost analysis.","tokens_in":21439,"tokens_out":2721,"duration_ms":71984,"significance":"If the results hold, this is a useful contribution to the content-integrity literature: misleading-thumbnail detection has a thin prior literature (CHECKER is essentially the only direct comparator), and the paper supplies (i) a publicly released, balanced, cross-country dataset with a codebook — a genuinely scarce resource; (ii) a systematic comparison of six models across three prompting regimes with modality ablations; (iii) an honest failure analysis and per-country/per-category slices showing where the method degrades; and (iv) a concrete cost model showing platform-scale feasibility. The qualitative findings — frontier proprietary LLMs substantially outperform open-weight VLMs, dynamic few-shot prompting helps most models, and prompt-based detection is competitive with a supervised pipeline without task-specific training — are supported by consistent model ordering across configurations and are plausible. However, the headline quantitative claims (93.8% accuracy; \"surpassed CHECKER on all metrics\") rest on an evaluation protocol with no held-out data and a comparator gap within sampling noise, so the magnitude of the contribution depends on revisions to the evaluation rathera","major_comments":[{"comment":"No held-out evaluation anywhere; the dynamic few-shot setting is transductive. §2.4 describes retrieving, for every test video, one labeled MTV and one labeled NMTV from the same 2,843-video dataset being scored, and the prompt design (CoT structure, 200-word truncation, example template) was iteratively refined while observing accuracy on this same data (§2.3–2.4, §4.3). The 93.8% headline figure (Table 1, abstract) is therefore a development-set number: labeled neighbors of each test point are in the prompt, and prompt hyperparameters were selected on the evaluation set. This is load-bearing for Contribution 5 ('state-of-the-art accuracy'). The fix is feasible within the paper's scope: e.g., a held-out split frozen before prompt tuning, or leave-one-country-out retrieval pools, which would also strengthen the cross-cultural claim in §4.5.","section":"§2.4, §4.2, Table 1"},{"comment":"Ground truth is filtered to annotator-consensus items only, and the human baseline contradicts the headline number. §2.2 reports κ=0.9633 and retains only videos with full agreement (3,200 → 2,843 after disagreements and availability losses), so models are scored only on unambiguous cases. Meanwhile §4.9 reports that five humans given exactly the model's inputs (thumbnail + subtitles + description) achieve only 71.6% accuracy. A 22-point gap between Claude and humans on identical inputs, on consensus-only labels, suggests the labels encode judgments not recoverable from the model's input modalities — e.g., mismatch evidence visible only by watching the full video, which neither annotators' inputs at labeling time nor the model's truncated/20-frame description fully capture. The paper should (i) report how many items were dropped for disagreement, (ii) characterize model performance on or","section":"§2.2, §4.9"},{"comment":"The CHECKER comparison is a 0.007 F1 gap on n=197 with no significance testing. §5 reports Claude F1 0.7227 vs CHECKER's best 0.7153 on CHECKER's 197-video test set. On ~197 samples, the standard error of an F1 difference of this size is far larger than the gap; a paired bootstrap or McNemar test would almost certainly show no significant difference. As written, the claim 'Claude surpassed CHECKER on all metrics' (Contribution 5, abstract) is not supported. The correct framing is parity with a supervised pipeline at zero training cost — which is still a meaningful result and should be claimed as such, with a significance test or confidence intervals added.","section":"§5"},{"comment":"Models are evaluated on different subsets of the data, so Table 1's cross-model ranking is not on identical inputs. Table 5 (Appendix A.5) shows Gemini processed 2,135 videos, Claude 2,759, GPT-4o-mini 2,769, GPT-4o 2,749, due to model-specific safety filters and the Twelve Labs 360p exclusion (§4.1). Accuracy differences of a few points (e.g., Gemini 82.8% vs GPT-4o-mini 84.8%) could be entirely explained by which videos each model was allowed to see — plausibly Gemini's stricter filters removed a non-random (more extreme, possibly easier) slice. §7 acknowledges this, but the main-text comparison in §4.2 and Figure 6 presents the numbers as directly comparable. At minimum, the paper should report all metrics on the intersection of processed videos.","section":"§4.1, §4.2, Table 5"},{"comment":"Claude both generates descriptions/explanations and is the top classifier; the self-reinforcement confound is acknowledged but not bounded. Claude wrote the video descriptions used by the open-weight models (§2.5), generated all thumbnail descriptions and the rationales embedded in every dynamic few-shot example (§2.4, Figure 3), and classified using its own descriptions. §7 concedes this 'may bias results in favor of Claude-based pipelines,' but no experiment quantifies the effect. A simple cross-ablation — classify Claude's outputs with Twelve Labs or Gemini descriptions, and classify with GPT-4o using Claude descriptions — would bound how much of Claude's 9–15-point lead is genuine reasoning ability versus representation alignment. Given that Contribution 3's model ranking depends on this, at least one such cell should be run.","section":"§2.4–2.5, §7"}],"minor_comments":[{"comment":"§4.9: the human baseline uses 5 annotators × 50 videos (250 labels). State whether the 50-video samples overlapped across annotators (enabling agreement statistics) or were disjoint, and report per-annotator variance. Also clarify whether these annotators were the same trained graduates from §2.2.","section":"§4.9"},{"comment":"Table 2: the per-category evaluation subsamples to min(MTV, NMTV) per category. Report the actual n per category and confidence intervals; categories like Film & Animation at 0.9000 accuracy look like n≈20, where a handful of flips moves the number by 5 points.","section":"§4.4, Table 2"},{"comment":"§2.2: κ=0.9633 should be reported on all 3,200 annotated items before exclusion, and the number of disagreement cases dropped should be stated explicitly (currently only inferable from the 3,200 → 2,843 funnel, which also includes availability losses).","section":"§2.2"},{"comment":"Paper checklist item 4(a) and §1: the repository is described as 'anonymized' but the URL (github.com/wajihanaveed) contains an author's name — fix for any double-blind venue.","section":"§1, Checklist"},{"comment":"Typos/formatting: 'LLaV A' appears with a stray space throughout (§1, §2.5, §4.8, references); Figure 11 in the appendix appears to duplicate Figure 7; §4.6 average accuracies (91.5/74.8/77.3/82.2) are cited to external benchmarks in a way that conflates this paper's numbers with prior-work averages — clarify provenance.","section":"Throughout"},{"comment":"§4.7: the $25.1M/month platform-scale estimate assumes every uploaded video gets the full pipeline; noting a triage design (e.g., cheap model first, Claude only on uncertain cases) would make the cost argument more realistic.","section":"§4.7"},{"comment":"The 'full stop' search phenomenon (§3.3, item 6) is interesting but anecdotal; either cite supporting documentation or soften the claim.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"The dataset release and breadth of evaluation are real assets, and §7 is unusually candid. The core problem is purely the evaluation protocol: every headline number comes from a setup with no held-out data, consensus-filtered labels, and (for the dynamic few-shot configuration) labeled neighbors of the test point inside the prompt. The CHECKER 'surpass' claim is a 0.007 F1 gap on n=197. None of this requires new data collection to fix — a frozen split, intersection-subset reporting, one cross-description-generator ablation, and significance tests would address it — but until then the abstract's quantitative claims are overstated relative to what the evidence supports. Hence major rather than minor revision. I also note the checklist's IRB/compensation answers are thin for a study involving human annotators and a survey; the editor may want to confirm venue policy on that point."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The real contribution here is the dataset and the controlled bake-off, not the headline accuracy. They release 2,843 videos from eight countries (balanced MTV/NMTV, κ=0.963 on agreed items), with thumbnails, subtitles, and video-to-text descriptions, plus code. That alone is useful for anyone working on visual clickbait or cross-cultural moderation. They then run the same multi-modal inputs through four proprietary and two open VLMs under zero/fixed/dynamic few-shot, with modality ablations, category and country slices, a small matched-input human baseline (71.6%), and a head-to-head on CHECKER’s 197-video set. Claude + dynamic few-shot sits on top; open models lag but improve with retrieval. Ordering is consistent and the qualitative finding—that frontier prompt-only models are competitive with a supervised thumbnail pipeline—looks real.\n\nWhat is soft is the evaluation hygiene behind the strongest claims. There is no held-out split. Dynamic few-shot retrieves labeled neighbors from the same 2,843-video pool being scored (transductive), and prompts were refined on this data. Annotator disagreements were dropped before scoring, so the test set is the easy slice; contested cases—the ones that matter for deployment—are unmeasured. The CHECKER edge is F1 0.7227 vs 0.7153 on n=197 with no significance test; that gap is noise. Safety filters also give models unequal subsets, and Claude-generated descriptions feed several pipelines (including open-model runs later classified in Claude-aligned settings), which the authors flag in §7 but which still qualifies transfer. The 71.6% human baseline under identical inputs versus 93.8% Claude is a useful red flag that either the labels encode more than the inputs or easy-case filtering is doing heavy work.\n\nNone of that kills the paper. The artifacts are real, the multi-model comparison is careful enough to be informative, and the failure modes (full-movie claims, celebrity grounding, cultural clickbait styles) are honestly discussed. This is for people building or evaluating moderation pipelines and for computational social scientists who want a cross-country MTV set. It deserves referee time; the numbers just need to be re-run on a true hold-out with independent description generators and a reported FPR on the disagreement slice. I would engage, cite the dataset, and treat 93.8% as an upper bound on easy cases rather than a deployable rate.","headline":"Useful multi-country dataset and solid multi-model bake-off showing prompt-only VLMs can flag misleading thumbnails; the 93.8% and CHECKER “win” are overstated by a no-holdout, agreement-filtered protocol.","tokens_in":22711,"tokens_out":621,"would_cite":true,"duration_ms":14317,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Multimodal LLMs can flag misleading YouTube thumbnails at over 93% accuracy without task-specific training.","keywords":["misleading thumbnails","clickbait detection","multimodal LLMs","YouTube","content moderation","few-shot prompting","cross-cultural dataset","vision-language models"],"falsifier":"Re-run the identical prompts on the same videos but replace every generated description with a human-written scene-by-scene summary (or full untruncated video input) and check whether Claude’s 93.8% accuracy and the open-model gains collapse or hold.","tokens_in":22501,"feed_emoji":"🖼️","tokens_out":851,"duration_ms":17036,"temperature":0.7,"pith_summary":"Misleading thumbnails lure clicks by exaggerating or misrepresenting what a video actually contains, eroding trust at global scale. This paper builds a balanced 2,843-video dataset spanning eight countries and shows that frontier multimodal language models can detect the mismatch by jointly reading the thumbnail image, subtitles, and a short video-to-text description. With dynamic few-shot prompting that retrieves culturally similar examples, Claude 3.5 Sonnet reaches 93.8% accuracy and beats a prior supervised specialist system on its own test set. The work argues that prompt-based LLMs offer a practical, training-free layer for pre-upload moderation that generalizes across regions and content categories.","feed_headline":"LLMs catch fake YouTube thumbnails at 93.8% accuracy","feed_subtitle":"Claude plus dynamic examples beats trained specialist systems with no fine-tuning needed","key_machinery":"A multi-modal detection pipeline that feeds the thumbnail image, subtitle transcript, and a concise video-to-text description into an LLM, guided by zero-shot, fixed few-shot, or dynamic few-shot (SBERT-retrieved) chain-of-thought prompts that define “misleading” as exaggeration, false promise, or thematic mismatch.","core_discovery":"When given a thumbnail, subtitles, and a generated video description, Claude 3.5 Sonnet under dynamic few-shot prompting detects misleading YouTube thumbnails at 93.8% accuracy (precision above 92%, recall above 94% in reported settings) on a new 2,843-video cross-country dataset and exceeds the best F1 of the prior supervised CHECKER pipeline on CHECKER’s own test set, all without any task-specific training.","pith_inferences":["If description quality is the hidden bottleneck, cheaper open models may close most of the gap once paired with stronger, model-agnostic video captioners.","The same thumbnail–description–subtitle triple could transfer to short-form platforms (Reels, TikTok) where titles are even weaker signals.","Failure modes concentrated on “full movie” claims and celebrity recognition suggest entity-linked visual grounding as the next concrete research lever."],"forward_implications":["Platforms can insert a lightweight pre-upload check that blocks or queues thumbnails whose semantic claim diverges from the video description and subtitles.","Dynamic few-shot retrieval with culturally matched examples shrinks cross-country accuracy gaps to under three points, supporting region-aware moderation.","Open-weight vision-language models improve substantially with the same retrieval prompts yet still lag proprietary systems by roughly 20 points, setting a clear target for open multimodal progress.","Cost estimates place full daily YouTube-scale screening at a fraction of one percent of platform revenue, making deployment economically plausible."],"fun_headline_variants":["Claude 3.5 Sonnet flags misleading YouTube thumbnails at 93.8% accuracy","LLMs detect bait thumbnails using video text, subs, and images","Cross-country set of 2843 videos shows Claude leading thumbnail checks","Dynamic few-shot Claude beats prior supervised thumbnail detectors","Multimodal LLM pipeline spots misleading thumbnails without fine-tuning"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"Machine-generated video descriptions (sometimes written by the same model family that later classifies) are faithful enough proxies for the full video that thumbnail–content mismatch can be judged from them alone.","fun_headline_variants_meta":{"raw":{"variants":["Claude 3.5 Sonnet flags misleading YouTube thumbnails at 93.8% accuracy","LLMs detect bait thumbnails using video text, subs, and images","Cross-country set of 2843 videos shows Claude leading thumbnail checks","Dynamic few-shot Claude beats prior supervised thumbnail detectors","Multimodal LLM pipeline spots misleading thumbnails without fine-tuning"]},"model":"grok-4.5","effort":"low","cost_usd":0.003704,"raw_usage":{"total_tokens":1241,"prompt_tokens":881,"num_sources_used":0,"completion_tokens":79,"cost_in_usd_ticks":37044000,"prompt_tokens_details":{"text_tokens":881,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":281,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":881,"tokens_out":79,"duration_ms":5552,"temperature":1.0,"reasoning_tokens":281,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-30T14:20:07.406728+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Re-run the identical prompts on the same videos but replace every generated description with a human-written scene-by-scene summary (or full untruncated video input) and check whether Claude’s 93.8% accuracy and the open-model gains collapse or hold.","supporting_citations":[],"review_version":1}