{"id":"29a80595-bd41-4fe3-820e-26b7d3c68f4d","arxiv_id":"2507.00045","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new 100-image benchmark shows that state-of-the-art multimodal models detect subtle, socially meaningful visual clues at near-chance levels and hallucinate accusations on innocent images.","lead":"The authors built CaughtCheating, a 100-photo benchmark in which a picture sent by a partner hides a subtle clue that contradicts the stated story. Top multimodal models such as GPT-o3 find the clue only about a quarter of the time, and they often invent suspicious clues for innocent photos.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's exact boundary numbers depend on unvalidated gold clue labels and an undocumented, internally inconsistent human baseline; both need independent verification before the central claim is fully supported.","rationale":"The reader's weakest assumption—that the quantitative conclusions depend on correct ground-truth annotations and a fairly measured human baseline—is the same condition I find least secure. I checked the full manuscript for independent validation of the gold labels: Section 3.4 reports only judge agreement (human vs. GPT-4.1), which is support for the evaluation pipeline but not for the labels. I also recomputed the human row from Appendix C and found the Precision/F1 inconsistency, which is an internal, checkable red flag rather than a disagreement with consensus. The paper does have real strengths: a novel, practically motivated task; a clear, consistent failure pattern across model families; released data and code; and kappa-based evaluation-pipeline validation. Those are why the concern does not push the verdict to REJECT. But because the exact boundary numbers (26.0% Clued Acc, 17.2% IoU, human F1 59.6) depend on unvalidated labels and an undocumented human protocol, the CONDITIONAL verdict stands. I set verdict_should_be to UNCHANGED because the reader already required exactly these annotation and protocol fixes.","tokens_in":22146,"tokens_out":8794,"duration_ms":91139,"concrete_test":"Run a pre-registered, blinded re-annotation study: recruit 3+ independent annotators who have not seen the paper's labels; for all 50 Clued images ask them to name the single most decisive suspicious clue and list other suspicious elements using the paper's question template; for the 50 Unclued images ask whether clear evidence is present. Compute Fleiss' kappa on exact-clue identification and use the independent labels to recompute Clued Acc/IoU/Unclued Acc for the evaluated models. Then run the human baseline under a documented no-hint protocol with fixed N and instructions, and verify the reported Precision/F1 from raw counts. If annotator agreement is high and human F1 reproduces ≈59.6, the concern is settled; otherwise the 26.0%/17.2% and human-comparison claims must be re-quantified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—'even the best performing model, GPT-o3, only achieves 26.0% accuracy and 17.2% IoU' (Section 4.1)—is only interpretable if the gold 'deterministic clue' labels are objective. Sections 3.2 and B.2 say the deterministic clue is selected by the authors from candidate clues; no inter-annotator agreement is reported for these labels. The kappa values in Section 3.4 (0.82/0.943) validate the GPT-4.1 judge relative to human evaluators; they do not validate the gold labels themselves. With only 50 Clued images and highly context-dependent social cues, a different annotation team might choose different deterministic clues, and a model naming a different but equally decisive clue would be scored wrong. The Unclued labels have the same issue. The human row in Table 1 is additionally undocumented and internally inconsistent: under the Appendix C transformation, Clued Acc=56.0 and Unclued Acc=63.6 imply Precision≈60.6 and F1≈58.2, not the reported 56.0 and 59.6. No participant count, recruitment, instructions, or hint policy is reported. Since Section 4.1 uses the human comparison to place the 'boundary,' both unvalidated gold labels and the unreliable human baseline are load-bearing for the paper's exact numbers, though the qualitative direction of model weakness is plausible.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CaughtCheating, a benchmark of 100 real-world social-media photographs (50 'Clued' images with subtle suspicious clues and 50 'Unclued' images) designed to test whether agentic multimodal large language models (MLLMs) can detect context-dependent visual clues that contradict a stated claim, such as 'dining alone.' The authors evaluate a wide range of open-source and proprietary MLLMs and report that even the best model, GPT-o3, achieves only 26.0% Clued Accuracy and 17.2% Clued IoU, while unreported human participants reach about 56.0% Clued Accuracy. They also report that strong models frequently hallucinate suspicious clues on unclued images, and they use decomposed perception/reasoning questions and Guided Search theory to explain the failures. The data and code are publicly released.","tokens_in":22525,"tokens_out":5512,"duration_ms":49412,"significance":"If the quantitative results are reliable, CaughtCheating fills a real gap: existing benchmarks largely test object recognition or well-defined reasoning, whereas this benchmark requires open-ended search for initially unspecified, socially contextual clues, a task with direct practical relevance. The reported hallucination behavior on unclued images is a useful and falsifiable finding that matters for real deployment. The paper's strengths include public data/code, a judge-validation step (Cohen's kappa 0.82 and 0.943 against human evaluators), decomposed questions that separate perceptual from reasoning failures, and detailed o3 reasoning traces. The qualitative direction — that current agentic MLLMs are far below human-level performance on this task — is plausible and important. However, the exact boundary numbers are supported only weakly, because the gold clue labels lack inter-annotator validation, the human baseline is undocumented and internally inconsistent, and the sample size is small with no confidence intervals.","major_comments":[{"comment":"The gold 'deterministic clue' labels and the 'unclued' ground-truth labels are load-bearing for the headline numbers, but the manuscript reports no inter-annotator agreement on these labels. The Cohen's kappa values in Section 3.4 (0.82 for Clued Acc and 0.943 for Unclued Acc) validate the GPT-4.1 judge against human evaluators; they do not validate the correctness of the deterministic clue selection or the absence-of-clue determination themselves. With only 50 clued images and highly context-dependent social cues, a different annotation team might plausibly select a different deterministic clue, and a model naming a different but equally decisive clue would be scored incorrect. Please add a second independent annotation pass (or a detailed annotation protocol with agreement statistics) and report agreement on the clue labels, not just on the final binary judgments.","section":"Section 3.2 / Appendix B.2"},{"comment":"The human row in Table 1 is internally inconsistent. Under the transformation in Appendix C, Clued Acc=56.0 and Unclued Acc=63.6 (with N_clued=N_unclued=50) imply Precision = 56/(56+18.2) ≈ 60.6 and F1 ≈ 58.2, but the table reports Precision=56.0 and F1=59.6. In addition, the paper does not report the number of human participants, their recruitment, the task instructions, or whether hints were allowed, although Figure 1 and Section 4.2 imply different hint conditions. Because Section 4.1 uses the human row to place the 'boundary' of MLLM capabilities, the human baseline must be documented fully and the reported P/R/F1 values corrected or derived from raw human judgments.","section":"Table 1 / Appendix C"},{"comment":"The central quantitative claims, including 'GPT-o3 only achieves 26.0% accuracy and 17.2% IoU,' are computed from 100 images (50 clued, 50 unclued) with no confidence intervals, no repeated-run variance, and no significance tests. For the best models, a one-sample difference (26.0 vs 20.0 for Gemini-2.5-pro) is within the expected sampling noise for N=50, so the exact ordering and the precision of the word 'boundary' are not supported. Please report bootstrap confidence intervals or exact binomial intervals for the main metrics, and soften claims that distinguish between near-ceiling models.","section":"Section 4.1"},{"comment":"The GPT-4.1 judge is validated for Clued Acc and Unclued Acc, but no human-model agreement is reported for the Clued IoU metric, which depends on a two-stage extraction-and-matching pipeline (Figures 9 and 10 of Appendix D). Since the 17.2% IoU figure is part of the central result, the IoU evaluation should either be validated against human annotations of the non-deterministic clue sets or reported with a caveat that its reliability is unmeasured.","section":"Section 3.4 / Appendix D"}],"minor_comments":[{"comment":"The abstract and introduction contain typos ('acheived', 'Butcan', 'refelction', 'femine', 'Uncledimage'); please proofread the text.","section":"Abstract and Section 1"},{"comment":"Table 3 uses 'with-clue' and 'without-clue' while the main text uses 'Clued' and 'Unclued'; please unify the terminology across the manuscript.","section":"Appendix B.3"},{"comment":"In case (c), 'an Uncledimage' should be 'an Unclued image' and 'really suspecious' should be 'really suspicious'.","section":"Section 5"},{"comment":"The appendix title 'Relate Work' should be 'Related Work', and the phrase 'detection-level benchmark' in Section 1 appears to be a typo for 'detective-level benchmark'.","section":"Appendix A and Section 1"},{"comment":"The caption 'The clue, such as a scattered lipstick, are clearly visible' should be 'The clues, such as a scattered lipstick, are clearly visible'.","section":"Figure 6 caption"}],"recommendation":"major_revision","confidential_remarks":"The benchmark is timely and the qualitative finding of hallucination on unclued images is likely robust, but the exact numbers require the annotation and human-baseline fixes described in the major comments. I would encourage the authors to state how many annotators produced the deterministic-clue labels and whether they were the authors, as this affects the credibility of the gold labels."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should look at this one. The CaughtCheating benchmark is a genuinely new stress test: it asks MLLMs to find unspecified, context-dependent suspicious clues in real photos that contradict a stated claim. That's different from standard VQA or subtle-object detection, and the failure pattern is coherent across many models: best model (o3) gets 26% clued accuracy, and the strong models over-hallucinate on unclued images. The Guided Search framing is a reasonable lens, and the decomposition into perception vs reasoning questions is a good analytical move.\n\nWhat the paper does well: it evaluates a broad range of open and proprietary models, uses a fixed template to avoid answer leakage, includes both clued and unclued categories, and validates the GPT-4.1 judge against human raters (kappa 0.82/0.943). The authors also explicitly acknowledge the demographic and scenario limitations of their social-media source pool, which is more than many benchmark papers do.\n\nThe soft spots are real but addressable. The dataset is only 100 images, with no confidence intervals or repeated-run variance; the exact numbers could move a lot. More importantly, the ground-truth \"deterministic clue\" labels are selected by the authors, with no inter-annotator agreement reported, and the human baseline is poorly documented. The stress-test note catches an internal inconsistency: with Clued Acc=56.0 and Unclued Acc=63.6, the Appendix C formulas give Precision≈60.6 and F1≈58.2, not the reported 56.0 and 59.6. That's a red flag for the precision of the human row, though the qualitative conclusion—humans do much better than models—almost certainly survives.\n\nIs the central claim supported? I think the qualitative direction is solid. No serious reviewer would expect a large model to reliably find a face in a spoon reflection when even half the humans miss it. But the paper asks us to believe that 26.0% is exactly where the boundary sits, and that number depends on labels and a human baseline we can't fully audit from the text.\n\nWho is this for? Anyone working on MLLM evaluation, visual grounding, or trust-signal verification. It deserves a serious referee: the novelty is there, the evaluation is honest, and the flaws are fixable. I'd recommend sending it out with a request for the dataset, full annotation protocols, and the human study details.\n\nRecommendation: engage with it, but push for transparency on the annotations and a re-run of the human baseline.","headline":"A genuinely new and practically relevant MLLM stress test, but the headline numbers rest on annotations and a human baseline that need better documentation before I'd trust the exact boundary.","tokens_in":22965,"tokens_out":2318,"would_cite":true,"duration_ms":21433,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper reports that even the strongest tested multimodal model detects only 26% of subtle \"cheating\" clues and invents clues on 92% of innocent photos, marking a concrete boundary of MLLM perception and reasoning.","keywords":["CaughtCheating","multimodal large language models","visual perception","visual reasoning","suspicious clue detection","Guided Search theory","hallucination","benchmark"],"falsifier":"Run a controlled human study on the same 100 images with at least 30 naive participants, no hints, and a pre-registered marking rule; if human F1 lands near the 20–25% range instead of 59.6%, the claimed human–model gap collapses. Independently, have fresh annotators label each clued image's deterministic clue and check agreement; low agreement would mean the accuracy scores are measured against a ground truth the authors themselves could not reliably reproduce.","tokens_in":21937,"feed_emoji":"🔍","tokens_out":7067,"duration_ms":65675,"temperature":0.7,"pith_summary":"The paper asks whether multimodal large language models (MLLMs) can play detective: look at an ordinary photo and notice a tiny, context-dependent clue that contradicts the story attached to it. To test this, the authors built CaughtCheating, a 100-photo benchmark of real social-media images in which a partner's claim is accompanied either by a subtle \"deterministic\" clue or by no clue at all. Their experiments report that every model they tried fails badly: the strongest, o3, detects the deterministic clue in only 26.0% of clued images and in only 8.0% of unclued images correctly says there is nothing suspicious, giving an overall F1 of 23.9%, while a human comparison reaches F1 59.6%. The paper's point is that near-ceiling scores on existing benchmarks conceal a real boundary: MLLMs cannot yet reliably perform open-ended search for unspecified, low-salience clues, and they hallucinate accusations on innocent images. If the measurements are right, CaughtCheating marks a concrete, practically relevant limit of current visual perception and reasoning.","feed_headline":"AI detectives miss three in four cheating clues","feed_subtitle":"On the 100-photo CaughtCheating benchmark, the strongest models score F1 23.9 while humans reach 59.6.","key_machinery":"The load-bearing object is the CaughtCheating benchmark itself: 100 real photographs, balanced 50/50 between images with a hidden deterministic clue and images without one, annotated with a primary claim-violation question, the deterministic clue, weaker non-deterministic clues, and decomposed perception and reasoning questions. The explanatory machinery is Guided Search theory from cognitive science, which says visual search is steered by bottom-up salience, top-down feature guidance, and scene structure and meaning. The paper uses those three factors to explain why o3's exhaustive grid-search reasoning traces succeed on Waldo-style tasks but collapse on CaughtCheating: the clue is low-salience, there is no predefined template telling the model what to look for, and interpreting the clue requires social reasoning about what contradicts a stated claim.","core_discovery":"The central claim is that open-ended detection of subtle, context-dependent visual clues is beyond current agentic MLLMs, even when the same models solve closed tasks like locating Waldo, finding a camouflaged snake, or geolocating a photo. On CaughtCheating, o3 achieves 26.0% accuracy and 17.2% IoU on clued images and 8.0% accuracy on unclued images, for an F1 of 23.9%; the reported human reference is 56.0% clued accuracy, 63.6% unclued accuracy, and F1 59.6%. The paper also shows through decomposed questions that models often identify the clue when it is explicitly pointed out, yet fail to find it when the query does not name it, and that the strongest models invent suspicious clues on innocent photographs. The authors interpret this pattern as a missing \"detective\" ability: MLLMs rely on exhaustive, region-by-region search, which fails when the target is invisible to bottom-up salience, has no known appearance to guide top-down attention, and draws its meaning from social context.","pith_inferences":["Editorial inference: the benchmark's 100 images are hand-selected and 50 of them lightly cropped to hide a visible person, so the measured difficulty may be partly a property of minimal image editing rather than of naturally occurring hidden clues; a larger set of uncropped, naturally ambiguous photos would test that.","Editorial inference: the paper's own limitation section notes that public social-media sourcing centers cisgender, heterosexual couples and hotel/dining scenes, so the \"boundary\" demonstrated is for one culturally specific form of social reasoning; other relationship norms or workplace/nightlife settings could behave differently.","Editorial inference: the strong models' low unclued accuracy (o3 at 8.0%) conflates two failures—perceptual hallucination and a cautious/refusal style that keeps generating possible suspicions—and the paper notes safety refusals on gender-related decomposed questions; separating refusal from perception needs neutral rephrased queries.","Editorial inference: a two-stage architecture that first proposes many candidate clues and then verifies each with an explicit \"does this contradict the claim\" step is a direct, testable extension suggested by the decomposed-perception result, and would predict a much higher clued accuracy if search guidance is the bottleneck."],"forward_implications":["Near-ceiling performance on existing multimodal benchmarks should not be read as detective-level perception, because tasks that name the target reward recognition rather than open-ended search.","A model that can identify a clue once told where to look cannot automatically find it in a free-form query; search guidance is a separate capability from perception and reasoning.","Agentic test-time scaling, such as o3's extended reasoning traces, does not by itself solve low-salience, unspecified-target detection.","Practical uses of MLLMs for photo verification, safety screening, or \"spot the problem\" applications would inherit both failure modes measured here: missed real clues and false accusations on innocent images.","Because the decomposed perception questions are answered far more accurately than the open question, a plausible route to improvement is teaching models to generate candidate hypotheses about context before scanning."],"supporting_citations":[{"why":"Introduces Guided Search theory, the framework the paper uses to explain why exhaustive region-by-region search fails on CaughtCheating.","marker":"(Wolfe et al., 1989)"},{"why":"Supplies the five attention-guiding factors, especially bottom-up salience, top-down feature guidance, and scene structure, that organize the failure analysis.","marker":"(Wolfe and Horowitz, 2017)"},{"why":"Source of the o3 model card; o3 is the strongest evaluated model and its 26.0% clued accuracy / 23.9% F1 define the claimed boundary.","marker":"(OpenAI, 2025)"},{"why":"Source of the Gemini-2.5-pro model card; Gemini-2.5-pro is the second-strongest proprietary baseline in the comparison.","marker":"(DeepMind, 2025b)"},{"why":"Humanity's Last Exam motivates the search for tasks beyond near-ceiling benchmark performance, framing CaughtCheating as a boundary probe.","marker":"(Phan et al., 2025)"},{"why":"Technical report for the Qwen2.5-VL family, one of the open-source model families whose near-zero clued accuracies anchor the lower end of the table.","marker":"(Bai et al., 2025)"},{"why":"Papers introducing the InternVL and InternVL2.5 families, which provide the 1B-to-72B open-source baselines.","marker":"(Chen et al., 2024e)"},{"why":"Introduces the LLaVA-OneVision family, another open-source baseline group in the evaluation.","marker":"(Li et al., 2024a)"},{"why":"GPT-4o system card; GPT-4o is the non-reasoning proprietary baseline at 4.0% clued accuracy.","marker":"(OpenAI et al., 2024)"}],"fun_headline_variants":["AI detectives miss 74% of cheating clues","CaughtCheating: MLLMs fail as human-level clue finders","GPT-o3 scores 23.9 F1 on cheating clue detection","MLLMs can't spot cheating clues humans catch easily","Why AI fails at detecting subtle cheating clues"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results stand on hand-assigned labels about which images contain a \"deterministic\" clue and which are genuinely unclued, together with a human F1 of 59.6% that is reported without a description of who the participants were, how many there were, what instructions they received, or whether hints were allowed.","fun_headline_variants_meta":{"raw":{"variants":["AI detectives miss 74% of cheating clues","CaughtCheating: MLLMs fail as human-level clue finders","GPT-o3 scores 23.9 F1 on cheating clue detection","MLLMs can't spot cheating clues humans catch easily","Why AI fails at detecting subtle cheating clues"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000265,"raw_usage":{"total_tokens":1633,"prompt_tokens":998,"completion_tokens":635,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":614,"completion_tokens_details":{"reasoning_tokens":552}},"tokens_in":614,"tokens_out":635,"duration_ms":6958,"temperature":1.0,"reasoning_tokens":552,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:34:47.433468+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled human study on the same 100 images with at least 30 naive participants, no hints, and a pre-registered marking rule; if human F1 lands near the 20–25% range instead of 59.6%, the claimed human–model gap collapses. Independently, have fresh annotators label each clued image's deterministic clue and check agreement; low agreement would mean the accuracy scores are measured against a ground truth the authors themselves could not reliably reproduce.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces Guided Search theory, the framework the paper uses to explain why exhaustive region-by-region search fails on CaughtCheating."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the five attention-guiding factors, especially bottom-up salience, top-down feature guidance, and scene structure, that organize the failure analysis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the o3 model card; o3 is the strongest evaluated model and its 26.0% clued accuracy / 23.9% F1 define the claimed boundary."}],"review_version":1}