{"id":"6bab1cb6-3534-4942-b6ec-26eb0f7367ca","arxiv_id":"2505.10541","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Internal attention patterns in multimodal LLMs are used to define an attention accuracy metric and a benchmark for detecting cases where a model answers correctly while attending to the wrong image.","lead":"This paper proposes a new way to test whether multimodal AI models truly look at the right image when answering picture questions, by reading their internal attention weights. It introduces a benchmark and an attention accuracy metric, but the metric's design raises questions about whether it measures understanding or just gaze.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Attention accuracy is not shown to directly measure visual understanding; the paper's own OCR-needle experiment (Sec. C.3, Table 5) exhibits correct attention alongside failed comprehension, contradicting the §2.1 biconditional.","rationale":"I read the paper as proposing a useful instrument: a new benchmark and an attention-based score for detecting cases where a model answers correctly while not using the right image. The STME dataset, the released code links, and the falsifiable protocol are real positives. My concern is not about the benchmark's existence but about the central interpretive claim in the abstract, that attention accuracy \"directly evaluates the model's visual understanding via internal mechanisms.\" The authors assert rather than prove the biconditional in Section 2.1, and their own Section C.3 sufficiency experiment is the decisive test. It reports high attention accuracy on OCR-needle samples where the model has demonstrably failed to extract the digit string from the image it attended to. On the most natural reading, that is a counterexample to sufficiency: the model looked at the right place but did not understand what it saw. Restricting \"understanding\" to \"image selection\" would make the metric tautological and would not justify the generality claimed for OCR-rich tasks in the STME benchmark. The additional protocol choices in Section 3.1, using each model's own correct-answer subset and taking the maximum over three metrics and several layer windows, further inflate and confound the reported numbers, but the conceptual counterexample is enough to keep the paper from supporting its headline claim. Thus the reader's REJECT remains appropriate; I would not change the verdict.","tokens_in":30171,"tokens_out":9369,"duration_ms":95433,"concrete_test":"Run the paper's released STME code on the Section C.3 OCR-needle condition (or an equivalent 100-item subset of hard tasks where the answer can only be obtained by reading or counting content inside the target image) and compute each model's attention accuracy with Definition 2.3, using the best of LND/M-LND/MC-LND as the paper does. Independently annotate each sample for whether the model's answer or stated reasoning actually uses the required visual content. If attention-correct samples include a nontrivial fraction, for example more than 10%, where the model's answer is wrong or the reasoning ignores the content of the attended image, then attention accuracy is not a direct measure of visual understanding and the abstract's central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the biconditional in Section 2.1: \"MLLM's attention converges onto the target image if and only if there are no IVMs during inference.\" The metric can only license the abstract's claim that attention accuracy \"directly evaluates the model's visual understanding\" if correct attention is sufficient as well as necessary for comprehension. The paper's own Section C.3 provides a counterexample to sufficiency. In the Image Needle in a Haystack task, models correctly localize the image containing the special digit string yet fail to recognize the full string because of limited OCR ability; Table 5 reports near-100% attention accuracy on these samples. So attention correctness is compatible with a clear failure to understand the visual content. The authors frame C.3 as supporting sufficiency, but if \"visual understanding\" is taken literally it supports the opposite conclusion. A model can look at the right image and still not understand what it saw, so the internal attention distribution cannot by itself certify understanding. The cross-model results are also weakened by the protocol in Section 3.1 of computing attention accuracy only on each model's own correctly answered samples, since the compared models are then scored on different subsets.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper defines implicit visual misunderstanding (IVM) as cases where a multimodal large language model (MLLM) gives a correct answer without genuinely understanding the relevant visual content. The authors propose to diagnose IVMs by inspecting the causal attention matrices inside the model: they define image-attention factors, introduce three layer-aggregation rules (LND, M-LND, MC-LND), and define attention accuracy as the proportion of correctly answered samples in which the resulting model-focused image matches the target image. They introduce the STME benchmark of multi-image single-target tasks, report attention-accuracy results across Qwen2VL, InternVL2, and LLaVA-OneVision at multiple scales, test robustness to image-order shuffling, and extend the approach to patch-level analysis and unimodal image-to-image matching.","tokens_in":30443,"tokens_out":7206,"duration_ms":68574,"significance":"If the central claim were fully supported, an attention-based metric that identifies which image a model's response is actually grounded in would be a useful complement to answer-accuracy benchmarks, especially for detecting correct answers reached through the wrong visual evidence. The paper provides a new benchmark, reproducible code/data links, and a clear attention-extraction methodology across several model families. However, the quantitative claims are weakened by the choice to report the maximum over metrics and layer counts, by evaluating attention accuracy only on each model's own correctly answered subset, and by the absence of the stated theorems that underpin the metric. The abstract's stronger claim that attention accuracy 'directly evaluates the model's visual understanding' is not supported by the evidence, including the paper's own OCR-needle experiment, so the contribution currently lands as a promising diagnostic framework rather than a validated direct measure of visual understanding.","major_comments":[{"comment":"The paper repeatedly refers to Theorem 2.1, Theorem 2.2, and Theorem 2.3, but no theorem statements or proofs appear anywhere in the manuscript. The load-bearing biconditional at the end of §2.1 ('MLLM's attention converges onto the target image if and only if there are no IVMs during inference') is asserted without proof and is the basis for defining attention correctness. Either provide precise theorem statements with proofs, or explicitly label these as empirical hypotheses and treat the experiments as tests of that hypothesis rather than consequences of a theorem.","section":"§2.1, §2.3"},{"comment":"The reported attention accuracy is the maximum over the three metrics (LND, M-LND, MC-LND) and over the number of last layers N, as stated in Definition 2.2 and Section 3.1 ('The highest attention accuracy obtained is used as the final value'). This best-of selection over free parameters on the test set inflates the headline numbers. Section E shows that attention-accuracy curves vary substantially with N for the same model, so reporting only the maximum is not a faithful estimate of a fixed evaluation rule. The authors should pre-specify a single metric and N, or report the full distribution over choices, including the average and worst-case values.","section":"§2.3, Definition 2.2, §3.1"},{"comment":"Attention accuracy is computed only on correctly answered samples, and the set of correctly answered samples differs across models. For example, Qwen2VL-7B and InternVL2-8B have substantially different answer accuracies, so their attention-accuracy scores in Table 1 are evaluated on different, non-comparable subsets. A model with lower answer accuracy may be scored on an easier subsample, making the cross-model comparison unreliable. The authors should compute attention accuracy on a common fixed subset (e.g., the intersection of correctly answered samples) or report the per-sample results and quantify how the choice of subset affects the ranking.","section":"§2.3, §3.1, Table 1"},{"comment":"The abstract claims that 'Attention accuracy directly evaluates the model's visual understanding via internal mechanisms,' but this is contradicted by the paper's own Section C.3. There, models achieve near-100% attention accuracy on Image Needle in a Haystack samples while failing to recognize the embedded digit string due to limited OCR ability, meaning attention can be correct while fine-grained visual comprehension fails. At best, the metric measures which image the model's response is based on, not whether the visual content was fully understood. The abstract, Section 1, and the conclusion should be revised to state this weaker claim, and the relationship between attention accuracy and full visual understanding should be discussed explicitly.","section":"Abstract, §2.1, §C.3"},{"comment":"The hard-task filtering pipeline relies on GPT-4o to answer questions and identify images related to the final answer, followed by manual review. Using a model to define the target-image labels introduces a potential model-dependent selection bias, especially if GPT-4o's attention or answer patterns correlate with the tested models' behaviors. The authors should report the number of samples removed at each filtering stage, provide inter-annotator agreement or a human-only validation subset, and discuss how GPT-4o-based filtering could affect the measured attention accuracy.","section":"§2.2"}],"minor_comments":[{"comment":"The LND definition is ambiguous: it defines a per-layer argmax over i for each l in N, but the text then treats the result as a single model-level prediction. Please clarify how the per-layer predictions are aggregated for a single sample.","section":"Eq. (3)"},{"comment":"Several figure axis labels in Section E appear as garbled token-like strings rather than readable text; the attention-accuracy curves are otherwise informative and should be relabeled.","section":"Section E figures"},{"comment":"The term 'equivariant measure' is used for attention accuracy but is never defined; either define it formally or replace it with a clearer description.","section":"§1, §6"},{"comment":"Table 5 has the same caption as Table 4 even though it reports a different experiment (Image Needle in a Haystack with incorrect or partial OCR outputs); the caption should describe the actual content and the sample-selection criterion.","section":"Table 5 caption"},{"comment":"Please provide a per-task breakdown of the 537 easy and 528 hard samples, the number of images per question in each task, and the dataset license and hosting details for the STME benchmark.","section":"§2.2"},{"comment":"There are several typos and formatting issues, including 'valuates' (§2.2), 'the the methodology' (§3.1), 'Extrcting submatrix' (§4.2), and 'Noted that' (Definition 2.1); a careful proofread is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The core attention-extraction methodology is potentially useful, but the paper's current evaluation protocol is not sound enough for the central quantitative claims: reporting the maximum over three metrics and over N, and conditioning on each model's own correct-answer subset, makes the headline attention-accuracy numbers inflated and the cross-model comparisons non-comparable. The missing theorem statements and the overclaim that attention accuracy directly measures visual understanding (contradicted by the paper's own OCR-needle experiment) also need to be addressed. I believe these issues are fixable within the scope of a major revision, so I do not recommend rejection, but the revised manuscript would need to substantially rework the evaluation protocol and reframe the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this paper is worth engaging with, but the headline claim doesn't survive contact with its own appendix. The STME benchmark and the observation that deep-layer attention converges on the target image are real contributions. Table 1 and the scale experiments give a plausible picture of cross-model differences, and the positional-bias shuffling in Table 2 is a nice robustness check. The authors also provide data and code links, which helps.\n\nWhat's actually new: a single-target multi-image benchmark, an attention-derived accuracy metric, and extensions to patch-level and unimodal settings. The convergence phenomenon itself—attention concentrating on the answer-bearing image as layers deepen—is worth documenting and is consistent with prior attention analysis work while going further in scale.\n\nNow the soft spots, and they are load-bearing rather than cosmetic.\n\nFirst, the formal claim in Section 2.1 is that attention converges onto the target image if and only if there are no IVMs. That biconditional is what licenses calling attention accuracy a direct measure of visual understanding. The paper's own Section C.3 gives a counterexample: in the Image Needle in a Haystack task, models attend to the correct image but fail to recognize the full digit string due to limited OCR ability. Table 5 shows near-perfect attention accuracy on exactly those samples. So a model can look at the right place and still not understand what it saw. Attention localization is at best necessary, not sufficient. The authors frame C.3 as support, but taken literally it undermines the abstract's central claim.\n\nSecond, the evaluation protocol. Attention accuracy is computed only on the subset of samples each model answered correctly (Section 3.1). Since different models get different subsets correct, the cross-model comparisons in Table 1 and Figure 5 are not apples-to-apples. A model that answers fewer hard questions but looks at the right image on the ones it does answer will look artificially strong. The paper needs to report the joint distribution of answer correctness and attention correctness, or otherwise control for the subset.\n\nThird, the reported numbers are selected as the maximum over three metrics (LND, M-LND, MC-LND) and over a sweep of N values. Appendix E shows that the three metrics disagree substantially, especially for smaller models. Reporting the max hides that instability and inflates the accuracy values. The \"scale-agnostic\" claim needs to be demonstrated, not asserted.\n\nWho is this for? People working on MLLM evaluation or attention-based interpretability. The STME dataset could be reused even if the metric is reframed more cautiously. The paper deserves a serious referee, but it needs a major revision: soften the sufficiency claim, fix the subset problem, and report the actual metric curves rather than the maximum.\n\nRecommendation: send it to peer review, but with the expectation of substantial changes before it can be accepted.","headline":"A useful new benchmark and a genuinely interesting attention phenomenon, but the central claim that attention accuracy measures visual understanding is contradicted by the paper's own OCR experiment and weakened by post hoc metric selection.","tokens_in":30953,"tokens_out":2445,"would_cite":false,"duration_ms":23689,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A model can give a correct answer while its attention is locked on the wrong image, and this paper introduces attention accuracy to expose that gap.","keywords":["implicit visual misunderstanding","attention accuracy","multimodal large language models","multi-image understanding","attention analysis","causal attention","STME benchmark"],"falsifier":"Take samples where attention accuracy is high, hide every image except the attended one, and ask whether the original answer can still be reproduced; if the answer cannot, attention was not carrying the understanding the metric credits it with.","tokens_in":29985,"feed_emoji":"👁️","tokens_out":10383,"duration_ms":97882,"temperature":0.7,"pith_summary":"Most benchmarks judge multimodal language models by whether the final answer is right. This paper argues that a model can answer correctly while actually looking at the wrong image or misusing the right one, a failure it names implicit visual misunderstanding (IVM). The central claim is that the model's causal attention reveals the gap: as layers deepen, attention concentrates on the image that supports the correct answer, and when it settles on a different image despite a right answer, an IVM has occurred. If true, attention accuracy would offer a more reliable way to evaluate visual understanding than answer correctness, and it would expose cases where a strong language model is masking a weak visual system.","feed_headline":"Attention accuracy catches AI answering right while viewing wrong image","feed_subtitle":"New metric reads internal attention to expose implicit visual misunderstandings that answer accuracy alone misses.","key_machinery":"The image-attention factor $\\sigma_i$: after softmax is applied to the causal attention matrix, the query rows belonging to the caption or question and to the model output are multiplied by the key columns of each image and averaged across attention heads and token positions, giving a per-image, per-layer concentration score. Three voting rules over the last layers — layer-focused image, mean layer-focused image, and majority-count layer-focused image — turn these scores into a single model-focused image, and attention accuracy is the fraction of correctly answered samples in which that image equals the target. The STME benchmark supplies the target-image association that makes this scoring possible, including a filtering pipeline that keeps only questions whose correct answer depends on exactly one image.","core_discovery":"The paper defines attention accuracy as a metric that compares the image a model's attention locks onto with the single image actually tied to the correct answer. To compute it, the authors separate the visual and textual blocks of the causal attention matrix and average soft attention scores over heads and tokens to form an image-attention factor per image per layer; three layer-based voting rules then name a model-focused image. On the new STME benchmark, which pairs easy caption-matching and retrieval tasks with harder document, textbook, slide, and spatial tasks, the paper reports that attention accuracy reaches 100 percent on easy tasks for Qwen2VL-7B, that it pulls apart model families whose answer accuracy looks similar, that it rises consistently with model scale, and that it stays stable under image-order shuffling. The same factor is extended to image patches and to image-to-image similarity, supporting the claim that the measure is scale- and architecture-agnostic.","pith_inferences":["The paper leaves training applications implicit, but attention accuracy could plausibly be used as a training signal or data filter that penalizes models for answering correctly while attending to the wrong image.","The paper's own OCR experiments in Section C.3 mark a boundary: attention can land on the correct image while the model still misreads fine-grained content, so attention accuracy is best read as measuring target localization, not full semantic comprehension.","Because the same factor applies to image-to-image attention, the approach could transfer to text-only or audio-visual transformers with little modification, though the paper only sketches this possibility.","The proposed if-and-only-if relation could be probed adversarially by giving the model samples whose answers are recoverable from the caption alone; stable target attention there would suggest convergence is tracking textual priors rather than visual understanding."],"forward_implications":["Answer accuracy and attention accuracy become two separate axes: a right answer with wrong attention reveals an implicit misunderstanding, while a wrong answer with right attention localizes the failure to recognition or reasoning rather than image selection.","Because attention accuracy varies by less than one percent when image order is shuffled, while answer accuracy varies several times more, model comparisons become fairer across different input arrangements.","Across scales within a model family, attention accuracy rises even where answer accuracy does not, isolating gains in visual capability from gains in the downstream language model.","On hard tasks the spread in attention accuracy across models is wider, so the metric can serve as a more sensitive diagnostic than answer accuracy when tasks demand close visual reading.","The patch-level extension identifies which regions of an image drive the answer, allowing hallucinations or misreadings to be traced to particular visual content."],"supporting_citations":[{"why":"Supplies the row-wise partition of attention matrices that the image-attention factor is built on.","marker":"Ben Melech Stan et al., 2024"},{"why":"Qwen2VL is the primary model where deep-layer attention convergence toward the target image is observed and benchmarked.","marker":"Wang et al., 2024b"},{"why":"InternVL2 provides the cross-series comparison showing that attention accuracy and answer accuracy diverge.","marker":"Chen et al., 2024c"},{"why":"LLaVA-OneVision's fixed-size image tokens enable the patch-level extension of the attention factor.","marker":"Li et al., 2024"},{"why":"Flickr30k supplies the target images and captions for the easy caption-matching task.","marker":"Young et al., 2014"},{"why":"OBELICS supplies candidate images for caption matching and for the image-to-image similarity set.","marker":"Laurençon et al., 2024"},{"why":"MileBench provides the long-context and multi-image samples that feed the STME easy and hard tasks.","marker":"Dingjie et al., 2024"},{"why":"Needle in a Multimodal Haystack serves as the easy retrieval task and as the basis for the OCR sufficiency analysis.","marker":"Wang et al., 2024c"},{"why":"GPT-4o is used to filter hard-task samples down to single-target questions and to validate the chain-of-thought reasoning.","marker":"OpenAI, 2024"}],"fun_headline_variants":["Attention accuracy exposes AI's wrong-image focus","Why AI can answer right while looking at wrong image","New metric reveals hidden visual blind spots in AI","Attention analysis uncovers implicit visual misunderstandings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the image a model's attention locks onto is the image the model has genuinely understood, so if attention can be correct while understanding fails, attention accuracy overstates visual comprehension.","fun_headline_variants_meta":{"raw":{"variants":["Attention accuracy exposes AI's wrong-image focus","Why AI can answer right while looking at wrong image","New metric reveals hidden visual blind spots in AI","Attention analysis uncovers implicit visual misunderstandings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000638,"raw_usage":{"total_tokens":2907,"prompt_tokens":884,"completion_tokens":2023,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":1966}},"tokens_in":500,"tokens_out":2023,"duration_ms":14106,"temperature":1.0,"reasoning_tokens":1966,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:06:54.555903+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take samples where attention accuracy is high, hide every image except the attended one, and ask whether the original answer can still be reproduced; if the answer cannot, attention was not carrying the understanding the metric credits it with.","supporting_citations":[{"cited_title":"H., Yu, F., Wan, X., and Wang, B","cited_arxiv_id":null,"evidence_quote":"MileBench provides the long-context and multi-image samples that feed the STME easy and hard tasks."},{"cited_title":"Gpt-4o system card, 2024","cited_arxiv_id":null,"evidence_quote":"GPT-4o is used to filter hard-task samples down to single-target questions and to validate the chain-of-thought reasoning."}],"review_version":1}