{"id":"08484f03-b20e-4440-b757-c8d15bc3d8fe","arxiv_id":"2607.05978","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":3,"one_line_summary":"MTLA is a training-free, post-hoc confidence score for multimodal LLM localization that restricts attention aggregation to the model's own predicted region and tokens, substantially improving hallucination detection and re-ranking across image, video, and audio.","lead":"This paper introduces a training-free method called MTLA that scores the reliability of an AI model's localized predictions (like bounding boxes or time intervals) by checking if the model's internal attention actually focuses on the region it predicted. A smart generalist might read it because it nearly doubles the zero-shot object detection accuracy of an 8B multimodal model without any additional training, narrowing the gap to specialized supervised detectors.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"Headline AP gain (20.4→37.0) bundles MTLA ranking with N=16 self-consistency voting; the paper does not report a self-consistency-only baseline (vote-weighted NMS without MTLA) that would isolate MTLA's marginal contribution to the final number.","rationale":"The reader identified the attention-as-proxy assumption as the weakest link, which is a reasonable concern but one the paper addresses empirically (AUROC 89.0, ablations in Table 6, size-bias analysis in Supp. C). My concern is different: it's about attribution of the headline AP gain. The N=1 MTLA result (32.1 AP, +57%) is strong and supports the core claim on its own. The 'nearly doubles' framing uses the N=16 number, which includes self-consistency voting gains that are orthogonal to MTLA's attention-based ranking. However, this is a framing issue, not a methodological flaw — the paper is transparent about the N=1 vs N=16 distinction in Table 3 and Figure 6a. The method is simple, well-motivated, validated across four benchmarks and three modalities, and the ablations confirm both the localized-attention and multi-token-aggregation components contribute. The code is public. The concern about the missing vote-only baseline is worth noting but does not undermine the paper's contribution: MTLA clearly outperforms all baselines at every N, and the N=1 result is already a substantial gain. Verdict remains ACCEPT.","tokens_in":18803,"tokens_out":5757,"duration_ms":407947,"concrete_test":"On COCO val2017 with Qwen3-VL-8B, run N=16 stochastic rollouts (temperature 0.7, top-p 0.95) and merge with NMS at IoU≥0.5 using vote count alone as confidence (no MTLA, no SVAR). Report COCO AP. If vote-only AP exceeds 34, MTLA's marginal contribution at N=16 is under 3 points and the 'nearly doubles' attribution is misleading; if vote-only AP is below 30, MTLA is the dominant contributor and the framing is fair.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that MTLA 'nearly doubles' COCO detection AP from 20.4 to 37.0. However, this figure uses N=16 stochastic rollouts with vote×MTLA fusion (Supp. A.1). The paper reports MTLA at N=1 as 32.12 AP (Table 3), so MTLA's ranking contribution alone accounts for +11.7 AP (a 57% increase, not 'nearly doubles'). The remaining ~5 points come from self-consistency voting (N=1→16). SVAR at N=16 reaches 34.6 (Fig. 6a), showing that self-consistency with any reasonable ranking captures much of the gain. The critical missing baseline is: N=16 rollouts merged with vote-weighted NMS but without MTLA (i.e., vote count only as confidence). If this baseline reaches, say, 33–35 AP, then MTLA's marginal contribution at N=16 is only 2–4 points, and the 'nearly doubles' framing substantially over-attributes the gain to MTLA rather than to the 16× oversampling. The paper is transparent about reporting N=1 separately, so this is a framing concern rather than a hidden flaw — the N=1 result (32.1 AP) already supports the core claim that MTLA is an effective confidence score. But the headline number and abstract attribute the full 37.0 to MTLA without isolating the self-consistency contribution.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"This paper proposes Multi-Token Localized Attention (MTLA), a training-free, post-hoc confidence score for localized predictions from multimodal large language models (MLLMs). The core idea is to measure how strongly a prediction's tokens attend to the input-modality tokens within the predicted region, aggregating across all prediction tokens. The method is evaluated across image detection (COCO), video temporal grounding (Charades-STA, QVHighlights), and audio localization (AudioSet-Strong), showing consistent improvements in hallucination AUROC and re-ranking accuracy over prior training-free baselines such as SVAR. The ablations in Section 5 cleanly decompose the contributions of the region mask and multi-token aggregation. The central claim—that MTLA is an effective, general confidence score for MLLM localization—is well-supported by the experiments.","tokens_in":19575,"tokens_out":1386,"duration_ms":290805,"significance":"The paper addresses a practical and timely problem: MLLMs hallucinate localized predictions prolifically, and token log-probabilities are nearly uninformative for grounding quality. The proposed MTLA score is parameter-free (no fitted weights, no external models), architecture-agnostic, and applies uniformly across image, video, and audio modalities with no per-benchmark tuning. The layer band (L8-21) is borrowed from prior work (SVAR) and reused without per-benchmark adjustment. The code is released. The falsifiable prediction—that attention mass inside the claimed region separates grounded from hallucinated predictions more sharply than global attention or token probabilities—is confirmed across four benchmarks and three model families. The nearly doubling of zero-shot COCO detection AP (20.4 to 37.0) is a notable result, though the framing of this headline number requires clarification (see major comments).","major_comments":[{"comment":"The headline claim that MTLA 'nearly doubles' COCO detection AP from 20.4 to 37.0 (Abstract, Sec. 1, Table 3) bundles MTLA re-ranking with N=16 self-consistency voting. Table 3 reports MTLA at N=1 as 32.12 AP, so MTLA's ranking contribution alone is +11.7 AP (a 57% increase). The remaining ~5 points come from self-consistency voting (N=1 to N=16). The critical missing baseline is: N=16 rollouts merged with vote-weighted NMS but without MTLA (i.e., vote count only as confidence). Fig. 6a shows SVAR at N=16 reaches 34.6 AP, suggesting that self-consistency with any reasonable ranking captures much of the gain. If a vote-count-only baseline at N=16 reaches 33-35 AP, then MTLA's marginal contribution at N=16 is only 2-4 points, and the 'nearly doubles' framing substantially over-attributes the gain to MTLA rather than to the 16x oversampling. The paper is transparent about reporting N=1 (32.","section":null},{"comment":"1 AP) separately, so this is a framing concern rather than a hidden flaw. The N=1 result already supports the core claim. However, the abstract and headline number attribute the full 37.0 to MTLA without isolating the self-consistency contribution. Adding the vote-count-only baseline at N=16 and adjusting the framing accordingly would resolve this.","section":null}],"minor_comments":[{"comment":"Abstract: the AUROC improvement range is stated as '+7 to +38' in the reader's report but as '+7 to +21' in the abstract text provided. Please reconcile the exact range with the reported numbers in Table 2 and Table 10.","section":null},{"comment":"Table 3: the Gemini API runs (Gemini 2.5 Flash, Pro, 3.0 Flash) are listed alongside training-free post-hoc scores but use a different model family and single-sample (N=1) API calls. A footnote or separator clarifying that these are not head-to-head comparisons with the Qwen3-VL re-ranking results would improve readability.","section":null},{"comment":"Supp. C: the region-size attention bias is acknowledged but not fully resolved. The paper states that normalization schemes degraded performance and that MTLA outperforms SVAR across all size categories. This is a reasonable empirical defense, but a brief discussion of why the bias does not invalidate cross-prediction comparisons (since AP is threshold-dependent) would strengthen the argument.","section":null},{"comment":"Sec. 3.2, self-consistency voting: the fusion rule differs between COCO (sum of cluster MTLA scores) and other modalities (max of cluster MTLA scores). The rationale for this difference is stated briefly but could be elaborated, as it affects the comparability of results across modalities.","section":null},{"comment":"Table 6: the 'label mean' row is marked as undefined (—) for QVHighlights because predictions are bare [start, end] timestamps. This is clear, but the table caption could note that the dash indicates the token set is not applicable to the output format, not that the experiment was not run.","section":null},{"comment":"Supp. A.4: the eager-attention forward pass requirement (bypassing FlashAttention) is a practical limitation. The paper notes this in Sec. 6, but the supplementary could briefly discuss the computational overhead of the eager pass relative to the standard fused-attention inference.","section":null},{"comment":"Sec. 4.4 / Table 5: the audio localization pipeline (Supp. A.3) involves a propose-and-localize cascade with an auxiliary language model for label normalization. This is more complex than the image/video setup. A brief note in the main text that the audio pipeline differs structurally would set appropriate expectations.","section":null}],"recommendation":"minor_revision","confidential_remarks":"The stress-test concern about the headline number (20.4 to 37.0) bundling MTLA with self-consistency is valid and well-targeted. The paper is transparent about reporting N=1 separately (32.12 AP), which already supports the core claim. The missing vote-count-only baseline at N=16 is a genuine gap, but it is addressable with a single additional experiment row and revised framing. I do not see this as a load-bearing error that invalidates the method—MTLA's contribution at N=1 is clear—but the framing should be corrected before publication. The paper is otherwise solid, with clean ablations and broad cross-modal validation."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for the careful reading and the constructive framing suggestion. The referee's single major comment is well-taken: the headline number of 37.0 AP bundles MTLA re-ranking with N=16 self-consistency voting, and the missing baseline (vote-count-only at N=16) is needed to isolate MTLA's marginal contribution at that setting. We agree this should be added and the framing adjusted.","responses":[{"response":"The referee is correct on both counts: (1) the headline number of 37.0 AP bundles MTLA with N=16 self-consistency voting, and (2) the vote-count-only baseline at N=16 is missing and is needed to isolate MTLA's marginal contribution at that setting. We will add this baseline and adjust the framing accordingly. We agree that the N=1 result (32.12 AP, a +11.7 / +57% improvement over raw 20.43) is the cleaner demonstration of MTLA's contribution as a confidence score, and we will foreground it in the abstract and introduction. We will also reframe the 37.0 AP figure to explicitly attribute the components (MTLA re-ranking at N=1 vs. the additional gain from N=16 self-consistency voting), rather than presenting the combined number as if it were solely due to MTLA. We note that Figure 6a already shows that SVAR at N=16 reaches 34.6 AP, which is consistent with the referee's expectation that a vote-count-only baseline would land in the 33–35 range; the vote-count-only baseline will make this explicit. We will add it to Table 3 and/or Figure 6a. We will retain the 37.0 AP result as the best achievable number under our full pipeline, but will no longer describe it as MTLA 'nearly doubling' AP without qualification. Instead, we will state that MTLA at N=1 improves AP by 57% (20.4 → 32.1) and that combining it with self-consistency voting yields 37.0 AP, with the vote-count-only baseline reported alongside to show the marginal contribution of each component.","revision_made":"yes","referee_comment":"The headline claim that MTLA 'nearly doubles' COCO detection AP from 20.4 to 37.0 bundles MTLA re-ranking with N=16 self-consistency voting. Table 3 reports MTLA at N=1 as 32.12 AP, so MTLA's ranking contribution alone is +11.7 AP (a 57% increase). The remaining ~5 points come from self-consistency voting (N=1 to N=16). The critical missing baseline is: N=16 rollouts merged with vote-weighted NMS but without MTLA (i.e., vote count only as confidence). Fig. 6a shows SVAR at N=16 reaches 34.6 AP, suggesting that self-consistency with any reasonable ranking captures much of the gain. If a vote-count-only baseline at N=16 reaches 33-35 AP, then MTLA's marginal contribution at N=16 is only 2-4 points, and the 'nearly doubles' framing substantially over-attributes the gain to MTLA rather than to the 16x oversampling. The paper is transparent about reporting N=1 separately, so this is a framing"}],"tokens_in":18568,"tokens_out":799,"duration_ms":87465,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"The main thing to know: MTLA is a real idea that works. The paper takes the existing SVAR approach (sum attention from a single response token to all visual tokens) and makes two changes — restrict the sum to tokens inside the predicted region, and average across all prediction tokens instead of reading one. Both changes are cleanly ablated in Table 6, and the region mask is the bigger lever, especially on video (+43 AUROC on QVHighlights). The method is training-free, parameter-free aside from a fixed layer band borrowed from SVAR, and ships code. The cross-modality generality (image, video, audio with the same recipe) is a genuine strength, not a stretch — the formulation is modality-agnostic by construction and the audio pipeline is honestly described as a cascade wrapper rather than claimed as native. AUROC improvements over SVAR are large and consistent across all four benchmarks and two model families. This is good work. The stress-test concern about the headline 20.4→37.0 AP number is valid but not damning. The paper does report N=1 separately (32.12 AP, Table 3), which already supports the core claim that MTLA is an effective confidence score. The jump from 32.1 to 37.0 at N=16 comes from self-consistency voting, and the abstract attributes the full gain to MTLA without isolating the voting contribution. The missing baseline — N=16 rollouts with vote-count-only confidence, no MTLA — would clarify how much of the final 5 points is MTLA vs. oversampling. Figure 6a shows SVAR at N=16 reaches 34.6, so a vote-only baseline probably lands somewhere in the 33–35 range, meaning MTLA's marginal contribution at N=16 is real but modest. The framing should be tightened, but the N=1 result stands on its own. The region-size bias from softmax normalization (Supp. C) is acknowledged and empirically addressed — MTLA beats SVAR across all size categories. The layer band is reused from SVAR without per-benchmark tuning, and Table 7 shows the full-layer-range result is nearly as strong, so this isn't a sensitivity problem. The eager-attention requirement (bypassing FlashAttention) is a practical limitation the paper is upfront about. Who benefits: researchers working on MLLM reliability, hallucination detection, or zero-shot localization. The paper also makes a useful contribution by benchmarking MLLMs under standard detection metrics (AP, Recall@1, PSDS1), which the community should adopt. Recommend accepting for peer review. The core method is novel, well-tested, and reproducible. The headline framing issue is fixable in revision and does not undermine the contribution.","headline":"MTLA: a simple, training-free attention-based confidence score for MLLM localization that works across image, video, and audio — the headline AP number over-attributes gains to MTLA by bundling self-consistency voting, but the core method is sound and genuinely novel.","tokens_in":19800,"tokens_out":673,"would_cite":true,"duration_ms":135939,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Attention Inside the Box: A Training-Free Score That Nearly Doubles MLLM Detection Accuracy","keywords":["multimodal large language models","object detection","hallucination detection","attention mechanisms","temporal grounding","training-free confidence estimation","zero-shot localization"],"falsifier":"If grounded and hallucinated predictions showed equal attention mass inside their proposed regions—e.g., if the model attended inside its claimed box regardless of whether a real object was there—MTLA would collapse to chance and provide no re-ranking benefit.","tokens_in":18923,"feed_emoji":"🎯","tokens_out":1223,"duration_ms":175522,"temperature":0.7,"pith_summary":"Multimodal large language models can emit bounding boxes and temporal windows, but between 58% and 68% of their localized predictions are hallucinations—regions that do not correspond to any real object or event. The model's own token probabilities are nearly useless for telling good predictions from bad ones, because coordinate tokens become near-deterministic once the model commits to a location. This paper proposes Multi-Token Localized Attention (MTLA), a post-hoc, training-free confidence score that works by reading the model's internal attention maps and checking whether the tokens that encode a predicted region actually attend to the input patches inside that same region. The core insight is twofold: grounded predictions concentrate their attention within their own proposed region, while hallucinated predictions spread attention across contextual distractors elsewhere; and this signal is distributed across all the prediction's tokens (coordinates and label), so aggregating across them yields a far more robust score than reading any single token. Used to re-rank predictions, MTLA lifts an 8B generalist model's zero-shot COCO detection AP from 20.4 to 37.0, approaching supervised specialist detectors, and produces similar gains across video temporal grounding and audio event localization—all without any task-specific training or tuning.","feed_headline":"Attention Inside the Box Nearly Doubles Zero-Shot MLLM Detection","feed_subtitle":"A training-free score that checks whether a model's prediction tokens attend to the region they claim lifts COCO AP from 20.4 to 37.0, no re","key_machinery":"Multi-Token Localized Attention (MTLA): a scalar score computed by (1) identifying the response tokens that encode a prediction's region coordinates and label, (2) masking the decoder's attention weights to keep only those directed at input-modality tokens falling inside the predicted region, (3) summing the masked attention per token, (4) averaging across all prediction tokens, and (5) averaging across attention heads and a fixed band of middle transformer layers.","core_discovery":"The paper's central discovery is that a multimodal LLM's decoder self-attention, when restricted to the modality tokens inside the model's own predicted region and averaged across all tokens comprising that prediction, functions as a reliable grounding confidence signal. Prior attention-based methods summed attention globally across the entire input and read a single response token; the paper shows these are weaker special cases. The localized restriction is what makes the signal discriminative—especially on longer inputs like video, where global attention is nearly uninformative (AUROC 41.5 for the global baseline vs. 80.0 for the localized version on QVHighlights). The multi-token聚合 is the","pith_inferences":["If MTLA is reading a real grounding mechanism rather than a correlate, then models trained with explicit attention-alignment losses (forcing prediction tokens to attend inside their proposed regions) should show both higher base localization accuracy and reduced MTLA gains—testing whether MTLA is measuring a cause or a symptom.","The region-size bias from softmax normalization (larger regions get more attention mass by construction) suggests that a non-softmax or entropy-normalized attention variant might recover the small-object performance that MTLA currently sacrifices, even though naive area normalization hurt in the paper's experiments.","The method's reliance on eager-attention forward passes (bypassing fused kernels like FlashAttention) creates a practical deployment tension: the very inference optimizations that make MLLMs fast also destroy the signal MTLA needs, which may motivate hardware-software co-design for attention map extraction.","If the grounding signal lives in middle-layer attention patterns regardless of modality, this raises the question of whether it emerges from the grounding training data itself or from a more general property of how transformers integrate spatial/temporal information—a question testable by applying MTLA to non-grounding-trained models."],"forward_implications":["Generalist MLLMs can serve as practical localization systems without task-specific training, provided a reliable post-hoc confidence signal is available to filter hallucinated predictions.","The same attention-localization principle transfers across image, video, and audio with no modality-specific modifications beyond constructing the region-to-token mask, suggesting the mechanism is a general property of how grounding MLLMs process spatial and temporal information.","Token log-probabilities are fundamentally inadequate as confidence signals for structured coordinate outputs, because they reflect decode-order artifacts and coarse-to-fine digit refinement rather than grounding quality.","Standard detection metrics (AP, Recall@1, PSDS1) should become a routine part of MLLM evaluation, not just captioning or VQA benchmarks, to expose the hallucination rates these models exhibit in localization tasks."],"fun_headline_variants":["Localize Attention to Predicted Regions to Catch MLLM Hallucinations","Restricted Self-Attention Doubles Zero-Shot MLLM Detection AP","Multi-Token Localized Attention Exposes MLLM Grounding Failures","Checking If Prediction Tokens Attend to Their Region Lifts COCO AP to 37.0","Training-Free Confidence Score From Localized Attention Narrows Gap to Supervised Detector"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The method assumes that the mass of decoder self-attention from prediction tokens onto input-modality tokens inside the predicted region is a faithful, unbiased indicator of whether the model actually grounded its prediction in the input. This requires extracting raw attention maps (bypassing optimized attention kernels), and the paper acknowledges a mathematical bias toward larger regions from softmax normalization that it does not fully resolve, relying instead on the empir","fun_headline_variants_meta":{"raw":{"variants":["Localize Attention to Predicted Regions to Catch MLLM Hallucinations","Restricted Self-Attention Doubles Zero-Shot MLLM Detection AP","Multi-Token Localized Attention Exposes MLLM Grounding Failures","Checking If Prediction Tokens Attend to Their Region Lifts COCO AP to 37.0","Training-Free Confidence Score From Localized Attention Narrows Gap to Supervised Detectors"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":700,"prompt_tokens":614,"completion_tokens":86,"prompt_tokens_details":null},"tokens_in":614,"tokens_out":86,"duration_ms":62948,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-08T19:27:48.936857+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If grounded and hallucinated predictions showed equal attention mass inside their proposed regions—e.g., if the model attended inside its claimed box regardless of whether a real object was there—MTLA would collapse to chance and provide no re-ranking benefit.","supporting_citations":[],"review_version":1}