{"id":"08f8754f-ad3a-4029-bc92-db0f5f656533","arxiv_id":"2509.06461","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CARVE contrasts attention maps from a general prompt and a specific question to mask out visual noise, then re-asks the question on the cropped and enlarged image, improving VQA accuracy by up to 75% on some benchmarks.","lead":"Vision-language models get distracted by busy images. This paper introduces CARVE, a training-free method that uses the model's own attention to find and zoom in on the relevant part of an image, improving accuracy on several visual question answering benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains may be inflated: key hyperparameters (p, K, λ, L, T, and even the general instruction) are selected on the evaluation sets with no held-out validation split or error bars; a validation-based protocol is needed before the 'consistently enhances' claim is supported.","rationale":"The reader's verdict is CONDITIONAL, and my stress-test supports the same verdict, so no change is needed. The reader's weakest_assumption was the theoretical claim that general-instruction attention approximates pure visual noise (Eq. 4.2). That is a real weakness, but it is not the most load-bearing issue for the paper's headline empirical assertion: CARVE could still improve accuracy even if the decomposition is only a useful heuristic. The more decisive concern is whether the reported improvements are real and generalizable. The paper tunes the general instruction in Appendix D and the masking hyperparameters in Figure 7 on subsets of the evaluation data, then reports the best numbers in Tables 1 and 2 without error bars or significance tests. Some improvements are small in absolute terms, and the V* benchmark has only 191 questions, so the 'up to 75%' headline gain is based on a small sample with favorable hyperparameter selection. The consistency across four models and four datasets is encouraging, but it does not rule out selection effects. A proper validation protocol is a concrete, minimal check that would settle whether the central empirical claim holds. The theoretical concern should also be addressed, ideally by comparing contrastive attention masks against human-labeled distractors, but the empirical protocol issue is the one that most directly affects the paper's acceptance as a reliable method.","tokens_in":19213,"tokens_out":3862,"duration_ms":39847,"concrete_test":"Split each benchmark into a validation fold and a held-out test fold before any hyperparameter search; fix p, K, λ, L, T, and G using only the validation fold, then report test-set accuracy with 95% bootstrap confidence intervals over questions for both CARVE and the w/o-CARVE baseline. If the validated gains are not positive for all model-dataset pairs, or if the confidence intervals overlap zero for the smaller deltas, the 'consistently enhances' claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that CARVE consistently improves VQA accuracy across models and datasets, but the comparisons in Tables 1 and 2 rely on hyperparameters chosen with access to evaluation data. Appendix D selects the general instruction G by maximizing TextVQA accuracy on a 1,000-instance sampled subset, and Figure 7 selects p and K on another 1,000-instance TextVQA subset; no separate validation split is used before reporting the best configurations. Many reported deltas are small in absolute terms (e.g., A-OKVQA QWEN2.5-VL-3B +4.79 to +7.26; POPE +0.23 to +1.73), and V* contains only 191 questions, so selection effects and sampling noise could account for part or all of the observed gains. The theoretical decomposition is also definitional: Appendix C.1 sets F_vis(I) equal to A^(G)(I), so Eq. 4.2 is an assumption rather than a derived result, and the subsequent 'proof' does not independently establish that general-instruction attention isolates image-inherent noise. However, the empirical claim could survive a weak theory; the evaluation protocol is what currently blocks verification of the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CARVE, a training-free method that improves VLM visual question answering by masking image regions deemed irrelevant. The method extracts attention maps under a task question and under a general instruction, computes a contrastive ratio A^(Q)/(A^(G)+lambda), thresholds and crops the resulting map, and re-runs the VLM on the refined image. The manuscript first reports correlational evidence that visual complexity (texture and color) correlates with attention entropy and that entropy correlates negatively with accuracy. It then presents a theoretical decomposition of attention into a visual-noise factor and a semantic-signal factor, claims a closed-form solution, and reports consistent accuracy gains over four datasets (A-OKVQA, POPE, V*, TextVQA) and four VLMs, with relative gains up to about 75% on V*. The paper also includes ablations over time steps, layer ranges, mask percentiles, and keep-region counts, plus a comparison with external tool-based baselines.","tokens_in":19548,"tokens_out":3599,"duration_ms":35951,"significance":"If the empirical claim is verified, CARVE would be a useful, simple, and inexpensive pre-processing method for improving VLM visual reasoning, and the attention-entropy analysis would contribute a genuinely interesting observation about how visual clutter degrades attention. The paper has several concrete strengths: it evaluates on four models and four datasets, includes ablations over time steps and layers, compares against SAM/YOLO/CLIP and ViCrop baselines, and gives detailed implementation information. The 'training-free' property and the pixel-level masking idea are clearly stated and easy to reproduce in principle. However, the current evaluation protocol does not yet support the 'consistently enhances' claim, and the theoretical decomposition is definitional rather than derived; these two issues are load-bearing and require revision before the central claims can be accepted.","major_comments":[{"comment":"Hyperparameters are selected on evaluation data, so the reported gains are likely optimistic. Appendix D chooses the general instruction G by maximizing accuracy on a 1,000-instance TextVQA subset, and Figure 7 selects p and K on another 1,000-instance TextVQA subset; the same TextVQA set then appears in Tables 1 and 2. No separate validation split is held out before the best configuration is reported. Because several reported deltas are small in absolute terms (e.g., A-OKVQA QWEN2.5-VL-3B +4.79 to +7.26; POPE +0.23 to +1.73), selection effects alone could account for part or all of the observed improvements. Please report a validation protocol with a disjoint test set, or at minimum provide per-configuration confidence intervals on a held-out split.","section":"Appendix D and Figure 7"},{"comment":"The attention-decomposition 'proof' is circular. In Definition 1, Eq. (4.1) postulates A(Q) = F_vis ⊗ F_sem, and Eq. (4.2) sets A(G) ≈ F_vis by assuming F_sem(G,I) ≈ 1_Nv. Theorem C.1 then defines F_vis through A(G) and obtains the decomposition by construction; the closed-form solution in Eq. (4.6) follows algebraically from these definitions rather than from an independent argument. The assumption that general-instruction attention isolates image-inherent noise is empirically testable, and it is not tested here. Please either re-frame Section 4.1 as a heuristic motivation (which is acceptable), or provide a validation of the assumption, for example by comparing the general-instruction mask against human-labeled distractors or an independent noise estimate.","section":"Section 4.1 and Appendix C.1"},{"comment":"The experiments lack error bars and significance tests, and the V* benchmark contains only 191 questions, so several headline improvements correspond to very small absolute counts. For instance, a 75.24% relative improvement for LLAVA1.5-13B on V* is roughly a 32-question swing, and the POPE deltas of +0.23 to +3.10 are within typical run-to-run variation for binary probing. Greedy decoding is deterministic, but the selection of attention layers and time steps still introduces multiple-comparison risk. Please report standard errors or bootstrap confidence intervals, and state how many protocol choices were explored before the reported configuration was selected.","section":"Tables 1, 2, and Appendix E"},{"comment":"The theoretical error analysis does not repair the circularity. Theorem C.4 assumes F_sem(G,I) = 1_Nv + epsilon and then bounds the error of the estimator A^(Q)/(A^(G)+lambda) relative to F_sem; this is a consistency check of the algebraic estimator, not an independent derivation. The paper would be strengthened by explicitly acknowledging that the decomposition is an assumption and by concentrating the theoretical contribution on the convexity and stability properties of the optimization, which are proved correctly in Appendices C.2 and C.3.","section":"Eq. (4.6) and Theorem C.4"}],"minor_comments":[{"comment":"The title and running header contain 'BYCONTRASTIVEATTENTION' with missing spaces; please fix the formatting.","section":"Title and header"},{"comment":"The notation a_{l,t,i} is described as the 'contrasted attention weight' but the contrast operation is not defined at this point; please specify the normalization or the contrast step used before entropy is computed.","section":"Eq. (3.3)"},{"comment":"The ViCrop variants 'rel-att', 'grad-att', and 'pure-grad' are not defined in the main text or appendix; please add a one-sentence description of each variant so the comparison is interpretable.","section":"Table 3"},{"comment":"Theorem C.6 invokes the Data Processing Inequality, but the Data Processing Inequality applies to Markov chains of random variables, not to arbitrary layer-wise attention maps in a transformer. The monotonic decrease of entropy with depth may hold empirically, but the proof as written is not valid; rephrase this as an empirical observation or provide a proper information-theoretic argument.","section":"Appendix C.6"},{"comment":"The correlation plots bin complexity into intervals and show the average entropy per bin, but they do not show the distribution of samples within bins or confidence intervals; adding scatter density and per-bin confidence intervals would make the claimed monotonic relationship more convincing.","section":"Figure 4"},{"comment":"The symbol M is used both for the VLM model and for the binary mask M*, which is confusing in lines 1, 3, and 7; please use distinct symbols, such as V for the model and B for the mask.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is plausible and potentially useful, but the empirical verification is the main obstacle: hyperparameter selection on the evaluation sets and missing error bars weaken the central 'consistently enhances' claim. The theoretical section should be reframed as a heuristic motivation, which I think the authors can do without losing the practical value of the method. I am not asking for a complete re-run of all experiments, but at least one held-out validation split with confidence intervals, and an honest statement of the number of protocol choices explored, would be necessary before I could recommend acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one-line summary: CARVE is a training-free method that masks and crops image regions based on the contrast between question-driven attention and attention under a generic 'describe the image' instruction. The empirical story is consistent — gains on four models across four benchmarks, sometimes large (V* is a small benchmark, so +75% is not as impressive as it sounds). The diagnostic work linking texture/color complexity to attention entropy is a genuine plus, and the paper is clearly written.\n\nWhat's actually new: pixel-level masking based on the ratio A(Q)/(A(G)+λ), plus the entropy analysis. That is a modest but legitimate extension of contrastive decoding and ViCrop. The improvement pattern is believable: older, smaller LLaVA models benefit more, which makes sense.\n\nSoft spots, in order of severity. First, the evaluation protocol. Appendix D picks the general instruction by maximizing accuracy on a 1,000-instance TextVQA subset; Figure 7 picks p and K on another 1,000-instance subset; no separate validation split is used before reporting the chosen configurations. That means the reported numbers are selected on the test set. Some deltas are small (POPE +0.23 to +1.73), and V* has 191 questions, so selection noise can easily explain part of the gains. There are no error bars or significance tests anywhere in Tables 1–2. This is the load-bearing issue: the method may well work, but the paper currently overstates its own support.\n\nSecond, the theory is not a theory. Appendix C.1 defines F_vis(I) to be A^(G)(I), so the decomposition A(Q)=F_vis⊗F_sem holds by definition; the 'proof' just restates the assumption. Theorem C.4 similarly builds in the approximation. If the general instruction also carries its own bias or focuses on salient objects, the ratio does not isolate noise. This is a valid concern, not just a framing quibble, though it does not invalidate the empirical method.\n\nThird, no code release. For a method with this many hyperparameters (λ, p, K, L, T, and the prompt itself), reproducibility is impossible without it.\n\nWho this is for: people working on training-free enhancement of VLMs, and anyone studying attention maps as a signal for cropping. The paper is worth a serious referee, but the referee should demand a held-out validation protocol, error bars, an honest theoretical framing, and ideally code before accepting the magnitude of the gains.","headline":"Plausible training-free visual refinement with a real eval-protocol problem: hyperparameters tuned on test subsets and a definitional 'proof'.","tokens_in":20067,"tokens_out":2493,"would_cite":false,"duration_ms":22382,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that contrasting a VLM's attention under a question with its attention under a generic description isolates the task-relevant visual signal, and that masking and magnifying only those regions improves visual question…","keywords":["vision-language models","attention entropy","contrastive attention","visual noise suppression","training-free enhancement","visual question answering","attention decomposition","pixel-level masking"],"falsifier":"Measure the attention map from the generic instruction on images with a single large salient object and no question. If those attention maps are not near-uniform (entropy significantly below the maximum $\\log N_v$), or if replacing the generic prompt with a deliberately uniform-attention prompt changes CARVE's accuracy gains substantially, then the assumption $F_{sem}(G,I) \\approx \\mathbf{1}_{N_v}$ fails and the theoretical basis collapses.","tokens_in":19022,"feed_emoji":"🔍","tokens_out":9961,"duration_ms":70927,"temperature":0.7,"pith_summary":"The paper argues that vision-language models lose accuracy on cluttered images because visual complexity, measured by texture and color, spreads attention diffusely. It shows that attention entropy rises with complexity and falls with accuracy. The paper then proposes CARVE, a training-free method that runs the model under a question and under a generic 'describe the image' prompt, divides the question-driven attention map by the generic one pixel-wise, thresholds and crops to the top regions, and re-asks the question on the magnified crop. Across four benchmarks and four open VLMs, accuracy consistently improves, with relative gains up to 75 percent on the hardest fine-grained benchmark.","feed_headline":"One attention trick improves visual question answering by up to 75%","feed_subtitle":"Divide question attention by generic-description attention, crop the top regions, and re-ask the VLM.","key_machinery":"The load-bearing identity is the attention decomposition $A^{(Q)}_{l,t}(I) = F_{vis}(I) \\otimes F_{sem}(Q,I)$, with $\\otimes$ the Hadamard product, together with the closed-form estimator $\\hat{A}_i = A^{(Q)}_i / (A^{(G)}_i + \\lambda)$ derived as the solution of a regularized least-squares problem. This estimator produces a pixel-level mask by fusing attention maps over a chosen layer range and generation time steps, thresholding at a top-p percentile, and cropping the top-K connected regions before resizing. The paper also introduces two complexity measures, edge density for texture and hue entropy for color, to support the motivating correlation between complexity, attention entropy, and accuracy.","core_discovery":"The central claim is that a VLM's attention map under a task question factorizes into an image-inherent visual-noise component and a task-driven semantic component, written as $A^{(Q)} = F_{vis}(I) \\otimes F_{sem}(Q,I)$. The paper argues that a generic instruction approximates the noise term alone because its semantic factor is uniform. Consequently, the ratio $\\hat{A}_i = A^{(Q)}_i / (A^{(G)}_i + \\lambda)$ estimates the semantic component, because the noise divides out when it dominates $\\lambda$. Using this estimated map to build a binary mask, CARVE crops the top regions and resizes them back to the original frame, which the paper shows moves correct-token probabilities ahead of incorrect ones and lifts accuracy consistently on A-OKVQA, POPE, V*, and TextVQA.","pith_inferences":["If the decomposition generalizes, the same contrast could be applied to prediction heads or detection outputs, turning the semantic map into a prompt-conditional prior rather than a one-shot crop.","The approach suggests a diagnostic tool: attention entropy or the contrast map could score how much of a VLM's error is due to visual clutter versus reasoning, supporting finer-grained failure analysis.","A direct stress test would be to replace the generic prompt with one that explicitly demands uniform scanning of the whole image; if CARVE's gains vanish or vary strongly, the noise-isolation assumption is fragile.","The same pixel-level contrast could be applied to intermediate features rather than attention maps, potentially recovering semantic signal even when attention itself is noisy."],"forward_implications":["CARVE requires no training or external segmentation tools; it relies only on the model's own attention, so any VLM with accessible attention maps can adopt it as a drop-in inference-time step.","Because the generic-instruction attention depends only on the image, it can be cached and reused across many questions on the same image, lowering the per-question computational overhead.","The entropy-complexity correlation suggests that image difficulty can be estimated before inference, potentially guiding when to apply CARVE and how aggressive the mask should be.","Gains are largest for weaker models and smallest for stronger recent ones, implying visual-noise interference is a substantial bottleneck in early VLMs and partly already mitigated in newer training."],"supporting_citations":[{"why":"TextVQA dataset, the primary benchmark for text-reading VQA used throughout the experiments.","marker":"Singh et al., 2019"},{"why":"A-OKVQA dataset, used to evaluate knowledge-based visual reasoning with CARVE.","marker":"Schwenk et al., 2022"},{"why":"POPE dataset, used to measure object hallucination and CARVE's effect on hallucination rates.","marker":"Li et al., 2023b"},{"why":"V* benchmark, the fine-grained visual search test set where the largest relative gains appear.","marker":"Wu & Xie, 2023"},{"why":"LLaVA-1.5 model family, one of the two base model groups CARVE is applied to.","marker":"Liu et al., 2023a"},{"why":"QWEN2.5-VL model family, the other base model group used in the experiments.","marker":"Qwen, 2025"},{"why":"Canny edge detection is used to define texture complexity in the correlation analysis.","marker":"Canny, 1986"},{"why":"ViCrop, the training-free cropping baseline CARVE is compared against on TextVQA.","marker":"Zhang et al., 2025"},{"why":"Shannon entropy is the metric used to quantify attention dispersion.","marker":"Shannon, 1948"},{"why":"Cited as the inspiration for using attention entropy to measure VLM attention distribution.","marker":"Yao et al., 2025b"}],"fun_headline_variants":["Attention contrast splits signal from noise, boosting VLMs up to 75%","CARVE: attention contrast crops away visual noise, lifting VQA","Training-free trick: contrast attention maps to sharpen VLM reasoning","Visual noise removed via attention division, improving VLMs by 75%","Crop from attention contrast: training-free VLM boost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central premise is that a generic description prompt produces attention that captures only image-inherent visual noise, with the semantic factor essentially uniform; if the generic prompt also focuses on salient objects or carries its own bias, the division does not isolate the task-relevant signal.","fun_headline_variants_meta":{"raw":{"variants":["Attention contrast splits signal from noise, boosting VLMs up to 75%","CARVE: attention contrast crops away visual noise, lifting VQA","Training-free trick: contrast attention maps to sharpen VLM reasoning","Visual noise removed via attention division, improving VLMs by 75%","Crop from attention contrast: training-free VLM boost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001343,"raw_usage":{"total_tokens":5451,"prompt_tokens":930,"completion_tokens":4521,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":4430}},"tokens_in":546,"tokens_out":4521,"duration_ms":26453,"temperature":1.0,"reasoning_tokens":4430,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:15:44.726715+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the attention map from the generic instruction on images with a single large salient object and no question. If those attention maps are not near-uniform (entropy significantly below the maximum $\\log N_v$), or if replacing the generic prompt with a deliberately uniform-attention prompt changes CARVE's accuracy gains substantially, then the assumption $F_{sem}(G,I) \\approx \\mathbf{1}_{N_v}$ fails and the theoretical basis collapses.","supporting_citations":[{"cited_title":"A-okvqa: A benchmark for visual question answering using world knowledge","cited_arxiv_id":null,"evidence_quote":"A-OKVQA dataset, used to evaluate knowledge-based visual reasoning with CARVE."}],"review_version":1}