{"id":"1f87495f-4cfb-4728-8cd2-84f68601606c","arxiv_id":"2412.10840","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A tuning-free method that aggregates selected attention maps in a pretrained multimodal LLM achieves GUI grounding accuracy comparable to fine-tuned systems, especially for text.","lead":"Researchers show that a pretrained multimodal AI model can locate buttons, text, and icons in screenshots by reading its own internal attention maps, without any extra training. This could make GUI automation cheaper and faster, since it removes the need to fine-tune models on expensive labeled interface data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported tuning-free advantage rests on K=10 and δ=0.5 selected from ablations on the same benchmarks whose headline numbers are then reported, with no validation split described.","rationale":"The reader's formal weakest_assumption is the Eq. (4) head-quality heuristic, and that is a legitimate secondary concern. But the ablation evidence (Table 4, Figure 6) shows that attention-map grounding alone already lifts mobile text accuracy from 40.3% to 71.8%, so even if the head-ranking heuristic is imperfect the central method does not collapse; it loses only a few points. By contrast, the choice of K and δ is shown to be decisive for final numbers (Figure 6: K=1 or K=1024 degrade accuracy), and the paper gives no evidence that those choices generalize. If they were selected on the same test sets, every headline comparison in Tables 1-3 is partly tied to test-set peeking. This is the most load-bearing issue because it directly affects the comparison to fine-tuned baselines that is the paper's main claim. I am not claiming intentional cherry-picking; the problem is that no validation protocol is reported, so the reader cannot tell whether K=10/δ=0.5 would have been chosen a priori. The proposed held-out test would settle this. The verdict remains conditional because the underlying attention mechanism is plausible, reproducible code is promised, and a clean held-out evaluation could restore the central claim.","tokens_in":15192,"tokens_out":7422,"duration_ms":73513,"concrete_test":"Use held-out splits. Split ScreenSpot (and OCG if used for selection) into two halves by screenshots; on the first half only, grid-search K in {1, 5, 10, 20, 1024} and δ in {0.3, 0.5, 0.7}; freeze the best pair; evaluate on the second half. Report the selected K/δ, the held-out average, and compare it to SeeClick's reported 53.4%. If the optimal pair changes across splits, or if the held-out average falls below SeeClick, the reported superiority is an artifact of test-set hyperparameter selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that a pretrained MLLM's attention maps, without any adaptation, can match or exceed GUI-grounding models trained with SFT. The weakest point in the evidence chain is not the attention heuristic per se but the way the two free parameters are set. Section 4.5 fixes K=10 and δ=0.5 \"based on these results\" (Figures 6 and 7), with no validation split described. The ablations appear to be run on the same ScreenSpot/OCG benchmarks whose final scores are then reported. Figure 6 explicitly shows performance is sensitive to K (K=1 and K=1024 both reduce accuracy), so the numbers in Tables 1-3 are conditional on hyperparameters that were selected after seeing the evaluation data. This makes the headline \"surpassing even GUI-specific SFT models\" overstate what has been demonstrated: a tuning-free pipeline with test-selected hyperparameters is not the same as a tuning-free pipeline evaluated fairly. The issue is fixable, which is why the finding is conditional rather than reject, but it is the load-bearing part of the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TAG, a tuning-free GUI grounding method that aggregates self-attention maps from a pretrained multimodal large language model (MiniCPM-Llama3-V 2.5) to localize elements described by a text query. The method constructs a prompt that asks the model to output an element description, then propagates attention from the generated description tokens to visual query tokens and from there to image patches via cross-attention (Eqs. 1-3). A head-selection heuristic (Eq. 4) keeps only the top-K self-attention heads per token, and a thresholded relevance map yields the final point prediction. The authors evaluate on a newly introduced OCR grounding benchmark (OCG), ScreenSpot, and Mind2Web, reporting that TAG outperforms fine-tuned GUI grounding models such as SeeClick and CogAgent on ScreenSpot average accuracy and on OCG, while matching SeeClick on Mind2Web element accuracy. The code is publicly available.","tokens_in":15431,"tokens_out":3364,"duration_ms":30417,"significance":"If the reported results hold under fair evaluation, the claim is significant: it would show that attention maps of a pretrained MLLM can substitute for task-specific fine-tuning in GUI grounding, with a simple and relatively efficient pipeline (Table 6). The paper also introduces a new benchmark (OCG) and demonstrates a generalization experiment on Qwen-VL-Chat. However, the central claim is weakened by the selection of the two hyperparameters K and delta on the same benchmarks whose final numbers are reported, and by the lack of a released OCG dataset. The paper's strengths include its clear method formulation, ablations showing each component contributes, and reproducible greedy generation.","major_comments":[{"comment":"The two free parameters K and delta are selected by ablating on the same evaluation benchmarks whose final scores are reported in Tables 1-3. The text states 'Based on these results, we use K = 10 in all experiments' and 'delta = 0.5 is used across all datasets,' but no validation split is described. Figure 6 shows strong sensitivity to K (e.g., K=1 substantially lowers accuracy), so the headline comparison against fine-tuned models is conditional on hyperparameters chosen after seeing the test data. Please add a validation split or otherwise demonstrate that the selection does not constitute tuning on the test set; if this is not possible, the claim of a 'tuning-free' advantage should be tempered accordingly.","section":"Section 4.5, Figures 6-7"},{"comment":"The OCG dataset is constructed from Mind2Web test screenshots with Azure OCR, but the dataset is not released. Moreover, the queries are exact OCR text, so this benchmark tests text grounding with perfectly aligned queries rather than the harder case of grounding natural-language commands. Since the OCG result (84.5% vs. 60.2% for SeeClick) is a primary demonstration of the method's advantage, the dataset and exact query-construction procedure must be released for independent verification, or the claim should be restricted to OCR-exact text grounding.","section":"Section 4.1, Table 1"},{"comment":"All results are single-run point estimates without standard errors or confidence intervals. For example, the ScreenSpot average gap between TAG (54.8%) and SeeClick (53.4%) is 1.4 percentage points; without an estimate of variance, it is unclear whether this difference is meaningful. Please report multiple runs (e.g., different random seeds or bootstrapping over the evaluation set) or provide statistical significance tests, especially for claims of surpassing fine-tuned baselines.","section":"Section 4, Tables 1-3"},{"comment":"The head-selection rule assumes that the sum of attention from a text token to all visual query tokens, \\tilde{A}^k_{T_j}, is a valid proxy for head quality for grounding. The justification is anecdotal, relying on Figure 3. The ablation in Table 4 shows the filter helps on average, but the paper does not analyze whether the heuristic holds for icon/widget grounding or under distribution shifts. Please provide a quantitative analysis of the correlation between \\tilde{A}^k and grounding success per element type, or a sensitivity analysis across different models and UI distributions, to support the generality of the selection rule.","section":"Section 3.5, Eq. (4)"}],"minor_comments":[{"comment":"The caption and the diagram use the term 'unfold' without explanation; please clarify that the head-wise attention is reshaped from the raw attention tensor.","section":"Figure 2"},{"comment":"The name 'Adaptive Text Token Selection' is somewhat misleading: the method does not explicitly select tokens from the user query but instead relies on the model's generated element description. Consider renaming or clarifying the mechanism.","section":"Section 3.3"},{"comment":"The sentence 'its pre-training data may make it impossible to include images of any aspect ratios' likely means 'may make it difficult to perform well on unseen aspect ratios' rather than 'impossible'; please rephrase.","section":"Section 4.1"},{"comment":"The OCG dataset statistics in Table 5 are useful, but please also state whether the dataset is released and under what license, since the main text does not provide a link.","section":"Section 6.3"},{"comment":"The column header 'MLLMs w/o SFT' could be made cleaner as 'Without SFT' to avoid inconsistent spacing; the table captions should also indicate whether the 'MLLMs' column refers to model size or model family.","section":"Tables 2 and 3"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is sound and interesting, but the evaluation methodology currently allows the two hyperparameters to be tuned on the test benchmarks, which is the main factor preventing acceptance. The OCG dataset release is also important for reproducibility. If the authors can address these issues with a proper validation split or a sensitivity analysis that demonstrates robustness, the paper could be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is genuinely new: instead of fine-tuning an MLLM for GUI grounding, TAG pulls localization out of the self-attention heads of a pretrained model by selecting text tokens from a model-generated element description, filtering heads by attention magnitude, and propagating through cross-attention to the image grid. It works better than I would have guessed, especially for text elements. On ScreenSpot text grounding, the gap over the base MiniCPMV2.5 is large and consistent, and the ablations in Table 4 show each component pulls its weight. The new OCG benchmark, built from Mind2Web screenshots with OCR boxes across 10 aspect ratios, is a reasonable test of text grounding even if it is simple. Code is public, and the method transfers to Qwen-VL-Chat in a small generalization experiment. That is a solid package. The soft spots match the stress-test note, and the note is accurate. Section 4.5 fixes K=10 and δ=0.5 based on ablations run on the same ScreenSpot/OCG benchmarks whose final scores are then reported. No validation split is described. Since Figure 6 shows real sensitivity to K, the headline \"surpassing even GUI-specific SFT models\" overstates what has been demonstrated: the method is tuning-free in the weights, but the pipeline has two hyperparameters dialed in on the test set. That is the main thing to fix. The OCG dataset is also not released, and its queries are OCR-exact, which makes the 84.5% number look less impressive than a paraphrase-robust benchmark would. And on icon/widget grounding the method still trails SeeClick on mobile (29.3% vs 52.0%), so the \"surpassing SFT\" claim is only true on average and mostly on text. Single-run results are a minor concern. None of this is fatal. The attention heuristic itself is not fitted to boxes; it is a deterministic aggregation, and the qualitative evidence in Figure 3 supports the head-selection rule. The circularity burden comes from the two scalar hyperparameters, not from the method's principle. I would send this to peer review. The right referee will push for a proper validation split, ideally leave-one-out or a held-out set of websites, and ask for the OCG data to be released. The method deserves a fair hearing and could become a useful baseline for tuning-free GUI grounding.","headline":"The tuning-free attention pipeline works surprisingly well for text grounding, but the headline numbers rest on test-set-selected hyperparameters, so the paper needs a validation split before the claims are fully established.","tokens_in":702,"tokens_out":912,"would_cite":true,"duration_ms":23161,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By reading out attention maps from a frozen multimodal LLM, a new method grounds GUI elements as accurately as models fine-tuned for the task.","keywords":["GUI grounding","attention map","multimodal large language model","tuning-free grounding","element localization","text grounding","ScreenSpot benchmark","parameter-free adaptation"],"falsifier":"Evaluate TAG on a different pretrained MLLM or on a GUI distribution where per-head attention sums do not correlate with whether a head's attention peak falls inside the ground-truth box; if top-K selection then yields worse accuracy than averaging all heads, the head-selection heuristic is falsified. Concretely, for each head compute both its Eq. (4) score and its localization accuracy (peak inside GT box), and check whether the top-K by score are also the top performers by accuracy.","tokens_in":14984,"feed_emoji":"🖱️","tokens_out":9765,"duration_ms":71288,"temperature":0.7,"pith_summary":"This paper establishes that a pretrained multimodal large language model can ground GUI elements—locate text and icons in a screenshot from a text command—without any GUI-specific fine-tuning, by reading out its internal attention maps. The proposed TAG method selects the tokens that describe the target element, aggregates their self-attention to the model's visual tokens, filters to the most informative attention heads, and projects the result back onto the image plane to produce a click point. On the ScreenSpot benchmark it reports 54.8% average accuracy, above fine-tuned GUI grounding models; on a new OCR grounding set built from Mind2Web screenshots it reaches 84.5% across ten aspect ratios. The significance is that spatial grounding may be an emergent, extractable property of off-the-shelf MLLMs, which could reduce the data and compute needed to build GUI agents.","feed_headline":"Attention readouts localize GUI elements, no fine-tuning needed","feed_subtitle":"A pretrained MLLM's internal attention outperforms fine-tuned GUI models on ScreenSpot.","key_machinery":"The machinery is a token-to-patch attention propagation chain. First, adaptive text token selection builds a prompt that makes the MLLM generate a short description of the target element, and the tokens of that description are the seeds. For each seed token, the method extracts per-head self-attention weights to the fixed set of visual query tokens in the LLM (Eq. 1 aggregates them with per-token, per-head weights α). The head selection rule (Eq. 4) scores each head by the sum of its attention from that token to all visual query tokens, keeps the top-K heads, and sets α=1 for them and 0 otherwise. The averaged token-to-query attention is then multiplied by the vision encoder's cross-attention matrix Across, which maps each visual query token to the H×W patch grid (Eq. 2), producing a per-token relevance map over image patches; these maps are averaged over selected tokens (Eq. 3). A threshold δ binarizes the map, connected components are found, and the center of the highest-average region becomes the prediction.","core_discovery":"The central claim of this paper is that a well-structured aggregation of attention maps from a pretrained MLLM can serve as a high-performance GUI element localizer without any fine-tuning. Concretely, the authors instruct MiniCPM-Llama3-V 2.5 to output a description of the element relevant to the user's command, then take the self-attention from those description tokens to the model's visual query tokens in each of the LLM's 1024 attention heads. Each head's contribution is weighted by a quality score—the total attention mass from the token to all visual query tokens—and only the top-K heads per token survive; the surviving maps are averaged and propagated through the vision encoder's cross-attention to image patches. Thresholding the resulting relevance map and taking the center of the highest-scoring connected region gives the predicted location. This pipeline achieves 54.8% average accuracy on ScreenSpot (above the fine-tuned SeeClick and CogAgent), 84.5% on the new OCG dataset across ten aspect ratios, and 87.4% on VisualWebBench's element grounding task. The authors also apply the same propagation to Qwen-VL-Chat, improving its OCG accuracy from 2.7% to 10.2%, which they read as evidence that the mechanism generalizes across MLLMs.","pith_inferences":["The head-selection heuristic might be replaced by a calibration step that measures per-head grounding quality on a handful of annotated screenshots, which could transfer TAG to models whose attention sums do not track localization accuracy.","The same propagation chain could be applied to other fine-grained visual grounding tasks, such as referring-expression comprehension or visual prompt pointing, where a natural-language expression must be mapped to an image region.","Because the method relies on the MLLM's own element description, prompt design (asking for exact text vs. keywords vs. a full sentence) may be a tuneable lever for different UI platforms and element types.","The paper's high text accuracy but lower icon accuracy suggests that attention readouts carry strong lexical-spatial alignment but weaker semantic-spatial alignment; training or prompting that strengthens icon-description tokenization could be a direct avenue for improvement."],"forward_implications":["GUI agents can be assembled from frozen pretrained MLLMs, sidestepping the cost of GUI-specific fine-tuning and the risk of overfitting to narrow UI distributions.","Text grounding becomes a reliable signal: the reported ScreenSpot text accuracies (88.3% mobile, 82.5% desktop, 70.9% web) are high enough to act as an OCR-coordinate backend for click actions.","The method extends to other MLLMs, as demonstrated by the Qwen-VL-Chat transfer experiment, so improvements in base models may automatically improve grounding.","The OCG dataset supplies a fixed benchmark with ten aspect ratios, letting the community measure how grounding degrades as screen shapes move away from square."],"supporting_citations":[{"why":"Provides MiniCPM-Llama3-V 2.5, the pretrained MLLM whose self-attention and cross-attention maps TAG reads to localize elements.","marker":"Yao et al. 2024"},{"why":"Provides the ScreenSpot dataset and the SeeClick fine-tuned baseline that TAG is directly compared against.","marker":"Cheng et al. 2024"},{"why":"Provides the Mind2Web screenshots and task structure used to construct the new OCG dataset and the agent-level evaluation.","marker":"Deng et al. 2024"},{"why":"Supplies Qwen-VL-Chat, the second MLLM used to test whether TAG's attention-propagation mechanism generalizes beyond MiniCPM.","marker":"Bai et al. 2023"},{"why":"Supplies the VisualWebBench element grounding task used as an additional evaluation of TAG against point-selection baselines.","marker":"Liu et al. 2024c"}],"fun_headline_variants":["Attention maps from MLLMs ground GUI elements with no fine-tuning","Zero-shot GUI grounding via MLLM attention aggregation","Pretrained MLLM attention outperforms fine-tuned GUI models","No fine-tuning needed: MLLM attention localizes GUI elements","MLLM attention maps beat fine-tuned models on GUI grounding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the sum of attention a head pays from a description token to all visual query tokens is a reliable measure of that head's spatial grounding quality, so discarding the low-sum heads cannot remove the heads that actually localize correctly.","fun_headline_variants_meta":{"raw":{"variants":["Attention maps from MLLMs ground GUI elements with no fine-tuning","Zero-shot GUI grounding via MLLM attention aggregation","Pretrained MLLM attention outperforms fine-tuned GUI models","No fine-tuning needed: MLLM attention localizes GUI elements","MLLM attention maps beat fine-tuned models on GUI grounding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000876,"raw_usage":{"total_tokens":3838,"prompt_tokens":1045,"completion_tokens":2793,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":661,"completion_tokens_details":{"reasoning_tokens":2705}},"tokens_in":661,"tokens_out":2793,"duration_ms":17652,"temperature":1.0,"reasoning_tokens":2705,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:33:19.629489+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate TAG on a different pretrained MLLM or on a GUI distribution where per-head attention sums do not correlate with whether a head's attention peak falls inside the ground-truth box; if top-K selection then yields worse accuracy than averaging all heads, the head-selection heuristic is falsified. Concretely, for each head compute both its Eq. (4) score and its localization accuracy (peak inside GT box), and check whether the top-K by score are also the top performers by accuracy.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Mind2Web screenshots and task structure used to construct the new OCG dataset and the agent-level evaluation."}],"review_version":1}