{"id":"32e79ded-0876-429e-9f68-961a345fe8c9","arxiv_id":"2608.09654","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A decoupled, regression-free GUI grounding pipeline using frozen MLLM description generation and CLIP-based layout matching reports large gains on ScreenSpot-Pro and Mind2Web.","lead":"The authors propose a two-step GUI grounding system: a frozen language model rewrites instructions into visual descriptions, and a matching module selects the best screen region from layout candidates without a coordinate regression head. If the reported gains hold, this could make screen-based GUI agents cheaper and less prone to wrong clicks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's headline 'regression-free' claim is internally contradicted: Eq. (5) trains the DINO proposer with L1 and GIoU box-regression losses, so the final coordinates are regressed detector outputs; the reported gains cannot be attributed to regression-free matching without further evidence.","rationale":"The reader's weakest assumption is candidate recall; I partially agree that recall is an untested ceiling. However, the more load-bearing problem is that the architecture as specified is not regression-free. Eq. (5) is not a minor implementation detail; it is the training objective of the model that produces the final box. If the DINO regression head is accurate on GUI screenshots, the CLIP matcher merely re-ranks high-quality regressed boxes, and the reported 20%/15% improvements may mostly measure DINO fine-tuning rather than the proposed regression-free matching. This is an internal inconsistency in the central claim, not a disagreement with field consensus. The reader's CONDITIONAL verdict already asks for ablations and error bars; my concern adds a mandatory condition: the regression-free claim must be either removed or verified by an ablation in which no box-regression parameters are learned. I therefore recommend REJECT for the current manuscript, with the possibility of re-review if the authors re-scope the claim and supply the missing recall/ablation data.","tokens_in":10874,"tokens_out":8219,"duration_ms":82030,"concrete_test":"One decisive check: rerun the benchmark with the DINO box-regression head ablated or frozen, so that candidate boxes come from a genuinely non-regressive proposal source (e.g., parsing-tool text/icon boxes or DINO with COCO weights and no GUI fine-tuning). If ScreenSpot-Pro accuracy stays near the reported 41.3%, the regression-free matching claim is plausible; if accuracy drops substantially, the DINO regression head is load-bearing and the 'regression-free' claim must be withdrawn. In the same run, report DINO candidate recall at the operating confidence threshold and the CLIP matching accuracy given recall; this separates the candidate-generation ceiling from the matching stage.","verdict_should_be":"REJECT","load_bearing_attack":"Section 3.2.1 and Eq. (5) describe a GUI-domain-adapted DINO detector trained with L_det = 5.0 L_reg + 2.0 L_giou + 2.0 L_cls. The DINO decoder learns continuous bounding-box regression, and the system's output box is the box regressed by this detector; CLIP only selects among these regressed candidates. This directly contradicts the abstract's 'without learning any coordinate regression' and 'requiring no coordinate regression parameters,' as well as contribution (1), which says coordinate regression is replaced by frozen cross-modal matching. At best the selection head is regression-free, not the grounding pipeline. Since the paper's mechanism for eliminating hallucinations is the absence of coordinate regression, the central claim is unsupported unless the authors either narrow the claim to the matching stage or show that the regression head is not responsible for the results. The manuscript also never reports candidate recall, so one cannot separate DINO regression accuracy from CLIP matching accuracy. A 'layout-prior candidate generator' trained with coordinate regression is exactly the kind of learned coordinate mapping the paper says it avoids.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage GUI grounding framework. A frozen MLLM first rewrites an abstract user instruction into a structured visual description rich in layout cues; a GUI-domain-adapted DINO detector then proposes candidate regions using only Text/Icon binary labels; finally, a frozen CLIP model selects the best-matching candidate, with a lightweight learned geometric fusion layer that injects spatial information. The authors report large improvements over end-to-end baselines on ScreenSpot, ScreenSpot-Pro, AITW, and Mind2Web, and claim that the pipeline is \"regression-free\" and hence avoids coordinate hallucinations. The manuscript argues that decoupling semantic understanding from layout-aware localization resolves the core difficulties of GUI grounding without expensive MLLM fine-tuning.","tokens_in":11127,"tokens_out":4077,"duration_ms":38421,"significance":"If the reported results are reproducible, the decoupled architecture is a practically attractive alternative to end-to-end MLLM grounding: it avoids full MLLM fine-tuning, uses only cheap Text/Icon labels for the candidate detector, and leverages frozen CLIP for open-vocabulary matching. The core idea is clear and easy to communicate, and the reported margins on ScreenSpot-Pro (41.3% vs. 18.9% for the best baseline) are striking. However, the central \"regression-free\" claim is currently not supported as stated, because the DINO candidate generator is trained with L1 and GIoU regression losses (Eq. 5), and the paper does not report candidate recall, ablations, error bars, or code/data. These gaps leave the attribution of the gains to the proposed mechanisms unresolved, though they are addressable with additional experiments and a more careful formulation of the claim.","major_comments":[{"comment":"The paper's central \"regression-free\" claim is internally contradicted by the training of the candidate generator. Eq. (5) includes L_reg, an L1 bounding-box regression loss, and L_giou, a generalized IoU loss, applied to the DINO Transformer decoder, and Section 3.2.1 states that the detector is fine-tuned to output predicted boxes. Since the final output box is the box produced by this detector and CLIP only selects among these regressed candidates, the pipeline as a whole does learn coordinate regression. The abstract and Contribution (1) claim that the model avoids \"learning any coordinate regression\" and \"requiring no coordinate regression parameters,\" which is not supported by the described method. At best the matching stage is regression-free, not the full grounding pipeline. The authors should either narrow the claim to the matching stage or provide a control experiment that removes or ablates the regression head and demonstrates that the reported gains do not depend on it.","section":"§3.2.1 and Eq. (5)"},{"comment":"No candidate-recall metric is reported. The method's accuracy is upper-bounded by the recall of the DINO Text/Icon candidate generator: if the target element is not among the proposals, the frozen CLIP matcher cannot recover it. The reported gains over baselines could therefore come from improved DINO candidate quality rather than from \"regression-free\" matching. Please report candidate recall (e.g., Recall@k at the relevant IoU threshold) on ScreenSpot-Pro and Mind2Web, separately for Text and Icon candidates, and also report an oracle accuracy where the correct candidate is selected by ground-truth information. This would separate the contribution of candidate generation from the contribution of cross-modal matching.","section":"§3.2.1 and §3.2.2"},{"comment":"The Introduction and Conclusion state that \"Controlled ablation studies\" confirm the contributions of layout-prior candidate generation and lightweight geometric fusion, but no ablation table or experiment appears in Section 4. Without ablations, the attribution of the reported gains to the two proposed mechanisms is unverified. Please add an ablation study that removes or disables each component (e.g., no geometric fusion, no layout-prior candidates, and a direct CLIP matching baseline on raw image patches) and reports the resulting accuracy on the same benchmarks.","section":"§4 (Experiments)"},{"comment":"The experimental section lacks essential reproducibility details and statistical support. The number of spatial training pairs for Stage 2 is not given, the sampling and annotation procedure for those pairs is not described, and no code, data, or trained checkpoints are provided. Moreover, the benchmark tables report no error bars or significance tests, even though the claimed margins are large. At minimum, please specify the size and construction of the spatial-pair set, report standard errors or confidence intervals over multiple runs, and release the code and candidate-generator checkpoints.","section":"§3.3 and §4"}],"minor_comments":[{"comment":"The rows for Fuyu and CogAgent are numerically identical (8.4, 6.6, 6.2, 2.9, 6.5, 3.4, 5.7); this is likely a copy-and-paste error and should be corrected.","section":"Table 1"},{"comment":"References [8] and [9] are the same paper (arXiv:2410.05243); the duplicate entry should be removed and the citation numbering adjusted.","section":"References"},{"comment":"The abstract says the model is trained \"with only Text/Icon binary labels,\" but Section 3.3 states that Stage 2 uses \"manually annotated spatial queries\" and Section 3.2.2 describes training the geometric fusion on spatial query-region pairs; please reconcile these statements.","section":"Abstract and §3.3"},{"comment":"The Preliminary paragraph calls the MLLM \"training-free\" while the rest of the paper says it is \"frozen-parameter\"; the wording should be consistent, since the MLLM is not trained but is used with in-context prompting.","section":"§3.1"},{"comment":"The caption says \"after pre-training\" and \"fine-tuning with visual captions,\" which conflicts with the paper's claim that the approach avoids fine-tuning; the caption should use terminology consistent with the method description.","section":"Figure 4"},{"comment":"Equation (2) defines cosine similarity with explicit norm normalization, but Eq. (6) uses sim(·,·) without defining it again; please state explicitly that sim is the same normalized cosine similarity from Eq. (2).","section":"§3.2.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be an early preprint. The central \"regression-free\" claim needs to be narrowed or supported by ablations, and the missing candidate-recall analysis is essential before the paper can be considered for publication. The duplicate reference and the identical baseline rows in Table 1 also suggest that the experimental section needs careful revision. If the authors address the major points, the paper could become a useful contribution to GUI grounding."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core design here is genuinely worth a look: a frozen MLLM that turns abstract instructions into layout-rich captions, a DINO detector fine-tuned only to propose Text/Icon regions, and frozen CLIP matching to pick the right box. That combination is not in the cited prior work, and the idea of avoiding expensive coordinate supervision by leveraging two-class layout labels is sensible. The reported gains on ScreenSpot-Pro and Mind2Web are large, and if they hold this would be a practical alternative for GUI agents. I also like the lightweight geometric fusion layer; it is cheap and plausibly effective for spatial queries.\n\nBut the headline claim is overstated. The paper says “without learning any coordinate regression,” yet Section 3.3 and Eq. (5) train the DINO proposer with L1 and GIoU box regression losses. The final output box is the regressed detector box; CLIP only selects among regressed candidates. At most the matching stage is regression-free, not the pipeline. Since the paper attributes hallucination suppression to the absence of coordinate regression, this is a load-bearing inconsistency, not a wording nit. The authors need to either narrow the claim or show the regression head is not responsible for the gains. They also never report candidate recall, so we cannot tell whether the ceiling is set by the detector or the matcher.\n\nThe other issues are more addressable: no code or data released, no error bars on any table, and the ablation study is referenced repeatedly but never shown. There also seem to be small citation slips, like UGround appearing twice as [8] and [9] and CLIP as [28] and [29]. None of that is fatal, but it does not meet the bar for a clean acceptance.\n\nOverall, I think this deserves a serious referee rather than a desk reject. The architecture is plausible, the task is important, and the numbers are strong enough to warrant verification. But my honest verdict is conditional: the authors must fix the regression-free claim, provide candidate recall, and release code/data before the results can be trusted.\n\nRecommendation: send it to review, but flag the regression-free contradiction and the missing evidence as major required revisions.","headline":"The decoupling idea is real and worth testing, but the 'regression-free' claim is contradicted by the paper's own Eq. (5), and missing code/ablation data make the headline numbers unverifiable.","tokens_in":11625,"tokens_out":1072,"would_cite":false,"duration_ms":15486,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that replacing coordinate regression with candidate generation plus frozen cross-modal matching removes GUI grounding hallucinations, yielding more than 20% accuracy gains on ScreenSpot-Pro.","keywords":["GUI grounding","regression-free localization","layout-aware matching","coordinate hallucination","CLIP","multimodal large language models","ScreenSpot-Pro","Mind2Web"],"falsifier":"Run the DINO candidate stage alone on ScreenSpot-Pro and measure recall: for each ground-truth target, does at least one proposed candidate overlap it? If recall is far below the reported 41.3% average accuracy, the matching stage cannot be doing the claimed work.","tokens_in":10681,"feed_emoji":"🖱️","tokens_out":7115,"duration_ms":64173,"temperature":0.7,"pith_summary":"This paper claims that the usual way of making GUI agents point at screen elements—training a multimodal model to output coordinates—is what causes them to hallucinate, and that a different division of labor avoids the problem. A frozen MLLM first rewrites the user instruction into a visual description; a detector adapted to GUI layouts proposes candidate text and icon regions; a frozen CLIP model then selects the best matching region. No component learns coordinate regression, so the final coordinates come from choosing an existing candidate. The consequence, if right, is that precise GUI grounding can be achieved with only Text/Icon labels and a small amount of spatial annotation, and on ScreenSpot-Pro the paper reports a gain of more than 20 percentage points over end-to-end systems while Mind2Web element accuracy and step success rate rise by more than 15%.","feed_headline":"Matching, not regression, powers GUI grounding's 20% leap","feed_subtitle":"Frozen MLLM rephrases instructions, a detector proposes text/icon candidates, and CLIP picks the right box.","key_machinery":"The load-bearing object is the Layout-Aware GUI Grounding Model, a two-step matcher. First, a GUI-domain-adapted DINO detector, fine-tuned on 200k screenshots labeled only with Text and Icon, outputs candidate bounding boxes plus a six-dimensional geometric vector encoding center, width, height, aspect ratio, and area. Then CLIP's frozen image encoder extracts a feature for each candidate crop, CLIP's text encoder encodes the MLLM's elaborated instruction, and cosine similarity selects the winner. For spatial queries, a learnable linear projection maps the geometric vector into the visual feature space and adds it to the candidate feature; this projection is trained with a contrastive loss on a small set of spatial query pairs and stays near zero when spatial cues are absent.","core_discovery":"The paper's central claim is that coordinate regression is the root cause of GUI grounding hallucinations, and that replacing it with candidate generation plus frozen cross-modal matching removes that failure mode. The Layout-Aware GUI Grounding Model first lets a GUI-adapted DINO detector propose text and icon regions, then uses frozen CLIP to match each candidate's visual feature against a description produced by a frozen MLLM. A single learnable linear projection adds geometric bias for spatial queries without reintroducing regression. On ScreenSpot-Pro the paper reports a gain of more than 20 percentage points over the strongest end-to-end baseline, and on Mind2Web it reports element accuracy and step success rate gains above 15%.","pith_inferences":["If the DINO candidate generator ever misses the target, the CLIP matcher cannot recover it, so the method's true ceiling is candidate recall rather than matching quality.","Replacing the frozen MLLM with a stronger or cheaper instruction parser should shift accuracy without retraining the grounding module, although the paper does not test this.","The same Text/Icon candidate-plus-frozen-matching design could transfer to other text-and-icon dominated visual domains, such as document or diagram grounding."],"forward_implications":["GUI agents can operate from screenshots alone, with no DOM or view hierarchy, while avoiding coordinate regression.","Only Text/Icon binary labels plus a small set of spatial query pairs are needed to train the grounding module, cutting annotation cost.","Spatially qualified instructions like 'upper-right close button' can be handled by a single learnable linear projection that stays near-zero for plain queries.","Accuracy gains on ScreenSpot-Pro and Mind2Web suggest the decoupling transfers across desktop, mobile, and web interfaces.","Coordinate hallucinations are structurally prevented, because coordinates are selected from detected candidates rather than generated."],"supporting_citations":[{"why":"Introduces the SeeClick GUI grounding baseline and the ScreenSpot benchmark used for the main grounding comparison.","marker":"[5]"},{"why":"Supplies the ScreenSpot-Pro benchmark of professional-application grounding tasks where the largest gains are reported.","marker":"[20]"},{"why":"Provides the Mind2Web dataset and the element accuracy and step success rate metrics for the agent evaluation.","marker":"[6]"},{"why":"The DINO detector architecture that is domain-adapted on Text/Icon labels for layout-prior candidate generation.","marker":"[40]"},{"why":"Grounding DINO is the open-vocabulary detection approach the candidate stage builds on and compares against.","marker":"[25]"},{"why":"The frozen CLIP model whose cross-modal similarity performs the hallucination-resistant final matching.","marker":"[29]"},{"why":"Qwen2.5-VL is the frozen MLLM used to elaborate instructions into structured visual descriptions and also serves as a baseline.","marker":"[3]"}],"fun_headline_variants":["GUI grounding sans regression: 20% accuracy gain","No regression, just matching: GUI grounding leaps 20%","Frozen CLIP matches layout, kills GUI hallucinations","Regression-free GUI grounding beats end-to-end by 20%","Layout-aware matching: the end of GUI hallucinations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The one load-bearing premise, introduced where the detector is described, is that the Text/Icon candidate generator almost always proposes the target element, because the frozen matcher can only select among those proposals and cannot recover a missed target.","fun_headline_variants_meta":{"raw":{"variants":["GUI grounding sans regression: 20% accuracy gain","No regression, just matching: GUI grounding leaps 20%","Frozen CLIP matches layout, kills GUI hallucinations","Regression-free GUI grounding beats end-to-end by 20%","Layout-aware matching: the end of GUI hallucinations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000363,"raw_usage":{"total_tokens":1942,"prompt_tokens":912,"completion_tokens":1030,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":951}},"tokens_in":528,"tokens_out":1030,"duration_ms":8218,"temperature":1.0,"reasoning_tokens":951,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:19:17.355617+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the DINO candidate stage alone on ScreenSpot-Pro and measure recall: for each ground-truth target, does at least one proposed candidate overlap it? If recall is far below the reported 41.3% average accuracy, the matching stage cannot be doing the claimed work.","supporting_citations":[{"cited_title":"Dino: Detr with improved denoising anchor boxes for end-to-end object detection","cited_arxiv_id":null,"evidence_quote":"The DINO detector architecture that is domain-adapted on Text/Icon labels for layout-prior candidate generation."}],"review_version":1}