Pith. sign in

REVIEW 2 major objections 4 minor 33 references

MissClick: Exploiting Digit-Serialized Coordinates to Attack GUI Grounding Models

T0 review · 2 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that attacks on GUI grounding models should exploit the place value of digit-serialized coordinates, and proposes two objectives—soft-coordinate displacement and place-weighted target-digit loss—that beat existing attacks

desk verdict Solid white-box attack with a genuinely useful objective-design insight; headline untargeted numbers are confounded by unparseable outputs counted as successes. read the letter →

arxiv 2608.03740 v1 pith:RDT5WKCT submitted 2026-08-04 cs.AI

Yu Ran , Wentao Zhao , Xin Zhang , Yi Pan This is my paper · ORCID
classification cs.AI
keywords GUIgroundingadversarialattackdigit-serializedcoordinatesplacevaluescreenvision-languagemodelswhite-boxcoordinategeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

GUI grounding models answer instructions by generating screen coordinates as sequences of digit tokens, then parse those tokens into numbers that determine a click. The paper's central claim is that adversarial attacks should exploit the place-value structure of this pipeline: flipping a hundreds-place digit moves the click roughly 100 pixels, while flipping a ones-place digit moves it one pixel, so token-level and embedding-level objectives leave attack performance on the table. To test the idea, the paper proposes MissClick-U, which maximizes the distance of a soft click built from probability-weighted digit values, and MissClick-T, which minimizes a place-weighted cross-entropy toward target digits. On two GUI grounding models across desktop, web, and mobile tasks, the untargeted objective reaches 75.07% and 72.93% attack success rates, and the targeted objective reaches 44.86% and 62.67%, outperforming previous attacks by up to 47 percentage points under identical perturbation budgets. The paper concludes that untargeted and targeted attacks favor different objectives, and that coordinate outputs should be treated as numbers, not text.

What carries the argument

The load-bearing identity is the place-value expansion of a parsed coordinate, c_k = Σ_j a_{k,j} d_{k,j} with a_{k,j} = 10^{m_k−j}: a coordinate component is a sum of digit values weighted by hundreds, tens, and ones. The attack turns this identity into a gradient path. A teacher-forced forward pass with the reference coordinate gives per-position digit distributions p_{k,j}; taking expected digit values and recombining them with the same place weights yields a differentiable soft coordinate, so MissClick-U can maximize displacement of the soft click via PGD. MissClick-T instead puts the place weights on per-digit cross-entropy terms, concentrating optimization on the high-order positions th

What would settle it

Run MissClick-U and MissClick-T, with the same epsilon=16/255 budget, against a GUI grounder that emits click locations through a coordinate-free action head or through multi-digit subword tokens. If attack success rates fall to baseline levels, the place-value digit-token interface is what the attack exploits. A second check: log per-position teacher-forced probabilities and greedy argmax digits during MissClick-T; if many target positions satisfy the expected digit equals target while argmax remains off-target, the Eq. (11) failure mode limits the targeted objective.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the discrete, digit-by-digit way GUI grounders emit coordinates creates a cheap attack surface: the parsed coordinate is a weighted sum of digit tokens with decimal place values, so a small pixel perturbation that changes one high-order digit can displace the executed click by hundreds of pixels. MissClick-U constructs a differentiable soft coordinate from teacher-forced digit distributions and maximizes its squared distance to the ground-truth click; MissClick-T minimizes a place-weighted target-digit cross-entropy that pushes high-order positions toward the target. The paper reports that on OS-Atlas and UGround over ScreenSpot-v2, MissClick-U

Load-bearing premise

The load-bearing premise is that each coordinate digit is its own token and that the digit distributions obtained from a teacher-forced pass with a reference coordinate string are a faithful enough proxy for greedy decoding that optimizing them moves the executed click; if a victim tokenizes digits differently, uses a coordinate-free action head, or has teacher-forced gradients that diverge from decoding, the reported gains would not transfer.

Editorial extensions

If this is right

  • Attack objectives for coordinate-generating GUI grounders should be defined in parsed numerical space; token-level cross-entropy and embedding-space objectives are suboptimal.
  • Untargeted and targeted attacks need different objectives: soft-coordinate displacement for untargeted, place-weighted target-digit loss for targeted.
  • Place-value weighting alone is worth 9.62 to 11.98 percentage points of targeted ASR over uniform per-digit weighting.
  • MissClick remains effective at smaller budgets: at epsilon=4/255 it beats baselines running at epsilon=16/255, and it reaches baseline-level ASR in 10–30 iterations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not tested in the paper, but implied by the mechanism: moving to multi-digit subword tokens or coordinate-free action heads would likely erase the reported gains, making such tokenization a natural defensive choice.
  • The same place-value insight likely transfers beyond GUI grounding to any autoregressive model that emits numeric coordinates, such as robotic end-effector positions or object bounding boxes, where high-order digit flips dominate downstream displacement.
  • The gap identified in Eq. (11) suggests a testable extension: an objective that combines soft-coordinate distance with per-position argmax pressure may push targeted ASR higher than pure digit CE.
  • Since teacher forcing is only a proxy for greedy decoding, a loss that differentiates through decoding (e.g., by soft-argmax or beam approximation) could close the remaining gap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper studies white-box adversarial attacks on GUI grounding models that output screen coordinates as autoregressively generated digit-token sequences. The authors observe that a one-token change at a high decimal place induces a much larger parsed-coordinate shift than at a low place (Eqs. (4)-(5)), and that untargeted and targeted attacks require different objectives. They propose MissClick-U, which maximizes the squared distance of a soft coordinate constructed from teacher-forced digit distributions (Eqs. (7)-(9)), and MissClick-T, which minimizes a place-value-weighted target-digit cross-entropy (Eq. (12)). Experiments on OS-Atlas and UGround over ScreenSpot-v2 report large ASR gains over Token CE and Representation attack under identical perturbation budgets, plus an objective ablation (Table 2) showing soft-coordinate distance is best for untargeted attacks while place-weighted digit CE is best for targeted attacks. The paper explicitly limits its scope to per-digit tokenized coordinate interfaces and white-box access.

Significance. If the reported results hold, the paper makes a meaningful contribution by connecting the numerical/place-value structure of generated coordinates to attack objective design. The evaluation is transparent in using the same epsilon, step size, iterations, and task subset across methods, and the ablations directly support the goal-specific design. The mathematical decomposition in Eqs. (4)-(5) is correct, and the counterexample after Eq. (11) correctly illustrates the gap between expected-digit and argmax-digit optimization. The authors also state limitations honestly. However, the headline untargeted results currently rest on a fragile evaluation convention: unparseable outputs are counted as successful untargeted attacks, and the attack objective is computed from a renormalized ten-digit distribution that can, in principle, reward shifting mass onto non-digit tokens. This measurement issue must be resolved before the central claim that soft-coordinate displacement is the best untargeted objective can be accepted.

major comments (2)
  1. [Sec. 3.2 / Sec. 3.3 / Eq. (11)] The untargeted metric is load-bearing and potentially inflated. Sec. 3.1 states that an unparseable output counts as a successful untargeted attack, and Sec. 4.1 repeats this. Meanwhile Eq. (6) renormalizes the digit distribution over the ten digit tokens only, discarding all other vocabulary logits, while evaluation uses greedy decoding over the full vocabulary. Thus optimizing MissClick-U can increase probability mass on non-digit tokens (commas, brackets, letters) at digit positions; the renormalized distribution may look like a shifted coordinate while the decoded string is unparseable. Since unparseable outputs are counted as untargeted successes, a substantial part of the +16.62/+30.72 pp gains in Table 1 may be format-breaking rather than click displacement. The paper never reports the parse-failure rate. Please report, for each method and model, the fraction of untargeted success
  2. [Sec. 3.2 / Sec. 3.3 / Eq. (11)] The teacher-forcing proxy is acknowledged for targeted attacks but not stress-tested for untargeted attacks. Eq. (11) shows that an expected digit equal to the target does not imply the target digit is the argmax. The same misalignment applies to the untargeted objective: maximizing soft-coordinate displacement (Eq. (9)) can be driven by changes in low-probability digit values that never survive greedy decoding. The evaluation is empirical, so this is not circular, but it means the reported ASR could arise from a proxy that correlates only weakly with the true decoded coordinate. Please quantify the agreement between the teacher-forced argmax digit sequence at the end of optimization and the actually decoded greedy sequence, or otherwise show that the soft-coordinate gradient direction is aligned with the greedy-decoded click. This is important because the central claim is specifically t
minor comments (4)
  1. [Fig. 1] The figure has garbled annotations (e.g., '×10 0 Δx = +0 +30 -2' and arrow symbols). Please clean up the per-position delta display and the parse/execute flow.
  2. [Tables 1-3] No variance or confidence intervals are reported. Since the dataset is modest (about 1,000 tasks) and Table 3 uses a 300-task subset, reporting standard errors or repeated-run variability would help assess the robustness of the cross-platform and cross-budget comparisons.
  3. [Sec. 4.3] The term 'crossover analysis' appears in the conclusion but is not defined in Sec. 4.3. Please either define it or use a more descriptive phrase.
  4. [Sec. 3.4] The phrase 'maximizes the probability assigned to the target digit' followed by minimizing a loss is clear, but the later sentence 'Minimizing L_T increases the probability' is fine. For readability, define Digit CE baseline explicitly in Sec. 4.1; it currently appears only in Table 2 without a formal description.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: attack objectives are engineered surrogates, evaluated independently.

full rationale

The paper's claimed derivation is not circular. MissClick-U's LU (Eq. 9) and MissClick-T's LT (Eq. 12) are surrogate objectives directly motivated by the respective success conditions (click outside B_gt / inside B_tgt), but success is evaluated by independent greedy autoregressive decoding and coordinate parsing, not by the surrogate itself. The paper explicitly acknowledges the proxy gap in Eq. (11) (expected digit equal to target does not imply argmax), and the empirical objective comparison in Table 2 shows the surrogates are not trivially identical to their metrics: soft-coordinate distance is best for untargeted but poor for targeted, while digit CE and place-weighted digit CE behave differently. The decimal place-value weighting is a mathematical property of the output format, not a fitted parameter; no parameter is tuned to the reported ASR values. Reported ASR values are computed on ScreenSpot-v2 tasks with fixed hyperparameters and identical budgets across methods, and the victim-model tokenization premise ('each decimal digit is represented by an individual token') is stated as a factual scope condition, not imported from the authors' own prior results. There are no load-bearing self-citations or uniqueness theorems. The untargeted success metric's counting of unparseable outputs as successes is a potential validity concern but does not constitute circularity, because the attack objective is not defined in terms of parse failure and the paper does not equate the metric with the objective. The limitations section is explicit about scope, and nothing in the derivation reduces to its own inputs.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central claim rests on the four domain assumptions above plus elementary decimal math; nothing is fitted to the reported success rates. The attack hyperparameters (epsilon, alpha, N, target sampling) are standard or swept and are shared with all baselines, so they do not confer circular advantage. The loss construction uses ground-truth and target boxes as labels for teacher forcing, which is normal threat-model label use, not data fitting. No new entities are introduced.

free parameters (4)
  • epsilon (perturbation budget) = 16/255 default; swept over 4, 8, 16/255 in Table 3
    Chosen by hand; standard l-infinity budget and sensitivity reported, so not a constant tuned to maximize headline ASR.
  • alpha (PGD step size) = 1/255
    Conventional PGD step used identically for all compared methods.
  • N (optimization iterations) = 100; saturates by 10-30 in Fig. 2
    Iteration budget chosen by hand; Fig. 2 shows MissClick reaches baseline ASR within 10-30 iterations, so results are not tight to N.
  • Target region sampling protocol = uniform sample among zero-IoU annotated elements; 896 eligible tasks
    Defines task difficulty for the targeted metric; fixed across methods, but target hardness is not otherwise controlled.
assumptions (6)
  • domain assumption Victim models OS-Atlas and UGround serialize each decimal digit of a coordinate as an individual token, in a normalized [0,1000] coordinate system
    Stated in Sec. 3.2 ('each decimal digit is represented by an individual token'); if a model used subword digit tokens or a coordinate-free head (GUI-Actor, excluded in Sec. 2), the attack's logit-extraction step does not apply.
  • domain assumption Teacher-forced digit distributions p_{k,j} are a usable differentiable proxy for the model's greedy-decoded coordinate output
    Secs. 3.2-3.4: gradients flow through teacher-forced logits while success is measured on greedy generation; the paper itself flags the expectation-vs-argmax gap in Eq. (11).
  • domain assumption White-box threat model: the attacker can add pixel perturbations to the screenshot and knows the architecture, weights, parse function, and click mapping r(.)
    Sec. 3.1 defines the threat model; all ASR results depend on it, and transferability/black-box settings are declared future work in Sec. 5.
  • domain assumption The executed click is fully determined by parsing the coordinate string and applying r(.) (point identity or box center), with no post-processing or safety layer
    Eqs. (1)-(3) in Sec. 3.1 equate success with r(parse(M(I+delta,q))); any deployment-side filtering would change the success condition.
  • domain assumption ScreenSpot-v2 ground-truth boxes are correct labels for the intended UI element, so clicks outside them count as failures
    Sec. 4.1 computes ASR only over tasks the victim grounds correctly and treats B_gt as authoritative; label noise would bias both ASR and attack difficulty.
  • standard math Base-10 place-value expansion: c_k = sum_j 10^{m_k-j} * d_{k,j}
    Sec. 3.2 Eq. (4): definition of decimal representation; no empirical content, used to build both objectives.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MissClick: Exploiting Digit-Serialized Coordinates to Attack GUI Grounding Models." pith.science (2026). https://pith.science/paper/RDT5WKCT

@misc{pith2026260803740,
  author       = {Pith},
  title        = {Pith review of: MissClick: Exploiting Digit-Serialized Coordinates to Attack GUI Grounding Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RDT5WKCT}},
  note         = {Machine review of arXiv:2608.03740}
}
read the original abstract

Recent GUI visual grounding models generate screen coordinates as sequences of digit tokens that are parsed into numerical values and mapped to executable clicks. The security implications of this coordinate generation process have been largely overlooked. We observe that each coordinate digit is predicted as a categorical token, yet after parsing, changing a hundreds-place digit by one changes the corresponding numerical coordinate component by 100 units, which can induce a large displacement of the executed click. This observation motivates attack objectives that account for the numerical and place-value structure of coordinate outputs rather than treating them as ordinary text. Moreover, untargeted and targeted attacks impose different success conditions--displacing the click outside the correct region versus into an attacker-specified region--and therefore benefit from different objectives. We propose MissClick, a simple and effective white-box adversarial attack with two goal-specific objectives: MissClick-U maximizes soft-coordinate displacement for untargeted disruption, while MissClick-T minimizes a place-weighted target-digit loss for targeted hijacking. Compared with existing attacks against GUI grounding models on OS-Atlas and UGround across desktop, web, and mobile platforms, MissClick-U achieves untargeted success rates of 75.07\% and 72.93\% (+16.62 and +30.72 pp), and MissClick-T achieves targeted success rates of 44.86\% and 62.67\% (+31.73 and +47.06 pp). Attack objective comparison further shows that soft-coordinate displacement yields the highest untargeted attack success rate, whereas place-weighted target-digit optimization yields the highest targeted attack success rate, revealing distinct objective preferences for the two attack goals.

Figures

Figures reproduced from arXiv: 2608.03740 by the authors.

Figure 1
Figure 1. The coordinate generation pipeline of GUI ground [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Executed clicks from UGround on ScreenSpot-v2 [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 2
Figure 2. Attack success rate under different optimization [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 27 canonical work pages

  1. [1]

    Findings of the Association for Computational Linguistics (ACL) , pages=

    Gui agents: A survey , author=. Findings of the Association for Computational Linguistics (ACL) , pages=

    work page

  2. [2]

    Advances in Neural Information Processing Systems (NeurIPS) , volume=

    Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for llm agents , author=. Advances in Neural Information Processing Systems (NeurIPS) , volume=

    work page

  3. [3]

    arXiv preprint arXiv:2507.23779 , year=

    Phi-ground tech report: Advancing perception in gui grounding , author=. arXiv preprint arXiv:2507.23779 , year=

    arXiv

  4. [4]

    Advances in Neural Information Processing Systems (NeurIPS) , volume=

    Are aligned neural networks adversarially aligned? , author=. Advances in Neural Information Processing Systems (NeurIPS) , volume=

    work page

  5. [5]

    Intriguing properties of neural networks , year =

    Szegedy, Christian and Zaremba, Wojciech and Sutskever, Ilya and Bruna, Joan and Erhan, Dumitru and Goodfellow, Ian and Fergus, Rob , booktitle=. Intriguing properties of neural networks , year =

    work page

  6. [6]

    Proceedings of Annual Meeting of the Association for Computational Linguistics (ACL) , pages=

    Advancing sequential numerical prediction in autoregressive models , author=. Proceedings of Annual Meeting of the Association for Computational Linguistics (ACL) , pages=

    work page

  7. [7]

    Proceedings of International Conference on Machine Learning (ICML) , pages=

    Image Hijacks: adversarial images can control generative models at runtime , author=. Proceedings of International Conference on Machine Learning (ICML) , pages=

    work page

  8. [8]

    Proceedings of International Conference on Machine Learning (ICML) , pages=

    Regress, Don’t Guess: A Regression-like Loss on Number Tokens for Language Models , author=. Proceedings of International Conference on Machine Learning (ICML) , pages=. 2025 , organization=

    work page 2025

Show all 33 references
  1. [9]

    International Conference on Learning Representations (ICLR) , year=

    Teaching Metric Distance to Discrete Autoregressive Language Models , author=. International Conference on Learning Representations (ICLR) , year=

  2. [10]

    IEEE Symposium on Security and Privacy (IEEE S&P) , organization=

    Towards evaluating the robustness of neural networks , author=. IEEE Symposium on Security and Privacy (IEEE S&P) , organization=

  3. [11]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages=

    Cogagent: A visual language model for gui agents , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages=

  4. [12]

    Proceedings of the International Conference on Machine Learning (ICML) , pages=

    Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks , author=. Proceedings of the International Conference on Machine Learning (ICML) , pages=. 2020 , organization=

    2020

  5. [13]

    Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , pages=

    Seeclick: Harnessing gui grounding for advanced visual gui agents , author=. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , pages=

  6. [14]

    International Conference on Learning Representations (ICLR) , year=

    OS-ATLAS: Foundation action model for generalist GUI agents , author=. International Conference on Learning Representations (ICLR) , year=

  7. [15]

    International Conference on Learning Representations (ICLR) , volume=

    Navigating the digital world as humans do: Universal visual grounding for gui agents , author=. International Conference on Learning Representations (ICLR) , volume=

  8. [16]

    arXiv preprint arXiv:2501.12326 , year=

    Ui-tars: Pioneering automated gui interaction with native agents , author=. arXiv preprint arXiv:2501.12326 , year=

    arXiv

  9. [17]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages=

    Showui: One vision-language-action model for gui visual agent , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages=

  10. [18]

    Proceedings of the International Conference on Machine Learning (ICML) , year=

    Aguvis: Unified pure vision agents for autonomous gui interaction , author=. Proceedings of the International Conference on Machine Learning (ICML) , year=

  11. [19]

    arXiv preprint arXiv:2510.03230 , year=

    Improving GUI Grounding with Explicit Position-to-Coordinate Mapping , author=. arXiv preprint arXiv:2510.03230 , year=

  12. [20]

    Advances in Neural Information Processing Systems (NeurIPS) , volume=

    Gui-actor: Coordinate-free visual grounding for gui agents , author=. Advances in Neural Information Processing Systems (NeurIPS) , volume=

  13. [21]

    International Conference on Learning Representations (ICLR) , year=

    Explaining and harnessing adversarial examples , author=. International Conference on Learning Representations (ICLR) , year=

  14. [22]

    International Conference on Learning Representations (ICLR) , year=

    Towards Deep Learning Models Resistant to Adversarial Attacks , author=. International Conference on Learning Representations (ICLR) , year=

  15. [23]

    Advances in Neural Information Processing Systems (NeurIPS) , volume=

    On evaluating adversarial robustness of large vision-language models , author=. Advances in Neural Information Processing Systems (NeurIPS) , volume=

  16. [24]

    Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops , pages=

    On the adversarial robustness of multi-modal foundation models , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops , pages=

  17. [25]

    International Conference on Learning Representations (ICLR) , volume=

    Jailbreak in pieces: Compositional adversarial attacks on multi-modal language models , author=. International Conference on Learning Representations (ICLR) , volume=

  18. [26]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , pages=

    On the robustness of gui grounding models against image attacks , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , pages=

  19. [27]

    International Conference on Learning Representations (ICLR) , volume=

    Dissecting Adversarial Robustness of Multimodal LM Agents , author=. International Conference on Learning Representations (ICLR) , volume=

  20. [28]

    Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    Webinject: Prompt injection attack to web agents , author=. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

  21. [29]

    Conference on Language Modeling (COLM) , year=

    VisualTrap: A Stealthy Backdoor Attack on GUI Agents via Visual Grounding Manipulation , author=. Conference on Language Modeling (COLM) , year=

  22. [30]

    arXiv preprint arXiv:2509.11250 , year=

    Environmental Injection Attacks against GUI Agents in Realistic Dynamic Environments , author=. arXiv preprint arXiv:2509.11250 , year=

  23. [31]

    Foundations and Trends in Privacy and Security , volume=

    Safety at scale: A comprehensive survey of large model and agent safety , author=. Foundations and Trends in Privacy and Security , volume=. 2026 , publisher=

    2026

  24. [32]

    Transactions on Machine Learning Research (TMLR) , year=

    Ufo2: The desktop agentos , author=. Transactions on Machine Learning Research (TMLR) , year=

  25. [33]

    Proceedings of the European Conference on Computer Systems (EuroSys) , pages=

    From Imperative to Declarative: Towards LLM-friendly OS Interfaces for Boosted Computer-Use Agents , author=. Proceedings of the European Conference on Computer Systems (EuroSys) , pages=

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.