Pith. sign in

REVIEW 4 major objections 5 minor 36 references

Attention-driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models without Fine-Tuning

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By reading out attention maps from a frozen multimodal LLM, a new method grounds GUI elements as accurately as models fine-tuned for the task.

desk verdict The tuning-free attention pipeline works surprisingly well for text grounding, but the headline numbers rest on test-set-selected hyperparameters, so the paper needs a validation split before the claims are fully established. read the letter →

arxiv 2412.10840 v1 pith:4PAF3LSZ submitted 2024-12-14 cs.CV

Hai-Ming Xu , Qi Chen , Lei Wang , Lingqiao Liu This is my paper · ORCID
classification cs.CV
keywords GUIgroundingattentionmapmultimodallargelanguagemodeltuning-freeelementlocalizationtextScreenSpotbenchmarkparameter-freeadaptation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper establishes that a pretrained multimodal large language model can ground GUI elements—locate text and icons in a screenshot from a text command—without any GUI-specific fine-tuning, by reading out its internal attention maps. The proposed TAG method selects the tokens that describe the target element, aggregates their self-attention to the model's visual tokens, filters to the most informative attention heads, and projects the result back onto the image plane to produce a click point. On the ScreenSpot benchmark it reports 54.8% average accuracy, above fine-tuned GUI grounding models; on a new OCR grounding set built from Mind2Web screenshots it reaches 84.5% across ten aspect ratios. The significance is that spatial grounding may be an emergent, extractable property of off-the-shelf MLLMs, which could reduce the data and compute needed to build GUI agents.

What carries the argument

The machinery is a token-to-patch attention propagation chain. First, adaptive text token selection builds a prompt that makes the MLLM generate a short description of the target element, and the tokens of that description are the seeds. For each seed token, the method extracts per-head self-attention weights to the fixed set of visual query tokens in the LLM (Eq. 1 aggregates them with per-token, per-head weights α). The head selection rule (Eq. 4) scores each head by the sum of its attention from that token to all visual query tokens, keeps the top-K heads, and sets α=1 for them and 0 otherwise. The averaged token-to-query attention is then multiplied by the vision encoder's cross-attention matrix Across, which maps each visual query token to the H×W patch grid (Eq. 2), producing a per-token relevance map over image patches; these maps are averaged over selected tokens (Eq. 3). A threshold δ binarizes the map, connected components are found, and the center of the highest-average region becomes the prediction.

What would settle it

Evaluate TAG on a different pretrained MLLM or on a GUI distribution where per-head attention sums do not correlate with whether a head's attention peak falls inside the ground-truth box; if top-K selection then yields worse accuracy than averaging all heads, the head-selection heuristic is falsified. Concretely, for each head compute both its Eq. (4) score and its localization accuracy (peak inside GT box), and check whether the top-K by score are also the top performers by accuracy.

Watch

Extended reading notes

Core claim

The central claim of this paper is that a well-structured aggregation of attention maps from a pretrained MLLM can serve as a high-performance GUI element localizer without any fine-tuning. Concretely, the authors instruct MiniCPM-Llama3-V 2.5 to output a description of the element relevant to the user's command, then take the self-attention from those description tokens to the model's visual query tokens in each of the LLM's 1024 attention heads. Each head's contribution is weighted by a quality score—the total attention mass from the token to all visual query tokens—and only the top-K heads per token survive; the surviving maps are averaged and propagated through the vision encoder's cross-attention to image patches. Thresholding the resulting relevance map and taking the center of the highest-scoring connected region gives the predicted location. This pipeline achieves 54.8% average accuracy on ScreenSpot (above the fine-tuned SeeClick and CogAgent), 84.5% on the new OCG dataset across ten aspect ratios, and 87.4% on VisualWebBench's element grounding task. The authors also apply the same propagation to Qwen-VL-Chat, improving its OCG accuracy from 2.7% to 10.2%, which they read as evidence that the mechanism generalizes across MLLMs.

Load-bearing premise

The method assumes that the sum of attention a head pays from a description token to all visual query tokens is a reliable measure of that head's spatial grounding quality, so discarding the low-sum heads cannot remove the heads that actually localize correctly.

Editorial extensions

If this is right

  • GUI agents can be assembled from frozen pretrained MLLMs, sidestepping the cost of GUI-specific fine-tuning and the risk of overfitting to narrow UI distributions.
  • Text grounding becomes a reliable signal: the reported ScreenSpot text accuracies (88.3% mobile, 82.5% desktop, 70.9% web) are high enough to act as an OCR-coordinate backend for click actions.
  • The method extends to other MLLMs, as demonstrated by the Qwen-VL-Chat transfer experiment, so improvements in base models may automatically improve grounding.
  • The OCG dataset supplies a fixed benchmark with ten aspect ratios, letting the community measure how grounding degrades as screen shapes move away from square.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The head-selection heuristic might be replaced by a calibration step that measures per-head grounding quality on a handful of annotated screenshots, which could transfer TAG to models whose attention sums do not track localization accuracy.
  • The same propagation chain could be applied to other fine-grained visual grounding tasks, such as referring-expression comprehension or visual prompt pointing, where a natural-language expression must be mapped to an image region.
  • Because the method relies on the MLLM's own element description, prompt design (asking for exact text vs. keywords vs. a full sentence) may be a tuneable lever for different UI platforms and element types.
  • The paper's high text accuracy but lower icon accuracy suggests that attention readouts carry strong lexical-spatial alignment but weaker semantic-spatial alignment; training or prompting that strengthens icon-description tokenization could be a direct avenue for improvement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes TAG, a tuning-free GUI grounding method that aggregates self-attention maps from a pretrained multimodal large language model (MiniCPM-Llama3-V 2.5) to localize elements described by a text query. The method constructs a prompt that asks the model to output an element description, then propagates attention from the generated description tokens to visual query tokens and from there to image patches via cross-attention (Eqs. 1-3). A head-selection heuristic (Eq. 4) keeps only the top-K self-attention heads per token, and a thresholded relevance map yields the final point prediction. The authors evaluate on a newly introduced OCR grounding benchmark (OCG), ScreenSpot, and Mind2Web, reporting that TAG outperforms fine-tuned GUI grounding models such as SeeClick and CogAgent on ScreenSpot average accuracy and on OCG, while matching SeeClick on Mind2Web element accuracy. The code is publicly available.

Significance. If the reported results hold under fair evaluation, the claim is significant: it would show that attention maps of a pretrained MLLM can substitute for task-specific fine-tuning in GUI grounding, with a simple and relatively efficient pipeline (Table 6). The paper also introduces a new benchmark (OCG) and demonstrates a generalization experiment on Qwen-VL-Chat. However, the central claim is weakened by the selection of the two hyperparameters K and delta on the same benchmarks whose final numbers are reported, and by the lack of a released OCG dataset. The paper's strengths include its clear method formulation, ablations showing each component contributes, and reproducible greedy generation.

major comments (4)
  1. [Section 4.5, Figures 6-7] The two free parameters K and delta are selected by ablating on the same evaluation benchmarks whose final scores are reported in Tables 1-3. The text states 'Based on these results, we use K = 10 in all experiments' and 'delta = 0.5 is used across all datasets,' but no validation split is described. Figure 6 shows strong sensitivity to K (e.g., K=1 substantially lowers accuracy), so the headline comparison against fine-tuned models is conditional on hyperparameters chosen after seeing the test data. Please add a validation split or otherwise demonstrate that the selection does not constitute tuning on the test set; if this is not possible, the claim of a 'tuning-free' advantage should be tempered accordingly.
  2. [Section 4.1, Table 1] The OCG dataset is constructed from Mind2Web test screenshots with Azure OCR, but the dataset is not released. Moreover, the queries are exact OCR text, so this benchmark tests text grounding with perfectly aligned queries rather than the harder case of grounding natural-language commands. Since the OCG result (84.5% vs. 60.2% for SeeClick) is a primary demonstration of the method's advantage, the dataset and exact query-construction procedure must be released for independent verification, or the claim should be restricted to OCR-exact text grounding.
  3. [Section 4, Tables 1-3] All results are single-run point estimates without standard errors or confidence intervals. For example, the ScreenSpot average gap between TAG (54.8%) and SeeClick (53.4%) is 1.4 percentage points; without an estimate of variance, it is unclear whether this difference is meaningful. Please report multiple runs (e.g., different random seeds or bootstrapping over the evaluation set) or provide statistical significance tests, especially for claims of surpassing fine-tuned baselines.
  4. [Section 3.5, Eq. (4)] The head-selection rule assumes that the sum of attention from a text token to all visual query tokens, \tilde{A}^k_{T_j}, is a valid proxy for head quality for grounding. The justification is anecdotal, relying on Figure 3. The ablation in Table 4 shows the filter helps on average, but the paper does not analyze whether the heuristic holds for icon/widget grounding or under distribution shifts. Please provide a quantitative analysis of the correlation between \tilde{A}^k and grounding success per element type, or a sensitivity analysis across different models and UI distributions, to support the generality of the selection rule.
minor comments (5)
  1. [Figure 2] The caption and the diagram use the term 'unfold' without explanation; please clarify that the head-wise attention is reshaped from the raw attention tensor.
  2. [Section 3.3] The name 'Adaptive Text Token Selection' is somewhat misleading: the method does not explicitly select tokens from the user query but instead relies on the model's generated element description. Consider renaming or clarifying the mechanism.
  3. [Section 4.1] The sentence 'its pre-training data may make it impossible to include images of any aspect ratios' likely means 'may make it difficult to perform well on unseen aspect ratios' rather than 'impossible'; please rephrase.
  4. [Section 6.3] The OCG dataset statistics in Table 5 are useful, but please also state whether the dataset is released and under what license, since the main text does not provide a link.
  5. [Tables 2 and 3] The column header 'MLLMs w/o SFT' could be made cleaner as 'Without SFT' to avoid inconsistent spacing; the table captions should also indicate whether the 'MLLMs' column refers to model size or model family.

Circularity Check

1 steps flagged · score 3.0 of 10

No derivation-level circularity; the attention aggregation equations are deterministic and independent of ground truth, but the two hyperparameters K and δ are selected on the same evaluation benchmarks whose headline numbers are reported.

  1. fitted input called prediction [Section 4.5, 'Impact of Top K' and 'Impact of Threshold δ' (Figures 6 and 7)]
    "Figure 6 shows that reducing K initially improves performance, with optimal results at K = 10 for both aspect ratios. ... Based on these results, we use K = 10 in all experiments. ... Figure 7 shows that with a lower threshold δ ≤ 0.3, the model's performance is suboptimal due to including too many fairly attended regions. As δ increases, the model's performance reaches its peak at δ = 0.5, but diminishes if δ is increased further. Thus δ = 0.5 is used across all datasets."

    The paper presents a 'tuning-free' pipeline, yet its two free constants are chosen by maximizing accuracy on the evaluation benchmarks themselves, with no validation split described. The final tables then report accuracy on those same benchmarks using the test-selected K = 10 and δ = 0.5, so the headline numbers are not independent predictions of a fully fixed method: they are the result of a benchmark-level fit. The attention aggregation itself (Eqs. 1-3) is deterministic and not fitted to ground-truth boxes, so this is a mild, partial circularity rather than a collapse of the derivation.

full rationale

The core derivation chain is self-contained: Eqs. (1)-(3) deterministically aggregate pretrained attention maps, Eq. (4) is a stated heuristic rather than a fitted quantity, and no load-bearing self-citation or imported uniqueness theorem is used. The main circularity-adjacent issue is that K and δ are tuned on the same ScreenSpot/OCG/Mind2Web benchmarks whose final scores are then reported, which makes the claimed advantage over SFT models conditional on test-set-selected constants. This is fixable with a held-out validation split and does not negate the independent content of the attention-based approach.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical or model-level entities. It relies on the pretrained model's existing attention mechanisms, plus two hyperparameters (K and δ) that are tuned on the evaluation benchmarks. The key assumptions are that attention maps carry spatial grounding information and that a specific head-quality heuristic is reliable.

free parameters (2)
  • K (number of top self-attention heads) = 10
    K is selected from ablation on the test benchmarks (Figure 6) and used across all experiments. It directly controls which attention heads contribute to the grounding heatmap.
  • δ (binarization threshold for the relevance heatmap) = 0.5
    δ is selected from ablation on the test benchmarks (Figure 7) and used across all datasets. It determines the high-relevance region from which the predicted location is derived.
assumptions (3)
  • domain assumption The self-attention and cross-attention maps of MiniCPM-Llama3-V 2.5 contain usable spatial correspondence between text tokens and image regions.
    The entire method assumes attention maps are a valid spatial grounding signal. No proof is offered; the paper relies on empirical validation in Sections 3 and 4.
  • ad hoc to paper The magnitude of the summed attention, \tilde{A}^k_{T_j} in Eq. (4), indicates the quality of a self-attention head for grounding.
    This heuristic is introduced specifically for this method and justified only by the anecdotal Figure 3, where heads with high sums are observed to lie inside the ground-truth box.
  • domain assumption The connected region with the highest average relevance score corresponds to the target GUI element.
    Used in Section 3.4 to convert the heatmap into a point prediction. The evaluation metric (prediction inside ground-truth box) implicitly relies on this assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Attention-driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models without Fine-Tuning." pith.science (2026). https://pith.science/paper/4PAF3LSZ

@misc{pith2026241210840,
  author       = {Pith},
  title        = {Pith review of: Attention-driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models without Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4PAF3LSZ}},
  note         = {Machine review of arXiv:2412.10840}
}
read the original abstract

Recent advancements in Multimodal Large Language Models (MLLMs) have generated significant interest in their ability to autonomously interact with and interpret Graphical User Interfaces (GUIs). A major challenge in these systems is grounding-accurately identifying critical GUI components such as text or icons based on a GUI image and a corresponding text query. Traditionally, this task has relied on fine-tuning MLLMs with specialized training data to predict component locations directly. However, in this paper, we propose a novel Tuning-free Attention-driven Grounding (TAG) method that leverages the inherent attention patterns in pretrained MLLMs to accomplish this task without the need for additional fine-tuning. Our method involves identifying and aggregating attention maps from specific tokens within a carefully constructed query prompt. Applied to MiniCPM-Llama3-V 2.5, a state-of-the-art MLLM, our tuning-free approach achieves performance comparable to tuning-based methods, with notable success in text localization. Additionally, we demonstrate that our attention map-based grounding technique significantly outperforms direct localization predictions from MiniCPM-Llama3-V 2.5, highlighting the potential of using attention maps from pretrained MLLMs and paving the way for future innovations in this domain.

Figures

Figures reproduced from arXiv: 2412.10840 by the authors.

Figure 1
Figure 1. Illustration of MiniCPMV2.5’s strong GUI image understanding but poor element localization. Our attention-driven [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overall pipeline of our TAG approach in Sec. 3.4 (top) and the self-attention selection module in Sec. 3.5 (bottom). [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Demonstrating how choosing top self-attention heads improves text-to-image token mapping (see Sec. 3.5 for details). [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Demonstration of the comparing methods on two cases of ScreenSpot. Our attention-driven grounding with element [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Demonstration of our method on Mind2Web to ground precisely at each step and successfully achieve the overall [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Ablation on Top K [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 9
Figure 9. Figure 9: It is almost the same as the templates used for our [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 8
Figure 8. Figure 8: Demonstration of the Full output for the query “Please do OCR on this image.” with MiniCPMV2.5. [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: The prompt templates used for MiniCPM-Llama3-V 2.5 and our model to accomplish tasks in the Mind2Web agent [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: Demonstration of our method on Mind2Web to ground precisely at each step and successfully achieve the overall [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 11 canonical work pages

  1. [1]

    Anthropic. 2024. Introducing the next generation of C laude

    work page 2024

  2. [2]

    Bai, J.; Bai, S.; Yang, S.; Wang, S.; Tan, S.; Wang, P.; Lin, J.; Zhou, C.; and Zhou, J. 2023. Qwen-VL : A Frontier Large Vision-Language Model with Versatile Abilities. arXiv preprint arXiv:2308.12966

    arXiv 2023

  3. [3]

    Chen, J.; Zhu, D.; Shen, X.; Li, X.; Liu, Z.; Zhang, P.; Krishnamoorthi, R.; Chandra, V.; Xiong, Y.; and Elhoseiny, M. 2023. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478

    arXiv 2023

  4. [4]

    Chen, W.; Cui, J.; Hu, J.; Qin, Y.; Fang, J.; Zhao, Y.; Wang, C.; Liu, J.; Chen, G.; Huo, Y.; et al. 2024. GUICourse: From General Vision Language Models to Versatile GUI Agents. arXiv preprint arXiv:2406.11317

    arXiv 2024

  5. [5]

    Cheng, K.; Sun, Q.; Chu, Y.; Xu, F.; Li, Y.; Zhang, J.; and Wu, Z. 2024. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935

    arXiv 2024

  6. [6]

    Deng, X.; Gu, Y.; Zheng, B.; Chen, S.; Stevens, S.; Wang, B.; Sun, H.; and Su, Y. 2024. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36

    work page 2024

  7. [7]

    Fan, Y.; Ding, L.; Kuo, C.-C.; Jiang, S.; Zhao, Y.; Guan, X.; Yang, J.; Zhang, Y.; and Wang, X. E. 2024. Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding. arXiv preprint arXiv:2406.19263

    arXiv 2024

  8. [8]

    Gao, D.; Ji, L.; Bai, Z.; Ouyang, M.; Li, P.; Mao, D.; Wu, Q.; Zhang, W.; Wang, P.; Guo, X.; et al. 2024. AssistGUI: Task-Oriented PC Graphical User Interface Automation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13289--13298

    work page 2024

Show all 36 references
  1. [9]

    He, H.; Yao, W.; Ma, K.; Yu, W.; Dai, Y.; Zhang, H.; Lan, Z.; and Yu, D. 2024. WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models. arXiv preprint arXiv:2401.13919

    2024 arXiv

  2. [10]

    Hong, W.; Wang, W.; Lv, Q.; Xu, J.; Yu, W.; Ji, J.; Wang, Y.; Wang, Z.; Dong, Y.; Ding, M.; et al. 2023. CogAgent: A Visual Language Model for GUI Agents. arXiv preprint arXiv:2312.08914

    2023 arXiv

  3. [11]

    P.; Russak, M.; Koh, J

    Kapoor, R.; Butala, Y. P.; Russak, M.; Koh, J. Y.; Kamble, K.; Alshikh, W.; and Salakhutdinov, R. 2024. OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web. arXiv preprint arXiv:2402.17553

    2024 arXiv

  4. [12]

    H.; Zheng, B.; Deng, X.; Su, Y.; and Chao, W.-L

    Kil, J.; Song, C. H.; Zheng, B.; Deng, X.; Su, Y.; and Chao, W.-L. 2024. Dual-View Visual Contextualization for Web Navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14445--14454

    2024

  5. [13]

    Li, G.; and Li, Y. 2023. Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus. In The Eleventh International Conference on Learning Representations

    2023

  6. [14]

    J.-J.; Mitchell, T.; and Myers, B

    Li, T. J.-J.; Mitchell, T.; and Myers, B. 2020. Interactive task learning from GUI-grounded natural language instructions and demonstrations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 215--223

    2020

  7. [15]

    Li, Y.; He, J.; Zhou, X.; Zhang, Y.; and Baldridge, J. 2020. Mapping Natural Language Instructions to Mobile UI Action Sequences. In Jurafsky, D.; Chai, J.; Schluter, N.; and Tetreault, J., eds., Proceedings of the 58th Annual Meeting of the Association for Computational Lingu...

    2020

  8. [16]

    Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2024 a . Visual Instruction Tuning. NeurIPS, 36

    2024

  9. [17]

    Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2024 b . Visual instruction tuning. Advances in neural information processing systems, 36

    2024

  10. [18]

    Y.; Lam, W.; Neubig, G.; Li, Y.; and Yue, X

    Liu, J.; Song, Y.; Lin, B. Y.; Lam, W.; Neubig, G.; Li, Y.; and Yue, X. 2024 c . VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding? arXiv preprint arXiv:2404.05955

    2024 arXiv

  11. [19]

    Lu, H.; Liu, W.; Zhang, B.; Wang, B.; Dong, K.; Liu, B.; Sun, J.; Ren, T.; Li, Z.; Sun, Y.; et al. 2024. DeepSeek-VL : Towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525

    2024 arXiv

  12. [20]

    Ma, X.; Zhang, Z.; and Zhao, H. 2024. CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation. In Findings of the Association for Computational Linguistics ACL 2024, 9097--9110

    2024

  13. [21]

    Nong, S.; Zhu, J.; Wu, R.; Jin, J.; Shan, S.; Huang, X.; and Xu, W. 2024. MobileFlow: A Multimodal LLM For Mobile GUI Agent. arXiv preprint arXiv:2407.04346

    2024 arXiv

  14. [22]

    OpenAI. 2023. GPT-4V(ision) System Card

    2023

  15. [23]

    Reid, M.; Savinov, N.; Teplyashin, D.; Lepikhin, D.; Lillicrap, T.; Alayrac, J.-b.; Soricut, R.; Lazaridou, A.; Firat, O.; Schrittwieser, J.; et al. 2024. Gemini 1.5 : Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530

    2024 arXiv

  16. [24]

    Wang, B.; Li, G.; and Li, Y. 2023. Enabling Conversational Interaction with Mobile UI using Large Language Models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI '23. New York, NY, USA: Association for Computing Machinery. ISBN 9781450394215

    2023

  17. [25]

    Wang, H.; Li, T.; Deng, Z.; Roth, D.; and Li, Y. 2024 a . Devil ' s Advocate: Anticipatory Reflection for LLM Agents. In Al-Onaizan, Y.; Bansal, M.; and Chen, Y.-N., eds., Findings of the Association for Computational Linguistics: EMNLP 2024, 966--978. Miami, Florida, USA: Ass...

    2024

  18. [26]

    Wang, J.; Xu, H.; Ye, J.; Yan, M.; Shen, W.; Zhang, J.; Huang, F.; and Sang, J. 2024 b . Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception. arXiv preprint arXiv:2401.16158

    2024 arXiv

  19. [27]

    Wang, L.; Deng, Y.; Zha, Y.; Mao, G.; Wang, Q.; Min, T.; Chen, W.; and Chen, S. 2024 c . MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents. arXiv preprint arXiv:2406.08184

    2024 arXiv

  20. [28]

    Wang, W.; Lv, Q.; Yu, W.; Hong, W.; Qi, J.; Wang, Y.; Ji, J.; Yang, Z.; Zhao, L.; Song, X.; et al. 2023. CogVLM : Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079

    2023 arXiv

  21. [29]

    Wu, Z.; Han, C.; Ding, Z.; Weng, Z.; Liu, Z.; Yao, S.; Yu, T.; and Kong, L. 2024. Os-copilot: Towards generalist computer agents with self-improvement. arXiv preprint arXiv:2402.07456

    2024 arXiv

  22. [30]

    J.; Cheng, Z.; Shin, D.; Lei, F.; et al

    Xie, T.; Zhang, D.; Chen, J.; Li, X.; Zhao, S.; Cao, R.; Hua, T. J.; Cheng, Z.; Shin, D.; Lei, F.; et al. 2024. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972

    2024 arXiv

  23. [31]

    Yang, J.; Zhang, H.; Li, F.; Zou, X.; Li, C.; and Gao, J. 2023. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441

    2023 arXiv

  24. [32]

    Yao, Y.; Yu, T.; Zhang, A.; Wang, C.; Cui, J.; Zhu, H.; Cai, T.; Li, H.; Zhao, W.; He, Z.; Chen, Q.; Zhou, H.; Zou, Z.; Zhang, H.; Hu, S.; Zheng, Z.; Zhou, J.; Cai, J.; Han, X.; Zeng, G.; Li, D.; Liu, Z.; and Sun, M. 2024. MiniCPM-V: A GPT-4V Level MLLM on Your Phone. arXiv pr...

    2024 arXiv

  25. [33]

    You, K.; Zhang, H.; Schoop, E.; Weers, F.; Swearngin, A.; Nichols, J.; Yang, Y.; and Gan, Z. 2024. Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs. arXiv preprint arXiv:2404.05719

    2024 arXiv

  26. [34]

    Zhu, D.; Chen, J.; Shen, X.; Li, X.; and Elhoseiny, M. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592

    2023 arXiv

  27. [35]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  28. [36]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.