Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

Less is More: Empowering GUI Agent with Context-Aware Simplification

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SimpAgent uses training-time screenshot masking and a consistency-guided token-dropping scheme to compress GUI history, improving step success rate while cutting 27% of inference FLOPs.

desk verdict Solid efficiency win for GUI agents; fix the Eq. (3) sign error, qualify the Mind2Web SOTA, and stop overselling the masking mechanism. read the letter →

arxiv 2507.03730 v1 pith:MXNZLQRW submitted 2025-07-04 cs.CV cs.AIcs.HCcs.LG

Gongwei Chen , Xurui Zhou , Rui Shao , Yibo Lyu , Kaiwen Zhou , Shuai Wang , Wentao Li , Yinchuan Li
show 2 more authors
Zhongang Qi Liqiang Nie
This is my paper · ORCID
classification cs.CVcs.AIcs.HCcs.LG
keywords agenthistorymodelingelementagentscomprehensivecompressioncontext
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

GUI agents are systems that look at a phone or computer screen and decide what to click or type next. Modern agents feed many screenshots into a multimodal language model, and that is expensive. The authors of SimpAgent argue that most of the screen is irrelevant and that old screenshots repeat information the agent already has.

Their first trick is to hide a random rectangular part of the current screenshot during training. Because a typical target is small, they argue the hidden area is usually irrelevant, and the model learns to rely on the important elements. Their second trick targets history. Instead of letting all previous screenshots flow through every layer of the language model, they remove all historical screenshot tokens after an early layer, keeping only the text of past actions. To stop this compression from losing useful information, they also train a second branch that keeps the full history and ask the two branches to make similar action predictions.

On four benchmarks, the compressed agent uses 27% fewer FLOPs and slightly improves step success rate over the Qwen2-VL baseline. The gains are modest, around one to three percentage points, and one test subset is actually worse than the baseline. The method is mostly a combination of known ideas: random masking resembles CutOut augmentation, and dropping tokens after early layers was explored in FastV and Victor. The new part is the consistency objective between the compressed and full branches, although the equation for that objective appears to have the sign of the KL term backwards.

Extended reading notes

Core claim

SimpAgent, through masking-based element pruning and consistency-guided history compression, reduces inference FLOPs by 27% and achieves superior GUI navigation performance on AITW, Mind2Web, GUI-Odyssey, and AndroidControl without extra GUI pre-training data. If correct, the central claim is that context simplification, rather than more pretraining data, is a viable route to both efficiency and accuracy for pure-vision GUI agents.

Load-bearing premise

The masking-based element pruning assumes that a random rectangular mask during training removes mostly unrelated elements and that the 2% click region is usually left intact, so the model learns better element comprehension. The paper's own comparison in Figure 5(b) shows that an inverse Gaussian center distribution, which directly avoids masking the click region, underperforms uniform masking, so the mechanism linking masking to element relevance is not established. If masking helps through generic regularization instead of element pruning, the context-aware narrative of the paper breaks down.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes SimpAgent, a pure-vision GUI agent built on Qwen2-VL-2B, with two training-time components: masking-based element pruning (randomly masking a rectangular region of the current screenshot) and consistency-guided history compression (dropping history vision tokens at a chosen LLM layer while training a KL-based consistency loss against an uncompressed branch). The model is evaluated on AITW, Mind2Web, GUI-Odyssey, and AndroidControl, reporting a 27% inference FLOPs reduction and modest step-success-rate gains over fine-tuned Qwen2VL without additional GUI pre-training data. The paper argues that context simplification, rather than more pre-training data, is an effective route to both efficiency and accuracy for pure-vision GUI agents.

Significance. The empirical direction is timely and potentially useful: the paper demonstrates that a cheap training-time masking augmentation plus token dropping can improve or at least preserve GUI navigation accuracy while cutting inference FLOPs, and it compares against several recent systems on four benchmarks. The authors also promise open-source code, and the attention-map analyses in Figures 6 and 7 provide some mechanistic illustration. However, the paper's central explanatory narrative—that masking prunes unrelated elements and that the consistency loss explicitly steers compression—is not established by the evidence, and one equation contradicts the stated objective. The results, if confirmed with proper error reporting, would still be a practical contribution, but the current framing overclaims.

major comments (4)
  1. [§4.2, Eq. (3)] The loss as written is L = -DKL[πθ(ã_t | om_t, H_t, G) || πθ(a_t | om_t, H_c_t, G)] minus two cross-entropy terms. Since training minimizes L, the DKL term is maximized, which is the opposite of the stated goal of "minimizing the Kullback-Leibler divergence" between the two branches. This is load-bearing because the consistency guidance is presented as the mechanism that recovers the accuracy lost by token dropping. Please correct the sign (likely a plus) and specify the direction of the KL divergence, since DKL(original||truncated) and DKL(truncated||original) have different mode-covering behavior.
  2. [§4.1, Figure 5(b)] The proposed mechanism is that random rectangular masking prunes mostly unrelated elements while sparing the ~2% click region. The inverse-Gaussian center sampling distribution was explicitly designed to mask away from the click point and therefore should realize this rationale, yet it underperforms uniform masking. The text concedes that the element distribution is "more complex than initially assumed," which removes the evidential basis for the element-pruning interpretation. Without direct evidence that masked regions correspond predominantly to unrelated elements, masking may simply be a generic occlusion/regularization augmentation. Since the title and abstract attribute the accuracy gains to "context-aware simplification" via element pruning, this is a load-bearing conceptual claim that should either be supported by targeted experiments or reframed.
  3. [§5.3, Tables 5 and Figure 5] Main results are reported without error bars; only Figure 5(b) gives standard deviations, without stating the number of seeds. In addition, the key hyperparameters—mask size bounds (a,b), masking probability p, and drop layer k—are selected based on performance on the same benchmarks that appear in the final tables. Given that some gains are small (e.g., +0.7% on AndroidControl in Table 4, +0.1 to +0.2 in several Table 9 rows), selection bias and run-to-run noise are material. Please report multiple seeds and a fixed hyperparameter-selection protocol, or clearly state which numbers are tuning-set results.
  4. [§5.2, Table 9] The text states that SimpAgent achieves "new state-of-the-art performances" on Mind2Web, but the full results show SimpAgent's Cross-Domain Step SR (43.3) is below the Qwen2VL baseline (44.6); only SimpAgent-M reaches 45.0. On Cross-Website, SimpAgent-M ties Qwen2VL at 42.2. This contradicts the unqualified superiority claim and needs qualification or explanation.
minor comments (6)
  1. [§4.1, Eq. (2)] The masking value v is never specified; please state the pixel value used (e.g., 0 or gray) for reproducibility.
  2. [§4.1 and §5.3] The exact masking probability p and the final mask size bounds (a,b) used for the main experiments are not reported; Figure 5 only shows ablations. These values should be listed in the implementation details.
  3. [§4.2, Eq. (3)] The notation ã_t and a_t for the two branches is not formally introduced; please define which branch uses the full history H_t and which uses the truncated history H_c_t before giving the objective.
  4. [Table 5] The abbreviations "C.p." and "C. C.p." are opaque; expand them in the table or caption to "history compression" and "consistency-guided history compression."
  5. [Abstract and §1] The phrase "superior GUI navigation performances" is too broad given the Cross-Domain regression on Mind2Web; please qualify the claim to the settings where the improvement actually holds.
  6. [Various] Minor typos and formatting: §4.1 "intractable cause" should be "intractable because"; §5.3 "inverse gaussian" should be "inverse Gaussian"; Appendix A "we follow the same data processing settings in SeeClick" is missing a period and the dataset descriptions could be tightened.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SimpAgent's claims rest on external benchmarks and measured ablations; the masking-mechanism critique is a correctness/novelty concern, not a definitional or self-citation circularity.

full rationale

SimpAgent is an empirical systems paper rather than a derivation. Its central claims are that masking-based element pruning and consistency-guided history compression improve GUI navigation accuracy while reducing FLOPs; the accuracy numbers are measured on four external benchmarks (AITW, Mind2Web, GUI-Odyssey, AndroidControl), and the FLOPs reduction follows arithmetically from dropping historical vision tokens in the truncated branch. No equation defines a predicted quantity in terms of a fitted parameter, and no benchmark result is forced by construction: the pilot experiments in Table 1 motivate the design but do not constitute the reported results, and the ablation studies, including the inverse-Gaussian comparison that underperforms uniform masking, are transparent empirical findings rather than self-fulfilling definitions. Self-citations in the paper are limited to background and related-work contexts (e.g., the authors' prior multimodal and GUI-agent works) and are not load-bearing for the claimed benchmark gains. The concern raised by external reviewers that the masking mechanism is not established as element pruning, or that random rectangular masking resembles generic augmentation such as Random Erasing, is a correctness and novelty critique rather than a circularity critique: even if that mechanism were wrong, the central efficiency claim from token dropping remains independent, and the accuracy results would still be externally measured rather than equivalent to the inputs. A separate concern about hyperparameters being selected on evaluation sets is not evidenced in the text and would relate to evaluation validity, not to circular derivation. Overall, no load-bearing step reduces to its own inputs by definition, by fitted-input renaming, or by a self-citation chain.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on several hand-chosen hyperparameters, mask size, mask probability, and drop layer, and on empirical assumptions about GUI screenshots and LLM attention aggregation. No new physical or conceptual entities are introduced. The main unstated dependence is that the consistency loss works as described despite the sign inconsistency in Eq. (3).

free parameters (4)
  • Mask size bounds a, b = 0.5, 0.7
    Chosen so the masked region covers 25% to 50% of the screenshot; selected by ablation in Fig. 5(a).
  • Masking probability p = 0.5
    Chosen by hand for training-time masking; not derived from data.
  • History vision token drop layer k = 3
    Selected from Table 6 because k=3 gives near-saturated step SR (67.3%) with 27% FLOPs reduction.
  • Mask center sampling distribution = Uniform over inverse Gaussian
    Uniform performs better than the inverse Gaussian prior in Fig. 5(b); the paper concludes the element distribution is more complex than assumed.
assumptions (4)
  • domain assumption High element density and loose element relations mean unrelated elements occupy a large portion of screenshots, so random masking removes mostly unrelated elements.
    Sec. 3.2 and Sec. 4.1; supported by a pilot experiment with OR versus O and click-region statistics, but random masking is only a proxy for relevance.
  • domain assumption Historical visual information is highly redundant; adding 4 observations costs 3.4x FLOPs for only 3.0% step SR gain.
    Sec. 3.3 and Table 1; motivates dropping history vision tokens.
  • domain assumption Causal self-attention in shallow LLM layers aggregates historical visual features into adjacent action tokens, so dropping all history vision tokens after layer k retains task-relevant information.
    Sec. 4.2 and Fig. 4; borrowed from FastV, Victor, and VoCo-LLaMA observations and assumed to hold for GUI agents.
  • ad hoc to paper Minimizing KL divergence between truncated-branch and original-branch action predictions steers compression to preserve useful information.
    Sec. 4.2 and Eq. (3); the stated equation has the opposite sign, so this axiom is not implemented as written in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Less is More: Empowering GUI Agent with Context-Aware Simplification." pith.science (2026). https://pith.science/paper/MXNZLQRW

@misc{pith2026250703730,
  author       = {Pith},
  title        = {Pith review of: Less is More: Empowering GUI Agent with Context-Aware Simplification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MXNZLQRW}},
  note         = {Machine review of arXiv:2507.03730}
}
read the original abstract

The research focus of GUI agents is shifting from text-dependent to pure-vision-based approaches, which, though promising, prioritize comprehensive pre-training data collection while neglecting contextual modeling challenges. We probe the characteristics of element and history contextual modeling in GUI agent and summarize: 1) the high-density and loose-relation of element context highlight the existence of many unrelated elements and their negative influence; 2) the high redundancy of history context reveals the inefficient history modeling in current GUI agents. In this work, we propose a context-aware simplification framework for building an efficient and effective GUI Agent, termed SimpAgent. To mitigate potential interference from numerous unrelated elements, we introduce a masking-based element pruning method that circumvents the intractable relation modeling through an efficient masking mechanism. To reduce the redundancy in historical information, we devise a consistency-guided history compression module, which enhances implicit LLM-based compression through innovative explicit guidance, achieving an optimal balance between performance and efficiency. With the above components, SimpAgent reduces 27% FLOPs and achieves superior GUI navigation performances. Comprehensive navigation experiments across diverse web and mobile environments demonstrate the effectiveness and potential of our agent.

Figures

Figures reproduced from arXiv: 2507.03730 by the authors.

Figure 1
Figure 1. Compared to the large-scale pre-training scheme, we [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The illustration of high density and loose relation of [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of our context-aware simplification framework for building SimpAgent. The main components are masking-based [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: The attention maps during the LLM decoding process [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: (a): The analysis of masking region (h, w ∼ U(a, b)) in masking-based element pruning module. (b): The analysis of masking strategy based on different sampling distributions. The analysis of masking region choice. In [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 7
Figure 7. Figure 7: Illustration of attention maps in agent models w/ and w/o [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: The distribution of element nums in AITW dataset. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: The distribution of target element ratio in Mind2Web [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Illustration of navigation steps in the GUI-Odyssey dataset. SimpAgent distinguishes the correct element among various [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Illustration of attention maps in agent models w/ and w/o consistency guidance, and their attention difference map. The [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: The real-world application of SimpAgent when adapted to downstream GUI navigation tasks. [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: The real-world application of SimpAgent when adapted to downstream GUI navigation tasks. [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Rethinking Inference-Time Scaling in Local Computer-Use Agents: Failure Modes and Compute Tradeoffs

    cs.AI 2026-07 conditional novelty 6.0 of 10

    On OSWorld, inference-time scaling for local CUAs mostly yields diminishing returns and shifts failures toward premature false successes rather than raising task success.

  2. Uncertainty-Aware GUI Agent: Adaptive Perception through Component Recommendation and Human-in-the-Loop Refinement

    cs.AI 2025-08 conditional novelty 4.0 of 10

    A GUI agent that trims UI input with a recommendation module and asks users when decisions are ambiguous reports state-of-the-art success rates, though the interaction module is not benchmarked.

Reference graph

Works this paper leans on

64 extracted references · 62 canonical work pages · cited by 2 Pith papers

  1. [1]

    Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities. arXiv preprint, 2023. 7, 14

    work page 2023

  2. [2]

    Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation

    Hyungjoo Chae, Namyoung Kim, Kai Tzu-iunn Ong, Minju Gwak, Gwanwoo Song, Jihoon Kim, Sunghwan Kim, Dongha Lee, and Jinyoung Yeo. Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation. In The Thirteenth International Confer- ence on Learning Representations, 2024. arXiv, 2024. 1

    work page 2024

  3. [3]

    Lion: Empowering multimodal large language model with dual-level visual knowledge

    Gongwei Chen, Leyang Shen, Rui Shao, Xiang Deng, and Liqiang Nie. Lion: Empowering multimodal large language model with dual-level visual knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26540–26550, 2024. 2

    work page 2024

  4. [4]

    An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models. InEuropean Conference on Computer Vision, 2024. arXiv, 2024. 2, 3, 4, 7, 8

    work page 2024

  5. [5]

    InternVL: Scaling up Vision Foundation Mod- els and Aligning for Generic Visual-Linguistic Tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. InternVL: Scaling up Vision Foundation Mod- els and Aligning for Generic Visual-Linguistic Tasks. arXiv preprint, 2023. 7

    work page 2023

  6. [6]

    SeeClick: Har- nessing GUI Grounding for Advanced Visual GUI Agents

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yan- tao Li, Jianbing Zhang, and Zhiyong Wu. SeeClick: Har- nessing GUI Grounding for Advanced Visual GUI Agents. In Proceedings of the 62nd Annual Meeting of the Associa- tion for Computational Linguistics . arXiv, 2024. 1, 2, 3, 6, 7, 12, 14

    work page 2024

  7. [7]

    Pangu-Agent: A Fine-Tunable Generalist Agent with Structured Reasoning

    Filippos Christianos, Georgios Papoudakis, Matthieu Zim- mer, Thomas Coste, Zhihao Wu, Jingxuan Chen, Khy- ati Khandelwal, James Doran, Xidong Feng, Jiacheng Liu, Zheng Xiong, Yicheng Luo, Jianye Hao, Kun Shao, Haitham Bou-Ammar, and Jun Wang. Pangu-Agent: A Fine-Tunable Generalist Agent with Structured Reasoning. arXiv preprint,

    work page

  8. [8]

    InstructBLIP: Towards General- purpose Vision-Language Models with Instruction Tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards General- purpose Vision-Language Models with Instruction Tuning. arXiv preprint, 2023. 3

    work page 2023

Show all 64 references
  1. [9]

    Mind2Web: Towards a Generalist Agent for the Web

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2Web: Towards a Generalist Agent for the Web. In Advances in Neural Information Processing Systems, 2023 . arXiv, 2023. 2, 6, 12

    2023

  2. [10]

    Iris: Breaking GUI Complexity with Adap- tive Focus and Self-Refining

    Zhiqi Ge, Juncheng Li, Xinglei Pang, Minghe Gao, Kaihang Pan, Wang Lin, Hao Fei, Wenqiao Zhang, Siliang Tang, and Yueting Zhuang. Iris: Breaking GUI Complexity with Adap- tive Focus and Self-Refining. arXiv preprint, 2024. 2, 6, 7, 14

    2024

  3. [11]

    Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents

    Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents. arXiv preprint, 2024. 1, 2, 4

    2024

  4. [12]

    CogAgent: A Visual Language Model for GUI Agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. CogAgent: A Visual Language Model for GUI Agents. In IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2023

  5. [13]

    iLLaV A: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models

    Lianyu Hu, Fanhua Shang, Liang Wan, and Wei Feng. iLLaV A: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models. arXiv preprint, 2024. 2, 4, 7

    2024

  6. [14]

    BLIV A: A Simple Multimodal LLM for Bet- ter Handling of Text-Rich Visual Questions

    Wenbo Hu, Yifan Xu, Yi Li, Weiyue Li, Zeyuan Chen, and Zhuowen Tu. BLIV A: A Simple Multimodal LLM for Bet- ter Handling of Text-Rich Visual Questions. arXiv preprint,

  7. [15]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint, 2024. 2, 14

    2024

  8. [16]

    Star: Learning diverse robot skill ab- stractions through rotation-augmented vector quantization

    Hao Li, Qi Lv, Rui Shao, Xiang Deng, Yinchuan Li, Jianye Hao, and Liqiang Nie. Star: Learning diverse robot skill ab- stractions through rotation-augmented vector quantization. International Conference on Machine Learning, 2025. 2

    2025

  9. [17]

    BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. In In- ternational Conference on Machine Learning . arXiv, 2023. 2, 3

    2023

  10. [18]

    On the Effects of Data Scale on Computer Control Agents

    Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the Effects of Data Scale on Computer Control Agents. In Advances in Neural Information Processing Systems . arXiv,

  11. [19]

    Lion-fs: Fast & slow video-language thinker as online video assistant

    Wei Li, Bing Hu, Rui Shao, Leyang Shen, and Liqiang Nie. Lion-fs: Fast & slow video-language thinker as online video assistant. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 3240–3251, 2025. 2

    2025

  12. [20]

    LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models. In Eu- ropean Conference on Computer Vision, 2024. arXiv, 2023. 2, 3, 4, 7

    2024

  13. [21]

    Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models. arXiv preprint, 2024. 3 9

    2024

  14. [22]

    Generative models in de- cision making: A survey

    Yinchuan Li, Xinyu Shao, Jianping Zhang, Haozhi Wang, Leo Maxime Brunswic, Kaiwen Zhou, Jiqian Dong, Kaiyang Guo, Xiu Li, Zhitang Chen, et al. Generative models in de- cision making: A survey. arXiv preprint, 2025. 1

    2025

  15. [23]

    Optimus-1: Hybrid multimodal memory empowered agents excel in long-horizon tasks

    Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dong- mei Jiang, and Liqiang Nie. Optimus-1: Hybrid multimodal memory empowered agents excel in long-horizon tasks. In Advances in Neural Information Processing Systems , pages 49881–49913, 2024. 2

    2024

  16. [24]

    Optimus-2: Multimodal minecraft agent with goal-observation-action conditioned policy

    Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, and Liqiang Nie. Optimus-2: Multimodal minecraft agent with goal-observation-action conditioned policy. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 9039–9049, 2025. 2

    2025

  17. [25]

    ShowUI: One Vision-Language-Action Model for GUI Visual Agent

    Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. ShowUI: One Vision-Language-Action Model for GUI Visual Agent. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 2, 4, 6, 7, 14

    2024

  18. [26]

    Visual Instruction Tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning, 2023. 2

    2023

  19. [27]

    TextMonkey: An OCR- Free Large Multimodal Model for Understanding Document

    Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. TextMonkey: An OCR- Free Large Multimodal Model for Understanding Document. arXiv preprint, 2024. 2, 3, 4, 7, 8

    2024

  20. [28]

    Thoughtful Interaction Design: A Design Perspective on Information Technology

    Jonas L ¨owgren. Thoughtful Interaction Design: A Design Perspective on Information Technology . MIT Press, Cam- bridge, Massachusetts, 2004. 1, 3, 4, 7

    2004

  21. [29]

    GUI Odyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices

    Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Box- uan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. GUI Odyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices. arXiv preprint, 2024. 2, 4, 6, 7, 12, 14

    2024

  22. [30]

    CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation

    Xinbei Ma, Zhuosheng Zhang, and Hai Zhao. CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation. In Findings of the Association for Compu- tational Linguistics. arXiv, 2024. 4

    2024

  23. [31]

    Ahmed, Puneet Mathur, Se- unghyun Yoon, Lina Yao, Branislav Kveton, Thien Huu Nguyen, Trung Bui, Tianyi Zhou, Ryan A

    Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, Xintong Li, Jing Shi, Hongjie Chen, Viet Dac Lai, Zhouhang Xie, Sungchul Kim, Ruiyi Zhang, Tong Yu, Mehrab Tanjim, Nesreen K. Ahmed, Puneet Mathur, Se- unghyun Yoon...

    2024

  24. [32]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023. 14

    2023

  25. [33]

    Large Language Models Can Self- Improve At Web Agent Tasks

    Ajay Patel, Markus Hofmarcher, Claudiu Leoveanu- Condrei, Marius-Constantin Dinu, Chris Callison-Burch, and Sepp Hochreiter. Large Language Models Can Self- Improve At Web Agent Tasks. arXiv preprint, 2024. 1

    2024

  26. [34]

    Agent Q: Advanced Reasoning and Learning for Au- tonomous AI Agents

    Pranav Putta, Edmund Mills, Naman Garg, Sumeet Mot- wani, Elan Markowitz, Chelsea Finn, and Rafael Rafailov. Agent Q: Advanced Reasoning and Learning for Au- tonomous AI Agents. arXiv preprint

  27. [35]

    WebRL: Train- ing LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning

    Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Xinyue Yang, Jiadai Sun, Yu Yang, Shuntian Yao, Tianjie Zhang, Wei Xu, Jie Tang, and Yuxiao Dong. WebRL: Train- ing LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning. arXiv preprint, 2024. 1, 2

    2024

  28. [36]

    UI-TARS: Pioneering Automated GUI Interaction with Native Agents

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shi- jue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng,...

    2025

  29. [37]

    Android in the Wild: A Large- Scale Dataset for Android Device Control

    Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Android in the Wild: A Large- Scale Dataset for Android Device Control. In Advances in Neural Information Processing Systems, 2023, 2023. 2, 3, 4, 6, 12

    2023

  30. [38]

    LLaV A-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. LLaV A-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models. arXiv preprint, 2024. 3

    2024

  31. [39]

    Multi-adversarial discriminative deep domain generalization for face presentation attack detection

    Rui Shao, Xiangyuan Lan, Jiawei Li, and Pong C Yuen. Multi-adversarial discriminative deep domain generalization for face presentation attack detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10023–10031, 2019. 3

    2019

  32. [40]

    Detecting and grounding multi-modal media manipulation

    Rui Shao, Tianxing Wu, and Ziwei Liu. Detecting and grounding multi-modal media manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 6904–6913, 2023

    2023

  33. [41]

    Detecting and grounding multi-modal media manip- ulation and beyond

    Rui Shao, Tianxing Wu, Jianlong Wu, Liqiang Nie, and Zi- wei Liu. Detecting and grounding multi-modal media manip- ulation and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 3

    2024

  34. [42]

    Mome: Mixture of multimodal experts for generalist multimodal large language models

    Leyang Shen, Gongwei Chen, Rui Shao, Weili Guan, and Liqiang Nie. Mome: Mixture of multimodal experts for generalist multimodal large language models. In Advances in Neural Information Processing Systems , pages 42048– 42070, 2024. 2

    2024

  35. [43]

    What does CLIP know about a red circle? Visual prompt engineering for VLMs, 2023

    Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does CLIP know about a red circle? Visual prompt engineering for VLMs, 2023. 3

    2023

  36. [44]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint,

  37. [45]

    Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration

    Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration. In Ad- vances in Neural Information Processing Systems . arXiv,

  38. [46]

    Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception

    Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception. arXiv preprint, 2024. 1, 2

    2024

  39. [47]

    Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin 10 Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Jun- yang Lin. Qwen2-VL: Enhancing Vision-Language Model...

  40. [48]

    GUI Agents with Foundation Models: A Comprehensive Survey

    Shuai Wang, Weiwen Liu, Jingxuan Chen, Weinan Gan, Xingshan Zeng, Shuai Yu, Xinlong Hao, Kun Shao, Yasheng Wang, and Ruiming Tang. GUI Agents with Foundation Models: A Comprehensive Survey. arXiv preprint, 2024. 2, 3

    2024

  41. [49]

    JARVIS-1: Open-World Multi-task Agents with Memory- Augmented Multimodal Language Models, 2023

    Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jin- bing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zi- long Zheng, Yaodong Yang, Xiaojian Ma, and Yitao Liang. JARVIS-1: Open-World Multi-task Agents with Memory- Augmented Multimodal Language Models, 2023. 2

    2023

  42. [50]

    Agent Workflow Memory

    Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent Workflow Memory. arXiv preprint, 2024. 2

    2024

  43. [51]

    AutoDroid: LLM-powered Task Automa- tion in Android

    Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. AutoDroid: LLM-powered Task Automa- tion in Android. In Proceedings of the 30th Annual Inter- national Conference on Mobile Computing and Networking....

    2024

  44. [52]

    DroidBot-GPT: GPT-powered UI Automation for Android

    Hao Wen, Hongming Wang, Jiaxuan Liu, and Yuanchun Li. DroidBot-GPT: GPT-powered UI Automation for Android. arXiv preprint, 2024. 2

    2024

  45. [53]

    Efficient Vision-Language Models by Sum- marizing Visual Tokens into Compact Registers

    Yuxin Wen, Qingqing Cao, Qichen Fu, Sachin Mehta, and Mahyar Najibi. Efficient Vision-Language Models by Sum- marizing Visual Tokens into Compact Registers. arXiv preprint, 2024. 2, 3, 4, 7, 8

    2024

  46. [54]

    OS-ATLAS: A Foun- dation Action Model for Generalist GUI Agents

    Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao. OS-ATLAS: A Foun- dation Action Model for Generalist GUI Agents. arXiv preprint, 2024. 1, 3, 4, 6, 7, 12

    2024

  47. [55]

    Gui-explorer: Au- tonomous exploration and mining of transition-aware knowl- edge for gui agent

    Bin Xie, Rui Shao, Gongwei Chen, Kaiwen Zhou, Yinchuan Li, Jie Liu, Min Zhang, and Liqiang Nie. Gui-explorer: Au- tonomous exploration and mining of transition-aware knowl- edge for gui agent. The 63rd Annual Meeting of the Associ- ation for Computational Linguistics, 2025. 1

    2025

  48. [56]

    Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction

    Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tian- bao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction. arXiv preprint, 2024. 1, 2

    2024

  49. [57]

    ReAct: Synergizing Reasoning and Acting in Language Models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing Reasoning and Acting in Language Models. https://arxiv.org/abs/2210.03629v3, 2022. 2

    2022 arXiv

  50. [58]

    mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration, 2023

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration, 2023. 4

    2023

  51. [59]

    V oCo-LLaMA: Towards Vision Compression with Large Language Models

    Xubing Ye, Yukang Gan, Xiaoke Huang, Yixiao Ge, Ying Shan, and Yansong Tang. V oCo-LLaMA: Towards Vision Compression with Large Language Models. arXiv preprint,

  52. [60]

    AppAgent: Mul- timodal Agents as Smartphone Users

    Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. AppAgent: Mul- timodal Agents as Smartphone Users. arXiv preprint, 2023. 1, 2

    2023

  53. [61]

    Falcon: Resolv- ing visual redundancy and fragmentation in high-resolution multimodal large language models via visual registers

    Renshan Zhang, Rui Shao, Gongwei Chen, Miao Zhang, Kaiwen Zhou, Weili Guan, and Liqiang Nie. Falcon: Resolv- ing visual redundancy and fragmentation in high-resolution multimodal large language models via visual registers. In Proceedings of the IEEE/CVF International Conferenc...

    2025

  54. [62]

    You Only Look at Screens: Multimodal Chain-of-Action Agents

    Zhuosheng Zhang and Aston Zhang. You Only Look at Screens: Multimodal Chain-of-Action Agents. In Find- ings of the Association for Computational Linguistics. arXiv,

  55. [63]

    MMInA: Benchmarking Multihop Multimodal Internet Agents, 2024

    Ziniu Zhang, Shulin Tian, Liangyu Chen, and Ziwei Liu. MMInA: Benchmarking Multihop Multimodal Internet Agents, 2024. 2

    2024

  56. [64]

    <image>Please generate the next move according to the ui screenshot, instruction and previous actions. Instruction: What’s on the menu at Domino’s?. Previous actions: Step0: {\

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. GPT-4V(ision) is a Generalist Web Agent, if Grounded. arXiv preprint, 2024. 1 11 A. GUI navigation tasks Android In The Wild (AITW) [37] consists of 30k in- structions and 715k operation trajectories in the context of s...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.