Chain-of-Memory: Enhancing GUI Agents for Cross-Application Navigation

Bin Chen; Chuanrui Hu; Teng Li; Xinzge Gao

REVIEW 4 major objections 5 minor 34 references

Chain-of-Memory: Enhancing GUI Agents for Cross-Application Navigation

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that explicit textual memory, kept as a chain of action-result summaries and long-term facts, improves cross-app GUI agents and lets a 7B model generate memory nearly as useful as a 72B model's after fine-tuning.

desk verdict A real memory mechanism and a large new dataset, but the headline '7B vs 72B' claim is not supported by the experiments as run, and the zero-shot gains come from oracle-provided memory rather than the agent's own generation. read the letter →

arxiv 2506.18158 v1 pith:U5BO4JG3 submitted 2025-06-22 cs.AI cs.CV

Xinzge Gao , Chuanrui Hu , Bin Chen , Teng Li This is my paper · ORCID

classification cs.AIcs.CV

keywords GUIagentmultimodallargelanguagemodelshort-termmemorylong-termcross-appnavigationscreen-actiondatasetmanagementtaskstate

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Chain-of-Memory (CoM), a method that lets GUI agents represent the state of a long cross-app task as explicit text: a short-term memory of recent action results and a long-term memory of key facts worth carrying across subtasks. The authors claim this beats relying on raw historical screenshots or action logs, because action logs are too sparse and screenshots too redundant to support accurate decisions. On the GUI Odyssey benchmark, adding CoM raises zero-shot action matching score (AMS) for Qwen2-VL-72B from 35.39 to 40.28 and success rate (SR) from 33.44 to 40.18, with similar gains after fine-tuning 7B models. They also contribute GUI Odyssey-CoM, 111,791 screen-action pairs annotated with memory information, and report that a 7B model fine-tuned on it generates memory nearly as useful in decisions as a 72B model's memory. Why care: cross-app tasks run long and information must survive hand-offs between apps, and CoM is a concrete recipe for storing that information in language an agent can reason over.

What carries the argument

The key machinery is a four-stage CoM loop: Information Perception detects the click target and compares the current screen with the previous one; STM Update appends a textual action result and drops the oldest entry when the list exceeds capacity $N=4$; LTM Storage extracts task-relevant ScreenInfo, evaluates with the task query and STM whether to save it, and updates per-app long-term entries; Action Decision feeds the current screen, STM, and LTM to the agent to pick the next operation. The other load-bearing object is GUI Odyssey-CoM, a re-annotation of 7,735 GUI Odyssey episodes into 111,791 screen-action pairs, each carrying action target, action result, ScreenInfo, STM, and LTM labels, generated by a strong multimodal model and selected by an evaluation model from three candidate responses.

What would settle it

Take an episode where a needed fact, such as a company name or price, appears only in a past screenshot; remove that fact from the generated ScreenInfo and STM, run the CoM agent, and compare against an agent that receives the full screenshot history. If the CoM agent's success drops at or below no-memory baselines on such deliberately degraded episodes, the assumption that text summaries preserve all task-critical information is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that explicitly writing down what changed on screen after each action, keeping a small rolling window of these action-result descriptions, and separately storing a distilled set of task-relevant facts makes a GUI agent substantially better at deciding its next click, type, home press, or completion signal. Short-term memory (STM) is a capped ordered list of recent action results; long-term memory (LTM) is built by evaluating, with the task query and STM in hand, whether extracted screen information is worth saving, with saved entries organized by app and the last entry replaced when the same app reappears. The paper's empirical claim is that STM mainly helps actions tied to the current state, such as HOME and STOP, while LTM mainly helps text input that must recall earlier search results, and combining both gives the best AMS, SR, and task-switching score. A second discovery is that a 7B model, once fine-tuned on GUI Odyssey-CoM, can generate its own CoM memory that performs in decisions almost as well as memory generated by a 72B model.

Load-bearing premise

The load-bearing premise is that the distilled text summaries (STM, LTM, and ScreenInfo) preserve all task-critical information from the screens, so the agent can safely work without ever seeing raw historical screenshots; if a summary omits or distorts a key detail, the agent has no way to recover it.

Editorial extensions

If this is right

Agents can avoid loading long screenshot histories into context and instead maintain compact text memory that transfers across apps.
Fine-tuning on GUI Odyssey-CoM can hand small models the ability to generate and use memory, lowering the compute threshold for deployable GUI agents.
STM and LTM play different roles: STM improves current-state actions like HOME and STOP, while LTM improves text-entry actions that must recall earlier search results.
Short-term memory length saturates near $N=4$, so increasing the rolling window beyond that yields little extra benefit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not test CoM beyond mobile phone navigation; a natural extension would be desktop or web GUI agents, where the same cross-app information hand-off problem occurs.
The per-app replacement rule for LTM is a design choice; a keyed or timestamped long-term store with retrieval would be a testable refinement when an app is revisited for a different subtask.
The ablation finding that raw ScreenInfo as input hurts performance suggests an explicit gating step could recover those losses by deciding when screen details are worth surfacing.
The paper reports results on episodes averaging 15.4 steps; an open stress test is whether CoM's gains persist on substantially longer tasks where memory accumulation and forgetting matter more.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about this paper. First, the core idea is sensible: give a GUI agent an explicit short-term and long-term memory, maintained as text summaries, instead of dumping raw screenshots or action logs into the context. Second, the experimental evidence as presented does not support the paper's most eye-catching claim—that a 7B model can match a 72B model at memory generation—because the zero-shot numbers are oracle-fed, not end-to-end, and the comparison mixes fine-tuned 7B with zero-shot 72B.

What is genuinely new: the CoM framework itself (STM with a fixed capacity, LTM with gated storage) and the GUI Odyssey-CoM dataset, which adds memory annotations to 7,735 episodes. That dataset could be a useful resource if released. The paper also deserves credit for reporting the ablation results honestly, including the fact that adding ScreenInfo hurts performance, and for stating the limitation that ScreenInfo introduces noise.

The soft spots are real but not fatal to the idea. The zero-shot evaluation feeds the model dataset-annotated STM and LTM, so those numbers are an upper bound on what the memory representation could provide, not a test of the full CoM pipeline. The only self-generated memory row (CoM* in the fine-tuning block) compares a 7B model against its own oracle-fed run, not against a 72B model under the same protocol, so the 'comparable to 72B' claim in the abstract is simply not tested. There are no error bars or significance tests; the runs are deterministic with greedy decoding, but without multiple seeds or any variance measure we cannot tell if the gains are stable. The full pipeline underperforms the ablated version, which suggests the ScreenInfo extraction stage needs work, not just mitigation. Finally, the same model (Qwen2-VL-72B) generated the annotations and is used as the zero-shot evaluator, which is a moderate circularity concern; the fine-tuning on 7B partially mitigates this.

Who is this for? Researchers working on memory-augmented GUI agents or mobile device control. The dataset, once released, could be valuable for training smaller models. The paper deserves serious peer review because the research direction is relevant and the dataset is substantial, but the authors need to provide an end-to-end evaluation (self-generated memory) for both zero-shot and fine-tuned settings, a fair head-to-head 7B vs 72B comparison, and some measure of variability. If they do that, the contribution could be solid. As it stands, the idea is promising but the evidence is incomplete.

My recommendation: send it to review, but with major revision expectations.

Referee Report

4 major / 5 minor

Summary. The paper proposes Chain-of-Memory (CoM), a method that augments GUI agents with explicit short-term memory (STM) and long-term memory (LTM) in the form of text summaries, together with a ScreenInfo extraction stage. The authors introduce GUI Odyssey-CoM, a dataset of 111,791 screen-action pairs annotated with memory information, and report zero-shot and fine-tuning experiments on Qwen2-VL and other models. The central claims are that CoM significantly improves cross-application GUI navigation performance and that fine-tuning on GUI Odyssey-CoM gives 7B models memory-management capabilities comparable to 72B models. The paper includes ablations of STM, LTM, and ScreenInfo components and an analysis of the STM capacity hyperparameter.

Significance. If the claims are validated, explicit text-based memory for GUI agents is a useful and timely direction, and the GUI Odyssey-CoM dataset could be a valuable resource for training smaller models on memory generation and use. The paper also provides a clear ablation design that separates the contributions of STM and LTM, and it ships a substantial annotation effort. However, the current evidence for the headline claims is weakened by the use of oracle-provided memory in the zero-shot experiments, the absence of a true end-to-end self-generated evaluation, and the fact that the complete CoM pipeline (with ScreenInfo) performs worse than its own STM+LTM ablation. The paper is therefore of moderate-to-high potential significance, but the central empirical claims need stronger support before publication.

major comments (4)

[§5.2, Table 2] The zero-shot '+CoM' rows in Table 2 are produced by feeding the model the dataset-provided STM and LTM annotations, not by the model executing Algorithm 1's memory-generation and update steps. These rows therefore measure an upper bound of the CoM pipeline, not its actual end-to-end behavior. The only fully self-generated row, '+CoM*' in the fine-tuning block, is for a 7B model and is not compared against a 72B model under the same self-generated protocol. Consequently, the abstract's claim that 'GUI Odyssey-CoM enables 7B models to achieve memory management capabilities comparable to 72B models' is not supported by the table, because the 7B self-generated result is not compared with a 72B self-generated result. The authors should add a self-generated (end-to-end) evaluation for the full pipeline, at least for Qwen2-VL-72B and ideally for the fine-tuned models, and restrict the comparability claim to matched protocols.
[§5.3, Table 3] Table 3 shows that adding ScreenInfo to STM+LTM decreases AMS in both zero-shot (40.28 to 38.72) and fine-tuned (84.4 to 82.05) settings, and similarly decreases SR (40.18 to 37.22 and 84.25 to 82.11). Since ScreenInfo extraction is an intrinsic stage of the CoM pipeline (Algorithm 1, lines 14-26), the complete CoM pipeline underperforms its own STM+LTM ablation. The explanation in §5.3 that extraneous screen information impedes decision-making is post hoc and not quantified. This result undermines the claim that the full CoM method, as described, is beneficial; at minimum, the paper needs to report the reliability of the ScreenInfo extraction stage (e.g., human or automatic quality scores) and to clarify whether the proposed method is STM+LTM only or the full Algorithm 1. If ScreenInfo is systematically harmful, the architecture should be revised or the claims scoped accordingly.
[§5.1-§5.3 and Figure 4] No error bars, confidence intervals, or significance tests are reported for any of the experimental results. Differences of 1-5 AMS points, such as the fine-tuned Qwen-VL-7b +CoM improvement over its baseline (80.2 to 81.62) or the zero-shot ScreenInfo ablation drop (35.39 to 32.55), may be within run-to-run noise. The STM capacity N is also a free parameter, and Figure 4 reports a single trajectory without variance. The authors should provide multiple seeds or bootstrap confidence intervals for the main tables and for Figure 4, and state how many evaluation runs each number represents.
[§4.2-§4.3, §5.1] The zero-shot evaluation uses Qwen2-VL-72B, which is the same model family and size used to generate the CoM annotations in §4.3. The zero-shot gains could therefore partly reflect the model's ability to consume its own output format rather than a general benefit of the CoM representation. The fine-tuning experiments on 7B models provide some independent grounding, but they do not test whether the memory annotations generalize to a different large MLLM in the zero-shot setting. Adding a zero-shot evaluation with a different model family (e.g., InternVL2.5-78B, which is already used in Table 2) using the same oracle annotations would address this concern and would also test the generalizability of the dataset annotations.

minor comments (5)

[§3.1, Eq. (1)] Equation (1) writes 'Mt+1 = Mt ∪ ri+1', but the subscript on the action result should be 'rt+1' to match the surrounding notation; as written, 'ri+1' is undefined in that context.
[Table 3] The row labels 'ZS' and 'FT' in Table 3 are never expanded in the table caption; they should be defined explicitly as 'zero-shot' and 'fine-tuned' for readability.
[Table 2] The column labeled 'Atomic' is not explained in the caption or in §5.1; the paper should define what 'Atomic' refers to and how it is computed.
[Figure 4] The horizontal axis of Figure 4 is not labeled in the main text; it should state explicitly that the x-axis is the STM capacity N, and the figure should include axis labels and, if possible, error bars.
[§5.2] The sentence 'the 7B model's self-generated memory information proved nearly as effective in decision-making as that generated by the 72B model' is not directly supported by the table, since the comparison is between a fine-tuned self-generated 7B run and a zero-shot oracle-fed 72B run; this should be reworded or qualified in line with the matched-protocol comparison requested above.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: CoM's reported gains are not equivalent to its inputs by construction; the zero-shot oracle-memory protocol is an upper bound, while the fine-tuned self-generated +CoM* row provides an independent end-to-end test.

full rationale

The claimed derivation chain is not circular. STM and LTM are defined as textual summaries of action results and distilled screen information (Equations 1-4), and the evaluations use GUI Odyssey's external ground-truth actions as the target labels, so no reported quantity reduces to its input by construction. The zero-shot '+CoM' condition in Table 2 does inject dataset-annotated STM/LTM rather than the model's own Algorithm 1 output, which is a legitimate validity caveat (it is an oracle-memory upper bound, not a full end-to-end pipeline test), but it is not a fitted parameter renamed as a prediction: the same annotation model's ScreenInfo output systematically hurts performance when added (Table 3), arguing against a generic self-preference artifact. The '+CoM*' row in Table 2 is a fully self-generated, end-to-end evaluation on the externally defined GUI Odyssey benchmark, and the paper's claim that the 7B model's self-generated memory is nearly as effective as the 72B-generated memory (Section 5.2) rests on that row. There are no load-bearing self-citations and no imported uniqueness theorems. Section 5.3's own admission that ScreenInfo reduces performance is a limitation of the intermediate extraction stage, not evidence of circularity. The central derivation is therefore self-contained against an external benchmark.

Assumptions & free parameters 1 free parameters · 3 assumptions · 2 invented entities

The main free parameter is the STM capacity N (set to 4). The key assumptions are that text summaries are lossless enough for decision-making, that automated annotation scoring is reliable, and that annotations from Qwen2-VL-72B transfer to other models. The conceptual STM/LTM/ScreenInfo components are introduced without independent evidence beyond the paper's own experiments.

free parameters (1)

Short-term memory capacity N = 4
Maximum number of action-result pairs retained in the STM (Section 3.1). Chosen by hand; the ablation in Figure 4 shows diminishing returns beyond 4.

assumptions (3)

domain assumption Textual summaries of action results capture task state as well as raw screenshots and action coordinates
The CoM pipeline (Algorithm 1) feeds only STM/LTM text to the agent; raw screenshots are not used as history, so any information lost in summarization is unrecoverable (Sections 3.1, 3.3).
domain assumption Automatically scored best-of-3 annotations are of sufficient quality for training
The dataset pipeline (Section 4.3) relies on an unspecified evaluation model to choose among three generated responses; only 20 episodes received manual verification (Section 4.2).
ad hoc to paper Qwen2-VL-72B produces memory annotations that generalize to other models
Generation model selection was based on 20 manually evaluated episodes (Section 4.2); the zero-shot evaluation uses the same model, so the annotations may be idiosyncratic to it.

invented entities (2)

Short-Term Memory (STM) and Long-Term Memory (LTM) modules
purpose: Store recent action-result text and distilled cross-task information, respectively, to condition the agent's next action
These are conceptual components of the proposed CoM framework; they are validated only through the paper's own experiments, not via an external benchmark.
ScreenInfo representation
purpose: Intermediate extracted text that feeds the long-term memory module
The ablation (Table 3) finds that including ScreenInfo reduces performance (zero-shot AMS 38.72 vs 40.28 for STM+LTM; fine-tuned 82.05 vs 84.4), so its value as an independent component is not established.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Chain-of-Memory: Enhancing GUI Agents for Cross-Application Navigation." pith.science (2026). https://pith.science/paper/U5BO4JG3

@misc{pith2026250618158,
  author       = {Pith},
  title        = {Pith review of: Chain-of-Memory: Enhancing GUI Agents for Cross-Application Navigation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U5BO4JG3}},
  note         = {Machine review of arXiv:2506.18158}
}

read the original abstract

Multimodal large language models (MLLMs) are attracting growing attention in the development of Graphical User Interface (GUI) agents. Existing approaches often rely on historical screenshots or actions to implicitly represent the task state. This reliance poses challenges for GUI agents in accurately understanding task states and underscores the absence of effective mechanisms to store critical information in complex and lengthy cross-app tasks. To address these challenges, we propose Chain-of-Memory (CoM), a novel approach for explicitly modeling short-term and long-term memory in GUI agents. CoM achieves this by capturing action descriptions, integrating task-relevant screen information, and maintaining a dedicated memory module to store and manage this information. By leveraging explicit memory representations, CoM enables GUI agents to better understand task states and retain critical historical information persistently. To equip GUI agents with memory management capabilities and evaluate the effectiveness of CoM, we developed the GUI Odyssey-CoM, a dataset comprising 111k screen-action pairs annotated with Chain-of-Memory. Experimental results demonstrate that CoM significantly improves GUI agents' performance in cross-application tasks. Additionally, GUI Odyssey-CoM enables 7B models to achieve memory management capabilities comparable to 72B models. The dataset and code will be open-sourced.

Figures

Figures reproduced from arXiv: 2506.18158 by the authors.

**Figure 2.** The architecture and workflow of Chain-of-Memory. CoM comprises four key stages: Information Perception, STM Updating, LTM Storage, and Action Decision. The agent perceives information from previous and current screens, updates the STM with recent action results, stores key information in the LTM, and makes the next action decision based on the current screen, STM, and LTM. al., 2024c; Chen et al., 2024c], MLLMs are… view at source ↗

**Figure 3.** GUI Odyssey-CoM collection pipeline. 4 GUI Odyssey-CoM 4.1 Data Collection Existing datasets lack CoM annotations, which hinders smaller models from acquiring memory management capabilities. To address this, we constructed a cross-app dataset annotated with CoM and designed a data generation pipeline to ensure high data quality.We constructed our dataset based on GUI Odyssey, the largest publicly available cross-ap… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Influence of short-term memory length. with CoM, likely due to its smaller scale, which limits its ability to directly process CoM information. However, after fine-tuning, the 7B model exhibited excellent performance, indicating that the GUI Odyssey-CoM dataset provide…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 12 canonical work pages

[1]

Working memory: looking back and looking forward

[Baddeley, 2003] Alan Baddeley. Working memory: looking back and looking forward. Nature reviews neuroscience, 4(10):829–839,

work page 2003
[5]

Expanding performance boundaries of open-source multimodal mod- els with model, data, and test-time scaling

[Chen et al., 2024b] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal mod- els with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271,

arXiv
[6]

Seeclick: Harnessing gui grounding for advanced visual gui agents

[Cheng et al., 2024] Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiy- ong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935,

arXiv 2024
[7]

Mind2web: Towards a generalist agent for the web

[Deng et al., 2024] Xiang Deng, Yu Gu, Boyuan Zheng, Shi- jie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Ad- vances in Neural Information Processing Systems , 36,

work page 2024
[8]

Human-level play in the game of diplomacy by combining language models with strategic reasoning

[(FAIR)† et al., 2022] Meta Fundamental AI Research Diplomacy Team (FAIR)†, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 378(6624):1067–1074,

work page 2022
[11]

Mapping natural language instructions to mobile ui action sequences

[Li et al., 2020] Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile ui action sequences. arXiv preprint arXiv:2005.03776,

arXiv 2020
[13]

Ap- pagent v2: Advanced agent for flexible mobile interac- tions

[Li et al., 2024b] Yanda Li, Chi Zhang, Wanqi Yang, Bin Fu, Pei Cheng, Xin Chen, Ling Chen, and Yunchao Wei. Ap- pagent v2: Advanced agent for flexible mobile interac- tions. arXiv preprint arXiv:2408.11824,

arXiv
[14]

Awq: Activation-aware weight quantization for on-device llm compression and acceleration

[Lin et al., 2024] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87–100,

work page 2024

Show all 34 references

[15]

An empirical study of code search in intelligent coding assistant: Per- ceptions, expectations, and directions

[Liu et al., 2024] Chao Liu, Xindong Zhang, Hongyu Zhang, Zhiyuan Wan, Zhan Huang, and Meng Yan. An empirical study of code search in intelligent coding assistant: Per- ceptions, expectations, and directions. In Companion Pro- ceedings of the 32nd ACM International Conference ...

2024
[16]

Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices

[Lu et al., 2024] Quanfeng Lu, Wenqi Shao, Zitao Liu, Fan- qing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. arXiv preprint arXiv:2406.08451,

2024 arXiv
[17]

Llama 3.2: Revolutionizing edge ai and vision with open, customizable models,

[Meta, 2024] Meta. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models,

2024
[18]

Communicative agents for software devel- opment

[Qian et al., 2023] Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software devel- opment. arXiv preprint arXiv:2307.07924, 6,

2023 arXiv
[19]

Learning transferable visual models from nat- ural language supervision

[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from nat- ural language supervision. In International confere...

2021
[20]

Android in the wild: A large-scale dataset for android device con- trol

[Rawles et al., 2023] Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Android in the wild: A large-scale dataset for android device con- trol. arXiv preprint arXiv:2307.10088,

2023 arXiv
[21]

Meta-gui: Towards multi-modal conversational agents on mobile gui

[Sun et al., 2022] Liangtai Sun, Xingyu Chen, Lu Chen, Tianle Dai, Zichen Zhu, and Kai Yu. Meta-gui: Towards multi-modal conversational agents on mobile gui. arXiv preprint arXiv:2205.11029,

2022 arXiv
[22]

Chameleon: Mixed- modal early-fusion foundation models

[Team, 2024] Chameleon Team. Chameleon: Mixed- modal early-fusion foundation models. arXiv preprint arXiv:2405.09818,

2024 arXiv
[23]

Ugif: Ui grounded instruc- tion following

[Venkatesh et al., 2022] Sagar Gubbi Venkatesh, Partha Talukdar, and Srini Narayanan. Ugif: Ui grounded instruc- tion following. arXiv preprint arXiv:2211.07615,

2022 arXiv
[24]

Omniparser: A unified frame- work for text spotting key information extraction and ta- ble recognition

[Wan et al., 2024] Jianqiang Wan, Sibo Song, Wenwen Yu, Yuliang Liu, Wenqing Cheng, Fei Huang, Xiang Bai, Cong Yao, and Zhibo Yang. Omniparser: A unified frame- work for text spotting key information extraction and ta- ble recognition. In Proceedings of the IEEE/CVF Confer- en...

2024
[25]

Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi- agent collaboration

[Wang et al., 2024a] Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi- agent collaboration. arXiv preprint arXiv:2406.01014 ,

arXiv
[26]

Mobile-agent: Autonomous multi-modal mo- bile device agent with visual perception

[Wang et al., 2024b] Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Ji- tao Sang. Mobile-agent: Autonomous multi-modal mo- bile device agent with visual perception. arXiv preprint arXiv:2401.16158,

arXiv
[27]

Qwen2-vl: Enhanc- ing vision-language model’s perception of the world at any resolution

[Wang et al., 2024c] Peng Wang, Shuai Bai, Sinan Tan, Shi- jie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhanc- ing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191,

arXiv
[28]

En- hancing the reasoning ability of multimodal large language models via mixed preference optimization

[Wang et al., 2024d] Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, and Jifeng Dai. En- hancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arXi...

arXiv
[29]

Chain-of-thought prompting elicits reasoning in large language models

[Wei et al., 2022] Jason Wei, Xuezhi Wang, Dale Schuur- mans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837,

2022
[30]

Ferret-ui: Grounded mobile ui under- standing with multimodal llms

[You et al., 2025] Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. Ferret-ui: Grounded mobile ui under- standing with multimodal llms. In European Conference on Computer Vision, pages 240–255. Springer,

2025
[32]

Llama-adapter: Efficient fine-tuning of language models with zero-init attention

[Zhang et al., 2023b] Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199,

arXiv
[33]

UFO: A UI-Focused Agent for Windows OS Interaction

[Zhang et al., 2024a] Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. UFO: A UI-Focused Agent for Windows OS Interaction. arXiv preprint arXiv:2402.07939,

arXiv
[34]

Android in the zoo: Chain-of-action-thought for gui agents

[Zhang et al., 2024b] Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. Android in the zoo: Chain-of-action-thought for gui agents. arXiv preprint arXiv:2403.02713, 2024

2024 arXiv
[2003]

Screenai: A vision-language model for ui and infographics understanding

[Baechler et al., 2024] Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Et- ter, Victor C ˘arbune, Jason Lin, Jindong Chen, and Ab- hanshu Sharma. Screenai: A vision-language model for ui and infographics understanding. arXiv preprint arXiv...

2024 arXiv
[2020]

On the effects of data scale on computer control agents

[Li et al., 2024a] Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on computer control agents. arXiv preprint arXiv:2406.03679,

arXiv
[2021]

Gui- world: A dataset for gui-oriented multimodal llm-based agents

[Chen et al., 2024a] Dongping Chen, Yue Huang, Siyuan Wu, Jingyu Tang, Liuyi Chen, Yilin Bai, Zhigang He, Chenlong Wang, Huichi Zhou, Yiqiang Li, et al. Gui- world: A dataset for gui-oriented multimodal llm-based agents. arXiv preprint arXiv:2406.10819,

arXiv
[2022]

Grounding language models to images for multimodal inputs and outputs

[Koh et al., 2023] Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal inputs and outputs. In International Confer- ence on Machine Learning , pages 17283–17300. PMLR,

2023
[2023]

Large language models are zero-shot reasoners

[Kojima et al., 2022] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems , 35:22199–22213,

2022
[2024]

Mobile app tasks with iterative feedback (mo- tif): Addressing task feasibility in interactive visual envi- ronments

[Burns et al., 2021] Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A Plummer. Mobile app tasks with iterative feedback (mo- tif): Addressing task feasibility in interactive visual envi- ronments. arXiv preprint arXiv:2104.08560,

2021 arXiv
[2025]

Appagent: Multimodal agents as smartphone users

[Zhang et al., 2023a] Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. arXiv preprint arXiv:2312.13771,

arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Working memory: looking back and looking forward

[Baddeley, 2003] Alan Baddeley. Working memory: looking back and looking forward. Nature reviews neuroscience, 4(10):829–839,

work page 2003

[2] [5]

Expanding performance boundaries of open-source multimodal mod- els with model, data, and test-time scaling

[Chen et al., 2024b] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal mod- els with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271,

arXiv

[3] [6]

Seeclick: Harnessing gui grounding for advanced visual gui agents

[Cheng et al., 2024] Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiy- ong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935,

arXiv 2024

[4] [7]

Mind2web: Towards a generalist agent for the web

[Deng et al., 2024] Xiang Deng, Yu Gu, Boyuan Zheng, Shi- jie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Ad- vances in Neural Information Processing Systems , 36,

work page 2024

[5] [8]

Human-level play in the game of diplomacy by combining language models with strategic reasoning

[(FAIR)† et al., 2022] Meta Fundamental AI Research Diplomacy Team (FAIR)†, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 378(6624):1067–1074,

work page 2022

[6] [11]

Mapping natural language instructions to mobile ui action sequences

[Li et al., 2020] Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile ui action sequences. arXiv preprint arXiv:2005.03776,

arXiv 2020

[7] [13]

Ap- pagent v2: Advanced agent for flexible mobile interac- tions

[Li et al., 2024b] Yanda Li, Chi Zhang, Wanqi Yang, Bin Fu, Pei Cheng, Xin Chen, Ling Chen, and Yunchao Wei. Ap- pagent v2: Advanced agent for flexible mobile interac- tions. arXiv preprint arXiv:2408.11824,

arXiv

[8] [14]

Awq: Activation-aware weight quantization for on-device llm compression and acceleration

[Lin et al., 2024] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87–100,

work page 2024