Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

MobileA3gent: Training Mobile GUI Agents Using Decentralized Self-Sourced Data from Diverse Users

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that mobile GUI agents can be trained from decentralized, auto-annotated user trajectories at about 1% of the cost of human-annotated centralized data, without sacrificing performance or privacy.

desk verdict Solid annotation pipeline and careful benchmark experiments, but privacy and real-world deployment claims outrun the evidence; worth a serious referee. read the letter →

arxiv 2502.02982 v2 pith:DK45RST2 submitted 2025-02-05 cs.AI

Wenhao Wang , Mengying Yuan , Zijie Yu , Guangyi Liu , Rui Ye , Tian Jin , Siheng Chen , Yanfeng Wang This is my paper · ORCID
classification cs.AI
keywords mobileGUIagentsfederatedlearningvision-languagemodelsautomaticdataannotationusertrajectoriesnon-IIDinstructiongenerationbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that the expensive, centralized, human-annotated datasets used to train mobile GUI agents can be replaced by a decentralized pipeline fed by ordinary users' daily phone interactions. It proposes a framework that automatically reconstructs what a user was trying to do from screenshots and touch actions, then trains the agent across many users without moving raw data off their devices. The central claim is that this pipeline produces instruction data of human-comparable quality, and that federated training with a custom aggregation rule can turn that data into agents matching or beating central human-annotation baselines. If true, it would make mobile agent training scale with the billions of active phone users at a fraction of current cost, and it would keep sensitive interaction data local. The experiments reported support this claim on three offline benchmarks and one online benchmark, at roughly 1% of the overall cost of the human-annotated baseline.

What carries the argument

The load-bearing mechanism is a two-stage annotation-and-training loop. Auto-Annotation first converts coordinate actions into readable action descriptions using the accessibility tree, then a local Descriptor VLM writes an atomic low-level instruction for each step, and a local Summarizer VLM, fed the concatenated screenshot strip plus all low-level instructions, writes the episode's high-level instruction. FedVLM-A then trains a LoRA-adapted VLM on each client and aggregates only the model updates; its adapted aggregation redefines each client's sample count as $n_k^* = \lambda n^{\mathrm{epi}}_k + n_k$, with $\lambda$ chosen near the average episode length, and weights the global model update by $\omega_k = n_k^* / \sum_j n_j^*$. This balances episode-level and step-level heterogeneity, degenerating to ordinary sample-count aggregation when $\lambda = 0$.

What would settle it

Run the MobileA3gent pipeline on real user devices or a high-fidelity on-device logging study, then compare the trained agent's step success rate against the paper's simulated results on AndroidControl and AndroidWorld; the central claim would be falsified if real-world trajectories yield materially worse instruction quality or agent performance than the crowdsourced partitions, or if on-device annotation with small local VLMs cannot reproduce the reported ~99% cost reduction without a large drop in quality.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that users' routine phone usage can be harvested as a training signal: converting raw actions into textual descriptions, generating per-step low-level instructions, and then summarizing those into a high-level instruction yields annotations nearly 80% similar to human-written ground truth. When a VLM-based mobile agent is trained on these auto-annotated episodes, it matches or exceeds the performance of the same model trained on human-annotated episodes, and with 1% of the overall cost the full MobileA3gent pipeline surpasses the centralized human-annotated result on AndroidControl. The second half of the discovery is that federated training of such agents benefits from an aggregation weight that counts both episodes and steps per client, rather than only raw samples, because mobile interaction data has a two-level structure with homogeneity inside each task and heterogeneity across tasks.

Load-bearing premise

The real-world promise depends on users' natural phone usage being capturable and locally annotatable at the same quality as the crowdsourced benchmark episodes used in the experiments, since the paper states it cannot yet run experiments on actual user phones.

Editorial extensions

If this is right

  • A mobile GUI agent can be trained from automatically annotated user interaction logs, removing the human-annotator bottleneck that currently limits dataset scale and diversity.
  • At 1% of the overall cost, the federated auto-annotated pipeline matches or surpasses the centralized human-annotated baseline, so the cost-quality trade-off of GUI agent training shifts dramatically.
  • Because performance improves as more clients contribute, aggregating daily usage from many users offers a scalable path toward ever-larger training corpora without collecting raw screenshots centrally.
  • Adapted aggregation that weights episodes and steps outperforms standard federated aggregation by at least 5% relative under non-IID splits, which changes how federated VLM training should be weighted for structured trajectory data.
  • The online AndroidWorld results indicate that agents trained only on auto-annotated AndroidControl episodes can complete previously unseen tasks, suggesting the approach yields generalizable behaviors, not just benchmark overfitting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If real-device capture matches the simulations, the same pipeline could support continual personalization, with agents improving on the habits and task distributions of their own users while still learning from the global population.
  • The episode-plus-step aggregation idea likely transfers to other hierarchically structured federated datasets, such as multi-turn conversations or logged sessions, where raw sample counts hide task-level skew.
  • The two-stage Descriptor/Summarizer design may be reusable outside mobile GUIs as a general method for turning interaction traces into natural-language task descriptions, for example in web automation or desktop assistants.
  • A deployment would still face a privacy gap the paper acknowledges only in passing: locally generated instructions and uploaded LoRA updates can encode sensitive user intent, so differential privacy or other protections would be needed before real adoption.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MobileA3gent, a two-component framework for training mobile GUI agents from decentralized user trajectories. Auto-Annotation converts raw screenshots and actions into low-level step descriptions and a high-level task instruction using a locally deployed VLM, while FedVLM-A trains VLM-based agents via federated learning with an adapted aggregation weight that combines episode counts and step counts. Experiments on AndroidControl, AitW, GUI Odyssey, and the online AndroidWorld benchmark compare against human annotation, synthetic data, and standard FL baselines, reporting that auto-annotated data are comparable to human-annotated data at much lower cost and that the adapted aggregation improves over FedAvg-family methods under non-IID splits. The paper also discusses privacy and limitations, including the absence of real-device experiments.

Significance. If the claims hold, MobileA3gent would be a practically valuable framework: it automates instruction annotation from interaction trajectories, makes a simple modification to FedAvg that can be dropped into existing federated VLM training stacks, and provides an unusually broad comparison across multiple benchmarks, annotation models, training models, and an online environment. Strengths include the equal-size comparison in Table 4, the model ablation in Section 4.5, the out-of-domain generalization splits in Appendix D.4, and the public code release. However, the headline claims about cost superiority and privacy go beyond what is demonstrated; the current evidence supports a more modest claim about simulated federated training with auto-annotated crowdsourced data.

major comments (4)
  1. [§3.3, Table 1, §E.2] The privacy component of the central claim is not substantiated. The protocol transmits LoRA adapters rather than raw data, but no differential-privacy guarantee, leakage bound, or attack-based evaluation (membership inference, model inversion, or memorization extraction) is provided. Section 3.3's Privacy Analysis addresses only eavesdropping, abuse, and peer exposure, while §E.2 concedes that VLMs memorize training data and that DP remains underexplored in this setting. Consequently, the abstract's and Section 1's 'rigorous user privacy protection' and Table 1's 'strongest protection' overstate what is demonstrated; either add a concrete privacy analysis with a threat model and experiments, or revise these claims to state that local data retention mitigates the three specified sharing risks.
  2. [Table 2 caption, Table 3, Table 4] The claim that 'with 1% overall cost, MobileA3gent even surpasses the centralized human-annotated data' is not supported by the equal-size experiment. Table 3 compares Human-Annotation on 1,000 samples with Auto-Annotation on 5,000 samples for AndroidControl and 3,000 for GUI Odyssey; the equal-size comparison in Table 4 shows Auto-Annotation is comparable but often numerically lower than Human-Annotation on AndroidControl-High at 5,000 samples (Type 77.49 vs 79.14, Ground 62.67 vs 66.56, SR 58.12 vs 61.70). The superiority claim should be qualified as holding at larger data volumes, or reframed as comparable performance at roughly 1% annotation cost with further improvements as data scale increases.
  3. [§4.4, Eq. (4), Table 12] The FedVLM-A improvement claim is only weakly tied to the proposed mechanism because the aggregation weight n*_k = λ·n_epi_k + n_k is defined in terms of the same episode and step counts used to create the skew, and λ is an empirically tuned hyperparameter taking values {3,5,7,9} in Table 12. The paper reports neither a λ-sensitivity analysis nor per-split, per-λ results, and the statement that the adapted aggregation 'inherits its convergence property' is asserted without proof or a supporting reference. Please report the λ used for each experimental setting, include a sensitivity sweep, and either provide a convergence argument for the adaptive weights or cite a result that covers them.
  4. [§B.2, §4.1] The real-world framing is not validated by the experiments. All 'users' are simulated by partitioning crowdsourced benchmarks (AndroidControl, AitW, GUI Odyssey), and annotation is run on RTX 4090 GPUs rather than on mobile devices; §B.2 explicitly states that experiments on actual user phones were not conducted. The abstract's claims about users' routine phone usage, negligible per-client cost, and practical deployment therefore rest on the unvalidated assumption that on-device trajectory capture, accessibility-tree-based action conversion, and local annotation can be performed with comparable quality. A feasibility study, or a clearly stated scoping of the contributions to the simulated setting, is needed before the deployment-oriented claims can be accepted.
minor comments (5)
  1. [§3.3] The phrase 'without comprising privacy' should read 'without compromising privacy'.
  2. [Figure 1] The label 'Executation LaborCost' appears to be a typo for 'Execution Labor Cost'.
  3. [Table 2, Table 8] The cost accounting is confusing: Table 8 reports per-1,000-sample costs while Table 2 reports an aggregate 'Anno. Cost' of 152.92 ¢; please state explicitly which dataset size and backend each aggregate cost corresponds to.
  4. [Appendix F.4] The Hugging Face URLs for Qwen2-VL-2B-Instruct and Qwen2-VL-7B-Instruct are identical; the 7B link should be corrected.
  5. [§4.1, §D.5] The annotation-cost formula is referenced as 'Equation 4.1' in Section D.5 but is not numbered in Section 4.1; please number the equation or fix the cross-reference.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the framework's claims are tested against external benchmarks, and the FedVLM-A weighting is a design choice evaluated on held-out task success rather than a fitted prediction.

full rationale

MobileA3gent's two central components are not circular by the paper's own equations. Auto-Annotation generates instructions from trajectories via the Descriptor and Summarizer prompts (Eqs. 1-2), and the generated instructions are then used to train agents that are evaluated on held-out benchmark test splits (AndroidControl, AitW, GUI Odyssey, AndroidWorld) against human-annotated ground truth; there is no identity between the generated annotation and the evaluation target. The data-quality metric is similarity to gold instructions, which is a stated proxy, not the claimed downstream result. FedVLM-A's adapted weight n*_k = λ·n_epi_k + n_k (Eq. 4) is a proposed aggregation scheme; the reported SR/Type/Ground numbers are measured on fixed test sets, so the improvement over FedAvg is an empirical outcome, not a quantity forced by the definition of the weight. λ is a scale-alignment hyperparameter (Table 12 lists 3,5,7,9), so tuning it is a standard model-selection caveat rather than a fitted input renamed as a prediction. Citations to the authors' OpenFedLLM [21] and ms-swift [20] are implementation/reproducibility references and do not carry a load-bearing uniqueness or existence claim. The paper's own limitations weaken the real-world and privacy assertions: B.2 states that experiments on actual user mobile phones were not conducted, and E.2 concedes that VLMs can memorize training data and that differential privacy remains underexplored in this setting. Those are evidence gaps affecting external validity, not circular reductions of the derivation. Accordingly, no step qualifies as circular under the required standard, and the appropriate score is 0.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central empirical claims rest on several domain assumptions about real-world data collection and on one fitted hyperparameter (λ). No new physical entities are introduced; the "two-level distribution" is a conceptual formulation, not an invented entity. The framework's real-world validity depends on assumptions that are acknowledged but not tested.

free parameters (1)
  • λ (episode-step balancing weight) = 3, 5, 7, 9 (per scenario)
    In Eq. (4), n*_k = λ·n_epi_k + n_k. The paper says λ is empirically set around the average step length, but Table 12 lists distinct values for different scenarios; the FedVLM-A results depend on this choice.
assumptions (5)
  • domain assumption Users' daily phone usage produces task-oriented trajectories, and screenshots/actions are collectible without user intervention.
    Assumed in Section 2.2 and 3.2; never validated with real user data (B.2).
  • domain assumption A locally deployed VLM can infer user intention from screenshots and converted actions accurately enough to train agents.
    This is the core premise of Auto-Annotation, evaluated only on crowdsourced benchmark trajectories.
  • domain assumption Benchmark trajectories (AndroidControl, AitW, GUI Odyssey) are representative of diverse real-world mobile usage.
    All experiments use these crowdsourced datasets as proxies for user self-sourced data; no real-device validation exists.
  • ad hoc to paper The adapted aggregation inherits FedAvg convergence properties.
    Stated in Section 3.3 without proof; the weighting n*_k alters the optimization objective, so convergence is non-trivial.
  • domain assumption Federated learning with LoRA adapters provides sufficient privacy protection.
    Privacy analysis in Section 3.3 is qualitative; E.2 acknowledges memorization risks in VLMs and that DP remains future work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MobileA3gent: Training Mobile GUI Agents Using Decentralized Self-Sourced Data from Diverse Users." pith.science (2026). https://pith.science/paper/DK45RST2

@misc{pith2026250202982,
  author       = {Pith},
  title        = {Pith review of: MobileA3gent: Training Mobile GUI Agents Using Decentralized Self-Sourced Data from Diverse Users},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DK45RST2}},
  note         = {Machine review of arXiv:2502.02982}
}
read the original abstract

The advancement of mobile GUI agents has opened new opportunities for automating tasks on mobile devices. Training these agents requires large-scale high-quality data, which is prohibitively expensive when relying on human labor. Given the vast population of global mobile phone users, if automated data collection from them becomes feasible, the resulting data volume and the subsequently trained mobile agents could reach unprecedented levels. Nevertheless, two major challenges arise: (1) extracting user instructions without human intervention and (2) utilizing distributed user data while preserving privacy. To tackle these challenges, we propose MobileA3gent, a collaborative framework that trains mobile GUI Agents using decentralized self-sourced data from diverse users. The framework comprises two components, each targeting a specific challenge: (1) Auto-Annotation, which enables the automatic collection of high-quality datasets during users' routine phone usage with minimal cost. (2) FedVLM-A, which enhances federated VLM training under non-IID distributions by incorporating adapted global aggregation based on both episode-level and step-level variability. Extensive experiments prove that MobileA3gent achieves superior performance over traditional approaches at only 1% of the cost, highlighting its potential for real-world applications

Figures

Figures reproduced from arXiv: 2502.02982 by the authors.

Figure 1
Figure 1. Comparing our pro￾posed paradigm with conven￾tional ones. By leveraging users’ daily phone usage, we achieve superior scalability with drastic cost savings. To tackle these challenges, we propose MobileA3gent, a collaborative learning framework that trains mobile agents using automatically collected user data from daily phone interactions while preserving user privacy. Specifically, MobileA3gent features two novel t… view at source ↗
Figure 2
Figure 2. System overview of MobileA3gent. During individual users’ daily phone usage, Auto￾Annotation automatically constructs training data through step-wise description and episode-wide summarization. Each user then participates in FedVLM-A through our training integration. By applying adapted global aggregation, we obtain the target mobile agent with enhanced capabilities. side of [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Performance and annota￾tion cost trade-off on AndroidWorld. Online Benchmark. We further evaluate our approach on the online benchmark AndroidWorld. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Data quality evaluation across comprehensive metrics. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Comparison between FedVLM-A and 7 base￾lines on non-IID splits of AndroidControl. FedVLM-A achieves SOTA performance on average. Transparent bars indicate average scores over skewed scenarios only. Baselines & Splits. We further con￾duct experiments under non-IID setti…
Figure 6
Figure 6. Figure 6: Ablation study on various base models. x+y indicates using model x for annotation and model y for training. Qwen refers to the consistent use of Qwen2-VL-7B-Instruct for fair comparison. X denotes a specific model. Arrows denote the relative improvement of Auto-Annotat…
Figure 7
Figure 7. Figure 7: Trends in mobile user statistics. The increasing number of mobile users and their rising [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Scaling law analysis on Android Control dataset with different training strategies. All [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Comprehensive visualization of different base models evaluated across multiple metrics [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: High-level SR across different action types within the action space of AndroidControl. The width of the pillars corresponds to the number of data samples in the evaluation test set; thus, the area reflects the weighted average performance. E Discussions and Future Dir…
Figure 11
Figure 11. Figure 11: Episode example from Android Control dataset. The high-level task is [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]
Figure 12
Figure 12. Figure 12: Prompt template for the Descriptor to generate low-level instruction [PITH_FULL_IMAGE:figures/full_fig_p032_12.png]
Figure 13
Figure 13. Figure 13: Prompt template for the Summarizer to generate high-level instruction [PITH_FULL_IMAGE:figures/full_fig_p033_13.png]
Figure 14
Figure 14. Figure 14: Prompt template for Visual-Sense to generate high-level instruction T high based on the list of converted actions and the concatenated screenshot sc . 33 [PITH_FULL_IMAGE:figures/full_fig_p033_14.png]
Figure 15
Figure 15. Figure 15: Prompt template for Chain-of-Thought to generate instruction step-by-step and finally obtain the high-level instruction. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_15.png]
Figure 16
Figure 16. Figure 16: Prompt template for the common part shared between different datasets during training of [PITH_FULL_IMAGE:figures/full_fig_p035_16.png]
Figure 17
Figure 17. Figure 17: Custom prompt template for training mobile GUI agents on AndroidControl. [PITH_FULL_IMAGE:figures/full_fig_p036_17.png]
Figure 18
Figure 18. Figure 18: Custom prompt template for training mobile GUI agents on GUI Odyssey. [PITH_FULL_IMAGE:figures/full_fig_p037_18.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SafeMobile: Chain-level Jailbreak Detection and Automated Evaluation for Multimodal Mobile Agents

    cs.AI 2025-07 conditional novelty 6.0 of 10

    A history-aware guard model with an LLM judge is reported to cut jailbreak success on mobile agent tasks from 86.1% to 8.4% while keeping task completion unchanged at 77.8%.

Reference graph

Works this paper leans on

71 extracted references · 40 canonical work pages · cited by 1 Pith paper

  1. [1]

    Di- giRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning, June 2024

    Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. Di- giRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning, June 2024

    work page 2024

  2. [2]

    MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents, June 2024

    Luyuan Wang, Yongyu Deng, Yiwei Zha, Guodong Mao, Qinmin Wang, Tianchen Min, Wei Chen, and Shoufa Chen. MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents, June 2024

    work page 2024

  3. [3]

    Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception, April 2024

    Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception, April 2024

    work page 2024

  4. [4]

    AppAgent: Multimodal Agents as Smartphone Users, December 2023

    Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. AppAgent: Multimodal Agents as Smartphone Users, December 2023

    work page 2023

  5. [5]

    AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents, July 2024

    Yuxiang Chai, Siyuan Huang, Yazhe Niu, Han Xiao, Liang Liu, Dingyu Zhang, Peng Gao, Shuai Ren, and Hongsheng Li. AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents, July 2024. 10

    work page 2024

  6. [6]

    AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectories, October 2024

    Yifan Song, Weimin Xiong, Xiutian Zhao, Dawei Zhu, Wenhao Wu, Ke Wang, Cheng Li, Wei Peng, and Sujian Li. AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectories, October 2024

    work page 2024

  7. [7]

    Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices, 2024

    Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices, 2024. URL https://arxiv.org/abs/2406.08451

    arXiv 2024

  8. [8]

    Android in the Zoo: Chain-of-Action-Thought for GUI Agents, July 2024

    Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. Android in the Zoo: Chain-of-Action-Thought for GUI Agents, July 2024

    work page 2024

Show all 71 references
  1. [9]

    On the Effects of Data Scale on Computer Control Agents, August 2024

    Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the Effects of Data Scale on Computer Control Agents, August 2024

    2024

  2. [10]

    Advances and open problems in federated learning

    Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(...

    2021

  3. [11]

    Breaking the centralized barrier for cross-device federated learning

    Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebas- tian U Stich, and Ananda Theertha Suresh. Breaking the centralized barrier for cross-device federated learning. Advances in Neural Information Processing Systems , 34:28663–28676, 2021

    2021

  4. [12]

    Communication-efficient learning of deep networks from decentralized data

    Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. PMLR, 2017

    2017

  5. [13]

    Measuring the effects of non-identical data distribution for federated visual classification

    Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335, 2019

    1909 arXiv

  6. [14]

    Adaptive federated optimization

    Sashank J Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Koneˇcn`y, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In International Conference on Learning Representations, 2020

    2020

  7. [15]

    Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms, October 2024

    Zhangheng Li, Keen You, Haotian Zhang, Di Feng, Harsh Agrawal, Xiujun Li, Mohana Prasad Sathya Moorthy, Jeff Nichols, Yinfei Yang, and Zhe Gan. Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms, October 2024

    2024

  8. [16]

    Ui-tars: Pioneering automated gui interaction with native agents

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326, 2025

    2025 arXiv

  9. [17]

    CogAgent: A Visual Language Model for GUI Agents, December 2023

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. CogAgent: A Visual Language Model for GUI Agents, December 2023

    2023

  10. [18]

    Gpt-4v(ision) is a generalist web agent, if grounded, 2024

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v(ision) is a generalist web agent, if grounded, 2024. URL https://arxiv.org/abs/2401.01614

    2024 arXiv

  11. [19]

    Gpt-4v(ision) system card. 2023. URL https://api.semanticscholar.org/CorpusID: 263218031

    2023

  12. [20]

    Swift:a scal- able lightweight infrastructure for fine-tuning, 2024

    Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, and Yingda Chen. Swift:a scal- able lightweight infrastructure for fine-tuning, 2024. URL https://arxiv.org/abs/2408. 05517

    2024

  13. [21]

    Openfedllm: Training large language models on decentralized private data via federated learning

    Rui Ye, Wenhao Wang, Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du, Yanfeng Wang, and Siheng Chen. Openfedllm: Training large language models on decentralized private data via federated learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and ...

    2024

  14. [22]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2021

    2021

  15. [23]

    On the convergence of fedavg on non-iid data

    Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations, 2019

    2019

  16. [24]

    Revisiting weighted aggregation in federated learning with neural networks

    Zexi Li, Tao Lin, Xinyi Shang, and Chao Wu. Revisiting weighted aggregation in federated learning with neural networks. arXiv preprint arXiv:2302.10911, 2023

    2023 arXiv

  17. [25]

    Qwen2-vl: Enhancing vision- language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision- language model’s p...

    2024 arXiv

  18. [26]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vis...

    2024

  19. [27]

    Android in the Wild: A Large-Scale Dataset for Android Device Control, October 2023

    Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Android in the Wild: A Large-Scale Dataset for Android Device Control, October 2023

    2023

  20. [28]

    AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents, June 2024

    Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Mary- beth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. AndroidWorld: A Dynamic Benchma...

    2024

  21. [29]

    Os-atlas: A foundation action model for generalist gui agents

    Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. Os-atlas: A foundation action model for generalist gui agents. arXiv preprint arXiv:2410.23218, 2024

    arXiv

  22. [30]

    OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis, December 2024

    Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, Ben Kao, Guohao Li, Junxian He, Yu Qiao, and Zhiyong Wu. OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis, Dece...

    2024

  23. [31]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. InText summarization branches out: Proceedings of the ACL-04 workshop, pages 74–81, 2004

    2004

  24. [32]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002

    2002

  25. [33]

    Meteor: An automatic metric for mt evaluation with improved correlation with human judgments

    Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72, 2005

    2005

  26. [34]

    DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agents, November 2024

    Taiyi Wang, Zhihao Wu, Jianheng Liu, Jianye Hao, Jun Wang, and Kun Shao. DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agents, November 2024

    2024

  27. [35]

    Dordis: Efficient federated learning with dropout-resilient differential privacy

    Zhifeng Jiang, Wei Wang, and Ruichuan Chen. Dordis: Efficient federated learning with dropout-resilient differential privacy. In Proceedings of the Nineteenth European Conference on Computer Systems, pages 472–488, 2024

    2024

  28. [36]

    Gpt-4: A large-scale multimodal model

    OpenAI. Gpt-4: A large-scale multimodal model. arXiv preprint arXiv:2303.08774, 2023. URL https://arxiv.org/abs/2303.08774

    2023 arXiv

  29. [37]

    Summact: Uncovering user intentions through interactive behaviour summarisation, 2024

    Guanhua Zhang, Mohamed Ahmed, Zhiming Hu, and Andreas Bulling. Summact: Uncovering user intentions through interactive behaviour summarisation, 2024. URL https://arxiv. org/abs/2410.08356. 12

    2024 arXiv

  30. [38]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instruc- tions, 2023. URL https://arxiv.org/abs/2212.10560

    2023 arXiv

  31. [39]

    Identifying user goals from ui trajectories

    Omri Berkovitch, Sapir Caduri, Noam Kahlon, Anatoly Efros, Avi Caciularu, and Ido Dagan. Identifying user goals from ui trajectories. arXiv preprint arXiv:2406.14314, 2024

    2024 arXiv

  32. [40]

    Federated optimization in heterogeneous networks

    Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429–450, 2020

    2020

  33. [41]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024

    2024 arXiv

  34. [42]

    SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association...

    2024 doi

  35. [43]

    Vision-language models for vision tasks: A survey

    Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , 46(8): 5625–5644, 2024. doi: 10.1109/TPAMI.2024.3369699

    2024

  36. [44]

    Appvlm: A lightweight vision language model for online app control, 2025

    Georgios Papoudakis, Thomas Coste, Zhihao Wu, Jianye Hao, Jun Wang, and Kun Shao. Appvlm: A lightweight vision language model for online app control, 2025. URL https: //arxiv.org/abs/2502.06395

    2025 arXiv

  37. [45]

    Lightweight neural app control

    Filippos Christianos, Georgios Papoudakis, Thomas Coste, HAO Jianye, Jun Wang, and Kun Shao. Lightweight neural app control. In NeurIPS 2024 Workshop on Open-World Agents, 2024

    2024

  38. [46]

    Learnact: Few-shot mobile gui agent with a unified demonstration benchmark, 2025

    Guangyi Liu, Pengxiang Zhao, Liang Liu, Zhiming Chen, Yuxiang Chai, Shuai Ren, Hao Wang, Shibo He, and Wenchao Meng. Learnact: Few-shot mobile gui agent with a unified demonstration benchmark, 2025. URL https://arxiv.org/abs/2504.13805

    2025 arXiv

  39. [47]

    Llm-powered gui agents in phone automation: Surveying progress and prospects, 2025

    Guangyi Liu, Pengxiang Zhao, Liang Liu, Yaxuan Guo, Han Xiao, Weifeng Lin, Yuxiang Chai, Yue Han, Shuai Ren, Hao Wang, Xiaoyu Liang, Wenhao Wang, Tianze Wu, Linghao Li, Hao Wang, Guanjing Xiong, Yong Liu, and Hongsheng Li. Llm-powered gui agents in phone automation: Surveying ...

    2025

  40. [48]

    Mobilegpt: Augmenting llm with human-like app memory for mobile task automation

    Sunjae Lee, Junyoung Choi, Jungjae Lee, Munim Hasan Wasi, Hojun Choi, Steve Ko, Sangeun Oh, and Insik Shin. Mobilegpt: Augmenting llm with human-like app memory for mobile task automation. In Proceedings of the 30th Annual International Conference on Mobile Computing and Netwo...

    2024

  41. [49]

    Omniparser for pure vision based gui agent, 2024

    Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based gui agent, 2024. URL https://arxiv.org/abs/2408.00203

    2024 arXiv

  42. [50]

    Retrieval-style in-context learning for few-shot hierarchical text classification

    Huiyao Chen, Yu Zhao, Zulong Chen, Mengjia Wang, Liangyue Li, Meishan Zhang, and Min Zhang. Retrieval-style in-context learning for few-shot hierarchical text classification. Transactions of the Association for Computational Linguistics, 12:1214–1231, 2024

    2024

  43. [51]

    Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning, August 2021

    Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning, August 2021. 13

    2021

  44. [52]

    Autowebglm: A large language model-based web navigating agent

    Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, and Jie Tang. Autowebglm: A large language model-based web navigating agent. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery a...

    2024

  45. [53]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  46. [54]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Syste...

    2023

  47. [55]

    Lmdeploy: A toolkit for compressing, deploying, and serving llm

    LMDeploy Contributors. Lmdeploy: A toolkit for compressing, deploying, and serving llm. https://github.com/InternLM/lmdeploy, 2023

    2023

  48. [56]

    Privacy-Preserving Instructions for Aligning Large Language Models, February 2024

    Da Yu, Peter Kairouz, Sewoong Oh, and Zheng Xu. Privacy-Preserving Instructions for Aligning Large Language Models, February 2024

    2024

  49. [57]

    Knowl- edgeSG: Privacy-preserving synthetic text generation with knowledge distillation from server

    WenHao Wang, Xiaoyu Liang, Rui Ye, Jingyi Chai, Siheng Chen, and Yanfeng Wang. Knowl- edgeSG: Privacy-preserving synthetic text generation with knowledge distillation from server. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference...

    2024 doi

  50. [58]

    The phantom menace: Unmasking privacy leakages in vision-language models, 2024

    Simone Caldarella, Massimiliano Mancini, Elisa Ricci, and Rahaf Aljundi. The phantom menace: Unmasking privacy leakages in vision-language models, 2024. URL https://arxiv. org/abs/2408.01228

    2024 arXiv

  51. [59]

    Déjà vu memorization in vision- language models, 2024

    Bargav Jayaraman, Chuan Guo, and Kamalika Chaudhuri. Déjà vu memorization in vision- language models, 2024. URL https://arxiv.org/abs/2402.02103

    2024 arXiv

  52. [60]

    Tea-fed: time-efficient asynchronous federated learning for edge computing

    Chendi Zhou, Hao Tian, Hong Zhang, Jin Zhang, Mianxiong Dong, and Juncheng Jia. Tea-fed: time-efficient asynchronous federated learning for edge computing. In Proceedings of the 18th ACM international conference on computing frontiers, pages 30–37, 2021

    2021

  53. [61]

    efedllm: Efficient llm inference based on federated learning

    Shengwen Ding and Chenhui Hu. efedllm: Efficient llm inference based on federated learning. arXiv preprint arXiv:2411.16003, 2024

    2024 arXiv

  54. [62]

    UI-Hawk: Unleashing the Screen Stream Understanding for GUI Agents

    Jiwen Zhang, Yaqi Yu, Minghui Liao, Wentao Li, Jihao Wu, and Zhongyu Wei. UI-Hawk: Unleashing the Screen Stream Understanding for GUI Agents. August 2024. doi: 10.20944/ preprints202408.2137.v1

    2024

  55. [63]

    Term-weighting approaches in automatic text retrieval

    Gerard Salton and Christopher Buckley. Term-weighting approaches in automatic text retrieval. Information processing & management, 24(5):513–523, 1988

    1988

  56. [64]

    Deepseek-vl: Towards real-world vision-language understanding, 2024

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseek-vl: Towards real-world vision-language understanding, 2024

    2024

  57. [65]

    Ovis: Structural embedding alignment for multimodal large language model

    Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural embedding alignment for multimodal large language model. arXiv:2405.20797, 2024

    2024 arXiv

  58. [66]

    Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan Ö. Arık. Learn- by-interact: A data-centric framework for self-adaptive agents in realistic environments, 2025. URL https://arxiv.org/abs/2501.10893

    2025 arXiv

  59. [67]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022. 14

    2022

  60. [68]

    Open the Zoho Meet app and view the scheduled meetings

    Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pages 5132–5143. PMLR, 2020. 15 A Related Work 1...

    2020

  61. [69]

    cooking" - Your Output: Click on the article titled

    uses 0.9/0.1 ratio for historical/current model interpolation. FedProx [ 40] applies proximal regularization with µ = 0.2 through ||w − wt||2 penalty terms. SCAFFOLD [ 68] configurations maintain server learning rate ηs = 1.0 with client momentum compensation, while FedAdam an...

  62. [70]

    They provide essential functionality and are defined with a specific format, ensuring consistency and reliability

    Basic Actions Basic actions are standardized and available across all platforms. They provide essential functionality and are defined with a specific format, ensuring consistency and reliability. • Basic Action 1: CLICK – purpose: Click at the specified position. – format: CLI...

  63. [71]

    They allow for flexibility and adaptability, enabling the model to support new and unseen actions defined by users

    Custom Actions Custom actions are unique to each user ´s platform and environment. They allow for flexibility and adaptability, enabling the model to support new and unseen actions defined by users. These actions extend the functionality of the basic set, making the model more...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.