Pith. sign in

REVIEW 16 cited by

AutoGLM: Autonomous Foundation Agents for GUIs

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2411.00820 v1 pith:XTKVQBFF submitted 2024-10-28 cs.HC cs.AIcs.CLcs.LG

Xiao Liu , Bo Qin , Dongzhu Liang , Guang Dong , Hanyu Lai , Hanchen Zhang , Hanlin Zhao , Iat Long Iong
show 22 more authors
Jiadai Sun Jiaqi Wang Junjie Gao Junjun Shan Kangning Liu Shudan Zhang Shuntian Yao Siyi Cheng Wentao Yao Wenyi Zhao Xinghan Liu Xinyi Liu Xinying Chen Xinyue Yang Yang Yang Yifan Xu Yu Yang Yujia Wang Yulin Xu Zehan Qi Yuxiao Dong Jie Tang
This is my paper
classification cs.HCcs.AIcs.CLcs.LG
keywords autoglmfoundationagentsautonomouscontrolagentdevelopedguis
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We present AutoGLM, a new series in the ChatGLM family, designed to serve as foundation agents for autonomous control of digital devices through Graphical User Interfaces (GUIs). While foundation models excel at acquiring human knowledge, they often struggle with decision-making in dynamic real-world environments, limiting their progress toward artificial general intelligence. This limitation underscores the importance of developing foundation agents capable of learning through autonomous environmental interactions by reinforcing existing models. Focusing on Web Browser and Phone as representative GUI scenarios, we have developed AutoGLM as a practical foundation agent system for real-world GUI interactions. Our approach integrates a comprehensive suite of techniques and infrastructures to create deployable agent systems suitable for user delivery. Through this development, we have derived two key insights: First, the design of an appropriate "intermediate interface" for GUI control is crucial, enabling the separation of planning and grounding behaviors, which require distinct optimization for flexibility and accuracy respectively. Second, we have developed a novel progressive training framework that enables self-evolving online curriculum reinforcement learning for AutoGLM. Our evaluations demonstrate AutoGLM's effectiveness across multiple domains. For web browsing, AutoGLM achieves a 55.2% success rate on VAB-WebArena-Lite (improving to 59.1% with a second attempt) and 96.2% on OpenTable evaluation tasks. In Android device control, AutoGLM attains a 36.2% success rate on AndroidLab (VAB-Mobile) and 89.7% on common tasks in popular Chinese APPs.

Discussion (0). Sign in to comment.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    OSWorld 2.0 is a benchmark of 108 realistic long-horizon computer-use tasks where current agents achieve only 20.6% binary completion, struggling with state inference and constraint tracking.

  2. SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Prediction

    cs.AI 2026-07 conditional novelty 6.0 of 10

    SeerGuard adds pre-execution instruction screening and action-level semantic next-state prediction to mobile GUI agents, improving safety-utility and risk-cost scores on MobileSafetyBench.

  3. GUI-AC: Enhancing Continual Learning in GUI Agents

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    GUI-AC stabilizes RFT for non-stationary GUI data by down-weighting noisy advantages and relaxing clipping bounds via a grounding certainty term.

  4. GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL

    cs.LG 2026-02 conditional novelty 6.0 of 10

    Action-weighted SFT plus KL-regularized GRPO on a curated 81K reasoning dataset lifts open-source GUI agents by 11–22 points on online task-completion benchmarks.

  5. Mind the Gap: Action Rebinding Attacks against Android GUI Agents

    cs.CR 2026-01 conditional novelty 6.0 of 10

    A zero-permission Android app can redirect a GUI agent's planned tap to a different app by switching the foreground during the agent's reasoning delay.

  6. MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents

    cs.CV 2025-07 conditional novelty 6.0 of 10

    MMBench-GUI provides a multi-platform, four-level benchmark and an efficiency-aware metric, and its experiments indicate visual grounding is the main bottleneck for current GUI agents.

  7. Context manipulation attacks : Web agents are susceptible to corrupted memory

    cs.CR 2025-06 conditional novelty 6.0 of 10

    Web agents with protected prompts can still be hijacked by injecting malicious steps into their stored task plans, reaching up to 63% success on privacy leaks.

  8. ZeroGUI: Automating Online GUI Learning at Zero Human Cost

    cs.AI 2025-05 conditional novelty 6.0 of 10

    ZeroGUI uses VLM-generated tasks and VLM-estimated rewards with two-stage GRPO to improve GUI agent success rates on OSWorld and AndroidLab without human annotations.

  9. TransBench: Breaking Barriers for Transferable Graphical User Interface Agents in Dynamic Digital Environments

    cs.HC 2025-05 conditional novelty 6.0 of 10

    TransBench is a new benchmark of 1,459 screenshots and 22,000 grounding instructions for measuring how well GUI agents transfer across app versions, platforms, and applications.

  10. HyMobileAgent: Data-Environment Co-Scaling for Efficient GUI Agents

    cs.CV 2026-07 reject novelty 5.0 of 10

    A 3B-scale GUI agent reportedly scores 82.6% on AndroidWorld and 42% on real-device tasks, but the evidence is not independently verified and may overlap with its RL training.

  11. Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling System

    cs.CL 2025-06 conditional novelty 5.0 of 10

    AI phone agents succeed on single-step commands but fail on multi-step compositions; a new benchmark measures this gap and a scheduler that decomposes tasks recovers most of it.

  12. GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents

    cs.CL 2025-05 conditional novelty 5.0 of 10

    GUI-G1-3B shows that for GUI grounding, removing reasoning chains, regularizing box size in the reward, and reweighting GRPO by difficulty yields 90.3% on ScreenSpot and 37.1% on ScreenSpot-Pro.

  13. VLM-3D:End-to-End Vision-Language Models for Open-World 3D Perception

    cs.CV 2025-08 reject novelty 4.0 of 10

    The paper promises VLM-3D but the body text is entirely the MVISU-Bench mobile-agent benchmark paper, so the stated result is unsupported.

  14. Uncertainty-Aware GUI Agent: Adaptive Perception through Component Recommendation and Human-in-the-Loop Refinement

    cs.AI 2025-08 conditional novelty 4.0 of 10

    A GUI agent that trims UI input with a recommendation module and asks users when decisions are ambiguous reports state-of-the-art success rates, though the interaction module is not benchmarked.

  15. GUI-G$^2$: Gaussian Reward Modeling for GUI Grounding

    cs.LG 2025-07 conditional novelty 4.0 of 10

    Modeling GUI elements as Gaussian distributions instead of binary targets yields 92.0% (ScreenSpot), 93.3% (ScreenSpot-v2), and 47.5% (ScreenSpot-Pro) for a 7B model, outperforming UI-TARS-72B by a relative 24.7% on t...

  16. A Comprehensive Survey of Deep Research: Systems, Methodologies, and Applications

    cs.AI 2025-06 conditional novelty 4.0 of 10

    A survey of 80+ Deep Research systems that proposes a four-layer taxonomy (foundation models, tool use, planning, synthesis) and compares commercial and open-source implementations.

Pith tools