Pith. sign in

REVIEW 3 major objections 6 minor 44 references

GUIDE: Resolving Domain Bias in GUI Agents through Real-Time Web Video Retrieval and Plug-and-Play Annotation

T0 review · 3 major / 6 minor · reviewed 2026-07-13 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GUIDE removes GUI-agent domain bias by retrieving live tutorial videos, turning them into planning and grounding knowledge, and injecting that knowledge as plug-and-play text without training.

desk verdict Solid systems paper: live tutorial retrieval + dual-channel injection gives real OSWorld gains without fine-tuning; the weak link is unmeasured retrieval mismatch, not the method design. read the letter →

arxiv 2603.26266 v3 pith:JSC2UTY6 submitted 2026-03-27 cs.AI cs.CV

Rui Xie , Zhi Gao , Chenrui Shi , Zirui Shang , Lu Chen , Qing Li This is my paper
classification cs.AIcs.CV
keywords GUIagentdomainbiasvideoretrieval-augmentedgenerationautonomouslearninginversedynamicsannotationplanningknowledgegroundingplug-and-play
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

GUI agents built on large vision-language models can see screens and follow instructions, but they still fail on specific apps because they do not know that app’s workflows or where its controls live. The paper argues this is a knowledge gap, not a capability gap, and that the gap can be closed at inference time from the open web. GUIDE searches YouTube for task-relevant tutorials, filters them with subtitles, and automatically converts consecutive video frames into natural-language planning steps and UI-element descriptions. Those texts are injected into existing single-model or multi-agent systems as reference material only. On OSWorld the same agents gain roughly 4.5–7.5 absolute points and often take fewer steps, without any weight or architecture change.

What carries the argument

Subtitle-driven Video-RAG plus inverse-dynamics annotation: progressive three-stage subtitle filtering (domain classification, topic extraction, relevance matching) selects up to two tutorials, then consecutive keyframes with UI-element graphs are fed to a VLM that outputs transferable planning trajectories and grounding element descriptions for prompt injection.

What would settle it

Run the full OSWorld suite again after deliberately replacing retrieved videos with high-scoring but procedurally mismatched tutorials (wrong goal, browser-based install, or slide-deck format) and check whether the reported gains disappear or reverse into more failures and wasted steps.

Watch

Extended reading notes

Core claim

Domain bias in GUI agents has two faces—wrong workflows (planning) and wrong UI layouts (grounding)—and both can be corrected by autonomously retrieving instructional videos, annotating them with an inverse-dynamics VLM pipeline, and injecting the resulting dual-channel natural-language knowledge into the agent at test time. The method is training-free and architecture-agnostic: the same extracted knowledge improves multi-agent systems and single-model agents alike on real desktop tasks.

Load-bearing premise

The method assumes that a live YouTube search will return procedurally matched screencasts whose steps and UI descriptions actually transfer to the target software and screen layout.

Editorial extensions

If this is right

  • Existing GUI agents can be improved at deployment by adding a retrieval-and-annotation sidecar rather than collecting new labeled trajectories or fine-tuning.
  • Planning-level domain knowledge accounts for most of the gain; grounding helps mainly on apps with dense or nonstandard UI layouts.
  • As software UIs change, agents can re-query the web for updated tutorials instead of waiting for a new training corpus.
  • The same dual-channel natural-language knowledge works for both multi-agent systems and single-model agents without architectural edits.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The bottleneck may shift from agent capability to retrieval quality: better procedural-consistency filters and screencast detectors would raise the ceiling more than stronger base VLMs alone.
  • The same pipeline could supply on-demand expertise for non-desktop interfaces (mobile, industrial HMIs) wherever public tutorial video exists.
  • Because knowledge is reference text rather than hard rules, residual failures from version drift become a measurable retrieval-evaluation problem rather than an opaque model failure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. GUIDE is a training-free, plug-and-play framework that aims to reduce domain bias in VLM-based GUI agents by retrieving task-relevant YouTube tutorial videos at test time and converting them into natural-language Planning and Grounding knowledge. A subtitle-driven three-stage Video-RAG pipeline (domain classification, topic extraction, relevance matching) selects up to K=2 videos; an inverse-dynamics annotation pipeline (keyframe pairs + OmniParser UI graphs + topic/subtitle context into a VLM, Eq. 1) produces transferable workflow and element descriptions that are injected into multi-agent or single-model agents without changing parameters or architecture. On OSWorld (361 tasks), the paper reports consistent absolute gains of roughly 4.5–7.5% across Seed-1.8, Qwen3-VL-8B, and AgentS3, with planning contributing most of the gain, complementary grounding benefits in UI-complex domains, ablations on element count and annotator strength, human checks of retrieval stages, cost accounting, and explicit failure cases.

Significance. If the reported gains hold under realistic retrieval conditions, GUIDE is a practically important contribution: it treats open-web tutorial video as a continuously updated knowledge source for GUI agents and does so without fine-tuning. Strengths that should be credited include architecture-agnostic dual-channel injection (Mode A multi-agent and Mode B single-model), full OSWorld evaluation on three agent stacks, channel and annotator ablations, human evaluation of Stage-1/2 retrieval quality, transparent cost breakdown, released code and dataset, and honest failure-case analysis (Sec. 4.4, App. F). Relative to Watch&Learn and offline video-mining lines, live retrieval plus structured planning/grounding decomposition is a clear systems advance for inference-time domain adaptation.

major comments (3)
  1. [Sec. 4.4 / App. F / Table 1] Sec. 4.4 and App. F document planning mismatches (wrong goal/workflow) and grounding mismatches (browser UI, animated slides) that waste steps or exhaust the 50-step budget, yet the paper never reports the fraction of the 299 annotated tasks that are procedurally mismatched, nor conditional success/step metrics for matched vs. mismatched injections. Table 1’s net +4.5–7.5% is therefore an aggregate that may mix large gains on well-matched videos with neutral or negative effects on mismatched ones. A load-bearing analysis for the claim that GUIDE supplies transferable domain expertise would be: (i) a human or automatic match-quality label for retrieved videos, and (ii) success rate and step-count stratified by match quality (and vs. no-injection on the same tasks).
  2. [Table 1 / Sec. 4.2] Table 1 compares GUIDE’s +4.47pp on AgentS3 (50.18→54.65) to Watch&Learn’s +2.2pp on Jedi under ICL. These are different agents, different retrieval corpora (live web vs. pre-built offline), and different evaluation setups; the text’s claim of a “substantially larger gain … on a comparably strong baseline” is therefore only weakly controlled. Either run a head-to-head on the same agent/protocol, or reframe the comparison as qualitative (live retrieval + dual-channel structure) rather than as a direct superiority claim on absolute points.
  3. [Sec. 4.5 / Table 1] Sec. 4.5 reports 82.8% retrieval coverage (299/361) and graceful degradation on the remaining 17.2%, but Table 1 does not break out covered vs. uncovered tasks. Because uncovered tasks contribute only baseline performance, the headline “over 5%” average can understate the effect when knowledge is available and overstate the method’s reliability as a general unbiasing mechanism. Please report overall and per-domain scores conditioned on whether any video was injected, and clarify how multi-app tasks (the main uncovered set) affect the Multi column.
minor comments (6)
  1. [Table 2(a) / Sec. 4.1] Table 2(a) peaks at k=5 (+11.99pp) while main results use default k=7, which is past the optimum on the 50-task subset. State the default choice more clearly in Sec. 4.1 and note the sensitivity in the main text, not only the ablation.
  2. [Sec. 4.1] Practical Notes (Sec. 4.1) mention Seed-1.8 malformed outputs and Chrome network failures that depress absolute scores. Confirm that relative GUIDE vs. baseline comparisons used identical retry/parsing policies so the deltas are not confounded by intermittent API or network noise.
  3. [Sec. 3.3 / App. D] Eq. (1) and Fig. 3 describe f_IDM well, but the Meaningful filter’s role in trajectory quality is mostly deferred to App. D. A short quantitative sentence in Sec. 3.3 (e.g., invalid-frame F1) would help readers assess annotation noise without leaving the main text.
  4. [Fig. 1 / Fig. 2] Fig. 1 and Fig. 2 use dense multi-panel layouts; ensure final camera-ready figures enlarge Stage-1/2/3 labels and score callouts for print readability.
  5. [Abstract] Abstract says “over 5% improvements” while the body reports 4.5–7.5% depending on agent; align the abstract with the full range or state “about 4.5–7.5% absolute.”
  6. [Sec. 2] Related Work could more sharply separate GUIDE from TongUI / VideoAgentTrek (offline fine-tuning from YouTube) in one comparative sentence on live vs. offline and plug-and-play vs. weight updates.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: GUIDE is empirical systems work whose gains are measured by external OSWorld state scripts, not by reconstructing fitted or self-defined quantities.

full rationale

The paper’s load-bearing claim is empirical: live YouTube retrieval + inverse-dynamics annotation + dual-channel NL injection improves the same agents on OSWorld without weight changes (Tab. 1: Seed-1.8 37.14→44.62, Qwen3-VL-8B 33.90→39.73, AgentS3 50.18→54.65). Success is scored by OSWorld’s execution-based gold-state scripts (App. B), independent of the retrieval/annotation pipeline. Eq. 1 (f_IDM) is a VLM procedure that maps keyframe pairs, OmniParser graphs, topic, and subtitles to transferable Thought/Action text; it does not define the OSWorld metric or fit parameters to the evaluation outcomes and then re-predict them. Knowledge is framed as optional reference, not as a quantity forced by construction. Related-work citations (Watch&Learn, TongUI, etc.) are comparative context, not uniqueness theorems or load-bearing premises that force the result. Ablations (planning vs grounding, k, annotator model) and failure cases (App. F) are reported as empirical evidence, not as self-definitional closures. Minor design thresholds (relevance ≥0.5, K≤2, k=7) are hyperparameters, not circular proofs. No self-definitional loop, fitted-input-as-prediction, or self-citation chain reduces the claimed gains to their inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 2 invented entities

GUIDE’s central claim rests on engineering assumptions and tunable pipeline choices rather than new physical entities. The free parameters are retrieval/annotation thresholds and injection budgets; the axioms are standard ML/systems assumptions about VLM competence, subtitle usefulness, and transfer of natural-language tutorial knowledge into agent prompts.

free parameters (4)
  • adaptive top-K video retention (K≤2, non-top1 relevance ≥0.5)
    Hand-chosen retrieval cutoff that determines which tutorials enter annotation and can change injected knowledge quality.
  • max grounding elements k (default 7; peak at 5 on 50-task subset)
    Injection budget chosen by ablation; performance is non-monotonic in k, so the reported dual-channel gains depend on this setting.
  • MOG2 foreground threshold (>10000 px) and keyframe sampling rules
    Controls which state pairs are annotated; different thresholds would change the trajectory and downstream planning/grounding text.
  • subtitle truncation and relevance scoring prompts
    Operational choices (e.g., 10k-character subtitle cap, dual-anchored topic prompt) that shape classification and ranking without a unique theoretical optimum.
assumptions (4)
  • domain assumption Public YouTube tutorials plus auto-generated subtitles contain enough operational semantics to recover task-relevant GUI workflows for many OSWorld apps.
    Load-bearing premise of the Video-RAG design; coverage is 82.8% of tasks, with graceful fallback otherwise.
  • domain assumption A VLM inverse-dynamics annotator given consecutive keyframes, OmniParser graphs, topic, and local subtitles can produce transferable natural-language planning and grounding knowledge.
    Core of Eq. 1 and the annotation pipeline; quality varies by annotator model but remains positive even for weaker annotators.
  • domain assumption Injected knowledge framed as reference material will be used productively by the base agent and overridden when screenshots conflict.
    Stated integration principle in Sec. 3.4; failure cases show this safeguard is incomplete when retrieval is mismatched.
  • standard math Standard LLM/VLM prompting, ASR, background subtraction, and UI parsing components behave as described under the chosen commercial/open models.
    Background tooling assumptions rather than novel theory; still necessary for the reported pipeline to work.
invented entities (2)
  • GUIDE dual-channel knowledge objects (Planning knowledge and Grounding knowledge)
    purpose: Structured natural-language artifacts that separately target workflow bias and UI-layout bias and can be injected into different agent modules.
    These are paper-defined intermediate representations, not physical entities; their value is measured only through downstream agent success.
  • Subtitle-driven three-stage Video-RAG filter (domain classification → topic extraction → relevance matching)
    purpose: Progressively select task-relevant GUI demonstration videos from noisy open-web search results.
    A methodological construct validated by human precision/recall and topic scores inside the paper, not an independently observed natural object.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GUIDE: Resolving Domain Bias in GUI Agents through Real-Time Web Video Retrieval and Plug-and-Play Annotation." pith.science (2026). https://pith.science/paper/JSC2UTY6

@misc{pith2026260326266,
  author       = {Pith},
  title        = {Pith review of: GUIDE: Resolving Domain Bias in GUI Agents through Real-Time Web Video Retrieval and Plug-and-Play Annotation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JSC2UTY6}},
  note         = {Machine review of arXiv:2603.26266}
}
read the original abstract

Large vision-language models have endowed GUI agents with strong general capabilities for interface understanding and interaction. However, due to insufficient exposure to domain-specific software operation data during training, these agents exhibit significant domain bias - they lack familiarity with the specific operation workflows (planning) and UI element layouts (grounding) of particular applications, limiting their real-world task performance. In this paper, we present GUIDE (GUI Unbiasing via Instructional-Video Driven Expertise), a training-free, plug-and-play framework that resolves GUI agent domain bias by autonomously acquiring domain-specific expertise from web tutorial videos through a retrieval-augmented automated annotation pipeline. GUIDE introduces two key innovations. First, a subtitle-driven Video-RAG pipeline unlocks video semantics through subtitle analysis, performing progressive three-stage retrieval - domain classification, topic extraction, and relevance matching - to identify task-relevant tutorial videos. Second, a fully automated annotation pipeline built on an inverse dynamics paradigm feeds consecutive keyframes enhanced with UI element detection into VLMs, inferring the required planning and grounding knowledge that are injected into the agent's corresponding modules to address both manifestations of domain bias. Extensive experiments on OSWorld demonstrate GUIDE's generality as a plug-and-play component for both multi-agent systems and single-model agents. It consistently yields over 5% improvements and reduces execution steps - without modifying any model parameters or architecture - validating GUIDE as an architecture-agnostic enhancement to bridge GUI agent domain bias.

Figures

Figures reproduced from arXiv: 2603.26266 by the authors.

Figure 1
Figure 1. Overview of GUIDE. (1) A Retrieval Agent filters YouTube candidates via three subtitle-driven stages to select top-K videos. (2) An Annotation Agent applies fIDM (Eq. 1) on keyframe pairs st/st+1 with UI element graphs Et/Et+1, topic Ttopic, and subtitle context Csub, producing planning and grounding knowledge. (3) Knowledge is injected into the GUI Agent in a plug-and-play manner—supporting multi-agent (Mode A) and… view at source ↗
Figure 2
Figure 2. Subtitle-driven Video-RAG pipeline. From 50+ YouTube candidates, a metadata pre-filter removes outliers, then three subtitle-driven stages progressively narrow results: (a) domain classification filters non-GUI content, (b) topic extraction (title + subtitle → semantic descriptor), and (c) dual-anchored relevance matching (topic as primary anchor, score 0–1). Final top-K (K≤2) videos proceed to annotation ( [PITH_F… view at source ↗
Figure 3
Figure 3. Fully automated annotation pipeline. Retrieved videos ( [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Qualitative example on a GIMP contrast-adjustment task [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Human evaluation of the Guide annotation pipeline. Each stage is independently evaluated by 3 annotators on 300 randomly sampled videos. (a) Stage 1: GUI domain classification on 300 candidate videos (pre-filter), achieving 100% precision with zero non-GUI contaminatio…
Figure 6
Figure 6. Figure 6: Representative examples for the Meaningful evaluation. Top: a non-GUI intro pair from a retrieved video segment, where OmniParser captures only sparse text regions rather than actionable UI structure. Bottom: an idle no-action GUI pair, where the interface remains sema…
Figure 7
Figure 7. Figure 7: Failure Cases: When Annotations Mislead the Agent. (a) Planning [PITH_FULL_IMAGE:figures/full_fig_p026_7.png]
Figure 8
Figure 8. Figure 8: Additional Failure Cases. (a) Planning mismatch: [PITH_FULL_IMAGE:figures/full_fig_p027_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 7 linked inside Pith

  1. [1]

    ICLR (2025)

    Agashe, S., Jimenez, C.E., Yang, B., Kim, N., Yang, J.: Agent S: An open agentic framework that uses computers like a human. ICLR (2025)

    2025

  2. [2]

    COLM (2025)

    Agashe, S., Wong, K., Tu, V., Yang, J., Li, A., Wang, X.E.: Agent S2: A composi- tional generalist-specialist framework for computer use agents. COLM (2025)

    2025

  3. [3]

    Anthropic: Claude Opus 4.6.https://www.anthropic.com/news/claude-opus-4- 6(2025)

    2025

  4. [4]

    arXiv preprint arXiv:2511.21631 (2025)

    Bai, S., Cai, Y., Chen, R., Chen, K., Cheng, Z., Ge, W., Hui, B., Li, K., Lin, J., Lu, D., et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025)

    arXiv 2025

  5. [5]

    Bytedance Seed Team: Seed-1.8 model card. Tech. rep., Bytedance (2025), avail- able at https : / / lf3 - static . bytednsdoc . com / obj / eden - cn / lapzild - tss / ljhwZthlaukjlkulzlp/research/Seed-1.8-Modelcard.pdf

    2025

  6. [6]

    ICLR (2025)

    Chen, D., Huang, Y., Wu, S., Tang, J., Chen, L., Bai, Y., He, Z., Wang, C., Zhou, H., Li, Y., Zhou, T., Yu, Y., Gao, C., Zhang, Q., Gui, Y., Li, Z., Wan, Y., Zhou, P., Gao, J., Sun, L.: GUI-World: A video benchmark and dataset for multimodal GUI-oriented understanding. ICLR (2025)

    2025

  7. [7]

    ACL (2024)

    Cheng, K., Sun, Q., Chu, Y., Xu, F., Li, Y., Zhang, J., Wu, Z.: SeeClick: Harnessing GUI grounding for advanced visual GUI agents. ACL (2024)

    2024

  8. [8]

    https://github.com/yt-dlp/yt-dlp(2024)

    yt-dlp contributors: yt-dlp: A feature-rich command-line audio/video downloader. https://github.com/yt-dlp/yt-dlp(2024)

    2024

Show all 44 references
  1. [9]

    CVPR (2024)

    Gao, D., Ji, L., Bai, Z., Ouyang, M., Li, P., Mao, D., Wu, Q., Zhang, W., Wang, P., Guo, X., Wang, H., Zhou, L., Shou, M.Z.: ASSISTGUI: Task-oriented desktop graphical user interface automation. CVPR (2024)

    2024

  2. [10]

    arXiv preprint arXiv:2510.02250 (2025)

    Gonzalez-Pumariega, G., Tu, V., Lee, C.L., Yang, J., Li, A., Wang, X.E.: Scaling agents for computer use. arXiv preprint arXiv:2510.02250 (2025)

    2025

  3. [11]

    EMNLP (2025)

    Guan, Z., Li, J.C.L., Hou, Z., Zhang, P., Xu, D., Zhao, Y., Wu, M., Chen, J., Nguyen, T.T., Xian, P., Ma, W., Qin, S., Chesi, G., Wong, N.: KG-RAG: Enhancing GUI agent decision-making via knowledge graph-driven retrieval-augmented generation. EMNLP (2025)

    2025

  4. [12]

    ICLR (2024)

    Gur, I., Furuta, H., Huang, A., Safdari, M., Matsuo, Y., Eck, D., Faust, A.: A real-world WebAgent with planning, long context understanding, and program synthesis. ICLR (2024)

    2024

  5. [13]

    CVPR (2024)

    Hong, W., Wang, W., Lv, Q., Xu, J., Yu, W., Ji, J., Wang, Y., Wang, Z., Zhang, Y., Li, J., et al.: CogAgent: A visual language model for GUI agents. CVPR (2024)

    2024

  6. [14]

    NeurIPS (2020) 16 R

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.t., Rocktäschel, T., et al.: Retrieval-augmented generation for knowledge-intensive NLP tasks. NeurIPS (2020) 16 R. Xie et al

    2020

  7. [15]

    arXiv preprint arXiv:2411.17465 (2024)

    Lin, K.Q., Xu, L., Guo, D., Zhong, Y., Liu, J., et al.: ShowUI: One vision-language- action model for GUI visual agent. arXiv preprint arXiv:2411.17465 (2024)

    2024 arXiv

  8. [16]

    arXiv preprint arXiv:2510.19488 (2025)

    Lu, D., Xu, Y., Wang, J., Wu, H., Wang, X., Wang, Z., Yang, J., Su, H., Chen, J., Chen, J., Mao, Y., Zhou, J., Lin, J., Hui, B., Yu, T.: VideoAgentTrek: Computer use pretraining from unlabeled videos. arXiv preprint arXiv:2510.19488 (2025)

    2025

  9. [17]

    arXiv preprint arXiv:2408.00203 (2024)

    Lu, Y., Yang, J., Shen, Y., Awadallah, A.: OmniParser for pure vision based GUI agent. arXiv preprint arXiv:2408.00203 (2024)

    2024 arXiv

  10. [18]

    arXiv preprint arXiv:2410.21276 (2024)

    OpenAI: GPT-4o system card. arXiv preprint arXiv:2410.21276 (2024)

    2024 arXiv

  11. [19]

    arXiv preprint arXiv:2601.03267 (2025)

    OpenAI: GPT-5 system card. arXiv preprint arXiv:2601.03267 (2025)

    2025 arXiv

  12. [20]

    ICML (2023)

    Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C., Sutskever, I.: Robust speech recognition via large-scale weak supervision. ICML (2023)

    2023

  13. [21]

    NeurIPS (2023)

    Rawles, C., Li, A., Rodriguez, D., Riva, O., Lillicrap, T.: Android in the wild: A large-scale dataset for android device control. NeurIPS (2023)

    2023

  14. [22]

    Song, C.H., Song, Y., Goyal, P., Su, Y., Riva, O., Palangi, H., Pfister, T.: Watch and learn:Learningtousecomputersfromonlinevideos.arXivpreprintarXiv:2510.04673 (2025)

    2025

  15. [23]

    NeurIPS (2024)

    Tan, W., Zhang, W., Xu, X., Xia, H., Ding, Z., Li, B., Zhou, B., Yue, J., Jiang, J., Li, Y., et al.: Cradle: Empowering foundation agents towards general computer control. NeurIPS (2024)

    2024

  16. [24]

    arXiv preprint arXiv:2502.17110 (2025)

    Wang, J., Xu, H., Zhang, X., Yan, M., Zhang, J., Huang, F., Sang, J.: Mobile- Agent-V: A video-guided approach for effortless and efficient operational knowledge injection in mobile automation. arXiv preprint arXiv:2502.17110 (2025)

    2025 arXiv

  17. [25]

    NeurIPS (2022)

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., Zhou, D.: Chain-of-thought prompting elicits reasoning in large language models. NeurIPS (2022)

    2022

  18. [26]

    arXiv preprint arXiv:2505.13227 (2025)

    Xie, T., Deng, J., Li, X., Yang, J., Wu, H., Chen, J., Hu, W., Wang, X., Xu, Y., Wang, Z., Xu, Y., Wang, J., Sahoo, D., Yu, T., Xiong, C.: Scaling Computer- Use Grounding via User Interface Decomposition and Synthesis. arXiv preprint arXiv:2505.13227 (2025)

    2025

  19. [27]

    NeurIPS (2024)

    Xie, T., Zhang, D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T.J., Cheng, Z., Gao, D., Lu, S., et al.: OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. NeurIPS (2024)

    2024

  20. [28]

    EMNLP (2025)

    Xu, R., Ma, K., Yu, W., Zhang, H., Ho, J.C., Yang, C., Yu, D.: Retrieval-augmented GUI agents with generative guidelines. EMNLP (2025)

    2025

  21. [29]

    arXiv preprint arXiv:2601.07779 (2026)

    Yang, B., Jin, K., Wu, Z., Liu, Z., Sun, Q., Li, Z., Xie, J., Liu, Z., Xu, F., Cheng, K., Li, Q., Wang, Y., Qiao, Y., Wang, Z., Ding, Z.: OS-Symphony: A holistic framework for robust and generalist computer-using agent. arXiv preprint arXiv:2601.07779 (2026)

    2026

  22. [30]

    ICLR (2023)

    Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., Cao, Y.: ReAct: Synergizing reasoning and acting in language models. ICLR (2023)

    2023

  23. [31]

    AAAI (2026)

    Zhang, B., Shang, Z., Gao, Z., Zhang, W., Xie, R., Ma, X., Yuan, T., Wu, X., Zhu, S.C., Li, Q.: TongUI: Internet-scale trajectories from multimodal web tutorials for generalized GUI agents. AAAI (2026)

    2026

  24. [32]

    arXiv preprint arXiv:2402.07939 (2024)

    Zhang, C., Li, L., He, S., Zhang, X., Qiao, B., Qin, S., Ma, M., Kang, Y., Lin, Q., Rajmohan, S., Zhang, D., Zhang, Q.: UFO: A UI-focused agent for Windows OS interaction. arXiv preprint arXiv:2402.07939 (2024)

    2024 arXiv

  25. [33]

    arXiv preprint arXiv:2601.07181 (2026)

    Zhang, Y., Guo, X., Goh, Y., Hu, J., Chen, Z., Wang, X., Gao, D., Shou, M.Z.: ShowUI-Aloha: Human-taught GUI agent. arXiv preprint arXiv:2601.07181 (2026)

    2026

  26. [34]

    ICPR (2004) GUIDE 17 A Prompt Structures for Agent Integration This appendix provides the full prompt structures used for injectingGuide knowledge into both integration modes

    Zivkovic, Z.: Improved adaptive Gaussian mixture model for background subtraction. ICPR (2004) GUIDE 17 A Prompt Structures for Agent Integration This appendix provides the full prompt structures used for injectingGuide knowledge into both integration modes. A.1 Mode A: AgentS...

    2004

  27. [35]

    Use the video planning to identify which stage of the workflow you are in and what to do next

    **Thought**: Analyze the current screenshot. Use the video planning to identify which stage of the workflow you are in and what to do next. Use the video grounding to help locate the relevant UI element -- then verify it exists in your current screenshot. Write 2-4 concise sentences

  28. [36]

    **Action**: A short imperative describing what to do

  29. [37]

    name": "computer_use

    A single <tool_call>...</tool_call> block with JSON: {"name": "computer_use", "arguments": {...}}. Rules: - Always start your response with "Thought:". - Use video planning for workflow understanding; use grounding to identify elements. - Always verify elements in your current...

  30. [38]

    Open image window inGIMP

  31. [39]

    review X/Y ppi◀

    Open Image→ScaleImage... review X/Y ppi◀

  32. [40]

    Change ppi to 150ppifor print output◀

  33. [41]

    The user opens Image→Scale Image... review X/Yresolution... choose ppi

    Click Scale to apply Task Needs vs Retrieved:×Needs visual enhancement(sharpen), NOT print ppitweak. Step 2 / 8 ×ppi check (dead end) Agent Thought: The task is to enhance thisphoto...According to retrieved planning: "The user opens Image→Scale Image... review X/Yresolution......

  34. [42]

    Research: browse RawTher-apee

  35. [43]

    Install darktable asRAWprocessor plugin◀

  36. [44]

    Open Image

    Open "Open Image" dia-log... Task Needs vs Retrieved:×Needs format conversion(Export As), NOT plugin in-stall. Step 7 / 13 ×W rong action Agent Thought: The GIMP error says no RAW loader isinstalled.According to the retrievedplanning: "The workflow begins bygathering informati...

Pith tools

Reviewed July 13, 2026 · model on record in the stance chip above.