Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agent

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GUI-explorer claims that a GUI agent can reach state-of-the-art task success by autonomously exploring apps and mining transition-aware screen-operation knowledge, with no parameter updates.

desk verdict A clever training-free GUI-knowledge pipeline whose headline numbers are currently uninterpretable because the exploration data and the benchmarks are not shown to be disjoint; deserves review but needs a decontamination check. read the letter →

arxiv 2505.16827 v1 pith:2VKGUAMM submitted 2025-05-22 cs.AI

Bin Xie , Rui Shao , Gongwei Chen , Kaiwen Zhou , Yinchuan Li , Jie Liu , Min Zhang , Liqiang Nie This is my paper · ORCID
classification cs.AI
keywords GUIagentautonomousexplorationtransition-awareknowledgevisual-semanticretrievalGUI-KRBbenchmarkSPA-BenchAndroidWorldmultimodallargelanguagemodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

GUI-explorer is a training-free GUI agent that tries to solve two failures of multimodal language models operating phones: misreading UI icons and acting on outdated app knowledge. It does so by first exploring each app autonomously, using goals derived from the app's declared activity list, and then mining the collected (observation, action, outcome) traces in an unsupervised way into a visual-semantic knowledge store. At task time, the agent retrieves and ranks entries relevant to the current screen and folds them into its reasoning prompt as Dynamic Guidance. The paper reports 53.7% task success on SPA-Bench single-app English Level 3 tasks and 47.4% on AndroidWorld, beating the cited baselines, and a 16.0 percentage-point reduction in prior-knowledge errors on its new GUI-KRB benchmark. If the benchmark scores are fair, the method matters because it upgrades an agent for a new app without any parameter updates or human-written demonstrations.

What carries the argument

The load-bearing object is the transition-aware knowledge store $K$: a multimodal index built from every exploration trajectory $\xi = \langle o_1, a_1, \dots, o_n, a_n\rangle$ by a function $F_{\mathrm{extract}}$ that maps each valid transition $(o_i, a_i, o_{i+1})$ to a key--value pair $\{k_i : v_i\}$, where $k_i$ is the visual patch of the element and $v_i$ is its operational description. Invalid transitions (where the observation does not perceptibly change) are discarded by perceptual hashing, and a refinement rule merges or inserts entries based on cosine similarity thresholds. The companion mechanism is Dynamic Guidance: at each step the agent extracts UI elements from the current screen, retrieves related entries from $K$, sorts them with an instruction-aware pairwise comparator implemented as a modified merge sort, and prepends the sorted knowledge to its reasoning prompt. This pair of mechanisms carries the direct causal claim that retrieval of mined transition knowledge, rather than model capacity or finetuning, produces the reported success rates.

What would settle it

Look for exact or near-exact matches between the knowledge store entries and the benchmark tasks' annotated gold steps (screenshot hash plus action). If an entry for a tested app replicates the test answer or gold action sequence, then the reported success rates are evidence of memorization, not of generalizable transition knowledge.

Watch

Extended reading notes

Core claim

The central claim is that the bottleneck in GUI automation is not the base model's reasoning but its lack of reliably grounded, current knowledge about specific interfaces, and that this knowledge can be manufactured automatically. GUI-explorer's autonomous exploration phase generates function-aware tasks from structural anchors (the activities declared in an app's manifest) and executes them with depth-first search, yielding diverse trajectories. From these, the Transition-aware Knowledge Extractor converts each valid state transition into an entry pairing a visual patch of the interacted element with a natural-language description of what that element does. At inference, visual-semantic retrieval pulls candidate entries for the visible elements, and a pairwise multimodal-language-model ranker orders them by relevance to the user instruction before they are injected into the reasoning prompt. The paper argues that this suppresses misinterpretation of UI components and replaces stale priors with app-specific transition-aware knowledge, a claim supported by the reported gains over the cited exploration-based, RL-trained, and zero-shot baselines.

Load-bearing premise

The whole gain rests on the assumption that the knowledge store built by exploring an app does not already contain the annotated answers or gold solutions used to score that same app on the benchmarks.

Editorial extensions

If this is right

  • Agents can be adapted to a new app by running the exploration phase once; no fine-tuning or human-collected demonstrations are required, so app updates can be absorbed by re-exploring rather than retraining.
  • The 16.0 percentage-point drop in prior-knowledge errors on GUI-KRB indicates that the method targets a specific and measurable failure mode: wrong or stale assumptions about what an icon does.
  • Because the knowledge is keyed to visual patches rather than element IDs, it can be transferred across UI states and, to a limited degree, across environments; the paper's cross-environment ablation shows a 4.3% gain on SPA-Bench from knowledge mined on AndroidWorld.
  • The 53.7% and 47.4% success rates on SPA-Bench and AndroidWorld set new reference numbers that future training-free and exploration-based agents on these benchmarks will be measured against.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: a disjoint-app re-run (knowledge store built from apps that never appear in the test split) would isolate the store's generalizability, since the paper's reported gains could in principle include memorized app-specific states.
  • Editorial extension: the appendix's small Android-to-web transfer experiment suggests cross-platform reuse of transition knowledge is plausible, but a proper test would scale to dozens of web tasks and control for the same base model, prompt template, and ranking budget.
  • Editorial extension: because the method is training-free and knowledge is stored as text-plus-visual patches, it can be audited directly: one can inspect every retrieved entry to see whether it states a wrong operation or encodes a test answer; such an audit would either strengthen or falsify the claim that the gains are knowledge-driven.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes GUI-explorer, a training-free GUI agent that (1) autonomously explores an application using a Function-aware Task Goal Generator guided by manifest-declared activity anchors and depth-first search; (2) mines 'transition-aware knowledge' as (visual patch, functionality) pairs from state-action transitions via an unsupervised extractor; and (3) at inference time performs visual-semantic retrieval of this knowledge to build dynamic guidance prompts for an MLLM (GPT-4o). The authors report 53.7% task success on SPA-Bench single-app English Level 3 tasks, 47.4% on AndroidWorld, and a 16.0-percentage-point reduction in prior-knowledge errors on a newly introduced GUI-KRB benchmark (500 samples, 43 apps). A clean ablation (Figure 4) shows that removing dynamic guidance drops SPA-Bench success by 12.2 percentage points, and cross-environment guidance still helps by 4.3 points. The method is open-sourced.

Significance. If the results hold, the contribution is significant: an automatic, unsupervised, training-free knowledge acquisition pipeline that is compatible with any MLLM and shows consistent gains over zero-shot and exploration-based baselines. The ablation design is a genuine strength because it isolates the knowledge component from the base model. The introduced GUI-KRB benchmark could be useful to the community for evaluating UI understanding, provided the benchmark is properly separated from the exploration data used to build the knowledge store. The main significance is conditional: the headline empirical claims currently rest on an untested assumption of information disjointness between exploration and evaluation, and on single-run evaluations without error bars.

major comments (4)
  1. [Section 3.1, Algorithm 1, Figure 4, Table 1] The headline SPA-Bench result (53.7%) is obtained with a knowledge store built from in-environment exploration on SPA-Bench itself, as explicitly labeled in Figure 4 ('in-environment exploration (SPA-Bench)'). The Task_Generator prompt in Appendix I.1 is designed to produce concrete tasks such as 'Search for "ocean waves" white noise, then sort by most played', which closely resembles SPA-Bench Level 3 instructions. The paper reports no overlap check between the 1,300+ knowledge items (Section 5.1.2) and the SPA-Bench test tasks, e.g., no task-text similarity, screenshot perceptual hashing, or element-level deduplication. Because retrieval can then return the exact action-effect pair that the evaluation later asks the agent to discover, the 53.7% number does not by itself demonstrate generalization. Please provide a concrete disjointness analysis (or, stronger, evaluate with the knowledge store built only from held-out apps/tasks) and rerun Table 1 under that strict separation.
  2. [Section 4.2, Appendix I.2 vs J.2, Table 4] The GUI-KRB evaluation is circular. The benchmark is constructed from over 300 trajectories collected in the same mobile environment (Section 4.2), and its Dynamic Comprehension task uses the very same transition-extraction prompt as the knowledge extractor (Appendix I.2 vs J.2). When GUI-explorer is evaluated on GUI-KRB, the retrieval component searches a knowledge vector store built from exploration trajectories; the retrieved entry for a UI element can therefore be the annotated functionality description that the evaluation is supposed to measure. The claimed 16.0% prior-knowledge error reduction (22.8% to 6.8% in Table 4) is uninterpretable without evidence that none of the 500 GUI-KRB elements, or their before/after screenshots, appear in the trajectories used to populate K. Please perform and report an element-level and screen-level overlap check, release the overlap-checking tool, and if necessary rebuild the benchmark from a disjoint trajectory set.
  3. [Abstract, Table 1] The abstract states 'a task success rate of 53.7% on SPA-Bench', but Table 1 reports results only for 'SPA-Bench single-app English Level 3 tasks'. SPA-Bench also contains multi-app tasks, other levels, and non-English variants. The claim as written overstates the measured scope. Please qualify the abstract to match the reported configuration, and if other SPA-Bench splits were evaluated, report them or state they were not.
  4. [Section 5.2, Tables 1 and 3] No repeated runs, seeds, or error bars are reported for any agent evaluation. The reported improvements (e.g., 42.0% to 53.7% on SPA-Bench, 40.5% to 47.4% on AndroidWorld) lack even a single standard deviation, so it is impossible to assess whether the gaps are meaningful given the stochasticity of GPT-4o-based agents and the environment. Please report at least three runs per condition with mean and standard deviation, or a statistical significance test for the main comparisons; if only one run is feasible per condition, state that explicitly as a limitation.
minor comments (4)
  1. [Section 5.2] The sentence 'This represents a 28.1% absolute improvement over M3A' is inconsistent with Table 1: 53.7 - 42.0 = 11.7 percentage points (or about 27.9% relative). Please correct the wording.
  2. [Table 4] The column header 'Dynamic Comprehen-sion Rrror Rate' contains a typo; it should be 'Error Rate'.
  3. [Section 3.1] The text says 'Each d-depth exploration with branching factor b yields O(bd) distinct trajectories' and later calls this 'Quadratic Coverage'; this should be O(b^d), which is exponential, not quadratic.
  4. [Section 5.3] The claim 'the ranking component reduced the error rate by 3%' should be '3 percentage points', since Table 4 reports 9.8% vs 6.8%.

Circularity Check

2 steps flagged · score 6.0 of 10

Partial circularity: the GUI-KRB Dynamic Comprehension task is defined as the paper's own knowledge-extraction operation, and the SPA-Bench/AndroidWorld gains come from a knowledge store populated by in-environment exploration of the same apps.

  1. self definitional [Eq. (1), Section 4.1, Section 5.3, Appendix J.2]
    "Fextract : (oi, ai, oi+1)→ {ki :v i} ... J.2 Prompting Template of Dynamic Comprehension Task Same as Appendix I.2. ... This task evaluates reasoning about cause-effect logic in GUI interactions. It simulates the knowledge extraction method we use in this paper."

    The benchmark's Dynamic Comprehension task is the same transition-extraction operation used to build the knowledge store K: Eq. (1) defines Fextract, and Appendix J.2 reuses the extractor prompt verbatim from Appendix I.2. Evaluating GUI-explorer on this task therefore measures the method against its own definition rather than against an independent criterion. Since K and GUI-KRB are both built from trajectories collected in the same mobile environment, retrieval can return the very transition-aware entry the benchmark asks the model to reproduce; no overlap check between the 1,300 knowledge items and the 500 GUI-KRB samples is reported.

  2. fitted input called prediction [Section 3.1, Figure 4, Appendix I.1, Tables 1 and 3]
    "(3) Guidance generated through in-environment exploration (SPA-Bench). ... Are concrete and specific (like searching for a particular item rather than just 'search') ... Example tasks from other apps (for reference only): 1. Search for "ocean waves" white noise, then sort results by most played"

    The knowledge store K is populated by autonomous exploration of the same benchmark applications that are later used for evaluation. The task generator is explicitly prompted to emit concrete, benchmark-like task instructions, yet the paper provides no check that generated exploration tasks are disjoint from SPA-Bench/AndroidWorld test tasks. The main 12.2% improvement attributed to dynamic guidance (Figure 4) can therefore be explained by retrieving action-effect pairs memorized during in-environment exploration, rather than by zero-shot generalization; the cross-environment experiment shows only a 4.3% gain, consistent with much of the benefit coming from in-environment lookup.

full rationale

The most serious circularity is in the GUI-KRB validation. Section 4.1 defines the Dynamic Comprehension task as 'simulat[ing] the knowledge extraction method we use in this paper', and Appendix J.2 reuses the exact extractor prompt from Appendix I.2. A model augmented with the paper's own knowledge store is then evaluated on the same operation that built the store. If any benchmark element (or a perceptually similar duplicate) is present in K, the 'similar UI elements' functionalities retrieved into the prompt can contain the annotated answer. The paper reports no deduplication or overlap check between the 1,300 knowledge items and the 500 GUI-KRB samples, so the claimed 16.0%/13.4% error-rate reductions are not interpretable as independent reasoning gains. For SPA-Bench and AndroidWorld, the issue is slightly weaker but still load-bearing: K is constructed by exploring the very apps used for testing, with a generator instructed to produce concrete user tasks. Without a demonstrated disjointness guarantee, the headline 53.7% and 47.4% success rates may be substantially memorization-based. Self-citations to the authors' own SPA-Bench paper are not themselves the problem, since SPA-Bench is a public benchmark with automated evaluation. Overall, the central claims are partially circular: one evaluation task is identical to the method by construction, and the main benchmark gains rely on an unexamined overlap between the exploration store and the test environment.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central contribution is empirical. The method rests on several hand-set exploration thresholds and on the assumption that MLLM-generated state-transition descriptions are accurate enough to guide actions. No new physical or scientific entities are introduced. The main unstated premise is that exploration data and evaluation data do not overlap in a way that leaks solutions or annotated answers.

free parameters (6)
  • DFS branching factor b = 10
    Exploration tree breadth chosen by hand; affects the number of trajectories and the coverage of app functionality (Section 5.1.2).
  • DFS max depth d = 5
    Exploration tree depth chosen by hand; controls how deeply nested exploration tasks are generated (Section 5.1.2).
  • step limit s = 30
    Per-task step cap during exploration; balances coverage and computational cost (Section 5.1.2).
  • key similarity threshold delta_k = 0.99
    Cosine threshold for treating two visual keys as the same knowledge entry in the refinement rule (Eq. 4).
  • value merge threshold delta_v = 0.1
    Cosine threshold below which new knowledge text is merged into an existing entry (Eq. 4).
  • GUI-KRB correctness threshold = 50% of expert keywords
    A response is correct if it contains at least 50 percent of expert-identified keywords (Section 4.1); this directly determines all GUI-KRB error rates.
assumptions (5)
  • domain assumption MLLM-generated functionality descriptions from before/after screenshots are sufficiently accurate to serve as operational knowledge.
    The entire knowledge store quality depends on GPT-4o's ability to infer action-effect relationships from two screenshots (Section 3.2, Appendix I.2).
  • domain assumption An unchanged screen, measured by perceptual hash similarity, implies an ineffective action.
    Transition Filtering discards transitions where the observation does not change; toggles, silent state changes, or animations that do not alter the screenshot are treated as invalid (Section 3.2).
  • domain assumption Manifest-declared activities are valid and complete functional anchors for task generation.
    Exploration Anchors come from app manifests; if the manifest omits or misnames features, generated tasks miss functionality (Section 3.1, Appendix D).
  • domain assumption The environment supports restoring to previous states via E.restore_to during exploration.
    DFS exploration relies on rollback to a saved state; on a real device this may fail for stateful or network-dependent apps (Algorithm 1).
  • ad hoc to paper SigLIP cosine similarity is a reliable proxy for visual and semantic equivalence of UI elements and knowledge text.
    Retrieval and knowledge merging use cosine similarity on SigLIP embeddings with hand-set thresholds; no evidence is provided that this embedding space preserves UI functional similarity (Sections 3.2, 3.3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agent." pith.science (2026). https://pith.science/paper/2VKGUAMM

@misc{pith2026250516827,
  author       = {Pith},
  title        = {Pith review of: GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agent},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2VKGUAMM}},
  note         = {Machine review of arXiv:2505.16827}
}
read the original abstract

GUI automation faces critical challenges in dynamic environments. MLLMs suffer from two key issues: misinterpreting UI components and outdated knowledge. Traditional fine-tuning methods are costly for app-specific knowledge updates. We propose GUI-explorer, a training-free GUI agent that incorporates two fundamental mechanisms: (1) Autonomous Exploration of Function-aware Trajectory. To comprehensively cover all application functionalities, we design a Function-aware Task Goal Generator that automatically constructs exploration goals by analyzing GUI structural information (e.g., screenshots and activity hierarchies). This enables systematic exploration to collect diverse trajectories. (2) Unsupervised Mining of Transition-aware Knowledge. To establish precise screen-operation logic, we develop a Transition-aware Knowledge Extractor that extracts effective screen-operation logic through unsupervised analysis the state transition of structured interaction triples (observation, action, outcome). This eliminates the need for human involvement in knowledge extraction. With a task success rate of 53.7% on SPA-Bench and 47.4% on AndroidWorld, GUI-explorer shows significant improvements over SOTA agents. It requires no parameter updates for new apps. GUI-explorer is open-sourced and publicly available at https://github.com/JiuTian-VL/GUI-explorer.

Figures

Figures reproduced from arXiv: 2505.16827 by the authors.

Figure 1
Figure 1. Comparison of GPT-4o and an user’s interpre [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of GUI-explorer. (a) Automatically constructing function-aware exploration goals by analyzing [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Without transition-aware knowledge as re [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Ablation study of operational guidance config [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Distribution of apps in GUI-KRB [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: A comprehensive sample from GUI-KRB [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Distribution of transition-aware knowledge [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skills

    cs.AI 2025-06 conditional novelty 5.0 of 10

    Mirage-1 combines a hierarchical multimodal skill memory with a skill-augmented Monte Carlo tree search to outperform prior GUI agents on Android and web online benchmarks.

Reference graph

Works this paper leans on

46 extracted references · 42 canonical work pages · cited by 1 Pith paper

  1. [1]

    Tap ‘Most played’, then share the top song to Instagram Story

    “Tap ‘Most played’, then share the top song to Instagram Story.” (Targeting anchor:ShareInstagramStory)

    work page

  2. [2]

    InThe Thirteenth International Conference on Learning Representations

    Navigating the digital world as humans do: Universal visual grounding for GUI agents. InThe Thirteenth International Conference on Learning Representations. Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. 2024. Cogagent: A visual language model for gui agents. InProceedings of ...

    arXiv 2024

  3. [3]

    Utilize the app’s full feature set based on the activity list

    work page

  4. [4]

    Rui Shao, Tianxing Wu, Jianlong Wu, Liqiang Nie, and Ziwei Liu

    Deepfake-adapter: Dual-level adapter for deep- fake detection.International Journal of Computer Vision, pages 1–16. Rui Shao, Tianxing Wu, Jianlong Wu, Liqiang Nie, and Ziwei Liu. 2024. Detecting and grounding multi-modal media manipulation and beyond.IEEE Transactions on Pattern Analysis and Machine Intel- ligence. Leyang Shen, Gongwei Chen, Rui Shao, We...

    arXiv 2024

  5. [5]

    Cover different user interaction patterns (viewing, editing, sharing, etc.)

    work page

  6. [6]

    modify" and

    Appagent: Multimodal agents as smartphone users. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI ’25, New York, NY , USA. Association for Computing Machinery. Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024. Gpt-4v(ision) is a generalist web agent, if grounded. InForty-first International Conference on Ma...

    work page 2025

  7. [7]

    Represent realistic user behaviors and goals

    work page

  8. [8]

    Tap the settings icon, then navigate to Drive Mode and enable it

    “Tap the settings icon, then navigate to Drive Mode and enable it.” (Targeting anchor:DriveModeActivity) E Error Analysis In this section, we categorize and discuss three primary error types observed in our evaluation tra- jectories, detailing their component-level manifes- tations and root causes. E.1 Perceptual Errors Perceptual errors occur when agents...

    work page 2023

Show all 46 references
  1. [9]

    Start from the current screen shown in the screenshot

  2. [10]

    Can be completed within 10-30 steps

  3. [12]

    Are concrete and specific (like searching for a particular item rather than just "search")

  4. [14]

    Include both basic and advanced features

  5. [16]

    Avoid excessive steps on form-filling or scrolling pages Important context: - App name: app name - Package name: package name - Available activities (app screens/features): activity list Format requirements:

  6. [17]

    List only the tasks without explanations or commentary

  7. [18]

    Each task should be a single, clear directive

  8. [19]

    Use specific examples (e.g., concrete search terms, actions, settings)

  9. [20]

    Include the expected outcome where relevant

  10. [21]

    Tasks should follow this pattern: [Starting action] + [Specific steps] + [End goal] Example tasks from other apps (for reference only):

  11. [22]

    ocean waves

    Search for "ocean waves" white noise, then sort results by most played

  12. [23]

    Great content!

    Open the first recommended video, then post "Great content!" as a comment

  13. [24]

    Watch Later

    Play the trending video, then add it to your "Watch Later" playlist

  14. [25]

    I.2 Prompting Template of Unsupervised Mining of Transition-aware Knowledge Objective: Describe the functionality of a specific UI element in a mobile app screenshot

    Navigate to the comments section of a featured video, then like the top comment Generate diverse tasks that would help a user explore and utilize all major features visible in the screenshot and implied by the activity list. I.2 Prompting Template of Unsupervised Mining of Tra...

  15. [29]

    UI element

    Use generic terms like "UI element" or appropriate pronouns Example: - Incorrect: "Tapping the element #3 displays David’s saved recipes in the results panel" - Correct: "Tapping this element will initiates a search and displays matching results" Guidance: - Describe the core ...

  16. [30]

    action_type

    Status Operations: - Task Complete: {"action_type": "status", "goal_status": "complete"} - Task Infeasible: {"action_type": "status", "goal_status": "infeasible"}

  17. [31]

    action_type

    Information Actions: - Answer Question: {"action_type": "answer", "text": "<answer_text>"}

  18. [32]

    action_type

    Screen Interactions: - Tap Element: {"action_type": "click", "index": <visible_index>} - Long Press: {"action_type": "long_press", "index": <visible_index>} - Scroll: Scroll the screen or a specific scrollable UI element. Use the ‘index‘ of the target element if scrolling a sp...

  19. [33]

    action_type

    Input Operations: - Text Entry: {"action_type": "input_text", "text": "<content>", "index": <text_field_index>} - Keyboard Enter: {"action_type": "keyboard_enter"}

  20. [34]

    action_type

    Navigation: - Home Screen: {"action_type": "navigate_home"} - Back Navigation: {"action_type": "navigate_back"}

  21. [35]

    action_type

    System Actions: - Launch App: {"action_type": "open_app", "app_name": "<exact_name>"} - Wait Refresh: {"action_type": "wait"} ## Current Objective User Goal: task goal ## Execution Context Action History: history Visible UI Elements (Only interact with *visible=true elements):...

  22. [36]

    - Verify element visibility (‘visible=true‘) before attempting any interaction (click, long_press, input_text)

    Path Optimization: - Prefer direct methods (e.g., open_app > app drawer navigation) - Always use the ‘input_text‘ action for entering text into designated text fields. - Verify element visibility (‘visible=true‘) before attempting any interaction (click, long_press, input_text...

  23. [37]

    Error Handling Protocol: - Switch approach after≥2failed attempts - Prioritize scrolling (‘scroll‘ action) over force-acting on invisible elements - If an element is not visible, use ‘scroll‘ in the likely direction (e.g., ’down’ to find elements below the current view). - Try...

  24. [38]

    Information Tasks: - MANDATORY: Use answer action for questions - Verify data freshness (e.g., check calendar date) ## Expert Techniques Here are some tips for you: knowledge ## Response Format STRICTLY follow: Reasoning: [Step-by-step analysis covering: - Visibility verificat...

  25. [42]

    UI element

    Use generic terms like "UI element" or appropriate pronouns Example: - Incorrect: "Tapping the element #3 displays David’s saved recipes in the results panel" - Correct: "Tapping this element will initiates a search and displays matching results" Guidance: - Describe the core ...

  26. [48]

    Tapping the element #3 displays David’s saved recipes in the results panel

    Only incorporate relevant patterns from similar elements if they align with the current context Example: - Incorrect: "Tapping the element #3 displays David’s saved recipes in the results panel" - Correct: "Tapping this element will initiates a search and displays matching res...

  27. [53]

    Consider similar elements’ functionalities as reference, but prioritize: - Current screen context - UI element attributes - Task description

  28. [54]

    Tapping the element #3 displays David’s saved recipes in the results panel

    Only incorporate relevant patterns from similar elements if they align with the current context Example: - Incorrect: "Tapping the element #3 displays David’s saved recipes in the results panel" - Correct: "Tapping this element will initiates a search and displays matching res...

  29. [60]

    Tapping the element #3 displays David’s saved recipes in the results panel

    Only incorporate relevant patterns from similar elements if they align with the current context Example: - Incorrect: "Tapping the element #3 displays David’s saved recipes in the results panel" - Correct: "Tapping this element will initiates a search and displays matching res...

  30. [61]

    Concise: 1-2 sentences

  31. [62]

    Focus on general function, not specific details

  32. [63]

    Avoid mentioning the numeric tag

  33. [64]

    UI element

    Use generic terms like "UI element" or appropriate pronouns

  34. [65]

    Consider similar elements’ functionalities as reference, with priority: - Higher-ranked (more relevant) reference functionalities - Current screen context - UI element attributes - Task description

  35. [66]

    Tapping the element #3 displays David’s saved recipes in the results panel

    Only incorporate relevant patterns from similar elements if they align with the current context Example: - Incorrect: "Tapping the element #3 displays David’s saved recipes in the results panel" - Correct: "Tapping this element will initiates a search and displays matching res...

  36. [2019]

    InProceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pages 10023– 10031

    Multi-adversarial discriminative deep domain generalization for face presentation attack detection. InProceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pages 10023– 10031. Rui Shao, Tianxing Wu, and Ziwei Liu. 2023. Detecting and grounding mu...

    2023

  37. [2025]

    Gongwei Chen, Leyang Shen, Rui Shao, Xiang Deng, and Liqiang Nie

    Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923. Gongwei Chen, Leyang Shen, Rui Shao, Xiang Deng, and Liqiang Nie. 2024. Lion: Empowering multi- modal large language model with dual-level visual knowledge. InProceedings of the IEEE/CVF Con- ference on Computer Visi...

    2024 arXiv

  38. [2710]

    Curran Associates, Inc. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhi- hao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024b. Qwen2-vl: Enh...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.