ReachAgent: Enhancing Mobile Agent via Page Reaching and Operation

Bin Wang; Jian Luan; Qinzhuo Wu; Wei Liu

REVIEW 3 major objections 6 minor 3 cited by

ReachAgent: Enhancing Mobile Agent via Page Reaching and Operation

T0 review · 3 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ReachAgent claims that decomposing mobile tasks into page-reaching and page-operation subtasks, then refining with preference-based reinforcement learning over whole GUI flows, improves task completion.

desk verdict Useful dataset and a sensible reach/operate decomposition, but the headline MobileReach gains need a subpath-overlap check before they convince. read the letter →

arxiv 2502.02955 v1 pith:QS2YWBRC submitted 2025-02-05 cs.CL cs.AI

Qinzhuo Wu , Wei Liu , Jian Luan , Bin Wang This is my paper · ORCID

classification cs.CLcs.AI

keywords mobileAIagentGUIautomationpagereachingoperationdirectpreferenceoptimizationactionalignmentReachvisuallanguagemodel

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that mobile GUI agents fail not because they cannot perform single steps but because they greedily chase task-relevant elements and lose sight of the whole GUI flow. To fix this, the authors build MobileReach, a training set that decomposes each task into page-reaching and page-operation subtasks, and ReachAgent, a two-stage framework: supervised fine-tuning on navigation, reaching, and operation data, followed by direct preference optimization over whole flows ranked by a four-level reward. On the in-house MobileReach benchmark, ReachAgent raises IoU accuracy and text accuracy by 7.12 and 7.69 percentage points at the step level and 4.72 and 4.63 at the task level over the strongest baseline, and it also edges out prior state of the art on the external Auto-UI benchmark. If the results hold, breaking tasks into reach/operate subtasks and rewarding entire successful flows is a recipe that transfers across apps.

What carries the argument

The load-bearing mechanism is the subtask decomposition: every GUI flow is split into 'reach' subtasks (get to a named page by any path) and 'operate' subtasks (reach a page and perform a specific action there). On top of this sit two supporting mechanisms: the action-alignment mechanism, which converts every XML element into a small set of candidate actions anchored at the element's center, and a four-level reward function used to build DPO preference pairs from whole flows, rewarding completion of all subtasks and penalizing longer, incomplete, or invalid flows.

What would settle it

Evaluate the released ReachAgent on a set of tasks written by human users (for example, crowd-sourced instructions for the same 49 apps) and compare task-success rate against the MobileVLM SFT baseline; if the gap shrinks or reverses, the reported advantage is tied to the automatically generated benchmark rather than to general task-completion ability.

Watch

Extended reading notes

Core claim

ReachAgent's central claim is that a mobile agent trained in two stages — first supervised on three task types (page navigation, page reaching, page operation), then refined with DPO on preference data built from a four-level reward (golden, longer-but-completing, incomplete, invalid) — produces GUI flows that complete tasks more often than agents trained only on step-level actions. The action-alignment mechanism contributes by restricting candidate actions to element centers and central scroll axes, shrinking the action space. The paper reports step-level IoU/text gains of +7.12/+7.69 points and task-level gains of +4.72/+4.63 points over fine-tuned MobileVLM on MobileReach, a task-success improvement of +6.59 points, and superior performance across all Auto-UI categories despite no subtask data from that benchmark.

Load-bearing premise

The benchmark's task labels and success criteria are generated automatically from random-walk GUI flows using image captions and a large language model, so the reported gains may not transfer to real user requests if those generated tasks do not match how people actually phrase or pursue mobile tasks.

Editorial extensions

If this is right

Task completion in mobile GUI agents improves when training data is decomposed into reach and operate subtasks rather than only full navigation traces.
Restricting the action space to element centers and scroll axes reduces step-level errors without hurting generalization.
Preference optimization over whole flows, with rewards defined by subtask completion and flow length, raises task-level accuracy more than step-level accuracy.
The gains transfer to an external benchmark (Auto-UI) even when no subtask data from that benchmark is used, suggesting the learned reach/operate abilities are app-general.
The released MobileReach dataset and code let other groups reproduce the two-stage recipe and test it on new apps.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reach/operate decomposition is generic enough that it could be applied to desktop or web agents, not just Android apps, by replacing the XML action space with the equivalent DOM or accessibility tree.
Because success is defined as completing all subtasks, the same four-level reward could be extended to partially-ordered subtasks or multi-condition tasks where subtask completion alone is not sufficient.
A direct test of the approach's robustness would be to evaluate on human-written tasks rather than GPT-4V-generated ones; the paper's own limitation note acknowledges that random-walk sampling may miss app functions.
The action-alignment mechanism's contribution suggests that constraining output coordinates to detected element centers is an inexpensive alternative to more complex grounding heads.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The thing to know about arXiv:2502.02955: the reach/operate decomposition is a genuinely nice idea for mobile GUI agents, and the MobileReach dataset is a real contribution. But the headline 7% gains on MobileReach are not yet credible, because the evaluation may be leaking through subpath overlap between the test flows and the subtask training data.

What's new and good: the paper constructs a large dataset with three task types (navigation, reaching, operation) from the Mobile3M graph, and a two-stage SFT + DPO recipe. The action alignment mechanism (restricting clicks to element centers and scrolls to central axes) is simple and clearly helps. The 4-level reward hierarchy for preference construction is sensible, and the ablation study shows each added component contributes something. The Auto-UI benchmark provides an independent check, although the gains there are small (about 0.3 points overall).

The soft spots are real. First, the main comparison is confounded: ReachAgent-stage1 adds action alignment and two extra training splits relative to the MobileVLM SFT baseline, so the +7% is not a clean measure of the subtask data. The ablation shows action alignment alone accounts for most of the step-level gain; the subtasks add about 1-2 points. Second, the Task Success Rate metric uses the same subtask-completion criterion as the DPO reward, so part of the RL gain is self-referential. The exact-match task accuracy is also reported and improves, which helps, but the concern remains. Third, and most importantly, the stress-test worry about subpath overlap is valid: the test flows are carved out of the same GUI graph, and the reaching/operation training data are deliberately extracted as subpaths of full flows. The paper says whole-flow duplicates were removed, but a test trajectory could appear as a contiguous subpath of a longer training flow. No overlap analysis at the state-action level is reported. If that overlap exists, the step-level and task-level numbers partly reflect memorization of exact transition sequences. The authors need to check this before the in-house results can be taken at face value.

There are no error bars, no significance tests, and no code or data released, which makes it hard to judge stability.

Bottom line: this is a solid engineering paper with a useful dataset and a plausible method. It deserves a serious referee, but the referee should ask for a subpath-overlap analysis, same-data baselines for the ablation, and ideally a release of the artifacts. If those checks come back clean, the paper would be a good fit for a main conference.

Referee Report

3 major / 6 minor

Summary. The paper proposes ReachAgent, a two-stage framework for mobile GUI agents. The authors construct MobileReach, a dataset built from Mobile3M's GUI graph, which decomposes navigation flows into page-reaching and page-operation subtasks. In the first stage, a MobileVLM backbone is fine-tuned on all three task types and uses an action-alignment mechanism that restricts actions to element centers and central axes. In the second stage, a four-level reward function based on subtask completion is used to construct preference pairs and apply DPO. On the MobileReach test split, the paper reports step-level IoU/Text accuracy gains of +7.12%/+7.69% and task-level gains of +4.72%/+4.63% over a MobileVLM SFT baseline, and on Auto-UI it reports small gains over prior SFT models. The claims rely on the validity of the constructed benchmark and on the soundness of the comparisons; the paper includes ablations isolating the action-alignment mechanism, the added subtask datasets, and the RL stage.

Significance. If the reported gains are robust, the paper offers a practical recipe for improving mobile GUI agents: decomposing navigation into reaching and operation subtasks, constraining the action space, and using preference optimization with a subtask-completion reward. The MobileReach dataset is a potentially useful resource, and the ablations are a positive feature because they separate the contributions of the main design choices. However, the significance is tempered by two concerns. First, the headline comparison is confounded because ReachAgent-stage1 differs from the MobileVLM baseline by both the action-alignment mechanism and the two additional training splits, and the ablations show that the alignment mechanism alone accounts for most of the step-level gain. Second, the MobileReach test set is drawn from the same GUI graph used to generate the reaching and operation training data, and the paper does not rule out subpath overlap, so the magnitude of the gains may be inflated by memorization. The Auto-UI results provide some external grounding, but the margins there are small.

major comments (3)

[Section 3.3 and 5.1] The MobileReach test set may overlap with the page-reaching and page-operation training data at the subpath level, and no overlap analysis is reported. The page-reaching and page-operation tasks are extracted as sub-flows of navigation flows (see Figure 7, where an 8-step flow is decomposed into reaching/operation tasks of lengths 2-8), and the test split is described only as 5% of navigation flows removed before construction. A test flow's state-action trajectory could therefore appear as a contiguous subpath of a different flow in the 67,920 reaching flows or 76,252 operation flows. The paper's validity checks ('whether this path has already appeared' and 'whether all actions have already appeared') appear to target full-flow duplication, not subpath coverage. Because the central quantitative claims are based on this test set, the paper must provide an analysis of subpath overlap at the state-action level, or re-split the data to ensure that no test subpath occurs in the reaching/operation training splits.
[Table 1 and Table 5] The headline comparison 'ReachAgent vs MobileVLM SFT' is a multi-variable comparison, not a single-technique improvement. ReachAgent-stage1 adds the action-alignment mechanism and two additional training splits (reaching and operation) on top of the MobileVLM SFT baseline, which is fine-tuned only on the page-navigation split. Table 5 shows that the action-alignment mechanism alone (+Al) accounts for most of the step-level gain (81.55% vs 76.20% IoU, 79.86% vs 74.08% Text), while adding reaching and operation data adds roughly +1.7% step-level and +1.87% task-level on average. The abstract and Section 5.4 attribute the full +7.12%/+7.69% step-level improvement to 'the SOTA agent' comparison, which conflates these factors. The paper should present a single-variable comparison against a baseline that also uses action alignment, or explicitly decompose the reported gain in the abstract and main text.
[Section 4.3 and A.2] The reported Task Success Rate metric is circular with respect to the DPO reward. The DPO preference data are constructed from the four-level reward in Section 4.3, where 'Golden' and 'Longer' are defined by whether the flow completes the instruction (i.e., all subtasks), and 'Incomplete' covers flows that do not. The Task Success Rate in Section A.2 is defined as 'a GUI flow that completes all subtasks as correct.' Thus the +6.59% Task Success Rate gain reported in Section 5.4 is expected from optimizing the model with DPO on exactly this criterion. The non-circular evidence is the exact-match task-level IoU/Text accuracy, which the paper also reports; the discussion should be reworded so that the exact-match metrics, not the Task Success Rate, are presented as the primary evidence for improved task completion.

minor comments (6)

[Section 4.2] In the sentence 'ReachAgent uses a position-aware visual language adapter to align visual representations with text representations, and generate the the action at:', the phrase 'the the' is a typo and should read 'generate the action'.
[Table 4] The baseline name 'CoCo-LLaV A' should be 'CoCo-LLaVA', and in Table 1 'MobileVLMseperate' should be 'MobileVLMseparate'.
[Section 4.3] The sentence 'Figure 4 shows several GUI flows for this task' is incorrect: Figure 4 is the framework overview, not a flow diagram. The authors likely mean Figure 2, which shows multiple GUI flows for the shopping-cart task.
[Table 3] The ablation row '+ action alignment & page reaching & page operation' reports a lower overall Text Acc (53.82%) than '+ action alignment & page operation' alone (54.37%), and a lower Operate SubTask Text Acc (43.33% vs 46.42%). This non-monotonicity is not discussed and deserves an explanation, as it weakens the claim that adding both subtask types is always beneficial.
[Section 5.1] The description 'For the Auto-UI dataset, we follow the official split and method for finetuning' is vague; the appendix later specifies 10% of GoogleApps, but the main text should state this explicitly.
[General] The notation for the model name is inconsistent: 'ReachAgent-stage 1', 'ReachAgent-stage1', 'ReachAgent-stage 2', and 'ReachAgent' are used interchangeably. Please unify these terms.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the DPO reward and Task Success Rate are aligned by design, but this is standard RL rather than a reduction, and the Auto-UI external benchmark independently grounds the central claims.

full rationale

I walked the chain from Mobile3M graph construction (Section 3.2) through subtask extraction (Section 3.3), SFT (Section 4.2), DPO preference construction (Section 4.3), and evaluation (Sections 5.1-5.4). The central claim is empirical: ReachAgent improves IoU/Text accuracy on MobileReach and outperforms baselines on Auto-UI (Tables 1 and 4). No step reduces by construction to its inputs. The 4-level reward (Golden/Longer/Incomplete/Invalid) and the Task Success Rate metric both define success as completing all subtasks (Sections 4.3 and A.2); this is reward-metric alignment, not circularity, because optimizing a stated reward and measuring the same objective is the standard RL setup, and the paper also reports strict task-level accuracy requiring exact match to golden actions, plus the external Auto-UI benchmark on which it did not build subtask data. The reliance on Mobile3M/MobileVLM from the same group is a normal use of a prior dataset/backbone, not a self-citation chain that forces the conclusion. The in-house test split is drawn from the same GUI graph as the training splits, and the paper's own Limitations section notes random-walk coverage and invalid-flow risk; those are benchmark-validity or data-leakage concerns, not circularity. Because the Auto-UI results are externally grounded and the reported MobileReach gains are not shown to be equivalent to the training objective by definition, I find no circular step.

Assumptions & free parameters 5 free parameters · 5 assumptions · 2 invented entities

All central quantities are anchored in assumptions about the Mobile3M graph, LLM-generated task validity, and the subtask-completion success criterion. The metric thresholds (14% IoU margin, 0.8 F1 threshold) are hand-set. No new physical entities are introduced; the reach/operate decomposition and the four-level reward are conceptual constructs whose independent validity is not demonstrated.

free parameters (5)

IoU accuracy margin = 14% of screen size
Actions count as matching if the predicted bounding box intersects the golden box within a 14% screen-size margin (Appendix A.2). The magnitude of the reported IoU Acc depends on this hand-set tolerance.
Input-text F1 threshold = 0.8
Text Acc treats input text as consistent only when F1 is at least 0.8 (Appendix A.2). This threshold changes the Text Acc numbers.
Max test steps = 15
During testing the agent is cut off at 15 interaction steps (Appendix A.3). This cap bounds task success rates and could favor shorter flows.
DPO beta = not reported
The DPO loss in Equation 6 includes beta, but its value is not given. The strength of the RL update is not reproducible from the text.
Similar page criterion for page operation extraction = not specified
Page operation subtasks are extracted when pages before and after an action are 'similar', based on page similarity and co-occurring element counts, but no threshold is given (Section 3.3).

assumptions (5)

domain assumption The Mobile3M random-walk graph covers representative GUI flows for the evaluated apps.
All training and test flows are sampled from this graph (Sections 3.1, 3.2, and 5.1). If the graph omits common user paths, the benchmark underrepresents real usage.
domain assumption GPT-4V and InternVL generated brief tasks and page names are valid ground-truth instructions.
Tasks and page names come from automatic captioning and LLM generation (Section 3.2 and Appendix C). No human validation is reported.
ad hoc to paper Completing all subtasks is a sufficient and correct definition of task success.
Task Success Rate and the DPO reward both use this criterion (Sections 4.3 and 5.2). This is the paper's own definition, not an external benchmark.
domain assumption The golden GUI flows are correct and complete solutions in which every step is necessary.
Task-level accuracy requires exact match to the golden flow (Appendix A.2), and preference data assumes golden actions are optimal (Section 4.3).
ad hoc to paper Clicking element centers and scrolling along central axes covers all useful interactions.
Action alignment restricts the action space to element centers (Section 4.1). This excludes valid off-center touches and may inflate IoU match.

invented entities (2)

Page reaching and page operation subtask decomposition
purpose: Frame every mobile task as a sequence of reaching a page and performing operations on it; used to construct training data, reward levels, and the Task Success Rate metric.
The paper provides no external benchmark showing this decomposition is the correct task abstraction; its validity is assumed throughout.
Four-level reward hierarchy (Golden, Longer, Incomplete, Invalid)
purpose: Assign a preference order to actions for DPO training and define task success.
Reward levels are defined by the authors; no user study or external reward signal validates them.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ReachAgent: Enhancing Mobile Agent via Page Reaching and Operation." pith.science (2026). https://pith.science/paper/QS2YWBRC

@misc{pith2026250202955,
  author       = {Pith},
  title        = {Pith review of: ReachAgent: Enhancing Mobile Agent via Page Reaching and Operation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QS2YWBRC}},
  note         = {Machine review of arXiv:2502.02955}
}

read the original abstract

Recently, mobile AI agents have gained increasing attention. Given a task, mobile AI agents can interact with mobile devices in multiple steps and finally form a GUI flow that solves the task. However, existing agents tend to focus on most task-relevant elements at each step, leading to local optimal solutions and ignoring the overall GUI flow. To address this issue, we constructed a training dataset called MobileReach, which breaks the task into page reaching and operation subtasks. Furthermore, we propose ReachAgent, a two-stage framework that focuses on improving its task-completion abilities. It utilizes the page reaching and page operation subtasks, along with reward-based preference GUI flows, to further enhance the agent. Experimental results show that ReachAgent significantly improves the IoU Acc and Text Acc by 7.12% and 7.69% on the step-level and 4.72% and 4.63% on the task-level compared to the SOTA agent. Our data and code will be released upon acceptance.

Figures

Figures reproduced from arXiv: 2502.02955 by the authors.

**Figure 2.** The complete 9-step GUI flow for a task. Green boxes represent the pages that need to be reached, and [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Actions and tasks for a GUI flow. The step-by-step description provides a set of action history, where each [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: The overview of our proposed ReachAgent. (a) Extracting action space from XML document. (b) In the [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Two cases of generated GUI flow by ReachAgent and MobileVLM. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

**Figure 6.** Figure 6: The prompt for step-by-step description and brief task generation. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

**Figure 7.** Figure 7: Page reaching subtasks and page operation subtasks extracted from the GUI flow example in Figure 3. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

BIMgent: Towards Autonomous Building Modeling via Computer-use Agents
cs.AI 2025-06 conditional novelty 6.0 of 10

BIMgent, a GUI-controlling LLM agent, completes 32% of BIM building modeling tasks end-to-end, outperforming baseline computer-use agents that complete none.
BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanism
cs.CL 2025-05 conditional novelty 6.0 of 10

BacktrackAgent adds a trained error detector and a rewriter to GUI agents, improving task success on Mobile3M and Auto-UI benchmarks.
MapAgent: Trajectory-Constructed Memory-Augmented Planning for Mobile Task Automation
cs.HC 2025-07 conditional novelty 5.0 of 10

A memory-augmented LLM planner that stores and retrieves page-level summaries from past trajectories improves success rates on mobile GUI task benchmarks.

Reference graph

Works this paper leans on

21 extracted references · 17 canonical work pages · cited by 3 Pith papers

[1]

Beauty" subcategory under the

In the "Beauty" subcategory under the "Recommendations" category of the shopping app, click the "Skin Care Set" product

work page
[2]

In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems , pages 1– 14

Vins: Visual search for mobile user interface design. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems , pages 1– 14. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. 2024. Internvl: Scaling up vision foundation models and aligning for generic visual-...

arXiv 2021
[3]

Advertisements

On the product details page, click the "Advertisements" label. Brief Task: Help me find a new skin care set that is on sale. Here is what you need to answer: Input: {chain_description} Please generate "Step-by-step description" and "Brief Task": Output: Figure 6: The prompt for step-by-step description and brief task generation. epoch. Similar to its offi...

work page
[4]

Step-by-step description

Android in the zoo: Chain-of-action-thought for gui agents. arXiv preprint arXiv:2403.02713. A Experiment Settings A.1 Baselines ReachAgent was compared to four other baselines as follows: GPT-4o, Qwen-VL, Auto-UI, Mo- bileVLM. • GPT-4o, GPT-4V (OpenAI, 2023) are most advanced VLMs currently available. • Qwen-VL, Qwen-VL-Max (Bai et al., 2023) are large-s...

arXiv 2023
[5]

Add to Cart

On the detailed information page, select and click "Add to Cart" to enter the parameter page of the phone

work page
[6]

New Arrivals

On the search results page for Skin Care Sets, continue to click on the product with the "New Arrivals" label

work page
[7]

On the parameter page, select “white" to confirm the parameters of the phone

work page
[8]

search”, [177,96][273,168]) P_2 a_2, input(“water purifier

First, we will observe such a GUI flow: GUI PageAction P_1 a_1, click(“search”, [177,96][273,168]) P_2 a_2, input(“water purifier”, [231,72][555,168], “xiaomi 14”) P_3 a_3, click(“search”, [597,48][702,192]) P_4 a_4, click(“Xiaomi 14”, [425,1074][628,1125]) P_5 a_5, click(“Add to cart”, [294,1122][429,1154]) P_6 a_6, scroll([0,585][720,1088], “up”) P_7 a_...

work page

Show all 21 references

[9]

Check the validity • Whether this path of P1, P2, ..., P9 has already appeared in the current training set

Then, we need to check the validity of this flow, generate corresponding tasks, and filter. Check the validity • Whether this path of P1, P2, ..., P9 has already appeared in the current training set. • Whether all actions in the action sequence have already appeared in the cur...
[10]

On the homepage of Xiaomi Mall, click the search icon to enter the search page
[11]

xiaomi 14

On the search page, enter "xiaomi 14" in the search box to search
[12]

On the search page, click the search icon to search
[13]

xiaomi 14

On the search results page, select the detailed information of "xiaomi 14"
[15]

xiaomi 14

On the parameter page, scroll the page to view more parameter information. Page Reaching Task Page Operation Task Actions On the search page, enter "xiaomi 14" in the search box to search click(“search”, [177,96][273,168]) input(“water purifier”, [231,72][555,168], “xiaomi 14”...
[17]

Brief Task: Help me find detailed information about xiaomi 14, and add a white one to the shopping cart

On the parameter page, click the "Confirm" button at the bottom to confirm the parameter configuration of the phone. Brief Task: Help me find detailed information about xiaomi 14, and add a white one to the shopping cart. Here, each step si corresponds to a GUI Page Pi and an ...
[18]

xiaomi 14

If a page is referred to by a unique name in the step-by-step description, we split out the sub-page flow leading to that page and assign a task through template to reach that page. For example: Description: 4. On the search results page, select the detailed information of "xi...
[19]

Xiaomi 14

If the element name of a click action does not appear in the existing dataset, we name the next page of clicking on the element with that element name and assign a task to reach that page. For example: Action: a4, click(“Xiaomi 14”, [425,1074][628, 1125]). Task: Help me naviga...
[20]

Action: a6, scroll([0,585][720,1088], “up”)

If the action type is scroll or input, we assign an operation task as referenced in the step-by-step description, including reaching the current page and performing this action. Action: a6, scroll([0,585][720,1088], “up”). Task: On the parameter page, scroll the page to view m...
[21]

Similar Pages: P7, P8 Task: On the parameter page, select “white" to confirm the parameters of the phone

If an action has similar GUI pages before and after execution, we assign an operation task ac- cording to the step-by-step description, describing the process of reaching this page to perform this action. Similar Pages: P7, P8 Task: On the parameter page, select “white" to con...
[2021]

arXiv preprint arXiv:2107.13731

Uibert: Learning generic multimodal representations for ui understanding. arXiv preprint arXiv:2107.13731. Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. 2024. Digirl: Training in-the-wild device-control agents with autonomous reinforc...

2024 arXiv
[2024]

Advances in Neural Information Processing Systems, 36

Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36. Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. 2023. Android in the wild: A large-scale dataset for andro...

2023 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

Beauty" subcategory under the

In the "Beauty" subcategory under the "Recommendations" category of the shopping app, click the "Skin Care Set" product

work page

[2] [2]

In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems , pages 1– 14

Vins: Visual search for mobile user interface design. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems , pages 1– 14. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. 2024. Internvl: Scaling up vision foundation models and aligning for generic visual-...

arXiv 2021

[3] [3]

Advertisements

On the product details page, click the "Advertisements" label. Brief Task: Help me find a new skin care set that is on sale. Here is what you need to answer: Input: {chain_description} Please generate "Step-by-step description" and "Brief Task": Output: Figure 6: The prompt for step-by-step description and brief task generation. epoch. Similar to its offi...

work page

[4] [4]

Step-by-step description

Android in the zoo: Chain-of-action-thought for gui agents. arXiv preprint arXiv:2403.02713. A Experiment Settings A.1 Baselines ReachAgent was compared to four other baselines as follows: GPT-4o, Qwen-VL, Auto-UI, Mo- bileVLM. • GPT-4o, GPT-4V (OpenAI, 2023) are most advanced VLMs currently available. • Qwen-VL, Qwen-VL-Max (Bai et al., 2023) are large-s...

arXiv 2023

[5] [5]

Add to Cart

On the detailed information page, select and click "Add to Cart" to enter the parameter page of the phone

work page

[6] [6]

New Arrivals

On the search results page for Skin Care Sets, continue to click on the product with the "New Arrivals" label

work page

[7] [7]

On the parameter page, select “white" to confirm the parameters of the phone

work page

[8] [8]

search”, [177,96][273,168]) P_2 a_2, input(“water purifier

First, we will observe such a GUI flow: GUI PageAction P_1 a_1, click(“search”, [177,96][273,168]) P_2 a_2, input(“water purifier”, [231,72][555,168], “xiaomi 14”) P_3 a_3, click(“search”, [597,48][702,192]) P_4 a_4, click(“Xiaomi 14”, [425,1074][628,1125]) P_5 a_5, click(“Add to cart”, [294,1122][429,1154]) P_6 a_6, scroll([0,585][720,1088], “up”) P_7 a_...

work page