Pith. sign in

REVIEW 3 major objections 4 minor 23 references

TRISHUL: Towards Region Identification and Screen Hierarchy Understanding for Large VLM based GUI Agents

T0 review · 3 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TRISHUL shows that a training-free, prompt-based pipeline of hierarchical screen parsing and spatially enhanced descriptions lets generalist vision-language models match or beat trained GUI agents on both action grounding and GUI referring.

desk verdict TRISHUL is a genuinely useful training-free GUI grounding pipeline, but as submitted the headline 'superior across AITW and Mind2Web' is not supported because the strongest baseline is excluded from exactly those benchmarks, and two tables report conflicting numbers for the same GPT-4V conditions. read the letter →

arxiv 2502.08226 v2 pith:4DAPOLSI submitted 2025-02-12 cs.CV cs.AIcs.LG

Kunal Singh , Shreyas Singh , Mukund Khanna This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords GUIagentsactiongroundingreferringhierarchicalscreenparsingspatiallyenhancedelementdescriptionset-of-markspromptingtraining-free
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a capable GUI agent does not need task-specific training. TRISHUL wraps a generalist vision-language model in two prompt-based modules: Hierarchical Screen Parsing, which groups on-screen elements into broad regions and local widgets, and Spatially Enhanced Element Description, which attaches function labels to icons by reading nearby text. With GPT-4V and GPT-4o as backbones, this training-free pipeline claims action-grounding accuracy that beats trained agents such as SeeClick and CogAgent on ScreenSpot and VisualWebBench, rivals OmniParser, and achieves the best image-only result on AITW, while also beating the Tree-of-Lens agent on the Screen Point-and-Read referring benchmark. The significance is that prompts and structure can substitute for dataset-specific fine-tuning, making GUI agents easier to deploy across platforms.

What carries the argument

The load-bearing mechanism is the pair of modules: Hierarchical Screen Parsing (HSP) and Spatially Enhanced Element Description (SEED). HSP runs SAM and OCR over the screenshot, thresholds boxes by area into Global Regions of Interest (GROIs) and local elements, scores each GROI by an information score (number of contained boxes over area adjusted by intersections), and applies non-maximum suppression to yield a clean hierarchy. SEED then takes the local elements with their OCR text, pairs nearby icons with text, labels each element as paired, standalone, actionable text, or picture, and generates a spatially grounded functional description for every element; these descriptions feed a Set-of-Marks prompt for grounding and the two-lens prompt for referring.

What would settle it

Run the released TRISHUL pipeline and the public OmniParser weights on ScreenSpot, AITW, and Mind2Web under identical API settings; if OmniParser's numbers exceed those in Table 2, or its AITW and Mind2Web results beat TRISHUL's, the claimed superiority over trained parsing is falsified.

Watch

Extended reading notes

Core claim

TRISHUL establishes that a training-free, prompt-based framework can unify the two core GUI skills: action grounding, mapping an instruction to the right on-screen element, and GUI referring, describing the content and layout under a user-selected point. Its central claim is that decomposing a screenshot into a hierarchy of broad regions (GROIs) and local elements, then enriching each local element with a spatially grounded functional description, lets a generalist LVLM perform both tasks at the level of, or above, dedicated fine-tuned systems. Concretely, the paper reports 72.2% overall action-grounding accuracy on ScreenSpot with GPT-4o (against OmniParser's 72.6%), 68.0% on VisualWebBench, 60.0% overall on AITW, and 71.58% description accuracy on ScreenPR with GPT-4o, surpassing the ToL agent on every backbone tested.

Load-bearing premise

The headline comparisons assume that the authors' reimplementation of OmniParser in Table 2 accurately reflects OmniParser's true performance, and that excluding OmniParser from the AITW and Mind2Web comparisons does not bias the claimed superiority.

Editorial extensions

If this is right

  • A GUI agent can be deployed on a new platform without collecting training data or fine-tuning: the same HSP and SEED prompts work with any sufficiently capable LVLM.
  • Augmenting an existing training-based parser with SEED improves it: replacing OmniParser's icon captions with SEED descriptions raises its ScreenSpot accuracy from 66.9% to 69.5% with GPT-4V.
  • Multi-candidate grounding is cheap and useful: pass@2 and pass@3 exceed pass@1 by more than 10 points on ScreenSpot and VisualWebBench, enabling a verification layer for GUI agents.
  • The benefit of region-level grounding is concentrated on dense web and desktop interfaces; the paper finds lower GROI counts and coverage on mobile screens, where the gain over flat grounding is smaller.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If HSP's Local Element Exhaustiveness is the binding constraint on Mind2Web, as the failure analysis suggests, then improving the proposal stage (better segmentation or OCR) should improve step success without any change to the prompts; this is directly testable by swapping detectors.
  • Because TRISHUL is training-free, its accuracy should rise automatically as stronger generalist LVLMs appear; the paper's observed gap between GPT-4o and GPT-4V is already in that direction.
  • The two-lens referring design implies that the same hierarchy could support accessibility tools, such as describing screen regions to a visually impaired user, a use case the paper names but does not evaluate quantitatively.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents TRISHUL, a training-free framework that augments large vision-language models (LVLMs) for GUI understanding. TRISHUL combines Hierarchical Screen Parsing (HSP), which uses SAM and OCR to build a two-level hierarchy of Global Regions of Interest (GROIs) and local elements, with Spatially Enhanced Element Description (SEED), which generates functionality descriptions via chain-of-thought and in-context examples. The authors evaluate action grounding on ScreenSpot, VisualWebBench, AITW, and Mind2Web, and GUI referring on ScreenPR, reporting that TRISHUL outperforms training-free baselines, rivals training-based methods such as OmniParser, and sets state-of-the-art results on several benchmarks.

Significance. If the reported results are reliable, TRISHUL is a valuable contribution: it is a modular, training-free pipeline that unifies action grounding and GUI referring, with a clearly specified parsing algorithm (Algorithm 1), ablations isolating the contributions of SEED and GROI grounding, and an honest failure analysis via the Local Element Exhaustiveness metric. The multi-sampling analysis in Section 4.1 is a useful practical extension. However, the central comparative claims rest on three load-bearing issues: the exclusion of OmniParser from AITW and Mind2Web, inconsistent numbers for the same configuration across tables, and the use of ScreenSpot examples for in-context learning while ScreenSpot is a headline benchmark. These need to be resolved before the stated superiority can be accepted.

major comments (3)
  1. [Section 3.1, Tables 2-4] The paper excludes OmniParser from the AITW and Mind2Web comparisons after failing to reproduce its ScreenSpot results, stating 'we have chosen to exclude their results for these benchmarks from our analysis.' OmniParser is the most directly relevant baseline, and its original paper reports results on AITW and Mind2Web. Without those numbers, the abstract's claim of 'superior performance in action grounding across ... AITW and Mind2Web' cannot be verified from the manuscript. Please include the published OmniParser results (clearly marked as reported by the original paper) in Tables 3 and 4, or provide a substantive justification for why they are invalid.
  2. [Table 2 vs. Table 6] The reported GPT-4V results are internally inconsistent. Table 2 lists TRISHUL with GPT-4V at 68.0% overall on VisualWebBench and 61.9% overall on ScreenSpot, while Table 6 reports Pass@1 values of 56.3% on VisualWebBench and 59.0% on ScreenSpot for what appears to be the same configuration. The difference on VisualWebBench (68.0 vs. 56.3) is too large to ignore. Please clarify the experimental setup, metric definitions, or sampling differences that explain this discrepancy, and ensure all tables report a consistent evaluation protocol.
  3. [Section 2.2] SEED uses 'six examples from the ScreenSpot dataset' for in-context learning, and ScreenSpot is one of the headline benchmarks. If any of these examples overlap with the evaluation set, the ScreenSpot results are contaminated. Even if they come from a disjoint split, the use of in-distribution examples as demonstrations for the same benchmark should be disclosed and discussed. Please specify the split of these examples and, ideally, run a variant that uses examples from other datasets to measure the effect.
minor comments (4)
  1. [Section 2.2] The ScreenSpot benchmark is cited as (Jurmu et al., 2008), but that reference is a different 'ScreenSpot' work on resource discovery; the GUI grounding benchmark ScreenSpot comes from SeeClick (Cheng et al., 2024). Please correct the citation.
  2. [Section 2.3, Table 1] GROI proposal accuracy is defined as whether the ground-truth bounding-box midpoint lies inside the proposed GROI. This metric can be trivially satisfied by very large GROIs; please also report the average area coverage of proposed GROIs or a localization metric that penalizes overly large regions.
  3. [Appendix A.2.3] The thresholds Sthresh and the IoS overlap thresholds are set to different values for the action-grounding and ScreenPR tasks. Since the paper claims a training-free framework, please discuss how these thresholds were chosen and whether the results are sensitive to them.
  4. [Section 3.4] The text reports human preference for TRISHUL over ToL as 62.8%, while Figure 4 states 63%; please make the numbers consistent.

Circularity Check

1 steps flagged · score 2.0 of 10

Main results are external-benchmark evaluations; the only circular element is minor: the SEED in-context examples are drawn from the ScreenSpot evaluation set, so the ScreenSpot headline number is not a clean out-of-sample test.

  1. fitted input called prediction [Section 2.2 (SEED), sentence after Eq. (3); used in Section 3.1 and Table 2.]
    "We use ICL (Brown et al., 2020) with six examples from the ScreenSpot (Jurmu et al., 2008) dataset, The full SEED prompt with specific details about the SEED module is available in Appendix A."

    ScreenSpot is the first benchmark whose action-grounding results are reported in Table 2. The SEED module is part of TRISHUL's inference pipeline for all action-grounding tasks, so the ScreenSpot numbers in Table 2 were produced with a prompt containing six examples taken from the ScreenSpot evaluation set itself. The claimed 'training-free, cross-dataset' ScreenSpot accuracy is therefore not an independent out-of-sample measurement: the evaluation set is partly an input to the method on that same benchmark. This does not make the numbers equal to the examples by construction, but it removes ScreenSpot as clean evidence for the paper's generalization claim and makes the comparison with training-based baselines (which did not receive ScreenSpot examples at inference) less apples-to-apples.

full rationale

TRISHUL's central results are obtained by running closed LVLMs with HSP+SEED prompts on public benchmarks (ScreenSpot, VisualWebBench, AITW, Mind2Web, ScreenPR) and comparing with published or reimplemented baselines. There is no self-citation chain, no imported uniqueness theorem, and no mathematically forced equivalence: HSP is a SAM/OCR post-processing recipe with explicit thresholds, and SEED is a prompting module whose outputs are evaluated through the LVLM's grounding decisions. The one genuinely circular element is that the SEED prompt uses six in-context examples from the ScreenSpot evaluation dataset, and the paper then reports ScreenSpot as a headline action-grounding benchmark; this is a test-set-contamination rather than a full by-construction reduction, and it does not affect VisualWebBench, AITW, Mind2Web, or ScreenPR in the same direct way. Two non-circular but important validity concerns should be weighed separately: OmniParser results in Table 2 are from the authors' own reimplementation, and Section 3.1 states that OmniParser results are excluded from AITW and Mind2Web after failed replication ('we have chosen to exclude their results for these benchmarks from our analysis'), making the 'superior performance' claim on those two datasets hard to verify; additionally, Table 2 gives GPT-4V ScreenSpot 61.9 and VisualWebBench 68.0, while Table 6 reports 59.0 and 56.3 for the same framework, an internal inconsistency. These concerns affect comparison fairness and reproducibility, not circularity. Overall, the core contribution is an empirical prompting pipeline evaluated against external benchmarks, so the circularity score is low.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The method's performance depends on hand-set detection thresholds, heuristics for filtering OCR/SAM outputs, and domain assumptions about the reliability of the underlying perception tools and evaluation proxies. No new physical entities are postulated.

free parameters (8)
  • Athresh-GROI = not reported
    Area threshold that separates Global Region of Interest candidates from local elements in Algorithm 1; hand-set.
  • Athresh-Icon / Athresh-Button = not reported
    Area thresholds that classify SAM boxes as icons or buttons; hand-set in Appendix A.2.2.
  • IOU threshold for NMS = 0.5 for action grounding, 0.0 for ScreenPR
    Used in GROI NMS (Appendix A.2.3); task-specific.
  • Sthresh (information score threshold) = 10 for ScreenPR, 25 for action grounding
    Rejects GROI candidates with low information score; task-specific (Appendix A.2.3).
  • IoS overlap thresholds = 0.5 for grounding, 0.0 for ScreenPR; icon redundancy IoS > 0.6
    Controls removal of overlapping or redundant boxes (Appendix A.2.2, A.2.3).
  • Icon aspect ratio range = [0.7, 1.3]
    Filters square-like icon boxes (Appendix A.2.2).
  • OCR false-positive character list = list of special characters and short strings
    Hand-curated list to drop spurious OCR text boxes (Appendix A.2.2).
  • ICL examples from ScreenSpot = 6 examples
    SEED prompt uses six examples from the ScreenSpot benchmark, which is also an evaluation set (Section 2.2).
assumptions (5)
  • domain assumption SAM and EasyOCR produce bounding boxes that are sufficiently accurate and complete to support hierarchical parsing.
    Algorithm 1 and the LEE analysis (Section 4.2) depend on detection quality; the paper itself notes low LEE as a bottleneck.
  • ad hoc to paper The information score heuristic and NMS thresholds generalize across the evaluated datasets without per-dataset tuning.
    Thresholds are set by hand and are task-specific (Sthresh differs between tasks), yet are treated as fixed for each task; no sensitivity analysis is provided.
  • domain assumption Midpoint containment (inside GROI or inside any local element box) is a meaningful proxy for grounding success.
    Used to define GROI proposal accuracy (Table 1) and the LEE metric (Section 4.2).
  • domain assumption Cycle-consistency evaluation in ScreenPR measures description quality faithfully.
    Section 3.4 adopts the ToL evaluation protocol; no independent verification that this proxy correlates with human preference beyond the reported study.
  • domain assumption Closed API LVLMs respond deterministically enough at temperature 0.0 for results to be stable.
    All experiments use temperature 0.0 (Appendix A.1), but no repeated-run variance is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TRISHUL: Towards Region Identification and Screen Hierarchy Understanding for Large VLM based GUI Agents." pith.science (2026). https://pith.science/paper/4DAPOLSI

@misc{pith2026250208226,
  author       = {Pith},
  title        = {Pith review of: TRISHUL: Towards Region Identification and Screen Hierarchy Understanding for Large VLM based GUI Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4DAPOLSI}},
  note         = {Machine review of arXiv:2502.08226}
}
read the original abstract

Recent advancements in Large Vision Language Models (LVLMs) have enabled the development of LVLM-based Graphical User Interface (GUI) agents under various paradigms. Training-based approaches, such as CogAgent and SeeClick, struggle with cross-dataset and cross-platform generalization due to their reliance on dataset-specific training. Generalist LVLMs, such as GPT-4V, employ Set-of-Marks (SoM) for action grounding, but obtaining SoM labels requires metadata like HTML source, which is not consistently available across platforms. Moreover, existing methods often specialize in singular GUI tasks rather than achieving comprehensive GUI understanding. To address these limitations, we introduce TRISHUL, a novel, training-free agentic framework that enhances generalist LVLMs for holistic GUI comprehension. Unlike prior works that focus on either action grounding (mapping instructions to GUI elements) or GUI referring (describing GUI elements given a location), TRISHUL seamlessly integrates both. At its core, TRISHUL employs Hierarchical Screen Parsing (HSP) and the Spatially Enhanced Element Description (SEED) module, which work synergistically to provide multi-granular, spatially, and semantically enriched representations of GUI elements. Our results demonstrate TRISHUL's superior performance in action grounding across the ScreenSpot, VisualWebBench, AITW, and Mind2Web datasets. Additionally, for GUI referring, TRISHUL surpasses the ToL agent on the ScreenPR benchmark, setting a new standard for robust and adaptable GUI comprehension.

Figures

Figures reproduced from arXiv: 2502.08226 by the authors.

Figure 1
Figure 1. Screen parsing results showing detected GUI elements and their function descriptors leveraging our HSP and SEED modules . Test-time assistance with visual perception tools: Studies have leveraged visual perceptions tools to assist general￾ist LVLMs like GPT-4V. MM-Navigator (Yan et al., 2023) leverages pre-trained icon detector module. A concurrent work to ours, Omniparser (Lu et al., 2024), trains a YOLO￾v8 (Jocher… view at source ↗
Figure 2
Figure 2. TRISHUL: Agentic Action Grounding Framework, Pink arrow, denotes our Hierarchical Screen Parsing (HSP) method, to generate GROIs and local element annotations, Green arrows represent our Spatially Enhanced Element Descriptor (SEED) workflow, Blue arrows represent our GROI proposal framework and Magenta Arrow shows, the Set of Marks (SoM) based Grounding workflow. tasks. Additionally, we validate TRISHUL’s effectiven… view at source ↗
Figure 3
Figure 3. TRISHUL: Agentic GUI Referring Framework, the 2 Lenses created using our HSP module for local and global context. Lens-1 contains the local element (blue) in the cropped GROI (red), Lens-2 contains the GROI (blue) in the full input screenshot (red).The selected point is represented as the black dot. Both lenses are fed to the LVLM to generate Layout and Task description. Bicon = {(i, bicon,i)} Nicon i=1 (1) Btext = … view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Human evaluation results on ScreenPR benchmark. TR￾ISHUL is preferred by human annotators 63% of the time over ToL agent and 73% of the time over baseline GPT-4o [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Local Element Exhaustiveness Score for ScreenSpot, Vi￾sual WebBench, AITW and Mind2Web ploy 10 human annotators from (https://www.indikaai.com/) and ask them to choose between the description generated by our approach and the alternative approach. Each evalua￾tor is pr…
Figure 6
Figure 6. Figure 6: Candidate bounding boxes generated from SAM + OCR to the left and the corresponding HSP results (Icon + text + picture) to the right 14 [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Prompt for Screen Point-and-Read [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Distribution of Number of GROIs per image for ScreenSpot and Visual WebBench [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 10
Figure 10. Figure 10: Prompt for instruction guided GROI Proposal generation 15 [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Prompt for SEED 16 [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: SoM grounding Prompt for ScreenSpot and VisualWebBench [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Agentic task following prompt for AITW 17 [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: Agentic task following prompt for Mind2Web 18 [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 7 canonical work pages

  1. [4]

    org/CorpusID:267069082

    URL https://api.semanticscholar. org/CorpusID:267069082. Deng, X., Gu, Y ., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y . Mind2web: Towards a generalist agent for the web. ArXiv, abs/2306.06070,

    arXiv

  2. [6]

    org/CorpusID:258823350

    URL https://api.semanticscholar. org/CorpusID:258823350. Gur, I., R ¨uckert, U., Faust, A., and Hakkani-T ¨ur, D. Z. Learning to navigate the web. ArXiv, abs/1812.09195,

    arXiv

  3. [8]

    org/CorpusID:260126067

    URL https://api.semanticscholar. org/CorpusID:260126067. 9 TRISHUL He, H., Yao, W., Ma, K., Yu, W., Dai, Y ., Zhang, H., Lan, Z., and Yu, D. Webvoyager: Building an end-to-end web agent with large multimodal models. In Annual Meeting of the Association for Computational Linguistics,

    work page

  4. [9]

    org/CorpusID:267211622

    URL https://api.semanticscholar. org/CorpusID:267211622. He, Z., Sunkara, S., Zang, X., Xu, Y ., Liu, L., Wichers, N., Schubiner, G., Lee, R. B., and Chen, J. Actionbert: Lever- aging user actions for semantic understanding of user interfaces. In AAAI Conference on Artificial Intelligence,

    work page

  5. [10]

    org/CorpusID:229363676

    URL https://api.semanticscholar. org/CorpusID:229363676. Hong, W., Wang, W., Lv, Q., Xu, J., Yu, W., Ji, J., Wang, Y ., Wang, Z., Zhang, Y ., Li, J.-Z., Xu, B., Dong, Y ., Ding, M., and Tang, J. Cogagent: A visual lan- guage model for gui agents. ArXiv, abs/2312.08914,

    arXiv

  6. [11]

    org/CorpusID:273102270

    URL https://api.semanticscholar. org/CorpusID:273102270. https://www.indikaai.com/. Indika.ai. JaidedAI. Easyocr: Ready-to-use ocr with 80+ supported languages and all popular writing scripts including latin, chinese, arabic, devanagari, cyrillic and etc. URL https://github.com/JaidedAI/EasyOCR. Jocher, G., Chaurasia, A., and Qiu, J. Ultralyt- ics yolov8,...

    work page 2023

  7. [13]

    org/CorpusID:3530344

    URL https://api.semanticscholar. org/CorpusID:3530344. Liu, J., Song, Y ., Lin, B. Y ., Lam, W., Neubig, G., Li, Y ., and Yue, X. Visualwebbench: How far have multimodal llms evolved in web page under- standing and grounding? ArXiv, abs/2404.05955,

    arXiv

  8. [14]

    org/CorpusID:269009925

    URL https://api.semanticscholar. org/CorpusID:269009925. Lu, Y ., Yang, J., Shen, Y ., and Awadallah, A. Omniparser. arXiv preprint arXiv:2408.00203, 2024. OpenAI. ”gpt-4v(ision) system card”, June, 2024a. URL https://openai.com/index/ gpt-4v-system-card/ . OpenAI. ”hello gpt-4o.”, June, 2024b. URL https:// openai.com/index/hello-gpt-4o/. Rawles, C., Li, ...

    arXiv 2024

Show all 23 references
  1. [15]

    org/CorpusID:258841249

    URL https://api.semanticscholar. org/CorpusID:258841249. Wang, B., Li, G., Zhou, X., Chen, Z., Grossman, T., and Li, Y . Screen2words: Automatic mobile ui summariza- tion with multimodal learning. The 34th Annual ACM Symposium on User Interface Software and Technology,

  2. [16]

    org/CorpusID:236957064

    URL https://api.semanticscholar. org/CorpusID:236957064. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. ”chain-of- thought prompting elicits reasoning in large language models”, 2023. URL https://arxiv.org/abs/ 2201.11903. Wu, ...

    2023 arXiv

  3. [17]

    org/CorpusID:237571719

    URL https://api.semanticscholar. org/CorpusID:237571719. Xie, T., Zhang, D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., Liu, Y ., Xu, Y ., Zhou, S., Savarese, S., Xiong, C., Zhong, V ., and Yu, T. Osworld: Benchmarking multimodal agents for ...

    arXiv

  4. [18]

    org/CorpusID:269042918

    URL https://api.semanticscholar. org/CorpusID:269042918. Yan, A., Yang, Z., Zhu, W., Lin, K. Q., Li, L., Wang, J., Yang, J., Zhong, Y ., McAuley, J. J., Gao, J., Liu, Z., and Wang, L. Gpt-4v in won- derland: Large multimodal models for zero-shot smartphone gui navigation. ArXi...

    arXiv

  5. [19]

    org/CorpusID:265149992

    URL https://api.semanticscholar. org/CorpusID:265149992. Yang, J., Zhang, H., Li, F., Zou, X., Li, C., and Gao, J. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441, 2023. Yao, S., Chen, H., Yang, J., and Narasimhan, K. W...

    arXiv

  6. [21]

    org/CorpusID:269005503

    URL https://api.semanticscholar. org/CorpusID:269005503. Zhang, C. X., Yang, Z., Liu, J., Han, Y ., Chen, X., Huang, Z., Fu, B., and Yu, G. Appagent: Multimodal agents as smartphone users. ArXiv, abs/2312.13771,

    arXiv

  7. [22]

    org/CorpusID:266435868

    URL https://api.semanticscholar. org/CorpusID:266435868. Zhang, T., Kishore, V ., Wu, F., Weinberger, K. Q., and Artzi, Y . Bertscore: Evaluating text generation with bert. ArXiv, abs/1904.09675, 2019. URL https: //api.semanticscholar.org/CorpusID: 127986044. Zhang, X., de Gre...

    1904 arXiv

  8. [23]

    @", "#",

    URL https://api.semanticscholar. org/CorpusID:231592643. Zheng, B., Gou, B., Kil, J., Sun, H., and Su, Y . Gpt-4v(ision) is a generalist web agent, if grounded. arXiv preprint arXiv:2401.01614, 2024. Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Bisk,...

    2024 arXiv

  9. [2008]

    org/CorpusID:192633

    URL https://api.semanticscholar. org/CorpusID:192633. Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y ., Doll´ar, P., and Girshick, R. Segment anything. arXiv:2304.02643, 2023. Koh, J. Y ., McAleer, S., Fr...

    2023 arXiv

  10. [2018]

    org/CorpusID:56657805

    URL https://api.semanticscholar. org/CorpusID:56657805. Gur, I., Furuta, H., Huang, A., Safdari, M., Mat- suo, Y ., Eck, D., and Faust, A. A real-world webagent with planning, long context understand- ing, and program synthesis. ArXiv, abs/2307.12856,

    arXiv

  11. [2020]

    org/CorpusID:218971783

    URL https://api.semanticscholar. org/CorpusID:218971783. Chen, J., Chen, C., Xing, Z., Xu, X., Zhu, L., Li, G., and Wang, J. Unblind your apps: Predicting natural- language labels for mobile gui components by deep learning. 2020 IEEE/ACM 42nd International Con- ference on Soft...

    2020

  12. [2021]

    org/CorpusID:236493482

    URL https://api.semanticscholar. org/CorpusID:236493482. Bai, H., Zhou, Y ., Cemri, M., Pan, J., Suhr, A., Levine, S., and Kumar, A. Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning. ArXiv, abs/2406.11896,

    arXiv

  13. [2022]

    org/CorpusID:250264533

    URL https://api.semanticscholar. org/CorpusID:250264533. You, K., Zhang, H., Schoop, E., Weers, F., Swearn- gin, A., Nichols, J., Yang, Y ., and Gan, Z. Ferret- ui: Grounded mobile ui understanding with multimodal llms. In European Conference on Computer Vision ,

  14. [2023]

    org/CorpusID:259129428

    URL https://api.semanticscholar. org/CorpusID:259129428. Fan, Y ., Ding, L., Kuo, C.-C., Jiang, S., Zhao, Y ., Guan, X., Yang, J., Zhang, Y ., and Wang, X. E. Read anywhere pointed: Layout-aware gui screen reading with tree-of- lens grounding, 2024. URL https://arxiv.org/ abs/...

    2024 arXiv

  15. [2024]

    org/CorpusID:270562229

    URL https://api.semanticscholar. org/CorpusID:270562229. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Ka- plan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-V oss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D...

    2005 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.