Pith. sign in

REVIEW 5 major objections 5 minor 3 cited by

Screen2AX: Vision-Based Approach for Automatic macOS Accessibility Generation

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Screen2AX claims that a single macOS screenshot contains enough visual information to reconstruct the application's full accessibility tree, and reports a 77% F1 score for tree reconstruction.

desk verdict A genuinely useful macOS accessibility dataset and pipeline, but the abstract's 77% 'complete tree' F1 and 2.2x agent gain are not what Tables 6 and 8 actually show. read the letter →

arxiv 2507.16704 v1 pith:NES4JT3W submitted 2025-07-22 cs.LG cs.AIcs.CVcs.HC

Viktor Muryn , Marta Sumyk , Mariya Hirna , Sofiya Garkot , Maksym Shamrai This is my paper
classification cs.LGcs.AIcs.CVcs.HC
keywords accessibilitymetadatamacOSUIelementdetectionhierarchygenerationAIagentsscreenparsingobjectdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that desktop accessibility metadata can be generated automatically from pixels rather than sourced from developers. It reports that only about a third of macOS applications expose full accessibility metadata, and that a vision pipeline can recover a usable accessibility tree from one screenshot. If true, screen readers and AI agents would no longer depend on developers having annotated custom UI elements. The paper backs this with three released datasets, a 77% F1 tree-reconstruction score, and agent benchmarks in which the generated hierarchy outperforms built-in metadata and the OmniParser V2 baseline.

What carries the argument

The central object is the accessibility tree itself: a window-rooted hierarchy whose internal nodes are semantic groups such as toolbars and panels, and whose leaves are UI elements carrying role, name, value, and bounding box. Screen2AX's mechanism is a three-stage vision pipeline: a fine-tuned YOLOv11 detector localizes and classifies elements; OCR plus a fine-tuned BLIP model supplies text and icon captions; and a second YOLOv11 model trained on a single AXGroup class predicts group boxes that are assembled into the final tree. The tree format is what lets the output plug into the same JSON grounding format used by agents and accessibility consumers.

What would settle it

Re-annotate a held-out set of the released screenshots with independent annotators who build trees from scratch without seeing built-in metadata; if inter-annotator agreement is low, or Screen2AX matches the original labels but not the independent trees, the 77% F1 cannot stand as evidence of accurate reconstruction.

Watch

Extended reading notes

Core claim

Screen2AX's central claim is that the full macOS accessibility hierarchy—element bounding boxes, roles, names, descriptions, and parent-child grouping—can be recovered from a single screenshot. The pipeline detects and classifies UI elements with a fine-tuned YOLOv11 model, extracts text with OCR, captions icon-only buttons with a fine-tuned BLIP model, and uses a second YOLOv11 model to predict semantic groups that are assembled into a tree mirroring the macOS AXWindow/AXGroup structure. On its own dataset, the method reaches 65.4% element-detection accuracy at IoU 0.5, a 77% F1 for edge-based tree reconstruction, and a 55% group-match score. When the resulting tree is fed to a GPT-4-based agent, task success rises to 33.7% versus 16.9% with built-in accessibility metadata and 28.0% with OmniParser V2, and ScreenSpot grounding success reaches 36.6% versus 31.9%.

Load-bearing premise

The load-bearing premise is that the human-corrected accessibility labels used as ground truth are accurate and consistent enough that a model trained on them is truly learning to reconstruct accessibility structure, not learning the labeling quirks of the annotators.

Editorial extensions

If this is right

  • Applications with missing or incomplete developer-authored accessibility metadata could receive a usable accessibility tree without any developer changes, simply from a screenshot.
  • AI agents grounded in Screen2AX-generated trees would perform roughly twice as well on simple UI tasks as agents grounded in native macOS accessibility metadata.
  • The hierarchical structure itself contributes to agent performance, with the full tree improving task success by about 3% over a flat list of the same detected elements.
  • The released datasets give future work a common benchmark for macOS UI element detection, grouping, and accessibility-tree reconstruction.
  • Real-time generation at roughly 0.2 seconds per element-detection pass makes the approach a plausible component of live accessibility tooling.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same pixel-to-tree pipeline could plausibly transfer to Windows, Linux, or web interfaces if similar labeled hierarchies existed, since the method itself is not macOS-specific.
  • Editorial inference: the reported one-third full-support statistic implies a large deployment surface, but real-world usefulness would need validation with screen-reader users, not only agent benchmarks.
  • Editorial inference: because a single screenshot captures one static state, popovers, hover states, and live-updating content would likely require temporal or multi-frame input to be handled reliably.
  • Editorial inference: the mobile-icon domain gap visible in the BLIP captioning failures suggests that a desktop-specific icon captioning dataset would directly improve both caption quality and agent task success.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper presents Screen2AX, a screenshot-only pipeline for generating hierarchical macOS accessibility metadata, combining YOLOv11-based element and group detection, OCR, BLIP-based icon captioning, and a heuristic or learned grouping step. It introduces three datasets (Screen2AX-Tree, Screen2AX-Element, Screen2AX-Group) and the Screen2AX-Task agent benchmark, and evaluates the pipeline on element detection, captioning, hierarchy reconstruction, and agent grounding. The central claims are that Screen2AX reconstructs complete accessibility trees at 77% F1 and yields a 2.2x improvement over native accessibility representations for agent task execution while surpassing OmniParser V2 on ScreenSpot.

Significance. The work addresses a real and under-served problem, and the released datasets and open-source code are valuable community assets. If the headline results survived closer scrutiny, this would be a meaningful step toward vision-only accessibility generation for desktop UIs. However, the evidence as presented does not yet support the complete-tree and 2.2x claims; the strengths are the resource contribution and a plausible pipeline, not the current quantitative overstatements.

major comments (5)
  1. [Abstract and Table 6] The '77% F1 score in reconstructing a complete accessibility tree' is an edge-based F1 (Table 6, 'F1 Score, % 77'), not a tree-level or complete-tree metric. The same table reports Leaves F1 of 31% and a Container Match (CM) of 55%; the abstract should either specify 'edge-based F1' or aggregate the full set of metrics, since 77% as stated materially overstates the measured reconstruction quality.
  2. [Sections 4.1, 5.3, and 6.1.1] The ground-truth trees and group labels were produced by manually correcting the same built-in macOS accessibility metadata that the paper documents as inconsistent, and Section 5.3 concedes that the dataset 'may contain inaccuracies or inconsistencies.' Because this label source supplies both the YOLOv11 hierarchy training targets and the evaluation targets, the reported 77% F1 can certify agreement with a noisy label convention rather than with true accessibility structure. The manuscript should report inter-annotator agreement on a re-annotated held-out subset and compare corrected labels against an independent expert standard.
  3. [Section 5.4.2 and Table 8] The abstract's '2.2x performance improvement over native accessibility representations' is not recoverable from Table 8: 33.7/16.9 = 1.99, not 2.2. The text's 16.8 percentage-point absolute gain is not a multiplicative improvement. Please correct the claim or provide the computation that yields 2.2x.
  4. [Tables 3 and 6, and Section 5.4.2] The component metrics are low enough to undercut the end-to-end story: element detection F1 is 43.8% (Table 3), leaves F1 is 31% (Table 6), and Screen2AX-Task success is 33.7% (Table 8). The paper should explain how an edge-based F1 of 77% is compatible with these numbers, and should not describe the detected result as a 'complete' accessibility tree.
  5. [Tables 7 and 8] The claimed advantages over OmniParser V2 (36.6 vs. 31.9 on ScreenSpot; 33.7 vs. 28.0 on Screen2AX-Task) and the 3.0-3.7 percentage-point gains from hierarchy are reported without confidence intervals, repeated trials, or significance tests, despite GPT-4 stochasticity and modest sample sizes (172 ScreenSpot instructions; 166 Screen2AX-Task images). Add variance estimates or significance testing before asserting superiority.
minor comments (5)
  1. [Section 5.4.2] The text contains the typo 'OmniPars r v2' in the sentence 'improvement of 5.7% compared to OmniPars r v2.'
  2. [Section 6.1.2] The sentence 'we initially use the ocrmac tool ... to extract text from bu tons' contains a typo; 'bu tons' should be 'buttons.'
  3. [Section 6] The opening sentence 'we qualitatively evaluate each core component of our stem' should read 'system' instead of 'stem.'
  4. [References [9] and [10]] Both references list the SeeClick paper with overlapping arXiv identifiers; consolidate them to avoid duplicate citations.
  5. [Section 4.1.4] The phrase 'we comprised 435 images' should be 'we compiled 435 images' or 'the dataset comprises 435 images' to match the intended meaning.

Circularity Check

1 steps flagged · score 3.0 of 10

One self-referential caption-evaluation loop; the central hierarchy and agent claims rest on standard supervised and external benchmarks and are not circular.

  1. other [Sections 4.3.2 and 5.2 (Table 5)]
    "Additionally, we collected a dataset of 988 macOS icons and captioned it using GPT-4 [27] for validation and testing purposes. / The GPT-measured accuracy is calculated by providing GPT-4[27] with the ground-truth caption, the predicted caption, and the corresponding image. GPT-4 is then asked to determine whether the ground truth and predicted captions have the same meaning."

    The ground-truth captions for the 988-icon validation/test set are produced by GPT-4, and the accuracy statistic is also computed by GPT-4, which judges whether each predicted caption 'has the same meaning' as that GPT-4-generated caption. The reported accuracy is therefore a self-consistency score: it measures agreement between BLIP and GPT-4's own caption distribution using GPT-4 as the arbiter, rather than agreement with an independent human label. If GPT-4 systematically prefers its own phrasing or semantics, the score is inflated by construction. This loop does not affect the hierarchy or agent results, which use human-annotated boxes or the external ScreenSpot benchmark, but it makes the Table 5 captioning comparison self-referential rather than independently grounded.

full rationale

Most of the paper's derivation chain is standard supervised learning and is not circular. The 77% hierarchy F1 is a held-out evaluation of YOLOv11 against Screen2AX-Group, a dataset the authors built by manually correcting macOS system metadata; training and testing on compatible splits of one's own dataset is the normal ML workflow, not a reduction of the prediction to its input. The Screen2AX-Task and ScreenSpot evaluations use human-annotated target boxes and an external benchmark, respectively, so the central claims have independent content. The admissions in Sections 5.3 and 6.1.1 about inconsistent labeling are correctness/quality caveats, not circularity. The one exhibited circular step is the icon-captioning metric: GPT-4 creates the reference captions for the 988 macOS icons and GPT-4 also judges whether Screen2AX's BLIP captions match them (Sections 4.3.2, 5.2). This makes the Table 5 'GPT-measured accuracy' a measure of consistency with GPT-4's own captions, not an independent semantic measure. Because the headline hierarchy and agent comparisons do not depend on this loop, the overall circularity is minor.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The work is an empirical ML systems paper, so the fitted parameters are learned model weights and manual hyperparameters rather than physical constants. The single most important assumption is that the corrected system accessibility tree is a valid ground truth, and the paper itself flags this data as noisy. No new physical or ontological entities are introduced.

free parameters (5)
  • YOLOv11 element detection model weights = not reported (trained on Screen2AX-Element)
    All downstream hierarchy and agent results depend on these fitted weights; the final weights and full hyperparameters are not provided.
  • YOLOv11 group detection model weights = not reported (trained on Screen2AX-Group)
    This model produces the hierarchy groups; the reported 77% F1 is its test-set performance on the same label distribution.
  • BLIP icon captioning model weights = fine-tuned on 5,000 mobile icons, 10 epochs, AdamW lr 5e-5
    Element descriptions from this model drive agent success; caption evaluation uses GPT-4 as judge rather than human annotation.
  • Heuristic grouping thresholds = 15 hyperparameters, e.g., vertical spacing min(h1,h2)+15px, 25%/40% overlap, 1.25x min-height, 40px edge difference…
    Used for the heuristic hierarchy baseline; the authors note these require manual fine-tuning per application.
  • Detection confidence threshold = not reported
    Section 4.2.2 says low-confidence detections are filtered, but the threshold value and its effect on metrics are not given.
assumptions (6)
  • domain assumption Built-in macOS accessibility trees, after manual correction, are a valid ground truth for the semantic hierarchy of a screen.
    All datasets and hierarchy metrics in Section 4.1 and Table 6 are derived from system metadata corrected via Roboflow; Section 5.3 says this data 'may contain inaccuracies or inconsistencies'.
  • domain assumption A single screenshot contains enough visual information to reconstruct the full accessibility tree.
    The method is vision-only, Figure 3; invisible and shifted elements discussed in Section 2.2 cannot be recovered from pixels, bounding any full-tree claim to what is visible.
  • domain assumption The seven-class simplification preserves the semantic information needed by assistive tools and agents.
    Section 4.1 merges 52 macOS classes into seven, for example AXCheckBox and AXRadioButton are folded into AXButton; no user study shows the merged roles are sufficient.
  • domain assumption GPT-4-based judgments of caption equivalence and agent element selection reflect true correctness.
    Section 5.2 uses GPT-4 as the caption judge and Section 5.4 uses GPT-4 as the agent; no human evaluation or inter-annotator agreement is reported.
  • domain assumption The 112 sampled macOS applications and the 166-image task subset are representative of macOS applications.
    Dataset construction in Section 4.1 and Screen2AX-Task in Section 4.1.4; no sampling plan is given, and the paper's own preliminary analysis shows large differences between popular and random apps.
  • domain assumption The heuristic grouping thresholds are transferable across layouts.
    Section 4.4.1 lists 15 hyperparameters and states they need manual fine-tuning per application, so this baseline is not a general grouping rule.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Screen2AX: Vision-Based Approach for Automatic macOS Accessibility Generation." pith.science (2026). https://pith.science/paper/NES4JT3W

@misc{pith2026250716704,
  author       = {Pith},
  title        = {Pith review of: Screen2AX: Vision-Based Approach for Automatic macOS Accessibility Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NES4JT3W}},
  note         = {Machine review of arXiv:2507.16704}
}
read the original abstract

Desktop accessibility metadata enables AI agents to interpret screens and supports users who depend on tools like screen readers. Yet, many applications remain largely inaccessible due to incomplete or missing metadata provided by developers - our investigation shows that only 33% of applications on macOS offer full accessibility support. While recent work on structured screen representation has primarily addressed specific challenges, such as UI element detection or captioning, none has attempted to capture the full complexity of desktop interfaces by replicating their entire hierarchical structure. To bridge this gap, we introduce Screen2AX, the first framework to automatically create real-time, tree-structured accessibility metadata from a single screenshot. Our method uses vision-language and object detection models to detect, describe, and organize UI elements hierarchically, mirroring macOS's system-level accessibility structure. To tackle the limited availability of data for macOS desktop applications, we compiled and publicly released three datasets encompassing 112 macOS applications, each annotated for UI element detection, grouping, and hierarchical accessibility metadata alongside corresponding screenshots. Screen2AX accurately infers hierarchy trees, achieving a 77% F1 score in reconstructing a complete accessibility tree. Crucially, these hierarchy trees improve the ability of autonomous agents to interpret and interact with complex desktop interfaces. We introduce Screen2AX-Task, a benchmark specifically designed for evaluating autonomous agent task execution in macOS desktop environments. Using this benchmark, we demonstrate that Screen2AX delivers a 2.2x performance improvement over native accessibility representations and surpasses the state-of-the-art OmniParser V2 system on the ScreenSpot benchmark.

Figures

Figures reproduced from arXiv: 2507.16704 by the authors.

Figure 1
Figure 1. Screen2AX is a screenshot-based method that addresses a key gap in the availability of accessibility support by [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Limitations of built-in accessibility: Elements with [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Class distribution of UI elements in the Screen2AX [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (13 more)
Figure 5
Figure 5. Figure 5: Histogram showing the bimodal distribution of [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 3
Figure 3. Figure 3: Screen2AX pipeline for automated accessibility generation. The system processes a UI screenshot through three stages: [PITH_FULL_IMAGE:figures/full_fig_p006_3.png]
Figure 6
Figure 6. Figure 6: An example of the labeled input screen (left) and the corresponding hierarchy tree (right). The tree captures every [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: AI Agent Task Execution Pipeline Using Screen2AX Metadata [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 9
Figure 9. Figure 9: OCR error: Bluetooth icon misinterpreted as the [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 8
Figure 8. Figure 8: Examples illustrating challenges in achieving perfect prediction accuracy. (Left) An application with rare red buttons at [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 10
Figure 10. Figure 10: Example of failure of task completion by AI agent. [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: Screenshot of the UI from macOS application "Ses [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Spatial density of UI elements. The plot shows the [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Size distribution by element type. Boxplots show [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: Distribution of element types in the dataset. The [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 18
Figure 18. Figure 18: Example of a macOS application with shifted UI el [PITH_FULL_IMAGE:figures/full_fig_p019_18.png]
Figure 17
Figure 17. Figure 17: An example screenshot of UI from an application [PITH_FULL_IMAGE:figures/full_fig_p019_17.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Do GUI Agents Believe Their Eyes? Diagnosing State-Belief Reliance on Pixels versus Structure

    cs.AI 2026-07 accept novelty 7.0 of 10

    Across four open-weight models and three OpenAI models, textual state beliefs of GUI agents follow a conflicting structured value on 30–75% of probes where the same model reads the pixels correctly, so agent state bel...

  2. Grounding Computer Use Agents on Human Demonstrations

    cs.LG 2025-11 conditional novelty 7.0 of 10

    GroundCUA, a 3.56M-element human-annotated desktop grounding dataset, and GroundNext models achieve strong UI grounding with less than one-tenth the SFT data of prior work.

  3. Tactile: Giving Computer-Using Agents Hands and Feet

    cs.AI 2026-07 conditional novelty 5.0 of 10

    Adding Tactile, an MCP tool layer that grounds agent actions in macOS accessibility semantics, OCR, and visual fallback, raised Codex Success@100 from 41.1% to 50.0% on macOSWorld-style tasks.

Reference graph

Works this paper leans on

46 extracted references · 21 canonical work pages · cited by 3 Pith papers

  1. [1]

    Appfigures. 2025. Top Apps & Games for Mac on the iOS App Store in the United States · Appfigures. https://appfigures.com/top-apps/mac-app-store/united- states/top-overall Accessed: 2025-03-12

    work page 2025

  2. [2]

    Apple Inc. 2004. Accessibility - Apple. https://www.apple.com/accessibility/ Accessed: 2025-03-11

    work page 2004

  3. [3]

    Apple Inc. 2018. Human Interface Guidelines | Apple Developer Documentation. https://developer.apple.com/design/human-interface-guidelines Accessed: 2025- 03-11

    work page 2018

  4. [4]

    Apple Inc. 2019. UniversalAccess.h | Apple Developer Documentation. https: //developer.apple.com/documentation/applicationservices/universalaccess_h Ac- cessed: 2025-03-11

    work page 2019

  5. [5]

    Apple Inc. 2023. Accessibility Inspector | Apple Developer Documentation. https: //developer.apple.com/documentation/accessibility/accessibility-inspector Ac- cessed: 2025-03-11

    work page 2023

  6. [6]

    Apple Inc. 2024. Use Switch Control to navigate your iPhone, iPad, or iPod touch. https://support.apple.com/en-us/119835 Accessed: 2025-03-11

    work page 2024

  7. [7]

    Apple Inc. 2024. VoiceOver User Guide for Mac. https://support.apple.com/lt- lt/guide/voiceover/welcome/mac Accessed: 2025-03-11

    work page 2024

  8. [8]

    Sara Bunian, Kai Li, Chaima Jemmali, Casper Harteveld, Yun Fu, and Magy Seif Seif El-Nasr. 2021. VINS: Visual Search for Mobile User Interface Design. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 423, 14 pages. https://doi.org/10...

    arXiv 2021

Show all 46 references
  1. [9]

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024. SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents. arXiv (Jan. 2024). https://doi.org/10.48550/arXiv.2401.10935 arXiv:2401.10935

    2024 doi arXiv

  2. [10]

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024. SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents. arXiv:2401.10935 [cs,HC] https://arxiv.org/abs/2401.10935

    2024 arXiv

  3. [11]

    Seyed Shayan Daneshvar and Shaowei Wang. 2024. GUI Element Detection Using SOTA YOLO Deep Learning Models. arXiv:2408.03507 https://arxiv.org/ abs/2408.03507

    2024 arXiv

  4. [12]

    Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017. Rico: A mobile app dataset for building data-driven design applications. In UIST 2017 - Proceedings of the 30th Annual ACM Symposium on User Interface...

    2017

  5. [13]

    Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian. 2019. CenterNet: Keypoint Triplets for Object Detection. arXiv:1904.08189 [cs.CV] https://arxiv.org/abs/1904.08189

    2019 arXiv

  6. [14]

    Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian

  7. [15]

    Hagberg, Daniel A

    Aric A. Hagberg, Daniel A. Schult, and Pieter J. Swart. 2008. Exploring Network Structure, Dynamics, and Function using NetworkX. In Proceedings of the 7th Python in Science Conference, Gaël Varoquaux, Travis Vaught, and Jarrod Millman (Eds.). Pasadena, CA USA, 11 – 15

    2008

  8. [16]

    Sabrina Haque and Christoph Csallner. 2024. AssistGUI: Task-Oriented Desktop Graphical User Interface Automation. (2024). arXiv:2312.13108 [cs.CV] https: //arxiv.org/abs/2312.13108

    2024 arXiv

  9. [17]

    Sabrina Haque and Christoph Csallner. 2024. Inferring Alt-text For UI Icons With Large Language Models During App Development. (2024). arXiv:2409.18060 [cs.HC] https://arxiv.org/abs/2409.18060

    2024 arXiv

  10. [18]

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. 2023. CogAgent: A Visual Language Model for GUI Agents. arXiv (Dec. 2023). https://doi.org/10.48550/arXiv.2312...

    2023 doi arXiv

  11. [19]

    Ko, Sangeun Oh, and Insik Shin

    Sunjae Lee, Junyoung Choi, Jungjae Lee, Munim Hasan Wasi, Hojun Choi, Steven Y. Ko, Sangeun Oh, and Insik Shin. 2023. Explore, Select, Derive, and Recall: Augmenting LLM with Human-like Memory for Mobile Task Automation. arXiv (Dec. 2023). https://doi.org/10.48550/arXiv.2312.0...

    2023 doi arXiv

  12. [20]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. arXiv:2201.12086 [cs.CV] https://arxiv.org/abs/2201.12086

    2022 arXiv

  13. [21]

    Kevin Qinghong Lin, Difei Gao Linjie Li, Shiwei Wu Zhengyuan Yang, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. 2024. ShowUI: One Vision-Language-Action Model for GUI Visual Agent. arXiv:2411.17465 [cs.CV] https://arxiv.org/abs/2411.17465

    2024 arXiv

  14. [22]

    Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. arXiv:1711.05101 [cs.LG] https://arxiv.org/abs/1711.05101

    2019 arXiv

  15. [23]

    Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. 2024. OmniParser for Pure Vision Based GUI Agent. (2024). arXiv:2408.00203 [cs.CV] https: //arxiv.org/abs/2408.00203

    2024 arXiv

  16. [24]

    J Matas, O Chum, M Urban, and T Pajdla. 2004. Robust wide-baseline stereo from maximally stable extremal regions. Image and Vision Computing 22, 10 (2004), 761–767. https://doi.org/10.1016/j.imavis.2004.02.006 British Machine Vision Computing 2002

    2004 doi

  17. [25]

    Runliang Niu, Jindong Li, Shiqi Wang, Yali Fu, Xiyu Hu, Xueyuan Leng, He Kong, Yi Chang, and Qi Wang. 2024. ScreenAgent: A Vision Language Model-driven Computer Control Agent. 10.48550/arXiv.2402.07945 (2024)

    2024 doi arXiv

  18. [26]

    NV Access. 2025. NV Access. https://www.nvaccess.org/download/ Accessed: 2025-03-11

    2025

  19. [27]

    OpenAI. 2023. GPT-4. https://openai.com/gpt-4 Accessed: 2025-03-11

    2023

  20. [28]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. https://aclanthology. org/P02-1040.pdf

    2002

  21. [29]

    Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lil- licrap. 2023. Android in the Wild: A Large-Scale Dataset for Android Device Control. (2023). arXiv:2307.10088 [cs.LG] https://arxiv.org/abs/2307.10088

    2023 arXiv

  22. [30]

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R- CNN: Towards Real-Time Object Detection with Region Proposal Networks. arXiv:1506.01497 [cs.CV] https://arxiv.org/abs/1506.01497

    2015 arXiv

  23. [31]

    Wobbrock

    Anne Spencer Ross, Xiaoyi Zhang, James Fogarty, and Jacob O. Wobbrock. 2018. Examining Image-Based Button Labeling for Accessibility in Android Apps through Large-Scale Analysis. In ACM Conferences. Association for Computing Machinery, New York, NY, USA, 119–130. https://doi.o...

    2018 doi

  24. [32]

    Sergii Kryvoblotskyi (MacPaw), Nataliia Stulova (MacPaw). 2024. macOS App Dataset. https://research.macpaw.com/publications/macos-app-dataset. Tech note

    2024

  25. [33]

    Maximilian Strauss, many hats, The Brogrammer, Vincent Liu, Wu Yifei, and Jun

  26. [34]

    Srinivas Sunkara, Maria Wang, Lijuan Liu, Gilles Baechler, Yu-Chung Hsiao, Jindong, Chen, Abhanshu Sharma, and James Stout. 2022. Towards Better Se- mantic Understanding of Mobile Interfaces. (2022). arXiv:2210.02663 [cs.HC] https://arxiv.org/abs/2210.02663

    2022 arXiv

  27. [35]

    Maryam Taeb, Amanda Swearngin, Eldon Schoop, Ruijia Cheng, Yue Jiang, and Jeffrey Nichols. 2023. AXNav: Replaying Accessibility Tests from Nat- ural Language. arXiv (Oct. 2023). https://doi.org/10.1145/3613904.3642777 arXiv:2310.02424

    2023

  28. [36]

    Lawrence Zitnick, and Devi Parikh

    Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2014. CIDEr: Consensus-based Image Description Evaluation. arXiv:1411.5726 [cs.CV] https: //arxiv.org/abs/1411.5726

    2014 arXiv

  29. [37]

    Bryan Wang, Gang Li, and Yang Li. 2023. Enabling Conversational Interaction with Mobile UI using Large Language Models. (2023). arXiv:2209.08655 https: //arxiv.org/pdf/2209.08655

    2023 arXiv

  30. [38]

    WebAIM. 2024. Screen Reader User Survey #10 Results. https://webaim.org/ projects/screenreadersurvey10/ Accessed: 2025-03-12

    2024

  31. [39]

    Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. 2023. Empowering llm to use smartphone for intelligent task automation. arXiv preprint arXiv:2308.15272 (2023)

    2023 arXiv

  32. [40]

    World Wide Web Consortium. 2024. Web content accessibility guidelines (WCAG) 2.2. https://www.w3.org/TR/WCAG/ Accessed: 2025-03-11

    2024

  33. [41]

    Jason Wu, Xiaoyi Zhang, Jeff Nichols, and Jeffrey P Bigham. 2021. Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots. InThe 34th Annual ACM Symposium on User Interface Software and Technology (UIST ’21) . ACM. https://doi.org/10.1145/3472749.3474763 Scre...

    2021

  34. [42]

    Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. 2023. Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks. arXiv:2311.06242 [cs.CV] https://arxiv.org/abs/ 2311.06242

    2023 arXiv

  35. [43]

    Shuhong Xiao, Yaxuan Song Yunnong Chen, Liuqing Chen, Lingyun Sun, Yankun Zhen, and Yanfang Chang. 2024. UI Semantic Group Detection: Grouping UI Elements with Similar Semantics in Mobile Graphical User Interface. (2024). arXiv:2403.04984 [cs.SE] https://arxiv.org/abs/2403.04984

    2024 arXiv

  36. [44]

    Ses- sion Pomodoro Focus Timer

    Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, Aaron Everitt, and Jeffrey P. Bigham. 2021. Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels. (2021)....

    2021 arXiv

  37. [2020]

    arXiv:2005.12872 [cs.CV] https://arxiv.org/abs/2005.12872

    End-to-End Object Detection with Transformers. arXiv:2005.12872 [cs.CV] https://arxiv.org/abs/2005.12872

    2005 arXiv

  38. [2024]

    https://github.com/straussmaximilian/ocrmac

    straussmaximilian/ocrmac. https://github.com/straussmaximilian/ocrmac

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.