Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

Morae: Proactively Pausing UI Agents for User Choices

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read UI agents fail blind and low-vision users most often when a valid choice exists but no one asks; Morae pauses at those points, presents options, and completes more tasks with real user preferences expressed.

desk verdict Worth reviewing: a useful dataset and a promising system, but the technical evaluation's 'task success' conflates pausing with completion—the user study carries the claim. read the letter →

arxiv 2508.21456 v1 pith:LOTZNLVD submitted 2025-08-29 cs.HC cs.CLcs.CV

Yi-Hao Peng , Dingzeyu Li , Jeffrey P. Bigham , Amy Pavel This is my paper
classification cs.HCcs.CLcs.CV
keywords UIagentsblindandlow-visionusersproactivepausingpreferenceelicitationmixed-initiativeinterfaceslargemultimodalmodelswebautomationaccessibility
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that UI automation agents let down blind and low-vision users not by doing tasks wrong, but by doing them without asking. A field study with four BLV participants showed that in 182 of 484 tasks the agent reported as complete, real ambiguity remained—multiple equally valid options or underspecified details—and in 95% of such cases users never knew choices existed. The authors built Morae, an agent that at each step runs an internal ambiguity-verification loop and pauses to present options through a screen-reader-friendly generated interface when a preference is genuinely unclear. In a 256-task technical evaluation it reached 55.2% task success versus 53.1% for OpenAI Operator, with the gain concentrated in pause-required tasks (65.6% versus 50.8%). In a study with 10 BLV participants, Morae users completed more tasks (5.50 of 9) than with Operator (3.90) or TaxyAI (2.60) and reported stronger awareness and control over choices.

What carries the argument

The load-bearing mechanism is the Dynamic Verification of Ambiguous Choices decision loop. At each interaction step the agent splits planned actions into critical and non-critical, formulates prioritized verification questions (multiple equally valid UI elements? underspecified query? ambiguous tie-breaker like 'cheapest'?), and answers each with yes, no, unanswerable and proceed, or not important and proceed. An ambiguity indicator A(i) and a UI-sufficiency indicator I(i) feed a decision function: execute critical actions if incomplete, pause for clarification when ambiguity is confirmed and enough detail is visible, gather more UI detail when ambiguity exists but context is thin, and other

What would settle it

For each of the 256 technical-evaluation tasks, have a separate set of BLV users independently mark the exact step at which they would want to intervene, and measure agreement with the authors' annotated pause step; low agreement would mean the ground truth does not reflect real user pauses. Equivalently, pre-register a user study in which participants state their preferences before each task and the final selection is checked against that statement.

Watch

Extended reading notes

Core claim

Morae's central claim is that the right moment for human involvement in UI automation is not fixed or user-initiated but detected dynamically: at each step the agent generates and answers internal ambiguity-verification questions about the user's command, the observed UI, and the action history, pausing only when a critical choice is genuinely ambiguous and enough context is visible for the user to decide. This mechanism, which the paper calls Dynamic Verification of Ambiguous Choices, outperformed simpler instructions to pause, first-step-only verification, per-step verification without planning, and OpenAI Operator. The paper presents this as evidence that proactive pausing materially impr

Load-bearing premise

The evaluation leans on the authors' manually annotated pause points being the places any user would actually want to choose; if real users would pause at different steps, the measured gains could come from the annotation rather than from pausing itself.

Editorial extensions

If this is right

  • Adding a lightweight self-verification loop to an existing UI agent recovers most preference-critical failures without task-specific fine-tuning: Morae's pause detector reaches 69.8% recall against human-annotated pause points.
  • BLV users complete real web tasks at higher rates with a pausing agent (5.50 of 9 tasks) than with fully automated agents (3.90 for Operator, 2.60 for TaxyAI), because pausing surfaces options users did not know existed.
  • The 638-query field-study dataset, annotated for query validity, completion accuracy, and ambiguity, becomes a resource for studying how blind users phrase automation commands and where agents silently diverge.
  • Pausing followed by a screen-reader-friendly generated form is a viable interaction pattern for preference capture, replacing long text re-specification and structured widgets beat text for specifying multiple missing details at once.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A graded pause-necessity score with a per-user threshold—a direction the authors themselves flag—is the natural next experiment; it would test whether the roughly 40% unnecessary pauses actually cost users or simply act as harmless reminders.
  • The mechanism is not inherently accessibility-specific: sighted users also lose agency when an agent silently picks among equal options, so a sighted-user replication would separate accessibility-specific benefits from general agency benefits of pausing.
  • The 'decisions aligned with user preferences' result could be sharpened by eliciting preferences before each task and measuring match afterward; the paper reports alignment without defining how it was judged.
  • A variant that pauses but shows only a text prompt, without the generated UI, would isolate how much of the gain comes from pausing versus from the structured presentation of options; the paper does not run this ablation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces Morae, a UI agent for blind and low-vision (BLV) users that proactively pauses at decision points so users can express preferences, using a multimodal LLM and an explicit three-stage decision rule (critical action planning, ambiguity verification, and decision function). The work is grounded in a one-week field study with four BLV users, producing an annotated dataset of real agent interactions. A technical evaluation on a 256-task test set compares Morae against prompting variants and OpenAI Operator on task success and pause-detection performance. A remote user study with 10 BLV participants reports higher task completion (5.50 vs 3.90 vs 2.60 tasks), more preference-aligned decisions, higher decision entropy, and higher subjective ratings for Morae relative to Operator and TaxyAI.

Significance. Morae targets an important and understudied problem: current UI agents frequently execute tasks end-to-end without surfacing alternatives, which is particularly harmful for BLV users. The design is concrete, and the decision rule A(i)/I(i)/D(i) is transparent and does not involve numeric parameters fitted to the test set. The dataset of real BLV-agent interactions is a useful community resource, and the user study is with the target population and compares against a strong commercial baseline. However, the headline quantitative claims currently rest on a technical metric that conflates pause correctness with task completion, and the user-study success/preference metrics lack operational definitions. These issues are load-bearing and need to be resolved before the magnitude of the claimed benefit can be assessed.

major comments (4)
  1. [§5.1, Table 1] The definition of 'Task Success Rate' for pause-required tasks is 'correctly pause exactly at the annotated step'; no check is made that the task is completed after the user provides input. Since 128 of the 256 test tasks are pause-required, the overall 55.2% vs 53.1% difference is largely a measure of pause-correctness, not end-to-end success. Notably, Morae is worse on no-pause tasks (44.8% vs 55.5% for Operator). The abstract's 'increases task success' claim is therefore not supported by this evaluation unless the metric is renamed or the evaluation is extended to verify post-pause completion. Please report end-to-end success separately for pause-required and no-pause tasks.
  2. [§6.1–§6.2] The user study reports that participants 'successfully completed significantly more tasks' (5.50 vs 3.90 vs 2.60) and made more 'preference-aligned choices' (4.03 vs 2.98 vs 1.92), but neither metric is operationally defined. It is not stated what counted as task success (e.g., evaluator-verified final state, correct item in cart, event saved) or how preference alignment was judged (author annotation, participant self-report, option matching). These are cornerstone results; without clear definitions, the reader cannot assess the validity or magnitude of the claimed benefit.
  3. [§5.2 / Table 1] The technical evaluation reports point estimates only. For 256 binary task outcomes, the 2.1-point overall difference corresponds to roughly five tasks. The paper does not provide confidence intervals, per-task variability, or a statistical test, despite running each task three times and acknowledging nondeterminism. The headline 'outperforming Operator by 2.1%' should be accompanied by an estimate of uncertainty (e.g., bootstrap CIs or a paired test at the task/run level).
  4. [§5.1 / §3.2] The ground truth for pause locations is the authors' manually annotated path 'leading up to the point when the agent paused to request user input' for each task. The reported precision/recall (59.7/69.8) therefore measures agreement with this self-constructed label. Because prompt design used 54 annotated tasks drawn from the same field-study pool, there is a risk of overfitting to the annotators' notion of the 'right' pause point. Please provide evidence that the annotated pause point corresponds to what diverse BLV users would actually want (e.g., a small agreement study), or explicitly discuss this circularity as a limitation.
minor comments (4)
  1. [§7.1] The text says the pausing module covers 'almost two-thirds of the 94% human upper bound,' but the 94% human upper bound is never defined or sourced. Please clarify how it was computed or remove it.
  2. [§5.1] The 'Verifying-First-Step' description says questions were 'resampled 3 times and only picked top-5 as the verification question set.' The resampling procedure is unclear (from what distribution?), and the sentence should be rewritten for reproducibility.
  3. [§5.1] The text says 'We used the official client interface (Figure 8)' for OpenAI Operator, but Figure 8 in Appendix G is a TaxyAI failure case; the correct cross-reference appears to be Figure 4. Please fix.
  4. [Appendix D] There is a typo in the participant quote: 'very helpfull' should be 'very helpful.' Also, Appendix G's Figure 7 would benefit from a zoomed inset or text description, as the screenshot is difficult to read in the PDF.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the core pause-detection algorithm is evaluated on a held-out test set with explicit decision rules, and the user study independently supports the central claim.

full rationale

The paper's central derivation is not circular. Morae's pause decision function D(i) is defined explicitly from A(i) and I(i), which are generated by an LMM answering verification questions at each step. No numeric parameters are fitted to the test set. The validation set (30% of ambiguous tasks) is used only to design prompt templates, and the test set (70%, plus non-ambiguous tasks) is held out. The ground-truth pause labels are manually annotated by the authors, but this is a benchmark construction, not a fitted input: Morae is not trained or tuned to match the test labels, and the evaluation measures generalization to unseen tasks from the same distribution. The task-success definition for pause-required tasks as 'correctly pause exactly at the annotated step' is a metric choice that conflates pause detection with end-to-end completion—a correctness/validity concern, not a circularity, because the score is measured against independent annotations rather than derived from the method's own outputs. The user study provides separate, direct evidence for the main claim: participants completed more tasks with Morae (μ=5.50) than with Operator (μ=3.90) or TaxyAI (μ=2.60), based on actual human task completion. Self-citations in the paper (e.g., CowPilot, prior accessibility work) are contextual and not load-bearing for the novel algorithm or its evaluation. There is no imported uniqueness theorem, no ansatz smuggled via self-citation, and no parameter renamed as a prediction. The 'preference-aligned choices' metric lacks an operational definition, but the paper does not specify that it is computed from Morae's own pause decisions, so circularity cannot be established from the text. Overall, the derivation chain is self-contained and the advertised results are empirical measurements rather than consequences of the system's definitions.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The method has no numeric free parameters fitted to data. The LMM temperature is fixed at 0, the ambiguity indicator A(i) and sufficiency indicator I(i) are hardcoded rules, and prompt templates were drafted from a disclosed validation set (54 of 182 ambiguous tasks) rather than fit numerically. No new physical or conceptual entities are postulated; Morae is a software system. The 'Dynamic Verification of Ambiguous Choices' is a mechanism name, not an invented entity in the graviton sense.

assumptions (4)
  • domain assumption GPT-4o can detect user-choice ambiguity by generating and self-answering verification questions from DOM and screenshot inputs.
    Section 4.1's Stage 2 assumes the LMM's 'yes/no/unanswerable/not important' answers to self-asked questions are accurate enough to drive pause decisions; this is evaluated empirically but not proven beyond the 8 UI types.
  • domain assumption The authors' manually annotated pause steps are the correct pause points for real users.
    Section 5.1 says 'we manually recorded and verified two interaction paths'; the technical accuracy metrics (65.6% pause-required success, 69.8% recall) are measured against this self-constructed ground truth.
  • domain assumption The simplified DOM and screenshot provide sufficient UI context for reliable ambiguity detection.
    Morae relies on TaxyAI's reduced-DOM (Section 3.1) plus a screenshot; if important alternatives are pruned from the DOM, ambiguity detection can miss choices.
  • domain assumption Results on the 256-task test set generalize to the broader web.
    The test set covers 8 UI types and 20 platforms (Section 5.1); generalization to other platforms is assumed in the paper's claims.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Morae: Proactively Pausing UI Agents for User Choices." pith.science (2026). https://pith.science/paper/LOTZNLVD

@misc{pith2026250821456,
  author       = {Pith},
  title        = {Pith review of: Morae: Proactively Pausing UI Agents for User Choices},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LOTZNLVD}},
  note         = {Machine review of arXiv:2508.21456}
}
read the original abstract

User interface (UI) agents promise to make inaccessible or complex UIs easier to access for blind and low-vision (BLV) users. However, current UI agents typically perform tasks end-to-end without involving users in critical choices or making them aware of important contextual information, thus reducing user agency. For example, in our field study, a BLV participant asked to buy the cheapest available sparkling water, and the agent automatically chose one from several equally priced options, without mentioning alternative products with different flavors or better ratings. To address this problem, we introduce Morae, a UI agent that automatically identifies decision points during task execution and pauses so that users can make choices. Morae uses large multimodal models to interpret user queries alongside UI code and screenshots, and prompt users for clarification when there is a choice to be made. In a study over real-world web tasks with BLV participants, Morae helped users complete more tasks and select options that better matched their preferences, as compared to baseline agents, including OpenAI Operator. More broadly, this work exemplifies a mixed-initiative approach in which users benefit from the automation of UI agents while being able to express their preferences.

Figures

Figures reproduced from arXiv: 2508.21456 by the authors.

Figure 1
Figure 1. Morae is an accessible user interface agent that proactively pauses automation at key decision points for blind and [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. TaxyAI’s interface features a task command input, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Morae’s interface features a task command input [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The UI of OpenAI’s Operator Agent (zoomed view). [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: The average ratings for each of three agent conditions (TaxyAI, Morae, Operator) across eight subjective evaluation [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Dataset characteristics from our field study. [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Failure case illustrating OpenAI’s Operator incorrectly reporting the completion of a UI task—changing the slide [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Failure case of TaxyAI highlighting a lack of proactive user preference handling: the agent automatically selected [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows

    cs.SE 2025-07 conditional novelty 7.0 of 10

    A 20-developer controlled study found that the coding agent OpenHands improved task completion by 35 percentage points and halved user effort versus GitHub Copilot, while reducing user understanding of outputs.

  2. Dark Patterns Meet GUI Agents: LLM Agent Susceptibility to Manipulative Interfaces and the Role of Human Oversight

    cs.HC 2025-09 conditional novelty 6.0 of 10

    GUI agents frequently fall for deceptive interface designs, often without recognizing them, and human supervision of agents improves avoidance only partially while introducing new attention and workload costs.

Reference graph

Works this paper leans on

84 extracted references · 56 canonical work pages · cited by 2 Pith papers

  1. [1]

    [n. d.]. Google Calendar. http://calendar.google.com/. Accessed: 2025-04-09

    2025

  2. [2]

    [n. d.]. Google Docs. https://docs.google.com/. Accessed: 2025-04-09

    2025

  3. [3]

    [n. d.]. Introducing Operator. https://openai.com/index/introducing-operator/. Accessed: 2025-04-09

    2025

  4. [4]

    [n. d.]. Project Mariner. https://deepmind.google/technologies/project-mariner/. Accessed: 2025-04-07

    2025

  5. [5]

    [n. d.]. Target. https://www.target.com/. Accessed: 2025-04-09

    work page 2025

  6. [6]

    Salvatore Andolina, Valeria Orso, Hendrik Schneider, Khalil Klouche, Tuukka Ruotsalo, Luciano Gamberini, and Giulio Jacucci. 2018. SearchBot: Support- ing voice conversations with proactive search. In Companion of the 2018 ACM Conference on Computer Supported Cooperative Work and Social Computing. 9–12

    work page 2018

  7. [7]

    Apple. 2024. Mac Automation Scripting Guide: Automating the User Interface. https://developer.apple.com/library/archive/documentation/ LanguagesUtilities/Conceptual/MacAutomationScriptingGuide/ AutomatetheUserInterface.html

    work page 2024

  8. [8]

    Vikas Ashok, Yevgen Borodin, Yury Puzis, and IV Ramakrishnan. 2015. Capti- speak: a speech-enabled web screen reader. InProceedings of the 12th International Web for All Conference. 1–10

    work page 2015

Show all 84 references
  1. [9]

    Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. 2024. Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning. arXiv preprint arXiv:2406.11896 (2024)

    2024 arXiv

  2. [10]

    Cristian-Paul Bara, Sky CH-Wang, and Joyce Chai. 2021. MindCraft: Theory of mind modeling for situated dialogue in collaborative tasks. arXiv preprint arXiv:2109.06275 (2021)

    2021 arXiv

  3. [11]

    Bigham, Anna C

    Jeffrey P. Bigham, Anna C. Cavender, Jeremy T. Brudvik, Jacob O. Wobbrock, and Richard E. Ladner. 2007. WebinSitu: a comparative analysis of blind and sighted browsing behavior. InProceedings of the 9th International ACM SIGACCESS Conference on Computers and Accessibility (Tem...

    2007

  4. [12]

    Bigham, Tessa Lau, and Jeffrey Nichols

    Jeffrey P. Bigham, Tessa Lau, and Jeffrey Nichols. 2009. Trailblazer: enabling blind users to blaze trails through the web. InProceedings of the 14th International Conference on Intelligent User Interfaces (Sanibel Island, Florida, USA) (IUI ’09). Association for Computing Mac...

    2009

  5. [13]

    Not Knowing What You Don’t Know

    Jeffrey P Bigham, Irene Lin, and Saiph Savage. 2017. The Effects of" Not Knowing What You Don’t Know" on Web Accessibility for Blind Web Users. InProceedings of the 19th international ACM SIGACCESS conference on computers and accessibility. 101–109

    2017

  6. [14]

    Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A Plummer. 2022. A dataset for interactive vision-language navigation with unknown command feasibility. In European Conference on Computer Vision . Springer, 312–328

    2022

  7. [15]

    Valerie Chen, Alan Zhu, Sebastian Zhao, Hussein Mozannar, David Sontag, and Ameet Talwalkar. 2024. Need Help? Designing Proactive AI Assistants for Pro- gramming. arXiv preprint arXiv:2410.04596 (2024). Morae: Proactively Pausing UI Agents for User Choices UIST ’25, September ...

    2024 arXiv

  8. [16]

    Weihao Chen, Xiaoyu Liu, Jiacheng Zhang, Ian Iong Lam, Zhicheng Huang, Rui Dong, Xinyu Wang, and Tianyi Zhang. 2023. MIWA: Mixed-Initiative Web Automation for Better User Control and Confidence. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Tec...

    2023

  9. [17]

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems 36 (2023), 28091–28114

    2023

  10. [18]

    Yue Fan, Handong Zhao, Ruiyi Zhang, Yu Shen, Xin Eric Wang, and Gang Wu

  11. [19]

    Daniel Fried, Jacob Andreas, and Dan Klein. 2017. Unified pragmatic models for generating and following instructions. arXiv preprint arXiv:1711.04987 (2017)

    2017 arXiv

  12. [20]

    Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis- Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. 2018. Speaker-follower models for vision-and-language navigation. Advances in neural information processing syst...

    2018

  13. [21]

    Jiaxin Ge, Zora Zhiruo Wang, Xuhui Zhou, Yi-Hao Peng, Sanjay Subramanian, Qinyue Tan, Maarten Sap, Alane Suhr, Daniel Fried, Graham Neubig, et al. 2025. Autopresent: Designing structured visuals from scratch. In Proceedings of the Computer Vision and Pattern Recognition Confer...

    2025

  14. [22]

    Timo Götzelmann. 2016. LucentMaps: 3D printed audiovisual tactile maps for blind and visually impaired people. In Proceedings of the 18th international ACM Sigaccess conference on computers and accessibility . 81–90

    2016

  15. [23]

    Meera Hahn, Wenjun Zeng, Nithish Kannen, Rich Galt, Kartikeya Badola, Been Kim, and Zi Wang. 2024. Proactive Agents for Multi-Turn Text-to-Image Genera- tion Under Uncertainty. arXiv preprint arXiv:2412.06771 (2024)

    2024

  16. [24]

    Leona Holloway, Kim Marriott, and Matthew Butler. 2018. Accessible maps for the blind: Comparing 3D printed models with tactile graphics. In Proceedings of the 2018 chi conference on human factors in computing systems . 1–13

    2018

  17. [25]

    Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems . 159–166

    1999

  18. [26]

    Forrest Huang, Gang Li, Tao Li, and Yang Li. 2024. Automatic Macro Mining from Interaction Traces at Scale. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1–16

    2024

  19. [27]

    Mina Huh, Yi-Hao Peng, and Amy Pavel. 2023. GenAssist: Making image gen- eration accessible. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 1–17

    2023

  20. [28]

    Mina Huh, Saelyne Yang, Yi-Hao Peng, Xiang’Anthony’ Chen, Young-Ho Kim, and Amy Pavel. 2023. AVscript: Accessible Video Editing with Audio-Visual Scripts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–17

    2023

  21. [29]

    Faria Huq, Zora Zhiruo Wang, Frank F Xu, Tianyue Ou, Shuyan Zhou, Jeffrey P Bigham, and Graham Neubig. 2025. CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigation. arXiv preprint arXiv:2501.16609 (2025)

    2025

  22. [30]

    R Dan Jacobson. 1998. Navigating maps with little or no sight: An audio-tactile approach. In Content Visualization and Intermedia Representations (CVIR’98)

    1998

  23. [31]

    Alon Jacovi, Swabha Swayamdipta, Shauli Ravfogel, Yanai Elazar, Yejin Choi, and Yoav Goldberg. 2021. Contrastive explanations for model interpretability. arXiv preprint arXiv:2103.01378 (2021)

    2021 arXiv

  24. [32]

    Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2024. Language models can solve computer tasks. Advances in Neural Information Processing Systems 36 (2024)

    2024

  25. [33]

    Satwik Ram Kodandaram, Utku Uckun, Xiaojun Bi, IV Ramakrishnan, and Vikas Ashok. 2024. Enabling Uniform Computer Interaction Experience for Blind Users through Large Language Models. arXiv preprint arXiv:2407.19537 (2024)

    2024 arXiv

  26. [34]

    Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, et al. 2024. AutoWebGLM: Bootstrap And Reinforce A Large Language Model-based Web Navigating Agent. arXiv preprint arXiv:2404.03648 (2024)

    2024 arXiv

  27. [35]

    Gierad P Laput, Mira Dontcheva, Gregg Wilensky, Walter Chang, Aseem Agar- wala, Jason Linder, and Eytan Adar. 2013. Pixeltone: A multimodal interface for image editing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2185–2194

    2013

  28. [36]

    Microsoft Learn. 2024. UI Automation Overview. https://learn.microsoft.com/en- us/dotnet/framework/ui-automation/ui-automation-overview

    2024

  29. [37]

    Toby Jia-Jun Li, Amos Azaria, and Brad A Myers. 2017. SUGILITE: creating multimodal smartphone automation by demonstration. In Proceedings of the 2017 CHI conference on human factors in computing systems . 6038–6049

    2017

  30. [38]

    Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang

  31. [39]

    Xing Han Lù, Zdeněk Kasner, and Siva Reddy. 2024. Weblinx: Real-world website navigation with multi-turn dialogue. arXiv preprint arXiv:2402.05930 (2024)

    2024

  32. [40]

    Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. 2024. Omniparser for pure vision based gui agent. arXiv preprint arXiv:2408.00203 (2024)

    2024 arXiv

  33. [41]

    Matthew Marge, Carol Espy-Wilson, Nigel G Ward, Abeer Alwan, Yoav Artzi, Mohit Bansal, Gil Blankenship, Joyce Chai, Hal Daumé III, Debadeepta Dey, et al

  34. [42]

    Matthew Marge and Alexander I Rudnicky. 2019. Miscommunication detection and recovery in situated human–robot dialogue. ACM Transactions on Interactive Intelligent Systems (TiiS) 9, 1 (2019), 1–40

    2019

  35. [43]

    Shikhar Murty, Christopher Manning, Peter Shaw, Mandar Joshi, and Kenton Lee. 2024. BAGEL: Bootstrapping Agents by Guiding Exploration with Language. arXiv preprint arXiv:2403.08140 (2024)

    2024 arXiv

  36. [44]

    Magnus Müller and Gregor Žunić. 2024. Browser Use: Enable AI to control your browser. https://github.com/browser-use/browser-use

    2024

  37. [45]

    Anjali Narayan-Chen, Prashant Jayannavar, and Julia Hockenmaier. 2019. Col- laborative dialogue in Minecraft. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics . 5405–5415

    2019

  38. [46]

    Khanh Nguyen and Hal Daumé III. 2019. Help, anna! visual navigation with natural multimodal assistance via retrospective curiosity-encouraging imitation learning. arXiv preprint arXiv:1909.01871 (2019)

    2019 arXiv

  39. [47]

    Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan-Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur. 2022. Teach: Task-driven embodied agents that chat. In Proceedings of the AAAI Conference on Artificial Intellige...

    2022

  40. [48]

    Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, et al. 2024. Webcanvas: Bench- marking web agents in online environments. arXiv preprint arXiv:2406.12373 (2024)

    2024 arXiv

  41. [49]

    Ajay Patel, Markus Hofmarcher, Claudiu Leoveanu-Condrei, Marius-Constantin Dinu, Chris Callison-Burch, and Sepp Hochreiter. 2024. Large Language Models Can Self-Improve At Web Agent Tasks. arXiv preprint arXiv:2405.20309 (2024)

    2024 arXiv

  42. [50]

    Yi-Hao Peng, Jeffrey P Bigham, and Amy Pavel. 2021. Slidecho: Flexible non- visual exploration of presentation videos. In Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility . 1–12

    2021

  43. [51]

    Yi-Hao Peng, Faria Huq, Yue Jiang, Jason Wu, Amanda Xin Yue Li, Jeffrey Bigham, and Amy Pavel. 2024. DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation. In Proceedings of the European Conference on Computer Vision (ECCV)

    2024

  44. [52]

    Yi-Hao Peng, JiWoong Jang, Jeffrey P Bigham, and Amy Pavel. 2021. Say it all: Feedback for improving non-visual presentation accessibility. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems . 1–12

    2021

  45. [53]

    Yi-Hao Peng, Muh-Tarng Lin, Yi Chen, TzuChuan Chen, Pin Sung Ku, Paul Taele, Chin Guan Lim, and Mike Y Chen. 2019. Personaltouch: Improving touchscreen usability by personalizing accessibility settings based on individual user’s touch- screen interaction. In Proceedings of the...

    2019

  46. [54]

    Yi-Hao Peng, Jason Wu, Jeffrey Bigham, and Amy Pavel. 2022. Diffscriber: Describ- ing Visual Design Changes to Support Mixed-ability Collaborative Presentation Authoring. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. 1–13

    2022

  47. [55]

    Zhenhui Peng, Yunhwan Kwon, Jiaan Lu, Ziming Wu, and Xiaojuan Ma. 2019. Design and evaluation of service robot’s proactivity in decision-making support process. In proceedings of the 2019 CHI conference on human factors in computing systems. 1–13

    2019

  48. [56]

    Mahika Phutane, Crescentia Jung, Niu Chen, and Shiri Azenkot. 2023. Speaking with My Screen Reader: Using Audio Fictions to Explore Conversational Access to Interfaces. In Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility . 1–18

    2023

  49. [57]

    Christopher Power, André Freire, Helen Petrie, and David Swallow. 2012. Guide- lines are only half of the story: accessibility problems encountered by blind users on the web. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Austin, Texas, USA) (CH...

    2012

  50. [58]

    Emanuele Pucci, Isabella Possaghi, Claudia Maria Cutrupi, Marcos Baez, Cinzia Cappiello, and Maristella Matera. 2023. Defining Patterns for a Conversational Web. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–17

    2023

  51. [59]

    André Rodrigues. 2015. Breaking barriers with assistive macros. In Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility . 351–352

    2015

  52. [60]

    Hua Shen, Tiffany Knearem, Reshmi Ghosh, Kenan Alkiek, Kundan Krishna, Yachuan Liu, Ziqiao Ma, Savvas Petridis, Yi-Hao Peng, Li Qiwei, et al . 2024. Towards bidirectional human-ai alignment: A systematic review for clarifications, framework, and future directions. arXiv prepri...

    2024

  53. [61]

    Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. 2024. Trial and error: Exploration-based trajectory optimization for llm agents. arXiv preprint arXiv:2403.02502 (2024)

    2024 arXiv

  54. [62]

    Arjun Srinivasan, Mira Dontcheva, Eytan Adar, and Seth Walker. 2019. Discover- ing natural language commands in multimodal interfaces. In Proceedings of the 24th International Conference on Intelligent User Interfaces . 661–672. UIST ’25, September 28-October 1, 2025, Busan, R...

    2019

  55. [63]

    Liangtai Sun, Xingyu Chen, Lu Chen, Tianle Dai, Zichen Zhu, and Kai Yu. 2022. Meta-gui: Towards multi-modal conversational agents on mobile gui. arXiv preprint arXiv:2205.11029 (2022)

    2022 arXiv

  56. [64]

    Maryam Taeb, Amanda Swearngin, Eldon Schoop, Ruijia Cheng, Yue Jiang, and Jeffrey Nichols. 2024. Axnav: Replaying accessibility tests from natural language. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1–16

    2024

  57. [65]

    TaxyAI. 2024. TaxyAI Browser Extension. https://github.com/TaxyAI/browser- extension. Accessed: 2024-09-10

    2024

  58. [66]

    Stefanie Tellex, Ross Knepper, Adrian Li, Daniela Rus, and Nicholas Roy. 2014. Asking for help using inverse semantics. (2014)

    2014

  59. [67]

    Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. 2020. Vision-and-dialog navigation. In Conference on Robot Learning . PMLR, 394–406

    2020

  60. [68]

    Jesse Thomason, Aishwarya Padmakumar, Jivko Sinapov, Nick Walker, Yuqian Jiang, Harel Yedidsion, Justin Hart, Peter Stone, and Raymond J Mooney. 2019. Improving grounded natural language understanding through human-robot dialog. In 2019 International Conference on Robotics and...

    2019

  61. [69]

    Gregg Vanderheiden and Crystal Yvette Marte. 2024. Will AI allow us to dis- pense with all or most accessibility regulations?. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI EA ’24). Association for Computing Machiner...

    2024

  62. [70]

    Sanidhya Vijayvargiya, Xuhui Zhou, Akhila Yerukola, Maarten Sap, and Gra- ham Neubig. 2025. Interactive Agents to Overcome Ambiguity in Software Engineering. arXiv preprint arXiv:2502.13069 (2025)

    2025

  63. [71]

    Bryan Wang, Gang Li, and Yang Li. 2023. Enabling conversational interaction with mobile ui using large language models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . 1–17

    2023

  64. [72]

    Tianlu Wang, Rohit Sridhar, Diyi Yang, and Xuezhi Wang. 2021. Identifying and mitigating spurious correlations for improving robustness in nlp models. arXiv preprint arXiv:2110.07736 (2021)

    2021 arXiv

  65. [73]

    Terry Winograd. 1971. Procedures as a representation for data in a computer program for understanding natural language. (1971)

    1971

  66. [74]

    Jason Wu, Yi-Hao Peng, Xin Yue Amanda Li, Amanda Swearngin, Jeffrey P Bigham, and Jeffrey Nichols. 2024. UIClip: a data-driven model for assessing user interface design. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology. 1–16

    2024

  67. [75]

    Jason Wu, Siyan Wang, Siman Shen, Yi-Hao Peng, Jeffrey Nichols, and Jeffrey P Bigham. 2023. Webui: A dataset for enhancing visual ui understanding with web semantics. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–14

    2023

  68. [76]

    wunderwuzzi23. 2024. System Prompts. https://github.com/wunderwuzzi23/ scratch/tree/master/system_prompts Accessed: April 10, 2025

    2024

  69. [77]

    Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. 2025. An Illusion of Progress? Assessing the Current State of Web Agents. arXiv preprint arXiv:2504.01382 (2025)

    2025

  70. [78]

    Xiaoyi Zhang, Lilian De Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, et al. 2021. Screen recognition: Creating accessibility metadata for mobile applications from pixels. In Proceedings of the 2021 CHI Confere...

    2021

  71. [79]

    Yu Zhong, TV Raman, Casey Burkhardt, Fadi Biadsy, and Jeffrey P Bigham. 2014. JustSpeak: enabling universal voice control on Android. InProceedings of the 11th Web for All Conference. 1–4

    2014

  72. [80]

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. 2023. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854 (2023)

    2023 arXiv

  73. [81]

    I’m not a robot

    Fengda Zhu, Yi Zhu, Xiaojun Chang, and Xiaodan Liang. 2020. Vision-language navigation with self-supervised auxiliary reasoning tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 10012–10022. Morae: Proactively Pausing UI Agents for U...

    2020

  74. [2018]

    arXiv preprint arXiv:1802.08802 (2018)

    Reinforcement learning on web interfaces using workflow-guided explo- ration. arXiv preprint arXiv:1802.08802 (2018)

    2018 arXiv

  75. [2022]

    Computer Speech & Language 71 (2022), 101255

    Spoken language interaction with robots: Recommendations for future research. Computer Speech & Language 71 (2022), 101255

    2022

  76. [2025]

    arXiv preprint arXiv:2501.13896 (2025)

    GUI-Bee: Align GUI Action Grounding to Novel Environments via Au- tonomous Exploration. arXiv preprint arXiv:2501.13896 (2025)

    2025 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.