pith. machine review for the scientific record. sign in

arxiv: 2604.23772 · v2 · submitted 2026-04-26 · 💻 cs.HC

Recognition: unknown

PageGuide: Browser extension to assist users in navigating a webpage and locating information

Anh Totti Nguyen, Chirag Agarwal, Runtao Zhou, Thang T. Truong, Tin Nguyen, Trung Bui

Authors on Pith no claims yet

Pith reviewed 2026-05-08 05:35 UTC · model grok-4.3

classification 💻 cs.HC
keywords browser extensionLLM groundingweb navigationvisual overlaysDOM elementsuser studytask assistanceinformation location
0
0 comments X

The pith

PageGuide is a browser extension that visually grounds LLM answers in webpage HTML elements for finding information, following steps, and hiding distractions.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces PageGuide, a browser extension that connects large language model responses directly to specific parts of a webpage through visual highlights and overlays. It targets three user needs by highlighting evidence for quick verification, showing one instruction at a time for tasks, and allowing selective removal of irrelevant content. In tests with 94 participants, the tool raised hiding accuracy by 26 percentage points, cut task times by as much as 70 percent, lifted guide completion rates by 30 points, and lowered use of manual search commands by 80 percent. This approach lets users see exactly where answers originate on the page instead of accepting outputs without visible proof or searching by hand.

Core claim

PageGuide grounds LLM answers in the HTML DOM via visual overlays for three modes: Find highlights relevant evidence in place for immediate verification; Guide presents step-by-step instructions one at a time so users can perform actions themselves; and Hide lets users conceal distracting elements after deciding on each one. In a user study with 94 participants, PageGuide outperformed unaided browsing with a 26 percentage point gain in hide accuracy, 70 percent faster hide task completion, 30 percentage point higher guide completion rate, 80 percent less Ctrl+F usage in find tasks, and 19 percent shorter find task times.

What carries the argument

Visual overlays that map LLM natural-language outputs to precise, non-overlapping DOM elements and render them as interactive highlights directly on the live webpage.

If this is right

  • Users can verify AI answers instantly by seeing the precise page locations highlighted instead of searching manually.
  • Multi-step tasks such as changing a password become easier because instructions appear one at a time with visual cues.
  • Distracting content can be hidden selectively, improving focus and reducing time spent on cluttered pages.
  • Reliance on browser search tools like Ctrl+F drops sharply when relevant sections are marked automatically.
  • Overall effort for locating information and completing web actions decreases across find, guide, and hide modes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same visual grounding method could be added to other browser agents so their automated actions also show users the affected page elements.
  • Repeated use might help people form clearer expectations about how language models interpret webpage content.
  • Allowing direct user edits to the highlighted elements could create a feedback loop that improves future mappings.
  • Combining this approach with existing web automation tools might let users intervene in real time when an action looks incorrect.

Load-bearing premise

The language model can reliably identify exact webpage elements from user queries without errors or overlaps that force users to double-check every suggestion.

What would settle it

A test case where PageGuide highlights the wrong or overlapping elements for a query, causing users to select incorrect information or take longer than they would without the extension.

Figures

Figures reproduced from arXiv: 2604.23772 by Anh Totti Nguyen, Chirag Agarwal, Runtao Zhou, Thang T. Truong, Tin Nguyen, Trung Bui.

Figure 1
Figure 1. Figure 1: (a) Existing web agents can fail in three ways: by providing answers that cannot be verified from the page, failing view at source ↗
Figure 2
Figure 2. Figure 2: Given the query “How to add ABC to this GitHub project?”, PageGuide (powered by Gemini-3-Flash ) generates a step-by-step plan and delivers it one step at a time. The target UI element is highlighted directly on the page (e.g., Settings , Collaborators )), while the sidebar panel shows the current instruction, the outcome hint (), and Next / Stop controls. The user always drives the pace: each step only a… view at source ↗
Figure 3
Figure 3. Figure 3: On social platforms such as X.com, users often encounter repetitive or distracting content. Given the query view at source ↗
Figure 4
Figure 4. Figure 4: Given a user query, the Router assigns it to one of view at source ↗
Figure 5
Figure 5. Figure 5: Task performance comparing control and extension conditions across all three features. view at source ↗
Figure 6
Figure 6. Figure 6: Task completion time (seconds) for the control and extension conditions, restricted to correctly completed tasks. view at source ↗
Figure 7
Figure 7. Figure 7: Behavioral signals (mean ± SE) comparing the control and extension conditions. Each bar shows the average count or distance per task for five metrics: Ctrl+F presses, text selections, mouse clicks, scroll count, and mouse movement distance. All five metrics decrease substantially with PageGuide, indicating that users rely less on manual search and perform fewer interactions to complete the same tasks. unai… view at source ↗
Figure 12
Figure 12. Figure 12: Using these responses, we address RQ3: do users per￾ceive completing tasks with PageGuide as easier and less effortful compared to completing the same tasks without it? RQ3) Users rate PageGuide as easier to use and effective in helping them locate answers. Participants’ subjective ratings view at source ↗
Figure 8
Figure 8. Figure 8: Behavioral metrics (mean ± SE) broken down by task type (Find, Guide, Hide) and condition. While Find and Hide show consistent reductions across all signals, Guide shows a different pattern: page visits and mouse movement distance increase with PageGuide, reflecting that the extension actively guides users to navigate to new pages as part of the task. F1: PageGuide can correctly locate the information need… view at source ↗
Figure 9
Figure 9. Figure 9: Post-study Likert ratings (1 = Strongly Disagree, 7 = Strongly Agree) for each interaction mode. Each mode includes view at source ↗
Figure 10
Figure 10. Figure 10: Task outcome distributions for Guide and Hide tasks, based on participants’ response to “Did you com￾plete the task?” after each trial (completed / partial / failed), shown as the percentage of participants per category. With PageGuide, completed outcomes increase and failures de￾crease in both features. Notably, Guide shows a rise in partial completions under the extension condition, indicating that user… view at source ↗
Figure 11
Figure 11. Figure 11: Task interfaces displayed to participants during the user study, covering each of view at source ↗
Figure 12
Figure 12. Figure 12: Post-Study Questionnaire view at source ↗
Figure 13
Figure 13. Figure 13: Routing prompt used to classify user queries into appropriate handlers ( view at source ↗
Figure 14
Figure 14. Figure 14: Prompt used to answer user queries grounded in webpage content, producing responses with interleaved references view at source ↗
Figure 15
Figure 15. Figure 15: Hide mode prompt. Given the page element index and the user’s natural language description of content to suppress, the LLM returns a ranked list of up to 15 matching element indices, each paired with a one-sentence hiding reason. The result is used to apply display:none to the identified elements and populate the inline placeholder shown to the user view at source ↗
Figure 16
Figure 16. Figure 16: Guide mode prompt. The LLM receives the current page element index, the user query, the current step number, and any previously completed steps. It returns one action at a time as a JSON object specifying the instruction text, the SoM index and text of the target element, the action type (click, input, scroll), and a next-step hint view at source ↗
Figure 17
Figure 17. Figure 17: Data generation pipelines for the three subsets of PageGuide. (A) The Find pipeline collects webpages, filters for view at source ↗
Figure 18
Figure 18. Figure 18: Users can upload a PDF and ask questions about its contents through the PDF upload feature. In this example, the view at source ↗
Figure 19
Figure 19. Figure 19: PageGuide allows users to upload an image and ask questions about it, while grounding its textual response in evidence from the webpage and specific areas of the user’s image. In this example, the user asks whether the tiger in the uploaded image is the same species as the tiger described in the webpage, and the agent responds with evidence-backed information. Note: The yellow bounding box highlighting th… view at source ↗
Figure 20
Figure 20. Figure 20: With the Page Off feature enabled, users can ask questions that are not related to the currently active tab. The view at source ↗
Figure 21
Figure 21. Figure 21: Example of Gemini Chatbot on the query “How many female casts in Stranger Things?”. The chatbot provides an answer with partial references to text spans, such as “Stranger Things,” but does not link evidence to individual cast names, making verification difficult. In addition, the relevant evidence is embedded within dense text sections, and the highlights include unrelated information, such as male cast … view at source ↗
Figure 22
Figure 22. Figure 22: Example of Browser Use on https://en.wikipedia.org/wiki/Stranger_Things. The agent generates a response that view at source ↗
Figure 23
Figure 23. Figure 23: Example of MolmoWeb [14] on https://en.wikipedia.org/wiki/Elon_Musk. The agent generates a response that correctly answers the query is unable to highlight corresponding evidence, such as through bounding boxes, which makes it difficult for the users to verify the response view at source ↗
Figure 24
Figure 24. Figure 24: Example of Gemini Agent on https://us.megabus.com/. The agent generates a response that correctly answers the view at source ↗
Figure 25
Figure 25. Figure 25: Given the query “How to find the time frame for finding a lost item?” on https://us.megabus.com/, PageGuide (powered by Gemini-3-Flash ) not only generates a step-by-step plan but also highlights the relevant text spans on the webpage, making it easier for users to locate and verify the information view at source ↗
read the original abstract

Users browsing the web daily struggle to quickly locate relevant information in cluttered pages, complete unfamiliar multi-step tasks, and stay focused amid distracting content. State-of-the-art AI assistants (e.g., ChatGPT, Gemini, Claude) and browser agents (e.g., OpenAI Operator, Browser Use) can answer questions and automate actions, yet they return answers without showing where the information comes from on the page, forcing users to manually verify results and blindly trust every automated steps. We present PageGuide, a browser extension that grounds LLM answers directly in the HTML DOM via visual overlays, addressing three core user needs: (a) Find-locating and highlighting relevant evidence in-situ so users can instantly verify answers on the page; (b) Guide-showing step-by-step instructions (e.g. how to change password) one at a time so users can follow and perform actions by themselves; and (c) Hide-hiding distracting content-giving users a chance to decide to hide an element or not. In a user study (N=94), PageGuide outperform unaided browsing across all modes: Hide accuracy improve by 26 percentage points (86.7% relative gain) and task completion time drops by 70%; Guide completion rate increases by 30 percentage points; and Find reduces manual search effort, with Ctrl+F usage falling by 80% and task time decreasing by 19%. Code and demo is at: pageguide.github.io.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript presents PageGuide, a browser extension that grounds LLM outputs directly in a webpage's HTML DOM using visual overlays and step-by-step guidance. It targets three user needs: Find (highlighting relevant evidence in place), Guide (sequential instructions for multi-step tasks), and Hide (selectively removing distractions). The central empirical claim, based on a controlled user study with N=94 participants, is that PageGuide yields large gains over unaided browsing: +26 percentage points in Hide accuracy (86.7% relative), +30pp in Guide completion rate, 80% drop in Ctrl+F usage for Find, and task-time reductions of 70% (Hide) and 19% (Find).

Significance. If the grounding reliability and study results hold, the work fills a practical gap between current LLM assistants (which answer without page context) and browser agents (which act without user verification). The measured effect sizes across multiple modes are substantial for an HCI system paper, and the public code/demo at pageguide.github.io supports reproducibility and extension. This could inform design of future grounded web-AI tools.

major comments (2)
  1. [§5] §5 (User Study) and §5.3 (Results): No quantitative metrics are reported for LLM-to-DOM grounding accuracy (precision, recall, or error rate on element selection). The observed gains presuppose reliable, non-overlapping highlights and instructions; without these figures or a failure-case analysis, it is unclear whether benefits persist when grounding errs or requires manual correction.
  2. [§5.2] §5.2 (Experimental Design): The description of baseline conditions (unaided browsing), statistical tests for the reported differences, and handling of grounding failures or task selection criteria is incomplete. These details are load-bearing for interpreting the N=94 results and the claimed outperformance across Hide, Guide, and Find modes.
minor comments (2)
  1. [Abstract] Abstract: The performance claims would be stronger if they briefly noted whether differences reached statistical significance or included confidence intervals.
  2. [§4] Figure captions and §4 (System Architecture): Clarify how overlapping or ambiguous DOM elements are resolved in the visual overlays, as this directly affects user experience in Find and Guide modes.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback on our manuscript. We have revised the paper to address the concerns about the user study by adding the requested quantitative details and clarifications.

read point-by-point responses

  1. Referee: [§5] §5 (User Study) and §5.3 (Results): No quantitative metrics are reported for LLM-to-DOM grounding accuracy (precision, recall, or error rate on element selection). The observed gains presuppose reliable, non-overlapping highlights and instructions; without these figures or a failure-case analysis, it is unclear whether benefits persist when grounding errs or requires manual correction.

    Authors: We agree that explicit grounding accuracy metrics and failure analysis would strengthen the interpretation of the user study results. In the revised manuscript, we have added a new paragraph in §5.3 reporting precision (0.87) and recall (0.82) for LLM-to-DOM element selection, computed from logged interactions across all 94 participants and tasks. We also include a failure-case analysis describing the 14% of cases with imperfect grounding and how the interface supported user correction via re-query or manual override. These additions demonstrate that the reported performance gains held even when occasional grounding errors occurred. revision: yes

  2. Referee: [§5.2] §5.2 (Experimental Design): The description of baseline conditions (unaided browsing), statistical tests for the reported differences, and handling of grounding failures or task selection criteria is incomplete. These details are load-bearing for interpreting the N=94 results and the claimed outperformance across Hide, Guide, and Find modes.

    Authors: We appreciate this observation and have expanded §5.2 substantially in the revision. The baseline is now described as participants using only native browser functionality (no AI, no overlays, standard Ctrl+F and scrolling). We report the statistical tests (paired t-tests for time measures and McNemar's tests for binary outcomes, all p < 0.01). Grounding failures were handled by allowing on-demand regeneration, and tasks were selected from a predefined set of 12 realistic scenarios balanced across the three modes. These details are now fully specified to support evaluation of the N=94 results. revision: yes

Circularity Check

0 steps flagged

No circularity: purely empirical user-study evaluation with no derivations or self-referential predictions

full rationale

The paper describes a browser extension implementing three interaction modes (Find, Guide, Hide) that rely on LLM-based DOM grounding, then reports direct performance measurements from a controlled user study (N=94) comparing PageGuide against unaided browsing. No equations, fitted parameters, or predictive models are present; the reported gains (e.g., +26pp Hide accuracy, +30pp Guide completion, -80% Ctrl+F usage) are raw empirical outcomes, not quantities derived from or forced by the input data. No self-citations, uniqueness theorems, or ansatzes are invoked to justify core claims. The LLM-to-DOM mapping is an implementation detail whose accuracy is implicitly tested via end-to-end task metrics rather than being presupposed in a circular manner. The derivation chain is therefore empty and self-contained.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The work is an applied system plus empirical evaluation; it relies on standard assumptions about LLM capabilities and user-study validity rather than introducing new mathematical entities or fitted constants.

pith-pipeline@v0.9.0 · 5577 in / 1186 out tokens · 55790 ms · 2026-05-08T05:35:13.728432+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

77 extracted references · 26 canonical work pages · 3 internal anchors

  1. [1]

    Chirag Agarwal, Sree Harsha Tanneru, and Himabindu Lakkaraju. 2024. Faithful- ness vs. plausibility: On the (un) reliability of explanations from large language models.arXiv preprint arXiv:2402.04614(2024)

    work page arXiv 2024

  2. [2]

    Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, et al. 2019. Guidelines for Human-AI Interaction. InProceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 3:1–3:13. doi:10.1145/ 3290605.3300233

    work page arXiv 2019

  3. [3]

    Anthropic. 2024. 3.5 Sonnet, Computer Use, and the API updates. Anthropic announcement. https://www.anthropic.com/news/3-5-models-and-computer- use Accessed: 2026-03-03

    2024

  4. [4]

    Anthropic. 2026. Get started with Claude in Chrome. Anthropic Help Cen- ter. https://support.claude.com/en/articles/12012173-get-started-with-claude- in-chrome Accessed: 2026-03-31

    work page arXiv 2026

  5. [5]

    Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Victor Carbune, Jason Lin, Jindong Chen, and Abhanshu Sharma

  6. [6]

    Screenai: A vision-language model for ui and infographics understanding

    ScreenAI: A vision-language model for UI and infographics understanding. arXiv preprint arXiv:2402.04615

    work page arXiv

  7. [7]

    Longformer: The Long-Document Transformer

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The Long- Document Transformer. arXiv preprint arXiv:2004.05150

    work page internal anchor Pith review arXiv 2020

  8. [8]

    browser-use. 2025. browser-use: Make websites accessible for AI agents. GitHub repository. https://github.com/browser-use/browser-use Accessed: 2026-03-02

    2025

  9. [9]

    Phil Cuvin, Hao Zhu, and Diyi Yang. 2025. DECEPTICON: How Dark Patterns Manipulate Web Agents. arXiv preprint arXiv:2512.22894

    work page arXiv 2025

  10. [10]

    Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers anchored in research papers. InProceedings of the 2021 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies. 4599–4610

    2021

  11. [11]

    DataReportal, We Are Social, and Meltwater. 2025. Digital 2025: Global Overview Report. Annual global digital report. https://datareportal.com/reports/digital- 2025-global-overview-report

    2025

  12. [12]

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2Web: Towards a Generalist Agent for the Web. InAdvances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., 28091–28114. https://proceedings.neurips.cc/pa...

    2023

  13. [13]

    Google. 2025. Bring Gemini to Chrome. Google Blog. https://blog.google/ products-and-platforms/products/chrome/gemini-3-auto-browse/ Accessed: 2026-03-02

    2025

  14. [14]

    Ananya Gubbi Mohanbabu, Yotam Sechayk, and Amy Pavel. 2025. Task Mode: Dynamic Filtering for Task-Specific Web Navigation using LLMs. InProceedings of the 27th International ACM SIGACCESS Conference on Computers and Accessibility. ACM, New York, NY, USA, 1–18

    2025

  15. [15]

    Tanmay Gupta, Piper Wolters, Zixian Ma, Peter Sushko, Rock Yuren Pang, Diego Llanes, Yue Yang, Taira Anderson, Boyuan Zheng, Zhongzheng Ren, Harsh Trivedi, Taylor Blanton, Caleb Ouellette, Winson Han, Ali Farhadi, and Ranjay Krishna. 2026. MolmoWeb: Open Visual Web Agent and Open Data for the Open Web. https://allenai.org/papers/molmoweb

    2026

  16. [16]

    GWI. 2025. Ad blockers in 2025: Key trends and what they mean for advertisers. GWI insights article. https://www.gwi.com/blog/ad-blockers

    2025

  17. [17]

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Assoc...

    work page doi:10.18653/v1/2024.acl-long.371 2024

  18. [18]

    Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(Pittsburgh, Pennsylvania, USA)(CHI ’99). Association for Computing Machinery, New York, NY, USA, 159–166. doi:10.1145/302979.303030

    work page doi:10.1145/302979.303030 1999

  19. [19]

    Shagun Jhaver, Alice Qian Zhang, Quan Ze Chen, Nikhila Natarajan, Ruotong Wang, and Amy X Zhang. 2023. Personalizing content moderation on social media: User perspectives on moderation choices, interface design, and labor. Proceedings of the ACM on Human-Computer Interaction7, CSCW2 (2023), 1–33

    2023

  20. [20]

    Jaylen Jones, Zhehao Zhang, Yuting Ning, Eric Fosler-Lussier, Pierre-Luc St- Charles, Yoshua Bengio, Dawn Song, Yu Su, and Huan Sun. 2026. When Benign Inputs Lead to Severe Harms: Eliciting Unsafe Unintended Behaviors of Computer- Use Agents. arXiv preprint arXiv:2602.08235

    work page arXiv 2026

  21. [21]

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. 2024. Vi- sualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), Lun-Wei Ku, ...

    work page doi:10.18653/v1/2024.acl-long.50 2024

  22. [22]

    Thomas Kuntz, Agatha Duzan, Hao Zhao, Francesco Croce, Zico Kolter, Nicolas Flammarion, and Maksym Andriushchenko. 2025. OS-Harm: A benchmark for measuring safety of computer use agents. arXiv preprint arXiv:2506.14866

    work page arXiv 2025

  23. [23]

    Transactions of the Association for Computational Linguistics , author =

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural Questions: A Benchmark for Question Answering Research. I...

    work page doi:10.1162/tacl_a_00276 2019

  24. [24]

    Hao-Ping (Hank) Lee, Yi-Shyuan Chiang, Lan Gao, Stephanie Yang, Philipp Winter, and Sauvik Das. 2025. Purpose Mode: Reducing Distraction through Toggling Attention Capture Damaging Patterns on Social Media Web Sites.ACM Trans. Comput.-Hum. Interact.32, 1, Article 10 (April 2025), 41 pages. doi:10.1145/ 3711841

    2025

  25. [25]

    Ido Levy, Ben Wiesel, Sami Marreed, Alon Oved, Avi Yaeli, and Segev Shlomov

  26. [26]

    arXiv preprint arXiv:2410.06703 , year=

    ST-WebAgentBench: A benchmark for evaluating safety and trustworthi- ness in web agents. arXiv preprint arXiv:2410.06703

    work page arXiv

  27. [27]

    Ping Liu, Karthik Shivaram, Aron Culotta, Matthew Shapiro, and Mustafa Bilgic

  28. [28]

    InProceedings of the International AAAI Conference on Web and Social Media, Vol

    How Does Empowering Users with Greater System Control Affect News Filter Bubbles?. InProceedings of the International AAAI Conference on Web and Social Media, Vol. 18. 943–957

  29. [29]

    Elita Lobo, Chirag Agarwal, and Himabindu Lakkaraju. 2025. On the impact of fine-tuning on chain-of-thought reasoning. InProceedings of the 2025 Confer- ence of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 11679–11698

    2025

  30. [30]

    Kai Mei, Jiang Guo, Shuaichen Chang, Mingwen Dong, Dongkyu Lee, Xing Niu, and Jiarong Jiang. 2026. R-WoM: Retrieval-augmented World Model For Computer-use Agents. arXiv:2510.11892 [cs.CL] https://arxiv.org/abs/2510.11892

    work page arXiv 2026

  31. [31]

    Yohei Nakajima. 2023. BabyAGI. GitHub repository. https://github.com/ yoheinakajima/babyagi Accessed: 2026-03-03

    2023

  32. [32]

    Tin Nguyen, Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen

  33. [33]

    arXiv preprint arXiv:2503.02003 (2025)

    HoT: Highlighted Chain of Thought for Referencing Supporting Facts from Inputs. arXiv preprint arXiv:2503.02003. doi:10.48550/arXiv.2503.02003 Version 5

    work page doi:10.48550/arxiv.2503.02003

  34. [34]

    OpenAI. 2025. ChatGPT Agent: Browsing and completing tasks on the web (Atlas). OpenAI product page. https://chatgpt.com/features/agent Accessed: 2026-03-02

    2025

  35. [35]

    OpenAI. 2025. Introducing Operator. OpenAI product announcement. https: //openai.com/index/introducing-operator/ Accessed: 2026-03-03

    2025

  36. [36]

    Significant Gravitas. 2023. Auto-GPT. GitHub repository. https://github.com/ Significant-Gravitas/AutoGPT Accessed: 2026-03-03

    2023

  37. [37]

    Stanford News. 2025. Social media research tool lowers the political temperature. (November 2025). https://news.stanford.edu/stories/2025/11/social-media-tool- polarization-user-control-research Accessed: 2026-03-26

    2025

  38. [38]

    John Sweller. 1988. Cognitive Load During Problem Solving: Effects on Learning. Cognitive Science12, 2 (1988), 257–285. doi:10.1207/s15516709cog1202_4

    work page doi:10.1207/s15516709cog1202_4 1988

  39. [39]

    Sree Harsha Tanneru, Dan Ley, Chirag Agarwal, and Himabindu Lakkaraju. 2024. On the Difficulty of Faithful Chain-of-Thought Reasoning in Large Language Models. InTrustworthy Multi-modal Foundation Models and AI Agents (TiFA). https://openreview.net/forum?id=3h0kZdPhAC

    2024

  40. [40]

    The Browser Company. 2025. Dia Browser. Official product page. https: //www.diabrowser.com/ Accessed: 2026-03-02

    2025

  41. [41]

    Ada Defne Tur, Nicholas Meade, Xing Han Lu, Alejandra Zambrano, Arkil Pa- tel, Esin Durmus, Spandana Gella, Karolina Stanczak, and Siva Reddy. 2025. SafeArena: Evaluating the safety of autonomous web agents. arXiv preprint arXiv:2503.04957

    work page arXiv 2025

  42. [42]

    Maria Wang, Srinivas Sunkara, Gilles Baechler, Jason Lin, Yun Zhu, Fedir Zubach, Lei Shu, and Jindong Chen. 2024. WebQuest: A benchmark for multimodal QA on web page sequences. arXiv preprint arXiv:2409.13711

    work page arXiv 2024

  43. [43]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. InAdvances in , , Nguyen et al., Tin Nguyen, Thang T. Truong, Runtao Zhou, Trung Bui, Chirag Agarwal, and Anh Totti Nguyen Neural Information Processing Systems, S. K...

    2022

  44. [44]

    Wikipedia contributors. 2026. List of most-visited websites — Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/wiki/List_of_most-visited_websites. Accessed: 2026-04-01

    2026

  45. [45]

    Chen Xiang, Yuchen Zeng, Xiaofei Wang, et al. 2025. CoW Pilot: A Framework for Human-in-the-Loop Web Agents. InProceedings of the 2025 Annual Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations). Association for Computational Linguistics, Albuquerque, New Mexico, 61–72. doi:10.18653/v1/2025.naacl-demo.6

    work page doi:10.18653/v1/2025.naacl-demo.6 2025

  46. [46]

    Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. 2025. An Illusion of Progress? Assessing the Current State of Web Agents. arXiv:2504.01382 [cs.AI] https://arxiv.org/abs/2504.01382

    work page arXiv 2025

  47. [47]

    Roosta, and Tianmin Shu

    Suyu Ye, Haojun Shi, Darren Shih, Hyokun Yun, Tanya G. Roosta, and Tianmin Shu. 2026. RealWebAssist: A Benchmark for Long-Horizon Web Assistance with Real-World Users.Proceedings of the AAAI Conference on Artificial Intelligence40, 40 (Mar. 2026), 34441–34449. doi:10.1609/aaai.v40i40.40742

    work page doi:10.1609/aaai.v40i40.40742 2026

  48. [48]

    Zhexin Zhang, Shiyao Cui, Yida Lu, Jingzhuo Zhou, Junxiao Yang, Hongning Wang, and Minlie Huang. 2024. Agent-SafetyBench: Evaluating the safety of LLM agents. arXiv preprint arXiv:2412.14470

    work page internal anchor Pith review arXiv 2024

  49. [49]

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024. Gpt-4v (ision) is a generalist web agent, if grounded.arXiv preprint arXiv:2401.01614(2024)

    work page arXiv 2024

  50. [50]

    Runtao Zhou, Giang Nguyen, Nikita Kharya, Anh Nguyen, and Chirag Agarwal

  51. [51]

    InProceedings of the 31st International Conference on Intelligent User Interfaces (IUI ’26)

    Improving Human Verification of LLM Reasoning through Interactive Explanation Interfaces. InProceedings of the 31st International Conference on Intelligent User Interfaces (IUI ’26). Association for Computing Machinery, New York, NY, USA, 456–473. doi:10.1145/3742413.3789134

    work page doi:10.1145/3742413.3789134

  52. [52]

    WebArena: A Realistic Web Environment for Building Autonomous Agents

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024. WebArena: A Realistic Web Environment for Building Autonomous Agents. arXiv:2307.13854 [cs.AI] https://arxiv.org/abs/2307.13854 PageGuide: Browser extension to assist users in navigating a webp...

    work page internal anchor Pith review arXiv 2024

  53. [53]

    guide" - For step-by-step

    "guide" - For step-by-step "how to" questions that need interactive guidance

  54. [54]

    "hide" - For requests to hide, remove, or suppress distracting/annoying content (ads, banners, popups, cookie notices, sidebars, recommendations, etc.)

  55. [55]

    image_find

    "image_find" - For questions about an UPLOADED IMAGE (finding similar items, comparing with page content)

  56. [56]

    pdf_find

    "pdf_find" - For questions about PDF documents (summarize, find specific content, extract info from PDFs)

  57. [57]

    find" - For questions, information lookup, finding content, highlighting elements (DEFAULT) ROUTING RULES: -

    "find" - For questions, information lookup, finding content, highlighting elements (DEFAULT) ROUTING RULES: - "guide": User wants to LEARN how to do something in steps (e.g.,"how do I report this video?", "where can I find settings?", "help me delete my account") - "hide": User wants to hide or remove something on the page (e.g.,"hide the ads", "remove th...

  58. [58]

    Answer the question based on the page content if possible

  59. [59]

    text"] citations inline to reference specific elements from the PAGE INDEX - N is the index number from PAGE INDEX -

    If the page content has the answer, use [N:"text"] citations inline to reference specific elements from the PAGE INDEX - N is the index number from PAGE INDEX - "text" is the EXACT text snippet to highlight (copy from the page content)

  60. [60]

    Each citation should point to an element that supports that part of your answer

  61. [61]

    For lists of items, cite each one with the specific text to highlight

  62. [62]

    Use ONE citation per item (if same text has multiple indices, pick the link)

  63. [63]

    The "text" should be a short, specific phrase (not the entire element text)

  64. [64]

    Consider conversation history for context, but always answer based on CURRENT page content

  65. [65]

    Wikipedia’s [1], [2], [3]) — only use [N:"text"] format where N comes from the PAGE INDEX above

    NEVER reproduce existing footnote markers from the webpage itself (e.g. Wikipedia’s [1], [2], [3]) — only use [N:"text"] format where N comes from the PAGE INDEX above

  66. [66]

    The information is not provided on this page

    **CRITICAL**: If the information is NOT provided on this page: - State exactly: "The information is not provided on this page. " - Then, providing the answer using your own general knowledge base is HIGHLY ENCOURAGED. Do not simply stop after stating it is not on the page. - You MUST include citations to real, valid source URLs using STANDARD MARKDOWN LIN...

  67. [67]

    PAGE INDEX - Visible elements on the page

  68. [68]

    USER QUESTION - What the user wants to do

  69. [69]

    STEP NUMBER - Current step (1 = first step)

  70. [70]

    " or

    PREVIOUS STEPS - What was done before (if any) Your job: Guide the user ONE STEP at a time. IMPORTANT CONCEPTS: - Some buttons/options are HIDDEN in menus (like "... " or " ..." three-dot menus) - If the target isn’t visible, guide user to open the menu FIRST - Common hidden locations: dropdown menus, "More" buttons, three-dot menus, right-click menus, se...

  71. [71]

    ONE step at a time - don’t overwhelm the user

  72. [72]

    If target is likely hidden in a menu, first step should open that menu

  73. [73]

    waitFor":

    Use "waitFor": "click" when user needs to click something

  74. [74]

    isLastStep

    Set "isLastStep": true only when the goal is achieved

  75. [75]

    Make instructions clear and specific

  76. [76]

    Highlight the element user needs to interact with EXAMPLES: PAGE INDEX:

  77. [77]

    How do I report this video?

    (button) Save Q: "How do I report this video?" (Step 1) → { "step":1, "instruction":"Click the three-dot menu ( ...) to see more options", "highlight":"index":5, "text":" ...", "wait- For":"click", "isLastStep":false, "nextStepHint":"The menu will open with Report option" } Q: "How do I report this video?" (Step 2, after menu opened) PAGE INDEX now shows:...

    work page arXiv