Pith. sign in

REVIEW 3 major objections 6 minor 59 references

Temac: Multi-Agent Collaboration for Automated Web GUI Testing

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Temac couples a fast web GUI crawler with four LLM agents that infer and execute untested functionalities, reporting 12.5–60.3% average coverage gains over state-of-the-art baselines and 445 unique faults in 20 real-world sites.

desk verdict Useful multi-agent architecture and mostly consistent empirical evidence, but the one-hour budget claim is not yet like-for-like and the headline improvement range is misreported. read the letter →

arxiv 2506.00520 v1 pith:VXLIG4OI submitted 2025-05-31 cs.SE

Chenxu Liu , Zhiyu Gu , Guoquan Wu , Ying Zhang , Jun Wei , Tao Xie This is my paper · ORCID
classification cs.SE
keywords webGUItestinglargelanguagemodelsmulti-agentcollaborationcodecoveragestatetransitiongraphLLMagentscoverage-guided
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that automated web GUI testing gets substantially better when a fast, non-LLM crawler and a team of LLM agents work in sequence rather than either working alone. After the crawler has explored the application for half an hour, Temac converts what the crawler saw into a concise textual knowledge base, uses an LLM to infer which complex functionalities are still untested, and then sends specialized LLM agents to navigate to and execute those functionalities. The payoff, if the paper is right, is a testing approach that keeps the broad coverage of classical crawlers while gaining the ability to complete long, meaningful action sequences that those crawlers abandon midway. The paper reports average code-coverage improvements of 12.5% to 60.3% over four state-of-the-art baselines on six open-source applications, and 445 unique faults found in 20 real-world web applications.

What carries the argument

The load-bearing mechanism is a four-part textual knowledge base plus four specialized LLM agents that consume it. The knowledge base stores natural-language descriptions of each GUI state (generated by Summarizer from screenshots), a simplified state transition graph with self-loops removed, the 50 source files with the lowest coverage, and application-specific knowledge such as login credentials. Reviser turns this knowledge base into ten targeted testing tasks; Navigator selects the single most relevant state per task and supplies the shortest path from the home state; Executor decouples planning from acting, with a planner choosing the next action and a grounding model (UI-TARS) locating and performing it. The state-transition-guided navigation is the part that injects application-specific domain knowledge into the LLM, addressing the known low success rate of LLM web agents.

What would settle it

A concrete check: on the same six applications, run Temac with the Navigator's chosen starting states replaced by randomly selected states from the transition graph, keeping the Summarizer, Reviser, and Executor unchanged. If average code coverage does not drop below the reported 49.7%, then state-transition-guided navigation is not what drives the improvement. A second check would replace the Summarizer's descriptions with generic page titles; if coverage remains unchanged, the semantic summarization step is not load-bearing.

Watch

Extended reading notes

Core claim

The central discovery is that the state transition graph and coverage reports produced by an automated web GUI tester can be repurposed as domain knowledge that makes LLM agents reliable enough to test uncovered complex functionalities. Temac runs an existing AWGT approach (in the implementation, WebRLED) for a fixed period; then the Summarizer agent translates each GUI state from a screenshot into a one-line natural-language description, the Reviser agent combines those descriptions with the 50 lowest-coverage source files to produce ten concrete testing tasks, the Navigator agent picks the most relevant state for each task and extracts the shortest path from the home state, and the Executor agent—split into a planner and a visual grounding actor—carries out the task along that path. The paper reports that this three-phase pipeline outperforms all four baselines on all six subjects, with average coverage rising from roughly 44% for the strongest baseline to 49.7% overall, and it attributes the gain to the complementarity of breadth (crawler) and depth (LLM agents).

Load-bearing premise

The approach rests on whether the Summarizer's natural-language page descriptions and the coverage report are accurate enough for the Reviser to infer genuinely missing functionalities, and for the Navigator to choose a starting state whose shortest path from the home state is a workable route for the Executor.

Editorial extensions

If this is right

  • Full Temac beats both of its ablations: on average 12.5% higher code coverage than the crawler alone (WebRLED) and 22.9% higher than the LLM agents alone (Temac-LLM).
  • Both knowledge-base components contribute: removing the state transition graph from task execution lowers average coverage by 3.5%, and removing the coverage report lowers it by 5.4%.
  • The coverage curves for several applications have not plateaued at the one-hour budget, so the paper expects extended budgets of 6 to 12 hours to reveal further gains.
  • Because Temac is modular, the same pipeline works with a different AWGT approach, state abstraction, MLLM, or actor model, so the reported gains are not tied to WebRLED or GPT-4o specifically.
  • Using the no-coverage-report variant on the top 20 real-world sites, Temac found 445 unique faults, with 78.65% of them traced to the applications themselves.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested corollary is that the same crawl-then-summarize-then-execute loop could transfer to mobile GUI testing, where state graphs are smaller and LLM grounding success rates are higher, so the knowledge-base benefit might be even larger there.
  • The paper uses low-coverage files as a proxy for missing GUI workflows; a direct test would be whether each Reviser-generated task, after execution, actually raises coverage in the specific files it was derived from, and if not, the coverage-report guidance is heuristic rather than causal.
  • Since the 445 real-world faults were found with the coverage-report component disabled, the full knowledge base might find additional faults on live sites; this is a measurable prediction, not something the paper itself claims.
  • The one-hour, roughly $1.7 cost estimate with GPT-4o suggests Temac is cheap enough for continuous testing, so a natural extension is to measure coverage-per-dollar against the baselines rather than coverage alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Temac, a multi-agent LLM-based testing system layered on top of a non-LLM automated web GUI testing (AWGT) tool. Temac operates in three phases: (i) exploration, where an existing AWGT approach (WebRLED) runs for 30 minutes and collects states, actions, screenshots, and coverage reports; (ii) knowledge-base construction, where a Summarizer agent converts screenshots and the state transition graph into natural-language descriptions and a Reviser agent infers not-covered functionalities; and (iii) task execution, where a Navigator agent selects a key state from the graph and a planner-actor-decoupled Executor agent performs the inferred tasks. The evaluation compares Temac against Crawljax, FragGen, WebExplor, and WebRLED on six open-source web applications, reporting line-coverage improvements, and applies a variant (Temac-noCR) to 20 popular real-world sites, reporting 445 unique faults. The paper claims Temac is the first LLM-enhanced AWGT approach targeting code coverage and that its results demonstrate effectiveness and general applicability.

Significance. If the reported results hold, Temac would be a useful and practical contribution: it combines the speed and breadth of non-LLM exploration with the semantic and planning capabilities of LLM agents, and it does so with a modular architecture that could accommodate different underlying AWGT tools, MLLMs, and actors. The paper also provides ablations (Temac-LLM, Temac-RL, Temac-noSTG, Temac-noCR) that give evidence for the contribution of each component, and the consistency of the coverage improvement across all six applications is encouraging. However, the headline quantitative claims contain a reporting error, and the evaluation's time-budget fairness is currently questionable because knowledge-base construction time is excluded from Temac's one-hour budget. These issues are load-bearing and need to be addressed before the central claims can be accepted as stated.

major comments (3)
  1. [V-B, Table III, Abstract] The claim of "average improvement ranging from 12.5% to 60.3%" is not supported by the reported numbers as a range. The value 12.5% is the average relative improvement of Temac over WebRLED computed from Table IV (44.18% to 49.71%, i.e., (49.71−44.18)/44.18 ≈ 12.5%), while 60.3% is the absolute line coverage achieved by Temac on the Timeoff application in Table III. The manuscript never reports per-application relative improvements, so no range of improvements of the form stated in the abstract, introduction, and conclusion exists. Please compute and report per-application relative improvements using the stated formula, or rephrase the summary statistic to distinguish average relative improvement from per-application absolute coverage.
  2. [IV and V-A] The claimed one-hour time budget comparison is unfair because Temac's knowledge-base construction time is explicitly excluded from the evaluation, and the implementation executes this phase sequentially after the 30-minute exploration phase. Thus Temac's actual wall-clock time is 60 minutes plus the duration of the Summarizer, Reviser, Navigator, and Executor setup and knowledge-base construction, while the four baselines are capped at exactly 60 minutes. Since Section V-B states that Temac's coverage curves for 4gaBoards, Gadael, and Parabank are "not yet stagnated," the additional wall-clock time is a plausible partial explanation for at least some of the observed coverage gain. Please either include the knowledge-base construction time in the one-hour budget, or run a control that grants the strongest baseline the same additional time, and report the time breakdown.
  3. [V-F and Abstract] The abstract states that "Temac" reveals 445 unique faults in the top 20 real-world web applications, but RQ5 explicitly uses the Temac-noCR variant, not the full Temac system, because coverage reports are unavailable for real-world sites. This is an overstatement of the evaluated system: the 445 faults were produced by a configuration that omits the coverage-report component whose contribution is a central part of the paper's contribution. Moreover, RQ5 does not run any baseline (e.g., WebRLED alone) on the same 20 applications, so the reader cannot assess how many of these faults are specifically attributable to Temac's multi-agent layer. Please qualify the fault-detection claim to refer to Temac-noCR and add a baseline comparison, or discuss this as an exploratory finding rather than as evidence for the full Temac approach.
minor comments (6)
  1. [V-A / Figure 6] The legend in Figure 6 uses "WebExplore" while the text and tables consistently use "WebExplor"; please unify the naming.
  2. [V-A] The paper reports three repetitions per experiment but provides no variance information, confidence intervals, or statistical tests, despite acknowledging that AWGT approaches exhibit randomness; please add at least the per-application variance or standard deviation.
  3. [V-A / Table II] The application named "4gaBoards" in the text and tables appears as "4ga Boards" in Table II; please make the naming consistent.
  4. [V-F] The list of the "top 20" Alexa applications and the date of the ranking used are not provided, which hinders reproducibility of the RQ5 fault-detection results; please include the list and the ranking date.
  5. [VII-B] The estimated cost of "approximately $1.7 per one-hour testing process" does not appear to account for the UI-TARS actor served on an A100 GPU or for any additional API calls beyond the four agents; please clarify what cost components are included in this estimate.
  6. [IV / Reference [29]] The provided open-source repository link points to a Google Drive folder labeled "Seeker" rather than "Temac"; please verify that the link and label match the project name.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Temac's coverage improvements are measured outcomes of a transparent feedback loop, not predictions equivalent to its inputs.

full rationale

Temac's claimed derivation chain is empirical rather than mathematical: it runs WebRLED for 30 minutes, constructs a knowledge base that explicitly includes the coverage report, asks the Reviser agent to infer tasks targeting low-coverage files, and then measures code coverage after task execution. The coverage report is an input to the Reviser prompt, but the final coverage values are not fitted parameters or definitions; they are independent measurements of lines executed after the executor agent acts. The ablation studies (Temac-noSTG and Temac-noCR) show that removing the state transition graph or the coverage report changes coverage measurably, which is consistent with a genuine feedback loop rather than a circular reduction. The self-citations in the paper (WebRLED as both the exploration-phase component and a baseline, and Judge for state abstraction) are not load-bearing in a circular way: WebRLED is an externally available system with its own evaluation, and Temac is also compared against Crawljax, FragGen, and WebExplor as external baselines. The exclusion of knowledge-base construction time from the one-hour budget is a validity threat regarding wall-clock comparison, but it does not make the coverage result equivalent to the input by construction. No specific circular step can be quoted and reduced to an equation, a fitted parameter, or a self-citation chain that forces the claimed result.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The paper's central claim rests on a small set of hand-chosen parameters (time budget, file count, task count) and on the accuracy of the underlying tools (WebRLED, WebEmbed, GPT-4o, UI-TARS). No new physical or formal entities are postulated.

free parameters (5)
  • exploration_time_budget = 30 minutes
    Heuristically set in Section IV based on prior work showing coverage stagnation after 30 minutes; it determines how much of the one-hour budget remains for LLM agents and directly affects the reported coverage.
  • low_coverage_file_count = 50
    Section III-B.3 selects the 50 files with the lowest coverage for the knowledge base; this threshold is arbitrary and affects the Reviser agent's task generation.
  • task_count = 10
    The Reviser prompt in Figure 4 requires exactly 10 tasks; the number is arbitrary and affects how much of the uncovered functionality is targeted.
  • action_interval = 2000 ms
    Section V-A sets a fixed 2000 ms interval between actions for all approaches, following WebRLED; it affects how many actions fit in the time budget.
  • repetitions = 3
    Section V-A repeats each experiment three times and averages the results; with only three runs, variance is not reported.
assumptions (3)
  • domain assumption Line coverage is a meaningful proxy for testing effectiveness.
    The evaluation uses line coverage as the primary metric (Section V-A), assuming it reflects exploration quality.
  • domain assumption The state transition graph built by WebEmbed after 30 minutes of WebRLED exploration is a sufficiently accurate map of the application states.
    The Navigator selects key states and shortest paths from this graph (Section III-C); if the graph omits or merges states incorrectly, planned paths may be invalid.
  • domain assumption GPT-4o and UI-TARS perform reliably enough to summarize, infer, and execute tasks as implied by the coverage results.
    The four agents are implemented with GPT-4o and UI-TARS (Section IV); the approach assumes their perception and grounding are accurate enough for the reported gains.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Temac: Multi-Agent Collaboration for Automated Web GUI Testing." pith.science (2026). https://pith.science/paper/VXLIG4OI

@misc{pith2026250600520,
  author       = {Pith},
  title        = {Pith review of: Temac: Multi-Agent Collaboration for Automated Web GUI Testing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VXLIG4OI}},
  note         = {Machine review of arXiv:2506.00520}
}
read the original abstract

Quality assurance of web applications is critical, as web applications play an essential role in people's daily lives. To reduce labor costs, automated web GUI testing (AWGT) is widely adopted, exploring web applications via GUI actions such as clicks and text inputs. However, these approaches face limitations in generating continuous and meaningful action sequences capable of covering complex functionalities. Recent work incorporates large language models (LLMs) for GUI testing. However, these approaches face various challenges, including low efficiency of LLMs, high complexity of rich web application contexts, and a low success rate of LLMs in executing GUI tasks. To address these challenges, in this paper, we propose Temac, an approach that enhances AWGT using LLM-based multi-agent collaboration to increase code coverage. Temac is motivated by our insight that LLMs can enhance AWGT in executing complex functionalities, while the information discovered during AWGT can, in turn, be provided as the domain knowledge to improve the LLM-based task execution. Specifically, given a web application, Temac initially runs an existing approach to broadly explore application states. When the testing coverage stagnates, Temac then employs LLM-based agents to summarize the collected information to form a knowledge base and to infer not-covered functionalities. Guided by this knowledge base, Temac finally uses specialized LLM-based agents to target and execute the not-covered functionalities, reaching deeper states beyond those explored by the existing approach. Our evaluation results show that Temac exceeds state-of-the-art approaches from 12.5% to 60.3% on average code coverage on six complex open-source web applications, while revealing 445 unique failures in the top 20 real-world web applications. These results strongly demonstrate the effectiveness and the general applicability of Temac.

Figures

Figures reproduced from arXiv: 2506.00520 by the authors.

Figure 1
Figure 1. An example action sequence generated by WebRLED on Gadael. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of Temac. tasks and to assist LLM-based agents in executing these tasks. Our knowledge base comprises four parts: (1) natural￾language descriptions of each state, (2) state transitions, (3) code coverage of files in the AUT, and (4) application-specific knowledge. Templates and instantiations illustrating each part of the knowledge base are shown in Table I. 1) Natural-Language Descriptions of Each State (D… view at source ↗
Figure 3
Figure 3. Our prompt template for the Summarizer agent. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Our prompt template for the Reviser agent. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Our prompt template for the Navigator agent. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Line coverage trends over time of Temac and baseline approaches. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 44 canonical work pages

  1. [29]

    Open Source Repository of Seeker,

    “Open Source Repository of Seeker,” 2025. [Online]. Available: https://drive.google.com/drive/folders/12vk2qz8EQa3P8kZ 7IdPN Y3hh9oBV-E?usp=sharing

    work page 2025

  2. [1]

    [Online]

    “Monkey,” 2022. [Online]. Available: https://developer.android.com

    work page 2022

  3. [2]

    Improving random GUI testing with image-based widget detection,

    T. D. White, G. Fraser, and G. J. Brown, “Improving random GUI testing with image-based widget detection,” in Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis , 2019, pp. 307–317

    work page 2019

  4. [3]

    Crawling Ajax-based web applications through dynamic analysis of user interface state changes,

    A. Mesbah, A. Van Deursen, and S. Lenselink, “Crawling Ajax-based web applications through dynamic analysis of user interface state changes,” ACM Transactions on the Web (TWEB) , vol. 6, no. 1, pp. 1–30, 2012

    work page 2012

  5. [4]

    Invariant-based automatic testing of modern web applications,

    A. Mesbah, A. Van Deursen, and D. Roest, “Invariant-based automatic testing of modern web applications,” IEEE Transactions on Software Engineering, vol. 38, no. 1, pp. 35–53, 2011

    work page 2011

  6. [5]

    Neural embeddings for web testing,

    A. Stocco, A. Willi, L. L. L. Starace, M. Biagiola, and P. Tonella, “Neural embeddings for web testing,” arXiv preprint arXiv:2306.07400, 2023

    arXiv 2023

  7. [6]

    Fragment-based test generation for web apps,

    R. K. Yandrapally and A. Mesbah, “Fragment-based test generation for web apps,” IEEE Transactions on Software Engineering , vol. 49, no. 3, pp. 1086–1101, 2022

    work page 2022

  8. [7]

    Judge: Effective state abstraction for guiding automated web GUI testing,

    C. Liu, J. Wang, W. Yang, Y . Zhang, and T. Xie, “Judge: Effective state abstraction for guiding automated web GUI testing,” ACM Transactions on Software Engineering and Methodology , 2025, Just Accepted

    work page 2025

Show all 59 references
  1. [8]

    Automatic web testing using curiosity-driven reinforcement learning,

    Y . Zheng, Y . Liu, X. Xie, Y . Liu, L. Ma, J. Hao, and Y . Liu, “Automatic web testing using curiosity-driven reinforcement learning,” in Proceed- ings of the 43rd International Conference on Software Engineering , 2021, pp. 423–435

    2021

  2. [9]

    QExplore: An exploration strategy for dynamic web applications using guided search,

    S. Sherin, A. Muqeet, M. U. Khan, and M. Z. Iqbal, “QExplore: An exploration strategy for dynamic web applications using guided search,” Journal of Systems and Software , vol. 195, no. 1, pp. 111 512–111 512, 2023

    2023

  3. [10]

    A reinforcement learning approach to generating test cases for web applications,

    X. Chang, Z. Liang, Y . Zhang, L. Cui, Z. Long, G. Wu, Y . Gao, W. Chen, J. Wei, and T. Huang, “A reinforcement learning approach to generating test cases for web applications,” in Proceedings of the 2023 International Conference on Automation of Software Test , 2023, pp. 13–23

    2023

  4. [11]

    UniRL- Test: universal platform-independent testing with reinforcement learning via image understanding,

    Z. Zhang, Y . Liu, S. Yu, X. Li, Y . Yun, C. Fang, and Z. Chen, “UniRL- Test: universal platform-independent testing with reinforcement learning via image understanding,” in Proceedings of the 31st International Symposium on Software Testing and Analysis , 2022, pp. 805–808

    2022

  5. [12]

    Effective, platform- independent GUI testing via image embedding and reinforcement learn- ing,

    S. Yu, C. Fang, X. Li, Y . Ling, Z. Chen, and Z. Su, “Effective, platform- independent GUI testing via image embedding and reinforcement learn- ing,” ACM Transactions on Software Engineering and Methodology , vol. 33, no. 7, pp. 1–27, 2024

    2024

  6. [13]

    Deep reinforcement learning for automated web GUI testing,

    Z. Gu, C. Liu, G. Wu, Y . Zhang, C. Yang, Z. Liang, W. Chen, and J. Wei, “Deep reinforcement learning for automated web GUI testing,” arXiv preprint arXiv:2504.19237 , 2025

    2025 arXiv

  7. [14]

    Q-learning,

    C. J. Watkins and P. Dayan, “Q-learning,” Machine Learning , vol. 8, no. 1, pp. 279–292, 1992

    1992

  8. [15]

    Leveraging large vision- language model for better automatic web GUI testing,

    S. Wang, S. Wang, Y . Fan, X. Li, and Y . Liu, “Leveraging large vision- language model for better automatic web GUI testing,” in 2024 IEEE International Conference on Software Maintenance and Evolution, 2024, pp. 125–137

    2024

  9. [16]

    Feature-driven end-to-end test generation,

    P. Alian, N. Nashid, M. Shahbandeh, T. Shabani, and A. Mesbah, “Feature-driven end-to-end test generation,” in Proceedings of the 47th International Conference on Software Engineering , 2025, pp. 678–678

    2025

  10. [17]

    Nav- iQAte: Functionality-guided web application navigation,

    M. Shahbandeh, P. Alian, N. Nashid, and A. Mesbah, “Nav- iQAte: Functionality-guided web application navigation,” arXiv preprint arXiv:2409.10741, 2024

    2024 arXiv

  11. [18]

    Near-duplicate detection in web app model inference,

    R. Yandrapally, A. Stocco, and A. Mesbah, “Near-duplicate detection in web app model inference,” in Proceedings of the 42nd international conference on software engineering , 2020, pp. 186–197

    2020

  12. [19]

    Fill in the blank: Context-aware automated text input generation for mobile GUI testing,

    Z. Liu, C. Chen, J. Wang, X. Che, Y . Huang, J. Hu, and Q. Wang, “Fill in the blank: Context-aware automated text input generation for mobile GUI testing,” in Proceedings of the 45th International Conference on Software Engineering, 2023, pp. 1355–1367

    2023

  13. [20]

    Testing the limits: Unusual text inputs generation for mobile app crash detection with large language model,

    Z. Liu, C. Chen, J. Wang, M. Chen, B. Wu, Z. Tian, Y . Huang, J. Hu, and Q. Wang, “Testing the limits: Unusual text inputs generation for mobile app crash detection with large language model,” in Proceedings of the 46th International Conference on Software Engineering , 2024, pp. 1–12

    2024

  14. [21]

    Make LLM a testing expert: Bringing human-like interaction to mobile GUI testing via functionality-aware decisions,

    Z. Liu, C. Chen, J. Wang, M. Chen, B. Wu, X. Che, D. Wang, and Q. Wang, “Make LLM a testing expert: Bringing human-like interaction to mobile GUI testing via functionality-aware decisions,” in Proceedings of the 46th International Conference on Software Engineering, 2024, pp. 1–13

    2024

  15. [22]

    Autonomous large language model agents enabling intent-driven mobile GUI testing,

    J. Yoon, R. Feldt, and S. Yoo, “Autonomous large language model agents enabling intent-driven mobile GUI testing,” arXiv preprint arXiv:2311.08649, 2023

    2023 arXiv

  16. [23]

    MobileGPT: Augmenting LLM with human-like app memory for mo- bile task automation,

    S. Lee, J. Choi, J. Lee, M. H. Wasi, H. Choi, S. Ko, S. Oh, and I. Shin, “MobileGPT: Augmenting LLM with human-like app memory for mo- bile task automation,” in Proceedings of the 30th Annual International Conference on Mobile Computing and Networking , 2024, pp. 1119– 1133

    2024

  17. [24]

    Mind2Web: Towards a generalist agent for the web,

    X. Deng, Y . Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y . Su, “Mind2Web: Towards a generalist agent for the web,” Advances in Neural Information Processing Systems , vol. 36, no. 1, pp. 28 091– 28 114, 2023

    2023

  18. [25]

    WebArena: A realistic web environment for building autonomous agents,

    S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y . Bisk, D. Fried et al. , “WebArena: A realistic web environment for building autonomous agents,” arXiv preprint arXiv:2307.13854 , 2023

    2023 arXiv

  19. [26]

    VisualWebArena: Evaluating multimodal agents on realistic visual web tasks,

    J. Y . Koh, R. Lo, L. Jang, V . Duvvur, M. C. Lim, P.-Y . Huang, G. Neubig, S. Zhou, R. Salakhutdinov, and D. Fried, “VisualWebArena: Evaluating multimodal agents on realistic visual web tasks,” arXiv preprint arXiv:2401.13649, 2024

    2024 arXiv

  20. [27]

    GPT-4V(ision) is a gen- eralist web agent, if grounded,

    B. Zheng, B. Gou, J. Kil, H. Sun, and Y . Su, “GPT-4V(ision) is a gen- eralist web agent, if grounded,” in Proceedings of the 41th International Conference on Machine Learning , 2024

    2024

  21. [28]

    Navigating the digital world as humans do: Universal visual grounding for GUI agents,

    B. Gou, R. Wang, B. Zheng, Y . Xie, C. Chang, Y . Shu, H. Sun, and Y . Su, “Navigating the digital world as humans do: Universal visual grounding for GUI agents,” arXiv preprint arXiv:2410.05243 , 2024

    2024 arXiv

  22. [30]

    [Online]

    “Gadael,” 2020. [Online]. Available: https://github.com/gadael/gadael

    2020

  23. [31]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog , vol. 1, no. 8, p. 9, 2019

    2019

  24. [32]

    Language models are few-shot learners,

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B...

    2020

  25. [33]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017, pp. 5998–6008

    2017

  26. [34]

    Web application testing: Using tree kernels to detect near-duplicate states in automated model inference,

    A. Corazza, S. Di Martino, A. Peron, and L. L. L. Starace, “Web application testing: Using tree kernels to detect near-duplicate states in automated model inference,” in Proceedings of the 15th International Symposium on Empirical Software Engineering and Measurement, 2021, pp. 1–6

    2021

  27. [35]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in Neural Information Processing Systems , vol. 35, no. 1, pp. 24 824–24 837, 2022

    2022

  28. [36]

    Google Style Guide,

    “Google Style Guide,” 2025. [Online]. Available: https://google.github. io/styleguide

    2025

  29. [37]

    Aguvis: Unified pure vision agents for autonomous GUI interaction,

    Y . Xu, Z. Wang, J. Wang, D. Lu, T. Xie, A. Saha, D. Sahoo, T. Yu, and C. Xiong, “Aguvis: Unified pure vision agents for autonomous GUI interaction,” arXiv preprint arXiv:2412.04454 , 2024

    2024 arXiv

  30. [38]

    OS-ATLAS: A foundation action model for generalist GUI agents,

    Z. Wu, Z. Wu, F. Xu, Y . Wang, Q. Sun, C. Jia, K. Cheng, Z. Ding, L. Chen, P. P. Liang et al., “OS-ATLAS: A foundation action model for generalist GUI agents,” arXiv preprint arXiv:2410.23218 , 2024

    2024 arXiv

  31. [39]

    UI-TARS: Pioneering automated GUI interaction with native agents,

    Y . Qin, Y . Ye, J. Fang, H. Wang, S. Liang, S. Tian, J. Zhang, J. Li, Y . Li, S. Huang et al., “UI-TARS: Pioneering automated GUI interaction with native agents,” arXiv preprint arXiv:2501.12326 , 2025

    2025 arXiv

  32. [40]

    Cogagent: A visual language model for GUI agents,

    W. Hong, W. Wang, Q. Lv, J. Xu, W. Yu, J. Ji, Y . Wang, Z. Wang, Y . Dong, M. Ding et al. , “Cogagent: A visual language model for GUI agents,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 14 281–14 290

    2024

  33. [41]

    CogVLM: Visual expert for pretrained language models,

    W. Wang, Q. Lv, W. Yu, W. Hong, J. Qi, Y . Wang, J. Ji, Z. Yang, L. Zhao, S. XiXuan et al. , “CogVLM: Visual expert for pretrained language models,” Advances in Neural Information Processing Systems , vol. 37, pp. 121 475–121 499, 2024

    2024

  34. [42]

    WebGLM: Towards an efficient and reliable web-enhanced question answering system,

    X. Liu, H. Lai, H. Yu, Y . Xu, A. Zeng, Z. Du, P. Zhang, Y . Dong, and J. Tang, “WebGLM: Towards an efficient and reliable web-enhanced question answering system,” in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023, pp. 4549– 4560

    2023

  35. [43]

    AutoWebGLM: A large language model- based web navigating agent,

    H. Lai, X. Liu, I. L. Iong, S. Yao, Y . Chen, P. Shen, H. Yu, H. Zhang, X. Zhang, Y . Dong et al. , “AutoWebGLM: A large language model- based web navigating agent,” in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024, pp. 5295– 5306

    2024

  36. [44]

    Set-of-mark prompting unleashes extraordinary visual grounding in GPT-4V,

    J. Yang, H. Zhang, F. Li, X. Zou, C. Li, and J. Gao, “Set-of-mark prompting unleashes extraordinary visual grounding in GPT-4V,” arXiv preprint arXiv:2310.11441, 2023

    2023 arXiv

  37. [45]

    Feedback-directed exploration of web ap- plications to derive test models,

    A. M. Fard and A. Mesbah, “Feedback-directed exploration of web ap- plications to derive test models,” inProceedings of the 24th International Symposium on Software Reliability Engineering , 2013, pp. 278–287

    2013

  38. [46]

    Web application testing: A systematic literature review,

    S. Do ˘gan, A. Betin-Can, and V . Garousi, “Web application testing: A systematic literature review,” Journal of Systems and Software , vol. 91, pp. 174–201, 2014

    2014

  39. [47]

    Realworld,

    “Realworld,” 2022. [Online]. Available: https://github.com/gothinkster/ realworld

    2022

  40. [48]

    4ga Boards,

    “4ga Boards,” 2025. [Online]. Available: https://github.com/RARgames/ 4gaBoards

    2025

  41. [49]

    Timeoff Management Application,

    “Timeoff Management Application,” 2023. [Online]. Available: https: //github.com/timeoff-management/timeoff-management-application

    2023

  42. [50]

    Parabank,

    “Parabank,” 2024. [Online]. Available: https://github.com/parasoft/ parabank

    2024

  43. [51]

    Agilenfant,

    “Agilenfant,” 2016. [Online]. Available: https://sourceforge.net/projects/ agilefant

    2016

  44. [52]

    Efficient memory management for large language model serving with PagedAttention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with PagedAttention,” in Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , 2023

    2023

  45. [53]

    Alexa Top Websites,

    “Alexa Top Websites,” 2025. [Online]. Available: https://www. expireddomains.net/alexa-top-websites

    2025

  46. [54]

    Fetch Standard - CORS Protocol and Credentials,

    “Fetch Standard - CORS Protocol and Credentials,” 2025. [Online]. Available: https://fetch.spec.whatwg.org/#cors-protocol-and-credentials

    2025

  47. [55]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” Ad- vances in Neural Information Processing Systems , vol. 36, pp. 34 892– 34 916, 2023

    2023

  48. [56]

    Vision-driven automated mobile GUI testing via multimodal large language model,

    Z. Liu, C. Li, C. Chen, J. Wang, B. Wu, Y . Wang, J. Hu, and Q. Wang, “Vision-driven automated mobile GUI testing via multimodal large language model,” arXiv preprint arXiv:2407.03037 , 2024

    2024 arXiv

  49. [57]

    Guardian: A runtime framework for LLM-based UI exploration,

    D. Ran, H. Wang, Z. Song, M. Wu, Y . Cao, Y . Zhang, W. Yang, and T. Xie, “Guardian: A runtime framework for LLM-based UI exploration,” in Proceedings of the 33rd International Symposium on Software Testing and Analysis , 2024, pp. 958–970

    2024

  50. [58]

    WorkArena: How capable are web agents at solving common knowledge work tasks?

    A. Drouin, M. Gasse, M. Caccia, I. H. Laradji, M. Del Verme, T. Marty, L. Boisvert, M. Thakkar, Q. Cappart, D. Vazquez et al. , “WorkArena: How capable are web agents at solving common knowledge work tasks?” arXiv preprint arXiv:2403.07718 , 2024

    2024 arXiv

  51. [59]

    WorkArena++: Towards compositional planning and reasoning-based common knowledge work tasks,

    L. Boisvert, M. Thakkar, M. Gasse, M. Caccia, T. de Chezelles, Q. Cap- part, N. Chapados, A. Lacoste, and A. Drouin, “WorkArena++: Towards compositional planning and reasoning-based common knowledge work tasks,” Advances in Neural Information Processing Systems , vol. 37, pp. ...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.