Pith. sign in

REVIEW 3 major objections 7 minor 63 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

Web agents can barely navigate real websites, and most benchmarks miss it

2026-07-08 15:51 UTC pith:3ZRBUH3J

load-bearing objection Large-scale web agent benchmark with a novel network-request-based evaluator; reliability claims need IAA evidence. the 3 major comments →

arxiv 2607.06118 v1 pith:3ZRBUH3J submitted 2026-07-07 cs.CV cs.MM

WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation

Wei Dong , Tianyu Fu , Zhe Yu , Hanning Wang , Anyang Su , Zhizhou Fang , Yuyang Chen , Shuo Wang
show 4 more authors
Minghui Wu Ping Jiang Zhen Lei Chenxu Zhao
This is my paper
classification cs.CV cs.MM
keywords evaluationnavigationagentacrosscapabilitiescomprehensiveinteractiontask
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces WebRetriever, a benchmark of 1,550 tasks across 800 real websites spanning consumer, professional, and enterprise domains, and argues that existing web-agent evaluations are systematically over-optimistic because they test too few sites, rely on brittle or coarse automated judges, and stop at navigation success without checking whether agents can actually extract correct information. The authors propose NavEval, an automated evaluator that combines the final screenshot with filtered network-request logs and URL trajectories—signals that reveal whether the agent actually performed the right searches, filters, and form submissions rather than merely landing on a plausible-looking page. Across three evaluation protocols—bare navigation, navigation with operational documentation, and end-to-end task completion with information extraction—state-of-the-art agents achieve only 21% success on basic navigation and 12% on full end-to-end tasks. NavEval matches human judgments at 91% agreement on WebRetriever and 97% on an external benchmark, substantially outperforming prior automated evaluators. The central claim is that reaching the right page is not the same as completing the task, and that request-level interaction logs are the missing signal for automated evaluation to approach human reliability.

Core claim

The paper's central discovery is that web-agent performance collapses when evaluation moves beyond navigation success to end-to-end information extraction, and that this gap is invisible to existing benchmarks. Even the strongest agents, which reach target pages roughly 37–45% of the time with operational guidance, succeed on full end-to-end tasks only 16–21% of the time—meaning that more than half of successful navigations still fail to extract the correct answer. The paper attributes NavEval's evaluation accuracy to its use of filtered network-request sequences as a complementary signal to screenshots: requests reveal whether specific filters, searches, and selections were actually applied

What carries the argument

The key mechanism is NavEval's combination of rule-based filtering of network requests with LLM reasoning. Given a task description, starting URL, action sequence, network requests, and final screenshot, NavEval first filters requests by subdomain matching and removes irrelevant fields, then feeds the cleaned request log alongside the action trajectory and final screenshot to an LLM that judges task success. The request log captures fine-grained interaction semantics—search queries, filter parameters, sort orders, form submissions—that screenshots alone cannot convey.

Load-bearing premise

The paper treats its human annotations as ground truth but uses a sequential multi-stage review process rather than parallel independent annotation, so it never reports how often human annotators disagree with each other. If annotators themselves disagree substantially on borderline cases, the 91% agreement rate may be near the ceiling of what is achievable, and the gap between NavEval and prior methods may be smaller than it appears.

What would settle it

Run a parallel double-annotation study on a subset of WebRetriever tasks and compute inter-annotator agreement; if IAA falls below 91%, NavEval's agreement rate is at the noise ceiling and its superiority over prior methods is not meaningfully distinguishable from annotation ambiguity.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If NavEval's 91% human-agreement rate holds, automated web-agent evaluation can largely replace costly human annotation, enabling much larger-scale benchmarking and faster iterative development.
  • The finding that end-to-end success rates are roughly half of navigation success rates suggests that current agents' information-extraction and reasoning capabilities, not their navigation, are the binding constraint for real deployment.
  • The three-protocol design separates navigation, knowledge-assisted navigation, and end-to-end completion, providing a diagnostic framework that can pinpoint whether an agent's failure stems from path-finding, knowledge integration, or information extraction.
  • Operational documentation improves success by about 8 percentage points, suggesting that coupling agents with task-specific knowledge bases is a viable near-term deployment strategy, though the modest gain also indicates agents struggle to fully leverage such documentation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The absence of inter-annotator agreement metrics means the 91% agreement ceiling is unknown; if human annotators disagree on borderline cases at rates above 9%, NavEval may already be at the noise ceiling, and further gains may require better task design rather than better evaluators.
  • The sharp drop from navigation success to end-to-end success suggests that benchmarks evaluating only navigation are measuring a capability that is necessary but far from sufficient, implying that published agent performance numbers from navigation-only benchmarks may overstate real-world utility by a factor of two or more.
  • NavEval's reliance on network-request logs may be brittle against websites that load content via client-side rendering without distinct network calls, or against single-page applications where filtering happens in-browser without new requests, potentially limiting generalization to certain site architectures.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces WebRetriever, a large-scale online web agent benchmark comprising 800 websites and 1,550 tasks across diverse domains and intent types. It also proposes NavEval, an LLM-as-Judge evaluation method that integrates filtered network requests, action trajectories, and final screenshots to achieve reported human agreement rates of ~91% on WebRetriever and ~97% on Online-Mind2Web. Three evaluation protocols are introduced: Protocol I (basic navigation), Protocol II (navigation with operational documentation), and Protocol III (end-to-end task completion with information extraction). The benchmark and evaluation framework are released with code and supplementary materials. The central claims are that WebRetriever provides broader and more realistic coverage than prior benchmarks, that NavEval is a reliable scalable substitute for human evaluation, and that navigation success alone is an insufficient predictor of end-to-end capability.

Significance. The paper makes a solid contribution to the web agent evaluation landscape. The benchmark scale (800 websites, 1,550 tasks) and the tri-protocol evaluation framework are genuinely useful additions. The three-protocol design that disentangles navigation, knowledge-augmented navigation, and end-to-end information extraction is a thoughtful contribution that exposes capability gaps overlooked by prior benchmarks. The ablation in Table 6 (Protocol III with/without extraction) effectively demonstrates that page arrival is not task success. The code and data release, along with the detailed prompt templates in Appendix E, support reproducibility. The NavEval ablation studies (Tables 9–10) on LLM backbone and rule-based filtering are informative. However, the significance of the NavEval contribution is tempered by the absence of inter-annotator agreement metrics for the ground-truth annotations, which leaves the noise ceiling unknown and makes it difficult to calibrate the reported agreement rates.

major comments (3)
  1. §B.4: No inter-annotator agreement (IAA) metric is reported for WebRetriever's ground-truth annotations. The five-stage sequential pipeline reports correction rates of 30%, 26%, and 27%, with 54% of tasks receiving at least one correction. These correction rates signal substantial initial disagreement, yet the paper does not report what fraction of tasks would be judged differently by two independent annotators. Without an IAA ceiling, it is impossible to determine whether NavEval's 91.2% average AR (Table 3) represents near-human performance or is simply at the noise ceiling. This is load-bearing for the central claim that NavEval is 'a reliable, discriminative, and fine-grained framework.' The paper should either (a) report IAA on a held-out subset of tasks with parallel independent annotation, or (b) explicitly discuss the noise ceiling and reframe the 91.2% AR relative to it.
  2. Table 4 and §4.2: The 97% AR on Online-Mind2Web is presented as evidence of NavEval's generalizability, but it is counterintuitively higher than the 91.2% AR on the authors' own benchmark. Examination of Table 4 shows this inflation is benchmark-wide: WebJudge (Claude-4.5-Sonnet) scores 81.0% on WebRetriever (Table 3) but WebJudge-7B scores 87.2% on Online-Mind2Web; Autonomous Eval (O4-mini) scores 70.3% on WebRetriever vs. 83.8% on Online-Mind2Web. This pattern suggests Online-Mind2Web tasks have more clear-cut success/failure boundaries, not that NavEval is especially robust. The paper does not acknowledge or investigate this confound. The 97% claim should be qualified, and the benchmark-wide inflation pattern should be discussed as an alternative explanation.
  3. §4.3 and Supplementary: Protocol III contains only 100 tasks (§3.3), and Table 2 reports results for six agents, yielding approximately 16–17 tasks per agent. The claim that 'navigation success alone is an insufficient predictor of real-world application effectiveness' (Abstract, §4.2) is central to the paper's framing, and the Protocol III results in Table 6 are the primary evidence. However, with 100 tasks total and per-agent sample sizes of roughly 16–17, the confidence intervals on these success rates are very wide. The paper should report confidence intervals or bootstrap intervals for Protocol III results, or at minimum acknowledge the statistical limitations of drawing strong conclusions from 100 tasks.
minor comments (7)
  1. Protocol III contains only 100 tasks while Protocols I and II each contain 1,000. The paper should discuss whether 100 tasks provide sufficient coverage for the diverse task categories listed in Table 7 (document extraction, form interaction, multi-source comparison, etc.), as some categories may reduce to only 5–6 tasks.
  2. Table 3: NavEval is only evaluated with Claude-4.5-Sonnet as the judge backbone, while all baselines are evaluated with three backbones (GPT-4o, O4-mini, Claude-4.5-Sonnet). Table 9 in the supplementary provides NavEval results with other backbones, but this should be integrated into the main comparison table to support the robustness claim.
  3. §3.1: The task difficulty bins (easy: n<6, medium: 6≤n≤15, hard: n>15) are defined but no per-difficulty results are reported in the experiments. Reporting SR and AR by difficulty bin would add diagnostic value.
  4. Fig. 8: The axis labels and tick marks are partially illegible in the current rendering. The x-axis label and unit should be clarified for the camera-ready version.
  5. §B.4: The annotation team composition (14 members with varied roles) is described, but the number of annotators per task and the total annotation effort (person-hours) are not reported. This information is standard for benchmark papers.
  6. Table 4: WebVoyager entries for SeeAct and Browser-Use are marked with dashes ('–'), indicating missing data. The paper should note why these entries could not be computed.
  7. The paper uses 'Claude-4.5' and 'Claude-4.5-Sonnet' interchangeably in different locations (e.g., Table 2 vs. Table 3). Consistent naming would avoid ambiguity.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive review. All three major comments identify legitimate gaps that we will address in revision. Specifically: (1) we will conduct a parallel-annotation IAA study on a held-out subset and report the noise ceiling alongside NavEval's agreement rate; (2) we will explicitly discuss the benchmark-wide inflation pattern on Online-Mind2Web and qualify the 97% AR claim accordingly; and (3) we will report bootstrap confidence intervals for Protocol III results and acknowledge the statistical limitations of drawing strong conclusions from 100 tasks. We agree with the substance of all three comments and will revise the manuscript accordingly.

read point-by-point responses

  1. Referee: §B.4: No inter-annotator agreement (IAA) metric is reported for WebRetriever's ground-truth annotations. The five-stage sequential pipeline reports correction rates of 30%, 26%, and 27%, with 54% of tasks receiving at least one correction. Without an IAA ceiling, it is impossible to determine whether NavEval's 91.2% average AR represents near-human performance or is simply at the noise ceiling. The paper should either (a) report IAA on a held-out subset of tasks with parallel independent annotation, or (b) explicitly discuss the noise ceiling and reframe the 91.2% AR relative to it.

    Authors: The referee is correct that the absence of an IAA ceiling is a genuine gap, and the correction rates we report (30%, 26%, 27%) do signal substantial initial disagreement that we did not adequately contextualize. We will address this in revision by conducting option (a): a parallel independent annotation study on a held-out subset of approximately 150 tasks. Two annotators who did not participate in the original sequential pipeline will independently label these tasks following the same annotation guidelines, and we will report Cohen's kappa and percent agreement as the IAA ceiling. We will then reframe NavEval's 91.2% AR relative to this ceiling in both the main text and the abstract. If the IAA ceiling turns out to be close to 91%, we will explicitly state that NavEval operates near the noise ceiling and adjust the claim of being 'reliable, discriminative, and fine-grained' accordingly. If the ceiling is higher, the current framing is supported. Either way, the noise ceiling will be transparently reported. revision: yes

  2. Referee: Table 4 and §4.2: The 97% AR on Online-Mind2Web is presented as evidence of NavEval's generalizability, but it is counterintuitively higher than the 91.2% AR on the authors' own benchmark. The benchmark-wide inflation pattern (WebJudge Claude-4.5-Sonnet: 81.0% on WebRetriever vs. 87.2% on Online-Mind2Web; Autonomous Eval O4-mini: 70.3% vs. 83.8%) suggests Online-Mind2Web tasks have more clear-cut success/failure boundaries. The paper does not acknowledge or investigate this confound. The 97% claim should be qualified, and the benchmark-wide inflation pattern should be discussed as an alternative explanation.

    Authors: The referee's observation is accurate and well-supported by the data. The pattern is indeed benchmark-wide: every automated evaluation method we tested scores higher on Online-Mind2Web than on WebRetriever, which strongly suggests that Online-Mind2Web tasks have more clear-cut success/failure boundaries rather than NavEval being especially robust on that benchmark. We failed to acknowledge this confound in the manuscript. In revision, we will add a paragraph in §4.2 explicitly noting the benchmark-wide inflation pattern, discussing the likely explanation (Online-Mind2Web tasks may have less ambiguous success criteria), and qualifying the 97% AR claim accordingly. Specifically, we will reframe the Online-Mind2Web results as demonstrating that NavEval generalizes to external benchmarks while noting that the higher absolute AR is partly attributable to the benchmark's task characteristics rather than solely to NavEval's robustness. We will also remove or soften the implication that the 97% AR on Online-Mind2Web is inherently more impressive than the 91.2% on WebRetriever. revision: yes

  3. Referee: §4.3 and Supplementary: Protocol III contains only 100 tasks (§3.3), and Table 2 reports results for six agents, yielding approximately 16–17 tasks per agent. The claim that 'navigation success alone is an insufficient predictor of real-world application effectiveness' is central to the paper's framing, and the Protocol III results in Table 6 are the primary evidence. However, with 100 tasks total and per-agent sample sizes of roughly 16–17, the confidence intervals on these success rates are very wide. The paper should report confidence intervals or bootstrap intervals for Protocol III results, or at minimum acknowledge the statistical limitations of drawing strong conclusions from 100 tasks.

    Authors: The referee is correct that 100 tasks with roughly 16–17 tasks per agent yields wide confidence intervals, and we did not acknowledge this limitation. We will address this in two ways. First, we will report bootstrap 95% confidence intervals (10,000 resamples) for all Protocol III success rates in Table 2 and Table 6. Second, we will add an explicit discussion of the statistical limitations in §4.3, noting that while the directional finding (success rates drop substantially from navigation-only to end-to-end completion) is consistent across all six agents and therefore unlikely to be an artifact of sampling noise, the precise magnitudes of the drops should be interpreted with caution given the small sample sizes. We will also soften the language in the abstract from a definitive claim to one that acknowledges the sample size constraint, e.g., noting that the results 'suggest' rather than 'demonstrate' that navigation success alone is an insufficient predictor. We agree that the Protocol III sample size is a limitation of the current work and will note expanding it as future work. revision: yes

Circularity Check

0 steps flagged

No significant circularity found; NavEval is defined independently of the human annotations it is evaluated against.

full rationale

NavEval (Eq. 1-2) is a fixed pipeline: rule-based filtering of network requests plus an off-the-shelf LLM-as-Judge with a provided prompt template. No parameter is fitted to human agreement data, and the method is not defined in terms of the agreement rate it claims to achieve. The ground-truth human annotations are generated independently (§B.4, §C.3), with explicit separation between NavEval outputs and human reviewers. While NavEval is used as a trajectory filter in the Protocol II documentation generation pipeline (§B.2), this does not create circularity in the evaluation because the success labels for Protocol II come from separate human annotation (Table 2, 'Human' column), not from NavEval. External validation on Online-Mind2Web (Table 4) provides independent grounding. The self-bias concern (Claude-4.5-Sonnet as both judge and evaluated agent) is a correctness issue, not circularity, and is partially addressed by the backbone ablation (Table 9). The one minor concern—shared infrastructure between the benchmark and the evaluation method—is standard for benchmark papers and does not constitute a construction-level reduction. Score 1 reflects this minor structural overlap with no impact on the independence of the central claim.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

No new physical entities or mathematical objects are invented. The benchmark and evaluator are engineering artifacts, not theoretical constructs.

free parameters (3)
  • NavEval LLM backbone = Claude-4.5-Sonnet
    Selected as the best-performing judge model from ablation (Table 9); not fitted to data but chosen empirically.
  • Rule-based filtering thresholds = Not explicitly stated
    Subdomain matching and field trimming rules are described qualitatively but exact thresholds/parameters are not specified in the main text.
  • Task difficulty bins = easy: n<6, medium: 6≤n≤15, hard: n>15
    Step-count thresholds for difficulty categorization are chosen by the authors without justification from data.
axioms (4)
  • domain assumption Human annotations on WebRetriever tasks are reliable ground truth
    §B.4: Sequential multi-stage review is used but no inter-annotator agreement metric is reported, assuming the review process produces reliable labels.
  • domain assumption Network request sequences contain sufficient signal to verify task completion
    §3.2: NavEval's design assumes that filtered request sequences reveal query execution and filtering operations that screenshots cannot capture.
  • domain assumption Live website content remains stable enough for reproducible evaluation
    §3.1: The paper assumes tasks remain executable over time, with a maintenance plan for replacement when they become obsolete.
  • domain assumption LLM-as-judge can reliably classify task success given multi-source context
    §3.2: The entire NavEval framework depends on the LLM's ability to correctly reason over task descriptions, request sequences, action trajectories, and screenshots.

pith-pipeline@v1.1.0-glm · 22298 in / 3599 out tokens · 194182 ms · 2026-07-08T15:51:36.649353+00:00 · methodology

0 comments
read the original abstract

As web agents increasingly demonstrate capabilities in automated task execution, the development of robust evaluation frameworks for assessing their navigation and task completion performance has emerged as a critical research priority. However, existing benchmarks exhibit fundamental limitations. First, they suffer from insufficient scale and limited domain diversity, constraining comprehensive evaluation of cross-domain generalization. Second, prevailing LLM-as-Judge evaluation methodologies inadequately capture fine-grained interaction semantics, particularly regarding precise query formulation and filtering operations. Third, current benchmarks predominantly emphasize navigation success metrics while neglecting critical requirements for real-world deployment scenarios. To address these limitations, we introduce WebRetriever, a large-scale benchmark encompassing 800 websites and 1,550 tasks across diverse domains, including consumer, professional, and enterprise sectors, with comprehensive coverage of user intent patterns. We propose NavEval (Navigation Evaluation), a novel LLM-as-Judge framework that leverages rich interaction context beyond visual screenshots, achieving state-of-the-art alignment with human judgment across multiple evaluation datasets. Furthermore, we establish three complementary evaluation protocols that collectively provide holistic assessment of web agent capabilities: navigation proficiency, knowledge-assisted interaction, and end-to-end task completion with information extraction. Extensive experimental analysis reveals substantial performance disparities across evaluation protocols, demonstrating that navigation success alone is an insufficient predictor of real-world application effectiveness. WebRetriever delivers fine-grained diagnostic insights into agent capabilities and establishes a rigorous foundation for advancing web agent research and development.

Figures

Figures reproduced from arXiv: 2607.06118 by Anyang Su, Chenxu Zhao, Hanning Wang, Minghui Wu, Ping Jiang, Shuo Wang, Tianyu Fu, Wei Dong, Yuyang Chen, Zhen Lei, Zhe Yu, Zhizhou Fang.

Figure 1
Figure 1. Figure 1: Motivation for the WebRetriever benchmark. WebRetriever addresses key lim￾itations of prior work from three aspects: dataset scale and diversity, automated eval￾uation reliability, and deployment-oriented evaluation protocols [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Visualization of WebRetriever’s website coverage: (a) distribution across indus￾try domains, (b) distribution of website types. Filter& Find Products 12.4% Local Services & Real Estate 10.9% Facts & Encyclopedia 10.2% Classes& Learning 7.9% Reading& Browsing 7.4% Tutorials & Life Solutions 6.6% Watch& Listen 6.1% Cart Action 4.7% Social& Community 4.5% Price Comparison 4.1% View Reviews & Recs 4.0% Real-ti… view at source ↗
Figure 3
Figure 3. Figure 3: Visualization of user intent distributions in WebRetriever tasks: (a) distribution of general user intents, (b) distribution of professional user intents. Finance, Education & Research, Culture, Entertainment & Travel, Life Ser￾vices, Healthcare, Industrial Manufacturing, and Public Services, as illustrated in [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Workflow of NavEval. Compared to existing methods, NavEval applies rule￾based filtering to extract fine-grained intermediate signals, which are then jointly rea￾soned with the task description, action trajectory, and final screenshot by an LLM to determine task success, enabling robust evaluation with higher human agreement rates [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Workflow of the semi-automated pipeline for constructing operational docu￾mentation in Protocol II. The process integrates automated exploration, evaluation, manual refinement, and LLM-based generation to produce high-quality operational documentation. operations after reaching target pages in realistic scenarios. Consequently, strong benchmark performance fails to translate into practical capability, reve… view at source ↗
Figure 6
Figure 6. Figure 6: Example of the semi-automated operational documentation generation pipeline for the task "If I want to make predictions using a model, how can I ensure that my predictions are safe?" For clarity, the figure shows only the trajectory screenshots and omits other intermediate information. refined manually to obtain high-quality trajectories. Finally, using the prompts designed in Sec. E.2, Gemini-2.5-Pro gene… view at source ↗
Figure 7
Figure 7. Figure 7: Details of websites in the WebRetriever benchmark. (a) Geographic distribution of websites. (b) Language distribution of websites [PITH_FULL_IMAGE:figures/full_fig_p022_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Distribution of task reference steps in WebRetriever, with x-axis as steps per trajectory and y-axis as task proportion (0–1 scale). as China and the US, as well as other geographic regions, including Europe (e.g., France, Spain), Asia, Africa, and Latin America. Complementing this geographic coverage, [PITH_FULL_IMAGE:figures/full_fig_p023_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: shows a task involving unexpected pop-up windows that appear during execution. Agents must interact with these elements to access hidden informa￾tion. D.2 Collapsible Content [PITH_FULL_IMAGE:figures/full_fig_p026_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Task: "List the locations in Washington where vaccines are available". (a) Click the ’J’ category on the A–Z index page to further browse related entries. (b) Click ’Joint Pain’ on the ’J’ category page to view the possible causes [PITH_FULL_IMAGE:figures/full_fig_p027_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Task: "Find information about the possible causes of joint pain". D.4 Hallucinations The Web agent occasionally exhibits hallucinations, performing unnecessary or incorrect interactions. In [PITH_FULL_IMAGE:figures/full_fig_p027_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Task: "View a comparative trend chart showing the year-to-date stock per￾formance of NVIDIA (NVDA) and Microsoft (MSFT)." [PITH_FULL_IMAGE:figures/full_fig_p028_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Task: "View the scoring performance of Stephen Curry in his most recent game." E Prompt Template E.1 NavEval Evaluation System Prompt: You are a benchmark validator. Your job is to decide whether a browser￾automation agent successfully completed a task by examining: 1. Task description – defines the goal and the key actions required. Every requirement, keyword, constraint, or objective in the task is trea… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

63 extracted references · 63 canonical work pages · 23 internal anchors

  1. [1]

    Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systems

    Abuelsaad, T., Akkil, D., Dey, P., Jagmohan, A., Vempaty, A., Kokku, R.: Agent- e: From autonomous web navigation to foundational design principles in agentic systems. arXiv preprint arXiv:2407.13032 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  2. [2]

    Official Product Announcement (2025), https://www.anthropic.com/news/claude-sonnet-4-5

    Anthropic: Introducing claude sonnet 4.5. Official Product Announcement (2025), https://www.anthropic.com/news/claude-sonnet-4-5

    work page 2025

  3. [3]

    Qwen2.5-VL Technical Report

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al.: Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  4. [4]

    Advances in Neural Information Processing Systems36, 78142–78167 (2023)

    Bai, Y., Ying, J., Cao, Y., Lv, X., He, Y., Wang, X., Yu, J., Zeng, K., Xiao, Y., Lyu, H., et al.: Benchmarking foundation models with language-model-as-an-examiner. Advances in Neural Information Processing Systems36, 78142–78167 (2023)

    work page 2023

  5. [5]

    Advances in Neural Information Processing Systems37, 5996–6051 (2024)

    Boisvert, L., Thakkar, M., Gasse, M., Caccia, M., De Chezelles, T.L., Cappart, Q., Chapados, N., Lacoste, A., Drouin, A.: Workarena++: Towards compositional planning and reasoning-based common knowledge work tasks. Advances in Neural Information Processing Systems37, 5996–6051 (2024)

    work page 2024

  6. [6]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Chen, Q., Pitawela, D., Zhao, C., Zhou, G., Chen, H.T., Wu, Q.: Webvln: Vision- and-language navigation on websites. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 1165–1173 (2024)

    work page 2024

  7. [7]

    ChatShop: Interactive Information Seeking with Language Agents

    Chen, S., Wiseman, S., Dhingra, B.: Chatshop: Interactive information seeking with language agents. arXiv preprint arXiv:2404.09911 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  8. [8]

    Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities

    Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al.: Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  9. [9]

    Advances in Neural Information Processing Systems36, 28091–28114 (2023)

    Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., Su, Y.: Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems36, 28091–28114 (2023)

    work page 2023

  10. [10]

    In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Deng, Y., Zhang, X., Zhang, W., Yuan, Y., Ng, S.K., Chua, T.S.: On the multi- turn instruction following for conversational web agents. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 8795–8812 (2024)

    work page 2024

  11. [11]

    Drouin, A., Gasse, M., Caccia, M., Laradji, I.H., Del Verme, M., Marty, T., Boisvert, L., Thakkar, M., Cappart, Q., Vazquez, D., et al.: Workarena: How ca- pable are web agents at solving common knowledge work tasks? arXiv preprint arXiv:2403.07718 (2024) 16 W. Dong, T. Fu et al

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  12. [12]

    In: European Conference on Computer Vision

    Fan, Y., Ma, X., Wu, R., Du, Y., Li, J., Gao, Z., Li, Q.: Videoagent: A memory- augmented multimodal agent for video understanding. In: European Conference on Computer Vision. pp. 75–92. Springer (2024)

    work page 2024

  13. [13]

    The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation

    Fernandes, P., Deutsch, D., Finkelstein, M., Riley, P., Martins, A.F., Neubig, G., Garg, A., Clark, J.H., Freitag, M., Firat, O.: The devil is in the errors: Leverag- ing large language models for fine-grained machine translation evaluation. arXiv preprint arXiv:2308.07286 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  14. [14]

    arXiv preprint arXiv:2509.17336 (2025)

    Fu, T., Su, A., Zhao, C., Wang, H., Wu, M., Yu, Z., Hu, F., Shi, M., Dong, W., Wang, J., et al.: Mano technical report. arXiv preprint arXiv:2509.17336 (2025)

    work page arXiv 2025

  15. [15]

    In: NeurIPS 2023 Foundation Models for Decision Making Workshop (2023)

    Furuta, H., Matsuo, Y., Faust, A., Gur, I.: Language model agents suffer from compositional generalization in web automation. In: NeurIPS 2023 Foundation Models for Decision Making Workshop (2023)

    work page 2023

  16. [16]

    REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites

    Garg, D., VanWeelden, S., Caples, D., Draguns, A., Ravi, N., Putta, P., Garg, N., Abraham, T., Lara, M., Lopez, F., et al.: Real: Benchmarking autonomous agents on deterministic simulations of real websites. arXiv preprint arXiv:2504.11543 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  17. [17]

    Seed1.5-VL Technical Report

    Guo, D., Wu, F., Zhu, F., Leng, F., Shi, G., Chen, H., Fan, H., Wang, J., Jiang, J., Wang, J., et al.: Seed1. 5-vl technical report. arXiv preprint arXiv:2505.07062 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  18. [18]

    WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models

    He, H., Yao, W., Ma, K., Yu, W., Dai, Y., Zhang, H., Lan, Z., Yu, D.: Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  19. [19]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Hong, W., Wang, W., Lv, Q., Xu, J., Yu, W., Ji, J., Wang, Y., Wang, Z., Dong, Y., Ding, M., et al.: Cogagent: A visual language model for gui agents. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14281–14290 (2024)

    work page 2024

  20. [20]

    GPT-4o System Card

    Hurst, A., Lerer, A., Goucher, A.P., Perelman, A., Ramesh, A., Clark, A., Os- trow, A., Welihinda, A., Hayes, A., Radford, A., et al.: Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  21. [21]

    In: European Conference on Computer Vision

    Kapoor, R., Butala, Y.P., Russak, M., Koh, J.Y., Kamble, K., AlShikh, W., Salakhutdinov, R.: Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. In: European Conference on Computer Vision. pp. 161–178. Springer (2024)

    work page 2024

  22. [22]

    In: ICLR 2025 Workshop on Foundation Models in the Wild (2025)

    Kara, S., Faisal, F., Nath, S.: Waber: Evaluating reliability and efficiency of web agents with existing benchmarks. In: ICLR 2025 Workshop on Foundation Models in the Wild (2025)

    work page 2025

  23. [23]

    VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks

    Koh, J.Y., Lo, R., Jang, L., Duvvur, V., Lim, M.C., Huang, P.Y., Neubig, G., Zhou, S., Salakhutdinov, R., Fried, D.: Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  24. [24]

    In: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

    Lai,H.,Liu,X.,Iong,I.L.,Yao,S.,Chen,Y.,Shen,P.,Yu,H.,Zhang,H.,Zhang,X., Dong, Y., et al.: Autowebglm: A large language model-based web navigating agent. In: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. pp. 5295–5306 (2024)

    work page 2024

  25. [25]

    ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents

    Levy, I., Wiesel, B., Marreed, S., Oved, A., Yaeli, A., Shlomov, S.: St- webagentbench: A benchmark for evaluating safety and trustworthiness in web agents. arXiv preprint arXiv:2410.06703 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  26. [26]

    Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., Hashimoto, T.B.: Alpacaeval: An automatic evaluator of instruction-following models (2023) WebRetriever 17

    work page 2023

  27. [27]

    Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration

    Liu, E.Z., Guu, K., Pasupat, P., Shi, T., Liang, P.: Reinforcement learning on web interfaces using workflow-guided exploration. arXiv preprint arXiv:1802.08802 (2018)

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  28. [28]

    Liu, J., Song, Y., Lin, B.Y., Lam, W., Neubig, G., Li, Y., Yue, X.: Visualwebbench: How far have multimodal llms evolved in web page understanding and grounding? arXiv preprint arXiv:2404.05955 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  29. [29]

    AgentBench: Evaluating LLMs as Agents

    Liu,X.,Yu,H.,Zhang,H.,Xu,Y.,Lei,X.,Lai,H.,Gu,Y.,Ding,H.,Men,K.,Yang, K., et al.: Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  30. [30]

    VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents

    Liu, X., Zhang, T., Gu, Y., Iong, I.L., Xu, Y., Song, X., Zhang, S., Lai, H., Liu, X., Zhao, H., et al.: Visualagentbench: Towards large multimodal models as visual foundation agents. arXiv preprint arXiv:2408.06327 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  31. [31]

    Lù, X.H., Kasner, Z., Reddy, S.: Weblinx: Real-world website navigation with multi-turn dialogue (2024)

    work page 2024

  32. [32]

    Official Product An- nouncement (2025),https://openai.com/index/introducing-o3-and-o4-mini/

    OpenAI: o4-mini: Fast and cost-efficient reasoning model. Official Product An- nouncement (2025),https://openai.com/index/introducing-o3-and-o4-mini/

    work page 2025

  33. [33]

    Autonomous Evaluation and Refinement of Digital Agents

    Pan, J., Zhang, Y., Tomlin, N., Zhou, Y., Levine, S., Suhr, A.: Autonomous eval- uation and refinement of digital agents. arXiv preprint arXiv:2404.06474 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  34. [34]

    WebCanvas: Benchmarking Web Agents in Online Environments

    Pan, Y., Kong, D., Zhou, S., Cui, C., Leng, Y., Jiang, B., Liu, H., Shang, Y., Zhou, S., Wu, T., et al.: Webcanvas: Benchmarking web agents in online environments. arXiv preprint arXiv:2406.12373 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  35. [35]

    UI-TARS: Pioneering Automated GUI Interaction with Native Agents

    Qin, Y., Ye, Y., Fang, J., Wang, H., Liang, S., Tian, S., Zhang, J., Li, J., Li, Y., Huang, S., et al.: Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  36. [36]

    In: International Conference on Machine Learning

    Shi, T., Karpathy, A., Fan, L., Hernandez, J., Liang, P.: World of bits: An open- domain platform for web-based agents. In: International Conference on Machine Learning. pp. 3135–3144. PMLR (2017)

    work page 2017

  37. [37]

    BEARCUBS: A benchmark for computer-using web agents

    Song, Y., Thai, K., Pham, C.M., Chang, Y., Nadaf, M., Iyyer, M.: Bearcubs: A benchmark for computer-using web agents. arXiv preprint arXiv:2503.07919 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  38. [38]

    In: Findings of the Association for Computational Linguis- tics: ACL 2025

    Tian, S., Zhang, Z., Chen, L.Y., Liu, Z.: Mmina: Benchmarking multihop multi- modal internet agents. In: Findings of the Association for Computational Linguis- tics: ACL 2025. pp. 13682–13697 (2025)

    work page 2025

  39. [39]

    Advances in Neural Information Processing Systems37, 115963–116021 (2024)

    Wornow, M., Narayan, A., Viggiano, B., Khare, I., Verma, T., Thompson, T., Her- nandez, M., Sundar, S., Trujillo, C., Chawla, K., et al.: Wonderbread: A benchmark for evaluating multimodal foundation models on business process management tasks. Advances in Neural Information Processing Systems37, 115963–116021 (2024)

    work page 2024

  40. [40]

    AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials

    Xu,Y.,Lu,D.,Shen,Z.,Wang,J.,Wang,Z.,Mao,Y.,Xiong,C.,Yu,T.:Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials. arXiv preprint arXiv:2412.09605 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  41. [41]

    In: Second Conference on Language Modeling (2025),https://openreview.net/forum?id=6jZi4HSs6o

    Xue, T., Qi, W., Shi, T., Song, C.H., Gou, B., Song, D., Sun, H., Su, Y.: An illusion of progress? assessing the current state of web agents. In: Second Conference on Language Modeling (2025),https://openreview.net/forum?id=6jZi4HSs6o

    work page 2025

  42. [42]

    In: ArXiv (preprint)

    Yao, S., Chen, H., Yang, J., Narasimhan, K.: Webshop: Towards scalable real-world web interaction with grounded language agents. In: ArXiv (preprint)

    work page

  43. [43]

    Yoran, O., Amouyal, S.J., Malaviya, C., Bogin, B., Press, O., Berant, J.: Assistant- bench: Can web agents solve realistic and time-consuming tasks? arXiv preprint arXiv:2407.15711 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  44. [44]

    GPT-4V(ision) is a Generalist Web Agent, if Grounded

    Zheng, B., Gou, B., Kil, J., Sun, H., Su, Y.: Gpt-4v (ision) is a generalist web agent, if grounded. arXiv preprint arXiv:2401.01614 (2024) 18 W. Dong, T. Fu et al

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  45. [45]

    Advances in neural information processing systems36, 46595–46623 (2023)

    Zheng, L., Chiang, W.L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al.: Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems36, 46595–46623 (2023)

    work page 2023

  46. [46]

    ICLR (2024)

    Zhou, S., Xu, F.F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Bisk, Y., Fried, D., Alon, U., et al.: Webarena: A realistic web environment for building autonomous agents. ICLR (2024)

    work page 2024

  47. [47]

    Support” l Click “Predicting with your Models

    Zunic, G.: Browser use = state of the art web agent. Blog Technical Report (2024), https://browser-use.com/posts/sota-technical-report WebRetriever 19 A Overview This supplementary material is structured into four main sections. First, we detail data construction, including the design of the three evaluation protocols, task creation procedures, and operat...

    work page 2024

  48. [48]

    Every requirement, keyword, constraint, or objective in the task is treated as a criterion

    Task description – defines the goal and the key actions required. Every requirement, keyword, constraint, or objective in the task is treated as a criterion

    work page

  49. [49]

    Network requests – simplified logs revealing user operations such as searches, filters, sorts, form submissions, item selections, report retrieval, etc

    work page

  50. [50]

    Final screenshot – shows the final state of the page and is used to verify that the result matches the task requirements

    work page

  51. [51]

    submit”, “search

    URL trajectory – the full ordered list of URLs visited during navigation, reflecting the steps taken to reach the final state. WebRetriever 29 You must use the combination of these three evidence channels to evaluate whether each key requirement in the task was fulfilled. However, if a single evidence source alone is already sufficient to confirm or rejec...

    work page

  52. [52]

    No quotes, no bullets, no numbering

    Output ONLY one English sentence. No quotes, no bullets, no numbering

    work page

  53. [53]

    Do NOT mechanically list actions; instead, describe actions as steps that move toward completing the goal

    The sentence MUST follow the task goal (title) as the primary constraint. Do NOT mechanically list actions; instead, describe actions as steps that move toward completing the goal

    work page

  54. [54]

    You must reflect ALL meaningful action types that appear in the trajectory, including Click/Type/Select/Toggle/ScrollDown/ScrollUp

    work page

  55. [55]

    Scroll actions MUST NOT be omitted and MUST be written with a goal/purpose inferred from the task goal and the subsequent non-scroll action

    work page

  56. [56]

    ** Click/selection wording constraints:

    Preserve the real execution order of actions, but compress redundant micro-steps when possible. ** Click/selection wording constraints:

    work page

  57. [57]

    When describing what to click/select, prefer describing the UI location/- position/role (e.g., navigation entry, search result position, list position, tab, filter option) rather than copying long content text

    work page

  58. [58]

    If a label is necessary, keep it short and functional

    Avoid quoting full article/news/product titles or long sentences from the page. If a label is necessary, keep it short and functional

    work page

  59. [59]

    ** Output constraints:

    If multiple candidates exist on the page, express which one by ordinal/posi- tion (first/second/top) or by function, so the instruction is goal-directed and reproducible. ** Output constraints:

    work page

  60. [60]

    Do NOT include coordinates, bbox, ids, file paths, or step numbers

    work page

  61. [61]

    first... then... next... during... finally

    Do NOT invent actions that do not exist in the trajectory. 3.Preferastructuredflowsuchas“first... then... next... during... finally...”, but keep it ONE sentence

    work page

  62. [62]

    Keep under 100 English characters if possible; correctness and goal-alignment 32 W. Dong, T. Fu et al. are more important than brevity

    work page

  63. [63]

    User Prompt: Task: <task> Website: <website> Trajectory: <trajectory> Now generate the operational document sentence:

    Preserve important UI labels/options in original casing if they are in English. User Prompt: Task: <task> Website: <website> Trajectory: <trajectory> Now generate the operational document sentence:

    work page