Pith. sign in

REVIEW 2 major objections 5 minor 9 cited by

Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Mind2Web 2 claims to be the first benchmark for long-horizon, time-varying agentic search, and its Agent-as-a-Judge pipeline evaluates complex citation-backed answers with about 99 percent verifier correctness.

desk verdict Genuinely new benchmark and judge framework with solid task quality; headline comparisons and judge-audit narrowness need revision before the numbers are taken at face value. read the letter →

arxiv 2506.21506 v2 pith:LLQV56A4 submitted 2025-06-26 cs.AI cs.CL

Boyu Gou , Zanming Huang , Yuting Ning , Yu Gu , Michael Lin , Weijian Qi , Andrei Kopanev , Botao Yu
show 18 more authors
Bernal Jiménez Gutiérrez Yiheng Shu Chan Hee Song Jiaman Wu Shijie Chen Hanane Nour Moussa Tianshu Zhang Jian Xie Yifei Li Tianci Xue Zeyi Liao Kai Zhang Boyuan Zheng Zhaowei Cai Viktor Rozgic Morteza Ziyadi Huan Sun Yu Su
This is my paper · ORCID
classification cs.AIcs.CL
keywords agenticsearchdeepresearchsystemsbenchmarkconstructionAgent-as-a-Judgetree-structuredrubriccitationattributionLLM-as-a-Judgelong-horizonwebtasks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Mind2Web 2 is a benchmark of 130 realistic, long-horizon web-search tasks whose answers are not fixed strings but change with the live web, and it argues that such tasks can be evaluated automatically and reliably. The paper introduces Agent-as-a-Judge: each task gets a tree-structured rubric whose leaf nodes are binary checks, implemented as a task-specific judge agent that extracts claims from an answer and verifies them against cited webpages. The authors report that this pipeline reaches about 99 percent agreement with a human evaluator on sampled leaf-node judgments. If the benchmark and judge hold up, the field gains a way to compare deep-research systems and humans on open-ended, citation-backed information work rather than on single-answer trivia.

What carries the argument

The load-bearing object is the rubric tree: a goal is decomposed into leaf-node binary verifications, and scores are aggregated bottom-up. Critical nodes act as gates (failure at any critical child zeroes the parent), non-critical children are averaged for partial credit, and sequential nodes short-circuit later checks after an earlier failure. This yields two metrics—Partial Completion (mean root score) and Success Rate (fraction of tasks scoring 1). Around this sits the judge agent, an agentic Python workflow per task with an Extractor that pulls structured claims out of the answer and a Verifier that checks each claim against cached webpage text and screenshots, using LLM calls for simple checking and URL-based verification.

What would settle it

Have two independent evaluators from outside the benchmark team re-score leaf nodes on a stratified sample of all 120 private-test tasks, or swap the Verifier's underlying model and compare leaf-level agreement; if agreement falls materially below the reported 99 percent, or if agent scores restricted to Subset-30 differ from full-set scores, the central reliability and comparability claims are weakened.

Watch

Extended reading notes

Core claim

The paper claims that the central obstacle—evaluating complex, time-varying, citation-backed answers—can be removed by exploiting the generation-verification asymmetry: even if two agents return very different texts, the requirements of each task are known in advance, so a rubric can specify what counts as correct. Each rubric is a tree of dozens to hundreds of binary leaf checks that verify both correctness (does the answer satisfy every stated criterion?) and attribution (is each claim actually backed by its cited URL?). The judge agent runs this tree with LLM-based extractors and verifiers. On a human evaluation of 15 tasks, the verifier's leaf judgments matched human judgment at 99 percent after accounting for human annotation errors. The paper also reports that the best-performing Deep Research system reaches 50–70 percent of human Partial Completion while taking less than half the average human time.

Load-bearing premise

The entire comparison rests on the judge agents' 99 percent correctness generalizing from 15 sampled tasks with one human evaluator from the same research group to all 120 private-test tasks, and on the 30-task human-tested subset representing the full test set.

Editorial extensions

If this is right

  • If the benchmark holds, agentic search systems can be compared on tasks requiring dozens to hundreds of web actions and multi-source synthesis, not just single-answer lookups.
  • Because answers are time-varying, future benchmarks no longer need to freeze answers or restrict queries to static factoids to stay automatically evaluable.
  • The two metrics give developers a partial-credit signal for incremental progress, not only an all-or-nothing success flag.
  • The reported 50-70 percent of human performance at half the time suggests deep-research systems may already be cost-effective for some tedious information-gathering work, while the error analysis shows hallucinated or invalid citations remain the main barrier.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to apply the rubric-tree judge design to other long-form, time-varying agent outputs, such as travel planning or procurement, whenever the criteria decompose into independent single-page verifications.
  • The reported 99 percent correctness may not transfer to the hardest private-test items, since collapsed webpage content already caused four of the eight remaining verifier discrepancies; measuring judge accuracy on those cases specifically would sharpen the claim.
  • The observed 50-70 percent human performance at half the time suggests that, as inference budgets grow, deep-research systems could overtake humans on tedious but constrained information-gathering tasks—while criterion violations by human participants show the human reference itself is noisy.
  • If the private-test rubrics and the judge-script generation pipeline stay hidden, leaderboard integrity depends on that secrecy, since a system optimized against this exact judge could inflate scores without improving real search ability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces Mind2Web 2, a benchmark of 130 long-horizon, time-varying web-search tasks with citation-backed answers, together with Agent-as-a-Judge, a rubric-tree evaluation framework implemented as task-specific judge agents. The authors report rubric complexity statistics, a human performance study on a randomly selected 30-task subset, evaluation of ten frontier agentic search systems and humans, an error analysis, and a human audit of judge agents on 15 tasks reporting 99.03% leaf-node correctness. The headline result is that OpenAI Deep Research achieves 50-70% of human performance while spending about half the time.

Significance. If the benchmark and judge pipeline hold up, this is a valuable community resource: it addresses a real evaluation gap for long-horizon agentic search, includes realistic live-web tasks, provides a public/private split, releases code and evaluation scripts, and offers a useful error-analysis taxonomy. The human-relative comparison is attention-worthy and timely. However, the evidence for the two strongest claims—the reliability of the judge agents and the 50-70% human-relative number—is currently thinner than the abstract suggests, so the paper needs strengthening before these claims can be taken at face value.

major comments (2)
  1. [§4.2, Table 3] The abstract's '50-70% of human performance' is not directly supported by Table 3. Human Partial Completion (0.79) and Success Rate (0.54) are computed on Subset-30 only, whereas agent scores are reported for all 120 private-test tasks; the paper does not report agent scores restricted to Subset-30. Unless the authors show agent performance on the same 30 tasks or argue convincingly that Subset-30 is representative, the quoted ratio is an apples-to-oranges comparison. Please add a Subset-30 column/row for all systems or revise the claim.
  2. [§4.4, Appendix D.5] The 99.03% verifier correctness is derived from 15 sampled tasks and 720 leaf-node checks scored by a single human evaluator, with trivial total-failure answers explicitly excluded. Because errors are nested within task-specific judge-agent scripts, pooled node-level accuracy can hide whole-task script failures: one flawed script can corrupt many leaf judgments for a task, and those errors are diluted when combined with hundreds of correct nodes from other tasks. The audit also relies on a human evaluator who participated in error analysis and on adjudication by a judge-agent developer, so it does not provide fully independent confirmation. Please report per-task error rates and a task-level confidence interval, justify the exclusion of total-failure answers, and clarify the independence of the evaluator and adjudication protocol.
minor comments (5)
  1. [Table 1] The WebWalkerQA row cites [41], which is AssistantBench; the correct reference appears to be [37]. Please fix this citation.
  2. [Appendix D.2] In the Extractor and Verifier prompts, 'http//' and 'https//' should be 'http://' and 'https://'.
  3. [Appendix E.1] Appendix E.1 states that Hugging Face Open Deep Research was run with OpenAI's o3 model as its base model; this important detail is absent from §4.1 and Table 3, and should be disclosed wherever the system is described as open-source.
  4. [Figure 4] The category 'Explicitly Time-varying Tasks' is defined only in the text (57 tasks); add the definition to the caption and consider showing error bars, since Figure 3 includes them.
  5. [§3.3] The rubric formula's 'otherwise' branch (s(v)=1 when there are no non-critical children) could be misread as applying to leaf nodes; a one-sentence clarification that leaves are assigned by the Verifier would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark's task construction, rubric evaluation, and human-relative performance figures are empirical measurements rather than derivations that reduce to their own inputs.

full rationale

The paper's central claims — Mind2Web 2's 130 tasks, the Agent-as-a-Judge rubric framework, the 99.03% verifier correctness estimate, and the 50–70% human-relative performance figure — do not reduce by construction to the paper's inputs. The tasks are collected and validated through multi-stage human labor, the judge agents are generated and refined against those rubrics, and system/human answers are external artifacts scored by those rubrics; this is standard benchmark construction, not a circular derivation in which a predicted quantity is definitionally equal to a fitted input. The judge-reliability claim rests on a human audit of 720 leaf verifications across 15 sampled tasks; although the audit uses one evaluator from the same research group and discrepancy adjudication by a judge-agent developer, the paper reports disagreements and an error rate as an empirical measurement rather than as a fitted parameter renamed as a prediction. Self-citations to prior OSU work (Mind2Web, Online-Mind2Web, AttributionBench) provide context and comparison baselines but are not load-bearing for the main claims. The 50–70% human-relative statement is an arithmetic ratio of the reported partial-completion means (0.54/0.79), which may be affected by the Subset-30 versus full-test-set comparison, but that is a measurement-validity concern, not circularity. No quoted passage exhibits a specific reduction of a claimed result to its own definition, so under the evidentiary standard no circular step can be flagged.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The benchmark rests on internal human annotation and LLM-based judgment rather than external ground truth. No new physical or formal entities are postulated; 'Agent-as-a-Judge' is a software framework built from existing LLM tools. No numerical parameters are fitted to data.

assumptions (5)
  • domain assumption URL-based attribution to single webpages is sufficient to verify task answers.
    Task design principles exclude tasks requiring simultaneous multi-page verification or dynamically loaded content (Section C.2, Appendix H.1), narrowing the benchmark to what URL-level checks can judge.
  • domain assumption OpenAI o4-mini performs extraction and verification reliably.
    Stated in Sections 3.4 and 4.4; reliability is claimed from human evaluation on 15 sampled tasks, not a formal guarantee.
  • domain assumption Cached webpage contents reflect the live web state at answer-generation time.
    Appendix E.2 pre-fetches and caches pages referenced in answers; if a page changes between answer generation and evaluation, the attribution check could be wrong.
  • domain assumption The 130 collected tasks represent realistic long-horizon agentic search queries.
    Tasks are proposed and validated by the authors' own annotators (Section 3.2, Appendix C); vague or subjective queries are excluded, so external representativeness is not proven.
  • domain assumption Subset-30 is representative of the private 120-task test set.
    Used to compare human performance to agent performance in Table 3; the paper says Subset-30 is a subset of the private test set but does not provide matched agent scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge." pith.science (2026). https://pith.science/paper/LLQV56A4

@misc{pith2026250621506,
  author       = {Pith},
  title        = {Pith review of: Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LLQV56A4}},
  note         = {Machine review of arXiv:2506.21506}
}
read the original abstract

Agentic search such as Deep Research systems-where agents autonomously browse the web, synthesize information, and return comprehensive citation-backed answers-represents a major shift in how users interact with web-scale information. While promising greater efficiency and cognitive offloading, the growing complexity and open-endedness of agentic search have outpaced existing evaluation benchmarks and methodologies, which largely assume short search horizons and static answers. In this paper, we introduce Mind2Web 2, a benchmark of 130 realistic, high-quality, and long-horizon tasks that require real-time web browsing and extensive information synthesis, constructed with over 1000 hours of human labor. To address the challenge of evaluating time-varying and complex answers, we propose a novel Agent-as-a-Judge framework. Our method constructs task-specific judge agents based on a tree-structured rubric design to automatically assess both answer correctness and source attribution. We conduct a comprehensive evaluation of ten frontier agentic search systems and human performance, along with a detailed error analysis to draw insights for future development. The best-performing system, OpenAI Deep Research, can already achieve 50-70% of human performance while spending half the time, highlighting its great potential. Altogether, Mind2Web 2 provides a rigorous foundation for developing and benchmarking the next generation of agentic search systems.

Figures

Figures reproduced from arXiv: 2506.21506 by the authors.

Figure 1
Figure 1. Mind2Web 2 features realistic and diverse long-horizon web search tasks and a novel [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Example of tree-structured rubrics. Top-down, task goals are decomposed into a tree [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. Average Partial Completion on explic￾itly time-varying tasks compared to other tasks. Comparison Between Agent Types. Unsurprisingly, ChatGPT Search and Perplexity Pro Search emerge as the weakest systems, primarily limited by their restricted search horizon and relatively shallow information synthesis abilities inherent to LLMs. In contrast, most Deep Research systems achieve superior performance. These systems are… view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Errors across agents and humans. The bars indicate the percentage of tasks exhibiting each [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SciHazard: A Benchmark for Measuring Scientific Safety Risks with Decomposed Harm Scoring

    cs.AI 2026-07 conditional novelty 7.0 of 10

    A new scientific-safety benchmark and a decomposed, retrieval-grounded metric that aligns with expert harm judgments substantially better than existing LLM-as-judge baselines.

  2. DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Histories

    cs.CV 2026-02 conditional novelty 6.5 of 10

    The paper reframes image retrieval as agentic exploration over personal visual histories and shows the best tested multimodal agent scores only 28.7 exact match on its new DISBench benchmark.

  3. From Simple QA to Deep Research: A Verifiable Benchmark Constructed through Iterative Task Evolution

    cs.AI 2026-08 conditional novelty 6.0 of 10

    An automatic pipeline evolves simple QA questions into 500 deep-research tasks with DAG-structured, fact-grounded rubrics that discriminate between models.

  4. Persona2Web: Benchmarking Personalized Web Agents for Contextual Reasoning with User History

    cs.CL 2026-02 conditional novelty 6.0 of 10

    Persona2Web is a new open-web benchmark where agents must infer a user's preferences from synthetic browsing history to solve intentionally ambiguous queries; current best agents score 13% success.

  5. DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generation

    cs.CL 2025-12 conditional novelty 6.0 of 10

    DEER uses 7 evaluation dimensions, 101 rubric items, task-specific expert guidance, and unsupported-claim backtracking to score deep-research reports; current systems score lowest on fulfilling expert requests and ana...

  6. LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Services

    cs.AI 2025-12 conditional novelty 6.0 of 10

    LocalSearchBench—1.3M merchant records and 900 multi-hop local-life QA tasks across 9 Chinese cities—shows the best reasoning agent reaches only 35.6% correctness.

  7. ResearcherBench: Evaluating Deep AI Research Systems on the Frontiers of Scientific Inquiry

    cs.AI 2025-07 conditional novelty 6.0 of 10

    A new benchmark evaluates AI deep research systems on 65 frontier AI questions, finding OpenAI and Gemini lead on rubric coverage while all systems cite accurately but leave much content unsupported.

  8. Sidekick: Designing Communication for Effective Multitasking with Computer Use Agents

    cs.HC 2026-07 conditional novelty 5.0 of 10

    Sidekick's multimodal feedback significantly improves multitasking performance with computer-use agents compared to chat-only or peripheral text feedback.

  9. SCOPE: Stochastic and Counterbiased Option Placement for Evaluating Large Language Models

    cs.CL 2025-07 reject novelty 4.0 of 10

    SCOPE estimates a model's position bias with nonsense prompts, puts correct answers in disliked slots, and spreads similar distractors apart to cap lucky guessing.

Reference graph

Works this paper leans on

95 extracted references · 60 canonical work pages · cited by 9 Pith papers

  1. [1]

    Claude computer use

    Anthropic. Claude computer use. https://www.anthropic.com/news/ 3-5-models-and-computer-use , 2024. Accessed: 2025-05-08

    2024

  2. [2]

    Claude takes research to new places, 2025

    Anthropic. Claude takes research to new places, 2025. URL https://www.anthropic.com/ news/research. Accessed 2025-07-01

    2025

  3. [3]

    The anatomy of a large-scale hypertextual web search engine

    Sergey Brin and Lawrence Page. The anatomy of a large-scale hypertextual web search engine. Computer networks and ISDN systems, 30(1-7):107–117, 1998

    1998

  4. [4]

    Learning to rank using gradient descent

    Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. Learning to rank using gradient descent. In Proceedings of the 22nd international conference on Machine learning, pages 89–96, 2005

    2005

  5. [5]

    Teaching large language models to self-debug

    Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. In The Twelfth International Conference on Learning Representations,

  6. [6]

    Mindsearch: Mimicking human minds elicits deep ai searcher

    Zehui Chen, Kuikun Liu, Qiuchen Wang, Jiangning Liu, Wenwei Zhang, Kai Chen, and Feng Zhao. Mindsearch: Mimicking human minds elicits deep ai searcher. arXiv preprint arXiv:2407.20183, 2024

    arXiv 2024

  7. [7]

    Chatbot arena: An open platform for evaluating llms by human preference

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-first International Conference on Machine Learning, 2024

    2024

  8. [8]

    Mind2web: Towards a generalist agent for the web

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091–28114, 2023

    2023

Show all 95 references
  1. [9]

    Enabling large language models to generate text with citations

    Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models to generate text with citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6465–6488, 2023

    2023

  2. [10]

    Gemini deep research

    Google. Gemini deep research. https://gemini.google/overview/deep-research/, 2025. Accessed: 2025-05-08

    2025

  3. [11]

    Navigating the digital world as humans do: Universal visual grounding for GUI agents

    Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for GUI agents. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openrev...

    2025

  4. [12]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  5. [13]

    Webvoyager: Building an end-to-end web agent with large multimodal models

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume1...

    2024

  6. [14]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021

    2021

  7. [15]

    Open deep research

    Hugging Face. Open deep research. https://huggingface.co/blog/open-deep-research,

  8. [16]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024. 12

    2024 arXiv

  9. [17]

    Visualwebarena: Evaluating multimodal agents on realistic visual web tasks

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649, 2024

    2024 arXiv

  10. [18]

    Attributionbench: How hard is automatic attribution evaluation? In Findings of the Association for Computational Linguistics ACL 2024, pages 14919–14935, 2024

    Yifei Li, Xiang Yue, Zeyi Liao, and Huan Sun. Attributionbench: How hard is automatic attribution evaluation? In Findings of the Association for Computational Linguistics ACL 2024, pages 14919–14935, 2024

    2024

  11. [19]

    Evaluating verifiability in generative search engines

    Nelson F Liu, Tianyi Zhang, and Percy Liang. Evaluating verifiability in generative search engines. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7001–7025, 2023

    2023

  12. [20]

    Learning to rank for information retrieval

    Tie-Yan Liu et al. Learning to rank for information retrieval. Foundations and Trends® in Information Retrieval, 3(3):225–331, 2009

    2009

  13. [21]

    Weblinx: Real-world website navigation with multi-turn dialogue

    Xing Han Lu, Zdenˇek Kasner, and Siva Reddy. Weblinx: Real-world website navigation with multi-turn dialogue. In International Conference on Machine Learning, pages 33007–33056. PMLR, 2024

    2024

  14. [22]

    Self-refine: Iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534–46594, 2023

    2023

  15. [23]

    Gaia: a benchmark for general ai assistants

    Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, 2023

    2023

  16. [24]

    Webgpt: Browser-assisted question-answering with human feedback

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. We...

    2021 arXiv

  17. [25]

    Introducing ChatGPT search

    OpenAI. Introducing ChatGPT search. https://openai.com/index/ introducing-chatgpt-search/, 2024

    2024

  18. [26]

    Deep research system card

    OpenAI. Deep research system card. Technical report, OpenAI, February 2025. URL https: //cdn.openai.com/deep-research-system-card.pdf

    2025

  19. [27]

    Operator system card

    OpenAI. Operator system card. Technical report, OpenAI, January 2025. URL https: //cdn.openai.com/operator_system_card.pdf

    2025

  20. [28]

    Au- tonomous evaluation and refinement of digital agents

    Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Au- tonomous evaluation and refinement of digital agents. In First Conference on Language Modeling, 2024

    2024

  21. [29]

    Webcanvas: Benchmarking web agents in online environments

    Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, et al. Webcanvas: Benchmarking web agents in online environments. arXiv preprint arXiv:2406.12373, 2024

    2024 arXiv

  22. [30]

    Perplexity ai

    Perplexity AI. Perplexity ai. https://www.perplexity.ai/, 2024. Accessed: 2025-05-08

    2024

  23. [31]

    Ui-tars: Pioneering automated gui interaction with native agents

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326, 2025

    2025 arXiv

  24. [32]

    A vector space model for automatic indexing

    Gerard Salton, Anita Wong, and Chung-Shu Yang. A vector space model for automatic indexing. Communications of the ACM, 18(11):613–620, 1975

    1975

  25. [33]

    Re- flexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Re- flexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634–8652, 2023. 13

    2023

  26. [34]

    Bearcubs: A benchmark for computer-using web agents

    Yixiao Song, Katherine Thai, Chau Minh Pham, Yapei Chang, Mazin Nadaf, and Mohit Iyyer. Bearcubs: A benchmark for computer-using web agents. arXiv preprint arXiv:2503.07919, 2025

    2025 arXiv

  27. [35]

    Paperbench: Evaluating ai’s ability to replicate ai research

    Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. Paperbench: Evaluating ai’s ability to replicate ai research. arXiv preprint arXiv:2504.01848, 2025

    2025 arXiv

  28. [36]

    Browsecomp: A simple yet challenging benchmark for browsing agents

    Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516, 2025

    2025 arXiv

  29. [37]

    Webwalker: Benchmarking llms in web traversal

    Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Linhai Zhang, Yulan He, Deyu Zhou, Pengjun Xie, et al. Webwalker: Benchmarking llms in web traversal. arXiv preprint arXiv:2501.07572, 2025

    2025 arXiv

  30. [38]

    Grok 3 beta — the age of reasoning agents.https://x.ai/blog/grok-3, 2025

    xAI. Grok 3 beta — the age of reasoning agents.https://x.ai/blog/grok-3, 2025. Accessed: 2025-05-08

    2025

  31. [39]

    An illusion of progress? assessing the current state of web agents

    Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. An illusion of progress? assessing the current state of web agents. arXiv preprint arXiv:2504.01382, 2025

    2025

  32. [40]

    Webshop: Towards scalable real-world web interaction with grounded language agents

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757, 2022

    2022

  33. [41]

    AssistantBench: Can web agents solve realistic and time-consuming tasks?, 2024

    Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. AssistantBench: Can web agents solve realistic and time-consuming tasks?, 2024. URL https://arxiv.org/abs/2407.15711

    2024 arXiv

  34. [42]

    Automatic evaluation of attribution by large language models

    Xiang Yue, Boshi Wang, Ziru Chen, Kai Zhang, Yu Su, and Huan Sun. Automatic evaluation of attribution by large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4615–4635, 2023

    2023

  35. [43]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on C...

    2024

  36. [44]

    GPT-4V(ision) is a generalist web agent, if grounded

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. GPT-4V(ision) is a generalist web agent, if grounded. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=piecKJ2DlB

    2024

  37. [45]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    2023

  38. [46]

    Webarena: A realistic web environment for building autonomous agents

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. Webarena: A realistic web environment for building autonomous agents. ICLR, 2024

    2024

  39. [47]

    good,” “effective,

    Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, et al. Agent- as-a-judge: Evaluate agents with agents. arXiv preprint arXiv:2410.10934, 2024. 14 Table of Contents i...

    arXiv

  40. [50]

    Extract only information explicitly mentioned in the provided answer exactly as it appears

    Do not add, omit, or invent any information. Extract only information explicitly mentioned in the provided answer exactly as it appears

  41. [51]

    If any required information is missing from the answer, explicitly returnnull as the JSON value

  42. [52]

    Understand it clearly, as it provides essential background for the extraction

    You will also receive the original task description as context. Understand it clearly, as it provides essential background for the extraction. You may apply common-sense reasoning to assist your extraction, but your final result must be accurately extracted from the answer tex...

  43. [53]

    Carefully follow those instructions when available

    Occasionally, additional instructions might be provided to aid your extraction. Carefully follow those instructions when available. SPECIAL RULES FOR URL EXTRACTION: These rules apply only when URL fields are required in the extraction

  44. [54]

    Do not create or infer any URLs

    Extract only URLs explicitly present in the answer text. Do not create or infer any URLs

  45. [55]

    Ignore obviously invalid or malformed URLs

    Extract only valid URLs. Ignore obviously invalid or malformed URLs

  46. [56]

    Correct" or

    If a URL is missing a protocol (http// orhttps// ), prependhttp// . Instruction for Extraction: {extraction_prompt} Original Task Description: {task_description} Complete Answer to the Task: {answer} Additional Instructions (if any): {additional_instruction} Prompt for Verifie...

  47. [57]

    Use logic, basic factual knowledge, or simple reasoning to determine its accuracy

    Carefully examine the provided claim or statement. Use logic, basic factual knowledge, or simple reasoning to determine its accuracy

  48. [58]

    Clearly understand the provided task description and complete answer, as they offer important context and may influence your decision

  49. [59]

    Your reasoning must be explicit, concise, and directly support your binary judgment

  50. [60]

    fact" is extracted directly from the answer, and the webpage provided is the URL source referenced in the answer. This verification step helps us determine whether the claim or

    Carefully follow any additional instructions provided. If none are provided, you may ignore this. Original Task Description: {task_description} 20 Complete Answer to the Task: {answer} Additional Instructions (if any): {additional_instruction} Claim or Statement to Verify: {cl...

  51. [61]

    Carefully examine the relevant sections of both the webpage text and the screenshot

    The provided webpage content may be lengthy. Carefully examine the relevant sections of both the webpage text and the screenshot. Determine clearly whether the claim or "fact" exactly matches or is explicitly supported by the webpage content. If the information appears to be n...

  52. [62]

    Understand them clearly, as they provide essential background for evaluating the claim

    You will also receive the original task description and the complete answer as context. Understand them clearly, as they provide essential background for evaluating the claim. You may apply common- sense reasoning (e.g., fuzzy matching for names differing only in letter casing...

  53. [63]

    If the provided webpage (the URL source mentioned in the answer) is entirely irrelevant, invalid, or inaccessible, you must conclude that the claim or "fact" is not supported

  54. [64]

    Carefully follow those instructions when available

    Occasionally, additional instructions might be provided to aid your judgment. Carefully follow those instructions when available. Original Task Description: {task_description} Complete Answer to the Task: {answer} Claim or Fact to Verify: {claim} Additional Instructions (if an...

    2025

  55. [65]

    All necessary requirements are provided in the task description — please strictly adhere to it to complete the task

    Do not ask for additional information or follow-up questions. All necessary requirements are provided in the task description — please strictly adhere to it to complete the task

  56. [67]

    In your answer, please include relevant links for each claim and piece of information

    Every claim and piece of information you provide must be supported by a source. In your answer, please include relevant links for each claim and piece of information. 23 Empirically, we find OpenAI Operator and Gemini Deep Research occasionally neglect the require- ments to pr...

  57. [70]

    In your answer, please include relevant links for each claim and piece of information

    Every claim and piece of information you provide must be supported by a source. In your answer, please include relevant links for each claim and piece of information. If the task requires a list of items (e.g., names, emails, affiliations, products), each item in the list must...

  58. [71]

    All necessary requirements are provided in the task description—please strictly adhere to it to complete the task

    Do not ask for additional information or follow-up questions. All necessary requirements are provided in the task description—please strictly adhere to it to complete the task

  59. [72]

    Do not provide critical information without actual searching

    To solve the task, you should search the web for online sources and use them to support all your claims and the information in your final answer. Do not provide critical information without actual searching

  60. [73]

    find_llava_commit

    Every claim and piece of information you provide must be supported by a source. In your answer, please include relevant links for each claim and piece of information. Even if the task explicitly requests some specific links, you must still provide URL sources for all the other...

    2004

  61. [74]

    Verify the task reflects real-world scenarios

    Realism a. Verify the task reflects real-world scenarios. Imagine yourself or someone you know performing this task in real life. b. Verify the task is not artificially combining many simple steps to increase complexity or tediousness. Note: • Certain subjectivity regarding th...

  62. [75]

    nice”, “good

    Clarity and Objectivity a. Ensure the task description is typo-free and grammatically correct. b. Verify that the description is clear and understandable. c. Ensure the task explicitly states the necessary background knowledge needed to complete the task. d. Make sure the task...

  63. [76]

    Confirm that the task takes more than 5 minutes to complete

    Tediousness and Feasibility a. Confirm that the task takes more than 5 minutes to complete. Try performing the task, ensure it can’t be quickly solved with only one or two simple searches. b. Confirm that the task required information can be found on publicly accessible websit...

  64. [77]

    list all

    Verifiability a. Ensure task verifiability. i. Draft an outline of the expected answer as a sanity check, as well as to assist future task validation. The outline should include: • All critical information explicitly required by the task, OR • Information that, while not expli...

  65. [78]

    • Avoid tasks with rapidly changing answers (e.g., stock prices, exchange rates)

    Additional Considerations • Tasks involving video understanding or non-English websites are currently not supported. • Avoid tasks with rapidly changing answers (e.g., stock prices, exchange rates). • Avoid tasks requiring extensive reasoning, complex calculations, or external...

  66. [79]

    Ensure all or most critical information can be practically located and verified from the URL sources

    Full Completion and End-to-End Testing 46 • Fully perform the task yourself from start to finish. Ensure all or most critical information can be practically located and verified from the URL sources

  67. [80]

    Feasibility of URL-based Verification • Verify that each URL provided as an information source uniquely and directly supports the expected statement. • Avoid scenarios that are beyond our evaluation framework capabilities, such as: – Tasks requiring simultaneous verification f...

  68. [81]

    • If ground-truth information is necessary for certain criteria, note it down and carefully validate its correctness

    Explicit Ground Truth and Evaluation Notes • Clearly document the related sources and information that can be helpful for understanding the task. • If ground-truth information is necessary for certain criteria, note it down and carefully validate its correctness. • Provide exp...

  69. [82]

    Understand the task clearly i

    Before you begin: a. Understand the task clearly i. Carefully read the task description and ensure you clearly understand what is asked (e.g., no language barriers or a lack of domain-specific knowledge). It’s okay if you do not yet know how to solve the task; planning and fig...

  70. [83]

    Start recording i

    Solving the task: a. Start recording i. Begin screen recording before you start planning or researching for the task. ii. Ensure all task-related activity is recorded, avoiding external screens. b. Research and Answer i. Search and browse to gather accurate and reliable inform...

  71. [84]

    You do not need to summarize every web page you visit

    Our basic expectation for answers is: All critical points required by the task should be included, along with URLs to verify them (i.e., the URL where you find each information). You do not need to summarize every web page you visit

  72. [85]

    You could also write some intermediate thoughts or the reasoning process on Google Docs for yourself when completing the task, though only the critical information asked by the task is required for the final answer. iii. For every piece of information or statement in your answ...

  73. [86]

    Use either inline hyperlinks or numbered citations

  74. [87]

    Ensure all URLs start with http orhttps

  75. [88]

    Export browsing data: i

    After completing the task: a. Export browsing data: i. Export statistics from the time tracker extension to CSV immediately ii. After exporting, you must stop gathering new information; only reformat your answer if you wish (e.g., if it is still cluttered). (Imagine you no lon...

  76. [89]

    Paste your final answer into the designated Answer column in the provided spreadsheet

    Upload your results: a. Paste your final answer into the designated Answer column in the provided spreadsheet. b. Convert your recording to MP4 using HandBrake (preset: Fast 1080p30) and upload to our OneDrive folder. c. Rename the CSV as taskID-yourName.csv, upload it to OneD...

  77. [90]

    Make sure you fully understand the task criteria, and discuss with us when necessary

    Read the task and the agent’s answer carefully. Make sure you fully understand the task criteria, and discuss with us when necessary

  78. [91]

    Evaluate the response using the error categories detailed below

  79. [92]

    Your comment should briefly explain what kind of error it is and why

    Whenever you identify an error, leave a comment in the Google Doc directly on the problematic part. Your comment should briefly explain what kind of error it is and why

  80. [93]

    Correctness Check This section is concerned only with the agent’s full response based on the text

    Collect all error types you identified, and check the corresponding labels in the annotation sheet. Correctness Check This section is concerned only with the agent’s full response based on the text

  81. [94]

    I tried but failed to find

    Incompleteness: Our definition of Incompleteness here is limited to immediately noticeable, surface- level omissions: The agent does not provide all content explicitly requested in the task. It does not cover more subtle or long-range reasoning failures. We further define two ...

  82. [95]

    find all

    Criteria Violation: This label is used when the agent’s answer breaks explicit constraints mentioned in the task. These constraints could be things like price ranges, required formats, word limits, or instructions such as “find all.” It also applies when the task comes with a ...

  83. [96]

    page not found

    Invalid Attribution: URL is expired, incorrectly formatted, or obviously fabricated, such as • The URL leads to a “page not found” error. • The provided arXiv link for a research paper (https://arxiv.org/abs/1234. 56789) is clearly fake

  84. [97]

    For example, in the following answer, a missing source and a valid one are presented

    Missing Attribution: No source URL is provided for the claim. For example, in the following answer, a missing source and a valid one are presented. • Totokaelo – Address: 913 Western Avenue, Seattle, WA 98104 → Missing Attribution: no URLs (including the following product page...

  85. [98]

    TODO”. Your task is to replace each “TODO

    Unsupported Answer: If a reachable attribution is provided, we need to examine whether it can support the claim. If not, there are 2 subtypes of errors: a. Retrieval Error: The provided sources are irrelevant to the task, such as: • The task requests the list of K-pop songs in...

  86. [2024]

    URL https://openreview.net/forum?id=KuPixIqPiq

  87. [2025]

    Accessed: 2025-05-08

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.