Pith. sign in

REVIEW 3 major objections 6 minor 3 cited by

Characterizing Deep Research: A Benchmark and Formal Definition

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A query is deep research when it needs many information units and non-trivial reasoning; a new 100-task benchmark ranks best current systems at 0.55 F1.

desk verdict A genuinely useful deep-research benchmark with a sensible two-axis characterization, but the 'formal definition' is a heuristic and the ground-truth expansion step needs tightening before the paper can be canonical. read the letter →

arxiv 2508.04183 v1 pith:KPWYTBV3 submitted 2025-08-06 cs.CL

Abhinav Java , Ashmit Khandelwal , Sukruta Midigeshi , Aaron Halfaker , Amit Deshpande , Navin Goyal , Ankur Gupta , Nagarajan Natarajan
show 1 more author
Amit Sharma
This is my paper
classification cs.CL
keywords deepresearchbenchmarkformaldefinitionsearchintensityreasoningclaim-levelevaluationprobleminversioninformationsynthesis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to define what makes a task deep research rather than merely a hard question or a long report. Its proposal: the defining feature is the shape of the work, not the output format—a query is deep research when answering it requires processing many separate information units and at least one of finding, digesting, or combining those units demands non-trivial reasoning. To make the definition testable, the paper recasts the task as a tuple of query, a list of claims (each possibly with supporting subclaims), and a corpus, so systems can be judged on the precision and recall of the claims they surface instead of on subjective report quality. It then contributes LiveDRBench, 100 such queries on science and world events, built by inverting existing long-context reasoning problems, and reports that current deep research products score between 0.02 and 0.72 F1 per category, with the highest-scoring system at 0.55 overall. If the characterization is right, progress in deep research becomes measurable, benchmarkable, and independent of the auxiliary skill of writing prose.

What carries the argument

Definition 1, the two-axis characterization: given a corpus, a query is a deep research query iff it has high search intensity (a large number of information units processed) and at least one of finding, processing, or combining those units requires non-trivial reasoning. Two companion mechanisms do the work. First, the claim-list intermediate representation—the answer is a list of dictionaries, each claim possibly nesting subclaims—makes the task objectively evaluable via claim-level precision and recall, with a strict variant that zeroes a correct claim whose subclaims are all wrong. Second, problem inversion builds benchmark queries: take a long-context reasoning problem (given a document

What would settle it

Gather a spread of queries straddling the stated thresholds (roughly 10 searches or 10 expert-minutes), have experienced researchers label each as deep research or not, and measure agreement with the definition's cutoffs: if the cutoffs misclassify a substantial share (say, more than a fifth of straddling queries), the formal characterization fails. A single crisp counterexample would also settle it: a query practitioners uniformly call deep research that an expert can answer in under 10 minutes with fewer than 10 searches.

Watch

Extended reading notes

Core claim

Deep research is defined by two intensities, not output format: a query qualifies when it requires processing many information units (search intensity) and at least one of finding, processing, or combining them needs non-trivial reasoning (reasoning intensity). The paper recasts the task as $\langle \text{query}, \text{answers-as-claims}, \text{corpus} \rangle$, separating search-and-reasoning from report writing and enabling objective claim-level precision and recall. LiveDRBench, 100 queries over science and world events, built by inverting long-context reasoning problems so no single web page answers them; across current deep research products, per-category F1 spans 0.02–0.72 with the bes

Load-bearing premise

The definition rests on uncalibrated thresholds—about 20 information units through at least 10 searches, or more than 10 minutes of an expert's time—and the paper gives no evidence these cutoffs separate deep research from ordinary multi-hop questions or simple list gathering.

Editorial extensions

If this is right

  • Progress on deep research becomes measurable by claim precision and recall, decoupled from report-writing quality, so the search-and-reasoning skill and the writing skill can be improved independently.
  • The strict metric (a correct claim scores zero when all its subclaims are wrong) makes grounding-checkable evaluation standard, penalizing answers that are right by memorization rather than by search.
  • Current failure patterns point to concrete fixes: enumerative tasks need explicit algorithm following, and material-identification tasks need joint optimization of the claim and its grounding—systems often get the material or the paper title, but not both.
  • Because queries are generated by inverting fresh documents and events, the benchmark can be periodically regenerated, diluting contamination from models trained on the web.
  • The trace analysis makes test-time search behavior a visible tunable axis: the best system branches and backtracks the most, while merely gathering more sources does not improve scores, so search strategy rather than scale may drive deep research quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The definition is corpus-relative by construction: a query that is deep research today can become trivial if a summarizing page appears, so the benchmark's requirement that no single document answers a query must be re-checked continuously—a tension the paper's own desiderata acknowledge without fully resolving.
  • The 10-minute and 20-information-unit thresholds are posits with no calibration; a natural follow-up is to fit them to expert labels across task families, possibly yielding a graded deep-research difficulty score instead of a binary tag.
  • Because the ground truth is expanded using the evaluated systems' own correct answers, benchmark scores are partly model-dependent; rankings could shift if a future system surfaces valid answers the current evaluators did not anticipate.
  • The claim-subclaim scoring scheme doubles as a cheap grounding audit—correct claim with wrong evidence scores zero—and could be reused outside benchmarks to validate agent search traces without full provenance verification.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a formal characterization of the deep research (DR) task, arguing that DR is defined not by long-form report output but by high 'search intensity' and 'reasoning intensity' over information units. It introduces Definition 1 (Section 3.1), a claim-based intermediate output representation, and precision/recall metrics over claim/subclaim structure (Section 3.2). The authors construct LiveDRBench, a benchmark of 100 tasks in scientific and public-interest domains, using an 'inversion' of long-context reasoning and other NLP tasks (Section 4). They evaluate three commercial DR systems and several LLM baselines, reporting F1 scores from 0.02 to 0.72 and an overall best F1 of 0.55 for OpenAI's DR model. They also analyze reasoning traces in terms of sources, branches, and backtracking (Section 5.3).

Significance. If the formal definition and benchmark construction are sound, the paper makes a useful contribution: it separates search/reasoning from report generation, provides a public benchmark with a recipe for periodic updates, and introduces objective claim-level metrics. The problem-inversion methodology is a practical way to generate DR queries that resist direct web lookup, and the trace analysis offers a starting point for studying agent behavior. The paper also reports baseline results across multiple commercial systems, which is valuable given the rapid deployment of DR models. However, the centrality of the formal definition and the claimed objectivity of the benchmark make the threshold operationalizations and ground-truth expansion procedure load-bearing.

major comments (3)
  1. [Section 3.1, Definition 1] The operationalization of 'reasoning-intensive' as 'more than ten minutes' of total expert task time collapses the two axes of Definition 1. A task with high search intensity but only routine filtering (e.g., the ENTITIES category, which scrapes 80-140 items) can exceed ten minutes without any non-trivial reasoning, while a single-document legal analysis requiring substantive inference could finish in under ten minutes by a fast expert. Similarly, the '20 information units (through at least 10 search queries)' threshold counts processed units, not units found via search, so a query over one very long document could cross the threshold without any search. Because the two axes are not independently operationalized, Definition 1 does not determine which queries are DR; in practice it reduces to a duration heuristic. The paper's caveat that quantifying intensity is subjective does not resolv
  2. [Section 4.4, Ground-truth expansion] The procedure of adding correct model-generated claims to the ground-truth set after reviewing all system outputs can inflate recall and makes the benchmark answers depend on the evaluated systems. This is especially concerning because the benchmark is intended to be objective and reproducible: if the ground truth is expanded only after seeing model outputs, then a system that produces a correct-but-absent claim gets credit, while a different system evaluated before the expansion would not. The paper does not report how many claims were added per category or whether the expansion was verified by any external criterion. A pre-registered or frozen ground truth with a separately documented 'supplementary correct answers' list would preserve objectivity while still acknowledging answer non-uniqueness.
  3. [Section 4.4 and Appendix B] The claim-agreement scores, the dictionary matching for NOVEL DS/FLIGHTS, and the trace analysis (Section 5.3) all rely on GPT-4o as the judge. While the authors provide a manual evaluation in Appendix D showing aggregate agreement, this manual check was performed by the authors themselves and not reported per-item or with inter-annotator agreement. Since the main evaluation claims—especially the F1 ranking and the 'objective benchmark' claim—depend on this judge, the paper should provide an error analysis of the judge (e.g., disagreement rates, examples, or a small independent annotation) to rule out systematic bias, particularly when comparing OpenAI systems against others.
minor comments (6)
  1. [Throughout] The benchmark name is typeset inconsistently as 'LIVE DRBENCH' in the abstract and title, 'LIVE DRBENCH' in some section headers, and 'LiveDRBench' elsewhere. Please unify.
  2. [Section 5.2] Typo: 'Retreival' should be 'Retrieval' in the sentence comparing Perplexity and Gemini on NOVEL DS tasks.
  3. [Tables 3-10] Table headers are inconsistent: Table 3 says 'Perplexity Pro' while other tables say 'Perplexity DR'. Also, Table 10 in the text is labeled 'Comparison of DR models' but includes baselines; please align captions with content.
  4. [Section 5.3] The reference to 'Figure 5.3' is incorrect; it should be 'Figure 5'. The figure caption also says 'Box Plots' but the figure is presented as box plots of trace characteristics; please clarify what the lower panel 'F1 Efficiency per event' shows.
  5. [References] Several references are incomplete or inconsistently formatted: the 'FutureSearch' entry has an author field with a colon and a stray name; the 'HuggingFace' author list is garbled; and the Sonar Reasoning URL points to the Sonar Pro page. Please check all URLs and author lists.
  6. [Section 3.2, Equation (1)] The notation 'Prec(Ai)' for atomic claims is used before the recursive definition is given; define 'subc' and the base case before using the shorthand. Also clarify whether the weights wi are set to 1 in the reported results and whether they are part of the benchmark release.

Circularity Check

1 steps flagged · score 2.0 of 10

Formal definition's reasoning-intensity axis is defined by the same 10-minute total-time heuristic as DR itself, collapsing the two axes; the benchmark evaluation itself is not circular.

  1. self definitional [Section 3.1, Definition 1 and following operationalization paragraphs]
    "While quantifying search or reasoning intensity is subjective, we posit that DR corresponds to any query that takes more than 10 minutes for an ideal human expert. ... While reasoning-intensive for a human expert is difficult to quantify, we posit that a problem is reasoning-intensive if the expected time taken by the expert to solve the task using available tools such as web search is more than ten minutes."

    Definition 1 makes 'search intensity' and 'reasoning intensity' independent conjuncts of a deep research query. The paper then defines 'reasoning-intensive' by the same >10-minute total expert time that it earlier used to characterize DR itself. Since total task time includes the search component, a high-search/low-reasoning task (e.g., an ENTITIES enumeration of 80-140 items) that crosses the threshold is automatically labeled reasoning-intensive even if no finding/processing/combining subtask requires non-trivial reasoning. Thus the reasoning-intensity conjunct adds no independent constraint, and the formal definition reduces by construction to the 10-minute heuristic rather than to the two promised axes.

full rationale

The paper's central benchmark construction and model evaluation are not circular: ground truth is obtained from external sources (CURIE, manually written abstracts, scraped event data) and from human-verified post-hoc expansion, not from the evaluated models as fitted parameters. The GPT-4o judge is cross-checked against human ratings in Table 11, so the evaluation does not reduce to a self-referential LLM judgment. The only real circularity is inside the 'formal definition': the 10-minute threshold is used both to define DR directly and to instantiate 'reasoning intensity,' making the two-axis definition collapse into a single time heuristic. This weakens the formal characterization but does not infect the benchmark results, which would stand even if the definitional thresholds were recalibrated. No load-bearing self-citations or imported uniqueness theorems are present.

Assumptions & free parameters 2 free parameters · 5 assumptions · 1 invented entities

The core definition rests on two hand-picked thresholds (10 minutes, 20 information units) and several domain assumptions about human expert time, claim decomposability, LLM judgment, and web corpus properties. No physical entities are invented; the claim representation is a conceptual tool with no external falsifiable handle.

free parameters (2)
  • reasoning intensity threshold = 10 minutes
    Posited in Section 3.1: 'we posit that a problem is reasoning-intensive if the expected time taken by the expert to solve the task using available tools such as web search is more than ten minutes'.
  • search intensity threshold = 20 information units / 10 search queries
    Posited in Section 3.1: 'a typical threshold may be 20 information units (through at least 10 search queries)'.
assumptions (5)
  • domain assumption An ideal human expert's solving time is a valid measure of reasoning intensity
    Invoked in Section 3.1 when defining reasoning intensity via the 10 minute threshold; no empirical evidence connects expert time to the proposed definition.
  • domain assumption Relevant information can be decomposed into claims and subclaims that capture correctness
    Used throughout Section 3.2 to define metrics; assumes a canonical and complete claim decomposition exists for each query.
  • domain assumption GPT-4o can reliably judge claim agreement
    Used in Section 4.4 for the main evaluation; the authors provide a manual cross-check in Appendix D but still rely on the LLM judge for the headline numbers.
  • domain assumption The entire web is the retrieval corpus and no single document directly answers each query
    Stated in Section 4.1 as a desideratum; the benchmark's difficulty depends on this being true, but it is verified only by construction and manual checks.
  • domain assumption Problem inversion from long-context QA preserves query difficulty and uniqueness
    Used in Section 4.1 to create queries from CURIE and other sources; the authors manually verify uniqueness, but the mapping from long-context QA to DR is not formally characterized.
invented entities (1)
  • claim-based intermediate output representation
    purpose: Abstracts deep research output into a list of claims with subclaims to enable objective metrics
    Introduced as a definitional choice in Section 3.2 rather than an empirically validated representation; it is a useful evaluation tool but not a discovered entity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Characterizing Deep Research: A Benchmark and Formal Definition." pith.science (2026). https://pith.science/paper/KPWYTBV3

@misc{pith2026250804183,
  author       = {Pith},
  title        = {Pith review of: Characterizing Deep Research: A Benchmark and Formal Definition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KPWYTBV3}},
  note         = {Machine review of arXiv:2508.04183}
}
read the original abstract

Information tasks such as writing surveys or analytical reports require complex search and reasoning, and have recently been grouped under the umbrella of \textit{deep research} -- a term also adopted by recent models targeting these capabilities. Despite growing interest, the scope of the deep research task remains underdefined and its distinction from other reasoning-intensive problems is poorly understood. In this paper, we propose a formal characterization of the deep research (DR) task and introduce a benchmark to evaluate the performance of DR systems. We argue that the core defining feature of deep research is not the production of lengthy report-style outputs, but rather the high fan-out over concepts required during the search process, i.e., broad and reasoning-intensive exploration. To enable objective evaluation, we define DR using an intermediate output representation that encodes key claims uncovered during search-separating the reasoning challenge from surface-level report generation. Based on this formulation, we propose a diverse, challenging benchmark LiveDRBench with 100 challenging tasks over scientific topics (e.g., datasets, materials discovery, prior art search) and public interest events (e.g., flight incidents, movie awards). Across state-of-the-art DR systems, F1 score ranges between 0.02 and 0.72 for any sub-category. OpenAI's model performs the best with an overall F1 score of 0.55. Analysis of reasoning traces reveals the distribution over the number of referenced sources, branching, and backtracking events executed by current DR systems, motivating future directions for improving their search mechanisms and grounding capabilities. The benchmark is available at https://github.com/microsoft/LiveDRBench.

Figures

Figures reproduced from arXiv: 2508.04183 by the authors.

Figure 1
Figure 1. Characterizing the deep research task. Left (a) shows the landscape of various multi-hop reasoning tasks. Compared to existing tasks, the deep research task involves both high search and high reasoning intensity. Right (b) shows a stylized process of generating an answer to a DR query: DR query → Claims → Long form report. LIVEDRBENCH focuses on the precision and completeness of the intermediate but crucial step of … view at source ↗
Figure 2
Figure 2. Average F1 score for deep research models and search-enabled LLMs on LIVEDR￾BENCH [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 4
Figure 4. Problem inversion process to create LIVEDRBENCH tasks. The first step is to find a long-context or document reasoning problem that includes a question based on the document and its ground-truth answer. In the second step, this problem is inverted to create a new question asking for an event or entity consistent with the properties mentioned in an answer. In the third step, the question is refined (e.g., more propert… view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: (Top) Box Plots illustrating the number of [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From Simple QA to Deep Research: A Verifiable Benchmark Constructed through Iterative Task Evolution

    cs.AI 2026-08 conditional novelty 6.0 of 10

    An automatic pipeline evolves simple QA questions into 500 deep-research tasks with DAG-structured, fact-grounded rubrics that discriminate between models.

  2. DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generation

    cs.CL 2025-12 conditional novelty 6.0 of 10

    DEER uses 7 evaluation dimensions, 101 rubric items, task-specific expert guidance, and unsupported-claim backtracking to score deep-research reports; current systems score lowest on fulfilling expert requests and ana...

  3. SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agents

    cs.AI 2025-09 conditional novelty 6.0 of 10

    A 20B autonomously reasoning deep-research agent trained with synthetic-data RL reaches 28.7% on Humanity's Last Exam, exceeding several larger and proprietary baselines.

Reference graph

Works this paper leans on

44 extracted references · 21 canonical work pages · cited by 3 Pith papers

  1. [1]

    Openscholar: Synthesizing scientific literature with retrieval-augmented lms

    Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D'arcy, et al. Openscholar: Synthesizing scientific literature with retrieval-augmented lms. arXiv preprint arXiv:2411.14199, 2024

    arXiv 2024

  2. [2]

    Deerflow: Deep exploration and efficient research flow

    Henry Li Bytedance, Daniel Walnut. Deerflow: Deep exploration and efficient research flow. https://github.com/bytedance/deer-flow, 2025

    work page 2025

  3. [3]

    Deep research comparator: A platform for fine-grained human annotations of deep research agents

    Prahaladh Chandrahasan, Jiahe Jin, Zhihan Zhang, Tevin Wang, Andy Tang, Lucy Mo, Morteza Ziyadi, Leonardo FR Ribeiro, Zimeng Qiu, Markus Dreyer, et al. Deep research comparator: A platform for fine-grained human annotations of deep research agents. arXiv preprint arXiv:2507.05495, 2025

    work page arXiv 2025

  4. [4]

    Patent claims revisited

    Dargaye Churnet. Patent claims revisited. Nw. J. Tech. & Intell. Prop., 11: 0 501, 2012

    work page 2012

  5. [5]

    Deepresearchgym: A free, transparent, and reproducible evaluation sandbox for deep research, 2025

    João Coelho, Jingjie Ning, Jingyuan He, Kangrui Mao, Abhijay Paladugu, Pranav Setlur, Jiahe Jin, Jamie Callan, João Magalhães, Bruno Martins, and Chenyan Xiong. Deepresearchgym: A free, transparent, and reproducible evaluation sandbox for deep research, 2025. URL https://arxiv.org/abs/2505.19253

    arXiv 2025

  6. [6]

    Curie: Evaluating llms on multitask scientific long-context understanding and reasoning

    Hao Cui, Zahra Shamsi, Gowoon Cheon, Xuejian Ma, Shutong Li, Maria Tikhanovskaya, Peter Christian Norgaard, Nayantara Mudur, Martyna Beata Plomecka, Paul Raccuglia, et al. Curie: Evaluating llms on multitask scientific long-context understanding and reasoning. In The Thirteenth International Conference on Learning Representations

    work page

  7. [7]

    Claim verification in the age of large language models: A survey

    Alphaeus Dmonte, Roland Oruche, Marcos Zampieri, Prasad Calyam, and Isabelle Augenstein. Claim verification in the age of large language models: A survey. arXiv preprint arXiv:2408.14317, 2024

    arXiv 2024

  8. [8]

    Deepresearch bench: A comprehensive benchmark for deep research agents, 2025

    Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. Deepresearch bench: A comprehensive benchmark for deep research agents, 2025. URL https://arxiv.org/abs/2506.11763

    arXiv 2025

Show all 44 references
  1. [9]

    Eli5: Long form question answering

    Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. Eli5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558--3567, 2019

    2019

  2. [10]

    Bosse, Jon Evans, Robert G

    FutureSearch, :, Nikos I. Bosse, Jon Evans, Robert G. Gambee, Daniel Hnyk, Peter Mühlbacher, Lawrence Phillips, Dan Schwarz, and Jack Wildman. Deep research bench: Evaluating ai web research agents, 2025. URL https://arxiv.org/abs/2506.06287

    2025 arXiv

  3. [11]

    Analysis of plan-based retrieval for grounded text generation

    Ameya Godbole, Nicholas Monath, Seungyeon Kim, Ankit Singh Rawat, Andrew McCallum, and Manzil Zaheer. Analysis of plan-based retrieval for grounded text generation. arXiv preprint arXiv:2408.10490, 2024

    2024 arXiv

  4. [12]

    We’re expanding our gemini 2.5 family of models

    Google. We’re expanding our gemini 2.5 family of models. https://blog.google/products/gemini/gemini-2-5-model-family-expands/, a . Accessed: 2025-08-04

    2025

  5. [13]

    Deep research is now available on gemini 2.5 pro experimental

    Google. Deep research is now available on gemini 2.5 pro experimental. https://blog.google/products/gemini/deep-research-gemini-2-5-pro-experimental/, b . Accessed: 2025-08-04

    2025

  6. [14]

    Precise information control in long-form text generation

    Jacqueline He, Howard Yen, Margaret Li, Shuyue Stella Li, Zhiyuan Zeng, Weijia Shi, Yulia Tsvetkov, Danqi Chen, Pang Wei Koh, and Luke Zettlemoyer. Precise information control in long-form text generation. arXiv preprint arXiv:2506.06589, 2025

    2025

  7. [15]

    Cuad: An expert-annotated nlp dataset for legal contract review

    Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. Cuad: An expert-annotated nlp dataset for legal contract review. arXiv preprint arXiv:2103.06268, 2021

    2021 arXiv

  8. [16]

    Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609--6625, 2020

    2020

  9. [17]

    Bright: A realistic and challenging benchmark for reasoning-intensive retrieval

    SU Hongjin, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang, Liu Haisu, Quan Shi, Zachary S Siegel, Michael Tang, et al. Bright: A realistic and challenging benchmark for reasoning-intensive retrieval. In The Thirteenth International Conference on Learnin...

  10. [18]

    Deep research agents: A systematic examination and roadmap, 2025

    Yuxuan Huang, Yihang Chen, Haozheng Zhang, Kang Li, Meng Fang, Linyi Yang, Xiaoguang Li, Lifeng Shang, Songcen Xu, Jianye Hao, Kun Shao, and Jun Wang. Deep research agents: A systematic examination and roadmap, 2025. URL https://arxiv.org/abs/2506.18096

    2025 arXiv

  11. [19]

    Open-source deepresearch – freeing our search agents

    Aymeri HuggingFace: Roucher, Albert Villanova del Moral, merve, Thomas Wolf, and Clémentine Fourrier. Open-source deepresearch – freeing our search agents. https://huggingface.co/blog/open-deep-research

  12. [20]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  13. [21]

    Researcharena: Benchmarking llms' ability to collect and organize information as research agents

    Hao Kang and Chenyan Xiong. Researcharena: Benchmarking llms' ability to collect and organize information as research agents. arXiv e-prints, pages arXiv--2406, 2024

    2024

  14. [22]

    Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation, 2024

    Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, and Manaal Faruqui. Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation, 2024. URL https://arxiv.org/abs/2409.12941

    2024 arXiv

  15. [23]

    Webthinker: Empowering large reasoning models with deep research capability

    Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji-Rong Wen, and Zhicheng Dou. Webthinker: Empowering large reasoning models with deep research capability. arXiv preprint arXiv:2504.21776, 2025

    2025 arXiv

  16. [24]

    Lost in the middle: How language models use long contexts

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12: 0 157--173, 2024

    2024

  17. [25]

    Veritrail: Closed-domain hallucination detection with traceability

    Dasha Metropolitansky and Jonathan Larson. Veritrail: Closed-domain hallucination detection with traceability. arXiv preprint arXiv:2505.21786, 2025

    2025

  18. [26]

    Gaia: a benchmark for general ai assistants

    Gr \'e goire Mialon, Cl \'e mentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, 2023

    2023

  19. [27]

    Introducing researcher and analyst in microsoft 365 copilot

    Microsoft and Jared Spataro. Introducing researcher and analyst in microsoft 365 copilot. https://www.microsoft.com/en-us/microsoft-365/blog/2025/03/25/introducing-researcher-and-analyst-in-microsoft-365-copilot/, 2025. Accessed: 2025-07-31

    2025

  20. [28]

    Introducing gpt-4.1 in the api

    OpenAI. Introducing gpt-4.1 in the api. https://openai.com/index/gpt-4-1/, a . Accessed: 2025-08-04

    2025

  21. [29]

    Introducing openai o3 and o4-mini

    OpenAI. Introducing openai o3 and o4-mini. https://openai.com/index/introducing-o3-and-o4-mini/, b . Accessed: 2025-08-04

    2025

  22. [30]

    Introducing deep research

    OpenAI. Introducing deep research. https://openai.com/index/introducing-deep-research/, 2025

    2025

  23. [31]

    Perplexity deep research

    PerplexityAI. Perplexity deep research. https://www.perplexity.ai/hub/blog/introducing-perplexity-deep-research, a . Accessed: 2025-08-04

    2025

  24. [32]

    Sonar pro

    PerplexityAI. Sonar pro. https://docs.perplexity.ai/getting-started/models/models/sonar-pro, b . Accessed: 2025-08-04

    2025

  25. [33]

    Sonar reasoning

    PerplexityAI. Sonar reasoning. https://docs.perplexity.ai/getting-started/models/models/sonar-pro, c . Accessed: 2025-08-04

    2025

  26. [34]

    Humanity's last exam

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity's last exam. arXiv preprint arXiv:2501.14249, 2025

    2025 arXiv

  27. [35]

    Assisting in writing wikipedia-like articles from scratch with large language models

    Yijia Shao, Yucheng Jiang, Theodore A Kanell, Peter Xu, Omar Khattab, and Monica S Lam. Assisting in writing wikipedia-like articles from scratch with large language models. arXiv preprint arXiv:2402.14207, 2024

    2024 arXiv

  28. [36]

    Pangu deepdiver: Adaptive search intensity scaling via open-web reinforcement learning

    Wenxuan Shi, Haochen Tan, Chuqiao Kuang, Xiaoguang Li, Xiaozhe Ren, Chen Zhang, Hanting Chen, Yasheng Wang, Lifeng Shang, Fisher Yu, et al. Pangu deepdiver: Adaptive search intensity scaling via open-web reinforcement learning. arXiv preprint arXiv:2505.24332, 2025

    2025

  29. [37]

    Geak: Introducing triton kernel ai agent & evaluation benchmarks

    Jianghui Wang, Vinay Joshi, Saptarshi Majumder, Xu Chao, Bin Ding, Ziqiong Liu, Pratik Prabhanjan Brahma, Dong Li, Zicheng Liu, and Emad Barsoum. Geak: Introducing triton kernel ai agent & evaluation benchmarks. arXiv preprint arXiv:2507.23194, 2025

    2025 arXiv

  30. [39]

    Browsecomp: A simple yet challenging benchmark for browsing agents

    Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516, 2025 b

    2025 arXiv

  31. [40]

    Grok 3 beta — the age of reasoning agents

    xAI. Grok 3 beta — the age of reasoning agents. https://x.ai/news/grok-3. Accessed: 2025-08-04

    2025

  32. [41]

    A comprehensive survey of deep research: Systems, methodologies, and applications, 2025

    Renjun Xu and Jingwen Peng. A comprehensive survey of deep research: Systems, methodologies, and applications, 2025. URL https://arxiv.org/abs/2506.12594

    2025 arXiv

  33. [42]

    Researcherbench: Evaluating deep ai research systems on the frontiers of scientific inquiry

    Tianze Xu, Pengrui Lu, Lyumanshan Ye, Xiangkun Hu, and Pengfei Liu. Researcherbench: Evaluating deep ai research systems on the frontiers of scientific inquiry. arXiv preprint arXiv:2507.16280, 2025

    2025 arXiv

  34. [43]

    Cohen, Ruslan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA : A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing ( EMNLP ) , 2018

    2018

  35. [44]

    Open deep research

    David Zhang. Open deep research. https://github.com/dzhng/deep-research, 2025

    2025

  36. [45]

    Deepresearcher: Scaling deep research via reinforcement learning in real-world environments

    Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments. arXiv preprint arXiv:2504.03160, 2025

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.