Pith. sign in

REVIEW 3 major objections 5 minor 19 cited by

BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A fixed, human-verified corpus of 830 BrowseComp queries lets researchers measure retrieval and reasoning separately in deep-research agents, and the measurements show the retriever often decides the score.

desk verdict A genuinely useful fixed-corpus benchmark for deep-research evaluation, with caveats about incomplete qrels and filtered queries. read the letter →

arxiv 2508.06600 v1 pith:S36GHMVD submitted 2025-08-08 cs.CL cs.IR

Zijian Chen , Xueguang Ma , Shengyao Zhuang , Ping Nie , Kai Zou , Andrew Liu , Joshua Green , Kshama Patel
show 12 more authors
Ruoxi Meng Mingyi Su Sahel Sharifymoghaddam Yanxi Li Haoran Hong Xinyu Shi Xuye Liu Nandan Thakur Crystina Zhang Luyu Gao Wenhu Chen Jimmy Lin
This is my paper
classification cs.CLcs.IR
keywords BrowseComp-Plusdeep-researchagentsbenchmarkretrievalevaluationCranfieldparadigmhardnegativesreproducibilitytool-usereasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that current evaluations of deep-research agents — LLMs that search iteratively and reason over results — conflate two distinct abilities: finding evidence and reasoning over it. BrowseComp-Plus is its answer: a fixed, human-verified corpus for 830 BrowseComp queries, with evidence documents, answer-containing gold documents, and mined hard negatives per query, so each component can be scored independently and runs are reproducible without a live web API. The paper demonstrates the payoff by pairing seven agent models with several retrievers: better retrieval lifts accuracy sharply (gpt-5 from 55.9% to 70.1%; gpt-4.1 from 14.6% to 35.4%) while cutting search calls, and oracle retrieval reaches 93.5%, locating the main bottleneck in retrieval rather than reasoning. If the benchmark is right, component-level diagnosis of deep-research systems becomes tractable and cheap.

What carries the argument

The load-bearing object is the per-query, human-verified evidence set: for each of 830 queries, on average 6.1 evidence documents, 2.9 gold documents (which contain the answer in implicit or variant form, not just as an exact string), and about 76 mined hard negatives, all inside one fixed 100,195-document corpus. This object carries the argument by turning the retriever into a measurable component: standalone retrieval is scored with Recall@k and nDCG under the Cranfield paradigm, and any accuracy difference between agent runs becomes attributable to the retriever, the reasoning model, or their interaction, rather than to a drifting web corpus.

What would settle it

Judge a random sample of corpus documents that are currently unlabeled for their query — including the Fineweb-augmented runs' top-ranked "false negatives" — with fresh annotators: if a meaningful share contain verifiable answer evidence, the completeness premise fails and the published Recall and nDCG figures are underestimates. A second check: rebuild the evidence labels with an independent discovery agent (not the one used to mine the original set) and see whether the relative ranking of retrievers at Recall@1000 survives the new label set.

Watch

Extended reading notes

Core claim

BrowseComp-Plus converts 830 of BrowseComp's 1,266 queries into a closed evaluation: a deduplicated, fixed corpus of 100,195 documents in which each query carries human-verified evidence documents, gold documents containing the final answer semantically (not just as a substring), and mined hard negatives. The paper's central claim is that this makes retrieval a controlled variable — retrieval alone can be scored under the Cranfield paradigm with Recall@k and nDCG, and end-to-end agent runs become reproducible because no live web API is involved. The demonstration is that retriever choice, more than model identity, separates agent scores: gpt-4.1 rises from 14.58% (BM25) to 35.42% (Qwen3-Embe

Load-bearing premise

The load-bearing premise is that the human-verified evidence set is complete enough for every unlabeled corpus document to be treated as non-relevant: annotators only verified pages surfaced by one web-enabled model plus about 20 minutes of extra searching, and Section 4.8.4 explicitly scores unjudged added documents as non-relevant — so if relevant evidence sits unlabeled in the corpus, the reported Recall@k, nDCG, and citation metrics are lower bounds and agent rankings cou

Editorial extensions

If this is right

  • Retriever choice becomes a first-order lever: replacing BM25 with Qwen3-Embedding-8B more than doubles accuracy for several agents (gpt-4.1: 14.58% to 35.42%; Sonnet 4: 14.34% to 36.75%) and raises gpt-5 from 55.90% to 70.12%.
  • Retrieval, not reasoning, is the measured bottleneck: the strongest retriever recalls only 14.5% of evidence documents at k=5, whereas oracle retrieval gets gpt-4.1 to 93.49%, so closing the retrieval gap is the largest available gain.
  • The open-source deficit is in interleaved tool use, not question answering: Qwen3-32B scores 83.25% when handed all positive documents versus 10.36% end-to-end.
  • Reasoning effort trades compute against coverage: gpt-oss-120B with Qwen3-Embedding-8B rises from 24.94% (low effort) to 42.89% (high), while search calls and cost rise with effort.
  • Tenfold corpus growth (adding Fineweb documents) leaves retriever and agent rankings intact, supporting the compact 100k corpus as a fast, valid experimental testbed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the oracle-gap decomposition (93.49% oracle vs. 14.58% with BM25 for gpt-4.1) quantifies headroom per system; the same decomposition could be applied to other multi-hop QA benchmarks if comparable human-verified evidence labels were produced.
  • Editorial extension: the paper leaves open whether retriever quality shapes reinforcement-learning dynamics; a testable follow-up would train Search-R1-style agents against BM25 versus a dense retriever and compare training curves and cross-retriever generalization.
  • Editorial extension: because unlabeled corpus documents count as non-relevant, the published Recall and nDCG figures are lower bounds; pooling and judging the top-ranked documents of many retrievers would produce a second-generation label set that measures the current evidence set's completeness.
  • Editorial extension: the citation metrics (coverage, precision, and recall against labeled evidence) form a cheap, corpus-grounded faithfulness proxy that other fixed-corpus agent benchmarks could adopt as a standard reporting block.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces BrowseComp-Plus, a fixed-corpus benchmark derived from BrowseComp for evaluating deep-research agents. It filters the original 1,266 queries to 830, attaches human-verified evidence and gold documents to each query, adds mined hard negatives, and assembles a 100,195-document corpus. The authors then evaluate a range of closed- and open-source LLM agents paired with BM25, Qwen3-Embedding, and ReasonIR retrievers, reporting end-to-end accuracy, recall, search calls, citation metrics, and standalone retriever effectiveness. The central claim is that this carefully curated fixed corpus enables fair, reproducible, and component-level evaluation of retrieval and LLM reasoning in deep-research systems.

Significance. If the construction is sound, BrowseComp-Plus fills a real gap: current BrowseComp-style evaluation depends on opaque, dynamic web search APIs, making it hard to disentangle retriever quality from agent reasoning. The paper's strengths are substantial: a fixed public corpus, 400+ hours of human annotation, over 80% inter-annotator agreement, a Fineweb-based robustness check, and an unusually broad set of agent/retriever pairings. The oracle-retrieval and reasoning-effort ablations are informative, and the observation that retriever quality affects accuracy and search-call efficiency is a useful empirical contribution. The main risk is that the evidence/qrel set is non-exhaustive, so several headline numerical claims are lower bounds whose ranking-level validity has not been demonstrated.

major comments (3)
  1. [§3.2.2 / §4.8.4] The relevance labels are non-exhaustive. Evidence documents are collected from o3's web search plus 'at least 20 minutes' of annotator searching, and §4.8.4 explicitly scores unjudged Fineweb documents as non-relevant. The same policy is implicit for unjudged documents in the 100k corpus. Consequently, Recall@k, nDCG@10, and citation recall in Tables 2 and 3 are lower bounds, and a retriever or agent that finds genuinely relevant but unlabeled evidence is penalized. Because this bears directly on the 'fair and transparent' claim, the paper should provide a sensitivity analysis—e.g., pooling top-ranked unjudged documents from each retriever, having annotators judge them, and recomputing metrics—or at least quantify the possible impact of incomplete qrels. This is the standard Cranfield incompleteness issue, but it is currently a premise rather than a verified property.
  2. [§3.2.1 / Table 1] The evidence set was generated using OpenAI o3, and o3 is later evaluated as a deep-research agent on that same corpus. Human verification ensures the documents are relevant, but the selection of which documents enter the corpus is o3-dependent. This can bias the benchmark in o3's favor and makes the retriever qrels a function of one system's search behavior. A concrete check would be to construct a held-out subset whose evidence is collected by a different pipeline (or from externally available evidence), and verify that retriever rankings and agent insights are stable. Without such a check, the 'fair comparison' claim is weakened.
  3. [§3.2.1–§3.2.2 / Appendix C] Of 1,266 BrowseComp queries, only 830 survive the pipeline: 124 lack evidence, 137 have unscrapable URLs, and 175 fail verification or are excluded for ambiguity. The paper does not analyze whether this subset is representative of BrowseComp or whether conclusions about retrieval difficulty and agent ranking transfer to excluded queries. Since BrowseComp-Plus is proposed as a benchmark for deep-research agents, not merely a diagnostic subset, the authors should report characteristics of excluded queries (category distribution, difficulty proxies, answer types) and show that the main results are stable under alternative inclusion criteria.
minor comments (5)
  1. [Abstract / §2.1] Typo: 'retrieval-agumented generation' should be 'retrieval-augmented generation'.
  2. [Table 6 vs Table 2] Table 6 reports BM25 Original Recall@1000 as 13.6%, while Table 2 reports 13.7% for the same setting. The source of the small discrepancy should be clarified or corrected.
  3. [§4.7] The phrase 'Qwen3-8B surpassing ReasonIR-8B' should read 'Qwen3-Embedding-8B' for consistency with the rest of the paper.
  4. [Throughout] Model-name capitalization is inconsistent (e.g., 'GPT-5' vs 'gpt-5', 'SearchR1' vs 'Search-R1'). Please unify.
  5. [Figure 1] The axes and markers in Figure 1 are not fully described in the caption; please define what each point and color represents and state which retriever/LLM combinations are shown.

Circularity Check

1 steps flagged · score 2.0 of 10

Benchmark construction is empirically grounded; only minor o3-seeded label bias and standard non-exhaustive-qrels caveats, no derivation-level circularity.

  1. other [Section 3.2.1 (Evidence Document Gathering) and Section 4.5 (Results)]
    "we leverage the OpenAI o3 model with web search enabled. We provide the question-answer pairs as input prompts and instruct the model to search online for web pages containing evidence that supports the answers."

    The relevance labels used to score o3 (Table 1: 63.49% accuracy with Qwen3-Embed-8B) are seeded by o3's own web-search output: candidate evidence URLs become the human-verified evidence set unless rejected or supplemented by annotators. o3 therefore helped define the target distribution against which it is later measured, so its high scores are partly a self-evaluation artifact. This is not a full deductive equivalence—human verification (80% agreement) and the 20-minute supplemental-search fallback break the identity—so it contributes only mild circularity.

full rationale

The paper contains no derived equations or fitted parameters called predictions; it constructs a benchmark and reports measurements. The central claim—that a fixed, human-verified corpus enables independent component evaluation—is grounded by the annotation pipeline (14 annotators, 400+ hours, >80% cross-annotator agreement) and by the oracle-retrieval check (gpt-4.1 reaches 93.49% when given all labeled positive documents). The only circularity-adjacent step is the o3-seeded label construction in §3.2.1: because o3 is among the evaluated agents (§4.5), its performance is partly measured on a label set it helped create; the human-verification step keeps this from being a definitional reduction. The paper itself flags the standard Cranfield completeness limitation in §4.8.4: 'Since these additional documents are unjudged, they are treated as non-relevant under standard TREC-style evaluation, inevitably lowering measured retrieval effectiveness.' That makes Recall/nDCG lower bounds but is a validity caveat, not circularity. Self-citations (Pyserini, Tevatron, corpus-subsampling work) are tooling references, not load-bearing justifications. Overall score 2 reflects the minor o3-seeding bias without treating it as fatal.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the completeness and accuracy of the document labels, which are human-verified but not exhaustive. There are no fitted parameters or invented physical entities; the main epistemic costs are the filtering of the original query set and the assumption that unlabeled documents are irrelevant.

assumptions (4)
  • domain assumption The 830 surviving queries are representative of the original 1,266 BrowseComp queries.
    Sections 3.2.1-3.2.2 and Appendix C show that 124 queries lacked o3 evidence, 137 failed scraping, and 175 failed verification or were problematic. About a third of the original set is gone, so aggregate accuracy and rankings may not transfer to full BrowseComp.
  • domain assumption Evidence relevance labels are exhaustive; unlabeled documents are non-relevant.
    Annotators verify only documents gathered by o3 plus up to 20 minutes of their own searching (Section 3.2.2). Retrieval metrics in Table 2 and citation recall in Table 3 treat all other documents as non-relevant, making recall a lower bound at best.
  • domain assumption GPT-4.1 as judge reliably maps model answers to BrowseComp ground truth.
    Accuracy in Table 1 is decided by gpt-4.1 with the prompt in Appendix F; no human agreement on judge decisions is reported, and judge errors would move accuracy numbers.
  • domain assumption Static-corpus results predict live deep-research agent behavior.
    The paper motivates the fixed corpus as a proxy for web-scale research (Sections 1 and 5). The oracle result of 93.49% accuracy (Section 4.8.1) shows the corpus is easier than open web search; transfer to live conditions is assumed, not demonstrated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent." pith.science (2026). https://pith.science/paper/S36GHMVD

@misc{pith2026250806600,
  author       = {Pith},
  title        = {Pith review of: BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S36GHMVD}},
  note         = {Machine review of arXiv:2508.06600}
}
read the original abstract

Deep-Research agents, which integrate large language models (LLMs) with search tools, have shown success in improving the effectiveness of handling complex queries that require iterative search planning and reasoning over search results. Evaluations on current benchmarks like BrowseComp relies on black-box live web search APIs, have notable limitations in (1) fairness: dynamic and opaque web APIs hinder fair comparisons and reproducibility of deep research methods; (2) transparency: lack of control over the document corpus makes it difficult to isolate retriever contributions. In other words, the current evaluations may compare a complete deep research system at a given time, but they do not foster well-controlled experiments to provide insights into the capability of underlying deep research LLMs. To address these challenges, we introduce BrowseComp-Plus, a benchmark derived from BrowseComp, employing a fixed, carefully curated corpus. Each query in BrowseComp-Plus includes human-verified supporting documents and mined challenging negatives, enabling controlled experimentation. The benchmark is shown to be effective in distinguishing the performance of deep research systems. For instance, the open-source model Search-R1, when paired with the BM25 retriever, achieves 3.86% accuracy, whereas the GPT-5 achieves 55.9%. Integrating the GPT-5 with the Qwen3-Embedding-8B retriever further enhances its accuracy to 70.1% with fewer search calls. This benchmark allows comprehensive evaluation and disentangled analysis of deep research agents and retrieval methods, fostering insights into retrieval effectiveness, citation accuracy, and context engineering in Deep-Research system.

Figures

Figures reproduced from arXiv: 2508.06600 by the authors.

Figure 1
Figure 1. Accuracy vs. number of search calls for Deep-Research agents with different retrievers. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The two-stage pipeline of collecting evidence documents in the corpus (Section 3.2). [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The pipeline of collecting hard negative documents in the corpus(Section 3.3). [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: (a) Token distribution of corpus length, showing up to 90th percentile for display; (b) [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: A screenshot of the annotation interface. [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 19 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning

    cs.CL 2026-08 conditional novelty 7.0 of 10

    Even with gold geographic facts and computation tools, LLMs cap below two-thirds exact match on MultiGlobeQA, with grid indexing and shape computation as the weakest families.

  2. LLM Agents Are Latent Context Managers: Eliciting Self-Managed Context via State Proprioception

    cs.CL 2026-06 unverdicted novelty 7.0 of 10

    VISTA supplies LLM agents with a visible proprioceptive dashboard of typed context blocks, enabling untrained self-management that lifts performance on long-horizon tool-use benchmarks across multiple model scales.

  3. HiEviDR-Bench: A Benchmark for Hierarchical Evidence Aggregation in Deep Research

    cs.IR 2026-07 conditional novelty 6.5 of 10

    A hierarchical evidence-graph benchmark reveals that multimodal deep-research models write fluent reports while failing citation, claim, and answer grounding.

  4. Fetch-then-Explore: Decoupling Selection from Extraction over a Persistent Workspace for Search Agents

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Fetch-then-Explore, which stores fetched pages in a per-question workspace and extracts evidence on demand with grep/read, beats visit-and-read and browsing baselines on BrowseComp across three LLM backbones.

  5. Diagnosing Search Behavior and Failure Modes in Long-Horizon Search Agents

    cs.AI 2026-08 conditional novelty 6.0 of 10

    For six deep search agents on BrowseComp-Plus, answer accuracy tracks cumulative retrieval recall, not search effort, and failures split into missing-evidence and evidence-misuse gaps.

  6. A New Role for Relevance: Guiding Corpus Interaction in Agentic Search

    cs.CL 2026-07 conditional novelty 6.0 of 10

    RARG turns relevance scores into an execution order for grep-based corpus search, improving accuracy per tool call over retrieval and direct-corpus-interaction agents.

  7. DRNOISE: Benchmarking Deep Research Agents in Misleading Evidence Environments

    cs.LG 2026-07 conditional novelty 6.0 of 10

    One ordinary-looking false document makes strong deep-research agents abandon record-level evidence and commit the false value on 66–88% of previously solved tasks.

  8. From Memory to Skills: Evidence-Grounded Co-Evolution Governance for Long-Horizon LLM Agents

    cs.CL 2026-07 conditional novelty 6.0 of 10

    MSCE, a training-free memory–skill co-evolution framework, converts grounded traces into callable skills through reflection-weighted value backfilling and outperforms prior agent memory/skill baselines on EvoAgentBenc...

  9. ECHO: Prune To Act, Trace To Learn With Selective Turn Memory In Agentic RL

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Source-indexed selective turn memory plus provenance-guided credit assignment raises BrowseComp-Plus accuracy to 43.4% over GRPO and SUPO while cutting turn proliferation.

  10. Revisiting Text Ranking in Deep Research

    cs.IR 2026-02 conditional novelty 6.0 of 10

    In deep-research agents, passage-level BM25 retrieval plus re-ranking beats larger neural rankers, and converting agent queries into natural questions (Q2Q) improves neural ranking.

  11. Why Your Deep Research Agent Fails? On Hallucination Evaluation in Full Research Trajectory

    cs.AI 2026-01 conditional novelty 6.0 of 10

    Evaluating deep research agents at every step (plan, search, summarize), not just on the final answer, reveals that all six tested agents hallucinate, with failures that cascade from early fabrications or late-context...

  12. BEAR: Budgeted Evidence Allocation for Multi-Document Reasoning

    cs.CL 2026-01 conditional novelty 6.0 of 10

    A budget-controlled two-path hierarchical retrieval method reports near-full-context accuracy on multi-hop QA with up to 94% token savings, but the reported numbers are internally inconsistent and no artifacts are released.

  13. DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generation

    cs.CL 2025-12 conditional novelty 6.0 of 10

    DEER uses 7 evaluation dimensions, 101 rubric items, task-specific expert guidance, and unsupported-claim backtracking to score deep-research reports; current systems score lowest on fulfilling expert requests and ana...

  14. SafeSearch: Automated Red-Teaming of LLM-Based Search Agents

    cs.AI 2025-09 conditional novelty 6.0 of 10

    An automated red-teaming framework and 300-case benchmark show that a single unreliable website can induce unsafe responses in LLM search agents, with attack success rates up to 90.5%.

  15. DeepTRACE: Auditing Deep Research AI Systems for Tracking Reliability Across Citations and Evidence

    cs.CL 2025-09 conditional novelty 6.0 of 10

    An audit framework and empirical study showing that generative search engines and deep research agents frequently produce one-sided answers and weakly supported citations, with citation accuracy between 40 and 80%.

  16. SearchArt: Training Long-Horizon Search Agent with Scalable Synthetic and Verified Task

    cs.IR 2026-07 conditional novelty 5.0 of 10

    SearchArt post-trains Qwen3.5-27B on verification-filtered synthetic search trajectories, scoring 74.39 on BrowseComp-ZH, 70.06 on BrowseComp, and 52.55 on DeepResearch-Bench, competitive with several 200B-700B agents.

  17. DeepResearch-9K: A Challenging Benchmark Dataset of Deep-Research Agent

    cs.AI 2026-03 conditional novelty 5.0 of 10

    A synthetic benchmark of 9,000 multi-hop web-research questions with difficulty tiers and teacher-generated search trajectories, plus an open-source RL training framework that reportedly lets 3B-parameter agents beat ...

  18. Rerank Before You Reason: Analyzing Reranking Tradeoffs through Effective Token Cost in Deep Search Agents

    cs.IR 2026-01 conditional novelty 5.0 of 10

    Listwise reranking of the top 10–50 retrieved documents delivers comparable or better deep-search accuracy than increasing reasoning effort, at substantially lower effective token cost on BrowseComp-Plus.

  19. Rethinking On-policy Optimization for Query Augmentation

    cs.CL 2025-10 conditional novelty 5.0 of 10

    Simple zero-shot prompt-based query expansion matches costly RL-trained query rewriting in retrieval benchmarks, and training the RL policy to generate pseudo-documents (OPQE) yields the best overall scores.

Reference graph

Works this paper leans on

48 extracted references · 19 canonical work pages · cited by 19 Pith papers

  1. [1]

    Browsecomp: A simple yet challenging benchmark for browsing agents

    Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv:2504.12516, 2025. URL https: //arxiv.org/abs/2504.12516

    arXiv 2025

  2. [2]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proceed- ings of the 34th International Conference on Neural Information Processing Systems, NIPS ...

    work page 2020

  3. [3]

    Self-RAG: Learning to retrieve, generate, and critique through self-reflection

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum? id=hSyW5go0v8

    work page 2024

  4. [4]

    Browsecomp-zh: Benchmarking web browsing ability of large language models in chinese

    Peilin Zhou, Bruce Leon, Xiang Ying, Can Zhang, Yifan Shao, Qichen Ye, Dading Chong, Zhiling Jin, Chenxuan Xie, Meng Cao, Yuxin Gu, Sixin Hong, Jing Ren, Jian Chen, Chao Liu, and Yining Hua. Browsecomp-zh: Benchmarking web browsing ability of large language models in chinese. arXiv:2504.19314, 2025. URL https://arxiv.org/abs/2504.19314

    arXiv 2025

  5. [5]

    Bitterman

    Shan Chen, Pedro Moreira, Yuxin Xiao, Sam Schmidgall, Jeremy Warner, Hugo Aerts, Thomas Hartvigsen, Jack Gallifant, and Danielle S. Bitterman. Medbrowsecomp: Benchmarking medical deep research and computer use. arXiv:2505.14963, 2025. URL https://arxiv.org/abs/ 2505.14963

    arXiv 2025

  6. [6]

    V oorhees.The Evolution of Cranfield, pages 45–69

    Ellen M. V oorhees.The Evolution of Cranfield, pages 45–69. Springer International Publishing, Cham, 2019. ISBN 978-3-030-22948-1. doi: 10.1007/978-3-030-22948-1_2. URL https: //doi.org/10.1007/978-3-030-22948-1_2

    doi:10.1007/978-3-030-22948-1_2 2019

  7. [7]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025

    Gemini 2.5 Team. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. URL https://arxiv.org/abs/ 2507.06261

    arXiv 2025

  8. [8]

    The claude 3 model family: Opus, sonnet, haiku

    Anthropic Team. The claude 3 model family: Opus, sonnet, haiku. 2024. URL https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/ Model_Card_Claude_3.pdf. 14

    work page 2024

Show all 48 references
  1. [9]

    OpenAI o3 and o4-mini system card

    OpenAI Team. OpenAI o3 and o4-mini system card. 2025. URL https://cdn.openai.com/ o3-mini-system-card-feb10.pdf

    2025

  2. [10]

    GPT-OSS-120B & 20B model card

    OpenAI Team. GPT-OSS-120B & 20B model card. 2025. URL https://cdn.openai.com/ pdf/419b6906-9da6-406c-a19d-1bb078ac7637/oai_gpt-oss_model_card.pdf

    2025

  3. [11]

    Introducing the model context protocol

    Anthropic Team. Introducing the model context protocol. November 2024. URL https: //www.anthropic.com/news/model-context-protocol

    2024

  4. [12]

    Search-r1: Training llms to reason and leverage search engines with reinforce- ment learning

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforce- ment learning. arXiv:2503.09516, 2025. URL https://arxiv.org/abs/2503.09516

    2025 arXiv

  5. [13]

    Websailor: Navigating super-human reasoning for web agent

    Kuan Li, Zhongwang Zhang, Huifeng Yin, Liwen Zhang, Litu Ou, Jialong Wu, Wenbiao Yin, Baixuan Li, Zhengwei Tao, Xinyu Wang, Weizhou Shen, Junkai Zhang, Dingchu Zhang, Xixi Wu, Yong Jiang, Ming Yan, Pengjun Xie, Fei Huang, and Jingren Zhou. Websailor: Navigating super-human rea...

    2025 arXiv

  6. [14]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025 arXiv

  7. [15]

    Dense passage retrieval for open-domain question answering

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on E...

    2020 doi

  8. [16]

    The faiss library

    Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. arXiv:2401.08281, 2024

    2024 arXiv

  9. [17]

    Robertson

    Stephen E. Robertson. Okapi at trec-3. In Proceedings of the Third Text REtrieval Conference (TREC-3). NIST, 1994

    1994

  10. [18]

    Bge m3- embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, 2024

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3- embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, 2024

    2024

  11. [19]

    Unsupervised corpus aware language model pre-training for dense passage retrieval

    Luyu Gao and Jamie Callan. Unsupervised corpus aware language model pre-training for dense passage retrieval. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, ed- itors, Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volu...

    2022 doi

  12. [21]

    DRAMA: Diverse augmentation from large language models to smaller dense retrievers

    Xueguang Ma, Xi Victoria Lin, Barlas Oguz, Jimmy Lin, Wen-tau Yih, and Xilun Chen. DRAMA: Diverse augmentation from large language models to smaller dense retrievers. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63...

    2025

  13. [22]

    Reasonir: Training retrievers for reasoning tasks

    Rulin Shao, Rui Qiao, Varsha Kishore, Niklas Muennighoff, Xi Victoria Lin, Daniela Rus, Bryan Kian Hsiang Low, Sewon Min, Wen tau Yih, Pang Wei Koh, and Luke Zettlemoyer. Reasonir: Training retrievers for reasoning tasks. arXiv:2504.20595, 2025. URL https: //arxiv.org/abs/2504.20595

    2025 arXiv

  14. [23]

    Fine-tuning llama for multi-stage text retrieval

    Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. Fine-tuning llama for multi-stage text retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval , SIGIR ’24, page 2421–2425, New York, NY , USA, 202...

    2024

  15. [24]

    Improving text embeddings with large language models

    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. arXiv:2401.00368, 2023

    2023 arXiv

  16. [25]

    Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R. Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, Yi Luan, Sai Meher Karthik Duddu, Gus- tavo Hernandez Abrego, Weiqiang Shi, Nithi Gupta, Aditya Kusupati, Prateek Jain, Sid- dhartha Reddy Jonnalagad...

    2024 arXiv

  17. [26]

    Qwen3 embedding: Advancing text embedding and reranking through foundation models

    Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv:2506.05176, 2025. URL https://arxi...

    2025 arXiv

  18. [27]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019 doi

  19. [28]

    TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min-Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Com- putational Li...

    2017 doi

  20. [29]

    RA- DIT: Retrieval-augmented dual instruction tuning

    Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Richard James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. RA- DIT: Retrieval-augmented dual instruction tuning. In The Twelfth International Conference on Lea...

    2024

  21. [30]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors, Proc...

    2018 doi

  22. [31]

    Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction

    Adrien Barbaresi. Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction. In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natu...

    2021

  23. [32]

    Corpus subsampling: Estimating the effectiveness of neural retrieval models on large corpora

    Maik Fröbe, Andrew Parry, Harrisen Scells, Shuai Wang, Shengyao Zhuang, Guido Zuccon, Martin Potthast, and Matthias Hagen. Corpus subsampling: Estimating the effectiveness of neural retrieval models on large corpora. In Advances in Information Retrieval: 47th European Conferen...

    2025 doi

  24. [33]

    Asyncval: A toolkit for asynchronously validating dense retriever checkpoints during training

    Shengyao Zhuang and Guido Zuccon. Asyncval: A toolkit for asynchronously validating dense retriever checkpoints during training. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’22, page 3235–3239, New Y...

    2022

  25. [34]

    Arik, and Jiawei Han

    Bowen Jin, Jinsung Yoon, Priyanka Kargupta, Sercan O. Arik, and Jiawei Han. An empirical study on reinforcement learning for reasoning-search interleaved llm agents, 2025. URL https://arxiv.org/abs/2505.15117

    2025 arXiv

  26. [35]

    Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford

    Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. Okapi at TREC-2. In Donna K. Harman, editor, Proceedings of The Second Text REtrieval Conference, TREC 1993, Gaithersburg, Maryland, USA, August 31 - September 2, 1993, volume 500-21...

    1993

  27. [36]

    MTEB: Massive text embedding benchmark

    Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. MTEB: Massive text embedding benchmark. In Andreas Vlachos and Isabelle Augenstein, editors, Proceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistics, pages 20...

    2014 doi

  28. [37]

    BRIGHT: A realistic and challenging benchmark for reasoning- intensive retrieval

    Hongjin SU, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han yu Wang, Liu Haisu, Quan Shi, Zachary S Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon, Sercan O Arik, Danqi Chen, and Tao Yu. BRIGHT: A realistic and challenging benchmark for reasoning- intensive retrie...

  29. [38]

    Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations

    Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th International ACM SIGIR Conference on Res...

    2021

  30. [39]

    Tevatron 2.0: Unified document retrieval toolkit across scale, language, and modality

    Xueguang Ma, Luyu Gao, Shengyao Zhuang, Jiaqi Samantha Zhan, Jamie Callan, and Jimmy Lin. Tevatron 2.0: Unified document retrieval toolkit across scale, language, and modality. SIGIR ’25, page 4061–4065, New York, NY , USA, 2025. Association for Computing Machinery. ISBN 97984...

    2025

  31. [40]

    Wang, John-Clark Levin, Mstyslav Kazakov, Fiona Feng, Steven Y

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, Dmitry Dodonov, T...

    2025 arXiv

  32. [41]

    The fineweb datasets: Decanting the web for the finest text data at scale

    Guilherme Penedo, Hynek Kydlíˇcek, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro V on Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Processing ...

    2024

  33. [42]

    Feb4rag: Evaluating federated search in the context of retrieval augmented generation

    Shuai Wang, Ekaterina Khramtsova, Shengyao Zhuang, and Guido Zuccon. Feb4rag: Evaluating federated search in the context of retrieval augmented generation. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR...

    2024

  34. [45]

    Each subquery should focus on a single piece of information or concept

  35. [46]

    this person

    Subqueries MUST be completely self-contained and answerable independently - do not use pronouns or references like "this person", "the author", "these conditions", "they", "the movie", etc

  36. [47]

    Each subquery should include all necessary context and constraints from the original query

  37. [48]

    Preserve all important details and constraints from the original query

  38. [49]

    Subqueries: [

    Return only the subqueries as a JSON array of strings Example: Original: "Please identify the fictional character who occasionally breaks the fourth wall with the audience, has a backstory involving help from selfless ascetics, is known for his hu- mor, and had a TV show that ...

  39. [2023]

    URL https://arxiv.org/abs/2308.03281

    arXiv

  40. [2025]

    URL https://openreview.net/forum?id=ykuc5q381b

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.