Pith. sign in

REVIEW 4 major objections 5 minor 63 references

RAVine: Reality-Aligned Evaluation for Agentic Search

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RAVine claims that agentic search evaluation can be made full-process and noise-resistant by grounding report-quality scores in attributable nuggets, and it reports that today's models cover few required facts, cite poorly, and lean on…

desk verdict A genuinely useful evaluation sandbox for agentic search with released code and clearly defined process metrics, but the LLM-generated nugget ground truth needs human validation before the headline findings are trusted. read the letter →

arxiv 2507.16725 v2 pith:5VB6TD3P submitted 2025-07-22 cs.CL cs.AIcs.IR

Yilong Xu , Xiang Long , Zhi Zheng , Jinhua Gao This is my paper · ORCID
classification cs.CLcs.AIcs.IR
keywords agenticsearchretrieval-augmentedgenerationevaluationbenchmarknuggetstaskcompletenesscitationfaithfulnessprocess-orientedLLM-as-a-judge
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Agentic search systems are supposed to autonomously browse the web and write long-form answers, but the paper argues that existing benchmarks misalign with that goal: they favor obscure puzzles with short answers, extract noisy ground truth, and judge only the final report. RAVine is proposed as a full-process, reproducible evaluation sandbox that fixes all three misalignments at once. It uses realistic multi-point queries from search logs over a static web corpus, and it constructs attributable nuggets—small required facts traced back to their source pages—so that a single block-level pass over a report can score both task completeness and citation faithfulness. The paper's benchmark of current models shows they cover only a small fraction of required nuggets, cite poorly, and rely heavily on internal knowledge that cannot be attributed to any retrieved page. If the framework is right, the field gains a cheap, process-aware way to measure whether an agent actually gathered and cited external information, not just whether its final text sounds good.

What carries the argument

The load-bearing object is the nugget, an atomic factual statement that a good answer must cover, labeled vital or okay and traced to the web pages it came from. Nuggets are produced by batching relevant text segments by source document, having an LLM extract candidate facts, embedding them, clustering with HDBSCAN, and prompting the LLM to merge each cluster into a small set of nuggets attributed to every source page in the cluster. The second mechanism is block-level assignment: the report is split at citation boundaries, an LLM judge labels each nugget as fully, partially, or not supported in each block, and the matched nuggets' source pages become the gold citations against which citation recall and precision are computed. Process metrics then track the iterative loop, with marginal search gain defined as the new relevant documents each search call contributes beyond what previous calls found. The paper's headline finding—that good process does not guarantee good output—comes from correlating these search metrics with final completeness and from the Compin ratio, which tags score-bearing nuggets whose sources were never retrieved.

What would settle it

Take a random sample of the 84 test queries, have independent human annotators produce and score the nuggets by hand, and compare the resulting task-completeness and citation-recall scores with RAVine's LLM-generated ones; if the rankings of the eight evaluated models change, then the nugget ground truth—not the models—is driving the reported findings.

Watch

Extended reading notes

Core claim

RAVine's central claim is that agentic search evaluation should be a full-process, reproducible sandbox rather than a final-report-only grading exercise, and that the key to making it work is attributable ground truth. The paper constructs that ground truth by turning relevant web segments into nuggets—atomic facts labeled vital or okay—and keeping each nugget linked to the pages it came from. It then argues that splitting a model's long-form report into citation-bounded blocks lets a single LLM judge compute task completeness and faithfulness together, because the pages behind any supported nugget are exactly the citations the model should have given. Benchmarking eight model configurations, the paper reports that current systems cover few required nuggets, cite poorly (best citation recall in the main 32k dense-index table is 14.9%), and earn a large share of their completeness credit from internal knowledge rather than retrieved sources; it also reports that intermediate search performance correlates only weakly with final report quality.

Load-bearing premise

The load-bearing premise is that the automatically extracted, unverified nuggets are complete and accurate enough to serve as ground truth, so any noise or bias in the LLM that creates them would propagate into every completeness, citation, and internal-knowledge finding.

Editorial extensions

If this is right

  • A single nugget-assessment pass over the final report yields both task completeness and citation quality, so fine-grained evaluation becomes cheaper than running separate completeness and faithfulness judges.
  • Because every nugget is traced to its source pages, evaluators can measure Compin—the share of task-completeness credit earned without retrieving the supporting pages—and current models are shown to rely on internal knowledge for a substantial share of that credit.
  • Process metrics such as search gain, search recall and precision, and fetch precision separate 'searched well' from 'answered well'; the data show strong search does not guarantee a strong report.
  • The benchmark is reproducible across dense and BM25 indexes, and the comparative results across the evaluated model configurations offer a reusable snapshot of current agentic search capabilities.
  • Evaluations using realistic multi-point queries and long-form answers reveal low citation recall that report-only RAG evaluations typically miss.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Going beyond the paper: Compin could be converted from a diagnostic into a training objective—penalizing reports for scoring points without having retrieved the supporting pages—since the paper shows the behavior is measurable but does not propose using it as a loss.
  • Going beyond the paper: the index-sensitivity results imply that any agentic search ranking produced inside one retrieval sandbox may be backend-dependent; a fair comparison across labs would report scores under at least one dense and one lexical index, as the paper's robustness check begins to do.
  • Going beyond the paper: the block-level citation cap of three gold sources per nugget could be tested against user preferences, because readers in some domains expect exhaustive citation lists rather than a strongest-sources rule.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces RAVine, a sandbox and evaluation framework for agentic LLMs with search, combining a static web corpus (MS MARCO V2.1), a test set derived from TREC 2024 RAG Track queries, an attributable nugget collection method, block-level report quality metrics (task completeness, citation recall/precision), and process-oriented metrics (search recall/gain, fetch precision, efficiency). The authors benchmark several Qwen and LLaMA models under two context lengths and two retrieval indexes, and report findings about limited task completeness and faithfulness, weak correlation between process and final performance, and a tendency to rely on internal knowledge (Compin). The code and datasets are released.

Significance. If the nugget-based ground truth is reliable, RAVine offers a useful reusable infrastructure: the static corpus, tool-calling sandbox, and process metrics address real gaps in agentic search evaluation. The paper is particularly strong in releasing code and data, in separating process from end-to-end metrics, and in proposing attributable nuggets that link claims to source pages, which is a sensible direction for reducing evaluation noise and cost. The main findings, however, are entirely conditional on the accuracy, completeness, and unbiasedness of the LLM-generated nuggets and on the LLM-judge support labels; those assumptions are not yet validated, which limits confidence in the quantitative conclusions.

major comments (4)
  1. [Section 3.1 and Appendix A.1] The load-bearing premise of the evaluation is that the nuggets extracted by gemini-2.5-flash (and the vital/okay labels and support labels) constitute accurate, complete, and unbiased gold. The paper reports no human validation of the nuggets or labels, and the post hoc exclusion of two queries after the LLM failed to extract nuggets (Appendix A.1) demonstrates that the extraction pipeline can fail at the query level. Since every headline metric (task completeness, citation recall/precision, search recall, search gain, and Compin) is computed against these nuggets, systematic extraction errors could make the reported findings artifacts. I recommend adding a human validation study on a sample of queries (e.g., 10-20 queries) measuring nugget precision, recall, and vital/okay agreement, and reporting the stability of the main scores under variations in the extraction model or prompt.
  2. [Appendix A.2] The nugget cap is not an independent design choice: it is set to the 95th percentile of the vital-nugget distribution produced by the same LLM extractor. This introduces a mild circularity, because the completion denominator is derived from the extraction output it is meant to evaluate. The paper should report sensitivity of the main results to the cap (e.g., cap values of 40, 60, 80, and no cap where feasible), and should state how many queries are actually affected by the cap. Without this, the reader cannot tell whether the reported completeness scores are an artifact of truncating the extractor's own distribution.
  3. [Section 5.3, Equations (12)-(13)] The Compin metric conflates 'not retrieved' with 'internal knowledge'. A nugget is counted as internal-knowledge-supported whenever its gold source pages (capped at three per nugget, per Appendix A.4) are not retrieved, even if the model retrieved a different web page that contains the same fact or if the gold cap simply excluded a relevant page. This could overstate the 'reliance on internal knowledge' finding. The paper should validate Compin on a small set of runs with human-annotated attribution (e.g., comparing output claims against retrieved non-gold pages), and should clarify what fraction of Compin hits have any retrieved page (gold or not) supporting the claim.
  4. [Section 5.3 and Tables 2-3] The claim that 'strong performance during the search process does not necessarily lead to high-quality final answers' is based on correlations computed within a relatively small number of runs (84 queries per model-configuration). The p-values for the overall correlations are marginal (e.g., r=0.22, p=0.064 and r=0.19, p=0.0993 in Figure 4), and the 'search-based runs' correlation is computed on a subset that is not characterized by size or composition. The paper should report the number of runs in each subset and the confidence intervals for the correlations, and should avoid presenting the 'illusion of progress' as a firm finding.
minor comments (5)
  1. [Title and Section 2] The framework name is written inconsistently as 'RA Vine' and 'RAVine'; please standardize the spacing.
  2. [Appendix B.2, Table 8] Table 8 is captioned 'Search Gain and Fetch Gain' but the reported metrics are 'Avg. Search Prec.' and 'Avg. Fetch Prec.'; the caption and the table body do not match. Please correct the caption or the columns.
  3. [Appendix C, Tables 9 and 10] The case study text contains literal 'textbackslash n' artifacts and the table formatting is hard to read; please clean these up.
  4. [Section 4.1] The definition of 'search gain' in Equation (8) uses D_seen_<t, but the symbol is not explicitly defined in the main text; please define it as the union of previous search results.
  5. [Section 2.3] The sentence 'Due to identified quality issues in the qrels of two instances... we exclude these two queries' should specify whether the exclusion decision was made before or after computing any of the reported aggregate results, and whether any excluded query appears in the case studies.

Circularity Check

1 steps flagged · score 5.0 of 10

Compin's 'reliance on internal knowledge' and the 'illusion of progress' correlation are definitional restatements of the zero-relevant-retrieval partition; the benchmark and attributable-nugget evaluation are otherwise self-contained.

  1. self definitional [Section 5.3 (Attribution is impacted by reliance on internal knowledge) and Appendix A.4, Eqs. (12)-(13)]
    ""To investigate this behavior, we calculate the proportion of task completeness scores that relies on internal knowledge, and denote this metric as Compin. Specifically, if a model fails to retrieve any web pages that hit the qrels during the iterative process, we consider the final score of this run to be based on internal knowledge, and this score contributes to Compin.""

    Compin is stipulated, not measured: Eq. (12) sets sigma_ij=1 exactly when no source page for the nugget was retrieved and the block satisfies the nugget. The headline finding that 'a significant portion of task performance across models can be traceable to internal knowledge' restates the defining condition of Compin, so it holds by construction rather than by evidence about the model's knowledge source. The paper itself concedes in Section 4.1 that useful information may already exist in titles or headings returned by search, so 'no qrel page retrieved' does not entail internal knowledge. The companion 'illusion of progress' regression uses the same definitional partition, making the near-zero correlation in the internal group an artifact of the grouping rule.

full rationale

The central RAVine contribution — the static MS MARCO V2.1 sandbox, the TREC 2024 RAG test split, attributable segment-level nugget collection, block-level completeness and citation metrics, and process-oriented efficiency and tool metrics — does not reduce to its inputs. Nuggets are extracted from external qrel segments and scored by an external LLM; no parameter is fitted to the evaluated models, and no benchmark score is a renamed training target. The paper's self-citations [35,36] support only peripheral related-work claims and are not load-bearing. The post hoc removal of two queries and the absence of human validation of the gemini-2.5-flash nuggets are validity limitations, not circularity. The one genuine reduction-by-definition is the Compin-based 'internal knowledge reliance' and 'illusion of progress' analysis, where the construct is defined as the absence of relevant retrieval and then reported as a discovered behavior. Because that insight is one of the paper's three headline findings, partial circularity is present; however, the benchmark infrastructure remains independently usable and empirically grounded.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The framework rests on a small number of domain assumptions and a few fitted caps. The most important is that LLM-generated nuggets faithfully represent the information needed for each query; without human validation, this assumption is fragile. The caps (60 nuggets, 3 gold citations) are data-derived and shape scores, but are disclosed.

free parameters (6)
  • Nugget cap per query = 60
    Set at the 95th percentile of vital nuggets (56.15 rounded up) in A.2 based on the authors' own extraction distribution; affects which nuggets are scored and thus task completeness.
  • Gold citation cap per nugget = 3
    A.4: 'a hard upper limit in the calculation of citation recall: the number of gold citations per nugget is capped at a maximum of three.' Shapes citation recall and precision.
  • Top-3 gold citations for weighted recall = 3
    A.4: 'we normalize the total weight by considering only the top three gold citations with the highest weights.' Affects whether a model can achieve full citation recall.
  • Relevant pages per nugget cap = 3
    A.4: 'we impose the cap of three relevant web pages per nugget' in search recall and gain calculations; prevents dilution.
  • Context length limits = 32k and 128k
    Section 5.1: evaluation is divided into two groups by max context length; affects task completion rate and tool call counts.
  • HDBSCAN hyperparameters = not reported
    A.2 uses HDBSCAN for nugget clustering but does not state min_cluster_size or other parameters, making the clustering step underspecified.
assumptions (5)
  • domain assumption TREC 2024 RAG Track queries are representative of real user search behavior
    Section 2.3: queries come from Bing logs and were manually curated; used as the test split.
  • domain assumption LLM-generated nuggets are accurate ground truth
    Section 3.1: gemini-2.5-flash extracts nuggets with no human verification; the two excluded queries show failure modes.
  • domain assumption MS MARCO V2.1 approximates a real-world web corpus
    Section 2.1: used as a static web environment; assumes relevance and content distribution are realistic enough for evaluation.
  • domain assumption Block-level splitting at citation boundaries yields semantically complete statements
    Section 3.2: the method assumes blocks are better evaluation units than sentences; no validation is provided.
  • standard math Standard set-theoretic definitions of recall and precision
    Equations 5-11 apply standard set operations; no nonstandard math is used.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RAVine: Reality-Aligned Evaluation for Agentic Search." pith.science (2026). https://pith.science/paper/5VB6TD3P

@misc{pith2026250716725,
  author       = {Pith},
  title        = {Pith review of: RAVine: Reality-Aligned Evaluation for Agentic Search},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5VB6TD3P}},
  note         = {Machine review of arXiv:2507.16725}
}
read the original abstract

Agentic search, as a more autonomous and adaptive paradigm of retrieval augmentation, is driving the evolution of intelligent search systems. However, existing evaluation frameworks fail to align well with the goals of agentic search. First, the complex queries commonly used in current benchmarks often deviate from realistic user search scenarios. Second, prior approaches tend to introduce noise when extracting ground truth for end-to-end evaluations, leading to distorted assessments at a fine-grained level. Third, most current frameworks focus solely on the quality of final answers, neglecting the evaluation of the iterative process inherent to agentic search. To address these limitations, we propose RAVine -- a Reality-Aligned eValuation framework for agentic LLMs with search. RAVine targets multi-point queries and long-form answers that better reflect user intents, and introduces an attributable ground truth construction strategy to enhance the accuracy of fine-grained evaluation. Moreover, RAVine examines model's interaction with search tools throughout the iterative process, and accounts for factors of efficiency. We benchmark a series of models using RAVine and derive several insights, which we hope will contribute to advancing the development of agentic search systems. The code and datasets are available at https://github.com/SwordFaith/RAVine.

Figures

Figures reproduced from arXiv: 2507.16725 by the authors.

Figure 1
Figure 1. Overview of the three primary misalignments addressed by our work. From left to right: (1) The divergence between [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The architecture of the agentic LLM with search. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of the evaluation framework of RAVine. Left (§3.1): Attributable nugget collection via batched extraction at [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Correlation analysis between Task Completeness and Search Precision. Blue and orange points represent runs based on [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Statistics of the proportion of task completeness [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Fitted distribution curves of the number of relevant [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Comparison of search gain scores across models [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Comparison of fetch gain scores across models un [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 21 canonical work pages

  1. [1]

    Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. arXiv:1611.09268 [cs.CL] https://arxiv.org/abs/1611.09268

    arXiv 2018

  2. [2]

    Rae, Erich Elsen, and Laurent Sifre

    Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Ruther- ford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, O...

    arXiv 2022

  3. [3]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    arXiv 2020

  4. [4]

    Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020. HybridQA: A Dataset of Multi-Hop Question Answer- ing over Tabular and Textual Data. InFindings of the Association for Computational Linguistics: EMNLP 2020, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 1026–1036. doi...

    doi:10.18653/v1/2020.findings- 2020

  5. [5]

    João Coelho, Jingjie Ning, Jingyuan He, Kangrui Mao, Abhijay Paladugu, Pranav Setlur, Jiahe Jin, Jamie Callan, João Magalhães, Bruno Martins, and Chenyan Xiong. 2025. DeepResearchGym: A Free, Transparent, and Reproducible Evalua- tion Sandbox for Deep Research. arXiv:2505.19253 [cs.IR] https://arxiv.org/abs/ 2505.19253

    arXiv 2025

  6. [6]

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Jimmy Lin

  7. [7]

    Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024. The Faiss library. (2024). arXiv:2401.08281 [cs.LG]

    arXiv 2024

  8. [8]

    Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long Form Question Answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , Anna Korhonen, David Traum, and Lluís Màrquez (Eds.). Association for Computational Linguistics, Florence, Italy, 3558–3567. doi:10.1...

    doi:10.18653/v1/p19-1346 2019

Show all 63 references
  1. [9]

    Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling Large Language Models to Generate Text with Citations. arXiv:2305.14627 [cs.CL] https://arxiv.org/abs/2305.14627

    2023 arXiv

  2. [10]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv:2312.10997 [cs.CL] https://arxiv.org/abs/2312.10997

    2024 arXiv

  3. [11]

    Gemini. 2025. gemini-2.5-flash-preview. https://ai.google.dev/gemini-api/docs/ models?hl=zh-cn#gemini-2.5-flash-preview, Accessed on 2025-06-20

    2025

  4. [12]

    Google. 2025. Gemini Deep Research. https://gemini.google/overview/deep- research/. Accessed: 2025-05-08

    2025

  5. [13]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, A...

    2018 arXiv

  6. [14]

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reason- ing Steps. arXiv:2011.01060 [cs.CL] https://arxiv.org/abs/2011.01060

    2020 arXiv

  7. [15]

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. arXiv:2503.09516 [cs.CL] https://arxiv.org/abs/2503.09516

    2025 arXiv

  8. [16]

    Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. Hurdles to Progress in Long-form Question Answering. arXiv:2103.06332 [cs.CL] https://arxiv.org/abs/ 2103.06332

    2021 arXiv

  9. [17]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.1140...

    2021 arXiv

  10. [18]

    Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. 2025. Search-o1: Agentic Search-Enhanced Large Reasoning Models. arXiv:2501.05366 [cs.AI] https://arxiv.org/abs/2501.05366

    2025 arXiv

  11. [19]

    Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji- Rong Wen, and Zhicheng Dou. 2025. WebThinker: Empowering Large Reasoning Models with Deep Research Capability. arXiv:2504.21776 [cs.CL] https://arxiv. org/abs/2504.21776

    2025 arXiv

  12. [20]

    Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: A Python Toolkit for Reproducible Informa- tion Retrieval Research with Sparse and Dense Representations. In Proceedings of the 44th Annual International ACM SIGIR Co...

    2021

  13. [21]

    Nelson Liu, Tianyi Zhang, and Percy Liang. 2023. Evaluating Verifiability in Generative Search Engines. In Findings of the Association for Computational Linguistics: EMNLP 2023 , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singa...

    2023

  14. [22]

    Claudia Malzer and Marcus Baum. 2020. A Hybrid Approach To Hierarchi- cal Density-based Cluster Selection. In 2020 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI) . IEEE, 223–228. doi:10.1109/mfi49285.2020.9235263

    2020

  15. [23]

    OpenAI. 2025. Deep Research System Card . Technical Report. OpenAI. https: //cdn.openai.com/deep-research-system-card.pdf

    2025

  16. [24]

    Ronak Pradeep, Nandan Thakur, Sahel Sharifymoghaddam, Eric Zhang, Ryan Nguyen, Daniel Campos, Nick Craswell, and Jimmy Lin. 2024. Ragnarök: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Track. arXiv:2406.16828 [cs.IR] https://arxiv.org/abs...

    2024 arXiv

  17. [25]

    Ronak Pradeep, Nandan Thakur, Shivani Upadhyay, Daniel Campos, Nick Craswell, and Jimmy Lin. 2024. Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework. arXiv:2411.09607 [cs.IR] https://arxiv.org/abs/2411.09607

    2024 arXiv

  18. [26]

    Ronak Pradeep, Nandan Thakur, Shivani Upadhyay, Daniel Campos, Nick Craswell, and Jimmy Lin. 2025. The Great Nugget Recall: Automating Fact Extrac- tion and RAG Evaluation with Large Language Models. arXiv:2504.15068 [cs.IR] https://arxiv.org/abs/2504.15068

    2025 arXiv

  19. [27]

    Zehan Qi, Rongwu Xu, Zhijiang Guo, Cunxiang Wang, Hao Zhang, and Wei Xu. 2024. 𝐿𝑂𝑁𝐺 2𝑅𝐴𝐺: Evaluating Long-Context & Long-Form Retrieval- Augmented Generation with Key Point Recall. In Findings of the Association for Computational Linguistics: EMNLP 2024 , Yaser Al-Onaizan, Moh...

    2024 doi

  20. [28]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025 arXiv

  21. [29]

    Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Frame- work: BM25 and Beyond. Found. Trends Inf. Retr. 3, 4 (April 2009), 333–389. doi:10.1561/1500000019

    2009 doi

  22. [30]

    Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. 2023. REPLUG: Retrieval-Augmented Black- Box Language Models. arXiv:2301.12652 [cs.CL] https://arxiv.org/abs/2301.12652

    2023 arXiv

  23. [31]

    Aditi Singh, Abul Ehtesham, Saket Kumar, and Tala Talaei Khoei. 2025. Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG. arXiv:2501.09136 [cs.AI] https://arxiv.org/abs/2501.09136

    2025 arXiv

  24. [32]

    Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. 2025. R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning. arXiv:2503.05592 [cs.AI] https: //arxiv.org/abs/2503.05592

    2025 arXiv

  25. [33]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal

  26. [34]

    Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese

  27. [35]

    Yilong Xu, Jinhua Gao, Xiaoming Yu, Baolong Bi, Huawei Shen, and Xueqi Cheng

  28. [36]

    Yilong Xu, Jinhua Gao, Xiaoming Yu, Yuanhai Xue, Baolong Bi, Huawei Shen, and Xueqi Cheng. 2025. Training a Utility-based Retriever Through Shared Context Attribution for Retrieval-Augmented Language Models. arXiv preprint arXiv:2504.00573 (2025)

    2025

  29. [37]

    Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. 2025. An Illusion of Progress? Assessing the Current State of Web Agents. arXiv:2504.01382 [cs.AI] https://arxiv.org/abs/2504.01382

    2025

  30. [38]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025 arXiv

  31. [39]

    Cohen, Ruslan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. arXiv:1809.09600 [cs.CL] https://arxiv.org/abs/1809.09600

    2018 arXiv

  32. [40]

    Zichun Yu, Chenyan Xiong, Shi Yu, and Zhiyuan Liu. 2023. Augmentation- Adapted Retriever Improves Generalization of Language Models as Generic Plug- In. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Anna Rog...

    2023 doi

  33. [41]

    Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, et al. 2024. mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval. In Proceedings of the 2024 Conference ...

    2024

  34. [42]

    Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. 2023. Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models. arXiv:2309...

    2023 arXiv

  35. [43]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong W...

    2025 arXiv

  36. [44]

    a", "b", ...] and a and b are strings with no mention of

    Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pen- grui Lu, and Pengfei Liu. 2025. DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments. arXiv:2504.03160 [cs.AI] https://arxiv.org/abs/2504.03160 RAVine: Reality-Aligne...

    2025 arXiv

  37. [49]

    You primarily respond in English

  38. [50]

    You can choose to call known tools and generate the correct parameters according to the tool description

  39. [51]

    You can generate any content that helps you complete the task during the intermediate iteration process according to your needs

  40. [52]

    When you consider the task complete, the last generated content is a long-form report that covers much useful information for the given question

  41. [53]

    You need to meet the following requirements for your final long-form report:

    In each iteration, you get to choose what to do next (call the search tool or complete the task and generate a final report), and you do not require assistance or response from users. You need to meet the following requirements for your final long-form report:

  42. [54]

    Your long-form report needs to be in markdown format

  43. [55]

    Your long-form report needs to be logically clear, comprehensive in key points, and able to effectively address the given question

  44. [56]

    Your long-form report needs to include citations of the websites retrieved through external search tools

  45. [57]

    The citations in your final long-form report need to meet the following requirements:

    In the final output, your report must be enclosed within <report> and </report>, that is, only the content between these tags will be evaluated. The citations in your final long-form report need to meet the following requirements:

  46. [58]

    Citations can only appear at the end of a sentence

  47. [59]

    For example: ([title](url))

    Citations must follow the Markdown format, including the website’s title and URL, and should be enclosed in brackets. For example: ([title](url))

  48. [60]

    For example: ([title1](url1); [title2](url2); [title3](url3))

    Multiple citations can appear at the same time in one position, separated by semicolons. For example: ([title1](url1); [title2](url2); [title3](url3))

  49. [61]

    Please try to generate citations after the entire statement is presented

    A complete statement may contain one or more sentences. Please try to generate citations after the entire statement is presented

  50. [62]

    Question: {question} A.4 Details of Metrics The implementation details of several metrics defined in RAVine are as follows: Task Completion Rate

    Do not list the cited websites at the end of the report to avoid unnecessary token usage. Question: {question} A.4 Details of Metrics The implementation details of several metrics defined in RAVine are as follows: Task Completion Rate. We require the agentic model to complete ...

    2018

  51. [63]

    death with dignity,

    to denote search queries. However, this approach can inter- fere with the model’s native generation capabilities and hinder its adaptability to other tasks or domains. To address this, we adopt the standard tool calling schema provided by OpenAI4, where search calls are encaps...

    2018

  52. [2021]

    Overview of the TREC 2021 Deep Learning Track. In TREC. https://trec. nist.gov/pubs/trec30/papers/Overview-DL.pdf

    2021

  53. [2022]

    arXiv:2108.00573 [cs.CL] https://arxiv.org/abs/2108.00573

    MuSiQue: Multihop Questions via Single-hop Question Composition. arXiv:2108.00573 [cs.CL] https://arxiv.org/abs/2108.00573

    arXiv

  54. [2024]

    Aliice: Evaluating positional fine-grained citation generation.arXiv preprint arXiv:2406.13375 (2024)

    2024

  55. [2025]

    arXiv:2504.12516 [cs.CL] https://arxiv.org/abs/2504.12516

    BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents. arXiv:2504.12516 [cs.CL] https://arxiv.org/abs/2504.12516

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.