Pith. sign in

REVIEW 6 major objections 7 minor 4 cited by

InfoDeepSeek: Benchmarking Agentic Information Seeking for Retrieval-Augmented Generation

T0 review · 6 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces InfoDeepSeek, a 245-question live-web benchmark for agentic information seeking, and reports that the strongest tested agent, Gemini-2.5-Pro, reaches only 22.45% accuracy while search engine quality outweighs model…

desk verdict InfoDeepSeek delivers a genuinely useful benchmark and metrics suite for agentic RAG, but the headline information-quality finding is only partly supported because the metrics don't isolate evidence quality from parametric memory. read the letter →

arxiv 2505.15872 v2 pith:NA2A4LRD submitted 2025-05-21 cs.IR cs.CL

Yunjia Xi , Jianghao Lin , Menghui Zhu , Yongzhao Xiao , Zhuoying Ou , Jiaqi Liu , Tong Wan , Bo Chen
show 5 more authors
Weiwen Liu Yasheng Wang Ruiming Tang Weinan Zhang Yong Yu
This is my paper
classification cs.IRcs.CL
keywords agenticRAGinformationseekinglive-webbenchmarkretrieval-augmentedgenerationLLMevaluationdynamicretrievalinterferencemultilingualquestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that existing RAG benchmarks cannot measure what agentic RAG systems actually do—multi-turn planning, searching, and evidence gathering on the live, changing web—because they are built on static corpora with fixed gold documents and simple queries. To fill that gap, it presents InfoDeepSeek, a set of 245 manually constructed, triple-validated questions that are determinate, hard for LLMs even with one search, and diverse across six difficulty attributes, 14 domains, and 19 languages. It also proposes an evaluation protocol for dynamic settings that needs no pre-defined gold documents, with four metrics: answer accuracy, information accuracy at top-k, effective evidence utilization, and information compactness. The central empirical claim is that even the strongest LLM agents answer only about a fifth of these queries correctly, and that the quality of the underlying search engine matters more than the choice of reasoning model.

What carries the argument

The load-bearing machinery is the agentic information-seeking loop itself—plan, act through search, browse, time, or termination tools, observe, reflect, and continue up to T steps—followed by an augmentation stage that ranks and condenses retrieved content into an evidence set. Around this loop the paper builds the evaluation scaffold, and because a dynamic web has no gold document set, the metrics judge outcomes instead. The central metric decomposition is ACC (whether the final answer matches ground truth), IA@$k$ (whether the top-$k$ ranked evidence alone yields the correct answer), $\mathrm{EEU} = \max_{1\le k\le n} \mathrm{IA}@k / \mathrm{ACC}$ (whether the agent selected evidence worse than the best available subset), and IC (evidence-set length relative to a human-annotated source set, penalized when the answer fails). This decomposition is what lets the paper attribute failures to retrieval, evidence selection, or generation rather than treating agentic RAG as a single black box.

What would settle it

Have two independent human raters score the agent outputs for all 245 questions using their own instructions rather than the paper's evaluation prompts, then compare each judgment to the LLM-judge majority verdict; if agreement on the false-premise subset falls well below the reported 99.29%, or if the judge systematically accepts answers that only say 'the premise is wrong' without naming the correct error, then ACC, IA@k, EEU, and IC would shift and the benchmark's headline numbers would need revision.

Watch

Extended reading notes

Core claim

InfoDeepSeek is presented as the first benchmark and evaluation framework for agentic information seeking in real-world, dynamic web environments. The paper's central claim is that evaluating agents on open-ended web search requires replacing fixed gold document sets with outcome-based metrics that ask whether the retrieved evidence, the selected evidence set, and the final answer are sufficient and compact. On this yardstick the reported results are stark: Gemini-2.5-Pro, the best model tested, scores 22.45% on ACC and 21.63% on IA@5, most models score below 15%, and most effective-evidence-utilization values sit at or below 1, meaning agents commonly fail to extract the useful evidence they already retrieved. The paper also reports that search engine choice changes accuracy more than model choice—DeepSeek-V3 with Google reaches 28.57% versus 8.98% with DuckDuckGo—and that retrieval interference, where web content makes a model abandon a correct answer it knew from parametric memory, occurs at rates from roughly 40% to 100% depending on the model.

Load-bearing premise

The load-bearing premise is that the automatic judge—two large language models scoring answers plus a third model as tiebreaker—correctly decides when an agent's answer matches the ground truth, especially on the 27 questions built on false premises, and the evidence for that correctness is the authors' own annotation set.

Editorial extensions

If this is right

  • If the benchmark's difficulty filter does its job, any system that scores well must engage in multi-turn retrieval and evidence selection; parametric memory or a single search will not suffice.
  • Search engine choice becomes a first-order experimental variable: switching from DuckDuckGo to Google lifts DeepSeek-V3's accuracy from 8.98% to 28.57%, a larger gain than swapping between many LLM choices.
  • Allocating more retrieval steps—from 1 up to 20—roughly doubles or triples accuracy for the tested models, establishing test-time compute for search as a real scaling axis.
  • Retrieval interference is widespread, so robust agents will need mechanisms that protect correct internal knowledge from being overridden by noisy or tangential web content.
  • Language-aware prompting to search in a query's predominant language improves accuracy over English-only or Chinese-only search, particularly for multilingual long-tail questions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A consequence the authors leave implicit: because the difficulty filter is tied to GPT-4o and DeepSeek-R1 at one point in time, some InfoDeepSeek questions will become single-turn answerable as those models and search indexes improve, so keeping the benchmark hard will require versioned re-filtering of the question set.
  • The EEU and IC metrics are generic enough to apply to any evidence-producing agent, including non-RAG tool use and human search sessions, as diagnostics of whether the selected evidence is better than the best available subset and whether it is compact.
  • A design the paper does not test is multi-engine fusion: since search engine quality dominates and different engines excel on different question attributes, an agent that queries several engines and reranks the union should outperform any single-engine configuration.
  • A second untested design follows from the interference numbers: a confidence-gated agent that answers from memory first and searches only when uncertain should reduce retrieval interference and outperform the current always-search pipeline on the questions the model initially knows.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 7 minor

Summary. The paper introduces InfoDeepSeek, a 245-question benchmark for agentic information seeking in live web environments. Questions are constructed to satisfy determinacy, difficulty, and diversity, with a difficulty filter that removes questions solvable by GPT-4o or DeepSeek-R1 in a single-turn search. The authors propose an Agentic RAG framework and four metrics: answer accuracy (ACC), information accuracy (IA@k), effective evidence utilization (EEU), and information compactness (IC), together with an LLM-judge protocol that is validated against human annotations (99.29% agreement). Experiments across ten LLM configurations, four search engines, and several ablations report low absolute accuracy (best ACC 22.45% for Gemini-2.5-Pro), large gains when switching from DuckDuckGo to Google/Yahoo, attribute-level differences, test-time scaling benefits, retrieval interference, and language effects.

Significance. If the results hold, the benchmark is a useful community resource for evaluating agentic RAG in dynamic settings. The construction pipeline is careful: fact-grounded reverse drafting, anchor-knowledge expansion, two-verifier plus adjudicator validation, and an explicit difficulty filter. The auto-eval protocol is validated against human judgments, with a clear improvement from 95.57% to 99.29% after the separate false-premise prompt strategy. The detailed appendices and project page make the protocol largely reproducible. The most robust empirical findings are likely that current agents score low on these deliberately hard questions, that search engine choice has a large effect (Table 4), and that retrieval interference is prevalent. The claims that need qualification are the interpretation of IA@k as pure evidence quality and the attribution of the language effect, as detailed in the major comments.

major comments (6)
  1. [Section 5.1 (IA@k) and Section 6.3 (Figure 5a)] IA@k is defined as the accuracy of ϕ(q, C1:k), where ϕ is the same LLM used for retrieval, augmentation, and generation. Because ϕ has parametric knowledge, IA@k measures whether the top-k evidence plus the model's memory is sufficient, not the quality of the evidence in isolation. This is not a hypothetical concern: Section 6.3 reports interference rates of 40–80%, meaning a substantial fraction of benchmark items are answerable without retrieval. Thus the low IA@k values in Table 3 cannot be cleanly interpreted as 'information quality remains low', and EEU and IC inherit the same contamination since they are derived from IA@k. Please report the no-retrieval ACC for every model in Table 3, and report IA@k restricted to questions that are not answerable from memory (or otherwise condition on no-retrieval correctness), so that retrieval quality and parametric knowledge can be separated.
  2. [Section 5.1 (IC definition)] The paper motivates its dynamic evaluation by saying that a 'gold evidence set in advance' is infeasible, yet IC = (nq or n+b)/|Sq| uses Sq, a human-annotated standard set of source webpages, as a gold reference. This is a static gold set and is in tension with the paper's central claim of evaluating information seeking without pre-defined gold documents. Please either justify the use of Sq for compactness (for example, as a stable answer-support set rather than a retrieval ranking) or redefine IC without it; otherwise the claim of a 'dynamic evaluation w/o golden doc set' in Figure 1 is not fully accurate.
  3. [Table 7 / Figure 3] Attribute-level analyses rest on small subsets (False Premise n=27, Freshness n=48) but no confidence intervals or significance tests are reported. For example, in Table 7 the false-premise accuracy difference between Llama-4-Maverick (20.00%) and Qwen3-32B w/o think (7.69%) is based on 27 questions and is within the range of sampling noise. Add bootstrap confidence intervals (or exact binomial intervals) for all attribute-level and search-engine comparisons, and temper conclusions that rely on small denominators.
  4. [Section 5.1 (EEU)] EEU is defined as max_k IA@k / ACC and is described as indicating suboptimal evidence selection when below 1. However, several models in Table 3 have EEU clearly above 1 (DeepSeek-R1 1.189, Gemini-2.5-Flash 1.171), and the paper does not explain what EEU > 1 means. Since ACC uses all retrieved observations O while IA@k uses the curated top-k set, an EEU above 1 likely reflects beneficial filtering of noisy observations; the current one-sided interpretation ('EEU significantly below 1 suggests... suboptimal') is incomplete and should be revised, with explicit discussion of the >1 cases.
  5. [Section 6.2, Figure 5(b), Appendix D.5] The predominant-language condition uses a different prompt that explicitly instructs the agent to switch search languages, whereas the Chinese and English conditions use a prompt that does not contain this instruction. Consequently, the observed superiority of 'Predominant Language' in Table 15 may be caused by the extra meta-instruction rather than by the language of retrieval. Please add an ablation in which the language-switching instruction is included in the English and Chinese conditions, or otherwise equate prompt content across conditions.
  6. [Section 4.2 (Difficulty filtering)] The difficulty filter discards any question that GPT-4o and DeepSeek-R1 can answer with a single-turn search, so the low absolute ACC values in Table 3 are partly a consequence of the benchmark construction and should not be presented as a surprising empirical discovery about LLM information-seeking ability in general. I do not regard this as circular, because multi-turn agents are still free to succeed, but the paper should report single-turn-search baselines for the evaluated models and frame the ACC numbers as 'performance on questions that are deliberately hard for single-turn search'. This would make the comparison transparent and address the concern that the headline accuracy is an artifact of filtering.
minor comments (7)
  1. [Table 1] The legend states 'Symbol ✗ signifies the lack of this attribute, while symbol ✗ means it is not explicitly considered', but both symbols are rendered identically; one of them should be a different glyph (for example, '—').
  2. [Section 5.1] The text says 'A higher RA@k implies better evidence relevance'; this should read 'IA@k'.
  3. [Section 6.2] The word 'infromation' appears in the sentence 'This highlights the importance of search engine quality in supporting effective agentic infromation seeking'; it should be 'information'.
  4. [Appendix C.2] The auto-eval agreement of 99.29% is reported overall; since the separate-prompt strategy was specifically designed for false-premise questions and only 27 such questions exist, please report agreement separately for false-premise and non-false-premise questions, with confidence intervals and the human inter-annotator agreement on that sample.
  5. [Section 6.3 / Tables 11–14] Interference-rate denominators are not reported. For example, Qwen3-32B w/o think is listed at 100.00%, which could correspond to a very small number of no-retrieval-correct questions; include the counts and exact binomial intervals for all interference rates.
  6. [Figure 5(a)] The figure legend labels one series 'DeepSeek-V1', while the text and the rest of the paper refer to DeepSeek-V3; the label should be corrected.
  7. [General reproducibility] Because the benchmark runs against live web search engines, results will drift over time; please add query timestamps, search-engine version/date information, and an archival release of agent trajectories to support reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark's difficulty filtering is disclosed selection, not a fitted prediction; IA@k's generator-dependence is a validity caveat, not a circular reduction.

full rationale

InfoDeepSeek is an empirical benchmark paper, not a derivation. The closest candidate for circularity is the difficulty filter (Sections 4.1 and 4.2): queries are discarded if GPT-4o and DeepSeek-R1 answer correctly with a single-turn search, and Table 3 later reports low ACC for these two models. This is a transparent selection criterion, not a fitted parameter: the evaluated protocol allows multi-turn retrieval, so low multi-turn ACC is not logically forced by the filter, and the top model in Table 3, Gemini-2.5-Pro, was not used in the filter at all. The paper discloses the criterion explicitly and does not present the resulting difficulty as an independent empirical discovery. The IA@k metric is defined as I(phi(q, C_{1:k}) = y_q) with phi being the same LLM used for generation, so it measures whether top-k evidence plus the model's parametric knowledge is sufficient, rather than evidence quality in isolation; this is a construct-validity limitation, especially given the retrieval-interference evidence in Section 6.3 and Appendix D.4, but it is not a circular derivation because no quantity is fitted to another and no claimed result is equivalent to its inputs by construction. Self-citations in related work, such as reference [61], are contextual and not load-bearing for the benchmark's methodology or metrics. No circular step meets the required evidence threshold.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The benchmark's metrics rest on several hand-chosen hyperparameters (n, T, b) and on the reliability of LLM judges and the difficulty-filter models. These are not fitted in a physical sense, but the headline numbers (e.g., ACC 22%) depend on them. No new physical or conceptual entities are postulated beyond metric definitions.

free parameters (5)
  • max_evidence_len_n = 5
    The maximum number of evidence items in the augmentation stage, set to 5 following prior work; affects IA@k, EEU, and IC values.
  • max_retrieval_steps_T = 5 default; up to 20 in scaling experiments
    The step limit for the information seeking loop; results vary markedly with T (Section 6.3), so the headline numbers depend on this hyperparameter.
  • IC_penalty_b = 1
    Penalty constant added to n when the agent fails to answer; chosen by hand and directly scales IC.
  • evaluator_LLMs = DeepSeek-V3, Gemini-2.0-Flash, GPT-4o-mini (tiebreak)
    The judge models are choices that determine ACC and IA@k; the paper validates them against human judgments but the metric values are contingent on these judges.
  • difficulty_filter_LLMs = GPT-4o, DeepSeek-R1
    The set of models used in the difficulty check; the benchmark's hardness is defined relative to these two models.
assumptions (5)
  • domain assumption The ground-truth answers are unique, correct, and temporally stable.
    Determinacy criterion in Section 4.1; relies on human annotation and multi-source cross-validation, which can err.
  • domain assumption LLM judges (with separate prompts) approximate human judgment.
    Section 5.2 and Appendix C.2; validated on the authors' own annotations (99.29% agreement), but no external human evaluation.
  • ad hoc to paper The difficulty filter (two-model single-turn search) correctly identifies questions that require multi-turn agentic behavior.
    Section 4.1; the criterion is relative to GPT-4o and DeepSeek-R1 at a specific date, so the benchmark is tuned to those models.
  • domain assumption The search-engine and browsing tools expose a faithful sample of the live web.
    Section A.2; results depend on DuckDuckGo/Google/Bing/Yahoo APIs and Selenium, which can be blocked, rate-limited, or return localized content.
  • ad hoc to paper The human-annotated source set Sq is a stable gold standard for compactness.
    Section 5.1 IC definition; reintroduces a static gold set despite the paper's argument that dynamic environments make gold sets infeasible.

how reviews work

0 comments
Cite this review

Pith. "Pith review of InfoDeepSeek: Benchmarking Agentic Information Seeking for Retrieval-Augmented Generation." pith.science (2026). https://pith.science/paper/NA2A4LRD

@misc{pith2026250515872,
  author       = {Pith},
  title        = {Pith review of: InfoDeepSeek: Benchmarking Agentic Information Seeking for Retrieval-Augmented Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NA2A4LRD}},
  note         = {Machine review of arXiv:2505.15872}
}
read the original abstract

Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by grounding responses with retrieved information. As an emerging paradigm, Agentic RAG further enhances this process by introducing autonomous LLM agents into the information seeking process. However, existing benchmarks fall short in evaluating such systems, as they are confined to a static retrieval environment with a fixed, limited corpus} and simple queries that fail to elicit agentic behavior. Moreover, their evaluation protocols assess information seeking effectiveness by pre-defined gold sets of documents, making them unsuitable for the open-ended and dynamic nature of real-world web environments. To bridge this gap, we present InfoDeepSeek, a new benchmark with challenging questions designed for assessing agentic information seeking in real-world, dynamic web environments. We propose a systematic methodology for constructing challenging queries satisfying the criteria of determinacy, difficulty, and diversity. Based on this, we develop the first evaluation framework tailored to dynamic agentic information seeking, including fine-grained metrics about the accuracy, utility, and compactness of information seeking outcomes. Through extensive experiments across LLMs, search engines, and question types, InfoDeepSeek reveals nuanced agent behaviors and offers actionable insights for future research.

Figures

Figures reproduced from arXiv: 2505.15872 by the authors.

Figure 1
Figure 1. Comparison between traditional RAG benchmark (up) and our InfoDeepSeek (bottom). [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The construction workflow of InfoDeepSeek dataset. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Performance of LLMs and search engines across different question attributes. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Performance with different maximum step T of information seeking. Test-time Scaling for Agentic Information Seeking. One of the key characteristics of an agent is that its performance scales with respect to the amount of compute available during test time. To investiga…
Figure 5
Figure 5. Figure 5: Retrieval interference (a) and the impact of languages (b). [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: The ratio of questions for different domains. [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Fetch-then-Explore: Decoupling Selection from Extraction over a Persistent Workspace for Search Agents

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Fetch-then-Explore, which stores fetched pages in a per-question workspace and extracts evidence on demand with grep/read, beats visit-and-read and browsing baselines on BrowseComp across three LLM backbones.

  2. LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Services

    cs.AI 2025-12 conditional novelty 6.0 of 10

    LocalSearchBench—1.3M merchant records and 900 multi-hop local-life QA tasks across 9 Chinese cities—shows the best reasoning agent reaches only 35.6% correctness.

  3. Search-Time Data Contamination

    cs.AI 2025-08 conditional novelty 5.0 of 10

    Web-searching AI agents sometimes find the test dataset's answer key on HuggingFace during evaluation, inflating their scores on roughly 3% of benchmark questions.

  4. A Vision for Geo-Temporal Deep Research Systems: Towards Comprehensive, Transparent, and Reproducible Geo-Temporal Information Synthesis

    cs.CL 2025-06 conditional novelty 4.0 of 10

    A research agenda calling for geo-temporal reasoning in deep research systems, with no experiments or system implementation.

Reference graph

Works this paper leans on

106 extracted references · 46 canonical work pages · cited by 4 Pith papers

  1. [1]

    Claude 3.7 sonnet, 2025

    Anthropic. Claude 3.7 sonnet, 2025. https://www.anthropic.com/claude/sonnet, Ac- cessed on 2025-05-05

    2025

  2. [2]

    Benchmarking large language mod- els in retrieval-augmented generation

    Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language mod- els in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17754–17762, 2024

    2024

  3. [3]

    Deepseek-v3-0324, 2025

    DeepSeek. Deepseek-v3-0324, 2025. https://api-docs.deepseek.com/news/ news250325, Accessed on 2025-05-05

    2025

  4. [4]

    A survey on rag meeting llms: Towards retrieval-augmented large language models

    Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. A survey on rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6491–6501, 2024

    2024

  5. [5]

    Retrieval-augmented generation for large language models: A survey

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2:1, 2023

    arXiv 2023

  6. [6]

    gemini-2.0-flash, 2025

    Gemini. gemini-2.0-flash, 2025. https://ai.google.dev/gemini-api/docs/models? hl=zh-cn#gemini-2.0-flash , Accessed on 2025-05-05

    2025

  7. [7]

    gemini-2.5-flash-preview, 2025

    Gemini. gemini-2.5-flash-preview, 2025. https://ai.google.dev/gemini-api/docs/ models?hl=zh-cn#gemini-2.5-flash-preview , Accessed on 2025-05-05

    2025

  8. [8]

    gemini-2.5-pro-preview, 2025

    Gemini. gemini-2.5-pro-preview, 2025. https://ai.google.dev/gemini-api/docs/ models?hl=zh-cn#gemini-2.5-pro-preview-05-06 , Accessed on 2025-05-06

    2025

Show all 106 references
  1. [9]

    Gemini deep research, 2025

    Gemini. Gemini deep research, 2025. https://gemini.google/overview/ deep-research/, Accessed on 2025-5-5. 10

    2025

  2. [10]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  3. [11]

    A survey on large language models: Applications, challenges, limitations, and practical usage

    Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muham- mad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints, 3, 2023

    2023

  4. [12]

    Rethinking with retrieval: Faithful large language model inference

    Hangfeng He, Hongming Zhang, and Dan Roth. Rethinking with retrieval: Faithful large language model inference. arXiv preprint arXiv:2301.00303, 2022

    2022 arXiv

  5. [13]

    Mintqa: A multi-hop ques- tion answering benchmark for evaluating llms on new and tail knowledge

    Jie He, Nan Hu, Wanqiu Long, Jiaoyan Chen, and Jeff Z Pan. Mintqa: A multi-hop ques- tion answering benchmark for evaluating llms on new and tail knowledge. arXiv preprint arXiv:2412.17032, 2024

    2024 arXiv

  6. [14]

    Pasa: An llm agent for comprehensive academic paper search

    Yichen He, Guanhua Huang, Peiyuan Feng, Yuan Lin, Yuchen Zhang, Hang Li, et al. Pasa: An llm agent for comprehensive academic paper search. arXiv preprint arXiv:2501.10120, 2025

    2025 arXiv

  7. [15]

    Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint arXiv:2011.01060, 2020

    2011 arXiv

  8. [16]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qiang- long Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Informat...

    2025

  9. [17]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  10. [18]

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017

    2017 arXiv

  11. [19]

    Chatgpt for good? on opportunities and challenges of large language models for education

    Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and indi...

    2023

  12. [20]

    Natural questions: a benchmark for question answering research

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computa...

    2019

  13. [21]

    Agent-g: An agentic framework for graph retrieval augmented generation

    Meng-Chieh Lee, Qi Zhu, Costas Mavromatis, Zhen Han, Soji Adeshina, Vassilis N Ioannidis, Huzefa Rangwala, and Christos Faloutsos. Agent-g: An agentic framework for graph retrieval augmented generation

  14. [22]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems...

    2020

  15. [23]

    Clickprompt: Ctr models are strong prompt generators for adapting language models to ctr prediction

    Jianghao Lin, Bo Chen, Hangyu Wang, Yunjia Xi, Yanru Qu, Xinyi Dai, Kangning Zhang, Ruiming Tang, Yong Yu, and Weinan Zhang. Clickprompt: Ctr models are strong prompt generators for adapting language models to ctr prediction. In Proceedings of the ACM Web Conference 2024, page...

    2024

  16. [24]

    Rella: Retrieval-enhanced large language models for lifelong sequential behavior comprehension in recommendation

    Jianghao Lin, Rong Shan, Chenxu Zhu, Kounianhua Du, Bo Chen, Shigang Quan, Ruiming Tang, Yong Yu, and Weinan Zhang. Rella: Retrieval-enhanced large language models for lifelong sequential behavior comprehension in recommendation. In Proceedings of the ACM Web Conference 2024, ...

    2024

  17. [25]

    How can recommender systems benefit from large language models: A survey

    Jianghao Lin, Xinyi Dai, Yunjia Xi, Weiwen Liu, Bo Chen, Hao Zhang, Yong Liu, Chuhan Wu, Xiangyang Li, Chenxu Zhu, et al. How can recommender systems benefit from large language models: A survey. ACM Transactions on Information Systems, 43(2):1–47, 2025

    2025

  18. [26]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  19. [27]

    Crud-rag: A comprehensive chinese benchmark for retrieval-augmented generation of large language models

    Y Lyu, Z Li, S Niu, F Xiong, B Tang, W Wang, H Wu, H Liu, T Xu, E Chen, et al. Crud-rag: A comprehensive chinese benchmark for retrieval-augmented generation of large language models. arxiv 2024. arXiv preprint arXiv:2401.17043

    2024 arXiv

  20. [28]

    Knowledge injection to counter large language model (llm) hallucination

    Ariana Martino, Michael Iannelli, and Coleen Truong. Knowledge injection to counter large language model (llm) hallucination. In European Semantic Web Conference, pages 182–185. Springer, 2023

    2023

  21. [29]

    Llama 4 maverick, 2025

    Meta. Llama 4 maverick, 2025. https://www.llama.com/docs/ model-cards-and-prompt-formats/llama4/ , Accessed on 2025-05-06

    2025

  22. [30]

    Ms marco: A human-generated machine reading comprehension dataset

    Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. Ms marco: A human-generated machine reading comprehension dataset. 2016

    2016

  23. [31]

    Gpt-4o mini, 2025

    OpenAI. Gpt-4o mini, 2025. https://platform.openai.com/docs/models/ gpt-4o-mini, Accessed on 2025-05-05

    2025

  24. [32]

    o3-mini, 2025

    OpenAI. o3-mini, 2025. https://platform.openai.com/docs/models/o3-mini, Ac- cessed on 2025-05-05

    2025

  25. [33]

    Introducing deep research, 2025

    OpenAI. Introducing deep research, 2025. https://openai.com/index/ introducing-deep-research/, Accessed on 2025-5-5

    2025

  26. [34]

    Kwaiagents: Generalized information-seeking agent system with large language models

    Haojie Pan, Zepeng Zhai, Hao Yuan, Yaojia Lv, Ruiji Fu, Ming Liu, Zhongyuan Wang, and Bing Qin. Kwaiagents: Generalized information-seeking agent system with large language models. arXiv preprint arXiv:2312.04889, 2023

    2023 arXiv

  27. [35]

    Llm evaluators recognize and favor their own generations, 2024

    Arjun Panickssery, Samuel R Bowman, and Shi Feng. Llm evaluators recognize and favor their own generations, 2024. URL https://arxiv. org/abs/2404.13076

    2024 arXiv

  28. [36]

    Ragnarök: A reusable rag framework and baselines for trec 2024 retrieval-augmented generation track

    Ronak Pradeep, Nandan Thakur, Sahel Sharifymoghaddam, Eric Zhang, Ryan Nguyen, Daniel Campos, Nick Craswell, and Jimmy Lin. Ragnarök: A reusable rag framework and baselines for trec 2024 retrieval-augmented generation track. In European Conference on Information Retrieval, pag...

    2024

  29. [37]

    Agentic retrieval- augmented generation for time series analysis

    Chidaksh Ravuru, Sagar Srinivas Sakhinana, and Venkataramana Runkana. Agentic retrieval- augmented generation for time series analysis. arXiv preprint arXiv:2408.14484, 2024

    2024 arXiv

  30. [38]

    Evaluating retrieval quality in retrieval-augmented gen- eration

    Alireza Salemi and Hamed Zamani. Evaluating retrieval quality in retrieval-augmented gen- eration. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2395–2400, 2024

    2024

  31. [39]

    Collex–a multimodal agentic rag system enabling interactive exploration of scientific collections

    Florian Schneider, Narges Baba Ahmadi, Niloufar Baba Ahmadi, Iris V ogel, Martin Semmann, and Chris Biemann. Collex–a multimodal agentic rag system enabling interactive exploration of scientific collections. arXiv preprint arXiv:2504.07643, 2025

    2025 arXiv

  32. [40]

    In chatgpt we trust? measuring and characterizing the reliability of chatgpt

    Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. In chatgpt we trust? measuring and characterizing the reliability of chatgpt. arXiv preprint arXiv:2304.08979, 2023

    2023 arXiv

  33. [41]

    Autogpt: Build, deploy, and run ai agents, 2025

    Significant-Gravitas. Autogpt: Build, deploy, and run ai agents, 2025. https://github.com/ Significant-Gravitas/AutoGPT, Accessed on 2025-05-06

    2025

  34. [42]

    Agentic retrieval-augmented generation: A survey on agentic rag

    Aditi Singh, Abul Ehtesham, Saket Kumar, and Tala Talaei Khoei. Agentic retrieval-augmented generation: A survey on agentic rag. arXiv preprint arXiv:2501.09136, 2025

    2025 arXiv

  35. [43]

    Multihop-rag: Benchmarking retrieval-augmented generation for multi-hop queries

    Yixuan Tang and Yi Yang. Multihop-rag: Benchmarking retrieval-augmented generation for multi-hop queries. arXiv preprint arXiv:2401.15391, 2024. 12

    2024 arXiv

  36. [44]

    Introducing perplexity deep research, 2025

    Perplexity Team. Introducing perplexity deep research, 2025. https://www.perplexity. ai/hub/blog/introducing-perplexity-deep-research , Accessed on 2025-5-5

    2025

  37. [45]

    Qwen3 technical report, 2025

    Qwen Team. Qwen3 technical report, 2025. https://github.com/QwenLM/Qwen3/blob/ main/Qwen3_Technical_Report.pdf, Accessed on 2025-05-14

    2025

  38. [46]

    Large language models in medicine.Nature medicine, 29(8):1930–1940, 2023

    Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine.Nature medicine, 29(8):1930–1940, 2023

    1930

  39. [47]

    Musique: Multihop questions via single-hop question composition

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554, 2022

    2022

  40. [48]

    Freshllms: Refreshing large language models with search engine augmentation

    Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, et al. Freshllms: Refreshing large language models with search engine augmentation. arXiv preprint arXiv:2310.03214, 2023

    arXiv

  41. [49]

    A survey on large language model based autonomous agents

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345, 2024

    2024

  42. [50]

    Emergent abilities of large language models

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022

    2022 arXiv

  43. [51]

    Browsecomp: A simple yet challenging benchmark for browsing agents

    Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516, 2025

    2025 arXiv

  44. [52]

    Retrieval-augmented generation for natural language processing: A survey

    Shangyu Wu, Ying Xiong, Yufei Cui, Haolun Wu, Can Chen, Ye Yuan, Lianming Huang, Xue Liu, Tei-Wei Kuo, Nan Guan, et al. Retrieval-augmented generation for natural language processing: A survey. arXiv preprint arXiv:2407.13193, 2024

    2024 arXiv

  45. [53]

    Towards open-world recommendation with knowledge augmentation from large language models

    Yunjia Xi, Weiwen Liu, Jianghao Lin, Xiaoling Cai, Hong Zhu, Jieming Zhu, Bo Chen, Ruiming Tang, Weinan Zhang, and Yong Yu. Towards open-world recommendation with knowledge augmentation from large language models. In Proceedings of the 18th ACM Conference on Recommender System...

    2024

  46. [54]

    Memocrs: Memory-enhanced sequential conversational recommender systems with large language models

    Yunjia Xi, Weiwen Liu, Jianghao Lin, Bo Chen, Ruiming Tang, Weinan Zhang, and Yong Yu. Memocrs: Memory-enhanced sequential conversational recommender systems with large language models. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Manage...

    2024

  47. [55]

    A decoding acceleration framework for industrial deployable llm-based recommender systems

    Yunjia Xi, Hangyu Wang, Bo Chen, Jianghao Lin, Menghui Zhu, Weiwen Liu, Ruiming Tang, Weinan Zhang, and Yong Yu. A decoding acceleration framework for industrial deployable llm-based recommender systems. arXiv preprint arXiv:2408.05676, 2024

    2024 arXiv

  48. [56]

    Bursting filter bubble: Enhancing serendipity recommenda- tions with aligned large language models

    Yunjia Xi, Muyan Weng, Wen Chen, Chao Yi, Dian Chen, Gaoyang Guo, Mao Zhang, Jian Wu, Yuning Jiang, Qingwen Liu, et al. Bursting filter bubble: Enhancing serendipity recommenda- tions with aligned large language models. arXiv preprint arXiv:2502.13539, 2025

    2025 arXiv

  49. [57]

    A critical evaluation of evaluations for long-form question answering

    Fangyuan Xu, Yixiao Song, Mohit Iyyer, and Eunsol Choi. A critical evaluation of evaluations for long-form question answering. arXiv preprint arXiv:2305.18201, 2023

    2023 arXiv

  50. [58]

    Crag-comprehensive rag benchmark

    Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Gui, Ziran Jiang, Ziyu Jiang, et al. Crag-comprehensive rag benchmark. Advances in Neural Information Processing Systems, 37:10470–10490, 2024

    2024

  51. [59]

    A survey of ai agent protocols

    Yingxuan Yang, Huacan Chai, Yuanyi Song, Siyuan Qi, Muning Wen, Ning Li, Junwei Liao, Haoyi Hu, Jianghao Lin, Gaowei Chang, et al. A survey of ai agent protocols. arXiv preprint arXiv:2504.16736, 2025. 13

    2025 arXiv

  52. [60]

    Babyagi, 2025.https://github.com/yoheinakajima/babyagi, Accessed on 2025-05-06

    yoheinakajima. Babyagi, 2025.https://github.com/yoheinakajima/babyagi, Accessed on 2025-05-06

    2025

  53. [61]

    Agentic information retrieval

    Weinan Zhang, Junwei Liao, Ning Li, Kounianhua Du, and Jianghao Lin. Agentic information retrieval. arXiv preprint arXiv:2410.09713, 2024

    2024 arXiv

  54. [62]

    Retrieval-augmented generation for ai-generated content: A survey

    Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang, Wentao Zhang, Jie Jiang, and Bin Cui. Retrieval-augmented generation for ai-generated content: A survey. arXiv preprint arXiv:2402.19473, 2024

    2024 arXiv

  55. [63]

    A survey of large language models

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 1(2), 2023

    2023 arXiv

  56. [64]

    Browsecomp-zh: Benchmarking web browsing ability of large language models in chinese

    Peilin Zhou, Bruce Leon, Xiang Ying, Can Zhang, Yifan Shao, Qichen Ye, Dading Chong, Zhiling Jin, Chenxuan Xie, Meng Cao, et al. Browsecomp-zh: Benchmarking web browsing ability of large language models in chinese. arXiv preprint arXiv:2504.19314, 2025. 14 Table of Contents A ...

    2025 arXiv

  57. [67]

    You can use the reference tools mentioned when planning

  58. [72]

    task_name

    If you have sufficient information to answer the given query, invoke the termination tool to terminate planning. Otherwise, continue planning new tasks while ensuring no duplication with prior tasks. {tool_specification} {current_date_and_time} {memory} Given Query: {query} Ba...

  59. [73]

    A webpage consists of a URL and webpage summary or information extracted from the webpage that is relevant to the query

  60. [74]

    If multiple pieces of information come from the same webpage (determined by identical URLs), merge them rather than listing duplicates

  61. [75]

    If the question has multiple sub-questions, the relevant webpages of each sub-question must be included

    The output webpage list must include relevant webpages necessary to answer the question. If the question has multiple sub-questions, the relevant webpages of each sub-question must be included

  62. [76]

    If it is more than {max_webpage_num}, select {max_webpage_num} of the most important ones

    The number of webpages in the output webpage list can be less than {max_webpage_num}. If it is more than {max_webpage_num}, select {max_webpage_num} of the most important ones

  63. [77]

    url": "The webpage's URL

    The output webpage list is sorted according to its importance to answering the question, that is, the webpage ranked first has the greatest contribution to answering the question. {current_date_and_time} {memory} Given Query: {query} You must generate the list of webpages stri...

  64. [78]

    If the user’s query contains multiple answers, list all of them

  65. [79]

    If the user’s query is based on a wrong premise, point out the error. {current_date_and_time} Given query: {query} Relevant webpages: {webpages} Generate a brief English answer to solve the user’s query: B Dataset Construction B.1 Query Generation Our query construction pipeli...

  66. [80]

    advantage languages

    Anchor knowledge identification . Annotators begin by selecting an underrepresented domain or language. They search authoritative sources such as Wikipedia, reputable news sites, fact-checking platforms, expert forums, and academic databases to extract candidate facts—particul...

  67. [81]

    Question composition. Based on the anchor knowledge, annotators either directly con- struct a question or further increase question difficulty by incorporating multiple challenge attributes or introducing additional domain-specific facts – either common or difficult – through ...

  68. [82]

    Who is the current president of the USA?

    Temporal stability check. Annotators have to verify whether the answer varies with time. For potentially unstable questions (e.g., “Who is the current president of the USA?”), we explicitly add time constraints (e.g., “In 2025”) to ensure the answer remains fixed and verifiable

    2025

  69. [83]

    This question contains a false premise:

    Diversity control. We aim for each question to include at least two difficulty attributes, and for the dataset to span a wide range of domains, languages, and countries. Annotators actively switch focus when certain attributes or domains become overrepresented. 19 B.1.2 Filter...

  70. [84]

    Content Verification: Check the correctness of the question and answer, and ensure that the listed sources support the answer. Special attention is paid to the accuracy of proper noun translations, which must be verified against online references or annotated using their origi...

  71. [85]

    Criteria Check: Evaluate whether the question meets the required conditions: Determinacy (Is the answer stable over time?) and Difficulty (Can GPT-4o and DeepSeek-R1 in a web- enabled, single-turn search model correctly answer the question?) Regardless of whether the question ...

  72. [86]

    Metadata Verification: Ensure the correctness of annotated attributes, domain, and advan- tage language

  73. [87]

    a president who is a comedian

    Validation Outcome: Verifiers must fill out three fields: their name, a binary result (pass/fail), and explanatory notes. A question is marked as passed only if it fully sat- isfies all requirements (content, criteria, metadata). A fail may indicate outright rejection or sugge...

    2022

  74. [88]

    The candidate answer must contain content consistent with groundtrue answer to be considered correct (pay attention to the consistency of time, place, and person), but more details can be provided

  75. [89]

    If there are multiple items in the groundtrue answer, the candidate answer must also contain all the items to be correct

  76. [90]

    If the groundtrue answer mentions that the premise of the question is wrong, such as some assumptions of the question are wrong, the candidate answer must point out the wrong premise to be considered correct

  77. [91]

    If the candidate answer says that the question has a wrong premise„ such as person/event may not exist, it is a wrong answer

    When the groundtrue answer does not mention the wrong premise, the question itself is not problematic. If the candidate answer says that the question has a wrong premise„ such as person/event may not exist, it is a wrong answer

  78. [92]

    This question contains a false premise:

    The given question has a clear answer and the given groundtrue answer is absolutely right. If the candidate answer says it doesn’t know or lacks information, it is a wrong answer Just answer Yes or No. Given Question: {query} 24 Table 6: The Accuracy of unified prompt and sepa...

  79. [93]

    If the candidate answer does not point out or correct this false premise, it is incorrect

    This question has a false premise, which has been pointed out in the groundtruth answer. If the candidate answer does not point out or correct this false premise, it is incorrect

  80. [94]

    If the false premise pointed out by the candidate answer is different from the groundtruth answer (time, place, event, person, meaning inconsistent), it is incorrect

  81. [95]

    If the groundtruth answer still answers the question in addition to pointing out the false premise, the candidate answer should also answer the question. In this case, if the candidate answer does not answer the question or the meaning of the answer content is inconsistent wit...

  82. [96]

    The candidate answer must contain content that is consistent with the groundtrue answer to be considered correct (pay attention to the consistency of time, place, person, and quantity), but more details can be provided

  83. [97]

    If there are multiple contents/events/persons in the groundtrue answer, the candidate answer must also contain all the contents/events/persons to be considered correct

  84. [98]

    If the 25 candidate answer proposes a wrong premise or cannot determine whether the person/event exists, it is a wrong answer

    The given question does not have a wrong premise, and the relevant person/event must exist and be unique. If the 25 candidate answer proposes a wrong premise or cannot determine whether the person/event exists, it is a wrong answer

  85. [99]

    If the candidate answer does not answer the question correctly but proposes the need to further query relevant information, it is a wrong answer

    The given question has a clear answer and the given groundtrue answer must be correct. If the candidate answer does not answer the question correctly but proposes the need to further query relevant information, it is a wrong answer. Just answer Yes or No. Given question: {quer...

    1904

  86. [100]

    Retrieval stage: 5 calls for planning and action of the agent

  87. [101]

    Augmentation stage: 1 call to extract relevant evidence

  88. [102]

    Answer-generation stage: 5 calls for answering based on five evidence from the evidence set and one call for answering based on all the observations

  89. [103]

    Each query also consumes about 24k input tokens and produces roughly 4k output tokens

    Evaluation stage: 12 calls (two LLMs) to evaluate six candidate answers. Each query also consumes about 24k input tokens and produces roughly 4k output tokens. Most of the input tokens come from the retrieval and augmentation stages, since those involve lengthy interaction his...

  90. [104]

    You have a short-term memory of approximately 4,000 characters

  91. [105]

    You do not require assistance or response from users

  92. [106]

    31 Table 17: Retrieval interference under different languages, measured by %

    You can use the reference tools mentioned when planning. 31 Table 17: Retrieval interference under different languages, measured by %. Pred. Lang. denotes Predominant Language. Model Language Interference Rate DeepSeek-V3 Chinese 84.21 English 64.71 Pred. Lang. 56.25 GPT-4o Ch...

  93. [107]

    Be sure to verify the truthfulness of the information

    Complex problems can be split into sub-problems and then information can be collected, aggregated and authenticated. Be sure to verify the truthfulness of the information

  94. [108]

    Stay humble and call the tool for questions you are not sure about, but do not call the same tool with the same parameters repeatedly

  95. [109]

    You can flexibly switch the language of the search term to get more information. You can choose to search in Chinese, English, or the language related to the entity involved in the question (for example, if the question involves a French person, you can search in French)

  96. [110]

    You can think and plan up to {max_iter_num} steps, so strive to plan tasks as efficiently as possible

  97. [111]

    You have the capability for reflection and self-criticism; reflect on past decisions and improve your strategies

  98. [112]

    task_name

    If you have sufficient information to answer the given query, invoke the termination tool to terminate planning. Otherwise, continue planning new tasks while ensuring no duplication with prior tasks. {tool_specification} {current_date_and_time} {memory} Given Query: {query} Ba...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.