Pith. sign in

REVIEW 3 major objections 4 minor 62 references

DRNOISE: Benchmarking Deep Research Agents in Misleading Evidence Environments

T0 review · 3 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single plausible false document causes deep research agents to drop 66–88 percentage points in accuracy, and the failure is premature stopping, not failed retrieval.

desk verdict Useful paired benchmark with a likely-real effect, but as submitted it asks referees to trust an unshipped corpus, and the headline mechanism is traced on one model. read the letter →

arxiv 2607.17291 v1 pith:AFPRGZR5 submitted 2026-07-19 cs.LG cs.CLcs.IR

Jun Nie , Zhiqin Yang , Zhenheng Tang , Yonggang Zhang , Xiaowen Chu , Xinmei Tian , Bo Han This is my paper · ORCID
classification cs.LGcs.CLcs.IR
keywords deepresearchagentsmisleadingevidencebenchmarkverificationinertiaprematurestoppingconditionaldeferenceretrieval-augmentedgenerationreconciliation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces DRNOISE, a 100-task benchmark where each answer must be reconstructed from two corroborating chains of indirect records, and the noisy condition adds one plausible document that states a conflicting answer directly. The central finding is that this single ordinary-looking page causes 66–88 percentage-point accuracy drops in deep research agents that otherwise solve the clean tasks, and that the failures are not caused by missing evidence. Trace and oracle analyses identify verification inertia: agents retrieve the truth but stop when they encounter an answer-shaped document and commit its false value. Generic verification prompts help only modestly, while making the true answer as direct as the false one or explicitly instructing distrust of summaries recovers much of the gap. The conclusion is that reliable deep research needs active reconciliation of direct claims with record-level evidence, not just retrieval and citation.

What carries the argument

The paired clean–noisy intervention: two corroborating indirect record routes (Route A and Route B) that entail a unique gold answer, plus a single direct false-summary document in the noisy condition. Its diagnostic power comes from conditional deference—the fraction of clean-correct tasks where the model commits the designated false value under noise—together with a retrieval-exposure funnel and an oracle full-context ablation that separate premature stopping from retrieval failure and from genuine reconciliation failure.

What would settle it

Have independent auditors check all 100 tasks for the three declared label properties: a unique gold answer entailed by two genuinely separate record routes, and a false summary that is plausible but refuted by the records; then recompute conditional deference on the audited subset. If even a few tasks fail audit, the headline accuracy drops are an artifact of benchmark labels rather than agent behavior.

Watch

Extended reading notes

Core claim

Each task pairs a clean corpus, whose two independent record routes entail one unique gold answer, with a noisy corpus identical except for one ordinary-looking document stating a directly conflicting value. On five search-agent models with strong clean-task accuracy, this single document causes 66–88 percentage-point accuracy drops across symbolic and numerical families. Trace analyses show the failures are not retrieval misses: in nearly every noisy failure the agent retrieved both truth-supporting records and the false summary, but stopped before completing an evidence route. An oracle full-context test, with all records in the prompt, restores 91/100 accuracy even with the false document

Load-bearing premise

The results assume that in every one of the 100 tasks the gold answer really is the unique value entailed by two independent record chains and that each injected false summary is genuinely plausible but falsifiable—if the labels are off, the deference numbers are off.

Editorial extensions

If this is right

  • If the finding holds, high clean-task accuracy is not evidence that a research agent will keep its standards when a plausible falsehood is present; robustness must be measured under conflicting evidence, not just clean retrieval.
  • Benchmarks for research agents should include paired clean–noisy conditions so that a failure is attributable to a single document, and should report conditional deference over clean-correct tasks rather than overall error only.
  • Generic verify-before-answering instructions are not sufficient; closing the gap appears to require structural changes that force completing and reconciling an evidence route, such as requiring record-level citations or making true evidence as directly accessible as the false summary.
  • Because the effect spans symbolic and numerical task families, the failure is not a narrow arithmetic quirk but a general evidence-use policy problem.
  • The residual where agents override an already complete truthful route shows that even with all evidence in hand, a directly stated false claim can occasionally beat a reconstructed truth.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper compares the behavior to sycophancy directed at the corpus; we infer the same stopping bias likely appears in real deployments such as financial or due-diligence tools, where a confident summary is retrieved alongside raw records, so monitoring should flag answer-shaped documents as stopping signals.
  • A testable extension: vary the length and formatting of the false summary to see whether deference tracks how answer-like the document appears, or whether any direct assertion of a value triggers the effect.
  • The saturation of the source-credibility ladder suggests that the decisive cue is directness rather than authority; we infer that source-credibility labels alone will not firewall agents.
  • An intervention worth testing: require the agent to explicitly compare any direct claim against at least one complete record route before producing a final answer, and measure whether the minority of complete-route overrides disappears.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. DRNOISE introduces a 100-task benchmark, organized into ten families, for evaluating deep research agents under a controlled misleading-evidence intervention. Each task pairs a clean corpus containing two indirect evidence routes that are claimed to entail a unique gold answer with a noisy corpus that adds one plausible document directly stating a conflicting false answer. The authors evaluate five agents in a fixed BrowseComp-Plus-style harness and report 66–88 percentage-point accuracy drops for high-clean-accuracy systems, together with a conditional-deference metric. Trace analyses of GPT-5.4 attribute most failures to premature stopping ('verification inertia'), and ablations show that generic verification prompts, source-credibility changes, and retriever changes do not close the gap, while attack-aware prompts and a direct-true-document structure recover much of the lost performance.

Significance. If the benchmark's asserted properties hold, DRNOISE is a valuable controlled diagnostic that separates a capability-deployment gap from raw retrieval or reasoning failure. The paired clean/noisy design, the conditional-deference metric, and the multi-way ablations (retrieval funnel, trace taxonomy, oracle full-context, direct-true evidence) are thoughtful and internally coherent. The oracle experiment and the direct-true ablation are particularly useful checks on the premature-stopping interpretation. However, the headline results and the mechanism claim rest on benchmark labels that are asserted but not shipped or independently validated, and the mechanism analysis is performed on a single model and a single run. These issues are load-bearing rather than cosmetic, so the contribution is promising but currently conditional on elements the manuscript does not provide.

major comments (3)
  1. [§3.1, Appendix B, Table 3] The validity of DRNOISE rests on asserted properties: each task has a unique gold answer entailed by two independent, corroborating indirect routes, and each false summary is plausible yet falsifiable. Appendix B states these are 'enforced during construction,' but the paper ships no corpus, route labels, audit code, or human/LLM plausibility judgments. Because clean/noisy accuracy, conditional deference (Table 3), the complete-route funnel (Table 5), and the trace taxonomy (Table 6) all inherit these labels, the 66–88 pp accuracy drops and the premature-stopping mechanism are conditional on unverified construction. If a route is not genuinely independent, or a single document within a route alone entails the gold answer, then 'only partial route retrieved' does not establish premature stopping; it could reflect a search-ranking artifact. Please release the dataset with route annotations
  2. [§6, Table 6, Abstract] The 'dominant failure mode' claim of verification inertia is established from GPT-5.4 only, on one fixed dense-retrieval baseline run. The abstract and introduction present it as a general finding about the evaluated agents, but the trace taxonomy in Table 6 is a single-model, single-run observation. The 77/100 'only partial truth route retrieved' category is also defined relative to the benchmark's route labels, so it inherits the validity concern above. Please either restrict the mechanism claim to GPT-5.4 or provide trace taxonomies and retrieval-exposure funnels for the other high-clean systems (DeepSeek, MiMo, Gemini) and across the three runs. Without this, the 'dominant' status is not established.
  3. [§7 Limitations and Table 11] The oracle full-context experiment is an internal consistency check using the same gold and route labels as the main benchmark; it cannot validate the benchmark's ground-truth claims. The limitations paragraph acknowledges that the corpus is constructed but does not address the absence of external validation of the unique-gold and route-independence criteria. A concrete falsifiability check would strengthen the paper: for a random sample of tasks, independent annotators should be able to verify that routes A and B are separate and jointly entail the unique gold, that the false summary is refuted by the records, and that the false summary is not distinguishable from ordinary business artifacts without reading it. This is necessary to separate a genuine deference effect from an artifact of the benchmark's own label construction.
minor comments (4)
  1. [Title/header] The running header on page 1 reads 'BENCHMARKINGDEEPRESEARCHAGENTS' with missing spaces; also in §1 'DRNOISEreveals' appears without a space.
  2. [Table 3] For Qwen3-32B, conditional deference is computed over only 13 clean-correct tasks; the 61.3% value should be accompanied by a confidence interval or explicit caution in the table caption, not only in the main text.
  3. [Appendix E] The document-level example omits most of the route documents ('not shown'), so the reader cannot actually see how the two routes independently entail the gold answer. Including the full route document sets for at least one task, or a link to the released corpus, would make the claimed construction properties concrete.
  4. [Appendix C] The grading definition 'correct if it matches the gold value and not the false value' is redundant since gold and false values are distinct; the wording is harmless but could be simplified.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: DRNOISE measures a constructed benchmark intervention with definitional metrics; the headline results do not reduce to their inputs.

full rationale

The paper's central quantities are measurements under a controlled paired intervention, not predictions derived from fitted parameters or from prior work by the same authors. The gold answer is defined by construction (two corroborating indirect record chains entailing a unique value), and the false summary is likewise an input label; accuracy drop and conditional deference are then computed directly from model outputs against these labels. There is no equation in which a fitted parameter is later presented as a prediction, no load-bearing self-citation chain, and no imported uniqueness theorem that forces the conclusion. The trace analyses use the benchmark's own route labels to distinguish 'complete route' from 'partial route,' which is an interpretive classification rather than a circular derivation, though one might ask for independent validation of the corpus properties. The limitation section explicitly acknowledges that the corpus is constructed rather than scraped from the live web, which is an external-validity caveat, not evidence of circular reasoning. The paper is self-contained as an empirical benchmark evaluation, so the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 1 invented entities

The benchmark's central measurement does not depend on fitted parameters; the design choices are fixed experimental settings. The three domain assumptions above are load-bearing: if the constructed corpus is not genuinely well-posed or plausible, the magnitude and interpretation of the measured deference change.

assumptions (3)
  • domain assumption Each task has a unique gold answer entailed by two independent, corroborating indirect record chains, and no single document states it directly.
    Stated in §3.1 and Appendix B as enforced during construction, but no released data or independent verification is provided. If a route is not actually independent or the answer is not unique, clean/noisy labels are uncertain.
  • domain assumption The false summaries are plausible, ordinary-looking, and of comparable authority to the direct-true documents.
    Assumed in §3.2 and Appendix D from genre and style; no human raters or external plausibility measurement are reported. If the false documents are not plausible, the benchmark does not test misleading evidence in the intended sense.
  • domain assumption The BrowseComp-Plus harness with a single dense retriever and five API models is representative of deep research agents.
    Used in §4.1 to generalize the central claim from one harness. The BM25 ablation partially addresses retriever dependence, but no other agent framework or search protocol is tested.
invented entities (1)
  • Verification inertia
    purpose: Explanatory construct for the dominant failure mode: agents stop searching and reconciling evidence after encountering an answer-like document.
    Introduced in §1 and §5.3 and named in the conclusion. It is a post-hoc label for observed GPT-5.4 traces, supported within this benchmark but without an independent, outside-the-paper falsifiable handle. This is a conceptual label, not a physical entity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DRNOISE: Benchmarking Deep Research Agents in Misleading Evidence Environments." pith.science (2026). https://pith.science/paper/AFPRGZR5

@misc{pith2026260717291,
  author       = {Pith},
  title        = {Pith review of: DRNOISE: Benchmarking Deep Research Agents in Misleading Evidence Environments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AFPRGZR5}},
  note         = {Machine review of arXiv:2607.17291}
}
read the original abstract

Deep research agents increasingly operate over the open web, where relevant records coexist with redundant summaries, outdated reports, and misleading documents. Existing evaluations offer limited insight into whether agents preserve sound evidential standards when an ordinary-looking false document is deliberately seeded into a searchable environment and offers a direct shortcut to a conflicting answer. We introduce DRNOISE, a 100-task benchmark for answer recovery under misleading evidence. Each task has a unique gold answer supported by two corroborating indirect record chains; the paired noisy condition adds one plausible document that states a conflicting answer directly. The benchmark spans ten families of evidence operations. Across agents with strong clean-task performance, this single intervention causes 66-88 percentage-point accuracy drops. Trace analyses identify verification inertia as the dominant failure mode: agents often retrieve truthful records but stop before completing and reconciling the evidence chain, instead deferring to the answer-like document. Generic verification prompts reduce but do not close this gap. The setting is especially relevant to open-web deployment, where plausible falsehoods arrive through ordinary-looking pages rather than explicit attacks. Reliable deep research therefore requires more than retrieval and citation; it requires active reconciliation of direct claims with record-level evidence.

Figures

Figures reproduced from arXiv: 2607.17291 by the authors.

Figure 1
Figure 1. Main benchmark result on DRNOISE. Left: clean vs. noisy accuracy on the 100-task benchmark. Adding a single misleading document sharply reduces the accuracy of agents with strong clean-task performance. Right: conditional deference, the fraction of a system’s clean-correct tasks that are answered with the injected false value once the misleading document is present. 1 INTRODUCTION Deep research agents are increasing… view at source ↗
Figure 2
Figure 2. Overview of the DRNOISE paired intervention. Left: in the clean condition, two corroborat￾ing chains of records converge on the same gold answer, which the agent must reconstruct. Right: the noisy condition is identical except for one added ordinary-looking document that states a conflicting answer directly; the truthful records remain available, but the agent stops before reconciling them and commits the false valu… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 6 linked inside Pith

  1. [1]

    Self-RAG : Learning to retrieve, generate, and critique through self-reflection

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avi Sil, and Hannaneh Hajishirzi. Self-RAG : Learning to retrieve, generate, and critique through self-reflection. In International conference on learning representations, ICLR , 2024

    2024

  2. [2]

    Benchmarking large language models in retrieval-augmented generation

    Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024

    2024

  3. [3]

    BrowseComp-Plus : A more fair and transparent evaluation benchmark of deep-research agent

    Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, et al. BrowseComp-Plus : A more fair and transparent evaluation benchmark of deep-research agent. arXiv preprint arXiv:2508.06600, 2025

    arXiv 2025

  4. [4]

    The power of noise: Redefining retrieval for RAG systems

    Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. The power of noise: Redefining retrieval for RAG systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2024

    2024

  5. [5]

    InteractComp : Evaluating search agents with ambiguous queries

    Mingyi Deng, Lijun Huang, Yani Fan, Jiayi Zhang, Fashen Ren, Jinyi Bai, Fuzhen Yang, Dayi Miao, Zhaoyang Yu, Yifan Wu, et al. InteractComp : Evaluating search agents with ambiguous queries. arXiv preprint arXiv:2510.24668, 2025

    arXiv 2025

  6. [6]

    Mind2Web : Towards a generalist agent for the web

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2Web : Towards a generalist agent for the web. Advances in Neural Information Processing Systems, NeurIPS , 2023

    2023

  7. [7]

    Enabling large language models to generate text with citations

    Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models to generate text with citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023

    2023

  8. [8]

    Large language models cannot self-correct reasoning yet

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In International conference on learning representations, ICLR , 2024

    2024

Show all 62 references
  1. [9]

    Survey of hallucination in natural language generation

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM computing surveys, 55 0 (12): 0 1--38, 2023

    2023

  2. [10]

    Search-R1 : Training LLMs to reason and leverage search engines with reinforcement learning

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-R1 : Training LLMs to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025

    2025 arXiv

  3. [11]

    Dense passage retrieval for open-domain question answering

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 conference on empirical methods in natural language processing, EMNLP , 2020

    2020

  4. [12]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in neural information processing...

    2020

  5. [13]

    Entity-based knowledge conflicts in question answering

    Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity-based knowledge conflicts in question answering. In Proceedings of the 2021 conference on empirical methods in natural language processing, 2021

    2021

  6. [14]

    Self-Refine : Iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-Refine : Iterative refinement with self-feedback. Advances in neural information processing systems, NeurIPS , 2023

    2023

  7. [15]

    GAIA : a benchmark for general AI assistants

    Gr \'e goire Mialon, Cl \'e mentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA : a benchmark for general AI assistants. In International Conference on Learning Representations, ICLR , 2024

    2024

  8. [16]

    FActScore : Fine-grained atomic evaluation of factual precision in long form text generation

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore : Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Me...

    2023

  9. [17]

    WebGPT : Browser-assisted question-answering with human feedback

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. WebGPT : Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021

    2021 arXiv

  10. [18]

    On the risk of misinformation pollution with large language models

    Yikang Pan, Liangming Pan, Wenhu Chen, Preslav Nakov, Min-Yen Kan, and William Wang. On the risk of misinformation pollution with large language models. In Findings of the association for computational linguistics: EMNLP 2023, 2023

    2023

  11. [19]

    Tell me more! towards implicit user intention understanding of language model driven agents

    Cheng Qian, Bingxiang He, Zhong Zhuang, Jia Deng, Yujia Qin, Xin Cong, Zhong Zhang, Jie Zhou, Yankai Lin, Zhiyuan Liu, et al. Tell me more! towards implicit user intention understanding of language model driven agents. In Proceedings of the 62nd Annual Meeting of the Associati...

    2024

  12. [20]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dess \` , Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in neural information processing systems, NeurIPS , 2023

    2023

  13. [21]

    Towards understanding sycophancy in language models

    Mrinank Sharma, Meg Tong, Tomek Korbak, David Duvenaud, Amanda Askell, Sam Bowman, Esin Durmus, Zac Hatfield-Dodds, Scott Johnston, Shauna Kravec, et al. Towards understanding sycophancy in language models. In International Conference on Learning Representations, ICLR , 2024

    2024

  14. [22]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems, NeurIPS , 2023

    2023

  15. [23]

    FEVER : a large-scale dataset for fact extraction and VER ification

    James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER : a large-scale dataset for fact extraction and VER ification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua...

    2018

  16. [24]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, NeurIPS , 2022

    2022

  17. [25]

    BrowseComp : A simple yet challenging benchmark for browsing agents

    Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. BrowseComp : A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516, 2025

    2025 arXiv

  18. [26]

    WebWalker : Benchmarking LLMs in web traversal

    Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Linhai Zhang, Yulan He, Deyu Zhou, Pengjun Xie, et al. WebWalker : Benchmarking LLMs in web traversal. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume ...

    2025

  19. [27]

    Knowledge conflicts for LLMs : A survey

    Rongwu Xu, Zehan Qi, Zhijiang Guo, Cunxiang Wang, Hongru Wang, Yue Zhang, and Wei Xu. Knowledge conflicts for LLMs : A survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024

    2024

  20. [28]

    Narasimhan, and Yuan Cao

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. ReAct : Synergizing reasoning and acting in language models. In International Conference on Learning Representations, ICLR , 2023

    2023

  21. [29]

    -bench : A benchmark for tool-agent-user interaction in real-world domains

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. -bench : A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045, 2024

    2024 arXiv

  22. [30]

    WebArena : A realistic web environment for building autonomous agents

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. WebArena : A realistic web environment for building autonomous agents. In International Conference on Learning Representations, ICLR , 2024

    2024

  23. [31]

    PoisonedRAG : Knowledge corruption attacks to Retrieval-Augmented generation of large language models

    Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia. PoisonedRAG : Knowledge corruption attacks to Retrieval-Augmented generation of large language models. In 34th USENIX Security Symposium (USENIX Security 25), 2025

    2025

  24. [32]

    Wei, Jason and Sun, Zhiqing and Papay, Spencer and McKinney, Scott and Han, Jeffrey and Fulford, Isa and Chung, Hyung Won and Passos, Alex Tachard and Fedus, William and Glaese, Amelia , journal=

  25. [33]

    International Conference on Learning Representations,

    Mialon, Gr. International Conference on Learning Representations,

  26. [34]

    Deng, Mingyi and Huang, Lijun and Fan, Yani and Zhang, Jiayi and Ren, Fashen and Bai, Jinyi and Yang, Fuzhen and Miao, Dayi and Yu, Zhaoyang and Wu, Yifan and others , journal=

  27. [35]

    Thorne, James and Vlachos, Andreas and Christodoulopoulos, Christos and Mittal, Arpit , booktitle=

  28. [36]

    Retrieval-augmented generation for knowledge-intensive

    Lewis, Patrick and Perez, Ethan and Piktus, Aleksandra and Petroni, Fabio and Karpukhin, Vladimir and Goyal, Naman and K. Retrieval-augmented generation for knowledge-intensive. Advances in neural information processing systems,

  29. [37]

    Proceedings of the 2020 conference on empirical methods in natural language processing,

    Dense passage retrieval for open-domain question answering , author=. Proceedings of the 2020 conference on empirical methods in natural language processing,

    2020

  30. [38]

    Advances in neural information processing systems,

    Toolformer: Language models can teach themselves to use tools , author=. Advances in neural information processing systems,

  31. [39]

    Yao, Shunyu and Shinn, Noah and Razavi, Pedram and Narasimhan, Karthik , journal=

  32. [40]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , year=

    Tell me more! towards implicit user intention understanding of language model driven agents , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , year=

  33. [41]

    Jin, Bowen and Zeng, Hansi and Yue, Zhenrui and Yoon, Jinsung and Arik, Sercan and Wang, Dong and Zamani, Hamed and Han, Jiawei , journal=

  34. [42]

    Wu, Jialong and Yin, Wenbiao and Jiang, Yong and Wang, Zhenglin and Xi, Zekun and Fang, Runnan and Zhang, Linhai and He, Yulan and Zhou, Deyu and Xie, Pengjun and others , booktitle=

  35. [43]

    Narasimhan and Yuan Cao , title =

    Shunyu Yao and Jeffrey Zhao and Dian Yu and Nan Du and Izhak Shafran and Karthik R. Narasimhan and Yuan Cao , title =. International Conference on Learning Representations,

  36. [44]

    Nakano, Reiichiro and Hilton, Jacob and Balaji, Suchir and Wu, Jeff and Ouyang, Long and Kim, Christina and Hesse, Christopher and Jain, Shantanu and Kosaraju, Vineet and Saunders, William and others , journal=

  37. [45]

    Zhou, Shuyan and Xu, Frank F and Zhu, Hao and Zhou, Xuhui and Lo, Robert and Sridhar, Abishek and Cheng, Xianyi and Ou, Tianyue and Bisk, Yonatan and Fried, Daniel and others , booktitle=

  38. [46]

    Deng, Xiang and Gu, Yu and Zheng, Boyuan and Chen, Shijie and Stevens, Sam and Wang, Boshi and Sun, Huan and Su, Yu , journal=

  39. [47]

    International Conference on Learning Representations,

    Towards understanding sycophancy in language models , author=. International Conference on Learning Representations,

  40. [48]

    Proceedings of the 2021 conference on empirical methods in natural language processing , year=

    Entity-based knowledge conflicts in question answering , author=. Proceedings of the 2021 conference on empirical methods in natural language processing , year=

    2021

  41. [49]

    Knowledge conflicts for

    Xu, Rongwu and Qi, Zehan and Guo, Zhijiang and Wang, Cunxiang and Wang, Hongru and Zhang, Yue and Xu, Wei , booktitle=. Knowledge conflicts for

  42. [50]

    Findings of the association for computational linguistics: EMNLP 2023 , year=

    On the risk of misinformation pollution with large language models , author=. Findings of the association for computational linguistics: EMNLP 2023 , year=

    2023

  43. [51]

    Zou, Wei and Geng, Runpeng and Wang, Binghui and Jia, Jinyuan , booktitle=

  44. [52]

    Proceedings of the AAAI Conference on Artificial Intelligence , year=

    Benchmarking large language models in retrieval-augmented generation , author=. Proceedings of the AAAI Conference on Artificial Intelligence , year=

  45. [53]

    The power of noise: Redefining retrieval for

    Cuconasu, Florin and Trappolini, Giovanni and Siciliano, Federico and Filice, Simone and Campagnano, Cesare and Maarek, Yoelle and Tonellotto, Nicola and Silvestri, Fabrizio , booktitle=. The power of noise: Redefining retrieval for

  46. [54]

    Asai, Akari and Wu, Zeqiu and Wang, Yizhong and Sil, Avi and Hajishirzi, Hannaneh , booktitle=

  47. [55]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , year=

    Enabling large language models to generate text with citations , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , year=

    2023

  48. [56]

    International conference on learning representations,

    Large language models cannot self-correct reasoning yet , author=. International conference on learning representations,

  49. [57]

    Advances in neural information processing systems,

    Chain-of-thought prompting elicits reasoning in large language models , author=. Advances in neural information processing systems,

  50. [58]

    Advances in neural information processing systems,

    Reflexion: Language agents with verbal reinforcement learning , author=. Advances in neural information processing systems,

  51. [59]

    ACM computing surveys , volume=

    Survey of hallucination in natural language generation , author=. ACM computing surveys , volume=. 2023 , publisher=

    2023

  52. [60]

    Min, Sewon and Krishna, Kalpesh and Lyu, Xinxi and Lewis, Mike and Yih, Wen-tau and Koh, Pang and Iyyer, Mohit and Zettlemoyer, Luke and Hajishirzi, Hannaneh , booktitle=

  53. [61]

    Madaan, Aman and Tandon, Niket and Gupta, Prakhar and Hallinan, Skyler and Gao, Luyu and Wiegreffe, Sarah and Alon, Uri and Dziri, Nouha and Prabhumoye, Shrimai and Yang, Yiming and others , journal=

  54. [62]

    Chen, Zijian and Ma, Xueguang and Zhuang, Shengyao and Nie, Ping and Zou, Kai and Liu, Andrew and Green, Joshua and Patel, Kshama and Meng, Ruoxi and Su, Mingyi and others , journal=

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.