Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

ManuSearch: Democratizing Deep Search in Large Language Models with a Transparent and Open Multi-Agent Framework

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ManuSearch, a transparent three-agent deep-search system built on open 32B models, claims to outperform prior open-source search agents and leading closed-source products like GPT-4o Search Preview and Perplexity Sonar Reasoning Pro on…

desk verdict Useful benchmark and a sensible modular design, but the abstract overstates what Table 2 shows; the open-source comparison needs a common evaluation protocol before the headline claims hold. read the letter →

arxiv 2505.18105 v1 pith:JZ3LLYGG submitted 2025-05-23 cs.CL

Lisheng Huang , Yichen Liu , Jinhao Jiang , Rongxiang Zhang , Jiahao Yan , Junyi Li , Wayne Xin Zhao This is my paper
classification cs.CL
keywords ManuSearchdeepmulti-agentLLMopen-sourcelong-tailentitiesORIONbenchmarkweb-augmentedreasoningReAct
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that answering complex questions with live web search—deep search—does not require a proprietary black box. It presents ManuSearch, a modular multi-agent system that splits the job into a solution planning agent, an internet search agent, and a structured webpage reading agent, all running on open 32B models. On FRAMES, GAIA, and a new benchmark called ORION, ManuSearch reports higher accuracy than prior open-source search agents and beats several closed-source search products, including GPT-4o Search Preview and Perplexity Sonar Reasoning Pro. ORION itself is a 310-question bilingual benchmark built around long-tail entities, on which even leading closed systems score below 30 percent. If the comparisons hold, the practical consequence is that an open research group can replicate near-frontier deep search with components it can inspect and modify.

What carries the argument

The load-bearing mechanism is a three-agent decomposition built on the ReAct planning-and-acting loop. A solution planning agent holds a memory container of previous sub-questions and answers, generates the next sub-query, and stops to answer only when its accumulated evidence is sufficient. A tool-augmented internet search agent runs multi-round searches, calling a WebSearch tool that returns the top-K pages and an Answer Question tool that produces the sub-answer from the extracted content. A structured webpage reading agent strips HTML to clean text and extracts task-relevant passages guided by a search intent produced by the searcher. The paper's final configuration uses 'full reading'—the reader processes all retrieved pages rather than selecting a few—which it reports as both simpler and more accurate than selective reading.

What would settle it

Re-run the asterisked baselines—ODS-v2+DeepSeek-R1, Search-o1-32B, WebThinker-32B-Base, and SimpleDeepSearcher—under ManuSearch's exact protocol (same search API, same evaluator prompt, same answer format, same GAIA subset) on FRAMES, GAIA, and ORION; if ManuSearch no longer beats them, the claimed advantage over prior open-source systems fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that decoupling deep search into three cooperating agents lets 32B open models reach or exceed the accuracy of much larger proprietary systems. The solution planning agent iteratively decomposes the user's question into sub-questions and only produces the final answer once it judges the evidence sufficient; the tool-augmented internet search agent issues searches and sub-answers through a web API; and the structured webpage reading agent converts messy HTML into clean text and extracts the evidence relevant to the current search intent. In the paper's experiments, the DeepSeek-R1-plus-DeepSeek-V3 configuration scores 71.8 percent on FRAMES and 47.6 percent on GAIA, while the QwQ-32B configuration scores 68.4 percent on FRAMES and 46.6 percent on GAIA—numbers the paper reports as surpassing every closed-source search system tested. The paper attributes these results to modularity: each agent works within its capability, and the whole pipeline is traceable.

Load-bearing premise

The asterisked baseline numbers in Table 2 were taken from other studies, and the paper assumes these are directly comparable to its own runs even though search APIs, judge prompts, answer formats, and possibly evaluation subsets differ.

Editorial extensions

If this is right

  • Any LLM, from 32B open models to large closed reasoning models, can be dropped into ManuSearch and show large accuracy improvements over the same model answering without search.
  • Each stage of the pipeline is traceable: a user can inspect the planner's sub-questions, the exact queries issued to the search engine, and the evidence the final answer relies on.
  • ORION provides a reusable bilingual benchmark, with source-grounded answers and five predefined reasoning patterns, for open-web reasoning over long-tail entities.
  • A 32B-parameter configuration can match or beat closed-source search products, narrowing the practical gap between open and proprietary deep search.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to swap the web search tool for a domain-specific index (scientific literature, legal records, private corpora) without retraining the agents, since the planner, searcher, and reader communicate only through text.
  • If the results replicate under an identical evaluation protocol, the implication is that architectural decomposition, not model scale, drives deep-search quality—a claim the paper supports but does not isolate.
  • The full-reading result suggests that for open web data, missing a key page hurts accuracy more than processing noisy pages, an effect that may carry over to other agentic retrieval systems.
  • The traceable chain of sub-questions would allow targeted interventions, such as providing feedback to the planner when the searcher returns unsatisfying evidence, which the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes ManuSearch, a transparent multi-agent deep search framework with three cooperative agents: a solution planning agent, a tool-augmented internet search agent, and a structured webpage reading agent. It also introduces ORION, a 310-question benchmark for open-web reasoning over long-tail entities in English and Chinese, with human-verified answers and source URLs. The authors evaluate ManuSearch on FRAMES, ORION, and GAIA using Qwen and DeepSeek backbones, and claim that it substantially outperforms prior open-source deep search systems and surpasses leading closed-source search systems. Code and data are promised at a public repository.

Significance. If the headline result were solid, the contribution would be significant: an open, modular, interpretable alternative to proprietary deep search that runs on 32B open models, plus a new benchmark targeting long-tail reasoning. The three-agent decomposition is simple, the framework requires no additional training or fitted parameters, and validation on independent FRAMES and GAIA benchmarks in addition to the authors' ORION is a strength. The release of code and data supports reproducibility. However, the central outperformance claim is not currently supported by the paper's own Table 2: the best open-source baseline (ODS-v2+DeepSeek-R1) is ahead on FRAMES, SimpleDeepSearcher ties on GAIA, and the asterisked baseline numbers come from other studies without evidence of protocol equivalence. The significance is therefore conditional on a fair, common-protocol re-evaluation.

major comments (4)
  1. [Abstract; Section 5.2, Table 2] The abstract's claim that ManuSearch 'substantially outperforms prior open-source baselines' is not supported by the paper's own Table 2: ManuSearch-R1-V3 scores 71.8 on FRAMES, below ODS-v2+DeepSeek-R1's 75.3, and ties SimpleDeepSearcher at 47.6 on GAIA, while on GAIA Level 3 it scores 8.3 versus 16.7 for both Search-o1 and SimpleDeepSearcher. The body text itself hedges to 'nearly matching' and 'almost matching.' The abstract should be revised to match the actual results, or the results should be re-established under a common evaluation protocol.
  2. [Section 5.1, Table 2] The asterisked baseline numbers in Table 2 are collected from other studies, and the paper does not establish that those studies used the same evaluation protocol as ManuSearch: the same GAIA subset, the same ChatGPT-4o judge prompt, the same answer-format instructions, or the same search API and top-k retrieval settings. This comparability concern applies to both the open-source baselines and the closed-source systems with asterisks (e.g., GPT-4o Search Preview, Perplexity Sonar Reasoning Pro). Because the open-source comparison is the paper's central claim, the authors should re-run the open-source baselines under the identical protocol used for ManuSearch, or explicitly document protocol differences and demonstrate robustness to them.
  3. [Section 5.1, Implementation Details] The implementation description states that 'when the ManuSearch fails to provide an answer, we switch to the direct resoning mode to generate the final answer.' This means the reported accuracy for ManuSearch is a mixture of the multi-agent pipeline and vanilla direct LLM reasoning. The paper does not report the frequency of fallback or analyze its contribution to the results in Tables 2-5, so it is unclear what fraction of the reported performance is attributable to the search framework itself. The fallback mechanism should be quantified, or the system should be evaluated without it.
  4. [Section 5.2, Table 2] Table 2 reports no open-source search-system baselines on ORION; the ORION columns for ODS, Search-o1, WebThinker, and SimpleDeepSearcher are empty. Consequently, the paper's claim that ManuSearch 'significantly outperforms prior open-source systems' is untested on the authors' own benchmark. Open-source baselines should be evaluated on ORION under the same protocol, or the claim should be restricted to FRAMES and GAIA.
minor comments (6)
  1. [Section 5.1] There is a typo in Section 5.1: 'direct resoning mode' should be 'direct reasoning mode.'
  2. [Table 2 caption] The caption states 'Boldfonts indicate the best performance among open-source models,' but ODS-v2+DeepSeek-R1's 75.3 on FRAMES is not bold, while the bolded ManuSearch rows do not achieve the best open-source FRAMES score. The bold marking and the caption should be corrected to consistently indicate the best open-source result in each column.
  3. [Section 5.1] The hyperparameters (max sub-queries per search = 3, max web text length = 64K, top-K = 5) are stated only in prose; a hyperparameter table would improve reproducibility.
  4. [Sections 4.3 and 5.1] Section 4.3 describes the search agent as iteratively searching 'until the retrieved information is sufficient,' but Section 5.1 fixes the maximum number of sub-queries per search to 3. Please clarify the stopping criterion and whether the limit applies per sub-question or per search call.
  5. [Tables 1 and 2] Table 1 reports results for 'Kimi Exploration Edition' with an ORION overall accuracy of 17.1, while Table 2 lists 'Kimi Search' with a different ORION overall accuracy of 12.8. Please clarify whether these are different products or different evaluation runs to avoid confusion.
  6. [Section 5.3, Table 4] The statement that 'Full Reading achieves better overall performance' is not supported for the MS-QwQ-Qwen configuration: Selective Reading attains a higher GAIA average (44.7 vs 42.7). The claim should be restricted to the configurations where it holds, or the analysis should be reported per configuration more carefully.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the framework is not fitted to any benchmark and the central comparisons rest on external evaluation runs.

full rationale

ManuSearch is a modular ReAct-style agent system with no parameters trained or fitted to FRAMES, GAIA, or ORION; its outputs are generated by fixed planning/search/reading loops over open-weight backbones, so no reported number is a re-statement of a fitted input. The main results are independently obtained on external benchmarks FRAMES and GAIA, and ORION—though author-created—was constructed by LLM-assisted generation with human verification before evaluation and was not used to tune the framework. The only self-citation is R1-Searcher (Song et al., 2025), referenced descriptively in Related Work and Section 4.1 to contrast with the proposed decoupled design; it is not invoked as a uniqueness theorem, a fitted input, or as evidence for the central claim, so it does not make the argument circular. The asterisked open-source baseline numbers are imported from other studies and may differ in evaluation protocol; that is a comparability/evidence-quality concern about the headline claim, but it is not a circularity because ManuSearch's own scores are produced by the stated pipeline rather than derived from those baselines. No step in the paper's derivation reduces by construction to its own input.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No theoretical free parameters or invented entities are needed for this empirical system. The main hand-chosen numbers are implementation hyperparameters. ORION is a dataset artifact, not a postulated entity requiring independent evidence. The load-bearing assumptions are evaluation comparability, judge reliability, and the direct-reasoning fallback.

free parameters (3)
  • top_K webpages per search = 5
    Section 5.1 sets top-5 relevant webpages retrieved per WebSearch tool call; retrieval volume affects answer availability.
  • max_sub_queries_per_search = 3
    Section 5.1 sets a maximum of 3 sub-queries per search call, limiting the breadth of the search process.
  • max_webpage_text_length = 64K
    Section 5.1 sets the maximum original web text length to 64K tokens, truncating pages before the reading agent processes them.
assumptions (5)
  • domain assumption Google Web Search API top-K results contain sufficient evidence to answer ORION and FRAMES/GAIA sub-questions.
    Section 4.3 assumes the returned document set D is enough for the search agent to iterate and answer each sub-question.
  • domain assumption ChatGPT-4o as LLM judge reliably scores correctness for all three benchmarks.
    Section 5.1 says LLM-as-Judge with ChatGPT-4o is used for Pass@1 accuracy, with no human agreement or calibration reported.
  • domain assumption Asterisked baseline results collected from other studies are comparable to in-house runs.
    Table 2 marks results collected from other studies with asterisks; comparability across search engines, judge prompts, and answer formats is assumed.
  • domain assumption ORION human annotations are factually correct and fully sourced.
    Section 3.2 says human annotators verify each question and record reasoning chains, but no inter-annotator agreement is reported.
  • ad hoc to paper Fallback to direct reasoning when ManuSearch fails does not bias reported results.
    Section 5.1 switches to direct reasoning when ManuSearch fails to provide an answer; this rule is introduced by the authors and its frequency is not reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ManuSearch: Democratizing Deep Search in Large Language Models with a Transparent and Open Multi-Agent Framework." pith.science (2026). https://pith.science/paper/JZ3LLYGG

@misc{pith2026250518105,
  author       = {Pith},
  title        = {Pith review of: ManuSearch: Democratizing Deep Search in Large Language Models with a Transparent and Open Multi-Agent Framework},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JZ3LLYGG}},
  note         = {Machine review of arXiv:2505.18105}
}
read the original abstract

Recent advances in web-augmented large language models (LLMs) have exhibited strong performance in complex reasoning tasks, yet these capabilities are mostly locked in proprietary systems with opaque architectures. In this work, we propose \textbf{ManuSearch}, a transparent and modular multi-agent framework designed to democratize deep search for LLMs. ManuSearch decomposes the search and reasoning process into three collaborative agents: (1) a solution planning agent that iteratively formulates sub-queries, (2) an Internet search agent that retrieves relevant documents via real-time web search, and (3) a structured webpage reading agent that extracts key evidence from raw web content. To rigorously evaluate deep reasoning abilities, we introduce \textbf{ORION}, a challenging benchmark focused on open-web reasoning over long-tail entities, covering both English and Chinese. Experimental results show that ManuSearch substantially outperforms prior open-source baselines and even surpasses leading closed-source systems. Our work paves the way for reproducible, extensible research in open deep search systems. We release the data and code in https://github.com/RUCAIBox/ManuSearch

Figures

Figures reproduced from arXiv: 2505.18105 by the authors.

Figure 1
Figure 1. Distribution of domains in ORION. AI Systems Chinese English Overall Kimi Exploration Edition 14.7 20.0 17.1 Doubao Search 23.5 30.7 26.8 Qwen2.5-Max Search 20.0 20.7 20.3 [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The overall framework of our proposed ManuSearch. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. A case of ManuSearch from GAIA benchmark. [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Fetch-then-Explore: Decoupling Selection from Extraction over a Persistent Workspace for Search Agents

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Fetch-then-Explore, which stores fetched pages in a per-question workspace and extracts evidence on demand with grep/read, beats visit-and-read and browsing baselines on BrowseComp across three LLM backbones.

  2. HierSearch: A Hierarchical Enterprise Deep Search Framework Integrating Local and Web Searches

    cs.IR 2025-08 conditional novelty 6.0 of 10

    Hierarchical reinforcement learning with separate local and web deep search agents, plus an evidence refiner, beats flat RL and single-source deep search baselines on multi-source QA benchmarks.

Reference graph

Works this paper leans on

22 extracted references · 4 canonical work pages · cited by 2 Pith papers

  1. [1]

    Salaheddin Alzubi, Creston Brooks, Purva Chiniya, Edoardo Contente, Chiara von Gerlach, Lucas Irwin, Yihan Jiang, Arda Kaz, Windsor Nguyen, Sewoong Oh, and 1 others. 2025. Open deep search: Democratizing search with open-source reasoning agents. arXiv preprint arXiv:2503.20201

    arXiv 2025

  2. [2]

    Anthropic-AI. 2025. https://www.anthropic.com/news/claude-3-7-sonnet Claude 3.7 sonnet and claude code

    work page 2025

  3. [3]

    Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Fan Yang, Zenan Zhou, Weipeng Chen, Haofen Wang, Jeff Z Pan, and 1 others. 2025. Learning to reason with search for llms via reinforcement learning. arXiv preprint arXiv:2503.19470

    arXiv 2025

  4. [4]

    Zehui Chen, Kuikun Liu, Qiuchen Wang, Jiangning Liu, Wenwei Zhang, Kai Chen, and Feng Zhao. 2024. Mindsearch: Mimicking human minds elicits deep ai searcher. arXiv preprint arXiv:2407.20183

    arXiv 2024

  5. [5]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    arXiv 2025

  6. [6]

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, and 1 others. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720

    arXiv 2024

  7. [7]

    Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, and Manaal Faruqui. 2024. Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation. arXiv preprint arXiv:2409.12941

    arXiv 2024

  8. [8]

    Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. 2025 a . https://doi.org/10.48550/ARXIV.2501.05366 Search-o1: Agentic search-enhanced large reasoning models . CoRR, abs/2501.05366

    arXiv doi:10.48550/arxiv.2501.05366 2025

Show all 22 references
  1. [9]

    Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji - Rong Wen, and Zhicheng Dou. 2025 b . https://doi.org/10.48550/ARXIV.2504.21776 Webthinker: Empowering large reasoning models with deep research capability . CoRR, abs/2504.21776

    2025 doi arXiv

  2. [10]

    Meta-AI. 2025. https://ai.meta.com/blog/llama-4-multimodal-intelligence/ The llama 4 herd: The beginning of a new era of natively multimodal ai innovation

    2025

  3. [11]

    OpenAI. 2025 a . https://platform.openai.com/docs/models/gpt-4o-search-preview Gpt-4o search preview

    2025

  4. [12]

    OpenAI. 2025 b . https://openai.com/index/gpt-4-1/ Introducing gpt-4.1 in the api

    2025

  5. [13]

    Perplexity-AI. 2025. https://www.perplexity.ai/hub/blog/introducing-the-sonar-pro-api Introducing the sonar pro api

    2025

  6. [14]

    Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, and 1 others. 2024. O1 replication journey: A strategic progress report--part 1. arXiv preprint arXiv:2410.18982

    2024 arXiv

  7. [15]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling

    2024

  8. [16]

    Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. 2025. R1-searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.05592

    2025 arXiv

  9. [17]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR)

    2023

  10. [18]

    Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejun Ma, and Junxian He. 2025. 7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient. https://hkust-nlp.notion.site/simplerl-reason. Notion Blog

    2025

  11. [19]

    Yuxiang Zhang, Shangxi Wu, Yuqi Yang, Jiangming Shu, Jinlin Xiao, Chao Kong, and Jitao Sang. 2024. https://doi.org/10.48550/ARXIV.2412.00154 o1-coder: an o1 replication for coding . CoRR, abs/2412.00154

    2024 doi arXiv

  12. [20]

    Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. 2025. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments. arXiv preprint arXiv:2504.03160

    2025 arXiv

  13. [21]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  14. [22]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.