Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Dr. Zero: Self-Evolving Search Agents without Training Data

T0 review · 4 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that search agents can be trained with zero human-annotated data by having one model write increasingly hard multi-hop questions and another learn to solve them, matching or beating supervised baselines on open-domain QA b

desk verdict Solid, well-engineered proposer-solver self-evolution for search agents, but the 'surpasses' framing overreaches and the proposer's source-document sampling is a load-bearing unknown. read the letter →

arxiv 2601.07055 v2 pith:NBYZLVZC submitted 2026-01-11 cs.AI

Zhenrui Yue , Kartikeya Upasani , Xianjun Yang , Suyu Ge , Shaoliang Nie , Yuning Mao , Zhe Liu , Dong Wang This is my paper · ORCID
classification cs.AI
keywords self-evolutionsearchagentsreinforcementlearningmulti-hopquestionansweringsyntheticdatagenerationpolicyoptimizationtooluseopen-domainQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Dr. Zero asks whether a search agent can improve itself without any curated questions or answers, using only a search engine as its environment. The paper's answer is yes: a proposer model generates diverse, multi-hop synthetic questions, a solver is trained on them with group relative policy optimization, and the solver's pass rate feeds back to push the proposer toward harder but still solvable questions. A hop-grouped variant of policy optimization (HRPO) makes this loop computationally tractable by grouping questions of the same hop count instead of sampling many responses per prompt. On several open-domain QA benchmarks, the resulting data-free agent matches or exceeds supervised search agents, especially at the smaller model size, suggesting complex evidence-grounded reasoning can emerge without human supervision.

What carries the argument

The load-bearing machinery is the proposer-solver feedback loop. The proposer is prompted with a source document and a hop count and must produce a deterministic question whose reasoning chain starts in that document; its reward is a pass-rate-shaped difficulty reward, maximized when the solver gets exactly one of n attempts correct, plus a format reward for well-formed search and reasoning traces. The solver is trained on these synthetic QA pairs with GRPO. To make proposer training cheap, HRPO replaces nested sampling with hop-grouped advantage estimation: questions sharing the same number of hops form the baseline group, so each prompt needs only one generated question and a handful of so

What would settle it

Take the exact Dr. Zero pipeline but sample the initial source documents from a corpus that is provably disjoint from the seven evaluation benchmarks, or deduplicate all generated questions against NQ, TriviaQA, PopQA, HotpotQA, 2WikiMQA, MuSiQue, and Bamboogle; if the margin over supervised baselines shrinks or reverses, the apparent data-free gains are partly benchmark overlap rather than pure self-evolution.

Watch

Extended reading notes

Core claim

The paper's central claim is that a proposer-solver self-evolution loop, seeded only with documents from an external search corpus, can produce a search agent on par with agents trained on human-written questions and answers. The proposer is rewarded not for easy or impossible questions but for questions the current solver can answer exactly once out of several attempts; this keeps the curriculum at the edge of the solver's ability. The solver is trained with GRPO on the generated questions, and the proposer with HRPO, which estimates advantages by standardizing rewards within groups of questions that need the same number of reasoning hops. The result is an automated curriculum that co-evolv

Load-bearing premise

The whole data-free story rests on how the initial source documents that seed each proposer-generated question are chosen; the paper does not report the sampling procedure, and its own ablation shows that removing these seeds cuts average performance from 0.304 to 0.245.

Editorial extensions

If this is right

  • A search agent can be trained without human-written questions, answers, or demonstrations, using only an external search engine and a base language model.
  • As the solver improves, the proposer is automatically pushed to generate harder questions, producing a curriculum that keeps challenging the solver.
  • HRPO cuts proposer training compute to about a quarter of nested-sampling GRPO while maintaining or slightly improving average performance.
  • Gains concentrate early: performance peaks within about 50 solver steps and after two to three self-evolution iterations, after which additional training plateaus or degrades.
  • On the reported benchmarks, the 3B data-free agent surpasses a supervised search-agent baseline on single-hop datasets; the 7B version approaches it and beats it on one multi-hop dataset.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves open how the initial source document for each generated question is sampled; since removing it causes the largest single drop in the ablation (0.304 to 0.245), seed selection is a hidden variable worth isolating before accepting the 'data-free' framing.
  • If seed documents could themselves be proposed by the model, the pipeline would be fully closed-loop; the current design still depends on an external choice of starting text.
  • HRPO's idea of grouping trajectories by structural difficulty instead of sampling many responses likely transfers to other multi-turn tool-use RL settings, such as code repair or web navigation, where rollouts are expensive.
  • The framework entangles 'skill' with the search engine's corpus: the same loop on a smaller or biased index would yield different curricula, so results are partly a property of the knowledge environment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Dr. Zero, a data-free self-evolution framework for multi-turn search agents. A proposer model generates structurally diverse multi-hop QA pairs, each grounded in an initial source document, and a solver model is trained on those pairs via GRPO. The proposer is then updated with a difficulty-guided reward and hop-grouped relative policy optimization (HRPO), using solver pass rates to encourage questions that are challenging but solvable. Both agents are initialized from the same base LLM, and the only external knowledge comes from a search engine over an English Wikipedia index. Experiments on NQ, TriviaQA, PopQA, HotpotQA, 2WikiMQA, MuSiQue, and Bamboogle with Qwen2.5-3B/7B report that Dr. Zero roughly matches or exceeds supervised Search-R1 on several one-hop datasets and is competitive on multi-hop datasets, while using no human-annotated training data. The paper claims this demonstrates that complex search and reasoning can emerge solely through self-evolution.

Significance. If the empirical claims are supported, this is a significant result: it would show that agentic search and evidence-grounded reasoning can be bootstrapped without curated QA labels, and that the proposer-solver loop plus hop-grouped advantage estimation can reduce the compute cost of self-evolution. The paper deserves credit for releasing code, reporting detailed ablations (format reward, reward shaping, training steps, hop ratios), providing honest failure examples, and comparing against data-free baselines. However, the strength of the central claim is not yet fully secured: the source-document sampling procedure is unspecified, no contamination analysis is reported, and the aggregate numbers in Table 1 do not actually show that Dr. Zero surpasses the supervised Search-R1 baseline. These issues are central to the 'data-free' and 'matches or surpasses' narrative, so the current evidence is promising but conditional.

major comments (4)
  1. [Appendix A / Figure 5 / Table 7] The proposer prompt in Figure 5 requires a source document ('{document}') as the starting point for every generated question, but the paper never states how these documents are selected or sampled. Table 7 shows that removing the initial document is the largest single ablation effect (average drops from 0.304 to 0.245, about 19% relative). Since the search engine and evaluation corpus are both described as an English Wikipedia dump, and the evaluation datasets (NQ, TriviaQA, PopQA, HotpotQA, 2WikiMQA, MuSiQue, Bamboogle) are all Wikipedia-derived, a real risk exists that the proposer is being given relevant Wikipedia articles for entities or topics that also appear in the test set. Without an overlap analysis between source documents / generated questions and the benchmark questions, the comparison to supervised Search-R1 is not yet protected against this confound. Please specify the doc
  2. [Abstract / Table 1] The abstract and introduction say Dr. Zero 'matches or surpasses' fully supervised search agents. The average scores in Table 1 do not support the 'surpasses' part: on Qwen2.5-3B, Dr. Zero averages 0.326 vs. Search-R1's 0.327; on Qwen2.5-7B, 0.372 vs. 0.384. Dr. Zero wins on NQ, TriviaQA, PopQA, and 2WikiMQA (7B) but consistently loses on HotpotQA, MuSiQue, Bamboogle, and on the overall average. The paper should reframe the central claim to 'matches' on one-hop, 'competitive but below' on multi-hop, and clearly discuss the multi-hop gap rather than claiming parity across the board.
  3. [Table 9] The caption says bold indicates 'statistically significant' gains, but the table only gives Dr. Zero's mean and standard deviation over what appears to be three runs, while the baselines are single point estimates with no variance. No test statistic or p-value is reported. A proper paired significance test (e.g., paired bootstrap over evaluation examples or a signed test) or confidence intervals for all methods is needed before claiming statistical significance. For example, the 7B MuSiQue result (0.112 vs. 0.168) is numerically worse, so the selective bold marking without a formal test is misleading.
  4. [Appendix B / Table 8] The appendix concludes that HRPO 'achieves superior performance' compared with GRPO (0.326 vs. 0.320 average). However, GRPO is better on HotpotQA (0.303 vs. 0.298), MuSiQue (0.100 vs. 0.091), and Bamboogle (0.272 vs. 0.200); HRPO's advantage comes entirely from the one-hop datasets. Given the paper's emphasis on multi-hop reasoning, the appropriate conclusion is that HRPO gives comparable aggregate performance at much lower compute, not that it is superior. Also, the compute comparison counts 1×5 solver rollouts for HRPO vs. 4×4 for GRPO, but it should state whether proposer-side search rollouts are included on both sides.
minor comments (5)
  1. [Appendix A / References] Appendix A says the base model is used as the 'generative reward (Yang et al., 2024)', but the reference list does not contain a 2024 Yang et al. paper about generative reward; the likely intended citation is Yuan et al. (2024) on self-rewarding LLMs. Please fix the citation.
  2. [Figure 3] The caption contains a typo: 'proser' should be 'proposer'.
  3. [Figure 9] The proposer example for hop=3 claims Robert Harvey is the answer to a question about a Melbourne Demons player in the 1988 and 2000 AFL Grand Finals. Robert Harvey played for St Kilda, not Melbourne, so the generated QA pair contains a factual error. If such errors are not rare, the synthetic training signal could contain false premises. Please either correct the example or acknowledge that this is a failure case.
  4. [Section 3.3 / Eq. (5)] The GRPO objective is written with a min of the clipped and unclipped ratio inside an expectation over n samples, which is standard, but the notation is dense. A short explanation of how the group baseline is computed in Eq. (5) would improve readability.
  5. [Section 4.1] The paper states that all baselines are evaluated with the same search engine and corpus but does not give the retrieval corpus size or how documents are chunked. Reporting these details would help reproducibility and would also clarify the contamination question.

Circularity Check

0 steps flagged · score 2.0 of 10

No derived benchmark score is equivalent to its training inputs; the proposer-solver loop is externally evaluated. Minor self-citations are contextual; the source-document sampling gap is a contamination risk, not a circular fit.

full rationale

The paper's derivation chain is a genuine co-evolution loop: the proposer is rewarded by Eq. (4) through the solver's pass rate on synthetic questions, and the solver is trained by Eq. (5) with exact-match rewards against the proposer's synthesized answers. Neither reward is defined in terms of the seven evaluation benchmarks (NQ, TriviaQA, PopQA, HotpotQA, 2WikiMQA, MuSiQue, Bamboogle), so the reported EM scores are external, held-out outcomes rather than quantities fitted into the training objective. The `difficulty-guided reward` acts as an adaptive curriculum, not as a benchmark-derived target. Self-citations appear only contextually: the retrieval setup follows `(Yue et al., 2025; Jin et al., 2025)` and Search-R1 is used as a supervised baseline; these citations do not supply the load-bearing premise that data-free self-evolution works. One genuine methodological gap must be weighed: Appendix B reports that removing the initial document drops the 3B average from 0.304 to 0.245 (`The most significant performance drop occurs when the initial document is removed`), and Figure 5 shows the proposer prompt is seeded with `{document}`. The paper never states how these source documents are selected or whether they are deduplicated against the English Wikipedia dump used for retrieval and evaluation. If the documents come from that same dump and overlap benchmark topics, evaluation scores could be inflated. This is a contamination/correctness risk, not circularity: no equation or fitted parameter reduces the benchmark results to the training inputs.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method depends on standard RL/self-play assumptions plus several manually chosen design parameters. The most notable unexamined assumption is the source of the proposer's initial documents, which is load-bearing because the ablation shows large degradation without it. No new physical or mathematical entities are introduced.

free parameters (5)
  • difficulty-reward sample count n = 5
    Number of solver rollouts per generated question; controls the I(0<k<n) reward and HRPO/GRPO variance. Chosen by the authors.
  • hop ratio for proposer questions = 4:3:2:1
    Distribution of 1/2/3/4-hop synthetic questions; Table 4 shows it changes downstream performance, so the default ratio is a tuned modeling choice.
  • format reward weights = 0.5 max
    Sum of four structural requirements for well-formed outputs; weights were set by hand and validated via ablatio.
  • KL regularization coefficients = 0 (HRPO), 0/0.001 (GRPO)
    Authors state they conducted a minimal search over max gradient norm and KL coefficients to maintain stability.
  • training steps and iterations = 50 steps x 3 iterations
    Training stops when performance peaks or plateaus; the stopping rule is based on observed dynamics, so it is a practical free choice.
assumptions (5)
  • domain assumption External search engine returns relevant documents for synthetic queries.
    The whole method assumes multi-turn search tool gives useful evidence; Appendix A describes ANN retrieval over an E5-indexed Wikipedia dump.
  • domain assumption Proposer-generated answers are reliable enough as training labels.
    The solver is trained by GRPO against answers y produced by the proposer without independent verification beyond a format reward.
  • domain assumption Exact match on the final answer is a sufficient reward signal.
    Both proposer and solver use exact match as the outcome; paraphrased or partially correct answers receive zero credit.
  • ad hoc to paper Initial source documents are available without human curation and are representative.
    The proposer prompt requires a source document; Table 7 shows removing it drops average performance from 0.304 to 0.245, but the paper never states how these documents are sampled or selected.
  • domain assumption Base model initialization carries enough priors for search-agent behavior.
    The data-free claim depends on the pretrained checkpoint already knowing how to follow tool-use instructions and reason over retrieved passages.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dr. Zero: Self-Evolving Search Agents without Training Data." pith.science (2026). https://pith.science/paper/NBYZLVZC

@misc{pith2026260107055,
  author       = {Pith},
  title        = {Pith review of: Dr. Zero: Self-Evolving Search Agents without Training Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NBYZLVZC}},
  note         = {Machine review of arXiv:2601.07055}
}
read the original abstract

As high-quality data becomes increasingly difficult to obtain, self-evolution without curated training data has emerged as a promising paradigm. This approach allows large language models (LLMs) to autonomously generate and solve complex problems, thereby improving their reasoning capabilities. However, multi-turn search agents struggle in this setting due to limited question diversity and the substantial compute required for multi-step reasoning and tool use. In this work, we introduce Dr. Zero, a framework that enables search agents to effectively self-evolve without human-annotated training data, relying solely on an external search engine as their knowledge environment. In particular, we design a self-evolution feedback loop where a proposer generates structurally diverse questions to train a solver initialized from the same base model. As the solver evolves, it incentivizes the proposer to produce increasingly difficult yet solvable tasks, thus establishing an automated curriculum to refine both agents. To enhance training efficiency, we also introduce hop-grouped relative policy optimization (HRPO). This method clusters structurally similar questions to construct group-level baselines, effectively minimizing the sampling overhead in evaluating each query's individual difficulty and solvability. Consequently, HRPO significantly reduces the compute requirements for proposer training and reward estimation without compromising performance or stability. Extensive experimental results demonstrate that Dr. Zero matches or surpasses fully supervised search agents on several question answering benchmarks, showing that strong agentic search and evidence-grounded reasoning can emerge solely through self-evolution.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Ouroboros-Spatial: Closing the Data-Model Loop for Spatial Reasoning

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    A self-evolving training loop that generates its own spatial QA data with executable code and difficulty feedback lifts Qwen3-VL-4B/8B to 62.7/63.3 on VSI-Bench using an order of magnitude less data.

Reference graph

Works this paper leans on

15 extracted references · cited by 1 Pith paper

  1. [1]

    Hop 1 is the starting entity found in the document

    Hop: A node in the reasoning chain. Hop 1 is the starting entity found in the document. Hop n is the final answer. ### Inputs

  2. [2]

    n: the exact number of hops in the reasoning chain (requiring n-1 searches)

  3. [3]

    ### Process & Tools

    Source document: the full source text. ### Process & Tools

  4. [4]

    - Select a specific entity, event or detail explicitly mentioned in the text

    Analyze the Document and Select the Starting Point - Read and analyze the source document. - Select a specific entity, event or detail explicitly mentioned in the text. This entity becomes Hop 1 (the initial clue)

  5. [5]

    The result is Hop 2.,→ - Iterate: Continue connecting the current Hop i to the next Hop i+1 using deterministic, verifiable relation found via search

    Design the Chain Forwards - From Hop 1 to Hop 2: Identify a factual attribute or relation of Hop 1 that is NOT in the text but can be found via search. The result is Hop 2.,→ - Iterate: Continue connecting the current Hop i to the next Hop i+1 using deterministic, verifiable relation found via search. - Stop at Hop n: Continue this process until you have ...

  6. [6]

    </think>`when you plan connections or receive new information

    Reasoning & Search Protocol - Always reason inside`<think> ... </think>`when you plan connections or receive new information. - For each hop transition that requires external information, issue search query using`<tool_call> ... </tool_call>`. - Search results will be provided between`<tool_response> ... </tool_response>`by the system

  7. [7]

    Output Format - Emit a numbered sequence of EXACTLY n-1 search steps. For each search i (1 to n-1), produce: `<think> Reasoning step i: Identify Hop i in document/search results, formulate query to reach Hop i+1 </think>` `<tool_call> Query to search Hop i+1 </tool_call>` `[Wait for search results in <tool_response> from system]` - After completing all se...

  8. [8]

    Example template for Hop n = 1, i.e. no search: `<think> [Explain how Hop 1 is selected from the source document and how the question is formulated] </think>` `<question> [Question based solely on the text entity Hop 1] </question>` `<answer> [Answer (Hop 1)] </answer>`

Show all 15 references
  1. [9]

    Example template for Hop n = 3, i.e. 2 searches: `<think> [Reasoning step 1: Find Hop 1 in the source document, formulate the query to reach Hop 2] </think>` `<tool_call> [Search query to find Hop 2 based on Hop 1] </tool_call>` `[Wait for search results in <tool_response> fro...

  2. [10]

    Every subsequent hop must be supported by the corresponding search results.,→

    Start in Document: Hop 1 must be explicitly present in the source text. Every subsequent hop must be supported by the corresponding search results.,→

  3. [11]

    Search is mandatory for n > 1: Each link between hops beyond Hop 1 must use the search engine

  4. [12]

    Exact search count: Emit exactly (n-1)`<tool_call>`entries, no more, no fewer

  5. [13]

    No spoilers: The question must mention only Hop 1; do not include or hint at intermediate hops

  6. [14]

    Clarity: The question is self-contained; the answer is concise and direct (no extra commentary, formatting or explanation)

  7. [15]

    type": "function

    Chain integrity: Each hop must depend strictly on the previous hop. No hop should be skippable or derivable without its immediate predecessor.,→ Now, generate a question and its answer with n = {hop} hops starting from the following source document: {document} Figure 5System p...

    1962

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.