{"id":"ad8a67c3-26a8-41c7-8cd0-51bd3f5a258f","arxiv_id":"2601.07055","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A data-free proposer-solver loop with a search engine and hop-grouped advantage estimates lets a small LLM match supervised search agents on several QA benchmarks.","lead":"A research team built a self-training loop where one AI proposes hard questions, another answers them using web search, and each round makes the questions harder and the answers better—with no human-made training data. Their agent Dr. Zero roughly matches a supervised search agent on several question-answering benchmarks, suggesting that data-free self-evolution may be viable for search agents.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The data-free claim hinges on how the proposer’s initial source document is sampled; the paper never specifies this, and the same Wikipedia dump is used for generation, retrieval, and evaluation, creating a contamination risk that the 'w/o initial document' ablation shows is substantive.","rationale":"The reader's weakest assumption already identifies exactly this issue: the proposer is seeded with an initial source document, the selection mechanism is unspecified, and the same Wikipedia corpus is used for training, retrieval, and evaluation. My review confirms that this is the most load-bearing concern because the central 'data-free' claim depends on the absence of any hidden external supervision or test-set leakage. The ablation in Table 7 shows the initial document is not a minor convenience—removing it causes the largest performance drop in the paper. Yet the paper never describes how that document is chosen, nor does it report any contamination analysis. This is not an internal inconsistency, but a missing control that could invalidate the headline comparison to supervised baselines. The reader's CONDITIONAL verdict already accounts for this uncertainty, so my stress-test does not move the verdict; it sharpens the condition: if the proposed contamination check comes back positive, the verdict should move toward REJECT. I found no other issue as directly connected to the central claim, and I credit the paper for including the relevant ablation and for making the code repository available, which should make the test straightforward to run.","tokens_in":21077,"tokens_out":4637,"duration_ms":48984,"concrete_test":"Open the released repository (github.com/facebookresearch/drzero) and locate the code that populates `{document}` in the proposer prompt. If it samples documents from the same English Wikipedia corpus used for E5 retrieval/evaluation, perform a contamination audit: for every generated training question and its source document, compute entity-level overlap with the test questions of NQ, TriviaQA, PopQA, HotpotQA, 2WikiMQA, MuSiQue, and Bamboogle (e.g., using the answer entity or the first-hop entity). Compare the overlap rate to a random-Wikipedia-page baseline. Then re-run Table 1 on the subset of test questions whose answer entity does not appear in any training source document. If the Dr. Zero vs. Search-R1 gap shrinks or reverses on the uncontaminated subset, the data-free generalization claim is materially weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Dr. Zero matches/surpasses supervised search agents with zero training data. A load-bearing precondition is that the proposer’s source document (`{document}` in Fig. 5) is not a form of privileged supervision and does not leak evaluation topics. Appendix A describes training rollouts but never states how these documents are selected. The only ablation touching this, Table 7 (`w/o initial document`), drops the 3B one-iteration average from 0.304 to 0.245 (~19% relative), the largest single ablation effect in the paper. If these documents are sampled from the same English Wikipedia dump used for retrieval and evaluation (and no deduplication/contamination control is described), then the proposer is being handed a relevant Wikipedia page for each synthetic question. The generated questions can then partially mirror the benchmark questions (same entities, same supporting passages), inflating the reported EM gains. Because the source document is essential for performance, this is not a minor implementation detail; it directly bears on whether the 'data-free, external-search-only' story holds. The paper provides no analysis of overlap between training source documents/generated questions and the evaluation sets, so the central comparison to Search-R1 is not yet protected against this confound.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Dr. Zero, a data-free self-evolution framework for multi-turn search agents. A proposer model generates structurally diverse multi-hop QA pairs, each grounded in an initial source document, and a solver model is trained on those pairs via GRPO. The proposer is then updated with a difficulty-guided reward and hop-grouped relative policy optimization (HRPO), using solver pass rates to encourage questions that are challenging but solvable. Both agents are initialized from the same base LLM, and the only external knowledge comes from a search engine over an English Wikipedia index. Experiments on NQ, TriviaQA, PopQA, HotpotQA, 2WikiMQA, MuSiQue, and Bamboogle with Qwen2.5-3B/7B report that Dr. Zero roughly matches or exceeds supervised Search-R1 on several one-hop datasets and is competitive on multi-hop datasets, while using no human-annotated training data. The paper claims this demonstrates that complex search and reasoning can emerge solely through self-evolution.","tokens_in":21377,"tokens_out":6445,"duration_ms":72305,"significance":"If the empirical claims are supported, this is a significant result: it would show that agentic search and evidence-grounded reasoning can be bootstrapped without curated QA labels, and that the proposer-solver loop plus hop-grouped advantage estimation can reduce the compute cost of self-evolution. The paper deserves credit for releasing code, reporting detailed ablations (format reward, reward shaping, training steps, hop ratios), providing honest failure examples, and comparing against data-free baselines. However, the strength of the central claim is not yet fully secured: the source-document sampling procedure is unspecified, no contamination analysis is reported, and the aggregate numbers in Table 1 do not actually show that Dr. Zero surpasses the supervised Search-R1 baseline. These issues are central to the 'data-free' and 'matches or surpasses' narrative, so the current evidence is promising but conditional.","major_comments":[{"comment":"The proposer prompt in Figure 5 requires a source document ('{document}') as the starting point for every generated question, but the paper never states how these documents are selected or sampled. Table 7 shows that removing the initial document is the largest single ablation effect (average drops from 0.304 to 0.245, about 19% relative). Since the search engine and evaluation corpus are both described as an English Wikipedia dump, and the evaluation datasets (NQ, TriviaQA, PopQA, HotpotQA, 2WikiMQA, MuSiQue, Bamboogle) are all Wikipedia-derived, a real risk exists that the proposer is being given relevant Wikipedia articles for entities or topics that also appear in the test set. Without an overlap analysis between source documents / generated questions and the benchmark questions, the comparison to supervised Search-R1 is not yet protected against this confound. Please specify the doc","section":"Appendix A / Figure 5 / Table 7"},{"comment":"The abstract and introduction say Dr. Zero 'matches or surpasses' fully supervised search agents. The average scores in Table 1 do not support the 'surpasses' part: on Qwen2.5-3B, Dr. Zero averages 0.326 vs. Search-R1's 0.327; on Qwen2.5-7B, 0.372 vs. 0.384. Dr. Zero wins on NQ, TriviaQA, PopQA, and 2WikiMQA (7B) but consistently loses on HotpotQA, MuSiQue, Bamboogle, and on the overall average. The paper should reframe the central claim to 'matches' on one-hop, 'competitive but below' on multi-hop, and clearly discuss the multi-hop gap rather than claiming parity across the board.","section":"Abstract / Table 1"},{"comment":"The caption says bold indicates 'statistically significant' gains, but the table only gives Dr. Zero's mean and standard deviation over what appears to be three runs, while the baselines are single point estimates with no variance. No test statistic or p-value is reported. A proper paired significance test (e.g., paired bootstrap over evaluation examples or a signed test) or confidence intervals for all methods is needed before claiming statistical significance. For example, the 7B MuSiQue result (0.112 vs. 0.168) is numerically worse, so the selective bold marking without a formal test is misleading.","section":"Table 9"},{"comment":"The appendix concludes that HRPO 'achieves superior performance' compared with GRPO (0.326 vs. 0.320 average). However, GRPO is better on HotpotQA (0.303 vs. 0.298), MuSiQue (0.100 vs. 0.091), and Bamboogle (0.272 vs. 0.200); HRPO's advantage comes entirely from the one-hop datasets. Given the paper's emphasis on multi-hop reasoning, the appropriate conclusion is that HRPO gives comparable aggregate performance at much lower compute, not that it is superior. Also, the compute comparison counts 1×5 solver rollouts for HRPO vs. 4×4 for GRPO, but it should state whether proposer-side search rollouts are included on both sides.","section":"Appendix B / Table 8"}],"minor_comments":[{"comment":"Appendix A says the base model is used as the 'generative reward (Yang et al., 2024)', but the reference list does not contain a 2024 Yang et al. paper about generative reward; the likely intended citation is Yuan et al. (2024) on self-rewarding LLMs. Please fix the citation.","section":"Appendix A / References"},{"comment":"The caption contains a typo: 'proser' should be 'proposer'.","section":"Figure 3"},{"comment":"The proposer example for hop=3 claims Robert Harvey is the answer to a question about a Melbourne Demons player in the 1988 and 2000 AFL Grand Finals. Robert Harvey played for St Kilda, not Melbourne, so the generated QA pair contains a factual error. If such errors are not rare, the synthetic training signal could contain false premises. Please either correct the example or acknowledge that this is a failure case.","section":"Figure 9"},{"comment":"The GRPO objective is written with a min of the clipped and unclipped ratio inside an expectation over n samples, which is standard, but the notation is dense. A short explanation of how the group baseline is computed in Eq. (5) would improve readability.","section":"Section 3.3 / Eq. (5)"},{"comment":"The paper states that all baselines are evaluated with the same search engine and corpus but does not give the retrieval corpus size or how documents are chunked. Reporting these details would help reproducibility and would also clarify the contamination question.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The paper is well-executed and the efficiency idea (HRPO) is interesting, but the central claim currently overstates the results and one load-bearing detail—the source-document sampling—is unspecified. I believe the authors can address the contamination concern with an overlap analysis and clarified setup, so I am not recommending rejection. I also noted that the qualitative example in Figure 9 is factually incorrect; if this indicates systematic proposer errors, it would strengthen the need for a careful reward/quality analysis. The paper's code release and honest reporting of failure modes are positive signals."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: this paper is worth reading and worth sending to referees, but the headline claim is stronger than the data, and there's one unresolved setup detail that could change the interpretation.\n\nWhat's new: the specific combination of proposer-solver co-evolution with multi-turn search tool use, a hop-grouped advantage estimator (HRPO), and a difficulty reward based on solver pass rates. They compare against data-free baselines (SQLM*, R-Zero*) and the supervised Search-R1, and they include ablations and significance tests. The loop is genuinely co-evolutionary: proposer reward is defined by solver pass rate, and both agents are evaluated on held-out benchmarks. Code is released. That's real evidence.\n\nThe soft spots, in order of severity.\n\nFirst, the 'matches or surpasses' framing. At 3B, Dr. Zero averages 0.326 vs Search-R1's 0.327 — essentially a tie, slightly below. At 7B it's 0.372 vs 0.384, clearly below. On several multi-hop datasets (Bamboogle, MuSiQue) it trails by a wider margin. The body text is more measured — they say parity and outperform on some single-hop — but the abstract and title don't carry that nuance. That's a fixable overclaim, not a fatal flaw.\n\nSecond, the proposer is seeded with a source document, and the paper never says how those documents are selected. Figure 5 shows the prompt requires a document; the ablation 'w/o initial document' drops the 3B one-iteration average from 0.304 to 0.245 — the largest ablation effect in the paper. If those documents are sampled from the same English Wikipedia dump used for retrieval and evaluation, and there's no deduplication or overlap control, the synthetic training questions could overlap with benchmark questions in entities or supporting passages. The paper reports no contamination analysis. This doesn't necessarily sink the central mechanism, but it's load-bearing for the data-free claim, and it's not a minor implementation detail.\n\nThird, minor: Table 5's hyperparameters (group size 1, reward size 5) don't obviously line up with the HRPO equation as written. Needs a clarifying pass.\n\nThe math and citation pattern look solid; the empirical claims are mostly backed by tables and significance tests. The main missing piece is the data-setup transparency.\n\nWho should read it: anyone working on self-evolving agents, RL for tool use, or synthetic data generation. I'd send it to peer review — the core idea is coherent and the results, while conditional, are competitive. I'd ask the reviewers to focus on the source-document sampling and the framing.\n\nRecommendation: engage with it. With a clear description of the document selection and some contamination analysis, this could be a solid contribution.","headline":"Solid, well-engineered proposer-solver self-evolution for search agents, but the 'surpasses' framing overreaches and the proposer's source-document sampling is a load-bearing unknown.","tokens_in":21919,"tokens_out":4483,"would_cite":true,"duration_ms":42050,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that search agents can be trained with zero human-annotated data by having one model write increasingly hard multi-hop questions and another learn to solve them, matching or beating supervised baselines on open-domain QA b","keywords":["self-evolution","search agents","reinforcement learning","multi-hop question answering","synthetic data generation","policy optimization","tool use","open-domain QA"],"falsifier":"Take the exact Dr. Zero pipeline but sample the initial source documents from a corpus that is provably disjoint from the seven evaluation benchmarks, or deduplicate all generated questions against NQ, TriviaQA, PopQA, HotpotQA, 2WikiMQA, MuSiQue, and Bamboogle; if the margin over supervised baselines shrinks or reverses, the apparent data-free gains are partly benchmark overlap rather than pure self-evolution.","tokens_in":20928,"feed_emoji":"🔍","tokens_out":6842,"duration_ms":67978,"temperature":0.7,"pith_summary":"Dr. Zero asks whether a search agent can improve itself without any curated questions or answers, using only a search engine as its environment. The paper's answer is yes: a proposer model generates diverse, multi-hop synthetic questions, a solver is trained on them with group relative policy optimization, and the solver's pass rate feeds back to push the proposer toward harder but still solvable questions. A hop-grouped variant of policy optimization (HRPO) makes this loop computationally tractable by grouping questions of the same hop count instead of sampling many responses per prompt. On several open-domain QA benchmarks, the resulting data-free agent matches or exceeds supervised search agents, especially at the smaller model size, suggesting complex evidence-grounded reasoning can emerge without human supervision.","feed_headline":"No training data: self-evolving search agents match supervised rivals","feed_subtitle":"A proposer writes hard multi-hop questions, a solver learns them, and both improve using only a search engine.","key_machinery":"The load-bearing machinery is the proposer-solver feedback loop. The proposer is prompted with a source document and a hop count and must produce a deterministic question whose reasoning chain starts in that document; its reward is a pass-rate-shaped difficulty reward, maximized when the solver gets exactly one of n attempts correct, plus a format reward for well-formed search and reasoning traces. The solver is trained on these synthetic QA pairs with GRPO. To make proposer training cheap, HRPO replaces nested sampling with hop-grouped advantage estimation: questions sharing the same number of hops form the baseline group, so each prompt needs only one generated question and a handful of so","core_discovery":"The paper's central claim is that a proposer-solver self-evolution loop, seeded only with documents from an external search corpus, can produce a search agent on par with agents trained on human-written questions and answers. The proposer is rewarded not for easy or impossible questions but for questions the current solver can answer exactly once out of several attempts; this keeps the curriculum at the edge of the solver's ability. The solver is trained with GRPO on the generated questions, and the proposer with HRPO, which estimates advantages by standardizing rewards within groups of questions that need the same number of reasoning hops. The result is an automated curriculum that co-evolv","pith_inferences":["The paper leaves open how the initial source document for each generated question is sampled; since removing it causes the largest single drop in the ablation (0.304 to 0.245), seed selection is a hidden variable worth isolating before accepting the 'data-free' framing.","If seed documents could themselves be proposed by the model, the pipeline would be fully closed-loop; the current design still depends on an external choice of starting text.","HRPO's idea of grouping trajectories by structural difficulty instead of sampling many responses likely transfers to other multi-turn tool-use RL settings, such as code repair or web navigation, where rollouts are expensive.","The framework entangles 'skill' with the search engine's corpus: the same loop on a smaller or biased index would yield different curricula, so results are partly a property of the knowledge environment."],"forward_implications":["A search agent can be trained without human-written questions, answers, or demonstrations, using only an external search engine and a base language model.","As the solver improves, the proposer is automatically pushed to generate harder questions, producing a curriculum that keeps challenging the solver.","HRPO cuts proposer training compute to about a quarter of nested-sampling GRPO while maintaining or slightly improving average performance.","Gains concentrate early: performance peaks within about 50 solver steps and after two to three self-evolution iterations, after which additional training plateaus or degrades.","On the reported benchmarks, the 3B data-free agent surpasses a supervised search-agent baseline on single-hop datasets; the 7B version approaches it and beats it on one multi-hop dataset."],"fun_headline_variants":["Self-evolving search agents match supervised rivals with no training data","Proposer-solver loop lets search agents evolve without curated data","Dr. Zero: search agents rival supervised ones using only a search engine","Automated curriculum breeds search agents that match supervised QA","Search agents self-evolve to supervised-level performance, no human data"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole data-free story rests on how the initial source documents that seed each proposer-generated question are chosen; the paper does not report the sampling procedure, and its own ablation shows that removing these seeds cuts average performance from 0.304 to 0.245.","fun_headline_variants_meta":{"raw":{"variants":["Self-evolving search agents match supervised rivals with no training data","Proposer-solver loop lets search agents evolve without curated data","Dr. Zero: search agents rival supervised ones using only a search engine","Automated curriculum breeds search agents that match supervised QA","Search agents self-evolve to supervised-level performance, no human data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001103,"raw_usage":{"total_tokens":4456,"prompt_tokens":781,"completion_tokens":3675,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":3588}},"tokens_in":525,"tokens_out":3675,"duration_ms":28194,"temperature":1.0,"reasoning_tokens":3588,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T11:13:49.917211+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the exact Dr. Zero pipeline but sample the initial source documents from a corpus that is provably disjoint from the seven evaluation benchmarks, or deduplicate all generated questions against NQ, TriviaQA, PopQA, HotpotQA, 2WikiMQA, MuSiQue, and Bamboogle; if the margin over supervised baselines shrinks or reverses, the apparent data-free gains are partly benchmark overlap rather than pure self-evolution.","supporting_citations":[],"review_version":1}