{"id":"cbabf306-1d6f-49dd-a771-6f7b0e541b96","arxiv_id":"2608.00974","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Restricting RL training retrieval to ground-truth-related documents improves a search agent's multi-hop QA exact match from 0.206 to 0.297 over Search-R1, when tested on the full corpus.","lead":"This paper trains a search-augmented language model by restricting its search engine during reinforcement learning to documents related to the correct answer, then evaluates on the full corpus. The method improves exact-match accuracy on multi-hop QA by over 40% compared with Search-R1, though numbers come from a single run with no released code.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The NQ pseudo-GT construction in §3 restricts training to documents similar to the query-plus-answer string, so the policy can earn exact-match reward by copying answers from the restricted pool; whether this transfers to full-corpus search is unestablished.","rationale":"The reader and I identify the same weak point. The method's novelty is that ground-truth-similar restriction densifies reward and teaches generalizable search behavior. But for NQ, the ground-truth string contains the answer, so top-300 E5 similarity will retrieve documents that contain the answer string. Exact-match reward then rewards copying. The full-corpus evaluation is the right setting to test transfer, but it only reports aggregate EM; without an ablation that removes answer leakage, we cannot distinguish 'learned better subqueries' from 'learned to exploit a small answer-bearing index.' The proposed ablation is cheap and decisive. I am not claiming the result is false; the paper has independent evidence (full-corpus validation reward, out-of-domain gains, retrieval-accuracy plots) that makes conditional acceptance appropriate. Additional weaknesses such as missing code, missing error bars, and no kappa sensitivity analysis reinforce the need for conditionality but are secondary to the transfer question.","tokens_in":11247,"tokens_out":10036,"duration_ms":97810,"concrete_test":"Retrain GRT on NQ with ResCorp(s) built from the user query alone, dropping the gold-answer concatenation from §3, while keeping kappa=300 and all other hyperparameters fixed. Evaluate on the NQ test set with the full corpus. If NQ EM and the TriviaQA/PopQA transfer results stay within a few points of Table 1, the answer-leakage concern does not land; if they drop materially toward the Search-R1 baseline, the answer string in ResCorp(s) is the load-bearing component and the general-QA claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 defines ResCorp(s) as the union over each i in GT_s of the top-kappa E5-nearest documents to i. For NQ, GT_s is formed by concatenating the user query and the ground truth answer. Thus the training index for every NQ example contains documents selected for similarity to the answer string itself. The RL reward is 1 only if the final answer exactly matches a gold answer, so a policy can obtain dense reward without learning to decompose the question: issue a query close to the user query, retrieve from the small answer-enriched pool, and copy the answer span. This is a shortcut relative to the inference MDP, where the retriever searches the full 2018 Wikipedia corpus and the same subquery may not surface the answer-bearing document. The paper's full-corpus validation reward (Fig. 2b) and retrieval-accuracy plots (Fig. 1a) are consistent with better answer extraction as well as with better search; they do not isolate subquery quality. The central claim—that restricting retrieval during training teaches generalizable search behavior—therefore rests on the unverified assumption that the policy learned robust query generation rather than exploitation of the small, answer-containing candidate set. The HotpotQA results use legitimate ground-truth passages and may be less affected, but the NQ pseudo-GT is the only support for the claim that the method works without passage-level supervision, and it is the weakest link in the transfer argument.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Guided Retrieval Training (GRT), a training-time intervention for RL-trained search agents. During PPO training on Natural Questions (NQ) and HotpotQA, the retriever is restricted to ResCorp(s): for each ground-truth item, the top-kappa E5-nearest documents in the 2018 Wikipedia corpus, using gold passages for HotpotQA and the concatenation of query and gold answer for NQ. At inference the retriever searches the full corpus. The experiments report consistent exact-match gains over Search-R1 and other baselines on seven QA datasets, with the MHQA average rising from 0.206 to 0.297 and the all-task average from 0.318 to 0.375; additional figures report higher retrieval accuracy, better answer synthesis, and higher training/validation rewards. No code or data are released.","tokens_in":11537,"tokens_out":4964,"duration_ms":43540,"significance":"If the results withstand scrutiny, GRT is a simple and transferable recipe: giving the policy access to answer-bearing documents during early RL removes the sparse-reward bottleneck and leads to better subquery generation and answer synthesis, with gains that persist when the retriever is unrestricted at inference. The paper deserves credit for reporting full-corpus evaluation and validation reward rather than only restricted-training metrics, for recomputing the Search-R1 baseline under a common setup, and for separating retrieval accuracy from answer synthesis in the analysis. The significance is currently conditional because the main support for the method's most interesting regime (NQ, no passage-level supervision) rests on an unvalidated pseudo-ground-truth construction, and because all headline numbers are single-run comparisons without variance or significance testing.","major_comments":[{"comment":"For Natural Questions, GT_s is the concatenation of the user query and the gold answer, and the training retriever is restricted to documents whose E5 similarity to that string is in the top kappa. Combined with the exact-match reward, this makes answer copying from a small, answer-enriched pool a viable training strategy that does not require learning to decompose multi-hop questions; whether such behavior transfers to full-corpus inference is untested. Please validate the pseudo-ground-truth construction (e.g., check that the restricted pool actually contains the supporting passage, measure how often the gold answer string appears in the top-kappa documents, and ablate with passage-level supervision on a subset) and provide a metric that isolates subquery quality, such as overlap with gold subqueries or retrieval of gold supporting passages. The current Section 5.1 definition of retrieval accuracy (presence of the gold answer string in retrieved documents) cannot distinguish good search from answer copying.","section":"Section 3, definition of ResCorp(s)"},{"comment":"All numbers are reported as point estimates from what appears to be a single training run. The headline MHQA gain (0.206 vs. 0.297) and all-task gain (0.318 vs. 0.375) need error bars across seeds and a paired significance test, especially because several evaluation sets are small (Bamboogle has 125 test examples and PopQA is a single test split). Without this, the improvement could be within run-to-run variance.","section":"Table 1"},{"comment":"kappa is fixed at 300 with no sensitivity analysis. The size of the restricted corpus controls how much the training distribution differs from the inference distribution, so the reader cannot tell whether the benefit is robust or requires a carefully tuned kappa. Please report performance for a range of kappa values (e.g., 100, 300, 1000) on at least NQ and HotpotQA.","section":"Section 4.3, kappa setting"},{"comment":"No code, data splits, or trained checkpoints are provided, so the ResCorp(s) construction and the recomputed Search-R1 numbers in Table 1 cannot be independently verified. In addition, the evaluation protocol for Figures 1 and 2 is under-specified: the paper should state whether retrieval accuracy and answer synthesis are computed with the full 2018 Wikipedia corpus or with the restricted corpus at each checkpoint, and how many retrieved documents are inspected for the 'any retrieved document contains the answer' criterion.","section":"Reproducibility and evaluation protocol"}],"minor_comments":[{"comment":"The word 'inital' in the Text generation paragraph should be 'initial'.","section":"Section 3, Text generation"},{"comment":"The first sentence contains the typo 'in-ditribution'; it should be 'in-distribution'.","section":"Section 4.1"},{"comment":"The column header 'A verage' contains an unintended space, and '2wiki' should be '2Wiki' for consistency with Table 2.","section":"Table 1"},{"comment":"The phrase 'any if the the retrieved documents' is a typo for 'any of the retrieved documents'.","section":"Section 5.1"},{"comment":"References [1] and [2] are the same paper (Ma et al., EMNLP 2023), and references [12] and [29] both cite IRCoT; please deduplicate.","section":"References"},{"comment":"Because training rewards are computed under different retrieval distributions, the caption should state that the comparison is between environments, not just policies.","section":"Figure 2a"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline major-revision. The core idea is simple and potentially useful, and the full-corpus validation prevents the result from being circular, but the NQ pseudo-ground-truth shortcut is a credible threat to the transfer claim. I would not reject outright because the concern is addressable with additional experiments rather than being a logical impossibility. I ask the editor to ensure the requested validation analyses are actually added before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: GRT is a plausible training-time trick, and the headline numbers are probably real, but the paper needs code and a sensitivity analysis before I'd trust the NQ half of the story.\n\nWhat's actually new: Search-R1 showed outcome-only RL can teach an LLM to search. GRT's twist is to restrict the search engine's corpus during training to documents related to ground truth, which densifies reward and speeds learning. That's a simple idea and a natural extension, but it's not in the prior work I know, and the paper gets credit for evaluating on the full corpus at inference, so the main table isn't circular.\n\nWhat's done well: the comparisons use the Search-R1 codebase, the setup is faithful to prior work, the gains are consistent across tasks, and the MHQA improvement (0.206 to 0.297 EM average) is large. Training efficiency is a nice secondary result. The citation pattern looks fine.\n\nWhere I worry: the NQ pseudo-GT construction. Concatenating query + gold answer and retrieving top-300 E5 neighbors gives the policy a small, answer-enriched pool. A model can earn exact-match reward by issuing a query similar to the user query and copying the answer span, without learning to decompose questions. The full-corpus validation reward (Fig 2b) is encouraging, and the OOD MHQA gains suggest something transfers, but those OOD gains could be driven by HotpotQA training, not NQ. The paper never validates that the pseudo-GT pool contains the right supporting context, never varies κ, and reports no error bars or seeds. Figure 1a retrieval accuracy is measured inside the restricted environment, so it's partly by construction. None of this is fatal, and I don't suspect manipulation, but the central claim is only as strong as the transfer story, and that story has gaps.\n\nWho it's for: people working on RL training of search agents; they'll want to try GRT and see if it holds. It deserves a serious referee, mainly to push for code, ablations, and variance. I'd probably cite it as a related method, though I'd wait for the code release before relying on the exact numbers.","headline":"Plausible training-time trick with a real but unverified transfer story; deserves review but not acceptance on current evidence.","tokens_in":12052,"tokens_out":2871,"would_cite":true,"duration_ms":23865,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Guided Retrieval Training restricts training-time search to ground-truth-similar documents and reports a 44% relative gain in multi-hop QA exact match over Search-R1.","keywords":["search agents","multi-hop question answering","reinforcement learning","guided retrieval training","retrieval-augmented generation","question answering","exact match","subquery generation"],"falsifier":"Train GRT on Natural Questions with the gold answer string removed from the query-plus-answer text used to build the restricted corpus, then measure exact match on the full Wikipedia corpus at inference; if the reported gain over Search-R1 collapses, the improvement came from answer-string leakage rather than from learned subquery generation.","tokens_in":11051,"feed_emoji":"🔍","tokens_out":10428,"duration_ms":83307,"temperature":0.7,"pith_summary":"This paper introduces Guided Retrieval Training (GRT), a training-time intervention for reinforcement-learned search agents. During RL training, GRT restricts the search engine to a small corpus of documents most similar to ground truth information for each example, instead of letting the agent retrieve from the full Wikipedia corpus. The paper claims this densifies the reward signal and teaches the agent to generate better subqueries and synthesize answers, without any change to the search process at inference time. On seven QA benchmarks, GRT raises multi-hop exact match from 0.206 to 0.297 relative to Search-R1—about a 44% relative gain—and raises the all-task average from 0.318 to 0.375. A sympathetic reader would care because it is a simple intervention with no inference-time cost and consistent gains across in-domain and out-of-domain tasks.","feed_headline":"Restricting training-time search docs lifts multi-hop QA by 44%","feed_subtitle":"Guided Retrieval Training cuts sparse-reward failures and beats Search-R1 across seven QA benchmarks.","key_machinery":"The load-bearing mechanism is the restricted retrieval corpus $\\mathrm{ResCorp}(s)$: for each item of ground truth information in a training example, the paper keeps only the top-300 documents by cosine similarity of text embeddings, and the search engine during RL training may retrieve only from that union. This acts as a training-time filter that puts evidence near the answer inside the agent's context, so the policy receives a denser reward signal even when its early subqueries are poor. Because the same policy is later run against the full corpus, the method transfers only if the model learns generalizable query decomposition and synthesis from these filtered contexts.","core_discovery":"GRT's central claim is that restricting the retriever during RL training to a ground-truth-guided subset of documents produces a search agent that retrieves and answers better on the full corpus at inference. For a training example $s$, the paper defines the restricted corpus as $\\mathrm{ResCorp}(s) = \\bigcup_{i \\in GT_s} \\{ d \\in \\Gamma \\mid \\text{similarity}(d,i) \\in \\text{top-}\\kappa \\}$, where $\\Gamma$ is the 2018 Wikipedia corpus, $\\text{similarity}$ is cosine similarity of text embeddings, and $\\kappa=300$; ground truth information $GT_s$ comes from gold passages on HotpotQA and from concatenating the query with the gold answer on Natural Questions. With a 3-billion-parameter language model trained by PPO and reward 1 only on exact answer match, GRT reports higher retrieval accuracy and higher answer accuracy conditional on correct retrieval than the Search-R1 baseline, and validation rewards computed with the unrestricted retriever are also higher. The authors interpret this as evidence that the model learns to generate better subqueries and to synthesize answers from retrieved documents, rather than relying on the restricted corpus at inference.","pith_inferences":["For Natural Questions, the pseudo-ground-truth of query plus gold answer may push documents containing the answer string itself into the restricted pool; if the model learns to copy from those, the reported gain could shrink or vanish on questions whose answer is not verbatim in the retrieved text.","The same guided-retrieval idea could be tested as a general reward-shaping recipe for other RL agents with oracle evidence, such as code repair or tool-use agents, by replacing the search corpus with a small set of expert-provided documents during training.","The choice of $\\kappa=300$ and the embedding model are untested hyperparameters; a sweep over $\\kappa$ or a change of embedding model would show how much of the effect depends on the exact similarity ranking."],"forward_implications":["On the four multi-hop benchmarks, GRT's average exact match is 0.297 versus 0.206 for Search-R1, a relative improvement of about 44%; if correct, guided retrieval alone accounts for most of the gain.","The all-task average across seven QA datasets rises from 0.318 to 0.375, so the benefit is not confined to the two training datasets and appears on out-of-domain tasks like TriviaQA, PopQA, 2Wiki, Musique, and Bamboogle.","GRT improves both retrieval accuracy and answer accuracy given correct retrieval, which implies the training signal strengthens subquery generation and reasoning over retrieved text, not just answer copying from a smaller corpus.","Training reward is higher in the early RL steps and better QA accuracy is reached in fewer training steps, so GRT is a sample-efficiency improvement as well as a performance improvement."],"supporting_citations":[{"why":"Supplies the Search-R1 baseline and the search-agent RL framework that GRT modifies.","marker":"[8]"},{"why":"Provides the dense retriever used as the search engine over Wikipedia.","marker":"[36]"},{"why":"Provides the text-embedding model whose cosine similarity ranks documents for the restricted corpus.","marker":"[37]"},{"why":"Natural Questions is one of the two training datasets and the source of the query-plus-answer pseudo ground truth.","marker":"[20]"},{"why":"HotpotQA is the other training dataset and a multi-hop benchmark whose gold passages define ground truth information.","marker":"[23]"},{"why":"Supplies the 3-billion-parameter base language model used as the RL policy.","marker":"[34]"},{"why":"Provides the PPO algorithm used for policy optimization.","marker":"[35]"}],"fun_headline_variants":["Guided retrieval training boosts multi-hop QA by over 40%","Training with ground-truth docs lifts search agent performance","Search-GRT beats Search-R1 with fewer training steps","Ground-truth retrieval guidance yields 40% multi-hop QA gain"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the documents most similar to the question plus its known answer really contain the supporting evidence needed to learn good search behavior, rather than just containing the answer text itself.","fun_headline_variants_meta":{"raw":{"variants":["Guided retrieval training boosts multi-hop QA by over 40%","Training with ground-truth docs lifts search agent performance","Search-GRT beats Search-R1 with fewer training steps","Ground-truth retrieval guidance yields 40% multi-hop QA gain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000555,"raw_usage":{"total_tokens":2688,"prompt_tokens":1034,"completion_tokens":1654,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":650,"completion_tokens_details":{"reasoning_tokens":1584}},"tokens_in":650,"tokens_out":1654,"duration_ms":11566,"temperature":1.0,"reasoning_tokens":1584,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:13:37.960509+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train GRT on Natural Questions with the gold answer string removed from the query-plus-answer text used to build the restricted corpus, then measure exact match on the full Wikipedia corpus at inference; if the reported gain over Search-R1 collapses, the improvement came from answer-string leakage rather than from learned subquery generation.","supporting_citations":[{"cited_title":"Other notable works include HiPRAG [16], R1-Searcher [17],β-GRPO [18] and ZeroSearch [19]","cited_arxiv_id":null,"evidence_quote":"Supplies the Search-R1 baseline and the search-agent RL framework that GRT modifies."},{"cited_title":"The knowledge corpus used for the search engine retrieval mechanism consists of the 2018 Wikipedia documents","cited_arxiv_id":null,"evidence_quote":"Provides the dense retriever used as the search engine over Wikipedia."},{"cited_title":"Natural questions: A benchmark for question answering research,","cited_arxiv_id":null,"evidence_quote":"Provides the text-embedding model whose cosine similarity ranks documents for the restricted corpus."},{"cited_title":"Search-r1: Training LLMs to reason and leverage search engines with reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"HotpotQA is the other training dataset and a multi-hop benchmark whose gold passages define ground truth information."}],"review_version":2}