{"id":"ff602514-ede2-40f0-b8ee-1d57b4ca6e61","arxiv_id":"2607.05752","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Counterfactual no-search vs forced-search outcomes yield a model-specific oracle that trains search-routing policies, raising macro-F1 from ~0.71 to ~0.82–0.84 on oracle-eligible examples.","lead":"The paper trains LLMs to decide when to search by labeling each question from paired no-search vs forced-search outcomes, then fine-tunes with SFT and preference optimization. This cuts both unnecessary and missed search on factual and boundary questions for two small models.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Oracle labels and reported routing gains rest on LLM judges whose agreement with human resolution criteria is unvalidated, especially for KUQ correction/clarification/abstention.","rationale":"The reader correctly isolates judge fidelity and forced-search adequacy as the weakest assumption and already assigns CONDITIONAL with high confidence. My stress test does not overturn that verdict: the experimental design is clear, the within-model gains are large and directionally coherent, and the paper is explicit that the oracle is model-/tool-conditional. The single most load-bearing remaining risk is the unvalidated KUQ LLM judges, because they define both training targets and the headline metrics on the search-boundary subset that the introduction emphasizes. A modest human re-label study would settle whether that risk lands; until then CONDITIONAL remains the right call, not REJECT. No stronger internal inconsistency appears in the routing math or the reported numbers.","tokens_in":17520,"tokens_out":617,"duration_ms":7216,"concrete_test":"On a stratified sample of ~200 KUQ FA+Amb. responses (balanced across N1/S1, N0/S1, N0/S0 and Ppre/SFT/PO), obtain independent human SUCCESS/FAIL labels under the paper’s own rubrics; recompute oracle labels and Table 2 macro-F1 / FA Acc. / Amb. Acc. If human–judge agreement is <0.8 Cohen’s κ or if re-labeled macro-F1 gains shrink by >0.05 absolute, the reported routing improvements are not trustworthy.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that the counterfactual oracle (N vs S success) is a usable learning signal that raises routing macro-F1 on oracle-eligible held-out examples. That claim is only as strong as the binary success labels y_N and y_S. For PopQA the alias-aware SimpleQA-style judge is relatively standard, but for KUQ False Assumption and Ambiguous the paper uses gpt-5.4 resolution judges that map free-form responses to SUCCESS/FAIL for premise correction, clarification, or abstention (Appendix A.1, Tables A.1–A.2, prompts C.7–C.8). These labels decide both which rows enter the oracle-eligible set and which first actions are treated as correct. The paper never reports human agreement, inter-judge reliability, or a sensitivity analysis on those KUQ labels. If the judges systematically reward or punish particular no-search styles (or mis-score search-conditioned resolutions), both the oracle buckets and the SFT/PO gains on FA Acc. / Amb. Acc. (Table 2) become unreliable. The forced-search protocol (single first-turn call, fixed Brave budget) is a secondary concern already noted by the reader; the more load-bearing gap is that the evaluation criterion itself is uncalibrated for the search-boundary regime that motivates the work.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper frames search use in tool-augmented LLMs as an instance-level first-action routing problem (NO SEARCH vs SEARCH), where NO SEARCH covers direct answering, premise correction, clarification, and abstention. Supervision is derived from paired counterfactual traces: no-search N(q) and forced-search S(q). Task-specific binary success induces an oracle over NO SEARCH, SEARCH, and UNSOLVED; UNSOLVED is excluded from routing training/metrics and retained for diagnostics. On PopQA and KUQ false-assumption/ambiguous questions, SFT and DPO-style preference optimization raise free-policy routing macro-F1 on held-out oracle-eligible examples from 0.7082 to 0.8235 (Gemma E2B) and from 0.7053 to 0.8365 (Qwen3.5-4B) relative to a fixed selective-search prompt baseline, with model-specific error-direction repairs and exclusive-rescue analysis of hard dual-failure cases.","tokens_in":17913,"tokens_out":1363,"duration_ms":27171,"significance":"If the result holds under a validated success criterion, the contribution is a practical and well-scoped protocol for learning when to call search from observed outcomes rather than from a separate usefulness classifier or heuristic. Strengths include: (i) an explicit model-specific, outcome-based oracle rather than dataset-level search labels; (ii) controlled comparison of Ppre, SFT, and preference optimization under a fixed free-policy prompt and search interface; (iii) directional over-/under-search metrics that expose different failure modes across model families; and (iv) a diagnostic treatment of UNSOLVED that avoids collapsing heterogeneous pipeline failures into binary routing targets. The work is relevant to adaptive retrieval, tool-use, and over-search literature, and the counterfactual construction is reusable beyond the two models studied.","major_comments":[{"comment":"The central claim treats the counterfactual oracle as a usable proxy for search need, but y_N and y_S for KUQ rest on unvalidated gpt-5.4 resolution judges that map free-form correction/clarification/abstention to SUCCESS/FAIL (Appendix A.1, Tables A.1–A.2, prompts C.7–C.8). These labels determine both oracle eligibility and the FA Acc./Amb. Acc. columns in Table 2. Without human agreement, inter-judge reliability, or sensitivity analysis on KUQ labels, the reported routing gains on search-boundary subsets—and the interpretation of the oracle as reflecting true task success—are not yet load-bearing. PopQA’s alias-aware factuality judge is more standard; the gap is concentrated in the regime that motivates the paper.","section":null},{"comment":"Section 4 and Table 2 evaluate agreement between free-policy first actions and the oracle, not whether post-training free-policy rollouts improve end-to-end task success relative to Ppre (or relative to always-search / never-search). Because free-policy search trajectories need not match successful forced-search S(q) traces, and free-policy no-search need not match N(q), routing macro-F1 gains do not by themselves establish improved task success—the quantity used to define the oracle in Section 2. Reporting free-policy success rates (overall and by oracle bucket) on the same test split is needed to close the loop on RQ2 as framed in the introduction.","section":null},{"comment":"Section 2 and Appendix A.2 define SEARCH benefit via a single forced first-turn search call under a fixed Brave budget, with invalid first calls dropped and max-tool-round failures counted as y_S=0. This is a reasonable controlled protocol, but it systematically understates search utility when multi-hop refinement or larger evidence budgets would flip N0S1/N0S0 labels (the R_TR probe in Section 5 already shows some hard cases are budget-limited). The paper should either (a) quantify how often richer search would re-label oracle buckets on a subsample, or (b) more sharply limit claims to “search under this first-call protocol,” so that residual UNSOLVED and under-search rates are not over-interpreted as intrinsic search need.","section":null}],"minor_comments":[{"comment":"Figure 1 and Figure 2 are described but the manuscript text does not fully specify visual content for readers of the text-only version; ensure captions are self-contained.","section":null},{"comment":"Table 2 reports identical SFT macro-F1 (0.8207) for both models; a brief note on whether this is coincidence or a shared checkpoint-selection effect would help.","section":null},{"comment":"Appendix A.4: conditional-error denominators (e.g., Gemma test NO SEARCH=306, SEARCH=502) are useful; consider promoting a short version into the main experimental setup for readability.","section":null},{"comment":"Related Work could more explicitly contrast the outcome-based oracle with confidence/self-knowledge triggers (Self-RAG, Adaptive-RAG, SKR) in a single comparison table of supervision source and decision granularity.","section":null},{"comment":"Typos/formatting: “F AGEN Workshop” spacing in the header; “google/gemma-4-E2B-it” model naming is fine but should be consistent with “Gemma E2B” in tables.","section":null},{"comment":"The free-policy prompt card (Figure C.5) is central to Ppre; quoting the key selective-search instruction in the main text (1–2 sentences) would reduce dependence on the appendix.","section":null}],"recommendation":"major_revision","confidential_remarks":"The core idea is solid and the experimental hygiene (fixed interface, model-specific oracles, directional errors, UNSOLVED diagnostics) is above average for this area. I would not reject on novelty grounds. The revision bar is mainly evaluation integrity: KUQ judge validation and free-policy task-success numbers. If those land cleanly, this is a strong workshop-to-journal candidate. Scope fit for a general ML venue is acceptable as a methods/systems paper; for a more theory-heavy track it may read as primarily empirical protocol design."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful takeaway is simple: pair no-search and forced-search free-form traces, label NO SEARCH / SEARCH / UNSOLVED from task success, train first-action routing with SFT then DPO, and you get clear held-out macro-F1 lifts (0.71→0.82/0.84) over a prompt-only selective-search baseline under a fixed Brave interface.\n\nWhat is actually new is not “adaptive retrieval” in the abstract—Self-RAG, When2Call, and over-search papers already live there—but the concrete outcome oracle from free-form counterfactual traces, the explicit UNSOLVED diagnostic bucket, and the demonstration that the same signal repairs model-specific error directions (Gemma over-search restraint; Qwen also cuts missed search). The experimental hygiene is good: fixed free-policy prompt and tool interface, oracle-eligible-only metrics, conditional over-/under-search rates, PopQA vs KUQ counterfactual marginals that go in opposite directions, and exclusive-rescue probes on hard N0S0 cases. Scope is honest: first action only, model-/tool-conditional labels, no claim to solve multi-hop synthesis.\n\nSoft spots, in proportion. The stress-test is right that KUQ SUCCESS/FAIL rests on gpt-5.4 resolution judges with no human agreement or sensitivity analysis; those labels decide eligibility and FA/Amb accuracies, so that is a real calibration gap, not a nit. PopQA’s alias-aware SimpleQA-style judge is on firmer ground. Forced single-turn search with a fixed budget is a secondary limitation the paper already owns. No public code/data is a practical annoyance, not a conceptual hole. Circularity is low: ordinary supervised agreement with an outcome-defined label.\n\nThis is for people building RAG/tool routers who care about when not to search as much as when to. It deserves a serious referee. I would engage with it, cite the oracle construction if I work on selective tool use, and bring it to reading group.","headline":"Clean first-action routing paper: counterfactual N/S oracle + SFT/DPO gives real macro-F1 gains on two models; main soft spot is unvalidated KUQ judges, not the framing.","tokens_in":18479,"tokens_out":516,"would_cite":true,"duration_ms":7142,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Comparing no-search and forced-search outcomes for the same question yields a usable oracle that trains LLMs to call search only when it improves task success.","keywords":["search routing","retrieval-augmented generation","counterfactual supervision","tool use","when not to call","preference optimization","known-unknown questions","selective search"],"falsifier":"Hold the same models, prompt, and Brave search interface fixed, but re-label a large held-out set with human task-success judgments (or with a multi-hop search protocol): if the counterfactual oracle then disagrees with human necessity labels on a large fraction of cases, or if SFT/preference training on the new labels fails to beat the prompt-only selective-search baseline on routing macro-F1, the central claim is falsified.","tokens_in":18413,"feed_emoji":"🔎","tokens_out":1070,"duration_ms":14451,"temperature":0.7,"pith_summary":"Search tools help language models answer questions their parameters cannot, but search is not always the right first move: it can be unnecessary, noisy, or worse than correcting a false premise, asking for clarification, or abstaining. This paper treats the decision as instance-level search routing—whether to answer without the tool or to issue a search call—and builds supervision without human labels of “search usefulness.” For each question it runs a no-search trace and a forced-search trace, scores both with task-specific success criteria, and labels the case NO SEARCH, SEARCH, or UNSOLVED. On the eligible labels it trains routing policies with supervised fine-tuning and preference optimization, lifting routing macro-F1 well above a prompt-only selective-search baseline for two small models. Residual dual-failure cases are kept as diagnostics, showing that many remaining errors sit outside the first-action choice.","feed_headline":"Train LLMs to search only when it actually helps","feed_subtitle":"Paired no-search and forced-search outcomes build an oracle that cuts both missed and unnecessary search","key_machinery":"Counterfactual routing oracle: for each question, compare binary task success under a no-search execution and a forced first-turn search execution; assign NO SEARCH if no-search already succeeds, SEARCH if search recovers a no-search failure, and UNSOLVED if both fail (excluded from routing training and accuracy).","core_discovery":"An outcome-based oracle built from paired no-search and forced-search traces for the same question is both a fair evaluation target and a learning signal for instance-level search routing. Training on that oracle with supervised fine-tuning and preference optimization raises routing macro-F1 on held-out oracle-eligible examples from 0.7082 to 0.8235 for Gemma E2B and from 0.7053 to 0.8365 for Qwen3.5-4B, correcting model-specific under-search and over-search rather than merely changing global search rate.","pith_inferences":["If paired counterfactual traces are cheap enough, the same supervision recipe could extend to other optional tools (calculator, code interpreter) where “when not to call” matters as much as correct arguments.","As base models grow, the NO SEARCH / SEARCH boundary will shift with parametric knowledge, so routing adapters may need periodic re-oracle collection rather than a fixed label set.","Production systems could use the UNSOLVED diagnostic probes (larger model, expanded retrieval, post-training policy) as a cheap triage queue for hard failures instead of forcing a binary route.","Preference optimization’s modest further gains after SFT suggest the remaining headroom may sit more in query formulation and evidence use than in the first-action bit alone."],"forward_implications":["Search-augmented systems can train the first-action call decision from automatic paired rollouts instead of hand labels of search usefulness.","Routing training should correct under-search and over-search jointly and model-specifically, not chase a single global search-call rate.","NO SEARCH is a success path that includes correct direct answers, premise correction, clarification, and abstention—not a failure mode.","Dual-failure (UNSOLVED) cases should stay outside binary routing supervision and be diagnosed for capacity, retrieval budget, evidence use, and later pipeline stages.","The same outcome-oracle template can evaluate free-policy selective-search prompts against observed recoveries rather than against dataset-level assumptions that every PopQA item needs search or every boundary item forbids it."],"fun_headline_variants":["Pair no-search vs forced-search to train LLM search routing","Counterfactual oracle labels decide when LLMs should search","Outcome pairs teach models to skip unneeded search calls","Instance-level search routing from paired execution traces","Oracle supervision lifts routing F1 for Gemma and Qwen"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The automated judges and the single first-turn forced-search setup correctly decide whether search was needed; if they systematically mis-score answers or if richer multi-step search would flip many dual-failure labels, the oracle and the reported gains no longer hold.","fun_headline_variants_meta":{"raw":{"variants":["Pair no-search vs forced-search to train LLM search routing","Counterfactual oracle labels decide when LLMs should search","Outcome pairs teach models to skip unneeded search calls","Instance-level search routing from paired execution traces","Oracle supervision lifts routing F1 for Gemma and Qwen"]},"model":"grok-4.5","effort":"low","cost_usd":0.004548,"raw_usage":{"total_tokens":1370,"prompt_tokens":822,"num_sources_used":0,"completion_tokens":62,"cost_in_usd_ticks":45480000,"prompt_tokens_details":{"text_tokens":822,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":486,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":822,"tokens_out":62,"duration_ms":10658,"temperature":1.0,"reasoning_tokens":486,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T02:30:40.104338+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Hold the same models, prompt, and Brave search interface fixed, but re-label a large held-out set with human task-success judgments (or with a multi-hop search protocol): if the counterfactual oracle then disagrees with human necessity labels on a large fraction of cases, or if SFT/preference training on the new labels fails to beat the prompt-only selective-search baseline on routing macro-F1, the central claim is falsified.","supporting_citations":[],"review_version":1}