{"id":"00173cee-4846-468b-abcf-5ceacc782faa","arxiv_id":"2506.18959","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper frames agentic deep research as the successor to web search and proposes, without derivation, a test-time scaling law for reasoning and search.","lead":"This paper argues that LLM-based 'agentic deep research' systems, which combine reasoning with iterative web search, will become the dominant way people seek information, replacing traditional keyword search. It is a position and survey paper, and its main new artifact is a hypothesized 'test-time scaling law' that is interpolated from two existing datasets.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'Test-Time Scaling law' in Figure 3b is the weakest load-bearing element: it is a min-max-normalized interpolation of two unrelated single-axis curves, with no equation, error bars, or validation on an integrated deep-research system.","rationale":"I read the paper as a position statement and survey, not as a new experimental study. Within that frame, the benchmark comparison in Figure 2 is suggestive but not decisive, and the GitHub star analysis is popularity evidence rather than correctness evidence. The one place where the paper goes beyond synthesis into a formal quantitative claim is Section 4's test-time scaling 'law,' which is also listed as a key contribution in Section 1. That law is constructed from two independent single-axis results and is used to predict continued gains for integrated agentic deep research; it is therefore the most load-bearing unsupported element. The concrete check I propose targets exactly that composition step: if the plane fails on a single integrated system with both budgets varied, the law should be relabeled as a heuristic or hypothesis. This is a serious but not fatal issue: the paper's central position can be defended as a research agenda without the law, and the manuscript itself contains hedging language ('hypothesis,' 'hypothetical,' 'anticipated') that points toward the needed revision. I therefore keep the reader's CONDITIONAL verdict unchanged.","tokens_in":18784,"tokens_out":10373,"duration_ms":116521,"concrete_test":"Run a single open-source agentic deep-research system (e.g., R1-Searcher or DeepResearcher) on MuSiQue with a factorial grid of reasoning-token budgets (for example 0, 512, 2K, 8K) and search/retrieval-iteration budgets (for example 0, 1, 4, 16), record normalized accuracy, and fit the surface z = a + b*t_R + c*t_S + d*t_R*t_S in the two log-token dimensions. If the interaction term d is non-negligible, or if the fitted surface deviates from a plane by more than the run-to-run noise, the two single-axis curves in Figure 3b do not compose into the claimed joint TTS law.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim that needs examination is the TTS law introduced in Section 4 and Figure 3b. The figure combines s1 on AIME24 (reasoning axis, roughly 2^9-2^13 inference tokens) with IterDRAG on MuSiQue (search axis, 16K-5M tokens), normalizes each curve independently to [0,1], and then draws a 'diagonal plane' between them as the hypothetical scaling law for agentic deep research. This is not a fit to any system that performs reasoning and search jointly: no equation is stated, no error bars or held-out points are shown, and the two token ranges barely overlap, so the interior of the 3D plot is pure interpolation. Because the two series are separately normalized, the slopes are dimensionless and not commensurable; a 'linear' trend in Figure 3b is therefore not evidence of a shared law. Section 4 itself calls this an 'hypothesis' and Section 4.3 says the trade-off law is 'anticipated,' while Section 1 and Figure 3 present it as a 'law'; that inconsistency is worth resolving. The paper's broader dominance claim could survive the loss of this 'law,' but the paper's only formal scaling contribution would then be unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper is a position paper and survey arguing that a new paradigm, 'Agentic Deep Research' (LLMs with tightly integrated reasoning, iterative search, and synthesis), will become the dominant mode of information seeking, superseding traditional web search, LLM chatbots, and naive RAG. It organizes the evolution of search into stages, surveys prompting/SFT/RL approaches for incentivizing search with reasoning agents, reports benchmark numbers for OpenAI Deep Research on BrowseComp, BrowseComp-ZH, and Humanity's Last Exam, analyzes GitHub star trends of open-source implementations, and proposes a 'test-time scaling law' claiming that normalized performance improves linearly with scaling of internal reasoning depth and external knowledge exploration. It also discusses the alternative view of human-centered search and lists open problems. The main contributions are the taxonomy, the curated resource collection, and the TTS conjecture.","tokens_in":18980,"tokens_out":5481,"duration_ms":54463,"significance":"If the position is correct, the paper identifies a real shift in information access and charts a useful design space around the search-reasoning trade-off. The survey of recent RL-based methods (Search-R1, R1-Searcher, DeepResearcher, and others) and the curated GitHub repository are valuable resources for the community. The 'test-time scaling law' is a thought-provoking conjecture that could stimulate useful empirical work, and the paper explicitly includes an alternative-view section and human-in-the-loop caveats, which is a strength for a position paper. However, the paper provides no new experimental evidence, and its only formal quantitative contribution, the TTS law, is an interpolation between two external single-axis curves rather than a validated empirical law. The significance lies in the framing and resource synthesis rather than in a demonstrated scaling result, and the central dominance claim is a prediction that the manuscript supports with citations and qualitative trends rather than with a decisive experiment.","major_comments":[{"comment":"The proposed 'Test-Time Scaling law' is not supported as stated. The data consist of two independent curves: s1 on AIME24 for the reasoning axis (roughly 2^9-2^13 inference tokens) and IterDRAG on MuSiQue for the search axis (16K-5M tokens), each normalized separately to [0,1]. The 'diagonal plane' is an interpolation between these two series, with no equation, no error bars, no held-out points, and no evaluation on any integrated deep-research system that jointly performs reasoning and search. Because the two normalizations are independent, the slopes are not commensurable, and the visual linearity in Figure 3b does not constitute evidence of a shared scaling law. Since this law is the paper's only formal quantitative contribution, it should be reframed as an illustrative conjecture with explicit caveats, or substantiated with an actual equation, error analysis, and validation on an integrated system.","section":"Section 4, Figure 3b"},{"comment":"The paper is internally inconsistent about the status of the TTS claim: the abstract and Section 1 call it a 'test-time scaling law,' Section 4 opens with 'we introduce the Test-Time Scaling (TTS) law' and then calls it 'an hypothesis,' and Section 4.3 says the trade-off law 'is anticipated.' A 'law' implies an empirical regularity, while the text repeatedly concedes that the claim is a hypothesis. This distinction matters because the paper's quantitative contribution rests entirely on this claim; the authors should choose one framing and apply it consistently throughout.","section":"Abstract, Section 1, Section 4, Section 4.3"},{"comment":"The benchmark comparison in Section 3.2 is based on external reports and product announcements, not on new evaluations by the authors. The text says 'we adopt and evaluate three representative benchmarks' and later concludes 'we demonstrate that agentic systems significantly outperform existing approaches,' but Figure 2 reports numbers from third-party sources (e.g., OpenAI Deep Research at 51.5% on BrowseComp, 42.9% on BrowseComp-ZH, 26.6% on HLE) without standard errors, system versions, evaluation protocols, or dates of access. This is a citation, not a demonstration. The authors should either perform an independent evaluation with transparent methodology or explicitly state that these are reported third-party results, and soften the conclusion accordingly.","section":"Section 3.2, Figure 2"},{"comment":"The GitHub-star analysis is presented as 'empirical momentum' that reinforces the dominance claim, but it excludes the two most-starred and two least-starred repositories and then asserts a 'clear upward trajectory' without statistical tests, a well-defined sample period, or a comparison baseline. This is an anecdotal, cherry-picked trend rather than a rigorous empirical result. The authors should either provide a formal analysis (e.g., regression on star counts with confidence intervals) or characterize this evidence qualitatively and not as 'empirical momentum' that supports the central position.","section":"Section 3.2, Figure 3a, Appendix B"}],"minor_comments":[{"comment":"The footnote 'Work does note related the author's position at Amazon' appears to be a typographical error for 'Work does not reflect the author's position at Amazon'; please fix.","section":"Page 1, footnote"},{"comment":"The phrase 'information seekin' in the final paragraph is missing a 'g'; it should read 'information seeking'.","section":"Section 3.2"},{"comment":"There are several run-on or unclear sentences, such as 'engage users through interactive dialogue with integrate and tailored solutions that not directly available online' and 'LLM chatbots maintain conversation history throughout interactions, Beyond that, they aggregate...'; the prose needs careful proofreading.","section":"Section 2.2"},{"comment":"Figure 2 does not state the data sources or the date each benchmark number was collected; adding a footnote with the source and access date would improve reproducibility and clarity.","section":"Figure 2"},{"comment":"Table 1 contains typos such as '2Wikimultihopqa' and inconsistent capitalization, and many entries under 'Evaluation Data' are 'N/A'; the table should be cleaned up and the 'N/A' cases clarified.","section":"Appendix B, Table 1"},{"comment":"The statement that 'under optimal inference parameters, the performance improves nearly linearly' would benefit from a specific reference to the corresponding figure or table in Yue et al. (2025) and a definition of what constitutes 'optimal inference parameters.'","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"This manuscript is better characterized as a position paper and survey than as a technical contribution. For a journal venue, the TTS section must be reframed from a 'law' to a clearly labeled conjecture, and the benchmark and GitHub-star evidence should be presented as third-party reports and qualitative trends, respectively. The paper contains a high number of self-citations to very recent arXiv preprints; this is not disqualifying, but the authors should ensure that the cited works are central and necessary. The resource repository is a useful community contribution, but the scientific novelty rests on the TTS claim, which currently is not established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a survey/position piece, not a new experimental result. Its three-stage taxonomy (web search, chatbots/RAG, agentic deep research) is a clean synthesis, and the curated resource list on GitHub is genuinely useful. The soft spot is Section 4's 'Test-Time Scaling law': Figure 3b is built from two independent single-axis curves (s1 on AIME24, IterDRAG on MuSiQue), each min-max normalized onto [0,1], with barely overlapping token ranges. The diagonal plane is interpolated between them; there is no equation, no error bars, no held-out validation on an actual integrated deep-research system. Section 4 itself calls it a 'hypothesis' while the abstract and Figure 3 call it a law — that inconsistency should be resolved. As stated, the 'law' is not a law; it is a plausible visual conjecture.\n\nThe benchmark table in Section 3.2 (OpenAI Deep Research on BrowseComp, etc.) is also cited from product announcements, not new runs, so the 'significantly outperforms' claim is a citation rather than a demonstration. The GitHub star analysis is post hoc and the exclusion of top/bottom two repos is arbitrary, though it does add color.\n\nThese are real weaknesses, but they are conditions rather than fatal flaws. The paper's central claim — that agentic deep research is becoming the dominant mode of complex information seeking — is plausible and supported by the broader literature, and the survey's coverage of RL-trained search agents (Search-R1, R1-Searcher, DeepResearcher, ReSearch) is timely and reasonably accurate. I also credit the authors for including an 'Alternative View' section on human primacy and for listing open problems; too many position papers pretend there is no counterargument.\n\nWho is this for? A researcher wanting an entry point into the agentic deep research area, or someone who needs a single citation for the term and a quick survey of methods and benchmarks. It deserves a serious referee, but with a clear request: relabel the scaling contribution as a hypothesis, provide actual curves or error bars if they want to claim a law, and soften the 'inevitable' language in the abstract and conclusion. If those changes are made, the paper is a useful community resource.","headline":"A timely, well-organized position paper on agentic deep research whose useful taxonomy is weakened by a scaling 'law' that is really an interpolation of two unrelated benchmarks.","tokens_in":19640,"tokens_out":2565,"would_cite":true,"duration_ms":24178,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that agentic deep research—an LLM-driven loop of planning, searching, reasoning, and synthesizing—will become the dominant paradigm for information seeking and introduces a test-time scaling law to explain why.","keywords":["agentic deep research","test-time scaling","retrieval-augmented generation","reinforcement learning","information seeking","LLM agents","multi-hop retrieval","search paradigm shift"],"falsifier":"Run one integrated deep-research agent on a fixed set of open-ended web-research questions at several total token budgets, holding the task constant. If normalized accuracy does not rise roughly linearly with total inference tokens, or if doubling the search budget and doubling the reasoning budget together yields less than the sum of the individual gains, the linear test-time scaling law is falsified.","tokens_in":18535,"feed_emoji":"🔍","tokens_out":6513,"duration_ms":59052,"temperature":0.7,"pith_summary":"This paper stakes out a position: the next dominant way people will seek information is not a search engine that returns links, nor a chatbot that answers from memory, but an agentic deep research system—an LLM that plans a series of search steps, reads and reasons over what it finds, searches again, and finally synthesizes a report. The authors trace a line from keyword web search through LLM chatbots and retrieval-augmented generation to these autonomous research agents, arguing that only the last handles complex, multi-step, open-ended queries. To formalize why, they introduce a test-time scaling law: normalized performance improves linearly as inference-time computation grows, along two axes—internal reasoning depth and external knowledge exploration. They support the position with benchmark results on BrowseComp, BrowseComp-ZH, and Humanity's Last Exam, where a deep-research agent scores 51.5%, 42.9%, and 26.6% against standard LLMs that stay below 10–20%, and with rising open-source repository star trends. If the position is right, the user-facing interface to recorded knowledge shifts from ranked results to synthesized narrative produced by an agent.","feed_headline":"Agentic deep research is poised to replace web search","feed_subtitle":"The paper ties a linear test-time scaling law to agents that plan, search, and synthesize in a loop.","key_machinery":"The central mechanisms are two. First, the Agentic Deep Research loop: an LLM plans queries, retrieves, reasons, and iterates, with reasoning and search co-evolving rather than staged in sequence. Second, the Test-Time Scaling law: a hypothesized linear relation between normalized performance and inference-time compute along two axes, internal reasoning depth and external knowledge exploration, visualized as a diagonal plane interpolating two empirical scaling curves. The law frames a task-dependent token-budget trade-off between search and reasoning that the paper expects future systems to allocate adaptively.","core_discovery":"The paper's core claim is that Agentic Deep Research—defined as an LLM-driven feedback loop in which reasoning decides when, what, and how to search, and retrieved evidence in turn refines reasoning—will inevitably become the dominant paradigm for information seeking. The supporting formal object is the Test-Time Scaling law for Deep Research: when performance is normalized, it improves linearly with scaling of internal reasoning depth (longer chains of thought, self-refinement, self-consistency) and of external knowledge exploration (more iterative retrieval steps and long-context access). The paper is explicit that this law is a hypothesis: the diagonal plane in Figure 3b interpolates between reasoning data from S1 on AIME24 and search data from IterDRAG on MuSiQue, and different task types align with different planes in the search–reasoning trade-off. It further argues that reinforcement learning, rather than prompting or supervised fine-tuning, is the principled mechanism for incentivizing reasoning-driven search in open-ended environments.","pith_inferences":["An editorial extension: if the linear test-time scaling law is real, search engines may evolve into compute-budget allocators that decide how many tokens go to retrieval versus reasoning for each query, making token cost a first-class search metric.","The paper's two-axis trade-off suggests a concrete design target: an agent that predicts whether a query is search-heavy or reasoning-heavy and sets its own budget accordingly, which could be trained by reinforcement learning on reward signals for answer quality minus compute cost.","A testable consequence the paper does not state: the same linear trend should appear on a single integrated benchmark if total inference budget is varied, and not just as an interpolation of two separate curves.","If the position is correct, evaluating information-seeking systems shifts from retrieval metrics like precision and recall to end-user outcomes such as report correctness, synthesis quality, and reduced cognitive load."],"forward_implications":["If the scaling law holds, spending more inference tokens on either deeper reasoning or more search steps should yield predictable, roughly linear accuracy gains on knowledge-intensive tasks.","Standard web search and single-pass retrieval-augmented generation will be inadequate for complex multi-step queries, pushing users toward agentic systems that plan and iterate.","Reinforcement learning, rather than prompting or supervised fine-tuning, becomes the key training method for teaching search agents to explore, self-correct, and adapt.","Benchmark gaps of the kind reported—51.5% versus below 10% on BrowseComp, 26.6% versus under 20% on HLE—suggest that agentic deep research is not a marginal improvement but a categorical jump.","Open-source momentum around agentic research implementations indicates the paradigm shift is already happening in the developer community, not just in commercial products."],"supporting_citations":[{"why":"Supplies the reasoning test-time scaling data (S1 on AIME24) that defines one axis of the TTS law.","marker":"(Muennighoff et al., 2025)"},{"why":"Supplies the search test-time scaling data (IterDRAG on MuSiQue) that defines the other axis.","marker":"(Yue et al., 2025)"},{"why":"Provides the BrowseComp benchmark used to show the gap between standard LLMs and a deep-research agent.","marker":"(Wei et al., 2025a)"},{"why":"Provides Humanity's Last Exam, the closed-book reasoning benchmark used to show agent superiority on non-retrievable questions.","marker":"(Phan et al., 2025)"},{"why":"Exemplifies reinforcement-learning-incentivized reasoning, the foundation the paper argues makes agentic research possible.","marker":"(Guo et al., 2025)"},{"why":"Supplies Search-R1, an RL-trained search agent that supports the claim that RL incentivizes reasoning-driven search.","marker":"(Jin et al., 2025)"},{"why":"Introduces ReAct, the prompting paradigm of interleaving reasoning and acting that the paper positions as groundwork for agentic search.","marker":"(Yao et al., 2023c)"},{"why":"Provides WebGPT, early evidence that reward-based training can guide multi-step web search behavior.","marker":"(Nakano et al., 2021)"}],"fun_headline_variants":["Agentic deep research: linear scaling with reasoning depth","Why agentic research beats web search: RL-driven loops","Search + reasoning agents: the new deep research paradigm","Test-time scaling law for agentic search loops"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The scaling law's load-bearing premise is that performance trends measured separately on a math benchmark and a retrieval benchmark can be merged into one linear law that predicts how an integrated agent behaves on real research tasks, even though no integrated system was measured to produce that law.","fun_headline_variants_meta":{"raw":{"variants":["Agentic deep research: linear scaling with reasoning depth","Why agentic research beats web search: RL-driven loops","Search + reasoning agents: the new deep research paradigm","Test-time scaling law for agentic search loops"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000206,"raw_usage":{"total_tokens":1391,"prompt_tokens":937,"completion_tokens":454,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":391}},"tokens_in":553,"tokens_out":454,"duration_ms":5056,"temperature":1.0,"reasoning_tokens":391,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:41:21.575326+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run one integrated deep-research agent on a fixed set of open-ended web-research questions at several total token budgets, holding the task constant. If normalized accuracy does not rise roughly linearly with total inference tokens, or if doubling the search budget and doubling the reasoning budget together yields less than the sum of the individual gains, the linear test-time scaling law is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the search test-time scaling data (IterDRAG on MuSiQue) that defines the other axis."}],"review_version":2}