{"id":"8a69128c-1f63-4551-b200-6997cd607b6d","arxiv_id":"2508.03012","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"ToolTrain combines rejection-sampled supervised fine-tuning and reinforcement learning to improve LLM-based repository deep search for issue localization, achieving state-of-the-art results.","lead":"ToolTrain trains large language models to use repository retrieval tools for issue localization through a two-stage fine-tuning and reinforcement learning process. On function-level localization, the 32B model reportedly beats Claude-3.7, and the gains translate to better end-to-end issue resolution.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'localization improves end-to-end resolution' claim lacks a clean comparison showing the gain is due to localization skill rather than generic tool-use RL; an oracle-ablation or task-matched control would settle it.","rationale":"The strongest claim has two components: SOTA localization and improved localization causes better end-to-end resolution. The second component is the one that supports the paper's broader conclusion and is the least secure. The abstract provides no evidence isolating the mechanism; the training process could improve end-to-end performance through many channels. The proposed oracle/control tests would directly test mediation. In contrast, benchmark representativeness is a standard concern that applies to any empirical paper and is less specific to this contribution. I agree with the reader's UNVERDICTED status due to abstract-only availability, and my concern does not change the verdict; it identifies the first checkpoint a full review should examine.","tokens_in":599,"tokens_out":2466,"duration_ms":29696,"concrete_test":"In the full paper, check for an ablation/control: train a model with the same SFT+RL pipeline but with the reward or task replaced by a non-localization tool-use objective (e.g., retrieving and summarizing relevant files without locating an edit site). If the end-to-end issue resolution improvement persists under this control, the localization-specific training is not the driver. Alternatively, run an oracle test: with ground-truth edit locations injected into the model's context, measure end-to-end resolution; compare against the ToolTrain model's own localization output. If injecting oracle locations does not raise resolution accuracy over ToolTrain's predicted locations, then the claimed translation from localization gains to resolution is not established by the current comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central causal claim is that improved localization performance 'translates to' better end-to-end issue resolution. But the training recipe (rejection-sampled SFT + tool-integrated RL) changes multiple factors at once: the model learns to follow tool-use conventions, to search longer, to format actions, and only incidentally to localize. If the end-to-end gain is observed when comparing ToolTrain-trained models to base pretrained models, the gain could come from general instruction-following or tool-use competence rather than from the localization objective specifically. The paper's conclusion that 'training for issue localization is a viable and effective strategy' requires that the localization signal is the operative one. The abstract does not report an ablation (e.g., SFT-only vs SFT+RL, or a control trained on a different retrieval task) nor an oracle experiment that directly ties localization accuracy to resolution success. Without this, the headline claim is underdetermined. Benchmarks representativeness is a separate, secondary concern; the more load-bearing gap is the missing mediation evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ToolTrain, a two-stage training framework that combines rejection-sampled supervised fine-tuning with tool-integrated reinforcement learning to improve LLMs' ability to localize code issues through multi-step repository retrieval, a task the authors call Repo Deep Search. The abstract reports that ToolTrain-trained models achieve state-of-the-art function-level localization performance, with a 32B model surpassing Claude-3.7, and that improved localization transfers to better end-to-end issue resolution. This review is based only on the abstract; the full manuscript was not available for inspection.","tokens_in":799,"tokens_out":1759,"duration_ms":21345,"significance":"If the reported results hold, the paper would make a meaningful contribution by demonstrating that a specialized training objective for issue localization can produce measurable gains on downstream automated software development. The Repo Deep Search formulation addresses a real and increasingly important challenge in LLM-based software engineering, and the comparison against frontier models gives the claim practical relevance. However, the significance depends on the strength of the causal mediation claim and on the integrity of the benchmarks, neither of which can be assessed from the abstract alone.","major_comments":[{"comment":"The central claim that 'improved localization performance translates to better end-to-end issue resolution performance' is a mediation claim, but the experimental comparison described in the abstract changes multiple factors at once: the training recipe adds tool-use conventions, search behavior, and action formatting alongside the localization objective. Without a clean ablation (e.g., SFT-only versus SFT+RL, or a control trained on a different retrieval task) or an oracle-locator experiment that directly ties localization accuracy to resolution success, the observed end-to-end gain could be attributed to generic tool-use competence rather than localization skill. The full text must provide such evidence or explicitly soften this claim.","section":"Abstract (end-to-end claim)"},{"comment":"The abstract reports state-of-the-art results and a specific comparison against Claude-3.7 on function-level localization, but it does not identify the benchmarks, dataset composition, baselines, error bars, or evaluation protocol. Since localization accuracy and end-to-end issue resolution are measured separately, the reader cannot determine whether the two evaluations are matched in repository complexity, issue type, or data source. The full manuscript must disclose these details to support the headline numbers.","section":"Abstract (benchmark evidence)"},{"comment":"The concluding claim that 'training for issue localization is a viable and effective strategy' generalizes from the evaluated benchmarks to software development broadly. The abstract provides no information about the diversity of repositories, issue types, programming languages, or project sizes covered, nor about protections against data leakage or reward hacking during tool-integrated RL. The full text should address benchmark representativeness and potential leakage risks to justify the general conclusion.","section":"Abstract (generalization)"}],"minor_comments":[{"comment":"The term 'Repo Deep Search' is introduced but not formally defined; the full text should provide a precise task definition that distinguishes it from standard retrieval-augmented generation or agentic search.","section":"Abstract (terminology)"},{"comment":"The abstract does not mention whether code, trained models, or evaluation harnesses will be released; public availability of these artifacts would substantially strengthen the contribution.","section":"Abstract (reproducibility)"}],"recommendation":"uncertain","confidential_remarks":"This review is based solely on the abstract, so I cannot issue a soundness verdict. The main risk is the mediation claim that localization improvement causes end-to-end improvement; the full manuscript should be checked for ablations or oracle experiments that isolate the localization signal. I recommend a full review once the complete text is available."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe only thing we have is the abstract, so anything beyond that is provisional. What the paper actually introduces is a two-stage training pipeline—rejection-sampled SFT followed by tool-integrated RL—for what they call Repo Deep Search: using retrieval tools over a whole repository to find the code location that resolves an issue. That framing is a fair way to characterize what LLM agents are being asked to do, and the reported numbers (32B model beating Claude-3.7 on function-level localization, plus downstream end-to-end gains) are the kind of result that would move the needle if they hold up. I think the method is genuinely new in its combination.\n\nThe soft spot is the causal claim. The abstract says improved localization 'translates to' better end-to-end issue resolution, and that training for localization is a viable strategy. But the training recipe changes multiple things at once: tool-use conventions, search length, action formatting, instruction following. Without an ablation separating the localization signal from general tool-use RL, or an oracle experiment that directly ties localization accuracy to resolution success, the end-to-end gain could just be a smarter agent, not a better localizer. The stress-test note is right that this is the load-bearing gap. Benchmark representativeness is a secondary worry; the mediation evidence is the one to settle.\n\nFor an abstract-only review, I can't say whether the full paper contains the missing control. If it does—or even if it honestly discusses the limitation—this is a solid, citable contribution. If not, the headline claim is oversold. As it stands, the abstract is promising and the method deserves a serious referee. I'd send it to review and ask the authors to either provide the ablation or soften the causal language.\n\nWho's it for: people working on LLM agents for software engineering, issue tracking, and repository-scale code understanding. Good reading-group material once the full text is out.","headline":"Promising two-stage training recipe for repo-level issue localization; the end-to-end causal claim needs a cleaner ablation to carry its weight.","tokens_in":1223,"tokens_out":2078,"would_cite":false,"duration_ms":25670,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ToolTrain-trained LLMs beat Claude-3.7 at finding the code that must change to resolve a software issue.","keywords":["issue localization","Repo Deep Search","tool-integrated reinforcement learning","rejection-sampled supervised fine-tuning","LLM agents","code retrieval tools","automated software development","function-level localization"],"falsifier":"Run the ToolTrain-trained 32B model and Claude-3.7 on held-out issue-localization benchmarks drawn from repositories and tool backends different from those used in training; the claim weakens if the ToolTrain model's function-level localization advantage disappears or if improved localization fails to raise end-to-end resolution rates outside the original benchmark suite.","tokens_in":468,"feed_emoji":"🔍","tokens_out":3194,"duration_ms":34530,"temperature":0.7,"pith_summary":"This paper argues that the hard part of resolving a software issue is not writing code but finding the code that must change, a task it names Repo Deep Search: multi-hop reasoning through repository structure with retrieval tools. It proposes ToolTrain, a two-stage training pipeline in which an LLM first learns from rejection-sampled traces of tool use and then refines its search behavior with tool-integrated reinforcement learning. The claim is that this recipe materially improves function-level issue localization, putting a 32B model ahead of Claude-3.7 on the benchmark tested, and that better localization carries through to more issues actually resolved end to end. If true, targeted training for localization becomes a practical route to stronger automated software development without scaling up model size.","feed_headline":"ToolTrain beats Claude-3.7 at code-localization search","feed_subtitle":"A two-stage training recipe makes LLMs search repos better, and better search means more issues actually fixed.","key_machinery":"The central object is Repo Deep Search, defined as the multi-step process in which an LLM uses repository retrieval tools to locate code needing modification. The machinery that carries the argument is ToolTrain's two-stage training loop: first rejection-sampled supervised fine-tuning, where the model is trained on tool-use trajectories that were sampled and then filtered by outcome, and second tool-integrated reinforcement learning, where the model learns to maximize a reward tied to localization success while actually calling retrieval tools. The second stage is what makes tool use trainable end to end rather than fixed by imitation data.","core_discovery":"On the paper's own terms, the central discovery is that issue localization can be treated as a trainable tool-use skill rather than a side effect of general reasoning. Given a natural language issue report, the model must decide which repository retrieval tool to call, in what order, and when to stop, across a multi-step navigation of code dependencies. ToolTrain attacks this by first using rejection-sampled supervised fine-tuning to teach the model high-quality search trajectories, then applying tool-integrated reinforcement learning to optimize the policy directly against localization success. The reported results position ToolTrain-trained models as state of the art, with the 32B model surpassing Claude-3.7 at function-level localization, and the paper further reports that this localization gain transfers to end-to-end issue resolution. The authors conclude that localization-specific training is a viable and effective strategy for improving automated software development.","pith_inferences":["The same two-stage recipe could plausibly transfer to other retrieval-heavy agent tasks, such as API search, documentation navigation, or test-failure diagnosis, since none of those depend on the specific repository structure of the benchmark.","Because the reported gain is measured on function-level localization, one testable extension is to measure precision and recall of exact edit locations against independent human labels, which would separate genuine search skill from benchmark artifacts.","If localization training is folded into a general agent-training pipeline, the tool-integrated RL stage may reduce the need for end-to-end issue-resolution rewards, which are sparse and expensive; the paper does not test this.","The apparent advantage over a much larger proprietary model suggests that targeted tool-use training can partially substitute for raw scale, though the paper does not make this comparison across multiple model families."],"forward_implications":["Models trained with ToolTrain reach state-of-the-art function-level localization, with the 32B variant ahead of Claude-3.7 on the reported benchmark.","Improvements in localization accuracy translate into better end-to-end issue resolution performance, not just better search metrics.","Training specifically for issue localization is a viable and effective strategy for improving automated software development.","The two-stage design implies that teaching high-quality tool-use trajectories first, then optimizing them with reinforcement learning, is the operative recipe for turning retrieval tools into reliable search behavior."],"supporting_citations":[],"fun_headline_variants":["ToolTrain's two-stage training sharpens repo search","RL fine-tunes LLM repo search to beat Claude-3.7","ToolTrain: better bug hunting via tool-aware RL","Two-stage training teaches LLMs to search repos","ToolTrain's RL beats Claude-3.7 on repo search"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the assumption that the benchmarks used to measure localization and end-to-end resolution faithfully represent real-world code repositories, so the gains are not an artifact of the specific training data, baselines, or random seeds.","fun_headline_variants_meta":{"raw":{"variants":["ToolTrain's two-stage training sharpens repo search","RL fine-tunes LLM repo search to beat Claude-3.7","ToolTrain: better bug hunting via tool-aware RL","Two-stage training teaches LLMs to search repos","ToolTrain's RL beats Claude-3.7 on repo search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000569,"raw_usage":{"total_tokens":2669,"prompt_tokens":900,"completion_tokens":1769,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":1686}},"tokens_in":516,"tokens_out":1769,"duration_ms":14403,"temperature":1.0,"reasoning_tokens":1686,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T04:42:07.706864+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the ToolTrain-trained 32B model and Claude-3.7 on held-out issue-localization benchmarks drawn from repositories and tool backends different from those used in training; the claim weakens if the ToolTrain model's function-level localization advantage disappears or if improved localization fails to raise end-to-end resolution rates outside the original benchmark suite.","supporting_citations":[],"review_version":1}