{"id":"b09184c5-12dd-4be1-89e2-7290ba17cbc5","arxiv_id":"2412.19031","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Training 7B code models on automatically constructed repository-localization and code-edit data improves their SWE-Bench-verified resolution rate from about 1% to 7%.","lead":"This paper proposes ReSAT, a method that creates training data from real GitHub issues and pull requests to teach small language models how to find the right files, functions, and lines in a codebase and how to edit them. Fine-tuning two 7B open-source models on this data raised their issue-resolution rates on the SWE-Bench-verified benchmark, but the gains are modest and the models remain far behind large commercial systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gains are not causally isolated: without a flat-SFT control on the same issue-edit data, the structure-aware decomposition may not be the cause; the reported improvements could reflect generic supervised fine-tuning on GitHub issue-fix data.","rationale":"The paper's central contribution is the structure-aware decomposition of training data. The evaluation only compares ReSAT against the base instruct models; the ablation separates localization data from code-edit data but not ReSAT from flat supervised fine-tuning on the same issue-fix corpus. Since both ablation variants improve over the base model, generic SFT on GitHub issue-fix data is a plausible confound. The magnitude of the Agentless gains (4.8 to 6.4 points on 500 instances) is large and unlikely to be pure noise, but the attribution to structure-awareness is not established. This is a correctness risk for the title claim, not for the practical observation that the method performs well in the tested configuration. Running the flat-SFT control and, ideally, releasing the training data and code would settle the causal question. The reader's CONDITIONAL verdict already captures this uncertainty, so no change to the verdict is needed.","tokens_in":13709,"tokens_out":6733,"duration_ms":65887,"concrete_test":"Train CodeQwen-7B on a flat control dataset derived from the same 80,260 samples: keep the same code-edit supervision and hyperparameters (2 epochs, lr 5e-6, batch 128, 32k context) but remove the hierarchical decomposition, e.g., issue + repository file list to Search/Replace edit, or flatten localization targets into one instruction. Evaluate under Agentless on SWE-Bench-verified with at least 3 random seeds. If the flat control matches ReSAT's percent resolved within about 1.5 points, the structure-aware decomposition is not the causal ingredient; if it trails by more than about 2 points, the central claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 1 compares ReSAT fine-tuned models only against the original instruct models: CodeQwen 0.8 to 7.2 and Deepseek-Coder 1.8 to 6.6 under Agentless. Section 3.4 describes training on 80,260 samples built from PRs, and the ablation in Figure 3 separates only-localization from only-code-edit, but no control is trained on the same underlying PR-derived issue-edit pairs without the multi-level file/function/line decomposition. Because both ablations already improve over the base model (onlyEdit: 0.8 to 1.8; onlyLoc: 0.8 to 4.8), any supervised fine-tuning on similar GitHub issue-fix data could plausibly produce part or all of the gain. The RepoQA results in Table 2, where Python-only training improves accuracy in other languages (Deepseek avg 10.6 to 15.0), further suggest a generic instruction-following or format effect rather than specifically repository-structure understanding. Thus the paper's title and abstract claim that structure-aware training is the cause of the improvement is not isolated; the numbers support that ReSAT works in this configuration, but not that the structure awareness is the operative ingredient.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ReSAT (Repository Structure-Aware Training), a method for constructing training data from GitHub issues and pull requests to improve the repository-level issue-resolving and long-context understanding capabilities of small language models (SLMs). The data consists of multi-level localization tasks (file, function, line) and code-edit tasks. The authors fine-tune CodeQwen1.5-7B-Chat and Deepseek-Coder-6.7B-Instruct, evaluate on SWE-Bench-verified and RepoQA, and report substantial improvements in %Resolved (e.g., from 0.8% to 7.2% for CodeQwen under Agentless). Ablation studies on single-task data are presented, along with a case study and comparisons with LLM-based baselines in the appendix.","tokens_in":13974,"tokens_out":2627,"duration_ms":25296,"significance":"If the reported gains are robust, ReSAT provides a low-cost, LLM-free approach to improving SLMs on repository-level tasks, and the paper offers a concrete data-construction pipeline that the community could build on. The evaluation covers two SLMs, two issue-resolving frameworks, and a long-context benchmark, and the authors exclude SWE-Bench and RepoQA repositories from the training data, reducing direct leakage. The ablation study separates the contribution of localization versus code-edit data. However, the paper's central causal claim — that the structure-aware decomposition is the operative ingredient — is not isolated, and several evaluation gaps (single runs, unexplained per-language drops) prevent the evidence from fully supporting that claim as stated.","major_comments":[{"comment":"The paper's main claim is that structure-aware training (the multi-level file/function/line decomposition) is the cause of the gains, but no control is trained on the same raw issue-edit pairs without this decomposition. The ablation in Figure 3 shows that both only-edit (0.8→1.8) and only-localization (0.8→4.8) improve over the base model, so any supervised fine-tuning on similar GitHub issue-fix data could plausibly account for part or all of the improvement. Without a flat-SFT control on the same PR-derived data, the attribution to structure awareness is not isolated.","section":"§3.4 / Figure 3"},{"comment":"The per-language RepoQA results contain a large, unexplained drop for Deepseek-Coder on C++ (from 21 to 14) and a smaller drop for CodeQwen on TypeScript (67 to 65). The text states that ReSAT improves accuracy in most languages, which is true, but the C++ drop is substantial and should be discussed or analyzed. This is particularly relevant because the paper argues that Python-only training improves cross-lingual understanding, and a decline in one language weakens that claim without an explanation.","section":"§4.5 / Table 2"},{"comment":"All experiments appear to be single runs with no confidence intervals or significance tests. Because SWE-Bench-verified has 500 tasks, the reported differences are small in absolute terms (e.g., 0.8% vs 7.2% corresponds to 4 vs 36 issues, and a 0.8% change is about 4 issues). The paper should either provide variance estimates from repeated runs or acknowledge the sensitivity of these percentages to small absolute changes and discuss whether the observed gaps are statistically meaningful.","section":"§4.2 / Table 1"},{"comment":"The data-construction pipeline links issues to PRs using regular expressions on PR titles and commit messages, with no manual verification or filtering for whether the PR indeed resolves the mentioned issue. This introduces label noise in the supervisions signals, and the paper does not quantify the quality of these automatically generated links. A small human-validated sample or a heuristic check (e.g., whether the PR diff touches files related to the issue text) would help establish the reliability of the 80,260 samples.","section":"§3.1"}],"minor_comments":[{"comment":"The manuscript contains numerous typographical errors and formatting inconsistencies, including 'flollow', 'furute', 'utilze', 'lanuages', 'Pull Requets', 'webaset', and 'capibility'. A thorough proofreading pass is needed.","section":"Throughout"},{"comment":"The figure lacks error bars or any indication of run-to-run variance, which is especially important given the small absolute numbers in the SWE-Bench-verified results.","section":"Figure 3"},{"comment":"The comparison with SWE-Llama and CodeQwen-Opendevin confounds the training method with the inference framework and the underlying base model; this is acknowledged in the text, but it would be clearer to present it strictly as an end-to-end comparison and not as evidence specifically for ReSAT's data-construction choices.","section":"Appendix B, Table 4"},{"comment":"The description of line-localization data says that 'we introduce irrelevant functions as distractions' and also includes the modified function name in the output; this conflates function-level and line-level localization and should be described more precisely to avoid ambiguity about the task format.","section":"§3.2.3"},{"comment":"The paper states that RepoQA has 500 test samples across five languages; providing the per-language sample counts would help assess the reliability of the per-language accuracies reported in Table 2.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The central idea is interesting and the empirical gains are non-trivial, but the manuscript currently overstates the causal role of the structure-aware decomposition. The missing flat-SFT control is the key blocker; without it, the paper reads as 'curated GitHub data helps SFT,' which is less novel than the title claims. The authors should either add such a control (ideally with the same data but without the file/function/line hierarchy) or temper the title and abstract accordingly. Also, the single-run evaluation and unexplained per-language drops should be addressed in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ReSAT is worth knowing about: it takes real GitHub issue-PR pairs and turns them into three levels of localization training plus code-edit training, and that fine-tuning lifts two 7B models from around 1% to 7% resolved on SWE-Bench-verified under Agentless. That is a real, practically meaningful gain for privacy-sensitive or on-premise issue fixing. The paper is honestly written and the main results are consistent across CodeQwen and Deepseek-Coder.\n\nThe genuinely new part is the decomposition of issue-PR pairs into file, function, and line localization alongside edit generation, built without LLM-based synthesis. The ablation study is a good step: both localization-only and edit-only training improve over base, and combining them helps further. That supports the value of the overall recipe.\n\nThe soft spot is the one the reader identified: the paper does not include a control that trains on the same issue-edit pairs without the file/function/line hierarchy. The onlyEdit and onlyLoc ablations both improve over the base model, so a generic supervised fine-tuning effect on similar GitHub issue-fix data could account for part or all of the gain. Because the title and abstract claim that structure awareness is the operative ingredient, this control really matters. The authors should add it, and the current numbers cannot distinguish 'ReSAT works' from 'any SFT on issue-fix data works.'\n\nOther soft spots are minor by comparison: single runs with no error bars, an unexplained drop for Deepseek on RepoQA C++ (21 to 14), no released data or code, and an appendix comparison to SWE-Llama and Opendevin that confounds framework, base model, and training data. The leakage handling is good: they exclude SWE-Bench and RepoQA repositories from the training crawl, so the evaluation is not directly overlapped, even if the training distribution is similar.\n\nI'd send this to peer review. The empirical recipe is potentially useful and the evaluation is on a credible benchmark. The reviewers should push for a flat-SFT control and repeated runs, and the authors should be asked to release data. If the control shows generic SFT matches ReSAT, the contribution shrinks to 'SFT on issue-fix pairs helps,' which is still worth knowing but not what the title promises.","headline":"ReSAT gives a real gain for 7B models on SWE-Bench-verified, but the paper does not isolate whether the structure-aware decomposition is the cause rather than generic SFT on issue-fix data.","tokens_in":14510,"tokens_out":2586,"would_cite":true,"duration_ms":25450,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Repository structure-aware training lifts a small code model's issue-fixing rate on SWE-Bench-verified from 0.8% to 7.2%.","keywords":["ReSAT","repository structure-aware training","small language models","issue resolving","localization training","code edit training","SWE-Bench-verified","RepoQA"],"falsifier":"Train the same two base models on the same 80,260 issue-to-edit samples but flatten the supervision—use only issue-to-diff or issue-to-modified-file pairs instead of the file/function/line hierarchy. If the flat models match ReSAT's 7.2% and 6.6% resolved rates on SWE-Bench-verified, the hierarchy is not the active ingredient; if the flat models fall back to the 0.8%–1.8% range, the structure-aware decomposition is what carries the gain.","tokens_in":13518,"feed_emoji":"🛠️","tokens_out":17015,"duration_ms":291726,"temperature":0.7,"pith_summary":"This paper introduces ReSAT, a fine-tuning recipe that teaches small language models (13 billion parameters or fewer) to resolve real software issues by exploiting the structure of code repositories. The authors automatically collect 80,260 training samples from 229 popular open-source Python projects by linking resolved issues to the pull requests that fixed them, then organize each sample into four supervised tasks: locate the files, locate the functions, locate the lines, and produce the code edit. On SWE-Bench-verified—a benchmark of real repository issues with executable tests—ReSAT raises the resolved-issue rate of CodeQwen1.5-7B-Chat from 0.8% to 7.2% and of Deepseek-Coder-6.7B-Instruct from 1.8% to 6.6%. Both models also gain on the RepoQA long-context benchmark, including in languages other than Python. The paper's thesis is that hierarchical localization practice, not raw code exposure, is what makes small models better issue resolvers.","feed_headline":"9x issue-fix gain for small code models after structure-aware training","feed_subtitle":"A 7B open model reaches 7.2% on a real issue benchmark, up from 0.8%, with no API call.","key_machinery":"The load-bearing object is the multi-level progressive localization dataset. It consists of (1) file-level localization, where the input is the issue plus the repository file tree and the output is the modified files; (2) function-level localization, where the input is the issue plus a file skeleton built from class and function declarations and the output is the modified functions; (3) line-level localization, where the input is the issue plus the full function content, with distractor functions added, and the output is the exact modified lines; and (4) code edit generation, where the input is the issue plus the localized snippet and the output is a Search/Replace edit. This hierarchy mirrors the inference-time steps of the Agentless framework, so fine-tuning on it is intended to teach the model to decompose a repository into candidate locations before making changes.","core_discovery":"The central claim is that supervised fine-tuning on repository-structure information can substantially improve small models' ability to solve real-world coding issues, without resorting to larger models or agentic scaffolding. The authors call the recipe ReSAT and build it from real merged pull requests: for each issue, the training target is the set of files modified by the fix, the set of functions modified within those files, the set of lines modified within those functions, and the final Search/Replace edit. Trained on this data, CodeQwen1.5-7B-Chat and Deepseek-Coder-6.7B-Instruct both show large jumps in resolved issues on SWE-Bench-verified when used inside the Agentless pipeline—a two-phase framework that first localizes relevant code and then generates edits—and improved accuracy on RepoQA. The paper interprets the ablation result—localization data alone helps more than edit data alone, and the two combine best—as evidence that the structure-aware localization task is the active ingredient.","pith_inferences":["A direct control that would separate the structure-aware contribution from generic supervised fine-tuning is to train on the same 80,260 issue-edit pairs with the hierarchy flattened; until that control is run, part of the gain should be attributed to additional supervised fine-tuning on real issue-fix data.","Because the issue–pull-request links are extracted by regular expressions from titles and commit messages, some fraction of the 80,260 samples is likely mislabeled, so link cleaning and denoising are a natural next step that could raise the measured ceiling.","The same file/function/line/edit hierarchy can be generated from any diff, so ReSAT-style objectives could transfer to other repository-level tasks such as code review, refactoring, or test generation.","Since Python-only ReSAT data improves RepoQA in four other languages, training on multilingual issue-fix pairs is an obvious extension; whether SWE-Bench-style resolution gains transfer to non-Python repositories remains an open test."],"forward_implications":["A 7B-parameter model fine-tuned with ReSAT resolves 7.2% of SWE-Bench-verified issues through Agentless, a roughly ninefold improvement over the untrained CodeQwen baseline.","Localization data contributes more to the gains than code-edit data does, and using both together gives the best result, so the two training objectives are complementary.","Training on Python-only ReSAT data improves RepoQA accuracy in C++, Rust, Java, and TypeScript, suggesting the learned localization skill transfers across programming languages.","ReSAT narrows but does not close the gap with much larger commercial models on the same pipelines.","ReSAT-trained small models solve some issues that a frontier model fails on, so small and large models can complement each other in an issue-resolving system."],"supporting_citations":[{"why":"Supplies SWE-Bench, the RAG-SWE baseline framework, and the SWE-Llama comparison used to frame ReSAT's evaluation.","marker":"Jimenez et al., 2024"},{"why":"Supplies Agentless, the two-phase localization-and-edit framework in which ReSAT's main gains are measured and whose prompts inspired the training data design.","marker":"Xia et al., 2024"},{"why":"Supplies CodeQwen1.5-7B-Chat, one of the two base small language models fine-tuned by ReSAT.","marker":"Team, 2024"},{"why":"Supplies Deepseek-Coder-6.7B-Instruct, the other base small language model fine-tuned by ReSAT.","marker":"Guo et al., 2024"},{"why":"Supplies RepoQA, the long-context repository understanding benchmark used to measure ReSAT's transfer effects.","marker":"Liu et al., 2024"},{"why":"Supplies SWE-Bench-verified, the human-verified issue-resolving test set where the headline resolution rates are measured.","marker":"OpenAI, 2024"}],"fun_headline_variants":["Structure-aware training lifts small code models to 9x issue-fix rate","ReSAT: small models solve real issues 9x better with repo-aware training","Small LM issue-fixing jumps from 0.8% to 7.2% after repository-aware training","Train small code models on repo structure to multiply issue-fix success","New training recipe gives 7B models a 9x boost on real coding issues"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automatically scraped issue-pull-request pairs carry correct, transferable supervision, and that the measured gains come from the structure-aware hierarchy rather than from extra fine-tuning on similar issue-fix data.","fun_headline_variants_meta":{"raw":{"variants":["Structure-aware training lifts small code models to 9x issue-fix rate","ReSAT: small models solve real issues 9x better with repo-aware training","Small LM issue-fixing jumps from 0.8% to 7.2% after repository-aware training","Train small code models on repo structure to multiply issue-fix success","New training recipe gives 7B models a 9x boost on real coding issues"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000251,"raw_usage":{"total_tokens":1546,"prompt_tokens":926,"completion_tokens":620,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":525}},"tokens_in":542,"tokens_out":620,"duration_ms":6118,"temperature":1.0,"reasoning_tokens":525,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:57:16.161676+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same two base models on the same 80,260 issue-to-edit samples but flatten the supervision—use only issue-to-diff or issue-to-modified-file pairs instead of the file/function/line hierarchy. If the flat models match ReSAT's 7.2% and 6.6% resolved rates on SWE-Bench-verified, the hierarchy is not the active ingredient; if the flat models fall back to the 0.8%–1.8% range, the structure-aware decomposition is what carries the gain.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies SWE-Bench-verified, the human-verified issue-resolving test set where the headline resolution rates are measured."}],"review_version":1}