{"id":"5a5a5223-9aeb-4400-b0bf-b60c32200998","arxiv_id":"2509.08724","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A pipeline that mirrors real GitHub issues into existing test environments of other repositories, producing over 60,000 verifiable issue-resolving tasks that improve code-agent training.","lead":"SWE-Mirror turns real GitHub bugs and feature requests from one software project into near-identical, testable tasks inside a different project that already has an automated test environment, creating 60,671 training examples. Training smaller code models on these examples roughly doubled to tripled their success rate on the SWE-Bench-Verified benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Train/eval repository overlap may confound benchmark gains: target repos are drawn from SWE-Gym, SWE-rebench, and Multi-SWE-RL and likely intersect SWE-Bench-Verified/Multi-SWE-Bench, so improvements could reflect repo familiarity rather than transferable skill.","rationale":"The reader's designated weakest assumption is semantic fidelity of LLM-generated test.patch and mirror.patch. That is a real risk, and the small human audit (184 tasks) leaves residual uncertainty. However, the most load-bearing condition for the paper's central claim is that the benchmark improvements reflect transferable issue-resolving ability. That condition is directly threatened by unanalyzed overlap between the 40 training repositories and the evaluation benchmarks, since the paper selects target Gyms from SWE-Gym, SWE-rebench, and Multi-SWE-RL, and SWE-Bench-Verified's repo set is small and well-known. The reader's rationale does mention this overlap concern in passing, so my agreement is partial rather than full. Because the concern is addressable with a straightforward analysis and the paper's other evidence (e.g., the scaling curves, human audit) is not fundamentally invalid, the correct verdict remains CONDITIONAL; my stress test does not move it to REJECT or ACCEPT. If the overlap check later shows large contamination, the verdict should be reconsidered downward.","tokens_in":19435,"tokens_out":9185,"duration_ms":433735,"concrete_test":"Release the list of 40 target repository full names and intersect it with the repository lists of SWE-Bench-Verified and Multi-SWE-Bench-Flash; then re-evaluate SWE-Mirror-LM-7B and 32B separately on instances from overlapping and non-overlapping repositories. If the resolve-rate gain on non-overlapping repos is substantially smaller than the overall gain (e.g., more than 30% relative), or if the gain concentrates in the overlapping repos, the generalizability claim is not supported. A stronger version is to retrain on SWE-Mirror trajectories restricted to repos absent from the evaluation benchmarks and confirm the improvement persists.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that training on SWE-Mirror trajectories materially improves issue-resolving, with SWE-Mirror-LM-7B rising from 1.0% to 22.8% and 32B from 6.2% to 52.2% on SWE-Bench-Verified (Table 6). This inference requires that gains on these benchmarks measure transferable skill rather than familiarity with the specific repositories used for training. The paper does not report whether the 40 target repositories, selected from SWE-Gym, SWE-rebench, and Multi-SWE-RL (Section 2), intersect the SWE-Bench-Verified or Multi-SWE-Bench-Flash repo sets, nor does it stratify results by overlap. Because SWE-Bench-Verified uses a small set of well-known Python repositories and SWE-Gym's 11 repos are known to include several of the same projects, overlap is plausible. If a model trains on many tasks in, e.g., django or sympy, its score on SWE-Bench-Verified tasks from those repos could rise due to repo-specific file layout, test conventions, and API idioms, even if the mirrored issues themselves are semantically faithful. The isolated scaling experiments in Figure 2 use only SWE-Mirror trajectories but evaluate on the same benchmarks and inherit the same confound; the final models also mix in 6,025 SWE-rebench trajectories (Section 3.1). Without an overlap analysis or a held-out-repo evaluation, the headline numbers do not establish the central claim of general issue-resolving improvement.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SWE-Mirror, a pipeline that takes real GitHub pull requests and issues from source repositories and re-instantiates them as verifiable tasks inside existing Gym environments of 40 target repositories across four languages. The released artifact is SWE-Mirror-60K, containing 60,671 tasks, each composed of a problem statement, a mirror.patch that injects a bug or reverts a feature, a hidden test.patch, and a fix.patch that is the inverse of mirror.patch. The authors report a 46.0% end-to-end mirroring yield, a human audit on 184 mirrored tasks, and post-training experiments on Qwen2.5-Coder-Instruct 7B/32B under the OpenHands agent, claiming improvements on SWE-Bench-Verified from 1.0% to 22.8% and from 6.2% to 52.2%, together with a new SOTA among Qwen2.5-Coder-Instruct-based models. They also report scaling experiments and cross-lingual transfer experiments.","tokens_in":19720,"tokens_out":5425,"duration_ms":46736,"significance":"The work has clear practical significance: it addresses a real bottleneck in scaling verifiable issue-resolving training data by reusing existing Gym environments, and the construction pipeline is transparent and execution-based. The isolated scaling experiments in Figure 2 show consistent monotonic gains from SWE-Mirror-only trajectories under all three training strategies, and the cross-lingual generalization results are interesting. The dataset release at 60k tasks is potentially valuable to the community. The central claim is plausible but not yet fully established because the headline results mix SWE-Mirror trajectories with prior SWE-rebench trajectories and no train/eval repository-overlap analysis is provided.","major_comments":[{"comment":"No train/eval repository-overlap analysis is given. The 40 target Gym repositories are drawn from SWE-Gym, SWE-rebench, and Multi-SWE-RL, while evaluation uses SWE-Bench-Verified and Multi-SWE-Bench-Flash, which share well-known Python repositories with SWE-Gym. Because the final models train on trajectories in those target repositories, the gains in Table 6 and Figure 2 may reflect repo-specific familiarity rather than transferable issue-resolving skill. Please report the exact repository overlap between the 40 target repos and the evaluation benchmarks, and include a held-out-repo condition, such as training only on repositories not appearing in the benchmark, to support the generalization claim.","section":"§2, §3.1, Table 6, Figure 2"},{"comment":"The headline SOTA attribution is ambiguous. The final 12,456-trajectory set consists of 6,431 SWE-Mirror trajectories combined with 6,025 trajectories from prior SWE-rebench experiments, yet the abstract presents the +46.0% improvement as validating SWE-Mirror. This experimental design does not isolate SWE-Mirror's contribution. Please add ablations on SWE-Mirror-only at the same data scale, SWE-rebench-only, and the mixed set, and report the incremental gain attributable to each component.","section":"§3.1, Table 6, Abstract"},{"comment":"The verification protocol checks only test-status transitions, and because fix.patch is defined as the inverse of mirror.patch, the 'effective fix' requirement is satisfied by construction and does not establish that the mirrored bug corresponds to the source issue's core logic. The human audit covers only 184 tasks, with 21 rated inconsistent and 7 unclassifiable; extrapolating to 60,671 tasks leaves an unknown but potentially material share of semantically mismatched tasks. Please provide a larger or stratified audit, or automated semantic checks, and quantify how inconsistent tasks affect post-training performance.","section":"§2.3, §2.4"},{"comment":"The claim of a 'strong scaling law' is not supported by the evidence presented. Each curve in Figure 2 has only four points, with no fitted functional form, no repeated seeds, and no error bars. The monotonic increase is encouraging, but the paper should either fit a scaling form and evaluate its extrapolation or substantially soften the scaling-law language.","section":"§3.3.1, Figure 2"}],"minor_comments":[{"comment":"The section heading 'Conlusion' should be 'Conclusion'.","section":"§5"},{"comment":"The caption reads 'Dataset statics' and should read 'Dataset statistics'.","section":"Table 5"},{"comment":"The row label 'SWE-Mirro-60K' is missing the letter 'r'; the main text also alternates between 'SWE-Mirror-60K' and 'SWE-Mirro-60K'.","section":"Table 1"},{"comment":"Please state how many random seeds were used and consider adding error bars, especially because the differences between methods at the same data scale are small relative to the reported gains.","section":"Figure 2"},{"comment":"There are typos: 'SKIPED' should be 'SKIPPED', 'appliable' should be 'applicable', and 'whithout' should be 'without'.","section":"§2.3"},{"comment":"Reference [26] is cited as 'GPT-4o mini' but the text refers to 'GPT-4o-2024-0513'; please reconcile the model name and version.","section":"References"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nSWE-Mirror is worth a look: it's a genuinely new way to scale issue-resolving training data. Instead of synthesizing bugs inside one repo (SWE-Smith, SWE-Synth) or trying to automate Gym setup (SWE-rebench), it takes real GitHub PRs and re-instantiates them in a different repo that already has an executable Gym. The three-phase pipeline (collect, mirror, verify) produces 60,671 tasks with a test.patch, mirror.patch, and the inverse fix.patch, and the execution-based validation using run/test/fix logs is a sensible filter. The per-language yield numbers are informative, and the small human audit (184 tasks, 88% high/moderate consistency) gives some evidence that mirrored tasks preserve the source issue's semantics.\n\nThe isolated scaling experiments in Figure 2 are the strongest evidence: training only on SWE-Mirror trajectories lifts 7B from 1.0% to 18.2% and 32B from 6.2% to 35.6% on SWE-Bench-Verified, with error masking consistently beating response-only and error pruning. That's a real result.\n\nThe soft spots are real but not fatal. First, the headline numbers in Table 6 mix 6,431 SWE-Mirror trajectories with 6,025 SWE-rebench trajectories. The +46.0% (32B) is a combined-dataset result, not a SWE-Mirror-only result. The paper should be much clearer about that split, especially because the abstract's 'by extending the dataset size' phrasing makes it easy to misread. Second, the training repos likely overlap with the evaluation benchmarks: Gyms are taken from SWE-Gym, SWE-rebench, and Multi-SWE-RL, and SWE-Bench-Verified draws from a small set of popular Python repos. There is no overlap analysis, no held-out-repo evaluation. Without that, part of the gain may be repo familiarity rather than transferable skill. Third, verification checks test-status transitions, not semantic fidelity; the human audit covers only 184 of 60,671 tasks. Fourth, single runs, no error bars. Fifth, the dataset is announced but no URL or code is given.\n\nWho is this for? People building training sets for coding agents. It deserves a serious referee. The method is novel, the empirical direction is promising, and the problems I listed are addressable. A revised version with a clear overlap analysis, a SWE-Mirror-only vs. combined breakdown, and a dataset release would be much more convincing.\n\nMy recommendation: send it to review, and ask the referees to check the train/eval overlap with SWE-Bench and Multi-SWE-Bench carefully.","headline":"Genuinely novel data-generation pipeline with real isolated gains, but the headline SOTA mixes in SWE-rebench trajectories and likely overlaps the eval repos; needs revision, not rejection.","tokens_in":20339,"tokens_out":3466,"would_cite":true,"duration_ms":28123,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mirroring real GitHub bugs into similar repositories produces 60,671 verifiable coding tasks, and post-training on them lifts a 7B model from 1.0% to 22.8% and a 32B model from 6.2% to 52.2% on SWE-Bench-Verified.","keywords":["SWE-Mirror","mirrored issues","issue-resolving datasets","Gym environment reuse","agent post-training","error masking","cross-lingual transfer","SWE-Bench"],"falsifier":"Draw a random sample of several hundred tasks from the released dataset, have annotators compare each mirrored task against its source pull request without knowing the pairing, and measure the inconsistent share; if that share is large, or if a model trained on audited-inconsistent tasks resolves SWE-Bench-Verified as well as one trained on audited-consistent tasks, the claim that the fail-to-pass transition encodes the intended bug is refuted. A cheaper proxy is to take a model's successful fix for a mirrored task and check whether it also resolves the original issue in the source repository.","tokens_in":19141,"feed_emoji":"🐛","tokens_out":11880,"duration_ms":88410,"temperature":0.7,"pith_summary":"SWE-Mirror claims that the scarcest resource in training coding agents -- an executable test harness, or \"Gym,\" that can verify whether a proposed fix works -- does not have to be built anew for every task. The pipeline takes a real bug fix from one GitHub repository, distills the bug's core logic into an abstract description, and re-instantiates it in a different, functionally similar repository that already has a working Gym. Applied to 40 repositories across four languages, this yields 60,671 verifiable tasks built from authentic human-reported issues. The paper further claims that post-training on agent trajectories through these tasks materially improves issue-resolving ability: a 7B model climbs from 1.0% to 22.8% and a 32B model from 6.2% to 52.2% on SWE-Bench-Verified, a human-curated set of 500 real GitHub issues. If the claims hold, the one-task-per-environment bottleneck that has capped the scale of authentic training data is broken.","feed_headline":"Mirroring real bugs into similar repos creates 60,671 training tasks","feed_subtitle":"Post-training lifts a 7B model to 22.8% and a 32B model to 52.2% on SWE-Bench-Verified.","key_machinery":"The load-bearing mechanism is the test.patch / mirror.patch / fix.patch triple, generated and checked by a three-phase pipeline. Here a \"Gym\" is an executable environment that packages a repository snapshot together with test commands and log parsers so that a proposed patch can be automatically verified. Task Collection finds candidate source pull requests using an LLM-driven filter; Task Mirroring runs a Test Agent (Agentless-style localization plus patch generation) to write a regression test that passes on clean code and fails on buggy code, then a Mirror Agent to surgically break that test, so the failing-test transition becomes the formal contract of the task, with the inverse mirror.patch as the ground-truth fix. Task Verification replays the full test suite in three states -- bug injected (run.log), bug plus test added (test.log), and all patches applied (fix.log) -- and accepts a task only if the transitions are unambiguous: the new test must go NONE to FAILED to PASSED, no previously passing test may regress, and flaky or abnormal transitions are discarded. That three-log transition check is what converts an LLM's guess about a bug into a verifiable training signal.","core_discovery":"The paper's central claim is that a real software issue can be transplanted across repository boundaries. A source issue and its merged pull request are first compressed into an abstract description of the bug pattern -- what functionality is involved, what the observed symptom is, and what the root cause is -- stripped of the source repository's specific file paths and variable names. A Test Agent then writes a test.patch for the target repository that passes on clean code and fails once the bug is present; a separate Mirror Agent introduces the bug with a mirror.patch that breaks exactly those tests; and the inverse of that patch becomes the ground-truth fix.patch. A synthesized problem statement completes the task, and only tasks that pass a three-log execution check -- run.log, test.log, fix.log -- with unambiguous fail-to-pass transitions are kept. The released result is SWE-Mirror-60K: 60,671 tasks across 40 repositories, of which an independent human audit rated 156 of the 177 tasks with majority agreement as high or moderate in consistency with the source issue. From these tasks the authors collect 6,431 expert agent trajectories, combine them with 6,025 from an existing pipeline, and post-train Qwen2.5-Coder-Instruct models to reach 22.8% (7B) and 52.2% (32B) on SWE-Bench-Verified and 6.33% and 21.33% on Multi-SWE-Bench-Flash.","pith_inferences":["If mirroring genuinely preserves a bug's logic, the learned fix should transfer back: a model that resolves the mirrored task in the target repository should also make progress on the original issue in the source repository, a round-trip test the paper does not run.","The audit's 21 inconsistent and 7 unclassifiable cases out of 184, extrapolated to 60,671 tasks, imply that thousands of instances may encode a pattern the mirror agent injected rather than the source bug; a statistically powered audit of the released dataset would bound how much of the training signal is genuine.","Mirroring also suggests a curriculum: the same issue transplanted into repositories of increasing dissimilarity would let researchers measure and train how well agents cope with codebase variation, a dimension static benchmarks do not exercise.","Because the mirror agent must break tests the test agent wrote, the pipeline's ceiling is set by LLM test-writing quality; verifying the tests themselves, for example by mutation testing, is the next bottleneck this approach creates."],"forward_implications":["Each reusable Gym can host thousands of distinct authentic tasks, so scaling training data no longer requires roughly one gigabyte of environment storage per task; the 60,671 tasks here fit in about 100 GB.","Training data can be refreshed continuously from GitHub's live stream of merged pull requests, reducing the stagnation and contamination risk of static benchmarks.","For agent post-training, masking the loss on erroneous actions while keeping the full recovery context outperforms both naive fine-tuning and pruning error turns, and the gap widens as data grows.","Training on non-Python languages transfers to Python issue resolution: Rust data gives the largest single boost, so multilingual data is useful even for Python-only benchmarks.","At 52.2% on SWE-Bench-Verified, the 32B model matches systems several times its size under the same agent scaffold, suggesting dataset scale and quality can substitute for raw model size."],"supporting_citations":[{"why":"Supplies the seed Gym environments whose reuse is the entire basis of the dataset, and the baseline the paper extends.","marker":"[29]"},{"why":"Supplies more seed Gym environments and contributes 6,025 of the 12,456 training trajectories.","marker":"[6]"},{"why":"Defines the SWE-Bench task instance format (issue, PR, codebase snapshot, tests) that mirrored instances reproduce.","marker":"[17]"},{"why":"Provides the SWE-Bench-Verified evaluation set and the quality criteria the synthesized problem statements follow.","marker":"[27]"},{"why":"Contributes the validation-transition rules that Task Verification borrows and the multilingual benchmark used for evaluation.","marker":"[40]"},{"why":"The synthetic-task scaling approach the paper contrasts with, and the source of the SWE-agent-LM baseline models.","marker":"[39]"},{"why":"Establishes the cross-project analogous-bug premise that motivates mirroring issues across repositories.","marker":"[11]"},{"why":"The OpenHands agent scaffold used to collect trajectories and to evaluate all models.","marker":"[33]"},{"why":"The Qwen2.5-Coder-Instruct models used as post-training bases and the Qwen3 model used for keyword generation in task collection.","marker":"[37]"},{"why":"Provides the scaling-law framing the paper uses to interpret resolve rate as a function of trajectory count.","marker":"[18]"}],"fun_headline_variants":["Mirroring real bugs across repos creates 60K training tasks","Transplant real issues into new repos to train coding agents","SWE-Mirror: 60,671 tasks from mirrored real-world issues","Reusing GitHub issues to build verifiable coding tasks at scale","Bug-mirroring pipeline lifts SWE-Bench resolve by 46% for 32B"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"That the LLM-written test and the LLM-written bug-injecting patch truly re-create the source issue's logic in the target repository: the three-log verification only checks test-status transitions, not whether the taught bug is the original issue's bug, and the human fidelity audit covers only 184 of the 60,671 tasks.","fun_headline_variants_meta":{"raw":{"variants":["Mirroring real bugs across repos creates 60K training tasks","Transplant real issues into new repos to train coding agents","SWE-Mirror: 60,671 tasks from mirrored real-world issues","Reusing GitHub issues to build verifiable coding tasks at scale","Bug-mirroring pipeline lifts SWE-Bench resolve by 46% for 32B"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000271,"raw_usage":{"total_tokens":1737,"prompt_tokens":1165,"completion_tokens":572,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":781,"completion_tokens_details":{"reasoning_tokens":474}},"tokens_in":781,"tokens_out":572,"duration_ms":4866,"temperature":1.0,"reasoning_tokens":474,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:00:36.204529+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Draw a random sample of several hundred tasks from the released dataset, have annotators compare each mirrored task against its source pull request without knowing the pairing, and measure the inconsistent share; if that share is large, or if a model trained on audited-inconsistent tasks resolves SWE-Bench-Verified as well as one trained on audited-consistent tasks, the claim that the fail-to-pass transition encodes the intended bug is refuted. A cheaper proxy is to take a model's successful fix for a mirrored task and check whether it also resolves the original issue in the source repository.","supporting_citations":[{"cited_title":"Introducing SWE–Bench Verified","cited_arxiv_id":null,"evidence_quote":"Provides the SWE-Bench-Verified evaluation set and the quality criteria the synthesized problem statements follow."}],"review_version":2}