{"id":"186d0789-d605-4b2d-bf17-26883b5bb3ca","arxiv_id":"2508.01550","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":2,"one_line_summary":"RepoForge trains an 8B software engineering agent to 17.4% on SWE-Bench-Verified, a new state-of-the-art for small non-thinking models, using an automated data generation and evaluation pipeline.","lead":"This paper presents RepoForge, an automated pipeline that creates training data, runs evaluations, and trains a small software engineering agent, claiming a new state-of-the-art score for 8-billion-parameter models. A smart generalist should read it because it shows how large language model agents for coding might be trained much more cheaply and quickly.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Auto-generated training data may overlap with SWE-Bench-Verified, potentially inflating the 17.4% SOTA claim; the abstract reports no deduplication.","rationale":"The reader correctly identified the auto-generated environments as a load-bearing assumption, but focused on their correctness and representativeness. My stress-test narrows this to a more specific and, if real, more damaging failure mode: contamination of the training data with the evaluation benchmark. SWE-Bench-Verified is a static set of real GitHub issues; an automated pipeline that scrapes 'real GitHub commits' could easily include the very commits that SWE-Bench-Verified uses as gold patches or base versions. The abstract gives no indication that such overlap was prevented. This is a standard concern in SWE-agent papers, and its absence is conspicuous. The proposed concrete test is straightforward and would settle the issue: compare instance identifiers and patch content. If the test passes, the SOTA claim remains plausible and the paper deserves a full-text review. If it fails, the paper's central contribution collapses. I therefore recommend CONDITIONAL rather than UNCHANGED: the paper should be accepted only after the contamination check is reported and, if needed, the evaluation is repeated on a disjoint subset. My agreement with the reader is partial because they point at environment quality while I point at benchmark leakage; both are serious, but leakage is the more direct threat to the headline result.","tokens_in":820,"tokens_out":1973,"duration_ms":26237,"concrete_test":"Obtain the list of the 7,304 auto-generated instances (repo, issue/PR number, base commit SHA, resolved patch) and the official SWE-Bench-Verified instance list. Compare exact repo+issue identifiers, then compute patch similarity (e.g., git diff overlap or BLEU on code tokens) and base-commit ancestry between the two sets. If any generated instance matches a SWE-Bench-Verified instance, or if a generated patch is nearly identical to a benchmark gold patch, the evaluation is contaminated. Then re-evaluate RepoForge-8B-Agent on the subset of SWE-Bench-Verified instances that have zero overlap with the training data; if the score drops materially, the SOTA claim is unsupported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that RepoForge-8B-Agent achieves 17.4% on SWE-Bench-Verified, a new SOTA for ≤8B non-thinking models. This claim depends on the evaluation being uncontaminated. Contribution (2) states that 7,304 executable environments are auto-generated from real GitHub commits with zero manual intervention. SWE-Bench-Verified is also constructed from real GitHub issues and pull requests. If any of the auto-generated instances are drawn from the same repositories, issues, or base commits as the benchmark instances, training data could leak directly into the evaluation set. A model trained on the exact resolved patches or test outcomes for SWE-Bench-Verified tasks would achieve artificially high scores. The abstract does not mention any filtering, deduplication, or hold-out of SWE-Bench-Verified repositories or issues. Because the entire headline result rests on a fair benchmark comparison, the absence of reported contamination controls is the single most load-bearing concern. This is distinct from the reader's worry about environment correctness: even if every generated environment is perfectly executable and correctly labeled, leakage alone would invalidate the SOTA claim. The concern is concrete and testable: the generated dataset's provenance must be checked against the benchmark's instance list. Without such a check, the 17.4% number cannot be taken at face value.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The abstract describes RepoForge, an autonomous end-to-end pipeline that generates executable training environments from real GitHub commits, labels them automatically via SPICE, and trains an 8B-parameter SWE agent using SFT and RL. The central claim is that the resulting RepoForge-8B-Agent achieves 17.4% on SWE-Bench-Verified, establishing a new state-of-the-art for at most 8B non-thinking LLMs. The abstract also reports several infrastructure metrics: 7,304 auto-generated environments with zero manual intervention, 14× storage reduction per instance, >70% faster evaluation, and 19,000× cheaper labeling. The paper argues that these contributions jointly demonstrate that small models can reach SOTA performance on SWE benchmarks through automated data curation and efficient evaluation.","tokens_in":1096,"tokens_out":2414,"duration_ms":28788,"significance":"If the claims hold, this is a significant result for the SWE agent community: it would show that a fully automated data-generation pipeline can replace expensive manual dataset construction, and that an 8B non-thinking model can outperform prior small-model approaches on SWE-Bench-Verified. The infrastructure metrics (storage, evaluation speed, labeling cost) are also of practical value. However, the abstract alone provides no methodological detail, no error analysis, and no comparison protocol, so the headline SOTA claim cannot be verified from the manuscript as submitted. The paper should be credited for attempting a genuinely end-to-end pipeline and for reporting concrete infrastructure numbers, but the evidentiary bar for the central claim is not met in the abstract.","major_comments":[{"comment":"The 17.4% SWE-Bench-Verified result is vulnerable to training/evaluation contamination because the 7,304 training environments are auto-generated from real GitHub commits, while SWE-Bench-Verified is also constructed from real GitHub issues and pull requests. The abstract reports no deduplication or provenance check against the benchmark's repositories, base commits, or issues. Since the entire SOTA claim rests on a fair evaluation, the authors must provide a contamination analysis (e.g., patch/commit/issue overlap checks) and report how many generated instances were excluded or matched the benchmark. This is a load-bearing point that the abstract does not address.","section":"Abstract, contributions (1) and (2)"},{"comment":"The infrastructure metrics—14× storage reduction, >70% faster evaluation, and 19,000× cheaper labeling—are stated without defining the baselines, measurement protocols, or hardware/software setups. For example, 'labeling' via SPICE is ambiguous: SPICE is a patch-correctness metric, so it is unclear what labels are being produced (difficulty scores? target patches?) and how the 19,000× cost multiplier is computed. These metrics need precise definitions and comparison points before they can be assessed as contributions.","section":"Abstract, contributions (3), (4), and (5)"},{"comment":"The phrase 'zero manual intervention' is an absolute claim that is not defined or supported. Even if environment generation is automated, the scope of 'intervention' is unclear: did the authors manually select repositories, filter broken environments, adjust the RL reward scaffolding, or tune the SPICE threshold? The abstract should state exactly what was automated and what, if any, manual steps remained, otherwise the claim is unfalsifiable from the information given.","section":"Abstract, contribution (2)"}],"minor_comments":[{"comment":"The abstract does not report any training hyperparameters (learning rates, SFT/RL steps, reward weights) or evaluation protocol (number of runs, pass@k, inference budget). Without these, the SOTA claim is not reproducible even with the full text.","section":"Abstract, contribution (1)"},{"comment":"The term 'SPICE-based difficulty assessment' is unclear because SPICE is designed to evaluate patch correctness, not to measure task difficulty. Please clarify how the SPICE score is converted into a difficulty label and which threshold was used.","section":"Abstract, contribution (5)"},{"comment":"'Non-thinking LLMs' is used as a distinction for the SOTA claim but is not defined or cited. A brief definition or reference would help the reader understand the comparison class.","section":"Abstract, contribution (1)"},{"comment":"The phrase 'bubble-free RL scaffold' is not defined in the abstract. Please provide a short explanation or a reference so that the reader understands what problem in the RL pipeline is being addressed.","section":"Abstract, contribution (4)"}],"recommendation":"major_revision","confidential_remarks":"This is an abstract-only review, so the central technical content (environment generation, RL training, benchmark evaluation) could not be inspected. The contamination concern is the most serious issue: if any of the 7,304 auto-generated environments overlap with SWE-Bench-Verified, the headline number would be invalid. The authors should be asked to supply the contamination analysis before the paper can be considered. The infrastructure metrics also need methodological grounding. The paper may fit a systems venue, but the abstract currently reads more like a technical report than a conference submission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a systems engineering advance with a strong but unverified SOTA claim. The stress-test concern about SWE-Bench contamination is the right thing to check first; if the auto-generated repo/commit pool overlaps with the benchmark's instance list, the 17.4% number is void.\n\nThe genuinely useful part is the pipeline itself. They have packed SFT/RL, sandbox dedup and pruning, Ray-based eval, and SPICE labeling into one automated loop, and the efficiency numbers (14x storage reduction, 70% faster eval, 19,000x cheaper labeling) are the sort of operational wins that help the community even if the accuracy claim changes. The integration is real and likely saves compute for groups without massive infrastructure.\n\nThe soft spots are exactly where you'd expect in an abstract-only submission. There's no protocol for how the 7,304 environments were validated, no benchmark-to-training set dedup, and no artifact link. The stress-test note nails the load-bearing risk: SWE-Bench-Verified is built from real GitHub issues, and the training data is auto-generated from real GitHub commits. Without an explicit statement that the generated instances exclude the benchmark's repos/issues/labels, the headline result could be inflated by leakage. This isn't a minor worry—it's the first question any referee should ask. The 'zero manual intervention' claim also needs a precise definition, and the 19,000x labeling cost reduction needs a baseline.\n\nI'm not saying the result is false; I can't know that from an abstract. But the abstract's job is to make the claim believable enough to justify the full check, and it currently doesn't. The authors could settle this with a simple overlap analysis, which suggests it's a fixable review process.\n\nBottom line: this paper deserves peer review, not desk rejection, because the claims are concrete, the engineering is real, and the contamination concern is testable. Send it to a referee who will audit the data provenance and the benchmark overlap. I'd be surprised if the exact 17.4% survives, but the pipeline contribution will stand on its own.","headline":"A concrete but unverified SOTA claim wrapped around a genuinely useful engineering pipeline; the contamination question decides the accuracy number, not the pipeline's value.","tokens_in":1613,"tokens_out":2317,"would_cite":false,"duration_ms":25548,"reading_group":"maybe","serious_thinker":"unclear","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fully automated pipeline trains an 8B-parameter agent to 17.4% on SWE-Bench-Verified, a new best for small non-thinking models.","keywords":["software engineering agents","SWE-Bench-Verified","reinforcement learning","supervised fine-tuning","automated data curation","executable environment generation","SPICE difficulty labeling","8B parameter models"],"falsifier":"Take a random sample of the 7,304 auto-generated environments and replay the gold patches: if a non-trivial fraction fail to show the intended failing-then-passing test behavior, or if the SPICE difficulty labels do not match human judgment or downstream performance ordering, the automated data-quality premise is broken. A simpler check: ablate the SPICE labeling and the bubble-free RL scaffold separately; if removing either leaves the 17.4% score essentially unchanged, the claimed mechanism is not what carries the result.","tokens_in":1309,"feed_emoji":"🤖","tokens_out":1380,"duration_ms":49539,"temperature":0.7,"pith_summary":"This paper claims that an 8-billion-parameter software-engineering agent, RepoForge-8B-Agent, reaches 17.4% on SWE-Bench-Verified, the best published score among models with 8B parameters or fewer that do not use extended thinking. The route there is an end-to-end pipeline that automatically generates executable task environments from real GitHub commits, labels their difficulty without human annotators, and trains the agent with supervised fine-tuning followed by reinforcement learning. The authors care because this would show that the expensive, manually curated data and large reasoning models previously thought necessary for strong SWE-bench performance are not required. They also report large infrastructure savings: 14x smaller stored environments, more than 70% faster evaluation, and a 19,000x cheaper labeling process.","feed_headline":"8B model sets small-model SWE record at 17.4%","feed_subtitle":"A fully automatic pipeline generates 7,300+ coding tasks from real commits and trains the agent with no manual labels.","key_machinery":"The load-bearing object is the RepoForge pipeline itself, an autonomous end-to-end system that turns real GitHub commits into training data and trained agents. Its components are: (1) executable environment generation that compresses each instance from 1.4GB to 102MB through dependency management and image pruning; (2) a Ray-powered distributed evaluation harness that cuts evaluation time by more than 70%; (3) SPICE, an automated difficulty-labeling technique that measures task difficulty without human annotators; and (4) a bubble-free RL scaffold for multi-turn training. The pipeline's role is to produce, label, and evaluate training tasks at scale, which is what lets an 8B model reach 17.4% on SWE-Bench-Verified.","core_discovery":"The central discovery is that a small, fast-thinking model can be trained to a new state of the art on SWE-Bench-Verified using a fully automated data pipeline. RepoForge-8B-Agent achieves 17.4% on SWE-Bench-Verified, surpassing prior ≤8B non-thinking LLMs, and does so with 7,304 executable environments auto-generated from real GitHub commits with zero manual intervention. The paper attributes this to unifying storage-efficient sandboxing, a Ray-powered distributed evaluation harness, automated data generation, SPICE-based difficulty labeling, and a bubble-free RL scaffold. The result is presented as evidence that the bottlenecks of SWE agent training—storage cost, slow evaluation, scarce data, and manual labeling—can be removed at scale without sacrificing benchmark performance.","pith_inferences":["If the auto-generated environments are genuinely representative, the same pipeline should transfer to other SWE benchmarks, such as SWE-Bench-Full or live repositories, and one test is whether the 17.4% score generalizes rather than overfits to SWE-Bench-Verified's distribution.","The 19,000x labeling-cost figure implies that difficulty-aware data selection can be applied to millions of commits, which would allow dataset-scale ablations of 'train on hard tasks only' strategies—something the paper does not directly test.","A natural extension is to use the pipeline to train agents that do use extended thinking; if the data quality is the real driver, a thinking version of the same 8B model should exceed the non-thinking result by a larger margin than in prior comparisons.","The 'bubble-free RL scaffold' is the least-specified component in the abstract; an ablation or a clear description of what bubble it removes would let others judge whether the RL stage or the SFT data does the heavy lifting."],"forward_implications":["≤8B non-thinking models can now claim the top score on SWE-Bench-Verified, so compute-limited teams can aim at state-of-the-art without large reasoning models.","The pipeline's 14x storage reduction and more than 70% faster evaluation make large-scale SWE-agent training feasible on modest infrastructure.","Automated SPICE labeling at 19,000x lower cost removes the manual quality-control bottleneck that previously capped dataset size.","Because the data comes from real GitHub commits, the same pipeline can generate fresh tasks continuously as new commits land, keeping training data current.","Supervised fine-tuning on auto-generated data followed by multi-turn RL is a scalable recipe for training capable software-engineering agents."],"supporting_citations":[],"fun_headline_variants":["8B model tops SWE-Bench at 17.4% record","Auto pipeline trains 8B agent to 17.4% SWE-Bench","RepoForge: zero-label pipeline yields 8B SWE record","17.4% SWE-Bench: 8B non-thinking LLM sets record"],"cache_read_input_tokens":3840,"weakest_assumption_plain":"The whole result rests on the assumption that the 7,304 auto-generated commit-derived environments are correct and representative enough to train on, even though no human verified them; any systematic error in environment construction or in the SPICE difficulty labels would propagate into the trained model and could invalidate the state-of-the-art claim.","fun_headline_variants_meta":{"raw":{"variants":["8B model tops SWE-Bench at 17.4% record","Auto pipeline trains 8B agent to 17.4% SWE-Bench","RepoForge: zero-label pipeline yields 8B SWE record","17.4% SWE-Bench: 8B non-thinking LLM sets record"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000272,"raw_usage":{"total_tokens":1671,"prompt_tokens":1021,"completion_tokens":650,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":637,"completion_tokens_details":{"reasoning_tokens":563}},"tokens_in":637,"tokens_out":650,"duration_ms":6926,"temperature":1.0,"reasoning_tokens":563,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T05:31:26.904555+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the 7,304 auto-generated environments and replay the gold patches: if a non-trivial fraction fail to show the intended failing-then-passing test behavior, or if the SPICE difficulty labels do not match human judgment or downstream performance ordering, the automated data-quality premise is broken. A simpler check: ablate the SPICE labeling and the bubble-free RL scaffold separately; if removing either leaves the 17.4% score essentially unchanged, the claimed mechanism is not what carries the result.","supporting_citations":[],"review_version":1}