{"id":"bcb7a5e6-8138-4340-b70f-17bd49c8f522","arxiv_id":"2608.08570","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"FailForge converts failed SWE-bench trajectories into training signal via distilled procedural skills, lifting Qwen3.5-4B pass@1 by 6.6 points over RFT.","lead":"FailForge turns failed code-agent attempts into new training data by having an agent write a reusable strategy, using that strategy to guide a second try, then removing the strategy before training the model. A reader might care because it offers a way to extract useful signal from the hardest and most expensive failures in software engineering agent training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Residual leakage through the GPT-5.4 filter could inflate the +6.6 gain; auditing the 218 retained skills would settle it.","rationale":"The reader's weakest assumption and the most load-bearing concern are the same: the leakage filter must actually prevent solution-specific information from entering the induced skills. The filter's reported recall of 0.778 means a substantial fraction of truly leaky skills can survive, and the paper provides no estimate of how many survived among the skills that actually generated training trajectories. The no-LF ablation (63.6) nearly matches the hint baseline (63.8), so leaky skills are empirically equivalent to instance-level hints, which directly undermines the claim of procedural transfer if present in the recovered corpus. The cross-instance transfer experiment is a useful partial control, but same-repository sibling skills do not eliminate the possibility that remaining leakage is doing the work. My recommendation is to keep the CONDITIONAL verdict: the concern is real but testable, and the paper has independent support in the fixed-budget data-quality swap, cross-harness transfer, and component ablations. A focused audit of the retained skills would settle whether the headline gain is due to distilled procedure or residual answer guidance.","tokens_in":12733,"tokens_out":7461,"duration_ms":87727,"concrete_test":"Run a reconstruction audit on the 218 retained skills: for each (s_i, p_i), have an independent model (not GPT-5.4) attempt to reconstruct p_i from s_i plus the issue text, scoring exact and structural match, and also have two human annotators blind to p_i label each skill as leaky or clean. Remove all skills flagged by either audit, retrain Qwen3.5-4B on the reduced recovered corpus, and re-evaluate on SWE-bench Verified. If pass@1 drops by more than 1.5 points relative to 66.2, residual leakage is load-bearing; if it stays within noise, the leakage filter is adequate.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that FailForge distills transferable procedural competence, not instance-specific answers. The only mechanism separating these is the leakage filter J in the Leakage Filtering section. J is a GPT-5.4 judge calibrated on a sample, with recall 0.778 for leakage positives and precision 0.875. This means roughly 22% of truly leaky skills pass the filter. Because the diagnosing agent is given the gold patch, the base leak rate among generated skills is plausibly non-trivial, yet the paper does not report the pre-filter leak rate, the number of regenerations, or the final leak rate among the 832 induced skills, and crucially among the 218 skills whose recovered trajectories entered training. The no-leakage-filter ablation drops pass@1 to 63.6, close to the hint baseline of 63.8, showing that leaky skills behave much like instance-level hints. If residual leaky skills are overrepresented among the recovered corpus—plausible because leaky guidance makes guided re-rollout easier—the +6.6 gain could reflect training on near-answer guidance rather than distilled procedure. The cross-instance transfer result (Fig. 3) is partially reassuring, but sibling skills still come from the same repository and can share code-level cues, so it does not rule out leakage-driven recovery. This is load-bearing because it is the only thing distinguishing FailForge's supervision from RFT + Hint, which trails by 2.4 points.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FailForge, a pipeline that converts persistently failed rejection-sampling fine-tuning (RFT) rollouts on SWE tasks into additional training signal. For each instance on which all K teacher rollouts fail, a diagnosing agent uses the failed trajectories, test outputs, and the gold patch to induce a structured procedural skill; a GPT-5.4 judge filters skills that leak implementation details; the teacher then re-rolls out the instance with the skill injected into context; trajectories that pass the hidden tests are retained, the skill is stripped, and the student is fine-tuned on the union of the original and recovered corpora. On SWE-bench Verified, training Qwen3.5-4B with FailForge improves pass@1 from 59.6 (RFT 5x) to 66.2, with ablations, cross-harness transfer, and a skill taxonomy. The central claim is that recovered supervision encodes transferable procedural competence rather than instance-specific answers, enabled by the leakage filter and by removing the skill at training time.","tokens_in":12993,"tokens_out":4579,"duration_ms":51263,"significance":"If the claims hold, FailForge addresses a real limitation of RFT: persistent failures are precisely the instances that frontier models cannot solve, and the paper shows a nontrivial pass@1 gain over a strong RFT baseline with a modest recovery budget. The paper includes useful ablations, a cost analysis, a behavioral analysis, and a cross-instance transfer experiment with matched and shuffled guidance. The method is clearly described and the comparisons are internally consistent. However, the significance is contingent on two unresolved points: the leakage filter has imperfect recall and its residual leakage is not audited, and all results are single-run with no error bars or released code. Because the paper's own ablation shows that removing the leakage filter reduces the gain to near the hint baseline, residual leakage is a load-bearing concern for the interpretation of the 6.6-point improvement as evidence of distilled procedural competence.","major_comments":[{"comment":"The leakage filter is the only mechanism separating FailForge from RFT + Hint, yet its imperfection is not propagated into the headline results. The paper reports judge recall of 0.778 and precision of 0.875, meaning roughly 22% of leaky skills are accepted, but it does not report the pre-filter leak rate, the number of regenerations, the retry budget, or the post-filter leak rate among the 832 induced skills and, more importantly, among the 218 skills whose trajectories entered training. This matters because Table 5 shows that removing the leakage filter drops pass@1 from 66.2 to 63.6, essentially the hint baseline of 63.8, so residual leakage could plausibly account for much of the claimed advantage over hints. I ask for a leakage audit on the retained skills and recovered trajectories: for example, re-judge a sample of accepted skills against the gold patch with human annotators, and measure how often identifiers or control-flow structure from the gold patch appear in the recovered trajectories.","section":"Methodology, Leakage Filtering; Table 5"},{"comment":"All reported numbers are single-run and no error bars or significance tests are provided. Several key comparisons are small absolute differences, such as the 2.4-point gap between FailForge (66.2) and RFT + Hint (63.8), the 2.6-point effect of leakage filtering in Table 5, and the 2.4-point gap in Table 3's fixed-budget swap. Without multiple seeds, bootstrap confidence intervals, or a significance test, it is not possible to determine whether these differences exceed run-to-run variance. I also note that no code or data is released, which prevents independent verification of the trajectory recovery and filtering steps. Please report variance estimates and release the code and filtered skill/trajectory data.","section":"Experiments, Experimental Setup and Main Results"},{"comment":"The abstract and conclusion claim that gains are 'concentrated on the hardest problems', but the paper does not directly stratify SWE-bench Verified instances by difficulty. The experiments show that the added training data come from the persistently failed set F in the training corpus, which is not the same as showing that the student's improved evaluation performance is concentrated on difficult SWE-bench instances. Please add a difficulty stratification on the evaluation set, for instance by grouping instances by the RFT baseline's success/failure or by an external difficulty proxy, and report per-group pass@1 differences.","section":"Abstract and Main Results"}],"minor_comments":[{"comment":"The checkmark notation in the ablation table is not self-explanatory; adding explicit row labels such as 'no LF, no RR, no SR' would improve readability.","section":"Table 5"},{"comment":"The retry budget for leakage-filter regeneration is mentioned but never specified; please state the exact budget or how it was chosen.","section":"Methodology, Agentic Skill Induction"},{"comment":"The text refers to 'Kimi-K2.6' but the cited reference appears to be 'KimiK2.5'; please resolve this version discrepancy.","section":"References"},{"comment":"The 'tokens per point of pass@1 improvement' metric would benefit from explicitly stating the total token budgets per method, since Tok/rec alone does not capture the induction and filtering overhead.","section":"Experiments, Cost Efficiency"},{"comment":"The figure caption reports both absolute counts and percentages, but the percentages are not obviously derived from the counts; please clarify the reconciliation (e.g., 83/140 vs 59.3%).","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The core idea is timely and the experimental design is thoughtful, but the residual-leakage concern is serious enough that I would not accept the paper without the requested audit and variance reporting. The lack of code release compounds the issue. I would also flag that the 'hardest problems' claim needs direct evidence, not just the construction of the training set."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: FailForge digs signal out of the uniformly failed set via high-level skills, filters leakage explicitly, strips the skill before training, and checks whether the recovered trajectories transfer across instances. The internal controls are mostly right. Every method starts from the same RFT corpus C, spends exactly five additional teacher rollouts per failed instance, and keeps only the first success. On that setup the +6.6 pass@1 on SWE-bench Verified (59.6 to 66.2) and the ablation pattern tell a coherent story: no leakage filter gives 63.6, close to the hint baseline of 63.8, while the full pipeline reaches 66.2. The fixed-design data swap is a good addition. The cross-instance transfer result is partial, but it goes beyond what most papers in this area show. No circularity: eval repos are disjoint from training, and the gold patch is not in the student's interface. So the method deserves a serious referee.\n\nThe soft spots are real but not fatal. One: the leakage filter is load-bearing. The judge J has 0.778 recall on leakage, and the paper never reports the pre-filter leak rate, the retry count, or the final leak rate among the 218 recovered trajectories that actually entered training. Since leaky skills may recover trajectories more easily, residual leakage could be overrepresented in the recovered corpus, and then part of the +6.6 is training on near-answer hints rather than distilled procedure. The ablation to 63.6 shows the filter is doing real work, but its imperfections leave the residual contamination unknown. Auditing the retained skills is the one experiment that would settle it.\n\nTwo: everything is single-run, no error bars, no code or data. SWE-bench Verified has 500 instances, so the 6.6 point gap is about 33 instances; variance across seeds and config details could move that noticeably. The 'gains concentrated on hardest problems' claim in the abstract is not directly stratified, though the localization numbers point the right way. The gold patch at synthesis time is an acknowledged practical assumption; fine for offline training, not for agents that must bootstrap without historical fixes. These are caveats, not contradictions.\n\nWho this is for: people training open-weight code agents with RFT and people working on failure reuse. It should be reviewed, with a request for code/data and for the leakage audit. I would cite it; I would also want to see the retained-skill audit before betting on the interpretation.","headline":"FailForge is a well-controlled, useful method for turning uniformly failed SWE rollouts into training signal; the headline gain is credible, but the leakage filter is load-bearing and needs an audit before the 'procedural competence' interpretation is fully trusted.","tokens_in":13570,"tokens_out":2451,"would_cite":true,"duration_ms":28101,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FailForge claims that persistent failures in rejection sampling fine-tuning can be converted into transferable procedural training signal, improving a code agent's resolve rate by 6.6 points pass@1 over a strong baseline.","keywords":["rejection sampling fine-tuning","code agents","software engineering","procedural skills","failure recovery","skill distillation","execution-based verification","agent training"],"falsifier":"Take a sample of skills the filter judged clean, give them to annotators who have never seen the issue, and ask them to reconstruct the patch; if they can recover the gold patch's key edits at above-chance rates, leakage is real and the transferability claim weakens. A complementary check: train the student using skills harvested from failures in unrelated repositories; if the 6.6-point pass@1 gain collapses toward the hint baseline, the skills were carrying repository-specific solution information rather than transferable procedure.","tokens_in":12523,"feed_emoji":"🔧","tokens_out":7470,"duration_ms":74721,"temperature":0.7,"pith_summary":"Standard rejection sampling fine-tuning (RFT) keeps only trajectories that pass hidden tests and throws away tasks where every sampled rollout fails. FailForge argues these discarded failures are the most valuable learning signal, and claims they can be converted into supervision: an agent reads the failed traces, test errors, and the reference patch, writes a concise repository-agnostic procedural skill, and uses that skill to guide a second rollout. Trajectories that then pass are added to the training corpus, but the skill itself is stripped out before training, forcing the student to internalize the procedure. The paper reports that this recovery lifts a 4-billion-parameter student's solve rate on a standard human-validated software-engineering benchmark by 6.6 points over a strong RFT baseline, with all gains coming from instances that standard RFT discards.","feed_headline":"Failed runs become training data, lifting code agents 6.6 points","feed_subtitle":"A skill distilled from failed rollouts then deleted lets a small model solve the hardest issues without hints.","key_machinery":"The central mechanism is a synthesized procedural skill, a structured four-part artifact consisting of an applicability condition, key principles, common pitfalls, and a recommended approach, phrased at the level of methodology rather than any specific bug or repository. A diagnosing agent produces the skill by reasoning over all failed trajectories, test outputs, reasoning traces, and the gold patch; a leakage judge filters out skills that could let a reader reconstruct the patch; and a removal operator strips the skill from the training target after guiding the successful re-rollout. This turns skill-conditioned success into skill-free student behavior, so the model must encode the procedure in its weights rather than copy a visible hint.","core_discovery":"The paper's central discovery is that persistent failures in RFT are not mostly sampling noise or missing instance-specific facts; they reflect absent transferable procedural competence, namely how to localize a fault across a repository, how to read a test harness, and how to propagate a change. Supplying that procedure as a skill during rollout recovers over 26% of persistently failed instances, nearly twice the recovery of doubling the sampling budget, and the recovered trajectories, trained without the skill, improve pass@1 by 6.6 points over RFT on the human-validated benchmark. Instance-level hints recover more trajectories but train a weaker student, so recovery count alone is not a measure of supervision quality: procedural skills yield higher per-sample training value and transfer beyond the source instance.","pith_inferences":["Beyond the paper: the same diagnose-distill-reroll-remove loop should apply to any verifiable long-horizon domain where failures reflect missing methodology, such as data-science workflows or formal proof search; a direct test is running FailForge on a non-code verifiable corpus and checking the recovered-trajectory gain.","Beyond the paper: the leakage filter's reconstruction test is a proxy, and a stronger guarantee could be obtained by measuring whether skills improve solve rates on held-out issues from entirely different repositories, which the shuffled-sibling experiment only partially approximates.","Beyond the paper: the result that rewriting traces to hide skill references hurts suggests that preserving the investigation trajectory's natural reasoning matters more than surface coherence; ablating the degree of trace rewording could locate an optimum.","Beyond the paper: if the skills are truly transferable, models trained with FailForge should show better performance on previously unseen benchmark splits with fewer attempts, and that is testable without retraining by evaluating the trained checkpoints on a newly curated issue set."],"forward_implications":["A standard RFT pipeline can reuse its own failures: at the same teacher rollout budget, skill-guided recovery saves nearly twice as many failed instances as doubling unguided sampling.","Recovered trajectories are more valuable per sample: replacing equal-sized portions of the original corpus with recovered trajectories raises pass@1 from 59.6 to 63.8 at fixed corpus size.","The recovered behavior transfers beyond the source instance: shuffled skills from sibling instances retain 59.3% of matched recovery, while shuffled hints retain only 20.8%.","The method improves both a 4B and a 9B student, transfers to a multilingual benchmark, and survives a change of agent harness, so the effect is not tied to one model scale, language, or interface.","Removing the skill before training is load-bearing: keeping it in the training target drops pass@1 from 66.2 to 64.6, so the gain depends on distilling the procedure into weights."],"supporting_citations":[{"why":"Introduces the STaR self-bootstrapping recipe that defines the rejection sampling baseline FailForge extends by recovering failed tasks.","marker":"Zelikman et al. 2022"},{"why":"Provides the human-validated benchmark used for evaluation; the central 6.6-point gain is measured on its Verified subset.","marker":"Jimenez et al. 2024"},{"why":"Supplies the training task pool with executable environments and hidden test suites used to build the RFT corpus and the persistently failed set.","marker":"Pan et al. 2024"},{"why":"Provides the agent harness used to collect teacher rollouts and guided re-rollouts, so all compared methods share the same interface.","marker":"Wang et al. 2025"},{"why":"Defines the instance-level hint distillation baseline that FailForge exceeds, isolating procedural skills from answer-bound hints.","marker":"Wang et al. 2026"},{"why":"Supplies the multilingual benchmark used to test whether recovered supervision transfers out of the training distribution.","marker":"Yang et al. 2026"},{"why":"Supports the premise that the hardest persistently failed instances are the most valuable learning signal.","marker":"Bengio et al. 2009"}],"fun_headline_variants":["Persistent failures become procedural skills for code agents","Recover 26% failed runs, gain 6.6 points on SWE-bench","Failed rollouts teach code agents: no hints at inference","Distilling failure into skill: 6.6 point boost on hard tasks","Turning failure into training: code agent gains 6.6 points"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the leakage filter reliably removes solution-specific details from induced skills, so the guided second attempt teaches a transferable investigation procedure rather than a veiled version of the answer.","fun_headline_variants_meta":{"raw":{"variants":["Persistent failures become procedural skills for code agents","Recover 26% failed runs, gain 6.6 points on SWE-bench","Failed rollouts teach code agents: no hints at inference","Distilling failure into skill: 6.6 point boost on hard tasks","Turning failure into training: code agent gains 6.6 points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1445,"prompt_tokens":956,"completion_tokens":489,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":396}},"tokens_in":572,"tokens_out":489,"duration_ms":5491,"temperature":1.0,"reasoning_tokens":396,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:30:23.685550+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of skills the filter judged clean, give them to annotators who have never seen the issue, and ask them to reconstruct the patch; if they can recover the gold patch's key edits at above-chance rates, leakage is real and the transferability claim weakens. A complementary check: train the student using skills harvested from failures in unrelated repositories; if the 6.6-point pass@1 gain collapses toward the hint baseline, the skills were carrying repository-specific solution information rather than transferable procedure.","supporting_citations":[],"review_version":1}