REVIEW 3 major objections 5 minor 16 references
FailForge: Distilling Procedural Competence from Persistent Failures into Code Agents
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read FailForge claims that persistent failures in rejection sampling fine-tuning can be converted into transferable procedural training signal, improving a code agent's resolve rate by 6.6 points pass@1 over a strong baseline.
desk verdict FailForge is a well-controlled, useful method for turning uniformly failed SWE rollouts into training signal; the headline gain is credible, but the leakage filter is load-bearing and needs an audit before the 'procedural competence' interpretation is fully trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a synthesized procedural skill, a structured four-part artifact consisting of an applicability condition, key principles, common pitfalls, and a recommended approach, phrased at the level of methodology rather than any specific bug or repository. A diagnosing agent produces the skill by reasoning over all failed trajectories, test outputs, reasoning traces, and the gold patch; a leakage judge filters out skills that could let a reader reconstruct the patch; and a removal operator strips the skill from the training target after guiding the successful re-rollout. This turns skill-conditioned success into skill-free student behavior, so the model must encode the procedure in its weights rather than copy a visible hint.
What would settle it
Take a sample of skills the filter judged clean, give them to annotators who have never seen the issue, and ask them to reconstruct the patch; if they can recover the gold patch's key edits at above-chance rates, leakage is real and the transferability claim weakens. A complementary check: train the student using skills harvested from failures in unrelated repositories; if the 6.6-point pass@1 gain collapses toward the hint baseline, the skills were carrying repository-specific solution information rather than transferable procedure.
Extended reading notes
Core claim
The paper's central discovery is that persistent failures in RFT are not mostly sampling noise or missing instance-specific facts; they reflect absent transferable procedural competence, namely how to localize a fault across a repository, how to read a test harness, and how to propagate a change. Supplying that procedure as a skill during rollout recovers over 26% of persistently failed instances, nearly twice the recovery of doubling the sampling budget, and the recovered trajectories, trained without the skill, improve pass@1 by 6.6 points over RFT on the human-validated benchmark. Instance-level hints recover more trajectories but train a weaker student, so recovery count alone is not a measure of supervision quality: procedural skills yield higher per-sample training value and transfer beyond the source instance.
Load-bearing premise
The load-bearing premise is that the leakage filter reliably removes solution-specific details from induced skills, so the guided second attempt teaches a transferable investigation procedure rather than a veiled version of the answer.
Editorial extensions
If this is right
- A standard RFT pipeline can reuse its own failures: at the same teacher rollout budget, skill-guided recovery saves nearly twice as many failed instances as doubling unguided sampling.
- Recovered trajectories are more valuable per sample: replacing equal-sized portions of the original corpus with recovered trajectories raises pass@1 from 59.6 to 63.8 at fixed corpus size.
- The recovered behavior transfers beyond the source instance: shuffled skills from sibling instances retain 59.3% of matched recovery, while shuffled hints retain only 20.8%.
- The method improves both a 4B and a 9B student, transfers to a multilingual benchmark, and survives a change of agent harness, so the effect is not tied to one model scale, language, or interface.
- Removing the skill before training is load-bearing: keeping it in the training target drops pass@1 from 66.2 to 64.6, so the gain depends on distilling the procedure into weights.
Reading between the lines
- Beyond the paper: the same diagnose-distill-reroll-remove loop should apply to any verifiable long-horizon domain where failures reflect missing methodology, such as data-science workflows or formal proof search; a direct test is running FailForge on a non-code verifiable corpus and checking the recovered-trajectory gain.
- Beyond the paper: the leakage filter's reconstruction test is a proxy, and a stronger guarantee could be obtained by measuring whether skills improve solve rates on held-out issues from entirely different repositories, which the shuffled-sibling experiment only partially approximates.
- Beyond the paper: the result that rewriting traces to hide skill references hurts suggests that preserving the investigation trajectory's natural reasoning matters more than surface coherence; ablating the degree of trace rewording could locate an optimum.
- Beyond the paper: if the skills are truly transferable, models trained with FailForge should show better performance on previously unseen benchmark splits with fewer attempts, and that is testable without retraining by evaluating the trained checkpoints on a newly curated issue set.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes FailForge, a pipeline that converts persistently failed rejection-sampling fine-tuning (RFT) rollouts on SWE tasks into additional training signal. For each instance on which all K teacher rollouts fail, a diagnosing agent uses the failed trajectories, test outputs, and the gold patch to induce a structured procedural skill; a GPT-5.4 judge filters skills that leak implementation details; the teacher then re-rolls out the instance with the skill injected into context; trajectories that pass the hidden tests are retained, the skill is stripped, and the student is fine-tuned on the union of the original and recovered corpora. On SWE-bench Verified, training Qwen3.5-4B with FailForge improves pass@1 from 59.6 (RFT 5x) to 66.2, with ablations, cross-harness transfer, and a skill taxonomy. The central claim is that recovered supervision encodes transferable procedural competence rather than instance-specific answers, enabled by the leakage filter and by removing the skill at training time.
Significance. If the claims hold, FailForge addresses a real limitation of RFT: persistent failures are precisely the instances that frontier models cannot solve, and the paper shows a nontrivial pass@1 gain over a strong RFT baseline with a modest recovery budget. The paper includes useful ablations, a cost analysis, a behavioral analysis, and a cross-instance transfer experiment with matched and shuffled guidance. The method is clearly described and the comparisons are internally consistent. However, the significance is contingent on two unresolved points: the leakage filter has imperfect recall and its residual leakage is not audited, and all results are single-run with no error bars or released code. Because the paper's own ablation shows that removing the leakage filter reduces the gain to near the hint baseline, residual leakage is a load-bearing concern for the interpretation of the 6.6-point improvement as evidence of distilled procedural competence.
major comments (3)
- [Methodology, Leakage Filtering; Table 5] The leakage filter is the only mechanism separating FailForge from RFT + Hint, yet its imperfection is not propagated into the headline results. The paper reports judge recall of 0.778 and precision of 0.875, meaning roughly 22% of leaky skills are accepted, but it does not report the pre-filter leak rate, the number of regenerations, the retry budget, or the post-filter leak rate among the 832 induced skills and, more importantly, among the 218 skills whose trajectories entered training. This matters because Table 5 shows that removing the leakage filter drops pass@1 from 66.2 to 63.6, essentially the hint baseline of 63.8, so residual leakage could plausibly account for much of the claimed advantage over hints. I ask for a leakage audit on the retained skills and recovered trajectories: for example, re-judge a sample of accepted skills against the gold patch with human annotators, and measure how often identifiers or control-flow structure from the gold patch appear in the recovered trajectories.
- [Experiments, Experimental Setup and Main Results] All reported numbers are single-run and no error bars or significance tests are provided. Several key comparisons are small absolute differences, such as the 2.4-point gap between FailForge (66.2) and RFT + Hint (63.8), the 2.6-point effect of leakage filtering in Table 5, and the 2.4-point gap in Table 3's fixed-budget swap. Without multiple seeds, bootstrap confidence intervals, or a significance test, it is not possible to determine whether these differences exceed run-to-run variance. I also note that no code or data is released, which prevents independent verification of the trajectory recovery and filtering steps. Please report variance estimates and release the code and filtered skill/trajectory data.
- [Abstract and Main Results] The abstract and conclusion claim that gains are 'concentrated on the hardest problems', but the paper does not directly stratify SWE-bench Verified instances by difficulty. The experiments show that the added training data come from the persistently failed set F in the training corpus, which is not the same as showing that the student's improved evaluation performance is concentrated on difficult SWE-bench instances. Please add a difficulty stratification on the evaluation set, for instance by grouping instances by the RFT baseline's success/failure or by an external difficulty proxy, and report per-group pass@1 differences.
minor comments (5)
- [Table 5] The checkmark notation in the ablation table is not self-explanatory; adding explicit row labels such as 'no LF, no RR, no SR' would improve readability.
- [Methodology, Agentic Skill Induction] The retry budget for leakage-filter regeneration is mentioned but never specified; please state the exact budget or how it was chosen.
- [References] The text refers to 'Kimi-K2.6' but the cited reference appears to be 'KimiK2.5'; please resolve this version discrepancy.
- [Experiments, Cost Efficiency] The 'tokens per point of pass@1 improvement' metric would benefit from explicitly stating the total token budgets per method, since Tok/rec alone does not capture the induction and filtering overhead.
- [Figure 3] The figure caption reports both absolute counts and percentages, but the percentages are not obviously derived from the counts; please clarify the reconciliation (e.g., 83/140 vs 59.3%).
Circularity Check
No significant circularity: FailForge evaluates on disjoint external benchmarks, and the residual-leakage concern is a data-purity risk, not a derivation-level reduction.
full rationale
The claimed derivation chain is self-contained rather than circular. Standard RFT produces the successful corpus C and the persistently failed set F by sampling K=5 teacher rollouts per task and verifying each against hidden tests; all training data, including the recovered corpus, comes from these rollouts, and the only fitted object in the pipeline is the leakage judge J, which is calibrated on human labels but never used to fit the reported pass@1 numbers. Evaluation is on SWE-bench Verified and SWE-bench MultiLingual, with repositories explicitly disjoint from the SWE-Gym training repositories, so the reported +6.6 gain is not defined in terms of the training inputs or the judge's outputs. The paper itself flags the residual-leakage risk in the Leakage Filtering section: 'It may nevertheless introduce solution specific information into the induced skill...', and the no-leakage-filter ablation shows that unfiltered skills behave like hints; that is a data-purity and interpretation risk, not a constructional circularity, because no equation in the paper reduces a predicted quantity to a fitted parameter or to a self-citation. No self-definitional step, fitted-input-called-prediction step, load-bearing self-citation, author-imported uniqueness theorem, ansatz smuggled in via citation, or renaming of a known result was found.
Assumptions & free parameters
free parameters (3)
- Initial and additional rollout count K =
5
- Leakage filter retry budget
- Claude Code trajectory mixture size =
100
assumptions (4)
- domain assumption Training and evaluation repository sets are disjoint.
- domain assumption Hidden test suites provide correct binary verification of task success.
- domain assumption A diagnosing agent can extract a concise, non-task-specific procedural skill from failed trajectories, test outputs, and the gold patch.
- domain assumption The leakage judge's calibration (96.7% agreement, Cohen's kappa 0.805) transfers to the full skill distribution.
Cite this review
Pith. "Pith review of FailForge: Distilling Procedural Competence from Persistent Failures into Code Agents." pith.science (2026). https://pith.science/paper/EPU4QWTS
@misc{pith2026260808570,
author = {Pith},
title = {Pith review of: FailForge: Distilling Procedural Competence from Persistent Failures into Code Agents},
year = {2026},
howpublished = {\url{https://pith.science/paper/EPU4QWTS}},
note = {Machine review of arXiv:2608.08570}
}
read the original abstract
Rejection sampling fine-tuning (RFT) is widely used to train code agents by generating trajectories on verifiable software engineering tasks, retaining those that pass the tests, and fine-tuning on the successful rollouts. However, even strong code agents repeatedly fail on a substantial fraction of such tasks, and standard RFT simply discards these failures. The discarded samples are precisely the hardest and most informative ones, drawn from verifiable instances that are costly to curate. Stronger base models may reduce the number of failures, but the remaining hard cases still define the frontier for further improvement. We propose FailForge, an agentic framework that converts failed rollouts into training signal. For each failed instance, an agent diagnoses the failure from error feedback and execution traces, distills the diagnosis into a concise and actionable skill, and injects the skill into the agent context for a guided second attempt. Trajectories that succeed under skill guidance are folded back into the RFT corpus. Crucially, the skill is removed at training time, so the model internalizes the recovered behavior rather than relying on external hints at inference. FailForge recovers over 26% of previously failed instances at marginal additional cost, and training Qwen3.5-4B on the augmented corpus improves the SWE-bench Verified resolve rate by 6.6 points over a strong RFT baseline, with gains concentrated on the hardest problems.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Bengio,Y.;Louradour,J.;Collobert,R.;andWeston,J.2009. Curriculum learning. InProceedings of the 26th annual international conference on machine learning, 41–48. Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; Pinto, H. P. D. O.; Kaplan, J.; Edwards, H.; Burda, Y.; Joseph, N.; Brockman, G.;etal.2021. Evaluatinglargelanguagemodelstrainedon code.arXiv preprint arXi...
arXiv 2009
-
[4]
Jain,N.;Singh,J.;Shetty,M.;Zhang,T.;Zheng,L.;Sen,K.; and Stoica, I
Agentcoder: Multi-agent-based code genera- tion with iterative testing and optimisation.arXiv preprint arXiv:2312.13010. Jain,N.;Singh,J.;Shetty,M.;Zhang,T.;Zheng,L.;Sen,K.; and Stoica, I
-
[7]
Pan,J.;Wang,X.;Neubig,G.;Jaitly,N.;Ji,H.;Suhr,A.;and Zhang, Y
Reasoningbank: Scaling agent self-evolving with reasoning memory.arXiv preprint arXiv:2509.25140. Pan,J.;Wang,X.;Neubig,G.;Jaitly,N.;Ji,H.;Suhr,A.;and Zhang, Y
-
[8]
Shao,Z.;Wang,P.;Zhu,Q.;Xu,R.;Song,J.;Bi,X.;Zhang, H.; Zhang, M.; Li, Y.; Wu, Y.; et al
Training software engineering agents and verifiers with swe-gym.arXiv preprint arXiv:2412.21139. Shao,Z.;Wang,P.;Zhu,Q.;Xu,R.;Song,J.;Bi,X.;Zhang, H.; Zhang, M.; Li, Y.; Wu, Y.; et al
-
[9]
Shinn, N.; Cassano, F.; Gopinath, A.; Narasimhan, K.; and Yao, S
Deepseekmath: Pushing the limits of mathematical reasoning in open lan- guage models.arXiv preprint arXiv:2402.03300. Shinn, N.; Cassano, F.; Gopinath, A.; Narasimhan, K.; and Yao, S
-
[10]
Step Rejection Fine-Tuning: A Practical Distillation Recipe
Reflexion: Language agents with verbal re- inforcement learning.Advances in neural information pro- cessing systems, 36: 8634–8652. Slinko,I.;Zavidnyi,I.;Bogomolov,E.;andZharov,Y.2026. Step Rejection Fine-Tuning: A Practical Distillation Recipe. arXiv preprint arXiv:2605.10674. Tan,H.-Z.;Yang,X.-W.;Chen,H.;Shao,J.-J.;Wen,Y.;Shen, Y.;Luo,W.;Du,X.;Guo,L.-Z....
work page Pith review arXiv 2026
-
[11]
Wang,S.;Li,G.;Yang,Z.;andGao,Y.2026
Voyager: An open- ended embodied agent with large language models.arXiv preprint arXiv:2305.16291. Wang,S.;Li,G.;Yang,Z.;andGao,Y.2026. HindsightHint Distillation: Scaffolded Reasoning for SWE Agents from CoT-free Answers.arXiv preprint arXiv:2605.11556. Wang, X.; Li, B.; Song, Y.; Xu, F. F.; Tang, X.; Zhuge, M.; Pan,J.;Song,Y.;Li,B.;Singh,J.;etal.2025.Op...
arXiv 2026
-
[13]
arXiv preprint arXiv:2407.01489
Agent- less: Demystifying llm-based software engineering agents. arXiv preprint arXiv:2407.01489. Yang, J.; Jimenez, C. E.; Wettig, A.; Lieret, K.; Yao, S.; Narasimhan, K. R.; and Press, O
Show all 16 references
-
[14]
InThe Thirty-eighth Annual Conference on Neural In- formation Processing Systems
SWE-agent: Agent- Computer Interfaces Enable Automated Software Engineer- ing. InThe Thirty-eighth Annual Conference on Neural In- formation Processing Systems. Yang, J.; Lieret, K.; Jimenez, C.; Wettig, A.; Khandpur, K.; Zhang,Y.;Hui,B.;Press,O.;Schmidt,L.;andYang,D.2026. Swe...
2026
-
[15]
Zelikman, E.; Wu, Y.; Mu, J.; and Goodman, N
Scaling relationship on learning mathematical reasoning with large language models.arXiv preprint arXiv:2308.01825. Zelikman, E.; Wu, Y.; Mu, J.; and Goodman, N
-
[16]
Zhang,K.;Li,J.;Li,G.;Shi,X.;andJin,Z.2024.Codeagent: Enhancing code generation with tool-integrated agent sys- tems for real-world repo-level coding challenges
Glm-5: from vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763. Zhang,K.;Li,J.;Li,G.;Shi,X.;andJin,Z.2024.Codeagent: Enhancing code generation with tool-integrated agent sys- tems for real-world repo-level coding challenges. InPro- ceedings of the 62nd Annual M...
2024 arXiv
-
[2021]
InInternational Conference on Machine Learning, 4940–4950
Prior- itized level replay. InInternational Conference on Machine Learning, 4940–4950. PMLR. Jimenez, C. E.; Yang, J.; Wettig, A.; Yao, S.; Pei, K.; Press, O.;andNarasimhan,K.R.2024.SWE-bench:CanLanguage Models Resolve Real-world Github Issues? InThe Twelfth International Conf...
2024
-
[2023]
Huang, D.; Zhang, J
Reinforced self-training (rest) for language modeling.arXiv preprint arXiv:2308.08998. Huang, D.; Zhang, J. M.; Luck, M.; Bu, Q.; Qing, Y.; and Cui, H
-
[2024]
InInternationalConference on Learning Representations, volume 2024, 57734–57811
Critic: Large language models can self-correct withtool-interactivecritiquing. InInternationalConference on Learning Representations, volume 2024, 57734–57811. Gulcehre, C.; Paine, T. L.; Srinivasan, S.; Konyushkova, K.; Weerts, L.; Sharma, A.; Siddhant, A.; Ahern, A.; Wang, M...
2024
-
[2025]
arXiv preprint arXiv:2504.13145
Exploring expert failures improves llm agent tuning. arXiv preprint arXiv:2504.13145. Liu,Z.;Liu,S.;Zhong,T.;andYuan,M.2026. RIFT:Repur- posingNegativeSamplesviaReward-InformedFine-Tuning. In Liakata, M.; Moreira, V. P.; Zhang, J.; and Jurgens, D., eds.,Findings of the Associa...
2026 arXiv
-
[2026]
Learn hard problems dur- ing RL with reference guided fine-tuning.arXiv preprint arXiv:2603.01223. Xia, C. S.; Deng, Y.; Dunn, S.; and Zhang, L
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.