REVIEW 3 major objections 5 minor 30 references
Why Does CLAUDE.md Keep Growing? Catastrophic Remembering in Agentic Coding
T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Agentic prompts grow without bound because instructions outlive their rationale, and comments stop it.
desk verdict A solid observational study of agentic prompt growth, with a real confound in the experimental half that needs fixing before the central causal claim holds. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the instruction's latent reasoning r_d, the unrecorded rationale that made a maintainer add instruction d. Its recoverability ρ(d,a) decays with instruction age a, and flow balance on prompt size, E[Δ|D|] = A_t − Σ_d h(a_d), gives a divergent equilibrium |D_∞| = A/(ρ̄ s) as ρ̄→0. The paper's controllable testbed inverts a verifiable instruction-following benchmark: the benchmark's original instructions become the known minimum cover D⋆, its verifiers become hidden constraints, and a fresh maintainer must reconstruct D⋆ from a lossy objective under censored feedback. In that world the intervention is prompt comments—annotations stripped before execution that pass the why from one maintainer to the next—and they are what turns the unbounded ratchet into settlement near cover.
What would settle it
Run the same inverted-benchmark maintenance loop with a delete-and-revert protocol allowed in the uncommented arm; if that arm also settles near the minimum cover, the imperfect-recall mechanism is not what drives growth. Complement it with a repository search for commits where an instruction deletion was followed by a same-file or same-task revert, which would show trial deletion happening in the wild.
Extended reading notes
Core claim
Catastrophic remembering is the paper's name for the divergence in which a maintainer keeps instructions that should have been removed, the mirror image of catastrophic forgetting in continual learning. The formal claim is that the equilibrium prompt size diverges as the recoverability of latent reasoning ρ̄(a) decays to zero, even with a constant arrival rate of additions and a fixed constraint set, because the deletion hazard factorizes as h(a)≈ρ̄(a)s(a). The empirical discovery is that in real agentic prompts the deletion hazard falls with instruction age (log-hazard slope −0.032 per commit, steeper in multi-author files), which matches the imperfect-recall signature and contradicts staleness and content-fragility explanations. The experimental discovery is that comments carrying latent reasoning settle maintained prompts at the minimum cover: excess size drops from +211.3% to +1.4% over 51 steps at parity constraint satisfaction, and real-prompt instruction-following rises by 11.6 percentage points.
Load-bearing premise
The result depends on the assumption that maintainers cannot safely delete an instruction just by trying it and reverting if something breaks; if trial deletion is cheap, the exponential-audit premise and the whole divergence argument lose their force.
Editorial extensions
If this is right
- If prompt comments encode latent reasoning, agentic prompt files can stabilize near their minimum cover instead of tripling in size.
- Instruction-following improves when noisy instructions are pruned, so adding comments buys both smaller prompts and higher compliance.
- Deletion policies should be tied to recoverable rationale, not to instruction age or file size, because age-based pruning targets the wrong mechanism.
- Wholesale rewrites are a temporary reset, not a cure: growth resumes immediately and faster after a rewrite, so durable maintenance requires preserving reasoning across edits.
- As coding agents get more capable at maintaining prompts, uncommented prompts ratchet harder, so the benefit of comments grows with maintainer capability.
Reading between the lines
- A natural next test is whether trial deletion—delete, observe failures, revert—is cheap enough in practice to bypass the exponential audit entirely; if it is, the divergence rate should depend on revert cost rather than recall alone.
- The same decaying-reasoning mechanism may apply to system prompts and agent skill files, which the paper explicitly leaves outside its scope.
- Comment fidelity will likely become the next weak point: stale or misleading rationale could reintroduce the code-comment divergence problem, so tooling that detects comment-instruction inconsistency would be a natural extension.
- The repository analysis is observational, so a field experiment that gives real maintainers comment syntax and measures growth before and after would test whether the controlled result transfers.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies the unbounded growth of agentic coding context files (CLAUDE.md, AGENTS.md, copilot-instructions.md). It proposes that growth is caused by imperfect recall: the latent reasoning behind an instruction decays over time, and because deleting an instruction without that reasoning risks an unverifiable correctness regression (claimed to cost O(2^|D|) counterfactual probes), rational maintainers only add and rarely delete. The paper formalizes this as "catastrophic remembering," reports an observational study of 247,694 instruction lifetimes from 1,867 GitHub repositories showing that instruction counts grow by a median +226%, that deletion hazard falls with instruction age (log-hazard slope -0.032/commit), and that this decline survives gamma-frailty correction. It then presents a controlled experiment using an inverted IFEval testbed with a known minimum cover, in which prompt comments encoding latent reasoning reduce excess size from +60.4% to -5.8% (66.2pp) at parity constraint satisfaction, and a WildIFEval-based experiment in which comments improve instruction-following by up to 23.1%. The paper concludes that comments encoding latent reasoning halt the ratchet and buy back instruction-following.
Significance. If the central claims hold, the paper identifies a novel and practically important failure mode in agentic prompt maintenance and offers a simple, cheap intervention (comments) that could substantially improve LLM-agent behavior in real repositories. The work is unusually careful in several respects: the observational analysis uses pre-registered gates, explicit competing-risk censoring, conservative bias arguments, gamma-frailty models, and a multi-author interaction test that discriminates imperfect recall from content fragility. The experimental design makes the minimum cover observable, enabling a direct measurement of excess size, and includes placebo arms and ablations. The paper also reports reproducible machinery and explicit limitations. These strengths are substantial. However, the central causal claim of the experiment is threatened by a confound: the treatment arm differs from control in both the content of the comments and an explicit deletion-policy addendum in the system prompt, so the reported 66.2pp reduction is not uniquely attributable to latent-reasoning information.
major comments (3)
- [§4.2, Figure 7, Table 2] The treatment arm is confounded with an explicit deletion policy. The informative-comments arm alone receives the protocol addendum of Figure 7(b), which instructs the maintainer when to delete ('Delete a directive when its comment's failure kept recurring anyway... or when it duplicates another directive's target'); control and placebo receive only the generic system prompt of Figure 7(a). The text in §4.2 states that "Arms differ only in the handoff," but this is contradicted by Figure 7, where the system prompts also differ. The 66.2pp excess-size reduction in Table 2 may therefore be produced by the addendum's explicit pruning rule rather than by the latent-reasoning content of the comments. Table 10 holds the addendum constant within the informed-comment family and never tests the addendum with comment-shaped noise or with no comments. Without an arm that combines the addendum with uninformative or absent comments, the paper's central claim that comments encoding latent reasoning halt growth is not established. A factorial design (comments content × deletion-policy addendum) or an explicit control for the addendum is required before the causal reading is licensed.
- [§2, Eq. (4)–(5)] The derived O(2^|D|) cost of safe deletion assumes that an instruction can be removed only after an exhaustive subset audit that estimates Δc(D|d) for every combination of instructions. This rules out the cheap trial-deletion strategy in which a maintainer deletes one instruction, observes subsequent failures or their absence, and re-adds it if needed. Real maintainers—and the agents modeled here, who can add and delete in each round—can implement this strategy, so the claim that deletion "without risking a correctness regression" costs O(2^|D|) is too strong. The divergence in Eq. (5) follows from the decay assumption ρ(a)→0 together with this infeasibility claim; if trial deletion is feasible, unbounded growth no longer follows from imperfect recall alone. The paper should address trial deletion explicitly, either by arguing why it is unavailable or prohibitively costly in the settings studied (e.g., one-shot task execution or high regression cost), or by extending the model to allow for probed deletions. As written, the theoretical foundation overstates the necessity of comments.
- [§4.2, Table 8, Appendix C.1] The headline experimental effect is largely a two-instruction-cover result. In the |D*|=3 stratum at T=15, the uncommented arm's excess size CI is +15.2% [0.5, 31.8], which grossly overlaps cover, and the pre-registered criterion (i) of Table 11 does not hold in this slice. The paper acknowledges in Limitations that covers "run to two or three instructions against a median file's 39," but this undercuts the generality of the claim "comments halt growth": the mechanism is demonstrated only at very small constraint counts, while the corpus observations concern far larger prompts. The pooled numbers in Table 2 combine strata that behave differently, so the 66.2pp reduction is not a stable estimate of the effect at realistic instruction counts. The paper should either present the small-cover scope as a direct limitation in the main text or provide evidence that the effect persists at larger covers, for example by increasing the number of verifiers per world.
minor comments (5)
- [Title and Abstract] The title "Why DoesCLAUDE.mdKeep Growing?" has missing spaces; it should read "Why Does CLAUDE.md Keep Growing?". Also "O(2|D|)" in the abstract is missing the superscript caret and should appear as O(2^{|D|}).
- [Table 2] The notation "± is half a 95% bootstrap CI, asymmetric" is confusing because the displayed intervals are not symmetric; it would be clearer to state that intervals are percentile bootstrap intervals and report the lower and upper bounds explicitly.
- [§4.1, Appendix B.1] The description of the inverse-IFEval transform says that the maintainer "reconstructs D_j from o_j" in T steps, but it does not say how the number of verifiers (|D*|) is hidden. Appendix B.1 states that covers run to two or three instructions, which is a small discretization; a brief note in the main text that the cover size distribution is {2,3} would make the scope of the experiment transparent earlier.
- [§4.2 and Figure 6] Figure 6's caption says "3 of 8,541 additions, one per constraint family" without defining whether these numbers refer to the full experimental run or to a single seed. Adding a sentence that clarifies the counting would improve reproducibility.
- [Limitations] The Limitations section is unusually candid, but it is placed at the end of the paper after the Discussion; in a journal format the key limitations (e.g., small-cover experiment, LLM-judged WildIFEval, single-annotation matcher) should be summarized in a shorter form near the introduction or the corresponding results sections.
Circularity Check
Equation 5's divergence is a restatement of the assumed ρ→0 decay, and the comment-arm effect is confounded by the deletion-policy addendum.
-
self definitional
[Section 2, Equations 2–5]
"Define the decay ρ(d, a) = Pr(rd recoverable at age a), ... both ρ(d, a)→0 and ¯ρ(a)→0 as a→∞. We call that decay imperfect recall ... A maintainer rationally deletes an instruction only when that reason survives and it verifies as excess, so the hazard in its own age factorizes ... h(a)≈¯ρ(a)s(a) ... Flow balance then yields unbounded growth: |D∞|= A/(¯ρ(a)s(a)) − − − − → ∞ as a→∞."
The model's input is the assumption that imperfect recall means recoverability decays to zero, ρ(a)→0. The deletion hazard is then set to h(a)≈ρ̄(a)s(a), and Equation 5 is just the fixed point of A = Σh: with constant additions A and hazard tending to zero, the equilibrium size A/h necessarily diverges. The headline 'catastrophic remembering' is therefore a formal restatement of the assumed decay, not a consequence derived from independent first principles. Section 3.4 then offers the falling deletion hazard as evidence for imperfect recall, but that fall was already encoded in the assumed ρ̄(a)→0, so the confirmation is circular.
-
other
[Section 4.2, Figure 7(b), Table 2]
"Arms differ only in the handoff: the prompt instructions along with its comments (or not), containing a maintainer's summarized latent reasoning. ... Before deleting any directive, read its comment: if its failure has not recurred since the round it was added, the directive is likely PREVENTING that failure -- keep it. Delete a directive when its comment's failure kept recurring anyway (falsified), or when it duplicates another directive's target."
The informative-comments arm is the only arm that receives the protocol addendum explicitly telling the maintainer when to delete directives. Table 2's 66.2pp reduction in excess size is therefore a contrast between (comments plus an explicit deletion rule) and (no comments plus no deletion rule), not a contrast between comment contents. The causal claim that 'comments encoding latent reasoning' halt growth is fixed by construction: the treatment's deletion behavior is commanded by the addendum. The ablations in Table 10 hold the addendum constant and vary only the comment text, so they cannot separate the addendum's instruction to delete from the information carried by the comments. No arm combines the addendum with uninformative comments, leaving the headline attribution unidentified.
full rationale
Two load-bearing reductions are visible in the paper's own equations and design. First, Equation 5's unbounded growth is a direct algebraic consequence of defining imperfect recall as ρ(a)→0 and factorizing the deletion hazard as h≈ρ̄s; the headline divergence is thus a restatement of the model's input rather than an independent prediction. Second, the controlled experiment's central contrast is confounded by construction: the informative-comments arm alone receives the Figure 7(b) deletion-policy addendum, so the reported reduction in excess size cannot be attributed to the comments' latent-reasoning content. The corpus study in Section 3 is an independent, honest measurement, and there are no load-bearing self-citations (the repository frame is external and the ablations and judge-robustness checks are informative within the protocol). However, those independent parts do not rescue the two headline claims: the theory's divergence is encoded in its definition, and the experiment's causal attribution is not identified. Appendix B.3's admission that no arm separated until five feedback conditions were tuned is an additional reverse-fitting concern, though disclosed; I treat it as a selection issue rather than a separate definitional reduction. Overall this is partial circularity rather than complete equivalence, so the score is 6.
Assumptions & free parameters
free parameters (6)
- Rewrite threshold (50% of live instructions) =
0.5
- Fuzzy match similarity threshold =
70
- Instruction segmentation grammar =
per-corpus clause-level grammar (markdown split plus sentence split)
- Five feedback conditions for arm separation =
vague verdicts, censored passes, withheld constraint identifiers, no brevity objective, horizon >= 10
- WildIFEval noise and holdout counts (G, K, N) =
G=16, K=1, N=5 or 6
- Gamma frailty variance (theta) =
1.44, 1.49, 3.41, etc. depending on clustering
assumptions (6)
- domain assumption Properties A1-A5 of the hidden constraint set (instructability, interference, redundancy, censoring, stochasticity)
- domain assumption Deleting an instruction safely costs O(2^|D|) subset probes, with no trial-deletion and revert option
- domain assumption Latent reasoning recoverability rho(d,a) decays to zero with age
- domain assumption Instructions can be identified across versions by exact, normalized, and fuzzy text matching
- domain assumption IFEval verifiers and WildIFEval human-written constraints are valid proxies for hidden constraints in agentic coding
- domain assumption The LLM judge's verdicts approximate ground truth for prose constraints
invented entities (3)
-
Latent reasoning r_d
-
Catastrophic remembering
-
Minimum cover D*
Cite this review
Pith. "Pith review of Why Does CLAUDE.md Keep Growing? Catastrophic Remembering in Agentic Coding." pith.science (2026). https://pith.science/paper/FRZA3CZQ
@misc{pith2026260811095,
author = {Pith},
title = {Pith review of: Why Does CLAUDE.md Keep Growing? Catastrophic Remembering in Agentic Coding},
year = {2026},
howpublished = {\url{https://pith.science/paper/FRZA3CZQ}},
note = {Machine review of arXiv:2608.11095}
}
read the original abstract
Agentic coding READMEs like CLAUDE.md grow without bound in real repositories, stopping only when the repository retires or someone rewrites the file wholesale. We trace this to imperfect recall: appending an instruction is always cheap, but once an instruction's rationale is gone, deleting it without risking a correctness regression costs O(2^|D|) in a prompt of |D| instructions. We name the resulting divergence catastrophic remembering, the inverse of catastrophic forgetting around which continual learning is organized. First, we characterize this phenomenon across 247,694 instruction lifetimes in 1,867 repositories: agentic prompts grow without bound, more than tripling over their lifetime (+226%), gaining +4.9 net instructions every commit; further, the older an instruction gets, the less likely it is to be deleted (log-hazard -0.032/commit). Then, we show that prompt comments can halt the growth: inverting IFEval yields verifiable worlds whose optimal prompts are known, and there comments encoding latent reasoning remove 99.3% of excess instructions (+211.3% to +1.4%). Finally, applying the same inversion to WildIFEval, we show that prompt comments can improve real-world agentic instruction-following by up to 23.1%. If English is the new code, why don't we have comments yet?
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Chatlatanagulchai, Worawalan and Li, Hao and Kashiwa, Yutaro and Reid, Brittany and Thonglek, Kundjanasith and Leelaprute, Pattara and Rungsawang, Arnon and Manaskasemsak, Bundit and Adams, Bram and Hassan, Ahmed E. and Iida, Hajimu , year =. Agent. 2511.12884 , archivePrefix =
-
[2]
Psychology of Learning and Motivation , volume =
Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem , author =. Psychology of Learning and Motivation , volume =. 1989 , publisher =
work page 1989
-
[3]
Trends in Cognitive Sciences , volume =
Catastrophic Forgetting in Connectionist Networks , author =. Trends in Cognitive Sciences , volume =. 1999 , doi =
work page 1999
-
[4]
Proceedings of the National Academy of Sciences , volume =
Overcoming Catastrophic Forgetting in Neural Networks , author =. Proceedings of the National Academy of Sciences , volume =. 2017 , doi =
2017
-
[5]
The Dynamics of Rules: Change in Written Organizational Codes , author =. 2000 , publisher =
work page 2000
-
[6]
Administrative Science Quarterly , volume =
Limits to Bureaucratic Growth: The Density Dependence of Organizational Rule Births , author =. Administrative Science Quarterly , volume =
-
[7]
American Journal of Sociology , volume =
The Dynamics of Organizational Rules , author =. American Journal of Sociology , volume =
-
[8]
The Impact of Heterogeneity in Individual Frailty on the Dynamics of Mortality , author =. Demography , volume =
Show all 30 references
-
[9]
Proceedings of the 22nd International Conference on Mining Software Repositories (MSR) , year =
Prompting in the Wild: An Empirical Study of Prompt Evolution in Software Repositories , author =. Proceedings of the 22nd International Conference on Mining Software Repositories (MSR) , year =. 2412.17298 , archivePrefix =
-
[10]
and Baltes, Sebastian and Treude, Christoph , year =
Lulla, Jai Lal and Mohsenimofidi, Seyedmoein and Galster, Matthias and Zhang, Jie M. and Baltes, Sebastian and Treude, Christoph , year =. On the Impact of. 2601.20404 , archivePrefix =
-
[11]
2604.20300 , archivePrefix =
Gu, Yingjie and Xiong, Wenjian and Wang, Liqiang and Ren, Pengcheng and Li, Chao and Zhang, Xiaojing and Guo, Yijuan and Sun, Qi and Ma, Jingyao and Shi, Shidang , year =. 2604.20300 , archivePrefix =
-
[12]
Proceedings of the IEEE , volume =
Programs, Life Cycles, and Laws of Software Evolution , author =. Proceedings of the IEEE , volume =
-
[13]
2023 , eprint =
Instruction-Following Evaluation for Large Language Models , author =. 2023 , eprint =
2023
-
[14]
2503.06573 , archivePrefix =
Lior, Gili and Yehudai, Asaf and Gera, Ariel and Ein-Dor, Liat , year =. 2503.06573 , archivePrefix =
-
[15]
2024 , url =
Jiang, Yuxin and Wang, Yufei and Zeng, Xiaojun and Zhong, Wanjun and Li, Liangyou and Mi, Fei and Shang, Lifeng and Jiang, Xin and Liu, Qun and Wang, Wei , booktitle =. 2024 , url =
2024
-
[16]
and Lin, Kevin and Wooders, Sarah and Gonzalez, Joseph E
Packer, Charles and Fang, Vivian and Patil, Shishir G. and Lin, Kevin and Wooders, Sarah and Gonzalez, Joseph E. , year =. 2310.08560 , archivePrefix =
-
[17]
2504.19413 , archivePrefix =
Chhikara, Prateek and Khant, Dev and Aryan, Saket and Singh, Taranjeet and Yadav, Deshraj , year =. 2504.19413 , archivePrefix =
-
[18]
2025 , eprint =
Zep: A Temporal Knowledge Graph Architecture for Agent Memory , author =. 2025 , eprint =
2025
-
[19]
Proceedings of CVPR , year =
Learning to Prompt for Continual Learning , author =. Proceedings of CVPR , year =
-
[20]
Wang, Zifeng and Zhang, Zizhao and Ebrahimi, Sayna and Sun, Ruoxi and Zhang, Han and Lee, Chen-Yu and Ren, Xiaoqi and Su, Guolong and Perot, Vincent and Dy, Jennifer and Pfister, Tomas , booktitle =
-
[21]
Smith, James Seale and Karlinsky, Leonid and Gutta, Vyshnavi and Cascante-Bonilla, Paola and Kim, Donghyun and Arbelle, Assaf and Panda, Rameswar and Feris, Rogerio and Kira, Zsolt , booktitle =
-
[22]
Proceedings of the 14th Working Conference on Reverse Engineering (WCRE) , pages =
Do Code and Comments Co-Evolve? On the Relation between Source Code and Comment Changes , author =. Proceedings of the 14th Working Conference on Reverse Engineering (WCRE) , pages =
-
[23]
Proceedings of ACL , year =
Learning to Update Natural Language Comments Based on Code Changes , author =. Proceedings of ACL , year =
-
[24]
Proceedings of AAAI , year =
Deep Just-In-Time Inconsistency Detection Between Comments and Source Code , author =. Proceedings of AAAI , year =
-
[25]
Proceedings of the 44th International Conference on Software Engineering (ICSE) , year =
What Makes a Good Commit Message? , author =. Proceedings of the 44th International Conference on Software Engineering (ICSE) , year =
-
[26]
2011 , howpublished =
Documenting Architecture Decisions , author =. 2011 , howpublished =
2011
-
[27]
The Computer Journal , volume =
Literate Programming , author =. The Computer Journal , volume =
-
[28]
2004 , publisher =
Code Complete: A Practical Handbook of Software Construction , author =. 2004 , publisher =
2004
-
[29]
Proceedings of the 28th International Conference on Software Engineering (ICSE) , pages =
Maintaining Mental Models: A Study of Developer Work Habits , author =. Proceedings of the 28th International Conference on Software Engineering (ICSE) , pages =. 2006 , doi =
2006
-
[30]
Proceedings of the 40th International Conference on Machine Learning (ICML) , series =
Large Language Models Can Be Easily Distracted by Irrelevant Context , author =. Proceedings of the 40th International Conference on Machine Learning (ICML) , series =. 2023 , publisher =
2023
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.