Pith. sign in

REVIEW 3 major objections 3 minor

Weight writes can store usable facts from broad data, but later writes make earlier facts unreachable no matter the intervention tested.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-15 09:02 UTC pith:T5H7FFRB

load-bearing objection Abstract-only: sequential weight writes create usable but question-keyed knowledge that later writes make unreachable; context wins for accumulation, methods still uncheckable. the 3 major comments →

arxiv 2607.11020 v2 pith:T5H7FFRB submitted 2026-07-13 cs.CL cs.LG

Can a Language Model Learn Facts Continually in Its Weights?

Charles O'Neill This is my paper
classification cs.CL cs.LG
keywords continual learninglanguage modelsweight writesfact retentioncatastrophic forgettingcontext vs weightsQwen3knowledge storage
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether a language model can keep learning new facts by writing them into its weights, and whether those facts stay usable after many later writes. Using invented facts trained into Qwen3 models and scored on five kinds of held-out questions, it shows that the breadth of the training data decides what kind of knowledge is created: bare statements mostly produce recitation, while diverse restatements shrink the gap between reciting a fact and using it, without ever showing the model a conclusion. That difference survives into sequential learning: after twenty later writes, bare-statement facts fall to near-zero accuracy while facts written from broad study data still answer at 46 percent. The same experiments show that facts can be behaviourally forgotten without being erased from the weights; most of the log-probability gain remains, wrong answers often contain the most recent fact, and supplying a forgotten fact in the prompt recovers performance to 77-80 percent. No intervention tested, even those that measure each write carefully, keeps earlier facts reachable once later writes arrive. When facts must be composed or survive later overwriting, the reliable channel is therefore context rather than the weights.

Core claim

Broad training data can create usable knowledge inside a model's weights, and a frozen reference model can preserve capability, but after sequences of later weight writes no tested intervention keeps earlier facts reachable; forgotten facts retain most of their log-probability gain yet become question-keyed and redirected by newer writes, so composition and long-term retention of facts require context rather than the weights.

What carries the argument

Sequential weight writes of invented facts into Qwen3, tracked from creation through 20-100 later writes and scored on five held-out question types against a reference that receives the fact only in its prompt; breadth of study data versus bare statements is the variable that determines recitation-to-use gap and later retention.

Load-bearing premise

That invented facts trained into one model family, scored on five kinds of held-out questions, with the original model given the fact in its prompt as reference, are a valid proxy for whether real knowledge is stored, usable, and retained under continual weight writes.

What would settle it

Train a sequence of real, non-invented facts into the same models using broad study data, then measure whether earlier facts remain answerable on the same five held-out question types after twenty later writes; recovery above the reported 46 percent or success of any intervention that restores reachability would contradict the central claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Bare-statement continual learning yields near-total loss of earlier facts after only twenty writes, so simple fact dumps cannot accumulate knowledge in weights.
  • Diverse restatements create knowledge that is usable and more durable, yet still fails to stay reachable once later writes arrive.
  • Behavioural forgetting does not erase stored log-probability; the fact remains but is no longer retrieved by its original questions.
  • Unrelated abilities degrade in proportion to KL divergence from the original model, independent of how the earlier fact was stored.
  • For any task that requires composing facts or keeping them after later updates, the reliable route is to supply the facts in context rather than rely on weight storage.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the same pattern holds for real knowledge, production systems that continually fine-tune on new facts will need an external memory or retrieval layer rather than pure weight accumulation.
  • The observation that wrong answers contain the most recent fact suggests interference is largely a routing or keying problem, inviting tests that re-key older facts without rewriting them.
  • A frozen reference that restores capability when the forgotten fact is re-supplied in the prompt points to hybrid designs that keep a clean base model and inject facts only at inference time.
  • The recitation-to-use gap reduction without showing conclusions may generalise to other forms of synthetic data diversity beyond the restatements tested here.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The manuscript studies whether continual weight updates can accumulate usable knowledge in language models. Invented facts are written into Qwen3 models and tracked through sequences of 20–100 later writes, evaluated on five types of held-out questions against a reference in which the original model receives the fact in its prompt. The abstract reports that training-data breadth determines knowledge quality: bare-statement training yields recitation, while diverse restatements shrink the recitation-to-use gap (27.4 → 5.4 points). After twenty sequential writes, bare facts retain ~1% accuracy versus ~46% for broad-study facts. Forgotten facts retain most of the log-probability added by their write, and under bare training ~70% of wrong answers contain the most recent fact; re-supplying a forgotten fact in context recovers 77–80%. Damage to unrelated abilities tracks KL from the original model. The authors conclude that knowledge is stored but question-keyed, that no tested weight-based intervention keeps earlier facts reachable, and that context is the reliable channel when facts must be composed or survive later writes.

Significance. If the experimental design holds under full scrutiny, the work is a substantial empirical contribution to continual learning and knowledge editing. It cleanly separates recitation from use, documents behavioural forgetting without erasure, quantifies interference under sequential writes, and supplies a strong negative result on weight-based retention interventions. The breadth-of-data finding and the frozen-reference preservation observation are practically useful. The claim that context, not weights, is the reliable channel for composition and long-horizon retention would reframe research priorities in the area. Credit is due for the multi-type held-out suite, the prompt-reference baseline, and the explicit tracking of log-probability retention alongside accuracy.

major comments (3)
  1. The central negative claim—that no tested weight intervention keeps earlier facts reachable—rests on the validity of invented facts, the five held-out question types as probes of use/composition rather than recitation, and the fairness of the original-model-with-fact-in-prompt reference. The abstract alone cannot establish that surface-form confounds are controlled, that the five types isolate composition, or that the reference fairly separates weight storage from context. These design choices are load-bearing: if any fails, the reported retention numbers (1% vs 46%, 70% recent-fact contamination, 77–80% context recovery) no longer support the conclusion. Full methods, item construction, and ablation of the question suite are required before the claim can be accepted.
  2. The abstract asserts that later writes cause interference 'regardless of how the earlier fact was stored' and that damage tracks KL divergence from the original model. Without the full experimental matrix (which interventions, which KL regimes, statistical uncertainty, and whether held-out items truly isolate use), it is impossible to judge whether the interference result generalises or is an artefact of the particular write schedule and model scale. This is load-bearing for the claim that weight writes cannot support accumulation.
  3. The strong conclusion that 'the reliable channel is context rather than the weights' is an existence claim over the interventions tested. The abstract does not enumerate those interventions or their local-measurement accuracy. A complete list, with negative results stated per method, is needed so readers can assess whether the search was exhaustive enough to support the generalisation.
minor comments (3)
  1. Abstract is carefully written and reports specific percentages and design elements; once the full paper is available, ensure that every numeric claim in the abstract is traceable to a table or figure with error bars or confidence intervals.
  2. The interpretive construct 'question-keyed knowledge' should be defined operationally in the main text (e.g., via the five question types and the log-probability retention result) so it is not read as a free theoretical entity.
  3. Clarify data-exclusion rules, hyperparameter choices, and whether any facts or questions were filtered post hoc; these are standard reproducibility items that the abstract cannot supply.

Circularity Check

0 steps flagged

No significant circularity: empirical continual-learning measurements against held-out questions and an external prompt reference, with no derivation that reduces by construction to its inputs.

full rationale

This abstract-only paper reports experimental measurements of invented facts written into Qwen3 weights, scored on five types of held-out questions, with the original model given the fact in its prompt as the reference baseline. There are no equations, fitted parameters renamed as predictions, uniqueness theorems, or self-citation chains that force the central claims by construction. Retention numbers (e.g., 1% bare vs 46% broad after twenty writes; 77–80% recovery when a forgotten fact is re-supplied in context) are outcomes of sequential training and evaluation, not tautological restatements of fitted targets. Defining “knowledge” via the same held-out question suite used for scoring is ordinary experimental self-consistency, not equation-level circularity. Proxy-validity concerns about invented facts or question design are correctness risks outside the circularity criteria. With only the abstract available and no load-bearing self-definitional or self-citation reduction quotable from the text, the score is 0 and steps is empty.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 1 invented entities

Empirical continual-learning study. No free parameters are fitted to produce a theoretical constant; the claim rests on domain assumptions about proxies (invented facts, held-out question types, prompt-supplied original model as reference) and on unstated training/evaluation details unavailable in the abstract. No new physical or mathematical entities are postulated—only interpretive labels for observed storage behavior.

axioms (4)
  • domain assumption Invented facts written into model weights are a valid proxy for real-world factual knowledge acquisition and retention.
    Abstract builds the entire experimental program on invented facts; if they do not behave like real knowledge under use and composition, the central claim about weight vs context does not transfer.
  • domain assumption Accuracy on five types of held-out questions distinguishes usable knowledge from mere recitation.
    The recitation-to-use gap and retention numbers depend on this measurement design as stated in the abstract.
  • domain assumption The original model given the fact in its prompt is an appropriate reference for what successful knowledge should look like.
    Abstract explicitly uses this condition as the reference against which weight-written knowledge is scored.
  • domain assumption Sequential fine-tuning writes of later facts are a fair operationalization of continual weight-based learning.
    Interference and reachability conclusions assume this training regime matches the intended continual-learning setting.
invented entities (1)
  • question-keyed knowledge (interpretive construct) independent evidence
    purpose: Label for the observed pattern that facts remain in log-probability mass but later writes redirect the questions that used to reach them.
    Descriptive name for a behavioral pattern, not a new mechanism with independent mass or force; independent_evidence is the held-out question and contamination measurements themselves.

pith-pipeline@v1.1.0-grok45 · 6214 in / 2747 out tokens · 47302 ms · 2026-07-15T09:02:23.995958+00:00 · methodology

0 comments
read the original abstract

Continual learning promises a language model that keeps acquiring knowledge after training, with each new fact written into its weights. Whether weight writes can support accumulation remains undecided. We follow invented facts written into Qwen3 models from creation through sequences of twenty to one hundred later writes, using held-out questions of five types, with the original model given the fact in its prompt as the reference. Across these experiments, the breadth of the training data determines the kind of knowledge created. Bare-statement training produces recitation, while diverse restatements reduce the recitation-to-use gap from 27.4 to 5.4 points without showing the model a conclusion. This difference carries into later writes: after twenty sequential writes, bare-statement facts retain 1% accuracy while facts written from broad study data retain 46%. We also find that facts can be behaviourally forgotten without being erased. Forgotten facts keep most of the log-probability added by their write, and under bare-statement training 70% of wrong answers about them contain the most recently written fact. The same writes barely degrade the model's use of facts in context, and a forgotten study fact supplied in the prompt recovers to 77-80% on its questions. These results describe knowledge that is stored but question-keyed: later writes redirect the questions that reached it. Damage to unrelated abilities tracks KL divergence from the original model, and the later writes cause interference regardless of how the earlier fact was stored. Broad data can create usable knowledge, and a frozen reference can preserve capability, but no intervention we tested, including those built on accurate local measurements of each write, keeps earlier facts reachable. When facts must be composed or survive later writes, the reliable channel is context rather than the weights.

Figures

Figures reproduced from arXiv: 2607.11020 by Charles O'Neill.

Figure 1
Figure 1. Figure 1: Strict accuracy by question type as the optimisation budget grows for bare-statement training (indigo) and study training (rose). The dashed line is the fact-in-prompt ceiling and the dotted line the floor from the original model. Counterfactual accuracy stays at 21–23% under bare-statement training and reaches 45–50% under study training; composition is 39–41% against 59–60%, below the 83% ceiling. the ki… view at source ↗
Figure 2
Figure 2. Figure 2: The entailment gap on the primary evaluation. Solid bars show strict accuracy and light caps extend them to lenient accuracy. The cap is the share of answers that state an entailing fact without giving the requested conclusion. use uncued questions, which never restate the training phrasing; cued phrasings can supply the missing step themselves and collapse use back to recitation, a boundary measured in Ap… view at source ↗
Figure 3
Figure 3. Figure 3: The factorial over objective, data, and update method. a, strict accuracy after one write, with each connected pair showing the bare-statement-to-study change. b, the conclusion-type entailment gap against survival after twenty later writes. c–d, capability damage against endpoint KL from the original model and cumulative local KL. Colour denotes training data, marker shape the objective, and fill the upda… view at source ↗
Figure 4
Figure 4. Figure 4: The entailment gap for bare-statement and study training in three configurations. The gap follows the training data, not rank or scale. A GRPO condition with a programmatic answer-match reward received almost no signal because the base model never produced the target counterfactuals, and the few reward gains did not install the fact; we leave reinforcement-learning objectives that can write new knowledge t… view at source ↗
Figure 5
Figure 5. Figure 5: Retention of earlier facts after twenty sequential writes, strict, with counterfactual questions excluded. Recitation dies; half of the stated conclusions survive. leaves the held-out capability tests at 54% (against 3–37% at the full budget and roughly 80% before any writing), bare-statement facts still retain only 6%, so capability and retention dissociate. The comparison also shows an on-policy advantag… view at source ↗
Figure 6
Figure 6. Figure 6: Retention on the prior-conflict evaluation. Left: strict accuracy on earlier facts after k writes, pooled over tiers and seeds. Facts written from bare statements are near zero by k = 5. Right: retention after all twenty writes by prior tier; prior-inverting facts survive better than neutral ones under both methods. 20 40 60 80 100 facts written so far (k) 0 20 40 60 retention, strict (%) 20 40 60 80 100 f… view at source ↗
Figure 7
Figure 7. Figure 7: One hundred sequential study writes on the prior-conflict evaluation. Left: retention decays to a 25–28% plateau, and periodic consolidation into the original model does not lift it. Right: consolidation does preserve the held-out capability tests, recovering them at each pass. prior-inverting ones (95% CI [−21.3,−2.8], n = 19 paired triples), under both training methods. The reversal remains uncertain: a … view at source ↗
Figure 8
Figure 8. Figure 8: One hundred sequential writes across the reduced factorial (three seeds per condition; descriptive, with pooled point estimates and no uncertainty bands). Left: strict retention of earlier facts; the bare-statement/study separation persists to k=100 under both objectives and both update methods. Right: the held-out capability tests over the same runs. We then test the mitigation suggested by §6: every twen… view at source ↗
Figure 9
Figure 9. Figure 9: Storage survives behavioural forgetting. Left: the written statement’s log-probability, as a fraction of the lift its own write produced (Eq. 5.1), stays far above the pre-write floor across all subsequent writes. Lines are medians and shading the interquartile range over facts, for three seeds. Middle: facts that fail every question at k=20 still retain most of their statement lift. Solid lines are median… view at source ↗
Figure 10
Figure 10. Figure 10: Two written facts largely cannot be used together. Bars show strict accuracy on questions requiring both facts of a pair, by placement. Within each placement, the model answers directly, answers after its own retrievals are placed in context, and answers with the true statements placed in context. The dotted line is the both-in-prompt reference; the dashed line is the same true-statements-in-context measu… view at source ↗
Figure 11
Figure 11. Figure 11: Continued writing does not preferentially erode in-context use. Left: strict accuracy on fresh facts placed in context (solid) and on the held-out capability tests (dashed), across the sequential runs. Right: the sequence’s own written facts at k=20, answered with their statements supplied in context. The sequence’s own forgotten facts complete the argument. With their statements supplied in context, ques… view at source ↗
Figure 12
Figure 12. Figure 12: The twelve continual-learning conditions, by regime, objective, and rank. Left: capability change in percentage points from the ≈80% baseline. Right: facts kept, measured as earlier-fact accuracy after the twentieth sequential write and as all-fact accuracy for the batch conditions. The two extremes are both context distillation. Accuracy immediately after writing is uniform (76–78% sequential, 62–72% bat… view at source ↗
Figure 13
Figure 13. Figure 13: Capability loss against KL divergence from the original model, one point per condition; ρ = 0.83. sequential, 20 facts CD teacher: own merges batch, 100 facts CD teacher: original batch, 200 facts CD teacher: original −30 −20 −10 0 Capability change (pp) -6 -4 -2 +1 -3 -11 -16 -28 -20 -26 -22 -19 SFT, rank 4 SFT, rank 16 CD, rank 4 CD, rank 16 [PITH_FULL_IMAGE:figures/full_fig_p014_13.png] view at source ↗
Figure 15
Figure 15. Figure 15: Sequential distillation from a frozen teacher. Distilling each write against a frozen copy of the original model (indigo) preserves capability, retains earlier facts, and stays close to the original. Distilling through the model’s own accumulated merges (rose) loses all three. The own-merges retention bar scores its capped outputs as failures (21%; 34% excluding them). We tested this account sequentially,… view at source ↗
Figure 16
Figure 16. Figure 16: A per-write KL penalty to the current base on bare-statement sequential SFT, swept over its weight λ. Left: capability and retention recover sharply once λ > 0, while write accuracy falls modestly. Right: the measured KL from the original does not fall systematically with λ, so the rescue is not achieved by reducing drift. 5 10 15 20 Facts written so far (k) 0 50 100 Accuracy (%) (a) earlier facts 1 5 10 … view at source ↗
Figure 17
Figure 17. Figure 17: Sequential writing of twenty facts. Left: accuracy on earlier facts after k writes. Right: accuracy on each new fact at the time it is written. The ability to write facts never degrades; the written facts do. Two dissociations in [PITH_FULL_IMAGE:figures/full_fig_p016_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Causal tests of creation and interference. A: bare-statement supervision leaves a 27.4-point entailment gap (95% CI [20.2,35.1]); diverse recitation, with no derived conclusions, reduces it to 5.4 points [1.8,8.8]. B: the linearised Adam update predicts the next update’s realised effect (ρ = 0.795) but not eventual forgetting (ρ = −0.258). C: change in retention after fifteen later writes, by later-write … view at source ↗
Figure 19
Figure 19. Figure 19: Paired effects on final strict retention in the crossed experiment. Intervals are 95% bootstrap intervals over 24 stored facts after averaging three seeds. 7.3 Local prediction and control We next ask whether interference can be read from an update. At steps 0, 1, 4, 16, 64, and 192, we measured changes on held-out prompts (statement, gold-answer, and default-answer log-probabilities). We also saved the a… view at source ↗
Figure 20
Figure 20. Figure 20: Intervention controls. A: Each row is an eligible tested 144-step bare-statement plus 48-step bridge condition; indigo marks a passed individual criterion. No row passes all four. B: Retention change relative to ordinary Adam. Targeted gradient projection gains 1.4 points, random projection 1.8, and magnitude masking 5.0. Intervals resample the 60 paired store–seed–fact units; only five stored facts are u… view at source ↗
Figure 21
Figure 21. Figure 21: The matched-triple construction. Three invented entities share a domain and statement structure and differ only in how the stated property relates to a prior belief: it confirms the default, is neutral to any default, or inverts the default. A regression on the existing facts was not enough: per-fact prior strength gave a null for counterfactual use (ρ ≈ 0 in every condition, n = 179), but that analysis i… view at source ↗
Figure 22
Figure 22. Figure 22: Floor and ceiling for the three tiers under the certified judging pipeline, with the fraction of questions surviving the screen in the tick labels. Floors of 0.9–6.2% show the questions do not leak their answers in any tier. −5 0 5 10 15 study − bare gap, strict (pp) invert neutral confirm all answers non-truncated [PITH_FULL_IMAGE:figures/full_fig_p030_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: The conflict experiment’s primary quantity (strict policy): the study advantage over bare-statement training on use-type questions, paired within triples, averaged over three seeds, with 95% CIs, by prior tier (invert 77 triples, neutral 80, confirm 61). Every interval crosses zero and none reaches the +10pp support band (dotted); recomputed on non-truncated rows only (“non-truncated”), the gaps collapse … view at source ↗
Figure 24
Figure 24. Figure 24: Left: under the strict policy the diversity effect is small and every interval crosses zero; the expected effect size (shaded band) sits at the upper edge of the intervals. Right: composition sits at 37–58% for every recipe in every tier while the in-context ceiling spans 77–95%. conditions (6.1% versus 0.1% at 96 steps), the same defect class as the answer-length cap of §2.1. Recomputed on non-truncated … view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.