REVIEW 3 major objections 4 minor 21 references
Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge
T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read The paper argues that when task-relevant information is abundant in the training context, language models can reduce loss by using it directly instead of encoding it in weights, and this shift from parametric internalization to…
desk verdict The SFT evidence for context addiction is strong and worth citing; the pretraining sweep is confounded by sequence count and the theory is thin. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the parametric information frontier, $I_k(\rho)=\inf_{\Pi,q_k} I(W;\tau)$ subject to risk $R_k(\Pi,q_k)\le\rho$, together with the monotonicity result that for nested input windows $X^{(k)}=T_{k,m}(X^{(m)})$, $I_m(\rho)\le I_k(\rho)$. It formalizes the claim that context and weights are substitutable carriers of task information, so longer contexts weakly shrink the parametric storage needed for a target risk. The empirical machinery consists of three aligned measurements: average training gradient norm as a proxy for solution complexity; the FFN-to-SA gradient-norm ratio as a proxy for where optimization pressure lands; and module-restricted fine-tuning (updating only attention heads or only feed-forward layers) plus inference-time attention mass, which turns the gradient shift into a causal and behavioral test.
What would settle it
Run the same pretraining sweep while matching sequences per optimization step instead of tokens per step: if the inverted-U disappears, the central pretraining claim is refuted. Independently, on a synthetic task where the learned rule is known, estimate $I(W;\tau)$ directly from weights trained with and without in-context demonstrations; if longer context does not reduce measured task information in the weights, the parametric information mechanism is falsified.
Extended reading notes
Core claim
The central claim is the Information Abundance Paradox: when training context makes task-relevant information available, the model can reduce next-token loss by using that information directly instead of encoding it parametrically, and the learning dynamics shift accordingly. The empirical signature is an inverted-U: across model scales from 20M to 750M parameters, SuperGLUE and closed-book MCQA accuracy peak at a training window of about 2048 tokens and language modeling loss bottoms out around 8192 tokens, then both degrade as the window grows. In fine-tuning, increasing the number of target-domain documents from $k=0$ to $k=8$ under a fixed eight-document budget improves supporting-context accuracy while reducing no-context accuracy and increasing the supporting–conflicting gap. The theoretical account defines the parametric information frontier $I_k(\rho)$, the minimum mutual information $I(W;\tau)$ between weights and task needed to reach risk $\rho$ with context size $k$, and proves $I_m(\rho)\le I_k(\rho)$ for nested contexts $k<m$. Causal interventions on modules tie the behavior to mechanism: attention-only fine-tuning increases context reliance, while feed-forward-only fine-tuning preserves no-context robustness.
Load-bearing premise
The pretraining result hinges on the assumption that changing the context window under a fixed token budget isolates context length; because longer windows put far fewer sequences into each batch, the inverted-U could instead come from the resulting change in gradient noise and per-batch document diversity, a confound the paper does not analyze (it only flags the FLOP difference).
Editorial extensions
If this is right
- Longer pretraining windows are not a free scaling axis: past an intermediate optimum, more context degrades closed-book knowledge and general understanding even when total tokens and updates are matched.
- Fine-tuning models on task-relevant context bakes in a robustness tradeoff: gains with supporting context come with measurable losses when context is absent or misleading, so deployment decisions should depend on how reliable the test-time context will be.
- The optimal training window is not absolute but tied to the evaluation distribution: benchmarks with shorter examples peak at shorter windows, suggesting a rough scaling rule of one to two orders of magnitude above mean example length.
- The shift is visible inside the model: longer-context training lowers the FFN-to-SA gradient ratio, and attention-only fine-tuning reproduces context addiction while feed-forward-only fine-tuning preserves no-context accuracy.
Reading between the lines
- The paper's pretraining sweep fixes tokens per step, so longer-window runs see roughly 64 times fewer sequences per batch; separating context length from batch composition is a natural next experiment, and the inverted-U could plausibly shift or disappear once sequences per step are matched.
- The information-theoretic frontier predicts that $I(W;\tau)$ itself should fall as training context grows; estimating mutual information directly on the synthetic tasks would test the mechanism without relying on behavioral benchmarks.
- If contextualization is systematically the lower-complexity solution, then the same inverted-U may appear for other training choices that make context easier to exploit, such as retrieval-augmented pretraining or in-context learning curricula; the paper does not test these.
- The results suggest a possible mitigation direction: curricula that mix context-free and context-rich examples, or objectives that reward prediction without supporting evidence, might preserve parametric knowledge while still exploiting long contexts; this is an extension, not a claim of the paper.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes the Information Abundance Paradox: when task-relevant information is available in the training context, a language model can reduce loss by using that information directly rather than encoding it in its parameters, thereby shifting from parametric internalization toward contextualization and producing "context addiction" at test time. The authors support this with three main bodies of evidence: (i) a motivating comparison of short- and long-context public models; (ii) pretraining sweeps over context windows showing inverted-U performance on language modeling, SuperGLUE, and closed-book MCQA; and (iii) supervised fine-tuning experiments that fix context length and vary the fraction of task-relevant documents, showing improved supporting-context accuracy but reduced no-context and conflicting-context robustness. They also present a theoretical monotonicity result, gradient-based mechanistic analyses, module-restricted fine-tuning interventions, and inference-time attention analyses. The manuscript argues that long-context training is not a neutral scaling axis and that scaling context windows can change what models internalize.
Significance. If the central claims hold, the paper makes a useful and timely contribution: it challenges the assumption that longer training context is a purely beneficial scaling axis, and it operationalizes a distinction between parametric internalization and contextualization that is relevant to long-context training and evaluation. A particular strength is the SFT design in Section 3.2, which fixes the context budget and varies only the relevance of the prepended documents; this cleanly isolates information abundance and shows consistent effects across five model scales and four domains. The mechanistic analyses, especially the module-restricted fine-tuning and attention-allocation results, provide convergent and falsifiable evidence for the proposed shift. However, the broadest natural-language evidence, the pretraining sweep in Section 3.1, is currently undermined by a sequence-count confound, and the theoretical account in Section 4 is an achievability statement rather than a derivation of the empirical pattern. With additional control experiments and a more careful framing, the contribution would be significant for the long-context training community.
major comments (3)
- [§3.1 (Training setup) and Figure 2] The pretraining comparison is described as "token-and-update-matched, isolating context window effects under a fixed token budget," but with the global batch fixed at approximately 1.05M tokens, the number of sequences per optimization step falls from about 2048 at W=512 to about 32 at W=32768. The sweep therefore changes gradient noise, per-batch document diversity, and the number of independent loss blocks per update, not only the context length. The inverted-U in Figure 2 and the declining FFN/SA gradient ratio in Figure 5b could in principle be produced by these optimization-level changes rather than by reduced pressure toward parametric storage. Footnote 1 flags only the FLOP difference, not the sequence-count confound. Because this pretraining result is the paper's broadest natural-language evidence for the paradox, the confound is load-bearing; please add an experiment or quantitative analysis that directly addresses it, for example by holding sequences per batch fixed while varying the token budget, or by demonstrating that gradient noise and document coverage cannot explain the U-shape.
- [§4, Proposition 4.2] Proposition 4.2 states that I_m(ρ) ≤ I_k(ρ) for nested contexts, and the proof shows that a longer-context predictor can simulate a shorter-context predictor by projection. This is a valid achievability statement, but it does not imply that the trained model actually stores less task information in its weights, and because it is monotone in context size it cannot by itself explain the inverted-U in Figure 2 or the supporting/no-context tradeoff in Figure 3. As written, the theoretical account is close to a formal restatement of the hypothesis rather than a derivation of the empirical results. To make this section load-bearing, the paper would need to connect I_k(ρ) to the training dynamics, for example by showing that the learned weight distribution approaches the infimum or by directly estimating I(W;τ) across context sizes, or it should explicitly reframe the proposition as a motivation rather than an explanation.
- [§5.2, Figure 6] The module-restricted fine-tuning comparison is presented as causal evidence that FFN updates confer parametric robustness and SA updates confer context reliance. However, with LoRA rank fixed at r=64, the FFN and SA adapters contain very different numbers of trainable parameters, so the two conditions also differ in update capacity and optimization geometry. The observed differences in no-context and conflicting-context accuracy could therefore reflect parameter-count or learning-rate effects rather than the functional role of the modules. Please match the trainable parameter budget between the FFN-only and SA-only conditions, for example by adjusting the LoRA rank per module, or otherwise control for update capacity before interpreting Figure 6 as causal evidence.
minor comments (4)
- [§3.1 and App. C.1] The main text says training uses 10B tokens from Project Gutenberg, while App. C.1 says the models are trained for 4 epochs of a 2.5B-token corpus; please reconcile the corpus size and epoch count in the main text.
- [Figure 2 and related figures] The legend entries such as "20M Mean 55M CI95% 259M Inflection Point 750M" are ambiguous about which curve and shaded band correspond to each model scale; use a conventional per-scale color legend with a separate marker for inflection points.
- [App. H.1, Table 16] The full results table is extremely dense; consider reporting the k=0 versus k=8 differences or plotting the deltas so that the domain-level pattern can be verified more easily.
- [§5.1 and App. C.3] The average gradient norm G_k is compared across context lengths k, but the synthetic batches are matched in number of examples rather than number of tokens; please state whether G_k is normalized for sequence length or show that the within-task differences are not an artifact of longer prompts.
Circularity Check
Empirical evidence is self-contained; only Proposition 4.2 is a definitional formalization of the hypothesis, explicitly labeled an achievability statement.
-
self definitional
[Section 4, Proposition 4.2]
"Proposition 4.2 (Monotonicity of the parametric information frontier). If X(k) = Tk,m(X(m)) almost surely for k<m, then Im(ρ) ≤ Ik(ρ) for all ρ."
I_k(ρ) is defined as the minimum parametric information over predictors attaining risk ≤ ρ with context size k. Under the nested-input assumption X(k)=T_{k,m}(X(m)), any k-context predictor can be simulated at context size m by projecting the longer input, so the feasible set for I_m contains the feasible set for I_k with the same weight channel; the inequality then follows directly from taking an infimum over a weakly larger set. Thus the theorem reformulates the hypothesis's 'can reduce' clause by construction rather than deriving it from independent mechanisms. The paper labels it an achievability statement and does not use it to predict the empirical curves, making this a minor definitional flavor rather than load-bearing circularity.
full rationale
The natural-language and synthetic experiments are measured directly through controlled comparisons: the context-window sweep in Section 3.1, the train-time relevance sweep in Section 3.2, the synthetic task families in Section 5.1, the gradient-allocation and module-restricted interventions in Section 5.2, and the attention-allocation analysis in Section 5.3, with significance tests reported in Appendix F. No fitted parameter is renamed as a prediction, and no central result is derived from the authors' prior work; citations involving the authors (e.g., WikiSPAN, MultiRC, Fang et al.) are not load-bearing for the paradox. The pretraining sweep's fixed token budget does change the number of sequences per optimization step across context windows, which is a real confound threatening the causal interpretation of the inverted-U curve, but that is an experimental-design risk, not a circularity in which an output equals an input by construction. Proposition 4.2 is the only near-restatement of the hypothesis, and the paper candidly calls it an achievability statement, so it does not force the empirical conclusions; the empirical claims therefore stand on their own measurements.
Assumptions & free parameters
assumptions (4)
- domain assumption Nested-input assumption: for k < m, X(k) = T_{k,m}(X(m)) almost surely for the packed token stream.
- domain assumption Latent task variable tau with distribution P_T, and weight channel W ~ Pi(·|tau).
- domain assumption Average training gradient norm G_k is a proxy for solution/function complexity.
- domain assumption FFN gradients correspond to parametric internalization and SA gradients to contextualization.
invented entities (3)
-
Information Abundance Paradox
independent evidence
-
context addiction
independent evidence
-
parametric information frontier I_k(rho)
Cite this review
Pith. "Pith review of Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge." pith.science (2026). https://pith.science/paper/65UIMDWQ
@misc{pith2026260812218,
author = {Pith},
title = {Pith review of: Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge},
year = {2026},
howpublished = {\url{https://pith.science/paper/65UIMDWQ}},
note = {Machine review of arXiv:2608.12218}
}
read the original abstract
Large language models are increasingly trained and deployed with long contexts that span documents, code repositories, and interaction histories. This scaling reflects the implicit assumption that training on longer contexts will only help the model by exposing it to richer evidence. We challenge this view by studying how the context window shapes a model's mode of learning, shifting it between parametric internalization and contextualization. We propose the Information Abundance Paradox, which hypothesizes that abundant relevant information in the training context can reduce the incentive to encode that information parametrically, thereby increasing reliance on context. In pretraining with long documents, increasing the context window improves language modeling, natural language understanding, and closed-book MCQA only up to an intermediate optimum, after which performance consistently declines. In supervised fine-tuning, more task-relevant train-time context improves performance with supporting context, but reduces robustness when context is absent or misleading at test time. Our analysis suggests that this behavior arises when longer context provides a lower complexity solution. Mechanistically, training with informative context shifts gradient pressure from feed-forward networks, often linked to parametric knowledge, toward attention modules, and causal interventions show that this shift increases reliance on context during inference. Overall, these findings support the Information Abundance Paradox and suggest that scaling toward near-infinite context is not simply a matter of supplying more data, even when high-quality long-context data is abundant.
Figures
Figures from the paper (12 more)
Reference graph
Works this paper leans on
-
[1]
Unary bitwise operations.Inputs are 16-bit binary strings. Each task applies a unary bitwise operation to the input string, such as NOT, which maps each bit to its complement.5 The valid output tokens are0and1. We train each model for 25 epochs
-
[2]
Each task applies a deterministic string transformation, such as REVERSE
String transformations.Inputs are 8-letter strings over a fixed alphabet. Each task applies a deterministic string transformation, such as REVERSE. 6 The output is another string over the same alphabet. We train each model for 25 epochs
-
[3]
Each task applies a digit-wise arithmetic operation modulo 10
Digit-wise mod10 arithmetic.Inputs are 5-digit strings. Each task applies a digit-wise arithmetic operation modulo 10. For example, a task may add a fixed digit-wise offset to each input digit, with all arithmetic performed modulo 10. We train each model for 10 epochs
-
[4]
Each task shifts every digit by a global offset modulo
Caesar cipher.Inputs are 5-digit strings. Each task shifts every digit by a global offset modulo
-
[7]
URLhttps://arxiv.org/abs/2206.03126. 15 Team Olmo, :, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morrison, Jake Poznan- ski, Kyle Lo, Luca Soldaini, Matt Jordan, Mayee Chen, Michael Noukhovitch, Nathan Lambert, Pete Walsh, Pradeep Dasigi, Robert Berry,...
arXiv 2026
-
[8]
Ian Tenney, Dipanjan Das, and Ellie Pavlick
URLhttps://arxiv.org/abs/2403.05530. Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline, 2019. URL https://arxiv.org/abs/1905.05950. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris...
arXiv 2019
-
[9]
URLhttps://arxiv.org/abs/2307.09288. 17 Chandra Shekhara Kaushik Valmeekam, Krishna Narayanan, Dileep Kalathil, Jean-Francois Cham- berland, and Srinivas Shakkottai. Llmzip: Lossless text compression using large language models,
-
[10]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N
URLhttps://arxiv.org/abs/2306.04050. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https://arxiv.org/abs/ 1706.03762. Elena V oita and Ivan Titov. Information-theoretic probing with minimum description length, 2020. URLhttps://arxiv.org/abs...
arXiv 2023
Show all 21 references
-
[11]
URLhttps://arxiv.org/abs/2203.08913. Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti ...
2023 arXiv
-
[12]
URLhttps://arxiv.org/abs/1810.12885. Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun.∞bench: Extending long context evaluation beyond 100k tokens, 2024b. URLhttps://arxiv.org/abs/24...
-
[13]
Shuyan Zhou, Frank F
URLhttps://arxiv.org/abs/2406.00605. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents, 2024. URL ht...
2024 arXiv
-
[18]
We train each model for 10 epochs
Importantly, the same offset is applied to all positions. We train each model for 10 epochs. 0.3M 1.5M 7.5M Architecture Layers 4 6 8 Hidden size 64 128 256 Attn. heads (Q) 2 4 8 Attn. heads (KV) 1 2 4 Intermediate size 256 512 1024 Optimization Optimizer AdamW Peak LR1.0×10 −...
-
[19]
This condition measures performance when the model can rely on useful in-context evidence
Supporting context.The context contains demonstrations generated by the correct task rule. This condition measures performance when the model can rely on useful in-context evidence
-
[20]
This condition measures whether the model follows the supplied context even when it conflicts with the parametrically correct rule
Conflicting context.The context contains demonstrations generated by a consistent but incorrect rule from the same task family. This condition measures whether the model follows the supplied context even when it conflicts with the parametrically correct rule. 5The full operati...
2016
-
[2018]
learning
URLhttps://arxiv.org/abs/1803.05457. Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. Multi-head attention: Collaborate instead of concatenate, 2021. URLhttps://arxiv.org/abs/2006.16362. Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual ...
2021 arXiv
-
[2020]
11 Amanda Bertsch, Maor Ivgi, Emily Xiao, Uri Alon, Jonathan Berant, Matthew R
URLhttps://arxiv.org/abs/2004.05150. 11 Amanda Bertsch, Maor Ivgi, Emily Xiao, Uri Alon, Jonathan Berant, Matthew R. Gormley, and Graham Neubig. In-context learning with long-context models: An in-depth exploration. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors,Proceedin...
2004 arXiv
-
[2022]
Jianghao Chen, Junhong Wu, Yangyifan Xu, and Jiajun Zhang
URLhttps://arxiv.org/abs/2210.05675. Jianghao Chen, Junhong Wu, Yangyifan Xu, and Jiajun Zhang. Ladm: Long-context training data selection with attention-based dependency measurement for llms, 2025. URL https://arxiv. org/abs/2503.02502. Shouyuan Chen, Sherman Wong, Liangjian ...
2025 arXiv
-
[2023]
Chenxin An, Jun Zhang, Ming Zhong, Lei Li, Shansan Gong, Yao Luo, Jingjing Xu, and Lingpeng Kong
URLhttps://arxiv.org/abs/2305.13245. Chenxin An, Jun Zhang, Ming Zhong, Lei Li, Shansan Gong, Yao Luo, Jingjing Xu, and Lingpeng Kong. Why does the effective context length of llms fall short?, 2024. URL https://arxiv. org/abs/2410.18745. Suraj Anand, Michael A. Lepori, Jack M...
2024 arXiv
-
[2024]
doi: 10.18653/v1/2024.acl-long.172
Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.172. URL https://aclanthology.org/2024.acl-long.172/. David G. T. Barrett and Benoit Dherin. Implicit gradient regularization, 2022. URL https: //arxiv.org/abs/2009.11162. Iz Beltagy, Matthew E. Peters, ...
2024 arXiv
-
[2025]
ISBN 979-8-89176-189-6
Association for Computational Linguistics. ISBN 979-8-89176-189-6. doi: 10.18653/v1/ 2025.naacl-long.605. URLhttps://aclanthology.org/2025.naacl-long.605/. Owen Bianchi, Mathew J. Koretsky, Maya Willey, Chelsea X. Alvarado, Tanay Nayak, Adi Asija, Nicole Kuznetsov, Mike A. Nal...
2025
-
[8192]
furthermore
This trend is consistent with the hypothesis that shorter evaluation tasks saturate at shorter training windows, while longer language modeling contexts benefit from longer train-time context before the long-context degradation appears. The downstream results in Table 5 follow...
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.