REVIEW 3 major objections 5 minor 7 references
Language models store frequent verb+up phrases as whole units, not word-by-word.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
Frequency and predictability shift the internal representation of “up” inside V+up phrases away from standalone “up” in text and audio models, a pattern interpreted as holistic storage.
T0 review reviewed 2026-08-04 challenge →
load-bearing objection Robust frequency-driven probe divergence across LM scales and Whisper, but the 'holistic storage' interpretation is undercut by a missing contextual control; predictability effects are inconsistent with the abstract's blanket claim. the 3 major comments →
The Holistic Storage of Verb+Up Phrases in Text-based and Audio-based Language Models
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The paper's central claim is that all tested models—three transformer language models trained from scratch on a 150-million-token corpus, a 7-billion-parameter text model, and a small encoder-decoder speech recognizer—show a frequency- and predictability-driven divergence: the internal representation of 'up' inside a verb+up phrase becomes progressively less like the representation of standalone 'up' as the phrase's frequency and predictability rise. The authors interpret this divergence as holistic storage of the phrase, paralleling human listening studies in which participants detect 'up' more slowly in high-frequency phrases. They also find the divergence deepens over layers, appears earl
What carries the argument
The central instrument is a per-layer logistic probe trained to distinguish hidden-layer embeddings of standalone 'up' from other tokens in the same sentence. Applied to held-out verb+up phrases, the probe's logit measures how similar that phrase's 'up' representation is to standalone 'up'; a lower logit is taken as evidence that the phrase has been stored holistically. The paper analyzes these logits with Bayesian mixed-effects regressions and generalized additive models over frequency, predictability, and layer index.
Load-bearing premise
The load-bearing premise is that a lower probe score for 'up' inside a frequent phrase means the whole phrase is stored, rather than that the word's meaning or sound is simply adjusted by context; if contextual modulation can produce the same drop, the holistic-storage reading loses its footing.
What would settle it
If the same frequency-driven logit drop appears for a control word that cannot form a stored phrase—for example, 'up' in a frequent bigram that is not a verb+particle construction, or for a particle with identical frequency but no phrasal predictability—then the effect would be contextual rather than holistic. A more direct test would ask whether the divergence predicts behavior: if models are no faster or more accurate at using high-frequency verb+up phrases than compositional controls, the representational difference is not storage in any processing sense.
If this is right
- Holistic storage is a byproduct of exposure to distributional patterns; no separate memory or rule mechanism is needed to explain it.
- Human-scale training data, roughly half the tokens a college-aged person has experienced, is enough for frequency-driven holistic storage to emerge.
- Sensitivity to predictability, rather than raw frequency alone, strengthens with model size, suggesting that encoding conditional statistics requires greater representational capacity.
- The effect generalizes across modalities: text-based models and the encoder and decoder of a speech recognizer show the same gradient, so holistic storage is not tied to written tokenization.
- Storage is a graded, layer-dependent phenomenon rather than an all-or-nothing property: divergence from standalone 'up' accumulates across layers and appears earlier in larger models.
Where Pith is reading between the lines
- The probe-logit gradient is never validated against behavior, so a natural test would pair it with generation or recognition tasks: if high-frequency verb+up phrases are also produced faster or recognized more accurately, the representational divergence would be tied to a real processing advantage.
- Because larger models likely also saw more data, a controlled experiment training one architecture at multiple scales on identical token budgets would isolate whether exposure or capacity drives the predictability effect.
- The same protocol could be applied to languages with different particle systems, such as German separable prefixes or Chinese resultative compounds, to test whether frequency-driven holistic storage is a general property of usage rather than an artifact of English orthography.
- The layer-by-layer pattern suggests a mechanistic prediction: interventions on later layers should selectively damage high-frequency phrase representations while leaving low-frequency compositional phrases intact, giving a causal handle on where holistic units live.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates whether transformer language models and an ASR model store V+up phrasal verbs holistically. The authors train three BabyLM-size OPT models, and also use OLMo-3 7B and Whisper-small. They train logistic regression probes on embeddings of standalone 'up', then apply the probes to embeddings of 'up' in V+up phrases. Lower probe logits for high-frequency and high-predictability phrases are interpreted as evidence that these phrases have holistic representations that diverge from the compositional 'up' representation. Experiment 1 uses a probe trained only on standalone 'up'; Experiment 2 broadens positive training examples to 'up' as a subword; Experiment 3 applies the standalone-up probe to Whisper's encoder and decoder. The main reported result is negative frequency and predictability effects on probe logits, with by-layer analyses suggesting earlier emergence in larger models.
Significance. If the central inference were valid, this would be a useful extension of usage-based accounts of holistic storage to neural language models, including a cross-modal ASR result and models trained on human-comparable data. The paper has concrete strengths: the three BabyLM models are released, corpus counts and predictor definitions are explicit, Bayesian mixed-effects models provide credible intervals, and the item-level statistics are reported in appendices. However, the core claim depends on an unvalidated interpretive step: that a decrease in probe logit specifically indexes holistic storage rather than a generic contextual-predictability effect. Experiment 2 also contains predictability coefficients for the BabyLM models that are positive or non-significant, which is inconsistent with the abstract's across-the-board claim. These issues are load-bearing for the paper's main conclusion, so the manuscript needs substantive revision.
major comments (3)
- [§3.1.2 and Conclusion] The inference from lower probe logits to 'holistic storage' is not uniquely identified. A transformer's representation of a token is a function of its context, and when 'up' is highly predictable given the preceding verb (Eqn. 2), the model may encode it with lower surprisal even if the phrase is not stored as a unit. The random intercept for verb_up in Eqn. (3) does not control for this, because log-predictability is exactly a property of the verb–up pair. The paper does not offer a control condition in which predictability is manipulated outside V+up contexts, nor any behavioral or generation-based validation that lower logits correspond to holistic phrase storage rather than contextual modulation. Without such a control, the central claim is underdetermined.
- [Experiment 2, Table 8] The abstract and conclusion state that all models show holistic storage driven by frequency and predictability, but the Experiment 2 final-layer predictability coefficients for the BabyLM models are absent or positive: BabyLM 125M has -0.09 (95% CI [-0.25, 0.06]), BabyLM 350M has +0.25 [0.11, 0.39], and BabyLM 1.3B has +0.14 [0.04, 0.24]. Positive coefficients mean high-predictability phrases yield higher 'up-like' logits, the opposite of the prediction. The text in §4.2 acknowledges 'absent or positive effects,' but the abstract and conclusion do not qualify the claim. This is a load-bearing inconsistency for the 'all models' claim and for the claim that predictability effects strengthen with scale.
- [§3.1.2, Figures 2/4/6] The by-layer GAM comparisons assume the probe logit is comparable across layers, but the logistic probes are fit separately at each layer with no layer-wise calibration. If the scale or class separation of logits differs across layers, the tensor-product smooth over predictor and layer can produce spurious 'early versus late emergence' patterns. The conclusion that larger models show effects in earlier layers depends on this assumption. The paper should report per-layer classifier calibration or use a layer-invariant measure (e.g., rank-based or standardized scores) before interpreting layer trajectories.
minor comments (5)
- [§5.1.1 vs Table 3] The text reports that 1,426 V+up types were retained for Whisper, but Table 3 lists 1,724 items for both encoder and decoder. The text also gives a median frequency of 243,412, while Table 3 reports 169,679. These numerical discrepancies should be reconciled.
- [§3.1.2, Eqn. (4)] Typo: 'with a a random intercept' should read 'with a random intercept.'
- [§3.1.1] The superscript '4' after 'preposition' appears to be a footnote marker, but no corresponding footnote text is present.
- [§4.1.1] Spacing issue: 'upembedded' should be 'up embedded.'
- [§7 Limitations] The limitations section mentions construction, language, and checkpoint coverage, but should also acknowledge the probe-validity concern raised above, and note that no direct validation against behavioral or mechanistic criteria was performed.
Circularity Check
No significant circularity: the probe measurements are inferred from independent labels and corpus counts, not from the conclusion.
full rationale
The paper's derivation chain is self-contained. Frequency and predictability are computed directly from corpus counts (Eqs. 1-2), the probe logit is produced by logistic classifiers trained on independent labels (standalone 'up' vs. other tokens; Sec. 3.1.1), and the statistical models (Eqs. 3-4) test whether that measured outcome covaries with the corpus-derived predictors. No parameter is fitted to the target conclusion ('holistic storage'), and no equation reduces by construction to the claim that high-frequency/high-predictability V+up phrases are stored holistically. The self-citations (Houghton 2025b; Pluth et al. 2026) are used to motivate and interpret the results, but the load-bearing empirical steps rest on held-out probe decisions and independent corpus statistics, not on those citations; the cited human results are externally collected behavioral evidence. The main substantive concern, that lower logits might reflect contextual predictability or layer-wise calibration rather than holistic storage, is a validity/confound issue rather than a circularity. Under the stated criteria, no circular step is exhibited.
Axiom & Free-Parameter Ledger
free parameters (2)
- Logistic regression probe weights (per layer, per model) =
d_v + 1 coefficients per layer (e.g., 769 for OPT-125M final layer)
- Test-set inclusion cutoffs =
Minimum 20 V+up occurrences for LLM test items, 5 for Whisper; maximum 20 sampled sentences per type
axioms (5)
- domain assumption Probe logit is a valid, layer-comparable measure of holistic storage of V+up phrases.
- domain assumption Dolma v1.7 corpus counts approximate the true training distributions of OLMo-3 7B and Whisper-small.
- domain assumption Classifier decision boundaries are comparable across layers and across models.
- domain assumption The BPE tokenizer's 'up' token is representationally stable across V+up contexts.
- standard math Mixed-effects and GAM model assumptions hold for probe logits.
Cite this review
Pith. "Pith review of The Holistic Storage of Verb+Up Phrases in Text-based and Audio-based Language Models." pith.science (2026). https://pith.science/paper/RO443JKX
@misc{pith2026260613993,
author = {Pith},
title = {Pith review of: The Holistic Storage of Verb+Up Phrases in Text-based and Audio-based Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/RO443JKX}},
note = {Machine review of arXiv:2606.13993}
}
read the original abstract
A crucial aspect of linguistic capability is the ability to trade off between stored representations and abstract knowledge: one must retrieve learned representations, but also generate novel ones by applying productive rules. While recent work has examined abstract knowledge in language models, holistic storage of multi-word units has received far less attention. We probe internal representations in text-based LLMs and an ASR model, testing whether V+up phrasal verbs develop distinct representations as a function of frequency and predictability. All models show evidence of holistic storage driven by frequency and predictability, further supporting usage-based theories of language.
Figures
Reference graph
Works this paper leans on
-
[3]
InProceedings of the 2024 conference on empirical methods in natu- ral language processing, pages 913–929
Language models learn rare phenomena from less rare phenom- ena: The case of the missing aanns. InProceedings of the 2024 conference on empirical methods in natu- ral language processing, pages 913–929. 9 Emily Morgan and Roger Levy
2024
-
[6]
Both direct and indirect evi- dence contribute to dative alternation preferences in language models.arXiv preprint arXiv:2503.20850. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, and 1 others
-
[1986]
Simon N Wood
Frequency and the lexical storage of regularly in- flected forms.Memory & Cognition, 14(1):17–26. Simon N Wood. 2017.Generalized additive models: an introduction with R. chapman and hall/CRC. Qing Yao, Kanishka Misra, Leonie Weissweiler, and Kyle Mahowald
2017
-
[2022]
Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068. 10 A BabyLM Model Training Models were trained from random initialization for 20 epochs using the AdamW optimizer with fused weight updates and bfloat16 mixed precision. The learning rate was set to 3×10 −4 for the 125M model and 1×10 −4 for the 350M and 1.3B models, each p...
-
[2024]
Infini-gram: Scaling unbounded n-gram language models to a tril- lion tokens.arXiv preprint arXiv:2401.17377. R. Thomas McCoy, Paul Smolensky, Tal Linzen, Jian- feng Gao, and Asli Celikyilmaz
-
[2025]
Babylm turns 3: Call for pa- pers for the 2025 babylm workshop.arXiv preprint arXiv:2502.10645. Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, Mingjie Jin, Sanjeev Khudanpur, Shinji Watanabe, Shuaijiang Zhao, Wei Zou, Xiangang Li, Xuchen Yao, Yongqing Wang, Yu- jun Wang, and 2 others
Pith/arXiv arXiv 2025
-
[2026]
Mechanistic interpretability of asr models using sparse autoencoders.Preprint, arXiv:2605.12225. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu
This paper was first reviewed by deepseek-v4-flash on August 4, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.