Pith. sign in

REVIEW 4 major objections 6 minor 72 references

Modes of Sequence Models and Learning Coefficients

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Local learning coefficient estimates from SGLD characterise a mode-truncated 'effective' data distribution rather than the full transformer sequence distribution.

desk verdict A serious, honestly-written conditional theorem linking mode truncation to LLC insensitivity, with a practical claim that outruns what the assumptions currently support. read the letter →

arxiv 2504.18048 v1 pith:VW7LQMCL submitted 2025-04-25 cs.LG

Zhongtian Chen , Daniel Murfet This is my paper · ORCID
classification cs.LG MSC 68T0762B1015A69
keywords sequencemodelslocallearningcoefficientsingularvaluedecompositioneffectivedistributionstochasticgradientLangevindynamicsmodetruncationtheorytransformers
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper develops a geometric account of what local learning coefficient (LLC) estimates measure when they are computed on transformer sequence models. Singular value decomposition of the conditional sequence-probability tensor produces principal modes, and truncating the small-amplitude modes yields an effective true distribution. The main theorem states that, when a model is insensitive to the truncated modes in both its gradients and its log-probabilities, SGLD-based LLC estimates cannot distinguish the original loss from the loss of the truncated distribution. The paper concludes that in practice LLC values describe the geometry of the effective distribution, with the inverse temperature of the sampler acting as a resolution dial.

What carries the argument

The load-bearing object is the mode decomposition of the conditional-probability map $C_{k,l}: V_k\to \mathbb{R}^{\Sigma^l}$, defined by $C_{k,l}(x)=\sum_y q(y|x)y$, where $V_k$ is the $q$-weighted Hilbert space on length-$k$ contexts. SVD produces right vectors $v_\alpha$, left vectors $u_\alpha$, and singular values $s_\alpha$; the index $\alpha$ is a mode with propensity $s_\alpha^2$, and truncating to modes $\alpha\le\chi$ followed by KL projection gives the effective true distribution $q^{(\chi)}$. The proof couples two SGLD chains driven by the empirical gradient $\nabla L_m$ and the truncated gradient $\nabla L_m^{(\chi)}$, and uses the two insensitivity conditions together with Lipschitz continuity to bound both the chain distance and the difference of the resulting LLC estimates.

What would settle it

Measure the spectral norm of the minibatch Hessian along an SGLD chain in a large transformer: if the resulting Lipschitz constant $M$ is not below $\gamma/(n\beta)$ (about 30 under the paper's Pythia-style settings), the central hypothesis $Mn\beta<\gamma$ fails and the theorem's conclusion does not apply.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the geometry probed by SGLD-based LLC estimation is the geometry of an effective distribution obtained by truncating the small modes of the data. Starting from the conditional-probability map $C_{k,l}(x)=\sum_{y\in\Sigma^l} q(y|x)y$, the paper performs an SVD in a natural Hilbert space; the singular components are the modes, whose propensities are the squared singular values, and retaining modes up to a cutoff $\chi$ defines an effective true distribution $q^{(\chi)}$. Theorem 7.4 then shows that if the model is gradient-insensitive and log-probability-insensitive to $q-q^{(\chi)}$ on a region containing the SGLD chain, and if the relevant losses are Lipschitz with $Mn\beta<\gamma$, the SGLD-based LLC estimates computed from $L$ and from $L^{(\chi)}$ cannot be distinguished within a bound controlled by the insensitivity constants. The paper's conclusion is that LLC estimates calculated in practice characterise the effective, mode-truncated distribution rather than the full sequence distribution.

Load-bearing premise

Everything hinges on the assumption that, in the region where SGLD runs, a transformer's gradients and log probabilities barely change when the small modes are removed, and that the loss is smooth enough that its gradient's Lipschitz constant $M$ satisfies $Mn\beta<\gamma$; with the paper's Pythia-style hyperparameters this requires $M<30$, which the paper does not measure for large transformers.

Editorial extensions

If this is right

  • LLC estimates on transformer language models should be read as measuring the geometry of an effective, mode-truncated distribution rather than the full sequence distribution.
  • Stable LLC estimates at parameters that are not local minimisers of the population loss become less puzzling, because such parameters can be stationary points of the effective potential $L^{(\chi)}$.
  • The inverse temperature $\beta$ acts as a resolution dial: raising $\beta$ lets the estimate count contributions from rarer modes, while lowering $\beta$ restricts the estimate to the dominant modes.
  • Reported LLC values carry meaning only relative to the SGLD hyperparameters, since the bound depends on $n\beta$, $\gamma$, the step sizes, and the Lipschitz constants of the loss.
  • The mode decomposition offers a principled coarse-graining of sequence data, and the leading empirical modes correspond to interpretable patterns such as punctuation and common subword collocations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the theorem survives contact with real transformers, systematic $\beta$-scans of LLC estimates become a way to map the mode spectrum of a model's training distribution, since lower $\beta$ sees only dominant modes and higher $\beta$ resolves rarer ones.
  • The result implies that comparing LLC values across models or training runs is only meaningful when SGLD hyperparameters are matched, because the effective mode cutoff that an estimate sees depends on those hyperparameters.
  • The paper's connection to tensor renormalisation group suggests a testable extension: hierarchical or entanglement-guided truncations of the fundamental tensor could yield effective potentials that make LLC estimation both more accurate and more interpretable.
  • A direct experimental check on synthetic sequence distributions with known mode spectra could measure how much LLC estimates change when small modes are removed at fixed $\beta$; large changes would indicate the insensitivity constants are bigger than the theorem requires.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper develops a tensor-decomposition framework for conditional sequence distributions, defines a Hilbert-space basis of 'modes' via SVD of the conditional-probability operator C_{k,l}, and introduces a coarse-grained 'effective true distribution' q^(χ) by truncating modes. It then introduces two formal conditions—-gradient-insensitivity and log-probability-insensitivity (Definitions 6.3 and 6.4)—-and proves, in Theorems 7.3 and 7.4, that under these conditions plus Lipschitz and hyperparameter assumptions, SGLD-based LLC estimates of the true loss L and of the truncated loss L^(χ) are close. The authors conclude that LLC estimates computed in practice characterize the effective, mode-truncated distribution rather than the full sequence distribution, and that the inverse temperature β acts as a resolution dial. The paper also reports empirical SVD modes computed on the Pile with a Pythia tokenizer.

Significance. If the main theorem could be turned into a quantitative statement about real transformers, the paper would make a useful conceptual contribution to singular learning theory and to the interpretation of LLC measurements in language models: it would explain why LLC estimates can be stable at parameters that are not strict minimizers of the population loss, and it would connect mode truncation to tempered posterior sampling. The tensor-algebraic setup is clean and the proofs of Theorems 7.3 and 7.4 are internally coherent given their hypotheses. The paper is also honest in Remark 7.5 about the strength of the Lipschitz assumption. The central limitation is that the insensitivity constants A and B and the Lipschitz constant M are never measured or bounded for transformers, and the theorem's bound is not shown to be smaller than the resolution of real LLC estimates. The practical conclusion in the abstract and conclusion therefore remains a conditional statement rather than an established fact.

major comments (4)
  1. [§7, Theorem 7.4 and Remark 7.5] The quantitative gap between the theorem and the paper's central claim is load-bearing and unresolved. The bound in Eq. (75) with ξ=κ=0 is (ϵmax/ϵmin)·nβQA/(γ/(nβ)−M) + 2nβB. Under the paper's own hyperparameters (Remark 7.5: nβ=10, γ=300, ϵmax=ϵmin) and the plausible estimate Q≈10, the first term is 100A/(30−M), which is 20A at M=25 and 100A at M=29, and the second term is 20B. For the theorem to support the statement that 'LLC calculated in practice characterises the effective rather than the true distribution,' the entire bound must be small compared with the resolution of the LLC estimator. The manuscript provides no estimates of A or B, no transformer Hessian spectral bound for M, and no comparison of the bound with estimator variance. Remark 7.5 concedes that the M<30 condition would not be realistic if the Lipschitz constant were orders of magnitude larger and that no Hessian estimates for large transformers were found. This is a fixable but essential gap: the abstract and conclusion assert a practical fact that the theorem, as quantified, does not deliver.
  2. [§6, Definitions 6.3 and 6.4] The two insensitivity conditions are effectively a formal restatement of the conclusion they are used to prove. Gradient-insensitivity for constant A requires that the pairing of Ψ_{w∗} with q−q^(χ) have norm less than A over W′, and log-probability-insensitivity similarly bounds ⟨Φ(w∗), q−q^(χ)⟩. For q^(χ) obtained by mode truncation, these conditions say precisely that the model's gradients and log-probabilities do not see the truncated modes. The proof then propagates this insensitivity through SGLD. Since A and B are never measured, the theorem is conditional on the very phenomenon it is meant to establish. The motivation in §6.1 appeals to finite capacity but gives no capacity-dependent bound, and the plausibility argument in §6.2 relies on the auxiliary assumptions that all large modes are absolute bigrams, that q(στ)≤2^{−(k+l)H}, and that ∥P_{≤χ}(q)−q^(χ)∥ is small; these are stated as reasonable rather than verified. A direct measurement of A, B, and the gradient/log-probability differences for a small transformer would be needed to make the bridge from the conditional theorem to the practical claim.
  3. [§4.5 and Theorem 7.4] The threshold χ is not given an operational characterization. The effective distribution q^(χ) is defined as the KL projection of q onto the submanifold P_{≤χ}, which is not the same as the orthogonal projection P_{≤χ}(q); Remark 4.18 explicitly notes that the naive truncation need not be a probability distribution. The main theorem is stated for an arbitrary q^(χ) satisfying the insensitivity conditions, but the paper never proves or estimates for which mode cutoffs χ those conditions hold with small A and B. Consequently, the 'data-dependent threshold' mentioned in the abstract remains unspecified, and the theorem cannot be used to predict, for a given model and dataset, which modes will be invisible to LLC estimation.
  4. [§7, Theorem 7.4 and Remark 7.5] The asymptotic regime of the theorem is not tied to a scaling of β with n. The theorem says 'there exists large enough n' while imposing Mnβ∈(γ−2/ϵmax,γ). If β is taken as the standard β0/logn from Watanabe (2013), then nβ grows with n and the condition eventually fails because γ/(nβ)→0. If instead nβ is held fixed at the practical value 10 used in Remark 7.5, then β shrinks as 1/n and it is unclear whether the Glivenko-Cantelli and local-minimum hypotheses of LLC estimation are satisfied in the asymptotic regime. The proof needs a precise statement of how n, β, m, and the insensitivity constants scale; otherwise the bound in Eq. (73) is not a well-defined asymptotic guarantee.
minor comments (6)
  1. [§4.5, Definition 4.16] The word 'minises' in Definition 4.16 should be 'minimizes'.
  2. [§7, paragraph before Eq. (61)] The text refers to 'Theorem 4.16' when citing the construction of q^(χ); this should be 'Definition 4.16'.
  3. [§7, Eq. (73)] The probability lower bound 1−12T(T+1)δ−κ can be negative or meaningless for large T unless T is restricted, for example Tδ<1; the theorem should state this restriction explicitly.
  4. [§6, Definitions 6.3 and 6.4] The notation would be clearer if the dependence of A and B on W′, q, and q′ were made explicit, since the constants are used later without restating their domain of validity.
  5. [§5.2 and Appendix B] The empirical modes are computed from smoothed, filtered conditional probability matrices, not from the exact q; this is described in Appendix B, but a sentence in Section 5.2 reminding the reader that these are approximate empirical modes would improve clarity.
  6. [§3.2, Example 3.3] The tokenized example is not typeset as a quoted string and is hard to read; using a monospaced or quoted format would improve presentation.

Circularity Check

1 steps flagged · score 6.0 of 10

Theorem 7.4 mainly restates its insensitivity premises as the LLC insensitivity conclusion; the link from mode amplitudes to constants A and B is asserted, not derived.

  1. self definitional [Definitions 6.3–6.4 and Lemma 7.1 / Theorem 7.4 (Section 7, Eqs. (52)–(73))]
    "Definition 6.3: 'The model is gradient-insensitive for the constant A at w∗ if ‖⟨Ψ_{w∗}(−), q−q′⟩_H‖_2 < A.' Lemma 7.1 proof: 'By the hypothesis of gradient-insensitivity we have that ‖∇_w L(w)−∇_w L^(χ)(w)‖_2 = ‖Σ_{x∈Σ^k,y∈Σ}(q^(χ)(x,y)−q(x,y))∇_w log p(y|x,w)‖_2 is less than A.' Definition 6.4 similarly bounds |⟨Φ(w∗), q−q′⟩_H| < B."

    The two insensitivity definitions are literally bounds on the differences that Theorem 7.4 concludes LLC cannot detect: Eq. (52) identifies gradient-insensitivity with ‖∇_w L(w)−∇_w L^(χ)(w)‖ < A, and Eq. (53) identifies log-probability-insensitivity with |L(w)−L^(χ)(w)| < B. The theorem then propagates these assumed small quantities through Lipschitz constants and a coupling argument to bound the difference of the LLC estimators. So the statement 'LLC cannot distinguish L from L^(χ)' is the same smallness that was assumed of the model's loss/gradients; the mode cutoff χ is never derived from the singular-value amplitudes, and A,B are never estimated.

full rationale

The tensor/SVD formalism and the SGLD coupling proof are mathematically self-contained, and there is no fitted input renamed as a prediction and no load-bearing self-citation: the citations to the authors' prior LLC work supply context, while the theorem is proved in-text. The circularity concern is narrower but central: the insensitivity hypotheses are defined as smallness of exactly the loss and gradient differences whose invisibility to LLC is then announced as the paper's main finding. Theorem 7.4 is a conditional continuity statement rather than a derivation of mode insensitivity from the mode spectrum. Remark 7.5's admission that no transformer Hessian estimates are available, together with the unmeasured constants A and B, leaves the practical conclusion numerically unsupported; that gap is an evidential weakness rather than a distinct circular step.

Assumptions & free parameters 4 free parameters · 10 assumptions · 1 invented entities

The central claim rests on the existence and smallness of insensitivity constants A and B, a Lipschitz condition on the empirical loss that may fail for large transformers, and an entropy-rate bound used in a plausibility argument. The effective distribution is a new construction without independent empirical support. Standard results from SVD and empirical process theory are the only fully external inputs.

free parameters (4)
  • Mode cutoff chi
    The effective distribution q^(chi) and all theorems depend on an arbitrarily chosen cutoff; the paper does not provide a data-dependent rule for choosing it, despite the abstract's wording.
  • Gradient-insensitivity constant A
    Existence and smallness are assumed in Definition 6.3; no estimate is given for any model.
  • Log-probability-insensitivity constant B
    Existence and smallness are assumed in Definition 6.4; no estimate is given for any model.
  • Entropy rate H = assumed within [1,10]
    Used in the plausibility argument of Section 6.2 to argue that log-probability insensitivity is a mild constraint; not measured for the Pile/Pythia tokenizer.
assumptions (10)
  • domain assumption q(x) > 0 for all sequences, and the language condition ties all sequence lengths into one consistent distribution.
    Used to define the Hilbert spaces and C_{k,l}; see Definition 3.2 and Section 4.1.
  • domain assumption The parametric model is defined on a compact W, is continuously differentiable, and is bounded away from zero.
    Stated at the start of Sections 6 and 7; needed for Glivenko-Cantelli and Lipschitz arguments.
  • domain assumption Gradient-insensitivity holds with constant A on W'.
    Definition 6.3; main hypothesis of Theorems 7.3 and 7.4.
  • domain assumption Log-probability-insensitivity holds with constant B on W'.
    Definition 6.4; main hypothesis of Theorem 7.4.
  • domain assumption grad L_m and L_n are Lipschitz with constants M and Q on W', with M n beta in (gamma - 2/epsilon_max, gamma).
    Required for the SGLD coupling bound; see Theorem 7.3 and Remark 7.5.
  • domain assumption SGLD chains remain inside W' and the two chains use the same injected noise.
    Assumed before Theorem 7.3.
  • ad hoc to paper In the plausibility argument, modes are absolute bigrams and q(sigma tau) <= 2^{-(k+l)H} for an entropy rate H.
    Section 6.2 uses this to motivate log-probability-insensitivity; it is not a general property of real text distributions.
  • ad hoc to paper q^(chi) is close to the orthogonal projection P_{<=chi}(q) for large chi.
    Section 6.2, equations (57)-(58); this is 'reasonable to assume' but not proved.
  • standard math Glivenko-Cantelli and uniform convergence results from Vaart (1998, Section 19.2).
    Used in Lemma 7.1 and Theorem 7.4 to justify convergence of empirical losses.
  • standard math SVD, spectral theorem, and Perron-Frobenius theorem.
    Used throughout Section 4 for the mode decomposition and in Remark 4.12.
invented entities (1)
  • Effective true distribution q^(chi)
    purpose: Serves as the coarse-grained distribution whose geometry LLC is claimed to measure.
    Defined in Definition 4.16 via KL projection onto a mode-truncated submanifold; there is no independent experimental evidence that real training dynamics converge to or sample this distribution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Modes of Sequence Models and Learning Coefficients." pith.science (2026). https://pith.science/paper/VW7LQMCL

@misc{pith2026250418048,
  author       = {Pith},
  title        = {Pith review of: Modes of Sequence Models and Learning Coefficients},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VW7LQMCL}},
  note         = {Machine review of arXiv:2504.18048}
}
read the original abstract

We develop a geometric account of sequence modelling that links patterns in the data to measurable properties of the loss landscape in transformer networks. First, we cast conditional sequence distributions into a Hilbert-space framework and apply tensor decompositions to identify their principal modes. Truncating the small-amplitude modes yields an effective data distribution that preserves dominant structure while discarding statistical detail. Second, we show theoretically that Local Learning Coefficient (LLC) estimates are insensitive to modes below a data-dependent threshold. Consequently, the LLC calculated in practice characterises the geometry of the effective rather than the true distribution. This insight clarifies why reliable LLC estimates can be obtained even when a network parameter is not a strict minimiser of the population loss, and it highlights how the inverse temperature in SGLD acts as a resolution dial on the landscape structure.

Figures

Figures reproduced from arXiv: 2504.18048 by the authors.

Figure 1
Figure 1. Empirical modes. We show an example of the x, y pair which are heavily loaded in the first empirical mode for k = 1, l = 1 in our experiments on the Pile. Shown are three text samples. Next to each X, Y token we show the index in the tokeniser [PITH_FULL_IMAGE:figures/full_fig_p016_1.png] view at source ↗
Figure 2
Figure 2. Empirical modes. We show an example of the x, y pair which are heavily loaded in the first empirical mode for k = 2, l = 1 in our experiments on the Pile. Shown are three text samples. Next to each X, Y token we show the index in the tokeniser. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_2.png] view at source ↗
Figure 3
Figure 3. Empirical modes. We show an example of the x, y pair which are heavily loaded in the first empirical mode for k = 2, l = 1 in our experiments on the Pile. Shown are three text samples. Next to each X, Y token sequence we show the indices in the tokeniser. Example 5.9. For k = 2, l = 1 component 16 has singular value 22.45 with left singular vector u = 22.44 ) + 0.57 ). + 0.28 ), + · · · (47) and the right singular v… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Empirical modes. We show an example of the x, y pair which are heavily loaded in the first empirical mode for k = 3, l = 3 in our experiments on the Pile. Shown are three text samples. Next to each X, Y token sequence we show the indices in the tokeniser. For large mat…
Figure 5
Figure 5. Figure 5: Empirical modes. We show an example of the x, y pair which are heavily loaded in the first empirical mode for k = 3, l = 3 in our experiments on the Pile. Shown are three text samples. Next to each X, Y token sequence we show the indices in the tokeniser. Natural candi…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

72 extracted references · 45 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  3. [3]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  4. [4]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    work page 2024

  5. [5]

    Information Geometry and Its Applications: Convex Function and Dually Flat Manifold, pp.\ 75--102

    Amari, S.-i. Information Geometry and Its Applications: Convex Function and Dually Flat Manifold, pp.\ 75--102. Springer Berlin Heidelberg, 2009

    work page 2009

  6. [6]

    Information Geometry and its Applications

    Amari, S.-i. Information Geometry and its Applications. Springer, 2016

    work page 2016

  7. [7]

    M., and Telgarsky, M

    Anandkumar, A., Ge, R., Hsu, D., Kakade, S. M., and Telgarsky, M. A method of moments for mixture models and hidden M arkov models. Conference on Learning Theory, pp.\ 33--1, 2012

    work page 2012

  8. [8]

    J., Kakade, S

    Anandkumar, A., Ge, R., Hsu, D. J., Kakade, S. M., Telgarsky, M., et al. Tensor decompositions for learning latent variable models. J. Mach. Learn. Res., 15 0 (1): 0 2773--2832, 2014

    work page 2014

Show all 72 references
  1. [9]

    Consideration on the learning efficiency of multiple-layered neural networks with linear units

    Aoyagi, M. Consideration on the learning efficiency of multiple-layered neural networks with linear units. Neural Networks, 172: 0 106132, 2024. ISSN 0893-6080

    2024

  2. [10]

    Studying S mall L anguage M odels with S usceptibilities, 2025

    Baker, G., Wang, G., Hoogland, J., and Murfet, D. Studying S mall L anguage M odels with S usceptibilities, 2025

    2025

  3. [11]

    S., Lepage, G., Svetitsky, B., and Wilson, K

    Batrouni, G., Katz, G., Kronfeld, A. S., Lepage, G., Svetitsky, B., and Wilson, K. Langevin simulations of lattice field theories. Physical Review D, 32 0 (10): 0 2736, 1985

    1985

  4. [12]

    A neural probabilistic language model

    Bengio, Y., Ducharme, R., Vincent, P., and Jauvin, C. A neural probabilistic language model. Journal of machine learning research, 3 0 (Feb): 0 1137--1155, 2003

    2003

  5. [13]

    G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M

    Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, ...

    2023

  6. [14]

    Gpt-neox-20b: An open-source autoregressive language model

    Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint arXiv:2204.06745, 2022

    2022 arXiv

  7. [15]

    E ckart- Y oung

    Carroll, J. D. and Chang, J.-J. Analysis of individual differences in multidimensional scaling via an N -way generalization of " E ckart- Y oung" decomposition. Psychometrika, 35 0 (3): 0 283--319, 1970

    1970

  8. [16]

    Dynamics of transient structure in in-context linear regression transformers, 2025

    Carroll, L., Hoogland, J., Farrugia-Roberts, M., and Murfet, D. Dynamics of transient structure in in-context linear regression transformers, 2025. URL https://arxiv.org/abs/2501.17745

    2025 arXiv

  9. [17]

    Dynamical versus Bayesian phase transitions in a toy model of superposition

    Chen, Z., Lau, E., Mendel, J., Wei, S., and Murfet, D. Dynamical versus Bayesian phase transitions in a toy model of superposition. Preprint arXiv:2310.06301 [cs.LG], 2023

    2023 arXiv

  10. [18]

    Cover, T. M. and Thomas, J. A. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). Wiley-Interscience, USA, 2006

    2006

  11. [19]

    and Weber, M

    Deuflhard, P. and Weber, M. Robust P erron cluster analysis in conformation dynamics. Linear algebra and its applications, 398: 0 161--184, 2005

    2005

  12. [20]

    Identification of almost invariant aggregates in reversible nearly uncoupled M arkov chains

    Deuflhard, P., Huisinga, W., Fischer, A., and Sch \"u tte, C. Identification of almost invariant aggregates in reversible nearly uncoupled M arkov chains. Linear Algebra and its Applications, 315 0 (1-3): 0 39--59, 2000

    2000

  13. [21]

    Monology/pile-uncopyrighted

    Devin Gulliver . Monology/pile-uncopyrighted. https://huggingface.co/datasets/monology/pile-uncopyrighted, March 2025

    2025

  14. [22]

    Efrati, E., Wang, Z., Kolan, A., and Kadanoff, L. P. Real-space renormalization in statistical mechanics. Reviews of Modern Physics, 86 0 (2): 0 647--667, 2014

    2014

  15. [23]

    and Vidal, G

    Evenbly, G. and Vidal, G. Tensor network renormalization. Physical review letters, 115 0 (18): 0 180405, 2015

    2015

  16. [24]

    HLAT : H igh-quality large language model pre-trained on AWS T rainium

    Fan, H., Zhou, H., Huang, G., Raman, P., Fu, X., Gupta, G., Ram, D., Wang, Y., and Huan, J. HLAT : H igh-quality large language model pre-trained on AWS T rainium. In 2024 IEEE International Conference on Big Data (BigData), pp.\ 2100--2109. IEEE, 2024

    2024

  17. [25]

    Firth, J. R. A synopsis of linguistic theory, 1930-1955. In Studies in Linguistic Analysis, pp.\ 1--32. Philological Society, 1957

    1930

  18. [26]

    An SVD approach to identifying meta-stable states of M arkov chains

    Fritzsche, D., Mehrmann, V., Szyld, D., and Virnik, E. An SVD approach to identifying meta-stable states of M arkov chains. Electron. Trans. Numer. Anal., 2007

    2007

  19. [27]

    The Pile: an 800GB dataset of diverse text for language modeling

    Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. The Pile: an 800GB dataset of diverse text for language modeling. Preprint arXiv:2101.00027 [cs.CL], 2020

    2020 arXiv

  20. [28]

    C., Petrov, T., Kubin, G., and Koeppl, H

    Geiger, B. C., Petrov, T., Kubin, G., and Koeppl, H. Optimal K ullback-- L eibler aggregation via information bottleneck. IEEE Transactions on Automatic Control, 60 0 (4): 0 1010--1022, 2014

    2014

  21. [29]

    Harris, Z. S. Distributional structure. Word, 10 0 (2-3): 0 146--162, 1954

    1954

  22. [30]

    explanatory

    Harshman, R. A. Foundations of the parafac procedure: Models and conditions for an "explanatory" multimodal factor analysis. UCLA Working Papers in Phonetics, 16: 0 1--84, 1970

    1970

  23. [31]

    The developmental landscape of in-context learning, 2024

    Hoogland, J., Wang, G., Farrugia-Roberts, M., Carroll, L., Wei, S., and Murfet, D. The developmental landscape of in-context learning, 2024. URL https://arxiv.org/abs/2402.02364

    2024 arXiv

  24. [32]

    M., and Zhang, T

    Hsu, D., Kakade, S. M., and Zhang, T. A spectral algorithm for learning hidden markov models. Journal of Computer and System Sciences, 78 0 (5): 0 1460--1480, 2012

    2012

  25. [33]

    Universal artificial intelligence: Sequential decisions based on algorithmic probability

    Hutter, M. Universal artificial intelligence: Sequential decisions based on algorithmic probability. Springer Science & Business Media, 2005

    2005

  26. [34]

    Kim, Y., Jernite, Y., Sontag, D., and Rush, A. M. Character-aware neural language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016

    2016

  27. [35]

    Kolda, T. G. and Bader, B. W. Tensor decompositions and applications. SIAM review, 51 0 (3): 0 455--500, 2009

    2009

  28. [36]

    and Richardson, J

    Kudo, T. and Richardson, J. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226, 2018

    2018 arXiv

  29. [37]

    The local learning coefficient: A singularity-aware complexity measure, 2024

    Lau, E., Furman, Z., Wang, G., Murfet, D., and Wei, S. The local learning coefficient: A singularity-aware complexity measure, 2024. URL https://arxiv.org/abs/2308.12108

    2024 arXiv

  30. [38]

    P., Hoogland, J., Farrugia-Roberts, M., Wei, S., Oldenziel, A

    Lehalleur, S. P., Hoogland, J., Farrugia-Roberts, M., Wei, S., Oldenziel, A. G., Wang, G., Carroll, L., and Murfet, D. You are what you eat -- ai alignment requires understanding how data shapes structure and generalisation, 2025. URL https://arxiv.org/abs/2502.05475

    2025 arXiv

  31. [39]

    and Nave, C

    Levin, M. and Nave, C. P. Tensor renormalization group approach to two-dimensional classical lattice models. Physical review letters, 99 0 (12): 0 120601, 2007

    2007

  32. [40]

    J., Alyafeai, Z., Salesky, E., Raffel, C., Dey, M., Gallé, M., Raja, A., Si, C., Lee, W

    Mielke, S. J., Alyafeai, Z., Salesky, E., Raffel, C., Dey, M., Gallé, M., Raja, A., Si, C., Lee, W. Y., Sagot, B., and Tan, S. Between words and characters: A B rief H istory of O pen- V ocabulary M odeling and T okenization in NLP , 2021. URL https://arxiv.org/abs/2112.10508

    2021 arXiv

  33. [41]

    Efficient estimation of word representations in vector space

    Mikolov, T., Chen, K., Corrado, G., and Dean, J. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013 a

    2013 arXiv

  34. [42]

    S., and Dean, J

    Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013 b

    2013

  35. [43]

    Pattern theory: a unifying perspective

    Mumford, D. Pattern theory: a unifying perspective. In First European Congress of Mathematics: Paris, July 6-10, 1992 Volume I Invited Lectures (Part 1), pp.\ 187--224. Springer, 1994

    1992

  36. [44]

    G., and Segarra, S

    Navarro, M., Rozada, S., Marques, A. G., and Segarra, S. Low-rank tensors for multi-dimensional markov models. arXiv preprint arXiv:2411.02098, 2024

    2024 arXiv

  37. [45]

    On the realization of hidden markov models and tensor decomposition

    Ohta, Y. On the realization of hidden markov models and tensor decomposition. IFAC-PapersOnLine, 54 0 (9): 0 725--730, 2021

    2021

  38. [46]

    and Faccioli, P

    Orioli, S. and Faccioli, P. Dimensional reduction of markov state models from renormalization group theory. The Journal of chemical physics, 145 0 (12), 2016

    2016

  39. [47]

    Oseledets, I. V. Tensor-train decomposition. SIAM Journal on Scientific Computing, 33 0 (5): 0 2295--2317, 2011

    2011

  40. [48]

    Future lens: Anticipating subsequent tokens from a single hidden state

    Pal, K., Sun, J., Yuan, A., Wallace, B., and Bau, D. Future lens: Anticipating subsequent tokens from a single hidden state. In Jiang, J., Reitter, D., and Deng, S. (eds.), Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pp.\ 548--560, Si...

    2023 doi

  41. [49]

    Pennington, J., Socher, R., and Manning, C. D. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp.\ 1532--1543, 2014

    2014

  42. [50]

    and Hutter, M

    Phuong, M. and Hutter, M. Formal algorithms for transformers. Preprint arXiv:2207.09238 [cs.LG], 2022

    2022 arXiv

  43. [51]

    Renormalization and effective lagrangians

    Polchinski, J. Renormalization and effective lagrangians. Nuclear Physics B, 231 0 (2): 0 269--295, 1984

    1984

  44. [52]

    and Simon, B

    Reed, M. and Simon, B. Methods of modern mathematical physics i: Functional analysis, acad. Press, San Diego, 1980

    1980

  45. [53]

    Rogers, T. T. and McClelland, J. L. Semantic Cognition: A Parallel Distributed Processing Approach. MIT Press, 2004

    2004

  46. [54]

    How good is your tokenizer? on the monolingual performance of multilingual language models

    Rust, P., Pfeiffer, J., Vuli \'c , I., Ruder, S., and Gurevych, I. How good is your tokenizer? on the monolingual performance of multilingual language models. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Proceedings of the 59th Annual Meeting of the Association for Co...

    2021 doi

  47. [55]

    and Wiskott, L

    Seabrook, E. and Wiskott, L. A tutorial on the spectral theory of markov chains. Neural Computation, 35 0 (11): 0 1713--1796, 2023

    2023

  48. [56]

    Neural machine translation of rare words with subword units

    Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 1715--1725, 2016

    2016

  49. [57]

    Transformers represent belief state geometry in their residual stream

    Shai, A., Teixeira, L., Oldenziel, A., Marzen, S., and Riechers, P. Transformers represent belief state geometry in their residual stream. Advances in Neural Information Processing Systems, 37: 0 75012--75034, 2024

    2024

  50. [58]

    Shannon, C. E. A mathematical theory of communication. The Bell system technical journal, 27 0 (3): 0 379--423, 1948

    1948

  51. [59]

    The low-rank hypothesis of complex systems

    Thibeault, V., Allard, A., and Desrosiers, P. The low-rank hypothesis of complex systems. Nature Physics, 20 0 (2): 0 294--302, 2024

    2024

  52. [60]

    Tucker, L. R. Some mathematical notes on three-mode factor analysis. Psychometrika, 31 0 (3): 0 279--311, 1966

    1966

  53. [61]

    and Urdshals, J

    Urdshals, E. and Urdshals, J. Structure development in list-sorting transformers, 2025. URL https://arxiv.org/abs/2501.18666

    2025 arXiv

  54. [62]

    Vaart, A. W. v. d. Asymptotic Statistics. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 1998

    1998

  55. [63]

    The generalization phase diagram

    Vaintrob, D. The generalization phase diagram. https://www.lesswrong.com/posts/TSe3qhe4kxgKPJmfD/the-generalization-phase-diagram, 2025

    2025

  56. [64]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  57. [65]

    Differentiation and specialization of attention heads via the refined local learning coefficient, 2024

    Wang, G., Hoogland, J., van Wingerden, S., Furman, Z., and Murfet, D. Differentiation and specialization of attention heads via the refined local learning coefficient, 2024. URL https://arxiv.org/abs/2410.02984

    2024 arXiv

  58. [66]

    Algebraic Geometry and Statistical Learning Theory

    Watanabe, S. Algebraic Geometry and Statistical Learning Theory . Cambridge University Press , USA , 2009

    2009

  59. [67]

    A Widely Applicable Bayesian Information Criterion

    Watanabe, S. A Widely Applicable Bayesian Information Criterion . Journal of Machine Learning Research, 14: 0 867--897, 2013

    2013

  60. [68]

    Mathematical Theory of Bayesian Statistics

    Watanabe, S. Mathematical Theory of Bayesian Statistics . CRC Press, Taylor and Francis group , USA , 2018

    2018

  61. [69]

    and Teh, Y

    Welling, M. and Teh, Y. W. Bayesian L earning via S tochastic G radient L angevin D ynamics. In Proceedings of the 28th International Conference on Machine Learning , 2011

    2011

  62. [70]

    X., and Levine, L

    Wu, W., Morris, J. X., and Levine, L. Do language models plan ahead for future tokens? In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=BaOAvPUyBO

    2024

  63. [71]

    Pyhessian: Neural networks through the lens of the hessian, 2020

    Yao, Z., Gholami, A., Keutzer, K., and Mahoney, M. Pyhessian: Neural networks through the lens of the hessian, 2020. URL https://arxiv.org/abs/1912.07145

    2020 arXiv

  64. [72]

    and Wang, M

    Zhang, A. and Wang, M. Spectral state compression of markov processes. IEEE transactions on information theory, 66 0 (5): 0 3202--3231, 2019

    2019

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.