Pith. sign in

REVIEW 4 major objections 5 minor 5 cited by

Deriving Neural Scaling Laws from the statistics of natural language

T0 review · 4 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Data-limited scaling exponents of language models are set by two measurable statistics of the training corpus: the entropy-decay exponent γ and the correlation-decay exponent β, yielding α_D = γ/(2β).

desk verdict First quantitative theory of LLM data-limited scaling from language statistics, with a clean formula and elegant collapse — but the fast-learning assumption underpinning it is tested only thinly and the AR exponent is never fitted, so treat as promising, not settled. read the letter →

arxiv 2602.07488 v3 pith:WQEG76N3 submitted 2026-02-07 cs.LG cs.AIstat.ML

Francesco Cagnetta , Allan Raventós , Surya Ganguli , Matthieu Wyart This is my paper
classification cs.LGcs.AIstat.ML MSC 68T0768T50
keywords neuralscalinglawsdata-limitedautoregressivelossconditionalentropydecaytoken-tokencorrelationshorizon-limitedlearningcollapsenaturallanguagestatistics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that how fast a language model's test loss falls with the amount of training data — the data-limited scaling exponent — is not set by the architecture but by two statistical properties of the language itself: how quickly next-token conditional entropy drops as context grows (exponent γ) and how quickly token-token correlations decay with separation (exponent β). The mechanism is horizon-limited learning: with P tokens a model can only exploit statistical dependencies up to a context length n*(P) ≍ P^{1/(2β)}, and the loss is dominated by the entropy at that horizon, giving excess loss ≍ P^{−γ/(2β)}. The prediction has no free parameters and is verified by measuring γ and β on two corpora and comparing with the observed loss curves of transformer language models, including a data-collapse of all n-gram losses onto one curve. If right, it turns a widely observed empirical law into a derived consequence of linguistic statistics, and identifies the effective horizon as the learning bottleneck in the data-limited regime.

What carries the argument

The load-bearing object is the data-dependent prediction horizon n*(P), defined by equating the strongest token-token correlation at lag n, ∥C(n)∥_op, with the sampling noise O(P^{−1/2}); this gives n*(P) ≍ P^{1/(2β)}. Around this horizon the loss decomposition L_AR(P) ≍ H_{n*(P)} + Σ_{n≤n*(P)} E_n(P) separates two learning mechanisms — extending the horizon versus improving prediction within it — and the power-law ansatze H_n − H_∞ ≍ n^{−γ} and ∥C(n)∥_op ≍ n^{−β} convert it into the exponent prediction. The supporting identity is the scaling collapse L_n(P) ≍ n^{−γ} ℓ(P/n^{2β}), which makes the theory testable curve-by-curve.

What would settle it

Train a transformer on a text corpus whose γ and β have been measured, and fit the late-P decay of individual n-gram losses L_n(P) − H_n. If for any n the fitted exponent δ_n is no larger than γ/(2β), then the theory's fast-learning regime fails: Eq. 38 predicts the full loss should decay as P^{−δ_n} (slower than γ/(2β)), so checking whether the observed autoregressive loss follows the slower decay or the faster one would settle which regime governs.

Watch

Extended reading notes

Core claim

The central discovery is that the data-limited neural scaling exponent α_D for autoregressive language modeling is, in the fast-learning regime, α_D = γ/(2β), where γ is the exponent of the power-law decay H_n − H_∞ ≍ n^{−γ} of the next-token conditional entropy with context length n, and β is the exponent of the power-law decay ∥C(n)∥_op ≍ n^{−β} of the operator norm of the token-token covariance matrix at lag n. The argument: a signal-to-noise threshold ∥C(n)∥_op ≍ P^{−1/2} sets the largest context length n*(P) ≍ P^{1/(2β)} that P tokens can support; the autoregressive loss then decomposes into the entropy at that horizon plus excess losses from suboptimal within-horizon prediction; if the

Load-bearing premise

The whole exponent identification depends on the fast-learning hypothesis — that within the prediction horizon the model already uses available context nearly optimally, so the dominant effect of more data is to lengthen the horizon; if the within-horizon excess losses decay with an exponent δ ≤ γ/(2β), the observed scaling would instead be P^{−δ} and the prediction would not be α_D = γ/(2β).

Editorial extensions

If this is right

  • If the formula holds, the data-limited scaling exponent of any new corpus is fixed once γ and β are measured on that corpus, with no model training needed.
  • Improving data efficiency in this regime must come from breaking the horizon bottleneck — e.g., architectures that capture longer-range correlations from fewer tokens — since within-horizon learning is already fast.
  • The predicted collapse of n-gram loss curves provides a strong, architecture-independent signature that the model is in the horizon-limited regime: all context lengths follow the same master curve.
  • The theory implies a universality class: different deep architectures trained on the same language share the same exponent (though prefactors may differ), while shallow networks and kernel methods fall outside it with worse exponents.
  • The finite-context corrections are controlled: when the maximal context T becomes comparable to n*(P), the loss is truncated by T, so the pure power law is only expected in the regime n*(P) ≪ T — a fact the paper uses to set the valid comparison range.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: measure γ and β on a corpus with engineered statistics (e.g., a finite-order Markov chain with known decays) and check that the trained transformer's exponent follows γ/(2β); this would isolate the mechanism from confounds of real language.
  • If the horizon-limited picture extends to compute-optimal or model-size-limited scaling, the same two exponents might reappear in different combinations there — a guess the paper does not make but that arises naturally from its mechanism.
  • The collapse prediction suggests a cheap diagnostic: plot n^γ L_n against P/n^{2β}; systematic deviations at large n would show where the horizon-limited assumption breaks down, possibly indicating a transition to within-horizon-limited learning.
  • Because the theory identifies the bottleneck as information-theoretic, it hints that data-selection or curriculum strategies that lower effective β (making longer-range correlations easier to detect) could improve scaling exponents — an implication the authors leave implicit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a theory to predict the data-limited neural scaling law exponent for autoregressive language models from two measurable corpus statistics: the power-law decay of next-token conditional entropy with context length (exponent γ) and the power-law decay of token-token correlation strength with temporal lag (exponent β). The central claim is that, in the 'horizon-limited' fast-learning regime, the autoregressive test loss decays as L_AR(P) − H_∞ ≍ P^{−γ/(2β)}, with the exponents being properties of the dataset alone. The derivation in App. A decomposes the loss into a boundary term from the data-dependent prediction horizon and a sum of within-horizon excess losses, and shows that the observed exponent is min(δ, γ/(2β)). The paper estimates γ and β on TinyStories and WikiText, demonstrates collapse of n-gram loss curves under the predicted rescaling, and compares the autoregressive loss decay to the predicted slope for GPT-2/LLaMA-style transformers.

Significance. If the central claim holds, this would be a major advance: the first quantitative, parameter-free prediction of neural scaling law exponents from the statistics of natural language, with a falsifiable collapse prediction. The paper's strengths include a self-contained analytical derivation, explicit statements of all hypotheses, cross-architecture consistency of the estimated entropy exponent, and empirical tests on two qualitatively different corpora. However, the validity of the headline exponent rests on the fast-learning condition δ > γ/(2β), and the current evidence for this condition is thin; moreover, the principal validation of the exponent itself is visual rather than quantitative. These gaps make the claim conditional rather than established.

major comments (4)
  1. [§5, Eq. (38), Fig. 6] The fast-learning condition δ > γ/(2β) is load-bearing: if instead δ < γ/(2β), the observed exponent is δ and the advertised prediction α_D = γ/(2β) fails. The only direct test, Fig. 6, has three weaknesses: (i) it covers only n ≤ 12, while the data-dependent horizon n*(P) reaches a few tens of tokens at the largest P, so the regime most relevant to Eq. (34) is not directly tested; (ii) H_n is not measured independently but chosen by a grid search that maximizes R² of the subsequent power-law fit, so δ_n and H_n are co-fitted and the decay rate can be inflated; (iii) no confidence intervals are reported, and δ_n ≈ 0.2–0.3 versus γ/(2β) ≈ 0.19 is a thin margin. This precondition must be verified more rigorously before the central claim can be accepted.
  2. [§4.2, Figs. 1 and 4] The headline validation of the exponent prediction is only visual: the autoregressive losses L(P) are plotted against a dashed line of slope −γ/(2β), but no fitted slope or confidence interval for L(P) is reported for any context length T or dataset. Since the central claim is a quantitative exponent prediction, the paper should report empirical exponents obtained from power-law fits over the stated scaling range, together with uncertainties, and compare them explicitly to γ/(2β). Without this, the match shown in the figures cannot be distinguished from a range of plausible exponents.
  3. [§4.1.1] The entropy exponent γ is estimated from n-gram losses of trained transformers from the same model families (GPT-2, LLaMA) that the theory subsequently predicts. The paper argues for convergence and architecture independence, which supports the interpretation that L_n approaches H_n, but this is not a fully independent measurement of a dataset property. To strengthen the claim that γ (and hence α_D) is determined by the corpus alone, the authors should provide an independent estimate of H_n (e.g., from a different estimator or model family) or at least a sensitivity analysis showing that γ is stable across model sizes and training budgets.
  4. [Limitations] The Limitations section concedes that n*(P_max) corresponds to a few tens of tokens and that the tests are at academic scale. This is a significant scope restriction: the fast-learning regime is asserted to be the relevant one for modern LLMs, but the empirical evidence is confined to very short effective horizons and dataset sizes up to 10^8 tokens. The paper should either temper the title-level claim ('neural scaling laws' of modern LLMs) or provide additional evidence, e.g., by testing on larger P to push n* further, or by studying a setting where the within-horizon excess losses can be verified at larger n.
minor comments (5)
  1. [§5, Fig. 6] The scatter plot of δ_n (right panel) has no error bars. Adding confidence intervals would help judge whether δ_n > γ/(2β) is robust, especially given the co-fitting procedure.
  2. [§4.1.2] For WikiText, the correlation decay is described as a broken power law with a localized peak near n≈10. The paper states that the short-lag regime is used, but it would help to report the sensitivity of β to the fitting range and to the treatment of the outlier peak.
  3. [App. C] Code is promised only 'upon publication.' For reproducibility, consider releasing code and hyperparameters at the time of submission, or at least providing a complete table of final hyperparameters for all experiments.
  4. [§5 / App. A] The notation δ_n is used in Sec. 5, while the derivation in App. A sets δ_n = δ for simplicity. The relationship between the estimated δ_n and the single δ used in Eq. (35)–(38) should be stated explicitly.
  5. [Eq. (22) / Eq. (30)] The factors (T − (n − 1))/T are dropped in Eq. (30) to obtain the asymptotic expression Eq. (31). This implicitly assumes T ≫ n*(P). The condition for this approximation to be valid should be stated in the main text, not only in the prose around Eq. (5).

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: the central exponent prediction is a composition of independently measured language statistics, with minor self-citation and a model-based estimate of gamma as caveats.

full rationale

The paper's central formula alpha_D = gamma/(2 beta) is derived by composing two dataset-level power laws: ||C(n)||_op ~ n^{-beta} (measured directly from token co-occurrence counts, Eq. 27) fixes n*(P) ~ P^{1/(2 beta)} via the SNR threshold Eq. 4, and H_n - H_infinity ~ n^{-gamma} (Eq. 6) then gives the boundary-term decay P^{-gamma/(2 beta)}. beta is fully independent of the trained models, and the relation to the autoregressive loss is not tautological: Eq. 38 explicitly contains min{delta, gamma/(2 beta)}, so alpha_D = gamma/(2 beta) is a conditional prediction that fails if within-horizon learning is slow. The two caveats are verification gaps rather than circularity: gamma is estimated from GPT-2/LLaMA n-gram losses on the same corpora (Sec. 4.1.1), though the paper reports consistency across APE, RoPE, and LLaMA, and the fast-learning check (Sec. 5) co-fits H_n by grid search with no confidence intervals. These limit the strength of the evidence but do not make the prediction equal to its input by construction. Self-citations to Cagnetta & Wyart (2024) introduce the horizon concept, but the horizon is re-derived here from the SNR argument, so the self-citation is not load-bearing. The Limitations section explicitly notes the academic scale (n* of a few tens of tokens), which is a scope limitation, not a circular step.

Assumptions & free parameters 4 free parameters · 7 assumptions · 0 invented entities

The central prediction rests on two measured power-law exponents (γ, β) and an ansatz about how differential losses transition at a data threshold. The deepest 'free' input is the fast-learning assumption δ > γ/(2β), which is structurally required for the exponent α_D to be γ/(2β) rather than δ. No new particles, forces, or physical entities are postulated; the data-dependent horizon n*(P) is a derived construct, not a new entity. The paper honestly labels these as hypotheses, but the 'no free parameters' claim in the abstract applies only to the final exponent formula, not to the measurement of γ and β or to the assumed power-law forms.

free parameters (4)
  • γ (conditional entropy decay exponent) = 0.34 (TinyStories), 0.27 (WikiText)
    Fit by power-law regression to the small-n portion of L_n-vs-n for the largest-P trained model (Sec. 4.1.1). This is an input to the prediction α_D = γ/(2β); it is measured from data, not fitted to the target scaling-law exponent.
  • β (token-token correlation decay exponent) = 0.88 (TinyStories), 0.94 (WikiText)
    Fit by power-law regression to ‖C(n)‖_op-vs-n (Sec. 4.1.2). For WikiText only the initial decay stage is used; the broken power law and the n≈10 peak are explicitly ignored.
  • δ_n (excess-loss exponents) = ≈0.2–0.3 (Fig. 6, n ≤ 12)
    Extracted by fitting L_n − H_n to A P^{−δ_n} with a grid search over the asymptote H_n (step 10^{−2}). Used to test the fast-learning condition δ > γ/(2β); not used directly in the final exponent formula.
  • Threshold constant c in Eq. 26 = unspecified O(1)
    Introduced in the signal-to-noise threshold P*_n = c^2/‖C(n)‖². It cancels in all exponent predictions, so it does not affect the central claim, but it is an extra undetermined constant in the framework.
assumptions (7)
  • domain assumption H_n − H_∞ decays as a power law n^{−γ} (Hypothesis 1, App. A Eq. 19).
    Empirically motivated; not derived from first principles. It is an input to the final exponent formula.
  • domain assumption ‖C(n)‖_op decays as a power law n^{−β} (Hypothesis 2, App. A Eq. 27).
    Empirically motivated; the WikiText data show a broken power law with a peak near n≈10, so the assumption is only approximately valid in the fitted stage.
  • ad hoc to paper Differential loss Δ_n(P) has the scaling form Δ_n(P) = (H_n − H_{n−1}) f_n(P/P*_n) with f_n(x)→0 for x≪1 and 1−f_n(x)→x^{−δ_n} for x≫1 (App. A Eq. 23).
    This ansatz is not derived; it is the backbone of the derivation of both the collapse form (Eq. 9) and the final exponent (Eq. 38).
  • ad hoc to paper Fast-learning condition: δ > γ/(2β) (equivalently, the excess-loss sum decays at least as fast as the boundary term; App. A Eqs. 36–38).
    Without this condition the predicted exponent would be δ, not γ/(2β). Verified only indirectly for n ≤ 12 in Sec. 5, with a thin margin.
  • standard math The trained model's n-gram loss L_n(P,M) is an upper bound on H_n and converges to H_n for infinite data at sufficient capacity (Eq. 10).
    Cross-entropy ≥ entropy is exact; convergence in the infinite-data limit is a standard consistency assumption for expressive models, and is used to estimate γ from finite P.
  • domain assumption Stationarity of the token process, i.e. (X_1:(n−1)) has the same distribution as (X_2:n) (needed for Eq. 21).
    Assumed so that Δ_n = L_n − L_{n−1}; real corpora are chunked and concatenated with EOS tokens, so stationarity is only approximate.
  • domain assumption Signal-to-noise model: the learner can detect a correlation lag n only if ‖C(n)‖_op > c/√P (Eq. 26).
    Uses O(P^{−1/2}) concentration of empirical covariance matrices; the constant c is unknown but cancels in the exponent.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Deriving Neural Scaling Laws from the statistics of natural language." pith.science (2026). https://pith.science/paper/WQEG76N3

@misc{pith2026260207488,
  author       = {Pith},
  title        = {Pith review of: Deriving Neural Scaling Laws from the statistics of natural language},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WQEG76N3}},
  note         = {Machine review of arXiv:2602.07488}
}
read the original abstract

Despite the fact that experimental neural scaling laws have substantially guided empirical progress in large-scale machine learning, no existing theory can quantitatively predict the exponents of these important laws for any modern LLM trained on any natural language dataset. We provide the first such theory in the case of data-limited scaling laws. We isolate two key statistical properties of language that alone can predict neural scaling exponents: (i) the decay of pairwise token correlations with time separation between token pairs, and (ii) the decay of the next-token conditional entropy with the length of the conditioning context. We further derive a simple formula in terms of these statistics that predicts data-limited neural scaling exponents from first principles without any free parameters or synthetic data models. Our theory exhibits a remarkable match with experimentally measured neural scaling laws obtained from training GPT-2 and LLaMA style models from scratch on two qualitatively different benchmarks, TinyStories and WikiText.

Figures

Figures reproduced from arXiv: 2602.07488 by the authors.

Figure 1
Figure 1. Measurable language statistics predict the exponents of data-limited neural scaling laws in language models. Top: The highly diverse n-gram losses Ln (Eq. 2) of a GPT-2–style transformer trained from scratch on P-tokens of the TinyStories dataset (left) collapse onto a single curve, when plotted in rescaled units (right). Here Ln is rescaled by Hn ≍ n −γ , where γ is the exponent of the power law temporal decay of H… view at source ↗
Figure 2
Figure 2. Conditional entropy decay with time horizon defines a characteristic exponent γ that is architecture-independent. We train three classes of models from scratch on P-token slices of the TinyStories dataset: GPT-2–style transformers with absolute positional embeddings (Left), GPT-2–style transformers with rotary positional embeddings (RoPE, Center Left), and LLaMA-style transformers (Center Right), training a separate… view at source ↗
Figure 3
Figure 3. Decay of two-point correlations as a function temporal separation defines a characteristic exponent β. The second dataset-level statistic we consider is the decay of the two-point correlation function, defined as the norm of the token-token co-occurrence matrix C(n)µν = P(Xi = µ, Xi+n = ν) − P(Xi = µ)P(Xi+n = ν), with the time separation n. We define β as the exponent of a power-law fit to this decay. C(n) is estima… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Same as [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Fitting γ for the WikiText dataset. (Similar to [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: n-gram losses decay to their asymptote faster than P −γ/(2β) . Left: We plot the n-gram losses of our GPT-2–style transformer trained on TinyStories for low n (n ≤ 12). To isolate the decay due to the suboptimal use of the time horizon, we fit the large-P portion of th…
Figure 7
Figure 7. Figure 7: n-gram loss collapse for GPT-2-style transformers trained on WikiText at T = 512. (Same as [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: n-gram collapse and data-limited scaling exponent prediction for GPT-2-style transformers with RoPE trained on WikiText. (Same as [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: n-gram loss collapse for GPT-2-style transformers with RoPE trained on WikiText at T = 512. (Same as [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: n-gram loss collapse for GPT-2-style transformers trained on TinyStories at T = 64, 256 and 512. (Same as [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: n-gram collapse and data-limited scaling exponent prediction for GPT-2-style transformers with RoPE trained on TinyStories. (Same as [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: n-gram loss collapse for LLaMA-style transformers trained on TinyStories at T = 512. These correspond to the same n-gram losses used in [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Bridging Compute- and Data-Optimal Pretraining

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Pretraining loss obeys a single law in which repeated or paraphrased tokens count as η(N, data-per-parameter, expansion-ratio) fresh tokens, with total effective data saturating as derived tokens grow.

  2. Smooth Scaling Laws Hide Stepwise Token Learning

    cs.CL 2026-06 unverdicted novelty 7.0 of 10

    Token loss trajectories follow localized sigmoids whose learning-time spectrum quantitatively reconstructs scaling-law derivatives on T, D, and M axes and enables faster training via distribution reshaping.

  3. Information-Theoretic Limits of Reliability and Scaling in Language Models

    cs.CL 2026-05 conditional novelty 6.0 of 10

    A theoretical framework derives a reliability ceiling and a max-form Chinchilla-type scaling law for LLMs from task entropy and dependency spectra.

  4. Geometry of Ordinal Representations in Language Models

    cs.LG 2026-07 conditional novelty 5.5 of 10

    1D place-cell manifolds emerge for locally token-computable ordinals across models, while cross-position or semantic ordinals yield higher-dimensional or incoherent geometry, with architecture-dependent twisting.

  5. Test-Time Training for Modality Order Consistency in Vision-Language Models

    cs.CV 2026-07 conditional novelty 5.0 of 10

    VLMs are less accurate when the question precedes the image; per-instance KL self-distillation using the image-first answer as teacher closes most of the gap and slightly improves the image-first branch.

Reference graph

Works this paper leans on

6 extracted references · 1 canonical work pages · cited by 5 Pith papers

  1. [4]

    URL https: //proceedings.neurips.cc/paper_files /paper/2024/file/1dccfc3ee01871d05e3 3457c61037d59-Paper-Conference.pdf

    doi: 10.52202/079017-0526. URL https: //proceedings.neurips.cc/paper_files /paper/2024/file/1dccfc3ee01871d05e3 3457c61037d59-Paper-Conference.pdf. Paszke, A. et al. Pytorch: An imperative style, high- performance deep learning library. InNeurIPS, 2019. Peters, M. E., Neumann, M., Zettlemoyer, L., and Yih, W. Dissecting contextual word embeddings: Archite...

    work page doi:10.52202/079017-0526 2024

  2. [2017]

    (wikitext-103-raw-v1) datasets. All text is tokenized using a two-stage procedure consisting of whitespace pre- tokenization followed by byte-pair encoding (BPE) (Gage, 1994; Sennrich et al., 2016) with a vocabulary size of 8192. The resulting token sequences are concatenated into a single stream, with an end-of-sequence (EOS) token inserted between docum...

    1994

  3. [2018]

    boundary term

    Association for Computational Linguistics. doi: 10.18653/v1/D18-1179. URL https://aclantho logy.org/D18-1179. Porian, T., Wortsman, M., Jitsev, J., Schmidt, L., and Car- mon, Y . Resolving discrepancies in compute-optimal scaling of language models, 2025. URL https://ar xiv.org/abs/2406.19146. 10 Deriving neural scaling laws from the statistics of natural...

    arXiv 2025

  4. [2022]

    Manning, C

    URL https://arxiv.org/abs/2210.1 6859. Manning, C. D., Clark, K., Hewitt, J., Khandelwal, U., and Levy, O. Emergent linguistic structure in artificial neural networks trained by self-supervision.Proceedings of the National Academy of Sciences, 117(48):30046–30054, 2020. Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. I...

    2020

  5. [2023]

    cc/paper_files/paper/2023/file/5b634 6a05a537d4cdb2f50323452a9fe-Paper-Con ference.pdf

    URL https://proceedings.neurips. cc/paper_files/paper/2023/file/5b634 6a05a537d4cdb2f50323452a9fe-Paper-Con ference.pdf. Nam, Y ., Fonseca, N., Lee, S. H., Mingard, C., and Louis, A. A. An exactly solvable model for emergence and scaling laws in the multitask sparse parity problem. InThe Thirty-eighth Annual Conference on Neural Information Processing Sys...

    2023

  6. [2024]

    URL https: //www.pnas.org/doi/abs/10.1073/pnas

    doi: 10.1073/pnas.2311878121. URL https: //www.pnas.org/doi/abs/10.1073/pnas. 2311878121. Barkeshli, M., Alfarano, A., and Gromov, A. On the origin of neural scaling laws: from random graphs to natural language.arXiv preprint arXiv:2601.10684, 2026. URL https://arxiv.org/abs/2601.10684. Bordelon, B., Canatar, A., and Pehlevan, C. Spectrum- dependent learn...

    arXiv 2026

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.