Pith. sign in

REVIEW 11 cited by

Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.05405 v1 pith:AFOMRFHZ submitted 2024-04-08 cs.CL cs.AIcs.LG

Zeyuan Allen-Zhu , Yuanzhi Li This is my paper
classification cs.CLcs.AIcs.LG
keywords knowledgemodelcapacitylanguagemodelsbitsstoragetraining
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Scaling laws describe the relationship between the size of language models and their capabilities. Unlike prior studies that evaluate a model's capability via loss or benchmarks, we estimate the number of knowledge bits a model stores. We focus on factual knowledge represented as tuples, such as (USA, capital, Washington D.C.) from a Wikipedia page. Through multiple controlled datasets, we establish that language models can and only can store 2 bits of knowledge per parameter, even when quantized to int8, and such knowledge can be flexibly extracted for downstream applications. Consequently, a 7B model can store 14B bits of knowledge, surpassing the English Wikipedia and textbooks combined based on our estimation. More broadly, we present 12 results on how (1) training duration, (2) model architecture, (3) quantization, (4) sparsity constraints such as MoE, and (5) data signal-to-noise ratio affect a model's knowledge storage capacity. Notable insights include: * The GPT-2 architecture, with rotary embedding, matches or even surpasses LLaMA/Mistral architectures in knowledge storage, particularly over shorter training durations. This arises because LLaMA/Mistral uses GatedMLP, which is less stable and harder to train. * Prepending training data with domain names (e.g., wikipedia.org) significantly increases a model's knowledge capacity. Language models can autonomously identify and prioritize domains rich in knowledge, optimizing their storage capacity.

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformer

    cs.LG 2025-06 conditional novelty 7.0 of 10

    Freezing query and key attention weights still lets transformers form induction heads and stay close to standard performance on language modeling, while random static attention (MixiT) fails on in-context tasks but su...

  2. Local Redundancy: An Information-Theoretic Measure of Plasticity from Synthetic Memorization

    cs.LG 2026-07 conditional novelty 6.0 of 10

    An information-theoretic 'local redundancy' is lower-bounded, via an entropy-cancellation argument, by the expected squared gradient norm on synthetic probe data, and this proxy modestly out-predicts existing plastici...

  3. Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers

    cs.CL 2025-12 conditional novelty 6.0 of 10

    Canon layers—residual 1-d causal convolutions over adjacent tokens—boost synthetic reasoning depth 2-4x, lift NoPE to RoPE level, and bring GLA up to Mamba2/GDN, with qualitative real-world confirmation.

  4. LRM-1B: Towards Large Routing Model

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A 1B-parameter routing model beats existing multi-task neural solvers on synthetic VRP benchmarks, and the authors fit power-law scaling curves for model size, trajectories, and compute.

  5. How much do language models memorize?

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A compression-based measurement puts GPT-style model memorization capacity at roughly 3.6 bits per parameter, with membership inference success following a sigmoid in the dataset-to-capacity ratio.

  6. Who Reasons in the Large Language Models?

    cs.CL 2025-05 conditional novelty 6.0 of 10

    The authors hypothesize, with evidence from merging and freezing experiments, that the attention output projection is the primary locus of mathematical reasoning in large language models.

  7. Enhancing LLMs via High-Knowledge Data Selection

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A knowledge-element density and coverage scorer selects pre-training data that improves LLM performance on knowledge-intensive and general understanding benchmarks by 2 to 3 points.

  8. Data Quality over Capacity: Internalizing Documents into LoRA Adapters for Closed-Book QA

    cs.CL 2026-07 conditional novelty 5.0 of 10

    For closed-book QA via LoRA adapters, once capacity is sufficient, training-data quality—especially answer brevity—outperforms rank, learning rate, and architecture changes, lifting 15-doc accuracy from 57.7% to 85.7%.

  9. Benchmarking and Rethinking Knowledge Editing for Large Language Models

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Under autoregressive and sequential editing, parameter-based knowledge editing methods perform poorly, while the retrieval-based SCR baseline consistently outperforms them across datasets and models.

  10. Unifying Learning Dynamics and Generalization in Transformers Scaling Law

    cs.LG 2025-12 reject novelty 4.0 of 10

    Claims a two-stage transformer scaling law (exponential then C^{-1/6}) with matching bounds, but the lower bounds are missing, the exponent is inconsistent (-1/7 vs -1/6), and the law is an artifact of hand-set M = Θ(...

  11. Perspective on Utilizing Foundation Models for Laboratory Automation in Materials Research

    cs.RO 2025-06 accept novelty 1.0 of 10

    A perspective article reviews the state of using foundation models for laboratory automation and proposes a roadmap for fully autonomous experiments.

Pith tools