REVIEW 10 cited by
Physics of Language Models: Part 1, Learning Hierarchical Language Structures
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Transformer-based language models are effective but complex, and understanding their inner workings and reasoning mechanisms is a significant challenge. Previous research has primarily explored how these models handle simple tasks like name copying or selection, and we extend this by investigating how these models perform recursive language structure reasoning defined by context-free grammars (CFGs). We introduce a family of synthetic CFGs that produce hierarchical rules, capable of generating lengthy sentences (e.g., hundreds of tokens) that are locally ambiguous and require dynamic programming to parse. Despite this complexity, we demonstrate that generative models like GPT can accurately learn and reason over CFG-defined hierarchies and generate sentences based on it. We explore the model's internals, revealing that its hidden states precisely capture the structure of CFGs, and its attention patterns resemble the information passing in a dynamic programming algorithm. This paper also presents several corollaries, including showing why absolute positional embeddings is inferior to relative and rotary embeddings; uniform attention alone is surprisingly effective (motivating our follow-up work on Canon layers); encoder-only models (e.g., BERT, DeBERTa) struggle with deep structure reasoning on CFGs compared to autoregressive models (e.g., GPT); and injecting structural or syntactic noise into pretraining data markedly improves robustness to corrupted language prompts.
Forward citations
Cited by 10 Pith papers
-
Smooth Scaling Laws Hide Stepwise Token Learning
Token loss trajectories follow localized sigmoids whose learning-time spectrum quantitatively reconstructs scaling-law derivatives on T, D, and M axes and enables faster training via distribution reshaping.
-
Linear Causal Representation Learning by Topological Ordering, Pruning, and Disentanglement
By combining topological ordering, pruning, and disentanglement, CREATOR identifies linearly mixed latent causal variables up to permutation-and-scale ambiguity using only non-Gaussian noise.
-
(How) Can Transformers Predict Pseudo-Random Numbers?
Transformers predict LCG sequences in-context for fixed moduli up to 2^32 and unseen moduli up to 2^16 by learning the modulus factorization and digit-wise periodic structure.
-
Reasoning Core: Designing Broad Procedural Data for Completion-Supervised Reasoning Training
A new collection of 50 procedural generators designed for completion-supervised fine-tuning beats three existing procedural collections and a no-procedural baseline on reasoning benchmarks at 3B scale in mean scores.
-
Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers
Canon layers—residual 1-d causal convolutions over adjacent tokens—boost synthetic reasoning depth 2-4x, lift NoPE to RoPE level, and bring GLA up to Mamba2/GDN, with qualitative real-world confirmation.
-
Rethinking Memorization Measures and their Implications in Large Language Models
Contextual memorization, defined by comparing a string's training loss against the best loss without training on that string, is stricter than counterfactual memorization and suggests that zero-memorization optimal le...
-
Position: We Need An Algorithmic Understanding of Generative AI
The paper argues for a systematic algorithmic understanding of LLMs and presents a case study suggesting that Llama models do not implement BFS or DFS on graph navigation tasks.
-
Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
A string-based, metadata-guided language model framework for universal offline black-box optimization, with two variants and two embedding regularizations.
-
A Statistical Physics of Language Model Reasoning
A switching linear dynamical system on a 40-dimensional projection of LLM hidden states captures about half the variance of reasoning trajectories and predicts belief shifts during adversarial prompts.
-
Unifying Learning Dynamics and Generalization in Transformers Scaling Law
Claims a two-stage transformer scaling law (exponential then C^{-1/6}) with matching bounds, but the lower bounds are missing, the exponent is inconsistent (-1/7 vs -1/6), and the law is an artifact of hand-set M = Θ(...
Discussion (0). Continue with ORCID to comment.