REVIEW 17 cited by
The pitfalls of next-token prediction
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Can a mere next-token predictor faithfully model human intelligence? We crystallize this emerging concern and correct popular misconceptions surrounding it, and advocate a simple multi-token objective. As a starting point, we argue that the two often-conflated phases of next-token prediction -- autoregressive inference and teacher-forced training -- must be treated distinctly. The popular criticism that errors can compound during autoregressive inference, crucially assumes that teacher-forcing has learned an accurate next-token predictor. This assumption sidesteps a more deep-rooted problem we expose: in certain classes of tasks, teacher-forcing can simply fail to learn an accurate next-token predictor in the first place. We describe a general mechanism of how teacher-forcing can fail, and design a minimal planning task where both the Transformer and the Mamba architecture empirically fail in that manner -- remarkably, despite the task being straightforward to learn. Finally, we provide preliminary evidence that this failure can be resolved using _teacherless_ training, a simple modification using dummy tokens that predicts multiple tokens in advance. We hope this finding can ground future debates and inspire explorations beyond the next-token prediction paradigm. We make our code available under https://github.com/gregorbachmann/Next-Token-Failures
Forward citations
Cited by 17 Pith papers
-
Hierarchical Latent Prediction for Language Models
HiLP adds a hierarchical latent prediction objective to LM pretraining, improving coding and multi-step reasoning benchmarks and speculative decoding acceptance, with zero inference-time overhead.
-
The Effect of Perceived Race and Gender on Police Language Use: Experimental Evidence from VR Simulations
In randomized VR police encounters, officers' language to Black male characters was less deferential by about 0.1 points per exchange on a 0-10 scale, while White and biracial or multiracial female officers showed mor...
-
WAM-Diff2: Hierarchical AR-to-Diffusion Distillation for Highly Efficient Autonomous Driving VLA
A three-stage distillation converts an autoregressive driving VLA into a block-causal masked diffusion model, preserving planning accuracy while decoding 2.8x faster (15.1x with optimized kernels).
-
Hierarchical Domain Generalization
Over infinite domains, hierarchy-uniform domain generalization is impossible for every nontrivial hypothesis class; a length-generalization bound is a property of the length hierarchy, not a hierarchy-free guarantee.
-
FlowEdit: Information-Theoretic Control of LLM Reasoning Flows for Ill-posed Problems Involving Conflicts
FlowEdit fine-tunes LLMs so they enumerate all self-consistent answers to conflicting-condition problems, improving exact-set-match accuracy by 68% on a synthetic benchmark.
-
DeepLook: Deeper Thinking with Lookahead
A training-free decoder that branches only at low-confidence reasoning segments and ranks continuations by average lookahead confidence reports equal or better accuracy at roughly 1/8 of a confidence-filtered baseline...
-
From Reasoning to Super-Intelligence: A Search-Theoretic Perspective
The Diligent Learner, a reverse-curriculum algorithm with explicit backtracking and a validator, is proven to learn chain-of-thought reasoning efficiently under two learnability assumptions, while standard methods fai...
-
Learning to Insert [PAUSE] Tokens for Better Reasoning
A likelihood-based [PAUSE] token insertion method for fine-tuning shows small gains on GSM8K and MBPP, but the AQUA-RAT result is unreliable because the test set contains training samples.
-
FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocities
A 1.5B unified multimodal model trained with discrete flow matching and metric-induced probability paths matches autoregressive baselines of similar size on generation and understanding benchmarks.
-
Refining Over Resampling: Test-Time Self-Correction for LLM Reasoning
Sampling multiple reasoning paths, refining each with self-critique and self-correction, then majority voting improves math reasoning accuracy over width-only or verifier-based test-time scaling on several open-weight LLMs.
-
Noise is not always detrimental: the capacity of quantum batteries is enhanced in black holes
Hawking radiation is claimed to enhance quantum battery capacity for bipartite mixed states, while environmental noise generally degrades it in type-dependent ways.
-
What am I missing here?: Evaluating Large Language Models for Masked Sentence Prediction
Commercial LLMs are poor at predicting a missing sentence in narrative and expository texts, though they perform better in structured procedural text.
-
STEP Planner: Constructing cross-hierarchical subgoal tree as an embodied long-horizon task planner
STEP builds a coarse-to-fine subgoal tree with LLM-based decomposition and termination checks, reporting higher task success than existing LLM planners on WAH-NL and a real robot.
-
Improving Large Language Models with Concept-Aware Fine-Tuning
Adding lightweight multi-token auxiliary heads with a weighted future-token loss improves supervised fine-tuning of Llama-3-8B-Instruct across five diverse tasks.
-
PIG-Nav: Key Insights for Pretrained Image Goal Navigation Models
A pretrained image-goal navigation model combining early-fusion ViT, auxiliary objectives, and game-video data reports higher success than GNM, ViNT, and NoMaD, though zero-shot generalization is clouded by possible p...
-
Large Language Models for Planning: A Comprehensive and Systematic Survey
A structured survey of LLM planning methods, benchmarks, and interpretability work, organized around a three-way taxonomy.
-
DeepSeek: Paradigm Shifts and Technical Evolution in Large AI Models
A survey of DeepSeek's V3 and R1 models covering MLA, MoE, MTP, GRPO, and training engineering, with no new experimental results.
Discussion (0). Continue with ORCID to comment.