REVIEW 13 cited by
Think before you speak: Training Language Models With Pause Tokens
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Language models generate responses by producing a series of tokens in immediate succession: the $(K+1)^{th}$ token is an outcome of manipulating $K$ hidden vectors per layer, one vector per preceding token. What if instead we were to let the model manipulate say, $K+10$ hidden vectors, before it outputs the $(K+1)^{th}$ token? We operationalize this idea by performing training and inference on language models with a (learnable) $\textit{pause}$ token, a sequence of which is appended to the input prefix. We then delay extracting the model's outputs until the last pause token is seen, thereby allowing the model to process extra computation before committing to an answer. We empirically evaluate $\textit{pause-training}$ on decoder-only models of 1B and 130M parameters with causal pretraining on C4, and on downstream tasks covering reasoning, question-answering, general understanding and fact recall. Our main finding is that inference-time delays show gains when the model is both pre-trained and finetuned with delays. For the 1B model, we witness gains on 8 of 9 tasks, most prominently, a gain of $18\%$ EM score on the QA task of SQuAD, $8\%$ on CommonSenseQA and $1\%$ accuracy on the reasoning task of GSM8k. Our work raises a range of conceptual and practical future research questions on making delayed next-token prediction a widely applicable new paradigm.
Forward citations
Cited by 13 Pith papers
-
DeepLoop: Depth Scaling for Looped Transformers
Looped Transformers need residual-scaling exponent p=1/2 instead of DeepNorm's 1/4 when shared blocks are revisited with aligned visit-wise gradients.
-
Next-Latent Prediction Transformers Learn Compact World Models
NextLat augments next-token prediction with latent next-state prediction, theoretically converging latents to belief states and showing empirical gains in world modeling, reasoning, planning, and faster inference via ...
-
Can Aha Moments Be Fake? Towards Quantifying Decorative and True Thinking in Chain-of-Thought
LLMs interleave true causal reasoning steps with decorative ones in CoT, with only ~2.3% of steps having high causal impact on AIME for Qwen-2.5, and a steering direction can force internal use of specific steps.
-
Learning Temporal Abstractions via Variational Homomorphisms in Option-Induced Abstract MDPs
A variational option-critic algorithm with latent option embeddings and an implicit chain-of-thought cold-start is presented; the central optimality-preservation proof has a gap and some reported benchmark wins are in...
-
AnalogFed: Privacy-Preserving Discovery of Analog Circuits at Scale with Federated Generative AI
AnalogFed combines federated learning with a generative analog-topology model, adding dummy-token input perturbation and partial homomorphic encryption to resist membership inference and model inversion attacks.
-
MADI: Masking-Augmented Diffusion with Inference-Time Scaling for Visual Editing
A masking-augmented diffusion objective plus pause-token inference scaling modestly improves instruction adherence and source preservation for OmniGen-based image editing.
-
Energy-Based Transformers are Scalable Learners and Thinkers
Energy-Based Transformers learn to predict by gradient-descent minimization of a learned energy function, and the paper reports faster pretraining scaling and inference-time thinking gains over Transformer++ and Diffu...
-
System-1.5 Reasoning: Traversal in Language and Latent Spaces with Dynamic Shortcuts
System-1.5 Reasoning lets LLMs reason in latent space with early exits and step-skipping, matching chain-of-thought accuracy at over 20x speedup on GSM8K and StrategyQA.
-
Skip-Thinking: Chunk-wise Chain-of-Thought Distillation Enable Smaller Language Models to Reason Better and Faster
Chunk-wise training with loss-guided chunking and skip-thinking training improves small-model reasoning accuracy and speed over standard chain-of-thought distillation.
-
Not All LLM Reasoning is Visible in the Chain-of-Thought
Semantically empty filler tokens improve accuracy across several frontier LLMs on synthetic math tasks and let Claude Opus 4.5 satisfy a hidden modular constraint, evidence of computation invisible in output tokens.
-
Is MoE Routing a Huffman Code? Discovering the Frequency-Diversity Law in Chain-of-Thought
MoE routers allocate expert diversity in proportion to operation rarity (the "Frequency-Diversity Law"), and subset-difference pruning can expose this pattern when load-balancing creates redundant experts.
-
HALO: Hybrid Adaptive Latent Reasoning for Language Models
Selective second-stage latent refinement on budgeted scored tokens beats uniform fixed-1 and fixed-2 refinement on frozen Phi-4-mini for average MMLU-Pro/GPQA score at lower average applied refine steps.
-
Knowing Before Saying: LLM Representations Encode Information About Chain-of-Thought Success Before Completion
LLM hidden states encode enough information to predict chain-of-thought success before any reasoning tokens are generated, outperforming a text-only classifier.
Discussion (0). Sign in to comment.