REVIEW 7 cited by
Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Language Models (LLMs) excel at reasoning and planning when trained on chainof-thought (CoT) data, where the step-by-step thought process is explicitly outlined by text tokens. However, this results in lengthy inputs where many words support textual coherence rather than core reasoning information, and processing these inputs consumes substantial computation resources. In this work, we propose a hybrid representation of the reasoning process, where we partially abstract away the initial reasoning steps using latent discrete tokens generated by VQ-VAE, significantly reducing the length of reasoning traces. We explore the use of latent trace abstractions in two scenarios: 1) training the model from scratch for the Keys-Finding Maze problem, 2) fine-tuning LLMs on this hybrid data with an extended vocabulary including unseen latent tokens, for both logical and mathematical reasoning problems. To facilitate effective learning, we introduce a simple training procedure that randomly mixes latent and text tokens, which enables fast adaptation to new latent tokens. Our approach consistently outperforms the baselines methods in various benchmarks.
Forward citations
Cited by 7 Pith papers
-
Dynamics Within Latent Chain-of-Thought: An Empirical Study of Causal Structure
Latent CoT steps are causally heterogeneous and non-local: a few steps dominate, influence skips adjacent steps, and output commitment precedes representational commitment.
-
Not All LLM Reasoning is Visible in the Chain-of-Thought
Semantically empty filler tokens improve accuracy across several frontier LLMs on synthetic math tasks and let Claude Opus 4.5 satisfy a hidden modular constraint, evidence of computation invisible in output tokens.
-
Is Your Model Thinking or Just Stagnating? PUMA: Diagnosing Reasoning Pathology via Phase-Momentum Alignment
PUMA detects when a reasoning model's entropy drop aligns with hidden-state momentum, truncates at that point, and reports improved accuracy-efficiency on 1.5B-32B reasoning models.
-
Implicit Reasoning in Large Language Models: A Comprehensive Survey
A survey organizing implicit (silent) reasoning in LLMs into three execution paradigms, plus evidence, benchmarks, and challenges.
-
A Survey on Latent Reasoning
A survey that organizes latent reasoning methods into vertical recurrence, horizontal recurrence, and infinite-depth diffusion, arguing that silent reasoning can beat explicit chain-of-thought.
-
PREMISE: Scalable and Strategic Prompt Optimization for Efficient Mathematical Reasoning in Large Models
The paper reports that a concise optimized prompt reduces reasoning tokens by up to 87.5% while roughly preserving accuracy on GSM8K, SVAMP, and MATH-500.
-
Towards Concise and Adaptive Thinking in Large Reasoning Models: A Survey
A comprehensive review that categorizes methods for shortening and adaptively triggering chain-of-thought reasoning in large language models.
Discussion (0). Sign in to comment.