REVIEW 18 cited by
Reasoning with Latent Thoughts: On the Power of Looped Transformers
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Large language models have shown remarkable reasoning abilities and scaling laws suggest that large parameter count, especially along the depth axis, is the primary driver. In this work, we make a stronger claim -- many reasoning problems require a large depth but not necessarily many parameters. This unlocks a novel application of looped models for reasoning. Firstly, we show that for many synthetic reasoning problems like addition, $p$-hop induction, and math problems, a $k$-layer transformer looped $L$ times nearly matches the performance of a $kL$-layer non-looped model, and is significantly better than a $k$-layer model. This is further corroborated by theoretical results showing that many such reasoning problems can be solved via iterative algorithms, and thus, can be solved effectively using looped models with nearly optimal depth. Perhaps surprisingly, these benefits also translate to practical settings of language modeling -- on many downstream reasoning tasks, a language model with $k$-layers looped $L$ times can be competitive to, if not better than, a $kL$-layer language model. In fact, our empirical analysis reveals an intriguing phenomenon: looped and non-looped models exhibit scaling behavior that depends on their effective depth, akin to the inference-time scaling of chain-of-thought (CoT) reasoning. We further elucidate the connection to CoT reasoning by proving that looped models implicitly generate latent thoughts and can simulate $T$ steps of CoT with $T$ loops. Inspired by these findings, we also present an interesting dichotomy between reasoning and memorization, and design a looping-based regularization that is effective on both fronts.
Forward citations
Cited by 18 Pith papers
-
Operational Proto-Introspection in Looped Language Models: Process-Quality Taps, Executable Branching, and the Readout-Control Boundary
Strictly pre-answer hidden states of a looped transformer add significant AUROC over surface shortcuts for predicting correctness, and the readout yields decision-level gains but no generative control.
-
DiscoLoop: Looping Discrete Embeddings and Continuous Hidden States for Multi-hop Reasoning
DiscoLoop adds a discrete embedding channel to looped transformers to fix representational misalignment in two-hop reasoning, yielding near-perfect accuracy on synthetic tasks and better pretraining loss on real data.
-
Beyond Test-Time Memory: State-Space Optimal Control for LLM Reasoning
A symplectic LQR layer inserted as an adapter into pretrained LLMs yields large gains on MATH-500, AMC and AIME by solving a latent optimal-control problem at inference time.
-
Turbo Connection: Reasoning as Information Flow from Higher to Lower Layers
An architecture that routes hidden states from higher to lower layers between consecutive tokens improves reasoning accuracy and length generalization in fine-tuned LLMs.
-
Scaling Latent Reasoning via Looped Language Models
Looped language models with latent iterative computation and entropy-regularized depth allocation achieve performance matching up to 12B standard LLMs through superior knowledge manipulation.
-
Mobius Learning: Cyclic Depth Folding in Transformers
Möbius Learning, which cyclically shifts block order across data streams, achieves lower validation loss than fixed-order looped training at loop depths 6, 10, and 15 in a 124M-parameter GPT-2 experiment.
-
DeepLoop: Depth Scaling for Looped Transformers
Looped Transformers need residual-scaling exponent p=1/2 instead of DeepNorm's 1/4 when shared blocks are revisited with aligned visit-wise gradients.
-
Adaptive Depth in Looped Transformers: Diagnosing Learned Halting Gates and Trajectory Readouts
In looped transformers, halting-gate failures come mainly from how gate training reshapes the trajectory; fixed-prior depth supervision plus simple confidence readouts yields better accuracy per unit of compute.
-
ELT: Elastic Looped Transformers for Visual Generation
Weight-shared looped transformers trained with intra-loop self-distillation match MaskGIT-class FID/FVD at roughly 4x fewer parameters and support any-time inference across loop counts.
-
Prioritize the Process, Not Just the Outcome: Rewarding Latent Thought Trajectories Improves Reasoning in Looped Language Models
RLTT, a method that credits every latent reasoning loop during RL, improves math reasoning accuracy over GRPO by 10–18 percentage points on a 2.6B looped language model.
-
LaRe: Latent Refocusing for Multimodal Reasoning
LaRe performs iterative visual refocusing in latent space and reports accuracy gains with fewer tokens, but its main experiments compare against baselines trained with less data.
-
Fast and Simplex: 2-Simplicial Attention in Triton
2-simplicial attention, implemented in Triton with a sliding window, is claimed to yield a steeper loss-versus-parameters scaling exponent than dot-product attention on math and reasoning benchmarks.
-
Energy-Based Transformers are Scalable Learners and Thinkers
Energy-Based Transformers learn to predict by gradient-descent minimization of a learned energy function, and the paper reports faster pretraining scaling and inference-time thinking gains over Transformer++ and Diffu...
-
SHIFT: Self-reconstruction Harnesses Implicit Fine-grained Thinking for Retrieval
SHIFT trains LLM retrievers to reason through latent continuous tokens and reconstruct explicit reasoning traces, improving reasoning-intensive retrieval.
-
HALO: Hybrid Adaptive Latent Reasoning for Language Models
Selective second-stage latent refinement on budgeted scored tokens beats uniform fixed-1 and fixed-2 refinement on frozen Phi-4-mini for average MMLU-Pro/GPQA score at lower average applied refine steps.
-
Implicit Reasoning in Large Language Models: A Comprehensive Survey
A survey organizing implicit (silent) reasoning in LLMs into three execution paradigms, plus evidence, benchmarks, and challenges.
-
A Survey on Latent Reasoning
A survey that organizes latent reasoning methods into vertical recurrence, horizontal recurrence, and infinite-depth diffusion, arguing that silent reasoning can beat explicit chain-of-thought.
-
Towards Concise and Adaptive Thinking in Large Reasoning Models: A Survey
A comprehensive review that categorizes methods for shortening and adaptively triggering chain-of-thought reasoning in large language models.
Discussion (0). Sign in to comment.