REVIEW 9 cited by
Looped Transformers are Better at Learning Learning Algorithms
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Transformers have demonstrated effectiveness in in-context solving data-fitting problems from various (latent) models, as reported by Garg et al. However, the absence of an inherent iterative structure in the transformer architecture presents a challenge in emulating the iterative algorithms, which are commonly employed in traditional machine learning methods. To address this, we propose the utilization of looped transformer architecture and its associated training methodology, with the aim of incorporating iterative characteristics into the transformer architectures. Experimental results suggest that the looped transformer achieves performance comparable to the standard transformer in solving various data-fitting problems, while utilizing less than 10% of the parameter count.
Forward citations
Cited by 9 Pith papers
-
When Does Recurrence Become an Algorithm? Convergence Selection in Weight-Tied Looped Transformers
Weight-tied looped transformers on group prefix products implement a linear computation frontier whose speed matches the training loop budget, and a new convergence-time instrument reveals it.
-
Beyond Test-Time Memory: State-Space Optimal Control for LLM Reasoning
A symplectic LQR layer inserted as an adapter into pretrained LLMs yields large gains on MATH-500, AMC and AIME by solving a latent optimal-control problem at inference time.
-
Scaling Latent Reasoning via Looped Language Models
Looped language models with latent iterative computation and entropy-regularized depth allocation achieve performance matching up to 12B standard LLMs through superior knowledge manipulation.
-
Adaptive Depth in Looped Transformers: Diagnosing Learned Halting Gates and Trajectory Readouts
In looped transformers, halting-gate failures come mainly from how gate training reshapes the trajectory; fixed-prior depth supervision plus simple confidence readouts yields better accuracy per unit of compute.
-
ELT: Elastic Looped Transformers for Visual Generation
Weight-shared looped transformers trained with intra-loop self-distillation match MaskGIT-class FID/FVD at roughly 4x fewer parameters and support any-time inference across loop counts.
-
Latent Reasoning in TRMs is Secretly a Policy Improvement Operator
Recursive reasoning in TRMs is reinterpreted as policy improvement, and a stepwise denoising supervision scheme (DIS) cuts forward passes 18× while improving small-model ARC scores.
-
Fast and Simplex: 2-Simplicial Attention in Triton
2-simplicial attention, implemented in Triton with a sliding window, is claimed to yield a steeper loss-versus-parameters scaling exponent than dot-product attention on math and reasoning benchmarks.
-
Skip a Layer or Loop it? Test-Time Depth Adaptation of Pretrained LLMs
Pretrained LLM layers can be skipped/repeated per input to build custom paths, but the search uses ground-truth answers, so the accuracy gains are fitted, not predicted.
-
A Survey on Large Language Models for Mathematical Reasoning
Recent advances in LLM mathematical reasoning are organized into comprehension and generation phases, covering methods from prompting to test-time scaling.
Discussion (0). Sign in to comment.