arxiv: 2604.07822 · v1 · submitted 2026-04-09 · 💻 cs.CL · cs.AI· cs.LG

Recognition: 2 theorem links

· Lean Theorem

Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers

Harsh Kohli, Huan Sun, Srinivasan Parthasarathy, Yuekun Yao

Pith reviewed 2026-05-10 17:46 UTC · model grok-4.3

classification 💻 cs.CL cs.AIcs.LG

keywords implicit reasoningrecurrent-depth transformerscompositional generalizationsystematic generalizationdepth extrapolationgrokkingtransformer models

0 comments

The pith

Recurrent-depth transformers achieve systematic generalization and depth extrapolation in implicit reasoning by iterating over shared layers.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests whether transformers can combine stored knowledge or rules inside one forward pass, an ability called implicit reasoning. Vanilla models often fail when required to form new combinations of facts or to chain reasoning steps longer than those seen in training. Recurrent-depth transformers reuse the same layers multiple times within a single pass and succeed at both challenges. Systematic generalization to never-before-combined knowledge appears after a three-stage grokking process during training, while depth extrapolation to longer chains is enabled simply by running more iterations at inference time. These results matter because they point to a route for building models that compose knowledge more flexibly without adding parameters or training data for every new depth.

Core claim

Recurrent-depth transformers enable effective implicit reasoning on compositional tasks by allowing iterative computation over the same transformer layers. While vanilla transformers struggle with systematic generalization to unseen knowledge combinations and with depth extrapolation beyond training depths, recurrent-depth models succeed. Systematic generalization emerges via a three-stage process of memorization, in-distribution generalization, and finally out-of-distribution systematic generalization. Depth extrapolation is unlocked by scaling the number of recurrent iterations at inference time, with training strategies influencing the extent of generalization, though excessive recurrence

What carries the argument

Recurrent-depth transformers, which reuse the same set of transformer layers for multiple iterations inside one forward pass to support iterative computation.

If this is right

Models trained only on shallow reasoning depths can handle deeper compositions when more recurrent iterations are used at test time.
Systematic generalization to novel knowledge combinations appears after the model passes through memorization and in-distribution stages.
Training choices such as the recurrence depth used during training control how far extrapolation extends.
Excessive recurrence produces overthinking that degrades accuracy and caps generalization at very high depths.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same iterative reuse might let large pre-trained models handle longer multi-hop chains on natural text without full retraining.
The three-stage grokking pattern could appear as measurable shifts in attention or hidden-state similarity during training on real data.
Overthinking might be mitigated by learned stopping criteria or depth-dependent regularization that the paper does not explore.

Load-bearing premise

The controlled results from models trained from scratch on synthetic tasks will transfer to the implicit reasoning behavior of large pre-trained language models on natural language.

What would settle it

Train a recurrent-depth transformer on synthetic compositional questions with a maximum of five reasoning hops and then measure whether raising the number of inference-time iterations produces accurate answers on ten-hop questions that require both deeper chaining and novel combinations of training knowledge.

Figures

Figures reproduced from arXiv: 2604.07822 by Harsh Kohli, Huan Sun, Srinivasan Parthasarathy, Yuekun Yao.

**Figure 1.** Figure 1: Recurrent depth model architecture. The transformer block is repeated R times. The embedding layer and language model head (LM Head) have tied weights. In our experiments, we use a simple looped transformer similar to Saunshi et al. (2025) without design elements such as input injection, gated halting, and middle looping. of John Lennon is Yoko Ono is only stored in shallow layers, deeper layers cannot acc… view at source ↗

**Figure 2.** Figure 2: Illustration of systematic and extrapola [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Figure 3: Accuracy curves for recurrent-depth models across training epochs and wall [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

**Figure 4.** Figure 4: Accuracy of predicting bridge and target entities using logit lens at corresponding [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

**Figure 5.** Figure 5: Accuracy of recurrent-depth models on multi-hop composition, trained under var [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

**Figure 6.** Figure 6: Cumulative gradient updates required to first generalize to each hop complexity. Phase transition from prolonged training to rapid learning. We observe that models require a prolonged training phase to acquire low-hop tasks, after which they rapidly generalize to much more complex samples (Figure 6). We illustrate this for the model trained with dynamic recurrence by plotting training steps against th… view at source ↗

**Figure 7.** Figure 7: Accuracy of models trained with fixed recurrence [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗

**Figure 8.** Figure 8: Average logit margin across recurrent iterations for fixed-recurrence ( [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗

**Figure 9.** Figure 9: Average recurrent iterations using adaptive halting vs. sample complexity. Adaptive halting improves inference efficiency. We flexibly halt recurrence in order to allocate compute proportionate to input complexity. Prior work (Geiping et al., 2025) proposes adaptive halting based on the output distribution pt(· | x), terminating when the change between successive iterations becomes small, measured by KL… view at source ↗

**Figure 10.** Figure 10: Comparison of adaptive halting based on KL-divergence and entropy (ours) [PITH_FULL_IMAGE:figures/full_fig_p010_10.png] view at source ↗

**Figure 11.** Figure 11: Results with default initialization for training recurrences [PITH_FULL_IMAGE:figures/full_fig_p015_11.png] view at source ↗

**Figure 12.** Figure 12: Five random-seed runs for the R = 5 model with default initialization. To further examine this instability, we train the r = 5 model with five different random seeds. As shown in [PITH_FULL_IMAGE:figures/full_fig_p016_12.png] view at source ↗

**Figure 13.** Figure 13: Generalization ratio with changing model size and train recurrence. [PITH_FULL_IMAGE:figures/full_fig_p016_13.png] view at source ↗

**Figure 14.** Figure 14: Results for Seed 1. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 R=1 (ID 3-hop) R=2 (ID 6-hop) R=3 (ID 9-hop) R=4 (ID 10-hop) 2 5 8 11 14 17 20 23 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 R=5 (ID 12-hop) 2 5 8 11 14 17 20 23 R=6 (ID 13-hop) 2 5 8 11 14 17 20 23 R=7 (ID 14… view at source ↗

**Figure 15.** Figure 15: Results for Seed 2. We similarly plot the performance of the dynamic recurrence model across three different seeds and observe a generally high ID generalization and OOD extrapolation. Across all 17 [PITH_FULL_IMAGE:figures/full_fig_p017_15.png] view at source ↗

**Figure 16.** Figure 16: Dynamic recurrence with 3 difference random seed initializations. [PITH_FULL_IMAGE:figures/full_fig_p018_16.png] view at source ↗

**Figure 17.** Figure 17: Apparent deep compositional generalization in the pre-permutation dataset. [PITH_FULL_IMAGE:figures/full_fig_p018_17.png] view at source ↗

**Figure 18.** Figure 18: Activation-patching analysis on a 60-hop example. Each cell shows the change [PITH_FULL_IMAGE:figures/full_fig_p019_18.png] view at source ↗

read the original abstract

We study implicit reasoning, i.e. the ability to combine knowledge or rules within a single forward pass. While transformer-based large language models store substantial factual knowledge and rules, they often fail to compose this knowledge for implicit multi-hop reasoning, suggesting a lack of compositional generalization over their parametric knowledge. To address this limitation, we study recurrent-depth transformers, which enables iterative computation over the same transformer layers. We investigate two compositional generalization challenges under the implicit reasoning scenario: systematic generalization, i.e. combining knowledge that is never used for compositions during training, and depth extrapolation, i.e. generalizing from limited reasoning depth (e.g. training on up to 5-hop) to deeper compositions (e.g. 10-hop). Through controlled studies with models trained from scratch, we show that while vanilla transformers struggle with both generalization challenges, recurrent-depth transformers can effectively make such generalization. For systematic generalization, we find that this ability emerges through a three-stage grokking process, transitioning from memorization to in-distribution generalization and finally to systematic generalization, supported by mechanistic analysis. For depth extrapolation, we show that generalization beyond training depth can be unlocked by scaling inference-time recurrence, with more iterations enabling deeper reasoning. We further study how training strategies affect extrapolation, providing guidance on training recurrent-depth transformers, and identify a key limitation, overthinking, where excessive recurrence degrades predictions and limits generalization to very deep compositions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

Recurrent-depth helps on synthetic composition tasks with a clear grokking pattern and inference scaling, but the motivating link to pre-trained LLMs stays untested.

read the letter

The paper's core finding is that recurrent-depth transformers, which reuse the same layers iteratively, handle two generalization problems better than standard ones on synthetic tasks: combining unseen rule combinations and reasoning to greater depths than seen in training. They track a three-stage process for the first—memorization, then in-distribution success, then full systematic generalization—plus mechanistic checks, and show that simply increasing recurrence steps at inference time extends depth performance. They also flag overthinking as a limit when iterations go too far. That last part gives concrete training advice and is a useful observation on its own terms. The experiments are controlled and from-scratch, which keeps the comparisons clean. The soft spot is the gap between the opening motivation and the actual work. The abstract frames the problem around large language models failing at implicit multi-hop composition over parametric knowledge, yet every result uses small models trained on synthetic data. No fine-tuning, no natural language, no pre-trained weights. So the grokking stages and the inference-time fix are demonstrated only in the toy regime; whether they appear when the same change is made to real LLMs is left open. This is still worth referee time for groups working on architectural tweaks for reasoning. The empirical patterns are specific enough and the limitation is acknowledged, so a serious editor should send it out even if revisions will be needed to tighten the connection to the stated goal.

Referee Report

2 major / 2 minor

Summary. The paper studies implicit reasoning in transformers by introducing recurrent-depth models that perform iterative computation over shared layers. On synthetic compositional tasks trained from scratch, it claims vanilla transformers fail at systematic generalization (combining unseen knowledge compositions) and depth extrapolation (generalizing to deeper hops than trained), while recurrent-depth variants succeed: systematic generalization emerges via a three-stage grokking process (memorization to in-distribution to systematic), and depth extrapolation is unlocked by scaling inference-time recurrence steps, with additional analysis of training strategies and the overthinking limitation.

Significance. If the empirical results and mechanistic analysis hold, the work provides concrete evidence that architectural recurrence can unlock compositional generalization behaviors absent in standard transformers, including a documented grokking trajectory and a practical inference-time scaling mechanism for depth extrapolation. The controlled synthetic setup and identification of overthinking offer useful guidance for future architectural variants.

major comments (2)

[Introduction] Introduction and abstract: the opening diagnosis of transformer LLMs failing on implicit multi-hop composition over parametric knowledge is not followed by any experiments, fine-tuning, or mechanistic analysis on pre-trained models or natural-language data; all results are confined to from-scratch training on synthetic tasks, so the claimed relevance to LLM implicit reasoning rests on an untested transfer assumption.
[Experiments] Experimental sections (methods and results): the controlled studies are described at a high level without reported effect sizes, exact data splits, ablation tables, or statistical tests for the three-stage grokking trajectory and recurrence-scaling claims; this makes it difficult to assess whether post-hoc task definitions or narrow synthetic distributions drive the reported generalization advantages.

minor comments (2)

[Abstract] The abstract and conclusion could explicitly qualify that all findings are demonstrated only in the synthetic from-scratch regime to avoid overgeneralization to pre-trained LLMs.
[Methods] Notation for recurrence depth and inference-time iteration count should be unified across sections to prevent confusion between training and test-time usage.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback. We address the major comments point by point below, providing clarifications and committing to revisions that strengthen the manuscript without misrepresenting our synthetic, from-scratch experimental scope.

read point-by-point responses

Referee: [Introduction] Introduction and abstract: the opening diagnosis of transformer LLMs failing on implicit multi-hop composition over parametric knowledge is not followed by any experiments, fine-tuning, or mechanistic analysis on pre-trained models or natural-language data; all results are confined to from-scratch training on synthetic tasks, so the claimed relevance to LLM implicit reasoning rests on an untested transfer assumption.

Authors: We acknowledge that all experiments use from-scratch training on synthetic tasks. This controlled setup was deliberately chosen to isolate the mechanisms of implicit reasoning, systematic generalization, and depth extrapolation, allowing precise manipulation of knowledge compositions and reasoning depths that are infeasible to control in pre-trained LLMs. The synthetic tasks are constructed to directly instantiate the multi-hop composition challenges described in the introduction. We agree that transfer to LLMs remains an assumption at this stage. In the revision we will add an explicit limitations paragraph in the introduction and a dedicated Limitations section that states this assumption, discusses why synthetic studies provide foundational mechanistic evidence, and outlines future work on fine-tuning or analyzing pre-trained models. revision: partial
Referee: [Experiments] Experimental sections (methods and results): the controlled studies are described at a high level without reported effect sizes, exact data splits, ablation tables, or statistical tests for the three-stage grokking trajectory and recurrence-scaling claims; this makes it difficult to assess whether post-hoc task definitions or narrow synthetic distributions drive the reported generalization advantages.

Authors: We agree that greater experimental detail is needed for reproducibility and to rule out concerns about task construction or distribution narrowness. In the revised manuscript we will expand the Methods and Results sections to report: exact data generation procedures and train/validation/test splits; quantitative effect sizes (accuracy deltas with confidence intervals); full ablation tables covering training strategies, recurrence steps, and model variants; and statistical tests (e.g., paired t-tests or bootstrap resampling across random seeds) supporting the three-stage grokking trajectory and inference-time scaling results. These additions will make the robustness of the findings clearer. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical study with experimental outcomes on synthetic tasks

full rationale

The paper reports controlled experiments training models from scratch on synthetic compositional tasks to evaluate systematic generalization and depth extrapolation in recurrent-depth transformers versus vanilla transformers. Claims rest on observed training dynamics (e.g., three-stage grokking), mechanistic analysis, and inference-time scaling results rather than any mathematical derivation, equation, or parameter fit that reduces to its own inputs by construction. No self-definitional steps, fitted quantities renamed as predictions, or load-bearing self-citations appear in the methodology or results chain.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The paper is an empirical machine-learning study. No mathematical axioms, free parameters fitted inside a derivation, or newly invented entities are invoked in the abstract; claims rest on experimental outcomes from controlled training runs.

pith-pipeline@v0.9.0 · 5562 in / 1119 out tokens · 83846 ms · 2026-05-10T17:46:22.159803+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Foundation/ArithmeticFromLogic.lean reality_from_one_distinction unclear
recurrent-depth transformers... iterative computation over the same transformer layers... scaling inference-time recurrence... three-stage grokking process
IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear
Jcost uniqueness... phi-ladder... 8-tick period

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

SMolLM: Small Language Models Learn Small Molecular Grammar
cs.LG 2026-05 unverdicted novelty 7.0

A 53K-parameter model generates 95% valid SMILES on ZINC-250K, outperforming larger models, by resolving chemical constraints in fixed order: brackets first, rings second, valence last.
Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models
cs.CL 2026-05 unverdicted novelty 6.0

MELT decouples reasoning depth from memory in looped LLMs by sharing a single gated KV cache per layer and using two-phase chunk-wise distillation from Ouro, delivering constant memory use while matching or beating st...
Hyperloop Transformers
cs.LG 2026-04 unverdicted novelty 5.0

Hyperloop Transformers outperform standard and mHC Transformers with roughly 50% fewer parameters by looping a middle block of layers and applying hyper-connections only after each loop.

Reference graph

Works this paper leans on

4 extracted references · 3 canonical work pages · cited by 3 Pith papers

[1]

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al

URLhttps://openreview.net/forum?id=2edigk8yoU. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027, 2020. Jonas Geiping, Sean Michael McLeish, Neel Jain, John Kirchenba...

work page doi:10.18653/v1/2023.emnlp-main.751 2020
[2]

Prabha, D., Aswini, J., Maheswari, B., Subramanian, R

URLhttps://aclanthology.org/2025.naacl-long.420/. Brenden Lake and Marco Baroni. Generalization without systematicity: On the composi- tional skills of sequence-to-sequence recurrent networks. InInternational conference on machine learning, pp. 2873–2882. PMLR, 2018. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori...

work page doi:10.18653/v1/2023.findings-emnlp.378 2025
[3]

I ’m born in 1 8 7 1

URLhttps://aclanthology.org/2024.acl-long.550/. Yuekun Yao, Yupei Du, Dawei Zhu, Michael Hahn, and Alexander Koller. Language models can learn implicit multi-hop reasoning, but only if they have lots of training data. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Conference on Empirical M...

work page arXiv 2024
[4]

Using 3 representative tasks, Dziri et al

and a proxy for analyzing how models can learn internal mechanisms for combining facts instead of emitting long rationales in the form of CoT. Using 3 representative tasks, Dziri et al. (2023) demonstrate that transformers reduce compositional tasks to linearized subgraph matching that fails with increasing complexity. Wang et al. (2024a) show that transf...

2023