Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

Implicit Language Models are RNNs: Balancing Parallelization and Expressivity

T0 review · 3 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that implicit SSMs—state-space models whose transformations are iterated to a fixed point—implement the non-linear, non-diagonal state transitions of RNNs, and shows they solve state-tracking problems that explicit SSMs…

desk verdict Empirically strong, theoretically overstated: the S5 and 1.3B pretraining results are real news, but the 'are RNNs' claim outruns what the theorem and the approximate fixed-point solver actually support. read the letter →

arxiv 2502.07827 v3 pith:OXGLJCVN submitted 2025-02-10 cs.LG cs.AI

Mark Schöne , Babak Rahmani , Heiner Kremer , Fabian Falck , Hitesh Ballani , Jannes Gladrow This is my paper · ORCID
classification cs.LGcs.AI MSC 68T0768Q45
keywords implicitmodelsstate-spacedeepequilibriumtransformersrecurrentneuralnetworksexpressivitystatetrackingregularlanguages
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that state-space models (SSMs), which are linear and thus limited in what they can compute, can be given the expressivity of recurrent neural networks by iterating their own transformation until it converges to a fixed point. The central theoretical result is that such an implicit SSM produces a non-linear, non-diagonal state-to-state transition map, exactly the kind of update that lets RNNs track state. Empirically, the paper finds that only approximate fixed-point convergence is needed, so training can remain largely parallel, and demonstrates that implicit SSMs solve the S5 word problem—which explicit SSMs and transformers provably cannot—and outperform their explicit counterparts on language-modeling benchmarks up to 1.3B parameters.

What carries the argument

The central mechanism is the fixed-point iteration $z^{(s)}_t = f_\theta(z^{(s-1)}_t, h^{(s-1)}_t, x_t)$ coupled to the SSM recurrence (Equations 6 and 7), which can be resolved in a simultaneous mode (parallel across tokens) or a sequential mode (constant memory for autoregressive generation). The load-bearing identity is the implicit-function-theorem Jacobian (Equation 10), which adds non-diagonal correction terms to the diagonal transition $\Lambda$ of an explicit SSM; this identity is what makes the implicit SSM behave like a non-linear RNN and is verified numerically against autograd. Training uses phantom gradients, which backpropagate only through a fixed number of steps, decoupling memory usage from the depth of self-iteration.

What would settle it

Feed a trained implicit SSM an out-of-distribution input—for example, a sequence longer than the training length packed with hard non-solvable tokens—and record the relative residual after the fixed-point iteration hits its cap. If the residual stays above the training threshold or the iteration oscillates, the fixed-point and gradient claims cease to hold in exactly the regime where the paper claims an advantage.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that an implicit SSM defined by Equations (8) and (9) implements a non-linear and non-diagonal state-to-state transition function $h^*_{t-1} \mapsto h^*_t$. Given generic weights and a non-singular Jacobian condition at the fixed point, the implicit function theorem yields a smooth map $\varphi$ with $z^*_t = \varphi(h^*_{t-1}, x_t, \theta)$, and the state-to-state Jacobian decomposes into the diagonal matrix $\Lambda$ plus two non-diagonal correction terms (Equation 10). This identity is what breaks the 'illusion of state' that limits linear SSMs: the self-iteration over $z^*_t$ allows the model to emulate an RNN's transition in the limit $s \to \infty$. The paper further claims and verifies empirically that only approximate fixed-point convergence is needed, which supports a training curriculum with bounded self-iteration and leads to state-tracking accuracy on S5 word problems with length extrapolation, plus competitive or better perplexity on the D-Pile after pretraining on 207B tokens.

Load-bearing premise

The construction assumes the fixed-point iteration actually converges to a fixed point within the allowed budget for the inputs the model sees; the paper checks only a relative residual threshold and never proves convergence or uniqueness, so a divergent or bias-prone iteration would invalidate the fixed-point interpretation and the implicit-function-theorem gradients.

Editorial extensions

If this is right

  • Implicit SSMs can solve the S5 word problem and generalize to longer sequences, a task explicit SSMs and transformers provably fail.
  • Training stays largely parallel because only a sparse subset of tokens requires full fixed-point convergence; the rest resolve in a few self-iterations.
  • Simultaneous and sequential evaluation modes match to about 97.6–97.7% token identity, enabling constant-memory autoregressive generation after parallel training.
  • At scales up to 1.3B parameters on 207B tokens, implicit models achieve lower D-Pile perplexity and higher scores on several downstream tasks than explicit counterparts.
  • The (4+1) models converge to fixed points without being explicitly constrained to do so, suggesting that approximate convergence is an emergent property of the curriculum.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I infer that the sparse-hard-token hypothesis is testable directly: measuring how often implicit models exceed their iteration budget on natural text would quantify how much of the claimed advantage comes from genuinely rare hard tokens versus model capacity.
  • Since the paper verifies the Jacobian identity only on in-distribution inputs, an editorial extension is to check Equation (10) against autograd on out-of-distribution inputs; mismatches would reveal where the implicit-function-theorem regime breaks down.
  • The duality between simultaneous and sequential modes suggests a path to hardware acceleration: if fixed-point iterations can be computed quickly in specialized hardware, the self-iteration overhead could be amortized on exactly the tokens that need it.
  • The method could be combined with mixture-of-experts or latent-space reasoning, since the fixed-point iterations act as latent thoughts; the paper mentions this as a possibility but does not test it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces 'implicit SSMs', in which the recurrence and output of an SSM are iterated in a depth loop until a fixed point is reached, and studies implicit versions of Mamba2 and Llama. It proves (Theorem 1 and Appendix A.2) that at an exact fixed point the state transition h*_{t-1} -> h*_t is generally nonlinear and non-diagonal, with the Jacobian given by Eq. (10). Empirically, the paper reports that implicit Mamba2 solves the S5 word problem with a single layer while explicit Mamba2 needs many layers and does not length-generalize; that implicit models trained on a sparse mixture of easy and hard tokens generalize out-of-distribution when only 8 training self-iterations are used; that CATBABI results improve over explicit baselines; and that implicit Llama/Mamba2 models pretrained up to 1.3B parameters on 207B tokens of D-Pile obtain lower perplexity and better downstream accuracy than explicit counterparts. The paper also reports test-time length extrapolation and a simultaneous/sequential mode agreement of 97.6-97.7% token match.

Significance. If the fixed-point convergence issue can be resolved, this is an important contribution: it gives a concrete architecture that lifts the TC0-style expressiveness bound of transformers and SSMs while retaining parallel training, and it provides evidence that sparse iterative depth is sufficient on natural language. The synthetic word-problem experiments are convincing, with 10 seeds and confidence intervals, and the numerical Jacobian check in Figure 7 supports Eq. (10) at exact fixed points. The release of the code is a plus. The main caveat is that the theoretical interpretation and the deployed approximation are connected by a convergence assumption that the manuscript does not yet substantiate; until that is verified, the strongest claims ('implicit SSMs implement the non-linear state-transitions of RNNs') remain conditional on a property that is only checked through a loose relative-difference criterion.

major comments (3)
  1. [Section 3.1; Theorem 1; Appendix A.1; Appendix D.3; Table 6] The central claim that implicit SSMs 'implement the non-linear state-transitions of RNNs' depends on the model actually solving Equations (8)-(9), i.e., on convergence to an exact fixed point. Yet the paper only stops self-iteration when the relative consecutive difference |z_t^(s) - z_t^(s-1)| / |z_t^(s-1)| falls below epsilon = 0.05 (language models) or 0.01 (state tracking), with iteration caps at test time set to four times the training cap (Appendix A.1 and Appendix D.3). Table 6 shows that even in-distribution final relative differences are 0.013-0.048, not zero, and no convergence diagnostics are reported for the out-of-distribution regimes where the method claims advantage (length extrapolation in Figure 4 and Table 7, and the p=0.5 tests). A small consecutive difference does not imply closeness to a fixed point, and for non-contractive maps there is no guarantee of convergence or uniqueness. If the deployed iteration has not converged, then Eq. (10) and the implicit-function-theorem gradient do not describe the model actually being evaluated. I request that the authors report the true fixed-point residual |z^(s) - f(z^(s))|, the fraction of tokens that hit the iteration cap, and iteration counts separately for in-distribution and out-of-distribution test sets, and that they re-run the main out-of-distribution experiments with a substantially tighter tolerance and a larger cap to show that the conclusions are stable.
  2. [Table 1; Section 5; Appendix D.3] The language-model results in Table 1 are single runs with no uncertainty estimates, and the baselines are almost entirely reproduced in-house (Mamba2* and Llama-dagger) rather than compared against externally reported numbers for the same data. Many of the downstream differences that drive the 'outperform' claim are on the order of 1-3 accuracy points, which can easily be within seed or evaluation noise at this scale. At minimum, the paper should provide seed variance or a statistical comparison for the smaller models (130M and 370M), and should state publicly available baseline numbers for the same benchmark harness and prompting setup where they exist, so that the comparison is not solely self-reported.
  3. [Section 5, 'Effective Duality'; Figure 2; Table 8] The simultaneous/sequential mode equivalence is a distinctive claim, but the empirical support is only a 97.6-97.7% token-match rate on finite test sets and no analysis of the 2-3% mismatches. Because fixed points may be non-unique or may not be reached under the iteration cap, mismatches could be concentrated in tokens where the model is not at a fixed point, which would weaken the 'functionally equivalent' interpretation. I ask for a breakdown of the mismatched tokens (e.g., their fixed-point residuals, convergence step counts, and whether the two modes' probabilities are close for those tokens) and a per-token perplexity difference distribution rather than a single aggregate match rate.
minor comments (6)
  1. [Appendix A.2] The proof of Theorem 1 uses informal phrases such as 'generic weights' and 'by extension' without a precise genericity statement; please make the assumption explicit, and state whether the non-singularity of I - df/dz is verified numerically for the trained models used in Figure 7.
  2. [Equation (5)] The notation 'd dPhi / dtheta' in Eq. (5) appears to be a typo; it should likely be 'dPhi / dtheta' or 'd Phi / d theta'.
  3. [Table 5] Task 11 is labeled 'Basic conference'; in the bAbI literature this task is usually called 'basic coreference'.
  4. [Appendix A.1 and Appendix D.3] The paper repeatedly says 'only approximate fixed-point convergence suffices,' but there is no systematic ablation over the tolerance epsilon; adding one (e.g., epsilon in {0.01, 0.05, 0.1}) would substantiate this claim.
  5. [Table 7] Table 7 notes that the original Mamba2 was trained on length 8192 while all other models were trained on 2048; this caveat should be stated in the main text near Figure 4, since it materially affects the length-extrapolation comparison.
  6. [Section 5] The claim of 'largest implicit model trained to date' is presented without a comparison or citation; please provide a reference or clarify the exact comparison set.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central theoretical claim follows from the architecture's fixed-point equations, and the main empirical results are evaluated against external benchmarks or held-out test sets.

full rationale

The paper's headline theoretical claim (Theorem 1) is that the implicit SSM defined by Eqs. (8)–(9) has a non-linear, non-diagonal state-to-state transition h*(t-1) -> h*(t). The proof in Appendix A.2 derives this from the fixed-point equation z*_t = f_theta(z*_t, h*_{t-1}, x_t) using the implicit function theorem; the conclusion is a mathematical consequence of the non-linear network f_theta and the coupling through Lambda(z*_t, x_t) and u(z*_t, x_t), not an assumption of the desired result. No fitted parameter is renamed as a prediction: the S5 and CATBABI results are comparisons against explicit Mamba2/Transformer baselines, the D-Pile perplexities are measured on a held-out test split, and the length-extrapolation and simultaneous/sequential-mode token-match results are out-of-distribution checks not used to fit constants. The stopping tolerance (epsilon = 0.01/0.05) and iteration caps (24/32) are validation-selected hyperparameters, but the headline achievements (e.g., solving S5, lower perplexity than explicit baselines, 97.6% mode-match) are not predictions of those hyperparameter values. The only overlapping-author citation, Brunner et al. (2025) on photonic hardware, appears in the introduction and discussion as motivation and is not load-bearing for any technical claim. The proof does contain informal steps ('generic weights', 'by extension'), and convergence is only checked via a relative-difference stopping rule rather than a proven fixed-point theorem; these are rigor and correctness risks, not circularity. The derivation chain is therefore self-contained against external evidence, and no circular step is exhibited.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several unproven but standard assumptions: the convergence of the fixed-point iteration (only checked empirically), the generic non-linearity of the network, and prior complexity-theory results used to frame the hardness of the S5 word problem. The training hyperparameters (epsilon, iteration caps, phantom-gradient settings) are chosen by hand or via small-scale validation, and are not constants fitted to the headline benchmarks.

free parameters (3)
  • Fixed-point convergence tolerance epsilon = 0.05 for language models; 0.01 for state-tracking models
    Defines 'approximate fixed-point convergence' on which the whole training curriculum and the claim that approximate convergence suffices are based (Appendix A.1, D.3). Chosen by hand.
  • Maximum free-phase self-iterations = 24 (Mamba2) / 32 (Llama)
    Set based on 130M validation runs (Figure 14b); controls compute and the point at which the model is said to have converged at test time.
  • Phantom gradient steps k and smoothing lambda = k = 1 or 4; lambda = 0.5 (CatbAbI)
    Truncation and smoothing of the implicit-gradient estimator (Eq. 5) are hyperparameters that affect training stability and the reported results.
assumptions (4)
  • domain assumption The fixed-point iteration z(s+1) = f_theta(z(s), h, x) converges to a fixed point for the inputs encountered, within the allowed budget.
    Required for the implicit function theorem (Eq. 4) and for the claim that self-iteration implements RNN-like transitions; only checked empirically via relative difference (Appendix A.1), not proven. This is the load-bearing premise.
  • domain assumption With generic weights the network f_theta is non-linear and has non-diagonal Jacobian.
    Used in the proof of Theorem 1 (Appendix A.2) to conclude the state-to-state transition is non-linear and non-diagonal; the argument is informal ('in general linear position').
  • standard math Implicit Function Theorem.
    Standard theorem used to differentiate through the fixed point (Section 2.3, Eq. 4).
  • standard math Transformers and linear SSMs are TC^0-bounded and the S5 word problem is NC^1-complete (Barrington).
    Used in Section 2.2 and Appendix B to motivate the hardness of state tracking; taken from cited prior work (Merrill et al., 2022; 2024; Barrington, 1989).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Implicit Language Models are RNNs: Balancing Parallelization and Expressivity." pith.science (2026). https://pith.science/paper/OXGLJCVN

@misc{pith2026250207827,
  author       = {Pith},
  title        = {Pith review of: Implicit Language Models are RNNs: Balancing Parallelization and Expressivity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OXGLJCVN}},
  note         = {Machine review of arXiv:2502.07827}
}
read the original abstract

State-space models (SSMs) and transformers dominate the language modeling landscape. However, they are constrained to a lower computational complexity than classical recurrent neural networks (RNNs), limiting their expressivity. In contrast, RNNs lack parallelization during training, raising fundamental questions about the trade off between parallelization and expressivity. We propose implicit SSMs, which iterate a transformation until convergence to a fixed point. Theoretically, we show that implicit SSMs implement the non-linear state-transitions of RNNs. Empirically, we find that only approximate fixed-point convergence suffices, enabling the design of a scalable training curriculum that largely retains parallelization, with full convergence required only for a small subset of tokens. Our approach demonstrates superior state-tracking capabilities on regular languages, surpassing transformers and SSMs. We further scale implicit SSMs to natural language reasoning tasks and pretraining of large-scale language models up to 1.3B parameters on 207B tokens representing, to our knowledge, the largest implicit model trained to date. Notably, our implicit models outperform their explicit counterparts on standard benchmarks. Our code is publicly available at http://github.com/microsoft/implicit_languagemodels .

Figures

Figures reproduced from arXiv: 2502.07827 by the authors.

Figure 1
Figure 1. Top Left: Minimum layers required to solve the S5 word problem, a theoretically hard formalization of state tracking, for different sequence lengths. Top Right: Length generalization for Mamba2 and our implicit Mamba2 trained on L = 32 and extrapolated up to L = 128. Center: Down￾stream task accuracy for a range of self-iterations at test￾time for our implicit 1.3B parameter Mamba2 and Llama3 models. Dashed lines in… view at source ↗
Figure 2
Figure 2. A: The simultaneous mode self-iterates the entire sequence such that trajectories interact during convergence. It exploits the parallelism of the backbone model. B: The sequential mode iterates each token individually. Only converged hidden states or kv-caches are passed on. This mode is used for generation. C: Difference in perplexity between the two modes for our 1.3B implicit models. Theorem 1. An implicit SSM de… view at source ↗
Figure 3
Figure 3. All models were trained and evaluated on sequences of length [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Length extrapolation performance of the per token [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Fixed-point iteration and phantom gradients: A neural network is iterated until convergence in the forward pass. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: The simultaneous mode exploits the parallelism of state-space models or transformers, while the sequential mode [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: State-to-state Jacobian according to Equation ( [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Examples of word problems with mixed solvable and [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Comparison of implicit Mamba2, unrolled Mamba2, and Mamba2 for [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Hyperparameter sweeps for explicit Mamba2 models over batch sizes 128, 256, layers 1,2,3 and various learning [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Hyperparameter sweeps for implicit Mamba2 models over batch sizes 128, 256, layers 1,2,3 and various learning [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Hyperparameter sweeps for implicit Mamba2 models over batch sizes 128, 256, layers 1,2,3 and various learning [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Small-scale reasoning advantages of implicit SSMs compared to explicit SSMs on the [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: Left: Impact of Bounded Phase (4+1) Training Duration on Model Performance: A comparison of perplexity on the validation split of the D-PILE obtained by 130M Mamba2 models trained with 70%, 80%, and 90% bounded phase durations, with a batch size of 0.5M tokens. Right:…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MesaNet: Sequence Modeling by Locally Optimal Test-Time Training

    cs.LG 2025-06 conditional novelty 7.0 of 10

    MesaNet uses conjugate-gradient-optimal test-time regression in a chunkwise-parallelizable recurrent layer, achieving strong language modeling and benchmark performance at up to 1B scale.

  2. ATLAS: Learning to Optimally Memorize the Context at Test Time

    cs.CL 2025-05 conditional novelty 5.0 of 10

    A new recurrent memory module (Atlas) that optimizes its memory over a window of past tokens and uses Muon-style second-order updates, reporting gains over Transformers and existing linear RNNs on long-context tasks.

Reference graph

Works this paper leans on

18 extracted references · 10 canonical work pages · cited by 2 Pith papers

  1. [1]

    there exists an identity element e ∈ M with e ◦ m = m ◦ e = m for all m ∈ M

    work page

  2. [2]

    (m1 ◦ m2) ◦ m3 = m1 ◦ (m2 ◦ m3) for all m1, m2, m3 ∈ M

    the operation ◦ is associative, i.e. (m1 ◦ m2) ◦ m3 = m1 ◦ (m2 ◦ m3) for all m1, m2, m3 ∈ M. Straight forward examples for monoids are natural, rational or real numbers with multiplication, or strings with string concatenation. Since monoid are associative, we can simplify notation and write m ◦ m = m2, and so on for all powers k ∈ N. Definition 3 (Aperio...

    work page 1989

  3. [7]

    Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A

    URL https://proceedings.mlr.press/ v235/merrill24a.html. Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018. Omlin, C. W. and Giles, C. Extraction of rules from discrete-time recurrent neural networks. Neural Networks, 9(1):41–5...

    arXiv 2018

  4. [9]

    Snell, C

    URL https://openreview.net/forum? id=Ai8Hw3AXqks. Snell, C. V ., Lee, J., Xu, K., and Kumar, A. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations,

    work page

  5. [10]

    Soulos, P., Terzic, A., Hersche, M., and Rahimi, A

    URL https://openreview.net/forum? id=4FWAwZtd2n. Soulos, P., Terzic, A., Hersche, M., and Rahimi, A. Recur- rent transformers trade-off parallelism for length gen- eralization on regular languages. In The First Work- shop on System-2 Reasoning at Scale, NeurIPS’24 ,

    work page

  6. [11]

    Strobl, L., Merrill, W., Weiss, G., Chiang, D., and Angluin, D

    URL https://openreview.net/forum? id=6PjZA4Jvge. Strobl, L., Merrill, W., Weiss, G., Chiang, D., and Angluin, D. What formal languages can transformers express? a survey. Transactions of the Association for Computa- tional Linguistics, 12:543–561, 2024. 12 Implicit Language Models are RNNs: Balancing Parallelization and Expressivity Tay, Y ., Dehghani, M....

    work page 2024

  7. [12]

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E., Azhar, F., et al

    URL https://openreview.net/forum? id=E9dH0BP5VW. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.13971, 2023. Vafa, K., Chen, J. Y ., Rambachan, A., Kleinberg, J., and Mullainathan, S. Ev...

    arXiv 2023

  8. [13]

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q

    URL https://openreview.net/forum? id=aVK4JFpegy. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V ., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022. Weiss, G., Goldberg, Y ., and Yahav, E. Thinking like transformers. In ...

    work page 2022

Show all 18 references
  1. [14]

    Weston, J., Bordes, A., Chopra, S., Rush, A

    URL https://proceedings.mlr.press/ v139/weiss21a.html. Weston, J., Bordes, A., Chopra, S., Rush, A. M., Van Merri¨enboer, B., Joulin, A., and Mikolov, T. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698, 2015. Yang, L., L...

    2015 arXiv

  2. [17]

    Which density of entities is required to learn the intrinsic algorithm of the state tracking task?

  3. [18]

    We refer to tokens from a non-solvable group as non-solvable tokens, and to tokens from a star free language as solvable tokens

    How many self-iterations do our implicit models need to conduct to learn the intrinsic algorithm? Legend solvable token non-solvable token Star-free language NC -hard language1 Sparse NC -hard language1 Figure 8: Examples of word problems with mixed solvable and non-solvable t...

    2024

  4. [2019]

    Deletang, G., Ruoss, A., Grau-Moya, J., Genewein, T., Wen- liang, L

    URL https://openreview.net/forum? id=HyzdRiR9Y7. Deletang, G., Ruoss, A., Grau-Moya, J., Genewein, T., Wen- liang, L. K., Catt, E., Cundy, C., Hutter, M., Legg, S., Veness, J., and Ortega, P. A. Neural networks and the chomsky hierarchy. In The Eleventh International Confer- e...

    2023

  5. [2021]

    Schmidhuber, J

    URL https://openreview.net/forum? id=TuK6agbdt27. Schmidhuber, J. Self-delimiting neural networks. (arXiv:1210.0118), September 2012. doi: 10.48550/ arXiv.1210.0118. URL http://arxiv.org/abs/ 1210.0118. arXiv:1210.0118 [cs]. Sch¨utzenberger, M. On finite monoids having only tr...

    2012 doi arXiv

  6. [2022]

    Guan, L., Valmeekam, K., Sreedharan, S., and Kambham- pati, S

    URL https://openreview.net/forum? id=uYLFoz1vlAC. Guan, L., Valmeekam, K., Sreedharan, S., and Kambham- pati, S. Leveraging pre-trained large language models to construct and utilize world models for model-based task planning. In Advances in Neural Information Pro- cessing Sys...

    2023 arXiv

  7. [2023]

    URL https://openreview.net/forum? id=DeG07_TcZvT. Lim, Y . H., Zhu, Q., Selfridge, J., and Kasim, M. F. Parallelizing non-linear sequential models over the se- quence length. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: //openreview.n...

    2024

  8. [2024]

    Merrill, W., Weiss, G., Goldberg, Y ., Schwartz, R., Smith, N

    URL https://openreview.net/forum? id=NjNGlPh8Wh. Merrill, W., Weiss, G., Goldberg, Y ., Schwartz, R., Smith, N. A., and Yahav, E. A formal hierarchy of rnn architec- tures. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 443–459,...

    2020

  9. [2025]

    URL https://openreview.net/forum? id=UvTo3tVBk2. Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. Gu, A., Goel, K., and Re, C. Efficiently modeling long sequences with structured state spaces. In In- te...

    2023 arXiv

  10. [5582]

    Elman, J

    PMLR, 2022. Elman, J. L. Distributed representations, simple recurrent networks, and grammatical structure. Machine learning, 7:195–225, 1991. 10 Implicit Language Models are RNNs: Balancing Parallelization and Expressivity Gao, L., Biderman, S., Black, S., Golding, L., Hoppe,...

    2022 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.