Pith. sign in

REVIEW 4 major objections 3 minor 200 references

Learning Model Successors

T0 review · 4 major / 3 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Easy-to-hard generalization becomes induction: learn a function that maps one difficulty level's model to the next, then apply it beyond what was observed.

desk verdict A serious attempt to formalize easy-to-hard generalization, with an honest proof-of-concept that is more bounded than the abstract suggests. read the letter →

arxiv 2502.00197 v2 pith:VASZAYZK submitted 2025-01-31 cs.LG stat.ML

Yingshan Chang , Yonatan Bisk This is my paper · ORCID
classification cs.LGstat.ML
keywords inductivelearningmodelsuccessorsout-of-domaingeneralizationdifficultyprogressionlengthextrapolationeasy-to-hardgracefuldegradationdyck1language
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that the common thread behind length extrapolation, logical extrapolation, and algorithmic extrapolation is induction: from finitely many training domains a learner should infer a principle that applies unboundedly along a difficulty progression. To make that precise, it defines an inductive problem as a sequence of domains ordered by a successor operation, and it defines the goal of an inductive learner as producing a model successor, a map from the hypothesis that handles difficulty level $k$ to a hypothesis that handles level $k+1$, with no access to data from level $k+1$. The paper demonstrates the idea on recognizing balanced brackets: an inductive learner trained on three model transitions, from depth 1 to depth 4, produces correct transitions for depths 4 through 52. If the claims hold, the familiar "easy-to-hard" failures of neural networks are not mainly capacity or data problems; they are a symptom of learning in the wrong space, and the remedy is to learn regularities over models rather than only over data.

What carries the argument

The central object is the model successor, $\mathrm{Ind}_k$: a function in $\mathcal{H}^{\mathcal{H}}$ that maps the base learner's hypothesis at level $k$ to a hypothesis at level $k+1$, embodying the capacity growth that the difficulty progression demands. Around it sit three pieces of machinery: the inductive problem (domains indexed by a successor $\mathrm{Succ}$, with constant difficulty gap in the limit and no simpler subsequence), the degradation metric $\mathrm{DGR}$ (a discounted sum of risks over all harder domains, used to score $\mathrm{Ind}_k$ relative to the static hypothesis), and the dyck1 realization, in which hypotheses become extracted finite automata and learning a model successor becomes a sequence-to-sequence prediction of the next automaton's added transition rules.

What would settle it

Run the model-successor pipeline on a variant of balanced-bracket recognition where the automaton occasionally gains two states at a single step instead of one; if the learned successor still extrapolates perfectly to depth 52, the constant-gap niceness condition is not load-bearing, and if it fails, the condition is doing the work.

Watch

Extended reading notes

Core claim

Formally, a sequence of domains $D_1, D_2, \ldots$ together with a data successor $\mathrm{Succ}$ is an inductive problem when it satisfies natural-number-style axioms, and the successor is required to be nice: the difficulty gap between consecutive domains converges to a constant and no subsequence has a smaller gap. Under these conditions, the inductive learner $L_{\mathrm{Ind}}$ receives the base learner's hypotheses $\hat{h}^*_1, \ldots, \hat{h}^*_k$ and outputs a model successor $\mathrm{Ind}_k \in \mathcal{H}^{\mathcal{H}}$ such that applying $\mathrm{Ind}_k$ repeatedly yields hypotheses for $D_{k+1}, D_{k+2}, \ldots$. Inductive learnability, Definition 5.1, is satisfied when $\mathrm{Ind}_k$ degrades at least $\epsilon$-more gracefully than the static hypothesis $\hat{h}^*_k$, where degradation is a discounted sum of risks over harder domains. In the dyck1 demonstration, the base hypotheses are RNNs for depths 1--4, re-expressed as extracted finite automata and then as symbol strings; a decoder-only model trained on three successor transitions $\hat{h}^*_1 \to \hat{h}^*_2$, $\hat{h}^*_2 \to \hat{h}^*_3$, $\hat{h}^*_3 \to \hat{h}^*_4$ predicts the transitions from depth 4 up to depth 52, yielding zero degradation over that range.

Load-bearing premise

Everything rests on the assumption that each later difficulty level is generated from the previous one by the same principled step and that the gap between levels settles down, so that a handful of observed transitions reveals the pattern for all later transitions.

Editorial extensions

If this is right

  • A learner can solve instances at difficulty levels it never trained on, provided the levels are ordered by a principled successor and the model progression is learnable from finitely many transitions.
  • The framework makes explicit that expressivity, learnability, and generalizability are separate questions, and that many reported extrapolation failures should be attributed to a mismatch between problem and learning paradigm rather than to model incapacity.
  • Static-hypothesis methods for out-of-domain generalization are the special case where the model successor is the identity, so inductive learning subsumes them and clarifies exactly which invariance assumption they rely on.
  • Lifelong and prospective learning can be syntactically transformed into inductive learning, but each trades different assumptions: lifelong learning needs continuing data, prospective learning needs an identifiable stochastic process, and inductive learning needs the niceness conditions on the difficulty gap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the dyck1 result is representative, any task whose hypotheses admit a symbolic or structured encoding (automata, programs, circuit graphs) may support model-successor learning by turning hypothesis transitions into sequence prediction, which is a testable recipe for arithmetic or graph algorithms.
  • The framework leaves the halting point of $\mathrm{Ind}$ unspecified; a natural extension is to learn a halting policy alongside the successor, applying $\mathrm{Ind}$ only while predicted degradation stays small.
  • The constant-gap assumption can be probed directly: on a progression whose required automaton grows by two states at a time, or accelerates, the learned successor should fail to extrapolate if the niceness conditions are doing the work.
  • Reinterpreting benchmark failures through this lens suggests that many length-generalization negative results should be re-run under inductive-learning conditions before being read as evidence about neural networks in general.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. This paper proposes a formal account of inductive generalization along a difficulty progression. It models progressions of domains as Peano structures, defines a data successor via probabilistic transducers, and imposes niceness conditions (constant difficulty gap, no simpler subsequence) intended to make the progression principled. It then defines inductive learnability as the ability of a meta-learner L_Ind to infer a model successor Ind_k that maps hypotheses at level k to hypotheses at level k+1 and degrades ϵ-more gracefully than the base hypothesis. The paper situates this inductive learning in a taxonomy with distributional-shift, lifelong, and prospective learning, and sketches practical instantiations. An appendix reports dyck1 experiments: RNNs are trained at each depth, FSAs are extracted and encoded as strings, a decoder-only language model is trained on three transitions h1→h2, h2→h3, h3→h4, and is reported to generalize to h51→h52 with DGR=0.

Significance. The formalization is a useful conceptual contribution: it gives a shared language for expressivity, learnability, and generalizability, makes explicit assumptions behind OODG, and offers a taxonomy that clarifies how inductive learning differs from lifelong and prospective learning. The paper is honest about failure modes (Table 3d), and App. B's syntactic-equivalence analysis is a thought-provoking way to relate paradigms. However, the empirical demonstration, as written, does not instantiate the unbounded success criterion of Definition 5.1, and the paper's central extrapolation claim currently rests on a finite truncation and an unproved re-representation step.

major comments (4)
  1. [App. A, Experiment 3; Definition 5.1] Definition 5.1 defines success through DGR(Ind_k,h_k) as an infinite sum over m=k+1 to ∞, but the experiment sets δ_m=1 for 4≤m≤52 and δ_m=0 otherwise (footnote 19). The reported DGR=0 is therefore only over 48 finite future levels. Because the state-name alphabet [a-zA-Z] has 52 letters, L_Ind cannot even represent a hypothesis for depth 53, so the claim of 'perfect generalization up to h_51 to h_52' is a bounded extrapolation of 48 steps and does not verify the unbounded inductive principle promised in the abstract. I recommend either weakening the claim to finite extrapolation or providing an encoding with an unbounded state-name alphabet and reporting behavior on a growing range.
  2. [App. A, Experiment 3; Definition 5.1] L_Ind is trained and evaluated on string encodings of FSAs extracted post hoc from trained RNNs, not on the actual hypotheses h_k produced by the base learner. The paper does not supply encoding and decoding maps between the hypothesis space H_k and the string space, nor a fidelity theorem showing that the extracted FSA represents the RNN's behavior on the relevant domain. The experiment therefore defines a mapping on symbolic encodings, whereas the central object Ind_k of Definition 5.1 maps hypotheses in H_k to hypotheses in H_{k+1}; as stated, that formal object is not empirically instantiated.
  3. [§3, Definitions 3.1 and 3.2] Verification of the niceness conditions is asserted rather than proved. App. A states that the PFST in Fig. A1 characterizes the difficulty gap, but it does not show that Eq. (1) holds for all k≥\bar{k}, that K(T) is minimal, or that no simpler subsequence exists; the entropy table only demonstrates monotonicity. Since the authors concede that without these restrictions the definition of D 'is inevitably vacuous', the formal guarantees of inductive learnability are currently conditional on unverified assumptions. A proof or a precise condition under which the PFST attains the bound is needed.
  4. [§3, Peano axiom 5] The 'No junk / Axiom of Induction' as stated is not the induction axiom. From D1∈A and 'every element in A can be derived via applying Succ a number of times to D1' it does not follow that A contains every element of D; for example, A={D1,D2} satisfies the stated condition for a natural-number-like progression but omits D3. The intended axiom should be closure under Succ: if A⊆D, D1∈A, and Succ(a)∈A whenever a∈A, then A=D. As written, the Peano basis for 'principled' extrapolation is logically misstated and should be corrected.
minor comments (3)
  1. [§2 and §3] The symbol D is used both for the set of all possible samples in §2 (D={Dn | n∈N}) and for the difficulty progression in §3; this is confusing and should be disambiguated, for example by using a different symbol for the set of all samples.
  2. [App. A, Table A5] When the same target continuation is written with '<ns>' in several training instances, it would help to state explicitly that '<ns>' is a placeholder for a new state name drawn from [a-zA-Z], rather than a literal state name, so the reader can see why the learned rule is state-name invariant.
  3. [References] Reference [5] is an anonymous submission marked 'under review'; in a journal submission, unpublished or anonymous references should either be replaced by a citable version or flagged more carefully, since the current citation format does not meet standard archival expectations.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: the framework is axiomatic and the dyck1 experiment performs held-out successor extrapolation; only minor motivational self-citation [26] and a finite-alphabet truncation limit the empirical scope.

full rationale

The paper does not exhibit a circular derivation. Its formal core is axiomatic: Section 3 defines an inductive problem by imposing Peano axioms and two explicit niceness properties (constant difficulty gap, no simpler subsequence), and the paper states that without such restrictions the definition 'is inevitably vacuous.' The success criterion in Def 5.1 is a relative DGR improvement, not a quantity fitted from the targets. Experiment 3 trains L_Ind on three observed transitions (h1→h2, h2→h3, h3→h4) and evaluates on held-out transitions (h4→h5 through h51→h52), so the extrapolated successors are not fitted inputs renamed as predictions. The only self-citation, [26], is motivational for the counting example and is not load-bearing for the framework or the dyck1 experiments. App. B explicitly concedes that LL, PL, and IL are syntactically equivalent and that the framework is 'a new arrangement using existing concepts,' which is an honest scope statement rather than a disguised renaming. The finite state-name alphabet and the δ_m=0 truncation beyond m=52 (App. A footnote) mean the abstract's 'unbounded' claim is not empirically established, but this is an evidence limitation, not circularity.

Assumptions & free parameters 3 free parameters · 6 assumptions · 1 invented entities

The framework arranges existing concepts (evolving hypotheses, meta-learning over models) into a new formal structure. The hand-chosen quantities are experimental configurations and an evaluation window, not parameters fitted to make a derivation work; no physical entities are introduced; the load-bearing assumptions are the regularity of the difficulty progression (Peano axioms plus niceness properties) and the faithfulness of the FSA re-representation.

free parameters (3)
  • DGR evaluation weights delta_m (Experiment 3) = delta_m = 1 for 4 <= m <= 52, 0 otherwise
    Hand-chosen in App A, Experiment 3; the DGR(Ind3, h3) = 0 result covers only the tested depth window, so the stated zero degradation is not evaluated on the infinite progression the framework claims.
  • L_Ind training configuration = hidden 64, dropout 0.1, lr 0.01, wd 0.01, batch 32, steps 300
    Hand-chosen settings for the model-successor language model (App A, Experiment 3). They support the empirical demonstration but are not derived from the formal framework.
  • Base RNN training configuration = one layer, hidden 16, batch 32, steps 15k, 5 seeds
    Hand-chosen settings for the dyck1 base learners (App A, Setup); standard small-scale settings that the authors argue are representative.
assumptions (6)
  • domain assumption The difficulty progression D with data successor Succ satisfies Peano's axioms: unique origin, closure, bijectivity, no loop, no junk.
    Section 3 defines an inductive problem this way. It restricts expected OOD instances to principled ones and is load-bearing for the whole framework.
  • domain assumption Succ satisfies niceness properties: constant difficulty gap in the limit (Def 3.1) and no simpler subsequence (Def 3.2).
    Section 3.2. Without these, the formalization is 'inevitably vacuous' per the authors, and finite observation cannot reveal the infinite pattern.
  • domain assumption Assumption 4.1: no issue with expressivity or learnability in any domain.
    Section 4, stated explicitly; the framework begins where in-domain learning is already near-perfect.
  • domain assumption Assumption 4.2: no issue with hard-to-easy generalization.
    Section 4, stated explicitly; it lets the paper conflate L(dk) with L(d<=k) and is unverified in general.
  • domain assumption Succ is realizable as a sequence of probabilistic transducers satisfying Eq. 1.
    Section 3; the entropy-based difficulty measure and the niceness properties are formalized on this representational assumption.
  • domain assumption RNNs trained on dyck1-m can be faithfully re-represented as extracted finite-state automata.
    App A, Experiment 3; the entire proof of concept depends on extraction techniques [104, 106, 169] preserving the behavior relevant to the successor pattern.
invented entities (1)
  • Model successor Ind_k
    purpose: A function in H^H that maps the base learner's hypothesis at difficulty level k to a hypothesis at level k+1, enabling extrapolation without data from harder domains.
    The model successor is a mathematical construct, not a physical entity; the only evidence for its usefulness is the paper's own dyck1 experiment, so there is no independent falsifiable handle outside the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning Model Successors." pith.science (2026). https://pith.science/paper/VASZAYZK

@misc{pith2026250200197,
  author       = {Pith},
  title        = {Pith review of: Learning Model Successors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VASZAYZK}},
  note         = {Machine review of arXiv:2502.00197}
}
read the original abstract

The notion of generalization has moved away from the classical one defined in statistical learning theory towards an emphasis on out-of-domain generalization (OODG). There has been a growing focus on generalization from easy to hard, where a progression of difficulty implicitly governs the direction of domain shifts. This emerging regime has appeared in the literature under different names, such as length/logical/algorithmic extrapolation, but a formal definition is lacking. We argue that the unifying theme is induction -- based on finite samples observed in training, a learner should infer an inductive principle that applies in an unbounded manner. This work formalizes the notion of inductive generalization along a difficulty progression and argues that our path ahead lies in transforming the learning paradigm. We attempt to make inroads by proposing a novel learning paradigm, Inductive Learning, which involves a central concept called model successors. We outline practical steps to adapt well-established techniques towards learning model successors. This work calls for restructuring of the research discussion around induction and generalization from fragmented task-centric communities to a more unified effort, focused on universal properties of learning and computation.

Figures

Figures reproduced from arXiv: 2502.00197 by the authors.

Figure 1
Figure 1. Hypotheses that are a [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

200 extracted references · 38 canonical work pages

  1. [1]

    Generalization on the unseen, logic reasoning and degree curriculum

    Abbe, E., Bengio, S., Lotfi, A., and Rizk, K. Generalization on the unseen, logic reasoning and degree curriculum. Journal of Machine Learning Research, 25(331):1–58, 2024

    2024

  2. [2]

    How far can transformers reason? the locality barrier and inductive scratchpad

    Abbe, E., Bengio, S., Lotfi, A., Sandon, C., and Saremi, O. How far can transformers reason? the locality barrier and inductive scratchpad. arXiv preprint arXiv:2406.06467, 2024

    arXiv 2024

  3. [3]

    and Mansouri, A

    Ahuja, K. and Mansouri, A. On provable length and compositional generalization. In ICML 2024 Workshop on Theoretical Foundations of Foundation Models , 2024. URL https: //openreview.net/forum?id=xuwtmXiHMT

    2024

  4. [4]

    and Rodríguez, C

    Alessandroni, N. and Rodríguez, C. On perception as the basis for object concepts: A critical analysis. Pragmatics & Cognition, 26(2-3):321–356, 2019

    2019

  5. [5]

    Autoregressive transformers are zero-shot video imitators

    Anonymous. Autoregressive transformers are zero-shot video imitators. In Submitted to The Thirteenth International Conference on Learning Representations, 2024. URL https: //openreview.net/forum?id=wkbx7BRAsM. under review

    2024

  6. [6]

    Invariant risk minimization

    Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez-Paz, D. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019

    arXiv 1907

  7. [7]

    Dynamic node creation in backpropagation networks

    Ash, T. Dynamic node creation in backpropagation networks. Connection Science, 1(4): 365–375, 1989. doi: 10.1080/09540098908915647. URL https://doi.org/10.1080/ 09540098908915647

    doi:10.1080/09540098908915647 1989

  8. [8]

    and Nagarajan, V

    Bachmann, G. and Nagarajan, V . The pitfalls of next-token prediction. InForty-first Interna- tional Conference on Machine Learning, 2024. URL https://openreview.net/forum? id=76zq8Wkl6Z

    2024

Show all 200 references
  1. [9]

    P., Köster, R., Chadwick, M

    Banino, A., Badia, A. P., Köster, R., Chadwick, M. J., Zambaldi, V ., Hassabis, D., Barry, C., Botvinick, M., Kumaran, D., and Blundell, C. Memo: A deep network for flexible combination of episodic memories. In International Conference on Learning Representations, 2020

    2020

  2. [10]

    Pondernet: Learning to ponder

    Banino, A., Balaguer, J., and Blundell, C. Pondernet: Learning to ponder. In 8th ICML Workshop on Automated Machine Learning (AutoML), 2021

    2021

  3. [11]

    L., Montanari, A., and Rakhlin, A

    Bartlett, P. L., Montanari, A., and Rakhlin, A. Deep learning: a statistical viewpoint. Acta numerica, 30:87–201, 2021

    2021

  4. [12]

    A model of inductive bias learning

    Baxter, J. A model of inductive bias learning. Journal of artificial intelligence research, 12: 149–198, 2000

    2000

  5. [13]

    and Schuller, R

    Ben-David, S. and Schuller, R. Exploiting task relatedness for multiple task learning. In Learn- ing Theory and Kernel Machines: 16th Annual Conference on Learning Theory and 7th Kernel Workshop, COLT/Kernel 2003, Washington, DC, USA, August 24-27, 2003. Proceedings, pp. 567–5...

    2003

  6. [14]

    Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan, J. W. A theory of learning from different domains. Machine learning, 79:151–175, 2010

    2010

  7. [15]

    and Boult, T

    Bendale, A. and Boult, T. Towards open world recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1893–1902, 2015

    1902

  8. [16]

    Deep learning of representations for unsupervised and transfer learning

    Bengio, Y . Deep learning of representations for unsupervised and transfer learning. In Proceedings of ICML workshop on unsupervised and transfer learning , pp. 17–36. JMLR Workshop and Conference Proceedings, 2012

    2012

  9. [17]

    Curriculum learning

    Bengio, Y ., Louradour, J., Collobert, R., and Weston, J. Curriculum learning. InProceedings of the 26th annual international conference on machine learning, pp. 41–48, 2009

    2009

  10. [18]

    On the practical ability of recurrent neural networks to recognize hierarchical languages

    Bhattamishra, S., Ahuja, K., and Goyal, N. On the practical ability of recurrent neural networks to recognize hierarchical languages. In Proceedings of the 28th International Conference on Computational Linguistics. International Committee on Computational Linguistics, 2020. 10

    2020

  11. [19]

    On the Ability and Limitations of Transformers to Recognize Formal Languages

    Bhattamishra, S., Ahuja, K., and Goyal, N. On the Ability and Limitations of Transformers to Recognize Formal Languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2020. URL http...

    2020

  12. [20]

    Simplicity bias in transformers and their ability to learn sparse Boolean functions

    Bhattamishra, S., Patel, A., Kanade, V ., and Blunsom, P. Simplicity bias in transformers and their ability to learn sparse Boolean functions. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Li...

    2023

  13. [21]

    and Schulz, E

    Binz, M. and Schulz, E. Using cognitive psychology to understand gpt-3. Proceedings of the National Academy of Sciences, 120(6):e2218523120, 2023

    2023

  14. [22]

    E., Cruz, S., Dhamija, A

    Boult, T. E., Cruz, S., Dhamija, A. R., Gunther, M., Henrydoss, J., and Scheirer, W. J. Learning and the unknown: Surveying steps toward open world recognition. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 9801–9807, 2019

    2019

  15. [23]

    Making neural programming architectures generalize via recursion

    Cai, J., Shin, R., and Song, D. Making neural programming architectures generalize via recursion. arXiv preprint arXiv:1704.06611, 2017

    2017 arXiv

  16. [24]

    Précis of the origin of concepts

    Carey, S. Précis of the origin of concepts. Behavioral and Brain Sciences, 34(3):113–124, 2011

    2011

  17. [25]

    Multitask learning

    Caruana, R. Multitask learning. Machine learning, 28:41–75, 1997

    1997

  18. [26]

    and Bisk, Y

    Chang, Y . and Bisk, Y . Language models need inductive biases to count inductively.arXiv preprint arXiv:2405.20131, 2024

    2024 arXiv

  19. [27]

    A convex formulation for learning shared structures from multiple tasks

    Chen, J., Tang, L., Liu, J., and Ye, J. A convex formulation for learning shared structures from multiple tasks. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML ’09, pp. 137–144. Association for Computing Machinery, 2009. doi: 10. 1145/1553...

    2009

  20. [28]

    W., Schwarz, J

    Chen, S., Tack, J., Yang, Y ., Teh, Y . W., Schwarz, J. R., and Wei, Y . Unleashing the power of meta-tuning for few-shot generalization through sparse interpolated experts. arXiv preprint arXiv:2403.08477, 2024

    2024 arXiv

  21. [29]

    and Liu, B

    Chen, Z. and Liu, B. Lifelong machine learning. Morgan & Claypool Publishers, 2018

    2018

  22. [30]

    Learning from multiple sources

    Crammer, K., Kearns, M., and Wortman, J. Learning from multiple sources. Journal of Machine Learning Research, 9(57):1757–1774, 2008. URL http://jmlr.org/papers/v9/ crammer08a.html

    2008

  23. [31]

    Learning higher-order logic programs

    Cropper, A., Morel, R., and Muggleton, S. Learning higher-order logic programs. Machine Learning, 109:1289–1322, 2020

    2020

  24. [32]

    and Feys, R

    Curry, H. and Feys, R. Combinatory Logic. Number v. 1 in Combinatory Logic. North- Holland Publishing Company, 1958. URL https://books.google.com/books?id= fEnuAAAAMAAJ

    1958

  25. [33]

    Bayesian multitask learning with latent hierarchies

    Daumé III, H. Bayesian multitask learning with latent hierarchies. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, pp. 135–142, 2009

    2009

  26. [34]

    B., Lu, T., Luu, T., and Pál, D

    David, S. B., Lu, T., Luu, T., and Pál, D. Impossibility theorems for domain adaptation. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 129–136. JMLR Workshop and Conference Proceedings, 2010

    2010

  27. [35]

    B., Giapitzakis, G., Yang, S., Veli ˇckovi´c, P., and Fountoulakis, K

    de Luca, A. B., Giapitzakis, G., Yang, S., Veli ˇckovi´c, P., and Fountoulakis, K. Positional attention: Out-of-distribution generalization and expressivity for neural algorithmic reasoning. arXiv preprint arXiv:2410.01686, 2024

    arXiv

  28. [36]

    Random deep neural networks are biased towards simple functions

    De Palma, G., Kiani, B., and Lloyd, S. Random deep neural networks are biased towards simple functions. Advances in Neural Information Processing Systems, 32, 2019. 11

    2019

  29. [37]

    H., Cowan, N

    De Silva, A., Ramesh, R., Ungar, L., Shuler, M. H., Cowan, N. J., Platt, M., Li, C., Isik, L., Roh, S.-E., Charles, A., et al. Prospective learning: Principled extrapolation to the future. In Conference on Lifelong Learning Agents, pp. 347–357. PMLR, 2023

    2023

  30. [38]

    Universal transformers

    Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., and Kaiser, L. Universal transformers. In International Conference on Learning Representations, 2019. URL https://openreview. net/forum?id=HyzdRiR9Y7

    2019

  31. [39]

    K., Catt, E., Cundy, C., Hutter, M., Legg, S., Veness, J., and Ortega, P

    Deletang, G., Ruoss, A., Grau-Moya, J., Genewein, T., Wenliang, L. K., Catt, E., Cundy, C., Hutter, M., Legg, S., Veness, J., and Ortega, P. A. Neural networks and the chomsky hierarchy. In The Eleventh International Conference on Learning Representations, 2023. URL https://op...

    2023

  32. [40]

    M., Helm, H

    Dey, J., Geisa, A., Mehta, R., Tomita, T. M., Helm, H. S., Xu, H., Eaton, E., Dick, J., Priebe, C. E., and V ogelstein, J. T. Towards a theory of out-of-distribution learning.arXiv preprint arXiv:2109.14501, 2021

    2021 arXiv

  33. [41]

    A closer look at distribution shifts and out-of-distribution generalization on graphs

    Ding, M., Kong, K., Chen, J., Kirchenbauer, J., Goldblum, M., Wipf, D., Huang, F., and Gold- stein, T. A closer look at distribution shifts and out-of-distribution generalization on graphs. In NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications, 2...

    2021

  34. [42]

    and Ma, T

    Dong, K. and Ma, T. First steps toward understanding the extrapolation of nonlinear models to unseen domains. arXiv preprint arXiv:2211.11719, 2022

    2022 arXiv

  35. [43]

    Location Attention for Extrapolation to Longer Sequences

    Dubois, Y ., Dagan, G., Hupkes, D., and Bruni, E. Location Attention for Extrapolation to Longer Sequences. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , Online, jul

  36. [44]

    L., Jiang, L., Lin, B

    Dziri, N., Lu, X., Sclar, M., Li, X. L., Jiang, L., Lin, B. Y ., Welleck, S., West, P., Bhagavatula, C., Le Bras, R., et al. Faith and fate: Limits of transformers on compositionality. Advances in Neural Information Processing Systems, 36, 2024

    2024

  37. [45]

    How can self-attention networks recognize Dyck-n languages? In Findings of the Association for Computational Linguistics: EMNLP 2020 , November 2020

    Ebrahimi, J., Gelda, D., and Zhang, W. How can self-attention networks recognize Dyck-n languages? In Findings of the Association for Computational Linguistics: EMNLP 2020 , November 2020. URL https://aclanthology.org/2020.findings-emnlp.384/

    2020

  38. [46]

    and Zilioli, M

    Ell, S. and Zilioli, M. Categorical Learning, pp. 509–512. Springer US, Boston, MA, 2012. ISBN 978-1-4419-1428-6. doi: 10.1007/978-1-4419-1428-6_98. URL https://doi.org/ 10.1007/978-1-4419-1428-6_98

    2012 doi

  39. [47]

    Elman, J. L. Learning and development in neural networks: the importance of starting small. Cognition, 48(1):71–99, 1993. doi: https://doi.org/10.1016/0010-0277(93)90058-4. URL https://www.sciencedirect.com/science/article/pii/0010027793900584

    1993

  40. [48]

    H., and Hutter, F

    Elsken, T., Metzen, J. H., and Hutter, F. Efficient multi-objective neural architecture search via lamarckian evolution. arXiv preprint arXiv:1804.09081, 2018

    2018 arXiv

  41. [49]

    H., and Hutter, F

    Elsken, T., Metzen, J. H., and Hutter, F. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1–21, 2019

    2019

  42. [50]

    Neural fine-tuning search for few-shot learning

    Eustratiadis, P., Dudziak, Ł., Li, D., and Hospedales, T. Neural fine-tuning search for few-shot learning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=T7YV5UZKBc

    2024

  43. [51]

    and Lebiere, C

    Fahlman, S. and Lebiere, C. The cascade-correlation learning architecture. Advances in neural information processing systems, 2, 1989

    1989

  44. [52]

    Looped transformers for length generalization

    Fan, Y ., Du, Y ., Ramchandran, K., and Lee, K. Looped transformers for length generalization. In The 4th Workshop on Mathematical Reasoning and AI at NeurIPS’24, 2024. 12

    2024

  45. [53]

    and Heit, E

    Feeney, A. and Heit, E. (eds.). Inductive Reasoning: Experimental, Developmental, and Computational Approaches. Cambridge University Press, 2007. doi: https://doi.org/10.1017/ CBO9780511619304

    2007

  46. [54]

    Gallant, S. I. Three constructive algorithms for network learning. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 8, 1986

    1986

  47. [55]

    D., Tenenbaum, J

    Goodman, N. D., Tenenbaum, J. B., Feldman, J., and Griffiths, T. L. A rational analysis of rule-based concept learning. Cognitive science, 32(1):108–154, 2008

    2008

  48. [56]

    K., Mattern, C., Aitchison, M., and Veness, J

    Grau-Moya, J., Genewein, T., Hutter, M., Orseau, L., Deletang, G., Catt, E., Ruoss, A., Wenliang, L. K., Mattern, C., Aitchison, M., and Veness, J. Learning universal predic- tors. In Forty-first International Conference on Machine Learning , 2024. URL https: //openreview.net/...

    2024

  49. [57]

    Neural turing machines

    Graves, A. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014

    2014 arXiv

  50. [58]

    Adaptive computation time for recurrent neural networks

    Graves, A. Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.08983, 2016

    2016 arXiv

  51. [59]

    and Lopez-Paz, D

    Gulrajani, I. and Lopez-Paz, D. In search of lost domain generalization. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum? id=lQdXeXDoWtI

    2021

  52. [60]

    Characterizing implicit bias in terms of optimization geometry

    Gunasekar, S., Lee, J., Soudry, D., and Srebro, N. Characterizing implicit bias in terms of optimization geometry. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. ...

    2018

  53. [61]

    Theoretical limitations of self-attention in neural sequence models

    Hahn, M. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics, 2020

    2020

  54. [62]

    Han, Z., Gao, C., Liu, J., Zhang, J., and Zhang, S. Q. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403.14608, 2024

    2024 arXiv

  55. [63]

    Formal language recognition by hard attention transformers: Perspectives from circuit complexity

    Hao, Y ., Angluin, D., and Frank, R. Formal language recognition by hard attention transformers: Perspectives from circuit complexity. Transactions of the Association for Computational Linguistics, 10:800–810, 2022

    2022

  56. [64]

    The problem of induction

    Henderson, L. The problem of induction. In Zalta, E. N. and Nodelman, U. (eds.), The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Winter 2024 edition, 2024. URL https://plato.stanford.edu/archives/win2024/ entries/induction-problem/

    2024

  57. [65]

    The many faces of robustness: A critical analysis of out-of-distribution generalization

    Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., Song, D., Steinhardt, J., and Gilmer, J. The many faces of robustness: A critical analysis of out-of-distribution generalization. In 2021 IEEE/CVF International C...

    2021

  58. [66]

    Universal length generalization with turing programs

    Hou, K., Brandfonbrener, D., Kakade, S., Jelassi, S., and Malach, E. Universal length generalization with turing programs. arXiv preprint arXiv:2407.03310, 2024

    2024 arXiv

  59. [67]

    Llm- adapters: An adapter family for parameter-efficient fine-tuning of large language models

    Hu, Z., Wang, L., Lan, Y ., Xu, W., Lim, E.-P., Bing, L., Xu, X., Poria, S., and Lee, R. Llm- adapters: An adapter family for parameter-efficient fine-tuning of large language models. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirica...

    2023 doi

  60. [68]

    Language models as zero-shot planners: Extracting actionable knowledge for embodied agents

    Huang, W., Abbeel, P., Pathak, D., and Mordatch, I. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv preprint arXiv:2201.07207, 2022

    2022 arXiv

  61. [69]

    A theory of universal artificial intelligence based on algorithmic complexity

    Hutter, M. A theory of universal artificial intelligence based on algorithmic complexity. Technical report, Bayerstr. 21, 80335 Munich, Germany, Apr 2000. URL http://xxx.lanl. gov/abs/cs.AI/0004001

    2000

  62. [70]

    Making a low-dimensional representation suitable for diverse tasks

    Intrator, N. Making a low-dimensional representation suitable for diverse tasks. Connection Science, 8(2):205–224, 1996

    1996

  63. [71]

    Going beyond linear transformers with recurrent fast weight programmers

    Irie, K., Schlag, I., Csordás, R., and Schmidhuber, J. Going beyond linear transformers with recurrent fast weight programmers. In Beygelzimer, A., Dauphin, Y ., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems , 2021. URL https://openrev...

    2021

  64. [72]

    Length general- ization in arithmetic transformers

    Jelassi, S., d’Ascoli, S., Domingo-Enrich, C., Wu, Y ., Li, Y ., and Charton, F. Length general- ization in arithmetic transformers. arXiv e-prints, pp. arXiv–2306, 2023

    2023

  65. [73]

    Fantastic generalization measures and where to find them

    Jiang*, Y ., Neyshabur*, B., Mobahi, H., Krishnan, D., and Bengio, S. Fantastic generalization measures and where to find them. In International Conference on Learning Representations,

  66. [74]

    The impact of positional encoding on length generalization in transformers

    Kazemnejad, A., Padhi, I., Natesan Ramamurthy, K., Das, P., and Reddy, S. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 36, 2024

    2024

  67. [75]

    URL https://openreview.net/forum?id=SJgIPJBFvH

  68. [76]

    Kearns, M. J. and Vazirani, U. V . An introduction to computational learning theory . MIT Press, Cambridge, MA, USA, 1994. ISBN 0262111934

    1994

  69. [77]

    and Liu, B

    Ke, Z. and Liu, B. Continual learning of natural language processing tasks: A survey. arXiv preprint arXiv:2211.12701, 2022

    2022 arXiv

  70. [78]

    S., Reid, M., Matsuo, Y ., and Iwasawa, Y

    Kojima, T., Gu, S. S., Reid, M., Matsuo, Y ., and Iwasawa, Y . Large language models are zero-shot reasoners. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum? id=e2TBb5y0yFf

    2022

  71. [79]

    W., Sagawa, S., Marklund, H., Xie, S

    Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al. Wilds: A benchmark of in-the-wild distribution shifts. In International conference on machine learning, pp. 5637–5664. PMLR, 2021

    2021

  72. [80]

    L., and Courville, A

    Krueger, D., Caballero, E., Jacobsen, J.-H., Zhang, A., Binas, J., Zhang, D., Priol, R. L., and Courville, A. Out-of-distribution generalization via risk extrapolation (rex). In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learnin...

  73. [81]

    Korsky, S. A. On the computational power of RNNs. PhD thesis, Massachusetts Institute of Technology, 2019

    2019

  74. [82]

    and Daumé, H

    Kumar, A. and Daumé, H. Learning task grouping and overlap in multi-task learning. In Proceedings of the 29th International Coference on International Conference on Machine Learning, ICML’12, pp. 1723–1730. Omnipress, 2012

    2012

  75. [83]

    J., Liu, Y ., and Van Roy, B

    Kumar, S., Marklund, H., Rao, A., Zhu, Y ., Jeon, H. J., Liu, Y ., and Van Roy, B. Continual learn- ing as computationally constrained reinforcement learning. arXiv preprint arXiv:2307.04345, 2023. 14

    2023 arXiv

  76. [84]

    Krueger, K. A. and Dayan, P. Flexible shaping: How learning in small steps helps. Cognition, 110(3):380–394, 2009. doi: https://doi.org/10.1016/j.cognition.2008.11.014. URL https: //www.sciencedirect.com/science/article/pii/S0010027708002850

    2009 doi

  77. [85]

    M., Salakhutdinov, R., and Tenenbaum, J

    Lake, B. M., Salakhutdinov, R., and Tenenbaum, J. B. Human-level concept learning through probabilistic program induction. Science, 350(6266):1332–1338, 2015

    2015

  78. [86]

    Can rnns learn recursive nested subject-verb agreements? arXiv preprint arXiv:2101.02258, 2021

    Lakretz, Y ., Desbordes, T., King, J.-R., Crabbé, B., Oquab, M., and Dehaene, S. Can rnns learn recursive nested subject-verb agreements? arXiv preprint arXiv:2101.02258, 2021

    2021 arXiv

  79. [87]

    D., and Johns, E

    Kwon, T., Palo, N. D., and Johns, E. Language models as zero-shot trajectory generators, 2023

    2023

  80. [88]

    L., Brockschmidt, M., and Kushman, N

    Li, C., Tarlow, D., Gaunt, A. L., Brockschmidt, M., and Kushman, N. Neural program lattices. In International Conference on learning representations, 2017

    2017

  81. [89]

    Li, D., Yang, Y ., Song, Y .-Z., and Hospedales, T. M. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp. 5542–5550, 2017

    2017

  82. [90]

    Learning a meta-level prior for feature relevance from multiple related tasks

    Lee, S.-I., Chatalbashev, V ., Vickrey, D., and Koller, D. Learning a meta-level prior for feature relevance from multiple related tasks. In Proceedings of the 24th International Conference on Machine Learning, ICML ’07, pp. 489–496. Association for Computing Machinery, 2007. ...

    2007

  83. [91]

    R., and Eisner, J

    Lin, C.-C., Jaech, A., Li, X., Gormley, M. R., and Eisner, J. Limitations of autoregressive models and their alternatives. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 514...

    2021

  84. [92]

    T., Goel, S., Krishnamurthy, A., and Zhang, C

    Liu, B., Ash, J. T., Goel, S., Krishnamurthy, A., and Zhang, C. Transformers learn shortcuts to automata. In The Eleventh International Conference on Learning Representations, 2022

    2022

  85. [93]

    and Vitanyi, P

    Li, M. and Vitanyi, P. An Introduction to Kolmogorov Complexity and Its Applications . Springer Publishing Company, Incorporated, 4th edition, 2019. ISBN 3030112977

    2019

  86. [94]

    Darts: Differentiable architecture search

    Liu, H., Simonyan, K., and Yang, Y . Darts: Differentiable architecture search. InInternational Conference on Learning Representations, 2018

    2018

  87. [95]

    G., and Tan, K

    Liu, Y ., Sun, Y ., Xue, B., Zhang, M., Yen, G. G., and Tan, K. C. A survey on evolutionary neural architecture search. arXiv preprint arXiv:2008.10937, 2020

    2008 arXiv

  88. [96]

    Hierarchical representa- tions for efficient architecture search

    Liu, H., Simonyan, K., Vinyals, O., Fernando, C., and Kavukcuoglu, K. Hierarchical representa- tions for efficient architecture search. InInternational Conference on Learning Representations, 2018

    2018

  89. [97]

    MacKay, D. J. Information theory, inference and learning algorithms. Cambridge university press, 2003

    2003

  90. [98]

    Auto-regressive next-token predictors are universal learners

    Malach, E. Auto-regressive next-token predictors are universal learners. arXiv preprint arXiv:2309.06979, 2023

    2023 arXiv

  91. [99]

    MacKay, D. J. Bayesian methods for adaptive models . PhD thesis, California Institute of Technology, 1992

    1992

  92. [100]

    McAllester, D. A. Pac-bayesian model averaging. In Proceedings of the Twelfth Annual Conference on Computational Learning Theory, COLT ’99, pp. 164–170, New York, NY , USA,

  93. [101]

    T., Yao, S., Friedman, D., Hardy, M., and Griffiths, T

    McCoy, R. T., Yao, S., Friedman, D., Hardy, M., and Griffiths, T. L. Embers of autoregression: Understanding large language models through the problem they are trained to solve. arXiv preprint arXiv:2309.13638, 2023. 15

    2023 arXiv

  94. [102]

    and Laurence, S

    Margolis, E. and Laurence, S. How to learn the natural numbers: Inductive inference and the acquisition of number concepts. Cognition, 106(2):924–939, 2008

    2008

  95. [103]

    Sequential neural networks as automata

    Merrill, W. Sequential neural networks as automata. In Proceedings of the Workshop on Deep Learning and Formal Languages: Building Bridges . Association for Computational Linguistics, 2019. URL https://aclanthology.org/W19-3901/

    2019

  96. [104]

    and Tsilivis, N

    Merrill, W. and Tsilivis, N. Extracting finite automata from rnns using state merging. arXiv preprint arXiv:2201.12451, 2022

    2022 arXiv

  97. [105]

    Rule extrapolation in language modeling: A study of compositional generalization on OOD prompts

    Mészáros, A., Ujváry, S., Brendel, W., Reizinger, P., and Huszár, F. Rule extrapolation in language modeling: A study of compositional generalization on OOD prompts. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024. URL https://openreview....

    2024

  98. [106]

    Medin, D. L. and Coley, J. D. Perception and cognition at century’s end , chapter 13, pp. 403–439. Academic Press, 1998. URL https://doi.org/10.1016/B978-012301160-2/ 50015-0

    1998 doi

  99. [107]

    Millikan, R. G. A common structure for concepts of individuals, stuffs, and real kinds: More mama, more milk, and more mouse. Behavioral and Brain Sciences, 21(1):55–65, 1997. doi: 10.1017/s0140525x98000405

    1997 doi

  100. [108]

    G., Rao, K., Sadigh, D., and Zeng, A

    Mirchandani, S., Xia, F., Florence, P., Ichter, B., Driess, D., Arenas, M. G., Rao, K., Sadigh, D., and Zeng, A. Large language models as general pattern machines. In Proceedings of the 7th Conference on Robot Learning (CoRL), 2023

    2023

  101. [109]

    Machine Learning

    Mitchell, T. Machine Learning. McGraw-Hill International Editions. McGraw-Hill, 1997. ISBN 9780071154673. URL https://books.google.com/books?id=EoYBngEACAAJ

    1997

  102. [110]

    J., Shah, A., Verma, A., Chaudhuri, S., and Patel, A

    Michalenko, J. J., Shah, A., Verma, A., Chaudhuri, S., and Patel, A. B. Finite automata can be linearly decoded from language-recognizing RNNs. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=H1zeHnA9KX

    2019

  103. [111]

    Mundt, M., Pliushch, I., Majumder, S., and Ramesh, V . Open set recognition through deep neural network uncertainty: Does out-of-distribution detection require generative classifiers? In Proceedings of the IEEE/CVF international conference on computer vision workshops, pp. 0–0, 2019

    2019

  104. [112]

    A wholistic view of continual learning with deep neural networks: Forgotten lessons and the bridge to active and open world learning

    Mundt, M., Hong, Y ., Pliushch, I., and Ramesh, V . A wholistic view of continual learning with deep neural networks: Forgotten lessons and the bridge to active and open world learning. Neural Networks, 160:306–336, 2023

    2023

  105. [113]

    Pushdown layers: Encoding recursive structure in transformer language models

    Murty, S., Sharma, P., Andreas, J., and Manning, C. Pushdown layers: Encoding recursive structure in transformer language models. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 3233–324...

    2023

  106. [114]

    Domain generalization via invariant feature representation

    Muandet, K., Balduzzi, D., and Schölkopf, B. Domain generalization via invariant feature representation. In International conference on machine learning, pp. 10–18. PMLR, 2013

    2013

  107. [115]

    Nate Gruver, Marc Finzi, S. Q. and Wilson, A. G. Large Language Models Are Zero Shot Time Series Forecasters. In Advances in Neural Information Processing Systems, 2023

    2023

  108. [116]

    Neal, R. M. Bayesian Learning for Neural Networks. Springer-Verlag, Berlin, Heidelberg,

  109. [117]

    Learning to extrapolate: A transductive approach

    Netanyahu, A., Gupta, A., Simchowitz, M., Zhang, K., and Agrawal, P. Learning to extrapolate: A transductive approach. arXiv preprint arXiv:2304.14329, 2023

    2023 arXiv

  110. [118]

    J., Ren, M., Finn, C., and McClelland, J

    Nam, A. J., Ren, M., Finn, C., and McClelland, J. L. Learning to reason with relational abstractions. arXiv preprint arXiv:2210.02615, 2022

    2022 arXiv

  111. [119]

    Investigating the limitations of transformers with simple arithmetic tasks

    Nogueira, R., Jiang, Z., and Lin, J. Investigating the limitations of transformers with simple arithmetic tasks. arXiv preprint arXiv:2102.13019, 2021

    2021 arXiv

  112. [120]

    R., Jung, S., Mohan, A

    O’Bryan, S. R., Jung, S., Mohan, A. J., and Scolari, M. Category learning selectively enhances representations of boundary-adjacent exemplars in early visual cortex.Journal of Neuroscience, 44(3), 2024

    2024

  113. [121]

    M., Hubin, A., Immer, A., Karaletsos, T., Khan, M

    Papamarkou, T., Skoularidou, M., Palla, K., Aitchison, L., Arbel, J., Dunson, D., Filippone, M., Fortuin, V ., Hennig, P., Hernández-Lobato, J. M., Hubin, A., Immer, A., Karaletsos, T., Khan, M. E., Kristiadi, A., Li, Y ., Mandt, S., Nemeth, C., Osborne, M. A., Rudner, T. G. J...

    2024

  114. [122]

    Newman, B., Hewitt, J., Liang, P., and Manning, C. D. The eos decision and length extrapo- lation. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pp. 276–291, 2020. 16

    2020

  115. [123]

    I., Kemker, R., Part, J

    Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., and Wermter, S. Continual lifelong learning with neural networks: A review. Neural networks, 113:54–71, 2019

    2019

  116. [124]

    Uncertainty in neural networks: Approximately bayesian ensembling

    Pearce, T., Leibfried, F., and Brintrup, A. Uncertainty in neural networks: Approximately bayesian ensembling. In Chiappa, S. and Calandra, R. (eds.), Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings ...

    2020

  117. [125]

    and Risteski, A

    Peng, B. and Risteski, A. Continual learning: a feature extraction formalization, an efficient algorithm, and fundamental obstructions. Advances in Neural Information Processing Systems, 35:28414–28427, 2022

    2022

  118. [126]

    Learning explanations that are hard to vary

    Parascandolo, G., Neitz, A., Orvieto, A., Gresele, L., and Schölkopf, B. Learning explanations that are hard to vary. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=hb1sDDSLbV

    2021

  119. [127]

    Efficient neural architecture search via parameters sharing

    Pham, H., Guan, M., Zoph, B., Le, Q., and Dean, J. Efficient neural architecture search via parameters sharing. In International conference on machine learning, pp. 4095–4104. PMLR, 2018

    2018

  120. [128]

    Large language models are zero shot hypothesis proposers

    Qi, B., Zhang, K., Li, H., Tian, K., Zeng, S., Chen, Z.-R., and Zhou, B. Large language models are zero shot hypothesis proposers. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023

    2023

  121. [129]

    On the spectral bias of neural networks

    Rahaman, N., Baratin, A., Arpit, D., Draxler, F., Lin, M., Hamprecht, F., Bengio, Y ., and Courville, A. On the spectral bias of neural networks. In International conference on machine learning, pp. 5301–5310. PMLR, 2019

    2019

  122. [130]

    Adapterhub: A framework for adapting transformers

    Pfeiffer, J., Rücklé, A., Poth, C., Kamath, A., Vuli ´c, I., Ruder, S., Cho, K., and Gurevych, I. Adapterhub: A framework for adapting transformers. In Liu, Q. and Schlangen, D. (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Sys...

    2020 doi

  123. [131]

    Raina, R., Battle, A., Lee, H., Packer, B., and Ng, A. Y . Self-taught learning: transfer learning from unlabeled data. In Proceedings of the 24th international conference on Machine learning, pp. 759–766, 2007

    2007

  124. [132]

    and De Freitas, N

    Reed, S. and De Freitas, N. Neural programmer-interpreters. arXiv preprint arXiv:1511.06279, 2015. 17

    2015 arXiv

  125. [133]

    Position: Understanding llms requires more than statistical generalization

    Reizinger, P., Ujváry, S., Mészáros, A., Kerekes, A., Brendel, W., and Huszár, F. Position: Understanding llms requires more than statistical generalization. In Forty-first International Conference on Machine Learning, 2024

    2024

  126. [134]

    and Mehrotra, S

    Rahimian, H. and Mehrotra, S. Distributionally robust optimization: A review. arXiv preprint arXiv:1908.05659, 2019

    1908 arXiv

  127. [135]

    J., Asmuth, J., and Bloomfield, A

    Rips, L. J., Asmuth, J., and Bloomfield, A. Giving the boot to the bootstrap: How not to learn the natural numbers. Cognition, 101(3):B51–B60, 2006

    2006

  128. [136]

    S., Piantadosi, S

    Rule, J. S., Piantadosi, S. T., Cropper, A., Ellis, K., Nye, M., and Tenenbaum, J. B. Symbolic metaprogram search improves learning efficiency and explains rule learning in humans.Nature Communications, 15(1):6847, 2024

    2024

  129. [137]

    and Eaton, E

    Ruvolo, P. and Eaton, E. Ella: An efficient lifelong learning algorithm. In International conference on machine learning, pp. 507–515. PMLR, 2013

    2013

  130. [138]

    Ring, M. B. Continual learning in reinforcement environments. PhD thesis, University of Texas at Austin, 1994

    1994

  131. [139]

    Transformers, parallel computation, and logarithmic depth

    Sanford, C., Hsu, D., and Telgarsky, M. Transformers, parallel computation, and logarithmic depth. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR, 2024. URL https://proceedings.mlr.press/ v235/sanford24a.html

    2024

  132. [140]

    Y ., Padmakumar, V ., Kazemi, S

    Saparov, A., Pawar, S., Pimpalgaonkar, S., Joshi, N., Pang, R. Y ., Padmakumar, V ., Kazemi, S. M., Kim, N., and He, H. Transformers struggle to learn to search. arXiv preprint arXiv:2412.04703, 2024

    2024 arXiv

  133. [141]

    Sarnecka, B. W. and Carey, S. How counting represents number: What children must learn and when they learn it. Cognition, 108(3):662–674, 2008

    2008

  134. [142]

    W., Hashimoto, T

    Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, P. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731, 2019

    1911 arXiv

  135. [143]

    Autoregressive large language models are computa- tionally universal

    Schuurmans, D., Dai, H., and Zanini, F. Autoregressive large language models are computa- tionally universal. arXiv preprint arXiv:2410.03170, 2024

    2024 arXiv

  136. [144]

    Deep Thinking Systems: Logical Extrapolation With Recurrent Neural Networks

    Schwarzschild, A. Deep Thinking Systems: Logical Extrapolation With Recurrent Neural Networks. PhD thesis, University of Maryland, College Park, 2023

    2023

  137. [145]

    Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks

    Schwarzschild, A., Borgnia, E., Gupta, A., Huang, F., Vishkin, U., Goldblum, M., and Goldstein, T. Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks. Advances in Neural Information Processing Systems, 34:6695–6706, 2021

    2021

  138. [146]

    On the building blocks of mathematical logic

    Schönfinkel, M. On the building blocks of mathematical logic. From Frege to Gödel, pp. 355–366, 1967

    1967

  139. [147]

    Causal language modeling can elicit search and reasoning capabilities on logic puzzles

    Shah, K., Dikkala, N., Wang, X., and Panigrahy, R. Causal language modeling can elicit search and reasoning capabilities on logic puzzles. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum? id=i5PoejmWoC

    2024

  140. [148]

    and Ben-David, S

    Shalev-Shwartz, S. and Ben-David, S. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, USA, 2014. ISBN 1107057132

    2014

  141. [149]

    Learnability, stability and uniform convergence

    Shalev-Shwartz, S., Shamir, O., Srebro, N., and Sridharan, K. Learnability, stability and uniform convergence. Journal of Machine Learning Research, 11(90):2635–2670, 2010. URL http://jmlr.org/papers/v11/shalev-shwartz10a.html. 18

    2010

  142. [150]

    The pitfalls of simplicity bias in neural networks

    Shah, H., Tamuly, K., Raghunathan, A., Jain, P., and Netrapalli, P. The pitfalls of simplicity bias in neural networks. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 9573–9585. Cur...

    2020

  143. [151]

    Retrieval-augmented retrieval: Large language models are strong zero-shot retriever

    Shen, T., Long, G., Geng, X., Tao, C., Lei, Y ., Zhou, T., Blumenstein, M., and Jiang, D. Retrieval-augmented retrieval: Large language models are strong zero-shot retriever. In Ku, L.-W., Martins, A., and Srikumar, V . (eds.), Findings of the Association for Com- putational L...

    2024 doi

  144. [152]

    D., Ramesh, R., Yang, R., Yu, S., V ogelstein, J

    Silva, A. D., Ramesh, R., Yang, R., Yu, S., V ogelstein, J. T., and Chaudhari, P. Prospective learn- ing: Learning for a dynamic future. InThe Thirty-eighth Annual Conference on Neural Informa- tion Processing Systems, 2024. URL https://openreview.net/forum?id=XEbPJUQzs3

    2024

  145. [153]

    and Kurtz, B

    Slonneger, K. and Kurtz, B. L. Formal syntax and semantics of programming languages , volume 340. Addison-Wesley Reading, 1995

    1995

  146. [154]

    E., Swartout, W

    Shaw, D. E., Swartout, W. R., and Green, C. C. Inferring lisp programs from examples. In Proceedings of the 4th International Joint Conference on Artificial Intelligence - Volume 1, IJCAI’75, pp. 260–267, San Francisco, CA, USA, 1975. Morgan Kaufmann Publishers Inc

    1975

  147. [155]

    T., Rota, P., and Sebe, N

    Soviany, P., Ionescu, R. T., Rota, P., and Sebe, N. Curriculum learning: A survey.International Journal of Computer Vision, 130(6):1526–1565, 2022

    2022

  148. [156]

    Self-supervision is all you need for solving rubik’s cube

    Takano, K. Self-supervision is all you need for solving rubik’s cube. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum?id= bnBeNFB27b

    2023

  149. [158]

    V ., Malviya, P., Abdelsalam, M., Janarthanan, J., and Chandar, S

    Sodhani, S., Faramarzi, M., Mehta, S. V ., Malviya, P., Abdelsalam, M., Janarthanan, J., and Chandar, S. An introduction to lifelong supervised learning. arXiv preprint arXiv:2207.04354, 2022

    2022 arXiv

  150. [159]

    Is learning the n-th thing any easier than learning the first? In Touretzky, D., Mozer, M., and Hasselmo, M

    Thrun, S. Is learning the n-th thing any easier than learning the first? In Touretzky, D., Mozer, M., and Hasselmo, M. (eds.), Advances in Neural Information Processing Systems, volume 8. MIT Press, 1995

    1995

  151. [160]

    Explanation-Based Neural Network Learning - A Lifelong Learning Approach

    Thrun, S. Explanation-Based Neural Network Learning - A Lifelong Learning Approach . Kluwer Academic Publishers, Boston, MA, April 1996

    1996

  152. [161]

    and Mitchell, T

    Thrun, S. and Mitchell, T. M. Lifelong robot learning. Robotics and autonomous systems, 15 (1-2):25–46, 1995

    1995

  153. [162]

    Tenenbaum, J. B. A Bayesian framework for concept learning. PhD thesis, Massachusetts Institute of Technology, 1999

    1999

  154. [163]

    Valiant, L. G. A theory of the learnable. In Proceedings of the Sixteenth Annual ACM Symposium on Theory of Computing, STOC ’84, pp. 436–445. Association for Computing Machinery, 1984. URL https://doi.org/10.1145/800057.808710

    1984

  155. [164]

    Q., and Louis, A

    Valle-Perez, G., Camargo, C. Q., and Louis, A. A. Deep learning generalizes because the parameter-function map is biased towards simple functions. In International Confer- ence on Learning Representations , 2019. URL https://openreview.net/forum?id= rye4g3AqFm

    2019

  156. [165]

    Vapnik, V . N. V . N.Statistical learning theory. Adaptive and learning systems for signal processing, communications, and control. Wiley, 1998. ISBN 0471030031

    1998

  157. [166]

    Utgoff, P. E. Machine learning of inductive bias, volume 15. Springer Science & Business Media, 2012

    2012

  158. [167]

    P., Budden, D., Pascanu, R., Banino, A., Dashevskiy, M., Hadsell, R., and Blundell, C

    Veliˇckovi´c, P., Badia, A. P., Budden, D., Pascanu, R., Banino, A., Dashevskiy, M., Hadsell, R., and Blundell, C. The clrs algorithmic reasoning benchmark. In International Conference on Machine Learning, pp. 22084–22102. PMLR, 2022

    2022

  159. [168]

    Efficient large language models: A survey

    Wan, Z., Wang, X., Liu, C., Alam, S., Zheng, Y ., Liu, J., Qu, Z., Yan, S., Zhu, Y ., Zhang, Q., et al. Efficient large language models: A survey. arXiv preprint arXiv:2312.03863, 2023

    2023 arXiv

  160. [169]

    G., Xing, X., Liu, X., and Giles, C

    Wang, Q., Zhang, K., Ororbia II, A. G., Xing, X., Liu, X., and Giles, C. L. An empirical evaluation of rule extraction from recurrent neural networks. Neural Computation, 30(9): 2568–2591, 2018

    2018

  161. [170]

    Adaptive recurrent vision performs zero-shot computation scaling to unseen difficulty levels

    Veerabadran, V ., Ravishankar, S., Tang, Y ., Raina, R., and de Sa, V . Adaptive recurrent vision performs zero-shot computation scaling to unseen difficulty levels. Advances in Neural Information Processing Systems, 36, 2024. 19

    2024

  162. [171]

    From decoding to meta-generation: Inference-time algorithms for large language models

    Welleck, S., Bertsch, A., Finlayson, M., Schoelkopf, H., Xie, A., Neubig, G., Kulikov, I., and Harchaoui, Z. From decoding to meta-generation: Inference-time algorithms for large language models. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://open...

    2024

  163. [172]

    Bananas: Bayesian optimization with neural architectures for neural architecture search

    White, C., Neiswanger, W., and Savani, Y . Bananas: Bayesian optimization with neural architectures for neural architecture search. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 10293–10301, 2021

    2021

  164. [173]

    Neural architecture search: Insights from 1000 papers

    White, C., Safari, M., Sukthanker, R., Ru, B., Elsken, T., Zela, A., Dey, D., and Hutter, F. Neural architecture search: Insights from 1000 papers. arXiv preprint arXiv:2301.08727, 2023

    2023 arXiv

  165. [174]

    Symbolic brittleness in sequence models: on systematic generalization in symbolic mathematics

    Welleck, S., West, P., Cao, J., and Choi, Y . Symbolic brittleness in sequence models: on systematic generalization in symbolic mathematics. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8629–8637, 2022

    2022

  166. [175]

    Wilson, A. G. The case for bayesian deep learning. CoRR, 2020. URL https://arxiv. org/abs/2001.10995

    2020 arXiv

  167. [176]

    Wilson, A. G. and Izmailov, P. Bayesian deep learning and a probabilistic perspective of generalization. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems , volume 33, pp. 4697–4708. Curran Associates...

    2020

  168. [177]

    Children’s acquisition of the number words and the counting system

    Wynn, K. Children’s acquisition of the number words and the counting system. Cognitive psychology, 24(2):220–251, 1992

    1992

  169. [178]

    Sub-task decomposition enables learning in sequence to sequence tasks

    Wies, N., Levine, Y ., and Shashua, A. Sub-task decomposition enables learning in sequence to sequence tasks. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=BrJATVZDWEH

    2023

  170. [179]

    J., Zhang, Y ., Luo, T., Xiao, Y ., and Ma, Z

    Xu, Z.-Q. J., Zhang, Y ., Luo, T., Xiao, Y ., and Ma, Z. Frequency principle: Fourier analysis sheds light on deep neural networks. arXiv preprint arXiv:1901.06523, 2019

    1901 arXiv

  171. [180]

    K., Kasai, J., and Yildirim, I

    Yamada, Y ., Bao, Y ., Lampinen, A. K., Kasai, J., and Yildirim, I. Evaluating spatial under- standing of large language models. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=xkiflfKCw3

    2024

  172. [181]

    and Piantadosi, S

    Yang, Y . and Piantadosi, S. T. One model for the learning of language. Proceedings of the National Academy of Sciences, 119(5):e2021865119, 2022

    2022

  173. [182]

    and Liu, B

    Xiao, C. and Liu, B. A theory for length generalization in learning to reason. arXiv preprint arXiv:2404.00560, 2024

    2024 arXiv

  174. [183]

    H., and Narasimhan, K

    Yao, S., Peng, B., Papadimitriou, C. H., and Narasimhan, K. Self-attention networks can pro- cess bounded hierarchical languages. In Annual Meeting of the Association for Computational Linguistics, 2021. URL https://api.semanticscholar.org/CorpusID:235166395. 20

    2021

  175. [184]

    How transferable are features in deep neural networks? In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., and Weinberger, K

    Yosinski, J., Clune, J., Bengio, Y ., and Lipson, H. How transferable are features in deep neural networks? In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., and Weinberger, K. (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014

    2014

  176. [185]

    T., and Kuhn, J

    Yu, X., Vu, N. T., and Kuhn, J. Learning the Dyck language with attention-based Seq2Seq models. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. Association for Computational Linguistics, 2019. URL https: //aclanthology.o...

    2019

  177. [186]

    Can large language models always solve easy problems if they can solve harder ones? arXiv preprint arXiv:2406.12809, 2024

    Yang, Z., Zhang, Y ., Liu, T., Yang, J., Lin, J., Zhou, C., and Sui, Z. Can large language models always solve easy problems if they can solve harder ones? arXiv preprint arXiv:2406.12809, 2024

    2024 arXiv

  178. [187]

    D., Tigges, C., Biderman, S., Raginsky, M., and Ringer, T

    Zhang, S. D., Tigges, C., Biderman, S., Raginsky, M., and Ringer, T. Can transformers learn to solve problems recursively? arXiv preprint arXiv:2305.14699, 2023

    2023 arXiv

  179. [188]

    Towards lifelong learning of large language models: A survey

    Zheng, J., Qiu, S., Shi, C., and Ma, Q. Towards lifelong learning of large language models: A survey. arXiv preprint arXiv:2406.06391, 2024

    2024 arXiv

  180. [189]

    M., Bengio, S., and Nakkiran, P

    Zhou, H., Bradley, A., Littwin, E., Razin, N., Saremi, O., Susskind, J. M., Bengio, S., and Nakkiran, P. What algorithms can transformers learn? a study in length generalization. In The Twelfth International Conference on Learning Representations, 2024. 21 Learning Model Succe...

    2024

  181. [190]

    C., Li, M., and Smola, A

    Zhang, A., Lipton, Z. C., Li, M., and Smola, A. J. Dive into Deep Learning . Cambridge University Press, 2023. https://D2L.ai

    2023

  182. [194]

    S → ( S ) 3

    S → ϵ 2. S → ( S ) 3. S → S S Invalid sequences are generated by corrupting valid sequences via one of the following steps

  183. [195]

    randomly delete a ‘(’ or a ‘)’

  184. [196]

    randomly insert a ‘(’ or a ‘)’

  185. [197]

    randomly substitute a ‘(’ with a ‘)’ or the other way around

  186. [198]

    ()" (p = 1) Figure A1: A constant-size PFST that trans- lates any sequence in the dyckm dataset into a sequence in the dyckm+1 dataset. “ ⋆ : ⋆

    pick two valid sequences X, Y, concatenate them “X ) Y ", then randomly insert ‘(’ into Y Let dyck1−m denote dyck1 language with nesting depth bounded by m. Hence, training data for dyck1−m will only include valid sequences with depth ≤ m. The data for each m is constructed wi...

  187. [199]

    learning under distributional shift

    In terms of our success criteria defined in §5, we achieve DGR(Ind3, h3) = 0 , in which δm = 1 if 4 ≤ m ≤ 52 and = 0 otherwise19. We obtain similar results even when the transition rules of ˆh∗ k in each training sequence are shuffled. The model correctly learns that it is sup...

  188. [200]

    zero-shot

    or non-recurrent architectures equipped with autoregressive decoding [171]. In the former, two families of approaches are most relevant to inductive generalization problems, both having the goal of simulating a recursive algorithm: (1) Deep thinking systems, featuring looping ...

  189. [201]

    pointer" associated with the halting decision starts to make incorrect advancements. Relatedly, the “ eos-problem

    maintains the entire pareto frontier of topologies, guiding the warm-starting of a child network 21For example, Reed & De Freitas [132] reported that Neural-Programmer Interpreters can length-generalize bubble sort from 20 to 60, beyond which the “pointer" associated with the ...

  190. [1999]

    ISBN 1581131674

    Association for Computing Machinery. ISBN 1581131674. doi: 10.1145/307400.307435. URL https://doi.org/10.1145/307400.307435

  191. [2020]

    URL https://aclanthology.org/2020

    Association for Computational Linguistics. URL https://aclanthology.org/2020. acl-main.39/

    2020

  192. [2021]

    URL https://proceedings.mlr.press/v139/krueger21a.html

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.