Pith. sign in

REVIEW 2 major objections 10 minor 1 cited by

Task Generalization With AutoRegressive Compositional Structure: Can Learning From $D$ Tasks Generalize to $D^{T}$ Tasks?

T0 review · 2 major / 10 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proves that an autoregressively compositional task class can be learned from about D training tasks and generalized to all $D^T$ tasks, and shows Transformers with chain-of-thought achieve this on sparse parity, arithmetic, and…

desk verdict The ARC framework and the O(D log D) bound are real, but the parity experiments don't instantiate the theorem's uniform-product sampling, so the headline empirical validation is looser than claimed. read the letter →

arxiv 2502.08991 v2 pith:UVTIJ6BC submitted 2025-02-13 cs.LG stat.ML

Amirhesam Abedsoltan , Huaqing Zhang , Kaiyue Wen , Hongzhou Lin , Jingzhao Zhang , Mikhail Belkin This is my paper · ORCID
classification cs.LGstat.ML MSC 68Q3268T07
keywords taskgeneralizationautoregressivecompositionalstructurein-contextlearningchain-of-thoughtsparseparitysamplecomplexityscaling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks when a model trained on a handful of tasks can solve a huge family of unseen tasks. It proposes an answer for task families that are autoregressively compositional: every task is a sequence of $T$ small conditional steps, and at each step there are only $D$ possible subtasks, so the full family has $D^T$ tasks. The main theoretical result is that sampling about $D \ln(100\,D\,T)$ training tasks uniformly is enough for a learner to identify every subtask and then recover an unseen task from a few demonstrations, with error probability at most $0.02$. The paper then shows that Transformers with chain-of-thought reasoning exhibit exactly this exponential generalization on sparse parity, arithmetic, and translation tasks, while standard in-context learning without chain-of-thought does not.

What carries the argument

The central object is the AutoRegressive Compositional (ARC) task class: a task is a tuple $(P_{\theta_1},\ldots,P_{\theta_T})$ of conditional distributions, with $P_{\theta_t}$ drawing $y_t$ from $x$ and $y_{<t}$; each $\theta_t$ has $D$ choices, so the class has $D^T$ tasks. The proof machinery is coupon-collector coverage: $n_\theta = D\ln(100\,D\,T)$ random tasks cover every subtask at every position with high probability, so the training stage can learn the true subtask families by maximum likelihood, and the inference stage then identifies the unseen task's subtasks one by one using a total-variation-based pairwise discrimination test (Lemma A.1).

What would settle it

Train a Transformer with CoT on sparse parity with $d=30, k=15$ while withholding every task whose second secret index is 5; if the model's accuracy on those held-out tasks stays near chance, the per-component coverage condition is necessary, not merely sufficient. A second check is to construct an ARC task class with two subtasks whose conditionals differ by TV distance smaller than $1/\sqrt{\ell}$ and confirm that inference demonstrations no longer identify the task.

Watch

Extended reading notes

Core claim

Under the AutoRegressive Compositional (ARC) model, task generalization reduces to per-component coverage of the atomic subtask families plus a distribution-discrimination test at inference. The paper proves Theorem 3.3: if each of the $D$ subtasks at each of $T$ positions has conditional distributions separated from all alternatives by total-variation distance at least $c>0$, then a learner that sees $n_\theta \ge D\ln(100\,D\,T)$ uniformly sampled training tasks, each with enough demonstrations, and then sees $\ell \ge 2\ln(100\,T\,n_\theta)/c^2$ demonstrations of a new task, recovers the new task's full subtask tuple with error probability at most $0.02$. Sparse parity with chain-of-thought is an ARC$(d,k)$ class with $c=1/2$, so about $d\ln(100\,k\,d)$ training tasks suffice; experiments reach above 98% accuracy on unseen parity tasks, including 306 training tasks covering about 155 million tasks for $d=30, k=15$. The paper also reports that adversarial task selection breaks the guarantee: omitting all tasks with second coordinate $s_2=5$ leaves accuracy at chance, while omitting pairs of values that still appear separately does not.

Load-bearing premise

The guarantee collapses unless distinct atomic operations at the same position produce conditionals that can be distinguished from a few demonstrations (total-variation separation $c>0$), and unless the randomly sampled training tasks happen to include every atomic operation at every position.

Editorial extensions

If this is right

  • Training-task complexity grows as $\sim D\log(DT)$, so increasing the number of compositional steps $T$ (and with it the class size $D^T$) barely raises the number of tasks that must be seen.
  • Representation decides the breadth $D$: with chain-of-thought, sparse parity changes from ARC$({d\choose k},1)$ with $\sim d^k$ training tasks to ARC$(d,k)$ with $\sim d\log d$ tasks, matching the observation that plain ICL fails while CoT succeeds.
  • For $d=30, k=15$, training on 306 tasks yields above 98% accuracy on roughly 155 million unseen parity tasks.
  • Task sampling matters: an adversarial training set that excludes one coordinate value at one position ($s_2=5$) leaves generalization at chance, while excluding a pair of values that still appear separately does not.
  • The same scaling appears in arithmetic (ARC$(2,d-1)$) and multi-step translation (ARC$(D,T-1)$), with translation showing a linear-in-$T$ dependence attributed to error accumulation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If ARC coverage is the right model, benchmark construction should count how many distinct atomic operations appear at each reasoning position in the training set; a model trained on tasks that cover all operation-position pairs should generalize, and one that misses any pair should fail on that slice.
  • The $1/c^2$ dependence on the separation constant suggests tasks with noisy or near-identical atomic operations need quadratically more demonstrations; testing this on arithmetic with small versus large numeric ranges would be a direct extension.
  • The paper's inference algorithm assumes the input distribution $P(x)$ is known; an editor-level extension is that under heavy covariate shift between training and inference inputs, the TV discrimination statistic may mis-calibrate, predicting degraded task generalization.
  • The missing-coordinate failure can be turned into a cheap audit: for any proposed CoT decomposition, remove all training tasks containing a particular atomic operation at a particular step and check whether accuracy collapses.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 10 minor

Summary. The paper studies task generalization under an "AutoRegressive Compositional" (ARC) structure, in which each task is an ordered composition of T subtasks drawn from D possible conditionals per position, so the task class has size D^T. The main theoretical result (Theorem 3.3) is that, under a compositional identifiability assumption (Assumption 3.2), a learner can identify any unseen task with error probability at most 0.02 after training on n_θ ≥ D ln(100 D T) tasks sampled uniformly from the full Cartesian product, each with sufficiently many demonstrations, plus ℓ ≥ 2 ln(100 T n_θ)/c^2 inference demonstrations; the proof combines a coupon-collector per-component coverage argument with a total-variation-based distribution discrimination test. The paper further shows that sparse parity with chain-of-thought (CoT) fits the ARC(d,k) template and presents experiments in which Transformers trained on roughly 3 d ln d parity tasks reach above 98% accuracy on held-out tasks, including a d=30,k=15 setting with a nominal task class of about 155 million combinations, and similar scaling behavior in arithmetic and multi-step translation. An adversarial task-selection experiment shows that removing all tasks with a particular coordinate value at position 2 causes chance-level performance, illustrating sensitivity to per-coordinate coverage.

Significance. The theoretical framework is a clean, parameter-free derivation from the stated assumptions, and the coupon-collector plus distribution-discrimination proof is a valid existence argument. If the empirical claims are taken at face value, the paper would provide a quantitative explanation of why CoT representations enable exponential task generalization, a finding of considerable interest to the ICL and compositional generalization communities. The paper's strongest points are that the theorem produces an explicit O(D log(DT)) task complexity, the empirical scaling law is predicted before fitting, and the linear-probe analysis offers a mechanistic check that the model identifies per-step subtasks. The adversarial missing-coordinate experiment is a valuable falsifiable prediction tied to the coverage condition. However, the connection between the empirical protocol and the theorem's assumptions is not fully specified, and at least one hidden oracle assumption (knowledge of the input marginal) is unstated; these issues need to be resolved before the empirical and theoretical claims can be considered as jointly established.

major comments (2)
  1. [Section 3.5, Table 1, Lemma A.2] The empirical parity task class is counted as C(d,k) (Table 1), but Theorem 3.3 requires tasks sampled uniformly from the Cartesian product Θ = ×_{t=1}^T Θ_t with |Θ_t| = D, which makes each position's subtask marginally uniform and independently covered. The paper does not specify how the CoT order of the secret indices is generated. Under the natural sorted order, the t-th CoT coordinate is the t-th order statistic of a uniform k-subset, whose marginal is not uniform: for d=30,k=15, the largest possible first coordinate has probability about 1/C(30,15) ≈ 6×10^{-9} and is almost surely absent from 306 training tasks. Because held-out tasks are sampled uniformly, the reported 98.10% accuracy therefore does not demonstrate generalization to the 155 million tasks; it is compatible with the model succeeding only on tasks whose per-position subtask values appeared in training, which is exactly the coverage regime of Lemma A.2. The authors should specify the ordering, stratify the held-out evaluation by whether each coordinate value was seen in training, or change the empirical task family to a true Cartesian product (and report the corresponding P(d,k) or d^k count). Without one of these changes, the claimed agreement between the empirical scaling and Theorem 3.3 is not established.
  2. [Section 3.4, Algorithm 2] Algorithm 2's discrimination test computes the expected values of its statistic under the candidate joint distributions (e.g., the sums over X × Y^t in line 5), which requires exact knowledge of the input marginal P(x) and of every candidate conditional distribution in P_{Ξ_t}. Neither Theorem 3.3 nor Assumption 3.2 states that the learner has access to P(x); the empirical Transformer does not have access to the analytic form of P(x) or the hypothesis class. This is a hidden oracle assumption. The theorem statement should either explicitly assume that the learner knows P(x) and the candidate family, or the algorithm should be modified to estimate these quantities from the available training and inference demonstrations. As written, the theorem's sample-complexity guarantee is for an algorithm with stronger access than the experimental setup provides.
minor comments (10)
  1. [Section 3.1] 'finte set' should be 'finite set'.
  2. [Section 4.2] 'praity' should be 'parity'.
  3. [Appendix B, Algorithm 3] The algorithm title 'Training Stage with Distribution Dislimination' contains a typo; it should be 'Discrimination'.
  4. [Assumption B.1] 'fixed tv marigin' should be 'fixed TV margin'.
  5. [Appendix D] The optimizer is called 'Wadam', but the cited reference (Kingma & Ba, 2015) is for Adam; the text should either say 'AdamW' or cite the appropriate source.
  6. [Section A.2] The proof header 'Proof of Theorem 3.5' should refer to Corollary 3.5, since the statement is labeled Corollary 3.5 in the main text.
  7. [Table 1 and Figure 2] The accuracy numbers are reported without error bars or variance across seeds; because the test evaluation randomly selects at most 200 held-out tasks, reporting standard errors would strengthen the scaling-law claim.
  8. [Section 3.3] The symbol D is used both for the number of subtasks per position and for training datasets (D_i, D_train, D_infer); this overloading makes the notation in the proofs harder to follow.
  9. [Equation (2)] The input marginal is denoted P_x(x) in Equation (2) but P(x) in Equation (1); unify the notation.
  10. [Appendix C] The paper correctly notes that the theory assumes an unbounded number of per-task demonstrations, while the experiments use context length 40; the non-asymptotic Appendix B requires an additional margin r whose value is not verified for the experimental tasks, so the empirical agreement with O(D log D) is a finite-sample phenomenon not directly covered by Theorem 3.3.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the O~(D) task-generalization bound is a direct derivation from the stated ARC identifiability and coverage assumptions, and the parity, arithmetic, and translation experiments test the predicted scaling rather than fitting it.

full rationale

The theoretical chain is self-contained. Theorem 3.3 is proved from Assumption 3.2 by (i) an MLE training stage that recovers each training task's per-step conditionals as nx tends to infinity, (ii) a total-variation-based discrimination lemma (Lemma A.1) for identifying the unseen task's conditionals at inference, and (iii) a coupon-collector argument showing that D ln(100 D T) uniformly sampled tasks cover every per-step subtask with high probability. The conclusion that O~(D) tasks suffice for D^T tasks is not equivalent to its input by construction: it relies on the identifiability margin c > 0 and on the per-component coverage condition, and it yields a concrete finite bound on the number of inference demonstrations. No parameter is fitted and then renamed a prediction; the experiments evaluate a Transformer at the theoretically suggested 3 d ln d training-task scale (and multiples) and report accuracy on held-out tasks, while the linear-probe experiment is an additional mechanistic check rather than an input to the bound. The citation to Wen et al. (2025), which shares authors with this paper, is used only to motivate that CoT decomposes parity into small XOR steps; the ARC(d, k) representation is written out explicitly in Section 3.5 and is not imported as an unverified premise. The paper itself flags the finite-sample limitation for translation error accumulation, which strengthens rather than hides the scope of the theory. The concern that parity tasks are counted as C(d, k) subsets while Theorem 3.3 assumes uniform sampling over [d]^T is a legitimate mismatch between the experiment's task distribution and the theorem's coverage argument, but it is an external-validity or correctness issue, not a circularity: the theorem's bound does not assume the experimental result. Therefore no circular step is exhibited.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The theoretical claim rests on the ARC decomposition and Assumption 3.2, plus an unstated modeling assumption about access to P(x). The empirical scaling uses hand-chosen hyperparameters but no fitted constants in the prediction. No new physical or conceptual entities are introduced.

free parameters (2)
  • Task training multiplier (3) = 3
    Used to set n_theta = 3 d ln d in all parity and arithmetic experiments. Chosen by hand; the theorem only requires a constant larger than 1, so the results are not fitted to make the scaling law hold.
  • Context length (40) = 40
    Used for parity and arithmetic experiments. Appendix C states this value works reasonably well for dimensions up to 20; it is chosen by hand and not fitted to optimize test accuracy.
assumptions (4)
  • domain assumption Assumption 3.2 (Compositional Identifiability): each step has D true subtask families inside a finite hypothesis class; distinct subtasks have TV at least c>0; false hypotheses have nonzero TV from the truth.
    Load-bearing for both training-stage MLE identification and inference-stage discrimination. If distinct operations are not separable by demonstrations, the O(D) task bound does not hold.
  • domain assumption The task class is exactly ARC(d,T): each task is an independent product of per-step parameter choices, and the input marginal P(x) is fixed across tasks.
    The exponential size D^T and the coverage argument both follow from this product structure. Real task families must be shown to fit this decomposition.
  • ad hoc to paper The learner can compute exact expected values of the discrimination statistic, which requires knowledge of the input marginal P(x) and the full conditional distributions.
    Algorithm 2 sums over X x Y^t using P(x); the paper never states that the learner knows P(x), and Transformers do not have closed-form access to it. This is an unstated assumption of the proof.
  • standard math Standard concentration and information inequalities (Hoeffding, Pinsker, union bound) hold as stated.
    Used in Lemma A.1 and in the proof of Theorem 3.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Task Generalization With AutoRegressive Compositional Structure: Can Learning From $D$ Tasks Generalize to $D^{T}$ Tasks?." pith.science (2026). https://pith.science/paper/UVTIJ6BC

@misc{pith2026250208991,
  author       = {Pith},
  title        = {Pith review of: Task Generalization With AutoRegressive Compositional Structure: Can Learning From $D$ Tasks Generalize to $D^T$ Tasks?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UVTIJ6BC}},
  note         = {Machine review of arXiv:2502.08991}
}
abstract

Large language models (LLMs) exhibit remarkable task generalization, solving tasks they were never explicitly trained on with only a few demonstrations. This raises a fundamental question: When can learning from a small set of tasks generalize to a large task family? In this paper, we investigate task generalization through the lens of autoregressive compositional structure, where each task is a composition of $T$ operations, and each operation is among a finite family of $D$ subtasks. This yields a total class of size $D^T$. We first show that generalization to all $D^T$ tasks is theoretically achievable by training on only $\widetilde{O}(D)$ tasks. Empirically, we demonstrate that Transformers achieve such exponential task generalization on sparse parity functions via In-context Learning (ICL) and chain-of-thought (CoT) reasoning. We further show generalization in arithmetic and translation, beyond parity functions.

Figures

Figures reproduced from arXiv: 2502.08991 by the authors.

Figure 1
Figure 1. We train a Transformer to learn parity functions through In-Context Learning (ICL): given a demonstration sequence (x1, f(x1)), . . . ,(xn, f(xn)), infer the target f(xquery) from a new input xquery. Each function f defines a distinct learning task. In this prototype experiment, tasks are sampled from the parity function family P arity(10, 2) with secret length k = 2 and bit length d = 10, totaling 45 tasks. To eval… view at source ↗
Figure 2
Figure 2. Test accuracy on unseen tasks. For parity task: D = d as the ambient dimension and T = k as the number of secret indices. We show that the empirical scaling closely follows the theoretical scaling of D ln(D). (A) For a fixed D = 15, as T increases, the test accuracy on unseen tasks remains similar, even though the total number of tasks (∼ D T ) grows exponentially with T. (B) For a fixed secret length is 3, as D inc… view at source ↗
Figure 4
Figure 4. Task generalization for language translation task: D is the number of languages and T is the length of steps. A possible explanation is error accumulation across sequen￾tial steps—longer sequences require higher precision in intermediate steps to maintain accuracy. This contrasts with our theoretical analysis, which focuses on asymptotic scal￾ing and does not explicitly account for compounding errors in finite-sampl… view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: shows that providing sufficient context length during both training and inference is crucial for strong performance. Empirically, we observed that a context length of 40 works reasonably well across all experiments with dimensions up to d = 20 [PITH_FULL_IMAGE:figures…
Figure 6
Figure 6. Figure 6: ICL without CoT even fails to generalize in distribution. D. Experiment Details Model and optimization. We used the transformers library from Hugging Face (Wolf et al., 2020) to instantiate and train our GPT-2 model from scratch. In all experiments, we used a 3-layer, …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Extrapolation by Association: Length Generalization Transfer in Transformers

    cs.CL 2025-06 conditional novelty 5.0 of 10

    Length generalization on a short-trained main task can be inherited from a longer-trained related auxiliary task trained jointly with it.

Reference graph

Works this paper leans on

39 extracted references · 28 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    How far can transformers reason? the locality barrier and inductive scratchpad

    Abbe, E., Bengio, S., Lotfi, A., Sandon, C., and Saremi, O. How far can transformers reason? the locality barrier and inductive scratchpad. Advances in Neural Information Processing Systems, 2024

    work page 2024

  3. [3]

    and Bengio, Y

    Alain, G. and Bengio, Y. Understanding intermediate layers using linear classifier probes. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings , 2017

    work page 2017

  4. [4]

    An, S., Lin, Z., Fu, Q., Chen, B., Zheng, N., Lou, J.-G., and Zhang, D. How do in-context examples affect compositional generalization? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 11027--11052, 2023

    work page 2023

  5. [5]

    Invariant risk minimization

    Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez-Paz, D. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019

    arXiv 1907

  6. [6]

    and Goyal, A

    Arora, S. and Goyal, A. A theory for emergence of complex skills in language models. arXiv preprint arXiv:2307.15936, 2023

    arXiv 2023

  7. [7]

    Transformers as statisticians: Provable in-context learning with in-context algorithm selection

    Bai, Y., Chen, F., Wang, H., Xiong, C., and Mei, S. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=liMSqUuVg9

    work page 2023

  8. [8]

    and Urner, R

    Ben-David, S. and Urner, R. Domain adaptation--can quantity compensate for quality? Annals of Mathematics and Artificial Intelligence, 70: 0 185--202, 2014

    work page 2014

Show all 39 references
  1. [9]

    Understanding in-context learning in transformers and LLM s by learning to learn discrete functions

    Bhattamishra, S., Patel, A., Blunsom, P., and Kanade, V. Understanding in-context learning in transformers and LLM s by learning to learn discrete functions. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=ekeyCgeRfC

    2024

  2. [10]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  3. [11]

    Learning bounds for importance weighting

    Cortes, C., Mansour, Y., and Mohri, M. Learning bounds for importance weighting. Advances in neural information processing systems, 23, 2010

    2010

  4. [12]

    S., Koushik, J., Singh, A., and P \'o czos, B

    Du, S. S., Koushik, J., Singh, A., and P \'o czos, B. Hypothesis transfer learning via transformation functions. Advances in neural information processing systems, 30, 2017

    2017

  5. [13]

    Distributionally robust losses for latent covariate mixtures

    Duchi, J., Hashimoto, T., and Namkoong, H. Distributionally robust losses for latent covariate mixtures. Operations Research, 71 0 (2): 0 649--664, 2023

    2023

  6. [14]

    L., Jiang, L., Lin, B

    Dziri, N., Lu, X., Sclar, M., Li, X. L., Jiang, L., Lin, B. Y., Welleck, S., West, P., Bhagavatula, C., Le Bras, R., et al. Faith and fate: Limits of transformers on compositionality. Advances in Neural Information Processing Systems, 36: 0 70293--70332, 2023

    2023

  7. [15]

    S., and Valiant, G

    Garg, S., Tsipras, D., Liang, P. S., and Valiant, G. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35: 0 30583--30598, 2022

    2022

  8. [16]

    Instruct-skillmix: A powerful pipeline for llm instruction tuning

    Kaur, S., Park, S., Goyal, A., and Arora, S. Instruct-skillmix: A powerful pipeline for llm instruction tuning. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  9. [17]

    Measuring compositional generalization: A comprehensive method on realistic data

    Keysers, D., Sch \"a rli, N., Scales, N., Buisman, H., Furrer, D., Kashubin, S., Momchev, N., Sinopalnikov, D., Stafiniak, L., Tihon, T., Tsarkov, D., Wang, X., van Zee, M., and Bousquet, O. Measuring compositional generalization: A comprehensive method on realistic data. In I...

    2020

  10. [18]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015

    2015

  11. [19]

    and Martinet, G

    Kpotufe, S. and Martinet, G. Marginal singularity and the benefits of labels in covariate-shift. The Annals of Statistics, 49 0 (6): 0 3299--3323, 2021

    2021

  12. [20]

    and Baroni, M

    Lake, B. and Baroni, M. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In International conference on machine learning, pp.\ 2873--2882. PMLR, 2018

    2018

  13. [21]

    Near-optimal linear regression under distribution shift

    Lei, Q., Hu, W., and Lee, J. Near-optimal linear regression under distribution shift. In International Conference on Machine Learning, pp.\ 6164--6174. PMLR, 2021

    2021

  14. [22]

    E., Papailiopoulos, D., and Oymak, S

    Li, Y., Ildiz, M. E., Papailiopoulos, D., and Oymak, S. Transformers as algorithms: Generalization and stability in in-context learning. In International Conference on Machine Learning, pp.\ 19565--19594. PMLR, 2023

    2023

  15. [23]

    and Stachenfeld, K

    Lippl, S. and Stachenfeld, K. When does compositional structure yield compositional generalization? a kernel theory. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  16. [24]

    Ma, C., Pathak, R., and Wainwright, M. J. Optimally tackling covariate shift in rkhs-based nonparametric regression. The Annals of Statistics, 51 0 (2): 0 738--761, 2023

    2023

  17. [25]

    Towards deep learning models resistant to adversarial attacks

    Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018

    2018

  18. [26]

    Domain adaptation: Learning bounds and algorithms

    Mansour, Y., Mohri, M., and Rostamizadeh, A. Domain adaptation: Learning bounds and algorithms. In Annual Conference on Learning Theory, 2009

    2009

  19. [27]

    M., Yang, F., Duchi, J., and Liang, P

    Raghunathan, A., Xie, S. M., Yang, F., Duchi, J., and Liang, P. Understanding and mitigating the tradeoff between robustness and accuracy. In International Conference on Machine Learning, pp.\ 7909--7919. PMLR, 2020

    2020

  20. [28]

    Y., Padmakumar, V., Joshi, N., Kazemi, M., Kim, N., and He, H

    Saparov, A., Pang, R. Y., Padmakumar, V., Joshi, N., Kazemi, M., Kim, N., and He, H. Testing the general deductive reasoning capacity of large language models using ood examples. Advances in Neural Information Processing Systems, 36: 0 3083--3105, 2023

    2023

  21. [29]

    M., Von Oswald, J., Pascanu, R., Sacramento, J., and Steger, A

    Schug, S., Kobayashi, S., Akram, Y., Wolczyk, M., Proca, A. M., Von Oswald, J., Pascanu, R., Sacramento, J., and Steger, A. Discovering modular solutions that generalize compositionally. In International Conference on Learning Representations, 2024

    2024

  22. [30]

    Out-of-distribution generalization via composition: a lens through induction heads in transformers

    Song, J., Xu, Z., and Zhong, Y. Out-of-distribution generalization via composition: a lens through induction heads in transformers. Proceedings of the National Academy of Sciences, 122 0 (6): 0 e2417182122, 2025

    2025

  23. [31]

    Covariate shift adaptation by importance weighted cross validation

    Sugiyama, M., Krauledat, M., and M \"u ller, K.-R. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research, 8 0 (5), 2007

    2007

  24. [32]

    H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W

    Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. Emergent abilities of large language models. Transactions on Machine Learning Researc...

    2022

  25. [33]

    From sparse dependence to sparse attention: Unveiling how chain-of-thought enhances transformer sample efficiency

    Wen, K., Zhang, H., Lin, H., and Zhang, J. From sparse dependence to sparse attention: Unveiling how chain-of-thought enhances transformer sample efficiency. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  26. [34]

    Compositional generalization from first principles

    Wiedemer, T., Mayilvahanan, P., Bethge, M., and Brendel, W. Compositional generalization from first principles. Advances in Neural Information Processing Systems, 36, 2024

    2024

  27. [35]

    Transformers: State-of-the-art natural language processing

    Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., and Brew, J. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processin...

    2020

  28. [36]

    Do large language models have compositional ability? an investigation into limitations and scalability

    Xu, Z., Shi, Z., and Liang, Y. Do large language models have compositional ability? an investigation into limitations and scalability. In Conference on Language Modeling, 2024

    2024

  29. [37]

    Towards a theoretical framework of out-of-distribution generalization

    Ye, H., Xie, C., Cai, T., Li, R., Li, Z., and Wang, L. Towards a theoretical framework of out-of-distribution generalization. Advances in Neural Information Processing Systems, 34: 0 23519--23531, 2021

    2021

  30. [38]

    Can models learn skill composition from examples? Advances in Neural Information Processing Systems, 37: 0 102393--102427, 2024

    Zhao, H., Kaur, S., Yu, D., Goyal, A., and Arora, S. Can models learn skill composition from examples? Advances in Neural Information Processing Systems, 37: 0 102393--102427, 2024

    2024

  31. [39]

    Zhou, K., Liu, Z., Qiao, Y., Xiang, T., and Loy, C. C. Domain generalization: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45 0 (4): 0 4396--4415, 2022

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.