Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Distinct Computations Emerge From Compositional Curricula in In-Context Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Putting subtask examples before composite examples in one prompt lets transformers answer an unseen two-step exponential on the first try.

desk verdict Solid behavioral result showing in-context subtask curricula boost zero-shot compositional generalization; the mechanistic story is plausible but explicitly correlational. read the letter →

arxiv 2506.13253 v1 pith:YW746FIU submitted 2025-06-16 cs.LG cs.AIcs.CL

Jin Hwa Lee , Andrew K. Lampinen , Aaditya K. Singh , Andrew M. Saxe This is my paper
classification cs.LGcs.AIcs.CL
keywords in-contextlearningcompositionalgeneralizationcurriculummodulararithmetictransformerslinearprobingzero-shotinferencetaskcomposition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that the ordering of examples inside a single prompt changes what a transformer computes, not just how well it performs. On a modular double-exponential task $b^{a^x} \bmod P$, the authors compare training on isolated examples of the full task against training on sequences that first show examples of the two single-exponential subtasks and then examples of their composition. Models trained with this in-context curriculum can answer the compositional task on the very first, zero-shot exemplar and make fewer errors than models trained without it, even though both regimes see the same input-output pairs. If this holds, it means that curriculum-like structure in text—component ideas followed by their combination—can push sequence models toward genuinely compositional inference.

What carries the argument

The load-bearing object is the in-context curriculum sequence itself: a block of $m$ exemplars for $y=a^x \bmod P$, a block of $m$ exemplars for $y=b^x \bmod P$, then $n$ exemplars for $y=b^{a^x} \bmod P$, with total length $2m+n$ and loss weighting that keeps the composite task's contribution to one-third of the total. The mathematical hinge is the identity $b^{a^x} \bmod P = b^{(a^x \bmod (P-1))} \bmod P$, which defines the intermediate values that a compositional solution must represent. The paper's evidence comes from comparing this curriculum condition with a vanilla condition that sees the same exemplar-level pairs but never mixes subtask and composite examples in one sequence, from linear probes that decode the intermediate values from hidden states, and from mismatch sequences that break the subtask-composition correspondence.

What would settle it

Run a causal intervention: patch or ablate the early-layer attention heads that the authors found attending from the compositional block back to the curriculum blocks (or path-patch the intermediate-value representations into the output logits). If zero-shot accuracy on the compositional task is unchanged when those heads are removed, the decoded intermediate values are not on the causal path and the compositional mechanism is not established.

Watch

Extended reading notes

Core claim

The central claim is that a subtask curriculum presented in context induces a compositional computation that vanilla training does not. On the double-exponential task $b^{a^x} \bmod P$, which the paper rewrites via the modular-exponent identity $b^{a^x} \bmod P = b^{(a^x \bmod (P-1))} \bmod P$, curriculum-trained transformers learn to decode and compose the intermediate values $a^x \bmod (P-1)$ and $b$: linear probes recover these values in the compositional block of unseen sequences, especially in early-to-middle layers, and a mismatch experiment in which the subtask blocks use parameters $(a,b)$ different from the compositional block makes the curriculum model fail. The authors interpret this as evidence that the model performs step-by-step composition rather than learning a direct lookup, and they show that the balance between this compositional strategy and ordinary few-shot learning shifts continuously with curriculum design—short compositional blocks favor composition, long compositional blocks mix in vanilla few-shot learning and change the order in which subtasks and the composite task are learned.

Load-bearing premise

The mechanistic conclusion rests on the assumption that linear-probe decodability of intermediate values reflects the computation the network actually performs, which the paper acknowledges is correlational; if the probe reads off information that is stored but not causally used, the behavioral gains could stand while the compositional-computation claim would not.

Editorial extensions

If this is right

  • A model trained with subtask-then-composition blocks answers an unseen $(a,b)$ double-exponential query on the first composite exemplar, something the vanilla model cannot do.
  • Curriculum and vanilla models see the same input-output pairs, so the robustness gain is attributable to sequence-level correlation structure rather than to seeing more data.
  • The intermediate values $a^x \bmod (P-1)$ and $b$ are linearly decodable in the compositional block for curriculum models but not vanilla models, supporting a step-by-step compositional readout.
  • Breaking the subtask-composition correspondence with mismatched parameters makes curriculum models fail, showing their composite answers depend on the subtask information in context.
  • Longer compositional blocks shift the model toward a mixture of compositional and vanilla few-shot strategies and change which task is learned first, so the strategy is graded rather than binary.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If this transfers to natural language, documents that introduce components before their combination—instructions, proofs, articles—may elicit more compositional processing in LLMs than randomly ordered passages; a testable prediction is that permuting such passages reduces zero-shot multi-step inference accuracy.
  • The results suggest that under short context budgets, curricula matter most: when few composite examples fit, relying on subtask information is the only route to zero-shot accuracy, while longer contexts let the model fall back on ordinary few-shot learning.
  • The authors' $P=41$ results already hint at a boundary condition: if the single-exponential blocks are too short for the model to learn the subtask, the curriculum benefit disappears; this predicts that curriculum gains depend on subtask learnability within the block length, which could be probed directly.
  • Because the mechanism evidence is correlational, a natural next step is path-patching: if the early-layer heads are causally responsible, then curriculum-trained models become a tractable test bed for circuit-level accounts of compositional in-context learning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies how presenting a compositional subtask curriculum in context changes the computations learned by transformers on a modular double-exponential task b^{a^x} mod P. Curriculum-trained models see single-exponential subtask examples (for a and b) followed by double-exponential compositional examples in the same sequence, while vanilla-trained models see matched exemplars without this in-context structure. The authors report that curriculum-trained models achieve lower error at the first compositional query (zero-shot) and higher overall robustness when the context length is held fixed. Linear probes show that intermediate subtask values (a^x mod (P-1) and b) are more decodable in the compositional block for curriculum models, and a mismatch experiment shows that curriculum models fail when the compositional block uses different task parameters than the subtask blocks. The paper also examines how the length of the compositional block modulates the model's strategy between compositional and standard few-shot in-context learning, and how curriculum design changes the order in which tasks are learned during training.

Significance. If the behavioral claims hold, the paper contributes a useful data-distributional perspective to in-context learning: curriculum-like context structure can enable compositional zero-shot inference and shift the model's internal strategy. The controlled task design, matched exemplar sets, shuffled-label probe control, and the mismatch prediction are notable strengths. The authors are transparent about the correlational nature of the mechanistic evidence. However, the central mechanistic claim ('distinct computations') rests on linear probe decodability and a mismatch experiment that admits alternative explanations, and all behavioral results are based on only two seeds. The paper is likely to interest the ICL and interpretability communities, but the strength of the conclusions currently exceeds the strength of the evidence.

major comments (3)
  1. [Section 3.1, Figures 2-3] All behavioral comparisons are based on 2 data seeds with no error bars or significance tests. Given that Appendix B.2 shows that for P=41 only one of several curriculum designs improves robustness, the absence of variance estimates makes it difficult to assess whether the central robustness claim is reliable. Please add at least one additional seed for the key comparisons, or provide bootstrap confidence intervals or error bars for Figures 2 and 3.
  2. [Section 3.2, Discussion, Limitations] The claim that the curriculum model 'employs a compositional strategy' rests on linear probe decodability and the mismatch experiment. The authors acknowledge in Limitations that this evidence is correlational. The mismatch experiment does not rule out the alternative that the model has learned a within-sequence parameter-consistency prior (i.e., it expects all blocks in a sequence to share (a,b)) rather than genuinely composing the two subtask computations. Since the title and abstract make a mechanistic claim ('distinct computations emerge'), please either add causal evidence (e.g., path patching or activation intervention) or temper the mechanistic language to match the correlational evidence.
  3. [Appendix B.2, P=41] The explanation for the P=41 result, that only the (11-11-2) curriculum works because shorter subtask blocks do not allow the model to identify a and b, is post hoc. This weakens the generality claim in the main text that findings extend to other values of P. Please provide a principled test (e.g., varying subtask block length for P=59 to show a similar threshold, or a pre-registered criterion) or explicitly frame the P=41 result as a boundary condition rather than as confirmation.
minor comments (5)
  1. [Throughout] There are several typos: 'compositioanl' (Section 3.2), 'predominanlty' (Section 3.3), 'occruing' (Appendix E.1), 'curriculm' and 'lerning' (Figure 8 caption), and 'skewedness' (Appendix B.1, should be 'skewness').
  2. [Section 2.1, Figure 3 caption] The notation 'm-m-n' in Figure 3 is used before it is clearly defined; please define it at first use or replace it with '(m, m, n)'.
  3. [Figure 1] The panel labels (a)-(d) are referenced in the caption, but the layout in the displayed figure appears to order panels differently; please ensure the panel letters match the figure layout.
  4. [Appendix C] The linear probe training details are sparse; please specify the exact input representation (which layer activations at which token positions), the regularization hyperparameters, and the number of training iterations used for the probe classifier.
  5. [Appendix E.3] Please state whether the Savitzky-Golay filtering of the loss curves affects the interpretation of learning dynamics, and specify the window length in terms of checkpoints rather than only in arbitrary units.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation chain: the empirical behavioral and probe results are independently measured on held-out data, and the mechanistic limitation is explicitly acknowledged rather than disguised.

full rationale

This paper is an empirical study rather than a derivation, and no prediction or first-principles result is equivalent to its inputs by construction. The central behavioral claims—zero-shot inference and robustness differences between curriculum and vanilla training—are measured on held-out (a, b) pairs, with the two conditions matched for exemplar-level exposure and loss weighting. The linear-probe evidence is trained on an 80/20 split of unseen evaluation sequences and validated with a shuffled-label control, so probe decodability is not fitted to the behavior it is used to explain. The mismatch experiment tests a prediction of the compositional hypothesis and is not used to set constants. The authors explicitly state in the Limitations that the mechanistic evidence is correlational and that causal manipulation such as path patching would be necessary; this is an evidentiary limitation, not circularity. Self-citations appear in related-work and motivation contexts and are not load-bearing for the main empirical claims. No equation or fitted parameter reduces to the target result, so no circular step can be exhibited.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No fitted parameters in the sense of a derived theory; listed experimental constants are hand-chosen. Standard number-theoretic facts are used. The main interpretive assumption is that linear probe decodability indicates causal computation, which the authors flag as a limitation.

free parameters (3)
  • loss weight for compositional task = 1/3 of total loss
    Chosen by hand to equalize subtask and compositional task importance across curriculum and vanilla; not fitted, but relative weighting could influence whether compositional strategy emerges.
  • curriculum block lengths (m,n) = (11,2), (10,4), (8,8), (6,12), (4,16)
    Experimental manipulation; the paper shows strategy mixes vary with these lengths. Not fitted to data but a hand-chosen design axis central to the claims.
  • total context length = 48 tokens (24 pairs)
    Fixed context window; all curriculum designs fit within it; not fitted but constrains results.
assumptions (4)
  • standard math Euler's theorem identity b^(a^x) mod P = b^(a^x mod (P-1)) mod P
    Used in Section 3.2 and Appendix C.1 to define the 'intermediate value' a^x mod (P-1) that probes decode.
  • domain assumption Sampling a and b from primitive roots of P with P prime yields a deterministic modular exponential task with no collisions across x
    The task assumes primitive-root bases make the modular exponential well-defined and hard; standard number theory.
  • domain assumption Linear probe decodability is evidence of task-relevant computation
    The mechanistic claim in Section 3.2 relies on linear probes; authors explicitly note this is correlational in Limitations.
  • domain assumption Behavior on 8-layer transformers trained from scratch transfers to larger models
    The discussion motivates implications for LLMs, but no experiments at scale appear in this preprint; this is stated as future direction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Distinct Computations Emerge From Compositional Curricula in In-Context Learning." pith.science (2026). https://pith.science/paper/YW746FIU

@misc{pith2026250613253,
  author       = {Pith},
  title        = {Pith review of: Distinct Computations Emerge From Compositional Curricula in In-Context Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YW746FIU}},
  note         = {Machine review of arXiv:2506.13253}
}
read the original abstract

In-context learning (ICL) research often considers learning a function in-context through a uniform sample of input-output pairs. Here, we investigate how presenting a compositional subtask curriculum in context may alter the computations a transformer learns. We design a compositional algorithmic task based on the modular exponential-a double exponential task composed of two single exponential subtasks and train transformer models to learn the task in-context. We compare (a) models trained using an in-context curriculum consisting of single exponential subtasks and, (b) models trained directly on the double exponential task without such a curriculum. We show that models trained with a subtask curriculum can perform zero-shot inference on unseen compositional tasks and are more robust given the same context length. We study how the task and subtasks are represented across the two training regimes. We find that the models employ diverse strategies modulated by the specific curriculum design.

Figures

Figures reproduced from arXiv: 2506.13253 by the authors.

Figure 1
Figure 1. Overview of the setup and key results. a) Task schema. In curriculum training, each training sequence is composed of m exemplars for two single-exponential tasks defined by a and b, respectively, followed by n composite double-exponential task exemplars. In vanilla training, the model is trained with a sequence of 2m + n in-context exemplars for the double-exponential task defined by task parameters (a, b). b) Examp… view at source ↗
Figure 2
Figure 2. Errors across entire context in vanilla vs. curriculum model. Gray dotted lines indicate the task bound￾aries in the curriculum. The curriculum model generalizes to each subtask after few-shots (first two single exponential tasks) and once the compositional task is present, fewer errors occur compare to vanilla setting from the zero-shot on (after the second gray dotted line). We train 8-layer transformers with sinu… view at source ↗
Figure 3
Figure 3. In-context error counts of compositional task in vanilla model vs. curriculum models. We zoom into the compositional task block of [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (17 more)
Figure 4
Figure 4. Figure 4: Linear probe decoding accuracy of target [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: The curriculum model performance on mismatch sequences:single exponential tasks of pa￾rameter (a, b) followed by double exponential task with mismatching paramters (a ′ , b′ ). The mod￾els trained with curriculum (8-8-8) and (10-10-4) fail on the mismatching compositio…
Figure 6
Figure 6. Figure 6: In mismatch exper￾iment, the model trained with longer compositional context shows slight decreasing of er￾ror suggesting the model does not only relies on composi￾tional strategy but vanilla few￾shot learning strategy as well. We demonstrated that the in-context curri…
Figure 7
Figure 7. Figure 7: Evidence of mixed strategy between vanilla and compositional solution modulated by [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: a) Top: In the curriculum (4-4-16), the model can solve compositional task after many examples before subtask learning (pink before gray). The zero-shot loss decreases sharply only after the subtask lerning (blue after gray). Bottom: In the curriculm (10-10-4), the sub…
Figure 9
Figure 9. Figure 9: Example loss and performance evolution on the curriculum (8-8-8) model and the vanilla [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Additional visualization of error counts complementary to Figure 2-3. [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Error counts in vanilla vs. curriculum model for [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: We show what variables we are decoding in each block in main Figure 4. Since [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Control linear probe decoding. We used shuffled labels for linear probe training to validate [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: Linear probe decoding results for other curricula design. We observe that having in [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: Attention pattern from selected heads in layer 4-7. x-axis shows token position [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: Mismatch experiment result on other curricula designs. [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]
Figure 17
Figure 17. Figure 17: Extended result of main Figure 7. We show the decoding of the intermediate values in the [PITH_FULL_IMAGE:figures/full_fig_p019_17.png]
Figure 18
Figure 18. Figure 18: Extended result of main Figure 7. We show loss curve of al shots. The loss curve above [PITH_FULL_IMAGE:figures/full_fig_p020_18.png]
Figure 19
Figure 19. Figure 19: Extended result of main Figure 8 [PITH_FULL_IMAGE:figures/full_fig_p020_19.png]
Figure 20
Figure 20. Figure 20: Linear probe of linear probe of curriculum (10-10-4) and (4-4-16) settings at more [PITH_FULL_IMAGE:figures/full_fig_p021_20.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Pretraining Curricula Enable Selective Fine-tuning

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Imbalanced pretraining curricula disentangle task circuits in transformers, improving in-context learning and the selectivity of refusal fine-tuning relative to balanced training.

Reference graph

Works this paper leans on

60 extracted references · 25 canonical work pages · cited by 1 Pith paper

  1. [1]

    Curriculum learning

    Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. InProceedings of the 26th annual international conference on machine learning, pages 41–48, 2009

    2009

  2. [2]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  3. [3]

    Data distributional properties drive emer- gent in-context learning in transformers.Advances in Neural Information Processing Systems, 35:18878–18891, 2022

    Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional properties drive emer- gent in-context learning in transformers.Advances in Neural Information Processing Systems, 35:18878–18891, 2022

    work page 2022

  4. [4]

    Derivation by phase

    Noam Chomsky. Derivation by phase. 1999. URL https://api.semanticscholar.org/ CorpusID:118158028

    work page 1999

  5. [5]

    Real-world visual statistics and infants’ first-learned object names.Philosophical Transactions of the Royal Society B: Biological Sciences, 372(1711):20160055, 2017

    Elizabeth M Clerkin, Elizabeth Hart, James M Rehg, Chen Yu, and Linda B Smith. Real-world visual statistics and infants’ first-learned object names.Philosophical Transactions of the Royal Society B: Biological Sciences, 372(1711):20160055, 2017

    work page 2017

  6. [6]

    Curriculum learning for human compositional generalization.Proceedings of the National Academy of Sciences, 119(41): e2205582119, 2022

    Ronald B Dekker, Fabian Otto, and Christopher Summerfield. Curriculum learning for human compositional generalization.Proceedings of the National Academy of Sciences, 119(41): e2205582119, 2022

    work page 2022

  7. [7]

    Faith and fate: Limits of transformers on compositionality.Advances in Neural Information Processing Systems, 36, 2024

    Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, et al. Faith and fate: Limits of transformers on compositionality.Advances in Neural Information Processing Systems, 36, 2024

    2024

  8. [8]

    The effects of information order and learning mode on schema abstraction.Memory & cognition, 12(1):20–30, 1984

    Renee Elio and John R Anderson. The effects of information order and learning mode on schema abstraction.Memory & cognition, 12(1):20–30, 1984

    1984

Show all 60 references
  1. [9]

    Connectionism and cognitive architecture: A critical analysis.Cognition, 28(1-2):3–71, 1988

    Jerry A Fodor and Zenon W Pylyshyn. Connectionism and cognitive architecture: A critical analysis.Cognition, 28(1-2):3–71, 1988

    1988

  2. [10]

    Ueber sinn und bedeutung.Philosophical Review, 57(n/a):209, 1948

    Gottlob Frege. Ueber sinn und bedeutung.Philosophical Review, 57(n/a):209, 1948

    1948

  3. [11]

    Dissecting recall of factual associations in auto-regressive language models.arXiv preprint arXiv:2304.14767, 2023

    Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models.arXiv preprint arXiv:2304.14767, 2023

    2023 arXiv

  4. [12]

    Finding neurons in a haystack: Case studies with sparse probing.arXiv preprint arXiv:2305.01610, 2023

    Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing.arXiv preprint arXiv:2305.01610, 2023

    2023 arXiv

  5. [13]

    Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks.arXiv preprint arXiv:2406.02550, 2024

    Tianyu He, Darshil Doshi, Aritra Das, and Andrey Gromov. Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks.arXiv preprint arXiv:2406.02550, 2024

    2024 arXiv

  6. [14]

    In-context learning creates task vectors.arXiv preprint arXiv:2310.15916, 2023

    Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors.arXiv preprint arXiv:2310.15916, 2023

    2023 arXiv

  7. [15]

    Curriculum learning for motor skills

    Andrej Karpathy and Michiel Van De Panne. Curriculum learning for motor skills. InAdvances in Artificial Intelligence: 25th Canadian Conference on Artificial Intelligence, Canadian AI 2012, Toronto, ON, Canada, May 28-30, 2012. Proceedings 25, pages 325–330. Springer, 2012. 10

    2012

  8. [16]

    When can transformers compositionally generalize in-context?ICML Next Generation of Sequence Modeling Architectures Workshop, 2024

    Seijin Kobayashi, Simon Schug, Yassir Akram, Florian Redhardt, Johannes von Oswald, Razvan Pascanu, Guillaume Lajoie, and João Sacramento. When can transformers compositionally generalize in-context?ICML Next Generation of Sequence Modeling Architectures Workshop, 2024

    2024

  9. [17]

    Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks

    Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. InInternational conference on machine learning, pages 2873–2882. PMLR, 2018

    2018

  10. [18]

    Human-like systematic generalization through a meta- learning neural network.Nature, 623(7985):115–121, 2023

    Brenden M Lake and Marco Baroni. Human-like systematic generalization through a meta- learning neural network.Nature, 623(7985):115–121, 2023

    2023

  11. [19]

    Andrew Kyle Lampinen, Stephanie C. Y . Chan, Aaditya K. Singh, and Murray Shanahan. The broader spectrum of in-context learning, 2024. URL https://arxiv.org/abs/2412.03782

    2024 arXiv

  12. [20]

    Geometric signa- tures of compositionality across a language model’s lifetime.arXiv preprint arXiv:2410.01444, 2024

    Jin Hwa Lee, Thomas Jiralerspong, Lei Yu, Yoshua Bengio, and Emily Cheng. Geometric signa- tures of compositionality across a language model’s lifetime.arXiv preprint arXiv:2410.01444, 2024

    2024 arXiv

  13. [21]

    Why do animals need shaping? a theory of task composition and curriculum learning.arXiv preprint arXiv:2402.18361, 2024

    Jin Hwa Lee, Stefano Sarao Mannelli, and Andrew Saxe. Why do animals need shaping? a theory of task composition and curriculum learning.arXiv preprint arXiv:2402.18361, 2024

    2024 arXiv

  14. [22]

    Capturing gradience, continuous change, and quasi-regularity in sound, word, phrase, and meaning.The handbook of language emergence, pages 53–80, 2015

    James L McClelland. Capturing gradience, continuous change, and quasi-regularity in sound, word, phrase, and meaning.The handbook of language emergence, pages 53–80, 2015

    2015

  15. [23]

    Universal linguistic inductive biases via meta-learning

    R Thomas McCoy, Erin Grant, Paul Smolensky, Thomas L Griffiths, and Tal Linzen. Universal linguistic inductive biases via meta-learning. InProceedings of the Annual Meeting of the Cognitive Science Society, volume 42, 2020

    2020

  16. [24]

    Circuit component reuse across tasks in transformer language models.arXiv preprint arXiv:2310.08744, 2023

    Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Circuit component reuse across tasks in transformer language models.arXiv preprint arXiv:2310.08744, 2023

    2023 arXiv

  17. [25]

    Progress measures for grokking via mechanistic interpretability, 2023

    Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability, 2023. URL https://arxiv.org/ abs/2301.05217

    2023 arXiv

  18. [26]

    Emergent linear representations in world models of self-supervised sequence models.arXiv preprint arXiv:2309.00941, 2023

    Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models.arXiv preprint arXiv:2309.00941, 2023

    2023 arXiv

  19. [27]

    Curriculum learning for reinforcement learning domains: A framework and survey.Journal of Machine Learning Research, 21(181):1–50, 2020

    Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey.Journal of Machine Learning Research, 21(181):1–50, 2020

    2020

  20. [28]

    Show your work: Scratchpads for intermediate computation with language models.arXiv preprint arXiv:2112.00114, 2021

    Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models.arXiv preprint arXiv:2112.00114, 2021

    2021 arXiv

  21. [29]

    Competition dynamics shape algorithmic phases of in-context learning.arXiv preprint arXiv:2412.01003, 2024

    Core Francisco Park, Ekdeep Singh Lubana, Itamar Pres, and Hidenori Tanaka. Competition dynamics shape algorithmic phases of in-context learning.arXiv preprint arXiv:2412.01003, 2024

    2024 arXiv

  22. [30]

    Pedregosa, G

    F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V . Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python.Journal of Machine Learnin...

    2011

  23. [31]

    Grokking: Generalization beyond overfitting on small algorithmic datasets.CoRR, abs/2201.02177, 2022

    Alethea Power, Yuri Burda, Harrison Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets.CoRR, abs/2201.02177, 2022. URLhttps://arxiv.org/abs/2201.02177

    2022 arXiv

  24. [32]

    Measuring and narrowing the compositionality gap in language models.arXiv preprint arXiv:2210.03350, 2022

    Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models.arXiv preprint arXiv:2210.03350, 2022. 11

    2022 arXiv

  25. [33]

    Milena Rabovsky and James L McClelland. Quasi-compositional mapping from form to meaning: A neural network-based approach to capturing neural responses during human language comprehension.Philosophical Transactions of the Royal Society B, 375(1791): 20190313, 2020

    2020

  26. [34]

    Pretraining task diversity and the emergence of non-bayesian in-context learning for regression.Advances in Neural Information Processing Systems, 36, 2024

    Allan Raventós, Mansheej Paul, Feng Chen, and Surya Ganguli. Pretraining task diversity and the emergence of non-bayesian in-context learning for regression.Advances in Neural Information Processing Systems, 36, 2024

    2024

  27. [35]

    Human curriculum effects emerge with in-context learning in neural networks

    Jacob Russin, Ellie Pavlick, and Michael J Frank. Human curriculum effects emerge with in-context learning in neural networks. InProceedings of the Annual Meeting of the Cognitive Science Society, volume 46, 2024

    2024

  28. [36]

    Meta-learning with memory-augmented neural networks

    Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning with memory-augmented neural networks. InInternational conference on machine learning, pages 1842–1850. PMLR, 2016

    2016

  29. [37]

    The transient nature of emergent in-context learning in transformers.Advances in Neural Information Processing Systems, 36, 2024

    Aaditya Singh, Stephanie Chan, Ted Moskovitz, Erin Grant, Andrew Saxe, and Felix Hill. The transient nature of emergent in-context learning in transformers.Advances in Neural Information Processing Systems, 36, 2024

    2024

  30. [38]

    What needs to go right for an induction head? a mechanistic study of in-context learning circuits and their formation.arXiv preprint arXiv:2404.07129, 2024

    Aaditya K Singh, Ted Moskovitz, Felix Hill, Stephanie CY Chan, and Andrew M Saxe. What needs to go right for an induction head? a mechanistic study of in-context learning circuits and their formation.arXiv preprint arXiv:2404.07129, 2024

    2024 arXiv

  31. [39]

    Strategy coopetition explains the emergence and transience of in-context learning.arXiv preprint arXiv:2503.05631, 2025

    Aaditya K Singh, Ted Moskovitz, Sara Dragutinovic, Felix Hill, Stephanie CY Chan, and Andrew M Saxe. Strategy coopetition explains the emergence and transience of in-context learning.arXiv preprint arXiv:2503.05631, 2025

    2025 arXiv

  32. [40]

    BF Skinner Foundation, 2019

    Burrhus Frederic Skinner.The behavior of organisms: An experimental analysis. BF Skinner Foundation, 2019

    2019

  33. [41]

    On the proper treatment of connectionism.Behavioral and brain sciences, 11 (1):1–23, 1988

    Paul Smolensky. On the proper treatment of connectionism.Behavioral and brain sciences, 11 (1):1–23, 1988

    1988

  34. [42]

    Discovering the compositional structure of vector representations with role learning networks

    Paul Soulos, R Thomas McCoy, Tal Linzen, and Paul Smolensky. Discovering the compositional structure of vector representations with role learning networks. InProceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 238– 254, 2020

    2020

  35. [43]

    Metaphysics Research Lab, Stanford Univer- sity, fall 2024 edition, 2024

    Zoltán Gendler Szabó.Compositionality. Metaphysics Research Lab, Stanford Univer- sity, fall 2024 edition, 2024. URL https://plato.stanford.edu/archives/fall2024/ entries/compositionality/

    2024

  36. [44]

    What do you learn from context? probing for sentence structure in contextualized word representations

    Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. What do you learn from context? probing for sentence structure in contextualized word representations. InInternational Conferenc...

    2019

  37. [45]

    A deep hierarchical approach to lifelong learning in minecraft

    Chen Tessler, Shahar Givony, Tom Zahavy, Daniel Mankowitz, and Shie Mannor. A deep hierarchical approach to lifelong learning in minecraft. InProceedings of the AAAI conference on artificial intelligence, volume 31, 2017

    2017

  38. [46]

    Function vectors in large language models.arXiv preprint arXiv:2310.15213, 2023

    Eric Todd, Millicent L Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models.arXiv preprint arXiv:2310.15213, 2023

    2023 arXiv

  39. [47]

    Matching networks for one shot learning.Advances in neural information processing systems, 29, 2016

    Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning.Advances in neural information processing systems, 29, 2016

    2016

  40. [48]

    Towards understanding chain-of-thought prompting: An empirical study of what matters

    Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, Will Wu, Luke Zettlemoyer, and Huan Sun. Towards understanding chain-of-thought prompting: An empirical study of what matters. InProc. of The 61st Annual Meeting of the Association for Computational Linguistics, 2023. URLhttps:/...

    2023 arXiv

  41. [49]

    Learning to reinforcement learn

    Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv:1611.05763, 2016

    2016 arXiv

  42. [50]

    A survey on curriculum learning.IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):4555–4576, 2021

    Xin Wang, Yudong Chen, and Wenwu Zhu. A survey on curriculum learning.IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):4555–4576, 2021

    2021

  43. [51]

    Emergent abilities of large language models.arXiv preprint arXiv:2206.07682, 2022

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models.arXiv preprint arXiv:2206.07682, 2022

    2022 arXiv

  44. [52]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  45. [53]

    When do curricula work? InInternational Conference on Learning Representations, 2021

    Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. When do curricula work? InInternational Conference on Learning Representations, 2021. URL https://openreview.net/forum? id=tW4QEInpni

    2021

  46. [54]

    An explanation of in-context learning as implicit bayesian inference

    Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. InInternational Conference on Learning Representations, 2022

    2022

  47. [55]

    Do large language models latently perform multi-hop reasoning?, 2024

    Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. Do large language models latently perform multi-hop reasoning?, 2024. URL https://arxiv.org/ abs/2402.16837

    2024 arXiv

  48. [56]

    Which attention heads matter for in-context learning?arXiv preprint arXiv:2502.14010, 2025

    Kayo Yin and Jacob Steinhardt. Which attention heads matter for in-context learning?arXiv preprint arXiv:2502.14010, 2025

    2025 arXiv

  49. [57]

    Parsel: Algo- rithmic reasoning with language models by composing decompositions.Advances in Neural Information Processing Systems, 36:31466–31523, 2023

    Eric Zelikman, Qian Huang, Gabriel Poesia, Noah Goodman, and Nick Haber. Parsel: Algo- rithmic reasoning with language models by composing decompositions.Advances in Neural Information Processing Systems, 36:31466–31523, 2023

    2023

  50. [58]

    The clock and the pizza: Two stories in mechanistic explanation of neural networks, 2023

    Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks, 2023. URL https://arxiv.org/abs/ 2306.17844

    2023 arXiv

  51. [59]

    Least-to-most prompting enables complex reasoning in large language models

    Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, et al. Least-to-most prompting enables complex reasoning in large language models. InThe Eleventh International Conference on Learning Repres...

    2023

  52. [60]

    Since bax modP=b ax mod (P−1) modP , the intermediate values from task a, bthat we try to decode from compositional task blocks are ax modP and b

    task parameter b, which are required for the compositional computation of bax . Since bax modP=b ax mod (P−1) modP , the intermediate values from task a, bthat we try to decode from compositional task blocks are ax modP and b. We train probes for the intermediate values in the...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.