Pith. sign in

REVIEW 3 major objections 5 minor 46 references

Rethinking Invariance in In-context Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that in-context learning can be made invariant to the order of context examples without sacrificing accuracy, by giving each example a leave-one-out encoding that never sees its own label yet still depends on all other…

desk verdict InvICL is a clever and empirically useful architecture, but its formal non-leakage characterization contradicts the paper's own definition and the 'principled design' story doesn't hold as written. read the letter →

arxiv 2505.04994 v1 pith:63NTWAVU submitted 2025-05-08 cs.CL cs.AI

Lizhe Fang , Yifei Wang , Khashayar Gatmiry , Lei Fang , Yisen Wang This is my paper · ORCID
classification cs.CLcs.AI
keywords in-contextlearningpermutationinvarianceinformationnon-leakagecontextinterdependenceattentionmaskleave-one-outencodinglengthgeneralizationpromptordersensitivity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's target is a class of algorithms for in-context learning (ICL) in which the model's prediction for a test query is unchanged when the order of the context examples is permuted. The authors claim that previous invariant ICL methods underperform standard auto-regressive ICL because they sacrifice one of two properties: information non-leakage, meaning the prediction for each example never sees that example's own label, and context interdependence, meaning each example's encoding depends on the other examples. The paper proposes InvICL, a two-stage scheme with leave-one-out encodings, and reports that it satisfies permutation invariance, non-leakage, and interdependence simultaneously. Empirically, it reports accuracy gains over both invariant and non-invariant baselines on most of a large benchmark suite, plus better length extrapolation and out-of-distribution generalization.

What carries the argument

The central object is the attention mask over context examples, interpreted as the adjacency matrix of a message-passing digraph. The argument runs on three equivalences: permutation invariance holds if and only if the mask is one of three constant patterns (full, diagonal, or off-diagonal); information non-leakage holds if and only if the mask can be reordered to lower-triangular, meaning the message-passing graph is acyclic; and the conjunction of the two forces a diagonal, bag-of-examples mask, in which no context example attends to any other. To recover interdependence, the paper introduces the leave-one-out (LOO) attention mask as the carrying mechanism: each example is first encoded by its own bag-of-examples attention so that its encoding never contains its own label, then re-encoded by attending to all other examples' independent encodings, and finally the test example aggregates the LOO encodings. A duplicated-input 'unrolling' runs both stages in a single forward pass at the same $O(n^2)$ attention complexity as the baselines.

What would settle it

Run the paper's own definition as a probe on a model with the diagonal (bag-of-examples) mask: fix all context examples except one, flip that example's label $y_i$, and check whether the model's prediction for that same example $x_i$ changes. If it changes, the diagonal mask leaks under Definition 3.2 and Proposition 3.6 fails, so the claimed principled derivation of the two-stage design collapses even if the implemented InvICL architecture remains non-leaky.

Watch

Extended reading notes

Core claim

The paper's central claim is that the three desiderata an invariant ICL algorithm should satisfy—permutation invariance, information non-leakage, and context interdependence—are jointly achievable, and that earlier invariant methods underperform because each sacrifices one of the last two. The paper proves a mask-level characterization: over the context examples, permutation invariance forces the attention mask into one of three forms (full attention, strictly diagonal, or strictly off-diagonal); information non-leakage forces the message-passing graph to be reorderable to lower-triangular; and the intersection of these conditions is a diagonal, bag-of-examples mask in which each example is aggregated independently. Interdependence is then restored in a second stage: each example is pre-encoded by bag-of-examples attention over all other examples but itself (a leave-one-out encoding), and the test example attends to these leave-one-out encodings. The paper claims that this design, implemented in parallel by duplicating the input sequence, outperforms auto-regressive, prefix, and bag-of-examples ICL on most of 142 real-world tasks, with the largest advantages in out-of-distribution settings and in length extrapolation beyond the trained number of demonstrations.

Load-bearing premise

The load-bearing premise is that an example attending to itself in the attention mask does not count as leaking its own label; if self-attention is treated as leakage, the proof that a diagonal mask preserves non-leakage, and with it the derivation of the two-stage leave-one-out design, no longer follows.

Editorial extensions

If this is right

  • Prompt-order sensitivity in ICL becomes a solved problem in the settings tested: InvICL reports zero order sensitivity under random permutations, so expensive search for a good prompt ordering becomes unnecessary.
  • Length generalization improves: models meta-trained with 8 demonstrations stay accurate from 1 to 16 demonstrations, and synthetic models trained on 40 in-context examples extrapolate beyond 40 where auto-regressive ICL degrades.
  • Out-of-distribution accuracy is where invariance pays most: on target tasks in unseen domains, InvICL beats every non-invariant baseline on all seven settings, and every invariant baseline on six of the seven.
  • The invariance comes at the same asymptotic cost: the duplicated-input parallel implementation keeps inference time and $O(n^2)$ attention complexity at the level of prefix or bag-of-examples ICL.
  • From the gradient-descent perspective, InvICL approximately implements full gradient descent on the regression objective, differing only by a second-order term, whereas auto-regressive ICL emulates online gradient descent and bag-of-examples ICL computes gradients only at the initial point.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I would flag that the proof of Proposition 3.5 (Appendix D.3) explicitly allows self-loops on the diagonal when declaring the mask lower-triangular, and a diagonal self-attention entry lets an example attend to its own value, which encodes its own label; on the strictest reading of Definition 3.2 that is leakage, so the formal derivation of the two-stage design is weaker than the implemented archi
  • Proposition 3.4's third mask (cross-attention between different examples with no self-attention) is invariant, but the paper's own acyclicity criterion marks it as leaky because pairs of examples share edges in both directions; testing whether that mask actually copies labels would probe whether the directed-acyclic-graph condition is the right formalization of information non-leakage.
  • The optimization analysis suggests a measurement the paper does not report: since Theorem C.1 says InvICL tracks full gradient descent, its in-context predictions on synthetic linear regression should approach the least-squares estimator more closely than auto-regressive ICL's do, and that gap could be quantified directly.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that the poor performance of prior permutation-invariant ICL algorithms stems from violating two additional desiderata: information non-leakage (the prediction for a context example must not depend on its own label) and context interdependence (each example's encoding should depend on other examples). It formalizes these as Definitions 3.1-3.3, proves in Propositions 3.4-3.6 that a single attention mask satisfying invariance and non-leakage must be diagonal (BoE), and then augments BoE with a leave-one-out pre-encoding step to restore context interdependence. The resulting InvICL is implemented in parallel by duplicating the context sequence and using a two-stage mask. Experiments on synthetic linear regression, sparse linear regression, decision trees, OOD shifts, and MetaICL finetuning on 142 tasks with GPT-2/GPT-Neo/Pythia report faster convergence, better length extrapolation, and higher average accuracy than AR, Prefix, PCW, SAICL, BatchICL, and NoPE baselines. Appendix C gives a gradient-descent approximation theorem. Code is publicly released.

Significance. Conditional on the correctness of the theoretical derivation, the paper would make a solid contribution: it identifies two design axes that are intuitive but previously scattered, gives a clean practical recipe, and demonstrates that order invariance need not cost accuracy. The evaluation is unusually broad: three base models, seven MetaICL settings, OOD subsets, multiple synthetic function classes, ablations of the mask and positional encoding, and a sensitivity measure that confirms exact permutation invariance (0.00 in Table 4). The parallel implementation and released code address the main practical concern with leave-one-out schemes. The principal reservation is that the 'principled design' chain in Section 3.2 is not internally consistent as stated, so the contribution is better described as a well-engineered architecture with a plausible formal motivation, pending repair of the propositions.

major comments (3)
  1. [Section 3.2, Proposition 3.5 and Appendix D.3] The non-leakage characterization in Proposition 3.5 is inconsistent with Definition 3.2. The proof explicitly allows self-loops on the diagonal ('Since the original graph allows self-loop, which corresponds to the diagonal elements, the adjacency matrix is lower triangular'), but a diagonal attention entry at (i,i) lets row i aggregate h_i, which by construction contains y_i, so the prediction f_i depends on y_i. A minimal counterexample is n=1 with mask M=[0]: the mask is lower triangular, yet the output read from h_1 after self-attention changes when y_1 changes. Consequently Propositions 3.5 and 3.6 do not establish that a diagonal mask is the only invariant non-leaky single-layer choice, and the derivation of the leave-one-out design from these propositions does not follow. The implemented InvICL may still be non-leaky because the second-stage LOO attention excludes the example's own label embedding, but the stated formal equivalence needs repair, for example by requiring strictly lower triangular masks (no self-loops) and by specifying how residual connections are treated in the graph abstraction.
  2. [Section 3.2 and Algorithm 1] The graph abstraction used in the propositions treats each context example as a single node with hidden state h_i = (x_i,y_i), whereas the implemented Algorithm 1 keeps h_x and h_y separate and updates h_y^{(k)} from other examples' embeddings. The propositions classify masks on the n-node joint graph, but the actual InvICL uses a duplicated sequence and a bipartite LOO attention pattern that is not among the single-layer masks {M1,M2,0} classified in Proposition 3.4. The paper should state explicitly that the two-stage design goes outside the single-layer mask classification and prove non-leakage and invariance for the two-stage mask directly. As written, the claim that InvICL is the unique principled combination of the three desiderata is not supported by the formal results.
  3. [Section 4.2, Table 2] The abstract and conclusion claim that InvICL 'surpasses' AR ICL in most benchmarks, but in the 'All target tasks' block several differences are within one standard deviation, for example HR→LR 45.1±1.31 versus 43.4±0.76, CLASS→CLASS 42.9±0.86 versus 43.4±1.36, and the average 42.4±0.87 versus 41.9±1.15. Only the 'Target tasks in unseen domains' block shows consistent and larger gains. Please provide significance tests or paired comparisons across seeds and temper the wording, or present the OOD setting as the primary empirical claim.
minor comments (5)
  1. [Figure 4 and Figure 6] Figure 4 and Figure 6(b) report length-generalization and decision-tree results without error bars; adding standard deviations over random seeds would make the claims easier to evaluate.
  2. [Appendix A.2] The BatchICL baseline uses the 15-th layer for aggregation with no sensitivity analysis; please clarify whether this choice was fixed a priori or selected on the evaluation tasks.
  3. [General presentation] There are several typos and formatting issues, including 'Desicion tree' in Figure 6, 'infty' in place of '-\infty' in Appendix D.2, and the inconsistently typeset author reference 'V on Oswald' in the body and bibliography.
  4. [Section 5, Theoretical Complexity] The discussion counts the number of '0' entries in the mask and concludes InvICL has the same order as Prefix ICL and twice the cost of AR ICL, while Table 3 reports nearly identical wall-clock times; please reconcile the analytical count with the measured inference time and note any memory overhead beyond the reported 14%.
  5. [Definition 3.3] Context interdependence is defined existentially, so any non-constant aggregation satisfies it; the paper should clarify whether this weak notion is intended or whether a stronger, quantitative notion (e.g., sensitivity to all other examples) is needed to support claims like 'maximizing context interdependence'.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the InvICL derivation is self-contained (abstract desiderata, in-appendix proofs, external baselines, no fit-to-target), with only non-load-bearing background self-citations; the diagonal self-loop allowance in Proposition 3.5's proof is a formal correctness gap, not a circular reduction.

full rationale

The derivation chain is essentially self-contained, and the empirical claims are not constructionally forced. The three desiderata are defined abstractly on the ICL function f (Definitions 3.1-3.3), not on the proposed attention mask, so the architecture is not produced by re-naming an input property. Proposition 3.4 is proved in Appendix D.2 (Lemmas D.1-D.2) rather than imported from prior work; Proposition 3.6 is the conjunction of Propositions 3.4 and 3.5; and Theorem C.1 is an existence result following the external Von Oswald et al. (2023) parametrization, with Section C explicitly stating that 'the actual weight may not be strictly follow its parametrization', so the theorem is not used to certify the empirical gains. No parameter is fitted to the reported benchmarks: the synthetic tasks have a known closed-form optimum (least squares), and the real-world protocol follows the external MetaICL framework with a disclosed truncation equalization (Table 5), unseen-domain splits, and a doubled-input control (Table 8). The only author-overlap citations (Wang et al. 2024; Wang et al. 2025) occur in the introduction as background on ICL and carry no load in the derivation. One formal flaw is flagged explicitly as a correctness risk rather than circularity: Definition 3.2 requires f_i to be invariant to y_i, yet the proof of Proposition 3.5 (Appendix D.3) permits self-loops - 'Since the original graph allows self-loop, which corresponds to the diagonal elements, the adjacency matrix is lower triangular' - so a diagonal BoE mask, which Proposition 3.6 certifies as non-leaky, lets row i attend to a value vector containing y_i, and the claimed equivalence between non-leakage and lower-triangularity does not follow from Definition 3.2. The implemented LOO mask may still be non-leaky (its second stage excludes example i), so this is an invalid supporting inference, not a reduction of the output to the input. Accordingly, no circular step is identified; the score of 2 reflects only the minor, non-load-bearing self-citations.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard transformer attention as message passing, on the i.i.d. and permutation-symmetry prior, and on the paper's characterization of non-leakage via acyclicity. The non-leakage-to-lower-triangular equivalence is the weakest load-bearing axiom because its proof allows self-loops, which are precisely the label-access route Definition 3.2 forbids. No novel physical entities are introduced; the duplicated input and LOO mask are algorithmic devices. No global constants are fitted to make the results come out; the only hand-set number is the 8-example training context used to align truncation rates.

free parameters (1)
  • Training context size for real-world experiments = 8 examples
    Chosen (reduced from MetaICL's 16) so that InvICL's duplicated input has a truncation rate comparable to standard ICL (Table 5). This affects the fairness of the comparison but is not fitted to the target claim.
assumptions (5)
  • standard math Permutation invariance of ICL outputs is equivalent to permutation equivariance of context-example embeddings under a single self-attention layer (Lemma D.2).
    Used in the proof of Prop 3.4; assumes no positional encoding and a single attention layer.
  • domain assumption The attention mask on context examples is the only controllable factor determining invariance; the test example always attends to all context examples.
    Section 3.2 states the test edge is trivial, which restricts the design space to the context sub-mask.
  • domain assumption Context examples are i.i.d., hence the true data distribution is permutation invariant and respecting this symmetry should improve generalization.
    Section 3.1 and the Discussion cite invariant-classifier theory; this motivates but does not prove the performance gain.
  • ad hoc to paper Information non-leakage is equivalent to the message-passing graph being acyclic except self-loops, so the mask can be reordered to lower triangular (Prop 3.5, Appendix D.3).
    Load-bearing and questionable: self-loops mean xi attends to its own label yi, which is exactly the leakage Definition 3.2 forbids.
  • standard math Under the Von Oswald et al. parametrization, one Transformer layer corresponds to one gradient descent step; InvICL's update is then GD plus a second-order term (Theorem C.1).
    Borrowed from prior theory; the authors acknowledge it is an existence proof, not a description of trained weights.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Invariance in In-context Learning." pith.science (2026). https://pith.science/paper/63NTWAVU

@misc{pith2026250504994,
  author       = {Pith},
  title        = {Pith review of: Rethinking Invariance in In-context Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/63NTWAVU}},
  note         = {Machine review of arXiv:2505.04994}
}
read the original abstract

In-Context Learning (ICL) has emerged as a pivotal capability of auto-regressive large language models, yet it is hindered by a notable sensitivity to the ordering of context examples regardless of their mutual independence. To address this issue, recent studies have introduced several variant algorithms of ICL that achieve permutation invariance. However, many of these do not exhibit comparable performance with the standard auto-regressive ICL algorithm. In this work, we identify two crucial elements in the design of an invariant ICL algorithm: information non-leakage and context interdependence, which are not simultaneously achieved by any of the existing methods. These investigations lead us to the proposed Invariant ICL (InvICL), a methodology designed to achieve invariance in ICL while ensuring the two properties. Empirically, our findings reveal that InvICL surpasses previous models, both invariant and non-invariant, in most benchmark datasets, showcasing superior generalization capabilities across varying input lengths. Code is available at https://github.com/PKU-ML/InvICL.

Figures

Figures reproduced from arXiv: 2505.04994 by the authors.

Figure 1
Figure 1. Performance of existing ICL algorithms un [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The attention masks of four types of ICL, corresponding to different types of ICL methods. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. ICL performance of different models that are trained with [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: The length generalization behavior of InvICL [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: The symmetric positional encoding applied in our work. [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: ICL performance on sparse linear regression and decision tree. [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: ICL performance on OOD tasks. The training dataset remains consistent with Section [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Intermediate results for InvICL and AR ICL on the linear regression setting. The line colors [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: The squared error at different training epochs. We set the number of context examples to [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: The linear probing results on InvICL and AR ICL. [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 20 canonical work pages

  1. [1]

    In-context examples selection for machine translation

    Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. In-context examples selection for machine translation. arXiv preprint arXiv:2212.02437, 2022

    arXiv 2022

  2. [3]

    What learning algorithm is in-context learning? investigations with linear models

    Ekin Aky \"u rek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In ICLR, 2022

    work page 2022

  3. [4]

    Transformers as statisticians: Provable in-context learning with in-context algorithm selection

    Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. arXiv preprint arXiv:2306.04637, 2023

    arXiv 2023

  4. [5]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In ICML, 2023

    2023

  5. [6]

    On the sample complexity of learning under geometric stability

    Alberto Bietti, Luca Venturi, and Joan Bruna. On the sample complexity of learning under geometric stability. In NeurIPS, 2021

    work page 2021

  6. [7]

    GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow , March 2021

    Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow , March 2021. URL https://doi.org/10.5281/zenodo.5297715

    doi:10.5281/zenodo.5297715 2021

  7. [8]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020

    2020

  8. [9]

    Scaling in-context demonstrations with structured attention

    Tianle Cai, Kaixuan Huang, Jason D Lee, and Mengdi Wang. Scaling in-context demonstrations with structured attention. In ICML 2023 Workshop on Efficient Systems for Foundation Models, 2023

    work page 2023

Show all 46 references
  1. [10]

    On the relation between sensitivity and accuracy in in-context learning

    Yanda Chen, Chen Zhao, Zhou Yu, Kathleen McKeown, and He He. On the relation between sensitivity and accuracy in in-context learning. arXiv preprint arXiv:2209.07661, 2022

    2022 arXiv

  2. [11]

    Positional information matters for invariant in-context learning: A case study of simple function classes

    Yongqiang Chen, Binghui Xie, Kaiwen Zhou, Bo Han, Yatao Bian, and James Cheng. Positional information matters for invariant in-context learning: A case study of simple function classes. arXiv preprint arXiv:2311.18194, 2023

    2023 arXiv

  3. [12]

    Introduction to algorithms

    Thomas H Cormen, Charles E Leiserson, Ronald L Rivest, and Clifford Stein. Introduction to algorithms. MIT press, 2022

    2022

  4. [13]

    Why can gpt learn in-context? language models secretly perform gradient descent as meta optimizers

    Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models secretly perform gradient descent as meta optimizers. arXiv preprint arXiv:2212.10559, 2022

    2022 arXiv

  5. [14]

    Causallm is not optimal for in-context learning

    Nan Ding, Tomer Levinboim, Jialin Wu, Sebastian Goodman, and Radu Soricut. Causallm is not optimal for in-context learning. arXiv preprint arXiv:2308.06912, 2023

    2023 arXiv

  6. [15]

    Transformers learn higher-order optimization methods for in-context learning: A study with linear models

    Deqing Fu, Tianqi Chen, Robin Jia, and Vatsal Sharan. Transformers learn higher-order optimization methods for in-context learning: A study with linear models. In NeurIPS 2023 Workshop on Mathematics of Modern Machine Learning, 2023

    2023

  7. [16]

    What can transformers learn in-context? a case study of simple function classes

    Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. In NeurIPS, 2022

    2022

  8. [17]

    Opt-iml: Scaling language model instruction meta learning through the lens of generalization

    Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint arXiv:2212.1...

    2022 arXiv

  9. [18]

    The impact of positional encoding on length generalization in transformers

    Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. In NeurIPS, 2024

    2024

  10. [19]

    In-context learning learns label relationships but is not conventional learning

    Jannik Kossen, Yarin Gal, and Tom Rainforth. In-context learning learns label relationships but is not conventional learning. In ICLR, 2024

    2024

  11. [20]

    Jiachang Liu, Dinghan Shen, Yizhe Zhang, William B Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Archit...

    2022

  12. [21]

    Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In ACL, 2022

    2022

  13. [22]

    Noisy channel language model prompting for few-shot text classification

    Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Noisy channel language model prompting for few-shot text classification. In ACL, 2022 a

    2022

  14. [23]

    Metaicl: Learning to learn in context

    Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. Metaicl: Learning to learn in context. In NAACL, 2022 b

    2022

  15. [24]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018

    2018

  16. [25]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. 2019

    2019

  17. [26]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21 0 (1): 0 5485--5551, 2020

    2020

  18. [27]

    Parallel context windows improve in-context learning of large language models

    Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. Parallel context windows improve in-context learning of large language models. arXiv preprint arXiv:2212.10947, 2022

    2022 arXiv

  19. [28]

    Do pretrained transformers really learn in-context by gradient descent? arXiv preprint arXiv:2310.08540, 2023

    Lingfeng Shen, Aayush Mishra, and Daniel Khashabi. Do pretrained transformers really learn in-context by gradient descent? arXiv preprint arXiv:2310.08540, 2023

    2023 arXiv

  20. [29]

    Giryes, G

    Jure Sokolić, R. Giryes, G. Sapiro, and M. Rodrigues. Generalization error of invariant classifiers. In AISTATS, 2016

    2016

  21. [30]

    The exact sample complexity gain from invariances for kernel regression

    Behrooz Tahmasebi and Stefanie Jegelka. The exact sample complexity gain from invariances for kernel regression. In NeurIPS, 2023

    2023

  22. [31]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017

    2017

  23. [32]

    Transformers learn in-context by gradient descent

    Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, Jo \ a o Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In ICML, 2023

    2023

  24. [33]

    Uncovering mesa-optimization algorithms in transformers

    Johannes von Oswald, Eyvind Niklasson, Maximilian Schlegel, Seijin Kobayashi, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Max Vladymyrov, Razvan Pascanu, et al. Uncovering mesa-optimization algorithms in transformers. arXiv preprint arXiv:2309.05858, 2023

    2023 arXiv

  25. [34]

    Can in-context learning really generalize to out-of-distribution tasks? In ICLR, 2025

    Qixun Wang, Yifei Wang, Yisen Wang, and Xianghua Ying. Can in-context learning really generalize to out-of-distribution tasks? In ICLR, 2025

    2025

  26. [35]

    A theoretical understanding of self-correction through in-context alignment

    Yifei Wang, Yuyang Wu, Zeming Wei, Stefanie Jegelka, and Yisen Wang. A theoretical understanding of self-correction through in-context alignment. In NeurIPS, 2024

    2024

  27. [36]

    Finetuned language models are zero-shot learners

    Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In ICLR, 2021

    2021

  28. [37]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022

    2022

  29. [38]

    Self-adaptive in-context learning

    Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Lingpeng Kong. Self-adaptive in-context learning. arXiv preprint arXiv:2212.10375, 2022

    2022 arXiv

  30. [39]

    Addressing order sensitivity of in-context demonstration examples in causal language models

    Yanzheng Xiang, Hanqi Yan, Lin Gui, and Yulan He. Addressing order sensitivity of in-context demonstration examples in causal language models. arXiv preprint arXiv:2402.15637, 2024

    2024 arXiv

  31. [40]

    An explanation of in-context learning as implicit bayesian inference

    Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In ICLR, 2021

    2021

  32. [41]

    Batch-icl: Effective, efficient, and order-agnostic in-context learning

    Kaiyi Zhang, Ang Lv, Yuhan Chen, Hansen Ha, Tao Xu, and Rui Yan. Batch-icl: Effective, efficient, and order-agnostic in-context learning. arXiv preprint arXiv:2401.06469, 2024

    2024 arXiv

  33. [42]

    Calibrate before use: Improving few-shot performance of language models

    Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In ICML, 2021

    2021

  34. [43]

    Xun Zheng, Bryon Aragam, Pradeep Ravikumar, and E. Xing. Dags with no tears: Continuous optimization for structure learning. In NeurIPS, 2018

    2018

  35. [44]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  36. [45]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  37. [46]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  38. [47]

    Xing , booktitle = NeurIPS , year = 2018 , @inproceedings sokoli2016generalization, title = Generalization Error of Invariant Classifiers , author = Jure Sokolić and R

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2018 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.