Pith. sign in

REVIEW 3 major objections 5 minor 31 references

Train Once, Reuse Everywhere: Generalizable Implicit In-Context Learning by Routing Attention

T0 review · 3 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read One training run can equip an LLM to do in-context learning on new tasks at zero-shot cost.

desk verdict A genuinely new mechanism for implicit ICL—attention logit routing—with solid but narrow OOD evidence; the 'diverse new tasks' claim outruns the evaluation. read the letter →

arxiv 2509.22854 v2 pith:4SFF2AUY submitted 2025-09-26 cs.CL

Jiaqian Li , Yanshu Li , Ligong Han , Ruixiang Tang , Wenya Wang This is my paper
classification cs.CL
keywords implicitin-contextlearningattentionroutingprincipalICLdirectionslogitszero-shotgeneralizationlargelanguagemodelslow-rankbiasout-of-distributiontransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes In-Context Routing (ICR), an implicit in-context learning method that captures a generalizable ICL pattern in the attention logits of large language models. Instead of injecting task-specific shift vectors into residual streams, ICR extracts low-dimensional 'Principal ICL Directions' from query and key projections across multiple domains, then uses a small trainable router to add a query-conditioned low-rank bias to attention logits during zero-shot inference. This enables a train-once-and-reuse framework: after one round of training on a few domains, ICR transfers directly to new, even out-of-domain, tasks without retrieval or retraining. If correct, this would give LLMs few-shot-like performance at zero-shot cost while avoiding the brittleness and task-specificity of prior implicit ICL approaches.

What carries the argument

The core mechanism is the PID-based attention routing: at each layer, a low-rank bias ΔA = γ·(Q U_q) diag(α) (K U_k)⊤ is added to the attention logits, where U_q and U_k are the top-r PCA directions of pooled last-token query/key projections (the Principal ICL Directions), γ is a per-head gate, and α is a per-layer routing vector produced by a small two-branch MLP conditioned on a frozen text-encoder representation of the query. This reparameterizes the attention kernel as I + U_q diag(α) U_k⊤, confining the intervention to a low-dimensional subspace of the query/key geometry.

What would settle it

Run pooled PCA on Q/K projections from two domains that share a strong spurious direction (e.g., both tasks are corrupted by the same input feature), then remove one domain and re-run PCA: if the top-r subspace shifts substantially, the shared-subspace assumption fails. Testing ICR on such paired domains should show its OOD gain disappear, while a per-task vector baseline remains competitive.

Watch

Extended reading notes

Core claim

ICR shows that the structural pattern underlying in-context learning can be extracted from attention projections rather than residual states. By pooling last-token query and key projections from ICL prompts on multiple domains and taking their top principal components, the method obtains reusable Principal ICL Directions (PIDs). A query-conditioned router then composes these directions into a low-rank bias added to attention logits, effectively reparameterizing the attention kernel. The paper argues this is the first implicit ICL method that can be directly adopted for zero-shot inference in diverse new tasks without retrieval or retraining, and reports consistent gains over vector-based bas

Load-bearing premise

The method assumes that domain-specific query/key variations cancel out when covariances are pooled, so that PCA on multi-domain projections recovers a shared, transferable ICL subspace; if those variations align instead, the extracted PIDs will not generalize to new tasks.

Editorial extensions

If this is right

  • ICR outperforms prior implicit ICL baselines that require task-specific retrieval or training, on both in-domain and out-of-domain datasets, and shows no performance collapse below zero-shot on any evaluated task.
  • ICR closely matches and sometimes surpasses few-shot prompting on in-domain tasks while using zero-shot prompts and cached PIDs plus a small router, making inference cheaper than explicit few-shot prompting.
  • The framework is train-once-and-reuse: one round of training on five domains transfers to seven unseen domains, including tasks with different label spaces, formats, and reasoning demands.
  • Ablations show that OOD robustness depends on the PCA-meaningful directions, not just low-rank routing; random orthogonal directions retain in-domain accuracy but collapse on OOD tasks.
  • The router's query-conditioned modulation is interpretable: it consistently upweights tokens related to reasoning and structural explanation across datasets, and layer/head/PID importance profiles are shared across semantically related tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If PIDs are truly domain-agnostic, the paper suggests that in-context learning in LLMs has a universal geometric signature in query/key space; one could attempt to transfer PIDs across models of different sizes or families, potentially amortizing the extraction step further.
  • The spiked covariance assumption implies a testable prediction: the top-r eigen-directions of pooled Q/K covariance should be stable under domain removal, and the eigengap should increase with the number of domains. Measuring this directly would indicate how many and which domains are needed for generalizable routing.
  • The router's input-conditioning opens a possible route to on-the-fly task identification: routing vectors might cluster by task type, and feeding an unlabeled query from a novel task could be used to infer which ICL direction to emphasize, without any labeled examples.
  • One could extend ICR to continual learning scenarios: as new tasks arrive, update only the router and possibly the PID subspace via incremental PCA, preserving earlier capabilities while adding new ones—this would test whether the shared ICL subspace truly remains stable under distribution shift.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes In-Context Routing (ICR), an implicit in-context learning method that operates on attention logits rather than on residual-stream vectors. ICR first extracts Principal ICL Directions (PIDs) by applying PCA to last-token query and key projections collected from explicit ICL prompts across five training domains, then trains a query-conditioned router with a small MLP to produce low-rank, head-gated additive biases for the attention logits of the last third of layers. The router is trained on zero-shot queries with cross-entropy, confidence-alignment, and sparsity losses while the backbone is frozen. The paper evaluates ICR on five in-domain and seven out-of-domain datasets using Llama2-7B, Qwen2.5-7B, and Llama3.1-8B, reporting consistent gains over prior vector-based implicit ICL baselines and claiming a train-once-and-reuse framework that generalizes to diverse new tasks without retrieval or retraining.

Significance. If the central claims hold, ICR is a useful step beyond vector-based implicit ICL: it moves the intervention into the attention geometry, makes the routing query-conditioned, and shows evidence that low-rank PCA directions extracted from multiple domains can be reused without per-task retraining. The paper has clear strengths: evaluation across 12 datasets and three LLMs, comprehensive ablations (PCA rank, random bases, loss components, domain alignment, routing layers), efficiency analysis, and released code. The main weaknesses are (i) the theoretical justification for why PIDs generalize is informal and rests on an unproven cancellation assumption, (ii) the reported numerical margins are sometimes very small with no error bars or significance tests, and (iii) the 'diverse new tasks' claim is not supported by the actual OOD benchmark, which consists of close variants of the training task families.

major comments (3)
  1. [Sec. 4.2 / Tables 1 and 7] The claim that ICR 'consistently outperforms' baselines is not backed by inferential statistics. The setup says each experiment is run with three seeds and averages are reported, but no standard deviations, confidence intervals, or significance tests are given. Several reported differences are tiny or even reversed: in Table 1 on Llama2-7B, SST-2 is 86.4 for both ICR and M2IV, and CSQA is 24.8 for both; on Qwen2.5-7B, TREC is 70.6 for ICR vs 70.8 for M2IV. The collapse counts (0 vs 2-4) could flip with a single seed. Please report per-seed results or at least error bars/paired tests for the headline comparisons.
  2. [Sec. 2.3 / Appendix A.3, Eq. (18)] The theoretical argument is load-bearing because it justifies PIDs as reusable directions, but Eq. (18) is stated without derivation. The term ρ_D is never defined, and the assertions that the numerator decreases with D and that gap_Q increases with N and D are not proven. Moreover, Eq. (6) already assumes that the domain-specific terms B_{q,d}Γ_{q,d}B^T_{q,d} 'average out toward isotropy'; this is precisely the condition that must hold for pooled PCA to recover the shared subspace, yet no diagnostic or experiment verifies it for the five chosen domains. Please provide a formal derivation or clearly frame the argument as a heuristic supported by the MATCHED-5 vs MATCHED-3 experiment.
  3. [Sec. 4.2 and Sec. D.3.1] The abstract and Sec. 4.2 describe the evaluation as spanning 'diverse new tasks' and claim that ICR is directly adoptable for 'zero-shot inference in diverse new tasks.' The seven OOD datasets do not support this wording. SST-5 and MR are sentiment classification, near-identical to training-domain SST-2; MRPC and CB are sentence-pair classification; COPA, CREAK, and AI2SciE are multiple-choice comprehension/reasoning tasks directly analogous to CSQA and PIQA. There are no structurally different tasks such as sequence labeling, generation, math/code, or non-English text. The evidence therefore supports interpolation across task-family variants, not generalization across diverse task types. Either add structurally out-of-domain tasks or limit the claim accordingly.
minor comments (5)
  1. [Appendix H / Table 12] The 'ICLness' token analysis is meant to support interpretability, but many of the top-50 tokens are subword fragments or generic tokens (e.g., dep, forme, fro, bro, catt, cyl, duch) rather than clearly reasoning-related words. The 'highlighted in red' formatting is lost in the text, and no quantitative comparison against a random or function-word baseline is provided. Please substantiate the claim that the selected tokens are especially tied to ICL.
  2. [Eq. (3) vs Eq. (10)] Equation (3) writes ΔA^l ∈ R^{T×T} as a layer-level bias shared across heads, while Eq. (10) applies a head-specific gate γ_{l,h}. Please clarify the dimensionality and broadcasting of the bias, especially how the shared low-rank term is combined with per-head gates.
  3. [Sec. 5.1 / Appendix H] The score definition uses 'stability_v = mean_v / (std_v + ε)' but ε is never specified. Please state its value or remove it.
  4. [Figure 4] The right panel showing Spearman correlations has small, overlapping labels and the cluster structure is hard to read. Consider a larger figure or a dendrogram-style plot.
  5. [Sec. 4.1] The paper reports using a single V100 GPU and deterministic settings, which is good for reproducibility, but the wall-clock time for PIDs extraction and router training is not reported. Including it would strengthen the efficiency discussion.

Circularity Check

1 steps flagged · score 1.0 of 10

Core ICR derivation and OOD evaluation are not circular; one peripheral analysis step (ICLness token scoring) is self-definitional but explicitly acknowledged and not load-bearing.

  1. self definitional [Appendix H ('ICLness' tokens), scoring definition and Table 12]
    "The final score is defined as scorev =stability v ·pos ratev ·log(1+borda v), which rewards tokens that are (i) strongly upweighted on average, (ii) consistently positive across datasets, and (iii) highly ranked across tasks. The top-50 tokens are listed in Table 12, with tokens strongly related to in-context reasoning or structural semantics ('ICLness' tokens) highlighted in red."

    The 'ICLness' property is operationalized as cross-dataset consistency of token-level log-probability shifts; the score explicitly rewards consistent positive upweighting across datasets. The paper then presents the top tokens selected by this criterion as evidence of 'generalizable ICLness,' so observing cross-dataset consistency among the top-scoring tokens is true by construction. The paper itself flags this ('One might argue that because we explicitly require consistency across datasets, the resulting tokens are trivially "cross-dataset"') and rebuts by noting that many consistent tokens are function words; that semantic characterization is post-hoc. This step is peripheral and does not support the main ICR prediction or OOD evaluation.

full rationale

The central derivation chain is not circular. PIDs are extracted by PCA from last-token Q/K projections collected from explicit ICL prompts on five training domains (Eq. 7), and the router is trained with supervised cross-entropy on those same domains (Eqs. 10-14). This is standard training on labeled data, not a prediction forced by construction. OOD performance is measured on seven held-out datasets (SST-5, MR, MRPC, CB, COPA, CREAK, AI2SciE) that were not used for extraction or training, so the 'train-once-and-reuse' claim is supported by genuine held-out evaluation rather than by re-reporting fitted values. The theoretical argument in Sec. 2.3 and Appendix A.3 is conditional: under the spiked covariance model with sufficiently diverse domain-specific subspaces that cancel in the pooled covariance, pooled PCA recovers the shared subspace. That is a stated identifiability assumption, and the Davis-Kahan bounds are standard perturbation results; the paper does not invoke the conclusion to justify the assumption. Self-citations (I2CL, M2IV, LIVE-related works) appear only as baselines, not as load-bearing justification for the method. The only self-referential component is the Appendix H 'ICLness' token scoring, which rewards cross-dataset consistency by design; the paper explicitly acknowledges this potential criticism and the step is peripheral. Overall circularity is minimal and the main empirical claims stand independently.

Assumptions & free parameters 6 free parameters · 5 assumptions · 1 invented entities

The method introduces a small set of hand-chosen hyperparameters (PCA rank, loss weights, layer range, prompt counts, training set size) and relies on domain assumptions about the spiked covariance structure of attention projections and the transferability of the router. No new physical entities are postulated; PIDs are data-derived constructs.

free parameters (6)
  • PCA rank r = 8
    Number of principal directions retained per layer; chosen by hand. Ablations show r=4 and r=12 alter results.
  • Loss weights lambda_conf, lambda_spar, lambda_gate = 0.01, 1e-3, 0.02
    Weights for confidence alignment, sparse routing, and gate sparsity in Eq. 14; chosen by hand.
  • Intervened layer set = last one-third of layers
    Routing applied only to layers 2L/3+1..L; this choice is motivated by ablation in Appendix G.3.
  • PIDs extraction prompt counts = 10k AGNews, 5k per remaining dataset
    Number of ICL prompts per domain used to build the ICL bases; affects PCA quality.
  • Router training set size = 25k queries (5k per domain)
    Training set size for the router; chosen arbitrarily and not ablated.
  • Frozen encoder choice = MiniLM
    Sentence encoder used for query conditioning; no ablation over encoder choice.
assumptions (5)
  • domain assumption Spiked covariance model for Q/K projections (Eq. 4)
    The last-token query/key projections follow a mixed spiked covariance with shared subspace, domain-specific terms, and isotropic noise. This is assumed, not verified directly.
  • domain assumption Domain-specific variations cancel in pooled covariance (Sec. 2.3)
    The paper argues that if domain-specific subspaces are diverse, their aggregate contribution becomes isotropic. This is an assumption about the diversity of training domains.
  • domain assumption Last token integrates context (Sec. 2.2)
    The last token of an ICL prompt is treated as the integration point for contextual information; the method's extraction relies on this.
  • domain assumption Router generalizes from MiniLM embedding to routing directions (Sec. 3.2)
    The trained router is assumed to transfer to queries from unseen tasks based on their frozen sentence embeddings.
  • standard math Davis-Kahan sin theta theorem
    Used in Appendix A.3 to bound subspace recovery; standard result but applied to assumptions not proven.
invented entities (1)
  • Principal ICL Directions (PIDs)
    purpose: Low-rank subspaces of attention logits used to route attention.
    PIDs are PCA components of collected Q/K projections; their meaningfulness is supported by ablations (random basis fails) but they are constructs of the method, not independently observable entities.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Train Once, Reuse Everywhere: Generalizable Implicit In-Context Learning by Routing Attention." pith.science (2026). https://pith.science/paper/4SFF2AUY

@misc{pith2026250922854,
  author       = {Pith},
  title        = {Pith review of: Train Once, Reuse Everywhere: Generalizable Implicit In-Context Learning by Routing Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4SFF2AUY}},
  note         = {Machine review of arXiv:2509.22854}
}
read the original abstract

Implicit in-context learning (ICL) has newly emerged as a promising paradigm that simulates ICL behaviors in the representation space of large language models (LLMs), aiming to attain few-shot performance at zero-shot cost. However, existing approaches largely rely on injecting shift vectors into residual flows, which are typically constructed from labeled demonstrations or task-specific alignment. Such designs fall short of utilizing the structural mechanisms underlying ICL and suffer from limited generalizability. To address this, we propose In-Context Routing (ICR), a novel implicit ICL method that captures and utilizes generalizable ICL patterns at the attention logits level. It extracts reusable structural directions that emerge during ICL and employs a learnable input-conditioned router to modulate attention logits accordingly, enabling an efficient train-once-and-reuse framework. We evaluate ICR on 12 real-world datasets spanning diverse domains and multiple LLMs. The results show that ICR consistently outperforms existing implicit ICL methods that require task-specific retrieval or training, while demonstrating robust generalization to out-of-domain tasks where they struggle. These findings position ICR to push the boundary of the practical value of ICL. The code is available at https://github.com/Lijiaqian1/In-Context-Routing.git.

Figures

Figures reproduced from arXiv: 2509.22854 by the authors.

Figure 1
Figure 1. Multi-task ICL on OOD targets. Multi-task few-shot prompting sometimes surpasses both [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of attention routing compared [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Pipeline of In-Context Routing (ICR). (a) We perform ICL across multiple domains to [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Left: Layer-importance visualization under ICR. Middle: Visualization of top-1 head in each layer, with rings for heads, spokes for layers (starting at layer 21), and green stars marking consensus heads (numbers denote head indices). Right: Correlation of per-dataset P…
Figure 5
Figure 5. Figure 5: Comparison of average per-sample inference time across five datasets for 5-shot, zero [PITH_FULL_IMAGE:figures/full_fig_p021_5.png]
Figure 6
Figure 6. Figure 6: Layer importance profiles. Curves show per-layer importance, computed from head gates [PITH_FULL_IMAGE:figures/full_fig_p024_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 22 linked inside Pith

  1. [2]

    Training dynamics of multi-head softmax attention for in-context learning: Emergence, convergence, and optimality.arXiv preprint arXiv:2402.19442,

    Siyu Chen, Heejune Sheen, Tianhao Wang, and Zhuoran Yang. Training dynamics of multi-head softmax attention for in-context learning: Emergence, convergence, and optimality.arXiv preprint arXiv:2402.19442,

    arXiv

  2. [5]

    More importantly, as the input length increases, the inference time of few-shot grows much faster than that of ICR

    The results show that ICR consistently requires less inference time than the 5-shot setting. More importantly, as the input length increases, the inference time of few-shot grows much faster than that of ICR. This demonstrates that ICR preserves the efficiency of implicit ICL, with the advantage becoming especially pronounced for longer contexts. G ADDITI...

    2021

  3. [7]

    cross-dataset

    add redundancy without benefit. G.3 ROUTINGLAYERS We investigate the effect of applying ICR at different depths within the model by evenly dividing it into early, middle, and late segments. Table 11 shows that intervening at the late layers yields the best overall performance. This outcome reflects a fundamental difference between ICR and prior vector-bas...

    2022

  4. [8]

    Qi Guo, Leiyu Wang, Yidong Wang, Wei Ye, and Shikun Zhang

    URL https://arxiv.org/abs/2407.21783. Qi Guo, Leiyu Wang, Yidong Wang, Wei Ye, and Shikun Zhang. What makes a good order of examples in in-context learning. InFindings of the Association for Computational Linguistics: ACL 2024, pp. 14892–14904,

    arXiv 2024

  5. [9]

    In-context learning creates task vectors.arXiv preprint arXiv:2310.15916,

    Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors.arXiv preprint arXiv:2310.15916,

    arXiv

  6. [14]

    Jack Merullo, Carsten Eickhoff, and Ellie Pavlick

    URL https://openreview.net/forum?id=1v3XEcRMyP. Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Language models implement simple word2vec- style vector arithmetic.arXiv preprint arXiv:2305.16130,

    arXiv

  7. [15]

    Metaicl: Learning to learn in context.arXiv preprint arXiv:2110.15943,

    Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. Metaicl: Learning to learn in context.arXiv preprint arXiv:2110.15943,

    arXiv

  8. [16]

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al

    11 Preprint. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads.arXiv preprint arXiv:2209.11895,

    arXiv

Show all 31 references
  1. [18]

    doi: 10.3115/1219840.1219855

    Association for Computational Linguistics. doi: 10.3115/1219840.1219855. URLhttps://aclanthology.org/P05-1015/. Yingzhe Peng, Xinting Hu, Jiawei Peng, Xin Geng, Xu Yang, et al. Live: Learnable in-context vector for visual question answering.Advances in Neural Information Proce...

  2. [20]

    Johannes V on Oswald, Eyvind Niklasson, Ettore Randazzo, Jo ˜ao Sacramento, Alexander Mordv- intsev, Andrey Zhmoginov, and Max Vladymyrov

    URLhttps://arxiv.org/abs/2307.09288. Johannes V on Oswald, Eyvind Niklasson, Ettore Randazzo, Jo ˜ao Sacramento, Alexander Mordv- intsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. InInternational Conference on Machine Learning, pp...

    arXiv

  3. [21]

    Elicit: Llm augmentation via external in- context capability.arXiv preprint arXiv:2410.09343, 2024a

    Futing Wang, Jianhao Yan, Yue Zhang, and Tao Lin. Elicit: Llm augmentation via external in- context capability.arXiv preprint arXiv:2410.09343, 2024a. Qixun Wang, Yifei Wang, Yisen Wang, and Xianghua Ying. Can in-context learning really general- ize to out-of-distribution task...

    arXiv

  4. [22]

    Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering.arXiv preprint arXiv:2212.10375,

    Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Lingpeng Kong. Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering.arXiv preprint arXiv:2212.10375,

    arXiv

  5. [23]

    Addressing order sensitivity of in-context demonstration examples in causal language models.arXiv preprint arXiv:2402.15637,

    Yanzheng Xiang, Hanqi Yan, Lin Gui, and Yulan He. Addressing order sensitivity of in-context demonstration examples in causal language models.arXiv preprint arXiv:2402.15637,

    arXiv

  6. [24]

    An explanation of in-context learning as implicit bayesian inference.arXiv preprint arXiv:2111.02080,

    Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference.arXiv preprint arXiv:2111.02080,

    arXiv

  7. [25]

    Pretraining data mixtures enable narrow model selection capabilities in transformer models.arXiv preprint arXiv:2311.00871,

    Steve Yadlowsky, Lyric Doshi, and Nilesh Tripuraneni. Pretraining data mixtures enable narrow model selection capabilities in transformer models.arXiv preprint arXiv:2311.00871,

    arXiv

  8. [26]

    Kayo Yin and Jacob Steinhardt

    URL https://arxiv.org/abs/2412.15115. Kayo Yin and Jacob Steinhardt. Which attention heads matter for in-context learning?arXiv preprint arXiv:2502.14010,

    arXiv

  9. [27]

    is a widely studied framework in random matrix theory and high-dimensional statistics. It assumes that the population covariance matrixΣ∈R d×d can be decomposed into an isotropic noise component plus a small number of low-rank “spikes”: Σ = rX i=1 θiuiu⊤ i +σ 2Id,(17) whereσ 2...

    1970

  10. [28]

    An identical argument applies toU k

    gives the bound: sin Θ span( bUq),span(U q) ≤ ∥∆Q∥op gapQ (19) Thus, the subspace stability depends on the relative size of the perturbation versus the eigengap. An identical argument applies toU k. Importantly, pooling across multiple domains helps enlargegap Q by amplifying ...

    2023

  11. [29]

    For training, we use the same number of few-shot examples as those contained in an ICL prompt during the construction of ICL bases, drawn from five in-domain datasets

    The LoRA module is applied to the token clas- sification head of the last layer with rank32. For training, we use the same number of few-shot examples as those contained in an ICL prompt during the construction of ICL bases, drawn from five in-domain datasets. Although LoRA re...

    2024

  12. [1970]

    URLhttps://doi.org/10

    doi: 10.1137/0707001. URLhttps://doi.org/10. 1137/0707001. Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: In- vestigating projection in naturally occurring discourse. Inproceedings of Sinn und Bedeutung, volume 23, pp. 107–124,

    doi

  13. [2001]

    Is attention required for icl? exploring the relationship between model architecture and in-context learning ability.arXiv preprint arXiv:2310.08049,

    Ivan Lee, Nan Jiang, and Taylor Berg-Kirkpatrick. Is attention required for icl? exploring the relationship between model architecture and in-context learning ability.arXiv preprint arXiv:2310.08049,

    arXiv

  14. [2002]

    M2iv: Towards efficient and fine-grained multimodal in-context learning via representation engineering

    Yanshu Li, Yi Cao, Hongyang He, Qisen Cheng, Xiang Fu, Xi Xiao, Tianyang Wang, and Ruixiang Tang. M2iv: Towards efficient and fine-grained multimodal in-context learning via representation engineering. InSecond Conference on Language Modeling, 2025a. Yanshu Li, JianJiang Yang,...

  15. [2005]

    A survey on in-context learning.arXiv preprint arXiv:2301.00234,

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey on in-context learning.arXiv preprint arXiv:2301.00234,

    arXiv

  16. [2018]

    Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers.arXiv preprint arXiv:2212.10559,

    Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers.arXiv preprint arXiv:2212.10559,

    arXiv

  17. [2019]

    Function vectors in large language models.arXiv preprint arXiv:2310.15213,

    Eric Todd, Millicent L Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models.arXiv preprint arXiv:2310.15213,

    arXiv

  18. [2020]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    1901

  19. [2021]

    URLhttps://arxiv.org/abs/2101. 06804. Sheng Liu, Haotian Ye, Lei Xing, and James Zou. In-context vectors: Making in context learning more effective and controllable through latent space steering.arXiv preprint arXiv:2311.06668,

    arXiv

  20. [2022]

    Creak: A dataset for common- sense reasoning over entity knowledge.arXiv preprint arXiv:2109.01653,

    Yasumasa Onoe, Michael JQ Zhang, Eunsol Choi, and Greg Durrett. Creak: A dataset for common- sense reasoning over entity knowledge.arXiv preprint arXiv:2109.01653,

    arXiv

  21. [2023]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. https://arxiv.org/abs/2106.09685,

    arXiv

  22. [2024]

    On the relation between sensitivity and accuracy in in-context learning.arXiv preprint arXiv:2209.07661,

    Yanda Chen, Chen Zhao, Zhou Yu, Kathleen McKeown, and He He. On the relation between sensitivity and accuracy in in-context learning.arXiv preprint arXiv:2209.07661,

    arXiv

  23. [2025]

    URLhttps://arxiv.org/abs/2410. 04468. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457,

    arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.