Pith. sign in

REVIEW 2 major objections 4 minor 1 cited by

The paper claims that multimodal in-context learning emerges because a primary modality installs a reusable induction circuit, so the secondary modality needs only low data diversity to be mapped onto it.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 07:11 UTC pith:W2BSJLQ2

load-bearing objection Careful empirical paper with a nice testbed and a solid knockout ablation; the headline asymmetric-diversity claim needs a late-fusion K1 sweep it doesn't have, plus some appendix cleanup. the 2 major comments →

arxiv 2601.20796 v2 pith:W2BSJLQ2 submitted 2026-01-28 cs.CL cs.LG

Dissecting Multimodal In-Context Learning: Modality Asymmetries and Circuit Dynamics in modern Transformers

Yiran Huang , Karsten Roth , Quentin Bouniot , Wenjia Xu , Zeynep Akata This is my paper
classification cs.CL cs.LG
keywords in-context learningmultimodal transformersinduction circuitsdata diversitymodality asymmetryRoPEpositional encodingssynthetic testbed
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that the ability to learn from in-context examples across two modalities does not require both modalities to have complex, high-diversity training data. Instead, pretraining a transformer decoder on a single high-diversity 'primary' modality installs a reusable induction circuit—an attention pattern that copies labels from matching context examples—and the second modality only needs enough structure to be mapped onto that existing circuit. The authors show that with 8,192 classes in the primary modality, as few as 256 classes in the secondary modality suffice for strong multimodal in-context learning. They also find that Rotary Position Embeddings (RoPE) raises the data-complexity threshold for ICL in both unimodal and multimodal settings, and that scaling decoder size helps multimodal ICL, opposite to the unimodal case. Mechanistically, both settings rely on previous-token and induction heads, and multimodal training primarily refines the label-matching induction head.

Core claim

The central claim is that multimodal ICL rests on an asymmetry: pretraining a decoder on a high-diversity primary modality (with 8,192 classes) installs a reusable induction circuit, so the secondary modality needs only modest diversity (256 classes suffice) to achieve comparable ICL once mapped into the decoder's embedding space. Scaling the decoder then favors ICL, because added capacity is used to map the secondary modality onto the existing circuit rather than to memorize. Mechanistically, both settings rely on the same induction circuit—an early previous-token head and a later induction head that copies labels from matching examples—and multimodal training primarily refines the label-ma

What carries the argument

The key object is the induction circuit: a two-step attention pattern in which an early-layer previous-token head copies the preceding item's identity and a later-layer induction head retrieves the label from a matching in-context exemplar. The paper quantifies these with progress measurements (previous-token head strength, induction head strength, target label association, and context-label accuracy) and validates them causally by knocking out identified heads. The argument is carried by a two-stage late-fusion curriculum: pretrain the decoder on the primary modality M1, then add an MLP projector (and optionally a pretrained encoder) to map M2 features into M1's embedding space.

Load-bearing premise

The asymmetry is demonstrated only under a specific two-stage late-fusion curriculum with the token order x_i, x'_i, l_i and a label set for M2 that is a subset of M1's; if that curriculum is not the right model of how real multimodal ICL arises, the primary/secondary claim may not generalize.

What would settle it

A decisive experiment would be a late-fusion sweep with K1 varied across a wide range (for example 256 versus 8,192) while keeping K2 fixed: if raising K1 does not lower the K2 requirement for strong ICL, the claim that the primary modality installs the reusable circuit is falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the asymmetry holds, multimodal ICL can be elicited with far less secondary-modality data diversity than previously thought, so image or video data may not need to match text diversity for few-shot generalization.
  • Scaling decoder capacity improves multimodal ICL, in contrast to unimodal ICL where scaling favors memorization at fixed data complexity.
  • RoPE and similar relative positional encodings raise the data-complexity threshold for ICL; absolute or hybrid encodings form induction circuits more readily, though sufficiently high data complexity can compensate.
  • A pretrained encoder for the secondary modality is valuable mainly for cross-modal alignment; encoder quality (validation accuracy) predicts downstream ICL performance.
  • The induction circuit (previous-token head plus induction head) is causally necessary for ICL; zeroing identified heads destroys performance, and zeroing either modality confirms genuine cross-modal integration.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The primary/secondary asymmetry appears to be a product of the training curriculum and sequence geometry rather than an intrinsic property of modalities: the paper's own early-fusion experiments reverse which modality anchors the circuit, showing that the first modality to learn the label-adjacent pattern becomes 'primary.'
  • A direct test of the claimed causal role of K1 would be a late-fusion sweep that varies K1 while holding K2 fixed; the paper does not report this control, so the asymmetry is demonstrated for one pretraining diversity level only.
  • If this reuse principle generalizes, a practical recipe for adding a new modality to a language model would be to invest in a strong encoder and alignment module rather than collecting extremely diverse multimodal corpora, because the decoder's existing ICL circuit can be leveraged.
  • Real MLLM observations in the paper (Qwen2.5-VL and IDEFICS) are qualitative and correlational; a stronger test would intervene on specific heads in those models to confirm the same circuit dependency, which the controlled setting shows.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper studies in-context learning (ICL) in small two-layer transformer decoders with modern components (RMSNorm, SiLU, RoPE) trained on synthetic Gaussian-mixture classification data. It first replicates established unimodal data-statistics findings, then reports that RoPE raises the data-complexity threshold for ICL and that model scaling favors in-weight memorization. The central multimodal contribution is a two-stage late-fusion curriculum: pretrain the decoder on a high-diversity primary modality M1 (K1=8192), then introduce a secondary modality M2 through a projector and joint training. The authors report an asymmetry whereby M2 requires comparatively little class diversity (K2=256) for multimodal ICL to emerge, attribute this to M1 installing a reusable induction circuit, and provide mechanistic evidence from previous-token/induction-head metrics, causal head knockouts, modality-zeroing ablations, and random-forest prediction. The findings are further validated on Omniglot and, qualitatively, on Qwen2.5-VL and IDEFICS.

Significance. The paper's controlled testbed, code release, careful ICL/IWL separation, and explicit causal ablations (Table 3) are substantial strengths. If the asymmetry claim survives the missing control, the paper provides a useful mechanistic account of why multimodal ICL can emerge with relatively low secondary-modality diversity and identifies induction-head refinement as a measurable circuit-level correlate. The RoPE finding is also a concrete, falsifiable claim that goes beyond prior simplified architectures. However, the significance is bounded by the curriculum-specific nature of the central claim: the paper itself shows that early-fusion joint training reverses the asymmetry, so the 'fundamental' asymmetry language in the abstract and Sec. 4.1 needs qualification unless the missing late-fusion K1 control is supplied.

major comments (2)
  1. [Sec. 4.1 / Figure 4a] The central asymmetry claim is not directly tested because K1 is never varied in the late-fusion protocol. Figure 4a fixes K1=8192 and sweeps K2 and B; the only K1 manipulation is early-fusion joint training in App. A.3.7, which simultaneously removes the pretraining stage and changes the position-to-label geometry. Thus the causal reading that 'high-diversity M1 installs the reusable ICL circuit' is confounded by two-stage training per se, by the decoder's already-known label embeddings (L2 is a 16-label subset of the 32 pretrained labels), and by sequence geometry. Please add a late-fusion K1 sweep (e.g., K1=256, 1024, 8192) with the same late-fusion schedule and x_i, x'_i, l_i sequence order, and report the K2 required to reach the ICL threshold. This is the control that would directly support the paper's headline claim. The paper's own acknowledgement (Sec. 4.1, 'Structural origins',
  2. [App. A.3.7 / Table 9] The early-fusion reversal evidence, used to support the structural-origin claim, is not convincingly quantified. Table 9 reports no error bars and contains striking non-monotonicities: at B=2, ICL for (K1,K2)=(2048,8192) is 0.62 while (4096,8192), (8192,8192), and (16384,8192) are all about 0.35-0.37; at B=4, (8192,8192) is 0.88/0.95 but (16384,8192) drops to 0.58. These entries make the purported reversal difficult to read and suggest high seed variance or an anomaly. Please report per-seed results, add standard deviations, and verify that the reversal is robust across the K1 grid before using it to argue that the pretraining phase is the cause of the main asymmetry.
minor comments (4)
  1. [Sec. 2, Eq. (1)] The formula for generating instances is garbled in the rendered text: 'xi = µk + εm ηp 1+ε 2m' is not readable. Please rewrite with explicit normalization, e.g., x_i = μ_k + ε_m η / sqrt(1+ε_m^2).
  2. [Table 4b (Appendix A.3.5)] There appears to be a duplication/inconsistency: PHStrength(1)_1 is listed twice, once with ρ=0.58 and once with ρ=-0.02. The main text (Sec. 4.4.2) says PHStrength(2)_1 shows negligible correlation, but Appendix Table 4b lists PHStrength(2)_1=0.48. Please check which metric is intended and align the text with the table.
  3. [Fig. 5] The y-axis label 'Data requirement' is defined only in the caption as K2·√B. Please specify in the main text the exact threshold (e.g., ICL accuracy >0.95), the sweep protocol, and whether error bars or seed ranges are included; the current figure reports no variability.
  4. [Abstract / Sec. 4.1] The phrase 'fundamental learning asymmetry' is stronger than what the evidence supports. The paper's own Sec. 4.1 and App. A.3.7 show the asymmetry is induced by the pretraining curriculum and sequence geometry. Please recalibrate the language, e.g., to 'curriculum-induced asymmetry' or 'asymmetry under two-stage late-fusion training.'

Circularity Check

0 steps flagged

No circularity found; the claimed derivations are empirical and the mechanistic claims are independently ablated.

full rationale

I walked the paper's main derivation chain—unimodal ICL premises (Sec. 3), the multimodal asymmetry claim (Sec. 4.1), scaling effects (Sec. 4.2), encoder alignment (Sec. 4.3), and circuit quantification (Sec. 4.4)—and found no step that reduces by construction to its own inputs. The central asymmetry claim ('after pretraining the decoder on a high-diversity M1 (K1=8192), M2 requires surprisingly little class diversity to achieve comparable ICL; a relatively small K2=256 is sufficient') is an empirical comparison from the K2 sweep in Figure 4a with K1 fixed; it is not derived from a fitted parameter or from a definition of the quantities involved. The causal attribution to high K1 is not fully controlled, since the late-fusion protocol never varies K1 and the early-fusion control changes more than K1, but that is a missing-control/validity concern, not circularity. In Sec. 4.4, the progress measurements (PHStrength, IndStrength, TLA, CLA) are defined independently from ICL accuracy and are validated causally by head knockout (Table 3), so the correlation and random-forest analyses do not smuggle the conclusion into the inputs; the regressor also uses a training/validation split. I found no load-bearing self-citation: prior work by Reddy, Chan et al., and Olsson et al. is external support, and the paper contains no uniqueness theorem or imported ansatz that forces its conclusions. Overall, the paper is self-contained against its own synthetic benchmarks and the mechanism story has independent causal support; score 0.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The paper makes empirical claims rather than deriving a theory. Most experimental knobs (K, B, α, ε) are controlled variables, not fitted parameters. The main load-bearing choices are the training curriculum, the label-set hierarchy, and the assumption that attention metrics capture the relevant mechanism.

free parameters (2)
  • Data complexity proxy K·√B
    Used as the x-axis and threshold metric for scaling and PE comparisons (Sec. 3.2, Fig. 2b). The square-root weighting is chosen by hand and could change the reported thresholds.
  • Default synthetic data settings (ε1=ε2=0.1, K1=8192, K2=256, L1=32, L2=16, D1=64, D2=32, B=4)
    Fixed by hand for the multimodal analysis (Sec. 2 and Sec. 4). The central asymmetry is demonstrated only within these settings.
axioms (4)
  • domain assumption L2 ⊂ L1 (secondary modality's labels are a subset of primary modality's vocabulary)
    Sec. 2: 'We set the label space L2 ⊂ L1 to reflect common practice...' — the asymmetry result is defined relative to this label hierarchy; it is not derived.
  • domain assumption Two-stage late-fusion training with pretrained decoder is representative of multimodal ICL
    Sec. 4.1 and Figure 3. The central asymmetry disappears in early-fusion joint training (App. A.3.7), so the choice of curriculum is load-bearing.
  • domain assumption Novel-class and swapped-label evaluation isolates ICL from IWL
    Inherited from Reddy (2024) and Chan et al. (2022); used throughout to define ICL. If context-label permutation does not fully remove memorization, ICL numbers are inflated.
  • domain assumption Attention-weight progress metrics (PHStrength, IndStrength, TLA, CLA) measure the mechanism
    Sec. 4.4.1 defines these metrics. They are correlational quantities; causality is shown only for two selected heads via knockout ablation.

pith-pipeline@v1.3.0-alltime-deepseek · 22649 in / 11228 out tokens · 120208 ms · 2026-08-03T07:11:00.802145+00:00 · methodology

0 comments
read the original abstract

Transformer-based multimodal large language models often exhibit in-context learning (ICL) abilities. Motivated by this phenomenon, we ask: how do transformers learn to associate information across modalities from in-context examples? We investigate this question through controlled experiments on small transformers trained on synthetic classification tasks, enabling precise manipulation of data statistics and model architecture. We begin by revisiting core principles of unimodal ICL in modern transformers. While several prior findings replicate, we find that Rotary Position Embeddings (RoPE) increases the data complexity threshold for ICL. Extending to the multimodal setting reveals a fundamental learning asymmetry: when pretrained on high-diversity data from a primary modality, surprisingly low data complexity in the secondary modality suffices for multimodal ICL to emerge. Mechanistic analysis shows that both settings rely on an induction-style mechanism that copies labels from matching in-context exemplars; multimodal training refines and extends these circuits across modalities. Our findings provide a mechanistic foundation for understanding multimodal ICL in modern transformers and introduce a controlled testbed for future investigation. Code is available at: https://github.com/YiranHuangIrene/multimodal-icl

Figures

Figures reproduced from arXiv: 2601.20796 by Karsten Roth, Quentin Bouniot, Wenjia Xu, Yiran Huang, Zeynep Akata.

Figure 1
Figure 1. Figure 1: Preliminaries in the multimodal setting. (a) The context consists of N triplets followed by the target query. The paired examples (xi, x′ i) from two modalities, with a shared label li, are generated from Gaussian Mixture Models (GMMs) by controlling within-class variation ε1 and ε2. (b) The distributional properties for the synthetic data. The burstiness B determines how often the class occurs in the cont… view at source ↗
Figure 2
Figure 2. Figure 2: (a) Impact of increasing model layers and attention heads on ICL-IWL tradeoff. Scaling up favours IWL over ICL for five seeds. (b) The data complexity (measured by K · √ B) required for models of different sizes to achieve the same ICL accuracy (> 0.95). A larger model needs more complex data for a strong ICL capability. (c) Across data regimes, RoPE yields lower ICL accuracy than absolute positional encod… view at source ↗
Figure 3
Figure 3. Figure 3: Multimodal setup. (a) Projector-only setup: an MLP projector aligns M2 features to the M1 embedding space. (b) Encoder-augmented setup: a pretrained M2 encoder is stacked before the projector and decoder. (c) Encoder pretraining: the M2 encoder is pretrained on M2-specific classes/labels. 64 128 256 512 Number of classes (K2) 1 2 4 B u r s tin e s s ( B ) (a) 64 128 256 512 Number of classes (K2) 1 2 4 B u… view at source ↗
Figure 4
Figure 4. Figure 4: (a) Fixing K1 = 8192, the impact of K2 and B on ICL performance. When the decoder is pretrained on M1, a signifi￾cantly lower data complexity for M2 is needed to achieve good ICL. (b) Consistently, IWL decreases with K2 and B. (c) Raising ε2 benefits ICL markedly more than raising ε1. (d) Fixing α1=1, the ICL–IWL balance is best when α2 ≈1 for five seeds. bedding space. We then jointly train it with the pr… view at source ↗
Figure 5
Figure 5. Figure 5: Data requirements (measured by K2 · √ B) needed for larger multimodal models to elicit strong ICL. Deeper or wider decoders achieve the same accuracy with lower data requirements. core ICL capability, while M2’s primary role is simply to provide a distinguishable signal that the projector can map onto the decoder’s pre-existing feature space. Further ex￾periments support this view. While within-class varia… view at source ↗
Figure 7
Figure 7. Figure 7: Pretrained encoders enhance multimodal ICL. (a) D1 = 64, increasing M2 feature dimension (D2) lowers ICL when using only a projector. (b) Fixing D2 = 512, adding a pretrained encoder outperforms a parameter-matched larger projector. 0.00 0.15 0.30 0.45 0.60 0.75 Encoder validation accuracy 0.4 0.5 0.6 0.7 0.8 0.9 1.0 ICL accuracy Train projector Train encoder + projector Train projector + decoder Train all… view at source ↗
Figure 8
Figure 8. Figure 8: Encoder quality predicts downstream ICL. Multi￾modal ICL on Omniglot increases with the encoder’s validation accuracy; gains saturate only when the decoder is frozen, while joint training of the decoder continues to yield improvements. three training strategies: training (1) only the projector, (2) the projector & pretrained decoder, and (3) all the compo￾nents. Results in Figure 7b indicate that the encod… view at source ↗
Figure 9
Figure 9. Figure 9: Demonstration of the head dynamics in the multimodal setting for five seeds. While CLA and previous token heads remain stable from the unimodal stage, the refinement of the induction head is the primary driver of accuracy gains [PITH_FULL_IMAGE:figures/full_fig_p007_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Overview of the preliminaries in the unimodal setting. (a) The context consists of N item-label pairs (xi, li) followed by the target query. K classes are assigned to L labels. (b) The distributional properties for the synthetic data. Class instances are obtained by controlling within-class variation ε. Class frequencies follow a Zipfian distribution with exponent α, and burstiness B determines repeated c… view at source ↗
Figure 11
Figure 11. Figure 11: The data distributional findings transfer from the GMM setting. (a) Number of classes and burstiness. Larger number of classes K and burstiness B promotes ICL while decreasing IWL. (b) Within-class variation.Increasing ε promotes ICL. However, it also slows down the emergence of ICL. (c) Class-distribution skew. Increasing α improves ICL. When α = 1, a balance is achieved between ICL and IWL. A.2.2. IMPAC… view at source ↗
Figure 12
Figure 12. Figure 12: Comparison of the impact of different positional encodings across data complexity and context length on ICL accuracy and the strength of previous token head and induction head. Three key patterns are visible: (1) ICL accuracy degrades for all PE types as context length (N) increases. (2) ALiBi and RoPE cluster together, while APE shows the highest accuracy, with the Hybrid PE performing in between. (3) Th… view at source ↗
Figure 13
Figure 13. Figure 13: In the multimodal setting, RoPE yields lower ICL accuracy than APE across data regimes. Consistent with unimodal setting, RoPE increases the data complexity threshold for strong ICL. A.3.2. IMPACT OF FEATURE DIMENSIONALITY ON CROSS-MODAL ALIGNMENT To investigate the relationship between feature dimensionality and ICL performance, we measured the alignment between the projected M2 features and M1 class pro… view at source ↗
Figure 14
Figure 14. Figure 14: The data distributional findings transfer from the GMM setting to the Omniglot dataset. (a) Larger number of classes K2 and burstiness B promotes ICL. The symmetry also transfers. When K1 = 8192, a smaller K2 = 256 is enough for the model to learn a very good ICL. (b) Within-class variation.Increasing ε1 εand ε2 promotes ICL. (c) Class-distribution skew. Increasing α improves ICL. When α1 = α2 = 1, a bala… view at source ↗
Figure 15
Figure 15. Figure 15: The correlation between statistical properties and regularization during encoder pretraining with the downstream ICL does not show a clear pattern across different training regimes. A.3.5. PROGRESS MEASUREMENT ANALYSIS Pearson Correlation. As shown in the Table 4b, PHStrength(1) 1 shows weaker correlation with ICL compared with IndStrength2 . During the unimodal pretraining stage, the model already learns… view at source ↗
Figure 16
Figure 16. Figure 16: The encoder size shows no clear correlation with multimodal ICL accuracy. induction head the most sensitive indicator of progress. Similarly, CLA exhibits a very low correlation with ICL accuracy because the model has learned in the first stage to predict the label from the context. In the multimodal training, CLA maintains at a very high value as shown in the [PITH_FULL_IMAGE:figures/full_fig_p017_16.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention

    cs.LG 2026-07 conditional novelty 7.0

    Positional schemes set the default spectral algebra of attention heads: previous-token heads are rotational under RoPE and content-like under absolute/ALiBi, as a post-function fingerprint rather than a hard constraint.

Reference graph

Works this paper leans on

19 extracted references · 9 linked inside Pith · cited by 1 Pith paper

  1. [1]

    J., Javaheripi, M., Kauffmann, P., et al

    Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gunasekar, S., Harrison, M., Hewett, R. J., Javaheripi, M., Kauffmann, P., et al. Phi-4 technical report.arXiv preprint arXiv:2412.08905,

    Pith/arXiv arXiv

  2. [2]

    using subsets of the progress measurements in the multimodal setting. Feature subsetR 2 (mean±std) PHStrength(1) 1 ,IndStrength 2 0.90±0.01 PHStrength(1) 1 ,IndStrength 2,TLA 1 0.96±0.06 PHStrength(1) 1 ,IndStrength 2,TLA 2 0.92±0.01 All metrics0.98±0.01 A.3.6. CROSS-MODALINTERACTIONANALYSIS VIAMODALITYZEROING To directly test whether the model performs g...

    2048

  3. [5]

    Sigmoid-weighted linear units for neural network function approximation in reinforcement learning.arXiv preprint arXiv:1702.03118,

    Elfwing, S., Uchibe, E., and Doya, K. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning.arXiv preprint arXiv:1702.03118,

    Pith/arXiv arXiv

  4. [6]

    Rethinking the role of demonstrations: What makes in-context learning work? InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing

    Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. Rethinking the role of demonstrations: What makes in-context learning work? InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics,

    2022

  5. [8]

    Shin, S., Lee, S

    URL https://openreview.net/ forum?id=aN4Jf6Cx69. Shin, S., Lee, S. W., Ahn, H., Kim, S., Kim, H. S., Kim, B., Cho, K., Lee, G., Park, W., Ha, J. W., et al. On the effect of pretraining corpora on in-context learning by a large-scale language model. In2022 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human La...

    2022

  6. [9]

    Roformer: Enhanced transformer with rotary position embedding

    Su, J., Lu, Y ., Pan, S., Wen, B., and Zhai, Y . Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864,

    Pith/arXiv arXiv

  7. [10]

    URL https:// qwenlm.github.io/blog/qwen2.5-vl/. 10 Dissecting Multimodal In-Context Learning: Modality Asymmetries and Circuit Dynamics in Transformers Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models.arXiv preprint...

    Pith/arXiv arXiv

  8. [12]

    Larger language models do in-context learning differently.arXiv preprint arXiv:2303.03846,

    Wei, J., Wei, J., Tay, Y ., Tran, D., Webson, A., Lu, Y ., Chen, X., Liu, H., Huang, D., Zhou, D., et al. Larger language models do in-context learning differently.arXiv preprint arXiv:2303.03846,

    Pith/arXiv arXiv

  9. [14]

    Elic- iting in-context learning in vision-language models for videos through curated data distributional properties

    Yu, K., Zhang, Z., Hu, F., Storks, S., and Chai, J. Elic- iting in-context learning in vision-language models for videos through curated data distributional properties. In Proceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing, pp. 20416–20431,

    2024

  10. [15]

    K., and Chan, S

    Zucchet, N., d’Angelo, F., Lampinen, A. K., and Chan, S. C. The emergence of sparse attention: impact of data distribution and benefits of repetition.arXiv preprint arXiv:2505.17863,

    arXiv

  11. [16]

    Appendix This appendix presents comprehensive methodological details, extended mechanistic analyses, and external validation of the findings reported in the main text

    11 Dissecting Multimodal In-Context Learning: Modality Asymmetries and Circuit Dynamics in Transformers A. Appendix This appendix presents comprehensive methodological details, extended mechanistic analyses, and external validation of the findings reported in the main text. Section A.1 formalizes the experimental framework, detailing the synthetic data ge...

    2024

  12. [19]

    target query

    on VL-ICL benchmark, a suite designed to directly measure the ICL ability of vision–language models Zong et al. (2024). We then compute the strength of the previous token head and induction head for Qwen2.5-VL (3B, 7B) to investigate whether the controlled findings on the correlation between ICL accuracy and the strength of the induction circuits hold in ...

    2024

  13. [64]

    DATA DISTRIBUTIONAL FINDINGS TRANSFER TO REAL IMAGE DATA

    D2 CKA w/o Enc CKA w/ Enc L2 w/o Enc L2 w/ Enc 32 0.16 0.17 0.95 0.91 128 0.12 0.14 1.47 1.37 256 0.09 0.11 1.82 1.43 512 0.07 0.11 2.15 1.45 A.3.3. DATA DISTRIBUTIONAL FINDINGS TRANSFER TO REAL IMAGE DATA. We extend our experiments on Omniglot dataset, which has 1.6k handwritten characters. We pre-train the decoder on GMM data and construct the multimoda...

    arXiv

  14. [2014]

    A., and Lewis, M

    Press, O., Smith, N. A., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapolation.arXiv preprint arXiv:2108.12409,

    Pith/arXiv arXiv

  15. [2021]

    Pretraining data mixtures enable narrow model selection capabilities in transformer models.arXiv preprint arXiv:2311.00871,

    Yadlowsky, S., Doshi, L., and Tripuraneni, N. Pretraining data mixtures enable narrow model selection capabilities in transformer models.arXiv preprint arXiv:2311.00871,

    Pith/arXiv arXiv

  16. [2022]

    Can multimodal large language models truly perform multimodal in-context learning? In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp

    Chen, S., Han, Z., He, B., Liu, J., Buckley, M., Qin, Y ., Torr, P., Tresp, V ., and Gu, J. Can multimodal large language models truly perform multimodal in-context learning? In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 6000–6010. IEEE, 2025a. Chen, S., Liu, J., Han, Z., Xia, Y ., Cremers, D., Torr, P., Tresp, V ., and...

    Pith/arXiv arXiv 2025

  17. [2023]

    Label words are anchors: An in- formation flow perspective for understanding in-context learning

    Wang, L., Li, L., Dai, D., Chen, D., Zhou, H., Meng, F., Zhou, J., and Sun, X. Label words are anchors: An in- formation flow perspective for understanding in-context learning. InProceedings of the 2023 Conference on Em- pirical Methods in Natural Language Processing,

    2023

  18. [2024]

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J

    URL https://arxiv.org/abs/ 2404.15736. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901,

    Pith/arXiv arXiv 1901

  19. [2025]

    A survey on in- context learning

    9 Dissecting Multimodal In-Context Learning: Modality Asymmetries and Circuit Dynamics in Transformers Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Chang, B., et al. A survey on in- context learning. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1107–1128,

    2024