Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

True Multimodal In-Context Learning Needs Attention to the Visual Context

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Multimodal in-context learning is mostly text imitation because models ignore demo images; a 160-parameter attention rebalancing makes them use the images.

desk verdict Useful diagnostic benchmark and a simple attention-rebalancing trick that works, but the mechanistic claim about reading visual context is overstated. read the letter →

arxiv 2507.15807 v2 pith:LBV7BVV5 submitted 2025-07-21 cs.CV cs.AI

Shuo Chen , Jianzhe Liu , Zhen Han , Yan Xia , Daniel Cremers , Philip Torr , Volker Tresp , Jindong Gu This is my paper
classification cs.CVcs.AI
keywords multimodalin-contextlearningvisualcontextneglectattentionreallocationDARATrueMICLparameter-efficientfine-tuningtaskvsrecognitionMLLMevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Current multimodal large language models mostly ignore the demonstration images when doing few-shot in-context learning, the paper argues, and instead copy textual answer patterns; benchmark gains on standard vision-language tasks often hide this because those tasks can be solved from text alone. To counter the neglect, the paper introduces DARA, a fine-tuning method that multiplies the pre-softmax attention-score columns belonging to image tokens by learned per-image, per-head scalars in the first decoder layer, so $S' = S\mathbf{F}$. With about 160 learnable parameters and a 30-example support set, DARA raises average TrueMICL accuracy from 77.67 to 83.00 on Qwen2-VL, from 41.95 to 48.67 on Idefics3, and from 44.62 to 52.81 on Phi-3.5-Vision relative to random four-shot inference. The paper also contributes TrueMICL, a benchmark of seven tasks and 867 samples built so that every correct answer requires understanding the demonstration images, which separates genuine task learning from mere text imitation. If the claims hold, a tiny and targeted rebalancing of attention, rather than heavy adaptation, is what decides whether multimodal in-context learning is actually multimodal.

What carries the argument

The load-bearing object is the modified attention-score matrix $S' = S\mathbf{F}$, where $S = QK^{\mathsf{T}}$ is the pre-softmax score matrix in one decoder layer and $\mathbf{F} = \operatorname{diag}(\mathbf{f})$ is a diagonal matrix of learnable scalars, nonzero only on columns belonging to visual tokens. Multiplying a column by $\mathbf{f} > 1$ raises that token's influence before softmax; DARA learns one scalar per image per attention head, trained by ordinary cross-entropy while all other parameters stay frozen. The appendix shows any such reweighting can be achieved by low-rank updates to the query and key projections, and the paper frames DARA as a constrained LoRA variant for MICL, placed in the first transformer layer across all heads because early cross-modal fusion starts there.

What would settle it

A decisive test would corrupt the demonstration images, for example replacing them with unrelated or blank images while keeping the demo text and query unchanged; if DARA's accuracy gain over its baseline persists, the method is not actually extracting information from the visual context.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that the dominant failure of multimodal in-context learning is not missing capacity but misplaced attention: current MLLMs allocate far more attention to text tokens than to demonstration images, so the visual context is effectively unused. The central corrective is DARA, which multiplies the columns of the pre-softmax attention-score matrix $S = QK^{\mathsf{T}}$ that correspond to image tokens by learned scalar factors $\mathbf{f}$, computes $S' = S\mathbf{F}$ with $\mathbf{F} = \operatorname{diag}(\mathbf{f})$, and then applies softmax as usual. Inserted only in the first transformer layer, with all other weights frozen, DARA learns factors that are consistently above 1 for demo images and lets different heads specialize on different demonstrations, lifting the reported image-token attention ratio from 28% to 46.7% in the Qwen2-VL example. The companion benchmark TrueMICL makes task success depend on the demo images by design; the paper reports consistent DARA gains across three open MLLMs, transfer of gains between tasks, further gains when combined with full LoRA, and no drop on standard vision-language benchmarks.

Load-bearing premise

The load-bearing premise is that MLLMs ignore demonstration images mainly because attention is allocated unevenly across image and text tokens, so multiplying pre-softmax attention-score columns by a few learned scalars is enough to release visual information that is already encoded; if the bottleneck were in the visual representation itself or in cross-modal alignment, DARA would move attention without moving accuracy.

Editorial extensions

If this is right

  • DARA trained on one TrueMICL task transfers to other tasks, improving unseen-task accuracy by 2 to 5 percentage points over the random four-shot baseline, so the learned rebalancing is not task-specific memorization.
  • Because DARA's update is a constrained LoRA update, it composes with LoRA; the paper reports a further 1 to 2 percentage point accuracy gain when DARA is applied on top of full LoRA.
  • Standard benchmarks such as VQAv2, GQA, A-OKVQA, and COCO show almost no difference between zero-shot, no-image, and random four-shot settings, so MICL improvements measured on them can be text imitation; TrueMICL is proposed as an evaluation where that escape route is closed.
  • Across shot counts from 2 to 32, DARA consistently improves accuracy over the matching baseline until the model's practical image limit, suggesting the benefit is not tied to one demonstration count.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the diagnosis is general, the same column-scaling operation should help any transformer whose in-context tokens of one type are being systematically under-attended, such as audio or video tokens in other few-shot multimodal settings; this is a testable consequence the authors do not run.
  • Beyond the paper: the paper's own heatmaps show increased attention over images but not that the extra attention locks onto task-relevant objects, so a masking experiment that removes the relevant object from the demo images would show whether DARA improves reasoning or merely raises visual weight.
  • Beyond the paper: because DARA leaves standard benchmark scores essentially unchanged, it looks like a safe drop-in for deployment, but the support sets used here are small and clean; whether the gains survive noisy real-world demonstrations with partial or irrelevant images is an untested extension.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents two contributions. First, DARA (Dynamic Attention Reallocation) is a parameter-efficient fine-tuning module that multiplies columns of the pre-softmax attention-score matrix by per-image learnable factors in the first decoder layer, with the goal of making MLLMs rely more on demonstration images during multimodal in-context learning. Second, TrueMICL is a benchmark of seven tasks (operator induction, clock math, outlier detection, CLEVR count, sudoku, palindrome, character classification) that are designed so that correct answers require learning a new image-to-text mapping from demonstrations. Experiments on Qwen2-VL, Idefics3, Phi-3.5-Vision, and GPT-4o report consistent accuracy gains for DARA over random 4-shot ICL and over a same-scale LoRA baseline, plus transfer across tasks and no degradation on standard VL benchmarks.

Significance. If the empirical claims hold, the paper makes a useful contribution: TrueMICL addresses a real evaluation gap, because many existing MICL benchmarks can be solved from text alone, and the no-image baseline in Table 2 confirms that most TrueMICL tasks do require the demonstration images. DARA is also an interesting low-parameter intervention, and the comparison against parameter-matched LoRA in Figure 5 is a genuine strength. The release of code and datasets supports reproducibility. The main caveat is that DARA's claimed mechanism—selectively recovering task-relevant visual content—is not directly established; the evidence shows a global shift of attention mass to image tokens, not content-level selectivity. The benchmark-validity claim also has a notable exception for the Sudoku task on Qwen2-VL.

major comments (4)
  1. [§3.1, §4.3, Fig. 2] DARA's mechanism is asserted but not tested at the content level. Equation (S' = SF) and Algorithm 1 apply one scalar per image to all visual-token score columns of that image, so the intervention can only scale the existing attention pattern; it cannot select the digits, colors, or characters that a TrueMICL task depends on. Section 4.3 reports a global image-token attention shift from 28% to 46.7%, and Figure 2(a) itself states that 'not all focus aligns with the specific object regions.' The layer/head ablation (Appendix D.3) and the hard-coded attention baseline (Section 4.5) test placement and gross over-amplification, not whether the boosted attention reads task-relevant content. The title-level claim that true MICL 'needs attention to the visual context' therefore rests on an interpretation that the experiments do not verify. Please add a content-level diagnostic (for example, attention selectivity toward the target object versus distractors, or an intervention that removes or perturbs only task-relevant image regions) or visibly soften the mechanistic claim.
  2. [Table 2, Sudoku row] TrueMICL's core validity principle is that tasks are unsolvable without the context images (Section 3.2). The No-image baseline in Table 2 violates this for the Sudoku task on Qwen2-VL: accuracy is 90.00% without demo images versus 93.33% with random 4-shot image demos, a gap of only 3.33 points. For Phi-3.5-Vision the gap is larger (52.67% vs 84.67%), but the Qwen2-VL result shows that at least one task is nearly solvable from the textual demo pattern alone for at least one model. This weakens the claim that every TrueMICL task 'must rely on the presence and understanding of demo images.' Please report per-task text-only ceilings explicitly and either redesign or re-justify the Sudoku task in light of this result.
  3. [Table 2, DARA vs Random framing] The abstract and Section 4.2 present DARA's gains primarily as improvements over the Random 4-shot baseline, but DARA is trained on the 30-sample support set with labels, while Random performs pure ICL with no training. This conflates the value of seeing the support-set labels with the specific attention-reallocation effect. The same-scale LoRA row is a reasonable control and should be moved to the foreground of the presentation: the cleanest claim supported by Table 2 is that DARA outperforms parameter-matched LoRA trained on the same support set, and that DARA also improves over Random. Please state this distinction explicitly in the abstract and in Section 4.2.
  4. [§4.5 Human evaluation] The human study is described in a single sentence: 20 participants, no demos versus with demos. No protocol details are given (which tasks, how many queries per participant, what the chance level was, whether the participants saw the same test items as the models, or how the 'generally failed' and 'improved' outcomes were measured). Because this study is used to validate TrueMICL, the missing details make the validation claim difficult to assess.
minor comments (6)
  1. [Abstract and §3.2] The total sample count is inconsistent: the abstract and Section 3.2 say 860 and 867 samples, respectively; Table 1 sums to 860 (210 support plus 650 test). Please align the numbers.
  2. [Appendix A] The proof statement contains a typo ('∃ ∆WQ and ∃ ∆WQ', the second should be ∆WK), and the claim that 'any matrix admits a low-rank factorization' should specify that the rank is at most the matrix rank; more importantly, the constructed ∆WK is input-dependent, which is precisely why the appendix must conclude that DARA is a constrained variant of LoRA rather than an equivalent. This caveat is currently only in the appendix and should be reflected in the main-text sentence describing the relationship.
  3. [Fig. 2(a)] The heatmaps are shown for a single example; given the authors' own remark that not all focus aligns with object regions, a quantitative summary over many examples would be more informative than one cherry-picked visualization.
  4. [Table 8] The standard VL benchmark table reports no standard deviations or number of runs, so it is unclear whether the small differences among Zero-shot, No-image, Random, LoRA, and DARA are within noise.
  5. [§4.2 and Table 2] The baseline is named 'RICES' in the text and table but 'RICE' in the sentence describing Table 2; please use one spelling. Also, the retrieval embedding used for RICES is not specified.
  6. [Fig. 6 and Table 9] The layer/head ablation and the LoRA+DARA combination are only reported for Qwen2-VL in Figure 6 and for all models in Table 9; adding the corresponding standard deviations or error bars would make the claimed small gains more convincing.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; DARA's gains are an empirical fine-tuning result and TrueMICL's visual dependency is validated independently of the method.

full rationale

The paper's central claims are empirical rather than derivational, and the reported results do not reduce to the method's own construction. DARA learns roughly 160 attention-scaling parameters on the support set and is then evaluated on held-out test queries in TrueMICL; this is standard few-shot fine-tuning, so the test accuracy is not a fitted value relabeled as a prediction. The benchmark's assertion that tasks require visual context is supported by the no-image baseline, which performs near zero-shot levels, and by a human study in which participants fail without demonstrations; therefore the benchmark is not rigged to DARA by definition. The attention reallocation itself, S' = SF, makes it true by construction that image-token attention is amplified, but the paper's substantive claim is that this amplification improves downstream accuracy, and that claim is tested empirically across models and tasks. The only self-citation with author overlap is Chen et al. (2023b), used to motivate attention imbalance as a possible cause of visual neglect; this is not load-bearing because the paper independently reproduces the phenomenon and measures attention ratios. Appendix A's LoRA-equivalence discussion explicitly states that DARA is a constrained variant, not a technical equivalent, and no experimental prediction is derived from that equivalence. The mechanism-level gap noted by skeptics, namely that per-image scalar scaling cannot select task-relevant objects, is a limitation in causal interpretation rather than circularity. Accordingly, no circular step satisfying the required evidentiary standard is present, and the appropriate finding is a score of 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper's empirical claims rest primarily on the design assumptions of DARA (attention scaling suffices, first layer is the right place) and of the benchmark (text alone cannot solve the tasks). No new physical or ontological entities are postulated; DARA is a parameterization of existing attention weights and TrueMICL is a dataset, not an entity in the graviton sense. The attention factors f are the main fitted parameters, and the layer/head placement is a hand-chosen design choice.

free parameters (2)
  • DARA attention-balancing factors f_j (per attention head, per demo or query image position) = Learned separately for each task support set; initialized to 1.0; Figure 3 shows values roughly 0.90-1.32 for Qwen2-VL…
    These are the parameters that constitute the method; they scale the columns of the pre-softmax attention score matrix for visual tokens (Section 3.1, Algorithm 1). They are fit to the 30-sample support set for each task, so their values are data-dependent rather than derived.
  • DARA placement: first transformer layer and all attention heads = Layer 0, all heads (e.g., 32 heads for Qwen2-VL, 160 params with 4 demos plus query)
    Chosen by hand, not learned; the paper justifies it with the layer ablation (Appendix D.3) showing comparable or worse performance with more layers and higher parameter cost.
assumptions (4)
  • domain assumption Scaling pre-softmax attention-score columns for image tokens is sufficient to make a frozen MLLM actually use the visual information in demonstrations.
    Invoked in Section 3.1 (S' = S F) and is the core mechanistic premise of DARA. Not proven; empirically supported only indirectly by task accuracy gains.
  • ad hoc to paper The first layer of the language backbone is the effective place to modulate cross-modal attention.
    Adopted in Section 4.1 (DARA inserted into the first transformer layer) and defended by the ablation in Appendix D.3; results show similar accuracy across layers, so the premise is a convenience rather than a necessity.
  • domain assumption The TrueMICL tasks are unsolvable from demonstration text alone and require understanding of demo images.
    Core design principle (Section 3.2, 'Context dependency'); supported by the no-image baseline results (Table 2) and the human evaluation (Section 4.5), but the benchmark is constructed by the authors.
  • domain assumption Pan et al.'s (2023) separation of ICL into task recognition and task learning carries over to multimodal models.
    Used to justify why standard VL datasets (e.g., captioning) do not test true MICL (Sections 2, 3.2). This is a conceptual prior from the language-only ICL literature.

how reviews work

0 comments
Cite this review

Pith. "Pith review of True Multimodal In-Context Learning Needs Attention to the Visual Context." pith.science (2026). https://pith.science/paper/LBV7BVV5

@misc{pith2026250715807,
  author       = {Pith},
  title        = {Pith review of: True Multimodal In-Context Learning Needs Attention to the Visual Context},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LBV7BVV5}},
  note         = {Machine review of arXiv:2507.15807}
}
read the original abstract

Multimodal Large Language Models (MLLMs), built on powerful language backbones, have enabled Multimodal In-Context Learning (MICL)-adapting to new tasks from a few multimodal demonstrations consisting of images, questions, and answers. Despite showing noticeable improvement on standard vision-language datasets, current MLLMs struggle to leverage visual information in the demonstrations. Specifically, they tend to neglect visual cues and over-rely on textual patterns, leading to mere text imitation rather than genuine multimodal adaptation. This behavior makes MICL still unimodal and largely restricts its practical utility. More importantly, this limitation is often concealed by the improved performance on tasks that do not require understanding the visual context. As a result, how to effectively enhance MICL ability and reliably evaluate the MICL performance remains underexplored. To address these issues, we first introduce Dynamic Attention Reallocation (DARA), an efficient fine-tuning strategy that encourages models to attend to the visual context by rebalancing attention across visual and textual tokens. In addition, we present TrueMICL, an MICL-dedicated dataset with both support and test sets that explicitly requires the integration of multimodal information-particularly visual content-for correct task completion. Extensive experiments demonstrate the effectiveness of our holistic solution, showcasing substantial improvements in the true multimodal in-context learning capabilities. Code and datasets are available at https://chenxshuo.github.io/true-micl-colm .

Figures

Figures reproduced from arXiv: 2507.15807 by the authors.

Figure 1
Figure 1. Examples of using MICL to solve image captioning from MSCOCO ( [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. DARA enhances visual attention both qualitatively (left) and quantitatively (right). [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Learned attention amplification factors across [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Performance comparison on Operator and Clock tasks across three models with [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Comparison between DARA (green point) and LoRA (blue curve). DARA outperforms the 4-shot baseline using only 140 parameters, whereas LoRA requires tens of thousands to reach similar performance, highlighting DARA’s parameter efficiency. Model Method Operator Clock Outl…
Figure 6
Figure 6. Figure 6: Comparison of performances when applying different settings of changed attention [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Dissecting Multimodal In-Context Learning: Modality Asymmetries and Circuit Dynamics in modern Transformers

    cs.CL 2026-01 conditional novelty 6.0 of 10

    In a two-modality transformer, a primary-modality pretraining stage installs an induction circuit, so the secondary modality needs only low class diversity to learn in-context from examples.

Reference graph

Works this paper leans on

33 extracted references · 7 canonical work pages · cited by 1 Pith paper

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219,

    arXiv

  2. [4]

    Each task is designed with adjustable difficulty levels, such as more diverse con- cepts in novel concept binding, more complex visual patterns in pattern interpretation, etc

    TrueMICL encompasses a wide range of MICL capabilities, including mathematical reasoning, novel concept binding, and pattern interpretation, providing a comprehensive dataset for multimodal in-context learning. Each task is designed with adjustable difficulty levels, such as more diverse con- cepts in novel concept binding, more complex visual patterns in...

    work page 2024

  3. [5]

    A survey for in-context learning

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in-context learning. arXiv preprint arXiv:2301.00234,

    arXiv

  4. [7]

    Wiest, Marta Ligero, Srividhya Sainath, Narmin Ghaffari Laleh, Omar S.M

    Dyke Ferber, Georg W ¨olflein, Isabella C. Wiest, Marta Ligero, Srividhya Sainath, Narmin Ghaffari Laleh, Omar S.M. El Nahhas, Gustav M ¨uller-Franzes, Dirk J¨ager, Daniel Truhn, and Jakob Nikolas Kather. In-context learning enables multimodal large language models to classify cancer pathology images. arXiv preprint arXiv:2403.07407,

    arXiv

  5. [8]

    In-context learning enables multimodal large language models to classify cancer pathology images

    URL https://doi.org/10.48550/arXiv.2403.07407. 40 pages, 5 figures. 11 Published as a conference paper at COLM 2025 Jun Gao, Qian Qiao, Tianxiang Wu, Zili Wang, Ziqiang Cao, and Wenjie Li. Aim: Let any multimodal large language models embrace efficient in-context learning. In Proceedings of the AAAI Conference on Artificial Intelligence, number 3 in 39, p...

    work page Pith review arXiv doi:10.48550/arxiv.2403.07407 2025

  6. [13]

    Chen, and Andrew Y

    Yixing Jiang, Jeremy Irvin, Ji Hun Wang, Muhammad Ahmed Chaudhry, Jonathan H. Chen, and Andrew Y. Ng. Many-shot in-context learning in multimodal foundation models. arXiv preprint arXiv:2405.09798,

    arXiv

  7. [14]

    2405.09798

    URL https://doi.org/10.48550/arXiv. 2405.09798. Benchmarking GPT-4o and Gemini 1.5 Pro across 14 multimodal datasets. Hugo Laurenc ¸on, Andr´es Marafioti, Victor Sanh, and L´eo Tronchon. Building and better understanding vision-language models: insights and future directions. In Workshop on Responsibly Building the Next Generation of Multimodal Foundation...

    arXiv doi:10.48550/arxiv.2405.09798

  8. [15]

    Mibench: Evaluating multimodal large language models over multiple images

    Haowei Liu, Xi Zhang, Haiyang Xu, Yaya Shi, Chaoya Jiang, Ming Yan, Ji Zhang, Fei Huang, Chunfeng Yuan, Bing Li, et al. Mibench: Evaluating multimodal large language models over multiple images. arXiv preprint arXiv:2407.15272, 2024a. Yilun Liu, Yunpu Ma, Shuo Chen, Zifeng Ding, Bailan He, Zhen Han, and Volker Tresp. Perft: Parameter-efficient routed fine...

    arXiv

Show all 33 references
  1. [16]

    A survey on lora of large language models

    12 Published as a conference paper at COLM 2025 Yuren Mao, Yuhang Ge, Yijiang Fan, Wenyi Xu, Yu Mi, Zhonghao Hu, and Yunjun Gao. A survey on lora of large language models. Frontiers of Computer Science, 19(7):197605,

    2025

  2. [17]

    Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, and et al

    OpenAI, Aaron Hurst, Adam Lerer, Adam P . Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, and et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276,

    arXiv

  3. [18]

    Cov- ers architecture, multimodal input/output capabilities, and safety evaluations of GPT-4o

    URL https://doi.org/10.48550/arXiv.2410.21276. Cov- ers architecture, multimodal input/output capabilities, and safety evaluations of GPT-4o. Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. What in-context learning “learns” in-context: Disentangling task recognition and tas...

    doi arXiv

  4. [19]

    doi: 10.18653/v1/2023.findings-acl.527

    As- sociation for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.527. URL https://aclanthology.org/2023.findings-acl.527. Libo Qin, Qiguang Chen, Hao Fei, Zhi Chen, Min Li, and Wanxiang Che. What factors affect multi-modal in-context learning? an in-depth explor...

    2023 arXiv

  5. [20]

    A systematic survey of prompt engineering in large language models: Techniques and applications

    Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:2402.07927,

    arXiv

  6. [21]

    12 pages, 2 figures; provides taxonomy, strengths/limitations, datasets and models for prompt engineering

    URL https://doi.org/10.48550/ arXiv.2402.07927. 12 pages, 2 figures; provides taxonomy, strengths/limitations, datasets and models for prompt engineering. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhi...

    doi arXiv

  7. [23]

    Accepted by EACL

    URL https://doi.org/10.48550/ arXiv.2307.07164. Accepted by EACL

    doi arXiv

  8. [24]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024b. Nan Xu, Fei Wang, Sheng Z...

    arXiv

  9. [25]

    Low-rank adaptation for foundation models: A comprehensive review

    Menglin Yang, Jialin Chen, Yifei Zhang, Jiahong Liu, Jiasheng Zhang, Qiyao Ma, Harshit Verma, Qianru Zhang, Min Zhou, Irwin King, et al. Low-rank adaptation for foundation models: A comprehensive review. arXiv preprint arXiv:2501.00365, 2024a. Xu Yang, Yongliang Wu, Mingzhuo Y...

  10. [26]

    In-context example selection via similarity search improves low-resource machine translation

    Armel Zebaze, Benoˆıt Sagot, and Rachel Bawden. In-context example selection via similarity search improves low-resource machine translation. arXiv preprint arXiv:2408.00397,

    arXiv

  11. [27]

    13 Published as a conference paper at COLM 2025 Haowei Zhang, Jianzhe Liu, Zhen Han, Shuo Chen, Bailan He, Volker Tresp, Zhiqiang Xu, and Jindong Gu

    URL https://doi.org/10.48550/arXiv.2408.00397. 13 Published as a conference paper at COLM 2025 Haowei Zhang, Jianzhe Liu, Zhen Han, Shuo Chen, Bailan He, Volker Tresp, Zhiqiang Xu, and Jindong Gu. Visual question decomposition on multimodal large language models. arXiv preprin...

    2025 doi arXiv

  12. [28]

    Mmicl: Empowering vision-language model with multi-modal in-context learning

    Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. Mmicl: Empowering vision-language model with multi-modal in-context learning. arXiv preprint arXiv:2309.07915,

    arXiv

  13. [29]

    Vl-icl bench: The devil in the details of benchmarking multimodal in-context learning

    Yongshuo Zong, Ondrej Bohdal, and Timothy Hospedales. Vl-icl bench: The devil in the details of benchmarking multimodal in-context learning. arXiv preprint arXiv:2403.13164,

    arXiv

  14. [30]

    Suppose we introduce an attention reallocation factor to the softmax operation by defining F := diag(f) ∈ RL×L, where f ∈ RL is a vector of learnable factors

    A DARA as a constrained version of LoRA Consider a transformer-based attention mechanism with query, key, and value transfor- mations defined by WQ, WK, WV, and let S := QK⊤ √ d be the attention score matrix, where Q := XWQ and K := XWK, where X is the input tensor to the tran...

    2025

  15. [32]

    As the model is finetuned using the settings described in Table 6, this parameter matrix is updated automatically. Parameter Value Batch size per device 1 Gradient accumulation steps 4 Epochs 5 Learning rate 1e-3 Warmup steps 5 Optimizer AdamW (8-bit) Weight decay 0.01 Learnin...

    2025

  16. [33]

    As shown in the figure, within the range of a few hundred parameters, different configurations have no significant difference in the impact on the final performance

    × (number of heads) × (number of layers). As shown in the figure, within the range of a few hundred parameters, different configurations have no significant difference in the impact on the final performance. D.4 Effect of Prompt Design We provide further analysis on the effect...

    2025

  17. [2015]

    Openflamingo: An open- source framework for training large autoregressive vision-language models.arXiv preprint arXiv:2308.01390,

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision-language models.arXiv preprint arXiv:2308.01390,

    arXiv

  18. [2017]

    Advanced multimodal deep learning architecture for image-text matching

    Jinyin Wang, Haijing Zhang, Yihao Zhong, Yingbin Liang, Rongwei Ji, and Yiru Cang. Advanced multimodal deep learning architecture for image-text matching. arXiv preprint arXiv:2406.15306, 2024a. URL https://doi.org/10.48550/arXiv.2406.15306. Proposes a novel architecture with ...

    doi arXiv

  19. [2019]

    Symdpo: Boosting in-context learning of large multi- modal models with symbol demonstration direct preference optimization

    Hongrui Jia, Chaoya Jiang, Haiyang Xu, Wei Ye, Mengfan Dong, Ming Yan, Ji Zhang, Fei Huang, and Shikun Zhang. Symdpo: Boosting in-context learning of large multi- modal models with symbol demonstration direct preference optimization. arXiv preprint arXiv:2411.11909,

    arXiv

  20. [2020]

    Benchmarking robustness of adaptation methods on pre-trained vision-language models

    Shuo Chen, Jindong Gu, Zhen Han, Yunpu Ma, Philip Torr, and Volker Tresp. Benchmarking robustness of adaptation methods on pre-trained vision-language models. Advances in Neural Information Processing Systems, 36:51758–51777, 2023a. Shuo Chen, Zhen Han, Bailan He, Mark Buckley...

    arXiv

  21. [2021]

    Mimicking or reasoning: Rethinking multi-modal in-context learning in vision-language models

    Chengyue Huang, Yuchen Zhu, Sichen Zhu, Jingyun Xiao, Moises Andrade, Shivang Chopra, and Zsolt Kira. Mimicking or reasoning: Rethinking multi-modal in-context learning in vision-language models. arXiv preprint arXiv:2506.07936,

    arXiv

  22. [2022]

    Towards multimodal in-context learning for vision & language models

    Sivan Doveh, Shaked Perek, M Jehanzeb Mirza, Amit Alfassy, Assaf Arbelle, Shimon Ullman, and Leonid Karlinsky. Towards multimodal in-context learning for vision & language models. arXiv preprint arXiv:2403.12736,

    arXiv

  23. [2023]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685,

    arXiv

  24. [2024]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901,

    1901

  25. [2025]

    A systematic survey of prompt engineering on vision-language foundation models

    Jindong Gu, Zhen Han, Shuo Chen, Ahmad Beirami, Bailan He, Gengyuan Zhang, Ruotong Liao, Yao Qin, Volker Tresp, and Philip Torr. A systematic survey of prompt engineering on vision-language foundation models. arXiv preprint arXiv:2307.12980,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.