Pith. sign in

REVIEW 4 cited by

Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2312.11420 v1 pith:RT3L3OB6 submitted 2023-12-18 cs.CL cs.AIcs.CV

Bingchen Zhao , Haoqin Tu , Chen Wei , Jieru Mei , Cihang Xie This is my paper · ORCID
classification cs.CLcs.AIcs.CV
keywords layernormmulti-modaltuningattentiondomainefficiencyefficientfinetuning
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

This paper introduces an efficient strategy to transform Large Language Models (LLMs) into Multi-Modal Large Language Models (MLLMs). By conceptualizing this transformation as a domain adaptation process, i.e., transitioning from text understanding to embracing multiple modalities, we intriguingly note that, within each attention block, tuning LayerNorm suffices to yield strong performance. Moreover, when benchmarked against other tuning approaches like full parameter finetuning or LoRA, its benefits on efficiency are substantial. For example, when compared to LoRA on a 13B model scale, performance can be enhanced by an average of over 20% across five multi-modal tasks, and meanwhile, results in a significant reduction of trainable parameters by 41.9% and a decrease in GPU memory usage by 17.6%. On top of this LayerNorm strategy, we showcase that selectively tuning only with conversational data can improve efficiency further. Beyond these empirical outcomes, we provide a comprehensive analysis to explore the role of LayerNorm in adapting LLMs to the multi-modal domain and improving the expressive power of the model.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CrossEarth-Gate: Fisher-Guided Adaptive Tuning Engine for Efficient Adaptation of Cross-Domain Remote Sensing Semantic Segmentation

    cs.CV 2025-11 conditional novelty 6.0 of 10

    A Fisher-information-guided dynamic selection over a toolbox of LoRA, adapter, and frequency-adapter modules improves cross-domain remote sensing segmentation over static PEFT methods.

  2. Implementing Adaptations for Vision AutoRegressive Model

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Fine-tuned Vision AutoRegressive models mostly beat a strong diffusion baseline on downstream image generation, but DP fine-tuning yields poor FID scores.

  3. Time Series Foundation Models for Multivariate Financial Time Series Forecasting

    q-fin.GN 2025-07 reject novelty 6.0 of 10

    Pretrained TTM shows large transfer and sample-efficiency gains in three financial forecasting tasks relative to training from scratch, but methodological flaws including possible look-ahead bias weaken the quantitati...

  4. Exploiting Layer Normalization Fine-tuning in Visual Transformer Foundation Models for Classification

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    LayerNorm parameter shifts after fine-tuning encode domain-transition information; rescaling them via an FSR-dependent scalar lambda plus a cyclic step improves ViT classification under data scarcity and domain shift.

Pith tools