Pith. sign in

REVIEW 5 cited by

Asymmetry in Low-Rank Adapters of Foundation Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.16842 v2 pith:SPXDELWA submitted 2024-02-26 cs.LG

Jiacheng Zhu , Kristjan Greenewald , Kimia Nadjahi , Haitz Sáez de Ocáriz Borde , Rickard Brüel Gabrielsson , Leshem Choshen , Marzyeh Ghassemi , Mikhail Yurochkin
show 1 more author
Justin Solomon
This is my paper
classification cs.LG
keywords fine-tuninglow-rankmatricesadaptersasymmetryboundeffectivefeatures
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Parameter-efficient fine-tuning optimizes large, pre-trained foundation models by updating a subset of parameters; in this class, Low-Rank Adaptation (LoRA) is particularly effective. Inspired by an effort to investigate the different roles of LoRA matrices during fine-tuning, this paper characterizes and leverages unexpected asymmetry in the importance of low-rank adapter matrices. Specifically, when updating the parameter matrices of a neural network by adding a product $BA$, we observe that the $B$ and $A$ matrices have distinct functions: $A$ extracts features from the input, while $B$ uses these features to create the desired output. Based on this observation, we demonstrate that fine-tuning $B$ is inherently more effective than fine-tuning $A$, and that a random untrained $A$ should perform nearly as well as a fine-tuned one. Using an information-theoretic lens, we also bound the generalization of low-rank adapters, showing that the parameter savings of exclusively training $B$ improves the bound. We support our conclusions with experiments on RoBERTa, BART-Large, LLaMA-2, and ViTs.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Noise Consistency Regularization for Improved Subject-Driven Image Synthesis

    cs.GR 2025-06 conditional novelty 6.0 of 10

    Adding consistency-to-pretrained and multiplicative-noise consistency losses to fine-tuning improves subject identity and background diversity over DreamBooth on a 30-subject benchmark.

  2. CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning

    cs.CV 2025-05 conditional novelty 6.0 of 10

    CL-LoRA adds a fixed random-orthogonal shared LoRA branch for cross-task knowledge and task-specific LoRA branches with block-wise weights, improving rehearsal-free class-incremental learning accuracy at low parameter cost.

  3. DeCAF: Decentralized Consensus-And-Factorization for Low-Rank Adaptation of Foundation Models

    cs.LG 2025-05 reject novelty 6.0 of 10

    A truncated-SVD consensus step for decentralized LoRA is claimed to reach O(1/sqrt T) convergence, matching decentralized SGD, with supporting CLIP and LLAMA2-7B experiments.

  4. Continual Knowledge Consolidation LORA for Domain Incremental Learning

    cs.LG 2025-10 conditional novelty 5.0 of 10

    CONEC-LoRA reports state-of-the-art accuracy on four domain-incremental benchmarks by combining task-shared and task-specific LoRAs with a stochastic classifier and a learned domain-ID selector.

  5. CoLA: Collaborative Low-Rank Adaptation

    cs.CL 2025-05 conditional novelty 4.0 of 10

    CoLA generalizes LoRA to multiple A and B matrices with a principal-component initialization and reports gains of roughly 2-4 accuracy points over PiSSA on low-sample fine-tuning benchmarks.

Pith tools