Pith. sign in

REVIEW 4 cited by

Mixture-of-Subspaces in Low-Rank Adaptation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.11909 v4 pith:X2E3PX63 submitted 2024-06-16 cs.LG cs.AI

Taiqiang Wu , Jiahao Wang , Zhe Zhao , Ngai Wong This is my paper · ORCID
classification cs.LGcs.AI
keywords loramosloraadaptationlow-rankmethodmixermixture-of-subspacessubspaces
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In this paper, we introduce a subspace-inspired Low-Rank Adaptation (LoRA) method, which is computationally efficient, easy to implement, and readily applicable to large language, multimodal, and diffusion models. Initially, we equivalently decompose the weights of LoRA into two subspaces, and find that simply mixing them can enhance performance. To study such a phenomenon, we revisit it through a fine-grained subspace lens, showing that such modification is equivalent to employing a fixed mixer to fuse the subspaces. To be more flexible, we jointly learn the mixer with the original LoRA weights, and term the method Mixture-of-Subspaces LoRA (MoSLoRA). MoSLoRA consistently outperforms LoRA on tasks in different modalities, including commonsense reasoning, visual instruction tuning, and subject-driven text-to-image generation, demonstrating its effectiveness and robustness. Codes are available at https://github.com/wutaiqiang/MoSLoRA.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. One Rank at a Time: Cascading Error Dynamics in Sequential Learning

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Errors from each rank-1 step in sequential low-rank learning compound through factors that grow when singular values are close, so early steps deserve more compute.

  2. Regularizing Subspace Redundancy of Low-Rank Adaptation

    cs.CV 2025-07 conditional novelty 5.0 of 10

    ReSoRA adds a penalty that reduces redundancy among rank-1 subspaces of LoRA-style adapters, producing modest accuracy improvements on vision-language retrieval and visual classification.

  3. Weight Spectra Induced Efficient Model Adaptation

    cs.LG 2025-05 reject novelty 4.0 of 10

    Fine-tuning mostly amplifies and reorients the top singular directions of weight matrices, and SpecLoRA learns to rescale a top-left block plus LoRA to improve PEFT performance.

  4. MAP: Revisiting Weight Decomposition for Low-Rank Adaptation

    cs.LG 2025-05 conditional novelty 4.0 of 10

    MAP decouples a weight matrix's direction and magnitude by normalizing the whole matrix and the low-rank update by their Frobenius norms and scaling each with a learnable scalar.

Pith tools