Pith. sign in

REVIEW 11 cited by

LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.13025 v2 pith:UXQ3QGRP submitted 2024-10-16 cs.CL cs.LG

Akshara Prabhakar , Yuanzhi Li , Karthik Narasimhan , Sham Kakade , Eran Malach , Samy Jelassi This is my paper · ORCID
classification cs.CLcs.LG
keywords skillcompositionloramergingdatalorasmodelskills
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Low-Rank Adaptation (LoRA) is a popular technique for parameter-efficient fine-tuning of Large Language Models (LLMs). We study how different LoRA modules can be merged to achieve skill composition -- testing the performance of the merged model on a target task that involves combining multiple skills, each skill coming from a single LoRA. This setup is favorable when it is difficult to obtain training data for the target task and when it can be decomposed into multiple skills. First, we identify practically occurring use-cases that can be studied under the realm of skill composition, e.g. solving hard math-word problems with code, creating a bot to answer questions on proprietary manuals or about domain-specialized corpora. Our main contribution is to show that concatenation of LoRAs (CAT), which optimally weights LoRAs that were individually trained on different skills, outperforms existing model- and data- merging techniques; for instance on math-word problems, CAT beats these methods by an average of 43% and 12% respectively. Thus, this paper advocates model merging as an efficient way to solve compositional tasks and underscores CAT as a simple, compute-friendly and effective procedure. To our knowledge, this is the first work demonstrating the superiority of model merging over data mixing for binary skill composition tasks. Code and data are available at https://github.com/aksh555/LoRA-Soups

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Harnessing Optimization Dynamics for Curvature-Informed Model Merging

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Optimization Trajectory Aware merging uses Adam second moments as a curvature proxy, first pruning task-vector edits with Fast Fisher Grafting, then reweighting survivors with a compressed curvature preconditioner.

  2. From Generation to Generalization: Emergent Few-Shot Learning in Video Diffusion Models

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Frozen CogVideoX1.5, adapted with LoRA on 3 to 30 input-output videos, performs segmentation, pose estimation, and abstract reasoning (ARC-AGI 16.75%) with modest but real generalization.

  3. FedRPCA: Enhancing Federated LoRA Aggregation Using Robust PCA

    cs.LG 2025-06 conditional novelty 6.0 of 10

    FedRPCA decomposes federated LoRA client updates with Robust PCA into common and client-specific components, averaging the common part and scaled-averaging the sparse part, which improves accuracy and convergence over...

  4. Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Initializing LoRA's A matrix with the smallest-eigenvalue eigenvectors of other tasks' feature covariance reduces interference when merging task-specific LoRA models, improving average merged accuracy.

  5. MultLFG: Training-free Multi-LoRA composition using Frequency-domain Guidance

    cs.CV 2025-05 conditional novelty 6.0 of 10

    MultLFG merges multiple LoRA adapters by adaptively weighting them in wavelet frequency subbands per denoising timestep, improving multi-concept composition on the ComposLoRA benchmark compared to prior training-free methods.

  6. CT-Merging: Consensus Directions and Task-Level Scaling for LoRA Adapter Merging

    cs.LG 2026-07 conditional novelty 5.0 of 10

    A LoRA-merging method using average-projector consensus directions and per-task RMS scaling beats prior SVD-based mergers on most CLIP merging benchmarks.

  7. Tensorized Clustered LoRA Merging for Multi-Task Interference

    cs.LG 2025-08 unverdicted novelty 5.0 of 10

    Clustering training data by embedding similarity and jointly CP-decomposing LoRA adapters cuts multi-task merging interference: +1.4% on Phi-3 and +2.3% on Mistral-7B over SVD baselines.

  8. Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Sparse adapters trained with max connection sensitivity outperform LoRA and full fine-tuning both alone and after merging 20 task experts, but still lag multitask training on unseen tasks.

  9. Continual Learning in Vision-Language Models via Aligned Model Merging

    cs.CV 2025-05 conditional novelty 5.0 of 10

    PAM merges a task-specific LoRA into a global LoRA and re-initializes sign-conflicting weights during training, reducing catastrophic forgetting in continual VLM learning.

  10. Sci-LoRA: Mixture of Scientific LoRAs for Cross-Domain Lay Paraphrasing

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Sci-LoRA dynamically mixes domain-specific LoRA adapters and achieves state-of-the-art lay paraphrasing across twelve domains without needing domain labels at inference.

  11. Why Do More Experts Fail? A Theoretical Analysis of Model Merging

    cs.LG 2025-05 reject novelty 4.0 of 10

    The paper claims to prove an upper bound and diminishing returns in model merging, but the proofs are not sound and the heavy-tailed claim is contradicted by its own equations.

Pith tools