Pith. sign in

REVIEW 23 cited by

LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2307.13269 v3 pith:IVYAGBOX submitted 2023-07-25 cs.CL cs.AI

Chengsong Huang , Qian Liu , Bill Yuchen Lin , Tianyu Pang , Chao Du , Min Lin This is my paper · ORCID
classification cs.CLcs.AI
keywords loralorahubmodulestaskscompositioncross-taskexamplesgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Low-rank adaptations (LoRA) are often employed to fine-tune large language models (LLMs) for new tasks. This paper investigates LoRA composability for cross-task generalization and introduces LoraHub, a simple framework devised for the purposive assembly of LoRA modules trained on diverse given tasks, with the objective of achieving adaptable performance on unseen tasks. With just a few examples from a new task, LoraHub can fluidly combine multiple LoRA modules, eliminating the need for human expertise and assumptions. Notably, the composition requires neither additional model parameters nor gradients. Empirical results on the Big-Bench Hard benchmark suggest that LoraHub, while not surpassing the performance of in-context learning, offers a notable performance-efficiency trade-off in few-shot scenarios by employing a significantly reduced number of tokens per example during inference. Notably, LoraHub establishes a better upper bound compared to in-context learning when paired with different demonstration examples, demonstrating its potential for future development. Our vision is to establish a platform for LoRA modules, empowering users to share their trained LoRA modules. This collaborative approach facilitates the seamless application of LoRA modules to novel tasks, contributing to an adaptive ecosystem. Our code is available at https://github.com/sail-sg/lorahub, and all the pre-trained LoRA modules are released at https://huggingface.co/lorahub.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 23 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SkillSmith: Learning to Compose Parametric Skills and Textual Knowledge

    cs.CL 2026-07 conditional novelty 7.0 of 10

    SkillSmith, an LLM augmented to ingest prefix-weights and text, generates target-task prefix-weights that beat text-only and weight-only baselines, especially as fine-tuning initialization.

  2. Rethinking Transfer in Continual Learning: A Replay-Based Realisation

    cs.LG 2026-07 conditional novelty 7.0 of 10

    In continual learning, forward transfer requires target headroom, a persistent carrier, and a compatible source; routing replay by gradient signatures improves accuracy and stability over uniform replay.

  3. Model Merging for Medical LVLMs: A Benchmark and a Winner-Take-All Approach

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A per-parameter winner-take-all merge beats averaging and subspace methods on a new 16-model, 8-modality medical LVLM benchmark.

  4. MED-DSLC: Multi-Expert-Domain Classification via Domain Supervision and Logit Calibration

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Domain-supervised MoE routing plus per-domain temperature scaling restores global logit calibration when merging frozen LoRA experts, cutting cross-domain interference in multi-domain VLM classification.

  5. Localized LoRA-MoE: Block-wise Low-Rank Experts With Adaptive Routing

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Block-wise and cell-wise LoRA-MoE routers break static LoRA gradient conflicts on multi-context matrix tasks, with cell-level gates matching a global router on uniform shifts and beating it on heterogeneous ones.

  6. Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Sleep-time Knowledge Seeding plus Dreaming lets LLMs expand capacity, distill fragile in-context memories into stable parameters, and self-improve without human labels.

  7. LoRA-Gen: Specializing Large Language Model via Online LoRA Generation

    cs.CL 2025-06 conditional novelty 6.0 of 10

    LoRA-Gen generates task-specific LoRA weights with a cloud-side LLM and reparameterizes them into a smaller edge model, enabling training-free specialization with compressed context.

  8. From Generation to Generalization: Emergent Few-Shot Learning in Video Diffusion Models

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Frozen CogVideoX1.5, adapted with LoRA on 3 to 30 input-output videos, performs segmentation, pose estimation, and abstract reasoning (ARC-AGI 16.75%) with modest but real generalization.

  9. Cartridges: Lightweight and general-purpose long context representations via self-study

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A per-corpus trained KV cache, called a Cartridge, matches full-context in-context learning quality on long-document benchmarks while using up to 38.6x less serving memory.

  10. Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptation

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Gradually increasing the probability that LoRA adapters stay active during fine-tuning improves generalization, merging, and pruning robustness.

  11. MultLFG: Training-free Multi-LoRA composition using Frequency-domain Guidance

    cs.CV 2025-05 conditional novelty 6.0 of 10

    MultLFG merges multiple LoRA adapters by adaptively weighting them in wavelet frequency subbands per denoising timestep, improving multi-concept composition on the ComposLoRA benchmark compared to prior training-free methods.

  12. FedHL: Federated Learning for Heterogeneous Low-Rank Adaptation via Unbiased Aggregation

    cs.LG 2025-05 reject novelty 6.0 of 10

    FedHL aggregates heterogeneous LoRA updates against a full-rank global baseline and claims O(1/sqrt T) convergence, with small gains on three LLM fine-tuning datasets.

  13. Model Merging is Secretly Certifiable: Non-Vacuous Generalisation Bounds for Low-Shot Learning

    cs.LG 2025-05 conditional novelty 6.0 of 10

    First non-vacuous PAC-Bayes certificates for large vision and language models in the 100-example low-shot regime, obtained by reinterpreting model merging as a low-dimensional posterior.

  14. GRIP: In-Parameter Graph Reasoning through Fine-Tuning Large Language Models

    cs.CL 2025-11 reject novelty 5.0 of 10

    An LLM can memorize a knowledge graph into LoRA weights and answer relation/reasoning queries about it without graph context, but the evaluation partly trains on the test task.

  15. Semantic-guided LoRA Parameters Generation

    cs.LG 2025-09 conditional novelty 5.0 of 10

    SG-LoRA generates LoRA parameters for unseen tasks from text descriptions alone, using semantic expert selection plus a conditional VAE, matching or exceeding oracle fine-tuning on retrieval benchmarks.

  16. Joint Information Extraction Across Classical and Modern Chinese with Tea-MOELoRA

    cs.CL 2025-09 conditional novelty 5.0 of 10

    Tea-MOELORA uses separate task and era gates over LoRA experts to jointly train relation and event extraction across classical and modern Chinese, improving F1 over joint LoRA and existing LoRA-MoE baselines on most datasets.

  17. Tensorized Clustered LoRA Merging for Multi-Task Interference

    cs.LG 2025-08 unverdicted novelty 5.0 of 10

    Clustering training data by embedding similarity and jointly CP-decomposing LoRA adapters cuts multi-task merging interference: +1.4% on Phi-3 and +2.3% on Mistral-7B over SVD baselines.

  18. Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Sparse adapters trained with max connection sensitivity outperform LoRA and full fine-tuning both alone and after merging 20 task experts, but still lag multitask training on unseen tasks.

  19. LiON-LoRA: Rethinking LoRA Fusion to Unify Controllable Spatial and Temporal Generation for Video Diffusion

    cs.CV 2025-07 conditional novelty 5.0 of 10

    LiON-LoRA adds a learned scaling token to video-diffusion LoRA adapters, enabling linear and independent control of camera trajectory and object motion strength.

  20. FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Experts

    cs.LG 2025-05 conditional novelty 5.0 of 10

    FLoE uses Fisher information to pick the transformer layers that matter and a Bayesian optimizer to set LoRA rank, cutting trainable parameters while keeping or improving accuracy.

  21. Sci-LoRA: Mixture of Scientific LoRAs for Cross-Domain Lay Paraphrasing

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Sci-LoRA dynamically mixes domain-specific LoRA adapters and achieves state-of-the-art lay paraphrasing across twelve domains without needing domain labels at inference.

  22. Neural Parameter Search for Slimmer Fine-Tuned Models and Better Transfer

    cs.LG 2025-05 conditional novelty 5.0 of 10

    Neural Parameter Search (NPS) prunes fine-tuned models by evolutionary reweighting of magnitude-based task vector subspaces, improving transfer, fusion, and compression.

  23. Why Do More Experts Fail? A Theoretical Analysis of Model Merging

    cs.LG 2025-05 reject novelty 4.0 of 10

    The paper claims to prove an upper bound and diminishing returns in model merging, but the proofs are not sound and the heavy-tailed claim is contradicted by its own equations.

Pith tools