REVIEW 8 cited by
Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Instruction Tuning has the potential to stimulate or enhance specific capabilities of large language models (LLMs). However, achieving the right balance of data is crucial to prevent catastrophic forgetting and interference between tasks. To address these limitations and enhance training flexibility, we propose the Mixture-of-LoRAs (MoA) architecture which is a novel and parameter-efficient tuning method designed for multi-task learning with LLMs. In this paper, we start by individually training multiple domain-specific LoRA modules using corresponding supervised corpus data. These LoRA modules can be aligned with the expert design principles observed in Mixture-of-Experts (MoE). Subsequently, we combine the multiple LoRAs using an explicit routing strategy and introduce domain labels to facilitate multi-task learning, which help prevent interference between tasks and ultimately enhances the performance of each individual task. Furthermore, each LoRA model can be iteratively adapted to a new domain, allowing for quick domain-specific adaptation. Experiments on diverse tasks demonstrate superior and robust performance, which can further promote the wide application of domain-specific LLMs.
Forward citations
Cited by 8 Pith papers
-
CL-Anomaly: Layer-Adaptive Mixture-of-Experts with Multimodal Large Language Model for Continual Learning in Anomaly Detection
A private-plus-shared LoRA MoE with layer-adaptive momentum transfer enables continual anomaly detection on MLLMs and beats prior continual-learning baselines across class, domain, and modality shifts.
-
Multi-Task GRPO: Reliable LLM Reasoning Across Tasks
MT-GRPO reweights tasks by reward and improvement and enforces those weights after zero-gradient filtering, improving worst-task accuracy by 6–28% over GRPO/DAPO baselines on 3- and 9-task setups.
-
Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation
PrunedLoRA tunes large-rank LoRA adapters and progressively structurally prunes A and B to a target low rank, empirically closing much of the LoRA-to-full-fine-tuning gap.
-
R^2MoE: Redundancy-Removal Mixture of Experts for Lifelong Concept Learning
R2MoE adds per-concept LoRA experts with routing distillation and expert pruning, reporting 0.19% forgetting and 15.2M added parameters on CustomConcept101.
-
LoRA-Gen: Specializing Large Language Model via Online LoRA Generation
LoRA-Gen generates task-specific LoRA weights with a cloud-side LLM and reparameterizes them into a smaller edge model, enabling training-free specialization with compressed context.
-
A Survey of LLM $\times$ DATA
A comprehensive survey of the bidirectional links between LLMs and data management, organized as DATA4LLM and LLM4DATA with a new 'IaaS' data-quality framework.
-
CoLA: Collaborative Low-Rank Adaptation
CoLA generalizes LoRA to multiple A and B matrices with a principal-component initialization and reports gains of roughly 2-4 accuracy points over PiSSA on low-sample fine-tuning benchmarks.
-
Survey of Specialized Large Language Model
A survey of 24 specialized LLMs (2022-2025) claims a shift from domain fine-tuning to native architectures, but the synthesis is undermined by citation errors and selection bias.
Discussion (0). Continue with ORCID to comment.