REVIEW 18 cited by
Mixture of LoRA Experts
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
LoRA has gained widespread acceptance in the fine-tuning of large pre-trained models to cater to a diverse array of downstream tasks, showcasing notable effectiveness and efficiency, thereby solidifying its position as one of the most prevalent fine-tuning techniques. Due to the modular nature of LoRA's plug-and-play plugins, researchers have delved into the amalgamation of multiple LoRAs to empower models to excel across various downstream tasks. Nonetheless, extant approaches for LoRA fusion grapple with inherent challenges. Direct arithmetic merging may result in the loss of the original pre-trained model's generative capabilities or the distinct identity of LoRAs, thereby yielding suboptimal outcomes. On the other hand, Reference tuning-based fusion exhibits limitations concerning the requisite flexibility for the effective combination of multiple LoRAs. In response to these challenges, this paper introduces the Mixture of LoRA Experts (MoLE) approach, which harnesses hierarchical control and unfettered branch selection. The MoLE approach not only achieves superior LoRA fusion performance in comparison to direct arithmetic merging but also retains the crucial flexibility for combining LoRAs effectively. Extensive experimental evaluations conducted in both the Natural Language Processing (NLP) and Vision & Language (V&L) domains substantiate the efficacy of MoLE.
Forward citations
Cited by 18 Pith papers
-
SAM+D: Parameter-Efficient Dimensional Lifting of SAM-Family Models via Depth-Routed LoRA and Depth Shifting
Depth-routed LoRA and a depth-shift module lift frozen SAM and SAM2 to 3D and 3D+T segmentation using less than ~3.7% trainable parameters.
-
DA-MergeLoRA: Hypernetwork-Based LoRA Merging for Few-Shot Test-Time Domain Adaptation
A hypernetwork generates per-column merging weights to combine source LoRA modules on CLIP, achieving state-of-the-art few-shot test-time domain adaptation.
-
Transferable Low-Rank Convolutional Bases for Onboarding Unseen Medical Imaging Modalities
A convolutional low-rank basis learned on CT and MRI transfers to chest X-ray, enabling onboarding of an unseen modality at 0.78% of full-fine-tuning parameters with exactly zero forgetting.
-
Parametric Memory Decoding for Zero-Shot Routing in LoRA-Based External Parametric Memory
PMDRouter selects LoRAs zero-shot by decoding scale-normalized linear response energy from one adapter-free backbone prefill, and leads most internal-signal baselines on a new multi-granularity EPM bench.
-
Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs
Hard top-1 token routing over frozen, independently RLVF-trained LoRA experts matches or beats soft-routing mixtures with about 8–10x fewer trainable parameters.
-
Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories
Sleep-time Knowledge Seeding plus Dreaming lets LLMs expand capacity, distill fragile in-context memories into stable parameters, and self-improve without human labels.
-
CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging
CoMoL represents every LoRA expert as a shared-basis core matrix and merges token-selected experts in that core space, reaching standard LoRA parameter counts while outperforming MoE-LoRA baselines on math and code.
-
R^2MoE: Redundancy-Removal Mixture of Experts for Lifelong Concept Learning
R2MoE adds per-concept LoRA experts with routing distillation and expert pruning, reporting 0.19% forgetting and 15.2M added parameters on CustomConcept101.
-
Cartridges: Lightweight and general-purpose long context representations via self-study
A per-corpus trained KV cache, called a Cartridge, matches full-context in-context learning quality on long-document benchmarks while using up to 38.6x less serving memory.
-
MultLFG: Training-free Multi-LoRA composition using Frequency-domain Guidance
MultLFG merges multiple LoRA adapters by adaptively weighting them in wavelet frequency subbands per denoising timestep, improving multi-concept composition on the ComposLoRA benchmark compared to prior training-free methods.
-
Guiding the Experts: Semantic Priors for Efficient and Focused MoE Routing
Guiding Soft MoE dispatch weights with foreground segmentation masks plus a zero-initialized LayerScale improves ImageNet-1K top-1 by 0.6% and ImageNet-100 by 1.4% over a reproduced baseline.
-
Multi-Dimensional Quality Assessment for AI-Generated Human-Centric Videos: Dataset and Model
A new large benchmark for AI-generated human-centric video quality with pairwise preferences, plus a Mixture-of-Experts MLLM that outperforms prior methods on rating, comparison, and Q&A.
-
EmoStyle: Affective Conditioning of Style-Specialist Experts for Emotional Image Generation
EmoStyle injects LLM-inferred valence-arousal and emotion labels into Z-Image via AdaLN-style residual modulation over style-bucket LoRA experts, plus VLM candidate ranking, and ranked first on AffectiveArt Track 1.
-
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.
-
Joint Information Extraction Across Classical and Modern Chinese with Tea-MOELoRA
Tea-MOELORA uses separate task and era gates over LoRA experts to jointly train relation and event extraction across classical and modern Chinese, improving F1 over joint LoRA and existing LoRA-MoE baselines on most datasets.
-
ICM-Fusion: In-Context Meta-Optimized LoRA Fusion for Multi-Task Adaptation
ICM-Fusion uses a conditional VAE plus task-vector guidance to fuse multiple LoRA adapters into one model, reporting marginal average gains on vision and language benchmarks and larger gains in a few-shot long-tail setup.
-
PFMBench: Protein Foundation Model Benchmark
A comprehensive benchmark of 17 protein foundation models across 38 tasks yields task correlations, a streamlined protocol, and identifies ProTrek as the strongest general performer.
-
$\mu$-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Experts
Test-time Wanda pruning, reframed as a mixture of micro-experts, adapts the sparse weight mask to each prompt and improves perplexity and VQA accuracy over static pruning baselines.
Discussion (0). Continue with ORCID to comment.