REVIEW 5 cited by
SiRA: Sparse Mixture of Low Rank Adaptation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Parameter Efficient Tuning has been an prominent approach to adapt the Large Language Model to downstream tasks. Most previous works considers adding the dense trainable parameters, where all parameters are used to adapt certain task. We found this less effective empirically using the example of LoRA that introducing more trainable parameters does not help. Motivated by this we investigate the importance of leveraging "sparse" computation and propose SiRA: sparse mixture of low rank adaption. SiRA leverages the Sparse Mixture of Expert(SMoE) to boost the performance of LoRA. Specifically it enforces the top $k$ experts routing with a capacity limit restricting the maximum number of tokens each expert can process. We propose a novel and simple expert dropout on top of gating network to reduce the over-fitting issue. Through extensive experiments, we verify SiRA performs better than LoRA and other mixture of expert approaches across different single tasks and multitask settings.
Forward citations
Cited by 5 Pith papers
-
Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs
Hard top-1 token routing over frozen, independently RLVF-trained LoRA experts matches or beats soft-routing mixtures with about 8–10x fewer trainable parameters.
-
CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging
CoMoL represents every LoRA expert as a shared-basis core matrix and merges token-selected experts in that core space, reaching standard LoRA parameter counts while outperforming MoE-LoRA baselines on math and code.
-
Reasoning and Tool-use Compete in Agentic RL:From Quantifying Interference to Disentangled Tuning
Jointly optimizing reasoning and tool use in agentic RL interferes with both; separating the two into disjoint LoRA adapters (DART) improves retrieval-augmented QA.
-
Regularizing Subspace Redundancy of Low-Rank Adaptation
ReSoRA adds a penalty that reduces redundancy among rank-1 subspaces of LoRA-style adapters, producing modest accuracy improvements on vision-language retrieval and visual classification.
-
Mixture of LoRA Experts with Multi-Modal and Multi-Granularity LLM Generative Error Correction for Accented Speech Recognition
A three-stage pipeline of mono-accent LoRA experts, hierarchical routing, and phoneme-plus-word LLM error correction cuts accented-English WER from 6.34% to 2.07% on a combined 9-accent test set.
Discussion (0). Sign in to comment.