Pith. sign in

REVIEW 11 cited by

MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.12851 v1 pith:PF25UWJ5 submitted 2024-02-20 cs.CL

Tongxu Luo , Jiahe Lei , Fangyu Lei , Weihao Liu , Shizhu He , Jun Zhao , Kang Liu This is my paper
classification cs.CL
keywords loraexpertsfine-tuningmodelsmoeloraparameterspeftreasoning
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Fine-tuning is often necessary to enhance the adaptability of Large Language Models (LLM) to downstream tasks. Nonetheless, the process of updating billions of parameters demands significant computational resources and training time, which poses a substantial obstacle to the widespread application of large-scale models in various scenarios. To address this issue, Parameter-Efficient Fine-Tuning (PEFT) has emerged as a prominent paradigm in recent research. However, current PEFT approaches that employ a limited set of global parameters (such as LoRA, which adds low-rank approximation matrices to all weights) face challenges in flexibly combining different computational modules in downstream tasks. In this work, we introduce a novel PEFT method: MoELoRA. We consider LoRA as Mixture of Experts (MoE), and to mitigate the random routing phenomenon observed in MoE, we propose the utilization of contrastive learning to encourage experts to learn distinct features. We conducted experiments on 11 tasks in math reasoning and common-sense reasoning benchmarks. With the same number of parameters, our approach outperforms LoRA significantly. In math reasoning, MoELoRA achieved an average performance that was 4.2% higher than LoRA, and demonstrated competitive performance compared to the 175B GPT-3.5 on several benchmarks.

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Six-Dimensional Taxonomy of Post-Training Adaptation Techniques with Applications in AI Governance

    cs.LG 2026-08 conditional novelty 6.0 of 10

    A new taxonomy characterizes 48 post-training AI adaptation techniques on six axes and maps them to regulatory documentation requirements.

  2. MoE$^2$-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Routing LoRA adapters with the frozen base router's logits plus a shared cross-layer adapter pool gives the best PEFT accuracy and retention on three MoE backbones.

  3. RoCo-ACE: Rollout-Conditioned Online Distillation for Retention-Aware Knowledge Injection

    cs.AI 2026-06 conditional novelty 6.0 of 10

    A rollout-conditioned contrastive distillation loss plus sparse anchored cross-entropy improves injected-knowledge accuracy in MLLMs while keeping retention close to the base model.

  4. CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging

    cs.CL 2026-02 conditional novelty 6.0 of 10

    CoMoL represents every LoRA expert as a shared-basis core matrix and merges token-selected experts in that core space, reaching standard LoRA parameter counts while outperforming MoE-LoRA baselines on math and code.

  5. Reasoning and Tool-use Compete in Agentic RL:From Quantifying Interference to Disentangled Tuning

    cs.AI 2026-02 conditional novelty 6.0 of 10

    Jointly optimizing reasoning and tool use in agentic RL interferes with both; separating the two into disjoint LoRA adapters (DART) improves retrieval-augmented QA.

  6. Onsager Principle-Based Domain Embedding for Thermodynamically Consistent Cahn-Hilliard Model in Arbitrary Domain

    math.NA 2025-08 unverdicted novelty 5.0 of 10

    As submitted, the paper is unverifiable: the abstract claims a thermodynamically consistent Cahn-Hilliard domain-embedding derivation, while the full text is a different paper on person re-identification.

  7. Sci-LoRA: Mixture of Scientific LoRAs for Cross-Domain Lay Paraphrasing

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Sci-LoRA dynamically mixes domain-specific LoRA adapters and achieves state-of-the-art lay paraphrasing across twelve domains without needing domain labels at inference.

  8. HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation

    cs.CV 2025-02 reject novelty 5.0 of 10

    HealthGPT unifies medical image comprehension and generation in a single autoregressive model using heterogeneous low-rank adaptation, reporting strong benchmark results.

  9. ScatterPrism: convergence for generative simulation and inverse problems in particle and nuclear physics

    cs.LG 2026-04 unverdicted novelty 4.0 of 10

    Standard Conditional Flow Matching loss is a misleading early plateau; physics-informed metrics keep improving, so ScatterPrism and multi-metric diagnostics are needed for kinematic fidelity.

  10. MoE-MLoRA for Multi-Domain CTR Prediction: Efficient Adaptation with Expert Specialization

    cs.IR 2025-06 conditional novelty 4.0 of 10

    A three-stage MoE variant of MLoRA improves multi-domain CTR prediction on sparse Taobao data but not on denser Movielens splits.

  11. Why Do More Experts Fail? A Theoretical Analysis of Model Merging

    cs.LG 2025-05 reject novelty 4.0 of 10

    The paper claims to prove an upper bound and diminishing returns in model merging, but the proofs are not sound and the heavy-tailed claim is contradicted by its own equations.

Pith tools