Pith. sign in

REVIEW 4 major objections 5 minor 67 references

This paper claims that fine-tuning MoE models with LoRA adapters works best when the adapter routing reuses the pretrained router's activations and draws from one global pool of LoRA experts shared across all layers, yielding state-of-the-a

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 06:10 UTC pith:YYXZDDPN

load-bearing objection Novel PEFT design for MoE models with a real confound: the SOTA comparison matches trainable parameters but not per-token active LoRA capacity, so treat accuracy claims as provisional. the 4 major comments →

arxiv 2607.21978 v1 pith:YYXZDDPN submitted 2026-07-24 cs.CL

MoE²-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation

Qingyu Yang , Haonan He , Minglei Li , Jingqi Ye , Tao Chen , Lei Bai , Peng Ye This is my paper
classification cs.CL
keywords MoE2-LoRAmixture-of-expertslow-rank adaptationparameter-efficient fine-tuningrouting-conditioned projectionglobal expert poolMoE-style LoRAgeneral capability retention
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Existing PEFT methods for MoE models either apply uniform LoRA to all experts or freeze a static subset of experts picked offline, while MoE-style LoRA methods train independent routers that ignore the pretrained router. The paper's thesis is that the two MoE mechanisms should be fused: adapter experts should be selected using the base router's own activations, and those adapters should live in one shared pool rather than per-layer pools. It introduces MoE2-LoRA, combining a Routing-Conditioned Projection module with a global shared LoRA expert pool, and reports state-of-the-art in-domain accuracy across several MoE backbones while retaining stronger general capabilities than baselines. The paper also documents an emergent layer-wise affinity in the shared pool, which it interprets as evidence that cross-layer sharing is structured and beneficial.

Core claim

On its own terms, MoE2-LoRA's core claim is that deeply coupling the two MoE mechanisms — the pretrained model's routing and the LoRA adaptation's routing — is the missing step in MoE-style PEFT. The Routing-Conditioned Projection (RCP) fuses the base router's logits with a low-rank hidden-state correction and learns a per-layer projection that produces LoRA routing probabilities, giving adapter selection a pretrained prior without freezing it to the base routing. A second ingredient, a single global LoRA expert pool shared across all layers, removes per-layer capacity choices and lets experts specialize by depth while being reused across nearby layers. The paper presents experiments on four

What carries the argument

The Routing-Conditioned Projection (RCP) is the central mechanism: it computes LoRA expert selection as p_L = softmax(W_l · [z; h_c]), where z are the frozen base router logits, h_c is a low-rank projection of the hidden state, and W_l is a per-layer learnable matrix. The global shared LoRA expert pool is the second mechanism: one set of LoRA experts, reused by all layers, with per-layer W_l deciding which experts fire. Together they let adapter routing inherit the pretrained router's structure while still adapting per task, and let experts be shared across layers without per-layer capacity hyperparameters.

Load-bearing premise

The load-bearing premise is that the reported scores from a single training run per method and backbone are representative enough to support state-of-the-art claims, even though no training seeds or variance are reported and two baselines were adjusted from their original recipes.

What would settle it

Run each method (MoE2-LoRA, DAS-LoRA, MoELoRA, MoLA) on Qwen3-30B-A3B math and code with five training seeds each, identical parameter budgets, and each baseline's original configuration; if the averaged in-domain gap between MoE2-LoRA and the best baseline shrinks below roughly one point — within typical seed noise on these benchmarks — the state-of-the-art claim is not supported.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • The pretrained router of an MoE model contains transferable routing knowledge; PEFT methods should condition adapter routing on it rather than learning routing from scratch.
  • Sharing LoRA experts across layers is not just a parameter-saving trick; the paper's analysis says it yields depth-specialized experts and balanced utilization, meaning static per-layer allocation schedules are suboptimal.
  • If the empirical results are correct, MoE2-LoRA is the strongest PEFT method on these benchmarks, with up to +2.56 in-domain average over the best baseline.
  • The method's efficiency profile, with training time and memory within roughly 10–20 percent of baselines, makes it practical at the tested scales.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A multi-seed replication on the largest backbone would settle whether the margins over DAS-LoRA (0.69 average in-domain, 0.21 on HumanEval) are real; the paper's single-run protocol cannot distinguish signal from seed noise.
  • The F-statistic comparing LoRA routing to base routing should be benchmarked against a passive oracle that perfectly copies base top-1 routing; without that ceiling, a high F value mainly shows the RCP coupling is doing what it was designed to do.
  • The learned non-uniform per-layer expert count suggests a direct experiment: give a static allocation method the same per-layer counts and see if dynamic routing still matters, isolating the value of RCP from the value of the shared pool.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MoE2-LoRA, a PEFT method for MoE LLMs. Its two components are (i) Routing-Conditioned Projection (RCP; Eq. 1), which computes LoRA routing p_L = softmax(W_l [z; h_c]) from base router logits z and a low-rank hidden-state projection h_c, and (ii) a single global LoRA expert pool shared across all layers, with per-layer projection matrices selecting top-K_L experts from this pool (Eq. 2). The method is evaluated on OLMoE-1B-7B, DeepSeek-V2-Lite, Qwen3-30B-A3B, and Qwen3.5-35B-A3B on math, code, general-retention, and medical VQA benchmarks. The central claim is that MoE2-LoRA achieves state-of-the-art downstream accuracy among PEFT methods while retaining stronger general capabilities, with gains of up to +2.56 in-domain average points over the strongest PEFT baseline (Table 1). Ablations on DeepSeek-V2-Lite (§5.1, Table 4) and analyses of routing alignment and global-pool depth structure (§5.2, §5.3) are used to attribute the gains to RCP and cross-layer sharing.

Significance. The core idea is attractive and well-motivated: reusing the pretrained MoE router's logits to inform adapter routing, and sharing a global LoRA expert pool across layers, are natural ways to couple task-specific adaptation with the base model's expert specialization. The paper has genuine strengths: a shared training/evaluation protocol is specified in Appendix A.1, hyperparameters and budgets are reported in Appendix A.2, FFT is honestly excluded from 'best PEFT' bolding, and the main analyses use external benchmarks rather than self-defined metrics. The depth-affinity and effective-expert-count analyses are informative, and the RCP ablation (31.56 to 32.23) is a clean step. However, the state-of-the-art claim rests on a comparison that matches trainable parameter counts but does not match per-token active LoRA capacity, and the component ablation is confounded by simultaneous changes to top-K and rank. The single-seed evaluation and modified baseline configurations further weaken the quantitative claims. If the compute-matched experiments confirm the reported margins, the contribution would be solid; as it stands, the central empirical claim needs substantial additional support.

major comments (4)
  1. [§4.1, Appendix A.2] The 'fair comparison' matches trainable parameters but not per-token active LoRA capacity. On OLMoE, MoELoRA uses top-2/rank-8 while MoE2-LoRA uses top-4/rank-16, a 4× larger active LoRA parameter count per token; DeepSeek is 4× (top-4/rank-6 vs top-6/rank-16); Qwen3 is 8× (top-4/rank-6 vs top-6/rank-32). Because the Limitations correctly state that dynamic adapters cannot be merged, this extra capacity is a real inference-time cost, not just a training anomaly. The SOTA gains (up to +2.56 in Table 1) and the global-pool benefit could therefore be explained by larger per-token LoRA compute rather than by the routing design. Please add a compute-matched baseline (e.g., MoELoRA with matched top-K × rank) or report per-token active FLOPs and show the gains persist.
  2. [§5.1, Table 4] The ablation chain does not isolate the global pool. The 'per-layer hidden-state' baseline uses the MoELoRA budget (top-4/rank-6 per Table 5/A.2), while '+ Global Pool (MoE2-LoRA)' uses the full method's top-6/rank-16 configuration. Thus the 32.23→35.32 improvement changes routing and sharing simultaneously with per-token capacity. The claimed complementarity of RCP and global sharing requires a global-pool variant with the same top-K and rank as the per-layer RCP row, or a per-layer row with the larger capacity.
  3. [Appendix A.3, Table 1] Each adapter is trained once; no seeds, variance, or significance statistics are reported. Several headline margins are small: on Qwen3, the in-domain average is 81.57 vs 80.88 for DAS-LoRA, and HumanEval is 82.32 vs 82.11; on OLMoE the +1.70 margin over DAS-LoRA is larger but still a single run. Appendix A.2 also modifies DAS-LoRA (no router or dense-backbone tuning) and rescales MoLA's rank, so the baselines may not be at their native optima. Without multiple seeds or error bars, the 'state-of-the-art' claim is not statistically supported.
  4. [§4.3, Table 2] The multimodal experiment is a one-run comparison on Qwen3.5-35B-A3B and does not report the MoE2-LoRA configuration used, nor any variance. The in-domain margins are modest (e.g., 74.20 vs 73.53 for DAS-LoRA). This evidence is consistent with the main claims but is not strong enough to extend the SOTA conclusion to multimodal settings.
minor comments (5)
  1. [§3.2, Eq. (1)] Notation is inconsistent: the heading uses 'W l' while the text uses W_l; d_b is introduced without a definition before Eq. (1); specify the dimension of [z; h_c] as N_E + d_b.
  2. [Appendix A.2] The claim 'fair comparison' would benefit from a sentence in the main text (not only the appendix) stating that DAS-LoRA is modified from its original setup and MoLA's rank is rescaled to match budgets; as written, the main text says only that budgets vary by ~10%.
  3. [Table 1] The method 'PERFT-E' is labeled 'per-expert LoRA (Liu et al., 2026)' in the table but is called PERFT-E in the text; use consistent names.
  4. [§5.3.2, Fig. 4] The caption reports 'Pearson r=0.85' without a p-value or confidence interval; this is a descriptive statistic on the same probe set and should be labeled as such.
  5. [Abstract] "the first attempt to fine-tune MoE models with MoE-style low-rank adaptation" is too strong given MoELoRA and MoLA are discussed as prior work; rephrase as 'a new framework' or 'the first to deeply bind the two MoE mechanisms' if that is the intended novelty.

Circularity Check

1 steps flagged

No significant circularity: central SOTA claims rest on external benchmarks; only a minor self-definitional F-statistic diagnostic is built into the RCP definition.

specific steps
  1. self definitional [Section 5.2 (Eq. 1; Appendix B.2)]
    "To verify that RCP’s per-layer projection Wℓ actually inherits information from the base MoE router rather than learning an independent routing function, we measure the F-statistic of structural alignment between LoRA routing and base routing... We treat F as a structural-fit diagnostic for the RCP projection Wℓ rather than a quality metric; passive coupling schemes (e.g., per-expert LoRA) attain trivially high F by construction."

    Eq. (1) defines pL = softmax(Wl·[z; hc]) with z = G(h), so the LoRA routing distribution is literally a learned function of the base router logits. The F-statistic partitions tokens by the base router's top-1 expert and asks whether pL varies with that partition; since z is an input to pL, high alignment is partly guaranteed by the architecture itself rather than being an independent confirmation that router information is 'inherited.' The paper's own appendix concedes that passive coupling attains trivially high F by construction. This diagnostic is a mechanistic sanity check, not the source of the SOTA claim; Table 1 scores come from external benchmarks, so the central result is not circular.

full rationale

The paper's central claims—state-of-the-art downstream accuracy and general-capability retention—are empirical measurements on external benchmarks (GSM8K, MATH-500, MBPP, HumanEval, MMLU, etc.) under a shared protocol, not derivations from the method's equations. The method definition (Eq. 1 and Eq. 2) does not encode the benchmark outcomes, and the comparisons are not fitted to the test sets. The only self-referential element is the routing-alignment F-statistic in Section 5.2: because pL is constructed from base router logits z, a high F-statistic partly reflects the design choice rather than an independent discovery. The paper explicitly labels F as a structural-fit diagnostic and not a quality metric, and this diagnostic is not load-bearing for the main accuracy claims. Single-run training variance (Appendix A.3) and the larger per-token active LoRA capacity of MoE2-LoRA (Appendix A.4) are genuine threats to the fairness and strength of the SOTA comparison, but they are empirical/experimental-design concerns, not circularity: the reported scores are not equivalent to the method's inputs by construction. Therefore the overall circularity score is low.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The paper makes no theoretical derivation; all central claims are empirical, so the ledger records the tunable hyperparameters the method depends on and the background assumptions that make the experiments interpretable. No new entities are postulated — the global pool and RCP are architectural components whose evidence is the benchmark evaluation itself.

free parameters (5)
  • Global pool size N_L = 128 (OLMoE), 256 (DeepSeek-V2-Lite), 256 (Qwen3)
    Table 5; tuned on validation, controls trainable budget; Appendix B.1 shows the depth-diagonal persists across N_L ∈ {128,256,512}, but main results use one value per backbone.
  • LoRA rank r = 16 (OLMoE/DeepSeek), 32 (Qwen3)
    Table 5; per-expert adapter rank, chosen to match ~10% budget parity with baselines; MoLA's rank is rescaled 16→22 to match budget.
  • Activated LoRA experts per token top-K_L = 4 (OLMoE), 6 (DeepSeek), 6 (Qwen3)
    Table 5; per-token activated LoRA experts; affects compute (Table 6) and the capacity claims.
  • Hidden-state bottleneck d_b = 16 (OLMoE), 64 (DeepSeek), 32 (Qwen3)
    Table 5; dimension of the auxiliary channel h_c in RCP; its removal lowers Ablation Avg on DeepSeek-V2-Lite (Table 4).
  • Per-layer projection W_l = dimensions N_L × (N_E + d_b), learned
    The RCP routing matrix; the paper's own Limitations note projection parameter cost grows with pool size.
axioms (4)
  • domain assumption Base MoE router logits carry task-relevant, transferable structure that improves adapter selection.
    Central to RCP (§3.2): the z channel is concatenated into the LoRA router input. The F-statistic (§B.2) shows W_l amplifies z, but the assumption that this helps task performance rests on the +0.67 ablation gain in Table 4.
  • domain assumption Cross-layer representational overlap licenses sharing LoRA experts across layers.
    §5.3.3 measures off-diagonal CKA ≈ 0.43 and infers 'a plausible basis for sharing LoRA experts across nearby or related layers'; the causal step from measured similarity to adaptation benefit is assumed, though supported by the +3.09 global-pool ablation gain.
  • domain assumption Single-run fine-tune scores are representative of each method.
    Appendix A.3: 'Each adapter is trained once per (method, backbone) configuration... Multi-run training-time statistics are not reported due to compute cost.' All headline comparisons in Tables 1–3 inherit this assumption.
  • standard math Standard LoRA update form ΔW = (α/r)BA captures the needed task-specific update.
    Used throughout §3.4; standard PEFT background from Hu et al. (2022).

pith-pipeline@v1.3.0-alltime-deepseek · 15669 in / 23108 out tokens · 195369 ms · 2026-08-01T06:10:13.195753+00:00 · methodology

0 comments
read the original abstract

Mixture-of-Experts (MoE) architectures have been widely adopted in large language models, yet parameter-efficient fine-tuning (PEFT) for MoE models remains underexplored. Existing PEFT methods for MoE either ignore router priors with uniform adapters, reducing efficiency and risking forgetting, or rely on static expert selection, limiting per-token capacity and cross-expert feature learning. In this paper, we make the first attempt to fine-tune MoE models with MoE-style low-rank adaptation: our method, entitled MoE$^2$-LoRA, deeply couples the pretrained expert specialization with task-specific adaptivity via a dual-channel Routing-Conditioned Projection (RCP) module, which reuses base router activations to inform LoRA routing. We further introduce a single global LoRA expert pool shared across all layers, enabling model-wide adaptation with emergent layer-wise affinities and balanced expert utilization. MoE$^2$-LoRA simultaneously benefits from the advantages of prior reuse, dynamic adapter routing, and model-wide knowledge sharing. Evaluated on multiple MoE backbones with varying scales and expert granularities, MoE$^2$-LoRA consistently achieves state-of-the-art downstream accuracy while retaining stronger general capabilities.

Figures

Figures reproduced from arXiv: 2607.21978 by Haonan He, Jingqi Ye, Lei Bai, Minglei Li, Peng Ye, Qingyu Yang, Tao Chen.

Figure 1
Figure 1. Figure 1: (I) Comparison of PEFT-on-MoE methods: (a) Per-expert LoRA, which equip each MoE expert with a [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the MoE2 -LoRA forward pass on a single MoE layer. 3.2 Routing-Conditioned Projection (Wl) MoE2 -LoRA introduces an additional LoRA ex￾pert pool {ELoRA j } NL j=1 and learns per-token rout￾ing pL ∈ R NL over this pool. The key design choice is the routing input: rather than learning routing from raw hidden state h or hardwiring it to base routing, MoE2 -LoRA uses a two-channel input: (1) Primar… view at source ↗
Figure 3
Figure 3. Figure 3: Per-layer routing-alignment F-statistic on [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Per-layer effective expert count on math (blue) [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Expert-affinity matrices on GSM8K for MoE2 -LoRA with different global pool sizes NL ∈ {128, 256, 512} on DeepSeek-V2-Lite. Rows are ex￾perts sorted by preferred depth; columns are MoE layers. A consistent depth-organized diagonal emerges across all pool sizes. Across pool sizes [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Expert-affinity matrices for the same trained [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

67 extracted references · 1 canonical work pages

  1. [2]

    Jun Bai, Minghao Tong, Yang Liu, Zixia Jia, and Zilong Zheng. 2025. Understanding and leveraging the expert specialization of context faithfulness in mixture-of-experts llms. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing

    2025

  2. [9]

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, Rongrong Ji, Caifeng Shan, and Ran He. 2025. https://arxiv.org/abs/2306.13394 Mme: A comprehensive evaluation benchmark for multimodal large language models . Preprint, arXiv:2306.13394

    Pith/arXiv arXiv 2025

  3. [10]

    Chongyang Gao, Kezhen Chen, Jinmeng Rao, Ruibo Liu, Baochen Sun, Yawen Zhang, Daiyi Peng, Xiaoyuan Guo, and VS Subrahmanian. 2025. Mola: Moe lora with layer-wise expert allocation. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 5097--5112

    2025

  4. [11]

    Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies . Transactions of the Association for Computational Linguistics (TACL)

    2021

  5. [13]

    Haonan He, Peng Ye, Yuchen Ren, Yuan Yuan, Luyang Zhou, Shucun Ju, and Lei Chen. 2025. https://arxiv.org/abs/2502.12171 Gora: Gradient-driven adaptive low rank adaptation . Preprint, arXiv:2502.12171

    arXiv 2025

  6. [15]

    Dan Hendrycks and 1 others. 2021. Measuring massive multitask language understanding. In ICLR

    2021

  7. [16]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR)

    2022

  8. [17]

    Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. 1991. Adaptive mixtures of local experts. Neural computation, 3(1):79--87

    1991

  9. [18]

    Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. 2018. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):180251

    2018

  10. [19]

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2023. https://arxiv.org/abs/2306.00890 Llava-med: Training a large language-and-vision assistant for biomedicine in one day . Preprint, arXiv:2306.00890

    Pith/arXiv arXiv 2023

  11. [22]

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. 2021. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th international symposium on biomedical imaging (ISBI), pages 1650--1654. IEEE

    2021

  12. [24]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2024 b . https://arxiv.org/abs/2307.06281 Mmbench: Is your multi-modal model an all-around player? Preprint, arXiv:2307.06281

    Pith/arXiv arXiv 2024

  13. [25]

    Tongxu Luo, Jiahe Lei, Fangyu Lei, Weihao Liu, Shizhu He, Jun Zhao, and Kang Liu. 2024. https://arxiv.org/abs/2402.12851 Moelora: Contrastive learning guided mixture of experts on parameter-efficient fine-tuning for large language models . Preprint, arXiv:2402.12851

    Pith/arXiv arXiv 2024

  14. [28]

    Qwen Team . 2026. https://qwen.ai/blog?id=qwen3.5 Qwen3.5 : Towards native multimodal agents

    2026

  15. [29]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99--106

    2021

  16. [30]

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149--4158

    2019

  17. [31]

    Yiru Tang, Kun Zhou, Xin Zhao, Jing Sha, Zhichao Sheng, and Shijin Wang. 2026. https://openreview.net/forum?id=zBgjWTWgCh Exploring expert concentration for parameter-efficient fine-tuning of mixture-of-expert LLM s

    2026

  18. [34]

    xAI . 2024. Grok-1.5 Vision Preview . https://x.ai/blog/grok-1.5v. RealWorldQA dataset, available at https://huggingface.co/datasets/xai-org/RealworldQA

    2024

  19. [36]

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2024. Metamath: Bootstrap your own mathematical questions for large language models. In International Conference on Learning Representations, volume 2024, pages 45040--45061

    2024

  20. [37]

    International Conference on Learning Representations (ICLR) , year=

    LoRA: Low-Rank Adaptation of Large Language Models , author=. International Conference on Learning Representations (ICLR) , year=

  21. [38]

    arXiv preprint arXiv:2409.02060 , year=

    OLMoE: Open Mixture-of-Experts Language Models , author=. arXiv preprint arXiv:2409.02060 , year=

    Pith/arXiv arXiv

  22. [40]

    2024 , eprint=

    MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Models , author=. 2024 , eprint=

    2024

  23. [41]

    arXiv preprint arXiv:2407.01906 , year=

    Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models , author=. arXiv preprint arXiv:2407.01906 , year=

    Pith/arXiv arXiv

  24. [42]

    Exploring Expert Concentration for Parameter-efficient Fine-tuning of Mixture-of-Expert

    Yiru Tang and Kun Zhou and Xin Zhao and Jing Sha and Zhichao Sheng and Shijin Wang , year=. Exploring Expert Concentration for Parameter-efficient Fine-tuning of Mixture-of-Expert

  25. [43]

    Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules

    Liu, Yilun and Ma, Yunpu and Lu, Yuetian and Chen, Shuo and Ding, Zifeng and Tresp, Volker. Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules. Findings of the A ssociation for C omputational L inguistics: EACL 2026. 2026. doi:10.18653/v1/2026.findings-eacl.232

    work page doi:10.18653/v1/2026.findings-eacl.232 2026

  26. [44]

    International Conference on Learning Representations (ICLR) , year=

    AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning , author=. International Conference on Learning Representations (ICLR) , year=

  27. [45]

    International Conference on Machine Learning (ICML) , year=

    DoRA: Weight-Decomposed Low-Rank Adaptation , author=. International Conference on Machine Learning (ICML) , year=

  28. [46]

    International Conference on Learning Representations (ICLR) , year=

    Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer , author=. International Conference on Learning Representations (ICLR) , year=

  29. [47]

    arXiv preprint arXiv:2006.16668 , year=

    GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding , author=. arXiv preprint arXiv:2006.16668 , year=

    Pith/arXiv arXiv 2006

  30. [48]

    Journal of Machine Learning Research , year=

    Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity , author=. Journal of Machine Learning Research , year=

  31. [49]

    arXiv preprint arXiv:2202.08906 , year=

    ST-MoE: Designing Stable and Transferable Sparse Expert Models , author=. arXiv preprint arXiv:2202.08906 , year=

    Pith/arXiv arXiv

  32. [50]

    arXiv preprint arXiv:2401.04088 , year=

    Mixtral of Experts , author=. arXiv preprint arXiv:2401.04088 , year=

    Pith/arXiv arXiv

  33. [52]

    arXiv preprint arXiv:2305.20050 , year=

    Let's Verify Step by Step , author=. arXiv preprint arXiv:2305.20050 , year=

    Pith/arXiv arXiv

  34. [55]

    ICLR , year=

    Measuring Massive Multitask Language Understanding , author=. ICLR , year=

  35. [56]

    ACL , year=

    HellaSwag: Can a Machine Really Finish Your Sentence? , author=. ACL , year=

  36. [58]

    arXiv preprint arXiv:2601.04823 , year=

    DR-LoRA: Dynamic Rank LoRA for Mixture-of-Experts Adaptation , author=. arXiv preprint arXiv:2601.04823 , year=

    arXiv

  37. [59]

    arXiv preprint arXiv:2603.12645 , year=

    LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing , author=. arXiv preprint arXiv:2603.12645 , year=

    arXiv

  38. [60]

    Neural computation , volume=

    Adaptive mixtures of local experts , author=. Neural computation , volume=. 1991 , publisher=

    1991

  39. [61]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    Pith/arXiv arXiv

  40. [62]

    Findings of the Association for Computational Linguistics: NAACL 2025 , pages=

    MoLA: MoE LoRA with layer-wise expert allocation , author=. Findings of the Association for Computational Linguistics: NAACL 2025 , pages=

    2025

  41. [63]

    arXiv preprint arXiv:2405.04434 , year=

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model , author=. arXiv preprint arXiv:2405.04434 , year=

    Pith/arXiv arXiv

  42. [64]

    arXiv preprint arXiv:2405.09673 , year=

    Lora learns less and forgets less , author=. arXiv preprint arXiv:2405.09673 , year=

    Pith/arXiv arXiv

  43. [65]

    arXiv preprint arXiv:2402.12354 , year=

    Lora+: Efficient low rank adaptation of large models , author=. arXiv preprint arXiv:2402.12354 , year=

    Pith/arXiv arXiv

  44. [66]

    2025 , eprint=

    GoRA: Gradient-driven Adaptive Low Rank Adaptation , author=. 2025 , eprint=

    2025

  45. [67]

    arXiv preprint arXiv:2603.24044 , year=

    MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning , author=. arXiv preprint arXiv:2603.24044 , year=

    arXiv

  46. [68]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , year=

    Understanding and leveraging the expert specialization of context faithfulness in mixture-of-experts llms , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , year=

    2025

  47. [69]

    arXiv preprint arXiv:2312.09979 , volume=

    Loramoe: Revolutionizing mixture of experts for maintaining world knowledge in language model alignment , author=. arXiv preprint arXiv:2312.09979 , volume=

    Pith/arXiv arXiv

  48. [70]

    International Conference on Learning Representations , volume=

    Metamath: Bootstrap your own mathematical questions for large language models , author=. International Conference on Learning Representations , volume=

  49. [71]

    arXiv preprint arXiv:2312.02120 , year=

    Magicoder: Empowering code generation with oss-instruct , author=. arXiv preprint arXiv:2312.02120 , year=

    Pith/arXiv arXiv

  50. [72]

    arXiv preprint arXiv:2110.14168 , year=

    Training verifiers to solve math word problems , author=. arXiv preprint arXiv:2110.14168 , year=

    Pith/arXiv arXiv

  51. [73]

    arXiv preprint arXiv:2108.07732 , year=

    Program synthesis with large language models , author=. arXiv preprint arXiv:2108.07732 , year=

    Pith/arXiv arXiv

  52. [74]

    arXiv preprint arXiv:2107.03374 , year=

    Evaluating large language models trained on code , author=. arXiv preprint arXiv:2107.03374 , year=

    Pith/arXiv arXiv

  53. [75]

    arXiv preprint arXiv:2009.03300 , year=

    Measuring massive multitask language understanding , author=. arXiv preprint arXiv:2009.03300 , year=

    Pith/arXiv arXiv 2009

  54. [76]

    Communications of the ACM , volume=

    Winogrande: An adversarial winograd schema challenge at scale , author=. Communications of the ACM , volume=. 2021 , publisher=

    2021

  55. [77]

    arXiv preprint arXiv:1803.05457 , year=

    Think you have solved question answering? try arc, the ai2 reasoning challenge , author=. arXiv preprint arXiv:1803.05457 , year=

    Pith/arXiv arXiv

  56. [78]

    Transactions of the Association for Computational Linguistics , volume=

    Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies , author=. Transactions of the Association for Computational Linguistics , volume=. 2021 , publisher=

    2021

  57. [79]

    Commonsenseqa: A question answering challenge targeting commonsense knowledge , author=. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages=

    2019

  58. [80]

    Proceedings of the 57th annual meeting of the association for computational linguistics , pages=

    Hellaswag: Can a machine really finish your sentence? , author=. Proceedings of the 57th annual meeting of the association for computational linguistics , pages=

  59. [81]

    Advances in neural information processing systems , volume=

    C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models , author=. Advances in neural information processing systems , volume=

  60. [82]

    Scientific data , volume=

    A dataset of clinically generated visual questions and answers about radiology images , author=. Scientific data , volume=. 2018 , publisher=

    2018

  61. [83]

    2021 IEEE 18th international symposium on biomedical imaging (ISBI) , pages=

    Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering , author=. 2021 IEEE 18th international symposium on biomedical imaging (ISBI) , pages=. 2021 , organization=

    2021

  62. [84]

    arXiv preprint arXiv:2003.10286 , year=

    Pathvqa: 30000+ questions for medical visual question answering , author=. arXiv preprint arXiv:2003.10286 , year=

    Pith/arXiv arXiv 2003

  63. [85]

    2024 , eprint=

    MMBench: Is Your Multi-modal Model an All-around Player? , author=. 2024 , eprint=

    2024

  64. [86]

    2025 , eprint=

    MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models , author=. 2025 , eprint=

    2025

  65. [87]

    2023 , eprint=

    LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day , author=. 2023 , eprint=

    2023

  66. [88]

    2024 , howpublished =

    2024

  67. [89]

    Geva, Mor and Khashabi, Daniel and Segal, Elad and Khot, Tushar and Roth, Dan and Berant, Jonathan , journal =