Pith. sign in

REVIEW 13 cited by

OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.01739 v2 pith:M4DLISHP submitted 2024-01-29 cs.CL cs.AIcs.DCcs.LG

Fuzhao Xue , Zian Zheng , Yao Fu , Jinjie Ni , Zangwei Zheng , Wangchunshu Zhou , Yang You This is my paper
classification cs.CLcs.AIcs.DCcs.LG
keywords llmsmodelsroutingearlyopenmoeanalysisdevelopmentfuture
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

To help the open-source community have a better understanding of Mixture-of-Experts (MoE) based large language models (LLMs), we train and release OpenMoE, a series of fully open-sourced and reproducible decoder-only MoE LLMs, ranging from 650M to 34B parameters and trained on up to over 1T tokens. Our investigation confirms that MoE-based LLMs can offer a more favorable cost-effectiveness trade-off than dense LLMs, highlighting the potential effectiveness for future LLM development. One more important contribution of this study is an in-depth analysis of the routing mechanisms within our OpenMoE models, leading to three significant findings: Context-Independent Specialization, Early Routing Learning, and Drop-towards-the-End. We discovered that routing decisions in MoE models are predominantly based on token IDs, with minimal context relevance. The token-to-expert assignments are determined early in the pre-training phase and remain largely unchanged. This imperfect routing can result in performance degradation, particularly in sequential tasks like multi-turn conversations, where tokens appearing later in a sequence are more likely to be dropped. Finally, we rethink our design based on the above-mentioned observations and analysis. To facilitate future MoE LLM development, we propose potential strategies for mitigating the issues we found and further improving off-the-shelf MoE LLM designs.

Discussion (0). Sign in to comment.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond Global Routing Aggregation: Phase-Aware Expert Merging for MoE Vision-Language Models

    cs.CV 2026-08 conditional novelty 7.0 of 10

    Phase-aware expert merging based on routing role profiles preserves more MoE-VLM accuracy than global routing aggregation at matched compression ratios.

  2. Modular Pretraining Enables Access Control

    cs.LG 2026-07 conditional novelty 7.0 of 10

    GRAM selectively trains auxiliary modules so that ablating one at inference removes a targeted capability while preserving the rest, closely tracking data-filtered models at 5x lower cost across 5 capability profiles.

  3. T^2MLR: Transformer with Temporal Middle-Layer Recurrence

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Routing recurrent state between middle layers of adjacent decoding steps improves Transformer reasoning more than full-layer recurrence, at under 8% inference overhead.

  4. Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook

    cs.CL 2026-03 unverdicted novelty 6.0 of 10

    DOVE measures LLM cultural value alignment via a rate-distortion value codebook and unbalanced optimal transport between human and model open-ended text distributions.

  5. Scaling depth capacity via zero/one-layer model expansion

    cs.LG 2025-11 conditional novelty 6.0 of 10

    Training GPT2 from a zero/one-layer model and expanding depth at 80% of the schedule reaches fixed-size loss with approximately 5x less compute.

  6. SHMoAReg: Spark Deformable Image Registration via Spatial Heterogeneous Mixture of Experts and Attention Heads

    cs.CV 2025-09 conditional novelty 6.0 of 10

    A deformable image registration network with mixture-of-attention heads in the encoder and per-voxel, per-direction mixture-of-experts convolutions in the decoder improves abdominal CT Dice from 60.58% to 65.58%.

  7. DriftMoE: A Mixture of Experts Approach to Handle Concept Drifts

    stat.ML 2025-07 conditional novelty 6.0 of 10

    An online mixture-of-experts model trained with a multi-hot correctness mask matches or approaches state-of-the-art adaptive ensembles on several concept drift benchmarks.

  8. Decoupled Relative Learning Rate Schedules

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Per-component relative learning-rate schedules speed up Transformer pretraining by up to 23% for MoE models, and the schedules tuned on a 34M model transfer to models 27x larger.

  9. MoGU V2: Toward a Higher Pareto Frontier Between Model Usability and Security

    cs.CL 2025-09 conditional novelty 5.0 of 10

    MoGUv2 embeds small routers in the deeper layers of LLMs to dynamically blend a helpful variant and a refusal variant, improving safety against jailbreak and fine-tuning attacks while preserving usability.

  10. Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights

    cs.LG 2025-06 conditional novelty 5.0 of 10

    At 56B total parameters, fine-grained MoE with smaller, more numerous experts beats standard Switch and Mixtral-style MoE on validation loss and average downstream accuracy at matched FLOPs.

  11. Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Experts

    cs.CV 2025-05 conditional novelty 5.0 of 10

    A sparse mixture-of-experts 3D multimodal LLM adaptively fuses RGB, RGBD, BEV, point cloud, and voxel tokens, achieving SOTA on several ScanNet-based 3D scene understanding benchmarks.

  12. The Future of Continual Learning in the Era of Foundation Models: Three Key Directions

    cs.LG 2025-06 conditional novelty 4.0 of 10

    Continual learning should pivot from weight-update-based methods to continual compositionality and orchestration of foundation models and agents.

  13. Scalable, Symbiotic, AI and Non-AI Agent Based Parallel Discrete Event Simulations

    cs.CL 2025-05 conditional novelty 4.0 of 10

    PDES orchestration of small language models with non-AI verifier agents raises accuracy on four toy tasks from about 23 percent to 68 percent, with the verifiers supplying most of the correctness.

Pith tools