REVIEW 7 cited by
LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Mixture-of-Experts (MoE) has gained increasing popularity as a promising framework for scaling up large language models (LLMs). However, training MoE from scratch in a large-scale setting still suffers from data-hungry and instability problems. Motivated by this limit, we investigate building MoE models from existing dense large language models. Specifically, based on the well-known LLaMA-2 7B model, we obtain an MoE model by: (1) Expert Construction, which partitions the parameters of original Feed-Forward Networks (FFNs) into multiple experts; (2) Continual Pre-training, which further trains the transformed MoE model and additional gate networks. In this paper, we comprehensively explore different methods for expert construction and various data sampling strategies for continual pre-training. After these stages, our LLaMA-MoE models could maintain language abilities and route the input tokens to specific experts with part of the parameters activated. Empirically, by training 200B tokens, LLaMA-MoE-3.5B models significantly outperform dense models that contain similar activation parameters. The source codes and models are available at https://github.com/pjlab-sys4nlp/llama-moe .
Forward citations
Cited by 7 Pith papers
-
Relax Within, Balance Across: Geometry-Guided Load Balancing for Vision-Language Mixture-of-Experts
The standard mixed image-text load-balancing loss lets modality errors cancel at one token mix; ReBA balances each modality separately and per image, reducing load across resolutions and tiles with comparable accuracy.
-
Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inference
Communication-aware expert placement plus device-level pruning yields 1.23–1.86× MoE inference throughput and better accuracy at equal speedup than load-balance or sequential baselines.
-
Cache Management for Mixture-of-Experts LLMs -- extended version
The paper formalizes expert caching in MoE LLMs as layered paging, proves near-tight competitive lower bounds, and shows a layer-aware LRU variant beats standard LRU on real traces.
-
What You Have is What You Track: Adaptive and Robust Multimodal Tracking
FlexTrack claims state-of-the-art multimodal tracking on complete and simulated missing-modality benchmarks, using heterogeneous mixture-of-experts fusion and a video-level masking training strategy.
-
Advancing Decoding Strategies: Enhancements in Locally Typical Sampling for LLMs
ASTS extends locally typical sampling with semantic scoring and dynamic thresholds, reporting improved perplexity, MAUVE, and diversity on story and summarization tasks.
-
Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding
CoSD fuses two LLMs at decoding time: it keeps the draft model's tokens unless a confidence rule or a decision tree over token probabilities says the assistant model's token is better.
-
SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decoupling
SkipGPT learns per-token, per-module routers that dynamically skip attention and MLP blocks, then uses LoRA to recover most of the original model's quality.
Discussion (0). Continue with ORCID to comment.