REVIEW 7 cited by
Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
A pivotal advancement in the progress of large language models (LLMs) is the emergence of the Mixture-of-Experts (MoE) LLMs. Compared to traditional LLMs, MoE LLMs can achieve higher performance with fewer parameters, but it is still hard to deploy them due to their immense parameter sizes. Different from previous weight pruning methods that rely on specifically designed hardware, this paper mainly aims to enhance the deployment efficiency of MoE LLMs by introducing plug-and-play expert-level sparsification techniques. Specifically, we propose, for the first time to our best knowledge, post-training approaches for task-agnostic and task-specific expert pruning and skipping of MoE LLMs, tailored to improve deployment efficiency while maintaining model performance across a wide range of tasks. Extensive experiments show that our proposed methods can simultaneously reduce model sizes and increase the inference speed, while maintaining satisfactory performance. Data and code will be available at https://github.com/Lucky-Lance/Expert_Sparsity.
Forward citations
Cited by 7 Pith papers
-
PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference
A training-free method that merges pairs of MoE experts at the individual-weight level and packs the required masks into unused exponent bits, cutting expert memory by 50% with minimal accuracy loss.
-
Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inference
Communication-aware expert placement plus device-level pruning yields 1.23–1.86× MoE inference throughput and better accuracy at equal speedup than load-balance or sequential baselines.
-
Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMs
DERN prunes SMoE LLMs by decomposing removed experts into neuron segments, reassigning the best-matching ones to kept experts, and clustering them into compact replacements, beating prior pruning baselines without retraining.
-
Unified Start, Personalized End: Progressive Pruning for Efficient 3D Medical Image Segmentation
PSP-Seg prunes redundant modules during training to make 3D segmentation networks much smaller and faster without losing accuracy.
-
Mixture-of-Experts for Personalized and Semantic-Aware Next Location Prediction
NextLocMoE combines a location-semantics expert mixer and a persona-based expert mixer inside a trimmed LLaMA backbone to set new accuracy marks on next-location prediction benchmarks.
-
MoE-Compression: How the Compression Error of Experts Affects the Inference Accuracy of MoE Model?
In a 26-layer MoE model, injecting Gaussian weight errors into middle-layer experts hurts math accuracy most, while deep-layer errors can sometimes improve instruction compliance.
-
LExI: Layer-Adaptive Active Experts for Efficient MoE Model Inference
LExI sets a different number of active experts per layer, found by weight-only sensitivity profiling and evolutionary search, improving MoE inference throughput with little accuracy loss.
Discussion (0). Sign in to comment.