REVIEW 7 cited by
LaCo: Large Language Model Pruning via Layer Collapse
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models (LLMs) based on transformer are witnessing a notable trend of size expansion, which brings considerable costs to both model training and inference. However, existing methods such as model quantization, knowledge distillation, and model pruning are constrained by various issues, including hardware support limitations, the need for extensive training, and alterations to the model internal structure. In this paper, we propose a concise layer-wise structured pruner called \textit{Layer Collapse (LaCo)}, in which rear model layers collapse into a prior layer, enabling a rapid reduction in model size while preserving the model structure. Comprehensive experiments show that our method maintains an average task performance of over 80\% at pruning ratios of 25-30\%, significantly outperforming existing state-of-the-art structured pruning methods. We also conduct post-training experiments to confirm that the \textit{LaCo} effectively inherits the parameters of the original model. Additionally, we perform ablation studies on various settings of \textit{LaCo}. Finally, we discuss our motivation from the perspective of layer-wise similarity and evaluate the performance of the pruned LLMs across various pruning ratios\footnote{\url{https://github.com/yangyifei729/LaCo}}.
Forward citations
Cited by 7 Pith papers
-
When Does Sparsity Mitigate the Curse of Depth in LLMs
Implicit and explicit sparsity reduce residual-stream variance and improve layer effectiveness metrics, enabling a depth-scaling recipe with about 4.6 points higher downstream accuracy.
-
GPTailor: Large Language Model Pruning Through Layer Cutting and Stitching
GPTailor searches over layer removal, layer selection, and layer merging across fine-tuned model variants to produce smaller LLMs that retain more benchmark performance than single-model pruning.
-
DaMoC: Efficiently Selecting the Optimal Large Language Model for Fine-tuning Domain Tasks Based on Data and Model Compression
DaMoC combines data filtering, token compression, and layer pruning to select the best LLM for domain fine-tuning, claiming ~20x faster training while preserving model rankings.
-
Boosting Parameter Efficiency in LLM-Based Recommendation through Sophisticated Pruning
PruneRec prunes attention heads, embedding dimensions, MLP units, and layers from a recommendation-tuned LLM, retaining 88% of accuracy with under 5% of non-embedding parameters.
-
Pruning General Large Language Models into Customized Expert Models
Cus-Prun identifies and removes neurons that are irrelevant to a user's target language, domain, and task, producing specialized expert models without post-training.
-
Pangu Light: Weight Re-Initialization for Pruning and Accelerating LLMs
Pangu Light prunes LLMs across width, depth, and attention heads, then re-initializes remaining weights, achieving up to 4.2x throughput with modest benchmark loss.
-
SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decoupling
SkipGPT learns per-token, per-module routers that dynamically skip attention and MLP blocks, then uses LoRA to recover most of the original model's quality.
Discussion (0). Sign in to comment.