REVIEW 7 cited by
Reassessing Layer Pruning in LLMs: New Insights and Methods
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Although large language models (LLMs) have achieved remarkable success across various domains, their considerable scale necessitates substantial computational resources, posing significant challenges for deployment in resource-constrained environments. Layer pruning, as a simple yet effective compression method, removes layers of a model directly, reducing computational overhead. However, what are the best practices for layer pruning in LLMs? Are sophisticated layer selection metrics truly effective? Does the LoRA (Low-Rank Approximation) family, widely regarded as a leading method for pruned model fine-tuning, truly meet expectations when applied to post-pruning fine-tuning? To answer these questions, we dedicate thousands of GPU hours to benchmarking layer pruning in LLMs and gaining insights across multiple dimensions. Our results demonstrate that a simple approach, i.e., pruning the final 25\% of layers followed by fine-tuning the \texttt{lm\_head} and the remaining last three layer, yields remarkably strong performance. Following this guide, we prune Llama-3.1-8B-It and obtain a model that outperforms many popular LLMs of similar size, such as ChatGLM2-6B, Vicuna-7B-v1.5, Qwen1.5-7B and Baichuan2-7B. We release the optimal model weights on Huggingface, and the code is available on GitHub.
Forward citations
Cited by 7 Pith papers
-
When Fewer Layers Break More Chains: Layer Pruning Harms Test-Time Scaling in LLMs
Removing just one or two layers from a reasoning-tuned LLM destroys its test-time scaling, and neither LoRA nor full fine-tuning restores it.
-
Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMs
DERN prunes SMoE LLMs by decomposing removed experts into neuron segments, reassigning the best-matching ones to kept experts, and clustering them into compact replacements, beating prior pruning baselines without retraining.
-
Latent Flow Transformer
LFT replaces up to 13 of 24 transformer layers of Pythia-410M with a single flow-based layer trained with Flow Walking, achieving KL 0.736 vs 0.932 for skipping three layers.
-
FCOS: A Two-Stage Recoverable Model Pruning Framework for Automatic Modulation Recognition
FCOS combines channel clustering and layer collapse diagnosis to prune AMR models by over 95% with minimal accuracy loss.
-
A Multimodal Deep Learning Framework for Early Diagnosis of Liver Cancer via Optimized BiLSTM-AM-VMD Architecture
The paper claims a BiLSTM-AM-VMD model achieves AUC 0.963 for early HCC diagnosis, but the evidence is undermined by contradictory dataset descriptions and missing artifacts.
-
Multi-Modal Machine Learning Framework for Predicting Early Recurrence of Brain Tumors Using MRI and Clinical Biomarkers
XGBoost combining MRI radiomics and clinical biomarkers reportedly reaches C-index 0.782 for early brain tumor recurrence, but the paper's methods describe a liver-cancer cohort and no evaluation of its claimed tempor...
-
ReStNet: A Reusable & Stitchable Network for Dynamic Adaptation on IoT Devices
ReStNet builds a hybrid model from two pre-trained networks by stitching at the most CKA-similar layer and fine-tuning only the stitching layer, claiming flexible runtime trade-offs for IoT devices.
Discussion (0). Continue with ORCID to comment.