REVIEW 7 cited by
Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Language Models (LLMs) with billions of parameters are prime targets for network pruning, removing some model weights without hurting performance. Prior approaches such as magnitude pruning, SparseGPT, and Wanda, either concentrated solely on weights or integrated weights with activations for sparsity. However, they overlooked the informative gradients derived from pretrained LLMs. In this paper, we present a novel sparsity-centric pruning method for pretrained LLMs, termed Gradient-based Language Model Pruner (GBLM-Pruner). GBLM-Pruner leverages the first-order term of the Taylor expansion, operating in a training-free manner by harnessing properly normalized gradients from a few calibration samples to determine the pruning metric, and substantially outperforms competitive counterparts like SparseGPT and Wanda in multiple benchmarks. Intriguingly, by incorporating gradients, unstructured pruning with our method tends to reveal some structural patterns, which mirrors the geometric interdependence inherent in the LLMs' parameter structure. Additionally, GBLM-Pruner functions without any subsequent retraining or weight updates to maintain its simplicity as other counterparts. Extensive evaluations on LLaMA-1 and LLaMA-2 across various benchmarks show that GBLM-Pruner surpasses magnitude pruning, Wanda and SparseGPT by significant margins. We further extend our approach on Vision Transformer. Our code and models are available at https://github.com/VILA-Lab/GBLM-Pruner.
Forward citations
Cited by 7 Pith papers
-
Weight-Adjusted Gradients Reveal Parameter Importance and Failure Modes in LLMs
Weight-adjusted gradients (weight times gradient) identify sparse LLM parameters whose masking induces rapid collapse and improve several efficiency and editing applications.
-
Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation
PrunedLoRA tunes large-rank LoRA adapters and progressively structurally prunes A and B to a target low rank, empirically closing much of the LoRA-to-full-fine-tuning gap.
-
Amber Pruner: Leveraging N:M Activation Sparsity for Efficient Prefill in Large Language Models
Amber Pruner proposes training-free N:M activation sparsity for LLM prefill; however, the supplied manuscript body is an unrelated paper.
-
SDMPrune: Self-Distillation MLP Pruning for Efficient Large Language Models
SDMPrune combines a self-distillation loss with Taylor-based importance scoring to prune only MLP neurons, improving zero-shot performance of compressed LLaMA models over existing pruning methods.
-
DipSVD: Dual-importance Protected SVD for Efficient LLM Compression
DipSVD combines channel-weighted whitening with layer-wise compression ratios and reports better perplexity and accuracy than existing SVD-based LLM compression methods.
-
Neural Parameter Search for Slimmer Fine-Tuned Models and Better Transfer
Neural Parameter Search (NPS) prunes fine-tuned models by evolutionary reweighting of magnitude-based task vector subspaces, improving transfer, fusion, and compression.
-
TriSP: Tri-Signal Structured Pruning for Large Language Models
TriSP prunes LLM attention heads and MLP neurons by combining magnitude, activation norm, and gradient sensitivity, achieving the best reported quality at 20–50% pruning on 7B models with LoRA recovery.
Discussion (0). Continue with ORCID to comment.