Pith. sign in

REVIEW 3 cited by

Movement Pruning: Adaptive Sparsity by Fine-Tuning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2005.07683 v2 pith:BKJDLOJG submitted 2020-05-15 cs.CL cs.LG

Victor Sanh , Thomas Wolf , Alexander M. Rush This is my paper · ORCID
classification cs.CLcs.LG
keywords pruningmodelmovementadaptivefine-tuningfirst-orderlanguagelearning
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Magnitude pruning is a widely used strategy for reducing model size in pure supervised learning; however, it is less effective in the transfer learning regime that has become standard for state-of-the-art natural language processing applications. We propose the use of movement pruning, a simple, deterministic first-order weight pruning method that is more adaptive to pretrained model fine-tuning. We give mathematical foundations to the method and compare it to existing zeroth- and first-order pruning methods. Experiments show that when pruning large pretrained language models, movement pruning shows significant improvements in high-sparsity regimes. When combined with distillation, the approach achieves minimal accuracy loss with down to only 3% of the model parameters.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LaPrune: Controllable Differentiable Sparsity at Million Scale

    cs.LG 2026-08 accept novelty 6.0 of 10

    A differentiable top-k mask layer that enforces an exact selection budget and uses a normalized hardness parameter to interpolate from equal-weight masks to hard binary masks, with saturation theory and million-scale results.

  2. Investigating Structural Pruning and Recovery Techniques for Compressing Multimodal Large Language Models: An Empirical Study

    cs.CL 2025-07 conditional novelty 5.0 of 10

    Structural pruning with finetuning plus hidden-state distillation recovers most performance in multimodal LLMs, with 5% of training data sufficient at moderate compression levels.

  3. The Carbon Cost of Conversation, Sustainability in the Age of Language Models

    cs.CY 2025-07 reject novelty 2.0 of 10

    A review-style essay that claims LLM training and use are environmentally expensive, but the quantitative evidence it presents is unreliable and internally inconsistent.

Pith tools