Pith. sign in

REVIEW 4 major objections 3 minor 1 cited by

LLM-BIP: Structured Pruning for Large Language Models with Block-Wise Forward Importance Propagation

T0 review · 4 major / 3 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLM-BIP prunes transformer blocks by scoring how much each channel and attention head changes the block's output, computing the scores in a single forward pass, and reports that this outperforms Wanda and LLM-Pruner at high sparsity.

desk verdict A simple, efficient block-wise pruning heuristic with attractive zero-shot numbers, but the derivation meant to justify it is written for an un-gated FFN and does not apply to the SwiGLU models actually pruned. read the letter →

arxiv 2412.06419 v1 pith:LWFNJ7AR submitted 2024-12-09 cs.CL cs.AI

Haihang Wu This is my paper · ORCID
classification cs.CLcs.AI MSC 68T5068T07
keywords structuredpruninglargelanguagemodelsblock-wiseimportancepropagationsingleforwardpassattentionheadFFNchannelLLaMAzero-shotevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LLM-BIP claims that the right unit for judging what to prune in a transformer is the transformer block, not a single layer: it derives closed-form importance scores for FFN channels and attention-head output channels that upper-bound the change in block output when those structures are removed, and it computes the scores from one forward pass. If correct, this gives a fast, gradient-free structured pruning method that preserves more of a model's reasoning ability at 20–70% sparsity than layer-wise or gradient-based alternatives. The paper reports that pruning LLaMA-7B, Vicuna-7B, and LLaMA-13B with these scores reduces perplexity on WikiText2 and PTB and raises zero-shot accuracy relative to Wanda and LLM-Pruner. The contribution is a block-aware importance metric that avoids both unreliable gradients and layer-wise error accumulation.

What carries the argument

The central objects are the block-wise importance scores in Definition 1: $s^F_j = |X^U_j|\cdot|W^D_j|$ and $s^H_j = |X^H_j|\cdot|W^O_j|(I+|W^U||W^D|)$, where $X^U_j$ and $X^H_j$ are the output values of the $j$-th FFN channel and attention-head output channel, and $W^D_j$ and $W^O_j$ are the corresponding output-weight rows. They come from an inequality that bounds the $\ell^1$ change in the block output by $C_\sigma \sum_j (1-s^F_j)|X^U_j||W^D_j|$, so selecting channels to keep is equivalent to keeping those with large scores. The same inequality is the reason the method needs only one forward pass: the scores are read off from activations and weights already computed during a single feed-forward sweep, and no gradient or Hessian is required.

What would settle it

On a gated LLaMA-7B block, compute the paper's score $|X^U_j||W^D_j|$ for every FFN channel and also measure the true block-output error from removing each channel alone over a fixed calibration set; if the rank correlation between score and measured error is low for channels with small $|X^U_j|$ but large gate outputs, the score fails to measure what the central claim requires. A simpler check is to test whether the Eq. (7) inequality actually holds on the gated architecture for randomly pruned masks; finding even one mask where the bound is violated would refute the derivation, though not necessarily the empirical heuristic.

Watch

Extended reading notes

Core claim

The central claim is that a channel's contribution to a transformer block can be measured by a product of its activation magnitude and the norm of its output weights: $s^F_j = |X^U_j|\cdot|W^D_j|$ for FFN channels and $s^H_j = |X^H_j|\cdot|W^O_j|(I+|W^U||W^D|)$ for attention-head output channels. The paper derives these scores as an upper bound on the $\ell^1$ distance between the original block output and the pruned block output, using Lipschitz continuity of the activation and treating pruning as zeroing entries of $X^U$ or $X^H$. Minimizing the bound leads to pruning channels with small scores, and the bound is computable from a single forward pass with no gradients or Hessians. The paper consequently claims that LLM-BIP prunes structural groups more accurately than global gradient-based methods and more robustly than layer-wise methods, with the gap widening at high sparsity.

Load-bearing premise

The derivation models the FFN as a single up projection followed by activation and down projection, while the evaluated LLaMA and Vicuna models use a gated FFN in which a separate gate output multiplies the up output before the down projection; if the gate activation materially changes which channels matter, the score is not a true upper bound on the block error.

Editorial extensions

If this is right

  • Pruning decisions can be made once, in a single forward pass, without computing or storing gradients; the paper reports pruning runs in about half the time of LLM-Pruner on an A100 GPU.
  • At 20% and 50% sparsity, the pruned LLaMA-7B, Vicuna-7B, and LLaMA-13B models keep higher zero-shot accuracy and lower perplexity than Wanda, LLM-Pruner, magnitude-based, random, and importance-propagation baselines.
  • Because importance is scored at block level, errors accumulate more slowly across layers than with layer-wise pruning; the paper shows LLM-BIP has smaller block-output reconstruction error than Wanda as depth increases.
  • The method remains usable without fine-tuning; fine-tuning with LoRA gives further gains mostly at higher sparsity, and at 20% sparsity the pruned model is already near the fine-tuned accuracy.
  • Structured pruning decisions translate to hardware speedup: the paper reports 24% and 40% inference latency reductions at 20% and 50% sparsity, respectively.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the derivation ignores the gate projection in gated FFNs such as SwiGLU, a natural extension is to multiply the score by the gate activation magnitude $|X^G_j|$; this variant could be tested directly against the paper's score on LLaMA models to see whether the high-sparsity gap closes further.
  • The same block-output upper-bound logic could be applied to other block-structured components or combined with quantization, since both are block-local reconstruction objectives.
  • The paper's upper bound sums independent per-channel contributions, so it does not model interactions between simultaneously pruned channels; at very high sparsity, a correction term for co-pruning effects may be needed.
  • If the metric is as calibration-robust as reported, it could be used in streaming or on-device settings where a pruning decision must be made from a tiny observation budget.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes LLM-BIP, a structured pruning method for LLMs that assigns importance scores to FFN channels and attention heads by measuring their influence on the transformer block output. The scores are derived from an upper bound on the block-level reconstruction error and are computed in a single forward pass without gradients. The method is evaluated on LLaMA-7B, Vicuna-7B, and LLaMA-13B at 20%, 50%, and 70% sparsity, reporting lower perplexity and higher zero-shot accuracy than Wanda, LLM-Pruner, and other baselines, both without fine-tuning and after LoRA fine-tuning.

Significance. If the proposed metric is sound, LLM-BIP would be a practically valuable contribution: it is fast (one forward pass), gradient-free, and shows large gains at high sparsity levels, especially at 50% and 70% where structured pruning of LLMs typically fails. The empirical tables are extensive and the method appears to be self-contained relative to the benchmarks, with no parameter fitted to downstream accuracy. However, the central derivation is built on an un-gated FFN model that does not match the evaluated LLaMA/Vicuna architectures, so the theoretical justification for the pruning score is not currently established. The paper also defers a key derivation to a supplementary file that is not present, making the MSA score in Eq. (8) unverifiable.

major comments (4)
  1. [Method, Eqs. (4)-(7), (9)] The derivation of the FFN importance score models the FFN as sigma(X'W_U)W_D (Eq. 4), but LLaMA-1, LLaMA-2, and Vicuna use the gated SwiGLU FFN (SiLU(X'W_G) ⊙ X'W_U)W_D. Eq. (7) therefore bounds the wrong object: the actual block-output change after pruning channel j is (1-s_j)SiLU(X'W_G)_j(X'W_U)_j W_{D,j}, which depends on the gate activation, and the proposed score |XU_j||WD_j| is not an upper bound for this change. A channel with large |XU_j| and near-zero gate output will be ranked important even though its contribution to the block output is negligible. The central claim that sF_j measures influence on the block output is not established for the evaluated architectures; the derivation must be redone with the gate included, or the score must be presented as a heuristic with supporting analysis.
  2. [Eq. (8), Definition 1, Remark] The MSA importance score is stated as |XH_j|·|WO_j|(I+|WU||WD|) with the derivation deferred to 'the supplementary materials', but no supplementary derivation is included in the manuscript, and the formula has dimensional ambiguities (for example, the roles of the per-token activation vector and the output-channel index are not specified). Since the coupling term (I+|WU||WD|) involves FFN weights and the FFN is gated in the evaluated models, the expression cannot be taken as established. The authors must provide the full derivation and state the exact dimensions of each term, or remove the claim that Eq. (8) is derived.
  3. [Tables 1-3 and Section 'Experimental Settings'] The experimental section states that LLaMA-2-13B is evaluated, but Table 3 labels the unpruned baseline as 'LLaMA-13B (Ma, Fang, and Wang 2023)' and lists a baseline perplexity of 11.58 for the original model. It is unclear whether the pruned model is LLaMA-1-13B or LLaMA-2-13B, and the baseline values appear to be taken from another paper. This matters because LLaMA-1 and LLaMA-2 have different architectures and tokenizers, and the 13B results are part of the central empirical claim. Please clarify the exact checkpoint and report the original model's numbers measured with the same evaluation pipeline.
  4. [Ablation Study and Algorithm 1] Algorithm 1 describes a sequential per-block pruning loop, but the text does not specify whether the forward pass for block l is computed with the original model or with the already-pruned earlier blocks. This distinction is important for the claim that block-wise pruning reduces error accumulation: if the activations are always taken from the unpruned model, then the measured per-block errors in Figure 3 are not the same as the errors that propagate through the pruned network. Please state the exact forward-propagation protocol used during pruning and clarify what Figure 3 measures.
minor comments (3)
  1. [Eq. (7)] The inequality chain in Eq. (7) uses both dot products and element-wise products without defining how the vector |XU| and the matrix |WD| are multiplied in the displayed expressions; the final sum suggests a per-channel product, but the preceding line is dimensionally ambiguous. Please make the notation explicit.
  2. [Tables 1-2] There are typographical errors in the tables: 'Importace Propagation' should be 'Importance Propagation', and the 'w/tune' and 'w/o tune' labels are inconsistent with the formatting elsewhere. These should be corrected for reproducibility.
  3. [Abstract and Introduction] The reported improvements are described variously as 3.26%, 4.49%, and 2.18% average accuracy gains; the abstract and introduction do not make clear which baselines and sparsity levels each number refers to. Please add a precise sentence tying each reported average to the corresponding table and conditioning regime.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the pruning importance scores are derived from an explicit upper bound and evaluated on held-out benchmarks; no parameter is fitted to the targets and no self-citation is load-bearing.

full rationale

The paper derives the FFN importance score sF_j = |XU_j| * |WD_j| from an upper bound on the transformer block output reconstruction error (Eq. 7), and the MSA score sH_j = |XH_j| * |WO_j| * (I + |WU| * |WD|) from an analogous derivation referenced to the supplementary material (Eq. 8). These scores are not fitted to the zero-shot benchmarks: they are computed in a single forward pass on calibration sentences from C4, then used to prune, and the resulting models are evaluated on held-out WikiText2, PTB, and common-sense reasoning tasks. No parameter of the pruning criterion is tuned to maximize benchmark accuracy, and no prior work of the same author is cited as the basis of the central claim. The selection rule follows directly from minimizing the stated upper bound, so the derivation is self-contained relative to the paper's stated objective. Potential weaknesses—such as the FFN model in Eq. (4) omitting the SwiGLU gate used in LLaMA and Vicuna, or the fact that minimizing an upper bound does not necessarily minimize the true reconstruction error—are matters of derivation validity or architectural mismatch, not circularity. The MSA formula is asserted rather than fully derived in the main text, but that is an omitted proof rather than a circular reduction. Accordingly, the circularity score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method introduces no fitted constants, and the central derivation rests on assumptions about the transformer architecture and on the existence of a supplementary derivation. The most consequential assumption is the simplified FFN model, which does not match the gated FFN of LLaMA/Vicuna.

free parameters (3)
  • LoRA rank
    Used in the fine-tuning step, but the rank is not reported in the paper, so the fine-tuned results are not fully reproducible.
  • Calibration set size = 128 sentences
    Chosen by hand for main experiments; robustness to size is tested, but all headline numbers use this fixed value.
  • Sparsity ratio r = 20%, 50%, 70%
    User-specified target sparsity, not fitted to data, but it is a key input to all reported results.
assumptions (5)
  • domain assumption Activation functions (GeLU, ReLU, Swish) are Lipschitz continuous with a uniform constant C_sigma.
    Used to bound |sigma(XU)-sigma(sF*XU)| in Eq (7); true for the cited activations, but C_sigma is never estimated and the bound drops it as a constant.
  • ad hoc to paper The FFN module can be written as sigma(X'W_U)W_D (Eq 4), i.e., a two-layer MLP without the gated SwiGLU structure used in LLaMA/Vicuna.
    Eq (4) omits the separate gate projection and element-wise multiplication that define the evaluated models; the pruning score in Eq (9) therefore ignores gate activations.
  • ad hoc to paper The attention-head importance score in Eq (8) follows from a derivation in the supplementary materials.
    The paper refers to supplementary details, but no supplementary file is present in the arXiv submission, so the derivation is unverified.
  • domain assumption Channel importances are independent: pruning decisions are made by sorting per-channel scores, ignoring interactions between pruned channels.
    The joint optimization in Eq (6) is replaced by per-channel thresholding; standard for pruning but not justified for correlated channels.
  • domain assumption Calibration activations XH and XU from 128 C4 sentences are representative of deployment data.
    The importance scores depend on calibration inputs; the authors test robustness to calibration size, but representativeness for other domains is not guaranteed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM-BIP: Structured Pruning for Large Language Models with Block-Wise Forward Importance Propagation." pith.science (2026). https://pith.science/paper/LWFNJ7AR

@misc{pith2026241206419,
  author       = {Pith},
  title        = {Pith review of: LLM-BIP: Structured Pruning for Large Language Models with Block-Wise Forward Importance Propagation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LWFNJ7AR}},
  note         = {Machine review of arXiv:2412.06419}
}
read the original abstract

Large language models (LLMs) have demonstrated remarkable performance across various language tasks, but their widespread deployment is impeded by their large size and high computational costs. Structural pruning is a prevailing technique used to introduce sparsity into pre-trained models and facilitate direct hardware acceleration during inference by removing redundant connections (structurally-grouped parameters), such as channels and attention heads. Existing structural pruning approaches often employ either global or layer-wise pruning criteria; however, they are hindered by ineffectiveness stemming from inaccurate evaluation of connection importance. Global pruning methods typically assess component importance using near-zero and unreliable gradients, while layer-wise pruning approaches encounter significant pruning error accumulation issues. To this end, we propose a more accurate pruning metric based on the block-wise importance score propagation, termed LLM-BIP. Specifically, LLM-BIP precisely evaluates connection importance by gauging its influence on the respective transformer block output, which can be efficiently approximated in a single forward pass through an upper bound derived from the assumption of Lipschitz continuity. We evaluate the proposed method using LLaMA-7B, Vicuna-7B, and LLaMA-13B across common zero-shot tasks. The results demonstrate that our approach achieves an average of 3.26% increase in accuracy for common reasoning tasks compared to previous best baselines. It also reduces perplexity by 14.09 and 68.76 on average for the WikiText2 dataset and PTB dataset, respectively.

Figures

Figures reproduced from arXiv: 2412.06419 by the authors.

Figure 1
Figure 1. The comparison among global pruning, layer-wise pruning, and our block-wise pruning. Global pruning methods (Ma, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison of the proposed method, Wanda and LLM-Pruner on LLaMA-7B (left) and Vicuna-7B (right) with [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Our method mitigates the pruning error accumu [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AutoChemSchematic AI: Agentic Physics-Aware Automation for Chemical Manufacturing Scale-Up

    cs.LG 2025-05 reject novelty 4.0 of 10

    The framework trains small models on synthetic AI-generated data to produce PFD/PID text, then validates two examples by manual DWSIM setup, leaving the industrial-viability claim unproven.

Reference graph

Works this paper leans on

59 extracted references · 52 canonical work pages · cited by 1 Pith paper

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Bai, H.; Hou, L.; Shang, L.; Jiang, X.; King, I.; and Lyu, M. R. 2022. Towards Efficient Post-training Quantization of Pre-trained Language Models . In NeurIPS

    work page 2022

  4. [4]

    Bisk, Y.; Zellers, R.; Le Bras, R.; Gao, J.; and Choi, Y. 2020. PIQA: Reasoning about Physical Commonsense in Natural Language . In AAAI

    work page 2020

  5. [5]

    Chen, M.; Shao, W.; Xu, P.; Lin, M.; Zhang, K.; Chao, F.; Ji, R.; Qiao, Y.; and Luo, P. 2023. DiffRate : Differentiable Compression Rate for Efficient Vision Transformers . In ICCV

    work page 2023

  6. [6]

    Chen, T.; Cheng, Y.; Gan, Z.; Yuan, L.; Zhang, L.; and Wang, Z. 2021. Chasing Sparsity in Vision Transformers: An End-to-End Exploration . In NeurIPS

    work page 2021

  7. [7]

    Clark, C.; Lee, K.; Chang, M.-W.; Kwiatkowski, T.; Collins, M.; and Toutanova, K. 2019. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions . In NAACL

    work page 2019

  8. [8]

    Clark, P.; Cowhey, I.; Etzioni, O.; Khot, T.; Sabharwal, A.; Schoenick, C.; and Tafjord, O. 2018. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

    work page 2018

Show all 59 references
  1. [9]

    S.; and Sol, S

    Cun, L.; Le Cun, Y.; Denker, J. S.; and Sol, S. A. 1989. Optimal Brain Damage . In NeurIPS

    1989

  2. [10]

    Dettmers, T.; Svirschevski, R.; Egiazarian, V.; Kuznedelev, D.; Frantar, E.; Ashkboos, S.; Borzunov, A.; Hoefler, T.; and Alistarh, D. 2023. SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression . arXiv preprint arXiv:2306.03078

    2023 arXiv

  3. [11]

    Dong, X.; Chen, S.; and Pan, S. J. 2017. Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon . In NeulPS

    2017

  4. [12]

    S.; and Elsen, E

    Evci, U.; Gale, T.; Menick, J.; Castro, P. S.; and Elsen, E. 2020. Rigging the Lottery: Making All Tickets Winners . In ICML

    2020

  5. [13]

    B.; and Wang, X

    Fang, G.; Ma, X.; Song, M.; Mi, M. B.; and Wang, X. 2023. DepGraph: Towards Any Structural Pruning . In CVPR

    2023

  6. [14]

    Frantar, E.; and Alistarh, D. 2023. SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot . In ICML

    2023

  7. [15]

    Frantar, E.; Ashkboos, S.; Hoefler, T.; and Alistarh, D. 2023. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers . In ICLR

    2023

  8. [16]

    P.; and Alistarh, D

    Frantar, E.; Singh, S. P.; and Alistarh, D. 2022. Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning . In NeurIPS

    2022

  9. [17]

    G.; and Ivolff, G

    Hassibi, B.; Stork, D. G.; and Ivolff, G. J. 1993. Optimal Brain Surgeon and General Network Pruning . In IEEE International Conference on Neural Networks

    1993

  10. [18]

    Hou, Z.; Qin, M.; Sun, F.; Ma, X.; Yuan, K.; Xu, Y.; Chen, Y.-K.; Jin, R.; Xie, Y.; and Kung, S.-Y. 2022. CHEX: CHannel EXploration for CNN Model Compression . In CVPR

    2022

  11. [19]

    J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W

    Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2022. LoRA: Low-Rank Adaptation of Large Language Models . In ICLR

    2022

  12. [20]

    Jiang, C.; Li, G.; Qian, C.; and Tang, K. 2018. Efficient DNN Neuron Pruning by Minimizing Layer-wise Nonlinear Reconstruction Error . In IJCAI

    2018

  13. [21]

    M.; Frankle, J.; and Dziugaite, G

    Jin, T.; Carbin, M.; Roy, D. M.; Frankle, J.; and Dziugaite, G. K. 2022. Pruning's Effect on Generalization Through the Lens of Training and Regularization . In NeurlPS

    2022

  14. [22]

    Kim, H.; Papamakarios, G.; and Mnih, A. 2021. The Lipschitz Constant of Self-Attention . In ICML

    2021

  15. [23]

    Kong, Z.; Dong, P.; Ma, X.; Meng, X.; Sun, M.; Niu, W.; Shen, X.; Yuan, G.; Ren, B.; Qin, M.; Tang, H.; and Wang, Y. 2022. SPViT: Enabling Faster Vision Transformers via Soft Token Pruning . In ECCV

    2022

  16. [24]

    Kurtic, E.; Campos, D.; Nguyen, T.; Frantar, E.; Kurtz, M.; Fineran, B.; Goin, M.; and Alistarh, D. 2022. The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models

    2022

  17. [25]

    Kurtic, E.; Frantar, E.; and Alistarh, D. 2023. ZipLM: Hardware-Aware Structured Pruning of Language Models

    2023

  18. [26]

    Lee, N.; Ajanthan, T.; and Torr, P. H. S. 2019. SNIP: Single-shot Network Pruning based on Connection Sensitivity . In ICLR

    2019

  19. [27]

    Liu, Z.; Wang, J.; Dao, T.; Zhou, T.; Yuan, B.; Song, Z.; Shrivastava, A.; Zhang, C.; Tian, Y.; R \' e , C.; and Chen, B. 2023. Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time . In ICML

    2023

  20. [28]

    Loshchilov, I.; and Hutter, F. 2017. SGDR: Stochastic gradient descent with warm restarts . ICLR, 1--16

    2017

  21. [29]

    Loshchilov, I.; and Hutter, F. 2019. Decoupled Weight Decay Regularization . In ICLR

    2019

  22. [30]

    Ma, X.; Fang, G.; and Wang, X. 2023. LLM-Pruner: On the Structural Pruning of Large Language Models . In NeurIPS

    2023

  23. [31]

    P.; Marcinkiewicz \ , M

    Marcus, M. P.; Marcinkiewicz \ , M. A.; and Santorini, B. 1993. Building a Large Annotated Corpus of English: The Penn Treebank . Computational Linguistics, 19(2): 313--330

    1993

  24. [32]

    Merity, S.; Xiong, C.; Bradbury, J.; and Socher, R. 2017. Pointer Sentinel Mixture Models . In ICLR

    2017

  25. [33]

    Mihaylov, T.; Clark, P.; Khot, T.; and Sabharwal, A. 2018. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering . In EMNLP

    2018

  26. [34]

    Molchanov, P.; Mallya, A.; Tyree, S.; Frosio, I.; and Kautz, J. 2019. Importance Estimation for Neural Network Pruning . In CVPR

    2019

  27. [35]

    Nova, A.; Dai, H.; and Schuurmans, D. 2023. Gradient-Free Structured Pruning with Unlabeled Data . In ICML

    2023

  28. [36]

    Pool, J.; and Yu, C. 2021. Channel Permutations for N:M Sparsity . In NIPS

    2021

  29. [37]

    Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2019. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer . The Journal of Machine Learning Research, 21(1): 5485--5551

    2019

  30. [38]

    L.; Bhagavatula, C.; and Choi, Y

    Sakaguchi, K.; Bras, R. L.; Bhagavatula, C.; and Choi, Y. 2019. WinoGrande: An Adversarial Winograd Schema Challenge at Scale

    2019

  31. [39]

    Shi, D.; Tao, C.; Jin, Y.; Yang, Z.; Yuan, C.; and Wang, J. 2023. UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformers . In ICML

    2023

  32. [40]

    P.; and Alistarh, D

    Singh, S. P.; and Alistarh, D. 2020. WoodFisher: Efficient Second-Order Approximation for Neural Network Compression . In NeurIPS

    2020

  33. [41]

    Sun, M.; Liu, Z.; Bair, A.; and Kolter, J. Z. 2024. A Simple and Effective Pruning Approach for Large Language Models . In ICLR

    2024

  34. [42]

    Sun, S.; Cheng, Y.; Gan, Z.; and Liu, J. 2019. Patient Knowledge Distillation for BERT Model Compression . In EMNLP, 4323--4332

    2019

  35. [43]

    Sun, S.; Gan, Z.; Cheng, Y.; Fang, Y.; Wang, S.; and Liu, J. 2020. Contrastive Distillation on Intermediate Representations for Language Model Compression . In EMNLP

    2020

  36. [44]

    Tang, Y.; Han, K.; Wang, Y.; Xu, C.; Guo, J.; Xu, C.; and Tao, D. 2022. Patch Slimming for Efficient Vision Transformers . In CVPR

    2022

  37. [45]

    Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozi \` e re, B.; Goyal, N.; Hambro, E.; Azhar, F.; Rodriguez, A.; Joulin, A.; Grave, E.; and Lample, G. 2023. LLaMA: Open and Efficient Foundation Language Models . arXiv preprint arXiv:2302.13971

    2023 arXiv

  38. [46]

    Gonzalez ; Ion Stoica ; and Eric P

    Wei-Lin Chiang ; Zhuohan Li ; Zi Lin ; Ying Sheng ; Zhanghao Wu ; Hao Zhang ; Lianmin Zheng ; Siyuan Zhuang ; Yonghao Zhuang ; Joseph E. Gonzalez ; Ion Stoica ; and Eric P. Xing . 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90 \

    2023

  39. [47]

    Wu, M.; Waheed, A.; Zhang, C.; Abdul-Mageed, M.; and Aji, A. F. 2024. LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions . In EACL

    2024

  40. [48]

    Xia, M.; Gao, T.; Zeng, Z.; and Chen, D. 2023. Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning

    2023

  41. [49]

    Xu, P.; Shao, W.; Chen, M.; Tang, S.; Zhang, K.; Gao, P.; An, F.; QIao, Y.; and Luo, P. 2024. BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation . In ICLR

    2024

  42. [50]

    Yang, H.; Yin, H.; Shen, M.; Molchanov, P.; Li, H.; and Kautz, J. 2023. Global Vision Transformer Pruning with Hessian-Aware Saliency . In CVPR

    2023

  43. [51]

    Yu, F.; Huang, K.; Wang, M.; Cheng, Y.; Chu, W.; and Cui, L. 2022 a . Width & Depth Pruning for Vision Transformers . In AAAI

    2022

  44. [52]

    I.; Han, X.; Gao, M.; Lin, C.-Y.; and Davis, L

    Yu, R.; Li, A.; Chen, C.-F.; Lai, J.-H.; Morariu, V. I.; Han, X.; Gao, M.; Lin, C.-Y.; and Davis, L. S. 2018. NISP: Pruning Networks using Neuron Importance Score Propagation . In CVPR

    2018

  45. [53]

    Yu, X.; Serra, T.; Ramalingam, S.; and Zhe, S. 2022 b . The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networks . In ICML

    2022

  46. [54]

    Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019. HellaSwag: Can a Machine Really Finish Your Sentence? In ACL

    2019

  47. [55]

    Zhang, M.; Chen, H.; Shen, C.; Yang, Z.; Ou, L.; Yu, X.; and Zhuang, B. 2023. LoRAPrune: Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning

    2023

  48. [56]

    Zhang, Q.; Zuo, S.; Liang, C.; Bukharin, A.; He, P.; Chen, W.; and Zhao, T. 2022 a . PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance . In ICML

    2022

  49. [57]

    Zhang, Y.; Lin, M.; Lin, Z.; Luo, Y.; Li, K.; Chao, F.; Wu, Y.; and Ji, R. 2022 b . Learning Best Combination for Efficient N:M Sparsity . In NIPS

    2022

  50. [58]

    Zhou, A.; Ma, Y.; Zhu, J.; Liu, J.; Zhang, Z.; Yuan, K.; Sun, W.; and Li, H. 2021. LEARNING N:M FINE-GRAINED STRUCTURED SPARSE NEURAL NETWORKS FROM SCRATCH . In ICLR

    2021

  51. [59]

    Zhuang, Z.; Tan, M.; Zhuang, B.; Liu, J.; Guo, Y.; Wu, Q.; Huang, J.; and Zhu, J. 2018. Discrimination-aware Channel Pruning for Deep Neural Networks . In NeuIPS

    2018

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.