Pith. sign in

REVIEW 6 major objections 6 minor 45 references

PGB: One-Shot Pruning for BERT via Weight Grouping and Permutation

T0 review · 6 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PGB prunes a fine-tuned BERT in one pass by grouping important weights into diagonal blocks, and reports higher GLUE and SQuAD accuracy than iterative pruning baselines, in a few hours.

desk verdict A useful one-shot BERT pruning method with believable results, but the paper needs code, error bars, and a fairer cost analysis before its headline claims are fully secure. read the letter →

arxiv 2502.03984 v1 pith:AEJ63OVM submitted 2025-02-06 cs.CL cs.AI

Hyemin Lim , Jaeyeon Lee , Dong-Wan Choi This is my paper · ORCID
classification cs.CLcs.AI
keywords BERTone-shotpruningsemi-structuredpretrainedlanguagemodelstask-specificweightgroupingpermutationGLUE
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PGB is a way to compress a fine-tuned BERT by pruning it once, in a single pass, instead of the usual loop of pruning, retraining, and distilling from a teacher model. The trick is to permute the rows and columns of each weight matrix until the weights the model actually relies on form compact diagonal blocks, then keep only those blocks and zero everything else; a second permutation puts each surviving weight back where it originally sat. The paper reports that this one-shot scheme beats three established structured-pruning methods on all seven GLUE tasks and on SQuAD at both 50% and 88% sparsity, and that the entire pruning-and-refinetuning process takes at most 2.1 hours, where the baselines take more than a day. If the results hold, BERT-sized models can be compressed for deployment quickly, with no distillation and little engineering.

What carries the argument

The machinery that carries the argument is the block-diagonal grouped weight matrix $\widehat{W}$, obtained by permuting the rows and columns of $W$ and keeping only $G$ diagonal blocks of size $\frac{M}{G} \times \frac{N}{G}$. The permutation is driven by an importance-maximization search: rows and columns are alternately sorted so that the total second-order importance in the top-left block of the permuted matrix is as large as possible, using a six-pass heuristic that the paper itself calls sub-optimal. The number of groups $G$ is set per matrix from the count of weights whose importance scores exceed a threshold $\tau$, capped at $G_{\max}$, so that the pruned matrix has about $MN/G$ surviving parameters; if the count is too high, the entire weight matrix is dropped, and if a whole layer forms no important group, the layer is dropped. The stored permutation vectors $\pi_r$ and $\pi_c$ are then inverted with $\mathrm{argsort}$, returning each surviving weight to its original position, so inference can run as $G$ independent dense blocks at $1/G$ the cost of the original linear operation.

What would settle it

Re-run PGB's exact pipeline on QQP at 88% sparsity but replace the alternating row/column sorting with random permutation of rows and columns, keeping all other hyperparameters and budgets identical; if the accuracy of the pruned model stays about the same, then the sorting heuristic contributes nothing and the reported advantage is coming from elsewhere (the block structure, the reconstruction step, or the re-finetuning).

Watch

Extended reading notes

Core claim

The central claim is that group-based pruning, previously applied to CNNs and to transformers trained from scratch, can be made to work on a fine-tuned BERT in one shot, provided the weights are first permuted into blocks of high importance. PGB assigns each weight a second-order importance score, adaptively sets the number $G$ of diagonal groups per matrix from the count of scores above a threshold, and runs an alternating row-and-column sorting heuristic to push the most important weights into the top-left corners of the permuted matrix. It then keeps only the $G$ diagonal blocks, prunes all other weights, and uses the stored permutation vectors to place every surviving weight back in its original row/column position; in layers where no important group forms, the entire layer is dropped. A short reconstruction-error fine-tune and three epochs of re-finetuning restore most of the lost accuracy. In the authors' experiments this pipeline yields models that are more accurate than CoFi, DynaBERT, and EBERT on all GLUE tasks at 88% sparsity, and it completes in under 2.1 hours.

Load-bearing premise

The whole method rests on the alternating row/column sorting heuristic clustering the weights the model truly depends on into the preserved blocks, and the paper itself labels that heuristic sub-optimal, so if it fails on real matrices the pruned model quietly throws away the wrong weights.

Editorial extensions

If this is right

  • If PGB's results hold, one-shot pruning can replace the default recipe of iterative pruning plus knowledge distillation for task-specific BERT, cutting compression time from more than a day to about two hours.
  • The largest accuracy gains over the baselines appear at 88% sparsity, so the grouping approach degrades more gracefully at extreme compression than the compared methods.
  • Because each pruned matrix executes as independent dense blocks, inference cost drops by a factor equal to the number of groups per matrix, without any special hardware.
  • The option to drop entire layers when no group forms means PGB produces models that are adaptive in both weights and depth, rather than forcing a fixed per-layer budget.
  • The method also applies to RoBERTa base and DistilBERT base, indicating the grouping-and-permutation principle is not specific to BERT's architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper never tests whether the alternating row/column sorting heuristic is actually finding good clusters: switching it for random permutations in a small ablation would reveal how much of the accuracy comes from the heuristic versus from the rest of the pipeline.
  • Because the method only needs any weight matrix, it should apply to decoder and cross-attention matrices in generative transformers, not just encoder BERT.
  • The block-diagonal structure left behind is a natural fit for hardware that accelerates structured sparsity (such as 2:4 patterns), which could turn the FLOPs saving into wall-clock speedups larger than the paper reports.
  • The short reconstruction-error step suggests that a few steps of low-rank adaptation on the surviving blocks could replace the full re-finetuning stage altogether.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. PGB is a one-shot semi-structured pruning method for task-fine-tuned BERT. For each weight matrix in the MHA and FFN sub-layers, PGB computes per-weight importance scores, adaptively selects a number G of groups, permutes rows and columns so that important weights are concentrated into diagonal blocks, prunes everything outside those blocks, and then re-permutes the surviving weights back to their original positions. Weight compensation and three epochs of re-finetuning follow. The method also drops whole FFN layers when no important group is found. Experiments on BERT_BASE cover seven GLUE tasks and SQuAD, with comparisons to EBERT, DynaBERT, CoFi, BMP, LayerDrop, and SNIP, plus additional results on DistilBERT and RoBERTa. The central claim is that PGB outperforms state-of-the-art structured pruning methods in both accuracy preservation and compression time, with the whole prune-and-finetune pipeline taking at most about 2.1 hours.

Significance. If the empirical results hold, the paper makes a useful practical contribution: it shows that a one-shot, distillation-free pruning scheme can be competitive with iterative structured-pruning baselines while reducing the compression pipeline from days to hours. The strengths are the breadth of evaluation, the explicit cost analysis for grouped inference, and the ablations over the main hyperparameters. The evidence is, however, conditional: all baselines are author re-implementations without released code, no variance is reported for the five-seed averages, and the grouping step relies on a permutation heuristic that the paper itself describes as sub-optimal and whose approximation quality is never measured. These gaps, rather than the core idea, are what currently prevent accepting the stated SOTA claim.

major comments (6)
  1. [Section 3.2 (Adaptive group numbers)] The rule for determining G is not fully specified: after Step 3 says 'set G to a value less than M x N / n_tau', there is no formula, rounding convention, or search procedure, and Gmax only supplies an upper cap. Since G determines the block size and the achieved sparsity, and since Figure 6 shows the final accuracy is sensitive to the group-number setting (Gmax=3 and Gmax=8 both degrade performance), this missing definition is load-bearing for reproducibility and for the interpretation of the hyperparameter ablation.
  2. [Section 3.2 (Eq. 3 and Algorithm 2)] Eq. (3) states the objective as maximizing the importance of the top-left M/G x N/G block of the full matrix W under one pair of row/column permutations. Algorithm 2, however, permutes the current remaining submatrix at each iteration and extracts the next block from that submatrix. These are not the same optimization problem, and the paper does not state the sequential objective that Algorithm 2 actually implements or give an argument for equivalence; the claim that the permutation step solves Eq. (3) is therefore unsupported.
  3. [Section 3.2 (Heuristic solution) and Table A.2] The permutation heuristic is explicitly described as sub-optimal and is fixed at Nperm=6 with no ablation over Nperm and no analysis of how close the found permutations are to the optimum of the stated objective. Because grouping quality is the central mechanism by which PGB preserves important weights, this omission is load-bearing: if the alternating-sort heuristic converges to a poor local arrangement, the extracted blocks can systematically exclude important weights, and the Table 1 margins over the baselines cannot be attributed to the grouping step as opposed to the three epochs of re-finetuning. Please add an Nperm ablation and, at least on small matrices, a comparison against optimal or random permutations.
  4. [Section 5.1 and Table 1] The experiments are reported as averages over five seeds, but no standard deviations, confidence intervals, or per-seed results are given. Several decisive differences are small (e.g., QQP 90.1 vs. 89.8 and SST-2 89.6 vs. 89.0 at 88% pruning), so without variance information the claim that PGB outperforms all compared methods on all seven GLUE tasks is not statistically supported. Please report variance or show that the margins are significant.
  5. [Section 5.1 and Table 1] All baselines were re-implemented by the authors, and no code is released for PGB or for the baseline re-implementations. Given that the superiority claim rests on these comparisons, the authors should release code and provide a validation protocol showing that their CoFi, DynaBERT, and EBERT reproductions match the originally reported accuracies within a small tolerance. This is especially unclear for CoFi and DynaBERT on SQuAD, since the paper says there is no publicly available code for those baselines on that benchmark.
  6. [Algorithm 2 and Algorithm 3] Algorithm 2 applies Permutation recursively to a shrinking submatrix, producing a sequence of permutations, whereas Algorithm 3 consumes a single pair (pi_r, pi_c) per weight matrix and indexes the groups as contiguous diagonal blocks. The paper does not explain how the per-group permutations are composed into the stored pi_r and pi_c, so the correctness of the re-permutation and of the PGB-Linear inference procedure is not established. Please specify the composition explicitly.
minor comments (6)
  1. [Section 5.3] The sentences 'When the Pruning rate surpasses 60%, RoBERTa demonstrates better performance compared to BERT' and 'However, as the sparsity ratio increases, the performance of BERT surpasses RoBERTa' are mutually contradictory; please clarify the intended claim.
  2. [Section 4] The claim of 'G times faster efficiency' should be stated as a FLOPs reduction for an individual matrix and should acknowledge that G varies across layers and that the permutation and concatenation operations in Algorithm 3 are not counted in the FLOPs comparison.
  3. [Section 3.2] The importance score I(.) is only referenced to [32,33,34]; please state the exact score formula used so the grouping step is reproducible.
  4. [References] Several reference entries have incorrect titles or authors (e.g., [6], [13], [14], and [32]); please re-check the entire bibliography against the original sources.
  5. [Figures 3 and 4] The axis labels in Figures 3 and 4 are garbled, with repeated or misplaced labels; please regenerate clean figures.
  6. [Algorithm 3] The notation in Line 2 of Algorithm 3 (eX <- -pi_c X) is unclear; please define the permutation operation used there.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PGB's central claims are empirical comparisons against external benchmarks, and the grouping heuristic is a correctness risk rather than a self-referential derivation.

full rationale

The paper proposes PGB, a one-shot semi-structured pruning method. Its central optimization is the permutation objective in Eq. (3), which maximizes the importance of the top-left block of a permuted weight matrix. This objective is solved with a heuristic borrowed from external work [16] and is explicitly described as sub-optimal in Section 3.2, but nothing in the formulation defines the predicted accuracy in terms of the method's own outputs. The reported results in Tables 1, 2, and 5 and Figures 3-5 are measured accuracies and F1 scores on held-out GLUE and SQuAD dev sets, compared against independently published baselines. No fitted parameter is renamed as a prediction; the adaptive group-count rule and the threshold tau are hyperparameters, and their sensitivity is reported in ablations. The inference cost analysis in Section 4 and Algorithm 3 is a straightforward arithmetic consequence of block-diagonal multiplication. Weight compensation in Eq. (4) is a reconstruction objective over retained group masks, not a circular definition of the pruning criterion. The paper does not rely on a self-citation chain: reference [16] is an external prior work, and no uniqueness theorem is imported from the present authors. The main risks identified by the reader - unvalidated 6-pass permutation heuristic, sensitivity to Gmax and tau, and absence of released code - are genuine empirical or correctness concerns, but they do not make the derivation circular. I find no definitional equivalence, no fitted-input-called-prediction step, and no load-bearing self-citation, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

PGB introduces no new physical entities. The method rests on three tuned hyperparameters (tau, Gmax, Nperm), on the assumption that second-order importance on 2K samples is trustworthy, and on a heuristic permutation with no quality guarantee. These are the main costs the reader pays beyond the standard tooling of BERT fine-tuning.

free parameters (3)
  • tau (importance threshold) = 1e-5
    Determines which weights count as important when choosing the number of groups; tuned over 1e-4, 1e-5, 1e-6 (Figure 7).
  • Gmax (max group number) = 6
    Limits group count per matrix; performance degrades at 3 and 8 (Figure 6), so the chosen value is tuned.
  • Nperm (permutation sort passes) = 6
    Number of alternating row/column sorts in the heuristic permutation; chosen by hand (Table A.2).
assumptions (4)
  • domain assumption Second-order importance scores computed on a 2K-sample subset of each task's training data reliably identify which weights to preserve.
    Section 5.1 states 2K samples are used; no analysis of the reliability of this estimate is provided.
  • domain assumption The alternating row/column sorting heuristic (from [16]) finds a permutation that clusters important weights well enough for the method to work.
    Section 3.2 calls the solution 'sub-optimal' and gives no approximation guarantees.
  • ad hoc to paper The adaptive group-number rule (set G < M*N / n_tau, with Gmax as a cap) produces matrices that meet the compression budget without excessive pruning.
    Section 3.2, 'Adaptive group numbers'; the rule is heuristic and its interaction with the global FLOPs budget is not derived.
  • domain assumption Re-finetuning with 3 epochs (same as original BERT fine-tuning) recovers most lost accuracy without distillation.
    The paper relies on re-finetuning to restore performance (Section 3.3) and provides no ablation on different re-finetuning budgets.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PGB: One-Shot Pruning for BERT via Weight Grouping and Permutation." pith.science (2026). https://pith.science/paper/AEJ63OVM

@misc{pith2026250203984,
  author       = {Pith},
  title        = {Pith review of: PGB: One-Shot Pruning for BERT via Weight Grouping and Permutation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AEJ63OVM}},
  note         = {Machine review of arXiv:2502.03984}
}
abstract

Large pretrained language models such as BERT suffer from slow inference and high memory usage, due to their huge size. Recent approaches to compressing BERT rely on iterative pruning and knowledge distillation, which, however, are often too complicated and computationally intensive. This paper proposes a novel semi-structured one-shot pruning method for BERT, called $\textit{Permutation and Grouping for BERT}$ (PGB), which achieves high compression efficiency and sparsity while preserving accuracy. To this end, PGB identifies important groups of individual weights by permutation and prunes all other weights as a structure in both multi-head attention and feed-forward layers. Furthermore, if no important group is formed in a particular layer, PGB drops the entire layer to produce an even more compact model. Our experimental results on BERT$_{\text{BASE}}$ demonstrate that PGB outperforms the state-of-the-art structured pruning methods in terms of computational cost and accuracy preservation.

Figures

Figures reproduced from arXiv: 2502.03984 by the authors.

Figure 1
Figure 1. Permutation and Grouping for BERT (PGB), where weight matrices are grouped [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The grouping procedure of PGB for weight matrices in BERT, which applies to [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Performance comparison with structured pruning methods varying the reduced [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Performance comparison with structured pruning methods varying the Reduced [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: PGB with BERTBASE and RoBERTaBASE Experiments on SQuAD Benchmarks. We compare additional pruning meth￾ods with PGB for SQuADv1.1 [20] and SQuADv2.0 [35]. The compared meth￾ods exclude knowledge distillation (KD) and data augmentation. Specifically, our approach is comp…
Figure 6
Figure 6. Figure 6: Ablation study with respect to the number of Group ( [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Ablation study with respect to threshold ( [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 35 canonical work pages

  1. [1]

    Gomez, Lukasz Kaiser, and Illia Polosukhin, Attention is All you Need , Advances in Neural Information Processing Systems, 2017, pp

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin, Attention is All you Need , Advances in Neural Information Processing Systems, 2017, pp. 5998–6008

    work page 2017

  2. [2]

    4171-–4186

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Proceedings of the 2019 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics, 2019, pp. 4171-–4186

    work page 2019

  3. [3]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov, RoBERTa: A Robustly Optimized BERT Pretraining Approach , CoRR, abs/1907.11692, 2019

    arXiv 1907

  4. [4]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    work page 2020

  5. [5]

    Dally, Learn- ing both Weights and Connections for Efficient Neural Network , Ad- vances in Neural Information Processing Systems, 2015, pp

    Song Han, Jeff Pool, Jeff Pool, John Tran, and William J. Dally, Learn- ing both Weights and Connections for Efficient Neural Network , Ad- vances in Neural Information Processing Systems, 2015, pp. 1135–1143. 20

    work page 2015

  6. [6]

    Hinton, Oriol Vinyals, and Jeffrey Dean

    Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Learning both Weights and Connections for Efficient Neural Network , Advances in Neural Information Processing Systems , CoRR, abs/1503.02531, 2015

    arXiv 2015

  7. [7]

    Lee, Namhoon, Thalaiyasingam Ajanthan, and Philip HS Torr, Snip: Single-shot network pruning based on connection sensitivity , arXiv preprint arXiv:1810.02340, 2018

    arXiv 2018

  8. [8]

    5058–5066

    Jian-Hao Luo, Jianxin Wu, and Weiyao Lin, Thinet: A filter level prun- ing method for deep neural network compression , In Proceedings of the IEEE international conference on computer vision, 2017, pp. 5058–5066

    work page 2017

Show all 45 references
  1. [9]

    15834-15846

    Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin, The lottery ticket hypothesis for pretrained bert networks, Advances in neural information processing systems, 2020, 33:pp. 15834-15846

    2020

  2. [10]

    Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Li, Dynabert: Dynamic BERT with adaptive width and depth , In Advances in neural information processing systems, 2020, 33

    2020

  3. [11]

    Rush, Block Pruning For Faster Transformers , In Empirical Methods in Nat- ural Language Processing (EMNLP), 2021, pp

    Fran¸ cois Lagunas, Ella Charlaix, Victor Sanh, and Alexander M. Rush, Block Pruning For Faster Transformers , In Empirical Methods in Nat- ural Language Processing (EMNLP), 2021, pp. 10619–10629

    2021

  4. [12]

    1513–1528

    Mengzhou Xia, Zexuan Zhong, and Danqi Chen, Structured Pruning Learns Compact and Accurate Models, In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 1513–1528

    2022

  5. [14]

    Xiaoqi Jiao , Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang and Qun Liu, Structured Pruning Learns Compact and Accurate Models, CoRR, abs/1910.01108, 2019

    1910 arXiv

  6. [15]

    Zhuo Su, Linpu Fang, Wenxiong Kang, Dewen Hu Matti Pietik¨ ainen, and Li Liu, Dynamic group convolution for accelerating convolutional 21 neural networks, In Computer Vision–ECCV 2020: 16th European Con- ference, 2020, pp. 138–155

    2020

  7. [16]

    1961–1970

    Ruizhe Zhao and Wayne Luk, Efficient structured pruning and architec- ture searching for group convolution , In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 2019, pp. 1961–1970

    2019

  8. [17]

    6883— 6893

    Sungrae Park, Geewook Kim, Junyeop Lee, Junbum Cha, Ji-Hoon Kim, and Hwalsuk Lee, Scale down Transformer by Grouping Features for a Lightweight Character-level Language Model, Proceedings of the 28th In- ternational Conference on Computational Linguistics, 2020, pp. 6883— 6893

    2020

  9. [18]

    Ivan Chelombiev, Daniel Justus, Douglas Orr, Anastasia Dietrich, Frithjof Gressmann, Alexandros Koliousis, and Carlo Luschi, Group- bert: Enhanced transformer architecture with efficient grouped struc- tured, arXiv preprint arXiv:2106.05822, 2021

    2021 arXiv

  10. [19]

    Bowman, GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding , In International Confer- ence on Learning Representations (ICLR), 2019

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman, GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding , In International Confer- ence on Learning Representations (ICLR), 2019

    2019

  11. [20]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang, SQuAD: 100, 000+ Questions for Machine Comprehension of Text , In Empirical Methods in Natural Language Processing (EMNLP), 2016

    2016

  12. [21]

    5776–5788

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou, MiniLM: Deep Self-Attention Distillation for Task-Agnostic Com- pression of Pre-Trained Transformers, In Advances in Neural Informa- tion Processing Systems, 2020, pp. 5776–5788

    2020

  13. [22]

    2158– 2170

    Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou, MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices, In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 2020, pp. 2158– 2170. 22

    2020

  14. [23]

    Victor Sanh, Thomas Wolf, and Alexander Rush, Movement pruning: Adaptive sparsity by fine-tuning , Advances in Neural Information Pro- cessing Systems, 2020, 33: pp.20378–20389

    2020

  15. [24]

    Paul Michel, Omer Levy, and Graham Neubig, Are sixteen heads really better than one? , Advances in Neural Information Processing Systems (Volume 32), 2019

    2019

  16. [25]

    Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov, Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned , arXiv preprint arXiv:1905.09418, 2019

    1905 arXiv

  17. [26]

    Zhengyan Zhang, Fanchao Qi, Zhiyuan Liu, Qun Liu, and Maosong Sun, Know what you don ’t need: Single-Shot Meta-Pruning for attention heads, AI Open, 2021, 2: pp. 36–42

    2021

  18. [27]

    Angela Fan, Edouard Grave, and Armand Joulin, Reducing Transformer Depth on Demand with Structured Dropout , CoRR, abs/1909.11556, 2019

    1909 arXiv

  19. [28]

    Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov, Poor Man ’s BERT: Smaller and Faster Transformer Models , CoRR, abs/2004.03844, 2020

    2004 arXiv

  20. [29]

    Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Zhangyang Wang, and Jingjing Liu, Earlybert: Efficient bert training via early-bird lottery tickets, arXiv preprint arXiv:2101.00063, 2020

    2020 arXiv

  21. [30]

    1231–1240

    Yani Ioannou, Duncan Robertson, Roberto Cipolla, and Antonio Crim- inisi, Deep roots: Improving cnn efficiency with hierarchical filter groups, Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1231–1240

    2017

  22. [31]

    8847–8856

    Guotian Xie, Jingdong Wang, Ting Zhang, Jianhuang, Lai, Richang Hong, and Guo-Jun Qi, Interleaved structured sparse convolutional neu- ral networks, In Proceedings of the IEEE Conference on Computer Vi- sion and Pattern Recognition, 2018, pp. 8847–8856. 23

    2018

  23. [32]

    Stork, and Gregory J

    Babak Hassibi, David G. Stork, and Gregory J. Wolff, Interleaved struc- tured sparse convolutional neural networks , In Proceedings of Interna- tional Conference on Neural Networks (ICNN’88), San Francisco, CA, USA, March 28 - April 1, 1993, pp. 293–299

    1993

  24. [33]

    11264–11272

    Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz , Importance Estimation for Neural Network Pruning , In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20,2019, pp. 11264–11272

    2019

  25. [34]

    Sidak Pal Singh and Dan Alistarh, WoodFisher: Efficient Second-Order Approximation for Neural Network Compression , In Advances in Neu- ral Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020

    2020

  26. [35]

    Rajpurkar, Pranav and Jia, Robin and Liang, Percy, Know what you don ’t know: Unanswerable questions for SQuAD , arXiv preprint arXiv:1806.03822, 2018

    2018 arXiv

  27. [36]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Brad- bury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas K¨ opf, Edward Z. Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner,...

    1912 arXiv

  28. [37]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien, Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R´ emi Louf, Morgan Funtowicz,Transformers: State-of-the-art natural language pro- cessing, In Proceedings of the 2020 conference on empirical methods in natur...

    2020

  29. [38]

    4814–4823

    Liu, Zejian and Li, Fanrong and Li, Gang and Cheng, Jian, EBERT: Efficient BERT inference with dynamic structured pruning , Findings of the Association for Computational Linguistics: ACL-IJCNLP, 2021, pp. 4814–4823. 24

    2021

  30. [39]

    Zi Lin, Jeremiah Z. Liu, Zi Yang, Nan Hua, and Dan Roth, Pruning Re- dundant Mappings in Transformer Models via Spectral-Normalized Iden- tity Prior , Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November, 2020, pp. 719–730

    2020

  31. [40]

    6848–6856

    Zhang, Xiangyu and Zhou, Xinyu and Lin, Mengxiao and Sun, Jian, Shufflenet: An extremely efficient convolutional neural network for mo- bile devices, In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 6848–6856

    2018

  32. [41]

    26809–26823

    Qingru Zhang, Simiao Zuo, Chen Liang, Alexander Bukharin, Pengcheng He, Weizhu Chen, and Tuo Zhao, Platon: Pruning large transformer models with upper confidence bound of weight importance , In Proceedings of the IEEE conference International Conference on Ma- chine Learning, ...

    2022

  33. [42]

    Manning, Andrew Y

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts, Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank , In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Pr...

    2013

  34. [43]

    Bowman, Neural Net- work Acceptability Judgments, Transactions of the Association of Com- putational Linguistics (TACL), 2019, 7: pp

    Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman, Neural Net- work Acceptability Judgments, Transactions of the Association of Com- putational Linguistics (TACL), 2019, 7: pp. 625–641

    2019

  35. [44]

    Cer, Mona T

    Daniel M. Cer, Mona T. Diab, Eneko Agirre, I˜ nigo Lopez-Gazpio, and Lucia Specia, SemEval-2017 Task 1: Semantic Textual Similarity Multi- lingual and Crosslingual Focused Evaluation, In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), 2017...

    2017

  36. [45]

    William B. Dolan and Chris Brockett, I˜ nigo Lopez-Gazpio, and Lucia Specia, Automatically Constructing a Corpus of Sentential Paraphrases , In Proceedings of the Third International Workshop on Paraphrasing, IWP@IJCNLP 2005, 2005. 25 Appendix A. Experimental Details Appendix ...

    2005

  37. [46]

    Each stage involves 20 epochs of training with layer-wise distillation

    consists of 2 stages: pruning and final finetuning. Each stage involves 20 epochs of training with layer-wise distillation. BMP [11] selects appropri- ate components to prune at the block level over 20 epochs training stages. Subsequently, prediction layer distillation is perf...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.