Pith. sign in

REVIEW 4 major objections 4 minor 32 references

MultiPruner: Balanced Structure Removal in Foundation Models

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MultiPruner prunes LLMs on three axes and reports better zero-shot accuracy than block-only pruning at comparable compression.

desk verdict A solid, incremental pruning recipe that mostly delivers what it claims; the main caveat is that the headline result is partly in-sample, but the paper deserves serious refereeing. read the letter →

arxiv 2501.09949 v1 pith:H2EM36ZS submitted 2025-01-17 cs.LG cs.AI

J. Pablo Muñoz , Jinjie Yuan , Nilesh Jain This is my paper · ORCID
classification cs.LGcs.AI
keywords structuredpruningtraining-freecompressionlargelanguagemodelsresidualblockMLPchannelattentionheadperplexityimportancezero-shotevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MultiPruner is a training-free structured pruning recipe for large language models. It tries to establish that pruning a Transformer along three dimensions—residual blocks, MLP channels, and attention heads—produces smaller models that retain more zero-shot accuracy than pruning any single dimension, and that this multidimensional balance is best achieved by pruning coarse structures first and fine ones later. If the paper is right, a standard 22% compression of Llama2-7B can actually improve average downstream accuracy by 2.66 points over BlockPruner at the same compression, with lower perplexity. The care a reader should take: the recipe's gains rest on using a text-likelihood score on a small calibration set as a universal importance signal.

What carries the argument

The load-bearing object is the minimal residual block—either the self-attention or the MLP sub-block within a Transformer layer—treated as the atomic unit for depth pruning, plus a weight-reordering step that sorts channel groups and heads by L1 norm so the width-pruning stages can remove the least important slices in groups. Importance is measured by the model's perplexity on 128 or 256 Alpaca calibration samples: each candidate removal is scored by the perplexity of the masked model, and the candidate with the smallest rise is removed. A fixed allocation of the total pruning ratio, Block:MLP Channel:Attention Head = 44:52:4, was discovered on Llama2-7B and then applied across models, based on sensitivity studies showing that attention heads degrade quickly beyond a small pruning budget while MLP channels tolerate much heavier pruning.

What would settle it

Run MultiPruner at 22% on Llama2-7B but replace the Alpaca calibration set with WikiText-2, a code corpus, or a task-mixed set for the importance scores, keeping everything else fixed; if the chosen structures differ from the paper's reported pruned model and the average zero-shot score drops by more than a few points relative to 62.83, the transferability of the calibration signal is undermined. Alternatively, search the Block:MLP:Attention ratio weights on Llama3.1-8B from scratch and compare the resulting model against the fixed 44:52:4 recipe at the same total ratio; the paper's cross-model generality stands or falls on that comparison.

Watch

Extended reading notes

Core claim

The paper claims that block-level pruning leaves redundancy on the table, and that reallocating a fixed pruning budget across depth and width yields a strictly better compressed model. Concretely, MultiPruner removes the least important minimal residual blocks (self-attention or MLP) one at a time until block pruning accounts for 44% of the target ratio, reorders weights by L1 norm so dispensable channels move to the end, then prunes MLP channel groups and attention heads until the remaining 56% of the budget is spent, with attention heads capped at about 4%. On Llama2-7B at 21.96% total pruning, this produces WikiText-2 perplexity 9.33 and an average score of 62.83 across five zero-shot tasks, against BlockPruner's 11.51 and 60.17 at 21.99%. The same pattern is reported for Llama2-13B, Llama3, Qwen, and Baichuan models, and an evolutionary-search variant occasionally trades a little perplexity for a higher average score.

Load-bearing premise

The method assumes that a text-likelihood score measured on 128 to 256 Alpaca samples tells which blocks, MLP channels, and attention heads can be safely removed for unrelated downstream tasks, and that the 44:52:4 split of the pruning budget stays near-optimal across different model families and target compression levels.

Editorial extensions

If this is right

  • At a fixed total pruning ratio, pruning blocks, MLP channel groups, and attention heads in that coarse-to-fine order yields lower perplexity and higher average zero-shot accuracy than block-only pruning on the Llama2, Llama3, Qwen, and Baichuan models tested.
  • Removing any of the three pruning stages degrades results; attention-head pruning alone is catastrophic, while block-plus-MLP-channel pruning without heads still outperforms block-only pruning on Llama2-7B.
  • The sensitivity analysis implies an allocation rule: keep attention-head pruning near 4% of the total ratio, give MLP channels roughly half the budget, and let block pruning take the remainder.
  • Weight reordering by L1 norm before width pruning is a necessary step; skipping it raises perplexity from 9.33 to 10.78 on Llama2-7B at 22%.
  • A 22%-pruned Llama2-7B can be deployed with roughly 1.3x prefill and decode speedups on a CPU, and two epochs of LoRA recovery tuning lift its average score from 62.83 to 64.18.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the 44:52:4 split was tuned on Llama2-7B and the appendix notes that GQA models may need fewer removed attention heads, the recipe likely needs per-architecture recalibration; a cheap test is to grid-search only the MLP versus block split on a GQA model and see whether the Pareto front matches the paper's ratio.
  • The paper's ablations suggest most of the accuracy gain may come from MLP channel pruning rather than the presence of three dimensions; isolating that by comparing block-plus-MLP against block-plus-head at equal total ratio would sharpen the causal story.
  • Since perplexity and downstream accuracy do not always move together, calibration perplexity is a proxy rather than the objective; a testable extension is to run the same pipeline with an importance metric tuned to downstream task loss and compare the selected structures.
  • The sequential coarse-to-fine ordering hints that block importance and channel importance are not independent; a joint search that re-evaluates channel importance after each block removal might dominate the fixed-order recipe at higher compression ratios.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes MultiPruner, a training-free structured pruning method for large language models that sequentially removes residual blocks (depth), MLP channel groups (width), and attention heads (width), with a weight-reordering step before width pruning. The method is evaluated on 10 LLMs (Llama2, Llama3, Qwen, Baichuan) at pruning ratios around 10-25%, comparing against BlockPruner, ShortGPT, SliceGPT, LaCo, and Relative Magnitude on Wikitext2 perplexity and five zero-shot tasks. The authors report that MultiPruner achieves the best perplexity and average score on 9 of 10 models, including a reproduced BlockPruner baseline, and provide ablations on pruning order, dimension existence, ratio sensitivity, weight reordering, inference speedups, and a LoRA-based recovery stage.

Significance. If the results hold, MultiPruner is a useful contribution to training-free structured pruning: it shows that combining depth and width pruning can beat depth-only block pruning at similar parameter ratios, and it provides a practical recipe (sequential coarse-to-fine, weight reordering, ratio weights). The paper's strengths include a reproduced BlockPruner baseline, broad model coverage, and explicit ablations. However, the central claim of cross-model superiority is currently weakened by the fact that the headline hyperparameters were selected on the same Llama2-7B model used as the main evidence, and no uncertainty estimates are provided.

major comments (4)
  1. [Section 3.5/3.6, Figure 4, Table 1] The Block:MLP:Attention ratio weights (44:52:4) and the coarse-to-fine pruning order are selected via ablations on Llama2-7B at a 22% target ratio (Figure 4, Table 4), and the same Llama2-7B row is then presented in Table 1 as the primary evidence of superiority over BlockPruner (average score 62.83 vs 60.17). This is an in-sample selection for the flagship result. To support the cross-model claim, the authors should provide out-of-sample validation, e.g., tune the configuration on a different model (or use nested selection on a holdout model) and then report results on Llama2-7B, or alternatively report results over multiple calibration subsets to show the 2.66-point gap is not a selection artifact.
  2. [Tables 1, 2, 3, 9] No error bars, standard deviations, or multiple runs are reported for any of the zero-shot accuracy or perplexity numbers. Since evaluation is on finite benchmark sets and the calibration subset is randomly drawn from Alpaca, differences of 1-2 points (e.g., PIQA 74.65 vs 74.21 in Table 1) may be within random variation. The authors should report variance across at least a few calibration subsets or seeds, or explicitly justify why the reported numbers are deterministic.
  3. [Table 1, Baichuan2-13B row] For Baichuan2-13B, MultiPruner achieves lower Wikitext2 perplexity (10.99 vs 15.36) but a lower average zero-shot score (59.12 vs 59.47) than BlockPruner. The abstract and Section 3.2 state that MultiPruner 'surpasses' or 'outperforms' other training-free pruning methods without qualification. The authors should explicitly acknowledge this exception and discuss its implications for the generality of the method, rather than relying on the word 'most' in a single sentence.
  4. [Table 10 and Section 3.1] The paper states that for GQA models MultiPruner 'generally avoids pruning attention heads' and that ratio weights 'may vary across different LLMs.' This means the method is not a fixed configuration but requires model-specific hyperparameter choices. The manuscript should provide a principled procedure for selecting these weights on a new model (e.g., based on model architecture or a small validation set), or clearly disclose the per-model tuning cost. Otherwise the claim of a 'training-free' method that works 'across various large pre-trained models' is overstated.
minor comments (4)
  1. [Algorithm 1] The thresholds τ1, τ2, and τ3 are cumulative (as clarified in Table 10), but the pseudocode uses a single τ that is overwritten. Adding a comment that τt are cumulative pruning-ratio thresholds would prevent confusion.
  2. [Section 1] The sentence 'MultiPruner removes this assumption and demonstrate that' contains a subject-verb agreement error; it should be 'demonstrates'.
  3. [Section 2.2] The 'Performance-Recovery Stage' is only one sentence long; since Table 5 reports a LoRA-based recovery result, this section should be expanded to describe the setup (e.g., which layers are adapted, training budget) or moved to the experimental section.
  4. [Section 3.4 and Algorithm 2] The evolutionary search variant (MultiPruner-Evol) is introduced, but the search space S and the number of evaluations N are not specified in the main text or Appendix D. Adding these details would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No construction-level circularity: pruning importance is calibration-set PPL while reported success is Wikitext2 PPL and external zero-shot tasks, so the evaluation is not the fitted objective.

full rationale

The method's selection signal is perplexity on the Alpaca calibration set ('The importance metric used for all pruning steps is the perplexity (PPL) on the calibration dataset'), while the reported evidence is Wikitext2 PPL and five LM-eval-harness zero-shot tasks ('regarding evaluation, MultiPruner computes the perplexity (PPL) of Wikitext2'). These are different datasets, so the reported metrics are not forced by the optimization by construction. The only load-bearing design choice is the Block:MLP:Attention ratio weight 44:52:4, which was selected on Llama2-7B at a 22% target ratio and then used to produce the Llama2-7B row in Table 1 ('The yellow point represents the ratio weight we adopted in the most experimental results, which is Block : MLP Channel : Attention Head = 44% : 52% : 4%.'); this is an in-sample hyperparameter-selection / benchmark-contamination concern, not a circular derivation, because downstream accuracy and Wikitext2 PPL are external to that selection and could in principle disagree with it. There is one self-citation (Muñoz et al., 2024) in a list of PEFT examples, but it is not load-bearing and does not support any premise of MultiPruner. No equation equates the calibration objective with the evaluation metric, no fitted parameter is renamed as a prediction, and no uniqueness claim is imported from the authors' prior work. The Baichuan2-13B row (BlockPruner 59.47 vs MultiPruner 59.12) further shows the advantage is not guaranteed, which is consistent with an empirical method rather than a definitionally forced result.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on an empirical pruning recipe rather than a mathematical derivation. The main free parameters are the per-dimension pruning ratio weights and the granularity choices; the main axioms are domain assumptions about PPL-based importance and the validity of removing residual sub-blocks. No invented entities are introduced.

free parameters (6)
  • Block:MLP:Attention pruning ratio weights = 44% : 52% : 4% of target ratio
    Chosen to maximize performance on Llama2-7B at 22% (Figure 4, Appendix D) and applied to most other models; selection is based on the same evaluation metrics reported.
  • MLP channel group size g_MLP = hidden size // 4 (e.g., 1024 for Llama2-7B)
    Hand-chosen granularity for channel pruning in Algorithm 1; not justified by a separate criterion.
  • Attention head group size g_ATTN = one head
    Hand-chosen; attention heads pruned one at a time.
  • Calibration set sizes = 256 samples for blocks, 128 for width
    Chosen by hand in Table 10; results may depend on this choice.
  • Weight reorder importance metric = L1 norm
    Selected by ablation in Appendix A (Table 7) on Llama2-7B; the paper notes it outperforms Wanda reordering.
  • Depth stopping heuristic for evolutionary variant = tau / 2
    Algorithm 2 lines 1-2 use half the target pruning ratio as the depth pruning stop, an experimentally discovered heuristic.
assumptions (4)
  • domain assumption Perplexity on 128-256 Alpaca calibration samples is a reliable global importance metric for residual blocks, MLP channels, and attention heads.
    Algorithm 1 uses PPL for all importance scores; if calibration PPL is not predictive of downstream zero-shot accuracy, the pruning decisions are misdirected.
  • domain assumption Transformer minimal residual blocks (attention and MLP) can be removed independently without breaking the residual stream.
    Inherited from BlockPruner (Zhong et al., 2024); Section 2 adopts BlockPruner's minimal residual block formulation.
  • domain assumption L1-norm weight reordering prior to pruning channels and heads preserves model function enough for iterative importance scoring.
    Appendix A ablation supports this empirically, but the mechanism is assumed; reordering changes weight layout.
  • domain assumption Zero-shot accuracy averaged over PIQA, WinoGrande, HellaSwag, ARC-e, and ARC-c is a meaningful proxy for pruned model quality.
    All comparisons use this average (Tables 1-3); the limitation section acknowledges accuracy does not capture all model behaviors.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MultiPruner: Balanced Structure Removal in Foundation Models." pith.science (2026). https://pith.science/paper/H2EM36ZS

@misc{pith2026250109949,
  author       = {Pith},
  title        = {Pith review of: MultiPruner: Balanced Structure Removal in Foundation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H2EM36ZS}},
  note         = {Machine review of arXiv:2501.09949}
}
read the original abstract

Recently, state-of-the-art approaches for pruning large pre-trained models (LPMs) have demonstrated that the training-free removal of non-critical residual blocks in Transformers is viable for reducing model size, achieving results that outperform previous training-free pruning approaches. Motivated by these findings, we extend BlockPruner (Zhong et al., 2024) and propose MultiPruner, a pruning approach that surpasses recent training-free pruning methods by adopting a multidimensional, iterative, fine-grained pruning strategy. In MultiPruner, multidimensional pruning reinstates the structural balance in block-pruned models by sequentially compressing along three dimensions: i) residual blocks, ii) channels of multilayer perceptrons (MLP), and iii) attention heads. This solution enhances zero-shot accuracy on downstream tasks compared to other techniques while improving model compression ratios, producing compressed models with fewer computing and memory requirements. Extensive experiments demonstrate the advantages of the proposed method across various large pre-trained models. The code and pruning configurations are available at https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning.

Figures

Figures reproduced from arXiv: 2501.09949 by the authors.

Figure 1
Figure 1. MultiPruner adopts a multidimensional fine-grained pruning method to make pruning more balanced, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Search progression for evolutionary search [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Comparison of BlockPruner, MultiPruner and [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The results of increasing/decreasing the weight of the target pruning ratio allocated to pruning MLP [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Details of the pruned Llama2-7B model obtained by MultiPruner, including the width of the self-attention [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Visualization of Table [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 6 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2024. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774

    arXiv 2024

  4. [4]

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebr \'o n, and Sumit Sanghai. 2023. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245

    arXiv 2023

  5. [5]

    Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman

    Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. 2024. https://openreview.net/forum?id=vXxardq6db Slice GPT : Compress large language models by deleting rows and columns . In The Twelfth International Conference on Learning Representations

    2024

  6. [6]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

    arXiv 2023

  7. [7]

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence

    2020

  8. [8]

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. https://crfm.stanford.edu/assets/report.pdf On the opportunities and risks of foundation models . ArXiv

    work page 2021

Show all 32 references
  1. [9]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. https://api.semanticscholar.org/CorpusID:3922816 Think you have solved question answering? try arc, the ai2 reasoning challenge . ArXiv, abs/1803.05457

    2018 arXiv

  2. [10]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  3. [11]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2023 doi

  4. [12]

    Mahoney, and Kurt Keutzer

    Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W. Mahoney, and Kurt Keutzer. 2021. https://arxiv.org/abs/2103.13630 A survey of quantization methods for efficient neural network inference . CoRR, abs/2103.13630

    2021 arXiv

  5. [13]

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. https://arxiv.org/abs/1503.02531 Distilling the knowledge in a neural network . Preprint, arXiv:1503.02531

    2015 arXiv

  6. [14]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR)

    2022

  7. [15]

    Fran c ois Lagunas, Ella Charlaix, Victor Sanh, and Alexander Rush. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.829 Block pruning for faster transformers . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10619--10629, Onli...

    2021 doi

  8. [16]

    Yann LeCun, John Denker, and Sara Solla. 1989. https://proceedings.neurips.cc/paper_files/paper/1989/file/6c9882bbac1c7093bd25041881277658-Paper.pdf Optimal brain damage . In Advances in Neural Information Processing Systems, volume 2. Morgan-Kaufmann

    1989

  9. [17]

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. https://openreview.net/forum?id=J8Ajf9WfXP LLM -pruner: On the structural pruning of large language models . In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  10. [18]

    Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. 2024. https://arxiv.org/abs/2403.03853 Shortgpt: Layers in large language models are more redundant than you expect . Preprint, arXiv:2403.03853

    2024 arXiv

  11. [19]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843

    2016 arXiv

  12. [20]

    Pablo Muñoz, Jinjie Yuan, and Nilesh Jain

    J. Pablo Muñoz, Jinjie Yuan, and Nilesh Jain. 2024. https://aclanthology.org/2024.findings-emnlp.749 SQFT : Low-cost model adaptation in low-precision sparse foundation models . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 12817--12832, Miami...

    2024

  13. [21]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. https://doi.org/10.1145/3474381 Winogrande: An adversarial winograd schema challenge at scale . Commun. ACM, 64(9):99–106

    2021 doi

  14. [22]

    Mohammad Samragh, Mehrdad Farajtabar, Sachin Mehta, Raviteja Vemulapalli, Fartash Faghri, Devang Naik, Oncel Tuzel, and Mohammad Rastegari. 2023. https://arxiv.org/abs/2312.09299 Weight subcloning: direct initialization of transformers using larger pretrained ones . Preprint, ...

    2023 arXiv

  15. [23]

    Victor Sanh, Thomas Wolf, and Alexander M. Rush. 2020. Movement pruning: adaptive sparsity by fine-tuning. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA. Curran Associates Inc

    2020

  16. [24]

    Zico Kolter

    Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. 2023. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695

    2023 arXiv

  17. [25]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  18. [26]

    Peng Xu, Wenqi Shao, Mengzhao Chen, Shitao Tang, Kaipeng Zhang, Peng Gao, Fengwei An, Yu Qiao, and Ping Luo. 2024. https://arxiv.org/abs/2402.16880 Besa: Pruning large language models with blockwise parameter-efficient sparsity allocation . Preprint, arXiv:2402.16880

    2024 arXiv

  19. [27]

    Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. 2023. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305

    2023 arXiv

  20. [28]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024 a . Qwen2 technical report. arXiv preprint arXiv:2407.10671

    2024 arXiv

  21. [29]

    Yifei Yang, Zouying Cao, and Hai Zhao. 2024 b . https://arxiv.org/abs/2402.11187 Laco: Large language model pruning via layer collapse . Preprint, arXiv:2402.11187

    2024 arXiv

  22. [30]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

    2019

  23. [31]

    Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, and Bohan Zhuang. 2024. https://openreview.net/forum?id=9KVT1e1qf7 Lo RAP rune: Pruning meets low-rank parameter-efficient fine-tuning

    2024

  24. [32]

    Longguang Zhong, Fanqi Wan, Ruijun Chen, Xiaojun Quan, and Liangzhi Li. 2024. https://arxiv.org/abs/2406.10594 Blockpruner: Fine-grained pruning for large language models . Preprint, arXiv:2406.10594

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.