Pith. sign in

REVIEW 4 major objections 5 minor 27 references

Pangu Light: Weight Re-Initialization for Pruning and Accelerating LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper's central claim is that weight re-initialization, not component selection, is what makes aggressive joint width and depth pruning of LLMs recoverable.

desk verdict Real signal in the controlled ablation, but the headline results are too confounded to prove the 'missing piece' claim. read the letter →

arxiv 2505.20155 v1 pith:XBWBH3ZD submitted 2025-05-26 cs.CL

Hanting Chen , Jiarui Qin , Jialong Guo , Tao Yuan , Yichun Yin , Huiling Zhen , Yasheng Wang , Jinpeng Li
show 10 more authors
Xiaojun Meng Meng Zhang Rongju Ruan Zheyuan Bai Yehui Tang Can Chen Xinghao Chen Fisher Yu Ruiming Tang Yunhe Wang (and Other Contributors)
This is my paper
classification cs.CL
keywords structuredpruningweightre-initializationLLMcompressioncross-layerattentionLayerNormstabilizationknowledgedistillationinferenceaccelerationGroupedQuery
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the reason aggressive joint pruning of large language models often collapses is not mainly which layers or channels get cut, but the state the surviving weights are left in. Its central proposal is a dedicated weight re-initialization phase immediately after structured pruning, before any recovery training, that restabilizes the network and preserves information from pruned components. On Pangu-38B, the authors report that this phase lets the pruned models recover nearly all of the original accuracy: Pangu Light-1.6x scores 81.6 on the reasoning benchmark average versus 82.0 for the unpruned model, while decoding at 2585 tokens/s versus 1631. A reader should care because the claim, if true, means the practical bottleneck in LLM compression is a fixable training-dynamics problem rather than a fundamental limit of pruning.

What carries the argument

CLAP and SLNP are weight re-initialization mechanisms, not importance metrics. CLAP's unit of transfer is the key-value group: within a consecutive layer pair where the second layer is pruned, each KV group's score is the average importance of its surviving query heads, all groups from both layers are ranked together, and the top groups are merged into the kept layer with their query, key, value, and output projection parameters. SLNP's unit is a per-layer scalar, $c_l = \|\gamma^{\mathrm{orig}}_l\|_2 / \|\gamma^{\mathrm{pruned}}_l\|_2$, applied as $\gamma^{\mathrm{new}}_l = c_l \times \gamma^{\mathrm{pruned}}_l$ to restore the RMSNorm output scale after width pruning. The third mechanism, Post-RMSNorm absorption, replaces a converged post-module RMSNorm with a constant per-channel scaling $\gamma^{\mathrm{abs}} = \bar{s}_{\mathrm{inv}} \times \gamma$, where $\bar{s}_{\mathrm{inv}}$ is the average inverse RMS scaling over a calibration set, and fuses that scaling into the preceding projection matrix so the normalization layer becomes zero-cost at inference.

What would settle it

Hold the base model, prune budget, recovery data, and compute budget fixed, and run the pipeline twice: once with CLAP and SLNP and once with the same pruning decisions but naive weight retention. If the benchmark gap between the two variants is within noise, the claim that weight re-initialization is the missing piece collapses.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that two re-initialization operations, Cross-Layer Attention Pruning (CLAP) and Stabilized LayerNorm Pruning (SLNP), are the missing piece that makes aggressive width-plus-depth pruning viable. CLAP handles depth reduction: instead of discarding a pruned layer's attention entirely, it ranks all key-value groups in the keep-prune layer pair by importance, transfers the top groups from the pruned layer into the kept layer, and re-initializes that kept layer's attention parameters group-wise. SLNP handles width reduction: after channel pruning shrinks each RMSNorm affine vector, it rescales the remaining gamma values by the ratio of original to pruned norm, restoring the layer's output magnitude so activations do not destabilize. The paper shows these moves on top of the Minitron baseline raise the average benchmark score from 30.2 to 33.8 in an 11B ablation, and the production models keep 98.9 percent of Pangu-38B's average score at 2.1x acceleration.

Load-bearing premise

The load-bearing premise is that the calibration set used for importance scoring and for the Post-RMSNorm absorption statistic represents real deployment data, and that those statistics remain valid after weights are re-initialized and fine-tuned; the paper never specifies that set's size, source, or composition.

Editorial extensions

If this is right

  • If the central claim is right, pruning recipes should stop treating component selection as the whole game; a short re-initialization step before recovery training is what unlocks aggressive compression ratios.
  • The 1.6x-pruned Pangu Light model is claimed to dominate a same-size Qwen3-32B on both accuracy (81.6 versus 80.9) and throughput (2585 versus 2225 tokens/s on Ascend NPUs).
  • The 4.2x-pruned variant still scores 79.6 average, above Qwen3-14B's 78.6, so the method's gains persist at much higher compression.
  • Post-RMSNorm absorption is claimed to recover almost all of the Sandwich-Norm performance (59.0 versus 59.9) while eliminating the norm's inference cost through fusion into adjacent weights.
  • The whole recipe, including the choice of which architectures to keep, is co-designed with Ascend NPU cost characteristics, meaning the throughput numbers are hardware-specific.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: CLAP depends only on grouped-query attention structure and per-head importance scores, so it should transfer to other GQA-based transformers without architectural changes; the paper only demonstrates it on Pangu models.
  • Editorial inference: SLNP's single global rescale per layer attributes the width-pruning breakdown to scale loss, but the same result could come from improved conditioning; a control that rescales to a random-but-matched norm would discriminate these explanations.
  • Editorial inference: the norm-absorption statistic $\bar{s}_{\mathrm{inv}}$ is frozen after calibration, so if deployment prompts have materially different activation norms the absorbed scaling will drift away from true RMSNorm behavior; monitoring this statistic across domains is a cheap validity check.
  • Editorial inference: since the calibration set is unspecified, ablating its size, domain mix, and sequence length would test how robust the pruning choices and the absorbed scalings are; the paper's reported margins may shrink or grow with that coverage.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Pangu Light, a structured pruning framework for LLMs that couples multi-axis importance-based pruning (channels, attention heads, FFN neurons, and layers) with two weight re-initialization techniques: CLAP, which merges the most important KV groups from a pruned layer into the preceding layer, and SLNP, which rescales RMSNorm affine parameters after channel pruning. It also introduces a Post-RMSNorm absorption technique for Sandwich-Norm architectures, replacing the dynamic normalization with a constant channel-wise scaling that can be fused into the preceding projection matrix. The method is evaluated by pruning Pangu-38B into 32B/2.1x/4.2x variants, reporting accuracy on six reasoning benchmarks and throughput on Ascend NPUs, plus an ablation on an 11B model and a 14B norm-absorption study. The central claim is that CLAP and SLNP are the 'missing piece' that makes aggressive joint width and depth pruning viable.

Significance. If the central claim holds, the paper makes a useful contribution by identifying weight re-initialization as an under-explored component in joint structured pruning, with two concrete techniques (CLAP and SLNP) and a hardware-aware norm absorption that is shown to be nearly lossless in Table 4. The transparent importance metrics and the controlled internal ablations (Tables 3 and 4) are strengths, as is the evaluation against comparable open models. However, the paper provides no code or artifacts, and the headline results bundle the proposed re-initialization with 300B-token annealing, KD, and post-training, so the causal attribution is not yet established at deployment scale. The missing specification of the calibration set further limits reproducibility.

major comments (4)
  1. [Section 3.2 and Table 2 vs Table 3] The load-bearing claim that CLAP and SLNP are the 'missing piece' for aggressive joint pruning is supported only by Table 3, which is an 11B model fine-tuned on 21B tokens and evaluated on LAMBADA/WPLC/MMLU/C-Eval/BigBench/HumanEval. The Table 2 results for Pangu Light-1.6x/2.1x/4.2x come from a pipeline that also includes 300B-token annealing with online KD and a 2M-instance post-training phase with a curated reasoning-heavy data mixture (Section 4.2). Since these components can independently add substantial capability, the 81.6/81.1/79.6 averages do not isolate CLAP/SLNP. Please provide a same-scale control that keeps the identical annealing/KD/post-training recipe but removes CLAP and SLNP, or explicitly reframe the central claim as 're-initialization contributes, and the full pipeline achieves...'.
  2. [Section 3.1, Section 3.3, Eqs. (1)-(4), (8)] The calibration set C is never specified: its size, source, domain composition, and token count are absent. All importance scores in Eqs. (1)-(4) and the expected inverse norm in Eq. (8) depend on C. If C is not representative of the deployment distribution, both the component selection and the frozen norm scalings can silently degrade the model. Please state how C was constructed and report sensitivity to C (e.g., varying its size or domain mix) to show the method is robust to this choice.
  3. [Section 3.3, Eqs. (8)-(10)] The text calls the absorbed Post-RMSNorm operation 'mathematically equivalent' and claims it preserves model integrity, but Eq. (8) computes an expected inverse norm, so Eq. (10) replaces a per-token normalization with a constant scaling. This is an approximation, not an equivalence, and Table 4 shows a 0.9-point average drop (59.9 vs 59.0) consistent with approximation error. Please either derive a bound on the approximation error or reframe the claim as an approximation validated empirically, and reserve 'mathematically equivalent' for the fusion of the fixed scaling into the weight matrix, which is exact.
  4. [Table 3] No error bars or repeated runs are reported for the ablation. The CLAP increment is 2.9 points and the SLNP increment is 0.7 points; without variance estimates or seed variations, it is unclear whether the SLNP gain is significant. Because Table 3 is the only controlled evidence for the paper's core causal claim, please report standard deviations across at least 2-3 seeds or a significance test.
minor comments (5)
  1. [Abstract and Section 4.3] The abstract claims the method outperforms 'prominent baseline pruning methods like Nemotron', but Nemotron is never defined or evaluated in the paper. Please either add a comparison or remove the reference.
  2. [Section 2 and Section 4.3] There are several typos: 'incluidng' in Section 2, 'Sm_acticvation' in Eq. (3), 're-initilization' in Section 5, and 'accleration' in Section 4.3. Please proofread the text.
  3. [Figure 2] Figure 2 is not readable in the current version: the axis labels and legend appear as garbled '/uni000000...' sequences. Please replace with a clean figure.
  4. [Section 4.1] The 'co-evaluation process' for selecting the pruned architectures is described only qualitatively, referencing simulation tools and a 'preliminary loss metric' without details. Please specify the search procedure and the final chosen configurations (number of retained layers, hidden dimensions, and KV groups) so that the pruning ratios are reproducible.
  5. [Table 2] Some values for QwQ-32B and Qwen3-32B/14B are marked with † as 'results from our internal evaluation', but the evaluation protocol for these entries is not given. Please state the exact setting used to obtain these numbers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the claimed re-initialization gains are tested by controlled ablations and do not reduce to fitted constants or self-citation.

full rationale

The paper's central claim is that CLAP and SLNP improve post-pruning recovery. This is supported by Table 3, where CLAP and SLNP are incrementally added to a Minitron baseline under the same 21B-token fine-tuning budget, and by Table 4 for the Post-RMSNorm absorption. The importance metrics (Eqs 1-4), CLAP's KV-group ranking (Eq 5), and SLNP's rescaling (Eq 6) are heuristics computed from model weights and a calibration set; none of the reported benchmark scores is a fitted value or a rearrangement of these equations. The Post-RMSNorm absorption (Eqs 8-10) is an approximation whose accuracy is then measured on benchmarks after fine-tuning. Self-citations to Pangu Ultra [25] supply the base model and evaluation protocol, which are inputs rather than conclusions, so no load-bearing derivation reduces to a self-citation. The lack of a deployment-scale controlled ablation (the headline results include 300B-token annealing and KD beyond Table 3) is a correctness or evidence concern, not circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The report's core recipe is heuristic. It adds at least four fitted or hand-chosen quantities (SLNP rescaling, norm absorption expectation, CLAP top-K, pruning ratios), assumes calibration-set transfer, and relies on massive continued training to recover quality. No new physical entities are introduced.

free parameters (5)
  • SLNP scaling factor c_l per RMSNorm layer = ||gamma_orig||_2 / ||gamma_pruned||_2
    Eq 6: re-scales pruned affine parameters to preserve their L2 norm; fitted from original and pruned weights.
  • Post-RMSNorm expected inverse norm s_inv per layer = Expectation in Eq 8 over calibration set
    Replaces the per-token normalization denominator with a calibration-set average; a fitted constant.
  • CLAP top-K KV groups retained per keep-prune layer pair = not disclosed
    Joint ranking of surviving groups from layers l and l+1; K controls how much attention capacity is transferred and is chosen by experiment.
  • Pruning ratios and target architectures for Pangu Light 1.6x/2.1x/4.2x = not disclosed in detail
    Section 4.1: configurations chosen via co-evaluation balancing simulated Ascend throughput and preliminary loss; these targets are hand-selected.
  • Importance thresholds and ranks for channel, head, and FFN pruning = not disclosed
    Eqs 1-4 produce scores, but the actual cut thresholds and ranks are not stated; they are calibrated to reach the target shapes.
assumptions (4)
  • domain assumption Activation statistics on calibration set C reveal which channels, heads, FFN neurons, and layers are unimportant for all downstream tasks.
    Eqs 1-4 depend on C; if C is unrepresentative, pruning selection is wrong.
  • domain assumption After convergence, RMSNorm input norms are stable enough that the expected inverse norm is a sufficient replacement for per-token normalization.
    Section 3.3, Eq 8-10: asserted without quantitative evidence or error bounds.
  • domain assumption Knowledge distillation plus 300B tokens of continued training can recover most performance lost by pruning.
    Section 3.4 and 4.2: a heavy compute commitment whose success is not guaranteed by the method and is not controlled against training from scratch.
  • ad hoc to paper Merging the most important KV groups from a pruned layer into the preceding layer preserves more information than deleting them.
    Section 3.2.1 CLAP: a heuristic validated only by one ablation, with no proof of information preservation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Pangu Light: Weight Re-Initialization for Pruning and Accelerating LLMs." pith.science (2026). https://pith.science/paper/XBWBH3ZD

@misc{pith2026250520155,
  author       = {Pith},
  title        = {Pith review of: Pangu Light: Weight Re-Initialization for Pruning and Accelerating LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XBWBH3ZD}},
  note         = {Machine review of arXiv:2505.20155}
}
read the original abstract

Large Language Models (LLMs) deliver state-of-the-art capabilities across numerous tasks, but their immense size and inference costs pose significant computational challenges for practical deployment. While structured pruning offers a promising avenue for model compression, existing methods often struggle with the detrimental effects of aggressive, simultaneous width and depth reductions, leading to substantial performance degradation. This paper argues that a critical, often overlooked, aspect in making such aggressive joint pruning viable is the strategic re-initialization and adjustment of remaining weights to improve the model post-pruning training accuracies. We introduce Pangu Light, a framework for LLM acceleration centered around structured pruning coupled with novel weight re-initialization techniques designed to address this ``missing piece''. Our framework systematically targets multiple axes, including model width, depth, attention heads, and RMSNorm, with its effectiveness rooted in novel re-initialization methods like Cross-Layer Attention Pruning (CLAP) and Stabilized LayerNorm Pruning (SLNP) that mitigate performance drops by providing the network a better training starting point. Further enhancing efficiency, Pangu Light incorporates specialized optimizations such as absorbing Post-RMSNorm computations and tailors its strategies to Ascend NPU characteristics. The Pangu Light models consistently exhibit a superior accuracy-efficiency trade-off, outperforming prominent baseline pruning methods like Nemotron and established LLMs like Qwen3 series. For instance, on Ascend NPUs, Pangu Light-32B's 81.6 average score and 2585 tokens/s throughput exceed Qwen3-32B's 80.9 average score and 2225 tokens/s.

Figures

Figures reproduced from arXiv: 2505.20155 by the authors.

Figure 1
Figure 1. Conceptual overview of the Pangu Light methodology, illustrating its integrated approach that com [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Performance ratio with respect to pruning ratio and acceleration ratio, illustrating the accuracy [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Distribution of the Sandwich-Norm’s affine scale parameters [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

27 extracted references · 1 canonical work pages

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Open AI. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Fluctuation-based adaptive structured pruning for large language models

    Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation-based adaptive structured pruning for large language models. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 10865–10873, 2024

    2024

  3. [3]

    Slicegpt: Compress large language models by deleting rows and columns.arXiv preprint arXiv:2401.15024, 2024

    Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns.arXiv preprint arXiv:2401.15024, 2024

    arXiv 2024

  4. [4]

    Layer normalization.arXiv preprint arXiv:1607.06450, 2016

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization.arXiv preprint arXiv:1607.06450, 2016

    arXiv 2016

  5. [5]

    Puzzle: Distillation-based nas for inference-optimized llms.arXiv preprint arXiv:2411.19146, 2024

    Akhiad Bercovich, Tomer Ronen, Talor Abramovich, Nir Ailon, Nave Assaf, Mohammad Dabbah, Ido Galil, Amnon Geifman, Yonatan Geifman, Izhak Golan, et al. Puzzle: Distillation-based nas for inference-optimized llms.arXiv preprint arXiv:2411.19146, 2024

    arXiv 2024

  6. [6]

    bert2bert: Towards reusable pretrained language models.ArXiv, abs/2110.07143, 2021

    Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang, Xiao Chen, Zhiyuan Liu, and Qun Liu. bert2bert: Towards reusable pretrained language models.ArXiv, abs/2110.07143, 2021

    arXiv 2021

  7. [7]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024. 12

    arXiv 2024

  8. [8]

    Sparsegpt: Massive language models can be accurately pruned in one-shot

    Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. InInternational Conference on Machine Learning, pages 10323–10337. PMLR, 2023

    2023

Show all 27 references
  1. [9]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  2. [10]

    Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  3. [11]

    Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  4. [12]

    Shortened llama: A simple depth pruning for large language models.arXiv preprint arXiv:2402.02834, 11, 2024

    Bo-Kyeong Kim, Geonmin Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung-Kyu Song. Shortened llama: A simple depth pruning for large language models.arXiv preprint arXiv:2402.02834, 11, 2024

    2024 arXiv

  5. [13]

    Lorap: Transformer sub-layers deserve differenti- ated structured compression for large language models.arXiv preprint arXiv:2404.09695, 2024

    Guangyan Li, Yongqiang Tang, and Wensheng Zhang. Lorap: Transformer sub-layers deserve differenti- ated structured compression for large language models.arXiv preprint arXiv:2404.09695, 2024

    2024 arXiv

  6. [14]

    Fastattention: Extend flashattention2 to npus and low-resource gpus

    Haoran Lin, Xianzhi Yu, Kang Zhao, Lu Hou, Zongyuan Zhan, Stanislav Kamenev, Han Bao, Ting Hu, Mingkai Wang, Qixin Chang, et al. Fastattention: Extend flashattention2 to npus and low-resource gpus. arXiv preprint arXiv:2410.16663, 2024

    arXiv

  7. [15]

    Slimgpt: Layer-wise structured pruning for large language models.arXiv preprint arXiv:2412.18110, 2024

    Gui Ling, Ziyang Wang, Yuliang Yan, and Qingwen Liu. Slimgpt: Layer-wise structured pruning for large language models.arXiv preprint arXiv:2412.18110, 2024

    2024 arXiv

  8. [16]

    Llm-pruner: On the structural pruning of large language models.Advances in neural information processing systems, 36:21702–21720, 2023

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models.Advances in neural information processing systems, 36:21702–21720, 2023

    2023

  9. [18]

    Shortgpt: Layers in large language models are more redundant than you expect.ArXiv, abs/2403.03853, 2024

    Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect.ArXiv, abs/2403.03853, 2024

    2024 arXiv

  10. [19]

    Compact language models via pruning and knowledge distillation.Advances in Neural Information Processing Systems, 37:41076–41102, 2024

    Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact language models via pruning and knowledge distillation.Advances in Neural Information Processing Sys...

    2024

  11. [20]

    Fusegpt: Learnable layers fusion of generative pre-trained transformers.arXiv preprint arXiv:2411.14507, 2024

    Zehua Pei, Hui-Ling Zhen, Xianzhi Yu, Sinno Jialin Pan, Mingxuan Yuan, and Bei Yu. Fusegpt: Learnable layers fusion of generative pre-trained transformers.arXiv preprint arXiv:2411.14507, 2024

    2024

  12. [21]

    A simple and effective pruning approach for large language models.arXiv preprint arXiv:2306.11695, 2023

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models.arXiv preprint arXiv:2306.11695, 2023

    2023 arXiv

  13. [22]

    Sheared llama: Accelerating language model pre-training via structured pruning.arXiv preprint arXiv:2310.06694, 2023

    Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning.arXiv preprint arXiv:2310.06694, 2023

    2023 arXiv

  14. [23]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  15. [24]

    Laco: Large language model pruning via layer collapse.arXiv preprint arXiv:2402.11187, 2024

    Yifei Yang, Zouying Cao, and Hai Zhao. Laco: Large language model pruning via layer collapse.arXiv preprint arXiv:2402.11187, 2024

    2024 arXiv

  16. [25]

    Pangu ultra: Pushing the limits of dense large language models on ascend npus

    Yichun Yin, Wenyong Huang, Kaikai Song, Yehui Tang, Xue-Fei Wu, Wei Guo, Peng Guo, Yaoyuan Wang, Xiaojun Meng, Yasheng Wang, Dong Li, Can Chen, Dandan Tu, Yin Li, Fisher Yu, Ruiming Tang, Yunhe Wang, Baojun Wang, Bin Wang, Bo Wang, Boxiao Liu, Changzheng Zhang, Duyu Tang, Fei ...

    2025

  17. [26]

    Root mean square layer normalization.Advances in Neural Information Processing Systems, 32, 2019

    Biao Zhang and Rico Sennrich. Root mean square layer normalization.Advances in Neural Information Processing Systems, 32, 2019. 13

    2019

  18. [27]

    Loraprune: Pruning meets low-rank parameter-efficient fine-tuning.arXiv preprint arXiv:2305.18403, 2023

    Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, and Bohan Zhuang. Loraprune: Pruning meets low-rank parameter-efficient fine-tuning.arXiv preprint arXiv:2305.18403, 2023

    2023 arXiv

  19. [28]

    Transformers without normaliza- tion.arXiv preprint arXiv:2503.10622, 2025

    Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, and Zhuang Liu. Transformers without normaliza- tion.arXiv preprint arXiv:2503.10622, 2025. 14 A Contributions and Acknowledgments Core ContributorsHanting Chen, Jiarui Qin, Jialong Guo, Tao Yuan, Yichun Yin, Huiling Zhen, Yash...

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.