Pith. sign in

REVIEW 4 major objections 5 minor 57 references

ACE: Exploring Activation Cosine Similarity and Variance for Accurate and Calibration-Efficient LLM Pruning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ACE claims that weighting LLM pruning scores by activation cosine-similarity loss and activation variance lets large language models be pruned accurately with only 16 calibration sequences, lowering perplexity by up to 18% and pruning…

desk verdict The empirical pruning results are real and worth engaging with, but the theoretical calibration-efficiency argument rests on a lambda that is treated as both constant and sequence-dependent, and the main text and appendix disagree on the variance score. read the letter →

arxiv 2505.21987 v1 pith:RANYXS3P submitted 2025-05-28 cs.LG

Zhendong Mi , Zhenglun Kong , Geng Yuan , Shaoyi Huang This is my paper · ORCID
classification cs.LG
keywords LLMpruningpost-trainingactivationcosinesimilarityvariancecalibrationefficiencyN:Msparsityperplexityone-shot
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ACE is a post-training pruning recipe for large language models built on a single idea: the weights worth keeping are the ones whose removal would rotate the output activation vectors in embedding space or flatten the differences between token representations. The paper derives two importance scores — CosP, based on the cosine-similarity loss introduced by zeroing a weight, and VarP, based on the variance of input activation columns — and combines them into one score. It claims that this combined score, used with only 16 calibration sequences, matches or beats the Wanda and RIA baselines across LLaMA, LLaMA-2, and OPT models at 50%, 2:4, and 4:8 sparsity, while cutting perplexity by up to 18% and pruning time by up to 63% in the settings tested. A sympathetic reader should care because calibration data and compute are scarce in real deployments; if the claim holds, one-shot pruning becomes cheaper and less data-hungry without sacrificing model quality.

What carries the argument

The load-bearing object is the combined score of Eq. (11), $$$S^{{\mathrm{(cos+var)}}$}_{ij} = |W_{ij}|\,\mathbb{E}\left[\frac{1}{1-\|X_j\|$_2^{2}$}\right] \cdot \frac{|W_{ij}|}{\sqrt{\sum_{i'} W_{i'j}^2/d_{\mathrm{out}}}}\, \sqrt{\sum_{j'} W_{ij'}^2},$$ where $X_j$ is the $j$-th input activation column across calibration tokens and $d_{\mathrm{out}}$ is the output dimension. The second line is the CosP factor: it approximates the relative cosine-similarity loss of setting $W_{ij}$ to zero, estimated from a first-order Taylor expansion of the output vector. The expectation in the first line is the VarP factor: it encodes the variance of input activations and, after a power-series expansion, penalizes weights whose input features show small spread across tokens. Together the two factors let a single pruning pass preserve both the direction and the discriminability of token embeddings.

What would settle it

Fix all hyperparameters, including the stabilization term, and measure WikiText-2 perplexity of the pruned LLaMA-7B at 50% unstructured sparsity using calibration sequence lengths 16, 32, 128, 512, 1024, and 2048 for both ACE and Wanda. The derivation predicts the gap should grow roughly as $\frac{1}{L+1}\mathbb{E}[\|X_j\|_2]$; if the observed gap does not increase monotonically as the sequence length $L$ shrinks, the claimed calibration-efficiency mechanism is not operating.

Watch

Extended reading notes

Core claim

The central claim is that pruning quality for LLMs is determined less by the magnitude of weights and activations than by two activation-geometry quantities: the angular deviation of the output vector after a weight is removed, and the spread of input activations across tokens. To act on this, the paper defines CosP from a first-order approximation of cosine loss, defines VarP from a variance-aware Hessian analysis, and multiplies them into Eq. (11), so the importance of $W_{ij}$ is $|W_{ij}|\,E[1/(1-\|X_j\|_2^2)]$ times a row-normalized weight factor. The paper's experiments report that this score, evaluated on only 16 calibration sequences, outperforms Wanda and RIA on WikiText-2 perplexity and seven zero-shot tasks for sparsities 50%, 2:4, and 4:8 across LLaMA-7B/13B/30B/65B, LLaMA-2-7B/13B, and OPT-350M/1.3B/2.7B/6.7B/13B/30B, and that in N:M settings it often uses 40%-60% of the baseline pruning time. Notably, the paper reports that at 60% unstructured sparsity on LLaMA-13B the 16-sequence version reaches a perplexity of 8.16 against 8.75 for Wanda with full sequences, which is the kind of result the calibration-efficiency argument is meant to explain.

Load-bearing premise

The calibration-efficiency claim rests on setting a stabilization term in the Hessian comparison to scale with calibration sequence length; if that term is fixed in practice, the paper's predicted short-sequence advantage is not established.

Editorial extensions

If this is right

  • A single calibration pass over 16 sequences is enough to prune models up to 65B parameters, making one-shot pruning practical with small calibration sets.
  • At N:M sparsity (2:4 and 4:8), the reported gains over baselines are larger, so hardware-accelerated sparse inference would see the biggest quality benefit.
  • Because the combined score is a drop-in replacement for the Wanda importance metric, existing pruning pipelines can adopt it without changing the pruning loop or the model architecture.
  • Pruning times as low as 40%-50% of baseline in 2:4 settings on LLaMA-30B mean large-model compression experiments become much cheaper to run.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper leaves implicit: if the geometric mechanism is right, the same cosine-plus-variance score should also improve quantization-aware pruning or sparse finetuning, since those operations also perturb output activations.
  • The closed form $E[1/(1-\|X_j\|_2^2)]$ suggests an outlier-amplification interpretation: columns whose squared norm approaches 1 get near-infinite importance, so the metric may implicitly protect a small set of high-norm features rather than treating all tokens uniformly.
  • Comparing VarP alone against CosP alone at sequence lengths 16 versus 2048 would isolate whether the calibration-efficiency claim is driven by variance rather than by cosine alignment; the paper does not report that ablation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ACE, a post-training pruning method for LLMs that combines two importance scores: CosP, based on cosine-similarity loss of output activations, and VarP, based on the mean and variance of input activation norms. The reported experiments cover LLaMA, LLaMA-2, and OPT at 50%, 2:4, and 4:8 sparsity, with both WikiText2 perplexity and zero-shot accuracy, and include timing comparisons. The headline claims are that the combined metric achieves lower perplexity than Wanda and RIA, can be run with as few as 16 calibration sequences, and requires substantially less pruning time, with a theoretical argument in §3.4 and Appendix B.2 intended to explain the calibration-efficiency behavior.

Significance. If the empirical claims hold, the paper would make a useful contribution: the evaluation is broad, covering multiple model families and sizes, and the 16-sequence pruning results show a consistent directional advantage over the baselines in many settings. The paper also ships a clear experimental protocol with reproducible baselines, and the runtime comparison is a practical strength. However, the theoretical derivation that is used to explain and advertise the calibration efficiency is not sound as written. Because the calibration-efficiency claim is a central advertised contribution, the theory needs to be either fixed or substantially downgraded before the paper is publishable in its current form.

major comments (4)
  1. [§3.4, Appendix B.2, Eqs. (42)–(44)] The derivation of the inverse-length scaling in Eq. (16) is circular. Equation (44) obtains diff = E[||X_j||^2]/(L+1) only after setting λ = Lk, but the preceding equations treat λ as a fixed regularization constant. A constant cannot depend on the sequence length L; if λ is fixed, the claimed 1/(L+1) scaling does not follow from the algebra. Moreover, Eq. (16) is interpreted as evidence of reduced reconstruction error, but the quantity diff is the difference between two denominator approximations, not a reconstruction error. Even if the scaling were valid, it would not by itself establish that the VarP score improves accuracy at short sequence lengths.
  2. [Eq. (10) vs. Appendix B.1, Eq. (39)] The metric for which results are reported is not the metric derived in Appendix B.1. The final VarP score in Eq. (10) is S_var^ij = |W_ij| · E[1/(1 - ||X_j||_2^2)], while the appendix derivation in Eq. (39) yields |W_ij|^2 · E[1/(1 - ||X_j||_2)], with a different power of the weight and a different norm convention. This mismatch means the Hessian-based analysis in §3.4 and Appendix B.2 does not actually connect to the deployed VarP score. The authors should either align the derivation with Eq. (10) or explicitly state that the Hessian analysis is a motivation rather than a proof.
  3. [§3.2, Eq. (9)] The power-series approximation E[||X_j||_2^4] + E[||X_j||_2^2] + 1 ≈ E[1/(1 - ||X_j||_2^2)] is presented as a derivation, but it omits all higher-order terms of the geometric series and requires the random variable ||X_j||_2^2 to be smaller than 1. The manuscript claims this follows because 'input activations are normalized,' but normalization does not in general bound the squared l2 norm below 1. The approximation selects exactly the terms that make the variance term appear, so the resulting form should be treated as a heuristic choice rather than a derived identity.
  4. [§3.1, Eq. (3) and Appendix A, Eqs. (27)–(29)] The CosP derivation uses Hölder's inequality and then drops the term |Σ_j W_tj X_j| / |Σ_j W_kj X_j|, arguing that it is invariant within a row and therefore does not affect ranking. However, the final metric in Eq. (3) combines row and column sums in a way that is not clearly justified by the derivation, and the index conventions in Eqs. (27)–(29) are inconsistent (the squared denominator sum over i is later treated as a row-independent constant). The resulting formula may still be a reasonable heuristic, but the derivation is not watertight and should be presented as such.
minor comments (5)
  1. [Table 3] The Wanda-FL pruning times for LLaMA-2-7B and LLaMA-2-13B in the 50% rows (98.8 and 183.5 seconds) appear to duplicate the OPT-13B/30B timing entries and are likely copy-paste errors; please verify and correct.
  2. [§3.2, Eqs. (4)–(8)] The notation for norms is confusing: ||X_j||_2, ||X_j||_2^2, and ||X_j||_4^2 are used with different meanings across equations, and Eq. (5) says 'E[||X_j||_2]' while the text says it represents the mean of squared values. Please introduce consistent notation for a column norm and its square.
  3. [Appendix B.2, Eqs. (42)–(44)] The sequence length is called L in Eq. (43), but the sum is written over n terms and the final expression uses E[||X_j||^2]; please align the summation index, the sequence-length symbol, and the expectation notation.
  4. [Introduction, reference [10]] Reference [10] is cited to support the claim that deploying LLaMA-65B has TTFT exceeding 100 ms, but the reference listed is 'Wanda++: Pruning large language models via regional gradients,' which does not appear to be the intended source.
  5. [Abstract and §1] The acronym ACE is used in the title and abstract but is never expanded in the body; please define it at first use.

Circularity Check

1 steps flagged · score 6.0 of 10

Calibration-efficiency proof is forced by choosing λ = nk/Lk: Eq. 44 restates that choice as Eq. 16, making the central 'fewer sequences' claim partly circular; the empirical comparisons remain non-circular.

  1. self definitional [Appendix B.2 (Efficiency Analysis), Eq. 44; relied on by Section 3.4, Eq. 16]
    "Since the λ is a constant, we can make it equals to nk, then: dif f= k λ + k E[||Xj||2] = 1 L + 1E[||Xj||2]"

    λ is introduced as a fixed regularizer ('λ is used to prevent algorithm failure due to singular matrices'), but Eq. 44 redefines it as nk, where n is the number/sequence length; the same equation then writes 1/(L+1), so either n=L or the algebra is inconsistent. Substituting a sequence-length-dependent λ=k·L into diff = k/(λ+k)E[||X_j||²] gives E[||X_j||²]/(L+1) by algebra; the inverse-length scaling is therefore built into the choice of λ, not derived from the VarP metric. Section 3.4 then presents Eq. 16's 1/(N+1) scaling as proof that 'our proposed method yields reduced reconstruction error ... particularly in scenarios with smaller input sequence length,' making the calibration-efficiency claim self-definitional. The bridge to the actual metric is also absent: Eq.

full rationale

The only circular step is in the theoretical calibration-efficiency argument. Appendix B.2 makes the regularization constant λ depend on the sequence length (λ = nk, with the final denominator written as L+1) and then algebraically obtains diff = E[||X_j||²]/(L+1); Section 3.4 quotes this as Eq. 16 and presents it as a proof of calibration efficiency. Because λ is chosen as a function of the sequence length, the derived inverse relationship is a restatement of that choice rather than an independent result. This makes the central 'fewer calibration sequences' claim partly circular. The paper's empirical evaluations (Tables 2-7 and the zero-shot tables) are self-contained comparisons against Wanda and RIA under fixed protocols and are not manufactured by Eq. 16; they stand or fall as experimental evidence independently. There is no load-bearing self-citation chain and no imported uniqueness theorem. I also flag that the theoretical derivation's metric (|W|², Eq. 39) does not match the implemented VarP metric (|W|, Eq. 10); this is a missing-link defect that weakens the claimed theoretical support, though it is not itself a circularity. Overall score 6: the calibration-efficiency conclusion reduces by construction to the choice of λ, while the pruning metrics and benchmark comparisons retain independent content.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a Wanda-style product of weight magnitude and activation norm plus two extra terms. The theoretical analysis adds hand-chosen polynomial coefficients and a post hoc regularization constant. No formal verification or code is supplied.

free parameters (2)
  • lambda_regularization = lambda = L * ||X||^2 (chosen in Appendix B.2)
    The derivation sets the SparseGPT regularization constant proportional to the number of calibration sequences L to force the 1/(L+1) scaling in Eq. 16, but lambda is normally a fixed hyperparameter in the Hessian inverse.
  • h polynomial coefficients = 1, 1, 1
    The coefficients in Eq. 5 are hand-picked so that the variance term telescopes into the first three terms of the geometric series 1/(1-u). They are not fit to data and have no independent justification.
assumptions (5)
  • domain assumption Layer-normed activations have approximately constant squared norm ||X||^2 = k.
    Used in Appendix B.1 to simplify the Hessian diagonal. RMS/layer normalization makes the norm scale with hidden dimension, but per-token feature norms still vary, so the constant-k approximation is not exact.
  • domain assumption Input activation values are less than 1.
    Invoked in Eq. 9 to justify the geometric power series expansion. Layer normalization does not guarantee that every component is below 1, so the expansion may not converge for all activations.
  • domain assumption The small-perturbation assumption |delta a_k| << |a_k| holds for pruned weights.
    Appendix A uses this to justify the first-order Taylor approximation of cosine loss. At 50% to 60% sparsity, many pruned weights are comparable in size to the remaining activations, so the assumption is not guaranteed.
  • domain assumption Semantic integrity is preserved primarily by minimizing angular deviation of output activations.
    The core motivation in Section 1 cites embedding-geometry results, but it is not proven that minimizing per-weight cosine loss is the right proxy for preserving model quality under pruning.
  • domain assumption For equal-importance weights, lower input activation variance better preserves token-level distinctions.
    The motivating example in Figure 2 and Section 3.2 is an empirical heuristic, not a derived property of transformer representations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ACE: Exploring Activation Cosine Similarity and Variance for Accurate and Calibration-Efficient LLM Pruning." pith.science (2026). https://pith.science/paper/RANYXS3P

@misc{pith2026250521987,
  author       = {Pith},
  title        = {Pith review of: ACE: Exploring Activation Cosine Similarity and Variance for Accurate and Calibration-Efficient LLM Pruning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RANYXS3P}},
  note         = {Machine review of arXiv:2505.21987}
}
read the original abstract

With the rapid expansion of large language models (LLMs), the demand for memory and computational resources has grown significantly. Recent advances in LLM pruning aim to reduce the size and computational cost of these models. However, existing methods often suffer from either suboptimal pruning performance or low time efficiency during the pruning process. In this work, we propose an efficient and effective pruning method that simultaneously achieves high pruning performance and fast pruning speed with improved calibration efficiency. Our approach introduces two key innovations: (1) An activation cosine similarity loss-guided pruning metric, which considers the angular deviation of the output activation between the dense and pruned models. (2) An activation variance-guided pruning metric, which helps preserve semantic distinctions in output activations after pruning, enabling effective pruning with shorter input sequences. These two components can be readily combined to enhance LLM pruning in both accuracy and efficiency. Experimental results show that our method achieves up to an 18% reduction in perplexity and up to 63% decrease in pruning time on prevalent LLMs such as LLaMA, LLaMA-2, and OPT.

Figures

Figures reproduced from arXiv: 2505.21987 by the authors.

Figure 1
Figure 1. Example of angular deviation before and after pruning Motivation. In Natural Language Processing (NLP), words are conventionally embedded into vector space. And semantically similar words exhibit approximate vector lengths and vector positions, where the latter is characterized by the small angular distances between them, re￾sulting in cosine similarity value approaching 1 [43, 44]. While pruning is utilized as a po… view at source ↗
Figure 2
Figure 2. The motivating example of our proposed activation variance-guided pruning metric [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 35 canonical work pages

  1. [1]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    arXiv 2018

  2. [2]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, et al. Language models are few-shot learners. NeurIPS, 33, 2020

    work page 2020

  3. [3]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  4. [4]

    Squad: 100,000+ questions for machine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016

    arXiv 2016

  5. [5]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. NeurIPS, 2020

    work page 2020

  6. [6]

    Recipes for building an open-domain chatbot

    Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, et al. Recipes for building an open-domain chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 300–325, 2021

    work page 2021

  7. [7]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

  8. [8]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) , pages 4171–4186, 2019

    2019

Show all 57 references
  1. [9]

    Llm inference performance engineering: Best practices

    Megha Agarwal, Asfandyar Qureshi, Linden Li Nikhil Sardana, Julian Quevedo, and Daya Khudia. Llm inference performance engineering: Best practices. URl: https://www. databricks. com/blog/llm-inference-performanceengineering-best-practices, 2023

    2023

  2. [10]

    Wanda++: Pruning large language models via regional gradients

    Yifan Yang, Kai Zhen, Bhavana Ganesh, Aram Galstyan, Goeric Huybrechts, Markus Müller, Jonas M Kübler, Rupak Vignesh Swaminathan, Athanasios Mouchtaris, Sravan Babu Boda- pati, et al. Wanda++: Pruning large language models via regional gradients. arXiv preprint arXiv:2503.04992, 2025

    2025 arXiv

  3. [11]

    Binarybert: Pushing the limit of bert quantization

    Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. Binarybert: Pushing the limit of bert quantization. arXiv preprint arXiv:2012.15701, 2020

    2012 arXiv

  4. [12]

    Spdy: Accurate pruning with speedup guarantees

    Elias Frantar and Dan Alistarh. Spdy: Accurate pruning with speedup guarantees. In Interna- tional conference on machine learning , pages 6726–6743. PMLR, 2022

    2022

  5. [13]

    Smoothquant: Accurate and efficient post-training quantization for large language models

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning , pages 38087–38099. PMLR, 2023

    2023

  6. [14]

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87–100, 2024

    2024

  7. [15]

    Optimal brain surgeon and general network pruning

    Gregory J Wolff, B Hassibi, and D Stork. Optimal brain surgeon and general network pruning. Technical report, Technical report, 1992

    1992

  8. [16]

    Optimal brain damage

    Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989

    1989

  9. [17]

    Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science

    Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9(1):2383, 2018. 10

    2018

  10. [18]

    A simple and effective pruning approach for large language models

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695, 2023

    2023 arXiv

  11. [19]

    Sparsegpt: Massive language models can be accurately pruned in one-shot

    Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. International Conference on Machine Learning , pages 10322–10337, 2023

    2023

  12. [20]

    Language model compression with weighted low-rank factorization

    Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. arXiv preprint arXiv:2207.00112, 2022

    2022 arXiv

  13. [21]

    Loretta: Low-rank economic tensor- train adaptation for ultra-low-parameter fine-tuning of large language models

    Yifan Yang, Jiajun Zhou, Ngai Wong, and Zheng Zhang. Loretta: Low-rank economic tensor- train adaptation for ultra-low-parameter fine-tuning of large language models. arXiv preprint arXiv:2402.11417, 2024

    2024 arXiv

  14. [22]

    How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings

    Kawin Ethayarajh. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 55–65, 2019

    2019

  15. [23]

    On the degeneration of neural text generation

    Jiwei Li, Dan Su, Yizhe Shen, and Pengwei Li. On the degeneration of neural text generation. In International Conference on Learning Representations (ICLR) , 2020

    2020

  16. [24]

    Efficient estimation of word representations in vector space

    Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013

    2013 arXiv

  17. [25]

    Simcse: Simple contrastive learning of sentence embeddings

    Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. In EMNLP, 2021

    2021

  18. [26]

    Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding

    Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015

    2015 arXiv

  19. [27]

    Rethinking the value of network pruning

    Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint arXiv:1810.05270, 2018

    2018 arXiv

  20. [28]

    Learning both weights and connections for efficient neural networks

    Song Han, Jeff Pool, John Tran, and William J Dally. Learning both weights and connections for efficient neural networks. Advances in neural information processing systems , 28, 2015

    2015

  21. [29]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks

    Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. International Conference on Learning Representations , 2019

    2019

  22. [30]

    Learning efficient convolutional networks through network slimming

    Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference on computer vision , pages 2736–2744, 2017

    2017

  23. [31]

    Importance estimation for neural network pruning

    Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11264–11272, 2019

    2019

  24. [32]

    Snip: Single-shot network pruning based on connection sensitivity

    Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations , 2018

    2018

  25. [33]

    Pruning filters for efficient convnets

    Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In 5th International Conference on Learning Representations, ICLR , 2017

    2017

  26. [34]

    Accelerating sparse deep neural networks

    Asit Mishra, Eriko Nurvitadhi, Jeffrey J Cook, and Debbie Marr. Accelerating sparse deep neural networks. In 2021 IEEE Hot Chips 33 Symposium (HCS) , pages 1–23. IEEE, 2021

    2021

  27. [35]

    Optimal brain compression: A framework for accurate post-training quantization and pruning

    Elias Frantar, Eldar Kurtic, Markus Stenström, and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems, 36, 2023

    2023

  28. [36]

    Plug- and-play: Hardware-aware semi-structured pruning for large language models

    Xin Zhang, Jiachen Wen, Yujun Zhou, Zhongzhi Huang, Xianglong Liu, and Dahua Lin. Plug- and-play: Hardware-aware semi-structured pruning for large language models. arXiv preprint arXiv:2402.07883, 2024. 11

    2024 arXiv

  29. [37]

    The state of sparsity in deep neural networks

    Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. In International Conference on Learning Representations (ICLR) , 2019. OpenReview preprint

    2019

  30. [38]

    Pruner-zero: Evolving symbolic pruning metric from scratch for large language models

    Peijie Dong, Lujun Li, Zhenheng Tang, Xiang Liu, Xinglin Pan, Qiang Wang, and Xiaowen Chu. Pruner-zero: Evolving symbolic pruning metric from scratch for large language models. arXiv preprint arXiv:2406.02924, 2024

    2024 arXiv

  31. [39]

    Wanda: Weight-norm based pruning for efficient large language models

    Mingjie Sun, Zhuang Cui, Shipeng Fang, Xiaotong Song, Jiacheng Wang, Zhiyuan Li, Ying Gao, Caiming Xiong, and Tong Liu. Wanda: Weight-norm based pruning for efficient large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processi...

    2023

  32. [40]

    Optishear: Towards efficient and adaptive prun- ing of large language models via evolutionary optimization

    Shuqi Liu, Bowei He, Han Wu, and Linqi Song. Optishear: Towards efficient and adaptive prun- ing of large language models via evolutionary optimization. arXiv preprint arXiv:2502.10735, 2025

    2025 arXiv

  33. [41]

    Compression of deep neural networks

    Seyyed Mohammad Saeed Damadi. Compression of deep neural networks. Master’s thesis, University of Maryland, Baltimore County, 2021

    2021

  34. [42]

    Hawq: Hessian aware quantization of neural networks with mixed-precision

    Zhen Dong, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed-precision. In Proceedings of the IEEE/CVF international conference on computer vision , pages 293–302, 2019

    2019

  35. [43]

    Distributed representations of words and phrases and their compositionality

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems (NeurIPS), volume 26, 2013

    2013

  36. [44]

    Glove: Global vectors for word representation

    Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014

    2014

  37. [45]

    Analyzing and measuring bert’s under- standing of syntax

    Kevin Li, Mark Yatskar, Wen-tau Yin, and Dan Hovy. Analyzing and measuring bert’s under- standing of syntax. In International Conference on Learning Representations (ICLR) , 2020

    2020

  38. [46]

    Optimal brain surgeon and general network pruning

    Babak Hassibi and David G Stork. Optimal brain surgeon and general network pruning. IEEE international conference on neural networks , pages 293–299, 1993

    1993

  39. [47]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron and ... Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  40. [48]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

  41. [49]

    A framework for few-shot language model evaluation

    Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. V ersion v0. 0.1. Sept, 10:8–9, 2021

    2021

  42. [50]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016

    2016 arXiv

  43. [51]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020

    2020

  44. [52]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Matt Gardner, Carissa Schoenick, Nikita Mishra, and Sameer Singh. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019

    2019

  45. [53]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019

    2019

  46. [54]

    Hellaswag: Can a machine really finish your sentence? In ACL, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In ACL, 2019. 12

    2019

  47. [55]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In AAAI, 2020

    2020

  48. [56]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. In arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  49. [57]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018. 13 A Appendix: Activation Cosine Similarity Loss Guided Pruning Metric Lemma: Consider a vector a ∈ Rd with ...

    2018

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.