Pith. sign in

REVIEW 4 major objections 3 minor 43 references

Symmetric Pruning of Large Language Models

T0 review · 4 major / 3 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Two leading pruning methods, Wanda and RIA, are special cases of one symmetric reconstruction error, and the same identity yields cheaper stochastic variants and a training-free fine-tune.

desk verdict The symmetric objective is a genuinely clean reformulation that recovers Wanda and RI, but the claimed recovery of RIA collapses under inspection, and the empirical gains are mostly marginal. read the letter →

arxiv 2501.18980 v1 pith:GBKQKYKM submitted 2025-01-31 cs.LG cs.AI

Kai Yi , Peter Richtárik This is my paper · ORCID
classification cs.LGcs.AI MSC 68T07
keywords post-trainingpruninglargelanguagemodelssymmetricreconstructionerrorWandarelativeimportancetraining-freefine-tuningdynamicsparsetrainingstochasticsampling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to give a theoretical foundation for two empirically successful but poorly understood post-training pruning methods for large language models: Wanda and RIA. Its central move is to replace the usual input-only reconstruction objective with a symmetric one that also tracks what each weight does to the layer's output, and to prove that deleting a single weight $W_{jk}$ changes this objective by exactly $|W_{jk}|(\|X_{:j}\|_2 + \|Y_{k:}\|_2)$. From that identity, both Wanda (set the output term to zero) and RIA (choose calibration matrices that encode reciprocal row and column norms) fall out as special cases, which the paper presents as the first theoretical explanation of why these heuristics work. Exploiting the same parametrization, the paper derives cheaper relative-importance scores that sample only 10% of each row and column, and a training-free prune-and-grow fine-tuning scheme, R2-DSnoT, that it reports outperforms strong baselines. A sympathetic reader cares because if the identity is right, a whole family of pruning methods with a common single-weight guarantee becomes available, not just two tuned heuristics.

What carries the argument

The load-bearing object is the single-weight identity $S_{jk} = |W_{jk}|(\|X_{:j}\|_2 + \|Y_{k:}\|_2)$, the exact change in the symmetric reconstruction error when weight $W_{jk}$ alone is zeroed; the freedom to choose the calibration matrices $X$ and $Y$ turns this one formula into a generator of pruning scores. Wanda is the corner $Y=0$, RIA is recovered by diagonal or constant-vector choices whose norms equal reciprocal $\ell^1$ row and column norms, and new methods arise by replacing those norms with $\ell^p$ norms or with norms over randomly sampled subsets of indices (StochRIA). The identity also drives the R2-DSnoT fine-tuning rules, which use the same relative-importance products plus an $\ell^p$ regularizer when deciding which weights to grow and which to prune.

What would settle it

Run the paper's exact setup on one model, say LLaMA2-7b at 50% sparsity with Wikitext-2 perplexity as the metric, two ways: one-shot pruning by globally sorting the scores $S_{jk}$, and greedy iterative pruning that recomputes the score after each deletion; if greedy is materially better, the multi-weight extrapolation of Lemma 3.1 is doing unproven work, and if the two match, the single-weight ranking is robust enough to support the method.

Watch

Extended reading notes

Core claim

The paper's core claim is that post-training pruning of a language model layer should minimize a symmetric reconstruction error $g(\tilde{W}) = \|X(\tilde{W}-W)\|_F + \|(\tilde{W}-W)Y\|_F$, where $X$ is the input calibration matrix and $Y$ is a newly introduced output calibration matrix. Lemma 3.1 gives the exact single-weight form: zeroing $W_{jk}$ costs $|W_{jk}|(\|X_{:j}\|_2 + \|Y_{k:}\|_2)$. Because the choice of $X$ and $Y$ is free, the paper shows that Wanda's score, $|W_{jk}|\|X_{:j}\|_2$, is exactly the $Y=0$ corner of this formula, and that RIA's relative-importance score, $|W_{jk}|(\|W_{j:}\|_1^{-1} + \|W_{:k}\|_1^{-1})$, is recovered by choosing $X$ and $Y$ whose row and column norms are reciprocals of the weight row and column $\ell^1$ norms. This is what the authors mean by providing theoretical insights into the empirical successes of Wanda and RIA, and it lets them generate new scores, including $\ell^p$-norm versions, random-unit-vector scalings, and a stochastic 10%-sampled relative importance, each carrying the same single-weight guarantee. On top of the pruning scores, the paper contributes R2-DSnoT, a training-free pruning-and-growing fine-tuner that applies the same relative-importance products plus a regularized decision boundary, reporting improved perplexity and zero-shot accuracy at 60% sparsity across LLaMA2, LLaMA3, and OPT models.

Load-bearing premise

The identity $S_{jk}$ is proved only for deleting a single weight, while every reported pruning result is obtained by sorting these scores and deleting many weights at once; the paper assumes this extrapolation holds, and the entire empirical evaluation rests on it.

Editorial extensions

If this is right

  • Wanda and RIA cease to be standalone heuristics and become two points in a parametrized family of pruning scores, each with an exact single-weight reconstruction guarantee.
  • Any new choice of calibration matrices $X$ and $Y$ produces a new pruning rule with the same guarantee, so the framework is a generator of candidate methods rather than a single algorithm.
  • Estimating relative importance from 10% of each row and column suffices to match, and on two of the four tested models beat, full RIA, which makes relative-importance pruning practical at large scale.
  • The training-free fine-tuning step R2-DSnoT improves perplexity and zero-shot accuracy on top of pruned models without any gradient updates, which, if it holds, removes the main cost barrier to iterative refinement of sparse language models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference, beyond the paper: because Lemma 3.1 is exact only for a single deletion, the paper's global-sort procedure is an untested approximation; comparing it against greedy iterative re-scoring on the same models would reveal how much of the reported performance the identity itself guarantees.
  • Editorial inference, beyond the paper: the theorem treats $X$ and $Y$ as free choices, so a natural unexplored variant is to set $Y$ from the actual output activations of the layer rather than from $W$ itself, which could capture downstream sensitivity that the current constructions miss.
  • Editorial inference, beyond the paper: the ablation finding that column-wise (output-side) relative importance matters more than row-wise suggests that output-calibrated pruning alone should already rival input-based Wanda; that prediction is directly testable on the paper's own benchmark suite.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. This paper proposes a symmetric pruning objective g(What)=||X(What-W)||_F + ||(What-W)Y||_F and derives a single-weight score S_jk=|W_jk|(||X_:j||_2 + ||Y_k:||_2). It claims this framework recovers Wanda and RIA as special cases, and uses it to motivate new pruning scores (generalized ℓp-norm and stochastic RIA, called stochRIA) as well as a training-free fine-tuning procedure R2-DSnoT. Experiments on LLaMA2-7b/13b, LLaMA3-8b and OPT-1.3b at several sparsity levels compare perplexity and zero-shot accuracy with Wanda, RIA and DSnoT.

Significance. The derivations in Lemma 3.1 and Theorem 3.5 are correct and show that Wanda and the activation-free relative-importance (RI) score are indeed special cases of the symmetric objective; the paper also provides simple constructions for ℓp and stochastic variants. If the RIA-recovery claim were true, the paper would offer a useful unification of two popular pruning heuristics. However, Lemma 3.6 is mathematically invalid as a global recovery of RIA (see major comment), so the central theoretical contribution does not hold as stated. The empirical study covers four models and multiple sparsities, which is a useful resource, but the headline claims of superior performance are not consistent with the reported tables. No code or machine-checked proofs are provided beyond the algebraic derivations.

major comments (4)
  1. [Section 3.3, Lemma 3.6] The construction in Lemma 3.6 does not recover RIA as a special case of the symmetric objective. For fixed matrices A and B, the score in (Sym) has the additive form |W_{jk}|(||A_{j:}||_2 + ||B_{:k}||_2). To match Eq. (2) for all (j,k), one would need ||A_{j:}||_2 + ||B_{:k}||_2 = ||W_{j:}||_1^{-1}||C_{:j}||_2^α + ||W_{:k}||_1^{-1}||C_{:j}||_2^α = h(j) + w_j v_k, with h(j)=||W_{j:}||_1^{-1}||C_{:j}||_2^α, w_j=||C_{:j}||_2^α, and v_k=||W_{:k}||_1^{-1}. Subtracting the equations for two rows j1 and j2 gives (w_{j1}-w_{j2})v_k = const for all k, which is impossible when the activation norms differ and the column norms v_k vary. In Lemma 3.6, B_{s,k} is set to ||C_{:j}||_2^α ||W_{:k}||_1^{-1} for a single row j, so B depends on j; no single pair (A,B) satisfies the identity for all (j,k). Thus Eq. (2) is recovered only for the one pair used in the construction, and Table 1's identification of RIA with Lemma 3.6 is incorrect. This undermines the abstract's claim that the framework provides theoretical insights into RIA's success.
  2. [Section 3.2, after Lemma 3.1] The exact score S_jk is derived for deleting a single weight while all other weights are unchanged. The text then states that 'the general approach can be extended to multiple weights iteratively,' and the experiments prune large fractions by sorting the single-weight scores. Sorting ignores the fact that after one weight is removed, the optimal next choice depends on the updated weight matrix and the remaining activations. No proof, bound, or experiment isolates this gap. Therefore the statement that the resulting batch-pruning strategy is 'backed by theoretical foundations' (end of Section 3.2) is not supported by the presented analysis.
  3. [Section 4.1, Tables 2 and 4] The text claims stochRIA 'consistently demonstrates the benefits' and has an accumulated perplexity 0.66 lower than RIA. The tables show mixed results: at 50% sparsity, stochRIA is worse than RIA on LLaMA2-7b (6.91 vs 6.88) and LLaMA3-8b (9.46 vs 9.44), and at 70% it is worse on LLaMA2-7b (72.85 vs 68.75) and LLaMA2-13b (62.15 vs 51.96). In addition, the listed deltas in Table 2 do not obviously sum to 0.66; for example, the 2:4 LLaMA3-8b entry reports +0.15 while 23.74 - 22.89 = 0.85. The aggregate claim should be recomputed from the full table, and the conclusion should be revised to say that stochRIA is comparable but cheaper, with mixed and sometimes worse outcomes at higher sparsity.
  4. [Section 4.3, Tables 5-6 and Appendix C.5] The abstract and Section 4.3 state that R2-DSnoT 'significantly outperforms strong baselines' and sets a new state of the art. The reported gains over DSnoT are small: for example, in Table 5 the RIA-based perplexity improves from 9.97 to 9.96 on LLaMA2-7b, and in Table 6 the RIA-based mean zero-shot accuracy improves from 50.2 to 50.3 on LLaMA2-7b and from 47.9 to 48.1 on LLaMA3-8b. No variance estimates or significance tests are reported. Moreover, Appendix C.5 selects hyperparameters per model via a 20-run random search and reports the best configuration, which is vulnerable to selection bias. The 'state of the art' claim is not supported by the evidence.
minor comments (3)
  1. [Appendix A.2 and Lemma 3.6] The notation is inconsistent: Theorem 3.5's proof refers to 'the j-th row of X' while using the column notation X:j, and Lemma 3.6 alternates between row and column norms. Please align the notation with Lemma 3.1, where X:j is a column and Y_k: is a row.
  2. [Lemma 3.1 and Section 3.6] The symbol S_jk is defined in Lemma 3.1 as the error value g(What), but later in Section 3.6 the notation D_{q,r} is used for a related reweighting quantity. Please clarify the relationship between the symmetric score and the R2-DSnoT growing/pruning criteria in Equations (5)-(6).
  3. [Table 1] The table uses footnote marker '(a)' both as a table footnote and as the dimension a in the header, which is confusing; a different symbol should be used for one of them.

Circularity Check

1 steps flagged · score 6.0 of 10

RIA is 'recovered' by defining A and B to match the RIA formula, so the claimed theoretical insight is a construction, not a derivation; the empirical contributions remain independent.

  1. self definitional [Section 3.3, Lemma 3.6 (and Eq. (2))]
    "we are aiming at finding the suitable A ∈ R^{a×b} and B ∈ R^{c×d} such as: ∥Aj:∥2 + ∥B:k∥2 = (∥Wj:∥−1_1 + ∥Wk:∥−1_1) · ∥C:j∥α_2 ... Define Auv = 0 for all (u, v) ̸= (j, p), and Aj,p = ∥C:j∥α_2 ∥Wj:∥−1_1. Similarly ... Bs,k = ∥C:j∥α_2 ∥Wk:∥−1_1. Then we recover Equation (2)."

    The equation the lemma claims to 'recover' is exactly the condition used to define A and B: the nonzero entry of row A_j: is set to ||W_j:||_1^{-1} ||C_:j||_2^α and the nonzero entry of column B_:k is set to ||W_k:||_1^{-1} ||C_:j||_2^α, so the norm identity is a restatement of the construction, not a consequence derived from the (Sym) objective. Moreover, the construction is performed for a single index pair (j,k) at a time; a single global B cannot carry a factor ||C_:j||_2^α in its k-th column that varies with the row index j. Thus the central claim that SymWanda 'provides theoretical insights' into RIA and recovers RIA as a special case is a definitional fitting of the framework to the target formula, not an independent derivation.

full rationale

The symmetric objective (Sym) and Lemma 3.1 are independently derived: for a single deleted weight, the reconstruction error is exactly |W_jk|(||X_:j||_2 + ||Y_k:||_2), and Corollary 3.2's reduction to Wanda (Y = 0) is a legitimate special case. The circularity is confined to the RIA/RI unification. Lemma 3.6 does not derive RIA from the objective; it defines A and B so that their row/column norms equal the RIA factors and then reads the target equation back. This is a construction fitted to the target, and because the construction is pointwise in (j,k), a single global pair (A,B) cannot actually realize the RIA score for all pairs. Theorem 3.5 is the same pattern at the RI level: the proof chooses X:j and Y_k: with norms equal to the inverse row/column norms, and the surrounding text says the methodology 'can invariably reconstruct' RI, which is true only because the calibration matrices are selected post hoc to match the score. Appendix B itself concedes the limitation of the unification claim for the squared-norm variant ('note that this is not always possible!'). The new algorithmic contributions (StochRIA, R2-DSnoT) and their empirical evaluations against external baselines are not circular; their value stands or falls on the experiments. The extrapolation from single-weight exactness to iterative multi-weight pruning is asserted rather than proved, but that is a validation gap, not circularity. Score 6 reflects that the central explanatory claim about RIA reduces by construction, while the paper retains independent empirical content.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central theoretical result rests on the standard layer-wise reconstruction assumption and on the extrapolation from single-weight to batch pruning. The recovery of RIA as a special case uses ad hoc constructions of X and Y that are reverse-engineered from the weight matrix. Several hyperparameters (α, β, p, γ1, γ2, reweighting booleans) are tuned on the evaluation data.

free parameters (6)
  • activation exponent α = 0.5 or 1; 0.5 chosen as optimal
    Controls the power of the input activation norm in Wanda/RIA scores; values 0, 0.5, 1, 2 are compared in Table 3, and α=0.5 is selected as default without a principled derivation.
  • sampling ratio β for StochRIA = 0.1
    Selected after scanning β ∈ {1, 0.9, 0.5, 0.1, 0.05, 0.01} (Appendix C.4); 0.1 chosen as the default setting.
  • norm order p for ℓp-norm variants = 1
    p ∈ {1, 2, 3, 4, 0, ∞} tested in Table 7; p=1 chosen as best after experiments.
  • growing regularizer γ1 = 0 (both Wanda and RIA best)
    Chosen via random search over {0, 0.0001, 0.001} per model and base method in Appendix C.5; impact is described as minimal.
  • pruning regularizer γ2 = 0.0001 (Wanda), 0.001 (RIA)
    Same random search in Appendix C.5; values vary by base method.
  • relative reweighting booleans (grow/prune) = Wanda: grow=True, prune=False; RIA: grow=False, prune=True
    Binary choices selected in the hyperparameter search; applying reweighting to both phases degrades performance.
assumptions (4)
  • domain assumption Layer-wise reconstruction error is a valid surrogate for final model loss.
    The paper minimizes ||X(Ŵ-W)||_F + ||(Ŵ-W)Y||_F per layer (Section 3.2), assuming this controls end-to-end perplexity, an assumption shared with Wanda and SparseGPT.
  • domain assumption Single-weight error extrapolates to batch pruning.
    Lemma 3.1 gives exact error for pruning one weight; the algorithms sort these scores and prune many weights simultaneously, ignoring interaction effects (Section 3.2).
  • ad hoc to paper X and Y can be chosen arbitrarily, not necessarily as real layer inputs or outputs.
    To recover RIA, X and Y are constructed from the weight matrix W (Theorem 3.5, Lemma 3.6) rather than being actual activation matrices; the score formula does not require X and Y to be real data.
  • domain assumption Calibration data distribution is representative of evaluation data.
    Calibration uses 128 C4 samples with 2048 tokens each; evaluation is on Wikitext-2. This is standard practice but an assumption that the calibration set captures the relevant activation statistics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Symmetric Pruning of Large Language Models." pith.science (2026). https://pith.science/paper/GBKQKYKM

@misc{pith2026250118980,
  author       = {Pith},
  title        = {Pith review of: Symmetric Pruning of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GBKQKYKM}},
  note         = {Machine review of arXiv:2501.18980}
}
abstract

Popular post-training pruning methods such as Wanda and RIA are known for their simple, yet effective, designs that have shown exceptional empirical performance. Wanda optimizes performance through calibrated activations during pruning, while RIA emphasizes the relative, rather than absolute, importance of weight elements. Despite their practical success, a thorough theoretical foundation explaining these outcomes has been lacking. This paper introduces new theoretical insights that redefine the standard minimization objective for pruning, offering a deeper understanding of the factors contributing to their success. Our study extends beyond these insights by proposing complementary strategies that consider both input activations and weight significance. We validate these approaches through rigorous experiments, demonstrating substantial enhancements over existing methods. Furthermore, we introduce a novel training-free fine-tuning approach $R^2$-DSnoT that incorporates relative weight importance and a regularized decision boundary within a dynamic pruning-and-growing framework, significantly outperforming strong baselines and establishing a new state of the art.

Figures

Figures reproduced from arXiv: 2501.18980 by the authors.

Figure 1
Figure 1. Visualization of the dense weight matrix in LLaMA2-7b. as illustrated in [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 12 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Deep rewiring: Training very sparse deep networks

    Bellec, G., Kappel, D., Maass, W., and Legenstein, R. Deep rewiring: Training very sparse deep networks. In International Conference on Learning Representations, 2018

    work page 2018

  3. [3]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019

    arXiv 1905

  4. [4]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    arXiv 2018

  5. [5]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  6. [6]

    Extreme compression of large language models via additive quantization

    Egiazarian, V., Panferov, A., Kuznedelev, D., Frantar, E., Babenko, A., and Alistarh, D. Extreme compression of large language models via additive quantization. In Forty-first International Conference on Machine Learning, 2024

    work page 2024

  7. [7]

    S., and Elsen, E

    Evci, U., Gale, T., Menick, J., Castro, P. S., and Elsen, E. Rigging the lottery: Making all tickets winners. In International conference on machine learning, pp.\ 2943--2952. PMLR, 2020

    2020

  8. [8]

    and Carbin, M

    Frankle, J. and Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2018

    work page 2018

Show all 43 references
  1. [9]

    and Alistarh, D

    Frantar, E. and Alistarh, D. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp.\ 10323--10337. PMLR, 2023

    2023

  2. [10]

    OPTQ : Accurate quantization for generative pre-trained transformers

    Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. OPTQ : Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=tcbBPnfwxS

    2023

  3. [11]

    A framework for few-shot language model evaluation

    Gao, L., Tow, J., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., McDonell, K., Muennighoff, N., et al. A framework for few-shot language model evaluation. Version v0. 0.1. Sept, 10: 0 8--9, 2021

    2021

  4. [12]

    Learning both weights and connections for efficient neural network

    Han, S., Pool, J., Tran, J., and Dally, W. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015

    2015

  5. [13]

    Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks

    Hoefler, T., Alistarh, D., Ben-Nun, T., Dryden, N., and Peste, A. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22 0 (241): 0 1--124, 2021

    2021

  6. [14]

    Javaheripi, M., Bubeck, S., Abdin, M., Aneja, J., Bubeck, S., Mendes, C. C. T., Chen, W., Del Giorno, A., Eldan, R., Gopi, S., et al. Phi-2: The surprising power of small language models. Microsoft Research Blog, 2023

    2023

  7. [15]

    Soft threshold weight reparameterization for learnable sparsity

    Kusupati, A., Ramanujan, V., Somani, R., Wortsman, M., Jain, P., Kakade, S., and Farhadi, A. Soft threshold weight reparameterization for learnable sparsity. In International Conference on Machine Learning, pp.\ 5544--5555. PMLR, 2020

    2020

  8. [16]

    Dynamic sparse training with structured sparsity

    Lasby, M., Golubeva, A., Evci, U., Nica, M., and Ioannou, Y. Dynamic sparse training with structured sparsity. arXiv preprint arXiv:2305.02299, 2023

    2023 arXiv

  9. [17]

    Optimal brain damage

    LeCun, Y., Denker, J., and Solla, S. Optimal brain damage. Advances in neural information processing systems, 2, 1989

    1989

  10. [18]

    Lee, N., Ajanthan, T., and Torr, P. H. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.02340, 2018

    2018 arXiv

  11. [19]

    E-sparse: Boosting the large language model inference through entropy-based n: M sparsity

    Li, Y., Niu, L., Zhang, X., Liu, K., Zhu, J., and Kang, Z. E-sparse: Boosting the large language model inference through entropy-based n: M sparsity. arXiv preprint arXiv:2310.15929, 2023

    2023 arXiv

  12. [20]

    C., and Pechenizkiy, M

    Liu, S., Yin, L., Mocanu, D. C., and Pechenizkiy, M. Do we actually need dense over-parameterization? in-time over-parameterization in sparse training. In International Conference on Machine Learning, pp.\ 6989--7000. PMLR, 2021

    2021

  13. [21]

    Pv-tuning: Beyond straight-through estimation for extreme llm compression

    Malinovskii, V., Mazur, D., Ilin, I., Kuznedelev, D., Burlachenko, K., Yi, K., Alistarh, D., and Richtarik, P. Pv-tuning: Beyond straight-through estimation for extreme llm compression. arXiv preprint arXiv:2405.14852, 2024

    2024 arXiv

  14. [22]

    Pointer sentinel mixture models

    Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016

    2016 arXiv

  15. [23]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018

    2018 arXiv

  16. [24]

    C., Mocanu, E., Stone, P., Nguyen, P

    Mocanu, D. C., Mocanu, E., Stone, P., Nguyen, P. H., Gibescu, M., and Liotta, A. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9 0 (1): 0 2383, 2018

    2018

  17. [25]

    and Wang, X

    Mostafa, H. and Wang, X. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning, pp.\ 4646--4655. PMLR, 2019

    2019

  18. [26]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  19. [27]

    L., Bhagavatula, C., and Choi, Y

    Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64 0 (9): 0 99--106, 2021

    2021

  20. [28]

    Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, 2023

    2023

  21. [29]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023 a

    2023 arXiv

  22. [30]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023 b

    2023 arXiv

  23. [31]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. Glue: A multi-task benchmark and analysis platform for natural language understanding. arxiv preprint arxiv: 180407461, 2018

    2018

  24. [32]

    Transformers: State-of-the-art natural language processing

    Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. Transformers: State-of-the-art natural language processing. EMNLP 2020, pp.\ 38, 2020

    2020

  25. [33]

    Smoothquant: Accurate and efficient post-training quantization for large language models

    Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp.\ 38087--38099. PMLR, 2023

    2023

  26. [34]

    Fedllm-bench: Realistic benchmarks for federated learning of large language models

    Ye, R., Ge, R., Zhu, X., Chai, J., Du, Y., Liu, Y., Wang, Y., and Chen, S. Fedllm-bench: Realistic benchmarks for federated learning of large language models. arXiv preprint arXiv:2406.04845, 2024

    2024 arXiv

  27. [35]

    Fedp3: Federated personalized and privacy-friendly network pruning under model heterogeneity

    Yi, K., Gazagnadou, N., Richtarik, P., and Lyu, L. Fedp3: Federated personalized and privacy-friendly network pruning under model heterogeneity. ICLR, 2024

    2024

  28. [36]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  29. [37]

    Magr: Weight magnitude reduction for enhancing post-training quantization

    Zhang, A., Wang, N., Deng, Y., Li, X., Yang, Z., and Yin, P. Magr: Weight magnitude reduction for enhancing post-training quantization. Advances in neural information processing systems, 2024 a

    2024

  30. [38]

    V., et al

    Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022 a

    2022 arXiv

  31. [39]

    Learning best combination for efficient n: M sparsity

    Zhang, Y., Lin, M., Lin, Z., Luo, Y., Li, K., Chao, F., Wu, Y., and Ji, R. Learning best combination for efficient n: M sparsity. Advances in Neural Information Processing Systems, 35: 0 941--953, 2022 b

    2022

  32. [40]

    Dynamic sparse no training: Training-free fine-tuning for sparse llms

    Zhang, Y., Zhao, L., Lin, M., Sun, Y., Yao, Y., Han, X., Tanner, J., Liu, S., and Ji, R. Dynamic sparse no training: Training-free fine-tuning for sparse llms. arXiv preprint arXiv:2310.08915, 2023

    2023 arXiv

  33. [41]

    Zhang, Y., Bai, H., Lin, H., Zhao, J., Hou, L., and Cannistraci, C. V. Plug-and-play: An efficient post-training pruning method for large language models. In The Twelfth International Conference on Learning Representations, 2024 b

    2024

  34. [42]

    Learning n: m fine-grained structured sparse neural networks from scratch

    Zhou, A., Ma, Y., Zhu, J., Liu, J., Zhang, Z., Yuan, K., Sun, W., and Li, H. Learning n: m fine-grained structured sparse neural networks from scratch. arXiv preprint arXiv:2102.04010, 2021

    2021 arXiv

  35. [43]

    A survey on model compression for large language models

    Zhu, X., Li, J., Liu, Y., Ma, C., and Wang, W. A survey on model compression for large language models. Transactions of the Association for Computational Linguistics, 12: 0 1556--1577, 2024

    2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.