Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ProxSparse claims learned 2:4 mask selection beats Wanda and SparseGPT across seven LLMs without retraining surviving weights.

desk verdict A practically strong 2:4 pruning method with a real but fixable proxy-target gap in the mask search, and an evaluation that needs error bars and a cleaner tuning protocol. read the letter →

arxiv 2502.00258 v2 pith:KZP6QXMO submitted 2025-02-01 cs.LG cs.CL

Hongyi Liu , Rajarshi Saha , Zhen Jia , Youngsuk Park , Jiaji Huang , Shoham Sabach , Yu-Xiang Wang , George Karypis This is my paper · ORCID
classification cs.LGcs.CL MSC 68T0790C2690C30
keywords semi-structuredpruning2:4sparsitylargelanguagemodelsmaskselectionregularizedoptimizationproximalgradientdescentmodelcompressionend-to-endlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper seeks to establish that the hard, combinatorial problem of choosing a 2:4 semi-structured sparsity mask for a pretrained large language model can be solved better by end-to-end regularized optimization than by the layer-wise heuristics that dominate current practice. Instead of scoring weights locally and pruning in one shot, ProxSparse relaxes the mask constraint into a smooth regularizer and lets gradient feedback from the whole model guide which weights survive. The method keeps surviving weights frozen at their original pretrained values, so the gain comes purely from mask choice, not from retraining or weight reconstruction. Across seven LLMs from four families, the paper reports consistently lower Wikitext perplexity and higher zero-shot accuracy than magnitude pruning, Wanda, and SparseGPT, using only a few hundred calibration samples. A fast proximal solver based on alternating minimization and iterative soft-thresholding makes the end-to-end search practical at the billion-parameter scale.

What carries the argument

The load-bearing object is a per-block regularizer for 2:4 sparsity, $\mathrm{Reg}_{2:4}(w) = |w_1w_2w_3| + |w_2w_3w_4| + |w_3w_4w_1| + |w_4w_1w_2|$, which is zero exactly when at least two of the four entries vanish and is permutation-invariant; a second regularizer $\mathrm{Reg}_{W_0}$ penalizes drift of retained weights away from the pretrained weights. Optimizing the relaxed objective with proximal gradient descent turns mask selection into gradual shrinkage, and the paper's fast solver EnumALM solves the proximal operator by alternating minimization over coordinates, where each step is a soft-thresholding operation. The paper proves convergence of the inner loop to a critical point and, under boundedness, convergence of the outer proximal-gradient loop for the regularized objective. This solver makes end-to-end search feasible on models with billions of parameters, with a reported 10x speedup over a projected-gradient solver and 100x over an interior-point solver for the proximal subproblem.

What would settle it

Run ProxSparse twice on the same model and calibration set, once with $\lambda_2 = 0$ to let weights drift and once with a large $\lambda_2$ to keep weights close to $W_0$, then evaluate both final masks against the frozen original weights. If the drifted-search mask does not beat the constrained-search mask on Wikitext perplexity and zero-shot accuracy, the claim that the gains come from relaxed global search under a frozen deployed model is not supported.

Watch

Extended reading notes

Core claim

The central claim is that end-to-end learned mask selection with a structure-inducing regularizer consistently outperforms previously proposed semi-structured mask selection methods at 2:4 sparsity while keeping retained weights frozen. On Llama-2-7b the paper reports Wikitext perplexity 8.51 against Wanda's 11.42 and SparseGPT's 10.30, and similar margins across Mistral, Qwen, OpenLlama, and Llama-3.1 models; reported average zero-shot accuracy gains reach about 20 percent. The paper attributes the improvement to global end-to-end feedback during mask search, which layer-wise heuristics lack, and to the smooth wiggle room of soft constraints that prevents premature commitment to a bad mask. The final deployed model is exactly 2:4 sparse with all surviving weights snapped back to their pretrained values, so the method isolates mask quality as the source of the gain.

Load-bearing premise

The load-bearing premise is that the mask selected while weights are allowed to move away from their pretrained values—the coefficient $\lambda_2$ is zero for six of seven models—remains near-optimal once the surviving weights are snapped back to the original pretrained weights; the paper reports about a 20 percent norm gap even with $\lambda_2 = 0$ and does not directly measure how much that gap degrades the final mask.

Editorial extensions

If this is right

  • At the reported margins, 2:4 semi-structured pruning no longer needs to trade mask quality for hardware-friendly structure: a few hundred calibration samples suffice to beat layer-wise one-shot methods.
  • Because surviving weights stay frozen, sparsified models can be served with standard 2:4 acceleration libraries; the paper measures 1.3x to 1.35x linear-layer speedups and a 1.26x end-to-end inference speedup.
  • The regularizer's wiggle-room account predicts that mask quality depends on a balanced regularization strength $\lambda_1$: too small under-shrinks, too large commits early, which the paper supports with PPL sweeps and mask-similarity curves.
  • The solver's efficiency claim means the method is not limited to a one-shot heuristic but can be rerun as calibration data grow; the paper reports PPL improving as sample counts increase from 100 to 400.
  • If the reported margins hold, the method can be applied to other semi-structured patterns such as 1:4 or 4:8 by reformulating the regularizer, though the paper notes the search space and solver cost grow.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the mask found under drifted weights remains near-optimal for frozen weights, the same regularized search could be used as a warm start for weight-reconstructing pruners, combining a better mask with SparseGPT-style error correction.
  • Beyond the paper: a direct probe of the core assumption would compare masks found with $\lambda_2 = 0$ against masks found with a large $\lambda_2$; if the constrained search matches or beats the drifted search under frozen weights, the current protocol's proxy-target mismatch is not the source of the reported gains.
  • Beyond the paper: the relaxation framework could serve as a general mask learner for future hardware sparsity patterns beyond 2:4, with the regularizer reformulated per pattern and solver cost as the main practical bottleneck.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces ProxSparse, a method for selecting 2:4 semi-structured sparsity masks in pretrained LLMs by solving a relaxed, regularized optimization problem over a small calibration set. The method replaces the combinatorial mask search with a continuous objective that combines the language-model loss with a 2:4-sparsity-inducing regularizer and an optional frozen-weight retention regularizer, and it solves the resulting problem with proximal gradient descent using a new alternating-minimization-based proximal operator (EnumALM). The authors prove convergence of the inner alternating-minimization loop and of the outer proximal-gradient scheme under stated assumptions. Experiments on seven LLMs report consistent perplexity and zero-shot accuracy improvements over magnitude pruning, SparseGPT, and Wanda, with the final deployed model obtained by projecting the learned weights to a 2:4 mask and snapping surviving weights back to the pretrained values.

Significance. If the empirical claims hold, ProxSparse is a practically useful contribution: it performs end-to-end mask learning with global loss feedback using only a few hundred calibration samples, requires no weight updates after mask selection, and includes an efficient proximal solver. The theoretical results in Propositions 1, 2, 6, and 7 are clearly stated with proofs in Appendix A, and the EnumALM solver is a reasonable algorithmic addition with a reported large speed advantage over earlier solvers. However, the paper's central empirical claim of 'consistent and significant improvement' currently rests on single-run comparisons without error bars, and the relationship between the optimized (drifted) weights and the deployed (snapped-to-W0) model is not directly tested. These gaps are load-bearing for the paper's main contribution, so the manuscript needs additional experimental validation before the claims can be accepted.

major comments (4)
  1. [Section 3.2, Algorithm 1; Table 8; Section 4.3.3] The deployed model is W0 ⊙ MaskProj2:4(Wk), but the mask is selected by optimizing (3) over weights W that are allowed to drift away from W0; Table 8 sets λ2 = 0 for six of the seven models, and Figure 2 reports a relative norm gap of about 20% even at λ2 = 0. The paper never directly measures how much performance is lost when the mask found under the drifted weights Wk is evaluated with the frozen weights W0, nor whether the same mask would be selected if the search were performed under a stronger frozen-weight penalty. This proxy-target mismatch directly concerns the paper's core claim that ProxSparse learns good masks for frozen-weight deployment. Please add a direct comparison, e.g., report L(W0 ⊙ M) versus L(Wk ⊙ M) for the same final mask M, and compare the final PPL of masks found with λ2 = 0 against masks found with a moderate λ2 that keeps W close to W0 throughout optimization.
  2. [Section 4.2, Table 2] No error bars, number of seeds, or statistical significance tests are reported anywhere in the empirical section. The abstract and Section 4.2 use the phrase 'significant improvement,' but with a single run per model and per baseline the claim is not statistically supported, especially since the calibration set, optimization randomness, and zero-shot evaluation all introduce variance. Please report means and standard deviations over at least three independent runs (or over different calibration draws), and state whether the observed improvements are stable across those runs.
  3. [Section 4.3.2, Table 4; Table 8] The main hyperparameter λ1 is tuned per model using Wikitext perplexity (Table 4), and Wikitext perplexity is also the headline metric in Table 2. The baselines, by contrast, use their standard published hyperparameters without a comparable per-model tuning budget. This selection procedure can inflate the reported gains and makes the comparison less informative. Please describe the exact model selection protocol, including whether the reported Wikitext numbers are the same numbers used to select λ1, and report the sensitivity of the zero-shot results to λ1 within the range shown in Table 4.
  4. [Section 3.4, Proposition 7; Algorithm 1; Table 8] The convergence guarantee in Proposition 7 is stated for Algorithm 1 as proximal gradient descent, but the experiments use AdamW as the optimizer (Table 8) and the proximal operator is solved by the approximate EnumALM solver (Algorithm 3) rather than an exact oracle. The proof invokes standard proximal-gradient convergence results and does not cover the actual implemented algorithm. Please either extend the theory to the implemented variant, or state clearly that the convergence guarantee applies to an idealized version and that the experiments use a heuristic optimization procedure.
minor comments (6)
  1. [Table 7 and Section 4.4.2] The text reports a peak-memory reduction of 29.5%–37.3%, while Table 7 lists memory gains of 70.50%, 67.50%, and 62.70%; these numbers are inconsistent and should be reconciled.
  2. [Figure 1; Section 4.3.2] The caption of Figure 1 uses duplicate labels ('b. Dense ratio evolvement' and 'b. Mask similarity evolvement'), and Section 4.3.2 refers to 'Figures 4(a), (b) and (c)' when the relevant figure in the main text is Figure 1; Appendix F contains another Figure 4, so the cross-references need to be fixed.
  3. [Table 3] The header row of Table 3 is garbled ('Both withrelaxationFronzen weightrelaxationSparsity constraintsrelaxation Both with hardConstraints') and the table layout is not readable; please reformat it.
  4. [Introduction and Table 8] There are typos such as 'accross' in the contributions list and 'parammeter' in Table 8; a careful proofread is needed.
  5. [Section 4.3.4] The comparison with MaskLLM in Table 6 uses statistics from the MaskLLM paper rather than runs in the same environment; this should be stated clearly in the text, as the current presentation implies a controlled comparison.
  6. [Table 2] The 'AVG' column is not defined; please state that it is the average over the seven zero-shot tasks and clarify how ties or missing values are handled.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central mask-quality claims are benchmarked against external baselines, and the self-cited proximal-operator theorem is an independent mathematical ingredient rather than the target result.

full rationale

ProxSparse's central empirical claim—consistent PPL/accuracy gains over Wanda, SparseGPT, magnitude pruning, and MaskLLM—is evaluated on external benchmarks (Wikitext, LM-Eval-Harness) with fixed pretrained models, so the reported improvements are not forced by construction. The regularizer Reg2:4 is explicitly designed so that its zero set is exactly the 2:4-sparse set; Proposition 1 states this defining property rather than deriving a prediction from it. The self-citation to Kübler et al. (2025) supplies Theorem 4, a parameter-free mathematical characterization of the proximal operator for (6); it does not assume the paper's own empirical conclusions, and the present paper adds its own ALM solver and convergence analysis. The remaining concerns—λ2=0 for six of seven models with final weight snapping to W0, and per-model λ1 selection—are proxy/validity risks about whether the relaxed search transfers to frozen weights, not cases where a prediction is identical to an input by definition. The paper itself flags the ~20% norm gap in Section 4.3.3 and leaves its cause to future investigation, which is a caveat about generalization rather than a circular step. No equation in the paper reduces to another by construction, and no fitted parameter is renamed as a prediction.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or architectural entities. It relies on two fitted hyperparameters per model (lambda1 and lambda2) plus a learning rate, and on several unproven regularity assumptions about the calibration distribution and the relaxed objective. The regularizer is borrowed from the authors' own earlier paper, so the ledger is mostly about tuning and domain assumptions rather than invented objects.

free parameters (3)
  • lambda1 (sparsity regularization strength) = Per model: 20 (Mistral-v0.1-7b), 25 (Mistral-v0.3-7b), 0.2 (Qwen2.5-14b), 1 (OpenLlama-7b-v2), 0.25 (Llama-2-7b), 0.5…
    Table 4 shows lambda1 is swept over Wikitext PPL and the value minimizing PPL is selected per model; this is a parameter fitted to the evaluation metric.
  • lambda2 (frozen weight retention strength) = 0 for six models, 0.25 for Llama-2-13b
    Table 5 shows lambda2=0 gives the best PPL on the two tested models, so the headline results rely on an ablation-derived choice rather than a principled default.
  • Learning rate = 5e-5 or 1e-4 depending on model
    Table 8 tunes the learning rate per model; no sensitivity analysis or selection rule is provided.
assumptions (5)
  • domain assumption Calibration data are drawn from the test-data distribution
    Section 2.3 explicitly assumes calibration samples are new data points from the test distribution; mask generalization to Wikitext and zero-shot tasks depends on this.
  • ad hoc to paper The relaxed regularized objective (3) is a faithful surrogate for the hard combinatorial mask problem (1)
    Corollary 3 only shows equality of feasible sets as lambda tends to infinity; at finite lambda the 'wiggling room' heuristic is not proven to preserve optimality of the original problem.
  • standard math The proximal operator theorem (Theorem 4) of Kubler et al. 2025 holds as stated
    The paper inherits the enumeration of candidate solutions from the authors' prior work without re-deriving it; EnumALM's correctness depends on this external result.
  • domain assumption Boundedness of iterates in Proposition 7
    The convergence proof assumes there exists B > 0 such that the norm of every iterate is bounded; the paper does not establish this from the algorithm or loss.
  • domain assumption The training loss L is continuously differentiable
    This regularity is needed for the proximal gradient convergence theory and is standard but not verified for the calibration loss on LLMs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs." pith.science (2026). https://pith.science/paper/KZP6QXMO

@misc{pith2026250200258,
  author       = {Pith},
  title        = {Pith review of: ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KZP6QXMO}},
  note         = {Machine review of arXiv:2502.00258}
}
read the original abstract

Large Language Models (LLMs) have demonstrated exceptional performance in natural language processing tasks, yet their massive size makes serving them inefficient and costly. Semi-structured pruning has emerged as an effective method for model acceleration, but existing approaches are suboptimal because they focus on local, layer-wise optimizations using heuristic rules, failing to leverage global feedback. We present ProxSparse, a learning-based framework for mask selection enabled by regularized optimization. ProxSparse transforms the rigid, non-differentiable mask selection process into a smoother optimization procedure, allowing gradual mask exploration with flexibility. ProxSparse does not involve additional weight updates once the mask is determined. Our extensive evaluations on 7 widely used models show that ProxSparse consistently outperforms previously proposed semi-structured mask selection methods with significant improvement, demonstrating the effectiveness of our learned approach towards semi-structured pruning.

Figures

Figures reproduced from arXiv: 2502.00258 by the authors.

Figure 1
Figure 1. Evolution of sparsity ratio on Llama-2-7b based on the degree of regularization. (a) Evolution of the 2:4 sparsity ratio over learning progress, where an insufficient regularization degree leads to under-learning. (b) With a larger λ1 parameters shrink more quickly towards 2:4 sparsity, resulting in early commitment to a suboptimal mask. (c) Comparison of the 2:4 sparse block ratios at early (0.1 epochs) and final s… view at source ↗
Figure 2
Figure 2. The relative norm difference over different λ2. The rela￾tive norm gap measures how closely retained weights match their original values post-training, with the semi-structured mask ap￾plied. The relative norm remained low (∼20%) with minimal change until a high lambda value was applied [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Illustration of the solution to (6) with an example input y = [1.4, 1.1, 1.0, 0.7] as λ increases. Observe that (1) the regularizer shrinks different coordinates differently according to their relative magnitude (2) all three algorithms return the same solution path. (3) the dashed lines indicate the two thresholds of λ from KKT conditions above which the 3-sparse and 2-sparse solutions become critical points (a nec… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Evolution of sparsity ratio on OpenLlama-7b-v2 based on the degree of regularization. (a) Evolution of the 2:4 sparsity ratio over learning, where an insufficient regularization degree leads to under-learning. (b) With a larger λ1 parameters shrink more quickly towards…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Amber Pruner: Leveraging N:M Activation Sparsity for Efficient Prefill in Large Language Models

    cs.LG 2025-08 unverdicted novelty 6.0 of 10

    Amber Pruner proposes training-free N:M activation sparsity for LLM prefill; however, the supplied manuscript body is an unrelated paper.

Reference graph

Works this paper leans on

38 extracted references · 6 canonical work pages · cited by 1 Pith paper

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Sparsellm: Towards global pruning for pre-trained language models, 2024

    Bai, G., Li, Y., Ling, C., Kim, K., and Zhao, L. Sparsellm: Towards global pruning for pre-trained language models, 2024. URL https://arxiv.org/abs/2402.17946

    arXiv 2024

  3. [3]

    An alternating semiproximal method for nonconvex regularized structured total least squares problems

    Beck, A., Sabach, S., and Teboulle, M. An alternating semiproximal method for nonconvex regularized structured total least squares problems. SIAM J. Matrix Anal. Appl., 37: 0 1129--1150, 2016

    work page 2016

  4. [4]

    Estimating or propagating gradients through stochastic neurons for conditional computation

    Bengio, Y., L \'e onard, N., and Courville, A. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013

    arXiv 2013

  5. [5]

    Fast and effective weight update for pruned large language models

    Bo z a, V. Fast and effective weight update for pruned large language models. arXiv preprint arXiv:2401.02938, 2024

    arXiv 2024

  6. [6]

    Learning to compress prompt in natural language formats, 2024

    Chuang, Y.-N., Xing, T., Chang, C.-Y., Liu, Z., Chen, X., and Hu, X. Learning to compress prompt in natural language formats, 2024. URL https://arxiv.org/abs/2402.18700

    arXiv 2024

  7. [7]

    A dynamic alternating direction of multipliers for nonconvex minimization with nonlinear functional equality constraints

    Cohen, E., Hallak, N., and Teboulle, M. A dynamic alternating direction of multipliers for nonconvex minimization with nonlinear functional equality constraints. J. Optim. Theory Appl., 193: 0 324--353, 2022

    work page 2022

  8. [8]

    Maskllm: Learnable semi-structured sparsity for large language models

    Fang, G., Yin, H., Muralidharan, S., Heinrich, G., Pool, J., Kautz, J., Molchanov, P., and Wang, X. Maskllm: Learnable semi-structured sparsity for large language models. arXiv preprint arXiv:2409.17481, 2024

    arXiv 2024

Show all 38 references
  1. [9]

    and Carbin, M

    Frankle, J. and Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018

    2018 arXiv

  2. [10]

    and Alistarh, D

    Frantar, E. and Alistarh, D. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp.\ 10323--10337. PMLR, 2023

    2023

  3. [11]

    Gptq: Accurate post-training quantization for generative pre-trained transformers

    Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022

    2022 arXiv

  4. [12]

    A framework for few-shot language model evaluation, 07 2024

    Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., a...

    2024

  5. [13]

    and Liu, H

    Geng, X. and Liu, H. Openllama: An open reproduction of llama, May 2023. URL https://github.com/openlm-research/open_llama

    2023

  6. [14]

    Han, S., Mao, H., and Dally, W. J. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015

    2015 arXiv

  7. [15]

    Pruning large language models with semi-structural adaptive sparse training

    Huang, W., Jian, G., Hu, Y., Zhu, J., and Chen, J. Pruning large language models with semi-structural adaptive sparse training. arXiv preprint arXiv:2407.20584, 2024

    2024 arXiv

  8. [16]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    2023 arXiv

  9. [17]

    M., Wang, Y.-X., Sabach, S., Ansari, N., Kleindessner, M., Budhathoki, K., Cevher, V., and Karypis, G

    K \"u bler, J. M., Wang, Y.-X., Sabach, S., Ansari, N., Kleindessner, M., Budhathoki, K., Cevher, V., and Karypis, G. A proximal operator for inducing 2: 4-sparsity. arXiv preprint arXiv:2501.18015, 2025

    2025 arXiv

  10. [18]

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration

    Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6: 0 87--100, 2024

    2024

  11. [19]

    F., Cheng, K.-T., and Chen, M.-H

    Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-C. F., Cheng, K.-T., and Chen, M.-H. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353, 2024

    2024 arXiv

  12. [20]

    W., and Yang, Y

    Lu, H., Zhou, Y., Liu, S., Wang, Z., Mahoney, M. W., and Yang, Y. Alphapruning: Using heavy-tailed self regularization theory for improved layer-wise pruning of large language models. Advances in Neural Information Processing Systems, 37: 0 9117--9152, 2024

    2024

  13. [21]

    Llm-pruner: On the structural pruning of large language models

    Ma, X., Fang, G., and Wang, X. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36: 0 21702--21720, 2023

    2023

  14. [22]

    Pointer sentinel mixture models, 2016

    Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models, 2016

    2016

  15. [23]

    A., Pool, J., Stosic, D., Stosic, D., Venkatesh, G., Yu, C., and Micikevicius, P

    Mishra, A., Latorre, J. A., Pool, J., Stosic, D., Stosic, D., Venkatesh, G., Yu, C., and Micikevicius, P. Accelerating sparse deep neural networks. arXiv preprint arXiv:2104.08378, 2021

    2021 arXiv

  16. [24]

    Gradient methods for minimizing composite functions

    Nesterov, Y. Gradient methods for minimizing composite functions. Mathematical programming, 140 0 (1): 0 125--161, 2013

    2013

  17. [25]

    Stochastic rounding for llm training: Theory and practice

    Ozkara, K., Yu, T., and Park, Y. Stochastic rounding for llm training: Theory and practice. arXiv preprint arXiv:2502.20566, 2025

    2025 arXiv

  18. [26]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  19. [27]

    Rockafellar, R. T. and Wets, R. J.-B. Variational Analysis. Springer Verlag, Heidelberg, Berlin, New York, 1998

    1998

  20. [28]

    Stop overthinking: A survey on efficient reasoning for large language models

    Sui, Y., Chuang, Y.-N., Wang, G., Zhang, J., Zhang, T., Yuan, J., Liu, H., Wen, A., Zhong, S., Chen, H., et al. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint arXiv:2503.16419, 2025

    2025 arXiv

  21. [29]

    Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695, 2023

    2023 arXiv

  22. [30]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  23. [31]

    Training llms with mxfp4

    Tseng, A., Yu, T., and Park, Y. Training llms with mxfp4. arXiv preprint arXiv:2502.20586, 2025

    2025 arXiv

  24. [32]

    Emergent abilities of large language models

    Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022

    2022 arXiv

  25. [33]

    K., Kang, D., Park, Y., and Hong, M

    Wei, Q., Yau, C.-Y., Wai, H.-T., Zhao, Y. K., Kang, D., Park, Y., and Hong, M. Roste: An efficient quantization-aware supervised fine-tuning approach for large language models. arXiv preprint arXiv:2502.09003, 2025

    2025 arXiv

  26. [34]

    Sheared llama: Accelerating language model pre-training via structured pruning

    Xia, M., Gao, T., Zeng, Z., and Chen, D. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694, 2023

    2023 arXiv

  27. [35]

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  28. [36]

    Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity

    Yin, L., Wu, Y., Zhang, Z., Hsieh, C.-Y., Wang, Y., Jia, Y., Li, G., Jaiswal, A., Pechenizkiy, M., Liang, Y., et al. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity. arXiv preprint arXiv:2310.05175, 2023

    arXiv

  29. [37]

    Kv cache compression, but what must we give in return? a comprehensive benchmark of long context capable approaches

    Yuan, J., Liu, H., Zhong, S., Chuang, Y.-N., Li, S., Wang, G., Le, D., Jin, H., Chaudhary, V., Xu, Z., et al. Kv cache compression, but what must we give in return? a comprehensive benchmark of long context capable approaches. arXiv preprint arXiv:2407.01527, 2024

    2024 arXiv

  30. [38]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.