Pith. sign in

REVIEW 3 major objections 4 minor 55 references

HASSLE-free: A unified Framework for Sparse plus Low-Rank Matrix Decomposition for LLMs

T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that directly minimizing the exact layer-wise objective with the full Hessian outperforms prior sparse-plus-low-rank LLM compression methods, cutting Llama3-8B WikiText-2 perplexity by 12%.

desk verdict Useful unified framework, but the printed algorithm has a D vs D^{-1} scaling error that makes the headline results unattributable to the described method. read the letter →

arxiv 2502.00899 v1 pith:IUCAJPPX submitted 2025-02-02 stat.ML cs.LG

Mehdi Makni , Kayhan Behdin , Zheng Xu , Natalia Ponomareva , Rahul Mazumder This is my paper · ORCID
classification stat.MLcs.LG MSC 68T0790C2615A23
keywords sparsepluslow-rankmatrixdecompositionLLMcompressionN:Msparsityone-shotpruninglayer-wisereconstructionerroralternatingminimizationfullHessianadaptation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that sparse-plus-low-rank compression of large language models should be solved by directly minimizing the local layer-wise reconstruction error built from the full input-activation Hessian, rather than the diagonal-Hessian relaxation used by prior work. It introduces HASSLE-free, an alternating-minimization algorithm that alternates between a sparse subproblem solved with a pruning method (SparseGPT by default) and a low-rank subproblem solved with Adam on a rescaled factorized objective. On Llama3-8B with a 2:4 sparsity component plus a rank-64 component, the method reduces WikiText-2 perplexity by 12% and narrows the average zero-shot gap to the dense model by 15% compared with OATS. The paper also shows that Wanda and OATS are exact minimizers of diagonal-approximation versions of the same objective, which positions the full-Hessian objective as the unifying target and explains where the earlier methods leave performance on the table.

What carries the argument

The central object is the layer-wise reconstruction objective with its Hessian $H=X^\top X$, which is built once from calibration activations and reused throughout alternating minimization. The load-bearing identity is that prior methods are exact solutions of diagonal-Hessian relaxations: with $H=\mathrm{diag}(X^\top X)$, the sparse subproblem's optimum is hard-thresholding $D\widetilde{W}$ with $D=\sqrt{\mathrm{diag}(X^\top X)}$, and the low-rank subproblem's closed form is $D^{-1}\mathcal{C}_r(D\bar{W})$, the rescaled truncated SVD. HASSLE-free replaces those relaxations by the full Hessian, reparametrizes the low-rank component as $UV^\top$, and applies a diagonal rescaling by $D$ to stabilize gradient descent; the paper argues the rescaling preserves the objective and lets one learning rate work across layers and models. SparseGPT is the plug-in solver for the sparse subproblem and Adam for the low-rank subproblem.

What would settle it

Run HASSLE-free with the sparse subproblem solver replaced by a diagonal-Hessian pruner such as Wanda while keeping the same alternating-minimization schedule and Adam low-rank solver; if the perplexity gains over OATS largely disappear, the full-Hessian objective is not the source of the improvement. A complementary check is to run OATS with its low-rank solver replaced by Adam: if OATS then matches HASSLE-free, the reported advantage comes from the optimizer choice rather than from minimizing the exact objective.

Watch

Extended reading notes

Core claim

The central claim is that existing sparse-plus-low-rank LLM compression methods solve only a relaxation of the true optimization problem. When the Hessian of the layer-wise squared reconstruction error is replaced by its diagonal, the sparse subproblem reduces to Wanda-style magnitude pruning and the low-rank subproblem has a closed-form OATS update; HASSLE-free instead minimizes the exact objective $\min_{W_S,M}\|X\widehat{W}-X(W_S+M)\|_F^2$ subject to a sparsity constraint and rank bound, using the full Hessian $H=X^\top X$. Empirically, this gives consistently lower local layer-wise reconstruction error and better perplexity and zero-shot accuracy across Llama3 and Llama3.2 models at several compression settings. The headline result is for Llama3-8B with 2:4 sparsity plus rank 64, where WikiText-2 perplexity drops by 12% and the average zero-shot gap to the dense model shrinks by 15% relative to OATS.

Load-bearing premise

The load-bearing premise is that the alternating-minimization loop, using SparseGPT for the sparse step and Adam for the factorized low-rank step, actually reaches near-optimal solutions of the full layer-wise objective within the 80 outer iterations used in the experiments; if the subproblem solvers stall at poor local points, the reported advantage over OATS could stem from solver behavior rather than from the full-Hessian objective itself.

Editorial extensions

If this is right

  • One-shot sparse-plus-low-rank compression of billion-parameter LLMs can be improved without retraining or fine-tuning by using the full-Hessian layer-wise objective instead of diagonal approximations.
  • The alternating-minimization template accepts any pruning method for the sparse subproblem, so future improvements in one-shot pruning should transfer directly into better sparse-plus-low-rank decompositions.
  • Because the Hessian is computed once and inverted once per layer, the extra cost of the exact objective over diagonal methods is amortized across the 80 outer iterations, keeping the method practical at the 1B–8B scale tested.
  • The analysis that Wanda and OATS exactly minimize diagonal-Hessian relaxations gives a mathematical explanation of their success and identifies their performance ceiling relative to the full-Hessian objective.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An ablation that keeps the alternating-minimization schedule and low-rank solver fixed but swaps SparseGPT for Wanda in the sparse subproblem would isolate whether the gains come from the full-Hessian objective or from the solver choice; the paper does not report this comparison.
  • The printed diagonal-scaling equivalence in equation (7) is stated with a scaling that the implementation does not literally execute, so the practical benefit of the scaling should be read as a stability device whose exact effect may depend on the code's rescaling.
  • The same modular framework should extend to quantization-plus-low-rank decomposition by replacing the sparse subproblem with a quantizer, a direction the paper names as future work.
  • If the 2:4 sparse-plus-rank-64 result holds at scale, the practical payoff is that models like Llama3-8B could be served on GPUs with 2:4 sparse tensor cores with much smaller quality loss than current sparse-plus-low-rank methods at the same memory budget.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces HASSLE-free, a one-shot sparse-plus-low-rank matrix decomposition framework for compressing large language models. The method minimizes a layer-wise reconstruction objective in the original weight space via alternating minimization: the sparse subproblem is solved with SparseGPT and the low-rank subproblem with Adam on a factored representation, optionally with a diagonal rescaling for numerical stability. The authors claim that prior work such as OATS solves a relaxed version of the objective and that HASSLE-free directly minimizes the exact objective. Experiments on Llama-3 and Llama-3.2 models report lower perplexity and better zero-shot accuracy than OATS in most settings, with a headline result of 12% perplexity reduction and 15% zero-shot gap reduction for Llama3-8B with 2:4 sparsity plus rank-64 low-rank component.

Significance. If the described algorithm is correct, HASSLE-free would provide a practical improvement over OATS for hardware-friendly 2:4 sparse-plus-low-rank compression, a setting with demonstrated inference acceleration on GPUs. The paper also contributes a useful taxonomy connecting existing methods (Magnitude Pruning, Wanda, OATS) as special cases of the proposed objective with different Hessian approximations. The empirical tables are extensive and cover multiple model sizes and compression settings. However, the correctness of the central algorithmic derivation is compromised by an algebraic error in the diagonal scaling (Eq. 7 and Algorithm 2), and the reported gains are not uniform across all settings. The strength of the empirical claims therefore cannot be fully assessed without either a corrected derivation or released code that matches the text.

major comments (3)
  1. [Section 5.2, Tables 1 and 3] The diagonal scaling reformulation is algebraically inconsistent. The original objective in Eq. (6) is Tr[(\bar W - UV^T)^T H (\bar W - UV^T)]. Performing the change of variables \tilde U = D U, \tilde V = V yields Tr[(D\bar W - \tilde U\tilde V^T)^T D^{-1} H D^{-1} (D\bar W - \tilde U\tilde V^T)], with the minimizer M^* = D^{-1}\tilde U_*\tilde V_*^T. In contrast, Eq. (7) states M^* = D U_* V_*^T, and Algorithm 2 rescales by U \leftarrow D U after the low-rank solve. If Low-Rank-GD returns U,V such that UV^T approximately equals D\bar W, then the final recovered low-rank matrix is approximately D^2\bar W, not \bar W. The same conclusion follows from Theorem 4.1, whose minimizer is D^{-1} C_r(D\bar W). Thus the printed algorithm does not minimize the layer-wise reconstruction objective of Eq. (1). Since the headline results in Table 2 (12% WT2 perplexity reduction and 15% zero-shot gap reduction for Llama3-8B 2:4+64LR) are attributed to this algorithm, the experiments cannot be tied to the method as written without a corrected equation/algorithm or a code release showing the implemented scaling.
  2. [Section 4.3 and abstract] The claim in the abstract that HASSLE-free 'substantially outperforms' existing methods is not uniform across the reported settings. In the fixed compression-ratio setting for Llama3-8B (Table 1 and Table 3), Ours-2:8+LR has higher WikiText-2 perplexity than OATS-2:8+LR (15.03 vs. 14.54), and the zero-shot average improvement is only 0.29 points (58.28 vs. 57.99). This contradicts the broad wording of the abstract and Section 1, which present the improvement as universal. The headline 12%/15% numbers are for a specific configuration (2:4+64LR); the paper should either qualify the summary statements or explain why the 2:8+LR case is worse.
  3. [Algorithm 2] The claims of 'exact' and 'approximation-free' minimization are overstated. While the objective in Eq. (1) is the full layer-wise reconstruction error and is not a relaxation of the kind used by OATS, the subproblems are solved by heuristic methods: SparseGPT for (P1) and Adam on a nonconvex factored objective for (P2), with a fixed budget of T_AM=80 outer iterations. These solvers do not return exact minimizers, so the method minimizes the objective only approximately. The text in Section 4.3 ('we aim to minimize Equation (1) in an approximation-free manner') should be rephrased to say that the objective itself is not approximated, not that the optimization is exact.
minor comments (4)
  1. [Section 4.2.3] The initialization line 'V ← NNout×r' is not valid notation; it should be 'V ← random Gaussian matrix in R^{Nout×r}' or similar.
  2. [Section 5.2] The statement that the scaling 'sets the diagonal of the new Hessian to 1_{Nin}' is not exact: for H = X^T X + λI, the diagonal of D^{-1}HD^{-1} is 1 + λ / diag(X^T X)_ii, not identically 1 when λ > 0.
  3. [Section 5.2] The table caption says 'rank r = 64' but some rows are labeled 'Ours-2:8+64LR' while the abstract and text use '2:8+LR'; please standardize the naming to avoid confusion.
  4. [Section 5.3] Figure 1 reports layer-wise reconstruction errors, but the y-axis scaling marks (×10^2, ×10^2, ×10, ×1, ×10^4, ×10^4, ×10) are unusual and make the figure hard to read. Please provide a single consistent scale or explicit per-panel error values.

Circularity Check

1 steps flagged · score 2.0 of 10

No load-bearing circularity: the central derivation is self-contained and the main Table 2 comparison to OATS is external, but Table 1 uses self-cited ALPS baselines and the printed diagonal rescaling is internally inconsistent (a non-circular correctness issue).

  1. self citation load bearing [Appendix A.1, Experimental Setup]
    "The LLM evaluation benchmarks reported in Table 1 are retrieved from the paper ALPS by Meng et al.[18] which uses the same evaluation strategy (and code) we do for the reported tasks [other zero-shot tasks are not reported in ALPS]."

    Table 1 is used as evidence that HASSLE-free outperforms OATS at a fixed compression ratio, and the baseline numbers are taken from ALPS, a paper by the present authors (Meng, Behdin, Ponomareva, Mazumder). This is a minor self-citation: the comparison in Table 1 is not fully external. It is not load-bearing for the paper's central 12%/15% claims, which are based on Table 2 where OATS results come from the OATS authors' own implementation rather than from the self-cited ALPS table. The self-citation supplies convenience baselines, not the target result, so it does not by itself force the claimed improvement.

full rationale

The derivation chain is not circular. Objective (1) is a standard layer-wise reconstruction error; the sparse subproblem is delegated to an external method (SparseGPT) and the low-rank subproblem is solved either by the classical Eckart-Young truncation (diagonal case, Theorem 4.1) or by first-order optimization on a factorized form. The main empirical claims in Table 2 are comparisons against the authors' own runs of OATS using OATS's public implementation, so the held-out perplexity and zero-shot numbers are independent of HASSLE-free's fitted outputs. The only self-citation entering the evidence is the ALPS-sourced baseline numbers in Table 1; this is minor and not load-bearing for the central claim. A separate, non-circular concern is that Section 4.2.3 and Algorithm 2 rescale with 'U ← D U' while claiming equivalence to Eq. (6): the stated change of variables would require rescaling the returned low-rank factor by D^{-1}, so the printed algorithm does not minimize the claimed objective. This is an internal consistency/implementation issue, not a circular reduction of the result to its inputs, and therefore it does not raise the circularity score beyond 2.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claim rests on standard compression surrogacy, an explicit invertibility assumption, and two heuristic subproblem solvers (SparseGPT and Adam) whose optimality is not established; no new physical or mathematical entities are introduced.

free parameters (5)
  • λ (Hessian regularization) = 0.01 * Tr(H)
    Added to H for numerical stability; taken from SparseGPT defaults and affects all pruning and low-rank updates.
  • T_AM (outer alternating-minimization iterations) = 80
    Set to the OATS default; performance depends on convergence of the alternating scheme.
  • T_LR (low-rank gradient-descent iterations) = 50
    Number of Adam steps for the low-rank subproblem; proposed as a default without sensitivity analysis.
  • η (learning rate for U and V) = 1e-2
    The paper states this learning rate works for all layers and models only with the diagonal scaling.
  • r (rank) in fixed-rank experiments = 64
    Target rank for all layers; in fixed-ratio experiments r is derived from the target compression ratio and sparsity pattern.
assumptions (6)
  • domain assumption Assumption 1: diag(X^T X) is full rank, equivalently no column of X is identically zero.
    Used in Theorem 4.1 and the diagonal rescaling in Eq (7); if a column is zero, D is singular.
  • domain assumption SparseGPT yields a sufficiently accurate minimizer of the sparse subproblem (P1).
    HASSLE-free's 'exact' claim relies on the pruning plug-in solving P1 well; SparseGPT itself is an approximate second-order heuristic.
  • domain assumption Adam on the factorized nonconvex low-rank problem converges to a good rank-r solution.
    No global optimality guarantee is provided for the low-rank subproblem; local minima could make the reported gains solver-dependent.
  • domain assumption Minimizing layer-wise reconstruction error improves end-to-end LLM quality.
    The whole compression pipeline assumes local reconstruction error is a valid surrogate for downstream perplexity and zero-shot accuracy.
  • domain assumption Calibration data (128 segments of 2048 tokens from C4) is representative for reconstructing test-time activations.
    The layer-wise Hessian H=X^T X is built once from this sample and used for all compression decisions.
  • standard math Eckart-Young theorem for truncated SVD.
    Used for the closed-form low-rank solution in the data-free and diagonal-approximation cases.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HASSLE-free: A unified Framework for Sparse plus Low-Rank Matrix Decomposition for LLMs." pith.science (2026). https://pith.science/paper/IUCAJPPX

@misc{pith2026250200899,
  author       = {Pith},
  title        = {Pith review of: HASSLE-free: A unified Framework for Sparse plus Low-Rank Matrix Decomposition for LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IUCAJPPX}},
  note         = {Machine review of arXiv:2502.00899}
}
read the original abstract

The impressive capabilities of large foundation models come at a cost of substantial computing resources to serve them. Compressing these pre-trained models is of practical interest as it can democratize deploying them to the machine learning community at large by lowering the costs associated with inference. A promising compression scheme is to decompose foundation models' dense weights into a sum of sparse plus low-rank matrices. In this paper, we design a unified framework coined HASSLE-free for (semi-structured) sparse plus low-rank matrix decomposition of foundation models. Our framework introduces the local layer-wise reconstruction error objective for this decomposition, we demonstrate that prior work solves a relaxation of this optimization problem; and we provide efficient and scalable methods to minimize the exact introduced optimization problem. HASSLE-free substantially outperforms state-of-the-art methods in terms of the introduced objective and a wide range of LLM evaluation benchmarks. For the Llama3-8B model with a 2:4 sparsity component plus a 64-rank component decomposition, a compression scheme for which recent work shows important inference acceleration on GPUs, HASSLE-free reduces the test perplexity by 12% for the WikiText-2 dataset and reduces the gap (compared to the dense model) of the average of eight popular zero-shot tasks by 15% compared to existing methods.

Figures

Figures reproduced from arXiv: 2502.00899 by the authors.

Figure 1
Figure 1. Local layer-wise reconstruction error ↓ (lower values are preferred) analysis of the decomposition of the layers of the first trans￾former block in Llama-3-8B into a 2:4 sparse com￾ponent plus a 64-rank low-rank component. All methods use the same number of Alternating￾Minimization steps 80. q_proj k_proj v_proj o_proj gate_proj up_proj down_proj Transformer blocks layers 0 1 2 3 4 5 6 7 O b j e c tiv e E r r o r ×1… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 22 canonical work pages

  1. [1]

    Sparks of artificial general intelligence: Early experiments with gpt-4.arXiv preprint arXiv:2303.12712, 2023

    Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4.arXiv preprint arXiv:2303.12712, 2023

    arXiv 2023

  2. [2]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team Google. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    arXiv 2023

  4. [4]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  5. [5]

    Optimalbraindamage

    YannLeCun,JohnDenker,andSaraSolla. Optimalbraindamage. Advancesinneuralinformation processing systems, 2, 1989

    work page 1989

  6. [6]

    Second order derivatives for network pruning: Optimal brain surgeon

    Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5, 1992

    1992

  7. [7]

    Fast as chita: Neural network pruning with combinatorial optimization

    Riade Benbaki, Wenyu Chen, Xiang Meng, Hussein Hazimeh, Natalia Ponomareva, Zhe Zhao, and Rahul Mazumder. Fast as chita: Neural network pruning with combinatorial optimization. arXiv preprint arXiv:2302.14623, 2023

    arXiv 2023

  8. [8]

    Fast convnets using group-wise brain damage

    Vadim Lebedev and Victor Lempitsky. Fast convnets using group-wise brain damage. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2554–2564, 2016

    work page 2016

Show all 55 references
  1. [9]

    Learning structured sparsity in deep neural networks.Advances in neural information processing systems, 29, 2016

    Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks.Advances in neural information processing systems, 29, 2016

    2016

  2. [10]

    Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned.arXiv preprint arXiv:1905.09418, 2019

    Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned.arXiv preprint arXiv:1905.09418, 2019

    1905 arXiv

  3. [11]

    Data-efficient structured pruning viasubmodularoptimization

    Marwa El Halabi, Suraj Srinivas, and Simon Lacoste-Julien. Data-efficient structured pruning viasubmodularoptimization. AdvancesinNeuralInformationProcessingSystems ,35:36613–36626, 2022

    2022

  4. [12]

    Learning n: m fine-grained structured sparse neural networks from scratch

    Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning n: m fine-grained structured sparse neural networks from scratch. arXiv preprint arXiv:2102.04010, 2021

    2021 arXiv

  5. [13]

    Learning both weights and connections for efficient neural network.Advances in neural information processing systems, 28, 2015

    Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network.Advances in neural information processing systems, 28, 2015

    2015

  6. [14]

    Dynamic network surgery for efficient dnns

    Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. Advances in neural information processing systems, 29, 2016

    2016

  7. [15]

    The optimal bert surgeon: Scalable and accurate second-order pruning for large language models, 2022

    Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal bert surgeon: Scalable and accurate second-order pruning for large language models, 2022. URLhttps://arxiv.org/abs/2203.07259

    2022 arXiv

  8. [16]

    Inducingandexploiting activation sparsity for fast inference on deep neural networks

    Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr, Michael Goin, WilliamLeiserson,SageMoore,BillNell,NirShavit,andDanAlistarh. Inducingandexploiting activation sparsity for fast inference on deep neural networks. In Hal Daumé III and Aarti Singh, edito...

    2020

  9. [17]

    Howwelldosparseimagenet models transfer?CoRR, abs/2111.13445, 2021

    EugeniaIofinova,AlexandraPeste,MarkKurtz,andDanAlistarh. Howwelldosparseimagenet models transfer?CoRR, abs/2111.13445, 2021. URLhttps://arxiv.org/abs/2111.13445

    2021 arXiv

  10. [18]

    Alps: Improved op- timization for highly sparse one-shot pruning for large language models.arXiv preprint arXiv:2406.07831, 2024

    Xiang Meng, Kayhan Behdin, Haoyue Wang, and Rahul Mazumder. Alps: Improved op- timization for highly sparse one-shot pruning for large language models.arXiv preprint arXiv:2406.07831, 2024

    2024 arXiv

  11. [19]

    Sparsegpt: Massive language models can be accurately pruned in one-shot

    Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. InInternational Conference on Machine Learning, pages 10323–10337. PMLR, 2023

    2023

  12. [20]

    Asimpleandeffectivepruningapproach for large language models.arXiv preprint arXiv:2306.11695, 2023

    MingjieSun,ZhuangLiu,AnnaBair,andJZicoKolter. Asimpleandeffectivepruningapproach for large language models.arXiv preprint arXiv:2306.11695, 2023

    2023 arXiv

  13. [21]

    Dynamic sparse no training: Training-free fine-tuning for sparse llms

    Yuxin Zhang, Lirui Zhao, Mingbao Lin, Yunyun Sun, Yiwu Yao, Xingjia Han, Jared Tanner, Shiwei Liu, and Rongrong Ji. Dynamic sparse no training: Training-free fine-tuning for sparse llms. arXiv preprint arXiv:2310.08915, 2023

    2023 arXiv

  14. [22]

    Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068, 2022

    SusanZhang,StephenRoller,NamanGoyal,MikelArtetxe,MoyaChen,ShuohuiChen,Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

  15. [23]

    Adam: Amethodforstochasticoptimization

    DiederikPKingma. Adam: Amethodforstochasticoptimization. arXivpreprintarXiv:1412.6980 , 2014

    2014 arXiv

  16. [24]

    Robust principal component pursuit via inexact alternating minimization on matrix manifolds.Journal of Mathematical Imaging and Vision, 51(3):361–377, 2015

    Michael Hintermüller and Tao Wu. Robust principal component pursuit via inexact alternating minimization on matrix manifolds.Journal of Mathematical Imaging and Vision, 51(3):361–377, 2015

    2015

  17. [25]

    Robust principal component analysis? Journal of the ACM (JACM), 58(3):1–37, 2011

    Emmanuel J Candès, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis? Journal of the ACM (JACM), 58(3):1–37, 2011

    2011

  18. [26]

    Linearized alternating direction method with adaptive penalty for low-rank representation.Advances in neural information processing systems, 24, 2011

    Zhouchen Lin, Risheng Liu, and Zhixun Su. Linearized alternating direction method with adaptive penalty for low-rank representation.Advances in neural information processing systems, 24, 2011

    2011

  19. [27]

    Parrilo, and Alan S

    Venkat Chandrasekaran, Sujay Sanghavi, Pablo A. Parrilo, and Alan S. Willsky. Sparse and low-rank matrix decompositions. In2009 47th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 962–967, 2009. doi: 10.1109/ALLERTON.2009.5394889

    2009

  20. [28]

    Godec: Randomized low-rank & sparse matrix decomposition in noisy case

    Tianyi Zhou and Dacheng Tao. Godec: Randomized low-rank & sparse matrix decomposition in noisy case. InProceedings of the 28th International Conference on Machine Learning, ICML 2011, 2011

    2011

  21. [29]

    Dimitris Bertsimas, Ryan Cory-Wright, and Nicholas A. G. Johnson. Sparse plus low rank matrix decomposition: A discrete optimization approach.Journal of Machine Learning Research, 24(267):1–51, 2023. URLhttp://jmlr.org/papers/v24/21-1130.html

    2023

  22. [30]

    Non-convex robust pca

    Praneeth Netrapalli, Niranjan U N, Sujay Sanghavi, Animashree Anandkumar, and Prateek Jain. Non-convex robust pca. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors,Advances in Neural Information Processing Systems, volume 27. Curran Associates...

    2014

  23. [31]

    On compressing deep models by low rank and sparse decomposition

    Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. On compressing deep models by low rank and sparse decomposition. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 7370–7379, 2017. 12

    2017

  24. [32]

    Losparse: Structured compression of large language models based on low-rank and sparse approximation

    Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Losparse: Structured compression of large language models based on low-rank and sparse approximation. InInternational Conference on Machine Learning, pages 20336–20350. PMLR, 2023

    2023

  25. [33]

    Oats: Outlier-aware pruning through sparse and low rank decomposition

    Stephen Zhang and Vardan Papyan. Oats: Outlier-aware pruning through sparse and low rank decomposition. arXiv preprint arXiv:2409.13652, 2024

    2024 arXiv

  26. [34]

    Slope: Double-pruned sparse plus lazy low-rank adapter pretraining of llms.arXiv preprint arXiv:2405.16325, 2024

    Mohammad Mozaffari, Amir Yazdanbakhsh, Zhao Zhang, and Maryam Mehri Dehnavi. Slope: Double-pruned sparse plus lazy low-rank adapter pretraining of llms.arXiv preprint arXiv:2405.16325, 2024

    2024 arXiv

  27. [35]

    Springer, 2020

    Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel S Emer.Efficient processing of deep neural networks. Springer, 2020

    2020

  28. [36]

    Loftq: Lora-fine-tuning-aware quantization for large language models.arXiv preprint arXiv:2310.08659, 2023

    Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, and Tuo Zhao. Loftq: Lora-fine-tuning-aware quantization for large language models.arXiv preprint arXiv:2310.08659, 2023

    2023 arXiv

  29. [37]

    Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning.arXiv preprint arXiv:2311.12023, 2023

    Han Guo, Philip Greengard, Eric P Xing, and Yoon Kim. Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning.arXiv preprint arXiv:2311.12023, 2023

    2023 arXiv

  30. [38]

    The approximation of one matrix by another of lower rank

    Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika, 1(3):211–218, 1936

    1936

  31. [39]

    Rank-sparsity incoherence for matrix decomposition.SIAM Journal on Optimization, 21(2):572–596, 2011

    Venkat Chandrasekaran, Sujay Sanghavi, Pablo A Parrilo, and Alan S Willsky. Rank-sparsity incoherence for matrix decomposition.SIAM Journal on Optimization, 21(2):572–596, 2011

    2011

  32. [40]

    Distributed opti- mizationandstatisticallearningviathealternatingdirectionmethodofmultipliers

    Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. Distributed opti- mizationandstatisticallearningviathealternatingdirectionmethodofmultipliers. Foundations and Trends®in Machine learning, 3(1):1–122, 2011

    2011

  33. [41]

    Osscar: One-shot structured pruning in vision and language models with combinatorial optimization.arXiv preprint arXiv:2403.12983, 2024

    Xiang Meng, Shibal Ibrahim, Kayhan Behdin, Hussein Hazimeh, Natalia Ponomareva, and Rahul Mazumder. Osscar: One-shot structured pruning in vision and language models with combinatorial optimization.arXiv preprint arXiv:2403.12983, 2024

    2024 arXiv

  34. [42]

    Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions.SIAM review, 53 (2):217–288, 2011

    Nathan Halko, Per-Gunnar Martinsson, and Joel A Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions.SIAM review, 53 (2):217–288, 2011

    2011

  35. [43]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of Machine Learning Research, 21(140):1–67, 2020. URLhttp: //...

    2020

  36. [44]

    URL https://huggingface.co/docs/transformers/ perplexity

    Perplexityoffixed-lengthmodels,2022. URL https://huggingface.co/docs/transformers/ perplexity

    2022

  37. [45]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. InInternationalConferenceonLearningRepresentations ,2017. URL https://openreview. net/forum?id=Byj72udxe

    2017

  38. [46]

    The penn treebank: Annotating predicate argument structure

    Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. The penn treebank: Annotating predicate argument structure. InProceedings of the Workshop on Human Language Technology, HLT ’94, page 114–119, USA...

    1994

  39. [47]

    A framework for few-shot language model evaluation, 12 2023.URL https://zenodo

    L Gao, J Tow, B Abbasi, S Biderman, S Black, A DiPofi, C Foster, L Golding, J Hsu, A Le Noac’h, et al. A framework for few-shot language model evaluation, 12 2023.URL https://zenodo. org/records/10256836, 7

    2023

  40. [48]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. InProceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020

    2020

  41. [49]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  42. [50]

    Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  43. [51]

    Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64(9):99–106, 2021

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64(9):99–106, 2021

    2021

  44. [52]

    A survey on recognizing textual entailment as an nlp evaluation.arXiv preprint arXiv:2010.03061, 2020

    Adam Poliak. A survey on recognizing textual entailment as an nlp evaluation.arXiv preprint arXiv:2010.03061, 2020

    2010 arXiv

  45. [53]

    Careful selection of knowledge to solve open book question answering.arXiv preprint arXiv:1907.10738, 2019

    Pratyay Banerjee, Kuntal Kumar Pal, Arindam Mitra, and Chitta Baral. Careful selection of knowledge to solve open book question answering.arXiv preprint arXiv:1907.10738, 2019

    1907 arXiv

  46. [54]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019

    1905 arXiv

  47. [55]

    Interactive supercomputing on 40,000 cores for machine learning and data analysis

    AlbertReuther,JeremyKepner,ChansupByun,SiddharthSamsi,WilliamArcand,DavidBestor, Bill Bergeron, Vijay Gadepally, Michael Houle, Matthew Hubbell, Michael Jones, Anna Klein, Lauren Milechin, Julia Mullen, Andrew Prout, Antonio Rosa, Charles Yee, and Peter Michaleas. Interactive ...

    2018

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.