Pith. sign in

REVIEW 4 major objections 5 minor 97 references

Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A LoRA adapter trained from a high-rank initialization and slowly pruned during fine-tuning can match full fine-tuning while deploying at rank 64.

desk verdict A genuinely useful empirical recipe for compact low-rank adapters, undercut by a circular robustness proof and some over-claiming. read the letter →

arxiv 2510.00192 v3 pith:5547JMA6 submitted 2025-09-30 cs.LG cs.AI

Xin Yu , Cong Xie , Xunmei Liu , Tiantian Fan , Lingzhou Xue , Zhi Zhang This is my paper
classification cs.LG cs.AI
keywords low-rank adaptationstructured pruninggradient-based pruningsecond-order optimizationparameter-efficient fine-tuninglarge language modelsloss-aware compressionover-parameterized initialization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PrunedLoRA is built on a simple trade: initialize a low-rank adapter with a much larger rank than the deployment budget, then prune it down during fine-tuning using structured, gradient-informed pruning. The paper aims to show that this recovers most of the performance of full fine-tuning — matching it on a math reasoning benchmark (73.38 vs. 73.31) and nearly matching on a natural-language-understanding suite (87.19 vs. 87.91) — while the adapter shipped at the end is only rank 64. The authors also try to prove that gradient-based structured pruning is more robust than activation-based pruning under weight perturbation, and to give a second-order interpretation of the pruning mask and update. If right, it would mean the long-standing gap between low-rank adaptation and full fine-tuning is mostly a matter of where you start and how you shrink, not of the final rank itself.

What carries the argument

The engine is a structured pruning step applied jointly to the two LoRA matrices A and B: the pruning mask is chosen by minimizing the second-order Taylor approximation of the loss change, using a Hessian estimate built from the gradient outer product, and a closed-form update readjusts the surviving rows and columns to compensate. Pruning B's columns and A's rows simultaneously lowers the adapter's rank while preserving, in approximation, the loss landscape. The same machinery yields an interpretation of the classic 'saliency' term in second-order pruning: select the columns whose Newton-updated values are least important under the Hessian-weighted quadratic metric.

What would settle it

Run a controlled pruning experiment at multiple checkpoints: from the same trained high-rank adapter, prune to the same target rank using a gradient-based criterion and an activation-based criterion, and record the actual loss immediately after pruning. If activation-based pruning ever shows smaller or equal loss increases, the claimed robustness ordering is falsified. A second check is to compare the gradient-outer-product Hessian estimate with a sampled true Hessian on a small adapter to see whether the approximation is reliable.

Watch

Extended reading notes

Core claim

The paper's central claim is that a low-rank adapter can carry almost all the expressiveness of full fine-tuning if it is trained from an over-parameterized high-rank initialization and then progressively pruned with a gradient-based, loss-aware structured pruning procedure. The resulting adapter matches full fine-tuning on GSM8K (73.38 vs. 73.31) and comes within about a point of it on GLUE (87.19 vs. 87.91) while using only a small fraction of the parameters at inference. The authors further claim, and attempt to prove on a single attention module, that gradient-based pruning is more robust than activation-based pruning with respect to the overall loss, because activation-based error bound

Load-bearing premise

The theoretical robustness conclusion rests on a definition that sets the gradient-based pruning error directly as the change in the overall loss, so the main bound is built into the definition, and the practical algorithm also assumes the gradient outer product and row independence give a faithful Hessian.

Editorial extensions

If this is right

  • LoRA can be trained at full-parameter scale and still end as a compact adapter: starting at rank 512 and pruning to rank 64.
  • Larger initialization ranks consistently improve final performance, so the initial parameter space — not the final rank — drives quality.
  • Gradual pruning with moderate intervals beats one-shot pruning at the same target rank, and the benefit is universal across pruning strategies.
  • Gradient-based pruning outperforms activation-based pruning at the same sparsity, including extreme compression to very small target ranks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: the theoretical comparison predicts that activation-based pruning's loss error grows with the magnitude of attention module weights during training, while gradient-based error stays flat; this could be checked directly while fine-tuning.
  • If the gradient-outer-product Hessian approximation holds, the same second-order pruning machinery could be applied to other adapter or projection schemes, not just LoRA.
  • The idea that initialization rank, not final rank, carries capacity suggests that memory-constrained settings could schedule adapter rank over time—train large early, shrink later—without changing the deployed artifact.
  • One could set heterogeneous per-layer target ranks chosen by the same loss-aware criterion rather than a uniform budget, potentially improving extreme compression.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes PrunedLoRA, a structured-pruning framework for LoRA fine-tuning. LoRA adapters are initialized at a high rank (128-512) and then progressively pruned down to a target low rank (e.g., 64) using a gradient-based importance criterion with a second-order update derived from an approximate Hessian H_hat = (∇L)^T(∇L). The paper claims a first theoretical analysis proving that gradient-based structured pruning is more robust than activation-based pruning with respect to overall loss, and reports experiments on Llama-3-8B GSM8K and T5 GLUE showing that PrunedLoRA outperforms LoRA, DoRA, AdaLoRA, SparseGPT, and LLM-Pruner, with init rank 512 reaching 73.38 on GSM8K versus 73.31 for full fine-tuning and 87.19 average GLUE versus 87.91 for full fine-tuning. The central theoretical claim, however, is not supported as stated: Definition 1 defines gradient-based perturbation error directly as loss error, so Proposition 2 restates the assumption rather than proving a comparison between pruning methods.

Significance. The empirical contribution is potentially valuable: the paper provides a clean operational recipe — high-rank initialization plus structured gradient-informed pruning to a low-rank adapter — and the ablations on initialization rank, scaling factor, pruning schedule, and target rank are useful. If the practical claims hold, the method narrows the LoRA/FFT gap while preserving low-rank inference efficiency. However, the advertised theoretical result is the main novelty emphasized in the abstract and contributions, and that result is currently circular. The Hessian approximation on which the algorithm relies is also unvalidated. No code or machine-checked proofs are provided, limiting reproducibility. As it stands, the paper's credible contribution is empirical, and the theoretical narrative needs to be removed or substantially reworked.

major comments (4)
  1. [Appendix B.1, Definition 1 and Proposition 2] Definition 1 defines gradient-based ε-perturbation error directly as |L(Ŵ) − L(W)| ≤ ε. Proposition 2's conclusion that the loss change is at most 3ε for Q, K, V is then a direct consequence of the definition plus the triangle inequality; no comparison between pruning strategies is established. The proof never connects an actual gradient-based pruning rule, such as Eq. (2) or Eq. (8), to this loss-error bound. Thus the abstract's 'provably show' and Section 3.2's theoretical motivation are unsupported. To keep the claim, derive a loss-error bound from the weight perturbation induced by the proposed pruning criterion under explicit Lipschitz/curvature assumptions; otherwise the theorem should be removed or restated as a definitional observation.
  2. [Appendix B.1, activation-based case of Proposition 2] The activation-based bound is not derived from Definition 1 either. The proof assumes ||Q − Qhat||_F ≤ ε, ||K − Khat||_F ≤ ε, and ||V − Vhat||_F ≤ ε, but Definition 1 defines activation-based perturbation as output reconstruction error ||ŴX − WX|| ≤ ε. These are not equivalent without assumptions on X (e.g., its smallest singular value), and no such connection is stated. Consequently the comparison between activation- and gradient-based perturbation is made with incompatible metrics, further undermining the claimed robustness result.
  3. [Section 3.2, Eqs. (5)-(8)] The algorithm's pruning criterion and update depend on the approximation H_hat = (∇_W L)^T (∇_W L) as an n×n Hessian under 'row independence.' This discards cross-row Hessian blocks and is a strong structural assumption. The exact second-order term in Eq. (4) is vec(δ)^T H vec(δ); replacing it by tr(δ^T H_hat δ) is not validated by any theorem or ablation. Since Eqs. (8), (10), and Algorithm 1 are derived from this approximation, the claim of a 'grounded interpretation' for the pruning metric is not supported without additional justification.
  4. [Section 4.1, Table 1] The text states that PrunedLoRA with init r=512 'matches or surpasses full fine-tuning (73.38 on GSM8K).' Full fine-tuning is 73.31 ± 0.32 and PrunedLoRA is 73.38 ± 0.42; the error bars overlap substantially, so this is not a statistically supported superiority claim. On GLUE (Table 3), PrunedLoRA's 87.19 is below Full FT's 87.91. Please soften to 'statistically indistinguishable on GSM8K' or provide paired significance tests.
minor comments (5)
  1. [Abstract and Section 4] The abstract mentions 'code generation' among the evaluated tasks, but the experimental section includes only mathematical reasoning and natural language understanding. Either add code-generation experiments or remove this from the list.
  2. [Eq. (6) vs Eq. (5)] Eq. (5) uses tr(δ^T H_hat δ), while Eq. (6) uses tr(δ H_hat δ^T). These are different quadratic forms unless additional trace identities are assumed; please unify the notation and state the intended expression.
  3. [Eq. (9)] In the optimization problem for A and B, the term for δ_A is written as 1/2 tr(δ_A^T cHB δ_A), which should presumably be cHA, not cHB. This appears to be a typo but affects reproducibility.
  4. [Section 3.2] The main text labels a statement as 'Proposition 1 (Unofficial Statement)' while the formal proposition appears only in Appendix B.1 as 'Proposition 2 (Official Statement).' The numbering and label are confusing; please align the main-text and appendix numbering.
  5. [Appendix D, Table 8] The text calls 4:8 and 2:4 patterns 'structured sparsity.' These are semi-structured N:M sparsity patterns, not structured pruning in the usual filter/column sense; please use the standard terminology.

Circularity Check

1 steps flagged · score 8.0 of 10

Proposition 2's theoretical robustness claim is definitional: gradient-based ε-perturbation is defined as a loss bound, so the 'proof' is a triangle-inequality restatement of the assumption.

  1. self definitional [Appendix B.1, Definition 1 and Proposition 2 (with proof)]
    "For gradient-based pruning strategies, we define ε-perturbation error as: |L(Ŵ) − L(W)| ≤ ε ... In contrast, under the perturbation error of gradient-based tuning strategies, if we assume that L(Q, K, V) is the loss of a single attention module, we know that |L(Q,K,V)−L(bQ,bK,bV)| ≤ 3ε, which is a direct consequence of the triangle inequality."

    Gradient-based ε-perturbation is defined directly as a bound on the task loss. Proposition 2 then 'proves' a loss bound of 3ε by applying the triangle inequality to that same assumed loss bound. No step connects the actual gradient-based pruning operation (the column-mask selection of Eq. 8 or the weight update of Eq. 11) to the assumed ε-level loss error. The advertised comparison is between a derived reconstruction/weight-perturbation bound (activation-based) and an assumed loss bound (gradient-based), so the headline 'provably show gradient-based pruning is more robust' is equivalent to its own definition rather than a derived result.

full rationale

The paper's empirical program—PrunedLoRA's high-rank initialization, structured pruning schedule, and comparisons against LoRA, DoRA, AdaLoRA, SparseGPT, LLM-Pruner, Wanda, and magnitude baselines—is self-contained and not circular: performance numbers come from direct experiments, and the pruning/update equations are derived from a stated second-order objective. The circularity is confined to the theoretical robustness claim in Appendix B.1, which is nonetheless advertised as a central contribution ('we provide the first theoretical analysis ... and provably show that ... gradient-based pruning is more robust than activation-based pruning'). Definition 1 defines gradient-based error as a bound on the loss itself, so Proposition 2's conclusion that the overall loss changes by at most 3ε is exactly the assumption repackaged via the triangle inequality. The proof language even says the 3ε bound is 'a direct consequence of the triangle inequality,' confirming the reduction. The activation-based side is a genuine calculation, but it is compared to an assumed loss bound, not to an analogous derived gradient-based bound. Because the comparison that the paper advertises as a proof is forced by definition, the score is high (8) despite the independent empirical content. I do not treat the paper's separate use of Ĥ = (∇L)ᵀ(∇L) under 'row independence' as circularity; it is an explicit Hessian approximation and is not disguised as a theorem. No self-citation load-bearing or uniqueness-imported circularity is present.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical or architectural entities. Its central calculation rests on a second-order Taylor expansion, a row-independence gradient-outer-product Hessian approximation, and a Lipschitz/attention toy model. The strongest theory claim is definitional rather than derived; see the circularity red flag.

free parameters (5)
  • Scaling factor alpha = alpha = r (best of {r/2, r, 2r}; e.g., 128 at init r=128, 512 at r=512)
    Grid searched in Table 4; GSM8K accuracy varies from 67.81 to 73.38 depending on alpha, and alpha is set proportional to current rank during training.
  • Initial rank r = 128, 256, or 512; best results at r=512
    Chosen from {128, 256, 512}; Tables 1, 4, and 5 show larger init ranks give better GSM8K accuracy. This is central to the method's motivation.
  • Pruning schedule (K1, K2) = K1=10, K2=2
    Ablation Table 6 selects the 10/2 schedule from K1 in {5,10} and K2 in {2,4}; other schedules reduce GSM8K accuracy.
  • Learning rate = not stated; grid searched over {1e-5, 5e-6, 1e-6}
    Appendix C.1 says the best result among these learning rates is reported, but the selected value for each task and model is not listed.
  • Hessian damping lambda (one-shot Appendix D) = unspecified scalar > 0
    Introduced for numerical stability in diag(H_hat + lambda I)^{-1} in Appendix D; value not reported.
assumptions (5)
  • domain assumption Second-order Taylor expansion of the loss around W is a good approximation when pruning (Eq. 4/5).
    Higher-order terms are ignored; this is the base of the pruning selection and update, and is not verified against the true loss landscape.
  • domain assumption Row independence assumption: the Hessian can be approximated by H_hat = (grad L)^T (grad L).
    Invoked in Section 3.2 as a standard technique for approximating the Hessian using gradients; unvalidated for LLM weight matrices.
  • domain assumption The loss is C-Lipschitz continuous in the general analysis of Appendix B.1.
    Used to bound activation-based perturbation effects via the Lipschitz constant; prior work cited by the paper states Lipschitz constants tend to grow during training.
  • domain assumption Gradients of the overall loss with respect to adapter matrices A and B are available through backpropagation.
    Required to compute grad_A L, grad_B L and the Hessian estimates; standard in fine-tuning but still a modeling assumption about the training pipeline.
  • ad hoc to paper A single attention module with Q, K, V of the same dimension is representative of extended LLM structured pruning.
    Proposition 2 is proven only for this toy model; the paper asserts the intuition extends to broader settings without a proof.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation." pith.science (2026). https://pith.science/paper/5547JMA6

@misc{pith2026251000192,
  author       = {Pith},
  title        = {Pith review of: Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5547JMA6}},
  note         = {Machine review of arXiv:2510.00192}
}
read the original abstract

Low-rank adaptation (LoRA) has become a widely used paradigm for parameter-efficient fine-tuning of large language models, yet its representational capacity often lags behind full fine-tuning. Within the context of LoRA, a key open question is how to obtain expressive low-rank adapters from over-parameterized spaces. We propose \textit{PrunedLoRA}, a new framework that leverages structured pruning to obtain highly representative low-rank adapters from an over-parameterized initialization. Unlike prior approaches that impose a fixed low-rank budget, PrunedLoRA dynamically prunes less important components during fine-tuning and prevents their reactivation, enabling flexible and adaptive rank allocation. For structured pruning, by minimizing the pruning error for overall loss, we provide fine-grained pruning and recovery updates in a gradient-based pruning strategy with grounded interpretation. We provide the first theoretical analysis of the robustness of structured pruning and provably show that under the impact of weight perturbation, gradient-based pruning is more robust than activation-based pruning with respect to overall loss. Empirically, PrunedLoRA consistently outperforms LoRA and its variants across supervised fine-tuning tasks in mathematical reasoning, code generation, and natural language understanding, and it also demonstrates advantages over existing structured pruning methods across diverse sparsity levels.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

97 extracted references · 24 linked inside Pith

  1. [1]

    Binarybert: Pushing the limit of bert quantization

    Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jin Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. Binarybert: Pushing the limit of bert quantization. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4334–4348, 2021

    2021

  2. [2]

    Federated fine-tuning of large language models under heterogeneous tasks and client resources.Advances in Neural Information Processing Systems, 37: 14457–14483, 2024

    Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, and Yaliang Li. Federated fine-tuning of large language models under heterogeneous tasks and client resources.Advances in Neural Information Processing Systems, 37: 14457–14483, 2024

    2024

  3. [3]

    Ce-lora: Computation-efficient lora fine-tuning for language models.CoRR, 2025

    Guanduo Chen, Yutong He, Yipeng Hu, Kun Yuan, and Binhang Yuan. Ce-lora: Computation-efficient lora fine-tuning for language models.CoRR, 2025

    2025

  4. [4]

    Lorashear: Efficient large language model structured pruning and knowledge recovery.arXiv preprint arXiv:2310.18356, 2023

    Tianyi Chen, Tianyu Ding, Badal Yadav, Ilya Zharkov, and Luming Liang. Lorashear: Efficient large language model structured pruning and knowledge recovery.arXiv preprint arXiv:2310.18356, 2023

    arXiv 2023

  5. [5]

    Hessian-free optimization for learning deep multidimensional recurrent neural networks.Advances in Neural Information Processing Systems, 28, 2015

    Minhyung Cho, Chandra Dhir, and Jaehyung Lee. Hessian-free optimization for learning deep multidimensional recurrent neural networks.Advances in Neural Information Processing Systems, 28, 2015

    2015

  6. [6]

    Heterogeneous low-rank approximation for federated fine-tuning of on-device foundation models.arXiv preprint arXiv:2401.06432, 2024

    Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi. Heterogeneous low-rank approximation for federated fine-tuning of on-device foundation models.arXiv preprint arXiv:2401.06432, 2024

    arXiv 2024

  7. [7]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv 2021

  8. [8]

    Beyond size: How gradients shape pruning decisions in large language models.arXiv preprint arXiv:2311.04902, 2023

    Rocktim Jyoti Das, Mingjie Sun, Liqun Ma, and Zhiqiang Shen. Beyond size: How gradients shape pruning decisions in large language models.arXiv preprint arXiv:2311.04902, 2023

    arXiv 2023

Show all 97 references
  1. [9]

    Predicting parameters in deep learning.Advances in neural information processing systems, 26, 2013

    Misha Denil, Babak Shakibi, Laurent Dinh, Marc’Aurelio Ranzato, and Nando De Freitas. Predicting parameters in deep learning.Advances in neural information processing systems, 26, 2013

    2013

  2. [10]

    Coreset-based neural network compression

    Abhimanyu Dubey, Moitreya Chatterjee, and Narendra Ahuja. Coreset-based neural network compression. In Proceedings of the European Conference on Computer Vision (ECCV), pages 454–470, 2018

    2018

  3. [11]

    Reducing transformer depth on demand with structured dropout

    Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. In International Conference on Learning Representations, 2019

    2019

  4. [12]

    Depgraph: Towards any structural pruning

    Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any structural pruning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16091– 16101, 2023

    2023

  5. [13]

    Geometric measure theory

    Herbert Federer. Geometric measure theory. Springer, 2014

    2014

  6. [14]

    Mixture-of-loras: An efficient multitask tuning for large language models.arXiv preprint arXiv:2403.03432, 2024

    Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, Yu Han, and Hao Wang. Mixture-of-loras: An efficient multitask tuning for large language models.arXiv preprint arXiv:2403.03432, 2024

    2024 arXiv

  7. [15]

    Optimal brain compression: A framework for accurate post-training quantization and pruning

    Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems, 35:4475–4488, 2022

    2022

  8. [16]

    Sparsegpt: Massive language models can be accurately pruned in one-shot

    Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International conference on machine learning, pages 10323–10337. PMLR, 2023

    2023

  9. [17]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  10. [18]

    Reweighted proximal pruning for large-scale language representation

    Fu-Ming Guo, Sijia Liu, Finlay S Mungall, Xue Lin, and Yanzhi Wang. Reweighted proximal pruning for large-scale language representation. arXiv preprint arXiv:1909.12486, 2019

    1909 arXiv

  11. [19]

    Learning both weights and connections for efficient neural network

    Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015

    2015

  12. [20]

    Flora: Low-rank adapters are secretly gradient compressors

    Yongchang Hao, Yanshuai Cao, and Lili Mou. Flora: Low-rank adapters are secretly gradient compressors. In International Conference on Machine Learning, pages 17554–17571. PMLR, 2024. 11

    2024

  13. [21]

    Second order derivatives for network pruning: Optimal brain surgeon.Advances in neural information processing systems, 5, 1992

    Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon.Advances in neural information processing systems, 5, 1992

    1992

  14. [22]

    Optimal brain surgeon and general network pruning

    Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference on neural networks, pages 293–299. IEEE, 1993

    1993

  15. [23]

    Lora+ efficient low rank adaptation of large models

    Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+ efficient low rank adaptation of large models. InProceedings of the 41st International Conference on Machine Learning, pages 17783–17806, 2024

    2024

  16. [24]

    Subspace optimization for large language models with convergence guarantees

    Yutong He, Pengrui Li, Yipeng Hu, Chuyan Chen, and Kun Yuan. Subspace optimization for large language models with convergence guarantees. InForty-secondInternational Conference on Machine Learning, 2024

    2024

  17. [25]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. InThirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track(R...

    2021

  18. [26]

    Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    2022

  19. [27]

    Network trimming: A data-driven neuron pruning approach towards efficient deep architectures.arXiv preprint arXiv:1607.03250, 2016

    Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures.arXiv preprint arXiv:1607.03250, 2016

    2016 arXiv

  20. [28]

    Accelerated sparse neural training: A provable and efficient method to find n: m transposable masks.Advancesin neural information processing systems, 34:21099–21111, 2021

    Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Joseph Naor, and Daniel Soudry. Accelerated sparse neural training: A provable and efficient method to find n: m transposable masks.Advancesin neural information processing systems, 34:21099–21111, 2021

    2021

  21. [29]

    Arankstabilizationscalingfactorforfine-tuningwithlora

    DamjanKalajdzievski. Arankstabilizationscalingfactorforfine-tuningwithlora. arXivpreprintarXiv:2312.03732, 2023

    2023 arXiv

  22. [30]

    Some fundamental aspects about lipschitz continuity of neural networks

    Grigory Khromov and Sidak Pal Singh. Some fundamental aspects about lipschitz continuity of neural networks. In The TwelfthInternational Conference on Learning Representations, 2023

    2023

  23. [31]

    The optimal bert surgeon: Scalable and accurate second-order pruning for large language models

    Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan-Adrian Alistarh. The optimal bert surgeon: Scalable and accurate second-order pruning for large language models. In Proceedings of the 2022 Conference on Empirical Meth...

    2022

  24. [32]

    Ziplm: Inference-aware structured pruning of language models

    Eldar Kurtić, Elias Frantar, and Dan Alistarh. Ziplm: Inference-aware structured pruning of language models. Advances in Neural Information Processing Systems, 36:65597–65617, 2023

    2023

  25. [33]

    Sparse fine-tuning for inference acceleration of large language models

    Eldar Kurtic, Denis Kuznedelev, Elias Frantar, Michael Goinv, Shubhra Pandit, Abhinav Agarwalla, Tuan Nguyen, Alexandre Marques, Mark Kurtz, and Dan Alistarh. Sparse fine-tuning for inference acceleration of large language models. Enhancing LLM Performance: Efficacy, Fine-Tuni...

    2025

  26. [34]

    Albert: A lite bert for self-supervised learning of language representations

    Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. InInternational Conference on Learning Representations, 2019

    2019

  27. [35]

    Lipschitz constant estimation of neural networks via sparse polynomial optimization

    Fabian Latorre, Paul Rolland, and Volkan Cevher. Lipschitz constant estimation of neural networks via sparse polynomial optimization. InInternational Conference on Learning Representations, 2020

    2020

  28. [36]

    Optimal brain damage.Advances in neural information processing systems, 2, 1989

    Yann LeCun, John Denker, and Sara Solla. Optimal brain damage.Advances in neural information processing systems, 2, 1989

    1989

  29. [37]

    T\’yr-the-pruner: Unlocking accurate 50% structural pruning for llms via global sparsity distribution optimization.arXiv preprint arXiv:2503.09657, 2025

    Guanchen Li, Yixing Xu, Zeping Li, Ji Liu, Xuanwu Yin, Dong Li, and Emad Barsoum. T\’yr-the-pruner: Unlocking accurate 50% structural pruning for llms via global sparsity distribution optimization.arXiv preprint arXiv:2503.09657, 2025

    2025

  30. [38]

    Pruning filters for efficient convnets

    Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations, 2017

    2017

  31. [39]

    Sepprune: Structured pruning for efficient deep speech separation.arXiv preprint arXiv:2505.12079, 2025

    Yuqi Li, Kai Li, Xin Yin, Zhifei Yang, Junhao Dong, Zeyu Dong, Chuanguang Yang, Yingli Tian, and Yao Lu. Sepprune: Structured pruning for efficient deep speech separation.arXiv preprint arXiv:2505.12079, 2025

    2025 arXiv

  32. [40]

    Memory-efficient llm training with online subspace descent

    Kaizhao Liang, Bo Liu,Lizhang Chen, and Qiang Liu. Memory-efficient llm training with online subspace descent. Advances in Neural Information Processing Systems, 2024. 12

    2024

  33. [41]

    Dynamic adaptation of lora fine-tuning for efficient and task-specific optimization of large language models

    Xiaoxuan Liao, Chihang Wang, Shicheng Zhou, Jiacheng Hu, Hongye Zheng, and Jia Gao. Dynamic adaptation of lora fine-tuning for efficient and task-specific optimization of large language models. InProceedings of the 2025 International Conference on Artificial Intelligence and C...

    2025

  34. [42]

    Galore+: Boosting low-rank adaptation for llms with cross-head projection.arXiv preprint arXiv:2412.19820, 2024

    Xutao Liao, Shaohui Li, Yuhui Xu, Zhi Li, Yu Liu, and You He. Galore+: Boosting low-rank adaptation for llms with cross-head projection.arXiv preprint arXiv:2412.19820, 2024

    2024 arXiv

  35. [43]

    Dora: Weight-decomposed low-rank adaptation

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. InForty-firstInternational Conference on Machine Learning, 2024

    2024

  36. [44]

    Ebert: Efficient bert inference with dynamic structured pruning

    Zejian Liu, Fanrong Li, Gang Li, and Jian Cheng. Ebert: Efficient bert inference with dynamic structured pruning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4814–4823, 2021

    2021

  37. [45]

    Learning efficient convolutional networks through network slimming

    Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference on computer vision, pages 2736–2744, 2017

    2017

  38. [46]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InInternationalConference on Learning Representations, 2017

    2017

  39. [47]

    Lcm-lora: A universal stable-diffusion acceleration module.arXiv preprint arXiv:2311.05556, 2023

    Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, and Hang Zhao. Lcm-lora: A universal stable-diffusion acceleration module.arXiv preprint arXiv:2311.05556, 2023

    2023 arXiv

  40. [48]

    Llm-pruner: On the structural pruning of large language models

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702–21720, 2023

    2023

  41. [49]

    Pissa: Principal singular values and singular vectors adaptation of large language models.Advances in Neural Information Processing Systems, 37:121038–121072, 2024

    Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models.Advances in Neural Information Processing Systems, 37:121038–121072, 2024

    2024

  42. [50]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017

    2017

  43. [51]

    Accelerating sparse deep neural networks.arXiv preprint arXiv:2104.08378, 2021

    Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks.arXiv preprint arXiv:2104.08378, 2021

    2021 arXiv

  44. [52]

    Pruning convolutional neural networks for resource efficient inference

    P Molchanov, S Tyree, T Karras, T Aila, and J Kautz. Pruning convolutional neural networks for resource efficient inference. In 5th International Conference on Learning Representations, ICLR 2017-Conference Track Proceedings, 2019

    2017

  45. [53]

    Importance estimation for neural network pruning

    Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. Importance estimation for neural network pruning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11264–11272, 2019

    2019

  46. [54]

    Sosp: Efficiently capturing global correlations by second-order structured pruning

    Manuel Nonnenmacher, Thomas Pfeil, Ingo Steinwart, and David Reeb. Sosp: Efficiently capturing global correlations by second-order structured pruning. InInternational Conference on Learning Representations, 2021

    2021

  47. [55]

    Gradient-free structured pruning with unlabeled data

    Azade Nova, Hanjun Dai, and Dale Schuurmans. Gradient-free structured pruning with unlabeled data. In International Conference on Machine Learning, pages 26326–26341. PMLR, 2023

    2023

  48. [56]

    Meta-kd: A meta knowledge distillation framework for language model compression across domains

    Haojie Pan, Chengyu Wang, Minghui Qiu, Yichang Zhang, Yaliang Li, and Jun Huang. Meta-kd: A meta knowledge distillation framework for language model compression across domains. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11...

    2021

  49. [57]

    Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    2020

  50. [58]

    Structural pruning via latency-saliency knapsack

    Maying Shen, Hongxu Yin, Pavlo Molchanov, Lei Mao, Jianna Liu, and Jose M Alvarez. Structural pruning via latency-saliency knapsack. Advances in Neural Information Processing Systems, 35:12894–12908, 2022

    2022

  51. [59]

    Parameter efficient reinforcement learning from human feedback

    Hakim Sidahmed, Samrat Phatale, Alex Hutcheson, Zhuonan Lin, Zhang Chen, Zac Yu, Jarvis Jin, Simral Chaudhary, Roman Komarytsia, Christiane Ahlheim, et al. Parameter efficient reinforcement learning from human feedback. arXiv preprint arXiv:2403.10704, 2024. 13

    2024 arXiv

  52. [60]

    Woodfisher: Efficient second-order approximation for neural network compression

    Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural network compression. Advances in Neural Information Processing Systems, 33:18098–18109, 2020

    2020

  53. [61]

    A simple and effective pruning approach for large language models

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. InThe TwelfthInternational Conference on Learning Representations, 2023

    2023

  54. [62]

    Patient knowledge distillation for bert model compression

    Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-...

    2019

  55. [63]

    Contrastive distillation on intermediate representations for language model compression

    Siqi Sun, Zhe Gan, Yuwei Fang, Yu Cheng, Shuohang Wang, and Jingjing Liu. Contrastive distillation on intermediate representations for language model compression. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 498–508, 2020

    2020

  56. [64]

    Improving lora in privacy-preserving federated learning

    Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving lora in privacy-preserving federated learning. arXiv preprint arXiv:2403.12313, 2024

    2024 arXiv

  57. [65]

    Mobilebert: a compact task-agnostic bert for resource-limited devices

    Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: a compact task-agnostic bert for resource-limited devices. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2158–2170, 2020

    2020

  58. [66]

    Darwinlm: Evolutionary structured pruning of large language models.arXiv preprint arXiv:2502.07780, 2025

    Shengkun Tang, Oliver Sieberling, Eldar Kurtic, Zhiqiang Shen, and Dan Alistarh. Darwinlm: Evolutionary structured pruning of large language models.arXiv preprint arXiv:2502.07780, 2025

    2025 arXiv

  59. [67]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  60. [68]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  61. [69]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  62. [70]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. InInternational Conference on Learning Representations, 2018

    2018

  63. [71]

    Lora-ga: Low-rank adaptation with gradient approximation.Advances in Neural Information Processing Systems, 37:54905–54931, 2024

    Shaowen Wang, Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approximation.Advances in Neural Information Processing Systems, 37:54905–54931, 2024

    2024

  64. [72]

    Lora-pro: Are low-rank adapters properly optimized? In The Thirteenth International Conference on Learning Representations, 2024

    Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. Lora-pro: Are low-rank adapters properly optimized? In The Thirteenth International Conference on Learning Representations, 2024

    2024

  65. [73]

    Assessing the brittleness of safety alignment via pruning and low-rank modifications

    Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. Assessing the brittleness of safety alignment via pruning and low-rank modifications. Proceedings of Machine Learning Research, 235:52588–52610, 2024

    2024

  66. [74]

    Structured optimal brain pruning for large language models

    Jiateng Wei, Quan Lu, Ning Jiang, Siqi Li, Jingyang Xiang, Jun Chen, and Yong Liu. Structured optimal brain pruning for large language models. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13991–14007, 2024

    2024

  67. [75]

    The iterative optimal brain surgeon: Faster sparse recovery by leveraging second-order information.Advances in Neural Information Processing Systems, 37:139621–139649, 2024

    Diyuan Wu, Ionut-Vlad Modoranu, Mher Safaryan, Denis Kuznedelev, and Dan Alistarh. The iterative optimal brain surgeon: Faster sparse recovery by leveraging second-order information.Advances in Neural Information Processing Systems, 37:139621–139649, 2024

    2024

  68. [76]

    Structured pruning learns compact and accurate models

    Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruning learns compact and accurate models. In 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022, pages 1513–1528. Association for Computational Linguistics (ACL), 2022

    2022

  69. [77]

    Pruning mixture-of-experts large language model using the hints from its router.arXiv preprint arXiv:2410.12013, 3, 2024

    Yanyue Xie, Zhi Zhang, Ding Zhou, Cong Xie, Ziang Song, Xin Liu, Yanzhi Wang, Xue Lin, and An Xu Moe-pruner. Pruning mixture-of-experts large language model using the hints from its router.arXiv preprint arXiv:2410.12013, 3, 2024. 14

    2024 arXiv

  70. [78]

    Deebert: Dynamic early exiting for accelerating bert inference

    Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. Deebert: Dynamic early exiting for accelerating bert inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2246–2251, 2020

    2020

  71. [79]

    Rethinking network pruning–under the pre-train and fine-tune paradigm

    Dongkuan Xu, Ian En-Hsu Yen, Jinxi Zhao, and Zhibin Xiao. Rethinking network pruning–under the pre-train and fine-tune paradigm. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p...

    2021

  72. [80]

    Theoretical characterization of how neural network pruning affects its generalization.OpenReview, 2023

    Hongru Yang, Yingbin Liang, Xiaojie Guo, Lingfei Wu, and Zhangyang Wang. Theoretical characterization of how neural network pruning affects its generalization.OpenReview, 2023

    2023

  73. [81]

    Mtl-lora: Low-rank adaptation for multi-task learning

    Yaming Yang, Dilixat Muhtar, Yelong Shen, Yuefeng Zhan, Jianfeng Liu, Yujing Wang, Hao Sun, Denvy Deng, Feng Sun, Qi Zhang, Weizhu Chen, and Yunhai Tong. Mtl-lora: Low-rank adaptation for multi-task learning. In AAAI Conference on Artificial Intelligence, 2024

    2024

  74. [82]

    Wanda++: Pruning large language models via regional gradients

    Yifan Yang, Kai Zhen, Bhavana Ganesh, Aram Galstyan, Goeric Huybrechts, Markus Müller, Jonas M Kübler, Rupak Vignesh Swaminathan, Athanasios Mouchtaris, Sravan Babu Bodapati, et al. Wanda++: Pruning large language models via regional gradients. InSparsity in LLMs (SLLM): Deep ...

    2025

  75. [83]

    Gradient-based intra-attention pruning on pre-trained language models

    Ziqing Yang, Yiming Cui, Xin Yao, and Shijin Wang. Gradient-based intra-attention pruning on pre-trained language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2775–2790, 2023

    2023

  76. [84]

    Zeroquant: Efficient and affordable post-training quantization for large-scale transformers.Advances in neural information processing systems, 35:27168–27183, 2022

    Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers.Advances in neural information processing systems, 35:27168–27183, 2022

    2022

  77. [85]

    Lora done rite: Robust invariant transformation equilibration for lora optimization

    Jui-Nan Yen, Si Si, Zhao Meng, Felix Yu, Sai Surya Duvvuri, Inderjit S Dhillon, Cho-Jui Hsieh, and Sanjiv Kumar. Lora done rite: Robust invariant transformation equilibration for lora optimization. InThe Thirteenth International Conference on Learning Representations, 2024

    2024

  78. [86]

    Metamath: Bootstrap your own mathematical questions for large language models

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In The TwelfthInternational Conference on Learning Representations, 2023

    2023

  79. [87]

    Understanding the statistical accuracy-communication trade-off in personalized federated learning with minimax guarantees.arXiv preprint arXiv:2410.08934, 2024

    Xin Yu, Zelin He, Ying Sun, Lingzhou Xue, and Runze Li. Understanding the statistical accuracy-communication trade-off in personalized federated learning with minimax guarantees.arXiv preprint arXiv:2410.08934, 2024

    2024 arXiv

  80. [88]

    Altlora: Towards better gradient approximation in low-rank adaptation with alternating projections.arXiv preprint arXiv:2505.12455, 2025

    Xin Yu, Yujia Wang, Jinghui Chen, and Lingzhou Xue. Altlora: Towards better gradient approximation in low-rank adaptation with alternating projections.arXiv preprint arXiv:2505.12455, 2025

    2025

  81. [89]

    Q8bert: Quantized 8bit bert

    Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized 8bit bert. In2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS), pages 36–39. IEEE, 2019

    2019

  82. [90]

    Block-diagonal hessian-free optimization for training neural networks.arXiv preprint arXiv:1712.07296, 2017

    Huishuai Zhang, Caiming Xiong, James Bradbury, and Richard Socher. Block-diagonal hessian-free optimization for training neural networks.arXiv preprint arXiv:1712.07296, 2017

    2017 arXiv

  83. [91]

    Loraprune: Struc- tured pruning meets low-rank parameter-efficient fine-tuning

    MingyangZhang, HaoChen, ChunhuaShen, ZhenYang, LinlinOu, XinyiYu, andBohanZhuang. Loraprune: Struc- tured pruning meets low-rank parameter-efficient fine-tuning. InFindings of the Association for Computational Linguistics ACL 2024, pages 3013–3026, 2024

    2024

  84. [92]

    Adap- tive budget allocation for parameter-efficient fine-tuning

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adap- tive budget allocation for parameter-efficient fine-tuning. InInternationalConference on Learning Representations. Openreview, 2023

    2023

  85. [93]

    Lora-one: One-step full gradient could suffice for fine-tuning large language models, provably and efficiently

    Yuanhe Zhang, Fanghui Liu, and Yudong Chen. Lora-one: One-step full gradient could suffice for fine-tuning large language models, provably and efficiently. InForty-secondInternational Conference on Machine Learning, 2025

    2025

  86. [94]

    Galore: Memory-efficient llm training by gradient low-rank projection

    Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection. InInternational Conference on Machine Learning, pages 61121–61143. PMLR, 2024

    2024

  87. [95]

    Adaptive activation-based structured pruning

    Kaiqi Zhao, Animesh Jain, and Ming Zhao. Adaptive activation-based structured pruning. arXiv preprint arXiv:2201.10520, 2022. 15

    2022 arXiv

  88. [96]

    Each rank could be an expert: Single-ranked mixture of experts lora for multi-task learning.arXiv preprint arXiv:2501.15103, 2025

    Ziyu Zhao, Yixiao Zhou, Zhi Zhang, Didi Zhu, Tao Shen, Zexi Li, Jinluan Yang, Xuwu Wang, Jing Su, Kun Kuang, et al. Each rank could be an expert: Single-ranked mixture of experts lora for multi-task learning.arXiv preprint arXiv:2501.15103, 2025

    2025 arXiv

  89. [97]

    To prune, or not to prune: exploring the efficacy of pruning for model compression

    Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878, 2017. 16 Appendix A The Use of LLMs LLMs were used to improve writing clarity and assist with code development. Specifically, LLMs ...

    2017 arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.