Pith. sign in

REVIEW 4 major objections 8 minor 43 references

Sparse Gradient Compression for Fine-Tuning Large Language Models

T0 review · 4 major / 8 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning memory can be cut by storing optimizer states in a k-dimensional subspace chosen freely, independent of model size.

desk verdict A promising but incomplete memory-efficient fine-tuning method whose central claim overstates savings by omitting the projection matrix and whose OMP step rests on an unproven sparsity assumption. read the letter →

arxiv 2502.00311 v1 pith:QRSRIGTH submitted 2025-02-01 cs.LG

David H. Yang , Mohammad Mohammadi Amiri , Tejaswini Pedapati , Subhajit Chaudhury , Pin-Yu Chen This is my paper · ORCID
classification cs.LG
keywords sparsegradientcompressionparameter-efficientfine-tuningcompressedsensingorthogonalmatchingpursuitoptimizerstateslargelanguagemodelsmemoryefficiencylow-rankadaptation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes sparse gradient compression (SGC), a training regime that keeps the Adam optimizer's two moment vectors—normally as large as the gradient itself—inside a k-dimensional subspace whose size the user chooses independently of the model's width. The paper argues that because fine-tuning gradients are approximately sparse, a fixed random projection down to k dimensions followed by orthogonal matching pursuit recovery preserves enough moment information to update weights as effectively as LoRA or GaLore. If the claim holds, the memory cost of optimizer states becomes 2k instead of 2d, and the memory-accuracy tradeoff can be swept in small, flexible steps rather than being locked to rank-1 LoRA or GaLore floors that depend on layer dimensions. Experiments on LLaMA2-7B, LLaMA3-8B, LLaMA2-13B, and Mistral-7B report accuracy comparable to or better than the baselines while using far fewer optimizer states, with the largest relative gains in data-limited and memory-limited settings.

What carries the argument

The load-bearing mechanism is the sparsify-project-recover round trip built around the AdamW update in equation (13): top-s sparsification of the gradient, projection onto a k-dimensional subspace via a fixed random matrix A that satisfies the RIP when k≥κs, accumulation of first and second moments in that subspace, and orthogonal matching pursuit (OMP) to map the moments back to the d-dimensional space for the weight update. Two variants make the approach practical: MESGC splits the gradient into c chunks and applies the same small projection matrix to each chunk, cutting projection-matrix storage by a factor of c, and CESGC prepends a GaLore-style SVD projection so that the vector entering SGC is already low-rank, reducing the OMP cost paid during fine-tuning.

What would settle it

Measure the union of the support sets of the top-s sparsified gradients across every step of a real fine-tuning run. If this union grows far beyond the OMP sparsity budget s while accuracy still matches full fine-tuning, the recovery in equation (13) is being helped by something other than bounded support; a direct check would compare the OMP-reconstructed moments against the true dense moments and see whether the relative error stays small on the exact tasks where SGC claims parity.

Watch

Extended reading notes

Core claim

The central claim is that Adam's first and second moments do not need to live in the parameter space at all. SGC sparsifies each gradient to its top s entries, projects the resulting sparse vector and its element-wise square through a fixed Gaussian matrix A∈$R^{{k×d}}$ satisfying the restricted isometry property, accumulates the two moments in R^k, and then reconstructs d-dimensional moment estimates with orthogonal matching pursuit, forming the update N_t = α OMP_A(M_t)/(√(OMP_A(V_t))+ε). Because G_t and $G_t^{2}$ share the same sparsity pattern, the reconstructed numerator and denominator have identical support, and the paper asserts that the total change in gradient sparsity across all steps can be bounded by a constant ilde{s}≪d, making the recovery almost exact. The payoff is that the optimizer stores only 2k = 2κs values, with k entirely decoupled from d; the paper's configurations reach 896–4096 optimizer states, about a 10× reduction relative to rank-1 LoRA and GaLore, while matching or exceeding their accuracy on the tested commonsense and knowledge benchmarks.

Load-bearing premise

That the coordinates where the largest gradient entries sit stay almost the same across all training steps, so that the sum of all moment vectors remains sparse enough for the recovery step to reconstruct it from just k measurements.

Editorial extensions

If this is right

  • Optimizer-state memory drops from 2d to 2k, with k = κs chosen by the user; the paper's configurations use 896–4096 states, about a 10× reduction relative to rank-1 GaLore and LoRA floors on 7B-scale models.
  • The memory–accuracy tradeoff becomes continuously adjustable in small increments (512-state steps in the paper's CESGC setup), because k is no longer tied to layer dimensions.
  • In data-limited regimes, SGC's greedy top-gradient selection yields higher accuracy than LoRA or GaLore at the same optimizer-state budget on the BoolQ subsets tested.
  • SGC is orthogonal to other gradient-compression methods: wrapping it around a GaLore-style projection (CESGC) preserves accuracy while shrinking the dimension entering the optimizer's subspace.
  • The method can be applied to many optimizers beyond AdamW, since the formulation only assumes that the optimizer's states are functions of the gradient and its element-wise square.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the support-union assumption holds broadly, SGC could be combined with quantized base weights to bring memory-constrained fine-tuning of 10B+ models within reach of single consumer GPUs, a combination the paper does not test.
  • A direct diagnostic—tracking how the union of top-s gradient supports grows across training steps—would let practitioners decide when the OMP sparsity budget needs to grow; the paper offers no such measurement, and the assumption is the method's most fragile link.
  • The same sparsify-project-recover cycle could transfer to other Adam-family optimizers and to vision or audio fine-tuning where gradient sparsity has also been observed, though the paper only demonstrates text tasks.
  • The convergence result proved is only a worst-case bound comparing chunk-based to global sparsification; closing the gap to a full convergence theorem for the OMP-based update is explicitly left as future work by the authors.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper proposes sparse gradient compression (SGC), a fine-tuning method that keeps AdamW's first and second moments in a k-dimensional subspace instead of the original d-dimensional parameter space. At each step, the gradient is sparsified to its top-s entries and projected with a fixed Gaussian matrix A ∈ R^{k×d}; the k-dimensional Adam moments are updated from these projections, and the d-dimensional update is reconstructed with orthogonal matching pursuit (OMP) run with sparsity s, where k = κs. Two variants are introduced: MESGC (chunking the gradient to shrink A) and CESGC (pre-projecting with GaLore's SVD basis before applying SGC). Experiments on LLaMA2-7B, LLaMA3-8B, LLaMA2-13B, and Mistral-7B report accuracy comparable to or slightly better than LoRA and GaLore at lower optimizer-state counts, together with ablations over chunk count, sparsity, and κ. The headline claim is that the number of optimizer states is independent of the model parameter count, enabling finer granularity than LoRA or GaLore.

Significance. If the central claims held as stated, SGC would be a useful addition to the PEFT toolbox: it decouples the optimizer-state count from the parameter dimension, is orthogonal to GaLore-style projection (CESGC), and the experimental comparison is against standard external baselines with detailed hyperparameters in Appendix D. The paper is honest about several of its limits: Section 4.6 explicitly defers convergence analysis, Section 5.1 flags the task-dependence of the small-data gain, and Appendix B discloses the Gram-matrix memory tradeoff of its OMP implementation. There is no circular reasoning: the gradient-sparsity premise is cited from prior work (Song et al., 2024), and the reported numbers are external comparisons without fitted quantities disguised as predictions. The contribution's significance hinges on two load-bearing points that the manuscript does not yet support: the recoverability of the accumulated Adam moments under support drift, and the memory accounting of the projection matrix. Both are empirical and, in principle, addressable within the scope of a revision.

major comments (4)
  1. [Section 4.3, Eqs. (12)–(13)] The recovery target is not s-sparse. The d-dimensional object recovered by OMP is the accumulated moment Σ_{i=1}^{t} h_i(β_1) G̃_i, whose support is the union of the per-step top-s supports. The sole justification is the sentence 'Assuming that the total changes in the sparsity of G_t over all t can be bounded by some constant s̃ ≪ d'; this assumption is never proved, never connected to the s used in OMP, and never measured. With roughly 1.1×10^4 gradient steps (170k examples, batch 16, one epoch), the union of drifting top-s supports can grow to min(d, s·T), orders of magnitude above the s = 1984–2496 used in Table 2; even taking the exponential-decay window of Adam into account (about 1/(1−β_2) ≈ 10^3 steps for V_t), the effective support can exceed s by orders of magnitude. In that regime OMP with sparsity s returns an s-sparse vector that is not the accumulated moment, and Eq. (13) does not implement AdamW in the original space, leaving the update in Algorithm 2 uncontrolled. I would like to see a measurement of the support-union size or, better, of the reconstruction error ‖OMP_A(M_t) − M_t^*‖/‖M_t^*‖ over training at the Table 2 hyperparameters, or a reformulation of the recovery step whose target is provably s-sparse.
  2. [Section 4.3, Eq. (13)] The shared-support claim for the recovered moments is not justified. The text states that because G̃_t and G̃_t^2 share a sparsity pattern, 'the indices of the non-zero entries in OMP_A(M_t) and OMP_A(V_t) are identical'. This holds only when OMP exactly recovers a common s-sparse signal. Under imperfect recovery, the greedy selections for M_t and V_t are driven by different measurement vectors and different coefficient magnitudes, so the selected supports will generally disagree; the coordinate-wise division in Eq. (13) then ceases to be the AdamW normalization, and coordinates where the supports differ produce either an effectively infinite ratio (M-support but not V-support) or a zero update (V-support but not M-support). The paper should report the fraction of coordinates on which the two recovered supports agree during training and define the update rule for coordinates where they differ.
  3. [Table 1, Sections 4.5 and 5.2] The SGC projection matrix A is omitted from the memory accounting. For MESGC, A has k_c × (d/c) entries (shared across chunks). At the Table 5 configuration (c = 256, s_c = 1, κ = 8) and with d ≈ 1.07×10^9 trainable q/v entries over 32 layers of LLaMA2-7B, this is 8 × (d/256) ≈ 3.4×10^7 entries, about 134 MB in fp32 — roughly four orders of magnitude more than the 4096 Adam moments counted as '# Params'; at the Section 5.2 configuration (c = 64, κ = 7) it is ≈ 1.2×10^8 entries. Even in the per-matrix convention of Table 1, storing A exceeds the counted optimizer states by orders of magnitude. If A is instead regenerated on the fly from a seed, the paper must state that explicitly and account for the added per-step cost (O(k_c · d/c) per chunk per OMP projection), which is comparable to a full fine-tuning step's optimizer cost and is absent from the throughput comparison in Table 4. As written, the abstract's memory-efficiency claim is not supported by the table's accounting, and the caption's parenthetical that projection matrices refer only to B_t does not address A.
  4. [Sections 4.3 and 4.6] The recovery and convergence guarantees presented do not cover the proposed algorithm. First, the RIP condition stated in Section 4.3 requires k ≥ C·s·log(d/s) measurements, but the paper fixes k = κs with κ = 7; at the per-chunk dimensions of the experiments (chunk lengths of order 10^4 to 10^7, s_c = 1 to 31), log(d/s) is about 5 to 10, so the cited compressed-sensing theory does not imply successful recovery at κ = 7. The claimed independence of k from d is therefore not supported by the cited theory, and Figure 3(c) shows the choice is fragile: κ = 6 already degrades performance sharply. Second, Section 4.6 explicitly leaves the convergence proof as future work, and Theorem 1 bounds only the single-step error of chunk-based versus global top-k sparsification; it does not address OMP reconstruction error or the accumulation of these errors over time. The 'almost accurately recover' assertion in Section 4.3 thus carries the full burden without theoretical support.
minor comments (8)
  1. [Section 4.2, Eq. (9)] The V_t update uses β_1 where AdamW requires β_2; as written the second moment adopts the wrong decay rate.
  2. [Section 4.4] The text says chunking reduces the projection matrix 'from k×d to (k×d)/c', but with a single matrix A shared across c chunks the per-chunk size is (k/c)×(d/c) = kd/c²; the stated and implied reductions differ by a factor of c.
  3. [Section 4.6, Theorem 1] The claim that in the uniform case E[‖G̃′ − G̃‖²] = 0 holds only when every chunk contains exactly s_c of the global top-s entries; for randomly distributed supports the chunk-selected sets differ from the global set and the expected error is positive.
  4. [Appendix B, Algorithm 3] Algorithm 3 precomputes the full Gram matrix A^H A, which at the chunk sizes used in the experiments (d/c up to roughly 10^7) would require on the order of 10^14 entries; please state what is actually precomputed or computed on demand in the implementation behind Tables 2–5.
  5. [Eq. (13)] The scaling factor α is fixed to 2 in all experiments without derivation or ablation; please state whether it is a tuned hyperparameter or a principled correction for the energy loss of the sparse recovery.
  6. [Section 5.1, Figure 2(a)] The claim of superior performance in data-limited settings rests on a single task (BoolQ); the self-flagged caveat that the effect 'may be task dependent' should be reflected in the abstract or conclusion.
  7. [Throughout] There are several copyediting issues: 'identify function' in Section 4.1, 'dimensonal' in the abstract, 'fune-tuning' in the Wu et al. reference, and the term 'quasi-sparse' is used without a definition.
  8. [Reproducibility] No code release is mentioned; given the implementation sensitivity of the GPU OMP routine and the Gram-matrix handling, releasing code would materially improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SGC's memory claim is a definitional design property, its performance claims are empirical against external baselines, and the unproven OMP sparsity assumption is a technical gap rather than a circular reduction.

full rationale

Walking the claimed derivation chain, SGC sparsifies each gradient using top-s selection, projects it with a fixed Gaussian matrix A, updates k-dimensional Adam moments, and recovers an s-sparse update via OMP. The optimizer-state count of 2k is a definition of the algorithm rather than a derived prediction: k = kappa*s is chosen by construction, so the statement that k is independent of the original parameter dimension is a design property, not a conclusion obtained from an equation that already contains that property. No fitted parameter is renamed as a prediction: the hyperparameters s, c, kappa, and alpha are fixed before evaluation, and the experimental comparisons are made against external methods (LoRA, GaLore, full fine-tuning) on standard benchmarks. The quasi-sparsity premise is attributed to Song et al. (2024), an external empirical result, and is not a self-citation chain. The unproved assertion in Section 4.3 that the total change in sparsity of G_t over all t is bounded by tilde{s} << d is a genuine and potentially fragile technical premise, and Section 4.6 explicitly defers the convergence proof ('we leave this as part of future work'); however, these are unsupported assumptions or omitted proofs, not cases where an output equation is equivalent to an input by construction. Theorem 1 bounds chunk-based versus global top-k sparsification error and does not secretly define the OMP recovery target, so it does not mask a circular equivalence. I therefore find no load-bearing circular step under the enumerated patterns.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical entities; it relies on a projection/compression scheme. The free parameters are tunable knobs that directly set the memory/accuracy tradeoff. The axioms include a standard CS assumption, a domain assumption about gradient sparsity, and two ad hoc assumptions about support behavior that are load-bearing for OMP recovery.

free parameters (5)
  • sparsity level s = varies (e.g., 1984, 256, 64)
    Controls the number of top gradient entries retained and hence k; chosen per task/experiment; no theory determines it.
  • chunk count c = varies (e.g., 64, 256, 4096)
    Chosen per experiment; affects memory/accuracy tradeoff.
  • RIP constant kappa = 7 (8 in some runs)
    Selected from ablation in Figure 3(c); not derived.
  • scaling factor alpha = 2
    Applied to OMP-recovered update; no derivation given.
  • GaLore rank r (CESGC) = 32
    First-stage projection rank in CESGC; hyperparameter.
assumptions (4)
  • domain assumption Gradients during fine-tuning are quasi-sparse, so top-s sparsification retains enough signal.
    Invoked in Section 4.2; cited from Song et al. (SIFT), not measured here.
  • ad hoc to paper The union of supports of sparsified gradients across all t is bounded by \tilde{s} << d.
    Stated after Eq. 12 to justify OMP recovery of accumulated moments; no proof or empirical support and likely violated as supports shift.
  • ad hoc to paper M_t and V_t share identical sparsity support after OMP recovery.
    Assumed in Eq. 13; not guaranteed by independent OMP runs.
  • standard math Random Gaussian A satisfies RIP when k >= kappa * s.
    Standard compressed sensing result; requires true sparsity of the recovered vector, which is the contested point.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sparse Gradient Compression for Fine-Tuning Large Language Models." pith.science (2026). https://pith.science/paper/QRSRIGTH

@misc{pith2026250200311,
  author       = {Pith},
  title        = {Pith review of: Sparse Gradient Compression for Fine-Tuning Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QRSRIGTH}},
  note         = {Machine review of arXiv:2502.00311}
}
read the original abstract

Fine-tuning large language models (LLMs) for downstream tasks has become increasingly crucial due to their widespread use and the growing availability of open-source models. However, the high memory costs associated with fine-tuning remain a significant challenge, especially as models increase in size. To address this, parameter efficient fine-tuning (PEFT) methods have been proposed to minimize the number of parameters required for fine-tuning LLMs. However, these approaches often tie the number of optimizer states to dimensions of model parameters, limiting flexibility and control during fine-tuning. In this paper, we propose sparse gradient compression (SGC), a training regime designed to address these limitations. Our approach leverages inherent sparsity in gradients to compress optimizer states by projecting them onto a low-dimensonal subspace, with dimensionality independent of the original model's parameters. By enabling optimizer state updates in an arbitrary low-dimensional subspace, SGC offers a flexible tradeoff between memory efficiency and performance. We demonstrate through experiments that SGC can decrease memory usage in optimizer states more effectively than existing PEFT methods. Furthermore, by fine-tuning LLMs on various downstream tasks, we show that SGC can deliver superior performance while substantially lowering optimizer state memory requirements, particularly in both data-limited and memory-limited settings.

Figures

Figures reproduced from arXiv: 2502.00311 by the authors.

Figure 1
Figure 1. Diagram comparing SGC (green) and PEFT methods LoRA and GaLore (blue) in terms of [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. (a). CESGC outperforms both GaLore and LoRA when fine-tuning with limited data on BoolQ. (b). Plot showing improvement of accuracy of CESGC when using a minimal number of optimizer states. Hollow blue points are interpolated values that indicate the granularity of CESGC across optimizer states. 5.4 Ablation Study Here, we investigate the effects of number of chunks c, total sparsity s, and the constant κ on fine-tun… view at source ↗
Figure 3
Figure 3. Ablation study for effects of number of chunks [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 6 canonical work pages

  1. [1]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, and et. al. Qwen technical report, 2023. URL https://arxiv.org/abs/2309.16609

    arXiv 2023

  2. [2]

    Decoding by Linear Programming

    Emmanuel Candes and Terence Tao. Decoding by linear programming, 2005. URL https://arxiv.org/abs/math/0502327

    work page Pith review arXiv 2005

  3. [3]

    Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information, 2004

    Emmanuel Candes, Justin Romberg, and Terence Tao. Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information, 2004. URL https://arxiv.org/abs/math/0409186

    arXiv 2004

  4. [4]

    The restricted isometry property and its implications for compressed sensing

    Emmanuel J Candes. The restricted isometry property and its implications for compressed sensing. Comptes rendus. Mathematique, 346 0 (9-10): 0 589--592, 2008

    work page 2008

  5. [5]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, and et. al. Palm: Scaling language modeling with pathways, 2022. URL https://arxiv.org/abs/2204.02311

    arXiv 2022

  6. [6]

    Boolq: Exploring the surprising difficulty of natural yes/no questions, 2019

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions, 2019. URL https://arxiv.org/abs/1905.10044

    arXiv 2019

  7. [7]

    Qlora: Efficient finetuning of quantized llms, 2023

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms, 2023. URL https://arxiv.org/abs/2305.14314

    arXiv 2023

  8. [8]

    Parameter-efficient fine-tuning of large-scale pre-trained language models

    Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5 0 (3): 0 220--235, 2023

    2023

Show all 43 references
  1. [9]

    Compressed sensing

    David L Donoho. Compressed sensing. IEEE Transactions on information theory, 52 0 (4): 0 1289--1306, 2006

    2006

  2. [10]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, and et. al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    2024 arXiv

  3. [11]

    Parameter-efficient fine-tuning for large models: A comprehensive survey, 2024

    Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey, 2024. URL https://arxiv.org/abs/2403.14608

    2024 arXiv

  4. [12]

    Flora: Low-rank adapters are secretly gradient compressors, 2024

    Yongchang Hao, Yanshuai Cao, and Lili Mou. Flora: Low-rank adapters are secretly gradient compressors, 2024. URL https://arxiv.org/abs/2402.03293

    2024 arXiv

  5. [13]

    Lora+: Efficient low rank adaptation of large models, 2024

    Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models, 2024. URL https://arxiv.org/abs/2402.12354

    2024 arXiv

  6. [14]

    Towards a unified view of parameter-efficient transfer learning, 2022

    Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning, 2022. URL https://arxiv.org/abs/2110.04366

    2022 arXiv

  7. [15]

    Parameter-efficient transfer learning for nlp, 2019

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp, 2019. URL https://arxiv.org/abs/1902.00751

    2019 arXiv

  8. [16]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https://arxiv.org/abs/2106.09685

    2021 arXiv

  9. [17]

    Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models, 2023

    Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models, 2023. URL https://arxiv.org/abs/2304.01933

    2023 arXiv

  10. [18]

    Lee, and Ernest K

    Uijeong Jang, Jason D. Lee, and Ernest K. Ryu. Lora training in the ntk regime has no spurious local minima, 2024. URL https://arxiv.org/abs/2402.11867

    2024 arXiv

  11. [19]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL https://arxiv.org/abs/1412.6980

    2017 arXiv

  12. [20]

    Kopiczko, Tijmen Blankevoort, and Yuki M

    Dawid J. Kopiczko, Tijmen Blankevoort, and Yuki M. Asano. Vera: Vector-based random matrix adaptation, 2024. URL https://arxiv.org/abs/2310.11454

    2024 arXiv

  13. [21]

    The power of scale for parameter-efficient prompt tuning, 2021

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning, 2021. URL https://arxiv.org/abs/2104.08691

    2021 arXiv

  14. [22]

    Prefix-tuning: Optimizing continuous prompts for generation, 2021

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation, 2021. URL https://arxiv.org/abs/2101.00190

    2021 arXiv

  15. [23]

    Memory-efficient llm training with online subspace descent, 2024

    Kaizhao Liang, Bo Liu, Lizhang Chen, and Qiang Liu. Memory-efficient llm training with online subspace descent, 2024. URL https://arxiv.org/abs/2408.12857

    2024 arXiv

  16. [24]

    Dora: Weight-decomposed low-rank adaptation, 2024

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation, 2024. URL https://arxiv.org/abs/2402.09353

    2024 arXiv

  17. [25]

    P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks, 2022

    Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks, 2022. URL https://arxiv.org/abs/2110.07602

    2022 arXiv

  18. [26]

    Decoupled weight decay regularization, 2019

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https://arxiv.org/abs/1711.05101

    2019 arXiv

  19. [27]

    Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks, 2021

    Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks, 2021. URL https://arxiv.org/abs/2106.04489

    2021 arXiv

  20. [28]

    A survey on lora of large language models, 2024

    Yuren Mao, Yuhang Ge, Yijiang Fan, Wenyi Xu, Yu Mi, Zhonghao Hu, and Yunjun Gao. A survey on lora of large language models, 2024. URL https://arxiv.org/abs/2407.11046

    2024 arXiv

  21. [29]

    A review of sparse recovery algorithms

    Elaine Crespo Marques, Nilson Maciel, Lirida Naviner, Hao Cai, and Jun Yang. A review of sparse recovery algorithms. IEEE access, 7: 0 1300--1322, 2018

    2018

  22. [30]

    Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition

    Yagyensh Chandra Pati, Ramin Rezaiifar, and Perinkulam Sambamurthy Krishnaprasad. Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition. In Proceedings of 27th Asilomar conference on signals, systems and computers, pp.\ 40--44...

    1993

  23. [31]

    Adapterfusion: Non-destructive task composition for transfer learning, 2021

    Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning, 2021. URL https://arxiv.org/abs/2005.00247

    2021 arXiv

  24. [32]

    Accurate lora-finetuning quantization of llms via information retention, 2024

    Haotong Qin, Xudong Ma, Xingyu Zheng, Xiaoyang Li, Yang Zhang, Shouda Liu, Jie Luo, Xianglong Liu, and Michele Magno. Accurate lora-finetuning quantization of llms via information retention, 2024. URL https://arxiv.org/abs/2402.05445

    2024 arXiv

  25. [33]

    Sparse is enough in fine-tuning pre-trained large language models, 2024

    Weixi Song, Zuchao Li, Lefei Zhang, Hai Zhao, and Bo Du. Sparse is enough in fine-tuning pre-trained large language models, 2024. URL https://arxiv.org/abs/2312.11875

    2024 arXiv

  26. [34]

    Stich, Jean-Baptiste Cordonnier, and Martin Jaggi

    Sebastian U. Stich, Jean-Baptiste Cordonnier, and Martin Jaggi. Sparsified sgd with memory, 2018. URL https://arxiv.org/abs/1809.07599

    2018 arXiv

  27. [35]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023

    2023

  28. [36]

    Llama 2: Open foundation and fine-tuned chat models, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and et al. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https://arxiv.org/abs/2307.09288

    2023 arXiv

  29. [37]

    Cg-fedllm: How to compress gradients in federated fune-tuning for large language models, 2024

    Huiwen Wu, Xiaohan Li, Deyi Zhang, Xiaogang Xu, Jiafei Wu, Puning Zhao, and Zhe Liu. Cg-fedllm: How to compress gradients in federated fune-tuning for large language models, 2024. URL https://arxiv.org/abs/2405.13746

    2024

  30. [38]

    Chain of lora: Efficient fine-tuning of language models via residual learning, 2024

    Wenhan Xia, Chengwei Qin, and Elad Hazan. Chain of lora: Efficient fine-tuning of language models via residual learning, 2024. URL https://arxiv.org/abs/2401.04151

    2024 arXiv

  31. [39]

    Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models, 2022

    Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models, 2022. URL https://arxiv.org/abs/2106.10199

    2022

  32. [40]

    Adalora: Adaptive budget allocation for parameter-efficient fine-tuning, 2023

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning, 2023. URL https://arxiv.org/abs/2303.10512

    2023 arXiv

  33. [41]

    Galore: Memory-efficient llm training by gradient low-rank projection, 2024

    Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection, 2024. URL https://arxiv.org/abs/2403.03507

    2024 arXiv

  34. [42]

    Efficient implementations for orthogonal matching pursuit

    Hufei Zhu, Wen Chen, and Yanpeng Wu. Efficient implementations for orthogonal matching pursuit. Electronics, 9 0 (9): 0 1507, 2020

    2020

  35. [43]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.