Pith. sign in

REVIEW 4 major objections 7 minor 1 cited by

CE-LoRA: Computation-Efficient LoRA Fine-Tuning for Language Models

T0 review · 4 major / 7 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CE-LoRA claims to replace LoRA's dense frozen-gradient backward product with a sparse approximation that preserves memory and matches LoRA's convergence rate.

desk verdict CE-LoRA is a plausible engineering contribution to LoRA backward-pass speedup, but its convergence theorem rests on an assumption that essentially asserts the result and is unlikely to hold near stationarity. read the letter →

arxiv 2502.01378 v1 pith:BADZKZSV submitted 2025-02-03 cs.LG

Guanduo Chen , Yutong He , Yipeng Hu , Kun Yuan , Binhang Yuan This is my paper · ORCID
classification cs.LG MSC 68T0768W4090C26
keywords low-rankadaptationcomputation-efficientfine-tuningapproximatematrixmultiplicationstructuredsparsitybackpropagationaccelerationlargelanguagemodelsmomentumSGDconvergenceanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CE-LoRA is a fine-tuning algorithm that tries to give LoRA's computation a speedup its memory savings never provided. The paper identifies the backward pass through the frozen weight matrix as the dominant remaining cost, and attacks it with an approximate matrix product that keeps only the most informative rows and columns, refreshed periodically. To stop the induced error from compounding as it travels backward, CE-LoRA first splits the frozen weight into a strong low-rank part and a small residual part via SVD, so the approximation acts only on the residual. The authors prove that momentum-SGD CE-LoRA converges at $\mathcal{O}(1/\sqrt{T})$, matching LoRA's rate, and report up to 3.39x faster backward passes and about 36% faster end-to-end fine-tuning with nearly unchanged accuracy on commonsense and arithmetic reasoning benchmarks.

What carries the argument

The machinery has two coupled parts. Approximated Matrix Multiplication (AMM) rewrites $PQ = \sum_{i=1}^n p_i q_i^\top$, scores each rank-one term by $\lVert p_i q_i^\top \rVert_F$, and approximates the product by the sum over the top-$s$ terms, giving $\mathcal{C}_p(PQ)$ at a fraction $p=s/n$ of the FLOPs. Double-LoRA first decomposes the frozen weight as $W_0 = B_0A_0 + W_s$ using an SVD, so the low-rank part is computed exactly and cheaply and the residual $W_s$ passed to AMM carries little signal; the activation gradient is then $G_x \approx \mathcal{C}_p(W_s^\top G_y) + A_0^\top(B_0^\top G_y) + A^\top G_z$. This split is what keeps the AMM-induced error from growing as it backpropagates through earlier layers.

What would settle it

Measure the relative error $\lVert \hat{g}_t - g_t \rVert_2 / \lVert g_t \rVert_2$ at every iteration of a standard LLM fine-tuning run; if it ever reaches or exceeds 1, or fails to stay below a fixed $\delta<1$ across all iterations, Assumption 4.4 is false and the $\mathcal{O}(1/\sqrt{T})$ theorem does not apply to CE-LoRA.

Watch

Extended reading notes

Core claim

The central claim is that LoRA's dominant backward cost is the dense product $W_0^\top G_y$ through the frozen weight matrix, and that this product can be approximated without harming fine-tuning. CE-LoRA computes $W_0^\top G_y$ as $\mathcal{C}_p(W_s^\top G_y) + A_0^\top(B_0^\top G_y)$, where $W_0 = W_s + B_0A_0$ is an SVD split into a dominant frozen low-rank part and a small residual, and $\mathcal{C}_p$ keeps only the rows and columns with the largest Frobenius-norm importance scores, refreshing the selection every $\tau$ steps. The paper proves that, under a uniform relative-error bound on this gradient estimate, CE-LoRA with momentum SGD converges at $\mathcal{O}(1/\sqrt{T})$, matching LoRA's order, and reports up to 3.39x backward-pass speedup, up to 36.3% end-to-end training-time reduction, and fine-tuning accuracy within about 1.6% of LoRA on commonsense and arithmetic reasoning benchmarks.

Load-bearing premise

The convergence proof assumes that every CE-LoRA gradient estimate stays within a fixed multiplicative factor ($1-\delta$) of the true gradient for the whole run, a property the paper spot-checks on three small GLUE tasks and that is hardest to satisfy near the end of training, where the true gradient is near zero.

Editorial extensions

If this is right

  • Fine-tuning runs up to 36.3% faster end-to-end and up to 3.39x faster in the backward pass than standard LoRA on the tested layer shapes and sparsity levels.
  • Memory stays at LoRA's level: the frozen double-LoRA part adds buffers rather than optimizer states, so CE-LoRA can use roughly the same rank budget.
  • CE-LoRA with momentum SGD has the same order of convergence as LoRA, so the computational savings do not, in the paper's model, change the statistical efficiency.
  • Fine-tuning accuracy on commonsense and arithmetic reasoning stays within about 1.6% of LoRA on average across models from roughly 7B to 13B parameters.
  • Sparsity is applied selectively: Q, K, and Gate layers stay dense, while V, O, Up, and Down layers run at sparsity levels 0.55-0.65, so the method depends on per-layer sensitivity to approximation error.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A stress test the paper does not run: vary the index-recompute period $\tau$; if accuracy degrades sharply as $\tau$ grows, the practical behavior depends on the historical importance-score heuristic more than the convergence theorem reveals.
  • The same residual-split-plus-AMM recipe could be applied to the other dense products in LoRA's backward pass, such as $G_A = G_z x^\top$, which the paper leaves untouched; that would roughly double the computational savings if the error-control mechanism transfers.
  • If Assumption 4.4 could be replaced by a weaker expected-error condition that holds only outside a neighborhood of stationary points, the convergence result would extend to late-stage fine-tuning without requiring the gradient estimate to stay accurate when the true gradient is tiny.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes CE-LoRA, a modification of LoRA fine-tuning that reduces backward-pass computation by approximating the dense multiplication W0^T * Gy with a sparsified multiplication (AMM) plus a low-rank correction (double-LoRA). It claims a convergence rate of O(1/sqrt(T)) under Assumption 4.4 and reports up to 3.39x backward acceleration and 36.3% training-step speedup with accuracy close to LoRA on commonsense and arithmetic reasoning benchmarks. The paper identifies the activation-gradient computation as the main LoRA backward bottleneck, provides a complexity analysis, and presents experiments on LLaMA-2 and LLaMA-3 models.

Significance. If the empirical speedup is real and the convergence claim were properly supported, this would be a useful contribution to parameter-efficient fine-tuning. The paper isolates a genuine computational bottleneck and proposes a plausible heuristics-based remedy, and the layer-wise sparsity analysis is a useful design study. However, the theoretical guarantee rests on a strong assumption that is not derived and is only spot-checked on small GLUE tasks, and the headline speedup is measured on a single linear layer with custom kernels that are not described. The current manuscript does not adequately support its central theoretical claim.

major comments (4)
  1. [Section 4, Assumption 4.4] The multiplicative gradient-error bound in (13)-(14) is not established from the AMM or double-LoRA constructions. The paper itself acknowledges the assumption is 'not standard' and justifies it only with empirical densities from gemma-2b on three GLUE tasks (Figures 3-4), which do not verify a uniform delta over all iterations or in the LLaMA setting. Since Theorem 4.5 and Corollary 4.6 depend entirely on this assumption, the claimed same-rate convergence is not demonstrated. Moreover, p and tau do not appear in the theorem, so the O(1/sqrt(T)) rate is not tied to the implemented sparsity and index-recomputation choices.
  2. [Section 4, Theorem 4.5 / Corollary 4.6] The convergence theorem is stated for momentum SGD with beta1 annealed to zero (see Corollary 4.6), while all experiments in Section 5 use AdamW. The stated O(1/sqrt(T)) guarantee therefore does not describe the algorithm that is actually evaluated, leaving a disconnect between theory and experiments.
  3. [Section 5.3, Figure 6] The 3.39x backward-pass speedup and the 36.3% 'end-to-end' speedup are measured on a single linear layer using purpose-built CUDA kernels, not on whole-model training. The abstract's '36.3% end-to-end training speedup' overstates the result. The kernel implementation is not described and no code is provided, so the wall-clock results are not reproducible.
  4. [Section 3.3] The layer-wise sparsity levels are selected from sensitivity experiments on the same benchmark families (Commonsense 14K and Math 7K) that are later used for evaluation (Commonsense 170K and Math 10k). This creates a risk of overfitting the sparsity configuration to the evaluation tasks. No cross-task validation or ablation of the sparsity-selection procedure is provided.
minor comments (7)
  1. [Abstract] The word 'iteartions' is a typo for 'iterations'.
  2. [Page 1 footnote] 'Coorresponding author' should be 'Corresponding author'.
  3. [Table 3] The column header 'SV AMP' should be 'SVAMP' (a single benchmark name).
  4. [Figures 3 and 4] The y-axis label 'ePDF' is unclear; the paper should say 'density' or 'estimated PDF'.
  5. [Section 3.1] The notation Cp(P·Q) is introduced in (9) but not formally defined; a sentence defining Cp as the AMM estimate with structured sparsity p would improve clarity.
  6. [Algorithm 1, line 13] The importance score uses W_{s,ell}^T[:,i] and G_{y_ell}[i,:]; the dimensions should be checked against the AMM construction in Section 3.1, since the selected indices I are used to index both rows and columns in line 20.
  7. [Appendix A, Lemma A.1] In the proof of Lemma A.1, the factor in the second term of (15) appears as 4L^2/(delta beta_1^2), but the derivation around (20) suggests 2L^2/(delta beta_1); the algebra should be reconciled.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the convergence theorem is a conditional statement under a stated gradient-error assumption, and no load-bearing step reduces to a fitted parameter or self-citation.

full rationale

The paper's O(1/sqrt(T)) convergence claim (Theorem 4.5, Corollary 4.6) is derived under Assumption 4.4, which bounds the CE-LoRA gradient estimator's deviation from the true gradient. This is a stated assumption rather than a derived consequence of the AMM or double-LoRA construction, and the theorem is explicitly conditional on it. The paper provides an empirical spot-check of the assumption on gemma-2b GLUE tasks, but that is not the same as fitting a parameter and then calling the fitted value a prediction; the convergence proof is a mathematical implication from the assumptions, not an empirical forecast forced by data. The assumption is strong and possibly hard to verify in all regimes, but that is a correctness or rigor concern, not circularity. The empirical speedup results are benchmark measurements against LoRA and are independent of the theoretical argument. The only self-citation (reference [52], by co-authors Yutong He, Yipeng Hu, and Kun Yuan) appears in the related-work discussion of subspace optimization methods and is not load-bearing for any claim in this paper. No equation is shown to equal another by construction, and no prediction reduces to its input by definition. Therefore the circularity score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on several assumptions: standard optimization assumptions, the ad hoc gradient-error bound in Assumption 4.4, the structured-sparsity premise for AMM, the claim that the SVD residual is more amenable to approximation, and the transferability of layer-wise sparsity choices. The free parameters include sparsity levels, double-LoRA rank, and the unreported index recomputation period tau. No new physical or conceptual entities are introduced.

free parameters (3)
  • layer-wise sparsity levels p = pV=pO=0.55, pUp=pDown=0.65, p=1 for Q/K/Gate
    Chosen from sensitivity experiments on Commonsense 14K and Math 7K (Section 3.3); these values directly set approximation quality and speedup in the main experiments.
  • double-LoRA rank r0 = 28 in sensitivity tests; not explicitly reported for main runs (trainable rank 14/56)
    A hyperparameter controlling the frozen low-rank correction; the memory budget is balanced by reducing the trainable rank, so r0 affects both memory and AMM error.
  • index recomputation period tau = not specified
    Controls how often importance scores are recomputed; this determines both accuracy and overhead, but no value is reported and its cost is excluded from the complexity analysis.
assumptions (5)
  • standard math Standard stochastic optimization assumptions: lower bounded loss, L-smoothness, unbiased stochastic gradient with bounded variance (Assumptions 4.1-4.3).
    Used in the convergence proof in Section 4 and Appendix A; these are conventional and acceptable.
  • ad hoc to paper Assumption 4.4: the CE-LoRA gradient estimate has a uniform multiplicative error bound of the form ||g_hat_t - g_t|| <= (1 - delta)||g_t||.
    This is the key premise of Theorem 4.5; it is not derived from AMM or double-LoRA and is only empirically spot-checked.
  • domain assumption Structured sparsity: a few outer products p_i q_i^T dominate the matrix product P Q.
    The entire AMM idea in Section 3.1 depends on this assertion about the weight-gradient outer products; it is plausible but unproven.
  • domain assumption The SVD residual W_s = W0 - B0 A0 has better structured sparsity and is more compatible with AMM.
    Section 3.2 claims this motivates double-LoRA, but no formal justification or measurement of the residual's sparsity is given.
  • domain assumption Sensitivity-derived layer-wise sparsity settings transfer across model sizes and task families.
    Section 3.3 selects sparsity levels on small datasets and then applies them to LLaMA-2/3 models; the paper does not validate this transfer on held-out settings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CE-LoRA: Computation-Efficient LoRA Fine-Tuning for Language Models." pith.science (2026). https://pith.science/paper/BADZKZSV

@misc{pith2026250201378,
  author       = {Pith},
  title        = {Pith review of: CE-LoRA: Computation-Efficient LoRA Fine-Tuning for Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BADZKZSV}},
  note         = {Machine review of arXiv:2502.01378}
}
abstract

Large Language Models (LLMs) demonstrate exceptional performance across various tasks but demand substantial computational resources even for fine-tuning computation. Although Low-Rank Adaptation (LoRA) significantly alleviates memory consumption during fine-tuning, its impact on computational cost reduction is limited. This paper identifies the computation of activation gradients as the primary bottleneck in LoRA's backward propagation and introduces the Computation-Efficient LoRA (CE-LoRA) algorithm, which enhances computational efficiency while preserving memory efficiency. CE-LoRA leverages two key techniques: Approximated Matrix Multiplication, which replaces dense multiplications of large and complete matrices with sparse multiplications involving only critical rows and columns, and the Double-LoRA technique, which reduces error propagation in activation gradients. Theoretically, CE-LoRA converges at the same rate as LoRA, $ \mathcal{O}(1/\sqrt{T}) $, where $T$ is the number of iteartions. Empirical evaluations confirm that CE-LoRA significantly reduces computational costs compared to LoRA without notable performance degradation.

Figures

Figures reproduced from arXiv: 2502.01378 by the authors.

Figure 1
Figure 1. An illustration of the Approximated Matrix Multiplication (AMM) technique (left) and the CE-LoRA framework (right). Compared with vanilla LoRA, can we develop a more computation-efficient fine￾tune algorithm by the same memory budget without sacrificing the statistical efficiency (i.e., convergence)? To answer this question, we first conduct a computational analysis of LoRA’s backward propagation procedure and ident… view at source ↗
Figure 2
Figure 2. Layer-wise Sensitivity Analysis of LLaMA3.2-1B. To estimate the product T computation-efficiently, we may assume the matrices P and Q enjoy some kinds of structured sparsity, such that a few (piq ⊤ i ) ′ s contribute to most of the result Pn i=1 piq ⊤ i , in which case we could estimate T by computing the most important parts only. Specifically, we identify s most important indices 1 ≤ i1 < · · · < is ≤ n, and the A… view at source ↗
Figure 3
Figure 3. Empirical validation of (13) on MRPC (left), RTE (middle) and CoLA (right). [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Empirical validation of (14) on MRPC (left), RTE (middle) and CoLA (right). [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Loss curve of commonsense reasoning fine-tune task. Each row in the figure corresponds to [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Comparison of training latency for CE-LoRA and LoRA at various sparsity levels (i.e., [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From over-reliance to smart integration: using Large-Language Models as translators between specialized modeling and simulation tools

    cs.SE 2025-06 conditional novelty 3.0 of 10

    Large language models should serve as translators in modeling and simulation workflows, coordinated with specialized tools and supported by LoRA-based task adapters on a shared backbone.

Reference graph

Works this paper leans on

61 extracted references · 18 canonical work pages · cited by 1 Pith paper

  1. [1]

    On the opportunities and risks of foundation models

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021

    arXiv 2021

  2. [2]

    Instruction tuning for large language models: A survey

    Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792, 2023

    arXiv 2023

  3. [3]

    Parameter-efficient fine-tuning for large models: A comprehensive survey

    Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403.14608, 2024

    arXiv 2024

  4. [4]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    arXiv 2021

  5. [5]

    Fast monte carlo algorithms for matrices i: Approximating matrix multiplication

    Petros Drineas, Ravi Kannan, and Michael W Mahoney. Fast monte carlo algorithms for matrices i: Approximating matrix multiplication. SIAM Journal on Computing, 36(1):132–157, 2006

    work page 2006

  6. [6]

    Accelerating large language models through partially linear feed-forward network

    Gansen Hu, Zhaoguo Wang, Jinglin Wei, Wei Huang, and Haibo Chen. Accelerating large language models through partially linear feed-forward network. arXiv e-prints, pages arXiv– 2501, 2025

    work page 2025

  7. [7]

    First Activations Matter: Training-Free Methods for Dynamic Activation in Large Language Models

    Chi Ma, Mincong Huang, Ying Zhang, Chao Wang, Yujie Wang, Lei Yu, Chuan Liu, and Wei Lin. First activations matter: Training-free methods for dynamic activation in large language models. arXiv preprint arXiv:2408.11393, 2024

    work page Pith review arXiv 2024

  8. [8]

    From galore to welore: How low-rank weights non-uniformly emerge from low-rank gradients

    Ajay Jaiswal, Lu Yin, Zhenyu Zhang, Shiwei Liu, Jiawei Zhao, Yuandong Tian, and Zhangyang Wang. From galore to welore: How low-rank weights non-uniformly emerge from low-rank gradients. arXiv preprint arXiv:2407.11239, 2024

    arXiv 2024

Show all 61 references
  1. [9]

    Lsaq: Layer-specific adaptive quantization for large language model deployment

    Binrui Zeng, Bin Ji, Xiaodong Liu, Jie Yu, Shasha Li, Jun Ma, Xiaopeng Li, Shangwen Wang, and Xinran Hong. Lsaq: Layer-specific adaptive quantization for large language model deployment. arXiv preprint arXiv:2412.18135, 2024

    2024 arXiv

  2. [10]

    Pushing the limits of large language model quantization via the linearity theorem.arXiv preprint arXiv:2411.17525, 2024

    Vladimir Malinovskii, Andrei Panferov, Ivan Ilin, Han Guo, Peter Richtárik, and Dan Alistarh. Pushing the limits of large language model quantization via the linearity theorem.arXiv preprint arXiv:2411.17525, 2024

    2024 arXiv

  3. [11]

    Q-galore: Quantized galore with int4 projection and layer-adaptive low-rank gradients

    Zhenyu Zhang, Ajay Jaiswal, Lu Yin, Shiwei Liu, Jiawei Zhao, Yuandong Tian, and Zhangyang Wang. Q-galore: Quantized galore with int4 projection and layer-adaptive low-rank gradients. arXiv preprint arXiv:2407.08296, 2024

    2024 arXiv

  4. [12]

    Training-free activation sparsity in large language models

    James Liu, Pragaash Ponnusamy, Tianle Cai, Han Guo, Yoon Kim, and Ben Athiwaratkun. Training-free activation sparsity in large language models. arXiv preprint arXiv:2408.14690, 2024

    2024 arXiv

  5. [13]

    Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models

    Zhiqiang Hu, Yihuai Lan, Lei Wang, Wanyu Xu, Ee-Peng Lim, Roy Ka-Wei Lee, Lidong Bing, and Soujanya Poria. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.01933, 2023

    2023 arXiv

  6. [15]

    Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641, 2019

    2019

  7. [16]

    Dolan and Chris Brockett

    William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005

    2005

  8. [17]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019. 11

    2019

  9. [18]

    BoolQ: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of ...

    2019

  10. [19]

    Piqa: Reasoning about phys- ical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020

    2020

  11. [20]

    Socialiqa: Com- monsense reasoning about social interactions

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Com- monsense reasoning about social interactions. arXiv preprint arXiv:1904.09728, 2019

    1904 arXiv

  12. [21]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  13. [22]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021

    2021

  14. [23]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  15. [24]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018

    2018 arXiv

  16. [25]

    Solving general arithmetic word problems

    Subhro Roy and Dan Roth. Solving general arithmetic word problems. arXiv preprint arXiv:1608.01413, 2016

    2016 arXiv

  17. [26]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  18. [27]

    Learning to solve arithmetic word problems with verb categorization

    Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. Learning to solve arithmetic word problems with verb categorization. InProceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 523–533, 2014

    2014

  19. [28]

    Program induction by rationale generation: Learning to solve and explain algebraic word problems

    Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Regina Barzilay and Min-Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Li...

  20. [29]

    Parsing algebraic word problems into equations

    Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Du- mas Ang. Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics, 3:585–597, 2015

    2015

  21. [30]

    Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editor...

    2021

  22. [31]

    MAWPS: A math word problem repository

    Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. MAWPS: A math word problem repository. In Kevin Knight, Ani Nenkova, and Owen Rambow, editors, Proceedings of the 2016 Conference of the North American Chapter of the Association for Computat...

    2016

  23. [32]

    Llama 2: Open foundation and fine-tuned chat models, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023

  24. [33]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024

    2024

  25. [34]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  26. [35]

    Improving language understanding by generative pre-training

    Alec Radford. Improving language understanding by generative pre-training. 2018

    2018

  27. [36]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019

    2019

  28. [37]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  29. [38]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

  30. [39]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  31. [40]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  32. [41]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  33. [42]

    Bloom: A 176b-parameter open-access multilingual language model

    Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili ´c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. 2023

    2023

  34. [43]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  35. [44]

    The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only

    Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv pr...

    2023 arXiv

  36. [45]

    Parameter-efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790–2799. PMLR, 2019

    2019

  37. [46]

    Adapterhub: A framework for adapting transformers

    Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vuli´c, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: A framework for adapting transformers. arXiv preprint arXiv:2007.07779, 2020. 13

    2007 arXiv

  38. [47]

    Relora: High- rank training through low-rank updates

    Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky. Relora: High- rank training through low-rank updates. In The Twelfth International Conference on Learning Representations, 2023

    2023

  39. [48]

    S$^{2}$FT: Efficient, scalable and generalizable LLM fine-tuning by structured sparsity

    Xinyu Yang, Jixuan Leng, Geyang Guo, Jiawei Zhao, Ryumei Nakada, Linjun Zhang, Huaxiu Yao, and Beidi Chen. S$^{2}$FT: Efficient, scalable and generalizable LLM fine-tuning by structured sparsity. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  40. [49]

    Sltrain: a sparse plus low-rank approach for parameter and memory efficient pretraining

    Andi Han, Jiaxiang Li, Wei Huang, Mingyi Hong, Akiko Takeda, Pratik Jawanpuria, and Bamdev Mishra. Sltrain: a sparse plus low-rank approach for parameter and memory efficient pretraining. arXiv preprint arXiv:2406.02214, 2024

    2024 arXiv

  41. [50]

    Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning

    Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning. arXiv preprint arXiv:2403.17919, 2024

    2024 arXiv

  42. [51]

    Galore: Memory-efficient llm training by gradient low-rank projection

    Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507, 2024

    2024 arXiv

  43. [52]

    Subspace optimization for large language models with convergence guarantees

    Yutong He, Pengrui Li, Yipeng Hu, Chuyan Chen, and Kun Yuan. Subspace optimization for large language models with convergence guarantees. arXiv preprint arXiv:2410.11289, 2024

    2024 arXiv

  44. [53]

    Flora: Low-rank adapters are secretly gradient compressors

    Yongchang Hao, Yanshuai Cao, and Lili Mou. Flora: Low-rank adapters are secretly gradient compressors. arXiv preprint arXiv:2402.03293, 2024

    2024 arXiv

  45. [54]

    Back razor: Memory-efficient transfer learning by self-sparsified backpropagation

    Ziyu Jiang, Xuxi Chen, Xueqin Huang, Xianzhi Du, Denny Zhou, and Zhangyang Wang. Back razor: Memory-efficient transfer learning by self-sparsified backpropagation. Advances in neural information processing systems, 35:29248–29261, 2022

    2022

  46. [55]

    Sheared back- propagation for fine-tuning foundation models

    Zhiyuan Yu, Li Shen, Liang Ding, Xinmei Tian, Yixin Chen, and Dacheng Tao. Sheared back- propagation for fine-tuning foundation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5883–5892, 2024

    2024

  47. [56]

    Mixed precision training

    Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXiv:1710.03740, 2017

    arXiv

  48. [57]

    Qlora: Efficient finetuning of quantized llms

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024

    2024

  49. [58]

    Apollo: Sgd-like memory, adamw-level performance

    Hanqing Zhu, Zhenyu Zhang, Wenyan Cong, Xi Liu, Sem Park, Vikas Chandra, Bo Long, David Z Pan, Zhangyang Wang, and Jinwon Lee. Apollo: Sgd-like memory, adamw-level performance. arXiv preprint arXiv:2412.05270, 2024

    2024 arXiv

  50. [59]

    Dropbp: Accelerating fine-tuning of large language models by dropping backward propagation

    Sunghyeon Woo, Baeseong Park, Byeongwook Kim, Minjung Jo, Sejung Kwon, Dongsuk Jeon, and Dongsoo Lee. Dropbp: Accelerating fine-tuning of large language models by dropping backward propagation. arXiv preprint arXiv:2402.17812, 2024. 14 A Missing Proofs In this section, we prov...

    2024 arXiv

  51. [60]

    (22) 15 Taking expectation and summing (22) for t = 0, 1, · · ·, Tyields inf x f (x) − f (x0) ≤ η 2 TX t=0 E[∥∇f (xt) − mt∥2 2] − 1 2η − L 2 TX t=0 E[∥xt+1 − xt∥2 2] − η 2 TX t=0 E[∥∇f (xt)∥2 2]. (23) Applying Lemma A.1 to (23) and noting that β1 ∈ (0, δ/(24 − 12δ)) implies (1...

  52. [61]

    (24) Since η ≤ min{1/2L, p (δβ 2 1 )/(8L2)} implies 1/(4η) ≥ L/2 and 1/(4η) ≥ (2ηL2)/(δβ 2 1 ), (21) is a direct result of (24). 16

  53. [2017]

    Association for Computational Linguistics

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.