Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that one step of full fine-tuning gradient already contains the low-rank shift LoRA needs, and proves that spectral initialization from that gradient makes LoRA converge linearly.

desk verdict Worth a serious referee, but the proof covers an initialization the algorithm does not use. read the letter →

arxiv 2502.01235 v3 pith:BQTNPBM3 submitted 2025-02-03 stat.ML cs.AIcs.LG

Yuanhe Zhang , Fanghui Liu , Yudong Chen This is my paper · ORCID
classification stat.MLcs.AIcs.LG MSC 65F5568T07
keywords low-rankadaptationspectralinitializationsubspacealignmentfine-tuninglargelanguagemodelsgradientdescentpreconditionedsingularvaluedecompositionlinearconvergence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LoRA-One argues that the information needed to fine-tune a large model with low-rank adapters is already present in a single gradient step of full fine-tuning. For a linear fine-tuning setup, the paper proves that gradient descent on LoRA adapters aligns $A_t$ and $B_t$ with the singular subspaces of the one-step full gradient $G^\natural$, and that initializing $A_0B_0$ from the top-$r$ SVD of $G^\natural$ gives $\|A_0B_0 - \Delta\|_{\mathrm{op}} \le \epsilon\|\Delta\|_{\mathrm{op}}$. Under a dominance condition on the pretrained weights, the same one-step initialization works for a ReLU model, and preconditioned gradient descent then converges linearly at a rate independent of the condition number of $\Delta$. On GLUE, GSM8K, MMLU, and HumanEval, LoRA-One matches LoRA's time and memory while improving accuracy. The paper therefore turns subspace alignment from a description of LoRA dynamics into an initialization principle.

What carries the argument

The central object is the one-step full fine-tuning gradient $G^\natural = -\nabla_W L(W^\natural)$, whose truncated singular value decomposition defines the spectral initialization $A_0 = \sqrt{\gamma}\, U_{[:,1:r]} S^{1/2}_{[1:r]}$ and $B_0 = \sqrt{\gamma}\, S^{1/2}_{[1:r]} V_{[:,1:r]}^\top$, so that $A_0B_0$ is the top-$r$ approximation of $G^\natural$. Two analytical devices carry the argument: the Schur decomposition of the stacked linear dynamics $H = \begin{pmatrix} I_d & \eta G^\natural \\ \eta (G^\natural)^\top & I_k \end{pmatrix}$, which decouples the spectral evolution of $A_t$ and $B_t$ and yields the alignment theorems; and, for the nonlinear case, the identity $\mathbb{E}_x[J_{W_t}] = \tfrac12(A_tB_t - \Delta) + \Psi(t)$, where the residual $\Psi(t)$ is shown to be $O(1/(\kappa r^*))$ under Assumption 4.1. Preconditioned updates $(B_tB_t^\top)^{-1}$ and $(A_t^\top A_t)^{-1}$ convert the local contraction into a rate independent of $\kappa(\Delta)$.

What would settle it

Fine-tune a ReLU teacher where the shift $\Delta$ has norm comparable to the pretrained weight, violating Assumption 4.1, run LoRA-One with spectral initialization, and check whether $\|A_tB_t - \Delta\|_F$ decays at the claimed $(1 - \eta/4)^t$ rate; the proof predicts failure once the ratio exceeds $C/(\kappa r^*)$. Equivalently, measure that ratio on real LLaMA layers during fine-tuning: if the histogram does not concentrate at $O(1/(\kappa r^*))$, the assumptions behind Theorem 4.2 are violated for those layers.

Watch

Extended reading notes

Core claim

The paper's central claim is that the one-step full fine-tuning gradient $G^\natural = -\nabla_W L(W^\natural)$ contains the low-rank structure needed to initialize LoRA adapters well. Under gradient descent with standard LoRA initialization, the iterates align: $B_t$ stays in the top-$r^*$ right singular subspace of $G^\natural$, and $A_t$ aligns with the top-$r^*$ left singular subspace after a short phase. If instead the adapters are initialized spectrally from the truncated SVD of $G^\natural$, so that $A_0B_0$ equals the best rank-$r$ approximation of $G^\natural$, then $A_0B_0$ is already close to the unknown low-rank shift $\Delta$ in linear models, and a similar conclusion holds for ReLU models once the pretrained weight norm dominates the shift. Continuing with gradient descent gives linear convergence of $\|A_tB_t - \Delta\|_F$, and adding preconditioners removes the dependence on the condition number $\kappa(\Delta)$. The same principle is implemented as the LoRA-One algorithm, which computes one gradient, takes its SVD, and then trains with AdamW; the reported experiments show improved accuracy over LoRA and LoRA-GA on natural language understanding, mathematical reasoning, and code generation benchmarks at nearly identical time and memory cost.

Load-bearing premise

For the nonlinear convergence theorem, each output neuron's pretrained weight must dominate its fine-tuning shift, meaning $\max\{\lambda^*_{r^*}, \|\Delta_m\|\} / \|\tilde{w}^\natural_m\|^2 = O(1/(\kappa r^*))$, so that the residual term in the expected gradient stays small; the paper verifies this empirically on one T5 model on SST-2 rather than on the LLaMA 2-7B benchmarks.

Editorial extensions

If this is right

  • If the guarantees hold, practitioners get a parameter-efficient fine-tuning recipe: compute one full-batch or large-batch gradient, take its top-$r$ SVD, and start LoRA there; no further full-gradient steps are needed.
  • The linear convergence results mean the adapter product error $\|A_tB_t - \Delta\|_F$ shrinks geometrically, and with preconditioning the rate does not degrade when the downstream shift $\Delta$ is ill-conditioned.
  • The subspace-alignment theorem doubles as a design rule: the correct initialization subspace is the top-$r$ singular subspace of the one-step gradient, not an arbitrary $2r$-dimensional approximation of that gradient.
  • Empirically, on the tested LLM benchmarks, LoRA-One improves accuracy over LoRA and LoRA-GA at essentially identical time and memory cost, with the largest gains on smaller GLUE tasks and on code generation.
  • The one-step gradient can suffice on small-scale natural language understanding datasets even with a low-rank approximation computed from a small sampled batch, sometimes matching LoRA's accuracy in under a second.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The alignment theory suggests a direct diagnostic: the principal angle between the top-$r$ singular subspace of the learned adapter product and the corresponding subspace of the one-step gradient should predict fine-tuning success, and this angle could be tracked without recomputing full gradients.
  • Because the theory identifies the correct singular subspace, it offers an explanation for why initializations that use other singular vectors of $G^\natural$ may stall; this could be tested by comparing LoRA-One, LoRA-GA, and LoRA-SB on synthetic linear problems with known rank $r^*$ in both the under-ranked and over-ranked regimes.
  • The preconditioned rate independent of $\kappa(\Delta)$ points to the shift's condition number as the true bottleneck of low-rank fine-tuning, suggesting that condition-number-aware preconditioners might transfer to other PEFT methods beyond LoRA-One.
  • The assumption check in the paper is limited to one T5 model on SST-2; a natural extension is to measure the same spectral ratios on LLaMA-scale layers, where the theory's dominance condition has not yet been verified.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies low-rank adaptation (LoRA) for fine-tuning pre-trained models, focusing on the singular-subspace alignment of the LoRA factors (A_t, B_t) with the one-step full fine-tuning gradient G♮. For linear models it proves alignment under random initialization, and shows that a spectral initialization based on the SVD of G♮ gives A0B0 ≈ Δ and linear convergence under (preconditioned) gradient descent. For nonlinear ReLU/Gaussian models it establishes a similar initial-error bound and linear convergence under preconditioned gradient descent, assuming a dominance condition on the pre-trained weights. The proposed algorithm, LoRA-One, uses this spectral initialization and is evaluated on GLUE, GSM8K, MMLU, and HumanEval, reporting improved accuracy over LoRA and LoRA-GA at comparable cost.

Significance. If the theoretical guarantees applied to the implemented algorithm, this would be a valuable contribution: it gives a parameter-free initialization principle, explicit finite-sample bounds, and demonstrates that a single full-gradient step can provide a good low-rank starting point. The paper is also honest in providing extensive appendices, detailed experimental settings, and code. The main theorem statements are nontrivial and the linear alignment analysis, in particular the Schur-decomposition argument for decoupling A_t and B_t, is a technically interesting contribution. However, the central 'provably and efficiently' claim is weakened by a concrete mismatch between the initialization analyzed in the theorems and the initialization evaluated in Algorithm 1, as well as by the fact that the convergence theorems cover GD/preconditioned GD while the experiments use AdamW.

major comments (4)
  1. [Algorithm 1, lines 3–6; Theorems 3.3, C.17, 4.2] Algorithm 1 does not implement the initialization analyzed in the theorems. Lines 3–6 set S ← S/S[0,0] and γ ← 1/s, so the initial product is A0B0 = U(S/S[0,0])V^T/s = G♮/(s·λmax(G♮)). In contrast, the proof of Theorem 3.3 (Lemma C.11) uses A0B0 = G♮, i.e. unnormalized S and γ=1, while the nonlinear result Lemma D.5 and Theorem 4.2 explicitly set γ=2. For the values s ∈ {16,64,128} used in Section 6, the product A0B0 is scaled by 1/s relative to G♮, so ∥A0B0 − Δ∥op is of order ∥Δ∥op rather than ≤ ε∥Δ∥op as required by Theorem 3.3. Moreover, the lower bounds λr*(A0), λr*(B0) ≥ √(λ*_r*/2) in Lemma C.11 fail because both factors are multiplied by 1/√s. Consequently Theorems C.17 and 4.2 cannot be invoked for the Algorithm 1 actually evaluated in Section 6. This is a load-bearing theory–implementation gap, not a cosmetic discrepancy.
  2. [Section 4, Eq. (9); Algorithm 1, line 9] The convergence theorems analyze gradient descent (Eq. (4)) and preconditioned gradient descent (Eq. (9) and Appendix C.3), whereas Algorithm 1 trains with AdamW (line 9). The preconditioned variant in Appendix B still passes the preconditioned gradients through AdamW, which is not the exact update analyzed in Theorem 4.2. Since the paper's title and abstract claim provable guarantees for the proposed algorithm, the authors should either state explicitly that the theory applies to GD/preconditioned GD and treat AdamW as an empirical choice, or provide a convergence result for the actual optimizer used in the experiments.
  3. [Assumption 4.1; Appendix G.5; Section 6.2] Assumption 4.1 is load-bearing for the nonlinear analysis: Lemma D.2 uses it to show that the residual term Ψ(t) is small, and Lemma D.5 uses it for the initial error bound. It is verified empirically only for T5-base on SST-2 (Appendix G.5), while the main large-model results in Section 6.2 use LLaMA 2-7B on MetaMathQA, Alpaca, and Code-Feedback. If the ratio max{λ*_r*, ∥Δ_m∥}/∥w̃♮_m∥² is not O(1/(κr*)) in those layers, the nonlinear recovery and convergence proof no longer applies. The authors should report the quantities in Assumption 4.1 for the actual benchmark layers, or restrict the nonlinear claim to settings where the assumption is verified.
  4. [Section 4, first paragraph; Section 6.2; Algorithm 1, line 1] Two further gaps between theory and experiments are left unquantified. First, the nonlinear theorem is stated only for the exact-rank case r = r* ('we consider the exact-rank case r = r∗ for delivery'), while all LLM experiments fix r = 8 without knowledge of the true rank of Δ. Second, Theorem 3.3 and Lemma D.5 concern the full-batch gradient G♮ built from N i.i.d. samples, whereas Algorithm 1 computes the gradient from a sampled batch B, and the experiments use gradient batch size 8 (Appendix G.2, G.4). The stochastic-batch and rank-mismatch effects are not analyzed, so it is unclear which theoretical result covers the evaluated configuration. The authors should either extend the analysis or explicitly mark these as empirical heuristics outside the proven regime.
minor comments (5)
  1. [Theorem 3.2] The simplified expression for α in the main text contains κ♮ twice with inconsistent exponents ('O(θ^{3/2} κ♮ d^{−3/4} κ♮^{−1/2} ...)'); please reconcile it with the full statement in Theorem C.9.
  2. [Lemma C.12] The statement 'BtV⊥ = 0d×(d−r∗)' has a dimension error: B_t is r×k and V⊥ is k×(k−r*), so the right-hand side should be 0_{r×(k−r*)}. The correct dimension appears later in Lemma C.18.
  3. [Figure 1 and Section 3.1] The phrase 'mean principle angle' should be 'principal angle' for consistency with the standard terminology used elsewhere.
  4. [Section 5, LoRA-GA comparison] The claim that LoRA-GA's estimate is 'biased, scaling with the model size' is stated in prose and in Appendix F but is not accompanied by a formal bound; adding a short quantitative statement would make the comparison easier to verify.
  5. [Table 2] The 'Pre-train' row lists only four numbers while the table has five task columns; please explicitly mark MNLI as not reported or add the missing value.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the one-step-gradient recovery bound is a covariance-concentration statement, not a self-referential prediction.

full rationale

The derivation chain is self-contained. In the linear model, the data model in Assumption 2.3 gives G♮ = −∇L(W♮) = (1/N)Xᵀ(Y − XW♮) = Σ̂Δ (Eq. (5) with Section 2.2), and the proof of Lemma C.11 states "Due to rank(G♮) = r∗ and r ≥ r∗, then A0B0 = G♮." Hence Theorem 3.3's bound ∥A0B0 − Δ∥op ≤ ϵ∥Δ∥op is exactly the external covariance concentration ∥(Σ̂ − I)Δ∥op ≤ ϵ∥Δ∥op from Lemma E.1; the conclusion is not assumed but is a statistical estimate with an explicit rate. The nonlinear analysis is derived rather than assumed: Lemma D.2 expands −E[JWt] = (AtBt − Δ)/2 + Ψ(t), Lemma D.5 proves ∥A0B0 − Δ∥F ≤ ρλ∗_r∗ by decomposing into low-rank approximation, concentration, and population errors under γ = 2, and Theorem D.10 proves linear convergence by induction. The only self-citation, Chen et al. (2021b, Theorem 2.9) for a sin Θ bound in Lemma D.9, is a standard, parameter-free perturbation result and is not load-bearing. Two non-circular concerns are flagged for correctness: (i) Algorithm 1 lines 3–6 set S ← S/S[0,0] and γ ← 1/s, so the implemented A0B0 = G♮/(s·λmax(G♮)), whereas Theorems 3.3, C.21, and 4.2 use γ = 1 or γ = 2; this is a theory–implementation gap, not a circular reduction. (ii) Assumption 4.1 is verified only for T5 base on SST-2 (Appendix G.5), not for the LLaMA 2-7B benchmarks in Section 6.2; this is a missing-support issue for the nonlinear theorem's applicability, again not circularity. Neither concern makes the derivation equivalent to its inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. The central derivation depends on the three domain assumptions above and on standard concentration/singular-subspace perturbation theorems; implementation constants (gamma, alpha, rank, learning rate) are tuned in the experimental sections but are not part of the proof. The nonlinear theorem additionally fixes gamma=2 rather than the tunable algorithm value.

free parameters (4)
  • Spectral scaling gamma (1/s) = Per dataset, e.g., 1/128 for MNLI, 1/16 for SST-2, 1/64 for MRPC
    Algorithm 1 normalizes S by S[0,0] and sets gamma=1/s with s tuned per benchmark; the nonlinear theorem fixes gamma=2.
  • LoRA alpha = 16 in large-scale experiments; tuned in GLUE
    Output scaling in Algorithm 1 line 10 is selected per task and interacts with gamma.
  • LoRA rank r = 8
    Rank is fixed to 8; theory requires r >= r* for exact recovery, so the unknown true rank is approximated.
  • Task-specific learning rates and batch sizes = Grid-searched per dataset, e.g., MetaMathQA LR 2e-4 batch 32
    Used for all compared methods; empirical comparison is not parameter-free.
assumptions (4)
  • domain assumption The downstream shift Delta is exactly low-rank with rank r* < min(d,k).
    Assumption 2.2; the theory is for exact low-rank teacher shifts and does not cover approximately low-rank or full-rank shifts.
  • domain assumption Inputs are isotropic sub-Gaussian (linear) or standard Gaussian (nonlinear), labels are noiseless, and d < N.
    Assumption 2.3; covariance concentration and the Gaussian ReLU population-gradient formulas depend on this.
  • domain assumption For the nonlinear theorem, pretrained neuron norms dominate the shift: max{lambda*_r*, ||Delta_m||}/||w~_m||^2 = O(1/(kappa r*)).
    Assumption 4.1; keeps the residual Psi(t) in Lemma D.2 small. Verified empirically for one T5/SST-2 setup only.
  • standard math Standard concentration, Weyl, and Wedin sin-theta results are used without proof.
    Invoked in Lemmas C.6, C.16, D.4, D.9; accepted external theorems.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently." pith.science (2026). https://pith.science/paper/BQTNPBM3

@misc{pith2026250201235,
  author       = {Pith},
  title        = {Pith review of: LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BQTNPBM3}},
  note         = {Machine review of arXiv:2502.01235}
}
read the original abstract

This paper explores how theory can guide and enhance practical algorithms, using Low-Rank Adaptation (LoRA, Hu et al. 2022) in large language models as a case study. We rigorously prove that, under gradient descent, LoRA adapters align with specific singular subspaces of the one-step full fine-tuning gradient. This result suggests that, by properly initializing the adapters using the one-step full gradient, subspace alignment can be achieved immediately and applicable to both linear and nonlinear models. Building on our theory, we propose a theory-driven algorithm, LoRA-One, where the linear convergence (as well as generalization) is built and incorporating preconditioners theoretically helps mitigate the effects of ill-conditioning. Besides, our theory reveals connections between LoRA-One and other gradient-alignment-based methods, helping to clarify misconceptions in the design of such algorithms. LoRA-One achieves significant empirical improvements over LoRA and its variants across benchmarks in natural language understanding, mathematical reasoning, and code generation. Code is available at: https://github.com/YuanheZ/LoRA-One.

Figures

Figures reproduced from arXiv: 2502.01235 by the authors.

Figure 1
Figure 1. (a) Illustration of the alignment behavior of LoRA with the certain singular subspaces. (b) The mean principle [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Under (LoRA-init), the risk and the alignment to full one-step GD of LoRA with different α 2 and d, trained via GD on task (3). Left: the log risk under dif￾ferent initialization variance α 2 . The risk is defined as 1 2 ∥AtBt − ∆∥ 2 F . Right: the best principal angle between the top-r singular subspace of G♮ and At during train￾ing. Smaller is closer. The principal angle is defined as mint ∥U⊤ r ∗,⊥(G♮ ) Ur ∗ (At)… view at source ↗
Figure 3
Figure 3. Comparison of the GD trajectories under (Spectral-init) and (LoRA-init) with two different starting points. See more experimental details in Appendix G.1. the alignment between Zlin t and G♮ can be successfully transferred to that of Zt, see Theorem C.9 for details. We remark that previous work on matrix sensing (Stoger ¨ & Soltanolkotabi, 2021; Soltanolkotabi et al., 2023) via a symmetrization technique cannot be d… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The log-risk curve under LoRA-One, LoRA-SB, and LoRA-GA, trained via GD on fine-tuning task (3) under: 1) under-ranked case r < r∗ , 2) over-ranked case r > r∗ . lowing initialization strategy (omit the scaling parameters) A0 ← − h UeG♮ i [:,1:r] , B0 ← h VeG♮ i⊤ [:,r+…
Figure 5
Figure 5. Figure 5: Accuracy comparison over epoch on GSM8K. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: The trajectory of test accuracy during fine-tuning for [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Histogram of the metric values defined presented on x-axis for all fine-tuned weight matrices. [PITH_FULL_IMAGE:figures/full_fig_p061_7.png]
Figure 8
Figure 8. Figure 8: Left: top-32 singular values for each pre-trained weight matrices W♮ . Right: top-32 singular values for each difference matrices ∆W = Wfine-tuned − W♮ after full fine-tuning. The Index is ranked from the largest to the smallest singular values [PITH_FULL_IMAGE:figure…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won't Fail)

    cs.LG 2025-02 conditional novelty 7.0 of 10

    Under restricted strong convexity and smoothness, every stable point of LoRA training is either a low-rank global minimum or a high-rank, large-magnitude spurious minimum, and practical initialization and weight decay...

  2. Between Gradient and Natural Gradient: A Continuum of LoRA Initializations

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Gradient-projection, Adam-like, and K-FAC-whitened LoRA initializations are all special cases of one two-parameter family, and the best exponents are task-dependent and usually interior.

  3. Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Non-zero initialization of both LoRA matrices improves robustness to small learning rates and preserves fine-tuning accuracy, so LoRA need not start exactly from the pretrained model.

Reference graph

Works this paper leans on

67 extracted references · 52 canonical work pages · cited by 3 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    B., and Misiakiewicz, T

    Abbe, E., Adsera, E. B., and Misiakiewicz, T. The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networks. In Conference on Learning Theory, pp.\ 4782--4887. PMLR, 2022

    work page 2022

  3. [3]

    B., Gheissari, R., and Jagannath, A

    Arous, G. B., Gheissari, R., and Jagannath, A. Online stochastic gradient descent on non-convex losses from high-dimensional inference. Journal of Machine Learning Research, 22 0 (106): 0 1--51, 2021

    work page 2021

  4. [4]

    A., Suzuki, T., Wang, Z., Wu, D., and Yang, G

    Ba, J., Erdogdu, M. A., Suzuki, T., Wang, Z., Wu, D., and Yang, G. High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representation . In Advances in Neural Information Processing Systems, pp.\ 37932--37946, 2022

    work page 2022

  5. [5]

    On learning gaussian multi-index models with gradient flow

    Bietti, A., Bruna, J., and Pillaud-Vivien, L. On learning gaussian multi-index models with gradient flow. arXiv preprint arXiv:2310.19793, 2023

    arXiv 2023

  6. [6]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language Models are Few-Shot Learners . In Advances in Neural Information Processing Systems, pp.\ 1877--1901, 2020

    work page 1901

  7. [7]

    and Globerson, A

    Brutzkus, A. and Globerson, A. Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputs . In International Conference on Machine Learning, pp.\ 605--614. PMLR, 2017

    work page 2017

  8. [8]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating Large Language Models Trained on Code . arXiv preprint arXiv:2107.03374, 2021 a

    arXiv 2021

Show all 67 references
  1. [9]

    Spectral Methods for Data Science: A Statistical Perspective

    Chen, Y., Chi, Y., Fan, J., Ma, C., et al. Spectral Methods for Data Science: A Statistical Perspective . Foundations and Trends in Machine Learning , 14 0 (5): 0 566--806, 2021 b

    2021

  2. [10]

    Training Verifiers to Solve Math Word Problems

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training Verifiers to Solve Math Word Problems . arXiv preprint arXiv:2110.14168, 2021

    2021 arXiv

  3. [11]

    Asymptotics of feature learning in two-layer networks after one gradient-step

    Cui, H., Pesce, L., Dandi, Y., Krzakala, F., Lu, Y., Zdeborova, L., and Loureiro, B. Asymptotics of feature learning in two-layer networks after one gradient-step. In International Conference on Machine Learning, pp.\ 9662--9695. PMLR, 2024

    2024

  4. [12]

    Neural Networks can Learn Representations with Gradient Descent

    Damian, A., Lee, J., and Soltanolkotabi, M. Neural Networks can Learn Representations with Gradient Descent . In Conference on Learning Theory, pp.\ 5413--5452. PMLR, 2022

    2022

  5. [13]

    A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilities

    Dandi, Y., Pesce, L., Cui, H., Krzakala, F., Lu, Y., and Loureiro, B. A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilities . In The 28th International Conference on Artificial Intelligence and Statistics, 2025. URL h...

    2025

  6. [14]

    Dayi, A. K. and Chen, S. Gradient dynamics for low-rank fine-tuning beyond kernels. arXiv preprint arXiv:2411.15385, 2024

    2024 arXiv

  7. [15]

    A Validation Approach to Over-parameterized Matrix and Image Recovery

    Ding, L., Qin, Z., Jiang, L., Zhou, J., and Zhu, Z. A Validation Approach to Over-parameterized Matrix and Image Recovery . arXiv preprint arXiv:2209.10675, 2022

    2022 arXiv

  8. [16]

    Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping

    Dodge, J., Ilharco, G., Schwartz, R., Farhadi, A., Hajishirzi, H., and Smith, N. Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping . arXiv preprint arXiv:2002.06305, 2020

    2002 arXiv

  9. [17]

    Guarantees of a Preconditioned Subgradient Algorithm for Overparameterized Asymmetric Low-rank Matrix Recovery

    Giampouras, P., Cai, H., and Vidal, R. Guarantees of a Preconditioned Subgradient Algorithm for Overparameterized Asymmetric Low-rank Matrix Recovery . arXiv preprint arXiv:2410.16826, 2024

    2024 arXiv

  10. [18]

    Han, Z., Gao, C., Liu, J., Zhang, J., and Zhang, S. Q. Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey . Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=lIsCS8b6zj

    2024

  11. [19]

    LoRA+: Efficient Low Rank Adaptation of Large Models

    Hayou, S., Ghosh, N., and Yu, B. LoRA+: Efficient Low Rank Adaptation of Large Models . In International Conference on Machine Learning, pp.\ 17783--17806. PMLR, 2024

    2024

  12. [20]

    Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

    He, K., Zhang, X., Ren, S., and Sun, J. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification . In Proceedings of the IEEE international conference on computer vision, pp.\ 1026--1034, 2015

    2015

  13. [21]

    Measuring Massive Multitask Language Understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring Massive Multitask Language Understanding . In International Conference on Learning Representations, 2021

    2021

  14. [22]

    Horn, R. A. and Johnson, C. R. Matrix Analysis . Cambridge university press, 2012

    2012

  15. [23]

    Parameter-Efficient Transfer Learning for NLP

    Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-Efficient Transfer Learning for NLP . In International Conference on Machine Learning, pp.\ 2790--2799, 2019

    2019

  16. [24]

    J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al

    Hu, E. J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. LoRA: Low-Rank Adaptation of Large Language Models . In International Conference on Learning Representations, 2022

    2022

  17. [25]

    Neural Tangent Kernel: Convergence and Generalization in Neural Networks

    Jacot, A., Gabriel, F., and Hongler, C. Neural Tangent Kernel: Convergence and Generalization in Neural Networks . In Advances in neural information processing systems, 2018

    2018

  18. [26]

    D., and Ryu, E

    Jang, U., Lee, J. D., and Ryu, E. K. LoRA Training in the NTK Regime has No Spurious Local Minima . In International Conference on Machine Learning, pp.\ 21306--21328. PMLR, 2024

    2024

  19. [27]

    Preconditioning Matters: Fast Global Convergence of Non-convex Matrix Factorization via Scaled Gradient Descent

    Jia, X., Wang, H., Peng, J., Feng, X., and Meng, D. Preconditioning Matters: Fast Global Convergence of Non-convex Matrix Factorization via Scaled Gradient Descent . Advances in Neural Information Processing Systems, 36, 2024

    2024

  20. [28]

    Kim, J., Kim, J., and Ryu, E. K. LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won't Fail) . In International Conference on Machine Learning, 2025

    2025

  21. [29]

    J., Blankevoort, T., and Asano, Y

    Kopiczko, D. J., Blankevoort, T., and Asano, Y. M. VeRA: Vector-based Random Matrix Adaptation . In The Twelfth International Conference on Learning Representations, 2024

    2024

  22. [30]

    The mnist database of handwritten digits

    LeCun, Y. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/, 1998

    1998

  23. [31]

    D., Oko, K., Suzuki, T., and Wu, D

    Lee, J. D., Oko, K., Suzuki, T., and Wu, D. Neural network learns low-dimensional polynomials with SGD near the information-theoretic limit. Advances in Neural Information Processing Systems, 37: 0 58716--58756, 2024

    2024

  24. [32]

    On the Crucial Role of Initialization for Matrix Factorization

    Li, B., Zhang, L., Mokhtari, A., and He, N. On the Crucial Role of Initialization for Matrix Factorization . In The Twelfth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=YTEwJaBdh0

    2025

  25. [33]

    Algorithmic Regularization in Over-parameterized Matrix Sensing and Neural Networks with Quadratic Activations

    Li, Y., Ma, T., and Zhang, H. Algorithmic Regularization in Over-parameterized Matrix Sensing and Neural Networks with Quadratic Activations . In Conference On Learning Theory, pp.\ 2--47. PMLR, 2018

    2018

  26. [34]

    On the Optimization Landscape of Low Rank Adaptation Methods for Large Language Models

    Liu, X.-H., Du, Y., Wang, J., and Yu, Y. On the Optimization Landscape of Low Rank Adaptation Methods for Large Language Models . In International Conference on Learning Representations, 2025

    2025

  27. [35]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled Weight Decay Regularization . arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  28. [36]

    Beyond Procrustes: Balancing-Free Gradient Descent for Asymmetric Low-Rank Matrix Sensing

    Ma, C., Li, Y., and Chi, Y. Beyond Procrustes: Balancing-Free Gradient Descent for Asymmetric Low-Rank Matrix Sensing . IEEE Transactions on Signal Processing, 69: 0 867--877, 2021

    2021

  29. [37]

    A Kernel-Based View of Language Model Fine-Tuning

    Malladi, S., Wettig, A., Yu, D., Chen, D., and Arora, S. A Kernel-Based View of Language Model Fine-Tuning . In International Conference on Machine Learning, pp.\ 23610--23641. PMLR, 2023

    2023

  30. [38]

    PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models

    Meng, F., Wang, Z., and Zhang, M. PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models . In Advances in Neural Information Processing Systems, pp.\ 121038--121072, 2024

    2024

  31. [39]

    A., and Sepulchre, R

    Mishra, B., Apuroop, K. A., and Sepulchre, R. A riemannian geometry for low-rank matrix completion. arXiv preprint arXiv:1211.1550, 2012

    2012 arXiv

  32. [40]

    A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networks

    Moniri, B., Lee, D., Hassani, H., and Dobriban, E. A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networks . In International Conference on Machine Learning, pp.\ 36106--36159. PMLR, 2024

    2024

  33. [41]

    A Simplified Neuron Model as a Principal Component Analyzer

    Oja, E. A Simplified Neuron Model as a Principal Component Analyzer . Journal of Mathematical Biology, 15: 0 267--273, 1982

    1982

  34. [42]

    Pretrained Transformer Efficiently Learns Low-Dimensional Target Functions In-Context

    Oko, K., Song, Y., Suzuki, T., and Wu, D. Pretrained Transformer Efficiently Learns Low-Dimensional Target Functions In-Context . Advances in Neural Information Processing Systems, 37: 0 77316--77365, 2024

    2024

  35. [43]

    Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning

    Ponkshe, K., Singhal, R., Gorbunov, E., Tumanov, A., Horvath, S., and Vepakomma, P. Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning . arXiv preprint arXiv:2411.19557, 2024

    2024

  36. [44]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer . Journal of Machine Learning Research, 21 0 (140): 0 1--67, 2020

    2020

  37. [45]

    Implicit Balancing and Regularization: Generalization and Convergence Guarantees for Overparameterized Asymmetric Matrix Sensing

    Soltanolkotabi, M., St \"o ger, D., and Xie, C. Implicit Balancing and Regularization: Generalization and Convergence Guarantees for Overparameterized Asymmetric Matrix Sensing . In The Thirty Sixth Annual Conference on Learning Theory, pp.\ 5140--5142. PMLR, 2023

    2023

  38. [46]

    and Soltanolkotabi, M

    St \"o ger, D. and Soltanolkotabi, M. Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction . In Advances in Neural Information Processing Systems, pp.\ 23831--23843, 2021

    2021

  39. [47]

    Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford Alpaca: An Instruction-following LLaMA model . https://github.com/tatsu-lab/stanford_alpaca, 2023

    2023

  40. [48]

    LaMDA: Language Models for Dialog Applications

    Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. LaMDA: Language Models for Dialog Applications . arXiv preprint arXiv:2201.08239, 2022

    2022 arXiv

  41. [49]

    Accelerating Ill-Conditioned Low-Rank Matrix Estimation via Scaled Gradient Descent

    Tong, T., Ma, C., and Chi, Y. Accelerating Ill-Conditioned Low-Rank Matrix Estimation via Scaled Gradient Descent . Journal of Machine Learning Research, 22 0 (150): 0 1--63, 2021

    2021

  42. [50]

    LLaMA 2: Open Foundation and Fine-Tuned Chat Models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. LLaMA 2: Open Foundation and Fine-Tuned Chat Models . arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  43. [51]

    Introduction to the non-asymptotic analysis of random matrices

    Vershynin, R. Introduction to the non-asymptotic analysis of random matrices. arXiv preprint arXiv:1011.3027, 2010

    2010 arXiv

  44. [52]

    High-Dimensional Probability: An Introduction with Applications in Data Science , volume 47

    Vershynin, R. High-Dimensional Probability: An Introduction with Applications in Data Science , volume 47. Cambridge university press, 2018

    2018

  45. [53]

    Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding . In International Conference on Learning Representations, 2019

    2019

  46. [54]

    LoRA-GA: Low-Rank Adaptation with Gradient Approximation

    Wang, S., Yu, L., and Li, J. LoRA-GA: Low-Rank Adaptation with Gradient Approximation . Advances in Neural Information Processing Systems, 37: 0 54905--54931, 2024

    2024

  47. [55]

    LoRA-Pro: Are Low-Rank Adapters Properly Optimized? In The Twelfth International Conference on Learning Representations, 2025

    Wang, Z., Liang, J., He, R., Wang, Z., and Tan, T. LoRA-Pro: Are Low-Rank Adapters Properly Optimized? In The Twelfth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=gTwRMU3lJ5

    2025

  48. [56]

    Perturbation bounds in connection with singular value decomposition

    Wedin, P.- . Perturbation bounds in connection with singular value decomposition. BIT Numerical Mathematics, 12: 0 99--111, 1972

    1972

  49. [57]

    V., Zhou, D., et al

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models . Advances in Neural Information Processing Systems, 35: 0 24824--24837, 2022

    2022

  50. [58]

    Xiong, N., Ding, L., and Du, S. S. How Over-Parameterization Slows Down Gradient Descent in Matrix Sensing: The Curses of Symmetry and Initialization . In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=xGvPKAiOhq

    2024

  51. [59]

    The Power of Preconditioning in Overparameterized Low-Rank Matrix Sensing

    Xu, X., Shen, Y., Chi, Y., and Ma, C. The Power of Preconditioning in Overparameterized Low-Rank Matrix Sensing . In International Conference on Machine Learning, pp.\ 38611--38654. PMLR, 2023

    2023

  52. [60]

    E., Luo, J., Tarmoun, S., Mallada, E., and Vidal, R

    Xu, Z., Min, H., MacDonald, L. E., Luo, J., Tarmoun, S., Mallada, E., and Vidal, R. Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorization . In International Conference on Artificial Intelligence and Statistics, 2025

    2025

  53. [61]

    MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models

    Yu, L., Jiang, W., Shi, H., Jincheng, Y., Liu, Z., Zhang, Y., Kwok, J., Li, Z., Weller, A., and Liu, W. MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models . In The Twelfth International Conference on Learning Representations, 2024. URL https://openre...

    2024

  54. [62]

    and Lee, K

    Zeng, Y. and Lee, K. The Expressive Power of Low-Rank Adaptation . In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=likXVjmh3E

    2024

  55. [63]

    and Pilanci, M

    Zhang, F. and Pilanci, M. Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models . In International Conference on Machine Learning, pp.\ 59641--59669. PMLR, 2024

    2024

  56. [64]

    Zhang, G., Fattahi, S., and Zhang, R. Y. Preconditioned Gradient Descent for Overparameterized Nonconvex Burer--Monteiro Factorization with Global Optimality Certification . Journal of Machine Learning Research, 24 0 (163): 0 1--55, 2023

    2023

  57. [65]

    Zhang, J., Fattahi, S., and Zhang, R. Y. Preconditioned Gradient Descent for Over-Parameterized Nonconvex Matrix Factorization . Advances in Neural Information Processing Systems, 34: 0 5985--5996, 2021

    2021

  58. [66]

    Y., Fu, J., Chen, W., and Yue, X

    Zheng, T., Zhang, G., Shen, T., Liu, X., Lin, B. Y., Fu, J., Chen, W., and Yue, X. OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement . In Findings of the Association for Computational Linguistics ACL 2024, pp.\ 12834--12859, 2024

    2024

  59. [67]

    Imbalance-Regularized LoRA: A Plug-and-Play Method for Improving Fine-Tuning of Foundation Models

    Zhu, Z., Wu, Y., Gu, Q., and Cevher, V. Imbalance-Regularized LoRA: A Plug-and-Play Method for Improving Fine-Tuning of Foundation Models . In NeurIPS Workshop on Adaptive Foundation Models, 2024

    2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.