Pith. sign in

REVIEW 3 major objections 5 minor 62 references

HRP: High-Rank Preheating for Superior LoRA Initialization

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Low-rank fine-tuning is decided by its starting direction, and a short high-rank preheat can supply that direction.

desk verdict A cheap and empirically solid LoRA initialization trick, let down by an incorrect central theorem. read the letter →

arxiv 2502.07739 v3 pith:SKQQHBJZ submitted 2025-02-11 cs.LG

Yuzhu Chen , Yingjie Wang , Shi Fu , Li Shen , Yongcheng Jing , Xinmei Tian , Dacheng Tao This is my paper · ORCID
classification cs.LG MSC 68T0715A23
keywords low-rankadaptationLoRAinitializationsingularvaluedecompositionmatrixfactorizationparameter-efficientfine-tuninghigh-rankpreheatinggradientflowasymmetric
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's target is a claim about Low-Rank Adaptation (LoRA): the performance of the fine-tuned model is largely settled by the direction in which the low-rank adapters are started, not just by how long or how well they are trained. The paper proves that the standard random start leaves LoRA with results no better than a random low-rank approximation in expectation, while starting along the main singular vectors of the true weight update would give exponential convergence to the best rank-$r$ approximation. Since the true update is unknown, the paper proposes High-Rank Preheating (HRP): run a short, higher-rank LoRA pass, extract the top singular vectors of what that pass learned, and use them as the initialization for the final low-rank fine-tune. It proves this initialization beats random initialization in expectation and improves as the preheating rank grows. If correct, this gives a cheap, storage-neutral way to close much of the gap between LoRA and full fine-tuning.

What carries the argument

The central object is the High-Rank Preheating (HRP) initialization: before the main fine-tuning, run $\mathrm{hrp\_step}$ steps of Asymmetric LoRA at rank $\mathrm{hrp\_rank}>r$ with a random orthogonal right sketch and a zero left matrix, then initialize the low-rank LoRA's $A$ matrix with the top $r$ right singular vectors of the preheated product $BA^\top$. The mechanism is that one preheating step turns the random basis into $\frac{\eta\alpha^2}{r^2}U\,\mathrm{diag}(I_{\mathrm{hrp\_rank}},0)\,U^\top M$, a projection of the target $M$ onto a random $\mathrm{hrp\_rank}$-dimensional subspace, so the leading singular vectors of the preheated update inherit the target's leading directions. The proof machinery is completed by closed-form gradient-flow solutions for Asymmetric LoRA, $X_t=(I-e^{-\eta Z_0 t})M$ (and the transpose form), plus the Eckart-Young characterization of the best rank-$r$ approximation.

What would settle it

Run HRP on a synthetic matrix-factorization problem whose target $M$ has nearly equal singular values (a flat spectrum) and compare expected final loss against random initialization; if HRP shows no improvement, or if in a real fine-tuning run the top $r$ right singular vectors of the preheated $BA^\top$ have no greater cosine similarity to the final $\Delta W$ than random directions do, the paper's central mechanism is falsified.

Watch

Extended reading notes

Core claim

Working in the matrix-factorization idealization of LoRA, where the adapters $A$ and $B$ are trained by gradient flow on $L_t=\frac{1}{2}\|\frac{\alpha}{r}B_tA_t^\top - M\|_F^2$ with $M=W^{\mathrm{target}}-W^{\mathrm{init}}$, the paper establishes three facts. First, under the standard zero-plus-random initialization, Asymmetric LoRA (one adapter frozen) has expected converged loss bounded below by the loss of a random rank-$r$ approximation, and classic LoRA either inherits that limitation early in training or can be trapped on a subspace orthogonal to the target's singular vectors. Second, if the initialization contains the top $r$ right (or left) singular vectors of $M$, both variants converge exponentially to the best rank-$r$ approximation. Third, because $M$ is unknown in practice, High-Rank Preheating (HRP) runs a few steps of higher-rank Asymmetric LoRA and takes the top $r$ right singular vectors of the resulting $BA^\top$ as the low-rank starting point; Theorem 6 bounds the expected converged loss by $\sum_{i=r+1}^{\mathrm{hrp\_rank}}\sigma_i(M)^2 + \frac{a-\mathrm{hrp\_rank}}{2a}\sum_i\sigma_i(M)^2$, which is lower than the random-initialization floor and decreases as the preheating rank grows.

Load-bearing premise

The method's advantage rests on the premise that a brief high-rank preheating pass on the real task loss makes the leading singular directions of the preheated update point at the leading singular directions of the true target update; if the real loss is not a matrix-factorization problem, or those directions shift during the main fine-tuning, the benefit can disappear.

Editorial extensions

If this is right

  • Random initialization leaves Asymmetric LoRA no better than a random low-rank approximation in expectation, which formalizes a concrete reason LoRA underperforms full fine-tuning.
  • Changing only the initial direction, with no architectural or optimizer change, restores exponential convergence to the best rank-$r$ approximation in the matrix-factorization setting.
  • The expected final loss bound for HRP shrinks as $\mathrm{hrp\_rank}$ grows, so spending a slightly higher preheating rank buys a provably better starting point.
  • HRP keeps the checkpoint and memory profile of ordinary low-rank LoRA because $B_0A_0^\top=0$ and $W_{\mathrm{init}}=W_{\mathrm{pre}}$, unlike SVD-based initializations that need extra storage.
  • On NLU and math-reasoning NLG benchmarks, HRP outperforms other initialization strategies and most LoRA variants and lands close to full-parameter fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The preheating idea could be recycled during training: if the target subspace drifts, re-preheat from the current checkpoint instead of only at initialization, turning HRP into a periodic re-anchoring routine.
  • The proof suggests a direct diagnostic: track the cosine similarity between the top $r$ right singular vectors of the preheated $BA^\top$ and the eventual $\Delta W$; a drop in alignment would predict when HRP's advantage erodes.
  • If the alignment mechanism transfers beyond the Frobenius factorization loss, HRP could also inform rank allocation, by preheating once at high rank and pruning to decide which directions deserve the low-rank budget.
  • Because preheating needs only a few steps and a small batch, HRP could be run on a tiny proxy dataset when the full training set is unavailable, using the proxy's singular directions as initialization.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies the role of initialization in LoRA fine-tuning. In a matrix-factorization benchmark, it claims that random initialization makes Asymmetric and classic LoRA converge to random low-rank approximations (Theorems 1–3), while a 'wise' initialization using the target's top singular vectors achieves exponential convergence to the best rank-r approximation (Theorems 4–5). Since the target is unknown, the authors propose High-Rank Preheating (HRP): run a few steps of high-rank Asymmetric LoRA, extract the top r right singular vectors of the preheated update, and use them to initialize the main low-rank LoRA. The central theoretical assertion, stated in the abstract and Section 4, is that HRP provably yields better fine-tuned results than random initialization in expectation, with the improvement growing with the preheating rank (Theorem 6). The empirical section evaluates HRP on GLUE with T5-base and on GSM8K/MATH with three LLMs, reporting gains over LoRA, PiSSA, LoRA-GA, and several LoRA variants, at negligible additional time and no extra GPU memory.

Significance. If the theoretical claim is established, the paper would contribute a useful, initialization-only mechanism for improving LoRA, with a transparent analysis in a matrix-factorization proxy and a practical procedure that is easy to implement on top of any LoRA pipeline. The empirical results are internally consistent: HRP improves over the compared initializations across most tasks, the ablations in Appendix C.1 show robustness in hrp_rank and hrp_step, and the time/memory tables support the claimed overhead. The main weakness is that the proof of the headline theorem contains a dimension error and an incorrect application of Theorem 1, so the paper's core theoretical guarantee is not currently established; the corrected bound does not imply the advertised improvement without additional assumptions. The practical contribution remains credible, but the central claim needs repair.

major comments (3)
  1. [Section 4, Theorem 6 and Appendix A.7] The stated bound in Theorem 6 does not follow from the proof in Appendix A.7. The proof derives ||X∞ - M||² ≤ ||M - Rhat{X}_∞||² + Σ_{i=r+1}^{hrp_rank} σ_i(M)², then halves and takes expectations. Since Rhat{X}_∞ is the limit of LSI Asymmetric LoRA with a random orthogonal matrix Rhat{B}_0 ∈ R^{b×hrp_rank}, Theorem 1 gives E||M - Rhat{X}_∞||² = (b - hrp_rank)/b · Σ σ_i(M)², not (a - hrp_rank)/a · Σ σ_i(M)². Therefore the correct expected-loss bound is (b - hrp_rank)/(2b) · Σ σ_i² + 1/2 Σ_{i=r+1}^{hrp_rank} σ_i². The theorem as printed uses the dimension a instead of b and drops the factor 1/2 from the singular-value tail. This is not cosmetic: for a=2, b=100, p=2, r=1, hrp_rank=2, M=diag(1,0.5), the stated bound is 0.25 while the proof's actual derivation evaluates to about 0.7375, so the printed statement is violated by the paper's own derivation.
  2. [Section 4, Theorem 6 vs Theorem 1] Even after correcting the dimension error, the resulting upper bound does not imply the abstract's claim that HRP achieves better results than random initialization in expectation. The random-initialization lower bound for RSI in Theorem 1 is (a - r)/(2a) · S. The corrected HRP upper bound is (b - hrp_rank)/(2b) · S + 1/2 Σ_{i=r+1}^{hrp_rank} σ_i². For architectures with b >> a, the first term is close to S/2, while the lower bound can be much smaller; e.g., with a=10, r=1, b=100, hrp_rank=2, any S, the upper bound is approximately 0.49S + 0.5σ_2², which exceeds the lower bound 0.45S whenever σ_2 > 0. Thus, without an additional spectral-decay assumption that makes the tail Σ_{i=r+1}^{hrp_rank} σ_i² small and also keeps (a-r)/(2a) sufficiently large relative to (b-hrp_rank)/(2b), the proof does not establish the advertised dominance over random initialization.
  3. [Algorithm 1 and Theorem 12 (Appendix A.7)] The dimension bookkeeping in the HRP specification is inconsistent and obscures the proof. Algorithm 1's step 2 writes Rhat{A}_0 = O_{a×r} while Rhat{B}_0 has hrp_rank columns; for an hrp_rank-rank preheating block, Rhat{A}_0 must be O_{a×hrp_rank}. In Theorem 12, the fine-tuning initialization is written as Rhat{A}_0 = Σ_{i=1}^{hrp_rank} v_i(Rhat{X}_t) eᵢ,hrp_rankRtop, but the fine-tuning LoRA has rank r, so the sum and the basis vector should run over r, not hrp_rank. The theorem statement in Section 4 also reads 'For any hrp_step > r', which should presumably be hrp_rank > r. These typos are load-bearing because they make it difficult to verify that the proof's objects match the stated algorithm.
minor comments (5)
  1. [Abstract and Section 1] The sentence claiming 'we prove that HRP makes LoRA achieve better fine-tuned results than random initialization in expectation' is stronger than what Theorem 6 currently establishes; if the theorem is repaired, the wording should be made conditional on the assumptions actually used.
  2. [Section 4, informal derivation after Algorithm 1] In the one-step update formula, Rhat{A}_1 = ηα/r MᵀRhat{B}_0 should read Rhat{A}_1 = ηα²/r² MᵀRhat{B}_0 (or the convention for the gradient should be stated consistently); the displayed expression drops a factor α/r.
  3. [Theorem 6 and Theorem 1] Theorem 6 writes the singular-value sum to max(a,b), while Theorem 1 sums to min(a,b); since singular values beyond min(a,b) vanish, this is harmless but should be made uniform to avoid confusion.
  4. [Figure 1 caption] The 'Target init' curve is not defined in the caption; it refers to the initialization of Theorem 4, and the caption should say so.
  5. [Section 4, Algorithm reference] The text says 'HRP can be formulated as Algorithm 4' but the algorithm is labeled Algorithm 1; internal cross-references should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found; the central derivation is self-contained, and the flagged Theorem 6 issue is a proof-correctness gap rather than an input-output identity.

full rationale

HRP's derivation chain is self-contained rather than circular. Section 3 builds the gradient-flow dynamics for LoRA and proves Theorems 1-5 directly from the flow equations in Appendices A.2-A.6, without assuming the later results. Theorem 6 (Appendix A.7) does not reduce to its inputs by construction: it constructs the preheated matrix from the same objective M used in the matrix-factorization benchmark, extracts its top-r singular vectors, and bounds the resulting fine-tuning error through Lemmas 1-3 and Theorem 1. The target M is the objective of the benchmark, not an input to Algorithm 1, and the comparison baseline (random rank-r approximation) is an independent lower bound from Theorem 1. No parameter is fitted to a subset of data and then presented as a prediction; HRP's initialization is computed from the preheated training dynamics and evaluated on held-out tasks. The paper's few self-references in the bibliography, e.g., [11] and [12], are background citations and are not load-bearing for the theoretical claim. The only passage that warrants explicit flagging is Appendix A.7: the expectation step invokes Theorem 1 with the coefficient (a - hrp_rank)/(2a), although the LSI preheating stage has a frozen sketch in the b-dimensional space, so the stated bound appears dimensionally inconsistent. That is a proof-correctness risk, not a circular definitional reduction, and therefore does not raise the circularity score.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

Everything the central claim rests on that the reader didn't pay for upstream: the matrix factorization model, the low-effective-rank assumption, and the alignment assumption between preheated directions and target directions.

free parameters (3)
  • hrp_rank = 128 (NLU), 256 (NLG)
    Preheating rank chosen by hand for experiments; ablation (Table 5) shows performance varies with it.
  • hrp_step = 100 (NLU), 200 (NLG)
    Number of preheating steps; ablation (Table 6) shows HRP active/inactive matters more than the exact value.
  • hrp_bs = 16 (NLU), 8 (NLG)
    Batch size for preheating, chosen to reduce GPU memory.
assumptions (5)
  • domain assumption Gradient flow approximates gradient descent for LoRA fine-tuning
    Section 3.1 replaces discrete updates with continuous flow, valid for small learning rates.
  • domain assumption The fine-tuning objective can be modeled as Frobenius-norm matrix factorization toward a fixed target M = W_target - W_pre
    Section 3.2, Eq. (2); real task losses are not Frobenius and W_target is not fixed during training.
  • domain assumption The target delta W has low effective rank (rank-dimension ratio small)
    Invoked after Theorem 1 to argue the lower bound is high; cited [46,47].
  • domain assumption Gradient norms are bounded and the gradient map is Lipschitz in Frobenius norm
    Assumed for Theorem 3 to bound the divergence between classic and Asymmetric LoRA; justified by bounded-domain backward propagation.
  • ad hoc to paper The preheated high-rank update's top singular vectors align with the target's top singular vectors
    Core premise of HRP in Section 4; not proven for non-Frobenius losses and short preheating.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HRP: High-Rank Preheating for Superior LoRA Initialization." pith.science (2026). https://pith.science/paper/SKQQHBJZ

@misc{pith2026250207739,
  author       = {Pith},
  title        = {Pith review of: HRP: High-Rank Preheating for Superior LoRA Initialization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SKQQHBJZ}},
  note         = {Machine review of arXiv:2502.07739}
}
abstract

This paper studies the crucial impact of initialization in Low-Rank Adaptation (LoRA). Through theoretical analysis, we demonstrate that the fine-tuned result of LoRA is highly sensitive to initialization, which is likely to lead suboptimal low-rank results. While this issue can be mitigated by adjusting the initial direction towards the main singular vectors of the target $\Delta W$, which is, however, typically unknown in real-world scenarios. To approximate this initial direction, we propose High-Rank Preheating (HRP), which first trains LoRA with a higher preheating rank for a few steps, then uses the main singular vectors of the derived $BA^\top$ as initialization for the main fine-tuning process. With only a modification in the initial direction, we prove that HRP makes LoRA achieve better fine-tuned results than random initialization in expectation, and the enhancement grows with the preheating rank. We validate our theoretical findings through extensive experiments in various models and tasks, where HRP significantly enhances LoRA's effectiveness and outperforms other initialization strategies and other LoRA variants.

Figures

Figures reproduced from arXiv: 2502.07739 by the authors.

Figure 1
Figure 1. Loss curves for matrix factorization targeting [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Loss curves for fine-tuning meta-llama/Llama-3.2-1B-Instruct on the MetaMathQA. [PITH_FULL_IMAGE:figures/full_fig_p024_2.png] view at source ↗
Figure 3
Figure 3. Loss curves for fine-tuning Qwen/Qwen3-1.7B on the MetaMathQA. [PITH_FULL_IMAGE:figures/full_fig_p024_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Loss curves for fine-tuning google/gemma-2-2b-it on the MetaMathQA. [PITH_FULL_IMAGE:figures/full_fig_p024_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 22 canonical work pages

  1. [35]

    Initialization using update approximation is a silver bullet for extremely efficient low-rank fine-tuning

    Kaustubh Ponkshe, Raghav Singhal, Eduard Gorbunov, Alexey Tumanov, Samuel Horvath, and Praneeth Vepakomma. Initialization using update approximation is a silver bullet for extremely efficient low-rank fine-tuning. arXiv preprint arXiv:2411.19557, 2024

    arXiv 2024

  2. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [2]

    Lora-xs: Low-rank adaptation with extremely small number of parameters

    Klaudia Bałazy, Mohammadreza Banaei, Karl Aberer, and Jacek Tabor. Lora-xs: Low-rank adaptation with extremely small number of parameters. arXiv preprint arXiv:2405.17604, 2024

    arXiv 2024

  4. [3]

    Lora learns less and forgets less

    Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, et al. Lora learns less and forgets less. arXiv preprint arXiv:2405.09673, 2024

    arXiv 2024

  5. [4]

    On the opportunities and risks of foundation models

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021

    arXiv 2021

  6. [5]

    Olora: Orthonormal low-rank adaptation of large language models

    Kerim Büyükakyüz. Olora: Orthonormal low-rank adaptation of large language models. arXiv preprint arXiv:2406.01775, 2024

    arXiv 2024

  7. [6]

    Nonconvex optimization meets low-rank matrix factorization: An overview

    Yuejie Chi, Yue M Lu, and Yuxin Chen. Nonconvex optimization meets low-rank matrix factorization: An overview. IEEE Transactions on Signal Processing, 67(20):5239–5269, 2019

    2019

  8. [7]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

Show all 62 references
  1. [8]

    Metainit: Initializing learning by learning to initialize

    Yann N Dauphin and Samuel Schoenholz. Metainit: Initializing learning by learning to initialize. Advances in Neural Information Processing Systems, 32, 2019

    2019

  2. [9]

    The approximation of one matrix by another of lower rank

    Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika, 1(3):211–218, 1936

    1936

  3. [10]

    Low rank adaptation for stable domain adaptation of vision transformers

    N Filatov and M Kindulov. Low rank adaptation for stable domain adaptation of vision transformers. Optical Memory and Neural Networks, 32(Suppl 2):S277–S283, 2023

    2023

  4. [11]

    On championing foundation models: From explainability to interpretability

    Shi Fu, Yuzhu Chen, Yingjie Wang, and Dacheng Tao. On championing foundation models: From explainability to interpretability. arXiv preprint arXiv:2410.11444, 2024

    2024 arXiv

  5. [12]

    Towards theoretical understandings of self-consuming generative models

    Shi Fu, Sen Zhang, Yingjie Wang, Xinmei Tian, and Dacheng Tao. Towards theoretical understandings of self-consuming generative models. arXiv preprint arXiv:2402.11778, 2024

    2024 arXiv

  6. [13]

    Understanding the difficulty of training deep feedfor- ward neural networks

    Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256. JMLR Workshop and Conference Proceedings, 2010

    2010

  7. [14]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  8. [15]

    The impact of initialization on lora finetuning dynamics

    Soufiane Hayou, Nikhil Ghosh, and Bin Yu. The impact of initialization on lora finetuning dynamics. arXiv preprint arXiv:2406.08447, 2024

    2024 arXiv

  9. [16]

    Lora+: Efficient low rank adaptation of large models

    Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402.12354, 2024

    2024 arXiv

  10. [17]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015

    2015

  11. [18]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  12. [19]

    The gronwall inequality

    Ralph Howard. The gronwall inequality. lecture notes, 1998

    1998

  13. [20]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 10

    2021 arXiv

  14. [21]

    Advlora: Adversarial low-rank adaptation of vision- language models

    Yuheng Ji, Yue Liu, Zhicheng Zhang, Zhao Zhang, Yuting Zhao, Gang Zhou, Xingwei Zhang, Xinwang Liu, and Xiaolong Zheng. Advlora: Adversarial low-rank adaptation of vision- language models. arXiv preprint arXiv:2404.13425, 2024

    2024 arXiv

  15. [22]

    A rank stabilization scaling factor for fine-tuning with lora

    Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:2312.03732, 2023

    2023 arXiv

  16. [23]

    Vera: Vector-based random matrix adaptation

    Dawid J Kopiczko, Tijmen Blankevoort, and Yuki M Asano. Vera: Vector-based random matrix adaptation. arXiv preprint arXiv:2310.11454, 2023

    2023 arXiv

  17. [24]

    Fine- tuning can distort pretrained features and underperform out-of-distribution

    Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint arXiv:2202.10054, 2022

    2022 arXiv

  18. [25]

    On the crucial role of initialization for matrix factorization

    Bingcong Li, Liang Zhang, Aryan Mokhtari, and Niao He. On the crucial role of initialization for matrix factorization. arXiv preprint arXiv:2410.18965, 2024

    2024 arXiv

  19. [26]

    Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning

    Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950–1965, 2022

    1950

  20. [27]

    Dora: Weight-decomposed low-rank adaptation

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353, 2024

    2024 arXiv

  21. [28]

    Decoupled weight decay regularization, 2019

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019

    2019

  22. [29]

    Randomized asymmetric chain of lora: The first meaningful theoretical framework for low-rank adaptation

    Grigory Malinovsky, Umberto Michieli, Hasan Abed Al Kader Hammoud, Taha Ceritli, Hayder Elesedy, Mete Ozay, and Peter Richtárik. Randomized asymmetric chain of lora: The first meaningful theoretical framework for low-rank adaptation. arXiv preprint arXiv:2410.08305, 2024

    arXiv

  23. [30]

    A survey on lora of large language models

    Yuren Mao, Yuhang Ge, Yijiang Fan, Wenyi Xu, Yu Mi, Zhonghao Hu, and Yunjun Gao. A survey on lora of large language models. Frontiers of Computer Science, 19(7):197605, 2025

    2025

  24. [31]

    Pissa: Principal singular values and singular vectors adaptation of large language models

    Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint arXiv:2404.02948, 2024

    2024 arXiv

  25. [32]

    On the explicit role of initialization on the convergence and implicit bias of overparametrized linear networks

    Hancheng Min, Salma Tarmoun, René Vidal, and Enrique Mallada. On the explicit role of initialization on the convergence and implicit bias of overparametrized linear networks. In International Conference on Machine Learning, pages 7760–7768. PMLR, 2021

    2021

  26. [33]

    All you need is a good init

    Dmytro Mishkin and Jiri Matas. All you need is a good init. arXiv preprint arXiv:1511.06422, 2015

    2015 arXiv

  27. [34]

    Global convergence and stability of stochastic gradient descent

    Vivak Patel, Shushu Zhang, and Bowen Tian. Global convergence and stability of stochastic gradient descent. Advances in Neural Information Processing Systems, 35:36014–36025, 2022

    2022

  28. [36]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020

    2020

  29. [37]

    Tied-lora: Enhacing parameter efficiency of lora with weight tying

    Adithya Renduchintala, Tugrul Konuk, and Oleksii Kuchaiev. Tied-lora: Enhacing parameter efficiency of lora with weight tying. arXiv preprint arXiv:2311.09578, 2023

    2023 arXiv

  30. [38]

    Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

    Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013

    2013 arXiv

  31. [39]

    Sharelora: Parameter efficient and robust large language model fine-tuning via shared low-rank adaptation

    Yurun Song, Junchen Zhao, Ian G Harris, and Sangeetha Abdu Jyothi. Sharelora: Parameter efficient and robust large language model fine-tuning via shared low-rank adaptation. arXiv preprint arXiv:2406.10785, 2024

    2024 arXiv

  32. [40]

    Understanding the dynamics of gradient flow in overparameterized linear models

    Salma Tarmoun, Guilherme Franca, Benjamin D Haeffele, and Rene Vidal. Understanding the dynamics of gradient flow in overparameterized linear models. In International Conference on Machine Learning, pages 10153–10161. PMLR, 2021

    2021

  33. [41]

    Gemma, 2024

    Gemma Team. Gemma, 2024

    2024

  34. [42]

    Qwen3, April 2025

    Qwen Team. Qwen3, April 2025. 11

    2025

  35. [43]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  36. [44]

    Rsvdpack: An implementation of randomized algorithms for computing the singular value, interpolative, and cur decompositions of matrices on multi-core and gpu architectures

    Sergey V oronin and Per-Gunnar Martinsson. Rsvdpack: An implementation of randomized algorithms for computing the singular value, interpolative, and cur decompositions of matrices on multi-core and gpu architectures. arXiv preprint arXiv:1502.05366, 2015

    2015 arXiv

  37. [45]

    Glue: A multi-task benchmark and analysis platform for natural language under- standing

    Alex Wang. Glue: A multi-task benchmark and analysis platform for natural language under- standing. arXiv preprint arXiv:1804.07461, 2018

    2018 arXiv

  38. [46]

    Pufferfish: Communication- efficient models at no extra cost

    Hongyi Wang, Saurabh Agarwal, and Dimitris Papailiopoulos. Pufferfish: Communication- efficient models at no extra cost. Proceedings of Machine Learning and Systems, 3:365–386, 2021

    2021

  39. [47]

    Cuttlefish: Low-rank model training without all the tuning

    Hongyi Wang, Saurabh Agarwal, Yoshiki Tanaka, Eric Xing, Dimitris Papailiopoulos, et al. Cuttlefish: Low-rank model training without all the tuning. Proceedings of Machine Learning and Systems, 5:578–605, 2023

    2023

  40. [48]

    Lora-ga: Low-rank adaptation with gradient approxima- tion

    Shaowen Wang, Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approxima- tion. arXiv preprint arXiv:2407.05000, 2024

    2024 arXiv

  41. [49]

    Asymmetric matrix sensing by gradient descent with small random initialization

    Johan S Wind. Asymmetric matrix sensing by gradient descent with small random initialization. arXiv preprint arXiv:2309.01796, 2023

    2023 arXiv

  42. [50]

    Chain of lora: Efficient fine-tuning of language models via residual learning

    Wenhan Xia, Chengwei Qin, and Elad Hazan. Chain of lora: Efficient fine-tuning of language models via residual learning. arXiv preprint arXiv:2401.04151, 2024

    2024 arXiv

  43. [51]

    Towards theoretically inspired neural initialization optimization

    Yibo Yang, Hong Wang, Haobo Yuan, and Zhouchen Lin. Towards theoretically inspired neural initialization optimization. Advances in Neural Information Processing Systems, 35:18983– 18995, 2022

    2022

  44. [52]

    Global convergence of gradient descent for asymmetric low-rank matrix factorization

    Tian Ye and Simon S Du. Global convergence of gradient descent for asymmetric low-rank matrix factorization. Advances in Neural Information Processing Systems, 34:1429–1439, 2021

    2021

  45. [53]

    Metamath: Bootstrap your own mathematical questions for large language models

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284, 2023

    2023 arXiv

  46. [54]

    The expressive power of low-rank adaptation

    Yuchen Zeng and Kangwook Lee. The expressive power of low-rank adaptation. arXiv preprint arXiv:2310.17513, 2023

    2023 arXiv

  47. [55]

    Adalora: Adaptive budget allocation for parameter- efficient fine-tuning

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter- efficient fine-tuning. arXiv preprint arXiv:2303.10512, 2023

    2023 arXiv

  48. [56]

    Gradinit: Learning to initialize neural networks for stable and efficient training

    Chen Zhu, Renkun Ni, Zheng Xu, Kezhi Kong, W Ronny Huang, and Tom Goldstein. Gradinit: Learning to initialize neural networks for stable and efficient training. Advances in Neural Information Processing Systems, 34:16410–16422, 2021

    2021

  49. [57]

    Asymmetry in low-rank adapters of foundation models

    Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz Sáez de Ocáriz Borde, Rickard Brüel Gabrielsson, Leshem Choshen, Marzyeh Ghassemi, Mikhail Yurochkin, and Justin Solomon. Asymmetry in low-rank adapters of foundation models. arXiv preprint arXiv:2402.16842, 2024. A Proof...

    2024 arXiv

  50. [58]

    Yt and Zt are symmetric and semi-positive

  51. [59]

    d Trace(Yt) = d Trace(Zt)

  52. [60]

    For zero+random initialization, Xt = Ob×a

  53. [61]

    For zero+random LSI, Yt = Oa×a

  54. [62]

    tilde” variables as those from Asymmetric LoRA, while the without “tilde

    For zero+random RSI, Zt = Ob×b. Besides, we denote ∥ · ∥F as the Frobenius norm, UX ΣX VX as the SVD decomposition of X where ΣX = diag(σ1(X), σ2(X) · · ·, ) with σ1(X) ≥ σ2(X) ≥ · · ·. Lemma 1. Gradient flow of matrix factorization 2 with Asymmetric LoRA in LSI ( ηA = η, ηB =...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.