Sharp Generalization Bounds for Foundation Models with Asymmetric Randomized Low-Rank Adapters

Anastasis Kratsios; Aurelien Lucchi; Haitz S\'aez de Oc\'ariz Borde; Tin Sum Cheng

REVIEW 5 major objections 4 minor 53 references

Sharp Generalization Bounds for Foundation Models with Asymmetric Randomized Low-Rank Adapters

T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proves a single asymmetric LoRA run concentrates its generalization gap at rate $\tilde{O}(\sqrt{r}/\sqrt{N})$ and that this sample-size rate is optimal.

desk verdict New high-probability upper bound for random-factor LoRA, but the claimed matching lower bound is broken and the upper-bound proof has a fixable gap. read the letter →

arxiv 2506.14530 v1 pith:JHDGGT3F submitted 2025-06-17 stat.ML cs.AIcs.LGcs.NEmath.STstat.TH

Anastasis Kratsios , Tin Sum Cheng , Aurelien Lucchi , Haitz Sáez de Ocáriz Borde This is my paper

classification stat.MLcs.AIcs.LGcs.NEmath.STstat.TH MSC 68T0760B20

keywords Low-RankAdaptationasymmetricLoRAfrozenrandomfactorsgeneralizationgapsamplecomplexitymatrixtheoryLipschitzwidthsanti-concentration

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that the generalization gap of a single LoRA fine-tuning run—not just the average over many random initializations—concentrates with high probability around its mean, at sample complexity $\tilde{O}(\sqrt{r}/\sqrt{N})$ for rank $r$ adapters trained on $N$ samples. The main upper bound (Theorem 1) is worst-case: it holds uniformly over data-generating distributions and training algorithms, for the random class of LoRA-perturbed networks with one frozen factor. A matching lower bound (Theorem 2) constructs a distribution on which no bound of this type can converge faster in the sample size $N$, so the rate is presented as optimal. The practical stake is that LoRA in production is a single run with a fixed random draw, and the paper argues that single-run behavior tracks the average closely enough to make asymmetric randomized LoRA reliable.

What carries the argument

The central object is the random parameter-to-LoRA map $\operatorname{LoRA}(\omega,(A^{(t)})_t)$ sending admissible trainable factors to a realized network, with a probabilistic bound on its Lipschitz constant $L_{\theta_{\mathrm{pre}}}^{\operatorname{LoRA}} \le 2^{c_2 T}(M\nu\sqrt{2r\log(2W/\epsilon)}+\|\theta_{\mathrm{pre}}\|_\infty)^{c_2 T}$, obtained from Lipschitz-width estimates for neural parameterizations and Gaussian concentration. That Lipschitz control yields the covering-number estimate $\mathcal{N}(\epsilon,\mathcal{F})\le ((2R+2R_0)^{cT+1}/\epsilon)^q$ with high probability, and Dudley's entropy integral converts the covering number into the Rademacher-complexity term in Theorem 1. The lower bound runs on a different machinery: the identity construction $A^{(t)}=B^{(t)\dagger}(\bar{I}_{d_{t+1},d_t}-W^{(t)})$ makes the perturbed network compute the identity map on the support of the data distribution, and Littlewood-Offord anti-concentration for the sample mean $\frac{1}{N}\sum_{n=1}^N X_n$ produces the required separation between true and empirical risk.

What would settle it

In the setting of Theorem 2 take $d=1$, any hidden width $W$, and rank $r<W$; a Gaussian $W\times r$ matrix $B$ cannot satisfy $BB^{\dagger}=I_W$, so for $d_{t+1}=W$ the identity map $W^{(t)}+B^{(t)}A^{(t)}=I$ cannot be realized by $A^{(t)}=B^{(t)\dagger}(\bar{I}-W^{(t)})$. A direct check is to sample many Gaussian $B$, compute that $A$, and measure $\|W^{(t)}+B^{(t)}A^{(t)}-I\|_{\mathrm{op}}$: the norm will be strictly positive with probability one, which means the constructed network is not the identity and the lower-bound mechanism of Theorem 2 does not apply to this admissible parameter regime.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that for the random class $\mathcal{F}$ of networks $f(x) = (W^{(T+1)}+B^{(T+1)}A^{(T+1)})x^{(T+1)}+b^{(T+1)}$, $x^{(t+1)}=\sigma((W^{(t)}+B^{(t)}A^{(t)})x^{(t)}+b^{(t)})$, with frozen random factors $B^{(t)}$ and trainable $A^{(t)}$, the worst-case gap $G=\sup_{f\in\mathcal{F}}|R(f)-R_N(f)|$ satisfies, with probability at least $1-\delta$, $G \le 4\min(1, 6\sqrt{q}\sqrt{A}/\sqrt{N}) + \sqrt{8\log(2/(1-\sqrt{1-\delta}))/N}$, where $q=r(W(T-1)+d+D)$ counts the trainable LoRA parameters. That is a $\tilde{O}(\sqrt{r}/\sqrt{N})$ sample complexity with high probability for a single fine-tuning run. Theorem 2 supplies a classification problem, with scalar input and outputs and i.i.d. Gaussian random factors, on which the realized LoRA network is the identity map and the empirical-mean fluctuations force $G>1/N$ with probability at least $(1-\delta/2)(1-\Theta(1/\sqrt{N}))$; the paper interprets this as a matching lower bound on the sample-size rate.

Load-bearing premise

The load-bearing premise is that every random frozen factor $B^{(t)}$ is full row-rank, so its Moore-Penrose pseudo-inverse is a right-inverse and the construction $A^{(t)}=B^{(t)\dagger}(\bar{I}-W^{(t)})$ really realizes the identity map; the paper's own setting assumes $1\le r<W$ and a positive minimum-width margin $\eta_\star=\min_t(\sqrt{d_{t+1}}-\sqrt{r})>0$, which requires $r<d_{t+1}$, and the lower-bound argument collapses if that row-rank premise fails.

Editorial extensions

If this is right

One fine-tuning run with frozen random LoRA factors has, with high probability, the same qualitative generalization behavior as the average over runs; practitioners can treat a single experiment's gap as tracking the bound.
Increasing the LoRA rank $r$ widens the worst-case generalization gap as $\sqrt{r}$, so the expressiveness gained by a larger rank comes with a predictable sample cost.
Freezing one factor keeps the parameter-to-LoRA map's derivative bounded in the frozen factor's largest singular value; training both factors would make that derivative grow linearly and enlarge covering numbers and generalization bounds.
The rate in $N$, namely $1/\sqrt{N}$, is presented as optimal for this random LoRA class: no uniform high-probability bound can converge faster in sample size without extra assumptions.
The upper bound is symmetric with respect to which factor is randomized, but the lower bound specifically requires randomizing the extractor $B$; randomizing the projector $A$ does not yield a matching lower bound.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's own identification of the optimality condition leaves open whether the upper-bound rate is also necessary when the random factor is the projector $A$; a natural next step is an anti-concentration lower bound for that setting using machinery other than smallest singular values.
Because the upper bound is uniform over training algorithms, it cannot distinguish between gradient-based fine-tuning and other procedures; the bound may be loose for realistic optimizers, and the paper's experiments show the $\sqrt{r}$ trend is visible but slower than the worst-case rate.
The lower-bound construction depends on the realized network being close to the identity map; in real models the pre-trained weights are far from identity, so the matching-lower-bound scenario is a stress test rather than a model of typical LoRA fine-tuning.
A reader could test the $r$-dependence directly on a vision-language model: with a fixed parameter budget, increasing $r$ while freezing one factor should increase the generalization gap sublinearly, consistent with the paper's bound but not with the worst-case exponent.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The genuinely new thing here is Theorem 1: a high-probability, single-run generalization bound for asymmetric LoRA with frozen random factors, giving an O~(sqrt(r)/sqrt(N)) rate. That's a real step beyond the average-case bound in Zhu et al., and the mix of Lipschitz-width machinery with random matrix theory is a fresh way to attack it. The CLIP experiments are a reasonable sanity check, though secondary. The soft spots are serious, and they are in the proofs. Lemma 4's Step 2 uses |sum_k B_ik A_kj| <= M |sum_k B_ik|, which fails when the A entries have mixed signs. A correct bound would go through the row norm or a Gaussian tail conditional on A, and the final rate probably survives up to log factors; the theorem may be true, but the proof as written does not establish it. This needs a genuine fix, not a typo patch. Theorem 2 is in worse shape. The construction requires B(t) B(t)^+ = I, which for a d_{t+1} x r matrix only holds if r >= d_{t+1}. That directly contradicts Setting 1 (1 <= r < W) and the eta* > 0 condition in Eq. (8). So the identity map is not realized, and the lower bound argument collapses. Even if one repaired the construction by moving to r >= W, the stated result only shows the gap exceeds 1/N with high probability, which is not a matching Omega(1/sqrt(N)) lower bound; the anti-concentration step could be retuned to get 1/sqrt(N), but that is not what the theorem states. There are also smaller slips: the definition xi = 2X - 2 has mean -1, not 0, in the proof. So where does that leave the paper? The upper-bound question is worth pursuing, and the authors have a plausible route to a correct result there. The lower bound, as advertised, is not established and the matching claim should be withdrawn or substantially reworked. This is not a desk-reject: the topic matters, the upper-bound idea is original, and the flaws are identifiable and possibly fixable. But no one should cite the lower bound, and the upper bound needs careful refereeing. Recommendation: send to peer review, with referees asked to focus on Lemma 4 and on whether the lower bound can be repaired or should be removed.

Referee Report

5 major / 4 minor

Summary. The paper studies asymmetric LoRA, in which one low-rank factor is random and frozen while the other is trained, and asks two questions: whether the typical generalization gap concentrates around its expectation for a single random draw, and what the optimal sample efficiency is. The main positive claim (Theorem 1) is a high-probability worst-case bound on the generalization gap of order Otilde(sqrt(r)/sqrt(N)); the matching-optimality claim (Theorem 2) is a lower bound constructed by making the LoRA-perturbed network realize the identity map and then applying an anti-concentration inequality. The proofs combine Lipschitz-regularity estimates for parameter-to-network maps, covering-number/Dudley arguments, Gordon's theorem, and a Littlewood-Offord-type concentration inequality. The paper also reports CLIP fine-tuning experiments showing that the generalization gap grows qualitatively with LoRA rank.

Significance. The question addressed is well motivated: existing LoRA generalization bounds are average-case, and a single-run tail guarantee would be a meaningful practical refinement. The upper-bound strategy, using Lipschitz widths and random-matrix estimates, is creative and potentially reusable. The experimental section is honest about the worst-case nature of the theory and reports only a qualitative rank trend. However, both main theoretical results have serious proof defects. Theorem 1 relies on an invalid inequality in the derivation of the key Lipschitz/covering estimate, and Theorem 2 is built on an unsatisfiable assumption and an internally contradictory construction. As submitted, the paper does not establish its advertised upper bound as proved, and it does not establish any matching lower bound at the claimed rate. The paper does not ship machine-checked proofs or reproducible derivations, and the experimental evidence does not substitute for correctness of the main theorems.

major comments (5)

[Section 3.2, Assumption 1 and Example 1] Assumption 1 postulates a distribution P_X supported on [0,1] with mean 1/2 and variance 1/2, but no such distribution exists: for any random variable taking values in [0,1], the variance is at most 1/4. Example 1 is also incorrect: a fair Bernoulli trial on {0,1} has variance 1/4, and the stated random variable 2X-2 takes values -2 and 0, not the Rademacher values +1 and -1. Thus the hypotheses of Theorem 2 are vacuous, and the lower bound as stated holds for no data-generating distribution.
[Appendix C, Step 1, Eqs. (25)-(28)] The identity construction defines A(t)=B(t)^dagger*(bar-I_{d_{t+1},d_t}-W(t)) and then uses B(t)B(t)^dagger=I_{d_{t+1}} to conclude that the LoRA-perturbed layer is the identity. This step requires B(t) to have a right inverse, which for a d_{t+1}xr matrix exists only if r>=d_{t+1}. In Setting 1 the hidden layers have d_{t+1}=W with r<W, so the premise fails; Eq. (28) does not follow, the conclusion f(x)=x is not established, and Eq. (30) cannot be used. The advertised lower bound therefore collapses.
[Section 3.2, Eq. (8) and Theorem 2 statement] The minimum-width condition eta* = min_{t=1,...,T}(sqrt(d_{t+1})-sqrt(r))>0 is unsatisfiable in the setting of Theorem 2, which sets d_{T+1}=1 while r>=1; hence sqrt(d_{T+1})-sqrt(r)<=0. Even if Assumption 1 were repaired, Theorem 2 would still be vacuous because no r>=1 can satisfy Eq. (8) when the output dimension is 1.
[Appendix C, Step 3 and conclusion of Theorem 2] The proof concludes P(G>1/N) >= 1 - Theta(1/sqrt(N)). This is not a matching lower bound for the claimed rate 1/sqrt(N): the threshold 1/N is far smaller than 1/sqrt(N) for large N, so the event G>1/N is compatible with an upper bound of order 1/sqrt(N). A genuine matching lower bound would require showing that G exceeds c/sqrt(N) for a positive constant c with non-vanishing probability. As written, Theorem 2 does not demonstrate optimality in sample size.
[Appendix B.2.1, Lemma 4, Step 2] The proof uses the inequality |sum_k B_{ik} A_{kj}| <= M |sum_k B_{ik}|, which is invalid when the entries of A have mixed signs. The correct bound is M sum_k |B_{ik}|, whose high-probability scale is O(M nu r), not M nu sqrt(r log(2W/epsilon)). This quantity R is propagated through Lemma 8 and Lemma 9 into Theorem 1, so the proof of the main upper bound as written is invalid. A repaired argument might preserve the Otilde(sqrt(r)/sqrt(N)) rate up to additional logarithmic factors, but that is not the proof contained in the manuscript.

minor comments (4)

[Section B.2.2, Lemmas 5-6 and 8] The symbol p is used both for the number of pre-trained parameters in Section 3 and for the dimension of the parameter ball in Lemmas 5-6, while Lemma 8 switches the covering exponent to q; please disambiguate these uses.
[Lemma 9 and proof of Theorem 1] Lemma 9 states a conditional probability at least 1-epsilon, but its proof concludes a conditional probability at least 1-2epsilon; the delta conversion in the proof of Theorem 1 uses the former. These failure probabilities should be reconciled.
[Theorem 1 statement] The displayed bound 4 min(1, sqrt(q) 6 sqrt(A)/sqrt(N)) is notationally hard to read; the proof works with 6 sqrt(qA)/sqrt(N), and the two forms should be written consistently.
[Example 1 and Appendix C, Eq. (40)] The centered Rademacher variable for a Bernoulli trial with X in {0,1} is 2X-1, not 2X-2; Eq. (40) also defines xi_n=2(X_n-1), which does not have zero mean under any distribution with mean 1/2. The centering should be corrected throughout the anti-concentration argument.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: both bounds are derived from external approximation-theoretic and random-matrix results, and the only author-overlapping citation is contextual rather than load-bearing.

full rationale

The upper bound (Theorem 1) is a standard covering-number/Rademacher derivation: Lemmas 4-9 chain Petrova-Wojtaszczyk Lipschitz width estimates, Gordon's Gaussian concentration, Lorentz et al. covering estimates, and Bartlett-Mendelson/Wainwright Rademacher bounds; no bound is defined through the generalization gap it estimates. The lower bound (Theorem 2) is an explicit adversarial construction, choosing A(t) = B(t)^dagger(\bar I - W(t)) and then applying Rudelson-Vershynin anti-concentration; it is not a fitted parameter renamed as a prediction. The only author-overlapping citation, Zhu et al. [2024], supplies the empirical/intuitive framing of asymmetric LoRA and a separate information-theoretic comparison; neither theorem's proof invokes it. A correctness concern exists in the proof of Theorem 2: the identity realization uses B(t)B(t)^+ = I, which would require r >= d_{t+1}, contradicting Setting 1's r < W and Eq. (8)'s eta* > 0; this is a validity flaw, not circularity, so it does not increase the circularity score.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities; its contributions are bounds built on standard tools. The free parameters M, nu, and eta are user-specified or proof-construction parameters, not fitted to data. The axioms are all cited background results plus the paper-specific Assumption 1 for the lower bound.

free parameters (3)

M (maximum admissible LoRA weight magnitude) = user-specified, not fitted
M bounds the entries of the trainable A factors; it appears in the covering number bound (R = M nu sqrt(2r log(2W/epsilon))) and in the lower bound's admissibility event. The paper does not estimate M from data.
nu (random initialization scale of B) = user-specified, not fitted
B_ij ~ N(0, nu^2); nu enters the Lipschitz constant bound in Lemma 4 and the covering number bound. Not fitted.
eta (anti-concentration slack in lower bound) = chosen in (0, eta*)
The lower bound's event A requires 0 < eta < eta* = min_t sqrt(d_{t+1}) - sqrt(r) to keep the smallest singular values positive; it is a proof-construction parameter tied to the failure probability delta.

assumptions (5)

standard math Lipschitz-width estimates of Petrova-Wojtaszczyk (2023a) on the local Lipschitz regularity of the MLP parameter-to-function map (Eq. (14)).
Invoked in Lemma 4 to bound the Lipschitz constant of the LoRA parameterization map; taken as a black box from approximation theory.
standard math Gordon's comparison theorem for Gaussian matrices (Vershynin 2010, Exercise 7.3.4), used to lower-bound smallest singular values of random B.
Used in the lower bound (Eq. (34)-(38)) to ensure the pseudo-inverse construction is well-posed with high probability.
standard math Rudelson-Vershynin (2008) anti-concentration / Littlewood-Offord inequalities for sums of i.i.d. variables (Eq. (41)).
Used in Step 3 of the lower bound to lower-bound the deviation of the sample mean from the expectation.
standard math Classical covering number estimate for balls in l_infinity (Lorentz et al. 1996) and Dudley's entropy integral (van der Vaart and Wellner 1996), and Rademacher complexity bounds (Bartlett and Mendelson 2002).
Backbone of the upper-bound proof.
ad hoc to paper Assumption 1: the sampling distribution P_X supported on [0,1] has mean and variance 1/2.
A sufficient condition introduced for the lower bound; the paper shows it is non-vacuous via Example 1 (fair Bernoulli trial). It limits the lower bound to this specific family of data distributions.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Sharp Generalization Bounds for Foundation Models with Asymmetric Randomized Low-Rank Adapters." pith.science (2026). https://pith.science/paper/JHDGGT3F

@misc{pith2026250614530,
  author       = {Pith},
  title        = {Pith review of: Sharp Generalization Bounds for Foundation Models with Asymmetric Randomized Low-Rank Adapters},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JHDGGT3F}},
  note         = {Machine review of arXiv:2506.14530}
}

abstract

Low-Rank Adaptation (LoRA) has emerged as a widely adopted parameter-efficient fine-tuning (PEFT) technique for foundation models. Recent work has highlighted an inherent asymmetry in the initialization of LoRA's low-rank factors, which has been present since its inception and was presumably derived experimentally. This paper focuses on providing a comprehensive theoretical characterization of asymmetric LoRA with frozen random factors. First, while existing research provides upper-bound generalization guarantees based on averages over multiple experiments, the behaviour of a single fine-tuning run with specific random factors remains an open question. We address this by investigating the concentration of the typical LoRA generalization gap around its mean. Our main upper bound reveals a sample complexity of $\tilde{\mathcal{O}}\left(\frac{\sqrt{r}}{\sqrt{N}}\right)$ with high probability for rank $r$ LoRAs trained on $N$ samples. Additionally, we also determine the fundamental limits in terms of sample efficiency, establishing a matching lower bound of $\mathcal{O}\left(\frac{1}{\sqrt{N}}\right)$. By more closely reflecting the practical scenario of a single fine-tuning run, our findings offer crucial insights into the reliability and practicality of asymmetric LoRA.

Figures

Figures reproduced from arXiv: 2506.14530 by the authors.

**Figure 2.** Generalization gap on different datasets in downstream classification tasks. [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 25 canonical work pages

[1]

Qwen technical report, 2023

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

arXiv 2023
[2]

Rademacher and gaussian complexities: Risk bounds and structural results

Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3 0 (Nov): 0 463--482, 2002

2002
[3]

Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huaj...

arXiv 2025
[4]

Qlora: Efficient finetuning of quantized llms

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. ArXiv, abs/2305.14314, 2023. URL https://api.semanticscholar.org/CorpusID:258841328

arXiv 2023
[5]

Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models, 2022

Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Hai-Tao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juanzi Li, and Maosong Sun. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models, 2022. U...

arXiv 2022
[6]

Efficient adaptation of large vision transformer via adapter re-composing

Wei Dong, Dawei Yan, Zhijun Lin, and Peng Wang. Efficient adaptation of large vision transformer via adapter re-composing. In Advances in Neural Information Processing Systems, 2023

work page 2023
[7]

The lottery ticket hypothesis: Finding sparse, trainable neural networks, 2018

Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks, 2018

2018
[8]

Robert M. French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3: 0 128--135, 1999. URL https://api.semanticscholar.org/CorpusID:2691726

work page 1999

Show all 53 references

[9]

An image is worth one word: Personalizing text-to-image generation using textual inversion

Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618, 2022

2022 arXiv
[10]

Mahoney, and Kurt Keutzer

Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W. Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference, 2021

2021
[11]

Majorization of gaussian processes and geometric applications

Yehoram Gordon. Majorization of gaussian processes and geometric applications. Probability theory and related fields, 91 0 (2): 0 251--267, 1992

1992
[12]

Xing, and Yoon Kim

Han Guo, Philip Greengard, Eric P. Xing, and Yoon Kim. Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning, 2024

2024
[13]

Svdiff: Compact parameter space for diffusion fine-tuning

Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning. arXiv preprint arXiv:2303.11305, 2023

2023 arXiv
[14]

Parameter-efficient fine-tuning for large models: A comprehensive survey, 2024

Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey, 2024. URL https://arxiv.org/abs/2403.14608

2024 arXiv
[15]

Towards a unified view of parameter-efficient transfer learning

Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=0RDcd5Axok

2022
[16]

Lora: Low-rank adaptation of large language models

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1 0 (2): 0 3, 2022

2022
[17]

Lee, and Ernest K

Uijeong Jang, Jason D. Lee, and Ernest K. Ryu. Lo RA training in the NTK regime has no spurious local minima. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=s1sdx6vNsU

2024
[18]

Nola: Networks as linear combination of low rank random basis, 2023

Soroush Abbasi Koohpayegani, KL Navaneet, Parsa Nooralinejad, Soheil Kolouri, and Hamed Pirsiavash. Nola: Networks as linear combination of low rank random basis, 2023

2023
[19]

Kopiczko, Tijmen Blankevoort, and Yuki M

Dawid J. Kopiczko, Tijmen Blankevoort, and Yuki M. Asano. Vera: Vector-based random matrix adaptation, 2024

2024
[20]

Fast randomized low-rank adaptation of pre-trained language models with pac regularization

Zijian Lei, Dong Qian, and William Cheung. Fast randomized low-rank adaptation of pre-trained language models with pac regularization. In Findings of the Association for Computational Linguistics ACL 2024, pages 5236--5249, 2024

2024
[21]

Graphadapter: Tuning vision-language models with dual knowledge graph

Xin Li, Dongze Lian, Zhihe Lu, Jiawang Bai, Zhibo Chen, and Xinchao Wang. Graphadapter: Tuning vision-language models with dual knowledge graph. In Advances in Neural Information Processing Systems, 2023

2023
[22]

PAC -tuning: Fine-tuning pre-trained language models with PAC -driven perturbed gradient descent

Guangliang Liu, Zhiyu Xue, Xitong Zhang, Kristen Johnson, and Rongrong Wang. PAC -tuning: Fine-tuning pre-trained language models with PAC -driven perturbed gradient descent. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https://openrevi...

2023
[23]

Black, Adrian Weller, and Bernhard Sch \"o lkopf

Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, Yandong Wen, Michael J. Black, Adrian Weller, and Bernhard Sch \"o lkopf. Parameter-efficient orthogonal finetuning via butterfly factorization. In ICLR, 2024

2024
[24]

Lorentz, Manfred v

George G. Lorentz, Manfred v. Golitschek, and Yuly Makovoz. Constructive approximation, volume 304 of Grundlehren der mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences]. Springer-Verlag, Berlin, 1996. ISBN 3-540-57028-4. doi:10.1007/978-3-642-60932...

1996 doi
[25]

Sanae Lotfi, Marc Anton Finzi, Yilun Kuang, Tim G. J. Rudner, Micah Goldblum, and Andrew Gordon Wilson. Non-vacuous generalization bounds for large language models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Feli...

2024
[26]

Score distillation via reparametrized DDIM

Artem Lukoianov, Haitz S \'a ez de Oc \'a riz Borde, Kristjan Greenewald, Vitor Campagnolo Guizilini, Timur Bagautdinov, Vincent Sitzmann, and Justin Solomon. Score distillation via reparametrized DDIM . In The Thirty-eighth Annual Conference on Neural Information Processing S...

2024
[27]

A kernel-based view of language model fine-tuning

Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A kernel-based view of language model fine-tuning. In International Conference on Machine Learning, pages 23610--23641. PMLR, 2023

2023
[28]

Gpt-4 technical report, 2023

OpenAI. Gpt-4 technical report, 2023. Available at https://openai.com/research/gpt-4

2023
[29]

Bronstein

Pantelis Papageorgiou, Haitz S \'a ez de Oc \'a riz Borde, Anastasis Kratsios, and Michael M. Bronstein. Graph low-rank adapters of high regularity for graph neural networks and graph transformers. In First Workshop on Scalable Optimization for Efficient and Adaptive Foundatio...

2025
[30]

Limitations on approximation by deep and shallow neural networks

Guergana Petrova and Przemys aw Wojtaszczyk. Limitations on approximation by deep and shallow neural networks. J. Mach. Learn. Res., 24: 0 Paper No. [353], 38, 2023 a . ISSN 1532-4435,1533-7928

2023
[31]

Lipschitz widths

Guergana Petrova and Przemys aw Wojtaszczyk. Lipschitz widths. Constructive Approximation, 57 0 (2): 0 759--805, 2023 b

2023
[32]

A dapter H ub: A framework for adapting transformers

Jonas Pfeiffer, Andreas R \"u ckl \'e , Clifton Poth, Aishwarya Kamath, Ivan Vuli \'c , Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. A dapter H ub: A framework for adapting transformers. In Qun Liu and David Schlangen, editors, Proceedings of the 2020 Conference on Empi...

2020 doi
[33]

Qwen2.5 technical report, 2025

Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

2025 arXiv
[34]

What’s hidden in a randomly weighted neural network? In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. What’s hidden in a randomly weighted neural network? In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, June 2020. doi:10.1109/cvpr42600.2020.01191. U...

2020
[35]

Pivotal tuning for latent-based editing of real images

Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or. Pivotal tuning for latent-based editing of real images. ACM Transactions on Graphics (TOG), 42 0 (1): 0 1--13, 2022

2022
[36]

High-resolution image synthesis with latent diffusion models, 2022

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2022. URL https://arxiv.org/abs/2112.10752

2022 arXiv
[37]

The littlewood--offord problem and invertibility of random matrices

Mark Rudelson and Roman Vershynin. The littlewood--offord problem and invertibility of random matrices. Advances in Mathematics, 218 0 (2): 0 600--633, 2008

2008
[38]

Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208.12242, 2022

2022 arXiv
[39]

A sharp inverse littlewood-offord theorem

Terence Tao and Van Vu. A sharp inverse littlewood-offord theorem. Random Structures & Algorithms, 37 0 (4): 0 525--539, 2010

2010
[40]

Inverse littlewood-offord theorems and the condition number of random discrete matrices

Terence Tao and Van H Vu. Inverse littlewood-offord theorems and the condition number of random discrete matrices. Annals of Mathematics, pages 595--632, 2009

2009
[41]

Llama: Open and efficient foundation language models

Hugo Touvron et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

2023 arXiv
[42]

van der Vaart and Jon A

Aad W. van der Vaart and Jon A. Wellner. Weak convergence and empirical processes. Springer Series in Statistics. Springer-Verlag, New York, 1996. ISBN 0-387-94640-3. doi:10.1007/978-1-4757-2545-2. URL https://doi.org/10.1007/978-1-4757-2545-2. With applications to statistics

1996 doi
[43]

Introduction to the non-asymptotic analysis of random matrices

Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices. arXiv preprint arXiv:1011.3027, 2010

2010 arXiv
[44]

Wainwright

Martin J. Wainwright. High-dimensional statistics, volume 48 of Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge, 2019. ISBN 978-1-108-49802-9. doi:10.1017/9781108627771. URL https://doi.org/10.1017/9781108627771. A non-asymp...

2019 doi
[45]

Tina: Tiny reasoning models via lora, 2025

Shangshang Wang, Julian Asilis, Ömer Faruk Akgül, Enes Burak Bilgin, Ollie Liu, and Willie Neiswanger. Tina: Tiny reasoning models via lora, 2025. URL https://arxiv.org/abs/2504.15777

2025 arXiv
[46]

Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation

Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openr...

2023
[47]

Compeft: Compression for communicating parameter efficient updates via sparsification and quantization

Prateek Yadav, Leshem Choshen, Colin Raffel, and Mohit Bansal. Compeft: Compression for communicating parameter efficient updates via sparsification and quantization. arXiv preprint arXiv:2311.13171, 2023

2023 arXiv
[48]

Yu, Jianlong Chang, Lingbo Liu, Qi Tian, and Chang Wen Chen

Bruce X.B. Yu, Jianlong Chang, Lingbo Liu, Qi Tian, and Chang Wen Chen. Towards a unified view on visual parameter-efficient transfer learning. arXiv preprint arXiv:2210.00788, 2022

2022 arXiv
[49]

Low-rank few-shot adaptation of vision-language models

Maxime Zanella and Ismail Ben Ayed. Low-rank few-shot adaptation of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1593--1603, 2024

2024
[50]

The expressive power of low-rank adaptation

Yuchen Zeng and Kangwook Lee. The expressive power of low-rank adaptation. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=likXVjmh3E

2024
[51]

Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning, 2023 a

Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning, 2023 a

2023
[52]

Adalora: Adaptive budget allocation for parameter-efficient fine-tuning, 2023 b

Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning, 2023 b

2023
[53]

Asymmetry in low-rank adapters of foundation models

Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz S \'a ez De Oc \'a riz Borde, Rickard Br \"u el Gabrielsson, Leshem Choshen, Marzyeh Ghassemi, Mikhail Yurochkin, and Justin Solomon. Asymmetry in low-rank adapters of foundation models. In Proceedings of the 41st Interna...

2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Qwen technical report, 2023

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

arXiv 2023

[2] [2]

Rademacher and gaussian complexities: Risk bounds and structural results

Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3 0 (Nov): 0 463--482, 2002

2002

[3] [3]

Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huaj...

arXiv 2025

[4] [4]

Qlora: Efficient finetuning of quantized llms

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. ArXiv, abs/2305.14314, 2023. URL https://api.semanticscholar.org/CorpusID:258841328

arXiv 2023

[5] [5]

Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models, 2022

Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Hai-Tao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juanzi Li, and Maosong Sun. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models, 2022. U...

arXiv 2022

[6] [6]

Efficient adaptation of large vision transformer via adapter re-composing

Wei Dong, Dawei Yan, Zhijun Lin, and Peng Wang. Efficient adaptation of large vision transformer via adapter re-composing. In Advances in Neural Information Processing Systems, 2023

work page 2023

[7] [7]

The lottery ticket hypothesis: Finding sparse, trainable neural networks, 2018

Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks, 2018

2018

[8] [8]

Robert M. French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3: 0 128--135, 1999. URL https://api.semanticscholar.org/CorpusID:2691726

work page 1999