Pith. sign in

REVIEW 4 major objections 5 minor 78 references

DeCAF: Decentralized Consensus-And-Factorization for Low-Rank Adaptation of Foundation Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that decentralized LoRA fine-tuning can converge at the same $O(1/\sqrt{T})$ rate as decentralized SGD, by deriving a new smoothness constant for the low-rank factors and resolving consensus interference with…

desk verdict The DLoRA rate improvement is plausible, but DeCAF's advertised convergence is not proven for the algorithm that actually runs; worth reviewing, not desk-rejecting. read the letter →

arxiv 2505.21382 v1 pith:NX6PDENT submitted 2025-05-27 cs.LG

Nastaran Saadati , Zhanhong Jiang , Joshua R. Waite , Shreyan Ganguly , Aditya Balu , Chinmay Hegde , Soumik Sarkar This is my paper
classification cs.LG MSC 68T0790C2668W15
keywords low-rankadaptationdecentralizedlearningconvergenceratetruncatedSVDconsensusinterferencefine-tuningfoundationmodelsnon-convexoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Decentralized LoRA fine-tuning currently trails standard decentralized SGD in theory, with a convergence rate of $O(1/T^{1/3})$ rather than $O(1/\sqrt{T})$. This paper claims that the gap can be closed: under a bounded-singular-value assumption and a Lipschitz-transfer assumption on the low-rank decomposition, the decentralized LoRA update converges at $O(1/\sqrt{T})$, with an error bound that explicitly shrinks as the LoRA rank $r$ grows. To address the "consensus interference" that appears when agents average the two low-rank factors separately, the paper introduces DeCAF, which averages the products $B_i A_i$ and re-factorizes with truncated SVD; it proves the SVD error is bounded and that DeCAF retains the same $O(1/\sqrt{T})$ rate. A reader should care because this would put decentralized fine-tuning of large vision-language and language models on the same theoretical footing as standard decentralized SGD while keeping LoRA's communication savings.

What carries the argument

The load-bearing object is the bilinear LoRA structure $W = W_0 + (\eta/r)BA$, which lets the chain rule convert smoothness in $W$ into smoothness in $(A,B)$ once the factors are bounded; this conversion is what the new constant $\hat{L} = \eta(2LC\sqrt{rc}+G)/r$ encodes. DeCAF's second engine is the truncated SVD operator $T(\cdot)=U_r\Sigma_r V_r^\top$, which takes the consensus product $\tilde{B}_i\tilde{A}_i = \sum_{j\in\mathcal{N}_b(i)}\pi_{ij}B_j A_j$ and returns two rank-$r$ factors so that agents can continue updating $A$ and $B$ separately. The supporting machinery is the mixing matrix $\Pi$ and its spectral gap $1-\sqrt{\rho}$, which controls how fast consensus spreads; the paper uses it both in the standard descent proof and in Theorem 2's bound on the difference between individual consensus and product consensus. These pieces together are what let the paper claim a rank-dependent, $O(1/\sqrt{T})$ rate for both DLoRA and DeCAF.

What would settle it

On a one-layer LoRA problem with $W_0 = 0$, run the paper's SGD trajectory and compute the ratio $\|\nabla f(w)-\nabla f(w')\|_F / \|w-w'\|_F$ between consecutive iterates; if at any step it exceeds the paper's $\hat{L}$ with the $C$ from Assumption 3.1, then Lemma 1 fails. A simpler check is whether $\|W-W'\|_F / (\|A-A'\|_F + \|B-B'\|_F)$ stays bounded along the trajectory; an unbounded ratio would directly falsify Assumption 3.1.

Watch

Extended reading notes

Core claim

The paper's central claim is that smoothness of the loss in the full weight matrix $W$ does not automatically carry over to the LoRA factors $(A,B)$, but it can be restored with a new smoothness constant $\hat{L} = \eta(2LC\sqrt{rc}+G)/r$ under the paper's Assumptions 1-3. With that constant, Theorem 1 gives a non-asymptotic bound on the average squared gradient, and Corollary 1 reduces it to $O(\sqrt{1/(NT)} + \sqrt{1/(NTr)})$, improving Dec-LoRA's $O(1/T^{1/3})$ rate and displaying for the first time how rank $r$ enters the error. For DeCAF, the consensus step operates on products $\sum_j \pi_{ij} B_j A_j$ and then applies the TSVD operator $T(\cdot)=U_r\Sigma_r V_r^\top$; Proposition 1 bounds the factorization error by $\sqrt{(|\mathcal{N}_b(i)|-1)r}\,c^2$, and Theorem 2 bounds the consensus gap between DLoRA and DeCAF by terms that vanish as $T$ grows and shrink as $r$ grows. The paper concludes that DeCAF converges at the same $O(1/\sqrt{T})$ rate as DLoRA, and that freezing one factor (DLoRA-FA) preserves the rate with a larger constant involving $k/r$.

Load-bearing premise

Everything rests on Assumption 3.1, which requires a single finite constant $C$ to bound changes in the frozen-weight output $W$ by changes in the two low-rank factors $A$ and $B$; the paper explicitly verifies $C$ only when one factor is frozen and otherwise assumes a sufficiently large $C$ exists.

Editorial extensions

If this is right

  • Decentralized LoRA fine-tuning converges at $O(1/\sqrt{T})$ for smooth non-convex objectives, matching decentralized SGD and improving on the earlier $O(1/T^{1/3})$ rate.
  • The convergence error bound shrinks with both agent count $N$ and LoRA rank $r$, giving a concrete theoretical reason that higher ranks train faster and that linear speedup in the number of agents holds.
  • DeCAF's TSVD-based product consensus has a bounded approximation error, and the consensus gap between DLoRA and DeCAF decays with $T$ and shrinks as $r$ grows, so DeCAF inherits the $O(1/\sqrt{T})$ rate.
  • Freezing one low-rank factor (DLoRA-FA) still converges at $O(1/\sqrt{T})$ but with a larger error constant proportional to $k/r$, quantifying the tradeoff between communication savings and accuracy.
  • Experiments on CLIP and LLAMA2-7B show DeCAF matching or exceeding federated averaging and beating local fine-tuning, including on ring-topology non-IID settings where individual-consensus DLoRA diverges.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because the consensus gap in Theorem 2 shrinks as the rank grows, existing methods that average $A$ and $B$ separately gain a rate certificate at large rank, even though they never run product consensus.
  • Beyond the paper: the smoothness-transfer argument depends only on the bilinear product $BA$, so the same $O(1/\sqrt{T})$ proof should apply to other low-rank parameter-efficient fine-tuning schemes, not just LoRA.
  • Beyond the paper: the TSVD step costs $O(rdk)$ per iteration, so at large rank a randomized or sketched SVD that preserves the top-$r$ subspace would be the natural way to keep the convergence guarantee while cutting the runtime bottleneck.
  • Beyond the paper: Corollary 1 implies a practical rank-selection guideline—pick $r$ large enough that the $1/\sqrt{NTr}$ term is below the target error, but no larger than communication and SVD budgets allow—which the paper does not state as a rule.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies decentralized fine-tuning of foundation models with LoRA. It analyzes DLoRA, in which agents run local SGD on the low-rank factors (A, B) and then perform separate consensus averaging on A and B, and it proposes DeCAF, in which agents first average the products B_i A_i and then use truncated SVD (TSVD) to factor the averaged product back into individual factors. The theoretical part introduces a new smoothness constant for LoRA objectives (Lemma 1), proves an O(1/sqrt(T)) non-convex convergence bound for DLoRA (Theorem 1), proves a constant bound on the TSVD reconstruction error (Proposition 1), and bounds a consensus discrepancy E_T between DLoRA and DeCAF (Theorem 2). The authors conclude that DeCAF also converges at O(1/sqrt(T)). Experiments compare DLoRA, DeCAF, and a frozen-A variant DLoRA-FA with local training and FedAvg on CLIP few-shot and LLaMA2-7B tasks under IID and non-IID splits.

Significance. The DLoRA part, if correct, is a useful theoretical step: replacing Dec-LoRA's O(1/T^{1/3}) bound with O(1/sqrt(T)), together with an explicit dependence on the LoRA rank, is a natural contribution to decentralized LoRA theory. The DLoRA-FA variant and the experiments spanning two model families and several topologies add practical value. However, the paper's flagship claim, the DeCAF matching-rate guarantee, is not established by the supplied proofs. Theorem 2 analyzes a TSVD-free auxiliary recursion rather than Algorithm 1, and Proposition 1's error bound is never incorporated into a descent or consensus argument. Because the title and abstract promise this guarantee, the significance of the manuscript in its present form is substantially reduced.

major comments (4)
  1. [§4, Theorem 2 and SM A.3.8] The claimed O(1/sqrt(T)) rate for DeCAF is not proven for the algorithm that actually runs. SM A.3.8 states that the proof of Theorem 2 assumes 'there is no TSVD in DeCAF' and replaces DeCAF by the auxiliary recursion (Eq. 35) on Delta W. Algorithm 1, by contrast, applies the TSVD operator at every communication round and continues SGD on the reconstructed pair (A_{t+1/2}, B_{t+1/2}). The quantity E_T in Theorem 2 bounds a difference between two consensus operations on DLoRA iterates; it is not a bound on E[||grad f(vbar_t, theta_0)||^2] for the TSVD-perturbed (A,B) iterates. The sentence 'Combining the convergence rate of DLoRA yields that the convergence rate of DeCAF still remains O(1/T^1/2)' is therefore an unproved assertion, not a consequence of Theorem 2.
  2. [Proposition 1 and §4] Proposition 1 gives only the T-independent bound ||Btilde_i^t Atilde_i^t - B_{t+1/2} A_{t+1/2}||_F <= sqrt((|N_b(i)|-1)r) c^2. This term is never inserted into the descent/consensus analysis leading to Theorem 1. A fixed-rank projection generally acts as a persistent bias rather than as vanishing noise; adding a constant perturbation of size O(c^2 sqrt(r)) to the update would typically leave an additive floor in the averaged gradient norm. The paper supplies no argument, such as a contraction-plus-error recursion or a step-size schedule that absorbs the bias, showing that the TSVD error is harmless. This is the load-bearing gap between the DLoRA theorem and the DeCAF claim.
  3. [Theorem 2] The stated bound does not imply that consensus differences vanish as the rank increases. The leading term is 2 alpha G eta c sqrt(r) (1 - sqrt(rho))^{-1}, which grows with sqrt(r), not decays. The subsequent displayed simplification writes E_T <~ sqrt(N) sqrt(r)/(sqrt(T) r (1 - sqrt(rho))) + ..., whereas the theorem's first term has been divided by an extra factor of r. After the step-size choice alpha <~ sqrt(N/T), the corrected leading term is of order sqrt(N r/T)/(1 - sqrt(rho)), which increases with r. Thus both the abstract's claim that consensus differences 'vanish as rank increases' and the main-text reading of Theorem 2 are contradicted by the theorem's own leading term.
  4. [Assumption 3.1] As written, Assumption 3.1 asserts a uniform inequality ||W - W'||_F <= C(||A - A'||_F + ||B - B'||_F) for arbitrary parameters W, W', A, A', B, B'. For arbitrary full-rank W unrelated to the factors this cannot hold, and for LoRA-parameterized W the paper justifies C only in the special cases A = A' or B = B'; the general case is dismissed with 'one can set a sufficiently large constant C'. Since C enters the smoothness constant Lhat linearly in Lemma 1 and hence every convergence rate, this is load-bearing. The assumption should be restricted to the LoRA parameterization and proved, ideally from Assumption 2, which would give a concrete C of order eta c / r under bounded singular values.
minor comments (5)
  1. [Algorithm 4] In Algorithm 4, line 22 sets B_i^{t+1/2} = A_i^t; this should almost certainly be B_i^t.
  2. [Lemma 1 and SM A.3.3] Lemma 1 states the smoothness constant in terms of c, but the proof and Remark 1 use sigma in the same role (for example, 2 L C sqrt(r) sigma + G). Please unify the notation.
  3. [SM A.3.8, Eq. (37)] In the proof of Theorem 2, Eq. (37) has rho^{(T-t)/2}; for the recursion in Eq. (35) the exponent should be rho^{(T-1-t)/2}. Please check the off-by-one and its effect on the subsequent bound.
  4. [Table 3] The BoolQ F1 scores in Table 3 are all near chance (around 60-64); a sentence explaining the evaluation protocol, number of classes, or task difficulty would help the reader interpret these numbers.
  5. [SM A.3.9, Theorem 7] Theorem 7 reuses the symbol sigma in the smoothness constant without defining it in that subsection; align it with the main-text constant c.

Circularity Check

2 steps flagged · score 6.0 of 10

DeCAF's advertised O(1/√T) rate is obtained by defining the DLoRA–DeCAF consensus difference without TSVD and then analyzing a TSVD-free surrogate recursion, so the rate claim partly reduces to the surrogate's rate.

  1. self definitional [Definition 1 and Theorem 2 (Section 4)]
    "Denote by ET =E[∥ P j∈N b(i)πijBj T Aj T − P j∈N b(i)πijBj T P j∈N b(i)πijAj T ∥F ] the consensus error between DLoRA and DeCAF at any T >0."

    This ET is exactly the Definition 1 model-consensus-interference term, which is defined for DLoRA's individual consensus and contains no TSVD operator. DeCAF is defined by the product consensus plus the TSVD step T(Σ πij Bj t Aj t) in Algorithm 1, Lines 10–11. Calling ET the 'consensus error between DLoRA and DeCAF' defines away the TSVD operation that distinguishes DeCAF from DLoRA. Consequently, the conclusion drawn from this quantity — that consensus differences between DLoRA and DeCAF vanish as rank increases — is a statement about the interference term inside DLoRA, not about the difference between the two algorithms. The actual DLoRA–DeCAF difference requires combining ET with Proposition 1's TSVD reconstruction error, which is never done.

  2. other [SM A.3.8, Eq. (35); Section 4 after Theorem 2]
    "Therefore, to theoretically shed light on the difference between DLoRA and DeCAF, we assume that there is no TSVD in DeCAF ... the different version of DeCAF is ∆Wi t+1/2 = P j∈N b(i)πij∆Wj t ,∆W i t+1 = ∆Wi t+1/2 −αg i,∆W t . ... Combining the convergence rate of DLoRA yields that the convergence rate of DeCAF still remainsO(1/ √ T)."

    The proof of Theorem 2 analyzes Eq. (35), a TSVD-free recursion on ∆W, after explicitly assuming 'there is no TSVD in DeCAF.' The main text then transfers DLoRA's O(1/√T) rate to DeCAF with 'Combining the convergence rate of DLoRA.' The transfer is the load-bearing step: DeCAF's actual update applies TSVD every communication round and feeds the reconstructed pair into the next SGD step, and Proposition 1's error bound √((|N b(i)|−1)r)c² is never inserted into the descent/consensus argument. Thus the advertised rate is the rate of the surrogate ∆W recursion relabeled as DeCAF's rate. Proving that the TSVD-perturbed (A,B) recursion inherits that rate is exactly what is assumed, so the matching-rate conclusion is partially circular.

full rationale

The empirical evaluation is self-contained: DeCAF is compared against external baselines, and no test metric is fitted from the theoretical rate, so no fitted-input circularity is present. The DLoRA analysis (Theorem 1, Lemma 1) is a conditional decentralized-SGD argument under Assumptions 1–5; Assumption 3.1 is an assumption (weakly justified but not a renamed prediction), so it does not by itself make the DLoRA rate circular. The circularity is specific to the DeCAF theoretical claim. First, the quantity ET labeled 'consensus error between DLoRA and DeCAF' is Definition 1's interference term, which excludes TSVD — DeCAF's defining operation — so the difference is defined in a way that removes DeCAF. Second, SM A.3.8 proves the bound for a no-TSVD ∆W recursion and the main text transfers DLoRA's rate to DeCAF without feeding Proposition 1's TSVD error into the convergence proof. The central advertised prediction for DeCAF therefore reduces, by construction of the surrogate recursion, to the surrogate's rate rather than to Algorithm 1's. Score 6 reflects partial circularity in the central claim; the DLoRA rate, DLoRA-FA analysis, and experiments retain independent content.

Assumptions & free parameters 1 free parameters · 6 assumptions · 0 invented entities

The central convergence claims rest on five assumptions about smoothness, bounded singular values, Lipschitz continuity, stochastic noise, and the mixing matrix. One constant, C, is effectively chosen by hand to force the key smoothness lemma, which weakens the parameter-free character of the derivation. No new physical or algorithmic entities are introduced; TSVD is an existing tool.

free parameters (1)
  • C (Assumption 3.1)
    Lipschitz constant relating distances in W to distances in (A,B). The paper explicitly says one can set it 'sufficiently large' to make the assumption hold, so it is an ad hoc unknown that enters the smoothness constant and every convergence bound.
assumptions (6)
  • domain assumption Assumption 1: f_i is L-smooth in the full parameter W.
    Standard smoothness assumption on the original objective, used in Lemma 1 and all convergence proofs.
  • domain assumption Assumption 2: largest singular values of all A_i and B_i are bounded by c.
    Needed to make the smoothness constant finite and to bound the TSVD error in Proposition 1; not guaranteed by LoRA initialization or training dynamics.
  • ad hoc to paper Assumption 3.1: there exists C such that ||W-W'|| <= C(||A-A'|| + ||B-B'||).
    Introduced specifically to recover smoothness in the low-rank parameterization; the paper only justifies it in special cases and otherwise sets C 'sufficiently large'.
  • domain assumption Assumption 3.2: f_i is G-Lipschitz continuous in W.
    Generic Lipschitz assumption used in the proof of Lemma 1.
  • domain assumption Assumption 4: bounded stochastic gradient variance and bounded gradient diversity.
    Standard noise and heterogeneity bounds used in the convergence analysis.
  • domain assumption Assumption 5: the mixing matrix is symmetric doubly stochastic with spectral gap.
    Standard assumption on network topology, used to control consensus error.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DeCAF: Decentralized Consensus-And-Factorization for Low-Rank Adaptation of Foundation Models." pith.science (2026). https://pith.science/paper/NX6PDENT

@misc{pith2026250521382,
  author       = {Pith},
  title        = {Pith review of: DeCAF: Decentralized Consensus-And-Factorization for Low-Rank Adaptation of Foundation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NX6PDENT}},
  note         = {Machine review of arXiv:2505.21382}
}
read the original abstract

Low-Rank Adaptation (LoRA) has emerged as one of the most effective, computationally tractable fine-tuning approaches for training Vision-Language Models (VLMs) and Large Language Models (LLMs). LoRA accomplishes this by freezing the pre-trained model weights and injecting trainable low-rank matrices, allowing for efficient learning of these foundation models even on edge devices. However, LoRA in decentralized settings still remains under explored, particularly for the theoretical underpinnings due to the lack of smoothness guarantee and model consensus interference (defined formally below). This work improves the convergence rate of decentralized LoRA (DLoRA) to match the rate of decentralized SGD by ensuring gradient smoothness. We also introduce DeCAF, a novel algorithm integrating DLoRA with truncated singular value decomposition (TSVD)-based matrix factorization to resolve consensus interference. Theoretical analysis shows TSVD's approximation error is bounded and consensus differences between DLoRA and DeCAF vanish as rank increases, yielding DeCAF's matching convergence rate. Extensive experiments across vision/language tasks demonstrate our algorithms outperform local training and rivals federated learning under both IID and non-IID data distributions.

Figures

Figures reproduced from arXiv: 2505.21382 by the authors.

Figure 1
Figure 1. DeCAF with ring topology: At step t, agent i exchanges Ai and Bi with neighbors, performs consensus on their products, applies truncated SVD, and updates using local data. Pre￾trained weights W0 remain frozen. decentralized SGD methods [12, 13]. Also, Dec-LoRA adopts the individual consensus update for low-rank matrices, leaving the so-called model consensus interference [14, 15, 16, 17] issue unsolved. In the theor… view at source ↗
Figure 2
Figure 2. Test accuracy on the Flowers dataset with CLIP: (2a) IID and (2b) non-IID topologies using DeCAF; (2c) decentralized algorithms on non-IID ring topology (40 shots/class, rank 2). (a) Low-Rank Variation (DeCAF) (b) Accuracy gap (c) Scalability: FC, IID [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Test accuracy on Flowers with CLIP: (a) DeCAF with varying ranks (IID, FC); (b) DeCAF vs. DLoRA accuracy gap; (c) DeCAF scalability with agent count (40 shots/class, rank 2). 5 Numerical Results In this section, we evaluate the performance of our proposed algorithms across various model architectures and datasets. We benchmark these algorithms against established baselines: Dec-LoRA (DLoRA) [11], Federated Averaging… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Illustration of DLoRA-FA by using two agents: agents [PITH_FULL_IMAGE:figures/full_fig_p021_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

78 extracted references · 39 canonical work pages

  1. [1]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021

    2021

  2. [2]

    Gpt-4 technical report, 2024

    OpenAI. Gpt-4 technical report, 2024

    2024

  3. [3]

    Gpt-3: Its nature, scope, limits, and consequences

    Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds Mach., 30(4):681–694, December 2020

    work page 2020

  4. [4]

    Scaling language models: Methods, analysis & insights from training gopher, 2022

    Google DeepMind. Scaling language models: Methods, analysis & insights from training gopher, 2022

    work page 2022

  5. [5]

    Rae, Oriol Vinyals, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre...

    2022

  6. [6]

    Skipping computations in multimodal llms, 2024

    Mustafa Shukor and Matthieu Cord. Skipping computations in multimodal llms, 2024

    work page 2024

  7. [7]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021

    2021

  8. [8]

    Decentralized federated learning: Fundamentals, state of the art, frameworks, trends, and challenges.IEEE Communications Surveys & Tutorials, 2023

    Enrique Tomás Martínez Beltrán, Mario Quiles Pérez, Pedro Miguel Sánchez Sánchez, Ser- gio López Bernal, Gérôme Bovet, Manuel Gil Pérez, Gregorio Martínez Pérez, and Alberto Huer- tas Celdrán. Decentralized federated learning: Fundamentals, state of the art, frameworks, trends, and challenges.IEEE Communications Surveys & Tutorials, 2023

    work page 2023

Show all 78 references
  1. [9]

    Decentralized federated learning: A survey on security and privacy.IEEE Transactions on Big Data, 2024

    Ehsan Hallaji, Roozbeh Razavi-Far, Mehrdad Saif, Boyu Wang, and Qiang Yang. Decentralized federated learning: A survey on security and privacy.IEEE Transactions on Big Data, 2024

    2024

  2. [10]

    El Mahdi El-Mhamdi, Sadegh Farhadkhani, Rachid Guerraoui, Arsany Guirguis, Lê-Nguyên Hoang, and Sébastien Rouault. Collaborative learning in the jungle (decentralized, byzan- tine, heterogeneous, asynchronous and nonconvex learning).Advances in neural information processing sy...

    2021

  3. [11]

    Decentralized low-rank fine- tuning of large language models.arXiv preprint arXiv:2501.15361, 2025

    Sajjad Ghiasvand, Mahnoosh Alizadeh, and Ramtin Pedarsani. Decentralized low-rank fine- tuning of large language models.arXiv preprint arXiv:2501.15361, 2025

    2025 arXiv

  4. [12]

    On the convergence of local descent methods in federated learning.arXiv preprint arXiv:1910.14425, 2019

    Farzin Haddadpour and Mehrdad Mahdavi. On the convergence of local descent methods in federated learning.arXiv preprint arXiv:1910.14425, 2019

    1910 arXiv

  5. [13]

    A unified theory of decentralized sgd with changing topology and local updates

    Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. A unified theory of decentralized sgd with changing topology and local updates. InInternational conference on machine learning, pages 5381–5393. PMLR, 2020

    2020

  6. [14]

    Improving lora in privacy-preserving federated learning.arXiv preprint arXiv:2403.12313, 2024

    Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving lora in privacy-preserving federated learning.arXiv preprint arXiv:2403.12313, 2024

    2024 arXiv

  7. [15]

    Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations

    Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations. arXiv preprint arXiv:2409.05976, 2024

    2024 arXiv

  8. [16]

    Selec- tive aggregation for low-rank adaptation in federated learning.arXiv preprint arXiv:2410.01463, 2024

    Pengxin Guo, Shuang Zeng, Yanran Wang, Huijie Fan, Feifei Wang, and Liangqiong Qu. Selec- tive aggregation for low-rank adaptation in federated learning.arXiv preprint arXiv:2410.01463, 2024

    2024 arXiv

  9. [17]

    Deer: Deviation eliminating and noise regu- lating for privacy-preserving federated low-rank adaptation.arXiv preprint arXiv:2410.12926, 2024

    Meilu Zhu, Axiu Mao, Jun Liu, and Yixuan Yuan. Deer: Deviation eliminating and noise regu- lating for privacy-preserving federated low-rank adaptation.arXiv preprint arXiv:2410.12926, 2024. 10

    2024 arXiv

  10. [18]

    Fast updating truncated svd for representation learning with sparse matrices.arXiv preprint arXiv:2401.09703, 2024

    Haoran Deng, Yang Yang, Jiahe Li, Cheng Chen, Weihao Jiang, and Shiliang Pu. Fast updating truncated svd for representation learning with sparse matrices.arXiv preprint arXiv:2401.09703, 2024

    2024 arXiv

  11. [19]

    Llama 2: Open foundation and fine-tuned chat models, 2023

    Meta. Llama 2: Open foundation and fine-tuned chat models, 2023

    2023

  12. [20]

    Fedlora: Model-heterogeneous personal- ized federated learning with lora tuning.arXiv preprint arXiv:2310.13283, 2023

    Liping Yi, Han Yu, Gang Wang, and Xiaoguang Liu. Fedlora: Model-heterogeneous personal- ized federated learning with lora tuning.arXiv preprint arXiv:2310.13283, 2023

    2023 arXiv

  13. [21]

    Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models.Advances in Neural Information Processing Systems, 36, 2024

    Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, et al. Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models.Advances in Neural Information Processing Syst...

    2024

  14. [22]

    Personalized collaborative fine-tuning for on-device large language models.arXiv preprint arXiv:2404.09753, 2024

    Nicolas Wagner, Dongyang Fan, and Martin Jaggi. Personalized collaborative fine-tuning for on-device large language models.arXiv preprint arXiv:2404.09753, 2024

    2024 arXiv

  15. [23]

    Learning multiple visual domains with residual adapters, 2017

    Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters, 2017

    2017

  16. [24]

    Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning, 2022

    Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning, 2022

    2022

  17. [25]

    Prefix-tuning: Optimizing continuous prompts for generation, 2021

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation, 2021

    2021

  18. [26]

    Aflora: Adaptive freezing of low rank adaptation in parameter efficient fine-tuning of large models, 2024

    Zeyu Liu, Souvik Kundu, Anni Li, Junrui Wan, Lianghao Jiang, and Peter Anthony Beerel. Aflora: Adaptive freezing of low rank adaptation in parameter efficient fine-tuning of large models, 2024

    2024

  19. [27]

    Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning.arXiv preprint arXiv:2308.03303, 2023

    Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning.arXiv preprint arXiv:2308.03303, 2023

    2023 arXiv

  20. [28]

    Loftq: Lora-fine-tuning-aware quantization for large language models.arXiv preprint arXiv:2310.08659, 2023

    Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, and Tuo Zhao. Loftq: Lora-fine-tuning-aware quantization for large language models.arXiv preprint arXiv:2310.08659, 2023

    2023 arXiv

  21. [29]

    Qlora: Efficient finetuning of quantized llms.Advances in Neural Information Processing Systems, 36, 2024

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms.Advances in Neural Information Processing Systems, 36, 2024

    2024

  22. [30]

    Qa-lora: Quantization-aware low-rank adaptation of large language models.arXiv preprint arXiv:2309.14717, 2023

    Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xiaopeng Zhang, and Qi Tian. Qa-lora: Quantization-aware low-rank adaptation of large language models.arXiv preprint arXiv:2309.14717, 2023

    2023 arXiv

  23. [31]

    Low-rank few-shot adaptation of vision-language models

    Maxime Zanella and Ismail Ben Ayed. Low-rank few-shot adaptation of vision-language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1593–1603, 2024

    2024

  24. [32]

    Dimat: Decentralized iterative merging-and-training for deep learning models

    Nastaran Saadati, Minh Pham, Nasla Saleem, Joshua R Waite, Aditya Balu, Zhanong Jiang, Chinmay Hegde, and Soumik Sarkar. Dimat: Decentralized iterative merging-and-training for deep learning models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2024

  25. [33]

    Dominating set model aggregation for communication-efficient decentralized deep learning.Neural Networks, 171:25–39, 2024

    Fateme Fotouhi, Aditya Balu, Zhanhong Jiang, Yasaman Esfandiari, Salman Jahani, and Soumik Sarkar. Dominating set model aggregation for communication-efficient decentralized deep learning.Neural Networks, 171:25–39, 2024

    2024

  26. [34]

    Collaborative deep learning in fixed topology networks.Advances in Neural Information Processing Systems, 30, 2017

    Zhanhong Jiang, Aditya Balu, Chinmay Hegde, and Soumik Sarkar. Collaborative deep learning in fixed topology networks.Advances in Neural Information Processing Systems, 30, 2017

    2017

  27. [35]

    Cross-gradient aggregation for decentralized learning from non-iid data

    Yasaman Esfandiari, Sin Yong Tan, Zhanhong Jiang, Aditya Balu, Ethan Herron, Chinmay Hegde, and Soumik Sarkar. Cross-gradient aggregation for decentralized learning from non-iid data. InInternational Conference on Machine Learning, pages 3036–3046. PMLR, 2021. 11

    2021

  28. [36]

    Stochastic gradient push for distributed deep learning, 2019

    Mahmoud Assran, Nicolas Loizou, Nicolas Ballas, and Michael Rabbat. Stochastic gradient push for distributed deep learning, 2019

    2019

  29. [37]

    Personalized collaborative fine-tuning for on-device large language models, 2024

    Nicolas Wagner, Dongyang Fan, and Martin Jaggi. Personalized collaborative fine-tuning for on-device large language models, 2024

    2024

  30. [38]

    Mhrc: Closed- loop decentralized multi-heterogeneous robot collaboration with large language models.arXiv preprint arXiv:2409.16030, 2024

    Wenhao Yu, Jie Peng, Yueliang Ying, Sai Li, Jianmin Ji, and Yanyong Zhang. Mhrc: Closed- loop decentralized multi-heterogeneous robot collaboration with large language models.arXiv preprint arXiv:2409.16030, 2024

    2024 arXiv

  31. [39]

    Yongchao Chen, Jacob Arkin, Yang Zhang, Nicholas Roy, and Chuchu Fan. Scalable multi-robot collaboration with large language models: Centralized or decentralized systems? In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 4311–4317. IEEE, 2024

    2024

  32. [40]

    A systematic literature review of blockchain-based federated learning: Architectures, applications and issues

    Dongkun Hou, Jie Zhang, Ka Lok Man, Jieming Ma, and Zitian Peng. A systematic literature review of blockchain-based federated learning: Architectures, applications and issues. In2021 2nd Information Communication Technologies Conference (ICTC), pages 302–307, 2021

    2021

  33. [41]

    Brendan McMahan, Timon Van Overveldt, David Petrou, Daniel Ramage, and Jason Roselander

    Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Koneˇcný, Stefano Mazzocchi, H. Brendan McMahan, Timon Van Overveldt, David Petrou, Daniel Ramage, and Jason Roselander. Towards federated learning at scale: S...

    2019

  34. [42]

    Federated learning: Chal- lenges, methods, and future directions.IEEE signal processing magazine, 37(3):50–60, 2020

    Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Chal- lenges, methods, and future directions.IEEE signal processing magazine, 37(3):50–60, 2020

    2020

  35. [43]

    Intrinsic dimensionality explains the effectiveness of language model fine-tuning.arXiv preprint arXiv:2012.13255, 2020

    Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning.arXiv preprint arXiv:2012.13255, 2020

    2012 arXiv

  36. [44]

    Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  37. [45]

    Randomized asymmetric chain of lora: The first meaningful theoretical framework for low-rank adaptation.arXiv preprint arXiv:2410.08305, 2024

    Grigory Malinovsky, Umberto Michieli, Hasan Abed Al Kader Hammoud, Taha Ceritli, Hayder Elesedy, Mete Ozay, and Peter Richtárik. Randomized asymmetric chain of lora: The first meaningful theoretical framework for low-rank adaptation.arXiv preprint arXiv:2410.08305, 2024

    arXiv

  38. [46]

    On nonconvex decentralized gradient descent.IEEE Transactions on signal processing, 66(11):2834–2848, 2018

    Jinshan Zeng and Wotao Yin. On nonconvex decentralized gradient descent.IEEE Transactions on signal processing, 66(11):2834–2848, 2018

    2018

  39. [47]

    Gradient tracking with multiple local sgd for decentralized non-convex learning

    Songyang Ge and Tsung-Hui Chang. Gradient tracking with multiple local sgd for decentralized non-convex learning. In2023 62nd IEEE Conference on Decision and Control (CDC), pages 133–138. IEEE, 2023

    2023

  40. [48]

    Decentralized stochastic projection-free learning with compressed push-sum

    Robin Francis and Sundeep Prabhakar Chepuri. Decentralized stochastic projection-free learning with compressed push-sum. In2023 IEEE 33rd International Workshop on Machine Learning for Signal Processing (MLSP), pages 1–6. IEEE, 2023

    2023

  41. [49]

    Bypassing the ambient dimension: Private sgd with gradient subspace identification.arXiv preprint arXiv:2007.03813, 2020

    Yingxue Zhou, Zhiwei Steven Wu, and Arindam Banerjee. Bypassing the ambient dimension: Private sgd with gradient subspace identification.arXiv preprint arXiv:2007.03813, 2020

    2007 arXiv

  42. [50]

    On the convergence of fedavg on non-iid data.arXiv preprint arXiv:1907.02189, 2019

    Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data.arXiv preprint arXiv:1907.02189, 2019

    1907 arXiv

  43. [51]

    Communication-efficient algorithms for statistical optimization.Advances in neural information processing systems, 25, 2012

    Yuchen Zhang, Martin J Wainwright, and John C Duchi. Communication-efficient algorithms for statistical optimization.Advances in neural information processing systems, 25, 2012

    2012

  44. [52]

    Local sgd converges fast and communicates little.arXiv preprint arXiv:1805.09767, 2018

    Sebastian U Stich. Local sgd converges fast and communicates little.arXiv preprint arXiv:1805.09767, 2018

    2018 arXiv

  45. [53]

    Sparsified sgd with memory

    Sebastian U Stich, Jean-Baptiste Cordonnier, and Martin Jaggi. Sparsified sgd with memory. Advances in neural information processing systems, 31, 2018. 12

    2018

  46. [54]

    Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning

    Hao Yu, Sen Yang, and Shenghuo Zhu. Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning. InProceedings of the AAAI conference on artificial intelligence, volume 33, pages 5693–5700, 2019

    2019

  47. [55]

    Openfedllm: Training large language models on decentralized private data via federated learning

    Rui Ye, Wenhao Wang, Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du, Yanfeng Wang, and Siheng Chen. Openfedllm: Training large language models on decentralized private data via federated learning. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and D...

    2024

  48. [56]

    Towards building the federatedgpt: Federated instruction tuning

    Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. Towards building the federatedgpt: Federated instruction tuning. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pag...

    2024

  49. [57]

    Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas

    H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication-efficient learning of deep networks from decentralized data, 2023

    2023

  50. [58]

    Automated flower classification over a large number of classes.2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722–729, 2008

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes.2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722–729, 2008

    2008

  51. [59]

    Ucf101: A dataset of 101 human actions classes from videos in the wild, 2012

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild, 2012

    2012

  52. [60]

    Food-101 – mining discriminative components with random forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 – mining discriminative components with random forests. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors,Computer Vision – ECCV 2014, pages 446–461, Cham, 2014. Springer International Publishing

    2014

  53. [61]

    Wic: the word-in-context dataset for evaluating context-sensitive meaning representations, 2019

    Mohammad Taher Pilehvar and Jose Camacho-Collados. Wic: the word-in-context dataset for evaluating context-sensitive meaning representations, 2019

    2019

  54. [62]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019

    2019

  55. [63]

    Flora: Low-rank adapters are secretly gradient compressors, 2024

    Yongchang Hao, Yanshuai Cao, and Lili Mou. Flora: Low-rank adapters are secretly gradient compressors, 2024

    2024

  56. [64]

    Federated lora with sparse communication, 2024

    Kevin Kuo, Arian Raje, Kousik Rajesh, and Virginia Smith. Federated lora with sparse communication, 2024

    2024

  57. [65]

    Fedms: Federated learning with mixture of sparsely activated foundations models.arXiv preprint arXiv:2312.15926, 2023

    Panlong Wu, Kangshuo Li, Ting Wang, and Fangxin Wang. Fedms: Federated learning with mixture of sparsely activated foundations models.arXiv preprint arXiv:2312.15926, 2023

    2023 arXiv

  58. [66]

    Towards building the federated gpt: Federated instruction tuning, 2024

    Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Yufan Zhou, Guoyin Wang, and Yiran Chen. Towards building the federated gpt: Federated instruction tuning, 2024

    2024

  59. [67]

    Openfedllm: Training large language models on decentralized private data via federated learning, 2024

    Rui Ye, Wenhao Wang, Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du, Yanfeng Wang, and Siheng Chen. Openfedllm: Training large language models on decentralized private data via federated learning, 2024

    2024

  60. [68]

    Differentially private low-rank adaptation of large language model using federated learning

    Xiao-Yang Liu, Rongyi Zhu, Daochen Zha, Jiechao Gao, Shan Zhong, Matt White, and Meikang Qiu. Differentially private low-rank adaptation of large language model using federated learning. ACM Transactions on Management Information Systems, 2023

    2023

  61. [69]

    On the kronecker product.Master’s thesis, University of Waterloo, 2004

    Kathrin Schacke. On the kronecker product.Master’s thesis, University of Waterloo, 2004

    2004

  62. [70]

    Asymmetry in low-rank adapters of foundation models.arXiv preprint arXiv:2402.16842, 2024

    Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz Sáez de Ocáriz Borde, Rickard Brüel Gabrielsson, Leshem Choshen, Marzyeh Ghassemi, Mikhail Yurochkin, and Justin Solomon. Asymmetry in low-rank adapters of foundation models.arXiv preprint arXiv:2402.16842, 2024. 13

    2024 arXiv

  63. [71]

    On the linear speedup analysis of communication efficient momentum sgd for distributed non-convex optimization

    Hao Yu, Rong Jin, and Sen Yang. On the linear speedup analysis of communication efficient momentum sgd for distributed non-convex optimization. InInternational Conference on Machine Learning, pages 7184–7193. PMLR, 2019

    2019

  64. [72]

    Local operator theory, random matrices and banach spaces

    Kenneth R Davidson and Stanislaw J Szarek. Local operator theory, random matrices and banach spaces. InHandbook of the geometry of Banach spaces, volume 1, pages 317–366. Elsevier, 2001

    2001

  65. [73]

    Flora: Low-rank adapters are secretly gradient compressors.arXiv preprint arXiv:2402.03293, 2024

    Yongchang Hao, Yanshuai Cao, and Lili Mou. Flora: Low-rank adapters are secretly gradient compressors.arXiv preprint arXiv:2402.03293, 2024

    2024 arXiv

  66. [74]

    Balancing communication and computation in distributed optimization.IEEE Transactions on Automatic Control, 64(8):3141–3155, 2018

    Albert S Berahas, Raghu Bollapragada, Nitish Shirish Keskar, and Ermin Wei. Balancing communication and computation in distributed optimization.IEEE Transactions on Automatic Control, 64(8):3141–3155, 2018

    2018

  67. [75]

    Instruction tuning with gpt-4, 2023

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, 2023

    2023

  68. [76]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023

    2023

  69. [77]

    Gpt-4o system card, 2024

    OpenAI. Gpt-4o system card, 2024

    2024

  70. [78]

    approximate

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023. 14 A Additional Analysis and Results This section presents the missing analysis and proof as well as the ad...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.