Pith. sign in

REVIEW 3 major objections 4 minor 97 references

FedRPCA: Enhancing Federated LoRA Aggregation Using Robust PCA

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FedRPCA splits client LoRA updates via Robust PCA into a common low-rank part to average and a sparse client-specific part to amplify, and reports gains over FedAvg, Task Arithmetic, TIES-Merging, FedProx, SCAFFOLD, and MOON on all six…

desk verdict A clean, well-tested server-side aggregation trick that consistently beats FedAvg by small margins, but the paper never really shows that RPCA is separating common from client-specific knowledge—treat the mechanism as an appealing hypothesis, not a demonstrated fact. read the letter →

arxiv 2506.01194 v1 pith:6KNOFNG6 submitted 2025-06-01 cs.LG cs.DC

Divyansh Jhunjhunwala , Arian Raje , Madan Ravi Ganesh , Chaithanya Kumar Mummadi , Chaoqun Dong , Jiawei Zhou , Wan-Yi Lin , Gauri Joshi
show 1 more author
Zhenzhen Li
This is my paper
classification cs.LGcs.DC
keywords federatedlearningLoRArobustprincipalcomponentanalysisdataheterogeneitymodelmergingtaskarithmeticparameter-efficientfine-tuninglow-rankplussparsedecomposition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that data heterogeneity degrades federated LoRA fine-tuning because averaging client updates washes out the knowledge each client holds that others do not, while preserving the shared knowledge that is already well covered. It proposes FedRPCA, a server-only aggregation rule that uses Robust PCA to split the stacked client LoRA updates into a low-rank common component and a sparse client-specific component, then averages the common part and applies scaled averaging to the personal part. FedRPCA is reported to beat FedAvg, Task Arithmetic, TIES-Merging, FedProx, SCAFFOLD, and MOON on all six vision and language benchmarks tested, with final-accuracy improvements of 0.28 to 1.01 percentage points over the second-best method, and the gains grow as heterogeneity and client count increase. If correct, this matters because the fix lives entirely at the server: no client-side training changes, no extra communication, and the method can be stacked on top of client-side methods such as FedProx and SCAFFOLD.

What carries the argument

The central object is the per-round Robust PCA decomposition $M = L + S$ of the matrix whose columns are the vectorized LoRA updates of all clients, computed separately for the two LoRA factors $A$ and $B$. Robust PCA, implemented here via an ADMM scheme with nuclear-norm and $\ell^1$ penalties, is meant to split the updates so that the low-rank part $L$ carries knowledge common to all clients and the sparse part $S$ carries per-client deviations; as indirect evidence, the paper reports average pairwise cosine similarity of 0.43 on raw updates, 0.71 on the low-rank parts, and 0.12 on the sparse parts. The aggregation rule then averages the columns of $L$ and applies a scaling factor $\beta$ to the averaged columns of $S$. The choice of $\beta$ is adaptive: $\beta_A^{(t)} = 1/E_A^{(t)}$ with $E_A^{(t)} = \|S_A^{(t)}\mathbf{1}\|/\|M_A^{(t)}\mathbf{1}\|$, the norm fraction of the update contributed by the sparse component, which the paper observes rising over training so that a fixed $\beta$ is suboptimal.

What would settle it

A synthetic experiment with planted structure would settle the question: build clients whose true updates are known sums of a shared low-rank direction and known client-specific sparse directions (the panda-cat-dog example made concrete), run FedRPCA, and check whether Robust PCA recovers the planted $L$ and $S$. If the recovered sparse components match the planted signals and the gains survive only when those components are used intact, the proposed mechanism is confirmed; if the gains persist even when the recovered sparse entries are permuted across clients or replaced by noise of equal norm, then the accuracy improvement comes from something other than the claimed signal separation.

Watch

Extended reading notes

Core claim

The central claim is that a client's LoRA update in federated fine-tuning is a mixture of a common signal shared across clients and a sparse client-specific signal, and that the right aggregation treats the two differently. The paper illustrates with a two-client example: if client 1 sees pandas and cats and client 2 sees pandas and dogs, with updates $\tau_1 = \tau_P + \tau_C$ and $\tau_2 = \tau_P + \tau_D$, the ideal global update is $\tau_P + \tau_C + \tau_D$. FedAvg produces $\tau_P + \tfrac{1}{2}(\tau_C + \tau_D)$, under-representing personal knowledge, while a direct Task Arithmetic scale of 2 produces $2\tau_P + \tau_C + \tau_D$, over-amplifying the shared panda signal. FedRPCA instead applies Robust PCA to the matrix of stacked vectorized client updates, for the $A$ and $B$ LoRA factors separately, decomposing $M = L + S$, then averages the low-rank $L$ and scaled-averages the sparse $S$; with scale $\beta = 2$ the toy example recovers exactly the ideal update. Empirically the paper reports FedRPCA as the best method on EuroSAT, SVHN, DTD, Stanford Cars, 20News, and MRQA, with the gap over baselines widening as Dirichlet heterogeneity strengthens and the number of clients grows.

Load-bearing premise

The load-bearing premise is that Robust PCA, applied round by round to the stacked LoRA updates, really separates shared knowledge from client-specific knowledge: the sparse component it recovers is genuine per-client signal rather than noise or an artifact of the low-rank fit, and the paper's evidence for this is indirect (cosine-similarity plots and visualizations) rather than a direct recovery check.

Editorial extensions

If this is right

  • Because FedRPCA changes only the server aggregation step, it can be composed with client-level heterogeneity fixes: combining it with FedProx and SCAFFOLD improves on either method alone (Figure 5).
  • The advantage over baselines grows with data heterogeneity (from +1.02 points at $\alpha = 10$ to +1.80 at $\alpha = 0.1$ on SVHN) and with client count (from +0.09 points at 20 clients to +1.68 at 100), the regimes where naive averaging most strongly suppresses client-specific signal.
  • At higher LoRA ranks the final-accuracy gap narrows, but the convergence advantage persists: at rank 8 FedRPCA needs 25 rounds to reach 90% of final accuracy versus FedAvg's 41 (1.52x faster), and at rank 32 it needs 12 versus 19 (1.42x faster).
  • The method adds no communication overhead and only modest server compute, about 1.5x FedAvg's per-round wall-clock time in the reported ViT-B/32 setup, since the RPCA input matrices are small (on the order of $10^3$ entries).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A randomization test the paper does not run would directly probe the mechanism: permuting the entries of the recovered sparse components across clients before aggregation should destroy the gains if $S$ truly carries client-specific knowledge, and the gains would survive scrambling only if the benefit comes from the scaling itself.
  • Because the decomposition is recomputed each round on fresh updates, the 'common' component is a within-round agreement rather than a tracked shared subspace; an online or streaming RPCA that carries the low-rank subspace across rounds could cut the per-round SVD cost and might stabilize the common component.
  • The same split-and-scale logic should transfer to other additive PEFT updates such as adapters or quantized LoRA variants, and to heterogeneous-rank settings where client updates must be padded or projected before stacking; the paper names transfer beyond LoRA only as future work, so this is an extension rather than a demonstrated result.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes FedRPCA, a server-side aggregation method for federated LoRA fine-tuning. At each communication round, the server stacks the vectorized LoRA updates from all clients into matrices and applies Robust PCA to decompose each matrix into a low-rank component L and a sparse component S. The low-rank components are averaged to consolidate common knowledge, while the sparse components are averaged with a scaling factor beta (set adaptively to 1/E, where E is the ratio of the sparse-update norm to the original-update norm) to amplify client-specific knowledge. The method is evaluated on six vision and language benchmarks, with ablations over heterogeneity, number of clients, LoRA rank, and combinations with FedProx and SCAFFOLD. The central modeling assumption is that RPCA, applied per round to the stacked LoRA updates, correctly separates common knowledge from client-specific knowledge.

Significance. If the low-rank/sparse separation is valid, FedRPCA is an attractive contribution: it is a simple server-side drop-in replacement for FedAvg, requires no client-side changes, and the reported results show consistent accuracy gains (0.28 to 1.01 percentage points) and faster convergence across six datasets, with larger gains under higher heterogeneity and more clients. The paper is also commendable for its broad empirical evaluation, including combinations with FedProx and SCAFFOLD and an analysis of computational overhead. However, the load-bearing assumption that RPCA recovers semantically meaningful common and client-specific components is not directly verified, and several strong LoRA-specific aggregation baselines are omitted. These issues currently temper the significance of the empirical claims.

major comments (3)
  1. [Section 4, Eqs. (7)-(9), Figure 1] The central claim that RPCA separates common knowledge from client-specific knowledge is not validated against ground truth. Because the input matrices M_A and M_B have dimensions p x M with p >> M (p is the LoRA parameter dimension and M is the number of clients), the low-rank component is not forced to represent a signal shared across clients; it can span all M clients while remaining low-rank. The cosine-similarity evidence in Figure 1 shows only that L columns are mutually similar and S columns are mutually dissimilar, which can hold for arbitrary low-rank/sparse decompositions and does not establish that the sparse entries are informative client-specific signals rather than noise or artifacts. Please provide a direct validation: for example, a synthetic setup with known common and client-specific components, a comparison of S columns against client-specific task vectors measured by leave-one-out or per-client held-out data, or an ablation that replaces S with noise of matched sparsity and norm. Without such a test, the gains could come from a generic sparse-residual correction plus adaptive scaling rather than from the proposed semantic separation.
  2. [Section 4 and Section 3] The sparsity assumption on client-specific updates is motivated by TIES-Merging, which applies to full-model task vectors after pruning and sign-conflict resolution. FedRPCA, however, applies RPCA to the raw LoRA factor matrices A and B. There is no evidence that client-specific knowledge in these factor matrices is sparse; if it is dense, RPCA will assign it to the low-rank component L, and it will not be amplified by beta. Please provide empirical evidence that the true client-specific LoRA factors are sparse (e.g., by measuring norm concentration of per-client updates relative to the global update) or a theoretical argument for why the TIES sparsity finding transfers to unfiltered LoRA factors.
  3. [Section 5, Table 1 and Related Work] The empirical comparison omits recent LoRA-specific aggregation methods that are discussed in the Related Work, such as FedEx-LoRA, FFA-LoRA, and FLoRA, which are designed to address the suboptimality of averaging A and B separately. Since FedRPCA also aggregates A and B separately before reshaping, it is important to show that its gains are not simply a byproduct of a different aggregation rule for the bilinear structure. Please add at least one or two of these baselines, or provide a concrete reason why they are not applicable to the considered setting.
minor comments (4)
  1. [Table 4] For rank 32, the reported accuracy improvement over the second-best method is +0.02 percentage points (94.05 vs 94.03), which is well within the reported standard deviations. The text correctly notes that the gap narrows, but labeling this as an 'improvement' overstates the result; please clarify that the rank-32 claim rests on the convergence speed-up, not on final accuracy, and report significance tests across seeds.
  2. [Section B.3, Figure 7] The claim that the optimal beta is 'approximately inversely proportional' to E(t) is based on only three measurement points (rounds 1, 25, and 50). A more systematic sweep across rounds would strengthen the justification for the beta = 1/E heuristic.
  3. [Section B.2] The description of the computational cost is ambiguous: Algorithm 1 applies RPCA to the full matrices M_A and M_B, while Section B.2 states that RPCA is also applied 'jointly across the (A, B) pairs in each layer.' Please clarify the exact per-layer versus global application and report the resulting cost consistently.
  4. [Section 5, Table 1 caption and text] The abstract and contributions state that FedRPCA 'boosts accuracy by 1%' on DTD and 20News, but the reported improvement for 20News is 0.99 percentage points (and for DTD it is 1.01). Please use consistent rounding and describe improvements as percentage points rather than percents.

Circularity Check

3 steps flagged · score 5.0 of 10

The RPCA cosine-similarity evidence is a by-construction tautology of the low-rank/sparse constraints, and the adaptive beta schedule is fit on SVHN and then evaluated on SVHN; the headline accuracy gains across the other benchmarks remain empirically independent.

  1. self definitional [Section 1, Figure 1 caption (page 2); also referenced in the Section 4 discussion of Figure 3]
    "The average pairwise cosine similarity is substantially higher for the low-rank components, confirming that they capture shared structure, and significantly lower for the sparse components, confirming that they capture client-specific variations."

    By Eq. (6), L is constrained to be low-rank and S is constrained to be sparse. For a rank-1 L, every column is a scalar multiple of a single vector, so pairwise cosine similarities are ±1 by algebra regardless of whether those columns encode shared knowledge; for an ℓ1-regularized sparse S with mostly disjoint supports, pairwise cosines are low regardless of whether the nonzeros are meaningful client-specific signals. The observed contrast in Figure 1 is therefore an algebraic consequence of the decomposition variables, not empirical confirmation of the semantic labels 'common' and 'client-specific'. Using this diagnostic as confirmation is circular: the pattern is guaranteed by how L and S are defined rather than by the data or by the recovered content.

  2. renaming known result [Section 4, paragraph introducing Eq. (6) and the aggregation Eqs. (10)-(11)]
    "In this decomposition, the columns in the low-rank matrix L captures the common signal across clients, while the columns in sparse matrix S captures the client-specific deviations."

    Because M = L + S by construction, the FedRPCA update in Eqs. (10)-(11) equals avg(M) + (β−1)avg(S): it is FedAvg plus a scaled version of the low-rank residual. Labeling L 'common signal' and S 'client-specific deviations' is a semantic renaming of the algebraic factors; RPCA guarantees only a low-rank-plus-sparse factorization under the Candès et al. identifiability conditions, not that L's columns are shared across clients nor that S's nonzeros are informative client-specific knowledge rather than noise or artifacts. The only offered evidence for the renaming is the Figure 1 cosine diagnostic, which is itself by construction. The explanatory mechanism is therefore a relabeling of the decomposition rather than an independently verified cause of the observed accuracy gains.

1 more flagged steps
  1. fitted input called prediction [Appendix B.3, 'Adaptive scaling in FedRPCA' (page 22); main result in Table 1 (SVHN row)]
    "We observe that the optimal β generally decreases as training progresses and appears roughly inversely proportional to the norm contribution of the sparse updates (obtained via Robust-PCA) to the original updates ... Motivated by this, we propose a simple heuristic: set β(t) A = 1/E(t) A and β(t) B = 1/E(t) B in every round."

    The adaptive schedule β = 1/E is inferred from the SVHN optimal-β measurements described in Figure 7 and is then used to produce FedRPCA's SVHN result in Table 1 and the 'adaptive schedule outperforms fixed choices' comparison in Figure 8. The SVHN experiments therefore test the heuristic on the same dataset that generated it, so the SVHN row is not an independent validation of the adaptive-β rule. Since the same schedule is transferred to the other five benchmarks, the headline cross-benchmark claim retains independent support, but the in-sample SVHN confirmation is fitted rather than predicted.

full rationale

The paper's final accuracy numbers are real empirical outputs, benchmarked against FedAvg, Task Arithmetic, TIES-Merging, FedProx, SCAFFOLD, and MOON, so the central empirical claim is not forced by the method's definitions. However, the paper's mechanistic explanation of why FedRPCA works contains two by-construction elements. First, the cosine-similarity diagnostic in Figure 1 is presented as 'confirming' that low-rank components capture shared structure and sparse components capture client-specific variation, but those cosine properties follow from the low-rank and sparse constraints themselves (rank-1 columns have cosine ±1; sparse columns with disjoint supports have low cosine), so the diagnostic is self-confirming. Second, the identification of L as 'common signal' and S as 'client-specific deviations' is a semantic renaming of the algebraic decomposition; algebraically the FedRPCA update equals FedAvg plus (β−1) times the average of the sparse residual, and RPCA's identifiability theory does not certify the semantic labels. A third, milder issue is that the adaptive β = 1/E heuristic was designed from SVHN optimal-β measurements and then evaluated on SVHN, making the SVHN result partially in-sample; the other benchmarks provide out-of-dataset transfer. Self-citations (e.g., FedExp) are not load-bearing, and no uniqueness theorem is invoked. Overall, a moderate partial circularity of the explanatory diagnostics, with the headline accuracy comparison still independently benchmarked, warrants a score of 5.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The main free parameters are the LoRA rank and the RPCA hyperparameters, which are standard choices. The adaptive beta schedule is a heuristic fitted to observed trends during training. The key axioms are the RPCA recovery assumptions and the sparsity of client-specific knowledge, neither of which are structurally verified beyond qualitative similarity plots.

free parameters (4)
  • LoRA rank r = 4 for main experiments, 8 and 32 for ablations
    Chosen by the authors, not derived from data; affects expressiveness and the experiments.
  • RPCA lambda = 1/sqrt(max(d1,d2))
    A standard RPCA default, but it controls sparsity of S and is not tuned to the FL setting.
  • RPCA mu = numel(M)/(4*||M||_1)
    Standard RPCA default from Candes et al.; controls the ADMM step size.
  • Adaptive beta schedule beta(t)=1/E(t) = beta_A(t)=1/E_A(t), beta_B(t)=1/E_B(t)
    The inverse relation between optimal beta and E(t) is observed empirically in Figure 7 on SVHN and then adopted as a general heuristic; this is a data-driven choice.
assumptions (4)
  • domain assumption Robust PCA decomposes a matrix M into low-rank L and sparse S with recovery guarantees, assuming incoherence conditions.
    The paper assumes RPCA will correctly separate common and client-specific knowledge without verifying the incoherence conditions on real update matrices. Section 4, Eq. (9).
  • domain assumption Client-specific knowledge is sparse in the LoRA update space.
    The paper motivates this by citing sparse task vectors from model merging literature (TIES-Merging), but does not verify sparsity in the federated LoRA setting. Section 4.
  • domain assumption The common knowledge across clients is low-rank when stacking vectorized updates.
    This is the premise of the RPCA decomposition. Figure 1 shows high cosine similarity among low-rank components, but that is descriptive, not a recovery guarantee. Section 4.
  • domain assumption Averaging the low-rank components and scaled-averaging the sparse components is a valid operation for LoRA A and B matrices separately.
    LoRA updates are bilinear (B*A), but the paper follows standard FedAvg in averaging A and B separately, without accounting for the bilinear structure. The paper acknowledges this in Section 4 but does not address it in the experiments. Section 3, Eq. (4).

how reviews work

0 comments
Cite this review

Pith. "Pith review of FedRPCA: Enhancing Federated LoRA Aggregation Using Robust PCA." pith.science (2026). https://pith.science/paper/6KNOFNG6

@misc{pith2026250601194,
  author       = {Pith},
  title        = {Pith review of: FedRPCA: Enhancing Federated LoRA Aggregation Using Robust PCA},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6KNOFNG6}},
  note         = {Machine review of arXiv:2506.01194}
}
read the original abstract

LoRA has emerged as one of the most promising fine-tuning techniques, especially for federated learning (FL), since it significantly reduces communication and computation costs at resource-constrained clients. However, data heterogeneity remains a significant challenge for LoRA-based FL, and the conventional aggregation strategy based on FedAvg suffers from slow convergence and suboptimal accuracy. Motivated by recent advances in model merging, particularly Task Arithmetic, we explore the idea of aggregating client LoRA parameters using scaled averaging. We first observe that a naive application of Task Arithmetic is ineffective due to the high cosine similarity between client updates, indicating significant common knowledge in the updates across clients. To address this issue, we propose decomposing client LoRA updates via Robust Principal Component Analysis (Robust-PCA) into a common low-rank component and client-specific sparse components. Our proposed algorithm FedRPCA aggregates the low-rank components through averaging, consolidating common knowledge, and applies scaled averaging to the sparse components to amplify client-specific knowledge. We evaluate our approach across a variety of vision and language tasks and demonstrate that it achieves higher final accuracy and faster convergence compared to competing baselines.

Figures

Figures reproduced from arXiv: 2506.01194 by the authors.

Figure 1
Figure 1. Cosine similarity matrices for the original client updates (Figure 1a), extracted low-rank components [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Performance comparison of FedAvg, Task Arithmetic, and FedRPCA on a feder￾ated LoRA task. A direct application of Task Arithmetic underperforms due to over￾amplification of the common signal across clients. In contrast, FedRPCA decomposes client updates into common and client-specific com￾ponents, selectively amplifying only the latter resulting in improved stability and performance. A promising approach to address … view at source ↗
Figure 3
Figure 3. Visualization of a subset of the original [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Evolution of E (t) A , which quanti￾fies the relative contribution of the client￾specific component to the overall update across training rounds. The increasing trend in later rounds highlights the need for an adaptive choice of β for best per￾formance. Choice of β. To…
Figure 5
Figure 5. Figure 5: Performance for FedRPCA com￾bined with FedProx and SCAFFOLD. Combining RPCA with Local Optimization. One of the key strengths of FedRPCA is its modularity–it operates entirely at the server level without making assumptions about the client-side training process. This d…
Figure 6
Figure 6. Figure 6: Per-round wall-clock times for different algorithms when fine-tuning ViT￾B/32 on SVHN dataset distributed across 50 clients [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]
Figure 7
Figure 7. Figure 7: Performance of FedRPCA-style aggregation for different values of the scaling factor β at various training rounds when fine-tuning a ViT-B/32 model on the SVHN dataset with 50 clients. At each round t, E(t) denotes the average of E (t) A and E (t) B , computed across al…
Figure 8
Figure 8. Figure 8: Performance of FedRPCA with differ￾ent fixed β values and the proposed adaptive β schedule when fine-tuning ViT-B/32 on SVHN dataset distributed across 50 clients. The pro￾posed adaptive schedule achieves higher final accuracy than all fixed choice of β. As discussed i…
Figure 9
Figure 9. Figure 9: Performance of FedRPCA and baselines across different vision and language datasets. FedRPCA consistently outperforms baselines, especially for harder datasets like DTD and 20News. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]
Figure 10
Figure 10. Figure 10: Effect of heterogeneity (α) on the performance of FedRPCA and baselines. Improvement offered by FedRPCA increases as heterogeneity increases. FedRPCA FedAvg FedProx TIES SCAFFOLD Task Arithmetic MOON 0 20 40 60 80 100 Round No. 75 80 85 90 95 Test Accuracy (%) (a) 20 …
Figure 11
Figure 11. Figure 11: Effect of number of clients on the performance of [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]
Figure 12
Figure 12. Figure 12: Effect of LoRA fine-tuning rank on the performance of FedRPCA and baselines. As the rank increases, the accuracy gap between FedRPCA and other baselines decreases; nonetheless FedRPCA still converges faster. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]
Figure 13
Figure 13. Figure 13: Effect of heterogeneity (α) on the performance of FedRPCA and baselines when fine-tuning on 20News dataset. FedRPCA FedAvg FedProx TIES Task Arithmetic MOON 0 50 100 150 200 Round No. 40 45 50 55 60 65 Test Accuracy (%) (a) 20 clients 0 50 100 150 200 Round No. 40 45 …
Figure 14
Figure 14. Figure 14: Effect of number of clients on the performance of [PITH_FULL_IMAGE:figures/full_fig_p026_14.png]
Figure 15
Figure 15. Figure 15: Effect of LoRA fine-tuning rank on the performance of FedRPCA and baselines when fine-tuning on 20News dataset. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_15.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

97 extracted references · 52 canonical work pages

  1. [1]

    Slora: Federated parameter efficient fine-tuning of language models

    Sara Babakniya, Ahmed Roushdy Elkordy, Yahya H Ezzeldin, Qingfeng Liu, Kee-Bong Song, Mostafa El-Khamy, and Salman Avestimehr. Slora: Federated parameter efficient fine-tuning of language models. arXiv preprint arXiv:2308.06522, 2023. 11

    arXiv 2023

  2. [2]

    Federated fine-tuning of large language models under heterogeneous tasks and client resources.arXiv preprint arXiv:2402.11505, 2024

    Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, and Yaliang Li. Federated fine-tuning of large language models under heterogeneous tasks and client resources.arXiv preprint arXiv:2402.11505, 2024

    arXiv 2024

  3. [3]

    Fedalt: Federated fine-tuning through adaptive local training with rest-of-the-world lora.arXiv preprint arXiv:2503.11880, 2025

    Jieming Bian, Lei Wang, Letian Zhang, and Jie Xu. Fedalt: Federated fine-tuning through adaptive local training with rest-of-the-world lora.arXiv preprint arXiv:2503.11880, 2025

    arXiv 2025

  4. [4]

    Lora-fair: Federated lora fine-tuning with aggregation and initialization refinement, 2025

    Jieming Bian, Lei Wang, Letian Zhang, and Jie Xu. Lora-fair: Federated lora fine-tuning with aggregation and initialization refinement, 2025

    2025

  5. [5]

    Boerner, Stephen Deems, Thomas R

    Timothy J. Boerner, Stephen Deems, Thomas R. Furlani, Shelley L. Knuth, and John Towns. Access: Advancing innovation: Nsf’s advanced cyberinfrastructure coordination ecosystem: Services & support. InPractice and Experience in Advanced Research Computing 2023: Computing for the Common Good, PEARC ’23, page 173–176, New York, NY, USA, 2023. Association for ...

    2023

  6. [6]

    On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258, 2021

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258, 2021

    arXiv 2021

  7. [7]

    On the applications of robust pca in image and video processing.Proceedings of the IEEE, 106(8):1427–1457, 2018

    Thierry Bouwmans, Sajid Javed, Hongyang Zhang, Zhouchen Lin, and Ricardo Otazo. On the applications of robust pca in image and video processing.Proceedings of the IEEE, 106(8):1427–1457, 2018

    2018

  8. [8]

    Robust pca via principal component pursuit: A review for a comparative evaluation in video surveillance.Computer Vision and Image Understanding, 122:22–34, 2014

    Thierry Bouwmans and El Hadi Zahzah. Robust pca via principal component pursuit: A review for a comparative evaluation in video surveillance.Computer Vision and Image Understanding, 122:22–34, 2014

    2014

Show all 97 references
  1. [9]

    Distributed optimization and statistical learning via the alternating direction method of multipliers.Foundations and Trends® in Machine Learning, 3(1):1–122, 2011

    Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers.Foundations and Trends® in Machine Learning, 3(1):1–122, 2011

    2011

  2. [10]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  3. [11]

    Buehler and Markus J

    Eric L. Buehler and Markus J. Buehler. X-lora: Mixture of low-rank adapter experts, a flexible framework for large language models with applications in protein mechanics and design.CoRR, abs/2402.07148, 2024

    2024 arXiv

  4. [12]

    Learned robust PCA: A scalable deep unfolding approach for high-dimensional outlier detection

    HanQin Cai, Jialin Liu, and Wotao Yin. Learned robust PCA: A scalable deep unfolding approach for high-dimensional outlier detection. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors,Advances in Neural Information Processing Systems, 2021

    2021

  5. [13]

    Robust principal component analysis? Journal of the ACM (JACM), 58(3):1–37, 2011

    Emmanuel J Candès, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis? Journal of the ACM (JACM), 58(3):1–37, 2011

    2011

  6. [14]

    Candès, Xiaodong Li, Yi Ma, and John Wright

    Emmanuel J. Candès, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis?J. ACM, 58(3):11:1–11:37, 2011

    2011

  7. [15]

    Feddat: An approach for foundation model finetuning in multi-modal heterogeneous federated learning

    Haokun Chen, Yao Zhang, Denis Krompass, Jindong Gu, and Volker Tresp. Feddat: An approach for foundation model finetuning in multi-modal heterogeneous federated learning. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 11285–11293, 2024

    2024

  8. [16]

    On the importance and applicability of pre-training for federated learning

    Hong-You Chen, Cheng-Hao Tu, Ziwei Li, Han-Wei Shen, and Wei-Lun Chao. On the importance and applicability of pre-training for federated learning. InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023

    2023

  9. [17]

    Fedtune: A deep dive into efficient federated fine-tuning with pre-trained transformers.arXiv preprint arXiv:2211.08025, 2022

    Jinyu Chen, Wenchao Xu, Song Guo, Junxiao Wang, Jie Zhang, and Haozhao Wang. Fedtune: A deep dive into efficient federated fine-tuning with pre-trained transformers.arXiv preprint arXiv:2211.08025, 2022. 12

    2022 arXiv

  10. [18]

    Heterogeneous LoRA for federated fine-tuning of on-device foundation models

    Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi. Heterogeneous LoRA for federated fine-tuning of on-device foundation models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conference on Empirical Methods in Natural Langu...

    2024

  11. [19]

    AdapterSoup: Weight averaging to improve generalization of pretrained language models

    Alexandra Chronopoulou, Matthew Peters, Alexander Fraser, and Jesse Dodge. AdapterSoup: Weight averaging to improve generalization of pretrained language models. In Andreas Vlachos and Isabelle Augenstein, editors,Findings of the Association for Computational Linguistics: EACL...

    2023

  12. [20]

    Describing textures in the wild

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014, pages 3606–3613. IEEE Computer Society, 2014

    2014

  13. [21]

    Qlora: Efficient finetuning of quantized llms

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors,Advances in Neural Information Processing Systems 36: Annual C...

    2023

  14. [22]

    Parameter competition balancing for model merging

    Guodong Du, Junlin Lee, Jing Li, Runhua Jiang, Yifei Guo, Shuyang Yu, Hanting Liu, Sim Kuan Goh, Ho-Kin Tang, Daojing He, and Min Zhang. Parameter competition balancing for model merging. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  15. [23]

    Online robust pca via stochastic optimization

    Jiashi Feng, Huan Xu, and Shuicheng Yan. Online robust pca via stochastic optimization. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors,Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013

    2013

  16. [24]

    Online robust PCA via stochastic optimization

    Jiashi Feng, Huan Xu, and Shuicheng Yan. Online robust PCA via stochastic optimization. In Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger, editors,Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Informa...

  17. [25]

    Mixture-of-loras: An efficient multitasktuningmethodforlargelanguagemodels

    Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, Yu Han, and Hao Wang. Mixture-of-loras: An efficient multitasktuningmethodforlargelanguagemodels. InNicolettaCalzolari, Min-YenKan, VéroniqueHoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, editors,Proceedings of the 2024 Joint...

    2024

  18. [26]

    MRQA 2019 shared task: Evaluating generalization in reading comprehension

    Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen, editors,Proceedings of the 2nd Workshop on ...

    2019

  19. [27]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  20. [28]

    Godec+: Fast and robust low-rank matrix decomposition based on maximum correntropy.IEEE Trans

    Kailing Guo, Liu Liu, Xiangmin Xu, Dong Xu, and Dacheng Tao. Godec+: Fast and robust low-rank matrix decomposition based on maximum correntropy.IEEE Trans. Neural Networks Learn. Syst., 29(6):2323–2336, 2018. 13

    2018

  21. [29]

    Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.IEEE J

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens., 12(7):2217–2226, 2019

    2019

  22. [30]

    Measuring the effects of non-identical data distribution for federated visual classification.CoRR, abs/1909.06335, 2019

    Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification.CoRR, abs/1909.06335, 2019

    1909 arXiv

  23. [31]

    Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    2022

  24. [32]

    Fedqlora: Federated quantization- aware lora for large language models

    Zhiwei Hu, Liang Zhang, Shan Dai, Shihua Gong, and Qingjiang Shi. Fedqlora: Federated quantization- aware lora for large language models. 2025

    2025

  25. [33]

    EMR-merging: Tuning- free high-performance model merging

    Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang. EMR-merging: Tuning- free high-performance model merging. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  26. [34]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco Túlio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023

    2023

  27. [35]

    Initialization matters: Unraveling the impact of pre-training on federated learning.arXiv preprint arXiv:2502.08024, 2025

    Divyansh Jhunjhunwala, Pranay Sharma, Zheng Xu, and Gauri Joshi. Initialization matters: Unraveling the impact of pre-training on federated learning.arXiv preprint arXiv:2502.08024, 2025

    2025 arXiv

  28. [36]

    Fedexp: Speeding up federated averaging via extrapolation

    Divyansh Jhunjhunwala, Shiqiang Wang, and Gauri Joshi. Fedexp: Speeding up federated averaging via extrapolation. InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023

    2023

  29. [37]

    Promoting data and model privacy in federated learning through quantized LoRA

    Zhu JianHao, Changze Lv, Xiaohua Wang, Muling Wu, Wenhao Liu, Tianlong Li, Zixuan Ling, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. Promoting data and model privacy in federated learning through quantized LoRA. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, edito...

    2024

  30. [38]

    Dataless knowledge fusion by merg- ing weights of language models

    Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merg- ing weights of language models. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  31. [39]

    Advances and open problems in federated learning.Foundations and trends®in machine learning, 14(1–2):1–210, 2021

    Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning.Foundations and trends®in machine learning, 14(1–2)...

    2021

  32. [40]

    Reddi, Sebastian U

    Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. SCAFFOLD: stochastic controlled averaging for federated learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 J...

    2020

  33. [41]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. InProceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023

    2023

  34. [42]

    3d object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In2013 IEEE International Conference on Computer Vision Workshops, ICCV Workshops 2013, Sydney, Australia, December 1-8, 2013, pages 554–561. IEEE Computer Soci...

    2013

  35. [43]

    Federated lora with sparse communication

    Kevin Kuo, Arian Raje, Kousik Rajesh, and Virginia Smith. Federated lora with sparse communication. arXiv preprint arXiv:2406.05233, 2024

    2024 arXiv

  36. [44]

    Newsweeder: Learning to filter netnews

    Ken Lang. Newsweeder: Learning to filter netnews. Carnegie Mellon University, http://qwone.com/ ja- son/20Newsgroups/, 1995

    1995

  37. [45]

    Model-contrastive federated learning

    Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. InIEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 10713–10722. Computer Vision Foundation / IEEE, 2021

    2021

  38. [46]

    Federated optimization in heterogeneous networks

    Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In Inderjit S. Dhillon, Dimitris S. Papailiopoulos, and Vivienne Sze, editors,Proceedings of the Third Conference on Machine Learning...

    2020

  39. [47]

    Differentially private low-rank adaptation of large language model using federated learning.ACM Transactions on Management Information Systems, 16(2):1–24, 2025

    Xiao-Yang Liu, Rongyi Zhu, Daochen Zha, Jiechao Gao, Shan Zhong, Matt White, and Meikang Qiu. Differentially private low-rank adaptation of large language model using federated learning.ACM Transactions on Management Information Systems, 16(2):1–24, 2025

    2025

  40. [48]

    A survey on lora of large language models.Frontiers of Computer Science, 19(7):197605, 2025

    Yuren Mao, Yuhang Ge, Yijiang Fan, Wenyi Xu, Yu Mi, Zhonghao Hu, and Yunjun Gao. A survey on lora of large language models.Frontiers of Computer Science, 19(7):197605, 2025

    2025

  41. [49]

    Merging models with fisher-weighted averaging

    Michael Matena and Colin Raffel. Merging models with fisher-weighted averaging. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors,Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Syste...

    2022

  42. [50]

    Merging models with fisher-weighted averaging

    Michael S Matena and Colin Raffel. Merging models with fisher-weighted averaging. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors,Advances in Neural Information Processing Systems, 2022

    2022

  43. [51]

    Communication-efficient learning of deep networks from decentralized data

    Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. InArtificial intelligence and statistics, pages 1273–1282. PMLR, 2017

    2017

  44. [52]

    Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. InNIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011

    2011

  45. [53]

    John Nguyen, Jianyu Wang, Kshitiz Malik, Maziar Sanjabi, and Michael G. Rabbat. Where to begin? on the impact of pre-training and initialization in federated learning. InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023...

    2023

  46. [54]

    Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2023

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2023

    2023 arXiv

  47. [55]

    Fedpft: federated proxy fine-tuning of foundation models.arXiv preprint arXiv:2404.11536, 2024

    Zhaopeng Peng, Xiaoliang Fan, Yufan Chen, Zheng Wang, Shirui Pan, Chenglu Wen, Ruisheng Zhang, and Cheng Wang. Fedpft: federated proxy fine-tuning of foundation models.arXiv preprint arXiv:2404.11536, 2024

    2024 arXiv

  48. [56]

    Kakade, Eran Malach, and Samy Jelassi

    Akshara Prabhakar, Yuanzhi Li, Karthik Narasimhan, Sham M. Kakade, Eran Malach, and Samy Jelassi. Lora soups: Merging loras for practical skill composition tasks.CoRR, abs/2410.13025, 2024. 15

    2024 arXiv

  49. [57]

    Fdlora: person- alized federated learning of large language model via dual lora tuning.arXiv preprint arXiv:2406.07925, 2024

    Jiaxing Qi, Zhongzhi Luan, Shaohan Huang, Carol Fung, Hailong Yang, and Depei Qian. Fdlora: person- alized federated learning of large language model via dual lora tuning.arXiv preprint arXiv:2406.07925, 2024

    2024 arXiv

  50. [58]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...

    2021

  51. [59]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and ...

    2021

  52. [60]

    Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    2019

  53. [61]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.J. Mach. Learn. Res., 21:140:1–140:67, 2020

    2020

  54. [62]

    Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan

    Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. ...

    2021

  55. [63]

    Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950, 2023

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950, 2023

    2023 arXiv

  56. [65]

    Closed-form merging of parameter-efficient modules for federated continual learning

    Riccardo Salami, Pietro Buzzega, Matteo Mosconi, Jacopo Bonato, Luigi Sabetta, and Simone Calderara. Closed-form merging of parameter-efficient modules for federated continual learning. InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore...

    2025

  57. [66]

    Ziplora: Any subject in any style by effectively merging loras

    Viraj Shah, Nataniel Ruiz, Forrester Cole, Erika Lu, Svetlana Lazebnik, Yuanzhen Li, and Varun Jampani. Ziplora: Any subject in any style by effectively merging loras. 2023

    2023

  58. [67]

    Donald Shenaj, Ondrej Bohdal, Mete Ozay, Pietro Zanuttigh, and Umberto Michieli. Lora. rar: Learning to merge loras via hypernetworks for subject-style conditioned image generation.arXiv preprint arXiv:2412.05148, 2024

    2024 arXiv

  59. [68]

    Gonzalez, and Ion Stoica

    Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonzalez, and Ion Stoica. S-lora: Serving thousands of concurrent lora adapters, 2023

    2023

  60. [69]

    Fed- sb: A silver bullet for extreme communication efficiency and performance in (private) federated lora fine-tuning.arXiv preprint arXiv:2502.15436, 2025

    Raghav Singhal, Kaustubh Ponkshe, Rohit Vartak, Lav R Varshney, and Praneeth Vepakomma. Fed- sb: A silver bullet for extreme communication efficiency and performance in (private) federated lora fine-tuning.arXiv preprint arXiv:2502.15436, 2025. 16

    2025

  61. [70]

    Fedex-loRA: Exact aggregation for federated parameter-efficient fine-tuning of foundation models

    Raghav Singhal, Kaustubh Ponkshe, and Praneeth Vepakomma. Fedex-loRA: Exact aggregation for federated parameter-efficient fine-tuning of foundation models. InNeurIPS 2024 Workshop on Fine- Tuning in Modern Machine Learning: Principles and Scalability, 2024

    2024

  62. [71]

    Improving loRA in privacy-preserving federated learning

    Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving loRA in privacy-preserving federated learning. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  63. [72]

    Improving lora in privacy-preserving federated learning

    Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving lora in privacy-preserving federated learning. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024

    2024

  64. [73]

    Merging multi-task models via weight-ensembling mixture of experts

    Anke Tang, Li Shen, Yong Luo, Nan Yin, Lefei Zhang, and Dacheng Tao. Merging multi-task models via weight-ensembling mixture of experts. InForty-first International Conference on Machine Learning, 2024

    2024

  65. [74]

    Parameter- efficient multi-task model fusion with partial linearization

    Anke Tang, Li Shen, Yong Luo, Yibing Zhan, Han Hu, Bo Du, Yixin Chen, and Dacheng Tao. Parameter- efficient multi-task model fusion with partial linearization. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenR...

    2024

  66. [75]

    Task arithmetic through the lens of one-shot federated learning.arXiv preprint arXiv:2411.18607, 2024

    Zhixu Tao, Ian Mason, Sanjeev Kulkarni, and Xavier Boix. Task arithmetic through the lens of one-shot federated learning.arXiv preprint arXiv:2411.18607, 2024

    2024 arXiv

  67. [76]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  68. [77]

    Robust subspace learning: Robust pca, robust subspace tracking, and robust subspace recovery.IEEE signal processing magazine, 35(4):32–55, 2018

    Namrata Vaswani, Thierry Bouwmans, Sajid Javed, and Praneeth Narayanamurthy. Robust subspace learning: Robust pca, robust subspace tracking, and robust subspace recovery.IEEE signal processing magazine, 35(4):32–55, 2018

    2018

  69. [78]

    LoRA-flow: Dynamic LoRA fusion for large language models in generative tasks

    Hanqing Wang, Bowen Ping, Shuo Wang, Xu Han, Yun Chen, Zhiyuan Liu, and Maosong Sun. LoRA-flow: Dynamic LoRA fusion for large language models in generative tasks. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Associatio...

    2024

  70. [79]

    Lo- calizing task information for improved model merging and compression

    Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-Jimenez, François Fleuret, and Pascal Frossard. Lo- calizing task information for improved model merging and compression. InForty-first International Conference on Machine Learning, 2024

    2024

  71. [80]

    Federated fine-tuning for pre-trained foundation models over wireless networks.IEEE Transactions on Wireless Communications, 2025

    Zixin Wang, Yong Zhou, Yuanming Shi, and Khaled B Letaief. Federated fine-tuning for pre-trained foundation models over wireless networks.IEEE Transactions on Wireless Communications, 2025

    2025

  72. [81]

    FLoRA: Federated fine-tuning large language models with heterogeneous low-rank adaptations

    Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. FLoRA: Federated fine-tuning large language models with heterogeneous low-rank adaptations. InThe Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  73. [82]

    One communication round is all it needs for federated fine-tuning foundation models.arXiv preprint arXiv:2412.04650, 2024

    Ziyao Wang, Bowei Tian, Yexiao He, Zheyu Shen, Luyang Liu, and Ang Li. One communication round is all it needs for federated fine-tuning foundation models.arXiv preprint arXiv:2412.04650, 2024

    2024

  74. [83]

    Federated fine-tuning of llms on the very edge: The good, the bad, the ugly

    Herbert Woisetschläger, Alexander Erben, Shiqiang Wang, Ruben Mayer, and Hans-Arno Jacobsen. Federated fine-tuning of llms on the very edge: The good, the bad, the ugly. InProceedings of the Eighth Workshop on Data Management for End-to-End Machine Learning, pages 39–50, 2024

    2024

  75. [84]

    Mixture of lora experts

    Xun Wu, Shaohan Huang, and Furu Wei. Mixture of lora experts. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. 17

    2024

  76. [85]

    Robust pca via outlier pursuit

    Huan Xu, Constantine Caramanis, and Sujay Sanghavi. Robust pca via outlier pursuit. In J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta, editors,Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc., 2010

    2010

  77. [86]

    Raffel, and Mohit Bansal

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors,Advances in Neural Information Processing ...

    2023

  78. [87]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  79. [88]

    Representation surgery for multi-task model merging

    Enneng Yang, Li Shen, Zhenyi Wang, Guibing Guo, Xiaojun Chen, Xingwei Wang, and Dacheng Tao. Representation surgery for multi-task model merging. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, edito...

    2024

  80. [89]

    Adamerging: Adaptive model merging for multi-task learning

    Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  81. [90]

    Sa-fedlora: Adaptive parameter allocation for efficient federated learning with lora tuning.arXiv preprint arXiv:2405.09394, 2024

    Yuning Yang, Xiaohong Liu, Tianrun Gao, Xiaodong Xu, and Guangyu Wang. Sa-fedlora: Adaptive parameter allocation for efficient federated learning with lora tuning.arXiv preprint arXiv:2405.09394, 2024

    2024 arXiv

  82. [91]

    pfedlora: model-heterogeneous personalized federated learning with lora tuning.arXiv preprint arXiv:2310.13283, 2023

    Liping Yi, Han Yu, Gang Wang, Xiaoguang Liu, and Xiaoxiao Li. pfedlora: model-heterogeneous personalized federated learning with lora tuning.arXiv preprint arXiv:2310.13283, 2023

    2023 arXiv

  83. [92]

    Language models are super mario: Absorbing abilities from homologous models as a free lunch

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. InForty-first International Conference on Machine Learning, 2024

    2024

  84. [93]

    Towards building the federatedgpt: Federated instruction tuning

    Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. Towards building the federatedgpt: Federated instruction tuning. InICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), page...

    2024

  85. [94]

    Composing parameter-efficient modules with arithmetic operation.Advances in Neural Information Processing Systems, 36:12589–12610, 2023

    Jinghan Zhang, Junteng Liu, Junxian He, et al. Composing parameter-efficient modules with arithmetic operation.Advances in Neural Information Processing Systems, 36:12589–12610, 2023

    2023

  86. [95]

    Merging loras like playing LEGO: pushing the modularity of lora to extremes through rank-wise clustering.CoRR, abs/2409.16167, 2024

    Ziyu Zhao, Tao Shen, Didi Zhu, Zexi Li, Jing Su, Xuwu Wang, Kun Kuang, and Fei Wu. Merging loras like playing LEGO: pushing the modularity of lora to extremes through rank-wise clustering.CoRR, abs/2409.16167, 2024

    2024 arXiv

  87. [96]

    rest-of-the-world

    Tianyi Zhou and Dacheng Tao. Godec: Randomized lowrank & sparse matrix decomposition in noisy case. In Lise Getoor and Tobias Scheffer, editors,Proceedings of the 28th International Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, June 28 - July 2, 2011, p...

    2011

  88. [98]

    " " De co mp ose D into low - rank L and sparse S ( M = L + S )

    interpret adapters as vectors in weight space and compose new skills using scaled vector arithmetic. A third direction explores dynamic or mixture-of-experts (MoE) approaches, which retain multiple adapters and learn to select or weight them at inference time. Methods such asM...

  89. [2013]

    Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pages 404–412, 2013

    2013

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.