FedHL: Federated Learning for Heterogeneous Low-Rank Adaptation via Unbiased Aggregation

BoYuan Li; Guo Li; Jiandian Zeng; Shengbo Chen; Tian Wang; Zihao Peng

REVIEW 4 major objections 4 minor 53 references

FedHL: Federated Learning for Heterogeneous Low-Rank Adaptation via Unbiased Aggregation

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Federated LoRA with per-client ranks converges at the same rate as homogeneous LoRA when updates are anchored to the full-rank global model.

desk verdict The aggregation-baseline idea is worth keeping; the advertised O(1/√T) guarantee does not cover the adaptive algorithm as written. read the letter →

arxiv 2505.18494 v1 pith:UWGFFGV7 submitted 2025-05-24 cs.LG cs.AI

Zihao Peng , Jiandian Zeng , Boyuan Li , Guo Li , Shengbo Chen , Tian Wang This is my paper

classification cs.LGcs.AI

keywords federatedlearninglow-rankadaptationheterogeneousLoRAtruncationbiasconvergenceanalysisunbiasedaggregationparameter-efficientfine-tuninglargelanguagemodels

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Federated fine-tuning of large models with LoRA is cheap, but when clients use different ranks the server must truncate the shared global model to each client's rank, and this paper argues that the bias from that truncation is what stops existing methods from having convergence guarantees. The authors propose FedHL, which aggregates each client's low-rank update as a correction on top of the full-rank global model $W_t$ instead of on top of the client's truncated copy $W_t^{r_i}$, so the truncation error never enters the update as a persistent term. They prove that this restores an $\mathcal{O}(1/\sqrt{T})$ convergence rate, the same rate as homogeneous LoRA, and that the optimal aggregation weights for the bound are inversely proportional to each client's squared truncation error, $\hat{r}_i(t) = \|W_t - W_t^{r_i}\|_F^2$. Experiments on math, code, and instruction-following datasets with 10 and 100 clients report 1-3% gains over prior heterogeneous-LoRA baselines. If the analysis holds, it gives a simple design rule: keep the full-rank model as the reference point and weight clients by inverse truncation error, and clients are free to choose their own ranks without sacrificing convergence.

What carries the argument

The load-bearing construction is the unbiased aggregation baseline: the server keeps the full-rank model $W_t$ from the previous round and writes every client's upload as a correction on that same reference point, $W_{t+1}^i - W_t^{r_i}$, where $W_t^{r_i}$ is the rank-$r_i$ SVD truncation of $W_t$ that client $i$ actually trained on. Because all client corrections share one baseline, different ranks can be aggregated by simple weighted addition, with no zero-padding and no stacking tricks, and the truncation inconsistency between clients cancels out of the update. The second mechanism is the weight rule $p_i(t) = \frac{1/(\hat{r}_i(t)^2 + \epsilon)}{\sum_j 1/(\hat{r}_j(t)^2 + \epsilon)}$, derived by minimizing the slowest-decaying term of the convergence bound in Theorem 2; it down-weights exactly the clients whose truncation error is largest. The proof couples a standard FedAvg descent lemma with a per-client bound on the gap between the truncated local trajectory and the ideal full-rank trajectory, a gap that grows like $D_0^K \hat{r}_i^2$ with $D_0 = 4(1 + L^2\eta_t^2) \geq 4$ and $K$ local steps, showing why removing the baseline truncation matters for the rate.

What would settle it

Run FedHL on a small convex problem, for example a quadratic whose optimum and gradients are known, with two clients holding different ranks, and compare three variants: the fixed-weight version the theorem analyzes, the published per-round weight recomputation, and a truncated-baseline method such as FLoRA. If the adaptive version clearly violates the $\mathcal{O}(1/\sqrt{T})$ decrease of the averaged squared gradient norm, or if the truncated-baseline method matches the same rate, the claim that unbiased aggregation is what restores convergence would be falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that the poor convergence of earlier heterogeneous-LoRA federated methods is caused by their aggregation rule, not by the rank heterogeneity itself. Existing approaches such as FlexLoRA and FLoRA reconstruct the next global model by adding each client's local increment to that client's truncated baseline, $W_t^{r_i} = B_t^i A_t^i$; Theorem 1 shows that this injects a model-truncation-bias term into the bound that does not shrink as the number of rounds $T$ grows. FedHL replaces the per-client baselines with the stored full-rank model $W_t$, aggregating $W_{t+1} = \sum_i p_i(t)\,[W_t + (W_{t+1}^i - W_t^{r_i})]$, which cancels the truncation-bias term. Under standard smoothness and bounded-variance assumptions and with learning rate $\eta_t = \frac{1}{LK\sqrt{T}}$, Theorem 2 bounds the average squared gradient norm by $\mathcal{O}(1/\sqrt{T})$, matching the homogeneous-LoRA rate. Theorem 3 shows that the weights minimizing the dominant gradient-drift term are $p_i(t) \propto 1/(\hat{r}_i(t)^2 + \epsilon)$, so clients whose rank-constrained view of the global model is more accurate carry more weight. The paper positions this as the first formal convergence guarantee for heterogeneous LoRA in federated learning.

Load-bearing premise

The convergence proof assumes the aggregation weights $p_i$ are fixed across rounds, so that the objective $\sum_i p_i f_i(W)$ being optimized does not change; the algorithm as published recomputes the weights from the current truncation errors each round, and the stated $\mathcal{O}(1/\sqrt{T})$ guarantee does not formally cover that adaptive schedule.

Editorial extensions

If this is right

Clients can pick their own LoRA ranks to match device constraints; the server no longer needs padding or stacking to reconcile them, since aggregation happens on reconstructed full matrices.
The truncation-bias term vanishes from the convergence bound, so the $\mathcal{O}(1/\sqrt{T})$ rate is the same as for homogeneous LoRA; the remaining drift term is controlled by the inverse-truncation-error weights.
The weight rule is directly computable on the server from the SVD it already performs, so the fix costs no extra communication and no extra storage beyond the uploaded low-rank factors.
The method carries over to cross-device settings with partial participation (15% and 30% sampling among 100 clients) and remains competitive when ranks are homogeneous, so it does not trade away standard scenarios to fix the heterogeneous one.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural next step the paper leaves implicit is an analysis of the fully adaptive procedure: the published algorithm recomputes the $p_i(t)$ every round from the current truncation errors, while the theorem treats the weights as fixed, so a time-varying-objective or two-timescale argument would close the gap between the guarantee and the implemented rule.
The inverse-truncation-error weighting treats $\hat{r}_i$ as a proxy for update quality; one could test alternatives that use gradient norms or local validation loss, which may behave differently when a high-rank client holds noisy data.
The same construction, aggregating around the full shared object and weighting by each client's projection error, transfers to other parameter-efficient federated methods such as adapters or prompt tuning, where client-specific projections also create alignment bias.
Because the truncation error is measured in the Frobenius norm, the optimal weights implicitly assume SVD truncation; a different rank-reduction scheme, such as random projection or rank allocated by layer importance, would change both the error profile and the optimal weighting, which the paper does not explore.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Zihao and co.'s FedHL has a genuinely useful algorithmic idea: use the full-rank global model as the aggregation baseline, and weight clients inversely to their truncation error. The experiments, while modest, consistently show 1–3% gains over three hetero-LoRA baselines across two FL settings. That part is credible.

The problem is the central convergence claim. Theorem 2 proves O(1/√T) for a version of the method with fixed aggregation weights p_i and fixed truncation errors ĥr_i. Algorithm 1 recomputes both every round (Eq. 15, line 17). The proof's descent lemma (Lemma 2) and the telescoping sum require a single time-invariant objective f(W)=Σ_i p_i f_i(W). With p_i(t) varying, the bound's coefficient no longer telescopes, and no argument shows the time-averaged weight-truncation product decays. So the guarantee does not cover the published algorithm. The stress-test note is right.

There's a second gap. The proof models local LoRA updates as full-matrix gradient descent from a truncated starting point. In reality clients update low-rank factors B_i, A_i; the local step is a projected or reparameterized update, not full gradient descent on the full matrix. The proof's Γτ recursion does not capture that.

A smaller but real problem: Remark 2 calls the O(1/√T) truncation term in Theorem 1 a 'persistent' bias that 'persists without guaranteed reduction.' That term is explicitly 1/√T-scaled, so it vanishes as T grows. The remark misreads the paper's own bound. This weakens the narrative that existing methods are non-convergent; Theorem 1 as written actually gives them an O(1/√T) rate, with a larger constant.

There are also minor loose ends: the ε in Eq. (15) is defined in Theorem 3 but set as a free hyperparameter in practice, and the softmax smoothing alters the derived weights.

Who is this for? People building hetero-LoRA FL systems will find the aggregation-baseline idea and the weighting scheme useful. The empirical section gives a reasonable sanity check. But readers who rely on the convergence theorem will be misled.

My recommendation: don't publish as is. The theory needs to be either corrected to cover adaptive weights (or explicitly downgraded to a heuristic), and the local-update modeling needs to match the algorithm. That's a substantial revision, but the core idea has enough merit that I'd send it to referees rather than desk-reject.

Referee Report

4 major / 4 minor

Summary. The paper proposes FedHL, a federated learning method for heterogeneous LoRA fine-tuning. The server aggregates client updates on top of the full-rank global model W_t rather than the truncated rank-r_i model, and assigns each client an adaptive aggregation weight p_i(t) that is inversely proportional to the client's current truncation error. The authors claim an O(1/sqrt(T)) convergence rate, derive closed-form 'optimal' aggregation weights in Theorem 3, and report 1--3% improvements over Zero-Padding, FlexLoRA, and FLoRA on three federated LLM benchmarks. The central contribution is the claimed formal convergence guarantee for heterogeneous LoRA, which is said to be lacking in prior work.

Significance. If the convergence guarantee were rigorously established, this would be a useful contribution: the unbiased full-rank aggregation baseline is a simple and plausible fix for truncation bias, and the paper provides a rare theoretical treatment of heterogeneous LoRA in FL. The experimental evaluation is reasonably broad, covering cross-silo and cross-device settings, varying rank heterogeneity, and per-client performance. However, the theoretical proof is the main advertised result, and as detailed below, it does not cover the algorithm as published. The paper's value is therefore substantially weakened unless the proof is repaired or the claims are scaled back.

major comments (4)

[Section V / Appendix VIII, Theorem 2 and Algorithm 1] The convergence proof treats the client weights p_i and truncation errors as constants independent of t, but the published algorithm updates both every round. Lemma 2 (Eq. (29)) is a descent inequality for a fixed objective f(W) = Σ_i p_i f_i(W), and the telescoping sum in Eq. (38) relies on a single fixed f across all rounds. In Algorithm 1, line 17, and Eq. (15), p_i(t) is recomputed from the current truncation error r-hat_i(t) = ||W_t - W_t^{r_i}||^2, which itself changes with W_t. Thus the algorithm minimizes a time-varying weighted objective, and the telescoping argument cannot be applied. The manuscript itself concedes in Section V that 'the definition of r-hat_i, which depends on t, is simplified in the notation for the convergence analysis.' As a result, Theorem 2 proves O(1/sqrt(T)) convergence only for an idealized fixed-weight, fixed-truncation-error version, not for FedHL as implemented. The same issue affects Theorem 3, since the 'optimal' weights are derived for a bound that is itself not a valid bound for the adaptive algorithm.
[Appendix VIII, Eq. (17) and local LoRA training] The proof models each client's local update as full-matrix gradient descent on W, writing W_{t+1} = Σ_i p_i (W_t - η_t Σ_τ ∇̃ f_i(W_{t,τ}^{r_i})). In the actual LoRA procedure, however, the client optimizes two low-rank matrices B_t^i and A_t^i with W = W_t^{r_i} + BA, and the effective update is a low-rank increment rather than the unconstrained full-matrix gradient of f_i. The proof never relates the LoRA-constrained update to the full-matrix gradient update it analyzes. This is not a cosmetic mismatch: the space of achievable local updates differs, and the claimed convergence guarantee for heterogeneous LoRA training therefore does not follow from the proof, even if the aggregation weights were fixed.
[Appendix VIII, Lemma 1, Eq. (28)] In bounding T1 = E⟨∇f(V_{t+1}), W_{t+1} - V_{t+1}⟩, the proof drops the term (1/2) E||∇f(V_{t+1})||^2 with the assertion that in standard FedAvg this term decreases over time and asymptotically approaches zero. This is not a quantitative bound, and the term is not negligible in the preceding inequality: it can contribute constants or terms that decay more slowly than O(1/sqrt(T)). Since Lemma 1 feeds directly into Eq. (37) and the final rate, dropping this term without a bound leaves the O(1/sqrt(T)) conclusion unsupported. A rigorous treatment must either bound E||∇f(V_{t+1})||^2 in terms of the quantities already appearing in the descent inequality or subtract it with a controlled coefficient.
[Appendix VIII, Eq. (32)-(33), Lemma 2] The recursion for Δ_τ in Lemma 2 uses the factor 1/(K-1) and 1/(2K-1). These denominators are undefined for K=1. The paper does not state a K≥2 assumption, and its own experiments use K=1 for Fed-CodeAlpaca (Section VI.A). Thus Theorem 2, as proven, does not cover a setting the paper evaluates. The theorem needs an explicit K≥2 condition, or a separate argument for the K=1 case.

minor comments (4)

[Section VI.A, hyperparameters] The paper states that aggregation weights are smoothed using a softmax function with temperature 1, but Theorem 3 and the convergence analysis are for the unsmoothed formula in Eq. (15). Please clarify whether the experimental algorithm uses exactly Eq. (15) or a softmax modification, and if the latter, how the theoretical guarantee is affected.
[Appendix X, Eq. (49)-(50)] The constant ε in Eq. (15) is defined as B/A with A and B depending on problem constants, but in Theorem 3 and Algorithm 1 it is treated as a free parameter. The paper does not specify how ε is chosen in practice, which weakens the claim that the weights in Eq. (15) are the theoretically optimal ones used in the experiments.
[Notation, Eq. (5)-(7)] Eq. (5) defines W_{t+1} = Σ_i p_i W_{t+1}^i, while Eq. (6) rewrites this as Σ_i p_i (W_t^{r_i} + ΔW_t^i) and Eq. (7) as Σ_i p_i (W_t + ΔW_t^i). The relationship between the p_i in these equations and the p_i(t) used later is not explained consistently; please align the notation.
[References] References [10] and [23] appear to be the same AdaLoRA paper under different titles; please consolidate or disambiguate. Also, reference [43] is listed with a different title than [41] but appears to be the same arXiv ID; please check.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: Theorem 3's weights minimize the paper's own bound (standard bound-optimization), and Theorem 2's proof does not assume its conclusion; the main gap (fixed p_i vs adaptive p_i(t)) is a correctness issue, not circularity.

full rationale

The claimed derivation chain is not circular. Theorem 2 derives an O(1/sqrt(T)) bound from Assumptions 1-3 using standard smoothness/SGD inequalities; the bound does not assume the convergence conclusion. Theorem 3 minimizes a term of that bound (the drift term sum_i p_i^2 r_hat_i^2) subject to sum_i p_i = 1, giving p_i^* proportional to 1/(r_hat_i^2 + epsilon) via Lagrange multipliers; optimizing a derived upper bound is a legitimate design step, not an inclusion of the target result. The regularization epsilon > 0 is introduced to make the objective well-posed and is a parameterization choice. The only self-citation of a co-author ([42], FedFMSL) appears in Remark 3 as a consistency comparison for the homogeneous-LoRA rate and is not load-bearing; the paper's own Theorem 2 supplies the rate. The manuscript does contain an explicit proof-coverage limitation: 'the definition of r_hat_i, which depends on t, is simplified in the notation for the convergence analysis' (Section V), and Algorithm 1/Eq. (15) recompute p_i(t) from r_hat_i(t) each round, whereas Lemmas 1-2 and the telescoping sum in Eq. (37)-(38) treat p_i and r_hat_i as fixed. This means Theorem 2 as stated does not cover the adaptive-weight algorithm as published, but this is a gap between an idealized proof and the implemented algorithm, not a reduction of the prediction to its inputs. No equation in the paper is equal to its own input by construction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard smoothness and noise assumptions plus two unstated simplifications: fixed client weights in the analysis despite adaptive weights in the algorithm, and full-matrix gradient updates despite low-rank LoRA training. The inverse-error weighting formula also depends on an unspecified regularizer eps. These are the main load-bearing assumptions beyond standard FL theory.

free parameters (2)

eps in Eq. (15) = not specified
Regularizer in the inverse-truncation-error weighting formula. The theorem defines it as a ratio of unknown constants (B/A), but the paper gives no practical value or selection rule, and it is needed to prevent division by zero when truncation error is zero.
softmax temperature for aggregation smoothing = 1
Chosen by hand to moderate extreme aggregation weights; no sensitivity analysis or stated justification.

assumptions (4)

domain assumption Assumptions 1-3: L-smooth loss, unbiased local gradients with bounded variance sigma_l^2, and bounded gradient heterogeneity sigma_g^2.
Standard federated learning assumptions, stated in Section V and used throughout Lemmas 1-4 and Theorems 1-2.
ad hoc to paper Client weights p_i are fixed during the convergence proof even though Algorithm 1 updates p_i(t) each round.
Lemma 2 and the telescoping sum in Eq. (37) rely on a fixed global objective f(W)=Σ p_i f_i(W). The algorithm described in Section IV.C and Eq. (15) changes p_i dynamically, so the proof does not cover the published method.
ad hoc to paper Local LoRA updates can be modeled as full-matrix gradient descent on W in Eq. (17).
The proof writes the local model change as -η Σ ∇f_i(W_{t,τ}^{r_i}), but Algorithm 1 trains B_i,A_i; the actual update lies in a low-rank subspace and is a projected gradient step. No chain-rule analysis is provided.
ad hoc to paper The term E||∇f(V_{t+1})||^2 can be neglected in Lemma 1.
Inequality (c) in Eq. (28) uses '≲' and cites the standard behavior of FedAvg without a finite-time bound or rigorous absorption into other terms, so the stated bound is not fully derived.

how reviews work

0 comments

Cite this review

Pith. "Pith review of FedHL: Federated Learning for Heterogeneous Low-Rank Adaptation via Unbiased Aggregation." pith.science (2026). https://pith.science/paper/UWGFFGV7

@misc{pith2026250518494,
  author       = {Pith},
  title        = {Pith review of: FedHL: Federated Learning for Heterogeneous Low-Rank Adaptation via Unbiased Aggregation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UWGFFGV7}},
  note         = {Machine review of arXiv:2505.18494}
}

abstract

Federated Learning (FL) facilitates the fine-tuning of Foundation Models (FMs) using distributed data sources, with Low-Rank Adaptation (LoRA) gaining popularity due to its low communication costs and strong performance. While recent work acknowledges the benefits of heterogeneous LoRA in FL and introduces flexible algorithms to support its implementation, our theoretical analysis reveals a critical gap: existing methods lack formal convergence guarantees due to parameter truncation and biased gradient updates. Specifically, adapting client-specific LoRA ranks necessitates truncating global parameters, which introduces inherent truncation errors and leads to subsequent inaccurate gradient updates that accumulate over training rounds, ultimately degrading performance. To address the above issues, we propose \textbf{FedHL}, a simple yet effective \textbf{Fed}erated Learning framework tailored for \textbf{H}eterogeneous \textbf{L}oRA. By leveraging the full-rank global model as a calibrated aggregation basis, FedHL eliminates the direct truncation bias from initial alignment with client-specific ranks. Furthermore, we derive the theoretically optimal aggregation weights by minimizing the gradient drift term in the convergence upper bound. Our analysis shows that FedHL guarantees $\mathcal{O}(1/\sqrt{T})$ convergence rate, and experiments on multiple real-world datasets demonstrate a 1-3\% improvement over several state-of-the-art methods.

Figures

Figures reproduced from arXiv: 2505.18494 by the authors.

**Figure 2.** Overview of FedHL. In heterogeneous LoRA, compared to ideal full-rank updates, truncation and gradient biases emerge, as illustrated in the upper [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Training Loss for LoRA-Based Configurations under Moderate and [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 6.** Figure 6: Comparison of Client Model Scores Across Different LoRA Ranks [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

**Figure 7.** Figure 7: Training Loss for LoRA-Based Configurations under Moderate and [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

**Figure 9.** Figure 9: Performance comparison under client participation rates (15% vs. [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 30 canonical work pages

[1]

When foundation model meets federated learning: Motivations, challenges, and future directions,

W. Zhuang, C. Chen, and L. Lyu, “When foundation model meets federated learning: Motivations, challenges, and future directions,”arXiv preprint arXiv:2306.15546, 2023

arXiv 2023
[2]

Advances and open challenges in federated learning with foundation models,

C. Ren, H. Yu, H. Peng, X. Tang, A. Li, Y . Gao, A. Z. Tan, B. Zhao, X. Li, Z. Liet al., “Advances and open challenges in federated learning with foundation models,”arXiv preprint arXiv:2404.15381, 2024

arXiv 2024
[3]

Communication-efficient learning of deep networks from decentralized data,

B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” inArtificial intelligence and statistics. PMLR, 2017, pp. 1273– 1282

2017
[4]

Review on security of federated learning and its application in healthcare,

H. Li, C. Li, J. Wang, A. Yang, Z. Ma, Z. Zhang, and D. Hua, “Review on security of federated learning and its application in healthcare,” Future Generation Computer Systems, vol. 144, pp. 271–290, 2023

work page 2023
[5]

Ten challenging problems in federated founda- tion models,

T. Fan, H. Gu, X. Cao, C. S. Chan, Q. Chen, Y . Chen, Y . Feng, Y . Gu, J. Geng, B. Luoet al., “Ten challenging problems in federated founda- tion models,”IEEE Transactions on Knowledge and Data Engineering, 2025

2025
[6]

Where to begin? on the impact of pre-training and initialization in federated learning,

J. Nguyen, J. Wang, K. Malik, M. Sanjabi, and M. Rabbat, “Where to begin? on the impact of pre-training and initialization in federated learning,”arXiv preprint arXiv:2206.15387, 2022

arXiv 2022
[7]

Promptfl: Let federated participants cooperatively learn prompts instead of models – federated learning in age of foundation model,

T. Guo, S. Guo, J. Wang, X. Tang, and W. Xu, “Promptfl: Let federated participants cooperatively learn prompts instead of models – federated learning in age of foundation model,”IEEE Transactions on Mobile Computing, vol. 23, no. 5, pp. 5179–5194, 2024

work page 2024
[8]

Fedbiot: Llm local fine-tuning in federated learning without full model,

F. Wu, Z. Li, Y . Li, B. Ding, and J. Gao, “Fedbiot: Llm local fine-tuning in federated learning without full model,” inProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024, pp. 3345–3355

work page 2024

Show all 53 references

[9]

Lora: Low-rank adaptation of large language models,

E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685, 2021

2021 arXiv
[10]

Adalora: Adaptive budget allocation for parameter-efficient fine-tuning,

Q. Zhang, M. Chen, A. Bukharin, N. Karampatziakis, P. He, Y . Cheng, W. Chen, and T. Zhao, “Adalora: Adaptive budget allocation for parameter-efficient fine-tuning,”arXiv preprint arXiv:2303.10512, 2023

2023 arXiv
[11]

Low-parameter federated learning with large language models,

J. Jiang, H. Jiang, Y . Ma, X. Liu, and C. Fan, “Low-parameter federated learning with large language models,” inInternational Conference on Web Information Systems and Applications. Springer, 2024, pp. 319– 330

2024
[12]

Fedlore: Communication-efficient and personalized edge intelligence framework via federated low-rank estimation,

Z. Shao, B. Li, P. Wang, Y . Zhang, and K.-K. R. Choo, “Fedlore: Communication-efficient and personalized edge intelligence framework via federated low-rank estimation,”IEEE Transactions on Parallel and Distributed Systems, vol. 36, no. 5, pp. 994–1010, 2025. 17

2025
[13]

Towards building the federatedgpt: Federated instruction tun- ing,

J. Zhang, S. Vahidian, M. Kuo, C. Li, R. Zhang, T. Yu, G. Wang, and Y . Chen, “Towards building the federatedgpt: Federated instruction tun- ing,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 6915–6919

2024
[14]

Heterogeneous lora for federated fine-tuning of on-device foundation models,

Y . J. Cho, L. Liu, Z. Xu, A. Fahrezi, M. Barnes, and G. Joshi, “Heterogeneous lora for federated fine-tuning of on-device foundation models,” inInternational Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023, 2023

2023
[16]

Heterogeneous low- rank approximation for federated fine-tuning of on-device foundation models,

Y . J. Cho, L. Liu, Z. Xu, A. Fahrezi, and G. Joshi, “Heterogeneous low- rank approximation for federated fine-tuning of on-device foundation models,”arXiv preprint arXiv:2401.06432, 2024

2024 arXiv
[17]

Federated fine-tuning of large language models under heterogeneous tasks and client resources,

J. Bai, D. Chen, B. Qian, L. Yao, and Y . Li, “Federated fine-tuning of large language models under heterogeneous tasks and client resources,” inThe Thirty-eighth Annual Conference on Neural Information Process- ing Systems, 2024

2024
[18]

Federated sketching lora: On-device collaborative fine-tuning of large language models,

W. Fang, D.-J. Han, L. Yuan, S. Hosseinalipour, and C. G. Brinton, “Federated sketching lora: On-device collaborative fine-tuning of large language models,”arXiv preprint arXiv:2501.19389, 2025

2025 arXiv
[19]

Smooth diffusion: Crafting smooth latent spaces in diffusion models,

J. Guo, X. Xu, Y . Pu, Z. Ni, C. Wang, M. Vasu, S. Song, G. Huang, and H. Shi, “Smooth diffusion: Crafting smooth latent spaces in diffusion models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 7548–7558

2024
[20]

Autore: Document-level relation extraction with large language models,

L. Xue, D. Zhang, Y . Dong, and J. Tang, “Autore: Document-level relation extraction with large language models,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), 2024, pp. 211–220

2024
[21]

mplug-owl: Modularization empowers large language models with multimodality,

Q. Ye, H. Xu, G. Xu, J. Ye, M. Yan, Y . Zhou, J. Wang, A. Hu, P. Shi, Y . Shiet al., “mplug-owl: Modularization empowers large language models with multimodality,”arXiv preprint arXiv:2304.14178, 2023

2023 arXiv
[22]

Qlora: Efficient finetuning of quantized llms,

T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,”Advances in Neural Information Processing Systems, vol. 36, 2024

2024
[23]

Adaptive budget allocation for parameter-efficient fine-tuning,

Q. Zhang, M. Chen, A. Bukharin, P. He, Y . Cheng, W. Chen, and T. Zhao, “Adaptive budget allocation for parameter-efficient fine-tuning,” inThe Eleventh International Conference on Learning Representations, 2023. [Online]. Available: https://openreview.net/ forum?id=lq62uWRJjiY

2023
[24]

LoRA+: Efficient low rank adaptation of large models,

S. Hayou, N. Ghosh, and B. Yu, “LoRA+: Efficient low rank adaptation of large models,” inForty-first International Conference on Machine Learning, 2024. [Online]. Available: https://openreview.net/forum?id= NEv8YqBROO

2024
[25]

Lorahub: Efficient cross-task generalization via dynamic lora composition,

C. Huang, Q. Liu, B. Y . Lin, T. Pang, C. Du, and M. Lin, “Lorahub: Efficient cross-task generalization via dynamic lora composition,”arXiv preprint arXiv:2307.13269, 2023

2023 arXiv
[26]

A kernel-based view of language model fine-tuning,

S. Malladi, A. Wettig, D. Yu, D. Chen, and S. Arora, “A kernel-based view of language model fine-tuning,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 23 610–23 641

2023
[27]

The expressive power of low-rank adaptation,

Y . Zeng and K. Lee, “The expressive power of low-rank adaptation,” inThe Twelfth International Conference on Learning Representations,
[28]

Lora training in the ntk regime has no spurious local minima,

U. Jang, J. D. Lee, and E. K. Ryu, “Lora training in the ntk regime has no spurious local minima,”arXiv preprint arXiv:2402.11867, 2024

2024 arXiv
[29]

Federated large language model: Solutions, challenges and future directions,

J. Hu, D. Wang, Z. Wang, X. Pang, H. Xu, J. Ren, and K. Ren, “Federated large language model: Solutions, challenges and future directions,”IEEE Wireless Communications, pp. 1–8, 2024

2024
[30]

A survey on lora of large language models,

Y . Mao, Y . Ge, Y . Fan, W. Xu, Y . Mi, Z. Hu, and Y . Gao, “A survey on lora of large language models,”arXiv preprint arXiv:2407.11046, 2024

2024 arXiv
[31]

Low-parameter federated learning with large language models,

J. Jiang, X. Liu, and C. Fan, “Low-parameter federated learning with large language models,”arXiv preprint arXiv:2307.13896, 2023

2023 arXiv
[32]

Fdlora: Personalized federated learning of large language model via dual lora tuning,

J. Qi, Z. Luan, S. Huang, C. Fung, H. Yang, and D. Qian, “Fdlora: Personalized federated learning of large language model via dual lora tuning,”arXiv preprint arXiv:2406.07925, 2024

2024 arXiv
[33]

Hyperflora: Federated learning with instantaneous personalization,

Q. Lu, D. Niu, M. S. Khoshkho, and B. Li, “Hyperflora: Federated learning with instantaneous personalization,” inProceedings of the 2024 SIAM International Conference on Data Mining (SDM). SIAM, 2024, pp. 824–832

2024
[34]

Improving loRA in privacy-preserving federated learning,

Y . Sun, Z. Li, Y . Li, and B. Ding, “Improving loRA in privacy-preserving federated learning,” inThe Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=NLPzL6HWNl

2024
[35]

Differentially private low-rank adaptation of large language model us- ing federated learning,

X.-Y . Liu, R. Zhu, D. Zha, J. Gao, S. Zhong, M. White, and M. Qiu, “Differentially private low-rank adaptation of large language model us- ing federated learning,”ACM Transactions on Management Information Systems, vol. 16, no. 2, pp. 1–24, 2025

2025
[36]

Federated lora with sparse communication,

K. Kuo, A. Raje, K. Rajesh, and V . Smith, “Federated lora with sparse communication,”arXiv preprint arXiv:2406.05233, 2024

2024 arXiv
[37]

Federated fine-tuning for pre-trained foundation models over wireless networks,

Z. Wang, Y . Zhou, Y . Shi, and K. B. Letaief, “Federated fine-tuning for pre-trained foundation models over wireless networks,”IEEE Trans- actions on Wireless Communications, vol. 24, no. 4, pp. 3450–3464, 2025

2025
[38]

Fed- erated low-rank adaptation for large models fine-tuning over wireless networks,

H. Sun, H. Tian, W. Ni, J. Zheng, D. Niyato, and P. Zhang, “Fed- erated low-rank adaptation for large models fine-tuning over wireless networks,”IEEE Transactions on Wireless Communications, vol. 24, no. 1, pp. 659–675, 2025

2025
[39]

Helora: Lora-heterogeneous federated fine-tuning for foundation models,

B. Fan, X. Su, S. Tarkoma, and P. Hui, “Helora: Lora-heterogeneous federated fine-tuning for foundation models,”ACM Transactions on Internet Technology, 2025

2025
[40]

FLoRA: Federated fine-tuning large language models with heterogeneous low-rank adaptations,

Z. Wang, Z. Shen, Y . He, G. Sun, H. Wang, L. Lyu, and A. Li, “FLoRA: Federated fine-tuning large language models with heterogeneous low-rank adaptations,” inThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [Online]. Available: https://openre...

2024
[42]

Fedfmsl: Federated learning of foundation models with sparsely activated lora,

P. Wu, K. Li, T. Wang, Y . Dong, V . C. M. Leung, and F. Wang, “Fedfmsl: Federated learning of foundation models with sparsely activated lora,” IEEE Transactions on Mobile Computing, vol. 23, no. 12, pp. 15 167– 15 181, 2024

2024
[43]

Fedlora: Model-heterogeneous personalized federated learning with lora tuning,

L. Yi, H. Yu, G. Wang, and X. Liu, “Fedlora: Model-heterogeneous personalized federated learning with lora tuning,”arXiv preprint arXiv:2310.13283, 2023

2023 arXiv
[44]

Training verifiers to solve math word problems,

K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakanoet al., “Training verifiers to solve math word problems,”arXiv preprint arXiv:2110.14168, 2021

2021 arXiv
[45]

Code alpaca: an instruction-following llama model for code generation, 2023,

S. Chaudhary, “Code alpaca: an instruction-following llama model for code generation, 2023,”URL https://github. com/sahil280114/codealpaca, 2023

2023
[46]

Evaluating large language models trained on code,

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockmanet al., “Evaluating large language models trained on code,”arXiv preprint arXiv:2107.03374, 2021

2021 arXiv
[47]

Dolly: Democratizing the magic of chatgpt with open models,

Databricks, “Dolly: Democratizing the magic of chatgpt with open models,” 2023, databricks Blog. https://www.databricks.com/blog

2023
[48]

Federated full-parameter tuning of billion-sized language models with communication cost under 18 kilobytes,

Z. Qin, D. Chen, B. Qian, B. Ding, Y . Li, and S. Deng, “Federated full-parameter tuning of billion-sized language models with communication cost under 18 kilobytes,” inForty-first International Conference on Machine Learning, 2024. [Online]. Available: https: //openreview.net...

2024
[49]

Llama 2: Open foundation and fine-tuned chat models,

H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosaleet al., “Llama 2: Open foundation and fine-tuned chat models,”arXiv preprint arXiv:2307.09288, 2023

2023 arXiv
[50]

Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning,

W. Kuang, B. Qian, Z. Li, D. Chen, D. Gao, X. Pan, Y . Xie, Y . Li, B. Ding, and J. Zhou, “Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning,” inProceed- ings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Dat...

2024
[51]

Ferret: Federated full-parameter tuning at scale for large language models,

Y . Shu, W. Hu, S.-K. Ng, B. K. H. Low, and F. R. Yu, “Ferret: Federated full-parameter tuning at scale for large language models,” inInternational Workshop on Federated Foundation Models in Conjunction with NeurIPS 2024, 2024. [Online]. Available: https://openreview.net/forum...

2024
[52]

On the convergence of zeroth-order federated tuning for large language models,

Z. Ling, D. Chen, L. Yao, Y . Li, and Y . Shen, “On the convergence of zeroth-order federated tuning for large language models,” inProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024, pp. 1827–1838

2024
[53]

Local sgd converges fast and communicates little,

S. U. Stich, “Local sgd converges fast and communicates little,” inICLR 2019-International Conference on Learning Representations, 2019

2019
[54]

Parallel restarted sgd with faster con- vergence and less communication: Demystifying why model averaging works for deep learning,

H. Yu, S. Yang, and S. Zhu, “Parallel restarted sgd with faster con- vergence and less communication: Demystifying why model averaging works for deep learning,” inProceedings of the AAAI conference on artificial intelligence, vol. 33, no. 01, 2019, pp. 5693–5700

2019
[2024]

Available: https://openreview.net/forum?id=likXVjmh3E

[Online]. Available: https://openreview.net/forum?id=likXVjmh3E

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

When foundation model meets federated learning: Motivations, challenges, and future directions,

W. Zhuang, C. Chen, and L. Lyu, “When foundation model meets federated learning: Motivations, challenges, and future directions,”arXiv preprint arXiv:2306.15546, 2023

arXiv 2023

[2] [2]

Advances and open challenges in federated learning with foundation models,

C. Ren, H. Yu, H. Peng, X. Tang, A. Li, Y . Gao, A. Z. Tan, B. Zhao, X. Li, Z. Liet al., “Advances and open challenges in federated learning with foundation models,”arXiv preprint arXiv:2404.15381, 2024

arXiv 2024

[3] [3]

Communication-efficient learning of deep networks from decentralized data,

B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” inArtificial intelligence and statistics. PMLR, 2017, pp. 1273– 1282

2017

[4] [4]

Review on security of federated learning and its application in healthcare,

H. Li, C. Li, J. Wang, A. Yang, Z. Ma, Z. Zhang, and D. Hua, “Review on security of federated learning and its application in healthcare,” Future Generation Computer Systems, vol. 144, pp. 271–290, 2023

work page 2023

[5] [5]

Ten challenging problems in federated founda- tion models,

T. Fan, H. Gu, X. Cao, C. S. Chan, Q. Chen, Y . Chen, Y . Feng, Y . Gu, J. Geng, B. Luoet al., “Ten challenging problems in federated founda- tion models,”IEEE Transactions on Knowledge and Data Engineering, 2025

2025

[6] [6]

Where to begin? on the impact of pre-training and initialization in federated learning,

J. Nguyen, J. Wang, K. Malik, M. Sanjabi, and M. Rabbat, “Where to begin? on the impact of pre-training and initialization in federated learning,”arXiv preprint arXiv:2206.15387, 2022

arXiv 2022

[7] [7]

Promptfl: Let federated participants cooperatively learn prompts instead of models – federated learning in age of foundation model,

T. Guo, S. Guo, J. Wang, X. Tang, and W. Xu, “Promptfl: Let federated participants cooperatively learn prompts instead of models – federated learning in age of foundation model,”IEEE Transactions on Mobile Computing, vol. 23, no. 5, pp. 5179–5194, 2024

work page 2024

[8] [8]

Fedbiot: Llm local fine-tuning in federated learning without full model,

F. Wu, Z. Li, Y . Li, B. Ding, and J. Gao, “Fedbiot: Llm local fine-tuning in federated learning without full model,” inProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024, pp. 3345–3355

work page 2024