Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

DP-FedLoRA claims that adding calibrated Gaussian noise to clipped LoRA updates makes federated fine-tuning of on-device LLMs differentially private with modest accuracy loss.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

DP-FedLoRA clips and adds Gaussian noise to per-client LoRA matrices in federated LLM fine-tuning, claiming unbiased updates and bounded variance, but the privacy calibration and experiments have significant gaps.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection A competent but forgettable combination of LoRA and DP-FL that gets the sensitivity wrong, undercuts its own privacy claim, and would only be citable as a cautionary tale. the 3 major comments →

arxiv 2509.09097 v1 pith:7GI65XQ5 submitted 2025-09-11 cs.CR cs.AI

DP-FedLoRA: Privacy-Enhanced Federated Fine-Tuning for On-Device Large Language Models

Honghui Xu , Shiva Shrestha , Wei Chen , Zhiyuan Li , Zhipeng Cai This is my paper
classification cs.CR cs.AI
keywords federated learningdifferential privacyLoRAlarge language modelson-device fine-tuningGaussian mechanismmembership inferenceprivacy budget
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to show that federated fine-tuning of large language models on edge devices can be made differentially private without destroying utility, by clipping each client's LoRA adaptation matrices to a Frobenius norm bound and adding Gaussian noise calibrated to that bound. If right, this gives a per-update (ε, δ)-DP guarantee against a semi-honest server performing membership inference, while preserving the communication efficiency and low-rank structure of LoRA. The paper further claims that the noisy update is unbiased in expectation, and its variance is bounded by a formula that grows with model dimensions and LoRA rank, which can be used to choose privacy parameters. Experiments with LLaMA-2-7B/13B across seven federated optimizers show modest drops on MMLU and BBH but a larger drop on CRASS counterfactual reasoning, which the paper attributes to noise sensitivity.

Core claim

The paper proposes DP-FedLoRA, a federated fine-tuning framework for on-device LLMs. Each client locally fine-tunes low-rank matrices B_k and A_k, clips each to a Frobenius norm bound, and adds isotropic Gaussian noise with scale set by the standard Gaussian-mechanism formula. The server aggregates the noisy matrices via a stacking operation that preserves the LoRA product structure. The paper's central claim is that this mechanism enforces (ε,δ)-differential privacy per client update, that the expected noisy adaptation equals the clean adaptation, and that the added variance is bounded by m σ_α² ||B||_F² + n σ_β² ||A||_F² + σ_β² σ_α² mn r. Experiments on LLaMA-2-7B and 13B with seven federa

What carries the argument

The load-bearing object is the clip-then-perturb mechanism applied independently to the two LoRA factors: each B_k and A_k is scaled down to Frobenius norm C_{Bk} or C_{Ak}, then isotropic Gaussian noise with standard deviation σ ≥ C √(2 log(1.25/δ))/ε is added. The server aggregates by horizontally stacking the noisy B's and vertically stacking the noisy A's, so the global adaptation remains a low-rank product. The theoretical results hinge on the independence and zero-mean of the noise matrices, giving unbiasedness Eq. (20), and on the variance bound Eq. (26), which is the tool for privacy-budget calibration.

Load-bearing premise

The guarantee assumes a clipped LoRA matrix has l2-sensitivity equal to its clip bound C; if neighboring datasets can produce opposite updates of norm C, the true sensitivity is 2C and the claimed (ε,δ) bound is off by a factor of two.

What would settle it

Compute the l2 sensitivity of the clip-and-noise mechanism for a pair of neighboring clients whose clipped updates are B and -B with ||B||_F=C; the output distributions are separated by 2C, so the noise scale must double to meet the same (ε,δ). A practical test is to run the released code with the paper's default ε=25 and measure membership-inference success on the noisy updates: an attacker success rate at chance level would support the claim, while near-perfect success would falsify it.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If a client's update is clipped to norm C and noise with σ ≥ C sqrt(2 log(1.25/δ))/ε is added, the released matrices satisfy (ε,δ)-DP, so the server cannot confidently perform membership inference on the update.
  • The noise-injected global update is unbiased: in expectation the noisy adaptation equals the clean adaptation, so the federated optimization trajectory is preserved on average.
  • The variance bound gives a calibration rule: for fixed clip bounds and privacy budget, the variance grows linearly with LoRA rank r and with model dimensions m,n, so smaller ranks and smaller models are preferable when noise is present.
  • Across seven federated optimizers, DP-FedLoRA yields only a ~4–5% average drop on MMLU and BBH, indicating that strong per-update DP can be compatible with competitive fine-tuning performance.
  • The stacking aggregation accommodates clients with heterogeneous ranks without zero-padding or averaging artifacts.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's sensitivity step likely undercounts the true noise needed: for two neighboring datasets whose clipped updates are B and -B, both with norm C, the release distributions are 2C apart, so the claimed (ε,δ) guarantee would require doubling σ or halving ε.
  • The analysis never composes the budgets spent on B and A, nor the budgets spent across the training rounds in which a client participates; a proper composition accounting would raise the effective privacy cost, and a DP-FedLoRA variant with a composition-aware accountant would be a natural extension.
  • The large CRASS drop suggests counterfactual-reasoning benchmarks are particularly sensitive to update noise; a testable extension would be per-task clip thresholds or rank-adaptive noise schedules to protect those tasks.
  • The variance formula implies a hard trade-off between LoRA rank and model scale under a fixed privacy budget, which could be measured empirically as a contour map of accuracy versus rank and noise scale.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes DP-FedLoRA, a method for privacy-preserving federated fine-tuning of large language models with LoRA. Each client trains low-rank matrices B_k and A_k locally, clips them to Frobenius norms C_B and C_A, and adds isotropic Gaussian noise; the server aggregates the noisy matrices by horizontal/vertical stacking into global matrices \tilde{B}, \tilde{A}. The paper claims that this provides (epsilon, delta)-differential privacy per client via the Gaussian mechanism (Eqs. 9-11), that the noisy update is unbiased (Eq. 20), and that its variance is bounded by Eq. (26). Experiments on LLaMA-2-7B/13B with Alpaca-GPT-4 data compare seven federated optimizers with and without DP on MMLU, BBH, and CRASS, and include ablations over epsilon, clipping norm, LoRA rank, and model size.

Significance. The application is timely and the experimental effort is substantial: the paper releases code, covers seven federated optimization algorithms, and empirically studies the effect of LoRA rank and model size on update variance. The unbiasedness result, while standard, is stated cleanly. However, the central privacy claim is not supported by the analysis. The Gaussian mechanism is calibrated with sensitivity C after clipping to a Frobenius ball of radius C, whereas the true L2 sensitivity of the clipped release is 2C. Furthermore, the paper does not compose the B and A releases within a round, nor does it account for clients participating in multiple communication rounds. The variance bounds in Eqs. (23)-(26) also contain dimension errors. These issues are load-bearing: they undermine the advertised (epsilon, delta)-DP guarantee and the analytical variance bound.

major comments (3)
  1. [Section V, Eqs. (9)-(11)] The Gaussian mechanism guarantee in Eq. (9) depends on the L2 sensitivity S. After clipping to ||B||_F <= C_B, the set of possible outputs is the Frobenius ball of radius C_B. Two neighboring datasets can yield clipped updates C_B u and -C_B u, so the sensitivity is sup_{D~D'} ||clip(B(D)) - clip(B(D'))||_F = 2C_B, not C_B. With sigma_B calibrated using S_B = C_B, the noise is half what it should be for the stated (epsilon, delta); the achieved per-release privacy loss is at least approximately 2 epsilon (or a much larger delta) for each matrix. The same issue applies to A. The paper's claim that DP-FedLoRA satisfies (epsilon, delta)-DP at the reported epsilon is therefore not delivered. The fix is to use S = 2C (equivalently, double sigma or halve epsilon) throughout the calibration.
  2. [Section V / Algorithm 1; Section VII-A] Even after correcting the sensitivity, the paper provides no composition analysis. A client emits both \tilde{B}_k and \tilde{A}_k in the same round; under basic composition the combined release has privacy budget at least epsilon_B + epsilon_A, not two independent guarantees. More importantly, Section VII-A runs 200 communication rounds with 2 of 20 clients sampled per round; an individual client may appear in many rounds. The manuscript never defines the unit of the DP guarantee (per update, per round, or over the whole training run) and never accounts for repeated participation. Without this accounting, the abstract's 'strong privacy guarantees' claim is unsupported. The paper must either provide a full composition analysis over the training process or explicitly restrict and justify a per-round-only claim.
  3. [Section VI, Eqs. (23)-(26)] The variance bounds contain dimension errors. For B in R^{m x r} and alpha in R^{r x n}, the total variance of B alpha (summed over all entries) is n sigma_alpha^2 ||B||_F^2, not m sigma_alpha^2 ||B||_F^2. Similarly, for beta in R^{m x r} and A in R^{r x n}, Var[beta A] = m sigma_beta^2 ||A||_F^2, not n sigma_beta^2 ||A||_F^2. Consequently Eq. (26) has the coefficients of the first two terms swapped. The third term, sigma_beta^2 sigma_alpha^2 m n r, is correct and matches the linear growth in rank reported in Section VII-D, but the stated bounds are not correct as written. This does not invalidate the unbiasedness result but weakens the claimed analytical variance characterization.
minor comments (4)
  1. [Section VII-B] The text says 'stricter privacy constraints (i.e., a larger value of epsilon)'; this is backwards. A larger epsilon corresponds to a weaker privacy guarantee. This terminology should be corrected throughout the ablation discussion.
  2. [Sections VI and VII] The claimed 'practical guidance for privacy-budget calibration' is not used to set the privacy budget. The experiments select epsilon = 25.0 based on the ablation in Fig. 1, not from the variance bounds. Either connect the analysis to the budget choice or temper the contribution claim.
  3. [Section III-A] The sentence 'This factorization drastically reduces the number of trainable parameters from m x r to (rd + rk)' contains typos; it should presumably be r(m + n) or similar. Please clarify.
  4. [Section V, Eq. (3)] The stacking operation is introduced as 'symbolized by L' but the displayed equation uses a different symbol. Please unify the notation and define the dimensions of the stacked matrices explicitly.

Circularity Check

0 steps flagged

No significant circularity: the DP guarantee and noise analyses are standard applications of the Gaussian mechanism and independent zero-mean noise, not reductions to fitted inputs.

full rationale

The paper's derivation chain is self-contained. The privacy calibration (Eqs. 9-11) invokes the standard Gaussian mechanism with an asserted L2-sensitivity; the unbiasedness claim (Eq. 20) follows directly from E[alpha]=0, E[beta]=0 and independence; and the variance bound (Eq. 26) is an analytic consequence of the same noise assumptions. No experimental result is fed back into the theorems, and no fitted parameter is renamed as a prediction. The epsilon=25 choice is an experimental hyperparameter selected by ablation, not a product of the theoretical analysis. The only author-overlapping citation is the survey in reference [2] (Z. Li, W. Chen), used for the general prevalence of on-device LLMs, which is not load-bearing for any technical derivation. A possible understatement of sensitivity (clipped matrices can differ by up to twice the clip norm) and the omission of composition across B/A matrices and rounds would affect the correctness of the privacy guarantee, but those are correctness/accounting issues, not circularity. Thus no circular step is identified; the minor self-citation accounts for the score of 2.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The central claims (unbiasedness, variance bound, DP guarantee) depend on standard Gaussian-mechanism math and one incorrect sensitivity assumption, plus several experimental hyperparameters (epsilon, clip norm, rank) that are chosen by hand after ablation.

free parameters (5)
  • privacy budget epsilon = 25.0
    Chosen from ablation study on FedAvg (Section VII-B); larger epsilon (weaker privacy) gave lower loss. This is the 'strong privacy' setting used for all experiments.
  • clipping threshold C (B and A) = 0.1
    Selected from ablation; values 0.1 to 1.0 tested, 0.1 chosen as default.
  • LoRA rank r = 32 (default), varied 8-128
    Set as hyperparameter; the rank experiment shows variance grows linearly with r.
  • LoRA scaling factor alpha = 64 (default), 128 in rank experiments
    Set as hyperparameter; affects the magnitude of updates.
  • noise scales sigma_B, sigma_A = not reported; depend on unstated delta
    Derived from Eq. (10)-(11) but the paper never reports the numerical values or delta, so the actual noise magnitude is unspecified.
axioms (4)
  • standard math Gaussian mechanism satisfies (epsilon, delta)-DP if sigma >= S * sqrt(2 log(1.25/delta)) / epsilon (Eq. 9)
    Standard result from Dwork et al.; the paper invokes it without proof.
  • ad hoc to paper Clipping a matrix to Frobenius norm C makes its l2-sensitivity C
    Load-bearing and incorrect; the sensitivity of clip-then-release is 2C. Used in Eqs. (10)-(11).
  • domain assumption Noise matrices beta and alpha are independent of the trained matrices B and A
    Reasonable because noise is drawn after training, but it is an assumption about the data generation process.
  • standard math Stacking identity: sum_k B_k A_k = (B_1 ... B_K)(A_1 ... A_K)^T when blocks align (Eq. 3)
    Block-matrix multiplication identity; correct.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of DP-FedLoRA: Privacy-Enhanced Federated Fine-Tuning for On-Device Large Language Models." pith.science (2026). https://pith.science/paper/7GI65XQ5

@misc{pith2026250909097,
  author       = {Pith},
  title        = {Pith review of: DP-FedLoRA: Privacy-Enhanced Federated Fine-Tuning for On-Device Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7GI65XQ5}},
  note         = {Machine review of arXiv:2509.09097}
}
Share X Bluesky LinkedIn Reddit HN
abstract

As on-device large language model (LLM) systems become increasingly prevalent, federated fine-tuning enables advanced language understanding and generation directly on edge devices; however, it also involves processing sensitive, user-specific data, raising significant privacy concerns within the federated learning framework. To address these challenges, we propose DP-FedLoRA, a privacy-enhanced federated fine-tuning framework that integrates LoRA-based adaptation with differential privacy in a communication-efficient setting. Each client locally clips and perturbs its LoRA matrices using Gaussian noise to satisfy ($\epsilon$, $\delta$)-differential privacy. We further provide a theoretical analysis demonstrating the unbiased nature of the updates and deriving bounds on the variance introduced by noise, offering practical guidance for privacy-budget calibration. Experimental results across mainstream benchmarks show that DP-FedLoRA delivers competitive performance while offering strong privacy guarantees, paving the way for scalable and privacy-preserving LLM deployment in on-device environments.

Figures

Figures reproduced from arXiv: 2509.09097 by Honghui Xu, Shiva Shrestha, Wei Chen, Zhipeng Cai, Zhiyuan Li.

Figure 1
Figure 1. Figure 1: Results of Ablation Study (a) FedAvg (b) FedProx (c) SCAFFOLD (d) FedAvgM (e) FedAdagrad (f) FedYogi (g) FedAdam [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Training Loss Curves of FedLLMs without Differential Privacy [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Training Loss Curves of DP-FedLLMs with Our Proposed DP-FedLoRA ( [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Performance scores of FedLLMs and DPFedLLMs for [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Expectation Value E[∆] ˜ and Variance Var[∆] ˜ of Noisy updates in DP-FedLoRA with Various Ranks [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Expectation E[∆] ˜ and Variance Var[∆] ˜ of Noisy Up￾dates in Our DP-FedLoRA over training rounds with Different Base Models and Fixed LoRA Rank TABLE IV: Converged Expectation E[∆] ˜ and Variance Var[∆] ˜ of Noisy Updates in Our DP-FedLoRA over training rounds with Different Base Models and Fixed LoRA Rank Base Model Expectation (E[∆] ˜ ) Variance (Var[∆] ˜ ) LLaMA-2 7B 1.04 × 10−8 151.47 LLaMA-2 13B 2.39… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Investigating Novice Researchers' Perceptions of Research Privacy Within LLM-Assisted Workflows

    cs.HC 2026-06 unverdicted novelty 6.0

    Interview study of 44 novice researchers finds privacy fears paradoxically accelerate LLM use for faster publication, with misconceptions about idea value and data dilution, and perceived ineffective mitigations.

  2. DP-FlogTinyLLM: Differentially private federated log anomaly detection using Tiny LLMs

    cs.CR 2026-04 unverdicted novelty 4.0

    DP-FLogTinyLLM combines federated learning, differential privacy, and LoRA-tuned tiny LLMs to match centralized log anomaly detection performance on Thunderbird and BGL datasets while preserving privacy.

Reference graph

Works this paper leans on

40 extracted references · 25 linked inside Pith · cited by 2 Pith papers

  1. [1]

    A review on edge large language models: Design, execution, and applications,

    Y . Zheng, Y . Chen, B. Qian, X. Shi, Y . Shu, and J. Chen, “A review on edge large language models: Design, execution, and applications,”ACM Computing Surveys, vol. 57, no. 8, pp. 1–35, 2025

    2025

  2. [2]

    On-device language models: A comprehensive review,

    J. Xu, Z. Li, W. Chen, Q. Wang, X. Gao, Q. Cai, and Z. Ling, “On-device language models: A comprehensive review,”arXiv preprint arXiv:2409.00088, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    A survey on parameter- efficient fine-tuning for foundation models in federated learning,

    J. Bian, Y . Peng, L. Wang, Y . Huang, and J. Xu, “A survey on parameter- efficient fine-tuning for foundation models in federated learning,”arXiv preprint arXiv:2504.21099, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Distributed llms and multimodal large language models: A survey on advances, challenges, and future directions,

    H. Amini, M. J. Mia, Y . Saadati, A. Imteaj, S. Nabavirazavi, U. Thakker, M. Z. Hossain, A. A. Fime, and S. Iyengar, “Distributed llms and multimodal large language models: A survey on advances, challenges, and future directions,”arXiv preprint arXiv:2503.16585, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    Decentralized federated learning: Fundamentals, state of the art, frameworks, trends, and challenges,

    E. T. M. Beltr ´an, M. Q. P ´erez, P. M. S. S ´anchez, S. L. Bernal, G. Bovet, M. G. P ´erez, G. M. P ´erez, and A. H. Celdr ´an, “Decentralized federated learning: Fundamentals, state of the art, frameworks, trends, and challenges,”IEEE Communications Surveys & Tutorials, vol. 25, no. 4, pp. 2983–3013, 2023

    2023

  6. [6]

    Decentralised learning in feder- ated deployment environments: A system-level survey,

    P. Bellavista, L. Foschini, and A. Mora, “Decentralised learning in feder- ated deployment environments: A system-level survey,”ACM Computing Surveys (CSUR), vol. 54, no. 1, pp. 1–38, 2021

    2021

  7. [7]

    Federated fine-tuning of llms: Framework comparison and research directions,

    N. Yan, Y . Su, Y . Deng, and R. Schober, “Federated fine-tuning of llms: Framework comparison and research directions,”arXiv preprint arXiv:2501.04436, 2025

    Pith/arXiv arXiv 2025

  8. [8]

    Active membership inference attack under local differential privacy in federated learning,

    T. Nguyen, P. Lai, K. Tran, N. Phan, and M. T. Thai, “Active membership inference attack under local differential privacy in federated learning,” arXiv preprint arXiv:2302.12685, 2023

    Pith/arXiv arXiv 2023

  9. [9]

    A survey on security and privacy threats to federated learning,

    J. Zhang, M. Li, S. Zeng, B. Xie, and D. Zhao, “A survey on security and privacy threats to federated learning,” in2021 International conference on networking and network applications (NaNA). IEEE, 2021, pp. 319–326

    2021

  10. [10]

    Privacy issues in large language models: a survey,

    H. Kibriya, W. Z. Khan, A. Siddiqa, and M. K. Khan, “Privacy issues in large language models: a survey,”Computers and Electrical Engineering, vol. 120, p. 109698, 2024

    2024

  11. [11]

    Understanding users’ security and privacy concerns and attitudes towards conversational ai platforms,

    M. Ali, A. Arunasalam, and H. Farrukh, “Understanding users’ security and privacy concerns and attitudes towards conversational ai platforms,” arXiv preprint arXiv:2504.06552, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    Parameter-efficient fine-tuning for pre-trained vision models: A survey,

    Y . Xin, J. Yang, S. Luo, H. Zhou, J. Du, X. Liu, Y . Fan, Q. Li, and Y . Du, “Parameter-efficient fine-tuning for pre-trained vision models: A survey,”arXiv preprint arXiv:2402.02242, 2024

    arXiv 2024

  13. [13]

    Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment,

    L. Xu, H. Xie, S.-Z. J. Qin, X. Tao, and F. L. Wang, “Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment,”arXiv preprint arXiv:2312.12148, 2023

    Pith/arXiv arXiv 2023

  14. [14]

    Bitfit: Simple parameter- efficient fine-tuning for transformer-based masked language-models,

    E. B. Zaken, S. Ravfogel, and Y . Goldberg, “Bitfit: Simple parameter- efficient fine-tuning for transformer-based masked language-models,” arXiv preprint arXiv:2106.10199, 2021

    arXiv 2021

  15. [15]

    On the effectiveness of adapter-based tuning for pretrained language model adaptation,

    R. He, L. Liu, H. Ye, Q. Tan, B. Ding, L. Cheng, J.-W. Low, L. Bing, and L. Si, “On the effectiveness of adapter-based tuning for pretrained language model adaptation,”arXiv preprint arXiv:2106.03164, 2021

    Pith/arXiv arXiv 2021

  16. [16]

    Compar- ative analysis of efficient adapter-based fine-tuning of state-of-the-art transformer models,

    S. M. Siddiqui, M. A. Sheikh, M. Aleem, and K. R. Singh, “Compar- ative analysis of efficient adapter-based fine-tuning of state-of-the-art transformer models,”arXiv preprint arXiv:2501.08271, 2025

    Pith/arXiv arXiv 2025

  17. [17]

    Lora: Low-rank adaptation of large language models

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, W. Chenet al., “Lora: Low-rank adaptation of large language models.” ICLR, vol. 1, no. 2, p. 3, 2022

    2022

  18. [18]

    Lora-ga: Low-rank adaptation with gradient approximation,

    S. Wang, L. Yu, and J. Li, “Lora-ga: Low-rank adaptation with gradient approximation,”Advances in Neural Information Processing Systems, vol. 37, pp. 54 905–54 931, 2024

    2024

  19. [19]

    Higher layers need more lora experts,

    C. Gao, K. Chen, J. Rao, B. Sun, R. Liu, D. Peng, Y . Zhang, X. Guo, J. Yang, and V . Subrahmanian, “Higher layers need more lora experts,” arXiv preprint arXiv:2402.08562, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    The impact of initialization on lora finetuning dynamics,

    S. Hayou, N. Ghosh, and B. Yu, “The impact of initialization on lora finetuning dynamics,”Advances in Neural Information Processing Systems, vol. 37, pp. 117 015–117 040, 2024

    2024

  21. [21]

    Iteris: Itera- tive inference-solving alignment for lora merging,

    H. Chen, R. Li, B. Zhu, Z. Wang, and L. Chen, “Iteris: Itera- tive inference-solving alignment for lora merging,”arXiv preprint arXiv:2411.15231, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    Flora: Federated fine-tuning large language models with heterogeneous low- rank adaptations,

    Z. Wang, Z. Shen, Y . He, G. Sun, H. Wang, L. Lyu, and A. Li, “Flora: Federated fine-tuning large language models with heterogeneous low- rank adaptations,”arXiv preprint arXiv:2409.05976, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    From demodulation to decoding: Toward complete lora phy understanding and implementation,

    Z. Xu, S. Tong, P. Xie, and J. Wang, “From demodulation to decoding: Toward complete lora phy understanding and implementation,”ACM Transactions on Sensor Networks, vol. 18, no. 4, pp. 1–27, 2023

    2023

  24. [24]

    Rbla: Rank-based-lora-aggregation for fine-tuning heterogeneous models in flaas,

    S. Chen, O. Tavallaie, N. Nazemi, and A. Y . Zomaya, “Rbla: Rank-based-lora-aggregation for fine-tuning heterogeneous models in flaas,” Sep 2024. [Online]. Available: https://arxiv.org/abs/2408.08699

    Pith/arXiv arXiv 2024

  25. [25]

    Fedfmsl: Federated learning of foundation models with sparsely activated lora,

    P. Wu, K. Li, T. Wang, Y . Dong, V . C. M. Leung, and F. Wang, “Fedfmsl: Federated learning of foundation models with sparsely activated lora,” IEEE Transactions on Mobile Computing, vol. 23, no. 12, pp. 15 167– 15 181, 2024

    2024

  26. [26]

    Autorank: Mcda based rank personalization for lora-enabled distributed learning,

    S. Chen, O. Tavallaie, N. Nazemi, X. Chen, and A. Y . Zomaya, “Autorank: Mcda based rank personalization for lora-enabled distributed learning,” 2024. [Online]. Available: https://arxiv.org/abs/2412.15553

    Pith/arXiv arXiv 2024

  27. [27]

    Towards efficient communication and secure federated recommendation system via low-rank training,

    N.-H. Nguyen, T.-A. Nguyen, T. Nguyen, V . T. Hoang, D. D. Le, and K.-S. Wong, “Towards efficient communication and secure federated recommendation system via low-rank training,” inProceedings of the ACM Web Conference 2024. New York, NY , USA: Association for Computing Machinery, 2024, p. 3940–3951. [Online]. Available: https://doi.org/10.1145/3589334.3645702

    arXiv 2024

  28. [28]

    Lora land: 310 fine-tuned llms that rival gpt-4, a technical report,

    J. Zhao, T. Wang, W. Abid, G. Angus, A. Garg, J. Kinnison, A. Sherstinsky, P. Molino, T. Addair, and D. Rishi, “Lora land: 310 fine-tuned llms that rival gpt-4, a technical report,” 2024. [Online]. Available: https://arxiv.org/abs/2405.00732

    Pith/arXiv arXiv 2024

  29. [29]

    Towards label-only membership inference attack against pre-trained large language models,

    Y . He, B. Li, L. Liu, Z. Ba, W. Dong, Y . Li, Z. Qin, K. Ren, and C. Chen, “Towards label-only membership inference attack against pre-trained large language models,” 2025. [Online]. Available: https://arxiv.org/abs/2502.18943

    Pith/arXiv arXiv 2025

  30. [30]

    Membership inference attacks against vision-language models,

    Y . Hu, Z. Li, Z. Liu, Y . Zhang, Z. Qin, K. Ren, and C. Chen, “Membership inference attacks against vision-language models,” 2025. [Online]. Available: https://arxiv.org/abs/2501.18624

    Pith/arXiv arXiv 2025

  31. [31]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark,

    Y . Wang, X. Ma, G. Zhang, Y . Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen, “Mmlu-pro: A more robust and challenging multi-task language understanding benchmark,” 2024. [Online]. Available: https://arxiv.org/abs/2406.01574

    Pith/arXiv arXiv 2024

  32. [32]

    Big-bench extra hard,

    M. Kazemi, B. Fatemi, H. Bansal, J. Palowitch, C. Anastasiou, S. V . Mehta, L. K. Jain, V . Aglietti, D. Jindal, P. Chen, N. Dikkala, G. Tyen, X. Liu, U. Shalit, S. Chiappa, K. Olszewska, Y . Tay, V . Q. Tran, Q. V . Le, and O. Firat, “Big-bench extra hard,” 2025. [Online]. Available: https://arxiv.org/abs/2502.19187

    Pith/arXiv arXiv 2025

  33. [33]

    Crass: A novel data set and benchmark to test counterfactual reasoning of large language models,

    J. Frohberg and F. Binder, “Crass: A novel data set and benchmark to test counterfactual reasoning of large language models,” 2022. [Online]. Available: https://arxiv.org/abs/2112.11941

    Pith/arXiv arXiv 2022

  34. [34]

    Decentralized federated averaging,

    T. Sun, D. Li, and B. Wang, “Decentralized federated averaging,” 2021. [Online]. Available: https://arxiv.org/abs/2104.11375

    Pith/arXiv arXiv 2021

  35. [35]

    Federated optimization in heterogeneous networks,

    T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V . Smith, “Federated optimization in heterogeneous networks,” 2020. [Online]. Available: https://arxiv.org/abs/1812.06127

    Pith/arXiv arXiv 2020

  36. [36]

    Scaffold: Stochastic controlled averaging for federated learning,

    S. P. Karimireddy, S. Kale, M. Mohri, S. J. Reddi, S. U. Stich, and A. T. Suresh, “Scaffold: Stochastic controlled averaging for federated learning,” 2021. [Online]. Available: https://arxiv.org/abs/1910.06378

    Pith/arXiv arXiv 2021

  37. [37]

    On the role of server momentum in federated learning,

    J. Sun, X. Wu, H. Huang, and A. Zhang, “On the role of server momentum in federated learning,” 2023. [Online]. Available: https://arxiv.org/abs/2312.12670

    Pith/arXiv arXiv 2023

  38. [38]

    Fedda: Resource-adaptive federated learning with dual-alignment aggregation optimization for heterogeneous edge devices,

    S. Cao, H. Wu, X. Wu, R. Ma, D. Wang, Z. Han, and W. Zhang, “Fedda: Resource-adaptive federated learning with dual-alignment aggregation optimization for heterogeneous edge devices,”Future Gener. Comput. Syst., vol. 163, p. 107551, 2025. [Online]. Available: https://doi.org/10.1016/j.future.2024.107551

    arXiv 2025

  39. [39]

    Not all federated learning algorithms are created equal: A performance evaluation study,

    G. A. Baumgart, J. Shin, A. Payani, M. Lee, and R. R. Kompella, “Not all federated learning algorithms are created equal: A performance evaluation study,” 2024. [Online]. Available: https: //arxiv.org/abs/2403.17287

    Pith/arXiv arXiv 2024

  40. [40]

    Accelerating fair federated learning: Adaptive federated adam,

    L. Ju, T. Zhang, S. Toor, and A. Hellander, “Accelerating fair federated learning: Adaptive federated adam,”IEEE Transactions on Machine Learning in Communications and Networking, vol. 2, pp. 1017–1032, 2024

    2024

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.