Pith. sign in

REVIEW 12 cited by

Improving LoRA in Privacy-preserving Federated Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.12313 v1 pith:EE56KQGK submitted 2024-03-18 cs.LG cs.CRcs.DC

Youbang Sun , Zitao Li , Yaliang Li , Bolin Ding This is my paper
classification cs.LGcs.CRcs.DC
keywords lorafederatedffa-loramatricesperformancecomputationalefficiencyfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Low-rank adaptation (LoRA) is one of the most popular task-specific parameter-efficient fine-tuning (PEFT) methods on pre-trained language models for its good performance and computational efficiency. LoRA injects a product of two trainable rank decomposition matrices over the top of each frozen pre-trained model module. However, when applied in the setting of privacy-preserving federated learning (FL), LoRA may become unstable due to the following facts: 1) the effects of data heterogeneity and multi-step local updates are non-negligible, 2) additive noise enforced on updating gradients to guarantee differential privacy (DP) can be amplified and 3) the final performance is susceptible to hyper-parameters. A key factor leading to these phenomena is the discordance between jointly optimizing the two low-rank matrices by local clients and separately aggregating them by the central server. Thus, this paper proposes an efficient and effective version of LoRA, Federated Freeze A LoRA (FFA-LoRA), to alleviate these challenges and further halve the communication cost of federated fine-tuning LLMs. The core idea of FFA-LoRA is to fix the randomly initialized non-zero matrices and only fine-tune the zero-initialized matrices. Compared to LoRA, FFA-LoRA is motivated by practical and theoretical benefits in privacy-preserved FL. Our experiments demonstrate that FFA-LoRA provides more consistent performance with better computational efficiency over vanilla LoRA in various FL tasks.

Discussion (0). Sign in to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. An Efficient Subspace Algorithm for Federated Learning on Heterogeneous Data

    cs.LG 2025-09 conditional novelty 7.0 of 10

    FedSub achieves O(rd) uplink communication and reduced gradient memory with a nonconvex convergence bound that includes a residual error floor from random projection variance.

  2. FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuning

    cs.LG 2026-02 conditional novelty 6.0 of 10

    A federated LoRA fine-tuning method that builds a client-similarity tree and adapts aggregation depth layer-by-layer outperforms flat or global aggregation baselines on NLU and NLG tasks.

  3. Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation

    cs.LG 2025-09 conditional novelty 6.0 of 10

    PrunedLoRA tunes large-rank LoRA adapters and progressively structurally prunes A and B to a target low rank, empirically closing much of the LoRA-to-full-fine-tuning gap.

  4. Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix

    cs.CR 2025-07 reject novelty 6.0 of 10

    FedASK uses double sketching and server-side SVD to aggregate federated LoRA updates under differential privacy, allowing both A and B matrices to be updated instead of freezing one.

  5. DeCAF: Decentralized Consensus-And-Factorization for Low-Rank Adaptation of Foundation Models

    cs.LG 2025-05 reject novelty 6.0 of 10

    A truncated-SVD consensus step for decentralized LoRA is claimed to reach O(1/sqrt T) convergence, matching decentralized SGD, with supporting CLIP and LLAMA2-7B experiments.

  6. FedGSA: Geometry-Consistent Subspace Aggregation for Differentially Private Federated LoRA

    cs.CR 2026-08 conditional novelty 5.0 of 10

    FedGSA aggregates differentially private federated LoRA updates as subspaces on a Grassmann manifold, improving GLUE and E2E NLG accuracy over Euclidean averaging baselines.

  7. Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learning

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Dysco reduces cross-client interference in federated LoRA fine-tuning by fixing each client's update subspace to directions insensitive to other clients' activations, yielding tighter bounds and better benchmark accuracy.

  8. Efficient and Privacy Aware Edge Cloud Collaborative Inference for Large Language Models

    cs.CR 2026-07 conditional novelty 5.0 of 10

    An edge-cloud LLM serving scheme splits the final projection along the hidden dimension, keeps cache control and LoRA modules on-device, and reports 29-46% lower per-token latency and 56-67% lower downlink payloads th...

  9. LAARA: Layer-Aware Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning

    cs.LG 2026-07 conditional novelty 5.0 of 10

    LAARA allocates LoRA ranks per layer from diagonal Fisher (gradient-based) estimates, reporting improved accuracy with fewer trainable parameters on GLUE and MathInstruct.

  10. A Language-Guided Bayesian Optimization for Efficient LoRA Hyperparameter Search

    cs.CL 2026-01 conditional novelty 5.0 of 10

    LLM embeddings plus Bayesian optimization find better LoRA hyperparameters in ~30 proxy trials than standard published settings.

  11. ParaBlock: Communication-Computation Parallel Block Coordinate Federated Learning for Large Language Models

    cs.LG 2025-11 conditional novelty 5.0 of 10

    ParaBlock hides communication latency in federated block-coordinate LLM fine-tuning by running last round's upload/download in parallel with current computation, preserving the O(1/√T) convergence rate.

  12. AFLoRA: Adaptive Federated Fine-Tuning of Large Language Models with Resource-Aware Low-Rank Adaption

    cs.LG 2025-05 conditional novelty 4.0 of 10

    AFLoRA prunes low-rank adapter dimensions per client, trains only the client-specific part locally and the shared part on the server, and aggregates heterogeneous updates by zero-padding and rank-aware weighting.

Pith tools