Pith. sign in

REVIEW 12 cited by

FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2409.05976 v1 pith:O5AR2KQP submitted 2024-09-09 cs.LG cs.DC

Ziyao Wang , Zheyu Shen , Yexiao He , Guoheng Sun , Hongyi Wang , Lingjuan Lyu , Ang Li This is my paper · ORCID
classification cs.LGcs.DC
keywords fine-tuningfederatedheterogeneousloraaggregationllmsadaptersclients
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The rapid development of Large Language Models (LLMs) has been pivotal in advancing AI, with pre-trained LLMs being adaptable to diverse downstream tasks through fine-tuning. Federated learning (FL) further enhances fine-tuning in a privacy-aware manner by utilizing clients' local data through in-situ computation, eliminating the need for data movement. However, fine-tuning LLMs, given their massive scale of parameters, poses challenges for clients with constrained and heterogeneous resources in FL. Previous methods employed low-rank adaptation (LoRA) for efficient federated fine-tuning but utilized traditional FL aggregation strategies on LoRA adapters. These approaches led to mathematically inaccurate aggregation noise, reducing fine-tuning effectiveness and failing to address heterogeneous LoRAs. In this work, we first highlight the mathematical incorrectness of LoRA aggregation in existing federated fine-tuning methods. We introduce a new approach called FLORA that enables federated fine-tuning on heterogeneous LoRA adapters across clients through a novel stacking-based aggregation method. Our approach is noise-free and seamlessly supports heterogeneous LoRA adapters. Extensive experiments demonstrate FLORA' s superior performance in both homogeneous and heterogeneous settings, surpassing state-of-the-art methods. We envision this work as a milestone for efficient, privacy-preserving, and accurate federated fine-tuning of LLMs. Our code is available at https://github.com/ATP-1010/FederatedLLM.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuning

    cs.LG 2026-02 conditional novelty 6.0 of 10

    A federated LoRA fine-tuning method that builds a client-similarity tree and adapts aggregation depth layer-by-layer outperforms flat or global aggregation baselines on NLU and NLG tasks.

  2. Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix

    cs.CR 2025-07 reject novelty 6.0 of 10

    FedASK uses double sketching and server-side SVD to aggregate federated LoRA updates under differential privacy, allowing both A and B matrices to be updated instead of freezing one.

  3. DeCAF: Decentralized Consensus-And-Factorization for Low-Rank Adaptation of Foundation Models

    cs.LG 2025-05 reject novelty 6.0 of 10

    A truncated-SVD consensus step for decentralized LoRA is claimed to reach O(1/sqrt T) convergence, matching decentralized SGD, with supporting CLIP and LLAMA2-7B experiments.

  4. LAARA: Layer-Aware Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning

    cs.LG 2026-07 conditional novelty 5.0 of 10

    LAARA allocates LoRA ranks per layer from diagonal Fisher (gradient-based) estimates, reporting improved accuracy with fewer trainable parameters on GLUE and MathInstruct.

  5. ParaBlock: Communication-Computation Parallel Block Coordinate Federated Learning for Large Language Models

    cs.LG 2025-11 conditional novelty 5.0 of 10

    ParaBlock hides communication latency in federated block-coordinate LLM fine-tuning by running last round's upload/download in parallel with current computation, preserving the O(1/√T) convergence rate.

  6. EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices

    cs.DC 2025-07 conditional novelty 5.0 of 10

    EdgeLoRA combines automatic adapter routing, LRU caching with a memory pool, and grouped LoRA batching to serve thousands of LoRA adapters on edge devices with up to 4x higher throughput than llama.cpp.

  7. Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning with Heterogeneous LoRA Allocation

    cs.LG 2025-06 conditional novelty 5.0 of 10

    Fed-HeLLo allocates different LoRA layers to clients of different resource levels using importance scores and geometric patterns, improving federated fine-tuning accuracy over random allocation baselines.

  8. FedNano: Toward Lightweight Federated Tuning for Pretrained Multimodal Large Language Models

    cs.LG 2025-06 reject novelty 5.0 of 10

    FedNano centralizes the frozen LLM on the server, trains lightweight NanoAdapters on clients, and reports higher federated VQA accuracy than FedAvg, FedProx, and FedDPA-F on ScienceQA and IconQA.

  9. AFLoRA: Adaptive Federated Fine-Tuning of Large Language Models with Resource-Aware Low-Rank Adaption

    cs.LG 2025-05 conditional novelty 4.0 of 10

    AFLoRA prunes low-rank adapter dimensions per client, trains only the client-specific part locally and the shared part on the server, and aggregates heterogeneous updates by zero-padding and rank-aware weighting.

  10. LLM-QFL: Distilling Large Language Model for Quantum Federated Learning

    cs.LG 2025-05 reject novelty 4.0 of 10

    LLM-QFL uses locally fine-tuned LLMs as controllers to reduce communication rounds and adapt optimizer steps in quantum federated learning.

  11. DP-FedLoRA: Privacy-Enhanced Federated Fine-Tuning for On-Device Large Language Models

    cs.CR 2025-09 reject novelty 3.0 of 10

    DP-FedLoRA clips and adds Gaussian noise to per-client LoRA matrices in federated LLM fine-tuning, claiming unbiased updates and bounded variance, but the privacy calibration and experiments have significant gaps.

  12. A Survey: Towards Privacy and Security in Mobile Large Language Models

    cs.CR 2025-09 conditional

    A survey of privacy and security challenges for mobile large language models, summarizing known attack types and defenses without introducing new results.

Pith tools