REVIEW 6 cited by
FDLoRA: Personalized Federated Learning of Large Language Model via Dual LoRA Tuning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models (LLMs) have emerged as important components across various fields, yet their training requires substantial computation resources and abundant labeled data. It poses a challenge to robustly training LLMs for individual users (clients). To tackle this challenge, the intuitive idea is to introduce federated learning (FL), which can collaboratively train models on distributed private data. However, existing methods suffer from the challenges of data heterogeneity, system heterogeneity, and model size, resulting in suboptimal performance and high costs. In this work, we proposed a variant of personalized federated learning (PFL) framework, namely FDLoRA, which allows the client to be a single device or a cluster and adopts low-rank adaptation (LoRA) tuning. FDLoRA sets dual LoRA modules on each client to capture personalized and global knowledge, respectively, and only the global LoRA module uploads parameters to the central server to aggregate cross-client knowledge. Finally, an adaptive fusion approach is employed to combine the parameters of the dual LoRAs. This enables FDLoRA to make effective use of private data distributed across different clients, thereby improving performance on the client without incurring high communication and computing costs. We conducted extensive experiments in two practice scenarios. The results demonstrate that FDLoRA outperforms six baselines in terms of performance, stability, robustness, computation cost, and communication cost.
Forward citations
Cited by 6 Pith papers
-
Safe-FedLLM: Delving into the Safety of Federated Large Language Models
Safe-FedLLM shows that a lightweight logistic-regression probe on LoRA B-matrix deltas can detect malicious federated clients, and step/client/shadow defense levels restore most of the safety lost to data poisoning.
-
Interpretable Mnemonic Generation for Kanji Learning via Expectation-Maximization
An EM-type algorithm jointly learns interpretable mnemonic rules and learner/kanji traits, yielding modest cold-start gains over fine-tuning for kanji mnemonic generation.
-
FedRPCA: Enhancing Federated LoRA Aggregation Using Robust PCA
FedRPCA decomposes federated LoRA client updates with Robust PCA into common and client-specific components, averaging the common part and scaled-averaging the sparse part, which improves accuracy and convergence over...
-
FedHL: Federated Learning for Heterogeneous Low-Rank Adaptation via Unbiased Aggregation
FedHL aggregates heterogeneous LoRA updates against a full-rank global baseline and claims O(1/sqrt T) convergence, with small gains on three LLM fine-tuning datasets.
-
Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learning
Dysco reduces cross-client interference in federated LoRA fine-tuning by fixing each client's update subspace to directions insensitive to other clients' activations, yielding tighter bounds and better benchmark accuracy.
-
LLM-QFL: Distilling Large Language Model for Quantum Federated Learning
LLM-QFL uses locally fine-tuned LLMs as controllers to reduce communication rounds and adapt optimizer steps in quantum federated learning.
Discussion (0). Continue with ORCID to comment.