REVIEW 8 cited by
Federated LoRA with Sparse Communication
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Low-rank adaptation (LoRA) is a natural method for finetuning in communication-constrained machine learning settings such as cross-device federated learning. Prior work that has studied LoRA in the context of federated learning has focused on improving LoRA's robustness to heterogeneity and privacy. In this work, we instead consider techniques for further improving communication-efficiency in federated LoRA. Unfortunately, we show that centralized ML methods that improve the efficiency of LoRA through unstructured pruning do not transfer well to federated settings. We instead study a simple approach, \textbf{FLASC}, that applies sparsity to LoRA during communication while allowing clients to locally fine-tune the entire LoRA module. Across four common federated learning tasks, we demonstrate that this method matches the performance of dense LoRA with up to $10\times$ less communication. Additionally, despite being designed primarily to target communication, we find that this approach has benefits in terms of heterogeneity and privacy relative to existing approaches tailored to these specific concerns. Overall, our work highlights the importance of considering system-specific constraints when developing communication-efficient finetuning approaches, and serves as a simple and competitive baseline for future work in federated finetuning.
Forward citations
Cited by 8 Pith papers
-
FlexOlmo: Open Language Models for Flexible Data Use
FlexOlmo merges independently trained language-model experts, trained on private data, into a single mixture-of-experts model without joint training.
-
FedChip: Federated LLM for Artificial Intelligence Accelerator Chip Design
FedChip applies federated fine-tuning to LLM-based AI accelerator design, adding a 30k-sample dataset and a Chip@k metric, with a reported 77% quality improvement over high-end LLMs.
-
FedRPCA: Enhancing Federated LoRA Aggregation Using Robust PCA
FedRPCA decomposes federated LoRA client updates with Robust PCA into common and client-specific components, averaging the common part and scaled-averaging the sparse part, which improves accuracy and convergence over...
-
Addressing the Collaboration Dilemma in Low-Data Federated Learning via Transient Sparsity
LIPS, a method that periodically prunes low-sensitivity middle-layer weights after aggregation, mitigates layer-wise inertia and improves low-data federated learning accuracy.
-
FedHL: Federated Learning for Heterogeneous Low-Rank Adaptation via Unbiased Aggregation
FedHL aggregates heterogeneous LoRA updates against a full-rank global baseline and claims O(1/sqrt T) convergence, with small gains on three LLM fine-tuning datasets.
-
Multi-Modal Multi-Task Federated Foundation Models for Next-Generation Extended Reality Systems: Towards Privacy-Preserving Distributed Intelligence in AR/VR/MR
The paper proposes M3T federated foundation models (FedFMs) as a privacy-preserving architecture for XR and codifies the key challenges as the SHIFT dimensions.
-
Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insights
A survey that maps methods for combining foundation models with federated learning into a training-customization-deployment taxonomy with practical ratings.
-
A Survey: Towards Privacy and Security in Mobile Large Language Models
A survey of privacy and security challenges for mobile large language models, summarizing known attack types and defenses without introducing new results.
Discussion (0). Sign in to comment.