REVIEW 12 cited by
SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Transfer learning via fine-tuning pre-trained transformer models has gained significant success in delivering state-of-the-art results across various NLP tasks. In the absence of centralized data, Federated Learning (FL) can benefit from distributed and private data of the FL edge clients for fine-tuning. However, due to the limited communication, computation, and storage capabilities of edge devices and the huge sizes of popular transformer models, efficient fine-tuning is crucial to make federated training feasible. This work explores the opportunities and challenges associated with applying parameter efficient fine-tuning (PEFT) methods in different FL settings for language tasks. Specifically, our investigation reveals that as the data across users becomes more diverse, the gap between fully fine-tuning the model and employing PEFT methods widens. To bridge this performance gap, we propose a method called SLoRA, which overcomes the key limitations of LoRA in high heterogeneous data scenarios through a novel data-driven initialization technique. Our experimental results demonstrate that SLoRA achieves performance comparable to full fine-tuning, with significant sparse updates with approximately $\sim 1\%$ density while reducing training time by up to $90\%$.
Forward citations
Cited by 12 Pith papers
-
FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuning
A federated LoRA fine-tuning method that builds a client-similarity tree and adapts aggregation depth layer-by-layer outperforms flat or global aggregation baselines on NLU and NLG tasks.
-
From LLMs to Edge: Parameter-Efficient Fine-Tuning on Edge Devices
On depthwise-separable CNNs, PEFT memory savings drop to about half of LLM levels, though LoRA and DoRA still cut update FLOPs by up to 95%.
-
Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix
FedASK uses double sketching and server-side SVD to aggregate federated LoRA updates under differential privacy, allowing both A and B matrices to be updated instead of freezing one.
-
ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraints
LoRA adapters can be initialized with a closed-form estimate derived from constraint sets linking source and target activations, improving fine-tuning speed and accuracy.
-
FedRPCA: Enhancing Federated LoRA Aggregation Using Robust PCA
FedRPCA decomposes federated LoRA client updates with Robust PCA into common and client-specific components, averaging the common part and scaled-averaging the sparse part, which improves accuracy and convergence over...
-
Assortment of Attention Heads: Accelerating Federated PEFT with Head Pruning and Strategic Client Selection
A federated fine-tuning method prunes 90% of attention heads, weights updates by attention importance, and selects clients by loss gap, cutting communication 1.8x and training compute 3.9x with under 2% accuracy drop.
-
LAARA: Layer-Aware Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning
LAARA allocates LoRA ranks per layer from diagonal Fisher (gradient-based) estimates, reporting improved accuracy with fewer trainable parameters on GLUE and MathInstruct.
-
ParaBlock: Communication-Computation Parallel Block Coordinate Federated Learning for Large Language Models
ParaBlock hides communication latency in federated block-coordinate LLM fine-tuning by running last round's upload/download in parallel with current computation, preserving the O(1/√T) convergence rate.
-
A Comprehensive Data-centric Overview of Federated Graph Learning
A data-centric taxonomy for Federated Graph Learning that classifies 79 studies by data characteristics and data utilization, plus a discussion of integration with pre-trained large models.
-
Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning with Heterogeneous LoRA Allocation
Fed-HeLLo allocates different LoRA layers to clients of different resource levels using importance scores and geometric patterns, improving federated fine-tuning accuracy over random allocation baselines.
-
AFLoRA: Adaptive Federated Fine-Tuning of Large Language Models with Resource-Aware Low-Rank Adaption
AFLoRA prunes low-rank adapter dimensions per client, trains only the client-specific part locally and the shared part on the server, and aggregates heterogeneous updates by zero-padding and rank-aware weighting.
-
Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
ACBench tests compressed LLMs on agentic tasks and finds 4-bit quantization keeps tool use and workflow generation strong while hurting real-world application performance.
Discussion (0). Continue with ORCID to comment.