REVIEW 6 cited by
FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Tasks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Increasing concerns and regulations about data privacy and sparsity necessitate the study of privacy-preserving, decentralized learning methods for natural language processing (NLP) tasks. Federated learning (FL) provides promising approaches for a large number of clients (e.g., personal devices or organizations) to collaboratively learn a shared global model to benefit all clients while allowing users to keep their data locally. Despite interest in studying FL methods for NLP tasks, a systematic comparison and analysis is lacking in the literature. Herein, we present the FedNLP, a benchmarking framework for evaluating federated learning methods on four different task formulations: text classification, sequence tagging, question answering, and seq2seq. We propose a universal interface between Transformer-based language models (e.g., BERT, BART) and FL methods (e.g., FedAvg, FedOPT, etc.) under various non-IID partitioning strategies. Our extensive experiments with FedNLP provide empirical comparisons between FL methods and helps us better understand the inherent challenges of this direction. The comprehensive analysis points to intriguing and exciting future research aimed at developing FL methods for NLP tasks.
Forward citations
Cited by 6 Pith papers
-
FedQuad: Adaptive Layer-wise LoRA Deployment and Activation Quantization for Federated Fine-Tuning
FedQuad adaptively picks how many final layers each device trains and how many layers' activations are compressed, reporting 1.4-5.3x faster convergence than federated fine-tuning baselines.
-
Assortment of Attention Heads: Accelerating Federated PEFT with Head Pruning and Strategic Client Selection
A federated fine-tuning method prunes 90% of attention heads, weights updates by attention importance, and selects clients by loss gap, cutting communication 1.8x and training compute 3.9x with under 2% accuracy drop.
-
ParaBlock: Communication-Computation Parallel Block Coordinate Federated Learning for Large Language Models
ParaBlock hides communication latency in federated block-coordinate LLM fine-tuning by running last round's upload/download in parallel with current computation, preserving the O(1/√T) convergence rate.
-
Mitigating Catastrophic Forgetting with Adaptive Transformer Block Expansion in Federated Fine-Tuning
FedBE appends zero-initialized transformer blocks to selected layers and allocates them across clients by resource and data profiles, reporting 12-74% better knowledge retention and 1.9-3.1x faster convergence in fede...
-
Privacy-Preserving Quantized Federated Learning with Diverse Precision
A differentially private stochastic quantizer with minimal distortion is coupled with SNR-based fusion weights and cluster-size optimization, improving utility in quantized federated learning.
-
Federated Learning-Enabled Hybrid Language Models for Communication-Efficient Token Transmission
FedHLM uses federated learning to learn token-level uncertainty thresholds that decide when to offload tokens from a small edge LM to a large cloud LM, claiming a 95 percent reduction in LLM transmissions.
Discussion (0). Continue with ORCID to comment.