Differentially Private Language Models Benefit from Public Pre-training
read the original abstract
Language modeling is a keystone task in natural language processing. When training a language model on sensitive information, differential privacy (DP) allows us to quantify the degree to which our private data is protected. However, training algorithms which enforce differential privacy often lead to degradation in model quality. We study the feasibility of learning a language model which is simultaneously high-quality and privacy preserving by tuning a public base model on a private corpus. We find that DP fine-tuning boosts the performance of language models in the private domain, making the training of such models possible.
This paper has not been read by Pith yet.
Forward citations
Cited by 2 Pith papers
-
ConfusionPrompt: Practical Private Inference for Online Large Language Models
ConfusionPrompt enables private black-box LLM inference via prompt decomposition and pseudo-prompt mixing, claiming better privacy-utility trade-off than perturbation methods and lower memory use than open-source loca...
-
SharedRequest: Privacy-Preserving Model-Agnostic Inference for Large Language Models
SharedRequest is a model-agnostic batch-level framework that mixes prompts with noise and groups equivalent instructions to achieve higher utility and lower query cost than individual differential privacy methods for ...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.