Pith. sign in

REVIEW 5 cited by

Large-Scale Differentially Private BERT

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2108.01624 v1 pith:JV4HDMDA submitted 2021-08-03 cs.LG cs.CLcs.CR

Rohan Anil , Badih Ghazi , Vineet Gupta , Ravi Kumar , Pasin Manurangsi This is my paper · ORCID
classification cs.LGcs.CLcs.CR
keywords batchbertdp-sgdimplementationsizeaccuracydifferentiallylarge-scale
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

In this work, we study the large-scale pretraining of BERT-Large with differentially private SGD (DP-SGD). We show that combined with a careful implementation, scaling up the batch size to millions (i.e., mega-batches) improves the utility of the DP-SGD step for BERT; we also enhance its efficiency by using an increasing batch size schedule. Our implementation builds on the recent work of [SVK20], who demonstrated that the overhead of a DP-SGD step is minimized with effective use of JAX [BFH+18, FJL18] primitives in conjunction with the XLA compiler [XLA17]. Our implementation achieves a masked language model accuracy of 60.5% at a batch size of 2M, for $\epsilon = 5.36$. To put this number in perspective, non-private BERT models achieve an accuracy of $\sim$70%.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Lightweight Method to Disrupt Memorized Sequences in LLM

    cs.LG 2025-02 conditional novelty 6.0 of 10

    A decoding-time intervention that substitutes a small model's probabilities for common function words into a large model's output reduces exact training-data recall by up to 10x with minimal measured quality loss.

  2. On the Privacy Risk of In-context Learning

    cs.LG 2024-11 conditional novelty 6.0 of 10

    A confidence-based membership inference attack identifies prompt demonstration data with AUC 0.69-0.86, more than fine-tuned models leak at matched utility, and ensembling reduces this to near random.

  3. FlashDP: Private Training Large Language Models with Efficient DP-SGD

    cs.LG 2025-07 conditional novelty 5.0 of 10

    FlashDP fuses per-sample gradient computation, norm calculation, clipping, and noise addition into a cache-friendly block-wise all-reduce workflow that avoids explicit per-sample gradient storage and redundant recomputation.

  4. SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attacks

    cs.CR 2025-06 conditional novelty 5.0 of 10

    SOFT paraphrases low-loss fine-tuning samples before training, reducing MIA AUC from about 0.82 to about 0.54 across six datasets at roughly 7% perplexity cost.

  5. The Impact of Generalization Techniques on the Interplay Among Privacy, Utility, and Fairness in Image Classification

    cs.LG 2024-12 conditional novelty 5.0 of 10

    Combining DP-SAT with De et al.'s generalization techniques reaches 81.11% accuracy under (8, 10^-5)-DP on CIFAR-10, while amplifying bias on skewed datasets.

Pith tools