REVIEW 13 cited by
Unlocking High-Accuracy Differentially Private Image Classification through Scale
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Differential Privacy (DP) provides a formal privacy guarantee preventing adversaries with access to a machine learning model from extracting information about individual training points. Differentially Private Stochastic Gradient Descent (DP-SGD), the most popular DP training method for deep learning, realizes this protection by injecting noise during training. However previous works have found that DP-SGD often leads to a significant degradation in performance on standard image classification benchmarks. Furthermore, some authors have postulated that DP-SGD inherently performs poorly on large models, since the norm of the noise required to preserve privacy is proportional to the model dimension. In contrast, we demonstrate that DP-SGD on over-parameterized models can perform significantly better than previously thought. Combining careful hyper-parameter tuning with simple techniques to ensure signal propagation and improve the convergence rate, we obtain a new SOTA without extra data on CIFAR-10 of 81.4% under (8, 10^{-5})-DP using a 40-layer Wide-ResNet, improving over the previous SOTA of 71.7%. When fine-tuning a pre-trained NFNet-F3, we achieve a remarkable 83.8% top-1 accuracy on ImageNet under (0.5, 8*10^{-7})-DP. Additionally, we also achieve 86.7% top-1 accuracy under (8, 8 \cdot 10^{-7})-DP, which is just 4.3% below the current non-private SOTA for this task. We believe our results are a significant step towards closing the accuracy gap between private and non-private image classification.
Forward citations
Cited by 13 Pith papers
-
Sequential Auditing for f-Differential Privacy
A new sequential auditor for f-differential privacy adaptively chooses its sample size, detects violations across the whole privacy tradeoff curve, and holds a user-set false-rejection rate.
-
Correlating Cross-Iteration Noise for DP-SGD using Model Curvature
Using Hessian eigenvalues from public data to design correlated noise for DP-SGD improves accuracy by 1–4% over current DP-MF methods.
-
Adapting to Linear Separable Subsets with Large-Margin in Differentially Private Learning
A private halfspace learner achieves empirical and population zero-one error O(1/(γ²εn) + |S_out|/(γn)) for the best separable subset S_in = S\S_out with margin γ, without knowing γ or S_out in advance.
-
Private Direct Preference Optimization for LLM Alignment
PrivDPO perturbs the DPO objective with an unbiased randomized rescaling to enforce epsilon-preference privacy, achieving near-DPO utility on three benchmarks and three LLM families up to 32B.
-
End-to-End Differential Privacy in Training Deep Neural Network Classifiers
Perturbing softmax outputs with the Dirichlet mechanism during training yields input-private, label-public classifiers that beat prior differentially private training accuracy on five image benchmarks.
-
Differentially Private Natural Gradient Descent
DP-NGD enables second-order optimization under differential privacy by decoupling curvature estimation onto public data, performing isotropic DP operations in a whitened space, and dynamically clamping curvature eigen...
-
An Interactive Framework for Finding the Optimal Trade-off in Differential Privacy
Modeling the privacy-accuracy Pareto front with sigmoid curves and letting users pick points on hypothetical fronts finds the optimal DP privacy budget with fewer queries and evaluations than pairwise baselines.
-
Lower Bounds for Public-Private Learning under Distribution Shift
For Gaussian mean estimation and linear regression with distribution shift, the paper claims that public data never provides complementary value: either public data alone suffices, or (for large shifts) private data a...
-
Optimizing Canaries for Privacy Auditing with Metagradient Descent
Optimized canary examples, crafted by metagradient descent on a small non-private model, more than double empirical epsilon lower bounds in black-box DP-SGD privacy audits on CIFAR-10.
-
Implementing Adaptations for Vision AutoRegressive Model
Fine-tuned Vision AutoRegressive models mostly beat a strong diffusion baseline on downstream image generation, but DP fine-tuning yields poor FID scores.
-
UniAud: A Unified Auditing Framework for High Auditing Power and Utility with One Training Run
UniAud uses synthetic uncorrelated canaries and self-comparison inference to reach near-optimal empirical epsilon lower bounds in one black-box DP audit run, while UniAud++ improves the utility-auditing trade-off via ...
-
The pretraining domain outweighs the training objective in setting the privacy-utility trade-off of differentially private medical image analysis
In DP-SGD chest X-ray classification, MIMIC-CXR supervised pretraining beats ImageNet and DINOv3 initializations, but the study cannot cleanly separate pretraining domain from objective because key comparison arms are...
-
Mitigating Disparate Impact of Differentially Private Learning through Bounded Adaptive Clipping
Bounded adaptive clipping puts a floor under the adaptive clipping threshold in DP-SGD and improves worst-class accuracy on skewed and Fashion MNIST.
Discussion (0). Continue with ORCID to comment.