REVIEW 1 major objections 1 minor 26 references
AdaDPIGU: Differentially Private SGD with Adaptive Clipping and Importance-Based Gradient Updates for Deep Neural Networks
T0 review · 1 major / 1 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims a differentially private training method that prunes low-importance gradient coordinates and adaptively clips per coordinate can match or beat non-private accuracy on MNIST and CIFAR-10 under an (ε,δ)-DP guarantee.
desk verdict The pruning mask is learned from private gradients, which invalidates the privacy theorem's data-independent-mask assumption, and the pretraining privacy loss is never accounted for; the reported epsilons understate the actual privacy cost. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing pieces are the importance score $s_j = \frac{1}{T_{\mathrm{pre}}}\sum_t |g_{t,j}|$, the binary mask $m$ built by sorting those scores and keeping the top $\lfloor r\cdot d\rfloor$ coordinates, and the per-coordinate standardization-plus-restore steps that let the algorithm clip and noise each coordinate in a scale-adapted space. A progressive unfreezing schedule gradually expands the mask so early noise concentrates on critical weights and later steps restore full capacity. The convergence theorems treat the mask as a fixed top-$k$ projection and quantify its effect through the energy-retention ratio $\alpha_t = \|m\odot\nabla L(\theta_t)\|^2/\|\nabla L(\theta_t)\|^2$.
What would settle it
Run the full AdaDPIGU pipeline with the importance-scoring pretraining phase included in the Rényi accountant, composing Tpre DPSGD steps with the main loop, and compare the resulting ε to the reported values; if the composed ε exceeds them, the (ε,δ)-DP claim does not cover the actual algorithm as run. A simpler check: derive the mask from a public dataset instead of the private training set and measure whether the CIFAR-10 accuracy at ε=4 still beats 71.12%.
Extended reading notes
Core claim
The central claim is that important-gradient selection and coordinate-wise adaptive clipping can reduce the noise burden of DPSGD far enough that sparse private training is not merely competitive, it can outperform a non-private model on CIFAR-10 at ε=4. The mechanism achieves this by scoring each parameter's importance as the time-averaged absolute clipped gradient during a DP pretraining run, retaining a fixed top-60% mask, and then performing updates only on the retained coordinates, with clipping thresholds set per coordinate in a standardized space. The paper asserts that because the mask is fixed and applied element-wise, the Gaussian mechanism's sensitivity bound is unchanged, so the same (ε,δ)-DP guarantee applies, and the convergence analysis shows the sparsified optimizer still converges at O(1/√T) with a penalty of $α_t^{{-1}}$ from the retained energy.
Load-bearing premise
The privacy proof treats the pruning mask as fixed and independent of the training data, but the algorithm builds that mask from the private data itself; if that selection step is not folded into the privacy budget, the reported ε values understate the true privacy cost.
Editorial extensions
If this is right
- Under the paper's claims, high-dimensional private training no longer needs to pay noise that scales with the full parameter count, only with the retained coordinates.
- The reported CIFAR-10 result implies sparsification can act as a regularizer under DP noise, not just a noise-saver.
- Coordinate-wise adaptive clipping in a standardized space should make the method less sensitive to the choice of a global clipping threshold.
- Accuracy at ε=2 and ε=4 on all three benchmarks would let practitioners keep a 60% update sparsity without giving up accuracy versus full DPSGD.
Reading between the lines
- Folding the pretraining phase into a single Rényi accountant would turn the per-phase ε values into an end-to-end privacy budget covering the data-dependent mask selection; that composition is a natural next step the paper leaves implicit.
- The importance-scoring idea could transfer to federated learning, where both communication cost and DP noise depend on dimension; testing on heterogeneous client data would be a direct extension.
- If the mask is recomputed mid-training from current gradients, the same argument could extend to adaptive masks, provided each recomputation is treated as a subsampled private query.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes AdaDPIGU, a differentially private SGD framework that combines importance-based gradient sparsification with coordinate-wise adaptive clipping. In a pretraining phase, per-coordinate importance scores are accumulated from private gradients, and a binary mask is formed from the top-k coordinates; the main training loop then updates only masked coordinates with clipped, noisy gradients. The authors claim that AdaDPIGU satisfies (ε,δ)-differential privacy and retains convergence guarantees, and they report strong empirical results on MNIST, FashionMNIST, and CIFAR-10, including 73.21% accuracy on CIFAR-10 at ε=4.
Significance. If the main claim were established, the paper would make a useful contribution: gradient sparsification is a plausible way to reduce the effective noise dimension in DP deep learning, and the empirical comparisons in Table 4 show large gains over DPSGD and competitive or better accuracy than several recent methods. The paper includes code links, detailed appendices, and an attempt at formal privacy and convergence analysis. However, the central privacy proof does not cover the algorithm as actually specified: Theorem 11 assumes a fixed, data-independent mask, while Algorithm 1 constructs the mask from private data, and the privacy cost of the pretraining phase is never composed with the main loop. This is a load-bearing gap, not a presentation issue, and it undermines the headline claims in the abstract and conclusion.
major comments (1)
- [Section 5.1, Theorem 11 and Remark 12; Section 4, Algorithm 1] The convergence analysis does not match the method described in Section 4 or Algorithm 2. Theorems 16 and 17 assume a fixed top-k mask applied in the same way at every iteration, but Section 4 describes per-sample pruning with a sample-specific threshold τ and a progressive unfreezing schedule {r_t} that gradually activates coordinates according to a time-varying mask m(t). Algorithm 2, as written, uses none of these components: it applies the single mask m from Algorithm 1 and does not include per-sample pruning or progressive unfreezing. Consequently, the theoretical convergence results, even if correct, do not establish convergence of the algorithm that produced the reported experimental numbers.
minor comments (1)
- [Appendix B.1] Algorithm 2, line 8, writes the masked gradient as ∇θℓ(m ⊙ θ_t, x_i), while the theoretical analysis in Section 5.1 defines the sparsified gradient as m ⊙ g_t(x_i). The two notations should be made consistent, and the exact masking convention should be stated unambiguously.
Circularity Check
Privacy guarantee for the actual algorithm assumes the data-independent mask that Algorithm 1 violates; the reported epsilon omits the data-dependent mask-selection step.
-
self definitional
[Section 5.1, Theorem 11 and Appendix B.1; Algorithm 1 lines 3-16]
"Let m ∈ {0,1}^d be a fixed, data-independent binary mask corresponding to a retention ratio r = ||m||0/d. ... Since the mask is fixed and applied element-wise, it is a post-processing operation, which does not increase sensitivity. ... By the post-processing theorem of differential privacy Dwork and Roth (2014), the masked mechanism satisfies the same (ε, δ)-DP guarantee as the unmasked Gaussian mechanism."
Algorithm 1 constructs m from private data: it runs DPSGD on private batches, accumulates s_j = (1/T_pre) Σ_t |g_t,j|, sorts s, and sets m_j = 1 for the top-k coordinates. Hence m is a function of D, not a fixed, data-independent mask. The proof of Theorem 11 is purely a post-processing argument: a fixed mask applied element-wise does not increase sensitivity and preserves DP. That argument is valid only when the post-processing map is independent of the original data, as Lemma 3 itself states. Because m depends on D, the proof assumes exactly the property the actual algorithm lacks.
-
fitted input called prediction
[Section 4, pretraining description; Algorithm 1 lines 4-16; Remark 12]
"Run DPSGD on batch B_t to obtain clipped gradient g_t ... s_j ← s_j + |g_t,j| ... m_j ← (1, if j ∈ I_s; 0, otherwise). Remark 12: 'Since the mask is constructed in a data-independent way (e.g., via offline importance estimation), it does not introduce additional privacy loss.'"
The importance scores that determine the mask are fitted statistics: they are accumulated clipped-gradient magnitudes on the private dataset, and the top-k rule is data-dependent. Remark 12 re-labels this construction as 'data-independent' and 'offline', and Theorem 11 then treats the mask as free through post-processing. No privacy analysis is given for the T_pre pretraining iterations that produce the scores; Algorithm 1 shows only clipped gradients with no noise, and the paper never composes the pretraining budget. Thus a quantity fitted to private data is renamed as an input and excluded from the privacy accounting, making the reported ε a privacy prediction that omits its own fitted selection step.
full rationale
The central claim that AdaDPIGU satisfies (ε,δ)-differential privacy is not derived for the procedure as written. Theorem 11 and Appendix B.1 prove DP only for a mechanism with a fixed, data-independent mask m, and the proof explicitly relies on post-processing. Algorithm 1, however, produces m by averaging per-coordinate clipped gradient magnitudes over T_pre DPSGD iterations on the private dataset and retaining the top-k coordinates. That makes m data-dependent, so the post-processing lemma does not apply, and the privacy loss of the pretraining phase is never composed into the reported budget. Consequently, the paper's headline privacy result follows from an assumption (data-independent mask) that the actual algorithm violates, rather than from a derivation covering the mask-selection step. This is the load-bearing circularity. The convergence theorems are not themselves circular: they are standard smoothness-and-noise bounds for the fixed-mask mechanism, but they share the same mask-independence premise and therefore also do not cover the actual Algorithm 1/2. No load-bearing self-citations are present; the related-work citations are background. The score reflects the fact that the central privacy guarantee reduces to its own premise for the algorithm actually run and reported in Table 4.
Assumptions & free parameters
free parameters (5)
- retention ratio r =
0.6
- noise multiplier σ =
per ε (e.g., 2.49 for ε=4 on MNIST, Table 3)
- clipping bound C =
not reported
- pretraining length Tpre =
not reported
- momentum factors γ1, γ2 and stability constant µ =
not reported
assumptions (5)
- domain assumption The loss function L is G-Lipschitz smooth
- ad hoc to paper The binary mask m is fixed and data-independent
- domain assumption Gradient bias is bounded, ∥z_t∥₂ ≤ σ_g²
- standard math RDP composition and subsampling amplification theorems
- domain assumption Running statistics α, β can be treated as fixed for sensitivity analysis
Cite this review
Pith. "Pith review of AdaDPIGU: Differentially Private SGD with Adaptive Clipping and Importance-Based Gradient Updates for Deep Neural Networks." pith.science (2026). https://pith.science/paper/5DDFVD7C
@misc{pith2026250706525,
author = {Pith},
title = {Pith review of: AdaDPIGU: Differentially Private SGD with Adaptive Clipping and Importance-Based Gradient Updates for Deep Neural Networks},
year = {2026},
howpublished = {\url{https://pith.science/paper/5DDFVD7C}},
note = {Machine review of arXiv:2507.06525}
}
abstract
Differential privacy has been proven effective for stochastic gradient descent; however, existing methods often suffer from performance degradation in high-dimensional settings, as the scale of injected noise increases with dimensionality. To tackle this challenge, we propose AdaDPIGU--a new differentially private SGD framework with importance-based gradient updates tailored for deep neural networks. In the pretraining stage, we apply a differentially private Gaussian mechanism to estimate the importance of each parameter while preserving privacy. During the gradient update phase, we prune low-importance coordinates and introduce a coordinate-wise adaptive clipping mechanism, enabling sparse and noise-efficient gradient updates. Theoretically, we prove that AdaDPIGU satisfies $(\varepsilon, \delta)$-differential privacy and retains convergence guarantees. Extensive experiments on standard benchmarks validate the effectiveness of AdaDPIGU. All results are reported under a fixed retention ratio of 60%. On MNIST, our method achieves a test accuracy of 99.12% under a privacy budget of $\epsilon = 8$, nearly matching the non-private model. Remarkably, on CIFAR-10, it attains 73.21% accuracy at $\epsilon = 4$, outperforming the non-private baseline of 71.12%, demonstrating that adaptive sparsification can enhance both privacy and utility.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Deep learning with differential privacy
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. InProceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages308–318,
work page 2016
-
[8]
Towards general deep leakage in federated learning.arXiv:2110.09074,
Jiahui Geng, Yongli Mou, Feifei Li, Qing Li, Oya Beyan, Stefan Decker, and Chunming Rong. Towards general deep leakage in federated learning.arXiv:2110.09074,
-
[12]
DPDR: Gradient decomposition and reconstruction for differentially private deep learning
44 AdaDPIGU: Priv ate SGD with Adaptive Clipping Yixuan Liu, Li Xiong, Yuhan Liu, Yujie Gu, Ruixuan Liu, and Hong Chen. DPDR: Gradient decomposition and reconstruction for differentially private deep learning. arXiv:2406.02744,
-
[14]
Milad Nasr, Reza Shokri, et al. Improving deep learning with differential privacy using gradient encoding and denoising.arXiv:2007.11524,
arXiv 2007
-
[15]
Privacy- preserving deep learning: Revisited and enhanced
Le Trieu Phong, Yoshinori Aono, Takuya Hayashi, Lihua Wang, and Shiho Moriai. Privacy- preserving deep learning: Revisited and enhanced. In Applications and Techniques in Information Security: 8th International Conference, ATIS 2017, Auckland, New Zealand, July 6–7, 2017, Proceedings, pages 100–110,
work page 2017
-
[16]
AdaClip: Adaptive clipping for private sgd.arXiv:1908.07643,
Venkatadheeraj Pichapati, Ananda Theertha Suresh, Felix X Yu, Sashank J Reddi, and Sanjiv Kumar. AdaClip: Adaptive clipping for private sgd.arXiv:1908.07643,
arXiv 1908
-
[17]
45 Zhang and Xie Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. ML-Leaks: Model and data independent membership inference attacks and de- fenses on machine learning models. arXiv:1806.01246,
-
[19]
Machine learning models that remember too much
Congzheng Song, Thomas Ristenpart, and Vitaly Shmatikov. Machine learning models that remember too much. InProceedings of the 2017 ACM SIGSAC Conference on computer and communications security, pages 587–601,
work page 2017
Show all 26 references
-
[20]
Subsampled rényi differ- ential privacy and analytical moments accountant
Yu-Xiang Wang, Borja Balle, and Shiva Prasad Kasiviswanathan. Subsampled rényi differ- ential privacy and analytical moments accountant. InThe 22nd international conference on artificial intelligence and statistics, pages 1226–1235, 2019a. Zhibo Wang, Mengkai Song, Zhifei Zhan...
2019
-
[22]
Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms.arXiv:1708.07747,
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms.arXiv:1708.07747,
-
[23]
Do not let privacy overbill utility: Gradient embedding perturbation for private learning.arXiv:2102.12677, 2021a
Da Yu, Huishuai Zhang, Wei Chen, and Tie-Yan Liu. Do not let privacy overbill utility: Gradient embedding perturbation for private learning.arXiv:2102.12677, 2021a. Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie-Yan Liu. Large scale private learning via low-rank reparametr...
2001 arXiv
-
[24]
Bypassingtheambientdimension: Private SGD with gradient subspace identification.arXiv:2007.03813,
YingxueZhou, ZhiweiStevenWu, andArindamBanerjee. Bypassingtheambientdimension: Private SGD with gradient subspace identification.arXiv:2007.03813,
2007 arXiv
-
[25]
R-GAP: Recursive gradient attack on privacy
Junyi Zhu and Matthew Blaschko. R-GAP: Recursive gradient attack on privacy. arXiv:2010.07733,
2010 arXiv
-
[26]
Improving differentially private SGD via randomly sparsified gradients
Junyi Zhu and Matthew B Blaschko. Improving differentially private SGD via randomly sparsified gradients. arXiv:2112.00845,
-
[1998]
Deepleakagefromgradientsinmultiple-label medical image classification
ZhengLi, MykolaHubchak, andYingyingZhu. Deepleakagefromgradientsinmultiple-label medical image classification. In 2021 IEEE 9th International Conference on Healthcare Informatics, pages 447–448,
2021
-
[2005]
FedSel: Federated SGD under local differential privacy with top-k dimension selection
Ruixuan Liu, Yang Cao, Masatoshi Yoshikawa, and Hong Chen. FedSel: Federated SGD under local differential privacy with top-k dimension selection. InDatabase Systems for Advanced Applications: 25th International Conference, DASFAA 2020, Jeju, South Korea, September 24–27, 2020,...
2020
-
[2009]
Bagging classifiers for fighting poisoning attacks in adversarial classification tasks
Battista Biggio, Igino Corona, Giorgio Fumera, Giorgio Giacinto, and Fabio Roli. Bagging classifiers for fighting poisoning attacks in adversarial classification tasks. In Multiple Classifier Systems: 10th International Workshop, MCS 2011, Naples, Italy, June 15-17,
2011
-
[2015]
DP- SUR: accelerating differentially private stochastic gradient descent using selective update and release
43 Zhang and Xie Jie Fu, Qingqing Ye, Haibo Hu, Zhili Chen, Lulu Wang, Kuncan Wang, and Xun Ran. DP- SUR: accelerating differentially private stochastic gradient descent using selective update and release. arXiv:2311.14056,
-
[2016]
Differential privacy meets neural network pruning
Kamil Adamczewski and Mijung Park. Differential privacy meets neural network pruning. arXiv:2303.04612,
-
[2017]
Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning
Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In 2019 IEEE symposium on security and privacy, pages 739–753,
2019
-
[2018]
PCDP-SGD: Improving the con- vergence of differentially private SGD via projection in advance.arXiv:2312.03792,
Haichao Sha, Ruixuan Liu, Yixuan Liu, and Hong Chen. PCDP-SGD: Improving the con- vergence of differentially private SGD via projection in advance.arXiv:2312.03792,
-
[2019]
Privacy- preserving learning via deep net pruning.arXiv:2003.01876,
Yangsibo Huang, Yushan Su, Sachin Ravi, Zhao Song, Sanjeev Arora, and K Li. Privacy- preserving learning via deep net pruning.arXiv:2003.01876,
2003 arXiv
-
[2020]
Secure multi-party computation problems and their applications: a review and open problems
Wenliang Du and Mikhail J Atallah. Secure multi-party computation problems and their applications: a review and open problems. InProceedings of the 2001 workshop on New security paradigms, pages 13–22,
2001
-
[2021]
Can machine learning be secure? InProceedings of the 2006 ACM Symposium on Information, computer and communications security, pages 16–25,
42 AdaDPIGU: Priv ate SGD with Adaptive Clipping Marco Barreno, Blaine Nelson, Russell Sears, Anthony D Joseph, and J Doug Tygar. Can machine learning be secure? InProceedings of the 2006 ACM Symposium on Information, computer and communications security, pages 16–25,
2006
-
[2022]
Bolt-on differential privacy for scalable stochastic gradient descent-based analytics
Xi Wu, Fengan Li, Arun Kumar, Kamalika Chaudhuri, Somesh Jha, and Jeffrey Naughton. Bolt-on differential privacy for scalable stochastic gradient descent-based analytics. In Proceedings of the 2017 ACM International Conference on Management of Data, pages 1307–1322,
2017
-
[2023]
Multiple classifier systems for adversarial classification tasks
Battista Biggio, Giorgio Fumera, and Fabio Roli. Multiple classifier systems for adversarial classification tasks. In Multiple Classifier Systems: 8th International Workshop, MCS 2009, Reykjavik, Iceland, June 10-12,
2009
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.