pith. sign in

arxiv: 2605.20450 · v1 · pith:V4EJ7K47new · submitted 2026-05-19 · 💻 cs.LG · cs.CR

SMA-DP: Spectral Memory-Aware Differential Privacy for Deep Learning

Mohammad Partohaghighi , Roummel Marcia This is my paper

Pith reviewed 2026-05-21 07:16 UTC · model grok-4.3

classification 💻 cs.LG cs.CR
keywords differential privacyDP-SGDspectral analysismemory-aware optimizationprivacy-utility trade-offdeep learning
0
0 comments X

The pith

SMA-DP-SGD adds a fractional memory branch to DP-SGD that adapts decay via power-law spectral exponents while keeping a clean conditional sensitivity structure.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes SMA-DP-SGD as an augmentation to standard differentially private stochastic gradient descent. It incorporates a memory branch built solely from prior privatized noisy gradient releases, with decay and depth adapted layer-wise by power-law spectral exponents drawn from spectral analysis. Privacy accounting stays transparent because the memory contribution is fixed once the private history is given, leaving only the current clipped gradient sum as the new data-dependent term scaled by a fixed coefficient. Experiments across CIFAR-100, CIFAR-10, and MNIST report competitive or higher accuracy than several DP optimization baselines, with the largest gains on the more challenging image datasets and with explicit controls showing how the memory weight trades off privacy and utility.

Core claim

SMA-DP-SGD preserves a clean conditional sensitivity structure: conditioned on the private release history the memory branch is fixed, so the only newly data-dependent term is the current clipped sum scaled by a fixed coefficient β. It therefore exactly recovers group-wise DP-SGD when β equals one. The adaptation of memory decay and effective depth is driven by WeightWatcher-inspired power-law spectral exponents that supply group-wise reliability signals, instantiated layer-wise, together with private-history alignment, norm matching, and warm-up activation to stabilize the contribution.

What carries the argument

The fractional memory branch constructed only from previously privatized noisy releases, with its decay and effective depth adapted by layer-wise power-law spectral exponents that serve as group-wise reliability signals.

If this is right

  • When the memory coefficient β is set to one the method reduces exactly to group-wise DP-SGD.
  • The single scalar β directly parametrizes the privacy-utility trajectory observed in ablations.
  • Private-history alignment together with norm matching keeps the memory contribution stable without additional data-dependent leakage.
  • Spectral and memory diagnostics confirm that the effective memory depth remains short to moderate while the memory-branch ratio stays small.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same spectral-control idea could be grafted onto other private first-order methods such as DP-Adam or DP-LAMB.
  • Empirical checks on transformer-based vision or language models would test whether the power-law signals remain reliable outside the convolutional regimes studied here.
  • The reported 2.94-fold runtime overhead indicates that practical deployment would benefit from cheaper approximations to the spectral exponent computation.

Load-bearing premise

Power-law spectral exponents supply stable group-wise reliability signals that can safely set memory decay and depth without introducing training instability or extra privacy leakage beyond the stated conditional sensitivity.

What would settle it

A run on CIFAR-10 or CIFAR-100 in which the spectral-exponent-driven memory branch produces either diverging loss curves or measured privacy leakage exceeding the bound implied by the conditional sensitivity analysis.

Figures

Figures reproduced from arXiv: 2605.20450 by Mohammad Partohaghighi, Roummel Marcia.

Figure 1
Figure 1. Figure 1: Overview of SMA-DP-SGD, combining a current private gradient branch with a spectrally [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Cross-dataset comparison of test accuracy versus training epoch for differentially private [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Ablation study of the fixed current-memory mixing parameter [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Stage-wise spectral dynamics and reliability-driven tempering of SMA-DP-SGD on [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Diagnostic effect of the fixed current-memory mixing parameter [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗
read the original abstract

Differentially private stochastic gradient descent (DP-SGD) enables private deep learning through per-example clipping and calibrated Gaussian noise, but its high-variance updates can reduce utility on challenging datasets. We propose \textbf{SMA-DP-SGD}, a \textbf{Spectral Memory-Aware Differentially Private Stochastic Gradient Descent} method that augments DP-SGD with a fractional memory branch built only from previously privatized noisy releases. WeightWatcher-inspired power-law spectral exponents provide group-wise reliability signals, instantiated layer-wise in our experiments, to adapt the decay and effective memory depth. Private-history alignment, norm matching, and warm-up activation stabilize the memory contribution. Privacy remains transparent: conditioned on the private release history, the memory branch is fixed, and the only newly data-dependent term is the current clipped sum scaled by a fixed coefficient \(\beta\). Hence, SMA-DP-SGD preserves a clean conditional sensitivity structure and exactly recovers group-wise DP-SGD when \(\beta=1\). Experiments on CIFAR-100, CIFAR-10, and MNIST show competitive or superior accuracy over several DP optimization baselines, with the largest gains on CIFAR-100 and CIFAR-10. CIFAR-10 ablations show that \(\beta\) controls the privacy--utility trajectory, while spectral and memory diagnostics confirm a controlled short-to-moderate effective memory depth and a small memory-branch ratio. Runtime analysis shows that the mechanism incurs additional overhead, about \(2.94\times\) DP-SGD in our CIFAR-10 implementation, revealing a practical trade-off between adaptive private memory and computational cost.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 3 minor

Summary. The manuscript proposes SMA-DP-SGD, an augmentation of DP-SGD that adds a fractional memory branch constructed exclusively from prior privatized noisy gradient releases. Layer-wise power-law spectral exponents (inspired by WeightWatcher) adapt the memory decay and effective depth, with private-history alignment, norm matching, and warm-up used for stability. The authors claim that, conditioned on the private release history, the memory branch is fixed so that the sole new data-dependent term is the current clipped gradient sum scaled by a fixed β; this preserves a clean conditional sensitivity and exactly recovers group-wise DP-SGD at β=1. Experiments on CIFAR-100, CIFAR-10 and MNIST report competitive or superior accuracy versus several DP optimization baselines, with ablations on β, spectral diagnostics, and a reported 2.94× runtime overhead relative to DP-SGD.

Significance. If the conditional privacy argument can be made rigorous, the method offers a principled route to improve utility in DP deep learning by controlled reuse of private history without breaking the sensitivity structure. The exact recovery of standard DP-SGD at β=1 and the reported gains on CIFAR-100/10 are attractive features; however, the computational overhead and the need for stable spectral signals limit immediate practicality.

major comments (1)
  1. [Privacy analysis / §4] Privacy analysis (abstract and §4): the claim that 'conditioned on the private release history, the memory branch is fixed' and that the only new data-dependent term is the current clipped sum scaled by β rests on the spectral exponents being non-data-dependent once the history is fixed. Because the exponents are computed from current model parameters (which accumulate all prior private updates), their values are implicitly data-dependent; this appears to violate the stated conditioning and may require additional noise or invalidate the exact recovery of group-wise DP-SGD at β=1. A formal argument showing that the exponent computation introduces no extra sensitivity (or that it is performed on a fixed snapshot) is needed.
minor comments (3)
  1. [Experiments] Abstract and experimental section: accuracy gains are stated without error bars, number of runs, or statistical tests; this weakens confidence in the reported superiority over baselines, especially given the stochastic nature of DP-SGD.
  2. [Runtime analysis] Runtime analysis: the 2.94× overhead figure is given without a component-wise breakdown (spectral computation vs. memory alignment vs. warm-up); a table or paragraph quantifying each source would clarify the practical trade-off.
  3. [Method] Notation: the precise definition of the 'fractional memory branch' and how the power-law exponent is instantiated layer-wise (e.g., which layers, how the exponent is mapped to decay rate) should be stated explicitly in the method section rather than left to the WeightWatcher reference.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their careful reading and constructive feedback on the privacy analysis. We address the single major comment point by point below.

read point-by-point responses

  1. Referee: [Privacy analysis / §4] Privacy analysis (abstract and §4): the claim that 'conditioned on the private release history, the memory branch is fixed' and that the only new data-dependent term is the current clipped sum scaled by β rests on the spectral exponents being non-data-dependent once the history is fixed. Because the exponents are computed from current model parameters (which accumulate all prior private updates), their values are implicitly data-dependent; this appears to violate the stated conditioning and may require additional noise or invalidate the exact recovery of group-wise DP-SGD at β=1. A formal argument showing that the exponent computation introduces no extra sensitivity (or that it is performed on a fixed snapshot) is needed.

    Authors: We thank the referee for highlighting this subtlety. The private release history is the full sequence of prior noisy gradient releases together with the deterministic model updates they induce. Conditioning on this history therefore fixes the current model parameters exactly (they are the result of applying the previous updates in order). The spectral exponents are obtained by a deterministic power-law fitting procedure applied to these fixed current parameters; consequently they are themselves fixed under the conditioning and introduce no additional sensitivity with respect to the new data batch. The memory branch (including its layer-wise decay rates derived from the exponents) is therefore a fixed, history-dependent quantity. The update takes the form β·(current clipped sum + calibrated noise) + fixed_memory_branch(history), so the only data-dependent contribution to sensitivity is the current clipped sum scaled by the constant β. This preserves the claimed conditional sensitivity. When β = 1 the memory-branch coefficient is set to zero by definition, recovering group-wise DP-SGD exactly and independently of the exponents. We will add a short formal paragraph and proof sketch to §4 making this conditioning argument explicit, including the deterministic accumulation of parameters given the history. revision: yes

Circularity Check

0 steps flagged

Privacy conditioning argument is self-contained; no circular reduction to inputs

full rationale

The paper's central privacy claim states that, conditioned on the private release history, the memory branch (including decay and depth adapted via layer-wise spectral exponents) is fixed, with only the current clipped sum scaled by fixed β being newly data-dependent. This directly supports the clean conditional sensitivity structure and exact recovery of group-wise DP-SGD at β=1. The current model parameters (source of the WeightWatcher-inspired exponents) are determined by prior private updates, so they remain fixed under the stated conditioning and introduce no additional data dependence for the current step. No self-definitional equations, fitted parameters renamed as predictions, or load-bearing self-citations appear in the derivation. Performance results are presented as empirical comparisons on CIFAR/MNIST, separate from the privacy argument. The mechanism is therefore self-contained against external DP benchmarks.

Axiom & Free-Parameter Ledger

2 free parameters · 1 axioms · 1 invented entities

The central claim rests on the assumption that spectral exponents yield reliable signals and that the memory branch adds no new data-dependent terms beyond the scaled current gradient.

free parameters (2)
  • beta
    Fixed coefficient scaling the current clipped sum; controls the privacy-utility trade-off and is chosen per experiment.
  • layer-wise spectral exponents
    Power-law exponents extracted per layer to set memory decay and depth; treated as reliability signals.
axioms (1)
  • domain assumption Memory branch is fixed once conditioned on the private release history.
    Stated to ensure the only new data-dependent term is the current clipped gradient sum.
invented entities (1)
  • fractional memory branch no independent evidence
    purpose: Augment DP-SGD updates with controlled historical information from prior noisy releases.
    New component introduced to reduce update variance while preserving privacy structure.

pith-pipeline@v0.9.0 · 5818 in / 1389 out tokens · 55556 ms · 2026-05-21T07:16:20.578815+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

18 extracted references · 18 canonical work pages

  1. [1]

    Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang

    Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. InProceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pages 308–318. ACM,

    work page 2016

  2. [2]

    Karame, Karl Wüst, Vasileios Glykantzis, Hubert Ritzdorf, and Srdjan Capkun

    1145/2976749.2978318. Galen Andrew, Om Thakkar, H. Brendan McMahan, and Swaroop Ramaswamy. Differentially private learning with adaptive clipping. InAdvances in Neural Information Processing Systems, volume 34, pages 17455–17466,

    work page arXiv

  3. [3]

    Private empirical risk minimization: Efficient algorithms and tight error bounds

    Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. InProceedings of the 2014 IEEE 55th Annual Symposium on Foundations of Computer Science, pages 464–473. IEEE,

    work page 2014

  4. [4]

    ISBN 9781479965175

    doi: 10.1109/FOCS.2014.56. Zhiqi Bu, Jinshuo Dong, Qi Long, and Weijie J. Su. Deep learning with gaussian differential privacy. Harvard Data Science Review, 2020(23),

    work page doi:10.1109/focs.2014.56 2014

  5. [5]

    Soham De, Leonard Berrada, Jamie Hayes, Samuel L

    doi: 10.1162/99608f92.cfc5dd25. Soham De, Leonard Berrada, Jamie Hayes, Samuel L. Smith, and Borja Balle. Unlock- ing high-accuracy differentially private image classification through scale.arXiv preprint arXiv:2204.13650,

    work page doi:10.1162/99608f92.cfc5dd25

  6. [6]

    Springer,

    Kai Diethelm.The Analysis of Fractional Differential Equations: An Application-Oriented Exposi- tion Using Differential Operators of Caputo Type, volume 2004 ofLecture Notes in Mathematics. Springer,

    work page 2004

  7. [7]

    Springer, Berlin (2010)

    doi: 10.1007/978-3-642-14574-2. John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization.Journal of Machine Learning Research, 12:2121–2159,

    work page doi:10.1007/978-3-642-14574-2

  8. [8]

    Foun- dations and Trends in Theoretical Computer Science 9(3-4), 211–407 (2013) https://doi.org/10.1561/0400000042

    doi: 10.1561/0400000042. Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. InTheory of Cryptography, pages 265–284. Springer,

    work page doi:10.1561/0400000042

  9. [9]

    doi: 10.1109/5. 726791. Charles H. Martin and Michael W. Mahoney. Traditional and heavy-tailed self regularization in neu- ral network models. InProceedings of the 36th International Conference on Machine Learning, volume 97 ofProceedings of Machine Learning Research, pages 4284–4293. PMLR,

    work page doi:10.1109/5

  10. [10]

    doi: 10.1038/s41467-021-24025-8. H. Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. InInternational Conference on Learning Representations,

    work page doi:10.1038/s41467-021-24025-8

  11. [11]

    Ilya Mironov

    doi: 10.1515/9783110258165. Ilya Mironov. R ´enyi differential privacy. In2017 IEEE 30th Computer Security Foundations Sym- posium, pages 263–275. IEEE,

    work page doi:10.1515/9783110258165

  12. [12]

    doi: 10.1109/CSF.2017.11. Yurii E. Nesterov. A method for solving the convex programming problem with convergence rate O(1/k2). InDoklady Akademii Nauk SSSR, volume 269, pages 543–547,

    work page doi:10.1109/csf.2017.11 2017

  13. [13]

    West, and YangQuan Chen

    Mohammad Partohaghighi, Roummel Marcia, Bruce J. West, and YangQuan Chen. When gradient clipping becomes a control mechanism for differential privacy in deep learning.arXiv preprint arXiv:2602.10584, 2026a. doi: 10.48550/arXiv.2602.10584. Mohammad Partohaghighi, Roummel Marcia, Bruce J. West, and YangQuan Chen. Statisti- cal roughness-informed machine unl...

    work page doi:10.48550/arxiv.2602.10584

  14. [14]

    Jeffrey Pennington and Pratik Worah

    1016/j.eswa.2026.131501. Jeffrey Pennington and Pratik Worah. Nonlinear random matrix theory for deep learning. InAd- vances in Neural Information Processing Systems, volume 30,

    work page arXiv 2026

  15. [15]

    Herbert Robbins and Sutton Monro

    doi: 10.1016/0041-5553(64) 90137-5. Herbert Robbins and Sutton Monro. A stochastic approximation method.The Annals of Mathemat- ical Statistics, 22(3):400–407,

    work page doi:10.1016/0041-5553(64

  16. [16]

    On Information and Sufficiency

    doi: 10.1214/aoms/1177729586. Florian Tram`er and Dan Boneh. Differentially private learning needs better features (or much more data). InInternational Conference on Learning Representations,

    work page doi:10.1214/aoms/1177729586

  17. [17]

    doi: 10.1103/RevModPhys.86.1169. 12 Appendix Overview This appendix provides supplementary experimental details, additional empirical results, diagnos- tic analyses, runtime comparisons, and the full theoretical analysis supporting the main paper. To improve readability and navigation, the supplementary material is organized into two main parts. Appendix ...

    work page doi:10.1103/revmodphys.86.1169

  18. [18]

    3, the choice ofβsubstantially affects the privacy–utility trajectory

    As shown in Fig. 3, the choice ofβsubstantially affects the privacy–utility trajectory. The limiting caseβ= 1.00requires a larger privacy budget to reach high test accuracy. In contrast, smaller val- ues ofβ, especiallyβ= 0.70andβ= 0.50, reach comparable or higher accuracy using a smaller privacy budget in this diagnostic. The curve forβ= 0.90provides an ...

    work page 2079