An exponential mechanism based on quadratic approximations for fine-tuning machine learning models with privacy guarantees
Pith reviewed 2026-05-21 07:02 UTC · model grok-4.3
The pith
A local quadratic approximation to the loss enables the exponential mechanism to sample fine-tuned model parameters exactly from a multivariate normal distribution while guaranteeing differential privacy.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
By constructing a utility function that combines a local quadratic approximation of the pretrained model with information from the new dataset, the exponential mechanism admits exact sampling from a multivariate normal distribution in closed form, for which privacy guarantees and accuracy estimates can be derived directly.
What carries the argument
The utility function formed by a local quadratic approximation of the pretrained model's loss combined with the new dataset, enabling closed-form multivariate normal sampling in the exponential mechanism.
Load-bearing premise
A local quadratic approximation of the pretrained model loss combined with the new dataset yields a utility function whose sensitivity can be bounded tightly enough to deliver meaningful privacy while preserving useful accuracy.
What would settle it
Observe whether the method maintains its claimed accuracy when applied to a model whose loss surface is known to be strongly non-quadratic away from the pretrained parameters, such as a deep network with many layers trained on complex data.
Figures
read the original abstract
Fine-tuning adapts a pretrained machine learning model to a small, sensitive dataset, but this process risks memorizing individual new data points, making the model vulnerable to adversaries who seek to extract sensitive information. In this work, we develop a randomized algorithm based on the exponential mechanism for fine-tuning while ensuring differential privacy. Our key idea is to construct a simple utility function that combines a local quadratic approximation of the pretrained model with information from the new dataset. The resulting exponential mechanism admits exact sampling from a multivariate normal distribution in closed form. We establish theoretical privacy guarantees, sensitivity bounds, and accuracy estimations for our method. We further introduce a random-projection strategy that makes the approach scalable to high-dimensional models. Numerical experiments on the MNIST benchmark and the MIMIC clinical dataset demonstrate competitive performance against existing differentially private fine-tuning techniques.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes a differentially private fine-tuning algorithm for pretrained ML models based on the exponential mechanism. A utility function is defined by combining a local quadratic Taylor approximation of the pretrained loss around the pretrained parameters with a term derived from the new dataset; the resulting quadratic utility permits exact sampling from a multivariate normal in closed form. Theoretical privacy guarantees via explicit sensitivity bounds on the utility, accuracy estimates, and a random-projection technique for high-dimensional scalability are presented. Experiments on MNIST and the MIMIC clinical dataset report competitive accuracy relative to existing DP fine-tuning baselines.
Significance. If the sensitivity bounds prove tight and the quadratic approximation remains faithful, the method supplies an efficient, closed-form alternative to noisy-gradient DP fine-tuning that avoids iterative optimization while preserving exact sampling. The exact MVN sampling and random-projection scalability are clear technical strengths that could improve reproducibility and applicability to high-dimensional models. Practical impact, however, rests on whether the derived Δ yields useful accuracy at meaningful privacy levels (ε, δ), which the current analysis leaves open.
major comments (3)
- [§3, Eq. (7)] §3 (Utility Construction) and Eq. (7): the sensitivity bound Δ on |u(D_new, θ) − u(D_new′, θ)| is claimed to be independent of the largest Hessian eigenvalue of the pretrained loss, yet the quadratic term explicitly involves this Hessian; the derivation therefore appears to require an additional uniform bound on curvature or gradient norms that is not stated or verified, risking a loose Δ that forces either large ε or degraded utility.
- [§5.3] §5.3 (Random Projection): the projection matrix is introduced post-hoc to reduce dimensionality, but no analysis is given of how the projection error affects either the exact quadratic form required for MVN sampling or the sensitivity bound itself; if the projected utility deviates from quadratic, the closed-form sampling claim no longer holds and the privacy guarantee must be re-derived.
- [§6, Table 2] §6 (Experiments), Table 2: accuracy is reported as competitive, yet the effective noise scale (determined by ε/Δ) and the numerical value of the sensitivity bound Δ are not tabulated; without these quantities it is impossible to judge whether the observed accuracy is achieved at a privacy level that is meaningfully stronger than the baselines.
minor comments (2)
- Notation for the utility function should explicitly separate the pretrained quadratic term from the new-data linear/quadratic term to avoid reader confusion when tracking sensitivity contributions.
- The abstract states 'exact sampling from a multivariate normal distribution in closed form,' but the manuscript never writes the explicit mean and covariance of that normal; adding this expression would clarify the sampling procedure.
Simulated Author's Rebuttal
We thank the referee for their careful reading and constructive comments on our manuscript. We address each major comment point by point below, providing clarifications and committing to revisions where appropriate to strengthen the presentation.
read point-by-point responses
-
Referee: [§3, Eq. (7)] §3 (Utility Construction) and Eq. (7): the sensitivity bound Δ on |u(D_new, θ) − u(D_new′, θ)| is claimed to be independent of the largest Hessian eigenvalue of the pretrained loss, yet the quadratic term explicitly involves this Hessian; the derivation therefore appears to require an additional uniform bound on curvature or gradient norms that is not stated or verified, risking a loose Δ that forces either large ε or degraded utility.
Authors: We appreciate the referee highlighting this potential source of confusion. The quadratic approximation is constructed exclusively from the pretrained loss function evaluated at the pretrained parameters and is therefore identical for any choice of D_new or D_new′. Consequently, when forming the difference |u(D_new, θ) − u(D_new′, θ)| the two quadratic terms cancel exactly, and the resulting sensitivity bound Δ depends only on the dataset-dependent linear term. This cancellation is implicit in the derivation of Eq. (7) but was not stated explicitly. In the revised manuscript we will add a short remark immediately after Eq. (7) that makes the cancellation explicit and confirms that no additional uniform bound on the Hessian eigenvalues is required for the sensitivity result. revision: yes
-
Referee: [§5.3] §5.3 (Random Projection): the projection matrix is introduced post-hoc to reduce dimensionality, but no analysis is given of how the projection error affects either the exact quadratic form required for MVN sampling or the sensitivity bound itself; if the projected utility deviates from quadratic, the closed-form sampling claim no longer holds and the privacy guarantee must be re-derived.
Authors: We agree that a quantitative treatment of the projection error is necessary for rigor. In the revised version we will expand §5.3 with a new lemma that bounds the deviation of the projected utility from the original quadratic form using the Johnson-Lindenstrauss lemma. We will show that the deviation can be absorbed into a slightly inflated sensitivity bound Δ′ = Δ + ε_proj, where ε_proj is an explicit function of the target dimension and failure probability. The privacy analysis will be updated to use Δ′, and we will note that the sampling distribution remains exactly multivariate normal (with the adjusted covariance) provided the projection is applied before forming the quadratic utility. This preserves the closed-form sampling property while making the privacy guarantee fully rigorous. revision: yes
-
Referee: [§6, Table 2] §6 (Experiments), Table 2: accuracy is reported as competitive, yet the effective noise scale (determined by ε/Δ) and the numerical value of the sensitivity bound Δ are not tabulated; without these quantities it is impossible to judge whether the observed accuracy is achieved at a privacy level that is meaningfully stronger than the baselines.
Authors: This observation is correct and improves the interpretability of the experimental results. We will augment Table 2 with three additional columns reporting, for each method and dataset: (i) the computed sensitivity bound Δ, (ii) the privacy parameter ε used, and (iii) the resulting effective noise scale ε/Δ. A short paragraph will be added to §6 explaining how Δ was evaluated numerically from the dataset-dependent term. These additions will allow direct comparison of the privacy-utility operating points with the baselines. revision: yes
Circularity Check
No significant circularity; standard DP exponential mechanism applied to a constructed quadratic utility
full rationale
The paper defines a utility u(θ) explicitly as a local quadratic Taylor approximation of the pretrained loss plus a term from the new dataset D_new. The exponential mechanism then yields an MVN because exp(ε u / 2Δ) is Gaussian when u is quadratic; this is a direct mathematical consequence of the definition, not a reduction of a claimed prediction back to fitted inputs. Privacy guarantees rest on an explicit sensitivity bound Δ for neighboring datasets, which is derived from the quadratic form rather than fitted or self-cited as a uniqueness theorem. No self-citation load-bearing steps, no ansatz smuggled via prior work, and no renaming of known results as new derivations. The construction is self-contained against external DP primitives and verifiable by direct computation of the Gaussian parameters.
Axiom & Free-Parameter Ledger
axioms (1)
- standard math The exponential mechanism provides differential privacy when the utility function has bounded sensitivity
Lean theorems connected to this paper
-
IndisputableMonolith/Cost/FunctionalEquation.leanwashburn_uniqueness_aczel unclear?
unclearRelation between the paper passage and the cited Recognition theorem.
We define the approximate loss function as ˜L(D, θ) = (θ−θ∗)⊤g(D) + ½(θ−θ∗)⊤H(D)(θ−θ∗) + λ/2 |θ−θ∗|² … U(D, θ) := −˜L(D, θ)
-
IndisputableMonolith/Foundation/AlphaCoordinateFixation.leancostAlphaLog_high_calibrated_iff unclear?
unclearRelation between the paper passage and the cited Recognition theorem.
The resulting exponential mechanism admits exact sampling from a multivariate normal distribution in closed form.
What do these tags mean?
- matches
- The paper's claim is directly supported by a theorem in the formal canon.
- supports
- The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
- extends
- The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
- uses
- The paper appears to rely on the theorem as machinery.
- contradicts
- The paper's claim conflicts with a theorem or certificate in the canon.
- unclear
- Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.
Reference graph
Works this paper leans on
-
[1]
Calibrating noise to sensitivity in private data analysis,
C. Dwork, F. McSherry, K. Nissim, and A. Smith, “Calibrating noise to sensitivity in private data analysis,” inTheory of Cryptography, S. Halevi and T. Rabin, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2006, pp. 265–284
work page 2006
-
[2]
Differential privacy: A survey of results,
C. Dwork, “Differential privacy: A survey of results,” inTheory and Applications of Models of Computation, M. Agrawal, D. Du, Z. Duan, and A. Li, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2008, pp. 1–19
work page 2008
-
[3]
Differentially private empir- ical risk minimization with input perturbation,
K. Fukuchi, Q. K. Tran, and J. Sakuma, “Differentially private empir- ical risk minimization with input perturbation,” inDiscovery Science, A. Yamamoto, T. Kida, T. Uno, and T. Kuboyama, Eds. Cham: Springer International Publishing, 2017, pp. 82–90
work page 2017
-
[4]
Certified robustness to adversarial examples with differential privacy,
M. L ´ecuyer, V . Atlidakis, R. Geambasu, D. Hsu, and S. Jana, “Certified robustness to adversarial examples with differential privacy,” 05 2019, pp. 656–672
work page 2019
-
[5]
N. Phan, M. N. Vu, Y . Liu, R. Jin, D. Dou, X. Wu, and M. T. Thai, “Heterogeneous gaussian mechanism: Preserving differential privacy in deep learning with provable robustness,” inProceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19. International Joint Conferences on Artificial Intelligence Organization, 7 ...
-
[6]
N. Phan, Y . Wang, X. Wu, and D. Dou, “Differential privacy preservation for deep auto-encoders: an application of human behavior prediction,”Proceedings of the AAAI Conference on Artificial Intelligence, vol. 30, no. 1, Feb. 2016. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/10165
work page 2016
-
[7]
Preserving differential privacy in convolutional deep belief networks,
N. Phan, X. Wu, and D. Dou, “Preserving differential privacy in convolutional deep belief networks,”Machine Learning, vol. 106, no. 9-10, pp. 1681–1704, Oct. 2017, publisher Copyright: © 2017, The Author(s)
work page 2017
-
[8]
Adaptive laplace mechanism: Differential privacy preservation in deep learning,
N. Phan, X. Wu, H. Hu, and D. Dou, “Adaptive laplace mechanism: Differential privacy preservation in deep learning,”2017 IEEE International Conference on Data Mining (ICDM), pp. 385–394, 2017. [Online]. Available: https://api.semanticscholar.org/CorpusID:1567787
work page 2017
-
[9]
Towards practical differentially private convex optimization,
R. Iyengar, J. P. Near, D. X. Song, O. Thakkar, A. Thakurta, and L. Wang, “Towards practical differentially private convex optimization,”2019 IEEE Symposium on Security and Privacy (SP), pp. 299–316, 2019. [Online]. Available: https://api.semanticscholar. org/CorpusID:52087126
work page 2019
-
[10]
Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang
M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, “Deep learning with differential privacy,” inProceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, ser. CCS ’16. New York, NY , USA: Association for Computing Machinery, 2016, p. 308–318. [Online]. Available: https://doi.org/10.1145/297674...
-
[11]
Concentrated differentially private gradient descent with adaptive per-iteration privacy budget,
J. Lee and D. Kifer, “Concentrated differentially private gradient descent with adaptive per-iteration privacy budget,” inProceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, ser. KDD ’18. New York, NY , USA: Association for Computing Machinery, 2018, p. 1656–1665. [Online]. Available: https://doi.org/10.1145/3...
-
[12]
Do not let privacy overbill utility: Gradient embedding perturbation for private learning,
D. Yu, H. Zhang, W. Chen, and T.-Y . Liu, “Do not let privacy overbill utility: Gradient embedding perturbation for private learning,” inInternational Conference on Learning Representations, 2021
work page 2021
-
[13]
Medical imaging deep learning with differential privacy,
A. Ziller, D. Usynin, R. Braren, M. Makowski, D. Rueckert, and G. Kaissis, “Medical imaging deep learning with differential privacy,” Scientific Reports, vol. 11, no. 1, p. 13524, 2021
work page 2021
-
[14]
Differential privacy for deep learning in medicine,
M. Mohammadi, M. Vejdanihemmat, M. Lotfinia, M. Rusu, D. Truhn, A. Maier, and S. T. Arasteh, “Differential privacy for deep learning in medicine,”arXiv preprint arXiv:2506.00660, 2025
-
[15]
Analysis of application examples of dif- ferential privacy in deep learning,
Z. Shen and T. Zhong, “Analysis of application examples of dif- ferential privacy in deep learning,”Computational intelligence and neuroscience, vol. 2021, no. 1, p. 4244040, 2021
work page 2021
-
[16]
Mechanism design via differential privacy,
F. McSherry and K. Talwar, “Mechanism design via differential privacy,” in48th Annual IEEE Symposium on Foundations of Computer Science (FOCS’07), 2007, pp. 94–103
work page 2007
-
[17]
Data mining with differential privacy,
A. Friedman and A. Schuster, “Data mining with differential privacy,” inProceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ser. KDD ’10. New York, NY , USA: Association for Computing Machinery, 2010, p. 493–502. [Online]. Available: https://doi.org/10.1145/1835804.1835868
-
[18]
M. Kapralov and K. Talwar,On differentially private low rank approximation, pp. 1395–1414. [Online]. Available: https://epubs. siam.org/doi/abs/10.1137/1.9781611973105.101
-
[19]
Differentially private hierarchical count-of-counts histograms,
Y .-H. Kuo, C.-C. Chiu, D. Kifer, M. Hay, and A. Machanavajjhala, “Differentially private hierarchical count-of-counts histograms,”Proc. VLDB Endow., vol. 11, no. 11, p. 1509–1521, Jul. 2018. [Online]. Available: https://doi.org/10.14778/3236187.3236202
-
[20]
Differential privacy without sensitivity,
K. Minami, H. Arai, I. Sato, and H. Nakagawa, “Differential privacy without sensitivity,” inProceedings of the 30th International Confer- ence on Neural Information Processing Systems, ser. NIPS’16. Red Hook, NY , USA: Curran Associates Inc., 2016, p. 964–972
work page 2016
-
[21]
R. A. Bridges, V . J. Tombs, and C. B. Stanley, “Are normalizing flows the key to unlocking the exponential mechanism? a path through the accuracy-privacy ceiling constraining differentially private ml,” 2024
work page 2024
-
[22]
The algorithmic foundations of differential privacy,
C. Dwork and A. Roth, “The algorithmic foundations of differential privacy,”Found. Trends Theor. Comput. Sci., vol. 9, no. 3–4, p. 211–407, Aug. 2014. [Online]. Available: https://doi.org/10.1561/ 0400000042
work page 2014
-
[23]
How to dp-fy ml: A practical tutorial to machine learning with differential privacy,
N. Ponomareva, S. Vassilvitskii, Z. Xu, B. McMahan, A. Kurakin, and C. Zhang, “How to dp-fy ml: A practical tutorial to machine learning with differential privacy,” inProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, ser. KDD ’23. New York, NY , USA: Association for Computing Machinery, 2023, p. 5823–5824. [Online]. Ava...
-
[24]
Opacus: User-friendly differential privacy library in pytorch
A. Yousefpour, I. Shilov, A. Sablayrolles, D. Testuggine, K. Prasad, M. Malek, J. Nguyen, S. Ghosh, A. Bharadwaj, J. Zhao, G. Cormode, and I. Mironov, “Opacus: User-friendly differential privacy library in PyTorch,”arXiv preprint arXiv:2109.12298, 2021
-
[25]
Model-agnostic meta-learning for fast adaptation of deep networks,
C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” inProceedings of the 34th International Conference on Machine Learning (ICML), 2017
work page 2017
-
[26]
Gradient-based learning applied to document recognition,
Y . Lecun, L. Bottou, Y . Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,”Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998
work page 1998
-
[27]
T. J. Pollard, A. E. W. Johnson, J. D. Raffa, L. A. Celi, R. G. Mark, and O. Badawi, “The eicu collaborative research database, a freely available multi-center database for critical care research,”Sci Data, vol. 5, p. 180178, 2018
work page 2018
-
[28]
Mimic-iv, a freely accessible electronic health record dataset,
A. E. W. Johnson, L. Bulgarelli, L. Shen, A. Gayles, A. Shammout, S. Horng, T. J. Pollard, S. Hao, B. Moody, B. Gow, L.-w. H. Lehman, L. A. Celi, and R. G. Mark, “Mimic-iv, a freely accessible electronic health record dataset,”Sci Data, vol. 10, no. 1, 2023
work page 2023
-
[29]
Introducing the blendedicu dataset, the first harmonized, international intensive care dataset,
M. Oliver, J. Allyn, R. Carencotte, N. Allou, and C. Ferdynus, “Introducing the blendedicu dataset, the first harmonized, international intensive care dataset,”Journal of Biomedical Informatics, vol. 146, p. 104502, 2023. [Online]. Available: https://www.sciencedirect.com/ science/article/pii/S153204642300223X
work page 2023
-
[30]
An Extensive Data Processing Pipeline for MIMIC-IV,
M. Gupta, B. Gallamoza, N. Cutrona, P. Dhakal, R. Poulain, and R. Beheshti, “An Extensive Data Processing Pipeline for MIMIC-IV,” inProceedings of the 2nd Machine Learning for Health symposium, ser. Proceedings of Machine Learning Research, vol. 12 VOLUME , <Society logo(s) and publication title will appear here.>
-
[31]
PMLR, 28 Nov 2022, pp. 311–325. [Online]. Available: https://proceedings.mlr.press/v193/gupta22a.html Hoang A. Tranreceived the M.S. degree in Mathematics from the Univer- sit´e d’Orl´eans, Orl ´eans, France, in 2008, and the Ph.D. degree in Applied Mathematics from the University of Pittsburgh, Pittsburgh, PA, USA, in
work page 2022
-
[32]
He is currently a mathematician with Data Analysis and Machine Learning Group, Computer Science and Mathematics Division, Oak Ridge National Laboratory, Oak Ridge, TN, USA. His research interests include compressed sensing, machine learning, high-dimensional approximations and numerical solution of partial differential equations. Jorge Ramirezis a Colombi...
work page 2022
-
[33]
She is currently a Postdoctoral Researcher at Oak Ridge National Laboratory. Her research interests include federated learning, differential privacy, synthetic data generation, and distributed optimization. Alberto Bocchinfusoreceived his BS in Computer Engineering from University of Calabria (Italy) in 2016, his MS in Automation and Control Engineering f...
work page 2016
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.