Pith. sign in

REVIEW 3 major objections 5 minor 22 references

On the interplay of Explainability, Privacy and Predictive Performance with Explanation-assisted Model Extraction

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Differential privacy at model and explainer reduces counterfactual-based model extraction, with measurable utility loss.

desk verdict A useful but incremental empirical study confirming that DP noise reduces counterfactual-assisted model extraction, undermined by missing privacy accounting and thin statistics. read the letter →

arxiv 2505.08847 v1 pith:H2GSWZE3 submitted 2025-05-13 cs.CR cs.AI

Fatima Ezzeddine , Rinad Akel , Ihab Sbeity , Silvia Giordano , Marc Langheinrich , Omran Ayoub This is my paper · ORCID
classification cs.CRcs.AI
keywords differentialprivacymodelextractionattackscounterfactualexplanationsknowledgedistillationexplainableAIMLaaSprivacy-utilitytrade-offDP-SGD
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

With explanation-equipped ML-as-a-service, counterfactual explanations can be turned against the provider: an attacker queries the model, collects prediction–counterfactual pairs, and trains a substitute via knowledge distillation. This paper tries to establish that injecting differential privacy into the classifier, into the counterfactual explainer, or into both reduces how well that extraction succeeds, with the strongest defense when both are protected. Using two tabular datasets and noise at three levels, it quantifies the price: accuracy, precision, recall, and F1 fall as noise rises, and counterfactual quality degrades in prediction gain and realism. The practical point is that a provider can choose where to spend privacy noise, and the measurements give a first map of that trade-off.

What carries the argument

The machinery is two noise-injection points plus a measurement protocol. On the model side, DP-SGD clips gradients and adds noise during training, at three settings labeled 0.1, 0.5, and 0.9. On the explainer side, the same noise levels are injected into the generator of a GAN that produces counterfactuals. Attack success is measured as agreement between the extracted substitute and the target on a held-out set, with queries ranging from 50 to 1,000; explanation quality is scored by prediction gain and by realism, the reconstruction error of an autoencoder. These pieces work together to turn 'where to add DP noise' into a measurable three-way trade-off.

What would settle it

Use the paper's exact DP settings and compute the achieved epsilon/delta for each noise level. If the privacy budget is large (for example, epsilon above 10) at the noise levels studied, the 'privacy protection' column of the trade-off is weak and the interpretation changes. Alternatively, repeat the extraction experiments under a fixed epsilon budget and check whether the reduction in agreement survives.

Watch

Extended reading notes

Core claim

Working with a DP-SGD-trained classifier and a differentially private GAN-based counterfactual generator, the paper's central claim is that differential privacy at the model, at the explainer, or at both simultaneously lowers the agreement between a target model and a knowledge-distillation substitute trained on counterfactual explanations. On one tabular dataset, no-DP agreement climbs from 70 to about 80 as queries grow to 1,000, while a DP-explainer noise of 0.9 keeps agreement near 50–72; comparable drops appear on a second dataset. Applying DP to both points produces what the authors call a synergistic improvement in resistance to extraction, beyond either defense alone. The same noise also degrades utility: prediction gain falls from 0.488 to 0.055 on one dataset when explainer noise is 0.9, and realism scores worsen, meaning counterfactuals become less realistic. The paper concludes that privacy protection against CF-assisted extraction is real but inseparable from losses in model quality and explanation quality.

Load-bearing premise

The results treat the chosen noise levels as privacy parameters without ever reporting the resulting epsilon/delta budget, so the claim that these settings provide meaningful differential privacy is an unverified premise.

Editorial extensions

If this is right

  • Defending the model and the explainer together gives lower extraction agreement than defending either alone, so providers facing CF-assisted extraction should consider both noise points.
  • Higher DP noise at either point reduces the attacker's fidelity, but only by paying a visible cost in accuracy, precision, recall, and F1.
  • Counterfactual quality degrades with noise: prediction gain drops and realism scores increase, so privacy protection directly weakens the usefulness of explanations to legitimate users.
  • Extraction agreement still rises with query count even under DP, so noise slows but does not fully stop a patient attacker.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper never reports the privacy budget (epsilon, delta) that its noise levels imply; a reader cannot yet tell whether the 'privacy' being bought is a strong guarantee or a weak one. An immediate extension is to recompute the experiments under fixed epsilon budgets rather than fixed noise levels.
  • If the actual epsilon at noise 0.1 is large, the apparent defense may be mostly utility loss rather than provable privacy, which would change the trade-off's interpretation.
  • The same two-point DP design could be tested against other explanation-based attacks, such as gradient- or saliency-based extraction; the mechanism suggests protection would generalize, but the paper does not test it.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies the interplay among differential privacy (DP), model extraction attacks (MEA) that leverage counterfactual explanations (CFs), and predictive performance. It compares four scenarios on two tabular datasets (Housing, EEG): no DP, DP applied to the target model via DP-SGD, DP applied to the CF-generating CounterGAN, and DP applied to both simultaneously. The experimental campaign measures model accuracy/precision/recall/F1, MEA agreement, prediction gain, and CF realism under noise levels 0.1, 0.5, and 0.9. The paper reports that higher DP noise reduces MEA success but also lowers model utility and CF quality, and it claims in Section 5.3 that applying DP at both the model and the explainer yields a synergistic improvement in MEA resistance.

Significance. If the central claim is established, the paper provides practical guidance for MLaaS providers on where to inject DP noise to mitigate CF-assisted model extraction. The paper has concrete strengths: it evaluates a realistic knowledge-distillation-based MEA that consumes counterfactuals, it covers two datasets, and it compares four deployment scenarios in a single framework. The qualitative finding that larger noise reduces attack success and utility is visible in the reported plots and is consistent with intuition. However, the paper's quantitative support is currently too weak for the claimed privacy trade-off: no privacy accounting is reported, so the meaning of the 'privacy' axis is unresolved, and the 'synergistic' claim in Section 5.3 is asserted from visual inspection rather than from a statistical interaction test. The contribution is therefore promising but not yet fully supported.

major comments (3)
  1. [Sections 3, 4.1, and 4.2] The paper never computes or reports a privacy budget (epsilon, delta) for any of the DP configurations. Section 3 calls the noise level the 'privacy parameter's noise level epsilon', and Section 4.1/4.2 state only that DP-SGD and the CounterGAN generator use 'noise levels of 0.1, 0.5 and 0.9' with TensorFlow Privacy's DPKerasAdamOptimizer. This conflates the noise multiplier with the privacy guarantee. In DP-SGD, the realized epsilon depends on the noise multiplier, the number of training steps, the sampling ratio, and the target delta; a noise level alone does not define a privacy regime. Without running a privacy accountant, the paper cannot support its central claim that DP provides meaningful protection against MEA, because the observed reduction in agreement could simply reflect model degradation rather than a formal privacy guarantee. The authors should report (epsilon, delta) for each reported configuration or explicitly frame the results as an ablation on noise magnitude rather than as a privacy evaluation.
  2. [Section 5.3, Discussion] The claimed 'synergistic improvement in resistance to MEA' when both the model and the explainer are protected is load-bearing but is asserted from visual comparison of the agreement plots in Figures 3 and 4 without any statistical test for interaction. The results are also reported as averages of 5 runs without error bars, confidence intervals, or per-run variability, so the reader cannot judge whether the observed differences between DP-Model and DP-Model-Explainer are meaningful. To support the synergy claim, the paper needs either an interaction test (e.g., two-way ANOVA or paired comparisons with the relevant contrast) or at least a clear presentation of variance and effect sizes. This is not a cosmetic issue: the synergy claim is highlighted as the main novel takeaway.
  3. [Section 4.1, model selection] Section 4.1 states that 'the best-performing model in term of accuracy was chosen' but does not specify whether this selection was performed separately for each DP configuration or globally across all configurations. If the selection is global, comparing DP-Model-0.1, DP-Model-0.5, and DP-Model-0.9 is confounded by the fact that different hyperparameter settings may have been selected for different noise levels. If the selection is per configuration, the same criterion should be described explicitly. This issue affects the quantitative comparisons in Figures 2-5 and should be clarified or corrected, especially because the paper does not report the selected hyperparameter values (learning rate, l2_norm_clip) for any configuration.
minor comments (5)
  1. [Section 4.1] There is a typo in 'optmizer' and the phrase 'a the softmax activation function' should be corrected to 'and a softmax activation function'.
  2. [Section 5.2 and Figures 3-4] The text says 'agreement ranges between 50 and 72 when the number of queries increases up to 1000' and later 'agreement starts at 70 with 50 queries', but it is not always clear whether these numbers are percentages or raw agreement counts; please standardize the units and axis labels in Figures 3 and 4.
  3. [Section 5.1] For the Housing dataset, the paper says 'results show a declining trend' but does not report the specific accuracy/precision/recall/F1 values, unlike the EEG case; please include the numbers or a supplementary table for completeness.
  4. [Section 4.2] The realism metric definition uses an autoencoder reconstruction but does not describe how the autoencoder was trained (architecture, training data, whether it is shared between datasets); please add this detail.
  5. [Section 4.1] The threat model description says the attacker generates random data points 'within a range of -3 to 3 for each feature' without specifying whether this range is per dataset, after standardization, or across all features; please clarify, since this range materially affects the query distribution and the MEA results.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical trade-off study whose conclusions rest on measured agreement and external quality metrics, not on definitional identities or fitted predictions.

full rationale

The paper is an empirical evaluation rather than a claimed derivation, and none of its central results are obtained by definitional identity or by fitting a parameter and renaming it a prediction. The MEA attack is taken from the authors' prior work [10], but the paper independently demonstrates the attack's effectiveness in the No-DP baseline, with agreement reaching roughly 96% on the EEG dataset at 1000 queries, so the self-citation is not load-bearing: the attack's validity is established within the paper itself. Similarly, the private CounterGAN from [10] is evaluated directly through realism and prediction-gain metrics, where realism is measured against an autoencoder reconstruction and prediction gain is measured as the change in the classifier's confidence score; these are external measurements, not definitions of the conclusions. The observed decline in MEA agreement under DP-Model, DP-Explainer, and their combination is an empirical trend, not a consequence of defining MEA success in terms of the noise level; the DP-Explainer corrupts only the CFs while leaving the model's prediction labels intact, and the paper quantifies that the effect is gradual and query-dependent rather than a forced identity. The absence of a privacy-accountant run and of reported epsilon/delta budgets is a substantive validity gap in the privacy claim, but it is not circularity: it concerns whether the chosen noise settings actually deliver a formal differential-privacy guarantee, not whether the derivation reduces to its inputs.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on hand-chosen noise levels and a specific attack threat model drawn from the authors' prior work. No new entities are introduced. The unquantified epsilon is the most significant assumption.

free parameters (3)
  • DP noise level (sigma multiplier) = 0.1, 0.5, 0.9 (explored)
    Hand-chosen noise levels for DP-SGD and DP in CounterGAN; the trade-off conclusions depend on these values, and no epsilon budget is reported.
  • l2_norm_clip = 1 and 1.5 for model; 1, 1.5, 3 for explainer
    Hand-chosen clipping bounds for DP optimizers; these affect the sensitivity and therefore the effective privacy of the training process.
  • learning rates = multiple (0.001, 0.002, 0.01 for model; 0.05, 0.005, 0.01, 0.001 for explainer)
    Values swept during training; the exact values used for the reported results are not stated, making the experiments difficult to reproduce.
assumptions (3)
  • domain assumption The threat model assumes an attacker with no knowledge of the training distribution who queries uniform random points in [-3,3] per feature and uses the KD-based attack from [10].
    The effectiveness of DP as a mitigation is measured only under this specific threat model; a stronger attacker could reduce the measured defense. Section 4.1 defines the threat model.
  • domain assumption Applying TensorFlow Privacy's DPKerasAdamOptimizer with noise levels 0.1, 0.5, 0.9 provides meaningful differential privacy guarantees.
    The paper never reports the resulting epsilon/delta values, so the actual privacy level is unverified. Sections 4.1 and 4.2.
  • domain assumption The realism metric based on autoencoder reconstruction error is a valid proxy for data distribution fit.
    The paper defines realism as reconstruction error but gives no details on the autoencoder's architecture or training. Section 4.2 defines realism.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the interplay of Explainability, Privacy and Predictive Performance with Explanation-assisted Model Extraction." pith.science (2026). https://pith.science/paper/H2GSWZE3

@misc{pith2026250508847,
  author       = {Pith},
  title        = {Pith review of: On the interplay of Explainability, Privacy and Predictive Performance with Explanation-assisted Model Extraction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H2GSWZE3}},
  note         = {Machine review of arXiv:2505.08847}
}
read the original abstract

Machine Learning as a Service (MLaaS) has gained important attraction as a means for deploying powerful predictive models, offering ease of use that enables organizations to leverage advanced analytics without substantial investments in specialized infrastructure or expertise. However, MLaaS platforms must be safeguarded against security and privacy attacks, such as model extraction (MEA) attacks. The increasing integration of explainable AI (XAI) within MLaaS has introduced an additional privacy challenge, as attackers can exploit model explanations particularly counterfactual explanations (CFs) to facilitate MEA. In this paper, we investigate the trade offs among model performance, privacy, and explainability when employing Differential Privacy (DP), a promising technique for mitigating CF facilitated MEA. We evaluate two distinct DP strategies: implemented during the classification model training and at the explainer during CF generation.

Figures

Figures reproduced from arXiv: 2505.08847 by the authors.

Figure 1
Figure 1. Model Extraction Attack within an MLaaS provider, depicting two different scenarios where DP is employed at the model or at the explainer to counter potential attacks. Given a dataset 𝐷 = {(𝑥𝑖 , 𝑦𝑖)} 𝑁 𝑖=1, where 𝑥𝑖 are feature vectors and 𝑦𝑖 are corresponding labels. A target model 𝑓(𝑥; 𝜃) trained and optimized to achieve high performance on 𝐷 is deployed as MLaaS and is queryable through an API (as shown in [PITH… view at source ↗
Figure 2
Figure 2. Model Performance achieved by the ML model across the two datasets for varying noise scales [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Housing dataset MEA agreement for various DP strategies, noise levels and number queries. 50 100 200 300 500 1,000 40 50 60 70 80 90 100 Queries Number Agreement No DP DP-Explainer-0.1 DP-Explainer-0.5 DP-Explainer-0.9 (a) No DP 50 100 200 300 500 1,000 40 50 60 70 80 Queries Number (b) DP-Model-0.1 50 100 200 300 500 1,000 50 60 70 Queries Number (c) DP-Model-0.5 50 100 200 300 500 1,000 40 50 60 70 Queries Number … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: EEG dataset MEA agreement for various DP strategies, noise levels and number queries. reaches 80 when 1000 queries are used. We now focus on the cases where DP is employed at the model level ( [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Prediction Gain and Realism achieved by explainers in the Housing and EEG datasets under various DP-Explainer noise levels. noise is higher. When the model is subjected to DP noise at levels of 0.1, 0.5, and 0.9, the prediction gains are consistently lower. We now focu…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 19 canonical work pages

  1. [10]

    Knowledge Distillation-Based Model Extraction Attack using GAN-based Private Counterfactual Explanations

    F. Ezzeddine, O. Ayoub, S. Giordano, Knowledge distillation-based model extraction attack using private counterfactual explanations, arXiv preprint arXiv:2404.03348 (2024)

    work page Pith review arXiv 2024

  2. [1]

    Tramèr, F

    F. Tramèr, F. Zhang, A. Juels, M. K. Reiter, T. Ristenpart, Stealing machine learning models via prediction APIs, in: 25th USENIX Security Symposium (USENIX Security 16), USENIX Association, 2016, pp. 601–618

    work page 2016

  3. [2]

    Shokri, M

    R. Shokri, M. Stronati, C. Song, V. Shmatikov, Membership inference attacks against machine learning models, in: 2017 IEEE Symposium on Security and Privacy (SP), IEEE, 2017, pp. 3–18. URL: https://ieeexplore.ieee.org/document/7958568. doi:10.1109/SP.2017.41

    arXiv 2017

  4. [3]

    Dwork, Differential privacy, in: International colloquium on automata, languages, and programming, Springer, 2006, pp

    C. Dwork, Differential privacy, in: International colloquium on automata, languages, and programming, Springer, 2006, pp. 1–12

    work page 2006

  5. [4]

    Abadi, A

    M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, L. Zhang, Deep learning with differential privacy, in: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, 2016, pp. 308–318

    work page 2016

  6. [5]

    Guidotti, A

    R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, F. Giannotti, D. Pedreschi, A survey of methods for explaining black box models, ACM Computing Surveys (CSUR) 51 (2018) 1–42

    work page 2018

  7. [6]

    Ezzeddine, Privacy implications of explainable ai in data-driven systems (2024)

    F. Ezzeddine, Privacy implications of explainable ai in data-driven systems (2024)

    work page 2024

  8. [7]

    Shokri, M

    R. Shokri, M. Strobel, Y. Zick, On the privacy risks of model explanations, in: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, 2021, pp. 231–241

    work page 2021

Show all 22 references
  1. [8]

    C. N. Spartalis, T. Semertzidis, P. Daras, Balancing xai with privacy and security considerations, in: European Symposium on Research in Computer Security, Springer, 2023, pp. 111–124

    2023

  2. [9]

    Wachter, B

    S. Wachter, B. Mittelstadt, C. Russell, Counterfactual explanations without opening the black box: Automated decisions and the gdpr, Harv. JL & Tech. 31 (2017) 841

    2017

  3. [11]

    Aïvodji, A

    U. Aïvodji, A. Bolot, S. Gambs, S. Mehnaz, R. Yvinec, Model extraction from counterfactual explanations, in: Proceedings of the 2020 conference on fairness, accountability, and transparency, 2020, pp. 99–109

    2020

  4. [12]

    Abbasi, P

    W. Abbasi, P. Mori, A. Saracino, Further insights: Balancing privacy, explainability, and utility in machine learning-based tabular data analysis, in: Proceedings of the 19th International Conference on Availability, Reliability and Security, 2024, pp. 1–10

    2024

  5. [13]

    A. C. Oksuz, A. Halimi, E. Ayday, Autolycus: Exploiting explainable artificial intelligence (xai) for model extraction attacks against interpretable models, Proceedings on Privacy Enhancing Technologies (2024)

    2024

  6. [14]

    A. Yan, R. Hou, X. Liu, H. Yan, T. Huang, X. Wang, Towards explainable model extraction attacks, International Journal of Intelligent Systems 37 (2022) 9936–9956

    2022

  7. [15]

    Y. Wang, H. Qian, C. Miao, Dualcf: Efficient model extraction attack from counterfactual explana- tions, in: Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, 2022, pp. 1318–1329

    2022

  8. [16]

    Dissanayake, S

    P. Dissanayake, S. Dutta, Model reconstruction using counterfactual explanations: A perspective from polytope theory, Advances in Neural Information Processing Systems (NeurIPS) (2024)

    2024

  9. [17]

    S. An, Y. Cao, Counterfactual explanation at will, with zero privacy leakage, Proceedings of the ACM on Management of Data 2 (2024) 1–29

    2024

  10. [18]

    F. Yang, Q. Feng, K. Zhou, J. Chen, X. Hu, Differentially private counterfactuals via functional mechanism, arXiv preprint arXiv:2208.02878 (2022)

    2022 arXiv

  11. [19]

    Pentyala, S

    S. Pentyala, S. Sharma, S. Kariyappa, F. Lécué, D. Magazzeni, Privacy-preserving algorithmic recourse, CoRR (2023)

    2023

  12. [20]

    Nemirovsky, N

    D. Nemirovsky, N. Thiebaut, Y. Xu, A. Gupta, Countergan: Generating counterfactuals for real-time recourse and interpretability using residual gans, in: Uncertainty in Artificial Intelligence, PMLR, 2022, pp. 1488–1497

    2022

  13. [21]

    URL: https://scikit-learn.org/stable/ modules/generated/sklearn.datasets.fetch_california_housing.html, accessed: 2024-01-04

    Scikit-learn Developers, California housing dataset, 2024. URL: https://scikit-learn.org/stable/ modules/generated/sklearn.datasets.fetch_california_housing.html, accessed: 2024-01-04

    2024

  14. [22]

    Roesler, Eeg eye state, UCI Machine Learning Repository, 2013

    O. Roesler, Eeg eye state, UCI Machine Learning Repository, 2013. URL: https://doi.org/10.24432/ C57G7J. doi:10.24432/C57G7J, accessed: 2024-01-04

    2013 doi

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.