Pith. sign in

REVIEW 4 major objections 6 minor 22 references

Privacy Drift: Evolving Privacy Concerns in Incremental Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Pith's one-line read Privacy leaks track training accuracy as models learn incrementally

desk verdict A useful empirical observation wrapped in a new term; the correlation is real, but the paper oversells 'formalization' and leaves the MIA pipeline a black box. read the letter →

arxiv 2412.05183 v1 pith:OL3VCUL6 submitted 2024-12-06 cs.LG cs.CR

classification cs.LGcs.CR
keywords privacydriftmembershipinferenceattackfederatedlearningincrementalconceptaccuracy-privacytrade-offnon-IIDdata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces "privacy drift," a concept analogous to concept drift: as a model is trained incrementally on new data, its vulnerability to membership inference attacks changes over time. Using a CIFAR-20 dataset split into four non-IID partitions, the authors show that the area under the curve (AUC) of membership inference attacks tracks training accuracy across training phases. This correlation holds in both centralized training and federated learning with two, five, and ten clients, leading the authors to conclude that privacy drift is independent of the number of clients. The finding matters because it suggests that accuracy improvements in incrementally updated models can quietly increase privacy risk, so static privacy evaluations may miss a moving target.

What carries the argument

The central object is "privacy drift," defined as the gradual variation in a model's vulnerability to membership inference over its training lifecycle, measured by the AUC of a membership inference attack. The experimental machinery consists of a CIFAR-20 dataset (the 20 superclasses of CIFAR-100) split into four non-IID partitions (A, B, C, D), on which an EfficientNet model is trained sequentially in different permutations, with test sets evaluated under both a uniform paradigm (a fixed mixed test set) and an additive paradigm (test sets that grow with each training phase). Privacy drift is operationalized by computing the Pearson correlation between training accuracy and MIA AUC across training phases, in centralized mode and in federated mode using federated averaging, with client counts of two, five, and ten.

What would settle it

Re-run the incremental training protocol with a fully specified membership inference attack (e.g., shadow-model training with balanced member/non-member sets) and test whether the correlation between training accuracy and MIA AUC persists; if the correlation disappears or AUC stays near chance under a properly specified attack, the privacy drift phenomenon as defined is an artifact of the attack configuration rather than a property of the evolving model.

Watch

Extended reading notes

Core claim

The paper's central claim is that privacy drift is a real, measurable phenomenon: the leakage of private information from a machine learning model, quantified by membership inference attack AUC, varies as the model undergoes incremental training on non-IID data. Across eight permutations of four data partitions and under two testing paradigms (uniform and additive), the authors observe a strong Pearson correlation between training accuracy and MIA AUC, even when test accuracy diverges from training accuracy. Repeating the same experiments in a federated setting with one, two, five, and ten clients, they find the correlation persists, and they conclude that the presence of privacy drift does not depend on the number of participating clients. The paper frames this as a dynamic accuracy-privacy trade-off: improvements in model performance tend to be accompanied by greater susceptibility to membership inference.

Load-bearing premise

The paper assumes that the membership inference attack AUC values it reports are a faithful measure of private information leakage, but it does not describe how the attack model is trained or how member and non-member test sets are balanced, so the observed correlation with training accuracy could be an artifact of the attack setup.

Editorial extensions

If this is right

  • Training accuracy can serve as a live proxy signal for privacy risk in incrementally updated models, since MIA AUC tracks it across phases and settings.
  • Privacy evaluations of continually learning systems must be repeated over time; a single static assessment at deployment will miss drift in leakage risk.
  • Federated learning does not remove privacy drift: the correlation holds whether training is centralized or distributed among up to ten clients.
  • The accuracy-privacy trade-off is phase-dependent, so techniques that boost accuracy mid-training may increase membership inference risk in the same period.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the correlation reflects the model's confidence calibration, then interventions that sharpen confidence (e.g., temperature scaling) could alter privacy drift without changing accuracy; this is a testable extension the paper does not explore.
  • Privacy drift could double as an early-warning diagnostic: a sudden spike in training accuracy during incremental updates may predict an imminent increase in membership inference risk.
  • The client-count independence implies that privacy-preserving defenses in FL should target the training dynamics themselves (e.g., per-phase differential privacy budgets) rather than assuming more clients dilute leakage.
  • A formal definition of privacy drift as a rate of change of leakage per unit of accuracy gain would make the concept quantifiable for comparing learning algorithms.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces the concept of "privacy drift" for federated learning (FL), defined informally as the evolving vulnerability of an incrementally trained model to membership inference attacks (MIA). Using CIFAR-100 converted to a 20-superclass task, the authors partition data into four non-IID splits, train EfficientNet models sequentially over permutations of these splits, and measure training accuracy, test accuracy, and MIA AUC under uniform and additive test paradigms. They also vary the number of FL clients (centralized, 2, 5, 10) and report that MIA AUC tracks training accuracy and that this behavior is independent of client count. The paper claims to formalize privacy drift and to provide empirical evidence of its dynamics.

Significance. If the central empirical claim holds—that MIA vulnerability in incremental FL tracks training accuracy and is insensitive to the number of clients—the paper would provide a useful framing for privacy monitoring in dynamically updated models. The experimental design is controlled and the code is publicly available, which are strengths. However, the absence of a formal definition of privacy drift, the lack of any specification of the MIA attack pipeline, and the absence of statistical inference for the client-count independence claims currently prevent the paper from delivering on its stated contributions. The non-circular portion of the work—the correlation between training accuracy and MIA AUC—is plausibly interesting, but it is not yet convincingly established given the underspecified attack methodology.

major comments (4)
  1. [Section I and Section IV] The paper promises in the contribution list (Section I) to "introduce and formalize" privacy drift, but Section IV, titled "Privacy Drift in FL: Dynamics and Trade-offs," contains no formal definition, equation, or operational metric. The Introduction defines privacy drift as a "gradual increase" in vulnerability, whereas Section IV describes "fluctuations" and a "complex and non-monotonic" relationship. This inconsistency leaves the central construct undefined. Please provide a precise mathematical definition, such as a drift score or rate based on MIA AUC or another specified measure, and state exactly which hypothesis (e.g., monotonic increase, correlation with training accuracy, or client-count independence) the experiments are designed to test.
  2. [Section III-B and Section V] The MIA AUC values that underlie all of the paper's quantitative claims are produced by an attack pipeline that is never described. Section III-B only states that the adversary has query access and "creates an attack model," but does not specify the attack architecture (e.g., shadow model training as in Shokri et al. [4]), the features used (confidence vector, predicted label, etc.), the number of shadow models, the member/non-member balance, or whether a fresh attack is trained at each incremental phase. Without this information, the reported AUC values could be artifacts of a misconfigured or biased attack setup, and the claimed correlation with training accuracy might reflect the attack's own behavior rather than a property of the target model. Please include a full description of the attack pipeline, ideally with pseudocode, and validate the attack by reporting its performance on a control model trained on data disjoint from the target members.
  3. [Section V-C] The headline claim that "privacy drift is independent of the number of clients in FL" is not supported by any statistical analysis. Figure 6 shows variation in the correlation across the individual experiments, but the text reports no error bars, confidence intervals, hypothesis tests, or effect sizes. The statement that there is a "strong correlation" is qualitative; no Pearson r values or p-values are given for Figures 4 or 6. To support the client-count independence claim, please report the per-permutation and per-client-number correlation coefficients with measures of uncertainty, and formally test whether the correlation differs across client counts (e.g., via bootstrap resampling or Fisher z-transformation).
  4. [Section V-A and Figures 5–7] Figures 5 and 7 show results averaged over eight permutations, but no variance or standard deviation is displayed. With only eight permutations of four partitions, the stability and representativeness of the averaged curves are unclear. The choice of eight permutations is also not justified; the text says this is "a reasonable number of variations." Please add error bars or shaded regions to the averaged plots, report the per-permutation results (at least in an appendix), and justify the permutation sampling procedure.
minor comments (6)
  1. [Abstract and Section V-A] The abstract says experiments are on CIFAR-100, but the paper actually reduces to the 20 superclasses and calls this CIFAR-20. Please clarify this in the abstract and first mention of the dataset.
  2. [Section V-C] "fed-average" should be written as "FedAvg" with the appropriate citation to McMahan et al. [20], and the model aggregation method should be described in one sentence for readers unfamiliar with the abbreviation.
  3. [Section III-B] The term "honest but curious" is typically used in secure multiparty computation to denote a semi-honest adversary; here it is used more loosely to mean an attacker who follows the query interface but attempts inference. Please define the term explicitly in the context of this threat model.
  4. [Figure 4] The caption states that Figure 4 presents the Pearson correlation, but no numerical values or axes units are described in the text. Please report the actual correlation coefficients, or at minimum state the ranges observed, in the caption or the main text.
  5. [Section II-C] The paragraph on "ML Model Attack Accuracy and the Privacy Drift" is largely definitional and repeats the abstract. It could be condensed, and the listed factors (data drift, model evolution, etc.) should be mapped to specific experiments in Section V.
  6. [References] Reference [18] is cited as prior work by the same authors; in Section IV this is referred to as "Ahamed et al." but since it is the authors' own work, consider phrasing it as "we previously showed" to avoid ambiguous self-citation.

Circularity Check

1 steps flagged · score 6.0 of 10

Privacy drift is defined as MIA-based leakage variation, so the empirical demonstration that MIA AUC fluctuates restates the definition; the accuracy-correlation and client-count claims are independent but secondary.

  1. self definitional [Section IV, 'Privacy Drift in FL: Dynamics and Trade-offs' (with the defining statements in Sections I and II-C)]
    "Our experiments reveal that privacy drift manifests as fluctuations in the susceptibility of models to privacy attacks during incremental updates. As the model undergoes continuous training with new data, the leakage of private information, as measured by the accuracy of MIA, varies."

    Section I defines 'privacy drift' as 'the gradual increase in a model's vulnerability to information leakage over its training lifecycle,' and Section II-C equates that leakage with MIA attack accuracy ('as MIA attack accuracy rises... thus emphasizing the presence of privacy drift'). The Section IV claim that privacy drift manifests as MIA-measured fluctuation is therefore the same statement as the definition: the 'empirical evidence' that privacy drift exists is true by construction rather than an independent discovery. The later claims about correlation with training accuracy and independence from client count are separate empirical observations and do not reduce to the definition.

full rationale

The paper's headline concept is self-definitional: 'privacy drift' is operationalized as variation in MIA accuracy/AUC, and the first empirical result is simply that MIA accuracy/AUC varies over incremental training. That part is a relabeling of the measured quantity rather than a testable prediction. The more substantive results—the strong correlation between training accuracy and MIA AUC, and the apparent independence from FL client count—are supported by the paper's own experiments (Figures 3, 4, 5, 6, and 7) and are not entailed by the definition alone. The paper does rely on the same-authors' prior work [18] for the accuracy-privacy trade-off and client-count independence, but the present experiments independently reproduce those observations, so the self-citation is not the sole load-bearing support. The underspecified MIA attack pipeline (no shadow-model training, attack features, or member/non-member balance) is a serious verifiability and correctness risk, but it is not itself a circularity: the paper does not construct MIA AUC from training accuracy by an explicit equation. Overall, the circularity is partial: the existence of 'privacy drift' reduces to its definition, while the accuracy-privacy correlation and client-count independence retain independent empirical content.

Assumptions & free parameters 2 free parameters · 3 assumptions · 1 invented entities

The paper introduces no fitted parameters in the derivation sense; the hand-chosen experimental design numbers above are the closest analogues. The main conceptual load is carried by the assumption that MIA AUC measures privacy and that random CIFAR partitions emulate data drift.

free parameters (2)
  • Number of incremental partitions (4) = 4
    Hand-chosen to create four training phases; the granularity of drift measurement depends on this choice, and no sensitivity analysis is provided.
  • Number of training permutations sampled (8) = 8
    The authors state 'trying eight permutations, as a reasonable number of variations' (Section V-B); correlation statistics are computed over these eight only, with no confidence intervals.
assumptions (3)
  • domain assumption Membership inference attack AUC is a valid and sufficient measure of private information leakage.
    The paper defines privacy drift in terms of MIA AUC and never considers other leakage vectors; see Section IV where privacy is operationalized as MIA accuracy.
  • domain assumption Random non-IID partitions of CIFAR-100 simulate meaningful data/concept drift for incremental learning.
    Section V-A creates four partitions by random sampling; the paper does not show that these partitions correspond to real-world distribution shifts, only that they are non-IID.
  • domain assumption Training accuracy is a monotonic proxy for memorization that drives membership inference success.
    The paper's core correlation (Figures 3-6) assumes the mechanism linking training accuracy to MIA AUC; this is an empirical regularity, not a derived result.
invented entities (1)
  • Privacy drift
    purpose: Conceptual label for the temporal variation in MIA vulnerability during incremental training.
    Privacy drift is defined as the variation in MIA AUC over time, so it has no independent falsifiable handle beyond the measurement that defines it; it is a descriptive term rather than a new mechanism.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Privacy Drift: Evolving Privacy Concerns in Incremental Learning." pith.science (2026). https://pith.science/paper/OL3VCUL6

@misc{pith2026241205183,
  author       = {Pith},
  title        = {Pith review of: Privacy Drift: Evolving Privacy Concerns in Incremental Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OL3VCUL6}},
  note         = {Machine review of arXiv:2412.05183}
}
read the original abstract

In the evolving landscape of machine learning (ML), Federated Learning (FL) presents a paradigm shift towards decentralized model training while preserving user data privacy. This paper introduces the concept of ``privacy drift", an innovative framework that parallels the well-known phenomenon of concept drift. While concept drift addresses the variability in model accuracy over time due to changes in the data, privacy drift encapsulates the variation in the leakage of private information as models undergo incremental training. By defining and examining privacy drift, this study aims to unveil the nuanced relationship between the evolution of model performance and the integrity of data privacy. Through rigorous experimentation, we investigate the dynamics of privacy drift in FL systems, focusing on how model updates and data distribution shifts influence the susceptibility of models to privacy attacks, such as membership inference attacks (MIA). Our results highlight a complex interplay between model accuracy and privacy safeguards, revealing that enhancements in model performance can lead to increased privacy risks. We provide empirical evidence from experiments on customized datasets derived from CIFAR-100 (Canadian Institute for Advanced Research, 100 classes), showcasing the impact of data and concept drift on privacy. This work lays the groundwork for future research on privacy-aware machine learning, aiming to achieve a delicate balance between model accuracy and data privacy in decentralized environments.

Figures

Figures reproduced from arXiv: 2412.05183 by the authors.

Figure 1
Figure 1. Four-split Non-IID partitioning of CIFAR-20 Dataset. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Design of training and test sets for incremental learning. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Training accuracy, test accuracy, and MIA AUC (area under the curve) for each permutation in the uniform test paradigm, [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Pearson correlation between training accuracy and MIA [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Privacy drift in CIFAR-20 under the uniform test [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Variation in the correlation for the individual exper [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Privacy drift in CIFAR-20 under the additive test [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 10 canonical work pages

  1. [4]

    Membership inference attacks against machine learning models,

    R. Shokri, M. Stronati, C. Song, and V . Shmatikov, “Membership inference attacks against machine learning models,” in 2017 IEEE symposium on security and privacy (SP) . IEEE, 2017, pp. 3–18

  2. [1]

    Large scale incremental learning,

    Y . Wu, Y . Chen, L. Wang, Y . Ye, Z. Liu, Y . Guo, and Y . Fu, “Large scale incremental learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2019, pp. 374–382

  3. [2]

    Incremental learning from noisy data,

    J. C. Schlimmer and R. H. Granger, “Incremental learning from noisy data,” Machine learning, vol. 1, pp. 317–354, 1986

  4. [3]

    Mia-bad: An approach for enhancing membership inference attack and its mit- igation with federated learning,

    S. Banerjee, S. Roy, S. F. Ahamed, D. Quinn, M. Vucovich, D. Nan- dakumar, K. Choi, A. Rahman, E. Bowen, and S. Shetty, “Mia-bad: An approach for enhancing membership inference attack and its mit- igation with federated learning,” in 2024 International Conference on Computing, Networking and Communications (ICNC) . IEEE, 2024, pp. 635–640

  5. [5]

    Deepview: View synthesis with learned gradient descent,

    J. Flynn, M. Broxton, P. Debevec, M. DuVall, G. Fyffe, R. Overbeck, N. Snavely, and R. Tucker, “Deepview: View synthesis with learned gradient descent,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 2367–2376

  6. [6]

    {PrivateEye}: Scalable and {Privacy-Preserving} compro- mise detection in the cloud,

    B. Arzani, S. Ciraci, S. Saroiu, A. Wolman, J. Stokes, G. Outhred, and L. Diwu, “ {PrivateEye}: Scalable and {Privacy-Preserving} compro- mise detection in the cloud,” in 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20) , 2020, pp. 797–815

  7. [7]

    Automatically detecting data drift in machine learning classifiers,

    S. Ackerman, O. Raz, M. Zalmanovici, and A. Zlotnick, “Automatically detecting data drift in machine learning classifiers,” arXiv preprint arXiv:2111.05672, 2021

  8. [8]

    Learning under concept drift: A review,

    J. Lu, A. Liu, F. Dong, F. Gu, J. Gama, and G. Zhang, “Learning under concept drift: A review,” IEEE transactions on knowledge and data engineering, vol. 31, no. 12, pp. 2346–2363, 2018

Show all 22 references
  1. [9]

    Imagenet large scale visual recognition challenge,

    O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al., “Imagenet large scale visual recognition challenge,” International journal of computer vision, vol. 115, pp. 211–252, 2015

  2. [10]

    Natural language processing,

    K. Chowdhary and K. Chowdhary, “Natural language processing,” Fundamentals of artificial intelligence , pp. 603–649, 2020

  3. [11]

    A unifying view on dataset shift in classification,

    J. G. Moreno-Torres, T. Raeder, R. Alaiz-Rodr ´ıguez, N. V . Chawla, and F. Herrera, “A unifying view on dataset shift in classification,” Pattern recognition, vol. 45, no. 1, pp. 521–530, 2012

  4. [12]

    Concept drift and covariate shift detection ensemble with lagged labels,

    Y . Xu and D. Klabjan, “Concept drift and covariate shift detection ensemble with lagged labels,” in 2021 IEEE International Conference on Big Data (Big Data) . IEEE, 2021, pp. 1504–1513

  5. [13]

    Adapting to online label shift with provable guarantees,

    Y . Bai, Y .-J. Zhang, P. Zhao, M. Sugiyama, and Z.-H. Zhou, “Adapting to online label shift with provable guarantees,” Advances in Neural Information Processing Systems , vol. 35, pp. 29 960–29 974, 2022

  6. [14]

    A survey on concept drift adaptation,

    J. Gama, I. ˇZliobait˙e, A. Bifet, M. Pechenizkiy, and A. Bouchachia, “A survey on concept drift adaptation,” ACM computing surveys (CSUR) , vol. 46, no. 4, pp. 1–37, 2014

  7. [15]

    Regional concept drift detection and density synchronized drift adaptation,

    A. Liu, Y . Song, G. Zhang, and J. Lu, “Regional concept drift detection and density synchronized drift adaptation,” in IJCAI International Joint Conference on Artificial Intelligence , 2017

  8. [16]

    A concept drift-tolerant case-base editing technique,

    N. Lu, J. Lu, G. Zhang, and R. L. De Mantaras, “A concept drift-tolerant case-base editing technique,” Artificial Intelligence, vol. 230, pp. 108– 133, 2016

  9. [17]

    Accuracy-privacy trade-off in deep ensemble: A membership inference perspective,

    S. Rezaei, Z. Shafiq, and X. Liu, “Accuracy-privacy trade-off in deep ensemble: A membership inference perspective,” in 2023 IEEE Sympo- sium on Security and Privacy (SP) . IEEE, 2023, pp. 364–381

  10. [18]

    Accuracy-privacy trade-off in the mitigation of membership inference attack in federated learning,

    S. F. Ahamed, S. Banerjee, S. Roy, D. Quinn, M. Vucovich, K. Choi, A. Rahman, A. Hu, E. Bowen, and S. Shetty, “Accuracy-privacy trade-off in the mitigation of membership inference attack in federated learning,” arXiv preprint arXiv:2407.19119 , 2024

  11. [19]

    Nvidia flare: Federated learning from simulation to real-world,

    H. R. Roth, Y . Cheng, Y . Wen, I. Yang, Z. Xu, Y .-T. Hsieh, K. Kersten, A. Harouni, C. Zhao, K. Lu et al. , “Nvidia flare: Federated learning from simulation to real-world,” arXiv preprint arXiv:2210.13291, 2022

  12. [20]

    Communication-efficient learning of deep networks from decentralized data,

    B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial intelligence and statistics . PMLR, 2017, pp. 1273– 1282

  13. [21]

    A stochastic approximation method,

    H. Robbins and S. Monro, “A stochastic approximation method,” The annals of mathematical statistics , pp. 400–407, 1951

  14. [22]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 , 2014

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.