Pith. sign in

REVIEW 5 major objections 5 minor 96 references

MUBox: A Critical Evaluation Framework of Deep Machine Unlearning

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MUBox claims that even state-of-the-art machine unlearning methods, including competition winners, are inconsistent across different forgetting scenarios, and that no single metric can judge them.

desk verdict Broad, useful unlearning benchmark whose central 'inconsistency' claim is real but quantitatively under-backed by unvalidated implementations and non-identical base models. read the letter →

arxiv 2505.08576 v1 pith:VGLKDIKS submitted 2025-05-13 cs.LG cs.CR

Xiang Li , Bhavani Thuraisingham , Wenqi Wei This is my paper · ORCID
classification cs.LGcs.CR
keywords machineunlearningbenchmarkevaluationframeworkmembershipinferenceattackdatapoisoningdepoisoningreproducibilitydeeplearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MUBox argues that the field's confidence in machine unlearning is largely an artifact of easy benchmarks. The platform runs 23 published unlearning methods through six scenarios — partial-class, all-class, class-wise, worst-case, best-case, and depoisoning — with 11 utility, efficacy, and cost metrics, all on the same models and data. Its central finding: even methods published at top venues and winners of unlearning competitions perform inconsistently once the scenario changes, and no single metric tells you whether forgetting actually happened. If this is right, published comparisons have been over-optimistic, and evaluations need harder scenarios and multiple metrics to be meaningful.

What carries the argument

The load-bearing object is MUBox itself: a modular benchmark that fixes one training setup (ResNet-18 on CIFAR-10 and MobileViT on TinyImageNet) and runs 23 published unlearning algorithms through six forgetting scenarios, scoring each with 11 metrics grouped into model utility (test and retaining accuracy), unlearning efficacy (forgetting accuracy, five membership-inference scores, and $\ell^2$ parameter distance), and cost efficiency (runtime relative to retraining and storage overhead). The unifying device is the gold-standard Retrain model: every method is compared against a model retrained without the forgotten data. The scenario axis — especially worst-case (lowest-loss) and best-case (highest-loss) forgetting and depoisoning — is what exposes the inconsistency that the paper reports.

What would settle it

Run, say, SSD, SalUn, and PGU under MUBox's exact protocol with several random seeds and the original authors' released code, then check whether the rankings persist and whether the numbers match the original papers; if rankings flip or the reproduced values differ sharply from the original reports, the inconsistency findings could be implementation or noise artifacts.

Watch

Extended reading notes

Core claim

The paper's claim is that current machine-unlearning evaluations, dominated by class-wise and random forgetting, give a misleading picture. In MUBox's controlled comparison, methods that look strong in the standard class-wise scenario often degrade sharply when only part of a class is forgotten, when the forgotten samples are the easiest or hardest for the model, or when the task is depoisoning. The same method can look successful under one membership-inference metric and fail under another; for example, Bad-T and SalUn score near 100% on confidence- and entropy-based membership inference but near 0% on correctness-based membership inference. Parameter-space distance ($\ell^2$) does not track behavioral difference: SSD and PGU have identical $\ell^2$ distances yet opposite utility outcomes. For depoisoning, effectiveness is attack-dependent — methods that restore accuracy after label-flipping can still leave a backdoor trigger effective. The paper also reports a reproducibility survey of 49 papers from 2019 to 2024: 30.6% provide no code, and half of the code that exists lacks full instructions.

Load-bearing premise

The whole comparison rests on the assumption that MUBox's implementations faithfully reproduce the 23 original methods and that a single run with each original paper's hyperparameters is representative enough to rank them.

Editorial extensions

If this is right

  • Class-wise forgetting, the most common benchmark, is the easiest setting; methods that pass it can still fail when only part of a class is forgotten (Amnesiac, SSD, and L-CODEC lose at least 10 accuracy points) or when the forgotten samples are the model's best or worst examples.
  • A single efficacy metric can flip a method's verdict: Bad-T and SalUn look fully unlearned under confidence- and entropy-based membership inference yet almost un-unlearned under correctness-based membership inference, so multi-metric evaluation is necessary.
  • Depoisoning success is attack-dependent: many methods restore utility after label-flipping, but for backdoor attacks FCS, MSG, CT, and NIU leave a high attack success rate, so unlearning cannot be treated as a universal poison defense.
  • Cost claims are method- and budget-specific: most approximate methods are faster than retraining, but L-CODEC can be about eight times slower and Amnesiac needs roughly 428 GB of stored updates, so efficiency must be reported together with storage and unlearning budget.
  • Methods that work on CIFAR-10 often fail on TinyImageNet: SSD collapses, SISA's test accuracy falls to about 18%, and gradient-based methods degrade on well-trained models, showing poor transfer across datasets and architectures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the inconsistency finding holds, benchmark design should add worst-case and best-case forgetting as default settings, because leaderboards built only on class-wise forgetting may reward methods that are tuned to that one scenario.
  • The paper's single-run design is a lower bound on reliability; adding multiple seeds and testing hyperparameter sensitivity would separate genuine scenario-dependence from tuning fragility, which the current data cannot fully distinguish.
  • The depoisoning results suggest unlearning can serve as a poisoning defense only when the attack family is known; an attack-agnostic unlearning defense would need to be evaluated against mixed or adaptive poisoning, which MUBox does not yet cover.
  • The reproducibility statistics imply that a code-plus-README norm is necessary but not sufficient; a shared harness with fixed scenarios and metrics may matter more than individual implementations for making comparisons meaningful.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper presents MUBox, a benchmark and evaluation platform for deep machine unlearning. It integrates 23 unlearning methods, six unlearning scenarios, and eleven evaluation metrics, and it reports experiments on CIFAR-10 (main text) and TinyImageNet (appendix). The main empirical claims are that state-of-the-art unlearning methods, including competition winners, show inconsistent effectiveness across scenarios; that no single metric adequately captures unlearning quality; and that depoisoning performance varies strongly with the attack type. The paper also contains a reproducibility survey of 49 unlearning papers.

Significance. If the empirical findings are robust, MUBox would be a useful community resource: it offers broad method coverage, a public codebase, a systematic scenario taxonomy, and a concrete reproducibility audit of prior work. The headline finding that methods with strong papers or competition results fail outside the simplest settings is important and would justify more diverse evaluation protocols. The reproducibility statistics and the explicit multi-metric comparisons are also valuable. However, the current evidence does not yet support the strength of the claims, because several load-bearing aspects of the evaluation protocol are not demonstrated: the FA metric is reported on inconsistent scales, the base models are not identical across methods, implementations are not validated against original results, and all numbers come from single runs.

major comments (5)
  1. [§3.4 and Tables 1, 3-6] The FA metric is not defined or reported consistently. The text defines FA as the accuracy change of the unlearned model on D_f compared with the Retrain model, but in every table the Original and Retrain rows report raw forgetting accuracy (e.g., Table 1 shows Retrain FA = 93.40 and Original FA = 100.00), while all other rows report signed differences (e.g., PGU FA = -1.60, SSD FA = -93.40). This makes the cross-row comparisons in the text, such as 'only a 1.6% decrease' and the ranking arguments in Section 4, formally invalid unless the reader manually converts to a common scale. Please report a single consistent FA definition, or report both raw forgetting accuracy and the deviation from Retrain in separate columns.
  2. [§3.2 and Appendix 'Additional Implementation Details'] Not all methods are evaluated on the same base model, despite the abstract and Section 3.2 claiming a unified comparison. The appendix states that Fisher and NTK use a ResNet-18 reduced to 40% of the original size and pre-trained on CIFAR-100, that L-CODEC also uses the 40%-scale ResNet-18, and that Amnesiac is trained for only 50 epochs. Different architectures, training durations, and pretraining distributions change loss geometry and forgetting behavior, so the rows for Fisher, NTK, L-CODEC, and Amnesiac are not directly comparable with the rows for the other methods. The paper should either run all methods on the same base model, or present these rows in a clearly separated section and remove them from the cross-method inconsistency claims.
  3. [§3.2, §4, and Appendix] The implementations of the 23 methods are not validated against the original papers' reported results. Section 4 says 'we use the same unlearning hyperparameter settings as specified in each original paper,' but no table or figure compares any MUBox result with the accuracy, forgetting, or runtime numbers in the original publications. Four competition methods (FCS, MSG, CT, NIU) are taken directly from Kaggle discussion threads rather than from official codebases. Because the central conclusion is that methods are inconsistent, a single buggy or misconfigured implementation could change a row and alter the finding. Please include a validation table showing, for each method, the original reported metric and the corresponding MUBox reproduction, or explicitly state which implementations could not be validated.
  4. [§4, Tables 1-6] All reported results are single-run numbers with no error bars, confidence intervals, or random-seed information. The main conclusions are comparative rankings across methods, and several distinctions in the text are small, such as FA values of +6.60 versus +4.10 in Table 1. Without multiple seeds, these differences could be noise, and the 'inconsistent effectiveness' finding would be stronger if shown to be stable under training and unlearning stochasticity. Please report at least three to five runs with means and standard deviations for the primary metrics, or clearly label the results as preliminary single-run observations.
  5. [§4.1 and §4.3] The MIA metric set appears to be selected post hoc. In Section 4.1, m_entropy and probability-based MIA are declared 'unreliable' and 'should not be used to evaluate unlearning performance in this set of experiments,' and in Section 4.3 m_entropy is dropped because of its 'failure.' Yet the columns remain in the tables and are later used in the discussion, including the statement that only SSD and SISA are effective 'across all five metrics.' This selective treatment weakens the abstract claim that no single metric can capture unlearning performance, because the conclusion depends on which metrics are retained. Please justify the inclusion or exclusion of metrics before presenting results, or report sensitivity analyses over all five MIA variants consistently.
minor comments (5)
  1. [§4.1] The typo 'Secone-order' should read 'Second-order' in the model utility paragraph.
  2. [Table 3 and Table 9] There are formatting errors in the numeric entries: Table 3 shows '06.30' and '03.60' in the FCS row, and Table 9 shows '0.78.78' in the NIU row. These should be corrected to standard numeric formatting.
  3. [§4.3] The sentence containing 'while maintaining competitive while maintaining competitive TA and RA' contains a duplicated phrase and should be rewritten.
  4. [Abstract and Appendix] The abstract states that TinyImageNet results 'will be provided as supplementary material upon acceptance of the paper,' but the appendix already contains Tables 8 through 12 with these results. Please update the wording to reflect the actual structure.
  5. [Appendix, Table 7] The table entry for reference [27] is labeled 'Amnesaic' instead of 'Amnesiac,' and the appendix contains a stray fragment ', i.e., phys_theta_u = A(D_r),' in the 'Key Ideas of Implemented Unlearning Methods' section. Both should be removed or corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MUBox is an empirical benchmark whose conclusions are read from its own tables, not derived from its inputs.

full rationale

MUBox makes no first-principles derivation; its findings are observational summaries of a benchmark it constructs. The central claim that unlearning methods are inconsistent across scenarios is an aggregate reading of Tables 1-6 and Figures 3-7, not an equation derived from the methods' definitions. The framework's metric formulas, such as MIA-Efficacy = TN/|D_f|, are evaluation definitions, and no fitted parameter is later renamed as a prediction. The implementation choices, such as using original-paper hyperparameters or reduced ResNet-18 for Fisher/NTK/L-CODEC, are fidelity and comparability limitations that could threaten validity, but they are not circular: the conclusions are not forced by the inputs by construction. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via prior work by these authors. The paper is self-contained against external benchmarks in the sense that its numbers come from running the integrated methods, and any dispute about implementation correctness is an empirical reproducibility concern, not a circularity finding. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical entities. Its axioms are the standard definitions of unlearning, the assumption that implementations are faithful, the use of retraining as gold standard, the validity of MIA as a proxy, and generalizability from two datasets.

assumptions (4)
  • domain assumption The 23 unlearning method implementations in MUBox are faithful reproductions of the original methods.
    The paper uses hyperparameters from original papers (Section 4) but does not verify that outputs match the original papers' reported numbers.
  • domain assumption Retraining on the retaining set is the gold standard for unlearning.
    Used throughout as the reference model; this is standard in the unlearning literature, but it may not be the only valid gold standard.
  • domain assumption Membership inference attacks provide a valid proxy for unlearning efficacy.
    Used in MIA-Efficacy metric; the paper itself shows that different MIA metrics disagree, which undermines this assumption.
  • domain assumption Results on CIFAR-10 and TinyImageNet generalize to other datasets and models.
    Only two datasets and two architectures are evaluated, so generalization is assumed rather than demonstrated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MUBox: A Critical Evaluation Framework of Deep Machine Unlearning." pith.science (2026). https://pith.science/paper/VGLKDIKS

@misc{pith2026250508576,
  author       = {Pith},
  title        = {Pith review of: MUBox: A Critical Evaluation Framework of Deep Machine Unlearning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VGLKDIKS}},
  note         = {Machine review of arXiv:2505.08576}
}
read the original abstract

Recent legal frameworks have mandated the right to be forgotten, obligating the removal of specific data upon user requests. Machine Unlearning has emerged as a promising solution by selectively removing learned information from machine learning models. This paper presents MUBox, a comprehensive platform designed to evaluate unlearning methods in deep learning. MUBox integrates 23 advanced unlearning techniques, tested across six practical scenarios with 11 diverse evaluation metrics. It allows researchers and practitioners to (1) assess and compare the effectiveness of different machine unlearning methods across various scenarios; (2) examine the impact of current evaluation metrics on unlearning performance; and (3) conduct detailed comparative studies on machine unlearning in a unified framework. Leveraging MUBox, we systematically evaluate these unlearning methods in deep learning and uncover several key insights: (a) Even state-of-the-art unlearning methods, including those published in top-tier venues and winners of unlearning competitions, demonstrate inconsistent effectiveness across diverse scenarios. Prior research has predominantly focused on simplified settings, such as random forgetting and class-wise unlearning, highlighting the need for broader evaluations across more difficult unlearning tasks. (b) Assessing unlearning performance remains a non-trivial problem, as no single evaluation metric can comprehensively capture the effectiveness, efficiency, and preservation of model utility. Our findings emphasize the necessity of employing multiple metrics to achieve a balanced and holistic assessment of unlearning methods. (c) In the context of depoisoning, our evaluation reveals significant variability in the effectiveness of existing approaches, which is highly dependent on the specific type of poisoning attacks.

Figures

Figures reproduced from arXiv: 2505.08576 by the authors.

Figure 1
Figure 1. Reproducibility analysis of 49 collected papers. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Unlearning methods across different unlearning [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 5
Figure 5. TA and FA of different unlearning methods for [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figures from the paper (1 more)
Figure 6
Figure 6. Figure 6: TA and FA of different unlearning methods for [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

96 extracted references · 66 canonical work pages

  1. [1]

    Thomas Baumhauer, Pascal Schöttle, and Matthias Zeppelzauer. 2022. Machine unlearning: Linear filtration for logit-based classifiers. Machine Learning (2022), 3203–3226

    2022

  2. [2]

    Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hen- grui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2021. Machine unlearning. In IEEE Symposium on Security and Privacy

    2021

  3. [3]

    Yinzhi Cao and Junfeng Yang. 2015. Towards making systems forget with machine unlearning. In IEEE symposium on security and privacy

    2015

  4. [4]

    Sungmin Cha, Sungjun Cho, Dasol Hwang, Honglak Lee, Taesup Moon, and Moontae Lee. 2024. Learning to unlearn: Instance-wise unlearning for pre-trained classifiers. In AAAI Conference on Artificial Intelligence

    2024

  5. [5]

    Jihong Chen and Jiabin Sun. 2021. Understanding the chinese data security law. International Cybersecurity Law Review (2021), 209–221

    2021

  6. [6]

    Min Chen, Weizhuo Gao, Gaoyang Liu, Kai Peng, and Chen Wang. 2023. Boundary unlearning: Rapid forgetting of deep networks via shifting the decision boundary. In IEEE/CVF Conference on Computer Vision and Pattern Recognition . 7766–7775

    2023

  7. [7]

    Ruizhe Chen, Jianfei Yang, Huimin Xiong, Jianhong Bai, Tianxiang Hu, Jin Hao, Yang Feng, Joey Tianyi Zhou, Jian Wu, and Zuozhu Liu. 2024. Fast model debias with machine unlearning. In Conference on Neural Information Processing Systems

    2024

  8. [8]

    Eli Chien, Haoyu Wang, Ziang Chen, and Pan Li. 2024. Certified Machine Un- learning via Noisy Stochastic Gradient Descent. arXiv preprint arXiv:2403.17105 (2024)

    arXiv 2024

Show all 96 references
  1. [9]

    Eli Chien, Haoyu Wang, Ziang Chen, and Pan Li. 2024. Langevin unlearning: A new perspective of noisy gradient descent for machine unlearning. arXiv preprint arXiv:2401.10371 (2024)

    2024

  2. [10]

    Rishav Chourasia and Neil Shah. 2023. Forget unlearning: Towards true data- deletion in machine learning. In International Conference on Machine Learning

    2023

  3. [12]

    Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli

  4. [13]

    NeurIPS Machine Unlearning Competition. 2023. https://www.kaggle.com/ competitions/neurips-2023-machine-unlearning/discussion/458721 Accessed December 5, 2024

    2023

  5. [14]

    IEEE Transactions on Information Forensics and Security (2023)

    Zero-shot machine unlearning. IEEE Transactions on Information Forensics and Security (2023)

    2023

  6. [15]

    NeurIPS Machine Unlearning Competition. 2023. https://www.kaggle.com/ competitions/neurips-2023-machine-unlearning/discussion/458531 Accessed December 5, 2024

    2023

  7. [16]

    NeurIPS Machine Unlearning Competition. 2023. https://www.kaggle.com/ competitions/neurips-2023-machine-unlearning/discussion/459200 Accessed December 5, 2024

    2023

  8. [17]

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Ima- genet: A large-scale hierarchical image database. In IEEE conference on computer vision and pattern recognition

    2009

  9. [18]

    NeurIPS Machine Unlearning Competition. 2023. https://www.kaggle.com/ competitions/neurips-2023-machine-unlearning/discussion/459095 Accessed December 5, 2024

    2023

  10. [19]

    Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong, Dennis Wei, and Sijia Liu. 2024. SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation. InInternational Conference on Learning Representations

    2024

  11. [20]

    Chongyu Fan, Jiancheng Liu, Alfred Hero, and Sijia Liu. 2024. Challenging forgets: Unveiling the worst-case forget sets in machine unlearning. arXiv preprint arXiv:2403.07362 (2024)

    2024 arXiv

  12. [21]

    Jack Foster, Stefan Schoepf, and Alexandra Brintrup. 2024. Fast machine un- learning without retraining through selective synaptic dampening. In AAAI Conference on Artificial Intelligence. 12043–12051

    2024

  13. [22]

    Jack Foster, Kyle Fogarty, Stefan Schoepf, Cengiz Öztireli, and Alexandra Brintrup

  14. [23]

    Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. 2019. Mak- ing ai forget you: Data deletion in machine learning. In Conference on Neural Information Processing Systems

    2019

  15. [24]

    Aditya Golatkar, Alessandro Achille, Avinash Ravichandran, Marzia Polito, and Stefano Soatto. 2021. Mixed-privacy forgetting in deep networks. In IEEE/CVF conference on computer vision and pattern recognition

    2021

  16. [25]

    Badih Ghazi, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Ayush Sekhari, and Chiyuan Zhang. 2023. Ticketed learning–unlearning schemes. In Conference on Learning Theory. 5110–5139

    2023

  17. [26]

    Aditya Golatkar, Alessandro Achille, and Stefano Soatto. 2020. Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations. In European Conference on Computer Vision

    2020

  18. [27]

    Laura Graves, Vineel Nagisetty, and Vijay Ganesh. 2021. Amnesiac machine learning. In AAAI Conference on Artificial Intelligence

    2021

  19. [28]

    Aditya Golatkar, Alessandro Achille, and Stefano Soatto. 2020. Eternal sunshine of the spotless net: Selective forgetting in deep networks. InIEEE/CVF Conference on Computer Vision and Pattern Recognition

    2020

  20. [29]

    Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. 2019. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access 7 (2019), 47230–47244

    2019

  21. [30]

    Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten

  22. [31]

    Keltin Grimes, Collin Abidi, Cole Frank, and Shannon Gallagher. 2024. Gone but Not Forgotten: Improved Benchmarks for Machine Unlearning. arXiv preprint arXiv:2405.19211 (2024)

    2024 arXiv

  23. [32]

    Varun Gupta, Christopher Jung, Seth Neel, Aaron Roth, Saeed Sharifi-Malvajerdi, and Chris Waites. 2021. Adaptive machine unlearning. In Conference on Neural Information Processing Systems

    2021

  24. [33]

    Elizabeth Liz Harding, Jarno J Vanto, Reece Clark, L Hannah Ji, and Sara C Ainsworth. 2019. Understanding the scope and impact of the california consumer privacy act of 2018. Journal of Data Protection & Privacy (2019), 234–253

    2019

  25. [34]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In IEEE conference on computer vision and pattern recognition. 770–778

    2016

  26. [35]

    Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. 2020. Certified data removal from machine learning models. InInternational Conference on Machine Learning

    2020

  27. [36]

    Tuan Hoang, Santu Rana, Sunil Gupta, and Svetha Venkatesh. 2024. Learn to Unlearn for Deep Neural Networks: Minimizing Unlearning Interference with Gradient Projection. In IEEE/CVF Winter Conference on Applications of Computer Vision

    2024

  28. [37]

    Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. 2021. Approximate data deletion from machine learning models. In International Con- ference on Artificial Intelligence and Statistics

    2021

  29. [38]

    Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. 2023. Model Sparsity Can Simplify Machine Unlearning. In Conference on Neural Information Processing Systems

    2023

  30. [39]

    Alvin Heng and Harold Soh. 2024. Selective amnesia: A continual learning approach to forgetting in deep generative models. In Conference on Neural Infor- mation Processing Systems

    2024

  31. [40]

    Alex Krizhevsky and Geoff Hinton. 2010. Convolutional deep belief networks on cifar-10. Technical report (2010), 1–9

    2010

  32. [41]

    Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou

  33. [42]

    Rachel Layne. 2024. How to Make AI ’Forget’ All the Private Data It Shouldn’t Have. Harvard Business School Working Knowledge(2024). https://hbswk.hbs.edu/ item/qa-seth-neel-on-machine-unlearning-and-the-right-to-be-forgotten

    2024

  34. [43]

    Junyaup Kim and Simon S Woo. 2022. Efficient two-stage model retraining for machine unlearning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4361–4369

    2022

  35. [44]

    Klas Leino and Matt Fredrikson. 2020. Stolen memories: Leveraging model memorization for calibrated{White-Box} membership inference. In USENIX Security Symposium. 1605–1622

    2020

  36. [45]

    Guihong Li, Hsiang Hsu, Radu Marculescu, et al . 2024. Machine Unlearning for Image-to-Image Generative Models. In International Conference on Learning Representations

    2024

  37. [46]

    In Conference on Neural Infor- mation Processing Systems

    Towards Unbounded Machine Unlearning. In Conference on Neural Infor- mation Processing Systems

  38. [47]

    Shen Lin, Xiaoyu Zhang, Chenyang Chen, Xiaofeng Chen, and Willy Susilo

  39. [48]

    Sangyong Lee and Simon S Woo. 2023. Undo: Effective and accurate unlearning method for deep neural networks. InACM International Conference on Information and Knowledge Management

    2023

  40. [49]

    Alessandro Mantelero. 2013. The EU Proposal for a General Data Protection Regulation and the roots of the ‘right to be forgotten’. Computer Law & Security Review 29, 3 (2013), 229–235

    2013

  41. [50]

    Ronak Mehta, Sourav Pal, Vikas Singh, and Sathya N. Ravi. 2022. Deep Unlearning via Randomized Conditionally Independent Hessians. In IEEE/CVF Conference on Computer Vision and Pattern Recognition . 10422–10431. Xiang Li, Bhavani Thuraisingham, and Wenqi Wei

    2022

  42. [51]

    Na Li, Chunyi Zhou, Yansong Gao, Hui Chen, Anmin Fu, Zhi Zhang, and Yu Shui. 2024. Machine Unlearning: Taxonomy, Metrics, Applications, Challenges, and Prospects. arXiv preprint arXiv:2403.08254 (2024)

    2024 arXiv

  43. [52]

    Seth Neel, Aaron Roth, and Saeed Sharifi-Malvajerdi. 2021. Descent-to-delete: Gradient-based methods for machine unlearning. In Algorithmic Learning The- ory

    2021

  44. [53]

    In IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Erm-ktp: Knowledge-level machine unlearning via knowledge transfer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition . 20147–20155

  45. [54]

    Jiaqi Liu, Jian Lou, Zhan Qin, and Kui Ren. 2024. Certified minimax unlearn- ing with generalization rates and deletion capacity. In Conference on Neural Information Processing Systems

    2024

  46. [55]

    2020.{Updates-Leak}: Data set inference and reconstruction attacks in online learning

    Ahmed Salem, Apratim Bhattacharya, Michael Backes, Mario Fritz, and Yang Zhang. 2020.{Updates-Leak}: Data set inference and reconstruction attacks in online learning. In USENIX security symposium. 1291–1308

    2020

  47. [56]

    Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh

  48. [57]

    Sachin Mehta and Mohammad Rastegari. 2022. MobileViT: Light-weight, General- purpose, and Mobile-friendly Vision Transformer. In International Conference on Learning Representations

    2022

  49. [58]

    Shaofei Shen, Chenhao Zhang, Yawen Zhao, Alina Bialkowski, Weitong Chen, and Miao Xu. 2024. Label-agnostic forgetting: A supervision-free unlearning in deep models. arXiv preprint arXiv:2404.00506 (2024)

    2024 arXiv

  50. [59]

    Office of the Privacy Commissioner of Canada. 2018. Announcement: Pri- vacy Commissioner Seeks Federal Court Determination on Key Issue for Cana- dians’ Online Reputation. https://www.priv.gc.ca/en/opc-news/news-and- announcements/2018/an_181010/

    2018

  51. [60]

    Get in Researchers; We’re Measuring Reproducibility

    Daniel Olszewski, Allison Lu, Carson Stillman, Kevin Warren, Cole Kitroser, Alejandro Pascual, Divyajyoti Ukirde, Kevin Butler, and Patrick Traynor. 2023. " Get in Researchers; We’re Measuring Reproducibility": A Reproducibility Study of Machine Learning Papers in Tier 1 Secur...

    2023

  52. [61]

    Liwei Song and Prateek Mittal. 2021. Systematic evaluation of privacy risks of machine learning models. In USENIX Security Symposium

    2021

  53. [62]

    Liwei Song, Reza Shokri, and Prateek Mittal. 2019. Privacy risks of securing ma- chine learning models against adversarial examples. In ACM SIGSAC Conference on Computer and Communications Security

    2019

  54. [63]

    Vinith Suriyakumar and Ashia C Wilson. 2022. Algorithms that approximate data removal: New results and limitations. In Conference on Neural Information Processing Systems

    2022

  55. [64]

    Supreeth Shastri, Melissa Wasserman, and Vijay Chidambaram. 2019. The seven sins of Personal-Data processing systems under GDPR. In USENIX Workshop on Hot Topics in Cloud Computing

    2019

  56. [65]

    Ayush K Tarun, Vikram S Chundawat, Murari Mandal, and Mohan Kankanhalli

  57. [66]

    Takashi Shibata, Go Irie, Daiki Ikami, and Yu Mitsuzumi. 2021. Learning with Selective Forgetting.. In International Joint Conference on Artificial Intelligence

    2021

  58. [67]

    Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Mem- bership inference attacks against machine learning models. In IEEE symposium on security and privacy . 3–18

    2017

  59. [68]

    Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck

  60. [69]

    Ga Wu, Masoud Hashemi, and Christopher Srinivasa. 2022. Puma: Performance unchanged model augmentation for training data removal. In AAAI Conference on Artificial Intelligence. 8675–8682

    2022

  61. [70]

    Yinjun Wu, Edgar Dobriban, and Susan Davidson. 2020. Deltagrad: Rapid re- training of machine learning models. In International Conference on Machine Learning

    2020

  62. [71]

    Ryutaro Tanno, Melanie F Pradier, Aditya Nori, and Yingzhen Li. 2022. Repair- ing neural networks by leaving the right past behind. In Conference on Neural Information Processing Systems

    2022

  63. [72]

    Haonan Yan, Xiaoguang Li, Ziyao Guo, Hui Li, Fenghua Li, and Xiaodong Lin

  64. [73]

    Fast yet effective machine unlearning.IEEE Transactions on Neural Networks and Learning Systems (2023)

    2023

  65. [74]

    Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. 2022. Unrolling sgd: Understanding factors influencing machine unlearning. In IEEE European Symposium on Security and Privacy

    2022

  66. [75]

    Rossi, and Raman Arora

    Enayat Ullah, Tung Mai, Anup Rao, Ryan A. Rossi, and Raman Arora. 2021. Machine Unlearning via Algorithmic Stability. InConference on Learning Theory

    2021

  67. [76]

    Shuijing Zhang, Jian Lou, Li Xiong, Xiaoyu Zhang, and Jing Liu. 2023. Closed- form machine unlearning for matrix factorization. In ACM International Confer- ence on Information and Knowledge Management . Table 7: Statistical analysis. This table includes 51 different methods f...

    2023

  68. [77]

    In Network and Distributed Systems Security Symposium

    Machine unlearning of features and labels. In Network and Distributed Systems Security Symposium

  69. [80]

    Jie Xu, Zihan Wu, Cong Wang, and Xiaohua Jia. 2024. Machine unlearning: Solutions and challenges. IEEE Transactions on Emerging Topics in Computational Intelligence (2024)

    2024

  70. [83]

    Jingwen Ye, Yifang Fu, Jie Song, Xingyi Yang, Songhua Liu, Xin Jin, Mingli Song, and Xinchao Wang. 2022. Learning with recoverable forgetting. In European Conference on Computer Vision

    2022

  71. [84]

    Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. 2018. Pri- vacy risk in machine learning: Analyzing the connection to overfitting. In IEEE computer security foundations symposium

    2018

  72. [85]

    Peng-Fei Zhang, Guangdong Bai, Zi Huang, and Xin-Shun Xu. 2022. Machine unlearning for image retrieval: A generative scrubbing approach. In ACM Inter- national Conference on Multimedia

    2022

  73. [87]

    ✓ ✓ NeurIPS 2021 [56]* ✗ - NeurIPS 2021

    2021

  74. [88]

    ✗ - NeurIPS 2023 SCRUB[41] ✓ ✓ NeurIPS 2023 [48]* ✗ - NeurIPS 2023 ℓ1-Sparsity[38] ✓ ✓ NeurIPS 2023 FCS[13] ✓ ✓ — NeurIPS 2023 MSG[14] ✓ ✓ — NeurIPS 2023 CT[15] ✓ ✓ — NeurIPS 2023 NIU[16] ✓ ✓ — NeurIPS 2023 Langevin[9] ✓ ✓ NeurIPS 2024 PNSGD[8] ✓ ✓ NeurIPS 2024 SalUn[19] ✓ ✓ ICLR 2024

    2023

  75. [89]

    ✓ ✓ ICML 2020 [10]* ✗ - ICML 2023 Amnesaic[27] ✓ ✓ — AAAI 2021 Bad-T[11] ✓ ✓ — AAAI 2023

    2020

  76. [90]

    ✗ - AAAI 2022 SSD[21] ✓ ✓ AAAI 2024

    2022

  77. [91]

    ✗ - IJCAI 2022 Fisher[25] ✓ ✗ CVPR 2020

    2022

  78. [92]

    ✗ - CVPR 2022 L-CODEC[50] ✓ ✓ CVPR 2022 Boundary-S[6] ✓ ✓ — CVPR 2023 Boundary-E[6] ✓ ✓ — CVPR 2023

    2022

  79. [93]

    ✓ ✓ ECCV 2022 NTK[26] ✓ ✗ ECCV 2020

    2022

  80. [94]

    ✗ - CIKM 2023 [67]* ✗ - COLT 2021 [22]* ✗ - COLT 2023 [52]* ✗ - ALT 2021

    2023

  81. [95]

    ✗ - Machine Learning 2022

    2022

  82. [96]

    ✗ - ACM MM 2022 UNSIR[65] ✓ ✓ — TNNLS 2023 PGU[36] ✓ ✓ — WACV 2024

    2022

  83. [97]

    unseen" data with respect to𝜃𝑢. Formally, MIA-Efficacy = 𝑇𝑁 |D 𝑓|, (3) where𝑇𝑁 denotes the true negatives, i.e., the number of forget- ting data predicted as “unseen

    Zijie Zhang, Yang Zhou, Xin Zhao, Tianshi Che, and Lingjuan Lyu. 2022. Prompt certified machine unlearning with randomized gradient smoothing and quanti- zation. In Conference on Neural Information Processing Systems . Appendix Reproducibility Study Summary Table 7 presents 49...

    2022

  84. [2019]

    arXiv preprint arXiv:1911.03030 (2019)

    Certified data removal from machine learning models. arXiv preprint arXiv:1911.03030 (2019)

    2019 arXiv

  85. [2021]

    In Conference on Neural Information Processing Systems

    Remember what you want to forget: Algorithms for machine unlearning. In Conference on Neural Information Processing Systems

  86. [2022]

    In Interna- tional Joint Conference on Artificial Intelligence

    Arcane: An efficient architecture for exact machine unlearning. In Interna- tional Joint Conference on Artificial Intelligence

  87. [2023]

    In AAAI Conference on Artificial Intelligence

    Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. In AAAI Conference on Artificial Intelligence . 7210–7217

  88. [2024]

    arXiv preprint arXiv:2402.01401 (2024)

    Zero-shot machine unlearning at scale via lipschitz regularization. arXiv preprint arXiv:2402.01401 (2024)

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.