Pith. sign in

REVIEW 4 major objections 4 minor 3 cited by

Unlearning difficulty can be read from a model's internal circuits before any erasure, with hard-to-forget samples relying on deep, output-proximal pathways.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 10:31 UTC pith:L754SDXD

load-bearing objection Useful first cut at a circuit-based difficulty metric for unlearning, but the central claim of being method-independent is not yet supported — worth review, not acceptance as-is. the 4 major comments →

arxiv 2601.09624 v2 pith:L754SDXD submitted 2026-01-14 cs.LG cs.AIcs.CLcs.CV

A Mechanistic Perspective and Circuit-Guided Difficulty Metric for Unlearning

Jiali Cheng , Ziheng Chen , Chirag Agarwal , Hadi Amiri This is my paper
classification cs.LG cs.AIcs.CLcs.CV
keywords machine unlearningmechanistic interpretabilitycircuit analysisunlearning difficultypre-unlearning predictionattribution patchingLLMTOFU
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Machine unlearning erases some samples easily and fails on others. This paper argues that the difference is encoded in the model's internal circuits, not just in the data. It introduces CUD, a score computed before any unlearning run that measures how a sample's circuit sits between easy-to-unlearn and hard-to-unlearn reference circuits. Replacing the default TOFU forget set with CUD-selected hard sets lowers unlearning efficacy by 14.1 points on average across five methods, while easy sets raise it by 3.3 points. The paper also finds a mechanistic signature: easy samples are supported by shallow, early-to-mid MLP edges, while hard samples rely on deeper, late-stage and attention-involving pathways.

Core claim

CUD assigns each forget sample a continuous score in [0,1] before unlearning, defined as 1−s_E/((1−s_E)+(1−s_H)), where s_E and s_H measure similarity of the sample's extracted circuit to easy and hard anchor circuits. The anchors are selected by bi-level optimization that finds samples with high versus low post-unlearning loss, and the circuits themselves are extracted from the original model using integrated-gradient attribution patching. The central claim is that this pre-unlearning score measures intrinsic, method-independent difficulty: across five unlearning methods, CUD-selected hard forget sets drop unlearning efficacy by 14.1 points on average and also degrade retain and general-kno

What carries the argument

The core object is the CUD score, a normalized ratio 1−s_E/((1−s_E)+(1−s_H)) that places a sample's circuit between two reference circuits. The reference easy and hard circuits are obtained in two steps: bi-level optimization (Eqs. 5–6) selects forget samples with high vs. low post-unlearning loss, and circuit extraction via integrated-gradient attribution patching (EAP-IG) locates the causal edges supporting those samples in the original model. The ratio form makes CUD a relative geometric measure in circuit space, independent of absolute edge counts; the paper shows it is robust to the similarity metric (cosine, Jaccard, Hamming) and to the choice of unlearning loss used to build the ancho

Load-bearing premise

The load-bearing premise is that the easy and hard reference circuits, selected from one forget set using a GradDiff unlearning loss, represent an intrinsic difficulty axis that generalizes to all other samples and all other unlearning methods; if those anchors are specific to that calibration set or method, the observed easy/hard separation is an in-sample artifact.

What would settle it

Rebuild the CUD anchors with a different unlearning loss (e.g., NPO) on the same forget set and re-run the easy/hard split; if the efficacy gap shrinks to zero or reverses, CUD is not method-independent. Also, apply CUD to a model of different size or architecture (e.g., a 7B model or a non-transformer) and check whether samples ranked hard under one model remain hard under the other; if rankings flip substantially, the metric is not a stable intrinsic property.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Before any unlearning run, CUD can identify which samples will be hardest to erase, enabling construction of controlled easy/hard forget sets for benchmarking and stress-testing.
  • Because the separation holds across five unlearning methods and three similarity metrics, CUD provides a method-agnostic difficulty signal rather than an artifact of a single algorithm.
  • Hard-to-unlearn samples also cause collateral damage: selecting them degrades retain performance and general knowledge, implying difficult forgets are entangled with broader model knowledge.
  • The continuous score supports difficulty-aware unlearning strategies such as curriculum scheduling, adaptive sampling, and loss reweighting, which the paper identifies as future applications.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the shallow-vs-deep circuit signature generalizes beyond the tested models and datasets, CUD could be used to localize which layers or circuit communities encode resistant knowledge, suggesting targeted interventions (e.g., late-layer perturbation) rather than global updates.
  • The anchors are built with a single unlearning loss (GradDiff); whether CUD remains method-independent when anchors are built with a different loss is a testable extension, and the reported 0.76 correlation with UNDIAL-based CUD is only a partial check.
  • The weak negative correlation with MRD (-0.27) suggests unlearning difficulty is not one-dimensional; combining CUD's structural signal with a sensitivity-based signal might predict unlearning outcomes better than either alone.
  • Because the hard-set efficacy drop (-14.1) is accompanied by retain degradation, the true cost of erasing hard samples in a real deployment may be even larger than the efficacy metric alone suggests.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes CUD, a circuit-based metric for per-sample unlearning difficulty. Each sample's circuit is extracted using EAP/EAP-IG and compared against two reference circuits, C_E and C_H, which are built from easy/hard samples identified by bi-level optimization over post-unlearning loss. CUD is defined as 1−s_E / ((1−s_E)+(1−s_H)). On TOFU and LLMRec, CUD-selected hard forget sets reduce unlearning efficacy by about 14 points and easy sets improve it by about 3 points, with reported statistical significance across several unlearning methods. The paper also reports mechanistic differences: easy samples are said to rely on shallow early MLP edges, while hard samples involve deeper, output-proximal and attention-related pathways.

Significance. If the central claims hold, CUD would be a practical, interpretable pre-unlearning diagnostic and a step toward mechanistic explanations of unlearning difficulty. The consistency across unlearning methods, robustness to similarity metrics, and the reported correlation of 0.76 when the anchor loss is changed are strengths. However, the core claim that CUD measures an intrinsic, method-independent difficulty before unlearning is undercut by the in-sample calibration between anchor construction and evaluation. The manuscript needs a held-out validation and a clarification of what 'pre-unlearning' means before the significance can be fully assessed.

major comments (4)
  1. [Sec. 3.1 (Eqs. 5–6) and Sec. 4.1 (Table 1)] The anchor circuits C_E and C_H are obtained by solving bi-level optimizations over post-unlearning loss on D_f using GradDiff (Sec. 4). CUD (Eq. 8) then ranks samples from the same D_f, and Table 1 constructs hard/easy forget sets from that same D_f. The reported −14.1/+3.3 separation is therefore at least partly in-sample calibration: the extreme-CUD samples are those Eq. 5–6 already identified as easy/hard under GradDiff. Cross-method columns mitigate this, but the selected samples are still from the calibration set. Please add a held-out split: fit anchors on one subset, evaluate CUD-selected sets on the complement, and report both; this is necessary to support the claim of an intrinsic, pre-unlearning metric.
  2. [Sec. 3.1 and Abstract] The claim that CUD is computed 'prior to any unlearning intervention' (Abstract, Sec. 5.2) is overstated. Computing C_E and C_H requires training θ_u(w) in Eqs. 5–6, i.e., an unlearning run (GradDiff) on the calibration set, and the selection of D_f,E/D_f,H is based on post-unlearning loss. Thus CUD is not fully pre-unlearning; at most it is a pre-unlearning score for query samples once calibration runs are available. The manuscript should acknowledge this, or demonstrate that anchors transfer across independent base models and forget splits.
  3. [Eqs. 5–6] As written with λ>0, the term +λ||w||2^2 in Eq. 5 encourages selecting more samples when maximizing, and in Eq. 6 it penalizes small sets when minimizing. This contradicts the text in Sec. 3.1 that 'λ is a hyperparameter that encourages selecting a small set of samples' and Sec. 4.1's 'sparsity-inducing regularizer λ||w||'. The sign or definition of λ should be corrected, and the actual value/range of λ used in experiments reported. If the intent is λ<0, this must be stated explicitly.
  4. [Sec. 3.1 and Table 3] CUD's construction depends on several unspecified thresholds and parameters: the edge binarization/top-k threshold after EAP/EAP-IG, the number of interpolation steps in Eq. 4, the similarity metric, and λ in Eqs. 5–6. Table 3 ablates only the similarity metric. Please report the default values and sensitivity analyses for the circuit-extraction parameters; otherwise the claim that CUD is a stable, intrinsic metric cannot be fully evaluated or reproduced.
minor comments (4)
  1. [Appendix B, Tables 4–5] Tables 4 and 5 appear to report the same 'LLM Rec unlearning with Llama3' setting but with inconsistent default/hard/easy values (e.g., default unlearn efficacy 76.6 vs 78.2). Please clarify what differs between the tables, or merge them.
  2. [Table 1 and Appendix A] RMU is described in Appendix A but does not appear in Table 1, despite being listed as one of the evaluated unlearning methods. If RMU results are omitted, please say so and explain why; otherwise include them.
  3. [Figure 3] The x- and y-axis labels in Figure 3 are both numeric ranges (0.425–0.550) and are not identified; the caption should state which axis is the GradDiff-based CUD and which is the UNDIAL-based CUD.
  4. [Sec. 4.2, Table 6] The 'unique edges' in Table 6 are reported as frequency differences without statistical significance or confidence intervals. Given the small differences for several edges (e.g., m11→m15, −9.3), adding a significance test or effect size would strengthen the mechanistic interpretation.

Circularity Check

2 steps flagged

CUD's easy/hard anchors are fit to post-unlearning loss on the same forget set it later selects from, so Table 1's separation is partly in-sample calibration rather than an intrinsic pre-unlearning difficulty.

specific steps
  1. fitted input called prediction [Sec. 3.1 Eqs. 5–6; Sec. 3.2 Eqs. 7–8; Sec. 4.1 Table 1]
    "Eq. 5 finds samples that have increased loss post-unlearning (i.e., low memorization, easy-to-unlearn) ... denoted as D_f,E. While Eq. 6 finds samples that remain low loss (i.e., high memorization, hard-to-unlearn) ... denoted as D_f,H. After that, we use circuit finding methods to locate the circuits for D_f,E, D_f,H on the original model f_o, denoted C_E, C_H, respectively. ... CUD(zi) = 1−sE/((1−sE)+(1−sH)) ,(8)"

    The anchors C_E and C_H are not intrinsic reference circuits: they are the circuits of the exact samples that Eqs. 5–6 selected using post-GradDiff loss on the same D_f. Eq. 8 then defines 'difficulty' as proximity to those fitted anchors, and Table 1 builds the Easy/Hard sets from that same D_f. Thus ranking by CUD is a nearest-prototype classifier whose prototypes were fit to the outcome being measured; the reported −14.1/+3.3 separation largely re-detects the low-post-unlearning-loss vs high-post-unlearning-loss selection that created C_E and C_H. Cross-method columns and robustness checks add independent evidence, but the sample selection is never moved to a held-out forget set or a different base model, so the calibration loop is not broken.

  2. other [Abstract/Sec. 1; Sec. 3.1]
    "We propose Circuit-guided Unlearning Difficulty (CUD), a pre-unlearning metric that assigns each sample a continuous difficulty score using circuit-level signals. ... we repeat each unlearning method five times with different seeds and take the common samples in all runs to get the stable D_f,E, D_f,H."

    CUD is advertised as a 'pre-unlearning' and method-independent metric, yet the anchor circuits are obtained by running GradDiff unlearning on D_f multiple times and selecting samples by their post-unlearning loss. The 'intrinsic difficulty' scale is therefore calibrated on the very post-unlearning outcome it claims to predict. Calling the score 'pre-unlearning' is only true for scoring a new sample after the anchors have been calibrated; the calibration itself requires the unlearning runs and the post-hoc loss information.

full rationale

The central circularity is the anchor-construction loop: Eqs. 5–6 fit easy/hard sample masks to post-GradDiff loss on D_f, and CUD (Eq. 8) is defined as similarity to the resulting C_E and C_H. Evaluating CUD-selected sets on that same D_f (Table 1) therefore partly measures the selection objective that created the anchors. This is not a fully vacuous identity—CUD uses circuit structure rather than raw loss, and the cross-method (NPO, SimNPO, UNDIAL), cross-similarity, and cross-LMU robustness checks provide some independent confirmation—but the paper never validates on held-out samples or a different base model, so the claim of an intrinsic, method-independent, pre-unlearning difficulty is overstated. No load-bearing self-citation chain or imported uniqueness theorem is present. Score 6 reflects a partial but real by-construction component in the headline empirical claim.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 1 invented entities

The central metric rests on calibration labels produced by the paper's own unlearning runs (Eq. 5/6), plus several unstated circuit-extraction thresholds. These are not independently provided, so the contribution is a fitted index rather than a parameter-free derivation.

free parameters (5)
  • λ (sparsity regularizer in Eq. 5/6)
    Controls the size and composition of the easy/hard sets chosen as anchors; no value or range is reported, so the anchor definition is under-specified.
  • EAP-IG interpolation steps
    The number of α steps used to approximate the integral in Eq. 4 is not reported; it directly affects which edges are discovered and hence every CUD value.
  • Circuit binarization threshold / top-k edges
    Circuits are represented as binary edge matrices in §3.2, but the threshold or number of edges retained per sample is never specified.
  • Similarity metric = cosine (default); Jaccard/Hamming tested for robustness
    CUD depends on sim(·) in Eq. 7; robustness is shown for three choices, but the metric choice is still a free design decision.
  • Seed-consensus requirement = 5 unlearning runs
    Easy/hard sets are defined as samples common to five unlearning runs with different seeds; the number of runs and consensus rule are arbitrary choices.
axioms (4)
  • domain assumption EAP-IG scores faithfully approximate the causal contribution of edges to model predictions.
    The whole circuit extraction in §2–3 relies on this. EAP-IG is an integrated-gradient approximation and may miss nonlinear or saturated effects.
  • ad hoc to paper Bi-level optimization (Eq. 5/6) with sparsity regularization selects stable, representative easy/hard samples.
    The anchors C_E/C_H are defined by this optimization; no proof or external validation establishes that the selected sets are canonical rather than procedure-specific.
  • ad hoc to paper Circuit similarity to aggregated anchors is a valid proxy for unlearning difficulty.
    Eq. 8 defines CUD by this similarity; the mapping from circuit geometry to difficulty is asserted, not derived.
  • domain assumption TOFU and MovieLens-1M with Llama-3.2-1B / Llama3 expose mechanisms representative of real LLM unlearning.
    Claims of method-independence and generalizability rest on two datasets and two base models.
invented entities (1)
  • Reference anchor circuits C_E and C_H no independent evidence
    purpose: Prototypical easy/hard circuit patterns used as the similarity reference in Eq. 7–8.
    They are constructed from the paper's own optimization labels and are not externally validated; CUD is calibrated to these self-defined entities.

pith-pipeline@v1.3.0-alltime-deepseek · 14588 in / 11687 out tokens · 113875 ms · 2026-08-03T10:31:51.354739+00:00 · methodology

0 comments
read the original abstract

Machine unlearning is becoming essential for building trustworthy and compliant language models. Yet unlearning success varies considerably across individual samples: some are reliably erased, while others persist despite the same procedure. We argue that this disparity is not only a data-side phenomenon, but also reflects model-internal mechanisms that encode and protect memorized information. We study this problem from a mechanistic perspective based on model circuits--structured interaction pathways that govern how predictions are formed. We propose Circuit-guided Unlearning Difficulty (CUD), a {\em pre-unlearning} metric that assigns each sample a continuous difficulty score using circuit-level signals. Extensive experiments demonstrate that CUD reliably separates intrinsically easy and hard samples, and remains stable across unlearning methods. We identify key circuit-level patterns that reveal a mechanistic signature of difficulty: easy-to-unlearn samples are associated with shorter, shallower interactions concentrated in earlier-to-intermediate parts of the original model, whereas hard samples rely on longer and deeper pathways closer to late-stage computation. Compared to existing qualitative studies, CUD takes a first step toward a principled, fine-grained, and interpretable analysis of unlearning difficulty; and motivates the development of unlearning methods grounded in model mechanisms.

Figures

Figures reproduced from arXiv: 2601.09624 by Chirag Agarwal, Hadi Amiri, Jiali Cheng, Ziheng Chen.

Figure 1
Figure 1. Figure 1: A mechanistic perspective of unlearning difficulty. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: CUD score distribution of samples from TOFU. Red: Default forget set. Blue: All samples. The default forget samples closely matches the overall distri￾bution, with a wide coverage of all difficulty levels, sug￾gesting that TOFU’s default forget set represents a mid￾level unlearning difficulty. Using CUD, we can select a harder / easier forget sets than the default to control task difficulty and better stre… view at source ↗
Figure 3
Figure 3. Figure 3: CUD score is robust to choices of LMU in Eq. 5 and 6, since the regularizer leads to sparse and consistent selection of representative samples. Corre￾lation coefficient ρ = 0.76. Each point represent in the figure represent the CUD scores computed using the GradDiff (Maini et al., 2024) MU loss and the UN￾DIAL (Dong et al., 2024) MU loss. ilarity metrics, CUD consistently induces the same qualitative separ… view at source ↗
Figure 4
Figure 4. Figure 4: Edge distribution of statistically different easy [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: shows the relationship between CUD and MRD-based difficulty (i.e., 1 - MRD/2) for all samples in TOFU. Overall, the two metrics ex￾hibit a weak correlation (ρ = −0.27), indicating that they capture fundamentally different notions of unlearning difficulty. While CUD produces a roughly normal and balanced distribution, MRD as￾signs a large fraction of samples very small scores, effectively categorizing most … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. TRACER: Token ReAssignment for Concept ERasure in Generative Recommendation

    cs.IR 2026-06 unverdicted novelty 7.0

    TRACER uses token reassignment for concept-related items plus a coherence regularizer to unlearn specific concepts in generative recommendation while preserving utility better than baselines.

  2. CURE:Circuit-Aware Unlearning for LLM-based Recommendation

    cs.IR 2026-04 unverdicted novelty 7.0

    CURE disentangles LLM recommendation circuits into forget-specific, retain-specific, and task-shared modules with tailored update rules to achieve more effective unlearning than weighted baselines.

  3. Towards Understanding the Robustness of Sparse Autoencoders

    cs.LG 2026-04 unverdicted novelty 6.0

    Integrating pretrained sparse autoencoders into LLM residual streams reduces jailbreak success rates by up to 5x across multiple models and attacks.

Reference graph

Works this paper leans on

52 extracted references · 1 canonical work pages · cited by 3 Pith papers

  1. [1]

    Daiki Asami and Saku Sugawara. 2024. https://doi.org/10.18653/v1/2024.findings-acl.913 What makes language models good-enough? In Findings of the Association for Computational Linguistics: ACL 2024, pages 15453--15467, Bangkok, Thailand. Association for Computational Linguistics

    work page doi:10.18653/v1/2024.findings-acl.913 2024

  2. [2]

    Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2021. Machine unlearning. In IEEE Symposium on Security and Privacy (SP)

    2021

  3. [3]

    Yinzhi Cao and Junfeng Yang. 2015. Towards making systems forget with machine unlearning. In Proceedings of the IEEE Symposium on Security and Privacy

    2015

  4. [4]

    Chong Chen, Fei Sun, Min Zhang, and Bolin Ding. 2022. Recommendation unlearning. In Proceedings of the ACM web conference 2022, pages 2768--2777

    2022

  5. [5]

    Ziheng Chen, Jiali Cheng, Hadi Amiri, Kaushiki Nag, Lu Lin, Sijia Liu, Gabriele Tolomei, and Xiangguo Sun. 2025 a . Frog: Fair removal on graph. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, pages 415--424

    2025

  6. [6]

    Ziheng Chen, Jin Huang, Jiali Cheng, Yuchan Guo, Mengjie Wang, Lalitesh Morishetti, Kaushiki Nag, and Hadi Amiri. 2025 b . Future: Flexible unlearning for tree ensemble. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, pages 4680--4684

    2025

  7. [7]

    Jiali Cheng and Hadi Amiri. 2024. Mu-bench: A multitask multimodal benchmark for machine unlearning. arXiv preprint arXiv:2406.14796

    Pith/arXiv arXiv 2024

  8. [8]

    Jiali Cheng and Hadi Amiri. 2025 a . Multidelete for multimodal machine unlearning. In Computer Vision -- ECCV 2024, pages 165--184, Cham. Springer Nature Switzerland

    2025

  9. [9]

    Jiali Cheng and Hadi Amiri. 2025 b . https://doi.org/10.21437/Interspeech.2025-2412 Speech Unlearning . In Interspeech 2025 , pages 3209--3213

    doi:10.21437/interspeech.2025-2412 2025

  10. [10]

    Jiali Cheng and Hadi Amiri. 2025 c . Tool unlearning for tool-augmented LLM s. In Forty-second International Conference on Machine Learning

    2025

  11. [11]

    Jiali Cheng and Hadi Amiri. 2025 d . Understanding machine unlearning through the lens of mode connectivity. arXiv preprint arXiv:2504.06407

    Pith/arXiv arXiv 2025

  12. [12]

    Jiali Cheng, George Dasoulas, Huan He, Chirag Agarwal, and Marinka Zitnik. 2023. https://openreview.net/forum?id=X9yCkmT5Qrl GNND elete: A general strategy for unlearning in graph neural networks . In The Eleventh International Conference on Learning Representations

    2023

  13. [13]

    Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adri \`a Garriga-Alonso. 2023. Towards automated circuit discovery for mechanistic interpretability. Advances in Neural Information Processing Systems, 36:16318--16352

    2023

  14. [14]

    Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W

    Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2022. https://doi.org/10.1162/tacl_a_00459 Time-aware language models as temporal knowledge bases . Transactions of the Association for Computational Linguistics, 10:257--273

    doi:10.1162/tacl_a_00459 2022

  15. [15]

    Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, and Ivan Vuli \'c . 2024. Undial: Self-distillation with adjusted logits for robust unlearning in large language models. arXiv preprint arXiv:2402.10052

    Pith/arXiv arXiv 2024

  16. [16]

    Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, and Ivan Vuli \'c . 2025. https://doi.org/10.18653/v1/2025.naacl-long.444 UNDIAL : Self-distillation with adjusted logits for robust unlearning in large language models . In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguist...

    doi:10.18653/v1/2025.naacl-long.444 2025

  17. [17]

    Vineeth Dorna, Anmol Mekala, Wenlong Zhao, Andrew McCallum, Zachary C Lipton, J Zico Kolter, and Pratyush Maini. 2025. Openunlearning: Accelerating llm unlearning via unified benchmarking of methods and metrics. arXiv preprint arXiv:2506.12618

    arXiv 2025

  18. [18]

    Ali Ebrahimpour-Boroojeny, Hari Sundaram, and Varun Chandrasekaran. 2025. https://openreview.net/forum?id=BkrIQPREkn Not all wrong is bad: Using adversarial examples for unlearning . In Forty-second International Conference on Machine Learning

    2025

  19. [19]

    Ronen Eldan and Mark Russinovich. 2023. Who's harry potter? approximate unlearning in llms. arXiv preprint arXiv:2310.02238

    Pith/arXiv arXiv 2023

  20. [20]

    Chongyu Fan, Jinghan Jia, Yihua Zhang, Anil Ramakrishna, Mingyi Hong, and Sijia Liu. 2025. Towards llm unlearning resilient to relearning attacks: A sharpness-aware minimization perspective and beyond. arXiv preprint arXiv:2502.05374

    Pith/arXiv arXiv 2025

  21. [21]

    Chongyu Fan, Jiancheng Liu, Alfred Hero, and Sijia Liu. 2024 a . https://arxiv.org/abs/2403.07362 Challenging forgets: Unveiling the worst-case forget sets in machine unlearning . Preprint, arXiv:2403.07362

    Pith/arXiv arXiv 2024

  22. [22]

    Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, and Sijia Liu. 2024 b . Simplicity prevails: Rethinking negative preference optimization for llm unlearning. arXiv preprint arXiv:2410.07163

    arXiv 2024

  23. [23]

    Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong, Dennis Wei, and Sijia Liu. 2024 c . https://openreview.net/forum?id=gn0mIhQGNM Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation . In The Twelfth International Conference on Learning Representations

    2024

  24. [24]

    Xiaohua Feng, Yuyuan Li, Chengye Wang, Junlin Liu, Li Zhang, and Chaochao Chen. 2025. A neuro-inspired interpretation of unlearning in large language models through sample-level unlearning difficulty. arXiv preprint arXiv:2504.06658

    Pith/arXiv arXiv 2025

  25. [25]

    Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. 2021. https://doi.org/10.18653/v1/2021.acl-long.144 Causal analysis of syntactic agreement mechanisms in neural language models . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint...

    doi:10.18653/v1/2021.acl-long.144 2021

  26. [26]

    Tal Haklay, Hadas Orgad, David Bau, Aaron Mueller, and Yonatan Belinkov. 2025. https://doi.org/10.18653/v1/2025.acl-long.141 Position-aware automatic circuit discovery . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2792--2817, Vienna, Austria. Association for Computational Linguistics

    doi:10.18653/v1/2025.acl-long.141 2025

  27. [27]

    Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. 2024. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. arXiv preprint arXiv:2403.17806

    Pith/arXiv arXiv 2024

  28. [28]

    Yihuai Hong, Lei Yu, Haiqin Yang, Shauli Ravfogel, and Mor Geva. 2024. Intrinsic evaluation of unlearning using parametric knowledge traces. arXiv preprint arXiv:2406.11614

    Pith/arXiv arXiv 2024

  29. [29]

    Yihuai Hong, Lei Yu, Haiqin Yang, Shauli Ravfogel, and Mor Geva. 2025. https://doi.org/10.18653/v1/2025.emnlp-main.985 Intrinsic test of unlearning using parametric knowledge traces . In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 19513--19535, Suzhou, China. Association for Computational Linguistics

    doi:10.18653/v1/2025.emnlp-main.985 2025

  30. [30]

    Jinghan Jia, Jiancheng Liu, Yihua Zhang, Parikshit Ram, Nathalie Baracaldo, and Sijia Liu. 2024 a . Wagle: Strategic weight attribution for effective and modular unlearning in large language models. Advances in Neural Information Processing Systems, 37:55620--55646

    2024

  31. [31]

    Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. 2024 b . https://doi.org/10.18653/v1/2024.emnlp-main.245 SOUL : Unlocking the power of second-order optimization for LLM unlearning . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 427...

    doi:10.18653/v1/2024.emnlp-main.245 2024

  32. [32]

    Aravind Krishnan, Siva Reddy, and Marius Mosbach. 2025. https://openreview.net/forum?id=Kd97lfFfTu Not all data are unlearned equally . In Second Conference on Language Modeling

    2025

  33. [33]

    Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, and 1 others. 2024. Rethinking machine unlearning for large language models. arXiv preprint arXiv:2402.08787

    Pith/arXiv arXiv 2024

  34. [34]

    Lipton, and J

    Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter. 2024. https://arxiv.org/abs/2401.06121 Tofu: A task of fictitious unlearning for llms . Preprint, arXiv:2401.06121

    Pith/arXiv arXiv 2024

  35. [35]

    Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. 2024. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:2403.19647

    Pith/arXiv arXiv 2024

  36. [36]

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359--17372

    2022

  37. [37]

    Neel Nanda. 2023. https://www.neelnanda.io/mechanistic-interpretability/attribution-patching Attribution patching: Activation patching at industrial scale

    2023

  38. [38]

    Soumyadeep Pal, Changsheng Wang, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. 2025. Llm unlearning reveals a stronger-than-expected coreset effect in current benchmarks. arXiv preprint arXiv:2504.10185

    Pith/arXiv arXiv 2025

  39. [39]

    Vaidehi Patil, Elias Stengel-Eskin, and Mohit Bansal. 2025. Upcore: Utility-preserving coreset selection for balanced unlearning. arXiv preprint arXiv:2502.15082

    Pith/arXiv arXiv 2025

  40. [40]

    Jiaxin Qin, Zixuan Zhang, Chi Han, Pengfei Yu, Manling Li, and Heng Ji. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.700 Why does new knowledge create messy ripple effects in LLM s? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12602--12609, Miami, Florida, USA. Association for Computational Linguistics

    doi:10.18653/v1/2024.emnlp-main.700 2024

  41. [41]

    Hadi Reisizadeh, Jinghan Jia, Zhiqi Bu, Bhanukiran Vinzamuri, Anil Ramakrishna, Kai-Wei Chang, Volkan Cevher, Sijia Liu, and Mingyi Hong. 2025. Blur: A bi-level optimization approach for llm unlearning. arXiv preprint arXiv:2506.08164

    arXiv 2025

  42. [42]

    Hammad Rizwan, Mahtab Sarvmaili, Hassan Sajjad, and Ga Wu. 2024. Instance-level difficulty: A missing perspective in machine unlearning. arXiv preprint arXiv:2410.03043

    Pith/arXiv arXiv 2024

  43. [43]

    Smith, and Chiyuan Zhang

    Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A. Smith, and Chiyuan Zhang. 2025. https://openreview.net/forum?id=TArmA033BU MUSE : Machine unlearning six-way evaluation for language models . In The Thirteenth International Conference on Learning Representations

    2025

  44. [44]

    Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2013. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034

    Pith/arXiv arXiv 2013

  45. [45]

    Aaquib Syed, Can Rager, and Arthur Conmy. 2024. Attribution patching outperforms automated circuit discovery. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 407--416

    2024

  46. [46]

    Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. https://proceedings.neurips.cc/paper_files/paper/2020/file/92650b2e92217715fe312e6fa7b90d82-Paper.pdf Investigating gender bias in language models using causal mediation analysis . In Advances in Neural Information Processing Systems, volume ...

    2020

  47. [47]

    Yixin Wan, Anil Ramakrishna, Kai-Wei Chang, Volkan Cevher, and Rahul Gupta. 2025. Not every token needs forgetting: Selective unlearning to limit change in utility in large language model unlearning. arXiv preprint arXiv:2506.00876

    Pith/arXiv arXiv 2025

  48. [48]

    Hangyu Wang, Jianghao Lin, Bo Chen, Yang Yang, Ruiming Tang, Weinan Zhang, and Yong Yu. 2025. Towards efficient and effective unlearning of large language models for recommendation. Frontiers of Computer Science, 19(3):193327

    2025

  49. [49]

    Rongzhe Wei, Peizhi Niu, Hans Hao-Hsun Hsu, Ruihan Wu, Haoteng Yin, Mohsen Ghassemi, Yifan Li, Vamsi K Potluru, Eli Chien, Kamalika Chaudhuri, and 1 others. 2025. Do llms really forget? evaluating unlearning with knowledge correlation and confidence awareness. arXiv preprint arXiv:2506.05735

    arXiv 2025

  50. [50]

    Puning Yang, Qizhou Wang, Zhuo Huang, Tongliang Liu, Chengqi Zhang, and Bo Han. 2025. Exploring criteria of loss reweighting to enhance llm unlearning. arXiv preprint arXiv:2505.11953

    Pith/arXiv arXiv 2025

  51. [51]

    Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. 2024. https://openreview.net/forum?id=MXLBXjQkmb Negative preference optimization: From catastrophic collapse to effective unlearning . In First Conference on Language Modeling

    2024

  52. [52]

    Kairan Zhao, Meghdad Kurmanji, George-Octavian B a rbulescu, Eleni Triantafillou, and Peter Triantafillou. 2024. https://openreview.net/forum?id=QAbhLBF72K What makes unlearning hard and what to do about it . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024