{"id":"01bd47b0-b6d6-4c11-8743-c3d292c551e4","arxiv_id":"2412.02432","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"DEL, which pairs channel-level weighted-gradient localization with reset-and-finetune, reports state-of-the-art unlearning metrics on CIFAR-10, SVHN, and ImageNet-100.","lead":"This paper proposes a new localization strategy for machine unlearning that identifies a small set of network parameters to modify, then resets and fine-tunes only those parameters. The method, DEL, reports better forgetting and accuracy trade-offs than existing localized and full-model unlearning baselines, which matters for efficient data deletion and model maintenance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim depends on matching an oracle that is deliberately under-trained (20 epochs, half learning rate) rather than A(Dtrain\\S) from Definition 2.1; if the oracle is instead retrained to convergence, DEL's near-zero deltas and its advantage over full-parameter methods may not persist.","rationale":"I read the paper as an empirical unlearning paper whose main deliverable is a specific localization strategy (DEL) and a claim that it outperforms both localized and full-parameter unlearning on three datasets. The strongest independent support is the control experiment in Table 5 (random masks perform worse), the forget-set-specificity analysis in Table 6, and the inclusion of confidence intervals; these make the mechanism story credible. The load-bearing weakness is not the mechanism but the reference point against which all success is measured. Definition 2.1 defines unlearning as closeness to A(Dtrain\\S), and A is the training algorithm. Appendix A.2 changes A for the oracle by reducing epochs from 50 to 20 and halving the learning rate, so the gold standard is a different, deliberately underfitted distribution. Since all Δ metrics are relative to this distribution, \"matching the oracle\" does not establish matching a properly retrained model, and the easy match can be an artifact of underfitting. This is more specific than the reader's weakest assumption, though related; I partially agree with the reader. The proposed check is deliberately narrow: train the oracle to convergence on one dataset and recompute Table 2. If DEL remains SOTA, the concern fails; if not, the paper should be conditional on repeating comparisons against a stronger oracle. I would keep the reader's CONDITIONAL verdict (no change from my pass), because the concern is substantive but addressable and the paper contains enough structure for a clean re-run.","tokens_in":21017,"tokens_out":6349,"duration_ms":67732,"concrete_test":"Re-run the CIFAR-10/ResNet-18 comparison in Table 2 with the oracle retrained from scratch on the retain set using the full original recipe (50 epochs, base learning rate 0.1, cosine decay, momentum 0.9, batch size 128, same seeds), while keeping all unlearning methods identical. Then recompute Δforget, ΔMIA, and Δtest for DEL and the strong baselines (SalLoc-RL, Shallowest-RL, Fine-tuning, Random Label). If DEL still matches the new oracle within one standard error and remains the best method on both IID and non-IID forget sets, the weak-oracle objection is resolved. If the deltas shift materially or DEL no longer matches, the SOTA claim is not supported until comparisons are repeated against a correctly trained reference.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that DEL matches retrain-from-scratch and thereby sets a new state of the art. But the \"oracle\" used as the gold standard is not retraining with the same algorithm A required by Definition 2.1. Appendix A.2 states it follows the original procedure \"except the number of epochs, which we set to 20, and learning rate, which is the half of that in original training.\" A model trained for 20 epochs at half LR is systematically underfitted relative to the 50-epoch original model. Because every metric is reported as Δ = Oracle − Method, all conclusions are relative to this weak reference. A weak oracle has lower forget-set accuracy and lower confidence on forget examples, so any post-hoc method that partially damages the network (e.g., resetting 30% of channels and short finetuning) can approach the oracle's forget accuracy without removing the influence of S. This is exactly the regime DEL operates in; full-parameter methods such as Finetune or Random Label \"overshoot\" the weak oracle (negative Δforget) partly because they retain more information than a 20-epoch model, not necessarily because they unlearn badly. If the oracle were trained with the full recipe, the absolute forget-accuracy target would be higher and the ranking among methods could change. The issue is compounded by tuning each method's learning rate and budget on the same test metrics (Appendix A.2, Table 7) and by three-seed confidence intervals with many overlapping margins, but the under-trained oracle is the specific point that makes the SOTA claim load-bearing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies localized machine unlearning, where only a small subset of parameters is modified. It proposes a localization strategy that scores each output channel/neuron by the average of the top-h magnitudes of weighted gradients over the forget set, then selects the top channels within a parameter budget. Pairing this strategy with a reset-and-finetune unlearning algorithm yields DEL. The paper reports experiments on CIFAR-10/ResNet-18, SVHN/ViT, and ImageNet-100/ResNet-50, with IID and non-IID forget sets, comparing against full-parameter and localized baselines using forget accuracy, MIA score, and test accuracy, all measured as differences from a retrain-from-scratch oracle. It claims DEL achieves state-of-the-art unlearning quality and utility, and includes ablations on localization-strategy components, random-mask controls, and robustness to parameter budget.","tokens_in":21341,"tokens_out":5386,"duration_ms":49586,"significance":"If the empirical claims hold, DEL is a practically attractive localized unlearning method: it is simple, modifies a small fraction of parameters, and the paper provides useful design insights (channel-level granularity and weighted gradients) that are plausibly transferable. The paper's strengths include a random-mask control (Table 5) that directly supports the necessity of data-dependent parameter selection, a factorial comparison of granularity and criticality criteria (Table 1), and evaluation on multiple architectures and forget-set types. The main reservations concern the evaluation protocol: the oracle is trained with only 20 epochs at half learning rate, and per-method budgets and learning rates are selected on the same test metrics used for reporting, with several key margins within one standard error. Because all claims are expressed as deviations from this oracle, the state-of-the-art conclusion is contingent on the oracle being a faithful reference point.","major_comments":[{"comment":"The oracle ('Retraining') is trained for 20 epochs with half the original learning rate, whereas the original model is trained for 50 epochs. This under-trained oracle likely has lower forget-set accuracy and lower confidence on forget examples; consequently, a method that partially damages the network (e.g., DEL's reset of 30% of channels) can achieve near-zero Δforget without actually removing the influence of S. This is load-bearing because every metric in the main tables is defined as a difference from this oracle, and the abstract's 'matching retrain-from-scratch' claim is only as strong as the oracle. The authors should retrain the oracle with the full 50-epoch, full-learning-rate recipe (or justify the reduced schedule as a deliberate and unbiased reference point) and re-run the comparisons; otherwise Definition 2.1's gold standard is not met.","section":"Section A.2 (Oracle training) and Section 6, Tables 2-4"},{"comment":"The paper reports each method's best budget and learning rate chosen using the same test-set metrics on which final numbers are reported (e.g., 'we report results using its best identified parameter budget and its best-paired unlearning algorithm for that setting'; Table 7 tunes learning rates on the same IID/non-IID forget sets). This selection bias is compounded by the fact that many key margins are within one or two standard errors (e.g., Table 2 Non-IID Δforget: DEL 0.43±1.06 vs SalLoc-RL -2.8±1.45; IID Δforget: DEL 0.97±0.42 vs L1-sparse 1.80±1.20). To support the SOTA claim, the authors should use a validation split for hyperparameter and budget selection, or report results for a fixed budget across all methods, and provide a multiple-comparison-aware analysis.","section":"Section 6 and Appendix A.2, Table 7"},{"comment":"The abstract and Section 6 state that DEL outperforms all compared methods on two unlearning metrics across all datasets and forget sets. However, on SVHN/ViT with the IID forget set, the confidence-based MIA in Table 8 shows DEL ΔMIA = -5.48±0.64, while Fine-tuning achieves -1.97±0.32 (closer to the oracle); Table 3 shows the same for correctness-based MIA (DEL -4.26±0.32 vs Fine-tuning -1.05±0.42). The paper acknowledges this in A.4 but does not qualify the abstract or the concluding SOTA claim. This inconsistency is load-bearing because it is part of the paper's headline contribution.","section":"Section A.4, Table 8 and Table 3 (SVHN IID)"},{"comment":"The proposed localization score c_oi is defined as the average of the top h scores of the channel's parameters, but neither the main text nor Appendix A.2 reports the value of h or its tuning range. Since h controls the granularity of the channel criticality estimate and is one of the few method-specific hyperparameters, its absence makes the method irreproducible and leaves a free parameter in the comparison. The authors should specify h for each dataset (or explain how it is selected).","section":"Section 5.2 and Algorithm 1"}],"minor_comments":[{"comment":"The legend entries 'Earliest-0.14' and 'Earliest-0.25' should presumably read 'Shallowest', since the plot labels use 'Shallowest-14%' and 'Shallowest-25%'.","section":"Figure 3"},{"comment":"The caption 'on Non-IID and IID forget set when training a ViT model on SVHN dataset' mixes singular and plural; it should be 'forget sets' and 'the SVHN dataset'.","section":"Table 3 caption"},{"comment":"The text says Reset + Finetune 'resets the identified critical parameters ... and finetunes only the identified critical parameters and the classifier layer', but Algorithm 1 suggests the mask alone determines the updated parameters; please clarify whether the classifier layer is always updated regardless of the mask.","section":"Section 4, experimental setup"},{"comment":"The table caption mentions a shaded region for 'our method', but the table as typeset does not show shading; please add the shading or describe the region in text.","section":"Table 6"}],"recommendation":"major_revision","confidential_remarks":"The main risk is that the SOTA claim is an artifact of a weak oracle and test-set-based hyperparameter selection. The core method is reasonable, but the evaluation protocol needs to be fixed before publication. The paper is within scope for an ML venue, and the related-work discussion is appropriate. A fully retrained oracle and a validation-based selection protocol would substantially increase confidence in the conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is worth taking seriously. The authors turn memorization localization intuitions into a cheap channel-level selection rule—top-h averaged |weight × gradient|, computed in one-shot over batches—and pair it with reset-and-finetune to get DEL. The ablations are the best part: Table 1 shows channel granularity and weighted gradients each help; Table 5’s random-mask control is exactly the right test and shows the mask itself matters, not just how many parameters you touch. The budget robustness story is plausible and the paper covers three datasets/architectures and IID/non-IID forget sets.\n\nThe soft spot is the oracle, and it is load-bearing. Appendix A.2 says the retrain-from-scratch oracle uses 20 epochs and half the learning rate of the original recipe (50 epochs). Definition 2.1 calls for A(Dtrain\\S) with the same algorithm A, but this oracle is deliberately under-trained. Every metric is reported as Δ = Oracle − Method, so all conclusions are relative to that weak reference. A 20-epoch model has lower forget accuracy and lower confidence on forget examples, so DEL’s near-zero deltas may mean it is approximating an undertrained model, not genuinely removing S’s influence. The full-parameter baselines that ‘overshoot’ with negative Δforget might be retaining more information than the 20-epoch oracle, rather than unlearning poorly. If the oracle were trained with the full recipe, the absolute targets would shift and the ranking could change. Section A.4 already hints at this: on ViT/SVHN with an IID forget set, full-parameter fine-tuning beats DEL on confidence-based MIA.\n\nSecondary issues: each method’s budget and learning rate are tuned on the same test metrics used for final evaluation (Table 7), which biases the comparison. Three seeds give overlapping confidence intervals on several key margins. No code is released, so replication is harder than it should be.\n\nNone of this sinks the paper. The localization strategy is likely to be useful, and the ablations are honest and informative. But the state-of-the-art claim as written is not yet supported. This deserves a serious referee: the evaluation-protocol question it raises is valuable in itself. My recommendation: major revision, with a properly converged oracle (or at least a validation split for tuning), and code release. I would bring it to reading group.","headline":"A smart localization strategy and honest ablations, but the SOTA claim leans on an under-trained oracle that makes every delta relative to a weak gold standard.","tokens_in":21914,"tokens_out":3628,"would_cite":true,"duration_ms":35563,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that selecting parameters to unlearn by channel-level weighted gradients on the forget set, then resetting and finetuning them, makes localized machine unlearning match retraining-from-scratch while updating only about…","keywords":["machine unlearning","localized unlearning","memorization","parameter localization","weighted gradient saliency","membership inference attack","Deletion by Example Localization (DEL)","forget set"],"falsifier":"Train a model, compute DEL's mask, then replace it with a random mask that has the same number of selected channels per layer; if the random mask matches DEL's forget accuracy and MIA score within noise, the criticality score is not doing the work. Alternatively, measure whether a neuron's top-$h$ weighted-gradient score predicts the change in forget-set loss when that neuron is reset; a near-zero correlation across layers would falsify the selection mechanism.","tokens_in":20858,"feed_emoji":"🧹","tokens_out":6029,"duration_ms":57157,"temperature":0.7,"pith_summary":"This paper argues that machine unlearning is best done locally: instead of updating all weights, an unlearning algorithm should identify a small set of parameters most responsible for the forget set and modify only those. Drawing on hypotheses about where neural networks memorize training examples, the authors derive a localization strategy that scores each neuron by the average magnitude of its weights times their gradients on the forget set, then resets the top-scoring neurons and finetunes them. The resulting method, Deletion by Example Localization (DEL), is reported to match a retrained-from-scratch oracle on forget-accuracy and membership-inference metrics across three dataset and architecture pairs and both IID and non-IID forget sets, while updating about 30% of the parameters and preserving utility. The paper's central claim is that this channel-level, forget-set-specific selection is what makes localized unlearning succeed.","feed_headline":"Unlearning matches retraining by resetting 30% of the network","feed_subtitle":"Channel-level weighted-gradient masks pick which parameters to delete, keeping accuracy while erasing the forget set.","key_machinery":"The load-bearing object is a localization mask $m_\\alpha$ constructed from a per-neuron criticality score. For each parameter $j$, the score is $s_j = |\\theta_j \\cdot g_j(\\theta_o, S)|$, the magnitude of the parameter value times its gradient on the forget set; for each output channel, the criticality is the average of the top $h$ parameter scores in that channel. The mask selects the largest number of channels that fit inside a budget $\\alpha$ of total parameters, and the unlearning step resets those parameters and finetunes them on the retain set. The channel grouping acts as smoothing that makes the heuristic criticality decisions less error-prone than per-parameter choices, and the weighted-gradient criterion acts as a regularizer that down-weights small parameters. Paired with Reset plus Finetune, this mask is the algorithm DEL.","core_discovery":"The central discovery is that the success of localized unlearning depends less on which unlearning algorithm is applied than on which parameters are chosen to modify, and that the right choice is a channel-level, forget-set-specific mask based on weighted gradients. The paper shows that data-agnostic strategies fail: modifying only deepest layers preserves utility but does not forget, while modifying shallowest layers forgets but destroys utility. A direct translation of memorization-localization hypotheses, CritMem, forgets too little and is expensive. DEL instead computes, for each parameter, $|\\theta_j \\cdot g_j(\\theta_o, S)|$, averages the top scores within each output channel, and resets the highest-scoring channels within a parameter budget before finetuning them on the retain set. Across CIFAR-10/ResNet-18, SVHN/ViT, and ImageNet-100/ResNet-50, DEL is reported to come closest to the retrain-from-scratch reference on forget accuracy and membership-inference score, outperforming both full-parameter methods and prior localized methods, and to beat previous localized methods on test accuracy.","pith_inferences":["One could test whether the channel-level weighted-gradient score approximates the Fisher information or the empirical influence of forget examples, which might unify DEL with other localized unlearning methods like SSD.","Since DEL's advantage over SalUn grows when budgets shrink, a natural stress test is to push budgets to 5-10% and see whether channel-level smoothing continues to dominate parameter-level saliency.","The paper compares against an oracle trained for fewer epochs at half learning rate; a stronger oracle would likely make all approximate methods look worse, so DEL's margins over the oracle should be re-measured against a fully-trained retrain baseline.","For IID forget sets on ViT, the paper notes some full-parameter methods beat localized methods on MIA, suggesting DEL's superiority may not transfer to all architectures or forget-set distributions."],"forward_implications":["If DEL's reported results hold, an unlearning request can be serviced by retraining only about 30% of a network's parameters, which changes the economics of data deletion in deployed classifiers.","The finding that forget-set-specific localization matters more for non-IID than IID forget sets suggests unlearning systems should adapt their parameter selection to the distribution of the deletion request.","The result that random masks with the same layer structure perform far worse implies that where you modify matters as much as how much, guiding future localized unlearning designs.","Because DEL pairs well with different unlearning algorithms, its localization mask could be dropped into existing pipelines such as Random Label and NegGrad+ to improve them."],"supporting_citations":[{"why":"Supplies the channel-level criticality criterion and the memorization-localization algorithm that DEL's localization strategy builds on.","marker":"Maini et al. (2023)"},{"why":"Provides the SalUn/SalLoc baseline, the gradient-saliency localization approach DEL is compared against, and the evaluation protocol and metrics.","marker":"Fan et al. (2023)"},{"why":"Introduces EU-k and CF-k, the deepest-layer localized unlearning baselines used in the comparison.","marker":"Goel et al. (2022)"},{"why":"Supplies SSD, a Fisher-information-based localized unlearning method used as a state-of-the-art baseline.","marker":"Foster et al. (2024)"},{"why":"Supplies NegGrad+ and SCRUB, full-parameter unlearning baselines that DEL outperforms.","marker":"Kurmanji et al. (2024)"},{"why":"Supplies the L1-sparse finetuning baseline that combines sparsity with unlearning.","marker":"Liu et al. (2024)"},{"why":"Supplies Random Label, the unlearning algorithm paired with SalUn in the comparison.","marker":"Graves et al. (2021)"},{"why":"Supplies the 'deep layers memorize' hypothesis behind the Deepest localization strategy examined in the paper.","marker":"Baldock et al. (2021)"},{"why":"Defines label memorization, the formal notion connecting memorization to unlearning.","marker":"Feldman (2020)"},{"why":"Provides empirical evidence linking memorization scores to unlearning difficulty, motivating the investigation.","marker":"Zhao et al. (2024)"}],"fun_headline_variants":["Localized unlearning: select parameters, not just algorithms","DEL resets critical channels to unlearn better","Weighted-gradient masks pin down what to forget","Parameter choice outweighs algorithm in unlearning","Channel-level masks drive effective machine unlearning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that tuning each method's parameter budget and hyperparameters on the same test set used for the metrics gives a fair picture, even though the oracle retrained model is trained for only 20 epochs at half the learning rate.","fun_headline_variants_meta":{"raw":{"variants":["Localized unlearning: select parameters, not just algorithms","DEL resets critical channels to unlearn better","Weighted-gradient masks pin down what to forget","Parameter choice outweighs algorithm in unlearning","Channel-level masks drive effective machine unlearning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000935,"raw_usage":{"total_tokens":4001,"prompt_tokens":949,"completion_tokens":3052,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":2981}},"tokens_in":565,"tokens_out":3052,"duration_ms":25927,"temperature":1.0,"reasoning_tokens":2981,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:27:06.204815+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a model, compute DEL's mask, then replace it with a random mask that has the same number of selected channels per layer; if the random mask matches DEL's forget accuracy and MIA score within noise, the criticality score is not doing the work. Alternatively, measure whether a neuron's top-$h$ weighted-gradient score predicts the change in forget-set loss when that neuron is reset; a near-zero correlation across layers would falsify the selection mechanism.","supporting_citations":[],"review_version":1}