{"id":"54c8a972-9630-424b-948d-939e3f0f335e","arxiv_id":"2412.14505","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A hybrid unlearning framework switches between partial retraining and direct parameter updates, with an optimized variant that retrains the last few slices after subtraction, reporting 1.5x to 8x speedups at comparable accuracy.","lead":"This paper combines two known machine unlearning approaches: retraining only part of the model when deletion is cheap, and directly subtracting parameter updates when it is not. It reports 1.5x to 8x faster unlearning than exact retraining, with similar accuracy, on four tabular datasets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sequential HS/OHS can resurrect previously forgotten samples: a later partial-retraining branch reloads stale checkpoints that still contain earlier DPUS-removed data, so the 100-request experiments are not well-defined.","rationale":"The reader's conditional verdict is reasonable, and the missing l/φ reporting is a genuine reproducibility defect. However, I found a more fundamental correctness gap: after any DPUS-style removal in HS/OHS, saved checkpoints θ_i and slice contents are not updated. A later PRS request reloads a stale checkpoint that includes the previously removed sample, so that sample's influence re-enters the model. This directly undermines the 'sequential 100 requests' protocol in §4.4 and the subsequent MIA validation. It is an internal inconsistency in the method as described, not a disagreement with external consensus. I therefore recommend REJECT as written, because the central empirical claim is not well-defined without an explicit state-update rule. If the authors supply such a rule or show experimentally that these revocation patterns do not occur, the verdict could reasonably return to CONDITIONAL.","tokens_in":10344,"tokens_out":8044,"duration_ms":76155,"concrete_test":"Run the minimal two-request test on HS/OHS with S=10, l=3, unique revocation samples d1∈D1 and d2∈D10. Process d1 (DPUS path), then d2 (PRS path). After the second request, apply the membership-inference attack from §4.4 to d1. If d1 is predicted 'in training' (or if accuracy differs from running the two requests in the opposite order), the checkpoints are stale and the sequential claim fails. Also inspect the implementation for where D_i, θ_i, and Δ_j are updated after each request; without such updates, the resurrection argument applies.","verdict_should_be":"REJECT","load_bearing_attack":"HS and OHS are specified against a single revocation d and a fixed set of saved checkpoints θ_i and increments Δ_j (Alg. 1–3). The experiments, however, claim to \"sequentially handle 100 randomly generated unlearning requests\" (§4.4). Nothing in Alg. 2 or Alg. 3 updates the saved θ_i or the slice membership after a DPUS-style removal. Concretely, let S=10, l=3, and first request d1∈D1 (i<l): Alg. 2 sets θ'_S=θ_S−Δ_j, but D1 is not modified and θ_1..θ_9 retain d1's contribution. If a later request d2∈D10 (i≥l), the PRS branch loads θ_9 and retrains to θ_S; that checkpoint was trained on D1..D9 and therefore contains d1. The final model has d1's influence restored, so d1 is no longer unlearned. OHS Alg. 3 has the same issue because its PRS branch (line 7) also reloads the original θ_{i−1}. Hence the reported accuracy and MIA effectiveness after 100 sequential requests are not well-defined unless the authors specify and implement an update of all affected checkpoints/increments after every request. This correctness gap is independent of the unreported l/φ values.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes two hybrid machine-unlearning methods, HS and OHS, that combine SISA-style partial retraining (PRS) with Amnesiac-style direct parameter update (DPUS). A threshold l, supposedly derived from an acceptable retraining overhead φ via Eq. (1), decides per request whether to retrain from a saved checkpoint or subtract a saved parameter increment. OHS subtracts from an intermediate checkpoint and retrains the remaining slices to recover accuracy. Experiments on Adult, Purchase, Census, and Olympics with a two-hidden-layer MLP, varying the number of slices S from 5 to 30, report accuracy and unlearning time over 100 sequential requests and claim 1.5×–8× speedups over SISA with comparable accuracy, plus a qualitative membership-inference validation.","tokens_in":10616,"tokens_out":6635,"duration_ms":53059,"significance":"If substantiated, the framework offers a practical accuracy/efficiency trade-off in unlearning, and OHS's subtract-then-fine-tune idea is a reasonable repair for accumulated DPUS error. The paper's strengths are the simplicity of the hybrid selection rule and the breadth of datasets; however, the central quantitative claims are not yet reproducible or fully supported: the threshold values are not reported, Eq. (1) is inconsistent, the sequential protocol is not well-defined under the given algorithms, and the MIA validation is only qualitative. The claimed speedups are plausible but currently a function of unstated choices.","major_comments":[{"comment":"Equation (1) does not describe a retraining workload: it sums slice indices times n/S rather than the number of affected slices times n/S. Additionally, the definition l=⌊S−m⌋ is inconsistent with the algorithm and the example: for S=4 and m=3 it gives l=1, so Alg. 2 would apply PRS to every slice and DPUS would never be used, whereas Fig. 2 states l=3. The paper never reports the φ or l values used for any dataset or S, so the 1.5×–8× speedup and the accuracy trade-off are controlled by an unstated, hand-chosen threshold. Please correct the workload formula, reconcile l with m, and report l/φ for every experimental configuration.","section":"§3.1, Eq. (1)"},{"comment":"The algorithms do not maintain a consistent unlearning state across sequential requests. When Alg. 2 uses DPUS for d1 in an early slice, it updates only θ'S and does not delete d1 from D_i or refresh the saved checkpoints θ_k and increments Δ_j. If a later request d2 falls in a later slice, the PRS branch reloads the original θ_{i−1}, which was trained on data including d1, and retrains to θ'S, thereby reintroducing d1's influence. The same flaw appears in Alg. 3's line 7. Consequently, the 100-request experiments reported in §4.4 are not well-defined under the published algorithms. Please specify how checkpoints, slice membership, and increments are updated after every request, or use a protocol that avoids stale checkpoints.","section":"§3.1–3.2 and §4.4"},{"comment":"The membership-inference validation is entirely qualitative. The text says 'through validation, we find that our methods achieve effective unlearning' but reports no attack success rate, AUC, membership advantage, or comparison with SISA/DPUS. Without quantitative MIA results, the claim that revocation data is 'thoroughly erased' is unsupported. Please report attack metrics for all methods and the revoked-data set, ideally with confidence intervals.","section":"§4.4 (MIA)"},{"comment":"All accuracy and timing comparisons are shown as single curves with no error bars, no number of independent runs, and no seeds. In this setting, an accuracy gap of 0.5% (HS vs SISA) or 5% (DPUS vs HS) cannot be assessed without variance information. Please provide means and standard deviations over multiple runs, or at a minimum the number of seeds and a significance test for the main accuracy comparisons.","section":"§4.4, Figs. 4–5"}],"minor_comments":[{"comment":"The phrase 'extract MU' appears to be a typo for 'exact MU'.","section":"§2"},{"comment":"Reference entries [18] and [19] are duplicates of the same Koh and Liang paper; please consolidate them.","section":"References"},{"comment":"The axis label 'Number of slice' should be pluralized to 'Number of slices'.","section":"Figs. 4–5"},{"comment":"The shadow-model MIA description omits the number of shadow models, the attack model architecture, and the dataset split used to train the attack model; these details are needed for reproducibility.","section":"§4.4"}],"recommendation":"major_revision","confidential_remarks":"The hybrid idea is straightforward and the contribution is incremental relative to SISA [3] and Amnesiac [13], but that alone would not be disqualifying if the empirical claims were airtight. The main risk is that the reported speedups depend on an unreported threshold and the sequential protocol is not well-defined; both must be fixed before publication. I would also suggest the editor ask for a reproducibility statement or code release, given the difficulty of reconstructing the l/φ settings from the current text."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's core idea is simple and plausible: train on slices, keep checkpoints and per-batch increments, use partial retraining when the deleted sample is in a late slice, and use increment subtraction otherwise. The OHS variant—subtract an increment from an earlier checkpoint and retrain the last slices—is a reasonable way to recover accuracy after subtraction. For a single unlearning request, the algorithms are coherent. The empirical trends across four datasets and several slice counts are internally consistent, and the authors are honest that DPUS degrades accuracy.\n\nThe soft spots are serious. The stress-test concern about sequential requests holds up. Algorithms 2 and 3 define unlearning for one request and never update the saved checkpoints or slice contents after a DPUS removal. The experiments run 100 sequential requests. So a later request in a late slice triggers partial retraining from a checkpoint that still contains earlier DPUS-removed samples. Those samples come back. The reported accuracy and MIA results after 100 requests are therefore not measuring what the text claims. This is a load-bearing flaw, not a missing error bar.\n\nThe unreported threshold l/phi is a real second problem. Eq. (1) is ambiguous, and no values of l or phi are given for any dataset. The 1.5x–8x speedups are controlled by that unstated choice, so they cannot be independently reproduced. There are also no error bars, no code, and the MIA validation is a hand-wave with no numbers.\n\nThe paper would be salvageable: report single-request results or fix the sequential protocol (e.g., update all checkpoints after every request, or restrict sequential requests to slices whose checkpoints are clean), and release code with l/phi values. As it stands, the central empirical claim is not well-defined.\n\nFor practitioners who need cheap approximate deletion in tabular MLPs, the single-request idea is worth looking at, but I wouldn't build on the paper's current sequential results. Yes, send it to serious peer review—a good referee could force the fixes—but the current version should not be accepted as is.","headline":"A sensible hybrid unlearning recipe whose 100-request experiments are undermined by stale checkpoints that resurrect previously deleted samples; fixable, but as is it shouldn't be the basis for the headline claims.","tokens_in":11166,"tokens_out":3954,"would_cite":false,"duration_ms":34729,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes a hybrid machine-unlearning strategy that chooses between partial retraining and direct parameter subtraction based on estimated retraining cost, achieving 1.5x-8x faster unlearning than SISA with comparable accuracy.","keywords":["machine unlearning","exact unlearning","approximate unlearning","SISA","direct parameter update","hybrid strategy","right to be forgotten","model accuracy"],"falsifier":"Run HS and OHS on Adult with $S=10$ while setting $l$ strictly from a pre-specified $\\varphi$ (for example, retraining at most 20% of the data is acceptable), without tuning $l$ after seeing unlearning time or accuracy; if the 1.5x-8x speedup over SISA does not appear, or the accuracy gap to SISA exceeds the reported 0.5%, the boundary-selection claim fails.","tokens_in":10145,"feed_emoji":"🧠","tokens_out":5697,"duration_ms":42263,"temperature":0.7,"pith_summary":"The paper tries to show that machine unlearning need not choose between exact but expensive retraining and fast but inaccurate parameter subtraction. It proposes a hybrid strategy (HS) that estimates the retraining workload caused by a revocation and, depending on where the revoked sample sits in the training schedule, either retrains the affected slices from a saved checkpoint or subtracts the batch's parameter increment directly. A second strategy (OHS) subtracts the increment at an earlier checkpoint and then retrains a fixed number of slices to recover accuracy. The authors claim this yields 1.5x to 8x faster unlearning than SISA while keeping accuracy close to SISA on Adult, Purchase, Census, and Olympics. If true, service providers could honor many right-to-be-forgotten requests at a fraction of the retraining cost.","feed_headline":"Hybrid unlearning cuts retraining by up to 8x at similar accuracy","feed_subtitle":"It picks cheap subtraction or retraining per request, keeping accuracy near exact unlearning.","key_machinery":"The load-bearing object is the slice threshold $l$, derived from the acceptable retraining overhead $\\varphi$ in Eq. (1), together with the saved checkpoint parameters $\\theta_i$ and saved batch increments $\\Delta_j$. The threshold turns the hybrid into a decision rule: if the revoked sample lies before position $l$, retraining the remainder would exceed the acceptable data-read budget, so the method uses the cheap DPUS subtraction; if it lies at or after $l$, it retrains from the nearest checkpoint. OHS modifies the subtraction path by rewinding to $\\theta_{S-l}$, subtracting there, and retraining $l$ slices, which corrects the accumulated error that repeated DPUS calls would otherwise build up.","core_discovery":"The central claim is that the exact/approximate split in machine unlearning is a false choice: an acceptable-retraining budget can divide the training run into a cheap zone and an expensive zone, and the unlearning procedure can pick per request. Concretely, given S slices, saved checkpoints $\\theta_i$, and saved per-batch increments $\\Delta_j$, the threshold $l = \\lfloor S - m \\rfloor$ is computed from the largest $i$ whose cumulative data reads $i \\cdot n/S + (i+1) \\cdot n/S + \\dots + S \\cdot n/S$ fit within the tolerable overhead $\\varphi$. Revocations in slices $i < l$ take the approximate path—subtract $\\Delta_j$ from $\\theta_S$ for HS, or subtract from $\\theta_{S-l}$ and retrain $l$ slices for OHS—while revocations in $i \\geq l$ take the exact path of deleting the sample and retraining from $\\theta_{i-1}$. The experiments on four real datasets with MLP classifiers report that HS is 6x to 8x faster than SISA and OHS is 1.5x to 6x faster, with OHS accuracy close to SISA.","pith_inferences":["Editorial inference: the same threshold logic could become adaptive, raising or lowering $l$ as the distribution of incoming revocation requests shifts.","Editorial inference: the paper's timing numbers do not count the memory overhead of storing every batch increment $\\Delta_j$; a deployment-level comparison would include that storage cost.","Editorial inference: the subtract-then-retrain correction in OHS is a template that could be applied to influence-function-based certified removal, where the analogous accuracy loss is also corrected by local retraining."],"forward_implications":["A provider can answer a stream of unlearning requests at a fraction of SISA's retraining cost while keeping the delivered model nearly as accurate as an exactly unlearned one.","Models trained with saved checkpoints and batch increments, as in SISA-style pipelines, can be retrofitted with HS or OHS without changing the training architecture or hyperparameters.","OHS provides a concrete accuracy-for-time knob: retraining $l$ slices after subtraction bounds the accuracy loss from repeated approximate unlearning, so the operator can trade a little speed for near-exact accuracy.","Because only the affected slice position matters, the method's benefit grows as the number of slices $S$ increases, since more checkpoints mean shorter retraining spans when the exact path is taken."],"supporting_citations":[{"why":"Supplies the SISA framework and the partial-retraining strategy (PRS) that HS and OHS build on and compare against.","marker":"[3]"},{"why":"Supplies the direct parameter update strategy (DPUS) that subtracts per-batch increments and serves as the approximate-unlearning baseline.","marker":"[13]"},{"why":"Provides the membership-inference attack used to validate that revoked data no longer influences the unlearned model.","marker":"[23]"},{"why":"Supplies the Adult dataset used in the efficiency and accuracy evaluation.","marker":"[2]"},{"why":"Supplies the Purchase dataset used in the efficiency and accuracy evaluation.","marker":"[17]"},{"why":"Supplies the Census dataset used in the efficiency and accuracy evaluation.","marker":"[9]"},{"why":"Supplies the Olympics dataset used in the efficiency and accuracy evaluation.","marker":"[16]"}],"fun_headline_variants":["Machine unlearning hybrid: 8x faster, accuracy near exact","Adaptive unlearning picks exact or cheap retraining per request","Hybrid unlearning cuts cost 8x, keeps accuracy","Unlearning: choose retrain or subtract, up to 8x speedup"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes the threshold $l$ (equivalently the acceptable retraining overhead $\\varphi$) can be fixed in advance and accurately separates cheap from expensive retraining, but the paper never reports the $l$ or $\\varphi$ values used, so the claimed speedups may depend on hindsight selection of that threshold.","fun_headline_variants_meta":{"raw":{"variants":["Machine unlearning hybrid: 8x faster, accuracy near exact","Adaptive unlearning picks exact or cheap retraining per request","Hybrid unlearning cuts cost 8x, keeps accuracy","Unlearning: choose retrain or subtract, up to 8x speedup"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000198,"raw_usage":{"total_tokens":1405,"prompt_tokens":1016,"completion_tokens":389,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":313}},"tokens_in":632,"tokens_out":389,"duration_ms":3468,"temperature":1.0,"reasoning_tokens":313,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:10:18.524607+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run HS and OHS on Adult with $S=10$ while setting $l$ strictly from a pre-specified $\\varphi$ (for example, retraining at most 20% of the data is acceptable), without tuning $l$ after seeing unlearning time or accuracy; if the 1.5x-8x speedup over SISA does not appear, or the accuracy gap to SISA exceeds the reported 0.5%, the boundary-selection claim fails.","supporting_citations":[{"cited_title":"In: 20 21 IEEE Symposium on Security and Privacy (SP)","cited_arxiv_id":null,"evidence_quote":"Supplies the SISA framework and the partial-retraining strategy (PRS) that HS and OHS build on and compare against."},{"cited_title":"In: Proceedings of the AAAI Conference on Artificial Intelligence","cited_arxiv_id":null,"evidence_quote":"Supplies the direct parameter update strategy (DPUS) that subtracts per-batch increments and serves as the approximate-unlearning baseline."},{"cited_title":"In: 2017 IEEE symposium on security and privacy (SP)","cited_arxiv_id":null,"evidence_quote":"Provides the membership-inference attack used to validate that revoked data no longer influences the unlearned model."},{"cited_title":"http://archive.ics.uci.edu/ml (1996)","cited_arxiv_id":null,"evidence_quote":"Supplies the Adult dataset used in the efficiency and accuracy evaluation."},{"cited_title":"https://www.kaggle.com/competitions/acquire-valued-shoppers-chal","cited_arxiv_id":null,"evidence_quote":"Supplies the Purchase dataset used in the efficiency and accuracy evaluation."},{"cited_title":"https://archive.ics.uci.edu/ml/machine-learning-databases/census-income-mld","cited_arxiv_id":null,"evidence_quote":"Supplies the Census dataset used in the efficiency and accuracy evaluation."},{"cited_title":"https://www.kaggle.com/heesoo37/120-years-of-olympic-history-athletes-and-results","cited_arxiv_id":null,"evidence_quote":"Supplies the Olympics dataset used in the efficiency and accuracy evaluation."}],"review_version":1}