{"id":"24879590-4bac-4a88-84d0-e83f7141b70d","arxiv_id":"2508.18509","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"SalUn unlearning on MedMNIST achieves near-retraining accuracy on BloodMNIST and OrganAMNIST but a roughly 8 to 10 point accuracy gap on PathMNIST, at a fraction of the runtime.","lead":"This paper applies an existing machine unlearning method, SalUn, to three public medical image datasets and compares its performance with full model retraining. The authors report that unlearning is much faster, but the accuracy gap versus retraining is small on two datasets and larger on the third.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PathMNIST test-accuracy gap and irreproducible AG values undercut the unqualified 'close to retraining' claim","rationale":"The reader's CONDITIONAL verdict is appropriate, but the most load-bearing concern is not primarily the absence of repeated runs; it is the direct contradiction between the paper's own PathMNIST numbers and the unqualified central claim. The reader's rationale does mention the PathMNIST accuracy gap and the AG inconsistency, but the weakest_assumption field emphasizes the single-split issue, which is secondary. The paper does provide a useful first application of SalUn to medical benchmarks and reports substantial runtime savings (e.g., 7.6 min vs 160 min on PathMNIST at δ=10%), and those parts deserve credit. However, the 'close to retraining' claim needs to be restricted to BloodMNIST and OrganAMNIST, or supported by multi-seed variance estimates plus a corrected AG calculation. These are fixable through revision, so the verdict should remain CONDITIONAL rather than REJECT or ACCEPT. My agreement with the reader is partial: we agree on the bottom line, but the load-bearing point is the unqualified claim contradicted by the PathMNIST table and the irreproducible AG, not just missing error bars.","tokens_in":6083,"tokens_out":8777,"duration_ms":96617,"concrete_test":"Run the PathMNIST experiments (δ=10% and 50%) with at least 5 distinct random forget-set splits, training both retrained and SalUn models from the same seeds, and report mean ± std for TA. If the mean SalUn–retrain TA gap is ≥5 points and its 95% CI excludes 0, the 'close to retraining' claim is false for PathMNIST. Also recompute AG from the per-metric differences in each split to confirm whether the reported AG values are reproducible.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The abstract/conclusion claim that SalUn is 'close to full retraining' on PathMNIST, OrganAMNIST, and BloodMNIST is not supported by the paper's own tables. In Table 1 (δ=10%), PathMNIST SalUn TA is 77.49 vs 87.77 for retraining, a 10.28-point gap; in Table 2 (δ=50%), 83.87 vs 91.80, a 7.93-point gap. On a 9-class medical image task, an 8–10 point drop is not 'close,' and the text never qualifies this. The AG metric intended to quantify closeness is also not reproducible: §3.3 defines AG as |mean of differences|, but the table values behave like a mean of absolute differences; even under that reading, recomputing Table 2 yields e.g. BloodMNIST (1.25+0.07+0.29+3.16)/4 = 1.19, not the reported 1.57, OrganAMNIST 0.54 not 0.70, PathMNIST 4.23 not 4.91. Since each setting uses a single random forget split with no variance, the PathMNIST gap cannot be dismissed as noise. Thus the central claim is overstated and the metric supporting it is internally inconsistent.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates SalUn, a state-of-the-art approximate machine-unlearning method, on three MedMNIST medical image classification datasets (BloodMNIST, OrganAMNIST, PathMNIST) using a ResNet-18 backbone. It compares SalUn against full retraining using UA, RA, TA, MIA, AG, and RTE metrics at forget rates of 10% and 50%, and also studies the effect of data augmentation (none, default random crop+horizontal flip, and default plus RandomAug) on the AG metric. The central claim, stated in the abstract and conclusion, is that SalUn achieves performance close to full retraining and is an efficient solution for medical applications.","tokens_in":6398,"tokens_out":2100,"duration_ms":22228,"significance":"If the claim were substantiated, this would be a useful early evaluation of an existing unlearning method in a domain (medical imaging) where privacy compliance is important. The paper also raises the underexplored question of how data augmentation interacts with unlearning quality. However, the evidence provided is uneven: the paper's own Table 1 and Table 2 show a 10.28-point and 7.93-point testing-accuracy gap on PathMNIST, which is not 'close' to retraining for that dataset. Moreover, the AG metric that quantifies closeness is defined inconsistently with the numbers in the tables and is not reproducible from the reported data. Because the central claim depends directly on these tables and AG, the paper currently overstates its findings. The use of a recognized unlearning method and a standard benchmark, and the focus on RTE, are positive features, but the evaluation lacks repeated runs, error bars, and statistical testing.","major_comments":[{"comment":"The claim that SalUn 'achieves performance close to full retraining' is contradicted by the PathMNIST TA results: in Table 1 (δ=10%) TA is 77.49 vs 87.77 (10.28-point gap), and in Table 2 (δ=50%) TA is 83.87 vs 91.80 (7.93-point gap). For a 9-class medical classification task, an 8–10 point drop is not 'close'. The text should either remove the unqualified claim or clearly restrict it to BloodMNIST and OrganAMNIST and characterize PathMNIST as substantially degraded.","section":"Abstract, §4, Tables 1 and 2, §5"},{"comment":"The AG metric is not defined consistently with the reported values. §3.3 defines AG as the absolute value of the mean of the differences between UA, RA, TA, and MIA ('módulo da média das diferenças'), which would be |(d1+d2+d3+d4)/4|. But the table values for Table 1 match a mean of absolute differences: e.g., BloodMNIST (0.84+0.12+0.32+1.59)/4 = 0.72. In Table 2, the reported values match neither definition: BloodMNIST would be (1.25+0.07+0.29+3.16)/4 = 1.19, not 1.57; OrganAMNIST 0.54, not 0.70; PathMNIST 4.23, not 4.91. The definition in the text must be corrected and the tables recomputed, or a clear explicit formula for AG must be given and the numbers checked.","section":"§3.3, Tables 1 and 2"},{"comment":"All experiments appear to be run on a single random split of the training set into forget and remaining sets, with no repeated seeds, no error bars, and no significance tests. Given that the PathMNIST TA gap varies between 10.28 and 7.93 across forget rates, the reader cannot determine whether these differences are stable or artifacts of a particular split. At minimum, report mean ± std over at least 3–5 runs, or justify why a single split is representative for these datasets.","section":"§4, Tables 1 and 2"},{"comment":"The augmentation analysis is only presented as bar values of AG in Figure 2, with no numerical table, no standard deviation, and no description of how the unlearning was run under each augmentation scenario (e.g., whether the same forget set and same random seed were used). The statement that 'Default+RA showed improvement for most analyses' is therefore not verifiable. Please provide the full quantitative results for all three augmentation settings and the associated variance.","section":"§4, Figure 2"}],"minor_comments":[{"comment":"Typo: 'RestNet-18' should be 'ResNet-18'. Also, sets are denoted inconsistently (D⟩, Di, Dr, Df); clean up notation.","section":"§3.2"},{"comment":"The metric list says AG uses 'MU' but this should be 'MIA'. Also the AG definition sentence is ambiguous and should be rewritten as a formula.","section":"§3.3"},{"comment":"The method name is inconsistently spelled 'Salun' and 'SalUn'; use one form.","section":"Throughout"},{"comment":"Figure 1 is in Portuguese while the body is in English; this may be acceptable for the intended venue, but the caption and labels should at least be understandable or translated.","section":"Figure 1"},{"comment":"Some recent unlearning works on medical data are not discussed; if none exist, the claim that this is the first evaluation should be stated more carefully and supported by a more systematic search.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The evaluation framework is simple but the main claim currently overreaches. The AG inconsistency in Table 2 is particularly concerning because the whole comparison rests on that metric. With corrected AG definitions, repeated runs, and a nuanced interpretation of the PathMNIST results, the paper could become a valid 'first look' at unlearning in medical imaging, but it is not ready in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things: this is the first test of SalUn, a state-of-the-art unlearning method, on medical image benchmarks, and the runtime numbers are genuinely useful. But the paper's central claim is not supported by its own tables. On PathMNIST, the test-accuracy gap is 10.28 points at a 10% forget rate and 7.93 points at 50%—that is not 'close,' and the text never qualifies it.\n\nWhat's new: the authors apply SalUn (ResNet-18) to PathMNIST, OrganAMNIST, and BloodMNIST at forget rates of 10% and 50%, and they add a data-augmentation comparison. That is a reasonable first step for an underexplored area, and the runtime advantage is real (SalUn takes 1–9 minutes versus 22–160 minutes for retraining). On BloodMNIST and OrganAMNIST, the TA gap is under 1.3 points, so for those two datasets 'close' is defensible. The augmentation analysis is thin but shows a plausible effect.\n\nThe soft spots are significant. First, the overstatement: PathMNIST is a 9-class medical image task, and an 8–10 point drop is not 'close to full retraining.' Second, the AG metric, which is supposed to summarise closeness, is computed inconsistently. Section 3.3 defines AG as the absolute value of the mean of differences, but Table 2's AG values (1.57, 0.70, 4.91) do not match the differences in the same table (recomputed: 1.19, 0.54, 4.23). That is a load-bearing error because the abstract and conclusion lean on AG. Third, there are no error bars or repeated runs; each number comes from a single random forget split, so we do not know whether the PathMNIST gap is stable. No code or data are released.\n\nThis paper is for researchers in machine unlearning and medical imaging who want a baseline for SalUn on MedMNIST. It is a short empirical note, not a transformative result. I would send it to peer review only after major revision: temper the abstract and conclusion, fix the AG computation, add repeated runs with standard deviations, and release code. Right now the inconsistencies make the central claim unreliable. Still, the question is worth answering, so it deserves a referee's time rather than a desk reject.","headline":"SalUn on MedMNIST: a useful first benchmark, but the 'close to retraining' claim collapses on PathMNIST and the AG metric in Table 2 is internally inconsistent.","tokens_in":6849,"tokens_out":4376,"would_cite":false,"duration_ms":42302,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that SalUn-based machine unlearning removes selected data from medical image classifiers with accuracy close to full retraining, at about a twentieth of the runtime.","keywords":["machine unlearning","medical image classification","SalUn","data augmentation","right to be forgotten","saliency","MedMNIST","ResNet-18"],"falsifier":"Run SalUn on the same three datasets with 20 independent random forget-set splits at the same 10% and 50% rates and record the distribution of Average GAP; if the spread across splits equals or exceeds the reported 0.5 to 4.9 point gap to retraining, the 'close to retraining' claim rests on an artifact of one random split rather than a stable effect.","tokens_in":6017,"feed_emoji":"🩺","tokens_out":5832,"duration_ms":64532,"temperature":0.7,"pith_summary":"The paper tries to establish that machine unlearning—selectively deleting the influence of specific training samples from an already-trained model—is viable for medical image classification, a setting where it had not been tested before. It applies SalUn to three medical imaging benchmarks, compares it with the gold standard of retraining from scratch, and reports that accuracy gaps are small while runtime drops by roughly a factor of twenty. It also claims that data augmentation, particularly adding random augmentation to the default crop-and-flip scheme, improves unlearning quality. If true, health-care deployments could answer right-to-be-forgotten requests by editing a deployed model instead of rebuilding it.","feed_headline":"Machine unlearning nearly matches retraining on medical images","feed_subtitle":"SalUn can honor right-to-be-forgotten requests on clinical classifiers at one-twentieth of retraining time.","key_machinery":"Saliency Unlearning (SalUn): a saliency-map-based unlearning procedure that identifies which model weights are most activated by the samples to be forgotten and selectively updates those weights so the model behaves as if it had never seen them. The paper applies it to a ResNet-18 trained for 200 epochs and runs unlearning for 10 epochs, evaluating with six metrics—UA, RA, TA, MIA, AG, and RTE—against the retrained model as the gold standard.","core_discovery":"The paper's central finding, on its own terms, is that Saliency Unlearning (SalUn) behaves nearly like full retraining on PathMNIST, OrganAMNIST, and BloodMNIST. Average GAP—the mean absolute difference across the UA, RA, TA, and MIA metrics relative to the retrained model—was 0.53 to 3.95 at a 10% forget rate and 0.70 to 4.91 at a 50% forget rate, while SalUn ran in 1.1 to 8.9 minutes versus 22.2 to 160 minutes for retraining. The authors interpret this as an efficient solution for medical applications, with harder forgetting at higher removal rates and on the more complex pathology dataset, and with data augmentation helping in most configurations.","pith_inferences":["I would expect the method's advantage to shrink on full-resolution clinical images: the 64x64 inputs used here are far smaller than typical pathology or MRI slices, so saliency localization may be easier; a test on higher-resolution data would bound the effect.","A natural extension is to combine SalUn with the label-smoothing and network-sparsity tricks the authors cite in related work, since those were shown to improve unlearning on general datasets and are orthogonal to saliency-based weight selection.","The comparison to retraining uses a single random split with no variance reporting; before clinical deployment, repeated random forget splits and confidence intervals would tell whether the near-zero average gap is stable or an artifact of one split.","If the near-retraining quality holds across splits, a practical certification pipeline could pair SalUn with membership-inference audits, letting hospitals demonstrate regulatory compliance without retraining."],"forward_implications":["A trained medical-image classifier could honor data-removal requests by running SalUn instead of retraining, cutting runtime from tens of minutes to a few minutes.","Unlearning quality degrades gracefully rather than catastrophically when the forget fraction goes from 10% to 50%; the worst average gap on PathMNIST stays below 5 percentage points.","Data augmentation, especially adding RandomAug to the default crop-and-flip, lowers the average gap in most configurations, so augmentation is a practical lever for better unlearning.","The same SalUn protocol transfers across three medical imaging modalities—histopathology, abdominal MRI, and blood microscopy—suggesting the method is not tied to one image type.","Because remaining accuracy and test accuracy stay near retrained levels, medical classifiers can forget selected data without sacrificing overall classification performance.","Higher forget rates and more complex datasets make unlearning harder, so deployment planning should budget for more unlearning epochs or additional augmentation when large fractions of data are removed."],"supporting_citations":[{"why":"Defines the SalUn method and supplies the training/unlearning protocol and default augmentation configuration that the paper follows.","marker":"[Fan et al. 2024]"},{"why":"Provides the three medical image benchmark datasets—PathMNIST, OrganAMNIST, and BloodMNIST—used in all experiments.","marker":"[Yang et al. 2021]"},{"why":"Supplies the ResNet-18 architecture on which both the original training and the retraining baseline are built.","marker":"[Wu et al. 2019]"},{"why":"Source of the data augmentation schemes (random crop, horizontal flip, and RandomAug) whose impact on unlearning quality is analyzed.","marker":"[Mumuni and Mumuni 2022]"},{"why":"Alongside Fan et al., supplies the evaluation metrics used, and provides the label-smoothing unlearning result the paper discusses in related work.","marker":"[Di et al. 2024]"},{"why":"Frames the right to be forgotten and data-protection regulation that motivates machine unlearning in the first place.","marker":"[Hoofnagle et al. 2019]"},{"why":"Connects the legal right to be forgotten to machine learning models, justifying the need for efficient data removal.","marker":"[Dang 2021]"},{"why":"Surveys machine unlearning methods and supports the paper's claim that unlearning has not been explored for medical image classification.","marker":"[Zhang et al. 2023]"}],"fun_headline_variants":["SalUn: near-perfect unlearning in under 9 minutes","Unlearning medical images: SalUn nearly as good as retraining","Forget at speed: SalUn matches retraining on medical datasets","Machine unlearning done right: SalUn rivals full retraining","Medical imaging unlearning: SalUn close to retraining, 20x faster"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The conclusion that SalUn is close to retraining assumes that the single random split of each dataset's training set into forget and keep sets—reported as point values with no repeated runs—is representative of unlearning quality for that dataset.","fun_headline_variants_meta":{"raw":{"variants":["SalUn: near-perfect unlearning in under 9 minutes","Unlearning medical images: SalUn nearly as good as retraining","Forget at speed: SalUn matches retraining on medical datasets","Machine unlearning done right: SalUn rivals full retraining","Medical imaging unlearning: SalUn close to retraining, 20x faster"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000441,"raw_usage":{"total_tokens":2018,"prompt_tokens":637,"completion_tokens":1381,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":381,"completion_tokens_details":{"reasoning_tokens":1298}},"tokens_in":381,"tokens_out":1381,"duration_ms":12710,"temperature":1.0,"reasoning_tokens":1298,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T16:27:02.606153+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SalUn on the same three datasets with 20 independent random forget-set splits at the same 10% and 50% rates and record the distribution of Average GAP; if the spread across splits equals or exceeds the reported 0.5 to 4.9 point gap to retraining, the 'close to retraining' claim rests on an artifact of one random split rather than a stable effect.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the SalUn method and supplies the training/unlearning protocol and default augmentation configuration that the paper follows."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the three medical image benchmark datasets—PathMNIST, OrganAMNIST, and BloodMNIST—used in all experiments."},{"cited_title":"and Mumuni, F","cited_arxiv_id":null,"evidence_quote":"Source of the data augmentation schemes (random crop, horizontal flip, and RandomAug) whose impact on unlearning quality is analyzed."},{"cited_title":"J., Van Der Sloot, B., and Borgesius, F","cited_arxiv_id":null,"evidence_quote":"Frames the right to be forgotten and data-protection regulation that motivates machine unlearning in the first place."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Connects the legal right to be forgotten to machine learning models, justifying the need for efficient data removal."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Surveys machine unlearning methods and supports the paper's claim that unlearning has not been explored for medical image classification."}],"review_version":1}