{"id":"c558de86-a3be-41d9-84c7-84b9f3fb879b","arxiv_id":"2412.08559","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Standard random-data LLM unlearning evaluations understate privacy leakage for minority data, as shown by canary and real rare-PII experiments across three datasets and two models.","lead":"LLM unlearning is usually tested on randomly picked data, which can hide how badly rare or minority data are forgotten. This paper injects rare phone numbers and legal years into language models and shows that these minority-style samples leak more private information after unlearning than average samples.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's 'at least 20% higher privacy leakage' is contradicted by the paper's own tables in several cells, and the single-seed design gives no variance estimates for the quantitative claim.","rationale":"The reader's weakest_assumption focuses on the generalization from synthetic canaries to real minority populations. That is a legitimate concern, partly mitigated by the paper's 'Minority' case, which uses real samples containing rare PII values rather than only artificial canaries. A more direct and verifiable weakness is the mismatch between the abstract's strong quantitative claim ('at least 20% higher') and the paper's own experimental tables, which contain multiple cells below 20% or with decreased |PL| magnitude. The single-seed design compounds this by providing no estimate of uncertainty, so even the cells that do exceed 20% cannot be assessed for statistical significance. I therefore see this overclaim as the most load-bearing issue for the paper's headline claim. The reader's rationale already mentions the overclaim and single seed, so my assessment partially agrees with theirs, though I would prioritize the numerical contradiction over the canary-generalization concern. The reader's CONDITIONAL verdict remains appropriate: the proposed minority-aware evaluation protocol is a useful contribution, but the abstract's quantitative claim should be softened or supported by error bars, and the construction of the minority set should be clarified.","tokens_in":38754,"tokens_out":8800,"duration_ms":93192,"concrete_test":"Tabulate all excess-ratio cells from Tables 2, 4, 5, 6, and 7. Count the fraction of Canary/Minority cells where the |PL| excess over Random is below 20% or negative. If that fraction is nontrivial (e.g., >10%), the abstract's unqualified 'at least 20%' claim is false as stated. Additionally, rerun the GPT-2 Enron-Phone experiment with at least 5 different seeds and report mean and standard deviation of the excess ratios; if the 95% confidence interval for the borderline cells (e.g., GA zlibMIA Minority) includes 20%, the quantitative claim lacks statistical support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim appears in the abstract as 'at least 20% higher privacy leakage across various unlearning methods, MIAs, datasets, and LLM scales.' However, the paper's own tables contain multiple cells where the excess ratio of |PL| for Canary or Minority versus Random is below 20% or even negative. For example, in Table 2 (GPT-2/Enron-Phone): GA zlibMIA Minority +7%, NegGrad+ zlibMIA Minority +13%, Langevin zlibMIA Minority +12%, GA Min-K% Minority +13%. Similar sub-20% cells appear in the ECHR tables (Table 5: EUk lossMIA Minority +10%, EUk zlibMIA Minority +4%). The main text hedges with 'in most cases' (Introduction) and 'on average' (Conclusion), but the abstract does not. In addition, App. B.2 fixes a single random seed (42) and no repeated seeds are reported, so there are no confidence intervals or significance tests; for the smaller excess values, the 20% threshold may be within noise. This matters because the 'at least 20%' bound is a headline contribution and is what makes the abstract's claim quantitatively specific. If the bound is not supported, the paper's main advertised result needs correction to 'most cases' or to a version with statistical error bars, even if the qualitative minority-awareness framework remains useful.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that standard LLM unlearning evaluations, which remove randomly chosen samples, systematically underestimate privacy leakage for minority or rare data. It proposes a minority-aware evaluation protocol that reports the maximum privacy leakage (PL) over three forget-set designs: Random, Canary (constructed by replacing PII with the least frequent value), and Minority (real samples with rare PII). Experiments on Enron and ECHR with GPT-2 and Llama-2, across six unlearning methods and three MIAs, find that Canary and Minority cases often exhibit larger PL magnitudes. The paper advertises this as \"at least 20% higher privacy leakage\" and further benchmarks unlearning methods under the proposed protocol, reporting that Langevin Unlearning offers a favorable privacy-utility trade-off compared with noiseless methods such as SCRUB and Gradient Ascent.","tokens_in":38999,"tokens_out":7195,"duration_ms":69221,"significance":"If the qualitative effect holds, this paper identifies a real blind spot in standard unlearning evaluation and offers a practical, low-cost fix: report worst-case results over several forget-set designs rather than only a randomly sampled one. The empirical scope is broad (six unlearning methods, three MIAs, three datasets, two model scales), the complexity budget is carefully defined, and the code is public. However, the headline \"at least 20%\" quantitative bound is not supported by the paper's own tables, and the single-seed experimental design leaves effect sizes without uncertainty estimates. The proposed framework is a useful contribution; the specific numerical claim needs correction and statistical grounding.","major_comments":[{"comment":"The abstract's claim that minority data \"suffer at least 20% higher privacy leakage across various unlearning methods, MIAs, datasets, and LLM scales\" is contradicted by the paper's own tables. For example, in Table 2 (GPT-2/Enron-Phone), the Minority excess ratios are +7% (GA, zlibMIA), +13% (NegGrad+, zlibMIA), +12% (Langevin, zlibMIA), and +13% (GA, Min-K%); in the Llama-2 panel, RL shows -65%/-77% for lossMIA and GA shows -27%/-17% for Min-K%. Table 5 (ECHR/GPT-2) contains EUk Minority +10% (lossMIA) and +4% (zlibMIA), and CFk Canary +13% (zlibMIA). The Introduction's \"in most cases\" and Conclusion's \"on average\" qualifiers are absent from the abstract. Please correct the abstract and any unqualified statements to match the actual evidence, or provide a statistical basis for a threshold claim.","section":"Abstract, §6.1, Table 2"},{"comment":"All experiments use a single random seed (seed 42), with no repeated runs or confidence intervals. Since several reported excess ratios lie between 4% and 20%, the \"at least 20%\" bound—and even the qualitative ordering in the smallest cells—may be within run-to-run noise. Please report multiple seeds (or bootstrap intervals over the forget-set draws) and significance tests for the key comparisons.","section":"App. B.2, §6.1"},{"comment":"The canary construction confounds rarity with repetition: all 100 canaries are created by replacing the area code with the single least frequent code (484), so the Canary condition tests one duplicated rare token rather than minority data generally. The Minority condition uses real emails with least-frequent area codes or years, which is more direct evidence, but App. A.3 is ambiguous about how Dminority is formed: Table 1 lists 484 with count 1, so a 100-sample minority set cannot all contain 484; the text alternates between \"the least frequent PII\" (singular) and \"the least frequent area codes\" (plural). Please clarify the exact construction of Dminority and report the Minority results as the primary evidence for the real-world claim, with the Canary results interpreted as a controlled audit of a single repeated rare token.","section":"§4.1, App. A.3, Fig. 3"},{"comment":"The forget-set size is stated as 1% of the training set, but App. A.3 says that 100 samples are used for the Random, Canary, and Minority sets. For Llama-2 the training set is 50,000 samples, so 1% would be 500, not 100. Please specify the actual sizes used and adjust the Complexity Unit budget and canary construction accordingly; if 100 was used throughout, the \"1%\" statement and the cross-scale comparison need revision.","section":"§6 (General Settings), App. A.3"}],"minor_comments":[{"comment":"The caption contains a typo: \"Canary & Minoirty settings\" should be \"Canary & Minority settings.\"","section":"Fig. 17 caption"},{"comment":"The sentence \"To rigorously show thatremoving data\" is missing a space; it should read \"that removing data.\"","section":"§4.1, page 4"},{"comment":"The caption uses \"GPT2\" while the rest of the paper uses \"GPT-2\"; please harmonize the notation.","section":"Table 3 caption"},{"comment":"The preprocessing step \"We randomly selected three coherent sentences from each sample\" should specify whether it applies to both datasets and whether it affects the context surrounding the PIIs of interest.","section":"App. A.3"},{"comment":"The treatment of \"year of judgment\" in ECHR as a PII would benefit from a brief justification, since a year alone is not personally identifying; this choice affects how the minority definition is interpreted.","section":"§6, App. A.2"}],"recommendation":"major_revision","confidential_remarks":"The paper's qualitative finding is plausible and the proposed minority-aware evaluation framework is a useful contribution to the unlearning evaluation literature. However, the headline \"at least 20%\" claim is not supported by the paper's own tables and needs to be corrected or qualified, and the single-seed design leaves the effect sizes without variance estimates. The canary confound and the forget-set size inconsistency should be resolved in revision. No concerns about novelty or citation practice."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is sound and worth publishing: rare or minority-associated data are harder to unlearn, and the standard random-forget-set evaluation misses that. The canary protocol, which swaps only the PII (e.g., area code 484) while keeping the rest of the email identical, is a clean way to isolate the effect of rarity. The breadth is impressive too: six unlearning methods, three MIAs, three datasets, and two model scales. The proposed minority-aware evaluation, reporting max PL over Random/Canary/Minority, is a practical step forward. The finding that Langevin Unlearning offers a stable privacy-utility trade-off is also useful.\n\nBut the headline quantitative claim does not survive contact with the paper's own tables. The abstract says \"at least 20% higher privacy leakage across various...\"; the introduction says \"in most cases\" and the conclusion says \"on average.\" The tables contain multiple cells below 20%, and several where minority/canary leakage is actually lower than random (e.g., Table 2: GA zlibMIA Minority +7%; Table 5: EUk lossMIA Minority +10%). This is not a fatal flaw, because the qualitative direction holds in the clear majority of cells, but the abstract overclaims and should be softened to match the evidence.\n\nThe single-seed design (App. B.2, seed 42) is a bigger problem for the quantitative claim. Without repeated seeds or confidence intervals, the smaller excess ratios could easily be noise. The \"at least 20%\" bound in particular needs error bars or a rephrasing.\n\nThe generalization from synthetic canaries to \"minority populations\" is a stretch. Replacing an area code with a rare one creates an artificial cluster; the real minority set (Case 3) helps, but it is still defined by rare phone area codes, not by demographic attributes. The title overpromises a bit relative to what is measured.\n\nThere is also an internal inconsistency about forget-set size. Section 6 says \"1% of the training set size,\" but App. A.3 says 100 samples are selected for the forget set, which is 1% for GPT-2 (10k) but only 0.2% for Llama-2 (50k). This needs to be reconciled.\n\nOverall: send it to review. The framework is a genuine contribution and the qualitative finding is credible. The authors need to fix the abstract, add variance estimates, and clarify the forget-set size. A serious referee can get this into shape.","headline":"Useful minority-aware evaluation framework for LLM unlearning, but the 'at least 20%' headline is not backed by the paper's own tables and the single-seed design leaves effect sizes uncertain.","tokens_in":39603,"tokens_out":3022,"would_cite":true,"duration_ms":29814,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Standard LLM unlearning evaluations understate privacy leakage for minority data by at least 20%, so the paper proposes reporting worst-case results across random, canary, and minority forget sets.","keywords":["machine unlearning","privacy leakage","membership inference attacks","minority populations","canary injection","large language models","long-tail memorization","unlearning evaluation"],"falsifier":"Repeat the evaluation on real minority data with ground-truth membership (e.g., Enron emails from the least-represented senders or ECHR facts mentioning rare years) and compute PrivLeak for minority versus random forget sets; if the gap is consistently below 20% or reverses, the central claim fails. Also, run the canary construction with several different rare tokens (not just 484) and multiple seeds; if the 20% gap disappears for some tokens, the effect is token-specific rather than a general property of rarity.","tokens_in":38549,"feed_emoji":"🔒","tokens_out":6265,"duration_ms":56788,"temperature":0.7,"pith_summary":"The paper argues that the standard pipeline for evaluating LLM unlearning—randomly choosing data to forget and averaging membership-inference leakage—systematically understates privacy risk for data tied to minority groups. To show this, the authors inject canaries whose personal identifiers (e.g., phone area codes) are replaced with the rarest value in the dataset, and they also directly remove the rarest real samples. Across six unlearning methods, three membership-inference attacks, three datasets, and two model scales, privacy leakage for these minority cases is at least 20% higher than for random removal. The paper proposes that unlearning evaluations should report the worst case across random, canary, and minority forget sets, together with worst-case utility. If true, current benchmarks give a false sense of equality in the 'right to be forgotten'.","feed_headline":"Minority data leak 20% more after LLM unlearning","feed_subtitle":"Standard audits delete random samples; rare-identifier records stay more recognizable.","key_machinery":"The central object is the minority-aware evaluation protocol built from three forget-set constructions: Random (uniform sampling, the status quo), Canary (identical emails with the identifier replaced by the least frequent value, e.g., area code 484), and Minority (the actual least-frequent real samples). The carrying identity is the PrivLeak ratio, $\\text{PL} = \\Delta\\text{AUC} / \\text{AUC}(M_{\\text{retrain}})$, which measures how much more easily a membership-inference attack distinguishes the unlearned model from a retrained model; the protocol reports the maximum PL across the three cases and the corresponding worst-case perplexity. The canary construction isolates the effect of identifier rarity by holding all other text constant, so any excess leakage is attributable to the minority attribute.","core_discovery":"On the paper's own terms, the central discovery is that unlearning efficacy is not uniform across data points: records with rare, shared attributes—exemplified by the least frequent phone area code—retain measurably more membership signal after unlearning than randomly selected records. The PrivLeak metric, the normalized gap in MIA AUC between the unlearned model and a retrained model, is consistently at least 20% larger for canary and minority forget sets than for random forget sets, and often far larger (e.g., 300–800% on Llama-2). This holds across gradient ascent, NegGrad+, SCRUB, random labels, layer-wise retraining, and Langevin unlearning, and across lossMIA, zlibMIA, and Min-K% attackers. The paper concludes that average-case evaluation misses high-risk partitions of the training data and offers a worst-case, minority-aware protocol as a corrective.","pith_inferences":["If real minority populations behave like the injected 484 canaries, then the 'right to be forgotten' is currently met unequally: members of groups with rare identifiers are less likely to be effectively forgotten than members of majority groups, which bears directly on GDPR compliance audits.","The effect's dependence on rarity level is untested: varying the frequency of the injected token (e.g., 2, 5, 100 occurrences) would reveal whether leakage grows monotonically with rarity or is an artifact of the single 484 token.","Because all experiments use one random seed (42) and one canary value per dataset, the reported effect sizes lack variance estimates; multi-seed and multi-token replications would strengthen the quantitative claim.","A straightforward extension is to audit unlearning fairness: use the max-PL protocol to measure disparate privacy risk across subgroups defined by income, geography, or dialect, and to set per-group unlearning budgets accordingly."],"forward_implications":["Unlearning benchmarks that report only average-case PrivLeak understate privacy leakage for minority data by at least 20%, making published method comparisons over-optimistic for these groups.","Evaluation suites should add canary and minority forget sets and report the maximum privacy leakage across all three, plus worst-case utility, so a method that fails on any one setting is exposed.","Noise-based unlearning (Langevin Unlearning) yields the best privacy-utility trade-off and is stable, whereas gradient ascent and random labels are unstable and can destroy model utility.","The protocol applies to any under-represented subset defined by a shared value, not only PII, so privacy audits can be stratified by demographic or other sensitive attributes."],"supporting_citations":[{"why":"Supplies the premise that rare/long-tail samples are memorized more strongly, which motivates why minorities should be harder to unlearn.","marker":"Feldman & Zhang, 2020"},{"why":"Provides membership-inference methodology and evidence of non-uniform memorization; used to justify per-sample MIA and outlier memorization.","marker":"Carlini et al., 2022"},{"why":"Introduces canary injection for privacy auditing, the template for the paper's synthetic minority canaries.","marker":"Jagielski et al., 2020"},{"why":"Extends privacy auditing with canaries, supporting the one-training-run auditing approach adopted here.","marker":"Steinke et al., 2024"},{"why":"Defines the PrivLeak metric and the MIA-based unlearning evaluation setup that the paper modifies.","marker":"Shi et al., 2024b"},{"why":"Provides the NegGrad+ and SCRUB unlearning methods evaluated under the minority-aware protocol.","marker":"Kurmanji et al., 2024"},{"why":"Supplies Langevin Unlearning, the noise-based method that the paper finds best in privacy-utility trade-off.","marker":"Chien et al., 2024a"},{"why":"Highlights limitations of average-case unlearning evaluation, motivating the worst-case analysis.","marker":"Hayes et al., 2024"}],"fun_headline_variants":["Minority data stays at risk after LLM unlearning","Unlearning worst-case: minority records leak more","Unlearning audits miss high-risk minority data","LLM unlearning: rare data 20% more exposed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The synthetic canary—an email whose phone area code is replaced by the single rarest code (484)—is a faithful stand-in for real minority population data, so the observed higher leakage reflects a property of minority data rather than an artifact of one artificially repeated rare token.","fun_headline_variants_meta":{"raw":{"variants":["Minority data stays at risk after LLM unlearning","Unlearning worst-case: minority records leak more","Unlearning audits miss high-risk minority data","LLM unlearning: rare data 20% more exposed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000311,"raw_usage":{"total_tokens":1774,"prompt_tokens":947,"completion_tokens":827,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":764}},"tokens_in":563,"tokens_out":827,"duration_ms":8647,"temperature":1.0,"reasoning_tokens":764,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:44:46.899580+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the evaluation on real minority data with ground-truth membership (e.g., Enron emails from the least-represented senders or ECHR facts mentioning rare years) and compute PrivLeak for minority versus random forget sets; if the gap is consistently below 20% or reverses, the central claim fails. Also, run the canary construction with several different rare tokens (not just 484) and multiple seeds; if the 20% gap disappears for some tokens, the effect is token-specific rather than a general property of rarity.","supporting_citations":[{"cited_title":"and Zhang, C","cited_arxiv_id":null,"evidence_quote":"Supplies the premise that rare/long-tail samples are memorized more strongly, which motivates why minorities should be harder to unlearn."},{"cited_title":"Auditing differentially private machine learning: How private is private sgd? Advances in Neural Information Processing Systems, 33: 0 22205--22216, 2020","cited_arxiv_id":null,"evidence_quote":"Introduces canary injection for privacy auditing, the template for the paper's synthetic minority canaries."},{"cited_title":"Privacy auditing with one (1) training run","cited_arxiv_id":null,"evidence_quote":"Extends privacy auditing with canaries, supporting the one-training-run auditing approach adopted here."},{"cited_title":"Towards unbounded machine unlearning","cited_arxiv_id":null,"evidence_quote":"Provides the NegGrad+ and SCRUB unlearning methods evaluated under the minority-aware protocol."}],"review_version":1}