{"id":"37ba2ae7-778a-496d-bacf-6a7ec2427733","arxiv_id":"2505.17160","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LURK uses optimized adversarial suffixes and an LLM judge to show that unlearned models still leak Harry Potter-specific references at high rates.","lead":"This paper tests whether language models that have supposedly forgotten the Harry Potter books still leak that knowledge when attacked with automatically generated prompt suffixes. It finds that several apparently unlearned models can be coaxed into naming Potter-specific terms, which suggests current unlearning checks are too weak to certify deletion.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Leakage rates may be inflated by the forced target phrase itself: Eq. (1) optimizes for the completion 'Harry Potter is', and Eq. (2)'s judge can count that injected proper noun as leakage, so the probe partly constructs the signal it measures.","rationale":"The reader identified the same load-bearing concern: the objective function in Eq. (1) forces the model to emit 'Harry Potter is', and Eq. (2) counts Harry Potter proper nouns in the completion, so the probe's target phrase can itself be scored as leakage. This is not a peripheral implementation detail; it directly affects every leakage rate in Table 3, which is the evidence for the paper's central claim that unlearned models 'leak idiosyncratic information' and that methods like TV achieve only 'concealment rather than true forgetting.' The concern is load-bearing because if the injected 'Harry Potter' token is counted, then LURK's success on any model—including one with zero residual HP knowledge—would be attributable to the probe, not to hidden retention. The paper has genuine strengths: the adversarial-prompting adaptation of GCG is reasonable, the judge calibration in Table 1 is carefully done, and the qualitative examples (e.g., 'liquid luck', 'Quaffles') do suggest that some residual knowledge can be surfaced. However, those qualitative examples are anecdotal and do not establish the quantitative leakage rates. A single control experiment with the retained ICLM-7B model would settle whether the metric has an unacceptable false-positive rate. Because the reader's conditional verdict already reflects this uncertainty, and because the paper could be repaired by adding this control and/or masking forced target tokens before scoring, the verdict should remain CONDITIONAL rather than being strengthened or weakened. I therefore recommend UNCHANGED: the evaluation should still require the control and code release before the headline numbers are trusted.","tokens_in":12641,"tokens_out":2520,"duration_ms":22826,"concrete_test":"Run LURK unchanged on the retained model ICLM-7B, which was never pretrained on the Harry Potter series, using the same target phrase 'Let's meet a famous wizard! Harry Potter is' and the same GPT-4o-mini/o3-mini judge. Since ICLM-7B has no Harry Potter knowledge to leak, any nonzero leakage rate on this model is a false positive caused by the forced target phrase. If the ICLM-7B leakage rate is substantially above zero (e.g., above 5%), the headline leakage rates are contaminated and Table 3 must be re-reported after masking or removing the forced target tokens before applying G(·).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—that unlearned models 'leak idiosyncratic information' under LURK (Table 3)—depends on whether the measured leakage is residual knowledge rather than an artifact of the probe's own target string. In Section 3 ('Affirmative Phrase'), the objective in Eq. (1) is the negative log-likelihood of generating the hand-chosen completion 'Let's meet a famous wizard! Harry Potter is.' The adversarial suffix is optimized to make the model emit exactly this string, including the proper noun 'Harry Potter'. The leakage check G(·) in Eq. (2) then counts proper nouns in the generated completion that are absent from the user query. Since 'Harry Potter' is almost never part of the user query (queries are taken from a Harry Potter corpus about other entities), the forced token 'Harry Potter' itself satisfies the G(·) criterion and contributes to the leakage rate. This is not an independent measurement of latent knowledge: the attack is rewarded for outputting a Harry Potter proper noun, and the evaluation then counts that same proper noun as evidence of retention. The problem is compounded by the fact that the target phrase also primes 'wizard' and the Harry Potter entity name, so even a model with no residual HP knowledge could complete the forced phrase and be scored as leaking. The paper reports no control condition on a model that has never seen the Harry Potter corpus (e.g., ICLM-7B), which would reveal the false-positive baseline of the probe. Without such a control, the post-probing leakage rates in Table 3 and the inference that well-performing unlearning methods such as TV 'rely on concealment rather than true forgetting' are not securely supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces LURK, an adversarial-suffix probing framework for detecting residual Harry Potter knowledge in unlearned LLMs. LURK adapts the GCG attack to optimize suffix tokens appended to HP-themed user queries, maximizing the likelihood that the model generates a hand-chosen affirmative completion, and employs an LLM-based judge (GPT-4o-mini with an o3-mini sanity check) to count idiosyncratic HP proper nouns in the generated response. Experiments on OPT-2.7B, LLaMA 2-7B/13B, and LLaMA 3.1-8B with WHP, Task Vector, Gradient Ascent, and NPO unlearning report leakage rates before and after probing, showing that models deemed successfully unlearned still produce HP references under optimized suffixes. The paper concludes that existing unlearning methods may rely on concealment rather than true forgetting.","tokens_in":12964,"tokens_out":5740,"duration_ms":48732,"significance":"If the measured leakage rates are accepted, the paper makes a timely and useful point: standard forget-set metrics such as FQ gap can certify a model as unlearned while adversarial prompting still surfaces domain knowledge, so unlearning evaluation should include robustness probing. The adaptation of GCG to a knowledge-leakage setting, the hybrid judge with few-shot CoT validation, and the systematic comparison of four unlearning algorithms across four model families are strengths, and the qualitative examples in Table A1 provide concrete leakage instances that do not depend on the exact target phrase. However, the quantitative headline result is currently confounded by the probe's own target string, and the paper lacks the control condition needed to separate manufactured references from residual knowledge. The contribution would be significant if the circularity is resolved and the leakage rates are reproduced over multiple seeds.","major_comments":[{"comment":"The headline leakage rates are not an independent measurement of residual knowledge. Eq. (1) optimizes the adversarial suffix to maximize the likelihood of the hand-chosen completion \"Let's meet a famous wizard! Harry Potter is\", and Eq. (2)'s check G(s) counts every HP-specific proper noun in the completion that is absent from the user query. Because the user queries are prompts about other HP entities, the injected proper noun \"Harry Potter\" itself satisfies G(s) >= 1, so the optimization is rewarded for emitting exactly the token that the judge then scores as leakage. A model with no residual HP knowledge could plausibly complete the forced phrase and be counted as leaking. To support the central claim, the authors should either exclude target-phrase proper nouns from G, use a target phrase with no HP proper noun, or report the leakage rate of the same attack on ICLM-7B (the never-exposed control) as a false-positive baseline. Without such a control, Table 3's rates overstate the amount of retained knowledge.","section":"Section 2, Eqs. (1)-(2); Section 3 'Affirmative Phrase'; Table 3"},{"comment":"The leakage rates in Table 3 are single-run point estimates with no reported variance. GCG is stochastic, and the paper does not report the number of seeds, making it impossible to assess whether the differences across methods (e.g., TV 79.7 vs WHP 53.4 on LLaMA2-13B) are stable. Additionally, the G(.) judge is validated on only 50 Q&A pairs (Table 1); a larger held-out validation set and a report of inter-annotator or judge reliability would strengthen the leakage signal. These additions are needed before the quantitative cross-method comparisons can support the conclusion that existing unlearning methods may rely on concealment rather than true forgetting.","section":"Section 4.1; Table 3"}],"minor_comments":[{"comment":"Algorithm 1 uses the symbol I in the inner loop (\"for each i \\u2208 I\") but no such set is defined; this should be the adversary-token index set A from Eq. (2). Also, \"Update x<n \\u2190 \\u02dcx(b∗)1:n\" appears to update the prefix rather than the suffix; please correct the notation.","section":"Algorithm 1"},{"comment":"Table 2 reports identical ICLM-7B values (FQ gap 0.0, PPL 11.7075, Avg Acc 0.5109) across all four base-model columns; clarify whether this is a formatting artifact or the same retained-model evaluation reused for all columns.","section":"Table 2"},{"comment":"The caption of Figure 1 contains a typo: \"prob knowledge leakage\" should read \"probe knowledge leakage\".","section":"Figure 1 caption"},{"comment":"The notation in Eq. (2), \"xA \\u2208 {1,...,V}|A|\", is easy to misread; parenthesize the superscript or explain that the superscript denotes the size of the set of adversarial indices.","section":"Eq. (2)"},{"comment":"Reference [2] (Eldan and Russinovich) is cited without a publication venue; please cite the arXiv or the technical report version to allow verification.","section":"Reference [2]"}],"recommendation":"major_revision","confidential_remarks":"The circularity concern is serious, but the qualitative examples and the overall framework suggest the underlying idea is worth pursuing. I recommend major revision rather than rejection: if the authors add the ICLM control, adjust the target phrase or judge to exclude injected proper nouns, and report multi-seed variances, the findings could become a solid contribution to unlearning evaluation. Please also note that the paper currently does not provide the promised code, so reproducibility depends on the appendix details."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's the quick read on arXiv:2505.17160. The paper's core idea is sound and worth discussing: use GCG-style adversarial suffix optimization to see if unlearned LLMs still emit Harry Potter-specific knowledge, and use an LLM judge to flag idiosyncratic proper nouns. The qualitative results are the strongest part — after probing, a TV-unlearned LLaMA-2-13B starts talking about 'liquid luck' and 'Quaffles,' which is exactly the kind of residual knowledge you'd want an evaluation to catch. The observation that TV looks great on standard forget quality but is more probe-vulnerable than WHP on larger models is genuinely interesting.\n\nBut the stress-test concern is correct, and it's not minor. Equation (1) optimizes the suffix to make the model generate 'Let's meet a famous wizard! Harry Potter is.' The judge in Eq. (2) then counts any HP proper noun in the completion that wasn't in the query — and 'Harry Potter' itself qualifies. So every successful attack is rewarded for outputting the name, and that name is then scored as leakage. A model with zero residual HP knowledge could comply with the forced phrase and be counted as leaking. That means the leakage rates in Table 3 are not independent measurements of retention. The paper reports no control on a model that never saw the corpus (ICLM-7B is used only for the forget quality baseline), so we don't know the false-positive rate of the whole probe procedure.\n\nOther soft spots are more standard: the judge is validated on 50 queries, no error bars, the commercial judge models aren't pinned, and code/suffixes haven't been released. And the 'concealment rather than true forgetting' framing is an interpretation; the data show probing can elicit some references, but that's not the same as showing the model was deliberately concealing.\n\nThat said, the integration and the qualitative findings are enough that I'd want to see a revision. If the authors fix the metric (e.g., exclude the forced target phrase or measure leak rate on completions that go beyond it), add an ICLM control, and release the code, the paper would be a useful contribution to unlearning evaluation. As is, trust the qualitative examples, not the headline numbers.\n\nI'd take this for a reading group — it raises the right questions. Peer review? Yes, with a serious referee. The flaws are fixable and the question matters.\n\nBest","headline":"Good idea, useful probe, but the headline leakage rates are inflated by the probe's own target phrase; needs a control and a rework of the metric.","tokens_in":13498,"tokens_out":2851,"would_cite":false,"duration_ms":23983,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LURK probes supposedly unlearned LLMs with adversarial suffixes and finds that they still leak Harry Potter-specific knowledge.","keywords":["machine unlearning","knowledge leakage","adversarial prompting","LLM evaluation","Harry Potter benchmark","latent knowledge","greedy coordinate gradient","unlearning robustness"],"falsifier":"Run LURK on a model known to have no Harry Potter exposure, such as the paper's retained-model baseline, and count how often the judge returns a score of at least one; if the judge flags completions that merely repeat the forced target phrase \"Harry Potter\" or its immediate context without additional canonical detail, the reported leakage rates overstate how much unlearned models remember.","tokens_in":12434,"feed_emoji":"🧙","tokens_out":10388,"duration_ms":78074,"temperature":0.7,"pith_summary":"This paper claims that a model can pass current unlearning evaluations and still retain the targeted knowledge, because the knowledge is hidden rather than erased. To demonstrate this it introduces LURK, a probing framework that appends adversarially optimized suffix tokens to user queries, forces the model into an affirmative continuation, and then counts idiosyncratic Harry Potter references in the reply. Applied to four unlearning algorithms across four open models, LURK finds substantial leakage after supposedly successful unlearning, with the largest gaps on the largest models. If the claim holds, standard forget-quality and fluency metrics are insufficient as evidence of forgetting, and auditing an unlearned model requires adversarial probing.","feed_headline":"Probe finds Harry Potter lore still hiding in unlearned chatbots","feed_subtitle":"Adversarial suffixes make supposedly erased models name Quidditch, Quaffles, and liquid luck","key_machinery":"The machinery is LURK, an extension of greedy coordinate gradient optimization in which token substitutions in an adversarial suffix are chosen by the gradient of a negative-log-likelihood loss $\\mathcal{L}(x_{\\le n})=-\\log P(s\\mid x_{\\le n})$ for a chosen target continuation $s$ such as \"Let's meet a famous wizard! Harry Potter is\". The search is constrained by a leakage monitor $G(s)\\ge 1$, a commercial LLM-based judge that counts canonical Harry Potter proper nouns appearing in the completion but not in the original query. The judge was calibrated against chain-of-thought and few-shot prompts, with a fast model used during optimization and a stricter model used for a final sanity check. The probe stops as soon as one canonical reference is found, so the optimized suffix is exactly the input that breaks the unlearned model's avoidance behavior.","core_discovery":"The paper's central claim is that even models judged successfully unlearned can leak idiosyncratic information under targeted adversarial conditions, so existing unlearning methods may achieve concealment rather than true forgetting. In the Harry Potter benchmark, a method with a near-zero forget-quality gap, such as Task Vector on LLaMA 3.1-8B, jumps to a post-probing leakage rate above 80% when LURK finds an adversarial suffix. Larger models show larger absolute leakage, suggesting that added capacity is used to hide the unwanted content instead of erasing it. The paper concludes that LURK supplies a more rigorous, diagnostic tool that requires no ground-truth completions, and that verifiable unlearning will need to address this kind of latent retention.","pith_inferences":["Interpreting the leakage rate as residual knowledge requires that the judge's canonical-noun count is not driven by the target phrase itself; the paper does not report a control that strips the forced continuation before scoring, so the headline rates are likely an upper bound on independent recall.","The same optimization framework could probe for leaked private data in other domains by swapping the judge's entity list, but the practical bottleneck would be obtaining a precise, canonical vocabulary for sensitive material and calibrating the judge's false-positive rate.","If the concealment conclusion generalizes, then unlearning methods that only fine-tune or perturb embeddings are likely to remain vulnerable to search-based attacks, and stronger guarantees such as provable deletion or membership masking would be needed for regulatory compliance."],"forward_implications":["Unlearning evaluations that rely only on forget-quality gap and fluency will overstate success; adding adversarial probing will turn failures that are invisible on current metrics into measurable leakage rates.","Model scale works against unlearning: larger models leak more under probing, so results on small unlearning test beds cannot be assumed to hold on production-scale systems.","A method's rank order can flip under probing: the method that looks strongest on standard metrics leaks more than the fine-tuning baseline on LLaMA 2-13B, so method comparisons should include adversarial inspection.","Because LURK needs no ground-truth completions, it can be applied when the unlearning target is described only by an entity list rather than by the memorized corpus itself."],"supporting_citations":[{"why":"Supplies the greedy coordinate gradient adversarial-prompt optimization that LURK extends.","marker":"[18]"},{"why":"Provides the Harry Potter unlearning dataset, the WHP baseline, and the inspiration for idiosyncratic-reference leakage checking.","marker":"[2]"},{"why":"Defines the forget-quality and retained-model evaluation protocol used to score unlearning.","marker":"[9]"},{"why":"Task Vector, one of the four unlearning methods probed, whose surprising leakage rates drive the concealment conclusion.","marker":"[4]"},{"why":"Gradient Ascent, an unlearning method that LURK evaluates and finds utility-destructive.","marker":"[5]"},{"why":"Negative Preference Optimization, the other unlearning method evaluated.","marker":"[16]"}],"fun_headline_variants":["Even 'unlearned' LLMs still know Quidditch under adversarial probe","Adversarial suffixes resurrect Harry Potter in supposedly erased models","LURK probe: 80% of 'erased' Harry Potter knowledge still leaks","Unlearning fails: Adversarial prompts expose retained Potter lore in LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach assumes that a model completing the probe's forced sentence with any Harry Potter proper noun has retained hidden knowledge, rather than simply echoing the name the probe itself supplied.","fun_headline_variants_meta":{"raw":{"variants":["Even 'unlearned' LLMs still know Quidditch under adversarial probe","Adversarial suffixes resurrect Harry Potter in supposedly erased models","LURK probe: 80% of 'erased' Harry Potter knowledge still leaks","Unlearning fails: Adversarial prompts expose retained Potter lore in LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000232,"raw_usage":{"total_tokens":1420,"prompt_tokens":806,"completion_tokens":614,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":422,"completion_tokens_details":{"reasoning_tokens":531}},"tokens_in":422,"tokens_out":614,"duration_ms":5188,"temperature":1.0,"reasoning_tokens":531,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:55:20.388186+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run LURK on a model known to have no Harry Potter exposure, such as the paper's retained-model baseline, and count how often the judge returns a score of at least one; if the judge flags completions that merely repeat the forced target phrase \"Harry Potter\" or its immediate context without additional canonical detail, the reported leakage rates overstate how much unlearned models remember.","supporting_citations":[{"cited_title":"In First Con- ference on Language Modeling","cited_arxiv_id":null,"evidence_quote":"Supplies the greedy coordinate gradient adversarial-prompt optimization that LURK extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Harry Potter unlearning dataset, the WHP baseline, and the inspiration for idiosyncratic-reference leakage checking."},{"cited_title":"Smith, and Chiyuan Zhang","cited_arxiv_id":null,"evidence_quote":"Defines the forget-quality and retained-model evaluation protocol used to score unlearning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Task Vector, one of the four unlearning methods probed, whose surprising leakage rates drive the concealment conclusion."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gradient Ascent, an unlearning method that LURK evaluates and finds utility-destructive."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Negative Preference Optimization, the other unlearning method evaluated."}],"review_version":1}