{"id":"38152fbb-c144-4d47-9f47-045128425906","arxiv_id":"2509.02820","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"JensUn applies Jensen-Shannon Divergence to both forget and retain losses in LLM unlearning, and a new worst-case paraphrased evaluation shows it preserves utility better than baselines while resisting benign relearning.","lead":"A new unlearning method for large language models uses Jensen-Shannon Divergence to drive forgotten facts toward a refusal response, and it introduces a stricter evaluation protocol using an AI judge and paraphrased questions. The authors report the method forgets target facts while preserving the model's general abilities better than existing approaches, but the claim of 'almost irreversible' forgetting is only partially supported by their own relearning tests.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Finite paraphrase/query coverage cannot support 'almost irreversible forgetting'; the method learns refusal for specific prompts, not erasure, so the permanence claim needs an out-of-distribution query test.","rationale":"The reader's weakest-assumption analysis identifies exactly the load-bearing gap: the paper defines unlearning success via a finite paraphrase set, then uses that definition to support a strong permanence claim. My stress-test converges on the same point. The method only reshapes the output distribution toward a refusal string on the training-time forget prompts; it does not demonstrate that the factual association is absent from the model's parameters. The benign-relearning experiment is informative and well-designed, but it uses a small, disjoint real-knowledge set and still shows 8% recovery after 2000 unlearning steps, so it cannot establish 'almost irreversible' forgetting in any absolute sense. The proposed concrete test would directly measure whether the result generalizes beyond the specific paraphrase generator and query formats used in the paper. Because this concern was already flagged by the reader and the verdict is already CONDITIONAL, I do not recommend changing the verdict. The paper's contribution as an improved loss and evaluation protocol remains valuable, but the permanence headline should be conditioned on the tested query distribution and on the closed API judge's behavior.","tokens_in":37216,"tokens_out":5942,"duration_ms":74054,"concrete_test":"Using the released LKF dataset and the 2000-step JensUn checkpoint, build a held-out adversarial evaluation: for each of the 100 forget facts, generate 30 new paraphrases with GPT-4o (not used in training), 10 translations into German/French/Spanish, 10 cloze completions, and 10 four-option MCQs. Score all with the same Gemini-2.5-Flash judge used in the paper. If held-out worst-case accuracy is near 0%, the finite-paraphrase concern is resolved; if it exceeds ~10% for JensUn, the 'almost irreversible forgetting' claim fails and the conclusion should be restricted to the tested query distribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central permanence claim rests on the worst-case metric J_W defined in Sec. 4.2: a fact is considered forgotten only if the LLM-Judge fails all 15 paraphrases and in-context-retain variants. This set is finite, English-only, and generated by three small LLMs (Mistral-7B, Qwen2.5-3B, Phi-3.5). The training objective Eq. (4)+(6) teaches the model to emit 'No idea' for specific forget prompts and their training paraphrases; it does not remove the underlying factual association from the weights. The paper's own Table 3 shows benign relearning still recovers 8% J_W after 2000 unlearning steps, so the data already undercut a literal 'irreversible' reading. The unsupported leap is from 'not retrievable under our finite query set' to 'almost irreversible forgetting': a different paraphrase, a non-English question, a cloze completion, or a multiple-choice framing could elicit the fact. Since all methods are evaluated with the same J_W, the relative forget-utility ranking may survive, but the absolute 0% and the permanence claim do not follow from the evidence presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes JensUn, an LLM unlearning method that replaces the common log-likelihood/DPO-style forget and retain losses with Jensen-Shannon Divergence terms (Eqs. 4-6), arguing that boundedness and well-behaved gradients improve the forget-utility trade-off and stability during longer unlearning. It also introduces LKF, a new 100-forget/400-retain QA dataset of lesser-known historical facts, and proposes an evaluation protocol based on an LLM semantic judge plus worst-case accuracy over 15 paraphrases and in-context retain samples (JW). Experiments on LKF (Llama-3.2-3B, Phi-3) and on RWKU compare JensUn with GradAscent, GradDiff, NPO, SimNPO, RMU, DPO, RT, ICU, and NPO+SAM, reporting that JensUn achieves 0% JW on LKF while preserving utility, and lower forget accuracy on RWKU than competitors. The paper further studies benign relearning and reports that JensUn retains higher WR and lower relearned JW than NPO/NPO+SAM at longer unlearning steps.","tokens_in":37514,"tokens_out":5381,"duration_ms":67647,"significance":"If the results hold, the paper makes three useful contributions: a simple, precisely specified unlearning loss that is competitive with or better than established baselines; a realistic QA-style unlearning dataset (LKF); and a stronger evaluation protocol that replaces ROUGE with a human-aligned LLM judge and uses worst-case paraphrases. The authors publicly release code and dataset, perform a human study for the judge, and provide a gradient analysis of JSD versus KL. These are concrete strengths. However, the central rhetorical claim of 'almost irreversible forgetting' is not supported by the paper's own relearning numbers, and the evaluation, while more robust than standard benchmarks, remains a finite set of English paraphrases. The method and the relative rankings may survive such criticism, but the permanence claim needs substantial reframing or additional out-of-distribution evidence.","major_comments":[{"comment":"The 'almost irreversible' framing is contradicted by the paper's own data. In Table 3, JensUn's relearned JW is 27% at 200 unlearning steps, 24% at 400, 19% at 600, 14% at 1000, and 8% at 2000; in Appendix D.5, retain-set relearning of the 2000-step model gives JW=18%, a value the text describes as 'only slight'. These are non-negligible recovery rates under a single benign relearning curriculum. The claims in the abstract and §5.3 that the information has been 'truly removed' and that unlearning is 'almost irreversible' are therefore unsupported. The authors should either define an explicit irreversibility criterion with an attack/relearning budget and demonstrate it, or reframe the claim as 'resistant to the tested benign relearning protocols'.","section":"§5.3, Table 3; Appendix D.5, Table 13; title/abstract"},{"comment":"The worst-case metric JW uses 15 English paraphrases generated by three small LLMs plus in-context retain samples. This is a finite query family, and Eq. (4) explicitly trains the model to emit a refusal string ('No idea') for the forget prompts and their training paraphrases. Consequently, 0% JW is consistent with a model that has learned a refusal behavior for those specific phrasings rather than having removed the underlying factual association. The paper's own Table 6 shows that adding fill-in-the-blank and hint variants did not change results, but this only extends the query family by one more finite set. To support 'knowledge has been truly removed', the authors should add genuine out-of-distribution probes — e.g., non-English questions, cloze completions, multiple-choice framing, task reframings, and paraphrases from held-out generators not used in evaluation — and report JW on th","section":"§4.2, Eq. (4), §3.1"},{"comment":"All headline forget/retain/utility numbers depend on calls to Gemini-2.5-Flash through a closed API. The human validation is limited to 360 LKF samples, and approximately 1.5% of RWKU samples are removed due to judge/API failures. As reported, there is no analysis of judge variance, prompt sensitivity, or agreement with an independent judge. Since Tables 1–3 and the RWKU results are the basis for every comparative claim, this is a reproducibility risk. The authors should release judge outputs, report repeated-call variability, and provide at least one additional judge or a manual audit for a subsample of the main tables.","section":"§4.1, Appendix A.4, all main tables"},{"comment":"For RWKU, DPO and NPO use the benchmark's predefined retain templates, while GradDiff, SimNPO, and JensUn use a custom train-time retain set constructed from disjoint targets, because the RWKU evaluation retain set has a format mismatch. Therefore, Table 2 compares methods trained with different retain distributions. This is disclosed, but it biases the retain and utility comparisons in the main table. The authors should either construct one common train-time retain set for all methods, or explicitly state in the main text that RWKU retain columns are not directly comparable across these method groups.","section":"Appendix B.3, Table 2"}],"minor_comments":[{"comment":"'Problems with ROGUE-L based metrics' contains a typo: ROGUE should be ROUGE.","section":"Figure 17 caption"},{"comment":"The notation JP is described as 'average forget quality evaluated with paraphrases' in one place, but Eq. (7) defines a sample-wise worst-case (max) accuracy. Please make the definition of JP, JICR, and JW consistent throughout.","section":"§4.2 and Appendix E.1"},{"comment":"The table header says 'Forget-utility trade-off ... on the LKF dataset', but the 'selected' rows are not visually marked in the provided text. Please ensure the final selected λR values are clearly highlighted in the camera-ready version.","section":"Table 9"},{"comment":"The sentence 'we use 1k instruction based generated queries from AlpacaEval' is awkward; consider '1,000 instruction-following queries sampled from AlpacaEval'.","section":"Appendix B.4"}],"recommendation":"major_revision","confidential_remarks":"The reader's stress-test concern about finite paraphrase coverage is legitimate and lands on the paper's central permanence claim. The method itself is well specified and the empirical comparison is informative, but the title/abstract overstate irreversibility, and Table 3 plus Table 13 already provide counter-evidence. I would not reject: the contribution can be made sound by reframing the permanence claim and adding out-of-distribution/relearning experiments. The closed-API judge is a separate but important reproducibility issue that the authors should address transparently."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read this one carefully. The core contributions are real: a JSD-based forget/retain loss that behaves better than the usual log-likelihood or NPO variants, a worst-case evaluation protocol (LLM judge over paraphrases and in-context retain) that exposes over-optimistic scores in existing benchmarks, and a new LKF dataset of non-dichotomous, lesser-known facts. The experiments are extensive, the ablations are honest (they sweep lambda_R and LR and report trade-off curves), and they include a human study showing the judge agrees with human ratings. The gradient analysis for JSD vs KL is a nice, checkable detail. I would absolutely use their evaluation ideas and the LKF dataset in my own work.\n\nThe soft spots are mostly in the framing. The title and abstract promise \"almost irreversible forgetting,\" but Table 3 shows JensUn recovering 8% of forget accuracy after 2000 unlearning steps under benign relearning. That is better than the baselines but it is not irreversibility. The paper actually concedes this in the text, but the headline claim outruns the evidence. Second, the worst-case metric J_W is defined over a finite set of English paraphrases generated by three small models. That is a useful lower bound on retrieval, but it cannot support a claim that the fact is erased from the weights. A different phrasing, a non-English query, or a different task format could still elicit the answer. The authors do test fill-in-the-blank and hints in an appendix and find no improvement to J_W, which helps, but the set remains finite and closed.\n\nI also note the reliance on a closed API judge (Gemini-2.5-Flash) for all headline numbers. The judge agrees with humans on a 360-sample study, so it is not arbitrary, but the reproducibility of exact numbers is limited. Low-severity concern: the forget loss explicitly trains refusal, so the 0% forget accuracy is partly by construction. That is legitimate unlearning behavior, not metric-gaming, but it should be stated more clearly.\n\nOverall: the method and evaluation framework are solid, the fairness of the comparison is reasonable, and the empirical ranking of JensUn against baselines is probably robust. The permanence claim needs to be toned down or re-tested on out-of-distribution queries. This paper deserves a serious referee; I would send it out and ask for the permanence language to be fixed.\n\nTake it for your reading group.","headline":"A useful empirical package—JSD loss plus a stricter evaluation protocol—but the 'almost irreversible' headline overstates what their own relearning table shows.","tokens_in":37994,"tokens_out":1715,"would_cite":true,"duration_ms":21975,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Jensen-Shannon objective drives LLM unlearning to 0% worst-case recall of targeted facts while preserving utility, and the paper shows standard evaluations overstate how well previous methods forget.","keywords":["machine unlearning","Jensen-Shannon divergence","large language models","forget-utility trade-off","benign relearning","LLM-as-judge evaluation","worst-case evaluation","LKF dataset"],"falsifier":"Take a JensUn-unlearned model with 0% JW on LKF and probe it with prompt transformations outside the paper's set — e.g., translations of the forget questions into several unrelated languages, or fine-tuning on a curriculum built from facts topically adjacent to the forget set (the paper's own appendix D.5 shows retain-set relearning already lifts JW from 1% to 18%). If any translation or curriculum pushes worst-case recall well above zero, the permanence claim would be settled as suppression, not removal.","tokens_in":37122,"feed_emoji":"🧠","tokens_out":13398,"duration_ms":125390,"temperature":0.7,"pith_summary":"Large language models absorb facts during pre-training, and 'unlearning' aims to delete specific facts from the parameters instead of retraining from scratch. This paper argues that existing unlearning methods look more successful than they are, because evaluating with ROUGE scores and a single question format misses paraphrases and in-context prompts that still elicit the forgotten answer. To fix both sides, the authors propose JensUn, which uses the Jensen-Shannon Divergence for both the forget and retain objectives — a bounded, symmetric loss that stays stable through long unlearning — and a harder evaluation protocol: an LLM judge plus worst-case accuracy over many paraphrases. On their new LKF dataset of lesser-known real-world facts, JensUn reaches 0% worst-case forget accuracy while keeping win rate and MMLU near the base model, and after 2000 unlearning steps followed by benign relearning it recovers only 8% of forgotten answers, versus 26% for NPO and 58% for NPO+SAM. If the claim holds, unlearning can be both effective and safe enough to deploy in privacy and safety settings.","feed_headline":"LLM forgetting hits 0% recall and survives relearning","feed_subtitle":"A bounded Jensen-Shannon loss keeps the model useful while forgotten facts stay forgotten.","key_machinery":"The central object is the Jensen-Shannon Divergence (JSD), a symmetric distance between probability distributions bounded between 0 and log 2 per token. JensUn uses it in both terms of the unlearning objective: the forget loss minimizes JSD from the model's output distribution to a one-hot refusal target such as 'No idea', and the retain loss minimizes JSD from the unlearning model to the frozen base model. The paper proves the JSD gradient norm is at most (log 2)/2 times the KL gradient norm, so updates stay well-behaved over long fine-tuning, while the retain loss starts at zero and only grows as the model drifts — a natural balance between forgetting and utility. The second piece of machi","core_discovery":"Two claims carry the paper. First, standard unlearning evaluation overstates success: ROUGE does not track factual correctness, and single-format questions hide residual knowledge. The worst-case metric JW — LLM-judged accuracy over 15 paraphrases plus in-context retain samples, forgotten only if all fail — raises reported forget accuracy by up to 43%. Second, with both losses formulated as Jensen-Shannon Divergence, the bounded objective drives the forget set to a refusal string while staying near the base model: on the new LKF dataset JensUn reaches 0% JW with WR 0.47, and after 2000 unlearning steps plus benign relearning recovers only 8% of forgotten answers versus 26% for NPO and 58% fo","pith_inferences":["Editorial extension: if bounded gradients are the true driver of JensUn's stability, other symmetric bounded divergences (e.g., the t-Jensen-Shannon family from the GAN literature) should behave similarly; the paper does not test this.","Editorial extension: the permanence claim is established against 'benign' relearning on data disjoint from the forget set; the paper's own retain-set relearning experiment (appendix D.5) pushes JensUn's JW from 1% to 18%, so adversarial curricula built from forget-adjacent or rephrased data remain an open threat model.","Editorial extension: the worst-case protocol could plausibly be extended to cross-lingual prompts and membership-inference probing; the paper tested fill-in-the-blank and hint variants (no additional signal) but not languages.","Editorial extension: LKF's recipe — real lesser-known facts, non-dichotomous questions, a semantically adjacent retain set — transfers naturally to privacy-sensitive domains such as medical or legal records, where 'obscure but real' is exactly the relevant regime."],"forward_implications":["Benchmark results for existing unlearning methods (TOFU, RWKU, MUSE, WHP) should be re-read under worst-case evaluation; the paper shows forget accuracy jumps by up to 43%, meaning several published comparisons overstated forgetting.","Unlearning can be run much longer than the standard 200 steps: JensUn stays stable and useful past 2000 steps, and longer unlearning is what makes benign relearning ineffective.","A provider can choose the behavior on forgotten facts — explicit refusal ('No idea'), token gibberish ('#'), or silence — by picking the JSD target string, without sacrificing forget quality or utility.","The LKF dataset and the LLM-judge-with-worst-case protocol give a realistic, low-cost testbed for unlearning evaluation, including a more reliable ranking of methods than ROUGE-based scores.","Combining the better loss with the harder evaluation shrinks the gap between reported unlearning success and actual forgetting, making unlearning a more credible tool for privacy and safety compliance."],"supporting_citations":[{"why":"Supplies the unified benchmarking codebase and the default λF hyperparameters used to train every baseline unlearning method on the LKF dataset.","marker":"Dorna et al. (2025)"},{"why":"Provides the RWKU benchmark and its original ROUGE/adversarial evaluation, which the proposed worst-case protocol shows overestimates unlearning quality.","marker":"Jin et al. (2024)"},{"why":"Shows that reformulating forget queries reveals residual knowledge in supposedly unlearned models, motivating the paraphrase and in-context-retain worst-case evaluation.","marker":"Thaker et al. (2025)"},{"why":"Defines the benign relearning setup used to test whether forgotten facts return under fine-tuning, the basis of the irreversibility claim.","marker":"Hu et al. (2024)"},{"why":"NPO, the main preference-optimization baseline whose unbounded forget loss is analyzed and whose 26% relearning recovery is the strongest comparison point for JensUn.","marker":"Zhang et al. (2024b)"},{"why":"NPO+SAM, the sharpness-aware relearning-resilient baseline that JensUn outperforms (8% vs 58% forget accuracy after relearning).","marker":"Fan et al. (2025)"},{"why":"TOFU, the benchmark whose ROUGE-based and dichotomous-question design motivates the LKF dataset and the semantic-judge metric.","marker":"Maini et al. (2024)"},{"why":"ROUGE, the string-overlap metric that the paper shows misaligns with human judgments of factual answers and replaces with an LLM judge.","marker":"Lin (2004)"},{"why":"DPO, a preference-optimization method adapted for unlearning and compared as a baseline on both LKF and RWKU.","marker":"Rafailov et al. (2023)"},{"why":"RMU, the representation-level unlearning baseline whose worst-case forget accuracy jumps most (up to 43%) under the new evaluation.","marker":"Li et al. (2024)"}],"fun_headline_variants":["JensUn: 0% recall on forgotten facts after relearning","Worst-case eval exposes LLM unlearning failures","Jensen-Shannon loss makes LLM forgetting stick","Forget 0% recall, relearn only 8% — JensUn"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"That a fact is truly forgotten because the model fails a finite test suite — up to 15 paraphrases and in-context retain samples — with no argument that these variations cover the prompts or fine-tuning curricula a determined questioner could use; an untested phrasing or relearning recipe that surfaces the answer would break the 'almost irreversible' conclusion.","fun_headline_variants_meta":{"raw":{"variants":["JensUn: 0% recall on forgotten facts after relearning","Worst-case eval exposes LLM unlearning failures","Jensen-Shannon loss makes LLM forgetting stick","Forget 0% recall, relearn only 8% — JensUn"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000251,"raw_usage":{"total_tokens":1393,"prompt_tokens":741,"completion_tokens":652,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":579}},"tokens_in":485,"tokens_out":652,"duration_ms":8104,"temperature":1.0,"reasoning_tokens":579,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T11:22:08.148525+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a JensUn-unlearned model with 0% JW on LKF and probe it with prompt transformations outside the paper's set — e.g., translations of the forget questions into several unrelated languages, or fine-tuning on a curriculum built from facts topically adjacent to the forget set (the paper's own appendix D.5 shows retain-set relearning already lifts JW from 1% to 18%). If any translation or curriculum pushes worst-case recall well above zero, the permanence claim would be settled as suppression, not removal.","supporting_citations":[],"review_version":1}