{"id":"90c7dbb0-d7e1-46eb-9533-76284566778e","arxiv_id":"2505.13312","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"GUARD performs inference-time unlearning by classifying prompts, retrieving original answers, and penalizing token matches during beam search, preserving utility but with forget quality that collapses on larger TOFU forget sets.","lead":"GUARD is a wrapper for large language models that blocks forbidden content at generation time instead of retraining the model. It uses a prompt classifier, retrieves the original answer from a forget set, and penalizes matching tokens during beam search, preserving general performance because no weights change.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Leakage prevention depends on perfect prompt detection and answer retrieval, but the paper's own Appendix B/C report nonzero FNR and retrieval top-1 accuracy as low as 90.7%, so a missed or mismatched prompt leaks the full original answer.","rationale":"The reader's weakest assumption identifies exactly this load-bearing dependency: the prompt classifier must correctly identify every forget-related query, and Appendix B/Tables 7a report false-negative rates up to 4.29% on paraphrased, adversarial, and irrelevant-context prompts, while Appendix C/Table 8 reports retrieval top-1 accuracy of only 90.7–97.4%. The central claim of 'effectively preventing' leakage cannot survive a nonzero miss rate because the model parameters are untouched; all unlearning happens in the output filter, so any prompt not caught by the filter exposes the original memorized content. My concrete test directly measures the resulting leakage rate on paraphrased prompts and compares it with an oracle variant, which would settle whether the reported FNR/retrieval errors translate into real leakage. If the end-to-end leakage is negligible, the concern does not land and the central claim is strengthened. If it is not, the paper should either weaken the claim to 'conditional on detection success' or add a robust detection/retrieval guarantee. The reader's verdict of CONDITIONAL remains appropriate, so I do not move the verdict; I agree with the reader's weakest assumption and recommend the same conditionality be made explicit and tested.","tokens_in":28445,"tokens_out":6270,"duration_ms":66597,"concrete_test":"Take the TOFU 5% forget split and generate 500 paraphrases of the forget questions using a paraphraser not used in classifier training (e.g., GPT-4o or back-translation). Run GUARD end-to-end with the exact trained classifier (Appendix B) and SBERT retrieval (Appendix C), and measure the fraction of paraphrases whose generated response still contains the original ground-truth answer (verbatim or high ROUGE-L match). Then rerun with an oracle that sends every paraphrase directly to the blocking stage after exact retrieval. If the end-to-end leakage rate is approximately the product of the reported FNR (up to 4%) and retrieval error (3–9%), or exceeds zero while the oracle leakage is zero, the central no-leakage claim fails in the paper's own operating regime. A stricter variant isolates retrieval failures by removing the classifier and feeding all paraphrases to retrieval.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that GUARD 'effectively prevent[s] the model from leaking the forgotten content' rests on a chain of three components: the prompt classifier must flag every forget-related query, retrieval must return the corresponding original answer, and token/SBERT matching must block every surface form of that answer. The paper's own results show the first two links are imperfect: Table 7a reports FNR up to 4.29% on paraphrased/adversarial/irrelevant-context TOFU prompts, and Table 8 reports SBERT retrieval top-1 accuracy of 90.7–97.4% (96.4–97.4% with RoBERTa reranking). Because GUARD performs no parameter update, the unlearned model is the original model behind a filter: whenever the classifier misses a prompt or retrieval selects a non-corresponding QA pair, no forbidding token is applied and the model's unchanged parameters can emit the forgotten content verbatim. The paper's own examples in Table 13 show that the underlying Llama2-7B does produce the original answers when the guard is not engaged. Thus the 'no leakage' property holds only if detection and retrieval are perfect, which the reported numbers disprove. The additional near-zero FQ values for GUARD on TOFU 5%/10% with Llama2-7B (Tables 9 and 10) further weaken the broad 'strong forget quality' summary, but the core methodological soft spot is the unexamined dependence on the imperfect front-end detection/retrieval.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces GUARD, a training-free, generation-time unlearning framework for LLMs. Instead of fine-tuning, GUARD (i) trains a lightweight MLP prompt classifier to detect whether an input belongs to a forget target, (ii) retrieves the most semantically similar QA pair from the forget set and extracts forbidden phrases from its answer, and (iii) at generation time applies a token-level trie hard match plus an SBERT-based soft semantic match to penalize or prune beam-search candidates that would produce those phrases. Experiments are reported on TOFU (1%, 5%, 10%), MUSE-News, and Harry Potter, with Llama2-7B, Phi-1.5B, OPT-2.7B, and several instruction-tuned models. The central claims are that GUARD achieves strong forget quality across these tasks while causing almost no utility degradation because no parameters are updated.","tokens_in":28793,"tokens_out":4549,"duration_ms":46247,"significance":"If validated, GUARD would be an appealing complement to fine-tuning-based unlearning: it is modular, does not require re-training, and avoids catastrophic forgetting by construction. The paper's strengths include a concrete three-component architecture, ablations of the two matching mechanisms (Table 5) and of forbidden-token extraction strategies (Table 4), and a useful appendix reporting prompt-classifier and retrieval diagnostics. The method is falsifiable and reproducible in principle. However, the significance hinges on an end-to-end leakage-prevention claim that depends on perfect prompt detection and answer retrieval, and the paper's own appendix numbers show those front-end components are not perfect. In addition, the utility-preservation claim is largely definitional because no parameter updates are made, while the 5%/10% TOFU results contradict the paper's 'excellent forget quality' wording. These issues are fixable but currently limit the strength of the contribution.","major_comments":[{"comment":"The text states that GUARD 'consistently demonstrates excellent forget quality' on the 5% and 10% TOFU splits, but the reported FQ values are effectively zero: Table 9 reports 1.8266e-05 for Llama2-7B 5%, and Table 10 reports 5.7346e-07 for Llama2-7B 10%, with values of 0.0014 and 0.0023 for Phi-1.5B. These are far from the retained model's FQ of 1.0 and are orders of magnitude below, e.g., NPO-RT's 0.1779 on the 5% split. The broad claim of strong forget quality across tasks and splits is therefore not supported by the paper's own tables. This is load-bearing because the abstract and conclusion generalize from the 1% success to all settings; please either correct the claim or analyze why the guard degrades as the forget set grows.","section":"§5.2, Tables 9 and 10"},{"comment":"The leakage-prevention guarantee is conditional on front-end perfection: the prompt classifier must flag every forget-related input and retrieval must return the corresponding original answer. The paper's own Appendix B (Table 7a) reports FNRs up to 0.0429 (TOFU 10%, adversarial) and 0.0400 (TOFU 5%, irrelevant context), and Appendix C (Table 8) reports SBERT top-1 retrieval accuracy as low as 0.9070 (TOFU 10%), rising only to 0.9637 with reranking. Because Eq. (2) leaves model parameters unchanged, any missed or mismatched prompt means the original model can emit the forgotten content verbatim; Table 13 shows that without guard activation the underlying Llama2-7B indeed produces the exact original answers. The claim in the abstract that GUARD 'effectively prevent[s] the model from leaking the forgotten content' is thus an end-to-end claim that the reported front-end numbers disprove. Please report an end-to-end leakage rate over the test set, including classifier misses and retrieval mismatches, and condition the paper's claims on the measured component accuracies.","section":"§4.2–4.3, Appendix B and C"},{"comment":"Because GUARD performs no parameter update, the utility metrics MU, R-RL, PPL, and Avg. Acc. are definitionally equal to those of the original model whenever the guard does not fire; Table 1 shows exact equality (e.g., MU 0.6239 and R-RL 0.9818 for Llama2-7B). The paper presents this as 'the best trade-off' and 'no degradation,' but this is a direct consequence of the method's design, not an empirical demonstration of a favorable forget-utility trade-off. The method's real utility question is what happens when the guard does fire: e.g., false positives on benign prompts, or degraded fluency when long forbidden spans are suppressed. Please reframe the utility claims as definitional and provide a quantitative measure of utility conditional on guard activation, including false-positive behavior on retain and general prompts.","section":"§3.3, Tables 1, 3, 9, 10"},{"comment":"All reported results appear to come from a single run, with no error bars, multiple seeds, or significance tests. This matters particularly for FQ, which is a KS-test p-value, and for the component-wise results in Tables 7–8, where classifier training, ChatGPT-based forbidden-token extraction, and retrieval are all stochastic. Without repeated runs, comparisons such as the ablation in Table 5 (FQ 0.1649 vs. 0.0541 vs. 0.0030) or the hyperparameter sweep in Table 12 may reflect noise rather than real differences. Please provide multiple seeds and variance for at least the main TOFU results and the key ablations, or state clearly that single-seed results should be interpreted as preliminary.","section":"§5.2–5.4 and Tables 1–12"}],"minor_comments":[{"comment":"Appendix B.2 says 'we use OPT-2.7B for extracting embeddings,' while §4.2 says the frozen LLM 'will later be unlearned'; since the evaluation base models include Llama2-7B and Phi-1.5B, please clarify whether the classifier trained on OPT embeddings is applied to the hidden states of the actual base model or to OPT embeddings computed at test time.","section":"§B.2 vs. §4.2"},{"comment":"ARC-Challenge and ARC-Easy are cited to reference [12] (Chollet), but the ARC dataset should be cited to its own paper; please correct the reference.","section":"Appendix F.3"},{"comment":"With the hard-match threshold set to β=1, the piecewise definition is confusing: a one-token match both 'fully matches' a forbidden sequence and satisfies 'L_match < β' is false; the text clarifies that any nonzero match is pruned, but the equation should be rewritten to state that every match with L_match ≥ 1 receives the infinite penalty.","section":"Eq. (8)"},{"comment":"The phrase 'forbidden token' is used where a set of tokens is meant; please use 'forbidden tokens' or 'forbidden phrases' consistently.","section":"Throughout"},{"comment":"The escaped Unicode sequences such as '\\u0409' appear literally in the generation examples; please render or explain these characters so the reader can interpret the failure modes.","section":"Table 13"}],"recommendation":"major_revision","confidential_remarks":"The core idea is publishable in principle, but the paper currently overclaims on two fronts: the 5%/10% TOFU rows contradict the 'excellent forget quality' summary, and the end-to-end leakage guarantee is undermined by the authors' own classifier FNR and retrieval accuracy numbers. I would also ask the editor to ensure the reviewers understand that 'utility preservation' is definitional for a training-free method. These are correctable with reframing and additional reporting; I do not see an unfixable load-bearing error."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"GUARD is a real idea: a training-free, inference-time filter that detects a forget prompt, retrieves the original answer, and prunes or penalizes matching tokens during beam search. The specific combination—trie-based hard matching plus SBERT soft matching at decoding time—is new relative to the ICUL, output-filtering, and soft-prompt baselines, and the paper earns credit for showing in ablations that both matching components matter. It also avoids utility degradation by construction, not by magic: no parameters change, so retain-set metrics are identical to the original model.\n\nThe soft spots are not minor. The central claim that GUARD effectively prevents the model from leaking forgotten content is only true when the front-end classifier and retriever are perfect. The paper's own Appendix B reports false-negative rates up to 4.29% on paraphrased, adversarial, and irrelevant-context prompts, and Appendix C reports retrieval top-1 accuracy of 90.7–97.4%. A missed prompt or wrong retrieval means no penalty is applied, and the unchanged model can emit the original answer verbatim. Table 13 shows the base model doing exactly that when the guard is not engaged. That is a load-bearing dependence on unexamined components, and the paper does not report an end-to-end leakage rate that accounts for these errors.\n\nSecond, the text overclaims. The paper says GUARD shows excellent forget quality on TOFU 5% and 10%, but Tables 9 and 10 report FQ near zero on Llama2-7B (1.8e-05 and 5.7e-07). That is not a rounding difference; those numbers mean the guard is not producing retained-model-like behavior for that model and split. This contradiction should be fixed.\n\nOther issues are more moderate. The ChatGPT-4o-mini prompts for forbidden-token extraction are not disclosed, alpha_token and alpha_sbert appear in the equations but I did not find their values in the setup, and there are no error bars or multiple seeds. Replication would be hard without code. Conceptually, this is generation-time censorship, not unlearning in the parameter-removal sense; the model still contains the information, and the PrivLeak score of 109.6 on MUSE shows membership leakage remains, which the paper acknowledges.\n\nWho is this for? Anyone building practical content filters or responding to copyright and right-to-be-forgotten requests without retraining. It is not a substitute for true unlearning and should not be sold as one. I would send this to serious peer review rather than desk reject, because the mechanism is new and the empirical comparison is useful. But the revision needs to fix the overclaim, report end-to-end leakage under classifier and retriever errors, and release code and prompts. Without those changes, the strong claims should not be accepted.","headline":"A genuinely new inference-time filtering mechanism with a load-bearing dependence on perfect detection and retrieval, and text that overclaims forget quality on TOFU 5%/10%.","tokens_in":29329,"tokens_out":3921,"would_cite":true,"duration_ms":38791,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GUARD claims that LLM unlearning can be moved entirely to generation time: a prompt classifier, a retrieval step, and token-level penalties suppress forgotten content during beam search, while the model's weights and general capabilities…","keywords":["LLM unlearning","generation-time unlearning","inference-time control","beam search filtering","semantic matching","prompt classification","right to be forgotten","copyright unlearning"],"falsifier":"Take a forget-set question, paraphrase it into one of the forms the paper's own classifier is shown to miss (paraphrase, adversarial, or irrelevant-context), and run the full GUARD pipeline: if the model reproduces the original answer verbatim in any such case, the unlearning guarantee fails at the detection gate rather than at the generation penalties. A second check targets the retrieval stage, whose top-1 accuracy is 90–97% in the appendix: when retrieval returns the wrong answer, the wrong phrases are blocked and the correct forbidden content can leak through.","tokens_in":28227,"feed_emoji":"🛡️","tokens_out":8856,"duration_ms":84281,"temperature":0.7,"pith_summary":"Generative unlearning usually means fine-tuning the model to unlearn, which risks degrading everything else; this paper tries to establish that forgetting can instead be enforced at generation time. GUARD detects prompts that concern the forget target, retrieves the original answer that must no longer be spoken, and during beam search penalizes or prunes any candidate tokens that match the forbidden phrases, either exactly or by semantic similarity. Because the weights are never modified, the model's general behavior is preserved exactly: in the paper's tables, model utility and retain-set ROUGE-L equal the original model's values, while forget-quality scores improve to the strongest among the compared methods. If the claim holds, an LLM could honor deletion requests and copyright restrictions without retraining, swapping a gradient update for an inference-time gate.","feed_headline":"New guard makes LLMs forget at generation time, no retraining","feed_subtitle":"GUARD prunes forbidden tokens during beam search, keeping utility identical to the original model.","key_machinery":"The load-bearing device is the penalty-augmented beam search: at each decoding step the total penalty $P_{\\text{total}} = P_{\\text{token}} + P_{\\text{SBERT}}$ is added to the negative log-likelihood to form each candidate's cost, and any candidate with infinite penalty is pruned. The hard matcher uses a trie—a tree structure for fast suffix matching—of forbidden sequences and prunes on any suffix match of length at least $\\beta=1$; the soft matcher uses SBERT (Sentence-BERT, a sentence-embedding model) to compute the cosine similarity between the last generated word and each forbidden-token embedding, pruning above the threshold $\\delta=0.5$. A simple MLP over averaged penultimate-layer hidden states gates whether the whole mechanism engages. Together, these parts are what distinguish GUARD from a mere output filter: the suppression acts on candidates during decoding rather than on finished text.","core_discovery":"On the paper's own terms, GUARD's discovery is that a fixed, fully trained LLM can be made to approximate the behavior of a retrained model on the forget set by intervening in the output space only. For each query the pipeline computes $h(x;\\theta_u) = \\mathrm{Unlearn}(h(x;\\theta_o))$: a lightweight MLP classifies whether the prompt belongs to the forget target; if it does, the most semantically similar question-answer pair is retrieved from the forget data and key phrases of the original answer are extracted as forbidden tokens; and during generation, a beam search augmented with token-level hard matching (trie suffix matching) and SBERT-based soft semantic matching assigns penalties, with infinite penalties pruning any candidate that resembles the forbidden content. The result is that the model produces fluent, alternative completions—\"The father of Hsiao Yun-Hwa is unemployed\" instead of \"…is a civil engineer\"—while its behavior on all other inputs is untouched. The paper reports the best forget quality among the compared methods on TOFU 1%/5%/10%, MUSE-News, and the Harry Potter copyright task, with model utility (MU), retain-set ROUGE-L, perplexity, and nine-benchmark zero-shot accuracy identical to the original model's, and it states its own main weakness: the privacy-leakage score on MUSE remains high (109.6), comparable to several baselines, which the paper attributes in part to measurement caveats.","pith_inferences":["Because GUARD prunes tokens without changing the underlying likelihoods, membership-inference signals on the forget data remain intact; a hybrid that also flattens logits or reweights probabilities on forget-triggered prompts could address the privacy-leakage weakness the paper reports on MUSE.","The pipeline is black-box friendly: it only needs decode-time token probabilities and embeddings, so it could sit on top of a hosted model, and the forbidden-token list could be updated live as new takedown or deletion requests arrive.","A testable extension is to replace the binary classifier gate with a confidence-weighted penalty—scaling the soft penalty by classifier uncertainty—which would soften the failure mode where a misclassified prompt receives no guarding at all."],"forward_implications":["Unlearning becomes a deployment-time decision: the same checkpoint serves both \"forget\" and \"retain\" regimes, and updating the forbidden-token trie updates what is forgotten, with no retraining.","Because no parameters change, utility metrics (MU, R-RL, perplexity, and nine-benchmark zero-shot accuracy) are preserved at the original model's level, eliminating catastrophic forgetting as a failure mode of unlearning.","The two matching mechanisms are complementary: ablations on TOFU 1% show that removing either the trie hard matching or the SBERT soft matching drops forget quality from 0.1649 to 0.0541 or 0.0030 respectively.","The same three-step pipeline transfers across three task shapes—entity QA (TOFU), news memorization (MUSE-News), and book-copyright continuation (Harry Potter)—with only classifier training specific to each task.","The paper's reported trade-off means a provider could serve one model to all users and still satisfy targeted takedown or deletion requests by changing the guard data, not the model.","editorial placeholder"],"supporting_citations":[{"why":"Supplies the sentence-embedding model (SBERT) used both for retrieving the most relevant answer from the forget data and for soft semantic matching during generation.","marker":"[77]"},{"why":"Provides the TOFU benchmark, its forget-quality and model-utility metrics, and the 1%/5%/10% entity-unlearning splits used in the main experiments.","marker":"[60]"},{"why":"Provides the MUSE-News benchmark and the VerbMem, KnowMem, and PrivLeak criteria used to evaluate news-memorization unlearning.","marker":"[81]"},{"why":"Supplies the Harry Potter experiment setup, the forget-chunk construction, the fine-tuned base models, and the baseline numbers GUARD is compared against on the copyright task.","marker":"[91]"},{"why":"Supplies the Harry Potter approximate-unlearning baseline and the book-based forget content for copyright unlearning.","marker":"[23]"},{"why":"The in-context training-free unlearning baseline that GUARD must exceed in forget quality on the compared tasks.","marker":"[72]"},{"why":"The output-filtering baseline that post-processes finished text, which GUARD contrasts with its in-generation penalty mechanism.","marker":"[87]"},{"why":"Provides the external chat model used in the default configuration to extract the forbidden tokens from retrieved answers, which the paper treats as the upper bound for extraction quality.","marker":"[1]"}],"fun_headline_variants":["GUARD: unlearn at inference, keep LLM fluent","LLM unlearning without retraining: GUARD prunes tokens","Generation-time unlearning: GUARD adaptively blocks forbidden tokens","Forget on the fly: GUARD filters tokens during generation","No fine-tuning needed: GUARD blocks forbidden knowledge at inference"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The guard only engages when the prompt classifier decides that the query touches the forgotten material: the paper's own appendix reports false-negative rates up to 4% on paraphrased, adversarial, and irrelevant-context prompts, and any query the classifier misses is answered with the forgotten content fully available, since the model's weights were never changed.","fun_headline_variants_meta":{"raw":{"variants":["GUARD: unlearn at inference, keep LLM fluent","LLM unlearning without retraining: GUARD prunes tokens","Generation-time unlearning: GUARD adaptively blocks forbidden tokens","Forget on the fly: GUARD filters tokens during generation","No fine-tuning needed: GUARD blocks forbidden knowledge at inference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000495,"raw_usage":{"total_tokens":2507,"prompt_tokens":1102,"completion_tokens":1405,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":718,"completion_tokens_details":{"reasoning_tokens":1318}},"tokens_in":718,"tokens_out":1405,"duration_ms":10766,"temperature":1.0,"reasoning_tokens":1318,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:15:59.585764+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a forget-set question, paraphrase it into one of the forms the paper's own classifier is shown to miss (paraphrase, adversarial, or irrelevant-context), and run the full GUARD pipeline: if the model reproduces the original answer verbatim in any such case, the unlearning guarantee fails at the detection gate rather than at the generation penalties. A second check targets the retrieval stage, whose top-1 accuracy is 90–97% in the appendix: when retrieval returns the wrong answer, the wrong phrases are blocked and the correct forbidden content can leak through.","supporting_citations":[],"review_version":1}