arxiv: 2604.14644 · v1 · submitted 2026-04-16 · 💻 cs.CL · cs.LG

Recognition: unknown

CURaTE: Continual Unlearning in Real Time with Ensured Preservation of LLM Knowledge

Seyun Bae , Seokhan Lee , Eunho Yang

Authors on Pith no claims yet

Pith reviewed 2026-05-10 11:58 UTC · model grok-4.3

classification 💻 cs.CL cs.LG

keywords machine unlearninglarge language modelscontinual unlearningknowledge preservationsentence embeddingsrefusal mechanismsreal-time systemsforget requests

0 comments

The pith

CURaTE lets LLMs forget targeted information on demand by training a separate embedding model to refuse matching prompts, without ever changing the original model's parameters.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents CURaTE as a method for removing specific knowledge from large language models after they have been trained. It trains a sentence embedding model on a dataset of forget requests so that the model can draw sharp lines around inputs that match those requests. When a new prompt arrives, the embedding model checks its similarity to the stored requests and triggers a refusal if a match is found. This leaves the language model itself untouched, which the authors show preserves its performance on all other topics at near-perfect levels no matter how many unlearning steps are applied. The approach supports immediate, ongoing unlearning that prior methods lose as updates accumulate.

Core claim

CURaTE trains a sentence embedding model on a forget-request dataset to form sharp decision boundaries, then uses similarity scoring on incoming prompts to decide whether to answer with the language model or return a refusal. Because the language model parameters are never modified, knowledge unrelated to the forget requests is preserved at near-perfect levels through any number of unlearning operations, and each request can be acted on in real time.

What carries the argument

A separately trained sentence embedding model that creates decision boundaries around forget requests and gates whether the unchanged language model generates a response or refuses.

If this is right

Unlearning requests can be handled instantly upon arrival with no retraining of the language model.
Performance on all retained knowledge remains near perfect across an unlimited sequence of updates.
Forgetting effectiveness exceeds that of methods that edit model parameters directly.
Continual unlearning becomes feasible for deployed models without cumulative loss of utility.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The separation of a lightweight detector from the main model could let existing LLM services add unlearning with little added latency.
Similar detectors might be used for other runtime controls such as safety checks without retraining the base model.
Over repeated requests this method could lower the long-term cost of meeting data-deletion regulations compared with periodic full retraining.

Load-bearing premise

Training a sentence embedding model on forget-request data will produce decision boundaries sharp enough to catch relevant prompts in real inputs without refusing too many unrelated ones.

What would settle it

A test set containing both direct variants of stored forget requests and clearly unrelated prompts, with measurement of whether the embedding model refuses the former while answering the latter at high accuracy.

Figures

Figures reproduced from arXiv: 2604.14644 by Eunho Yang, Seokhan Lee, Seyun Bae.

**Figure 1.** Figure 1: An overview of the CURaTE framework. CURaTE consists of a training phase carried out prior to deployment (upper part) and a three-step inference process after deployment (lower part). In the training phase, the embedder U is trained on three types of synthetic data generated from a seed dataset (training does not require any data from the forget set or retain set). For inference, real-time continual unlear… view at source ↗

**Figure 2.** Figure 2: Continual unlearning results on RETURN. (a) indicates performance on the unlearning target, while [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Figure 3: Continual unlearning results on ScienceQA. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

**Figure 4.** Figure 4: The F1 score resulting from each value of the threshold δ from 0.01 to 0.99 in intervals of 0.01 on four different datasets. needs to be tuned with our method—which is far less than existing methods—and that tuning δ can be carried out purely through inference, whereas hyperparameter tuning for other methods require multiple rounds of training (which is far more time consuming and expensive). Moreover, our… view at source ↗

**Figure 5.** Figure 5: Prompt and code for generating the three types of data based on the seed dataset. The input prompt of [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗

**Figure 6.** Figure 6: Core prompt and code for generating the near utility evaluation datasets on the four benchmarks RETURN, TOFU, TruthfulQA and ScienceQA [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗

**Figure 7.** Figure 7: The set R consists of 229 refusal expressions, all generated using GPT-4o [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗

**Figure 8.** Figure 8: Generated responses from CURaTE and other baselines on the forget set from stage 10 of the RETURN benchmark. M Qualitative Results In this section we show the text responses from all methods to some sample queries taken from the final stage of the RETURN benchmark. As mentioned above, we use a paraphrased variant of the original query to test performance on the forget set as using the original query would… view at source ↗

**Figure 9.** Figure 9: Generated responses from CURaTE and other baselines on the retain set (used) from stage 10 of the RETURN benchmark [PITH_FULL_IMAGE:figures/full_fig_p021_9.png] view at source ↗

**Figure 10.** Figure 10: Generated responses from CURaTE and other baselines on the retain set (not used) from stage 10 of the RETURN benchmark [PITH_FULL_IMAGE:figures/full_fig_p021_10.png] view at source ↗

**Figure 11.** Figure 11: Generated responses from CURaTE and other baselines on the non-target dataset from stage 10 of the RETURN benchmark [PITH_FULL_IMAGE:figures/full_fig_p021_11.png] view at source ↗

**Figure 12.** Figure 12: Generated responses from CURaTE and other baselines on the near utility dataset from stage 10 of the RETURN benchmark [PITH_FULL_IMAGE:figures/full_fig_p022_12.png] view at source ↗

**Figure 13.** Figure 13: Generated responses from CURaTE and other baselines on the WinoGrande dataset from stage 10 of the RETURN benchmark [PITH_FULL_IMAGE:figures/full_fig_p022_13.png] view at source ↗

**Figure 14.** Figure 14: Continual unlearning results on RETURN. (a) indicates performance on the unlearning target, while [PITH_FULL_IMAGE:figures/full_fig_p024_14.png] view at source ↗

**Figure 15.** Figure 15: Continual unlearning results on ScienceQA. (a) shows the unlearning target, while (b)–(e) illustrate performance on data that should be preserved [PITH_FULL_IMAGE:figures/full_fig_p024_15.png] view at source ↗

read the original abstract

The inability to filter out in advance all potentially problematic data from the pre-training of large language models has given rise to the need for methods for unlearning specific pieces of knowledge after training. Existing techniques overlook the need for continuous and immediate action, causing them to suffer from degraded utility as updates accumulate and protracted exposure of sensitive information. To address these issues, we propose Continual Unlearning in Real Time with Ensured Preservation of LLM Knowledge (CURaTE). Our method begins by training a sentence embedding model on a dataset designed to enable the formation of sharp decision boundaries for determining whether a given input prompt corresponds to any stored forget requests. The similarity of a given input to the forget requests is then used to determine whether to answer or return a refusal response. We show that even with such a simple approach, not only does CURaTE achieve more effective forgetting than existing methods, but by avoiding modification of the language model parameters, it also maintains near perfect knowledge preservation over any number of updates and is the only method capable of continual unlearning in real-time.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

CURaTE uses an external embedding model to refuse forget-request matches in real time without touching the LLM, but the abstract's performance claims have no numbers or details to evaluate.

read the letter

CURaTE tries to solve continual unlearning by training a separate embedding model to spot forget requests and refuse them on the fly, leaving the main LLM unchanged. The new part is the non-parametric design. Instead of editing weights or running expensive unlearning algorithms, it just computes similarity in embedding space and decides to answer or refuse. This sidesteps the utility degradation that comes with repeated updates to the model. It does well on the real-time angle. Adding a new forget request means updating the stored set and retraining or fine-tuning the embedding model, which should be faster than anything that touches the LLM. The preservation claim follows logically from never touching the parameters. The weak point is the evidence. The abstract claims superior forgetting and perfect preservation, but there are no results, no comparison tables, and no description of how they measure success or build the training data for the embedder. The assumption that a sentence embedding will draw clean lines between forget cases and everything else is doing a lot of work here. Paraphrases, context shifts, or clever rephrasings could break it, and the paper needs to show that doesn't happen at scale. The stress-test note about decision boundaries is on target given what's in the abstract. If the full paper has ablation studies on threshold choice and false refusal rates, that would help. This paper is aimed at practitioners who need to comply with data deletion requests in deployed models. Anyone thinking about lightweight unlearning methods would find it relevant. It should go to peer review so the experiments can be checked.

Referee Report

3 major / 2 minor

Summary. The manuscript proposes CURaTE for continual unlearning of specific knowledge from LLMs in real time. It trains a sentence embedding model on a forget-request dataset to form sharp decision boundaries, then computes similarity between an input prompt and stored forget requests to decide whether to answer or refuse. By avoiding any modification of the underlying LLM parameters, the method claims to deliver more effective forgetting than prior techniques while maintaining near-perfect knowledge preservation across arbitrary numbers of updates and enabling the only real-time continual unlearning approach.

Significance. If the empirical claims are substantiated, the work would be significant for machine unlearning: it directly targets the degradation of utility that accumulates in parameter-modifying methods and offers a path to immediate, ongoing removal of sensitive information without retraining or fine-tuning the base model. The parameter-free preservation aspect is a notable potential strength.

major comments (3)

Abstract: the central claims of 'more effective forgetting than existing methods,' 'near perfect knowledge preservation over any number of updates,' and being 'the only method capable of continual unlearning in real-time' are asserted without any quantitative results, baselines, evaluation metrics, or experimental protocol, rendering the claims unevaluable from the provided text.
Method description (sentence-embedding component): the approach depends on the embedding model producing sufficiently sharp decision boundaries to classify real-world prompts (including paraphrases and indirect queries) against stored forget requests. No details are supplied on dataset construction (positive/negative examples, augmentation strategy), similarity-threshold selection, or quantitative validation of boundary quality or false-refusal rates.
Evaluation claims: the assertion of perfect preservation 'over any number of updates' and superior forgetting requires explicit multi-step continual-unlearning experiments that measure both leakage on forget cases and utility degradation on unrelated prompts; the current text provides none.

minor comments (2)

Clarify whether the similarity threshold is a fixed hyperparameter, a learned quantity, or chosen via a specific procedure, and report its sensitivity.
Ensure the full manuscript includes a dedicated experimental section with tables reporting forgetting efficacy, preservation metrics, runtime, and comparisons to baselines.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive comments, which help clarify how to better present our contributions. We address each major comment below and will revise the manuscript to strengthen the abstract, method details, and evaluation descriptions.

read point-by-point responses

Referee: Abstract: the central claims of 'more effective forgetting than existing methods,' 'near perfect knowledge preservation over any number of updates,' and being 'the only method capable of continual unlearning in real-time' are asserted without any quantitative results, baselines, evaluation metrics, or experimental protocol, rendering the claims unevaluable from the provided text.

Authors: We agree that the abstract would be strengthened by incorporating concrete quantitative summaries of our results. The full manuscript reports experimental outcomes supporting these claims, including comparisons to baselines on forgetting effectiveness and preservation metrics. We will revise the abstract to include key numerical findings (e.g., forgetting rates, preservation accuracy across update counts) and a brief reference to the evaluation protocol. revision: yes
Referee: Method description (sentence-embedding component): the approach depends on the embedding model producing sufficiently sharp decision boundaries to classify real-world prompts (including paraphrases and indirect queries) against stored forget requests. No details are supplied on dataset construction (positive/negative examples, augmentation strategy), similarity-threshold selection, or quantitative validation of boundary quality or false-refusal rates.

Authors: The manuscript outlines the training of the sentence embedding model on a forget-request dataset to create decision boundaries. To address the request for greater transparency, we will expand the method section with specifics on dataset construction (positive/negative example generation and augmentation), the similarity threshold selection procedure, and quantitative validation results for boundary sharpness, including false-refusal rates on paraphrased and indirect prompts. revision: yes
Referee: Evaluation claims: the assertion of perfect preservation 'over any number of updates' and superior forgetting requires explicit multi-step continual-unlearning experiments that measure both leakage on forget cases and utility degradation on unrelated prompts; the current text provides none.

Authors: We acknowledge that the evaluation section would benefit from more explicit multi-step experimental details. The manuscript includes continual unlearning experiments demonstrating effective forgetting and near-perfect preservation, but we will revise it to clearly describe the protocol, metrics for leakage on forget requests, utility on unrelated prompts, and results across increasing numbers of updates. revision: yes

Circularity Check

0 steps flagged

No significant circularity; claims rest on independent experimental validation

full rationale

The paper's derivation chain consists of (1) training a separate sentence embedding model on a forget-request dataset to produce decision boundaries, (2) using cosine similarity at inference to decide refusal vs. answer, and (3) evaluating forgetting efficacy and knowledge retention on held-out prompts. Knowledge preservation is a direct consequence of the design choice to leave LLM parameters untouched, not a fitted quantity or self-referential prediction. No equations are presented that reduce performance metrics to the training data by construction, no self-citations are invoked as load-bearing uniqueness theorems, and no ansatz or renaming of prior results occurs. The abstract and method description treat the embedding step as an external, trainable component whose generalization is tested rather than assumed tautologically. This is the normal case of a self-contained empirical proposal.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The approach rests on the assumption that an auxiliary embedding model can be trained to produce reliable binary decisions on forget-request membership and that refusal preserves all non-matching knowledge without side effects.

free parameters (1)

similarity threshold
A cutoff value for deciding whether an input embedding matches a stored forget request closely enough to trigger refusal; must be chosen to balance forgetting effectiveness against utility loss.

axioms (1)

domain assumption Sentence embedding models can be trained to form sharp decision boundaries separating forget-request prompts from normal prompts.
Invoked when the paper states the embedding model is trained to enable sharp boundaries for detection.

pith-pipeline@v0.9.0 · 5489 in / 1232 out tokens · 32971 ms · 2026-05-10T11:58:50.208179+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

7 extracted references · 6 canonical work pages

[1]

Alphaedit: Null-space constrained knowledge editing for language mod- els.ArXiv, abs/2410.02355, 2024

Alphaedit: Null-space constrained knowledge editing for language models.ArXiv, abs/2410.02355. Chongyang Gao, Lixu Wang, Kaize Ding, Chenkai Weng, Xiao Wang, and Qi Zhu. 2025. On large lan- guage model continual unlearning. InThe Thirteenth International Conference on Learning Representa- tions. Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006. Dimensiona...

work page arXiv 2025
[2]

Zhenhua Liu, Tong Zhu, Chuanyuan Tan, and Wen- liang Chen

Rethinking machine unlearning for large lan- guage models.Nature Machine Intelligence, pages 1–14. Zhenhua Liu, Tong Zhu, Chuanyuan Tan, and Wenliang Chen. 2024. Learning to refuse: Towards mitigating privacy risks in llms.ArXiv, abs/2407.10058. Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and...

work page arXiv 2024
[3]

Pratiksha Thaker, Yash Maurya, Shengyuan Hu, Zhiwei Steven Wu, and Virginia Smith

Guardrail baselines for unlearning in llms. ArXiv, abs/2403.03329. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foun- dation and fine-tuned chat models.arXiv preprint arXiv:2307.09288. An Yang, Anfeng Li, Baoso...

work page arXiv 2023
[4]

Miao Yu, Liang Lin, Guibin Zhang, Xinfeng Li, Junfeng Fang, Ningyu Zhang, Kun Wang, and Yang Wang

Qwen3 technical report. Miao Yu, Liang Lin, Guibin Zhang, Xinfeng Li, Junfeng Fang, Ningyu Zhang, Kun Wang, and Yang Wang
[5]

Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei

Unierase: Unlearning token as a universal era- sure primitive for language models.arXiv preprint arXiv:2505.15674. Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. 2024. Negative preference optimization: From catastrophic collapse to effective unlearning.arXiv preprint arXiv:2404.05868. A Discussion on the Cost of Retain Sets Table 5:Retain setsizes for met...

work page arXiv 2024
[6]

Right to be Forgotten

and test split of OpenbookQA (Mihaylov et al., 2018). C.2 Training Configuration We employed ‘multi-qa-mpnet-base-dot- v1’ (Reimers and Gurevych, 2019) as the base model for the unlearning sentence embedder U. This model has only around 109 million parameters so our training cost is orders of magnitude smaller than existing gradient-based approaches, whic...

work page arXiv 2018
[7]

N.1 Privacy Data Unlearning In Figure 14 we can see that for LLaMA-3.2- 1B, gradient-based methods exhibit the same phe- nomenon of overforgetting as in the case of the 7B model

and Qwen3-1.7B (Yang et al., 2025). N.1 Privacy Data Unlearning In Figure 14 we can see that for LLaMA-3.2- 1B, gradient-based methods exhibit the same phe- nomenon of overforgetting as in the case of the 7B model. O3 shows even worse performance on theforget set, indicating greater difficulty in for- getting the necessary information. Of all baselines, U...

work page arXiv 2025