REVIEW 4 major objections 5 minor 16 references
Few-Shot Concept Unlearning with Low Rank Adaptation
T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The paper claims that updating only the final layers of CLIP's text encoder with a low-rank weighted loss can make Stable Diffusion forget a concept in about 50 seconds using 4-5 images.
desk verdict Modest extension of few-shot unlearning, but a sign error in the loss and a questionable mechanism make the reported results uninterpretable. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the low-rank perturbation $\Delta P = AB^\top$ of the CLIP text encoder's projection matrix, combined with a weighted loss over forget, retain, and image-reconstruction terms. The low rank (4-8) restricts changes to a small subspace so few images suffice and unrelated concepts are supposedly preserved; the retain loss anchors the projection for non-target prompts, and the forget loss drives target embeddings toward the random vector $F_{\text{forget}}$.
What would settle it
Run the unlearning procedure on a Stable Diffusion v2 model for a concept such as Siberian Husky, once with $F_{\text{forget}}$ drawn as $\mathcal{N}(0,2.0)$ and once with $F_{\text{forget}}$ set to a constant vector; if the concept disappears in both cases or unrelated prompts degrade in the first case, the random target is not semantically erasing the concept but simply corrupting the embedding space.
Extended reading notes
Core claim
On its own terms, the paper's discovery is that forgetting a concept in a text-to-image diffusion model can be reduced to a small perturbation of the CLIP text encoder's final projection matrix $P$. The update is $\Delta P = AB^\top$ with $A,B \in \mathbb{R}^{d\times r}$, $r$ between 4 and 8, learned by backpropagating a weighted loss that combines an image-noise reconstruction term, a retain loss keeping $(P+\Delta P)f_r^\top$ close to $Pf_r^\top$, and a forget loss pushing $(P+\Delta P)f_f^\top$ toward a random normal target $F_{\text{forget}} \sim \mathcal{N}(0,2.0)$. The authors report forget CLIP scores between 0.259 and 0.294, detection rates between 0 and 0.02, and runtimes around 50-58 seconds for Taj Mahal, Mickey Mouse, and Siberian Husky, with retain CLIP scores near 0.32-0.35.
Load-bearing premise
The claim depends on the assumptions that a randomly initiated target vector for the forget loss erases the concept semantically and that updating only the final text-encoder layers is enough to stop the U-Net from generating it.
Editorial extensions
If this is right
- If the claim holds, concept erasure in Stable Diffusion v2 becomes a one-minute operation using only a handful of images, replacing days of retraining.
- Copyrighted or private concepts such as Mickey Mouse can be suppressed while prompts that mention only peripheral objects still generate those objects, as in the reported case of gloves without the mouse.
- The method reports near-zero detection rates (0-0.02) and forget CLIP scores well below the baselines it compares against, implying the erased concept rarely appears in generations.
- Because only the CLIP text encoder is modified, the same low-rank perturbation could be applied to other CLIP-conditioned diffusion pipelines without retraining their image decoders.
Reading between the lines
- The paper leaves implicit that the random normal target $F_{\text{forget}}$ may be doing most of the work; if the same erasure happens with a zero or arbitrary fixed target, the method is likely corrupting the embedding rather than learning a semantic erasure.
- The reported high FID scores are interpreted by the authors as successful distancing from the forget concept, but a high FID could also signal degraded image quality overall; comparing FID on unrelated prompts would separate those two readings.
- The paper's own future-work note suggests a zero-shot variant by randomizing latents instead of encoding forget images; that points to a direct extension where the few-shot images become unnecessary if the random target alone remains sufficient.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes a few-shot concept unlearning method for Stable Diffusion v2. The authors add a low-rank perturbation ΔP = AB^T to the projection matrix P of the CLIP text encoder and update a small set of trainable text-encoder layers using a weighted loss composed of a noise-prediction term, a retain-consistency term, a forget term that maps the forget embedding to a randomly sampled target F_forget ~ N(0,2.0), and a low-rank regularization. Experiments on three concepts (Taj Mahal, Mickey Mouse, Siberian Husky) report forget CLIP scores of 0.259–0.294, detection rates below 0.02, and unlearning times around one minute, with qualitative before/after images and comparisons to UCE, EraseDiff, and a few-shot unlearning baseline.
Significance. If the claims held, the method would be valuable: it is parameter-efficient, few-shot, and much faster than retraining, and it targets the text encoder rather than the U-Net. The authors also provide quantitative metrics and qualitative samples for multiple concepts. However, the mechanism is not established: the optimized projection is not the text feature that conditions the U-Net in Stable Diffusion v2, the loss sign is internally inconsistent, and the forget target is arbitrary, so the observed effects could be generic image degradation rather than selective concept unlearning. The paper therefore does not yet support its central claim.
major comments (4)
- [Section III-A, Algorithm 1 line 12, Eq. (4)] The text states that reconstruction fidelity is enforced by a standard noise-prediction loss, but Algorithm 1 line 12 defines L_img = −MSE(ẑ, noise) and Eq. (4) minimizes the total loss L = L_img + λ1 L_retain + λ2 L_forget + λ3 L_reg. Minimizing −MSE maximizes the noise-prediction error on the forget prompt; this is not a standard noise-prediction loss and is never justified. Because the loss weights λ1, λ2, λ3 are not reported, the observed drop in Forget CLIP and Detection Rate could reflect general degradation rather than targeted unlearning. The authors must either correct the sign or explicitly justify the maximization, and they must report the loss weights.
- [Section III-A, 'Prompt Conditioning with CLIP' and Algorithm 1] The method optimizes quantities of the form (P + ΔP) f_f^T and (P + ΔP) f_r^T, treating P as the matrix that maps intermediate token features to final embeddings. In Stable Diffusion v2, the conditioning signal supplied to the U-Net is the token-level last hidden state of the CLIP text encoder, not the pooled or projected CLIP embedding. Since P is frozen and only ΔP and selected text-encoder layers are updated, L_forget and L_retain act on a projected output that is not directly consumed by the generative model. Gradients through f_f can still alter the unfrozen text-encoder layers, but the objective provides no mechanism to push the actual conditioning features in a semantically meaningful direction. The paper must demonstrate that the optimized projection is causally connected to generation, for example by ablating against a variant that optimizes the last hidden state directly.
- [Algorithm 1 line 4, Eq. (4)] The forget target F_forget ~ N(0,2.0) is arbitrary; no derivation, ablation, or sensitivity analysis is given for the target distribution or for the rank r (fixed at 8 in Section IV). The forget objective is MSE((P + ΔP) f_f^T, F_forget), so any observed reduction in forget-embedding similarity is by construction an optimization target rather than independent evidence. The detection rate is partially independent, but it is reported on only three concepts without error bars. A random target can degrade embeddings globally, so the authors need to ablate F_forget (for example, zero vector, a learned textual-inversion vector, or an orthogonal direction) and evaluate retain-set generation quality beyond a single CLIP score.
- [Section V.A and Table I] The evaluation conflates forgetting with image degradation. Section IV.A.2 states that a higher FID is desirable for unlearning, but high FID only indicates low similarity to the reference distribution and can result from general quality collapse; the reported FID values (151.6–343.0) are much larger than the baselines in Table II, yet retain CLIP is the only retain-quality metric and it is reported without variance. In addition, Section V.A says the forget loss should be as high as possible while the optimized L_forget is a squared error that is minimized to a random target; these contradictory statements make the loss graphs in Figure 7 uninterpretable. The paper should report FID on retain prompts, include standard deviations, specify the detector and prompts used for Detection Rate, and define exactly what the plotted losses represent.
minor comments (5)
- [Throughout] There are numerous typos and grammar issues, including 'componet', 'upto', 'Adversial', 'de-noisning', and 'co-relation'; the manuscript needs a careful proofread.
- [Section II.B] The text says Stable Diffusion has 'three major components' and then lists four items (VAE, U-Net, text encoder, scheduler).
- [Section IV and Table II] Table II appears without a clear lead-in sentence, and the baseline results are averaged across concepts without standard deviations or per-concept breakdowns, making comparison with Table I difficult.
- [Section IV, reproducibility] The manuscript does not state the number of forget and retain images used per concept, the prompt templates, the number of diffusion sampling steps, the classifier-free guidance scale, the optimizer settings, or the loss weights λ1, λ2, λ3; these are necessary for reproducibility.
- [Section III.A, Eq. (4)] The notation is inconsistent: L_forget is defined both as a matrix-valued MSE in the text and as a scalar loss in Eq. (4); the dimensions of (P + ΔP) f_f^T and F_forget should be spelled out.
Circularity Check
The headline Forget CLIP metric is directly optimized by L_forget; success on that metric is substantially the training objective rather than an independent prediction.
-
fitted input called prediction
[Algorithm 1, lines 14-15; Section IV-A.1 (CLIP Scores); Table I]
"Lforget← MSE((P + ∆P )f⊤ f ,F forget) ... Forget CLIP score measures how well the model forgets the target concept. A lower forget CLIP score means the model has successfully unlearned or removed that concept."
The unlearning objective L_forget directly regresses the projected forget-prompt text embedding (P+ΔP)f_f^T to a random target F_forget, while the reported Forget CLIP score measures cosine similarity between text and image embeddings in the same CLIP space. With the modified text encoder, the forget prompt's projected embedding is literally the quantity being optimized, so a low Forget CLIP score is the training target by construction. The paper does not state that the CLIP scorer uses an independent frozen text encoder, so the values in Table I (0.259-0.294) are not an emergent test of unlearning.
full rationale
There is no load-bearing self-citation or imported uniqueness theorem: the method builds on external prior work ([8] few-shot unlearning, [9] textual inversion), and no argument is forced by the authors' own prior results. The genuine circularity is narrower but central to the evaluation. L_forget fits (P+ΔP)f_f^T to F_forget ~ N(0, 2.0), and the headline Forget CLIP score is the cosine similarity between text and image embeddings in that same CLIP space; unless the evaluator freezes an independent CLIP text encoder (not stated), the lowered score is the direct product of the optimization. The Retain CLIP score is similarly anchored by L_retain to the original projection P f_r^T. Independent evidence exists—Detection Rate and the qualitative figures—but it is not quantified against a no-op baseline, and L_img = -MSE actively rewards worse noise prediction, so apparent forgetting may partly reflect generic degradation. Those are correctness concerns rather than circularity. Overall, the headline Forget CLIP 'prediction' reduces by construction to the training objective, while the central generative claim retains some independent content via detection and images, giving a partial-circularity score of 6.
Assumptions & free parameters
free parameters (4)
- rank r =
8
- F_forget (target forget representation) =
N(0, 2.0) random sample per Algorithm 1 line 4
- loss weights lambda_1, lambda_2, lambda_3 =
unspecified
- number of epochs and noise level =
10 epochs; t sampled uniformly
assumptions (4)
- domain assumption Low-rank update ΔP = AB^T with r much less than d is sufficient to express a useful text-encoder perturbation.
- domain assumption Modifying only the final layers of the CLIP text encoder is enough to erase a concept from Stable Diffusion generations.
- domain assumption A small set of retain prompts used in L_retain is representative of all concepts that must be preserved.
- ad hoc to paper A random normal target F_forget is a valid forget direction in embedding space.
Cite this review
Pith. "Pith review of Few-Shot Concept Unlearning with Low Rank Adaptation." pith.science (2026). https://pith.science/paper/TOBC32PW
@misc{pith2026250512395,
author = {Pith},
title = {Pith review of: Few-Shot Concept Unlearning with Low Rank Adaptation},
year = {2026},
howpublished = {\url{https://pith.science/paper/TOBC32PW}},
note = {Machine review of arXiv:2505.12395}
}
read the original abstract
Image Generation models are a trending topic nowadays, with many people utilizing Artificial Intelligence models in order to generate images. There are many such models which, given a prompt of a text, will generate an image which depicts said prompt. There are many image generation models, such as Latent Diffusion Models, Denoising Diffusion Probabilistic Models, Generative Adversarial Networks and many more. When generating images, these models can generate sensitive image data, which can be threatening to privacy or may violate copyright laws of private entities. Machine unlearning aims at removing the influence of specific data subsets from the trained models and in the case of image generation models, remove the influence of a concept such that the model is unable to generate said images of the concept when prompted. Conventional retraining of the model can take upto days, hence fast algorithms are the need of the hour. In this paper we propose an algorithm that aims to remove the influence of concepts in diffusion models through updating the gradients of the final layers of the text encoders. Using a weighted loss function, we utilize backpropagation in order to update the weights of the final layers of the Text Encoder componet of the Stable Diffusion Model, removing influence of the concept from the text-image embedding space, such that when prompted, the result is an image not containing the concept. The weighted loss function makes use of Textual Inversion and Low-Rank Adaptation.We perform our experiments on Latent Diffusion Models, namely the Stable Diffusion v2 model, with an average concept unlearning runtime of 50 seconds using 4-5 images.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[8]
Erasing concepts from text-to-image diffusion models with few-shot unlearning,
M. Fuchi and T. Takagi, “Erasing concepts from text-to-image diffusion models with few-shot unlearning,” in 35th British Machine Vision Conference 2024, BMVC 2024, Glasgow, UK, November 25-28, 2024 . BMV A, 2024. [Online]. Available: https://papers.bmvc2024.org/0216.pdf
work page 2024
-
[1]
The eu general data protection regulation (gdpr),
P. V oigt and A. V on dem Bussche, “The eu general data protection regulation (gdpr),” A Practical Guide, 1st Ed., Cham: Springer International Publishing , vol. 10, no. 3152676, pp. 10–5555, 2017
2017
-
[2]
An introduction to the california consumer privacy act (ccpa),
E. Goldman, “An introduction to the california consumer privacy act (ccpa),” Santa Clara Univ. Legal Studies Research Paper , 2020
2020
-
[3]
Lexica - ai art generator and search tool,
Lexica, “Lexica - ai art generator and search tool,” https://lexica. art/, 2025, accessed: 2025-05-03
work page 2025
-
[4]
Novelai - ai-assisted storytelling and art generation,
NovelAI, “Novelai - ai-assisted storytelling and art generation,” https://novelai.net/, 2025, accessed: 2025-05-03
work page 2025
-
[5]
Picsart - ai photo and video editing tools,
Picsart Inc., “Picsart - ai photo and video editing tools,” https: //picsart.com/, 2025, accessed: 2025-05-03
work page 2025
-
[6]
Adobe firefly - ai generative creative tools,
Adobe Inc., “Adobe firefly - ai generative creative tools,” https: //www.adobe.com/products/firefly.html, 2025, accessed: 2025-05- 03
work page 2025
-
[7]
Erasing concepts from diffusion models,
R. Gandikota, J. Materzynska, J. Fiotto-Kaufman, and D. Bau, “Erasing concepts from diffusion models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 2426–2436
work page 2023
Show all 16 references
-
[9]
An image is worth one word: Personalizing text-to-image generation using textual inversion,
R. Gal, Y . Alaluf, Y . Atzmon, O. Patashnik, A. H. Bermano, G. Chechik, and D. Cohen-Or, “An image is worth one word: Personalizing text-to-image generation using textual inversion,” arXiv preprint arXiv:2208.01618 , 2022
2022 arXiv
-
[10]
Denoising diffusion probabilistic models,
J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in neural information processing systems , vol. 33, pp. 6840–6851, 2020
2020
-
[11]
Learning transferable visual models from natural language supervision,
A. Radford, J. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” 02 2021
2021
-
[12]
Forget-me- not: Learning to forget in text-to-image diffusion models,
E. Zhang, K. Wang, X. Xu, Z. Wang, and H. Shi, “Forget-me- not: Learning to forget in text-to-image diffusion models,” arXiv preprint arXiv:2211.08332, 2023
2023 arXiv
-
[13]
Scissorhands: Scrub data influ- ence via connection sensitivity in networks,
J. Wu and M. Harandi, “Scissorhands: Scrub data influ- ence via connection sensitivity in networks,” arXiv preprint arXiv:2401.06187, 2024
2024 arXiv
-
[14]
Erasediff: Erasing data influence in diffusion models,
J. Wu, T. Le, M. Hayat, and M. Harandi, “Erasediff: Erasing data influence in diffusion models,” arXiv preprint arXiv:2401.05779 , 2024
2024 arXiv
-
[15]
Unlearncanvas: Stylized image dataset for enhanced machine unlearning evaluation in diffusion models,
Y . Zhang, C. Fan, Y . Zhang, Y . Yao, J. Jia, J. Liu, G. Zhang, G. Liu, R. R. Kompella, X. Liu et al. , “Unlearncanvas: Stylized image dataset for enhanced machine unlearning evaluation in diffusion models,” arXiv preprint arXiv:2402.11846 , 2024
2024 arXiv
-
[16]
Unified concept editing in diffusion models,
R. Gandikota, H. Orgad, Y . Belinkov, J. Materzy ´nska, and D. Bau, “Unified concept editing in diffusion models,” in Pro- ceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024, pp. 5111–5120
2024
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.