{"id":"f430fddf-a720-4314-966a-87ade44af6fe","arxiv_id":"2505.12395","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"The authors combine few-shot unlearning with low-rank adaptation on the CLIP text encoder to erase concepts from Stable Diffusion v2 in under a minute, reporting low forget-CLIP scores and detection rates on three concepts.","lead":"A short paper proposes a method to make Stable Diffusion forget specific concepts, such as Mickey Mouse, in about a minute using only a handful of images and low-rank updates to the text encoder. If it works, it offers a fast way to handle privacy and copyright removal requests for image generation models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The optimized loss acts on the CLIP projection head, but Stable Diffusion conditions the U-Net on pre-projection token features, so the claimed forgetting mechanism is not causally connected to generation; the random target further makes any effect uncontrolled.","rationale":"The paper's central assertion is that a few-shot, low-rank update to the final layers of the CLIP text encoder can selectively erase a concept from Stable Diffusion v2 in about one minute. For this to hold, the quantities being optimized must be the quantities that determine generation. On reading the full text, the weakest point is not the choice of F_forget per se, but the location of the loss. Section III-A and Algorithm 1 show that L_retain and L_forget are computed on (P+ΔP)f^T, where P is the projection from intermediate token features to final embeddings. The manuscript does not specify whether these 'final embeddings' are the same representations that Stable Diffusion feeds into the U-Net. In the standard SDv2 pipeline, the U-Net receives the last_hidden_state of the CLIP text transformer (per-token features), while the final text_projection/P is used only for CLIP contrastive embeddings. If that is the case, the loss operates on a head that is disconnected from the generative path. The gradient flow through the shared last transformer layers is indirect and uncontrolled: because P is frozen and ΔP is low-rank, the direction imposed on f_f is determined by P^T times the residual, which need not align with any direction that erases the concept in the hidden-state space. The random Gaussian target compounds this: it is never justified, and any effect it has on the hidden states is coincidental. The negative sign of L_img in Algorithm 1 is a further red flag: maximizing the noise-prediction MSE can degrade generation quality globally, and the absence of λ values, seeds, and retain-quality metrics beyond a single CLIP score means the reported low Forget CLIP and Detection Rate are compatible with generic corruption. This is not an attack on the authors, and the idea of few-shot unlearning through the text encoder is, in principle, promising; but as written, the manuscript does not establish the causal link that the headline claim requires.","tokens_in":8483,"tokens_out":8482,"duration_ms":90542,"concrete_test":"Run the official Stable Diffusion v2 text encoder and print the tensor passed to U-Net cross-attention (likely text_encoder(...)[0], the last_hidden_state). If it is not the output of the final text-projection, implement Algorithm 1 but stop the gradient between f_f and the L_forget/L_retain terms (or remove the projection loss entirely) while keeping the rest fixed; if Forget CLIP and Detection Rate do not change, the projection loss is not the operative mechanism. Also ablate the target by replacing F_forget ~ N(0,2) with a zero vector and with the embedding of an unrelated prompt; report the three result sets. If results are insensitive to the target, the random choice is not load-bearing; if they are sensitive, the paper must justify the specific distribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the assumption that updating the final projection of the CLIP text encoder erases the concept in the generation path. In Section III-A, the paper defines P as the matrix that 'maps intermediate token features to final embeddings' and Algorithm 1 (lines 14-15) computes L_retain and L_forget on (P+ΔP)f^T. However, in Stable Diffusion v2 the tensor that conditions the U-Net is the token-level last_hidden_state of the CLIP text encoder, not the pooled, projected CLIP embedding produced by P. Thus L_forget optimizes an auxiliary projection head that is never read by the U-Net. Gradients can reach the last encoder layers only through f_f, but because P is frozen (only ΔP is learned) and the target F_forget is random Gaussian noise, there is no reason to expect the hidden states used for conditioning to change in a way that selectively erases the concept. The negative L_img in line 12 (L_img = -MSE) actively rewards bad noise prediction; without reported λ values or error bars, the observed low Forget CLIP and Detection Rate could reflect generic image degradation rather than targeted unlearning. The random target is not derived from any principle and no ablation is given, so the mechanism is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a few-shot concept unlearning method for Stable Diffusion v2. The authors add a low-rank perturbation ΔP = AB^T to the projection matrix P of the CLIP text encoder and update a small set of trainable text-encoder layers using a weighted loss composed of a noise-prediction term, a retain-consistency term, a forget term that maps the forget embedding to a randomly sampled target F_forget ~ N(0,2.0), and a low-rank regularization. Experiments on three concepts (Taj Mahal, Mickey Mouse, Siberian Husky) report forget CLIP scores of 0.259–0.294, detection rates below 0.02, and unlearning times around one minute, with qualitative before/after images and comparisons to UCE, EraseDiff, and a few-shot unlearning baseline.","tokens_in":8787,"tokens_out":8995,"duration_ms":96809,"significance":"If the claims held, the method would be valuable: it is parameter-efficient, few-shot, and much faster than retraining, and it targets the text encoder rather than the U-Net. The authors also provide quantitative metrics and qualitative samples for multiple concepts. However, the mechanism is not established: the optimized projection is not the text feature that conditions the U-Net in Stable Diffusion v2, the loss sign is internally inconsistent, and the forget target is arbitrary, so the observed effects could be generic image degradation rather than selective concept unlearning. The paper therefore does not yet support its central claim.","major_comments":[{"comment":"The text states that reconstruction fidelity is enforced by a standard noise-prediction loss, but Algorithm 1 line 12 defines L_img = −MSE(ẑ, noise) and Eq. (4) minimizes the total loss L = L_img + λ1 L_retain + λ2 L_forget + λ3 L_reg. Minimizing −MSE maximizes the noise-prediction error on the forget prompt; this is not a standard noise-prediction loss and is never justified. Because the loss weights λ1, λ2, λ3 are not reported, the observed drop in Forget CLIP and Detection Rate could reflect general degradation rather than targeted unlearning. The authors must either correct the sign or explicitly justify the maximization, and they must report the loss weights.","section":"Section III-A, Algorithm 1 line 12, Eq. (4)"},{"comment":"The method optimizes quantities of the form (P + ΔP) f_f^T and (P + ΔP) f_r^T, treating P as the matrix that maps intermediate token features to final embeddings. In Stable Diffusion v2, the conditioning signal supplied to the U-Net is the token-level last hidden state of the CLIP text encoder, not the pooled or projected CLIP embedding. Since P is frozen and only ΔP and selected text-encoder layers are updated, L_forget and L_retain act on a projected output that is not directly consumed by the generative model. Gradients through f_f can still alter the unfrozen text-encoder layers, but the objective provides no mechanism to push the actual conditioning features in a semantically meaningful direction. The paper must demonstrate that the optimized projection is causally connected to generation, for example by ablating against a variant that optimizes the last hidden state directly.","section":"Section III-A, 'Prompt Conditioning with CLIP' and Algorithm 1"},{"comment":"The forget target F_forget ~ N(0,2.0) is arbitrary; no derivation, ablation, or sensitivity analysis is given for the target distribution or for the rank r (fixed at 8 in Section IV). The forget objective is MSE((P + ΔP) f_f^T, F_forget), so any observed reduction in forget-embedding similarity is by construction an optimization target rather than independent evidence. The detection rate is partially independent, but it is reported on only three concepts without error bars. A random target can degrade embeddings globally, so the authors need to ablate F_forget (for example, zero vector, a learned textual-inversion vector, or an orthogonal direction) and evaluate retain-set generation quality beyond a single CLIP score.","section":"Algorithm 1 line 4, Eq. (4)"},{"comment":"The evaluation conflates forgetting with image degradation. Section IV.A.2 states that a higher FID is desirable for unlearning, but high FID only indicates low similarity to the reference distribution and can result from general quality collapse; the reported FID values (151.6–343.0) are much larger than the baselines in Table II, yet retain CLIP is the only retain-quality metric and it is reported without variance. In addition, Section V.A says the forget loss should be as high as possible while the optimized L_forget is a squared error that is minimized to a random target; these contradictory statements make the loss graphs in Figure 7 uninterpretable. The paper should report FID on retain prompts, include standard deviations, specify the detector and prompts used for Detection Rate, and define exactly what the plotted losses represent.","section":"Section V.A and Table I"}],"minor_comments":[{"comment":"There are numerous typos and grammar issues, including 'componet', 'upto', 'Adversial', 'de-noisning', and 'co-relation'; the manuscript needs a careful proofread.","section":"Throughout"},{"comment":"The text says Stable Diffusion has 'three major components' and then lists four items (VAE, U-Net, text encoder, scheduler).","section":"Section II.B"},{"comment":"Table II appears without a clear lead-in sentence, and the baseline results are averaged across concepts without standard deviations or per-concept breakdowns, making comparison with Table I difficult.","section":"Section IV and Table II"},{"comment":"The manuscript does not state the number of forget and retain images used per concept, the prompt templates, the number of diffusion sampling steps, the classifier-free guidance scale, the optimizer settings, or the loss weights λ1, λ2, λ3; these are necessary for reproducibility.","section":"Section IV, reproducibility"},{"comment":"The notation is inconsistent: L_forget is defined both as a matrix-valued MSE in the text and as a scalar loss in Eq. (4); the dimensions of (P + ΔP) f_f^T and F_forget should be spelled out.","section":"Section III.A, Eq. (4)"}],"recommendation":"reject","confidential_remarks":"I do not see evidence of misconduct, but the manuscript is not ready for a serious venue in its current form. The central mechanism is architecturally questionable for Stable Diffusion v2, the loss sign is internally inconsistent, and the evaluation lacks the controls needed to distinguish selective unlearning from global degradation. A resubmission that addresses these points would need to be judged on a substantially reworked method and experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"At a glance, this is a modest but legitimate extension of Fuchi and Takagi's few-shot unlearning [8]: add a low-rank perturbation to the CLIP text encoder's projection matrix and use a weighted retain/forget/reg loss. The authors are honest about the backbone, and they do run experiments on three concepts with runtimes under a minute. The specific combination appears to be new, and the paper is easy to read. But the execution has two load-bearing problems that make the current claims unsupported.\n\nFirst, the paper contradicts itself on the image loss. Section III-A says a standard MSE noise-prediction loss is used, but Algorithm 1 line 12 sets L_img = -MSE(hat z, noise), and the later equations repeat the negative. Gradient descent on that term rewards worse noise prediction. If that is actually what was run, image quality should degrade globally, which would explain the odd results. If it's a typo, the manuscript needs fixing. Either way, the main metric table has no retain image quality, no error bars, and no code, so you cannot tell whether the concept is forgotten or the model is just broken.\n\nSecond, the mechanism is suspect. Stable Diffusion v2 conditions the UNet on the token-level hidden states of the CLIP text encoder, not on the pooled, projected CLIP embedding produced by the projection matrix P. The losses L_retain and L_forget act on (P+ΔP)f^T, so they optimize an auxiliary head that the UNet never reads. Gradients can reach the last encoder layers through f, but the forget target F_forget is just random Gaussian noise with no derivation. There is no reason to expect that to produce selective erasure rather than generic corruption of the hidden states. The generated images in the paper do show the concepts disappearing, but the examples look degraded (e.g., the Mickey prompt yields only gloves), and the FID interpretation is backwards: the authors claim a higher FID is desirable, but FID measures distributional similarity, and higher FID generally indicates worse image quality, not successful forgetting.\n\nI don't think this is a case of dishonest work; it reads like an early-stage student project that needs another pass. The literature engagement is fine, the idea has some potential, and the runtime claim is plausible. But as written, the sign error, the random target, and the lack of code/data/hyperparameters would let almost any result pass. I'd desk reject it and suggest the authors fix the loss, justify or replace the random target, add retain-quality metrics, and release code. The paper could then be a reasonable workshop submission; it's not ready for a serious venue.","headline":"Modest extension of few-shot unlearning, but a sign error in the loss and a questionable mechanism make the reported results uninterpretable.","tokens_in":9293,"tokens_out":3509,"would_cite":false,"duration_ms":37250,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that updating only the final layers of CLIP's text encoder with a low-rank weighted loss can make Stable Diffusion forget a concept in about 50 seconds using 4-5 images.","keywords":["machine unlearning","concept unlearning","diffusion models","low-rank adaptation","CLIP text encoder","few-shot learning","Stable Diffusion v2"],"falsifier":"Run the unlearning procedure on a Stable Diffusion v2 model for a concept such as Siberian Husky, once with $F_{\\text{forget}}$ drawn as $\\mathcal{N}(0,2.0)$ and once with $F_{\\text{forget}}$ set to a constant vector; if the concept disappears in both cases or unrelated prompts degrade in the first case, the random target is not semantically erasing the concept but simply corrupting the embedding space.","tokens_in":8295,"feed_emoji":"🧹","tokens_out":5826,"duration_ms":52295,"temperature":0.7,"pith_summary":"The paper proposes a fast concept-unlearning method for Stable Diffusion that works by updating the final layers of the CLIP text encoder with a weighted loss and low-rank adaptation. The central claim is that using 4-5 images of a concept, the model can be made to stop generating that concept in about 50 seconds, without retraining the U-Net or VAE. This matters because retraining diffusion models can take days, while privacy and copyright requests such as erasing a character like Mickey Mouse require quick, targeted removal.","feed_headline":"One-minute unlearning erases concepts from Stable Diffusion","feed_subtitle":"Low-rank CLIP edits erase Taj Mahal, Mickey Mouse, and Siberian Husky from SD v2 in about a minute.","key_machinery":"The load-bearing object is the low-rank perturbation $\\Delta P = AB^\\top$ of the CLIP text encoder's projection matrix, combined with a weighted loss over forget, retain, and image-reconstruction terms. The low rank (4-8) restricts changes to a small subspace so few images suffice and unrelated concepts are supposedly preserved; the retain loss anchors the projection for non-target prompts, and the forget loss drives target embeddings toward the random vector $F_{\\text{forget}}$.","core_discovery":"On its own terms, the paper's discovery is that forgetting a concept in a text-to-image diffusion model can be reduced to a small perturbation of the CLIP text encoder's final projection matrix $P$. The update is $\\Delta P = AB^\\top$ with $A,B \\in \\mathbb{R}^{d\\times r}$, $r$ between 4 and 8, learned by backpropagating a weighted loss that combines an image-noise reconstruction term, a retain loss keeping $(P+\\Delta P)f_r^\\top$ close to $Pf_r^\\top$, and a forget loss pushing $(P+\\Delta P)f_f^\\top$ toward a random normal target $F_{\\text{forget}} \\sim \\mathcal{N}(0,2.0)$. The authors report forget CLIP scores between 0.259 and 0.294, detection rates between 0 and 0.02, and runtimes around 50-58 seconds for Taj Mahal, Mickey Mouse, and Siberian Husky, with retain CLIP scores near 0.32-0.35.","pith_inferences":["The paper leaves implicit that the random normal target $F_{\\text{forget}}$ may be doing most of the work; if the same erasure happens with a zero or arbitrary fixed target, the method is likely corrupting the embedding rather than learning a semantic erasure.","The reported high FID scores are interpreted by the authors as successful distancing from the forget concept, but a high FID could also signal degraded image quality overall; comparing FID on unrelated prompts would separate those two readings.","The paper's own future-work note suggests a zero-shot variant by randomizing latents instead of encoding forget images; that points to a direct extension where the few-shot images become unnecessary if the random target alone remains sufficient."],"forward_implications":["If the claim holds, concept erasure in Stable Diffusion v2 becomes a one-minute operation using only a handful of images, replacing days of retraining.","Copyrighted or private concepts such as Mickey Mouse can be suppressed while prompts that mention only peripheral objects still generate those objects, as in the reported case of gloves without the mouse.","The method reports near-zero detection rates (0-0.02) and forget CLIP scores well below the baselines it compares against, implying the erased concept rarely appears in generations.","Because only the CLIP text encoder is modified, the same low-rank perturbation could be applied to other CLIP-conditioned diffusion pipelines without retraining their image decoders."],"supporting_citations":[{"why":"The few-shot text-encoder unlearning method this work extends; it supplies the approach of updating the CLIP text encoder using a small number of images.","marker":"[8]"},{"why":"Textual Inversion, the technique used to obtain concept embeddings from a few images.","marker":"[9]"},{"why":"Defines the diffusion forward/reverse process and the noise-prediction loss used as the image reconstruction term.","marker":"[10]"},{"why":"CLIP, the pretrained text encoder whose final projection matrix is perturbed.","marker":"[11]"},{"why":"EraseDiff baseline compared for unlearning time, CLIP score, FID, and detection rate.","marker":"[14]"},{"why":"Unified Concept Editing baseline compared for the same metrics.","marker":"[16]"}],"fun_headline_variants":["Low-rank text encoder tweak erases concepts in 50s","Few-shot unlearning via CLIP projection update","Tiny matrix edit removes concepts from SD v2","Few-shot low-rank unlearning erases concepts quickly"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim depends on the assumptions that a randomly initiated target vector for the forget loss erases the concept semantically and that updating only the final text-encoder layers is enough to stop the U-Net from generating it.","fun_headline_variants_meta":{"raw":{"variants":["Low-rank text encoder tweak erases concepts in 50s","Few-shot unlearning via CLIP projection update","Tiny matrix edit removes concepts from SD v2","Few-shot low-rank unlearning erases concepts quickly"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000966,"raw_usage":{"total_tokens":4163,"prompt_tokens":1049,"completion_tokens":3114,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":665,"completion_tokens_details":{"reasoning_tokens":3049}},"tokens_in":665,"tokens_out":3114,"duration_ms":21947,"temperature":1.0,"reasoning_tokens":3049,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:34:53.842686+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the unlearning procedure on a Stable Diffusion v2 model for a concept such as Siberian Husky, once with $F_{\\text{forget}}$ drawn as $\\mathcal{N}(0,2.0)$ and once with $F_{\\text{forget}}$ set to a constant vector; if the concept disappears in both cases or unrelated prompts degrade in the first case, the random target is not semantically erasing the concept but simply corrupting the embedding space.","supporting_citations":[{"cited_title":"Learning transferable visual models from natural language supervision,","cited_arxiv_id":null,"evidence_quote":"CLIP, the pretrained text encoder whose final projection matrix is perturbed."},{"cited_title":"Unified concept editing in diffusion models,","cited_arxiv_id":null,"evidence_quote":"Unified Concept Editing baseline compared for the same metrics."},{"cited_title":"Erasing concepts from text-to-image diffusion models with few-shot unlearning,","cited_arxiv_id":null,"evidence_quote":"The few-shot text-encoder unlearning method this work extends; it supplies the approach of updating the CLIP text encoder using a small number of images."}],"review_version":1}