REVIEW 4 major objections 5 minor 1 cited by
The Linear Geometry of Interpretable Tokens: Jailbreaking Attacks and Defenses for Unlearned Diffusion Models
T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Unlearned diffusion models still tie harmful concepts to hidden words
desk verdict SubAttack is a solid transferable embedding attack and SubDefense a useful defense, but the paper's claim that unlearned models retain concepts via specific implicit tokens is not causally established. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the CLIP token embedding space of the frozen text encoder. The attack learns a token embedding $v_{\text{att}} = \sum_i \alpha_i v_i$ with non-negative coefficients $\alpha_i = g_\Theta(v_i)$ produced by an MLP, so the learned embedding is interpretable as a weighted combination of existing vocabulary tokens. Orthogonality is enforced through deflation: after learning one embedding, each vocabulary token is projected to the subspace orthogonal to it, and the next embedding is learned on the projected vocabulary, yielding a set of orthogonal attack embeddings spanning a low-dimensional subspace. The defense (SubDefense) applies the same projection idea to every token in the vocabulary: $v_{\text{def},i} = v_i - \mathrm{Proj}_{V_{\text{att}}}(v_i)$, removing the learned attack subspace from all token embeddings.
What would settle it
A direct disproof would be to retrain the CLIP text encoder or replace it (e.g., with a fine-tuned text encoder) while keeping the same unlearned UNet, and then check whether the SubAttack token embeddings learned on the original CLIP space still achieve high attack success rates on the unlearned model; if the attack success drops to near the no-attack baseline, then the residual vulnerability is not inherent to the token embedding space but an artifact of the frozen encoder. Alternatively, one could remove only the top-weighted interpretable tokens identified for a concept (e.g., 'slave', 'nip' for nudity) from the vocabulary and measure whether the attack success rate falls as much as when the entire learned subspace is projected out; if the ASR does not drop, then the interpretable tokens are not the causal carriers of the residual concept.
Extended reading notes
Core claim
The central claim is that an unlearned text-to-image diffusion model has not actually forgotten the erased concept; it has merely suppressed its explicit textual associations, while the concept remains accessible through a low-dimensional subspace of implicit associations. These implicit associations can be learned as orthogonal attack token embeddings, each expressible as a non-negative linear combination of interpretable CLIP vocabulary tokens (bag-of-words style), and these embeddings achieve high attack success rates across prompts, initial noises, and unlearned models. Moreover, the same embeddings transfer back to the original Stable Diffusion model with high success, indicating that the residual vulnerability is inherited from the base model rather than created by unlearning. The paper then shows that projecting the CLIP vocabulary onto the null space of these learned attack embeddings substantially reduces the attack success rate of both their own SubAttack and other jailbreaking methods (UnlearnDiff, CCE) while maintaining generation quality.
Load-bearing premise
The load-bearing premise is that the residual vulnerability of an unlearned model is fully captured in the shared CLIP token embedding space and can be excised by projecting every vocabulary vector onto the null space of a set of attack embeddings learned on that same space; this premise is not established because the text encoder is frozen and identical across all victim models, making the attack embeddings live in the same space regardless of the UNet's unlearning.
Editorial extensions
If this is right
- Existing unlearning methods that only fine-tune the denoising UNet while freezing the text encoder leave the target concept accessible in the token embedding space; a defense that modifies the token embedding vocabulary directly can reduce attack success rates across multiple attack types.
- Attack token embeddings learned from one unlearned model transfer to other unlearned models and back to the original Stable Diffusion model, implying that robustness evaluations should consider transfer-based attacks rather than only per-model optimization.
- The interpretability of the attack embeddings provides a diagnostic tool: unlearned models with higher explicit-token retention (e.g., FMN) are more vulnerable to simple attacks, while those with more implicit associations (e.g., ESD, UCE, SPM) still fail under the subspace attack.
- The defense is plug-and-play: it can be applied on top of existing unlearned models without retraining the UNet, and it preserves generation quality on safe prompts better than the RECE baseline.
- Defending against CCE (which learns unconstrained token embeddings) is harder than defending against attacks tied to the vocabulary, and requires blocking more tokens at the cost of utility—pointing to a trade-off that future defenses must address.
Reading between the lines
- If the residual vulnerability is truly inherited from the frozen text encoder, then improving unlearning robustness may require modifying or replacing the CLIP text encoder itself, rather than only fine-tuning the UNet; this follows from the paper's finding that attack embeddings transfer back to the original SD model.
- The paper's interpretation of implicit associations (e.g., 'slave', 'nip' for nudity) is based on vocabulary tokens with high weights, but the defense removes the entire learned subspace rather than isolating these specific tokens; a testable extension would be to remove only the top-weighted interpretable tokens and see whether the attack success rate drops proportionally.
- Because the attack embeddings are learned on the frozen CLIP space, the same embedding could potentially be transferred to other text-to-image models that share or approximate the CLIP text encoder, extending the attack beyond the specific unlearned SD models tested.
- The sparsity analysis (top few hundred tokens carry most of the weight) suggests that a sparse, token-level defense might be possible—blocking only dozens of tokens instead of hundreds could preserve utility while still cutting the largest attack channels, a direction the paper leaves implicit.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper addresses jailbreaking and defense for unlearned text-to-image diffusion models. The authors propose SubAttack, which learns K orthogonal token embeddings in the frozen CLIP token embedding space, each constrained to be a non-negative superposition of vocabulary embeddings. Replacing the target concept word with these learned tokens regenerates unlearned concepts (nudity, Van Gogh, church, etc.) across ESD, FMN, UCE, and SPM. The paper claims the learned embeddings are interpretable in a bag-of-words sense and that word-cloud inspection reveals unlearned models retain the target concept through implicit textual associations (e.g., 'slave', 'nip', 'babes'). It further proposes SubDefense, which projects every CLIP vocabulary embedding onto the null space of the learned attack embeddings, and reports reduced ASR against SubAttack, UnlearnDiff, and CCE while preserving FID/CLIP on COCO-10k. The appendices include transfer tables, ablations on K and on number of blocked tokens, and visualizations.
Significance. If the claims hold, SubAttack is a useful global, transferable attack for auditing concept erasure, and SubDefense is a simple plug-in defense. Strengths: the attack evaluation spans four unlearned models and six concepts; the transfer tables and I2P defense results are extensive; and the paper explicitly attempts to close the attack-defense loop. The paper does not ship code or data, and no error bars or significance tests are reported. The conceptual contribution—that residual vulnerability is carried by human-interpretable implicit tokens—is currently not causally established, and the defense is evaluated largely against the same embeddings used to construct it. The manuscript's value would be materially higher if those gaps were closed.
major comments (4)
- [4.1, C.1, Tables 1–2] The ASR comparison is asymmetric in inference budget. For each prompt-seed pair, SubAttack generates K=5 images (one per attack embedding) and counts success if any of the five contains the concept, whereas CCE generates one image from a single learned embedding and UnlearnDiff is evaluated with its optimized prompt. The headline claim that SubAttack 'surpasses CCE in many circumstances' (Section 4.2) may therefore partly reflect the fivefold generation chance rather than a genuinely stronger attack. Please report per-embedding ASR and per-image hit rate, and provide a budget-matched comparison such as allowing CCE or a single SubAttack embedding to draw multiple seeds.
- [4.2, Fig. 3, Eq. (2), Fig. 13] The central interpretability claim—that unlearned models retain the target concept through implicit textual components such as 'slave', 'nip', and 'babes'—lacks causal evidence. In Eq. (2), v_att is a non-negative combination over the entire CLIP vocabulary, and SubDefense in Eq. (4) removes the full learned subspace V_att. No experiment isolates the displayed implicit tokens by ablating only those entries of alpha and measuring the resulting ASR change; Figure 13 is a sufficiency check (top ~1500 entries preserve the attack), not a necessity check for the handful of tokens highlighted in Figure 3. The ASR reductions in Tables 3, 4, and 17–20 are equally consistent with removal of a distributed, non-interpretable concept direction, so the paper's interpretive centerpiece is not currently established.
- [3.2, Tables 3–4] The defense evaluation is partly circular: SubDefense is constructed from exactly the attack embeddings it is then evaluated against. In Table 3, for UCE/nudity, SubAttack ASR drops from 81.67% to 34.11% after 20 blocked tokens, but those are the same tokens removed by Eq. (4). The independent evidence is weaker: UCE/nudity UnlearnDiff ASR drops only from 78.22% to 73.55% (Table 11), and on ESD, CCE drops from 85.11% to 75.67% (Table 4). To support the claim that the defense 'can defend unlearned models against various attacks,' the authors should evaluate against attacks re-optimized after the defense is applied, or hold out concepts and embeddings that were not used to build the projection.
- [4.2, Table 5, D.1] The 'inherited from the original SD' transfer argument is near-tautological given the fixed text encoder. The CLIP text encoder and token embedding table are frozen and identical across SD v1.4 and all victim models, so any token embedding learned in that space is by construction a valid embedding for the original SD; the high transfer ASRs in Table 5 (84–98%) may simply reflect that the learned vector is a good language-side trigger for the shared text encoder, independent of UNet unlearning. This does not by itself establish that the unlearned UNet retains a concept subspace. A control with a model whose text encoder has been modified, or a comparison of the same embeddings' ASR before and after UNet fine-tuning, is needed.
minor comments (5)
- [All tables] No error bars, confidence intervals, or significance tests are reported for ASR, FID, or CLIP scores. Differences of a few points, such as the Nudity UnlearnDiff row in Table 3 (73.55% vs 76.44%), should be accompanied by variance estimates across seeds or dataset subsamples.
- [Tables 11–12] The relative reduction arithmetic is inconsistent; for example, Church UnlearnDiff in Table 11 goes from 61.67% to 39.78% but is labeled '-64.34%', whereas the relative reduction is about 35.5%. Please use percentage-point changes or relative percentages consistently and label them clearly.
- [4.1] Section 4.1 has two items labeled '(iv)', with the Baselines paragraph as the second; the second should be '(v)'.
- [Throughout] No public code or dataset release is mentioned. Given the described dataset (900 prompt-seed pairs per main concept, six times larger than UnlearnDiff's) and the volume of reported numbers, releasing the constructed dataset and attack/defense code would substantially aid reproducibility.
- [F.2, J] The paper acknowledges in Section F.2 that interpretability relies on the predefined CLIP vocabulary and may miss nuanced retained representations; this important limitation should be stated in the main-text discussion rather than only in an appendix.
Circularity Check
Partial circularity: SubDefense's headline SubAttack results remove the very attack directions used in evaluation, and the 'implicit textual components' insight is largely imposed by the nonnegative-combination ansatz.
-
fitted input called prediction
[Section 3.2 (Eq. 4); Section 5.1; Table 4]
"The proposed defense will “block” the subspace spanned by V_att to defend against various jailbreaking attacks. Each token embedding v_i in V will be updated as follows: v_def,i = v_i − Proj_{V_att}(v_i), ∀i ∈ [N]. ... SubAttack with K=5 is used consistently before and after defense to ensure a fair comparison."
The defense projects every vocabulary vector onto the null space of V_att, where V_att is learned by the same SubAttack procedure and the post-defense evaluation reuses the same K=5 SubAttack embeddings. Consequently, the attack directions are zeroed out by construction, so the large SubAttack ASR drops in Tables 3 and 4 (e.g., UCE nudity 81.67% to 34.11%; ESD nudity 97.56% to 42.33%) are not independent evidence. The genuinely independent evidence is the effect on UnlearnDiff and CCE, which is real but weaker.
-
self definitional
[Section 3.1.1 (Eq. 2 and Remarks); Section 4.2; Table 23 / Fig. 13]
"vatt = PN i=1 αivi, αi = gΘ(vi) ≥ 0 ... Adopting the constraint in (2) following [22] further enables the “hidden word” to be interpretable in a bag-of-words manner."
Because v_att is defined as a nonnegative linear combination of existing CLIP token embeddings, its expression as a bag of vocabulary words is guaranteed by the ansatz, not discovered. The further claim that unlearned models retain the concept through implicit textual components such as “slave”, “nip”, and “babes” is a word-cloud reading of the same coefficients. The paper never isolates these displayed tokens as causal: Table 23 reports that about 1,455 coefficients are needed for the nudity attack, and Fig. 13 shows roughly 1,500 nonzero entries are required to sustain it. Thus the interpretability centerpiece largely restates the parameterization rather than demonstrating that the highlighted implicit tokens drive the residual vulnerability.
1 more flagged steps
-
other
[Section 4.2; Appendix C.1; Table 5]
"these unlearned models are fine-tuned on Stable Diffusion v1.4, and hence, they share the same CLIP text encoders. ... The transfer ASR turns out to be high, consistently being larger than 80% across all different concepts and models."
Since Eq. (2) confines every attack embedding to the shared CLIP token embedding space, and the text encoder is frozen and identical across SD v1.4 and all victim models, transferring the embeddings back to SD is largely a test of how the original SD UNet responds to a vector in the common vocabulary. The high Table 5 transfer ASR does not establish that the unlearned model’s residual vulnerability is specifically “inherited from the original SD”; any successful concept direction in the shared space would behave this way. The “inherited” conclusion is therefore partly a renaming of the design constraint plus an ASR measurement.
full rationale
The paper is not globally circular: SubAttack is a genuine new attack with independent empirical effectiveness and transferability results, and SubDefense’s effects on external attacks (UnlearnDiff, CCE) are not derived from the method’s definitions. The circularity is partial and localized. First, the headline defense evaluation against SubAttack removes the exact subspace used to construct the test attack, so those ASR reductions are forced by construction; the independent attack rows provide the real evidence for defense generality. Second, the interpretability discovery is partly self-definitional because v_att is by definition a nonnegative combination of vocabulary embeddings, and the paper’s own sparsity analysis shows that the attack is a broad direction rather than the few displayed implicit tokens. Third, the transfer-to-SD argument is weakened by the shared frozen CLIP encoder, making the “inherited from SD” conclusion closer to a restatement of the setup than an empirical discovery. No load-bearing self-citation chain or uniqueness theorem is used, so the paper does not reach score 8-10; the independent attack and external-defense results keep it at a moderate partial-circularity level.
Assumptions & free parameters
free parameters (2)
- K: number of attack token embeddings =
5 for attack; 100 for defense; 20 for RECE comparison
- Number of blocked tokens in SubDefense =
20, 100, or up to 390 in CCE ablation
assumptions (4)
- domain assumption The non-negative linear combination of CLIP token embeddings (Eq. 2) can express effective attack vectors, borrowed from prior concept-decomposition work.
- domain assumption All victim unlearned models share the frozen CLIP text encoder with SD v1.4.
- domain assumption The denoising loss (Eq. 1) with target-concept images is a sufficient objective to learn a hidden word for that concept.
- domain assumption Pretrained classifiers (NudeNet, WikiArt fine-tuned, ResNet-50) are valid proxies for the target concept's presence.
Cite this review
Pith. "Pith review of The Linear Geometry of Interpretable Tokens: Jailbreaking Attacks and Defenses for Unlearned Diffusion Models." pith.science (2026). https://pith.science/paper/GIYDQMDF
@misc{pith2026250421307,
author = {Pith},
title = {Pith review of: The Linear Geometry of Interpretable Tokens: Jailbreaking Attacks and Defenses for Unlearned Diffusion Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/GIYDQMDF}},
note = {Machine review of arXiv:2504.21307}
}
read the original abstract
Diffusion models excel at generating high-quality images but can memorize and reproduce harmful concepts when prompted. Although fine-tuning methods have been proposed to unlearn a target concept, they struggle to fully erase it while maintaining generation quality on other concepts, leaving models vulnerable to jailbreak attacks. Existing jailbreak methods demonstrate this vulnerability but offer limited insight into how unlearned models retain harmful concepts, limiting progress on effective defenses. In this work, we show that the erased concept persists as a coherent, interpretable linear subspace of the token embedding space, and that both an attack and a defense follow directly from this structure. We introduce SubAttack, a novel jailbreaking attack that reads out this subspace by learning an orthogonal set of attack token embeddings, each being a linear combination of human-interpretable textual elements, revealing that unlearned models still retain the target concept through related textual components. Furthermore, our attack is also more powerful and transferable across text prompts, initial noises, and unlearned models than prior attacks. Conversely, projecting out the same subspace yields SubDefense, a lightweight plug-and-play defense mechanism that suppresses the residual concept in unlearned models. SubDefense provides stronger robustness than existing defenses while better preserving safe generation quality. Extensive experiments across multiple unlearning methods, concepts, and attack types demonstrate that our approach advances both understanding and mitigation of vulnerabilities in diffusion unlearning.
Figures
Figures from the paper (26 more)
Forward citations
Cited by 1 Pith paper
-
Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks
A system-first taxonomy and literature synthesis of multimodal unlearning across vision, language, video, and audio, with datasets, benchmarks, metrics, applications, and open challenges.
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.