REVIEW 3 major objections 5 minor 33 references
DuMo: Dual Encoder Modulation Network for Precise Concept Erasure
T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read DuMo erases concepts by tuning only the U-Net skip connections with the backbone frozen, preserving non-target images.
desk verdict DuMo's TLMO modulation is a genuine improvement for concept erasure preservation, but the SOTA claim rests on an incomplete baseline comparison and an internal contradiction about the preservation loss. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The Eraser with PRior Knowledge (EPR) module is a copy of the U-Net encoder blocks, connected to the original skip connection features through zero-initialized 1x1 convolutions (ZeroConv); the original U-Net parameters stay frozen and the original skip features are added to the EPR output. The Timestep-Layer MOdulation process (TLMO) introduces per-layer and per-timestep scalar factors $M_t^l$ that scale each EPR output, with the final skip feature computed as $x_t^l + M_t^l S_{t,\mathrm{cera}}^l$. The modulation factors are learned with the erasing loss plus a preservation loss on empty-prompt outputs. The key mechanism is the combination: freezing the backbone preserves low-frequency structure, while the prior-knowledge skip features and learned scaling localize the erasure to the high-frequency details that carry style and concept identity.
What would settle it
On a set of concepts that require structural change (e.g., changing a person's posture or removing an object while keeping the scene), measure whether DuMo's frozen backbone leaves the structure untouched while the target concept remains; if erasing such a concept forces visible structural corruption of non-target content, the high-frequency-only assumption is shown to fail.
Extended reading notes
Core claim
The central discovery is that concept erasure can be achieved almost entirely on the high-frequency detail path of a U-Net, not on the structural backbone. The paper shows that a plug-in EPR module, trained with the backbone frozen and the original skip features kept as prior knowledge, removes a target concept while preserving the structure of non-target objects. The subsequent TLMO analysis reveals that erasure effectiveness varies strongly by skip-connection layer and denoising timestep, so a set of learned modulation factors per layer and timestep yields the final trade-off. On the I2P nudity benchmark DuMo detects 34 nude body parts versus 41 for the next-best method, and on cartoon and style erasure it reports LPIPSda of 0.428 and 0.387 respectively, the highest among compared methods. The paper presents these results as state-of-the-art for the three tasks on Stable Diffusion v1.4.
Load-bearing premise
The method assumes that the concept to be erased lives entirely in the high-frequency detail stream (the skip connections) and that the low-frequency backbone structure can be left untouched; if erasing a concept like nudity requires structural changes, such as putting clothes on a character, the preservation advantage weakens.
Editorial extensions
If this is right
- Removing a concept leaves the model's general image-generation ability largely intact, because the backbone parameters never change.
- Multiple concepts can be erased simultaneously by summing the outputs of several EPR modules on the skip connections.
- The per-layer, per-timestep modulation factors provide a controllable way to dial the erasure strength without retraining the whole model.
- The two-stage training procedure (EPR then TLMO) gives a practical recipe for extending the method to new concepts: train the eraser branch first, then calibrate its scaling.
Reading between the lines
- The FreeU-based separation of high/low frequency is approximate; DuMo's success suggests that the skip-connection path is a natural 'style knob', so the same architectural intervention could be used for style transfer or attribute editing, not just erasure.
- Because the modulation factors are learned per layer and timestep, their final values (e.g., zero on layer group 4 for 'Van Gogh') could be interpreted as a signature of where a concept lives in the network, potentially serving as a diagnostic for concept localization.
- The method's second-stage preservation loss is applied only on empty prompts; extending it to a set of frequent non-target prompts could further improve preservation, especially for concepts that overlap with common visual categories.
- The reported 34 nude parts vs 41 for the next best is on a single detector (Nudenet) at 0.6 threshold; the ranking could shift if detector or threshold changes.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes DuMo, a two-stage concept erasure method for text-to-image diffusion models. In the first stage, an Eraser with Prior Knowledge (EPR) module is trained to modify only U-Net skip-connection features while the backbone is frozen; in the second stage, a Timestep-Layer Modulation (TLMO) process learns per-timestep and per-layer scaling factors for the EPR outputs. The method is evaluated on explicit content erasure using the I2P benchmark and Nudenet detection, on cartoon concept removal, and on artistic style erasure, where it reports the lowest number of detected nude body parts and the highest LPIPSda scores in most comparisons.
Significance. DuMo is an attractive plug-and-play design: it leaves the backbone parameters untouched, intervenes only on skip connections, and the authors provide code. If the reported numbers hold, the method would be a useful contribution to concept erasure because it offers a better preservation of non-target concepts while still erasing the target concept. The use of external metrics (Nudenet, CLIP Score, FID, LPIPS) on held-out prompts is a strength. However, the evidence for the headline state-of-the-art claim is weakened by the omission of the RECE baseline from two of the three benchmarks and by an ablation result that appears to contradict the claimed benefit of the TLMO module, so the current version does not fully substantiate the strongest conclusions.
major comments (3)
- [Sec. 4 (Tables 2 and 3)] The paper states that it compares with 10 baseline methods, including RECE (Gong et al. 2024), but RECE appears only in Table 1 and is absent from both the Cartoon Concept Removal and Artistic Style Erasure tables. Since RECE is a general concept-erasure method and is listed among the baselines, its omission leaves the state-of-the-art claim for those two benchmarks unsupported. The authors should include RECE in Tables 2 and 3 or explain explicitly why RECE cannot be applied to those tasks.
- [Sec. 4.4, Table 5] Table 5 reports LPIPSda values of 0.426 for EPR alone and 0.358 for EPR+TLMO on the 'Van Gogh' concept, so adding the TLMO modulation stage decreases the composite trade-off metric by 0.068. The decrease comes from a drop in LPIPSe from 0.459 to 0.383, i.e., weaker erasure, even though LPIPSu improves from 0.033 to 0.025. This contradicts the narrative that TLMO improves the erasure-preservation balance and that the joint use of timestep and layer factors attains the optimal trade-off. The authors need to reconcile the quantitative result with the claim or revise the claim.
- [Sec. 4.1 and Eq. (6)] The 'Results of Explicit Content Erasure' paragraph says that 'our method achieves superior preservation effects without incorporating any preservation loss,' but Eq. (6) defines the TLMO training loss as L = L_era2 + lambda * L_pre. Unless the authors used lambda = 0 for the nudity experiments, the statement is inconsistent with the method description. Please clarify whether and where the preservation loss was used, and how this affects the interpretation of the explicit-content results.
minor comments (5)
- [Sec. 4, first paragraph] The sentence says 'we compare ours with 10 baseline methods' but only nine baseline methods are enumerated: ESD, UCE, SLD-Med, SA, CA, SDD, RECE, MACE, and SPM. Please correct the count or add the missing method.
- [Throughout] There are several typographical errors: 'noval' should be 'novel', 'briefy' should be 'briefly', 'protential' should be 'potential', 'our gold' should be 'our goal' in the Method section, and 'MOdulation' should be 'Modulation' in the module name.
- [Sec. 3.1, Eq. (1)] Eq. (1) and Fig. 2(a) do not clearly specify the internal structure of the EPR module, particularly how the copied encoder block relates to the function E_cera and the zero-convolution layers. A short architectural description would improve reproducibility.
- [Sec. 3.3] The text introduces 'Timestep-Layer MOdulation process (TLMO)' but the abstract and some later passages use 'Time-Layer MOdulation'; please unify the terminology.
- [Sec. 4.2] The definition of LPIPSe and LPIPSu is implicit; please state explicitly that LPIPSe is computed against images generated by the unmodified model for erased concepts and LPIPSu against images for unerased concepts, and specify which images are used as references.
Circularity Check
No significant circularity; DuMo is evaluated on external benchmarks with parameters trained from stated losses.
full rationale
The paper's central claims (lowest Nudenet count, highest LPIPSda on cartoon and style erasure) are supported by evaluations on I2P, COCO-30K, and artist-prompt datasets that are external to the training losses in Eqs. (3)-(6). The EPR module and TLMO factors are trained to minimize noise-prediction losses, not fitted to the reported Nudenet or LPIPS values. The design premise that skip connections carry high-frequency detail and backbone features carry low-frequency structure is imported from FreeU (Si et al. 2024), an external prior work by different authors, so it is independent support rather than a self-citation chain. The only adjacent concern is that the authors' own RECE baseline appears in Table 1 but not in Tables 2-3; this is a missing-comparison / selective-reporting issue, not a case where a prediction reduces to its input by construction. No equation in the paper defines a reported outcome in terms of the benchmark quantity it is said to predict, and no load-bearing step relies on an unverified self-citation. Therefore the derivation is self-contained and non-circular.
Assumptions & free parameters
free parameters (4)
- TLMO modulation factors M_l^t =
learned, shown in Fig. 5
- Erasure strength eta =
not reported
- Preservation weight lambda =
not reported
- Nudenet detection threshold =
0.6
assumptions (4)
- domain assumption U-Net backbone features carry low-frequency structural information, while skip connection features carry high-frequency detail and style (FreeU).
- domain assumption The ESD erasing loss (Eq. 3) is a valid training objective for concept erasure even without tuning the backbone.
- domain assumption LPIPSda = Avg(LPIPSe) - Avg(LPIPSu) is a meaningful trade-off metric for erasure vs preservation.
- ad hoc to paper The preservation loss Lpre with weight lambda adequately regularizes non-target concepts during TLMO training.
Cite this review
Pith. "Pith review of DuMo: Dual Encoder Modulation Network for Precise Concept Erasure." pith.science (2026). https://pith.science/paper/HGATVBWC
@misc{pith2026250101125,
author = {Pith},
title = {Pith review of: DuMo: Dual Encoder Modulation Network for Precise Concept Erasure},
year = {2026},
howpublished = {\url{https://pith.science/paper/HGATVBWC}},
note = {Machine review of arXiv:2501.01125}
}
read the original abstract
The exceptional generative capability of text-to-image models has raised substantial safety concerns regarding the generation of Not-Safe-For-Work (NSFW) content and potential copyright infringement. To address these concerns, previous methods safeguard the models by eliminating inappropriate concepts. Nonetheless, these models alter the parameters of the backbone network and exert considerable influences on the structural (low-frequency) components of the image, which undermines the model's ability to retain non-target concepts. In this work, we propose our Dual encoder Modulation network (DuMo), which achieves precise erasure of inappropriate target concepts with minimum impairment to non-target concepts. In contrast to previous methods, DuMo employs the Eraser with PRior Knowledge (EPR) module which modifies the skip connection features of the U-NET and primarily achieves concept erasure on details (high-frequency) components of the image. To minimize the damage to non-target concepts during erasure, the parameters of the backbone U-NET are frozen and the prior knowledge from the original skip connection features is introduced to the erasure process. Meanwhile, the phenomenon is observed that distinct erasing preferences for the image structure and details are demonstrated by the EPR at different timesteps and layers. Therefore, we adopt a novel Time-Layer MOdulation process (TLMO) that adjusts the erasure scale of EPR module's outputs across different layers and timesteps, automatically balancing the erasure effects and model's generative ability. Our method achieves state-of-the-art performance on Explicit Content Erasure, Cartoon Concept Removal and Artistic Style Erasure, clearly outperforming alternative methods. Code is available at https://github.com/Maplebb/DuMo
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Dhariwal, P.; and Nichol, A. 2021. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34: 8780--8794
2021
-
[4]
Fan, C.; Liu, J.; Zhang, Y.; Wei, D.; Wong, E.; and Liu, S. 2023. Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation. arXiv preprint arXiv:2310.12508
arXiv 2023
-
[5]
Gandikota, R.; Materzynska, J.; Fiotto-Kaufman, J.; and Bau, D. 2023. Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2426--2436
work page 2023
-
[6]
Gandikota, R.; Orgad, H.; Belinkov, Y.; Materzy \'n ska, J.; and Bau, D. 2024. Unified concept editing in diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 5111--5120
work page 2024
-
[7]
Gong, C.; Chen, K.; Wei, Z.; Chen, J.; and Jiang, Y.-G. 2024. Reliable and Efficient Concept Erasure of Text-to-Image Diffusion Models. arXiv preprint arXiv:2407.12383
arXiv 2024
-
[8]
Heng, A.; and Soh, H. 2024. Selective amnesia: A continual learning approach to forgetting in deep generative models. Advances in Neural Information Processing Systems, 36
work page 2024
Show all 33 references
-
[9]
L.; and Choi, Y
Hessel, J.; Holtzman, A.; Forbes, M.; Bras, R. L.; and Choi, Y. 2021. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718
2021 arXiv
-
[10]
Huang, C.-P.; Chang, K.-P.; Tsai, C.-T.; Lai, Y.-H.; and Wang, Y.-C. F. 2023. Receler: Reliable concept erasing of text-to-image diffusion models via lightweight erasers. arXiv preprint arXiv:2311.17717
2023 arXiv
-
[11]
Hunter, T. 2023. AI porn is easy to make now. For women, that's a nightmare. The Washington Post, NA--NA
2023
-
[12]
H.; Brown, L.; Cheng, J.; Khan, M.; Gupta, A.; Workman, D.; Hanna, A.; Flowers, J.; and Gebru, T
Jiang, H. H.; Brown, L.; Cheng, J.; Khan, M.; Gupta, A.; Workman, D.; Hanna, A.; Flowers, J.; and Gebru, T. 2023. AI Art and its Impact on Artists. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, 363--374
2023
-
[13]
Kim, S.; Jung, S.; Kim, B.; Choi, M.; Shin, J.; and Lee, J. 2023. Towards safe self-distillation of internet-scale text-to-image diffusion models. arXiv preprint arXiv:2307.05977
2023 arXiv
-
[14]
Kumari, N.; Zhang, B.; Wang, S.-Y.; Shechtman, E.; Zhang, R.; and Zhu, J.-Y. 2023. Ablating concepts in text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 22691--22702
2023
-
[15]
Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Doll \'a r, P.; and Zitnick, C. L. 2014. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 1...
2014
-
[16]
Lu, S.; Wang, Z.; Li, L.; Liu, Y.; and Kong, A. W.-K. 2024. Mace: Mass concept erasure in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6430--6440
2024
-
[17]
Luo, W.; Hu, T.; Zhang, S.; Sun, J.; Li, Z.; and Zhang, Z. 2024. Diff-instruct: A universal approach for transferring knowledge from pre-trained diffusion models. Advances in Neural Information Processing Systems, 36
2024
-
[18]
Lyu, M.; Yang, Y.; Hong, H.; Chen, H.; Jin, X.; He, Y.; Xue, H.; Han, J.; and Ding, G. 2024. One-dimensional Adapter to Rule Them All: Concepts Diffusion Models and Erasing Applications. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7559--7568
2024
-
[19]
Nichol, A.; Dhariwal, P.; Ramesh, A.; Shyam, P.; Mishkin, P.; McGrew, B.; Sutskever, I.; and Chen, M. 2021. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741
2021 arXiv
-
[20]
Q.; and Dhariwal, P
Nichol, A. Q.; and Dhariwal, P. 2021. Improved denoising diffusion probabilistic models. In International conference on machine learning, 8162--8171. PMLR
2021
-
[21]
Parmar, G.; Zhang, R.; and Zhu, J.-Y. 2022. On aliased resizing and surprising subtleties in gan evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11410--11420
2022
-
[22]
W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al
Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, 8748--8763. PMLR
2021
-
[23]
Ramesh, A.; Dhariwal, P.; Nichol, A.; Chu, C.; and Chen, M. 2022. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2): 3
2022 arXiv
-
[24]
Rando, J.; Paleka, D.; Lindner, D.; Heim, L.; and Tram \`e r, F. 2022. Red-teaming the stable diffusion safety filter. arXiv preprint arXiv:2210.04610
2022 arXiv
-
[25]
L.; Ghasemipour, K.; Gontijo Lopes, R.; Karagol Ayan, B.; Salimans, T.; et al
Saharia, C.; Chan, W.; Saxena, S.; Li, L.; Whang, J.; Denton, E. L.; Ghasemipour, K.; Gontijo Lopes, R.; Karagol Ayan, B.; Salimans, T.; et al. 2022. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing syste...
2022
-
[26]
Schramowski, P.; Brack, M.; Deiseroth, B.; and Kersting, K. 2023. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 22522--22531
2023
-
[27]
Schuhmann, C.; Beaumont, R.; Vencu, R.; Gordon, C.; Wightman, R.; Cherti, M.; Coombes, T.; Katta, A.; Mullis, C.; Wortsman, M.; et al. 2022. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems,...
2022
-
[28]
Setty, R. 2023. Ai art generators hit with copyright suit over artists’ images. Bloomberg Law. Accessed on February, 1: 2023
2023
-
[29]
Shi, Z.; Zhou, X.; Qiu, X.; and Zhu, X. 2020. Improving image captioning with better use of captions. arXiv preprint arXiv:2006.11807
2020 arXiv
-
[30]
Si, C.; Huang, Z.; Jiang, Y.; and Liu, Z. 2024. Freeu: Free lunch in diffusion u-net. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4733--4743
2024
-
[31]
Smith, M. 2022. Tutorial: How to remove the safety filter in 5 seconds
2022
-
[32]
Zhang, L.; Rao, A.; and Agrawala, M. 2023. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 3836--3847
2023
-
[33]
Zhang, Y.; Jia, J.; Chen, X.; Chen, A.; Zhang, Y.; Liu, J.; Ding, K.; and Liu, S. 2023. To generate or not? safety-driven unlearned diffusion models are still easy to generate unsafe images... for now. arXiv preprint arXiv:2310.11868
2023 arXiv
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.