Localizing Memorized Regions in Diffusion Models via Coordinate-Wise Curvature Differences

Gwangho Kim; Sungyoon Lee

arxiv: 2605.26756 · v2 · pith:F43XOTZTnew · submitted 2026-05-26 · 💻 cs.LG

Localizing Memorized Regions in Diffusion Models via Coordinate-Wise Curvature Differences

Gwangho Kim , Sungyoon Lee This is my paper

Pith reviewed 2026-06-29 19:54 UTC · model grok-4.3

classification 💻 cs.LG

keywords diffusion modelsmemorizationlocalizationcurvaturevariance collapseoverfittingStable Diffusion

0 comments

The pith

Curvature differences with an underfitted baseline localize memorized regions in diffusion model generations.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper characterizes local memorization in diffusion models as coordinate-wise variance collapse in the generated image. To separate this from intrinsic data constraints, it subtracts the curvature of a baseline model that is either unconditional or less trained. This yields a method for localizing memorized areas at the coordinate level. The approach also gives a geometric account of why score-difference metrics detect memorization. On Stable Diffusion, the curvature-difference localization beats the prior attention-based technique when measured against ground-truth masks.

Core claim

Local memorization in diffusion models manifests as coordinate-wise variance collapse that can be isolated from data-intrinsic effects by computing curvature differences relative to an underfitted baseline, either the unconditional model or a less-trained checkpoint.

What carries the argument

Coordinate-wise curvature differences, obtained by subtracting the curvature computed on an underfitted baseline from that of the trained model, to highlight overfitting-induced memorization.

If this is right

The method supplies a per-coordinate map of memorized content within an image.
It offers a geometric rationale for the effectiveness of score-difference detection.
Localization performance exceeds that of attention-map methods on Stable Diffusion.
Applicable to privacy audits by revealing which parts of outputs are memorized.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the baseline truly underfits, the method could generalize to detect memorization in other generative architectures.
Choosing different baselines might change which regions are flagged as memorized versus data-constrained.
Curvature-based signals could be combined with other geometric measures for more robust detection.

Load-bearing premise

Subtracting curvature from an underfitted baseline isolates overfitting-driven memorization rather than intrinsic data constraints.

What would settle it

If curvature-difference maps fail to align with ground-truth memorization masks on Stable Diffusion more closely than the attention baseline, the localization advantage would not hold.

Figures

Figures reproduced from arXiv: 2605.26756 by Gwangho Kim, Sungyoon Lee.

**Figure 1.** Figure 1: Qualitative comparison of memorization localization. For each example, we show the training image, the generated image, and the ground-truth memorization mask. We compare the proposed curvature-difference-based metrics ∆h∅ (Eq. 1) and ∆hθ˜ (Eq. 2), together with their score-difference-based surrogates ∆s∅ (Eq. 5) and ∆sθ˜ (Eq. 6), against the prior work, Bright Ending (BE) (Chen et al., 2025). Light region… view at source ↗

**Figure 2.** Figure 2: a, variability is distributed across all coordinates, resulting in relatively uniform coordinate-wise covariance. In contrast, in Figure 2b, the same two degrees of freedom are concentrated on a specific subset of coordinates, while the remaining coordinates are fixed. Although both distributions have identical intrinsic dimensionality, the latter leaves the two pixels fixed, providing a strong signal of … view at source ↗

**Figure 3.** Figure 3: Curvature-difference isolates overfitting-driven memorization. Heatmaps within each column share the same color scale, with light and dark regions indicating high and low curvature (or differences), respectively. Left: Generated samples; training instances are highlighted with red borders. Middle: Coordinate-wise curvature diag(−Hθ(xt, c)) computed at the final sampling step (t ≈ 0) of the first generate… view at source ↗

**Figure 4.** Figure 4: Qualitative results for SD v1.4. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗

**Figure 5.** Figure 5: Qualitative results for SD v2.1. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗

**Figure 6.** Figure 6: Synthetic experiment on curvature dynamics under progressive overfitting. (a) Training trajectories of the coordinate-wise curvature κ1(x) = (−∇xsθ(x, teval))11, measured at the duplicated outlier center xdup and at a representative rank-1 noisy manifold sample x1d. While κ1(x1d) rapidly saturates at the ground-truth curvature κ ⋆ 1, κ1(xdup) continues to increase throughout training. Notably, the model al… view at source ↗

**Figure 7.** Figure 7: Synthetic experiment on Curvature dynamics of κ1 at timesteps t = 20, 200, 800. E.2. Sensitivity to the Choice of the Baseline Model To evaluate the robustness of our framework with respect to the choice of the less-trained baseline (˜θ), we conduct an additional ablation study using Stable Diffusion v1.2 as an alternative baseline for the SD v1.4 target model. While SD v1.1 (used in our main experiments) … view at source ↗

read the original abstract

Diffusion models can unintentionally memorize training samples, raising concerns about privacy and copyright. While recent methods can detect memorization, they often rely on global or model-specific signals and provide limited insight into where memorization appears within a generated image. We provide a geometric characterization of local memorization as a coordinate-wise variance collapse. However, such collapse can also arise from intrinsic data constraints rather than overfitting. To isolate overfitting-driven memorization, we propose curvature-difference methods that subtract the curvature of an underfitted baseline, either the unconditional model or a less-trained version of itself. We further derive a score-difference proxy that provides a geometric explanation for the widely used score-difference-based detection metric. Experiments on Stable Diffusion, evaluated against ground-truth memorization masks, show that our method outperforms the prior attention-based localization method. Code is available at https://github.com/Gwangho99/mem-curv-diff.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper gives a geometric framing for localizing memorization via curvature subtraction from underfitted baselines, but the abstract supplies no numbers or baseline checks to back the outperformance claim.

read the letter

The main point is a method that treats local memorization in diffusion models as coordinate-wise variance collapse and subtracts curvature from an unconditional or less-trained baseline to isolate the overfitting part.

What the work does cleanly is supply a geometric story for why score-difference detection works and then test the resulting localization against ground-truth masks on Stable Diffusion. It reports beating the prior attention-based approach, and the code is released, which lets others inspect the implementation directly.

The soft spots sit in the experimental grounding. The abstract states the outperformance but gives no quantitative results, no error analysis, and no description of how the ground-truth masks were built or how the baselines were confirmed to lack the same collapse patterns. The central assumption—that the underfitted model removes only data-intrinsic effects—remains unverified in the text provided, so any reported gain could partly reflect residual non-memorization signal.

This is aimed at researchers who audit privacy and copyright issues in deployed generative models. Readers who already work on detection metrics or geometric views of overfitting will get the most from it. The idea is practical enough and the geometric angle fresh enough that it deserves a serious referee, though the authors will need to add the missing quantitative checks and baseline validation before it can be evaluated properly.

Referee Report

2 major / 1 minor

Summary. The paper claims that local memorization in diffusion models can be characterized geometrically as coordinate-wise variance collapse. To isolate overfitting-driven memorization from intrinsic data constraints, it proposes curvature-difference methods that subtract curvature computed on an underfitted baseline (unconditional model or less-trained checkpoint). It further derives a score-difference proxy that geometrically explains an existing detection metric. Experiments on Stable Diffusion, evaluated against ground-truth memorization masks, are reported to show outperformance over a prior attention-based localization method. Code is released.

Significance. If the experimental results hold and the baseline-subtraction premise is independently validated, the work supplies a geometric account of memorization localization together with a practical improvement over attention baselines. Reproducibility is aided by the public code release. The contribution would be relevant to privacy and copyright concerns in generative modeling.

major comments (2)

[Abstract] Abstract: the claim that curvature subtraction from an underfitted baseline isolates overfitting-driven memorization (rather than intrinsic data-driven collapse) is load-bearing for the reported outperformance against attention baselines, yet the manuscript provides no verification that the chosen baselines lack comparable coordinate-wise variance reduction on the same coordinates.
[Abstract] Abstract: the derived score-difference proxy is asserted to furnish a geometric explanation of the widely used score-difference metric, but the text does not clarify whether the proxy is independent or reduces to the original metric by construction; this circularity risk directly affects the claimed explanatory value.

minor comments (1)

[Abstract] Abstract: quantitative results, error bars, baseline implementation details, and ground-truth mask construction procedure are absent, preventing assessment of effect sizes and experimental robustness.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on the abstract. We address each major comment below, agreeing where clarification or additional evidence is needed and outlining the planned revisions.

read point-by-point responses

Referee: [Abstract] Abstract: the claim that curvature subtraction from an underfitted baseline isolates overfitting-driven memorization (rather than intrinsic data-driven collapse) is load-bearing for the reported outperformance against attention baselines, yet the manuscript provides no verification that the chosen baselines lack comparable coordinate-wise variance reduction on the same coordinates.

Authors: We agree this verification is important for the isolation claim. The manuscript currently relies on the underfitting design of the baselines (unconditional model or earlier checkpoints) without direct empirical checks on the memorized coordinates. In the revision we will add quantitative comparisons of coordinate-wise curvature and variance collapse between the trained model and both baselines on the ground-truth memorization masks, confirming that the baselines do not exhibit comparable reduction. revision: yes
Referee: [Abstract] Abstract: the derived score-difference proxy is asserted to furnish a geometric explanation of the widely used score-difference metric, but the text does not clarify whether the proxy is independent or reduces to the original metric by construction; this circularity risk directly affects the claimed explanatory value.

Authors: The proxy is obtained by algebraic rearrangement of the curvature-difference expression and is shown to approximate the score-difference metric; the derivation is independent and supplies the geometric reason the metric succeeds. It is not identical by construction. We will revise the relevant section to present the full derivation steps explicitly and state the independence of the proxy from the original metric. revision: yes

Circularity Check

0 steps flagged

No significant circularity; derivation and proxy remain independent of inputs

full rationale

The abstract presents a geometric characterization of memorization as coordinate-wise variance collapse, proposes curvature subtraction from underfitted baselines, and derives a score-difference proxy as an explanation for an existing detection metric. No equations, self-citations, or definitions are supplied that reduce the proxy or difference map to the input metric by construction. The experimental comparison against ground-truth memorization masks supplies independent falsifiable grounding outside any fitted parameters or prior author results, keeping the chain self-contained.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Abstract-only review; no explicit free parameters, axioms, or invented entities are stated. The approach relies on standard diffusion model assumptions and curvature definitions from prior literature.

axioms (1)

domain assumption Curvature can be meaningfully computed coordinate-wise on diffusion model outputs and baselines.
Invoked when defining the curvature-difference signal; location not specified beyond abstract description.

pith-pipeline@v0.9.1-grok · 5679 in / 1072 out tokens · 22825 ms · 2026-06-29T19:54:19.674333+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

4 extracted references · 3 canonical work pages

[1]

Karras, T., Aittala, M., Kynk ¨a¨anniemi, T., Lehtinen, J., Aila, T., and Laine, S

URL https://openreview.net/forum? id=ANvmVS2Yr0. Karras, T., Aittala, M., Kynk ¨a¨anniemi, T., Lehtinen, J., Aila, T., and Laine, S. Guiding a diffusion model with a bad version of itself.Advances in Neural Information Processing Systems, 37:52996–53021, 2024. Kong, X., Liu, O., Li, H., Yogatama, D., and Steeg, G. V . Interpretable diffusion via informati...

work page arXiv 2024
[2]

Pizzi, E., Roy, S

URL https://openreview.net/forum? id=fV0t65OBUu. Pizzi, E., Roy, S. D., Ravindra, S. N., Goyal, P., and Douze, M. A self-supervised descriptor for image copy detection. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 14532–14542, 2022. Ren, J., Li, Y ., Zeng, S., Xu, H., Lyu, L., Xing, Y ., and Tang, J. Unveiling...

work page doi:10.18653/v1/2023.acl-long.310 2022
[3]

Along the manifold direction, the marginal distribution pt(x) approximates a convolution of the data distributionN(0, σ 2 data) and the diffusion noise kernel N(0, σ 2 t )

DD-Mem saturates early:General data manifolds typically possess intrinsic noise ( σdata >0 ) even if the data lies on a low-dimensional manifold (Fefferman et al., 2016). Along the manifold direction, the marginal distribution pt(x) approximates a convolution of the data distributionN(0, σ 2 data) and the diffusion noise kernel N(0, σ 2 t ). Consequently,...

2016
[4]

However, crucially, the curvatureκ1 at this mode does not stop increasing; it continues to rise significantly as training proceeds to the final step (60,000 steps)

OD-Mem continues to sharpen:As shown in Figure 6, the model successfully generates samples from the outlier mode as early as the baseline checkpoint (20,000 steps). However, crucially, the curvatureκ1 at this mode does not stop increasing; it continues to rise significantly as training proceeds to the final step (60,000 steps). This indicates that memoriz...

work page arXiv 2025

[1] [1]

Karras, T., Aittala, M., Kynk ¨a¨anniemi, T., Lehtinen, J., Aila, T., and Laine, S

URL https://openreview.net/forum? id=ANvmVS2Yr0. Karras, T., Aittala, M., Kynk ¨a¨anniemi, T., Lehtinen, J., Aila, T., and Laine, S. Guiding a diffusion model with a bad version of itself.Advances in Neural Information Processing Systems, 37:52996–53021, 2024. Kong, X., Liu, O., Li, H., Yogatama, D., and Steeg, G. V . Interpretable diffusion via informati...

work page arXiv 2024

[2] [2]

Pizzi, E., Roy, S

URL https://openreview.net/forum? id=fV0t65OBUu. Pizzi, E., Roy, S. D., Ravindra, S. N., Goyal, P., and Douze, M. A self-supervised descriptor for image copy detection. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 14532–14542, 2022. Ren, J., Li, Y ., Zeng, S., Xu, H., Lyu, L., Xing, Y ., and Tang, J. Unveiling...

work page doi:10.18653/v1/2023.acl-long.310 2022

[3] [3]

Along the manifold direction, the marginal distribution pt(x) approximates a convolution of the data distributionN(0, σ 2 data) and the diffusion noise kernel N(0, σ 2 t )

DD-Mem saturates early:General data manifolds typically possess intrinsic noise ( σdata >0 ) even if the data lies on a low-dimensional manifold (Fefferman et al., 2016). Along the manifold direction, the marginal distribution pt(x) approximates a convolution of the data distributionN(0, σ 2 data) and the diffusion noise kernel N(0, σ 2 t ). Consequently,...

2016

[4] [4]

However, crucially, the curvatureκ1 at this mode does not stop increasing; it continues to rise significantly as training proceeds to the final step (60,000 steps)

OD-Mem continues to sharpen:As shown in Figure 6, the model successfully generates samples from the outlier mode as early as the baseline checkpoint (20,000 steps). However, crucially, the curvatureκ1 at this mode does not stop increasing; it continues to rise significantly as training proceeds to the final step (60,000 steps). This indicates that memoriz...

work page arXiv 2025