{"id":"0ea3ff4e-3244-4498-98f1-63d107efd329","arxiv_id":"2504.21292","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Diffusion models can keep most of their image quality after swapping global self-attention for a distilled multi-scale convolutional block, with FLOPs claimed to fall by up to 6929 times at 16K resolution.","lead":"This paper replaces the self-attention layers in pretrained text-to-image diffusion models with pyramid-shaped convolutional blocks trained by distilling attention patterns from the original model. The authors claim image quality stays comparable while compute drops sharply, especially at high resolutions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Efficiency claims in Tables 1 and 2 appear to count only the replaced self-attention module, so the 6929x/3.4x headline numbers are not end-to-end generation speedups as stated.","rationale":"The reader's weakest assumption already targeted the same issue, and the full manuscript strengthens it: Sec. 5.2 gives no explicit sentence limiting Tables 1 and 2 to 'only self-attention modules', but the magnitudes and ratios force that interpretation. The paper's own attention FLOPs formula in Sec. 3.1 and the ASM/ERF analysis are consistent with a module-level comparison; no wall-clock end-to-end timing, no profiler over the full UNet, and no code or weights are provided. I do not see a separate, more load-bearing flaw: the locality observation is qualitatively supported by attention-map visualizations, the ASM scaling analysis, and the Neighborhood Attention ablation; and the feature-level and output-level distillation losses could plausibly preserve quality even if the module does not perform true global reasoning. The main scientific proposal is plausible, but as written the central efficiency claim is overstated because it conflates module-level savings with generation-level savings. The existing CONDITIONAL verdict remains appropriate; this stress-test does not change it.","tokens_in":14924,"tokens_out":7169,"duration_ms":73963,"concrete_test":"Use an official or reimplemented SD1.5/DiT checkpoint with the DeltaConvBlock inserted and measure end-to-end generation on a single GPU: full 20-step DPM-Solver with CFG at 512x512 and 1024x1024, including text encoding and VAE decoding, profiling with torch.profiler or a full-model FLOPs counter over all UNet/DiT layers. Recompute the speedup factors in Tables 1 and 2 using these end-to-end per-image seconds and total FLOPs. If the 6929x and 3.4x ratios shrink to, say, below 2x end-to-end at 1024x1024 and below 1.2x at 512x512, the stated efficiency claims must be revised; if the end-to-end speedups remain large, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing problem is the scope of the efficiency comparison. The abstract claims 'reducing computational cost by 6929x ... surpassing LinFusion by 5.42x in efficiency', but Tables 1 and 2 compare FLOPs and latency only for the self-attention/DeltaConvBlock component, not for the whole denoising network. This reading follows from the paper's own numbers: for SD1.5 at 512x512 the 'Self-Attention' row is 49.67 G and 'DeltaConvFusion' is 2.82 G, yet a full SD1.5 UNet forward pass with all frozen ResNet, FFN, cross-attention, text-encoder, and VAE components costs far more than 2.82 G at this resolution. Likewise, Table 2's 0.78 ms 'Self-Attention' at 512x512 is too small to be whole-model latency; the 3.4x speedup at 1024x1024 compares one module against another module, not full generation. The 6929x figure is exactly the 16K-resolution self-attention FLOPs (11,010,934 G) divided by the DeltaConvBlock FLOPs (1,589 G), which ignores the frozen backbone. Consequently, the headline efficiency gain is not established for actual generation; at best, the paper shows a large reduction in the cost of the replaced module. The locality and quality arguments are separate and may survive, but the efficiency claim as stated is not supported by the tables.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether self-attention in pre-trained diffusion models is truly needed for global reasoning, and proposes a convolutional replacement module (DeltaConvBlock) with pyramid convolution branches and an average-pooling bias branch. The authors analyze attention maps in SD1.5, SDXL, and PixArt, arguing that attention is predominantly local and can be decomposed into a quadratic-decay high-frequency component plus a spatially invariant low-frequency component. They then replace self-attention blocks with DeltaConvBlocks, freeze all other parameters, and train only the new blocks using feature-level and output-level distillation. Experiments on SD1.5, SDXL, and PixArt report generative quality comparable to or better than baselines on DINOv2-based metrics, and large FLOP/latency reductions. The central claims are that self-attention can be replaced by convolutions without quality loss, and that this yields a 6929x computational reduction and 3.4x latency improvements over prior efficient attention alternatives.","tokens_in":15186,"tokens_out":4327,"duration_ms":42446,"significance":"If the central claims hold, the paper would make a useful contribution to efficient diffusion model inference: it provides evidence against the necessity of global attention at every block, introduces a simple convolution-based replacement module, and demonstrates that distillation from a frozen teacher can transfer the relevant spatial behavior. The paper ships concrete architecture details, a training recipe, and a series of qualitative and quantitative comparisons across U-Net and DiT backbones. However, the significance is tempered by two issues: the headline efficiency numbers are scoped to the replaced module rather than to end-to-end generation, and the ERF matching evidence is partly guaranteed by the feature-distillation objective. The quality evidence is suggestive but based on nonstandard metrics and lacks statistical uncertainty quantification.","major_comments":[{"comment":"The headline efficiency claims are not supported as stated because Tables 1 and 2 appear to measure only the self-attention/DeltaConvBlock module, not the full denoising network. For example, the 6929x reduction is exactly the ratio of the SD1.5 16K self-attention FLOPs (11,010,934 G) to the DeltaConvBlock FLOPs (1,589 G), and the 0.78 ms latency for 'Self-Attention' at 512x512 is far too small to be a whole-model forward pass. The abstract's 'reducing computational cost by 6929x' and 'surpassing LinFusion by 5.42x in efficiency' therefore refer to module-level savings, which do not translate to end-to-end generation speedups of the same magnitude because the frozen ResNet, FFN, cross-attention, text-encoder, and VAE components dominate the actual generation cost. The authors should either explicitly re-scope all such claims to the self-attention module or report full-model FLOPs and end-to-end per-image latency for a complete denoising run.","section":"Abstract; Sec. 5.2, Tables 1 and 2"},{"comment":"The effective receptive field matching in Fig. 9 is largely a consequence of the feature-level distillation loss Lf in Eq. (5), which directly minimizes the difference between DeltaConvBlock outputs and self-attention outputs at every layer. Thus the observation that 'DeltaConvFusion exhibits an ERF pattern closely matching that of self-attention-based models' is not an independent validation of the architectural decomposition; it demonstrates that the convolutional block can imitate the teacher when explicitly trained to do so. To support the claim that the pyramid-plus-pooling design intrinsically captures the two attention components, the authors should compare the ERF of an undestilled or randomly initialized DeltaConvBlock against the teacher, or at minimum explicitly acknowledge that the match is by construction due to distillation.","section":"Sec. 5.5, Fig. 9; Sec. 4.3, Eq. (5)"},{"comment":"The quantitative quality comparison is based on point estimates of newly introduced metrics (DS and FDD from DINOv2) and a custom evaluation set of 10,000 LAION images annotated by InternVL2-8B, with no error bars or statistical significance tests. Since some differences in Table 3 are small (e.g., PixArt 1024 CLIP scores of 30.60 vs 30.55), it is difficult to assess whether the claimed parity or improvement is meaningful. The paper should report confidence intervals or variance estimates, and ideally also evaluate on a standard benchmark (e.g., MS-COCO or a widely used FID/CLIP setup) to allow external comparison, while noting why a custom set was preferred.","section":"Table 3, Sec. 5.4, Sec. 5.1"}],"minor_comments":[{"comment":"The definition of the scaled simple gate rho(f) is garbled in the text; please rewrite it as a clear mathematical expression, including how the feature is split into two channel halves and how the scaling by sqrt(C') is applied.","section":"Sec. 4.1, Eq. (3) and gate definition"},{"comment":"The header of Table 1 contains a typo: '512 512 1024 1204' should be '512x512 1024x1024 2K 4K 8K 16K' (or similar), and the table should clarify in the caption that the FLOPs are for the self-attention/DeltaConvBlock component only, not the whole model.","section":"Table 1"},{"comment":"The phrase '4K curated real images from LAION' is ambiguous: it likely means 4,000 images, but '4K' usually denotes 3840x2160 resolution; please specify the exact number of images.","section":"Sec. 5.1"},{"comment":"The cross-reference 'In Equ.13' does not match the displayed equation numbering; the Min-SNR weighting appears as Eq. (6), so the reference should be corrected.","section":"Sec. 5.1, Implementation details"},{"comment":"The affiliations contain typos: 'Unviersity' should be 'University'; please proofread the author block.","section":"Author affiliations"},{"comment":"The text says 'apply Discrete Fourier Transform (DFT) and then utilize a high-pass Butterworth filter'; it would be clearer to specify whether the DFT is two-dimensional and how the filter cutoff frequency was chosen, since this affects reproducibility.","section":"Sec. 3.2, II. Frequency Analysis"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is interesting and potentially useful, but the efficiency claims in the abstract and Section 5.2 are currently scoped to the replaced self-attention module rather than to full generation, which inflates the headline numbers. This is fixable by re-scoping the claims or reporting end-to-end measurements. The ERF-matching validation is weaker than it appears due to the distillation objective, and the quality evaluation would benefit from standard benchmarks and error bars. None of these issues appear unfixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hey,\n\nThe thing to know about this paper: the conceptual claim is plausible and the analysis is readable, but the headline efficiency numbers are overstated. Tables 1 and 2 count only the replaced self-attention module, not the full denoising network. The 6929x FLOPs reduction is just self-attention FLOPs at 16K (11,010,934 G) divided by DeltaConvBlock FLOPs (1,589 G); the 3.4x latency speedup is module-versus-module. End-to-end generation still pays for the frozen ResNet/FFN/cross-attention blocks, text encoder, and VAE, so the practical speedup will be much smaller. The abstract doesn't qualify this, which is the main problem.\n\nWhat's genuinely good: the layer-wise analysis of attention maps across SD1.5, SDXL, and PixArt is a nice quantitative piece of evidence that attention in these models is mostly local. The decomposition into a distance-dependent high-frequency part and a spatially invariant low-frequency part is clean and motivates the pyramid-convolution-plus-average-pooling design. Showing that a distilled conv block can replace all self-attention layers while keeping the rest frozen, and that it generalizes across resolutions, is a meaningful existence proof. This is a real contribution to the \"do we need global attention?\" line of work.\n\nSoft spots beyond the efficiency framing: the quality evaluation uses DINOv2 Score and FDD, which are nonstandard, with no error bars or significance tests. The ERF matching in Fig. 9 is partly guaranteed by the feature-level distillation loss (Eq. 5), so it isn't independent confirmation. No code or checkpoints are released, so I can't verify the numbers. The training data (2M Midjourney + 4K LAION) is not the usual benchmark diet, which makes comparisons to LinFusion and DiTFastAttn hard to trust fairly.\n\nAll that said, the paper is not a dud. The idea is timely, the module is a sensible alternative to windowed attention and SSMs, and the locality analysis deserves a close look. I'd send it to peer review, but with the expectation of heavy revision: clarify the efficiency scope, report full-model FLOPs and end-to-end latency, use standard metrics with confidence intervals, and release code. Who is it for? Anyone working on efficient diffusion or questioning the necessity of global attention in pretrained models. I'd probably bring it to a reading group to debate the locality claim, though I wouldn't cite it as-is.\n\nRecommendation: accept for review, not desk reject. Read the efficiency tables carefully.","headline":"Worth engaging: the locality analysis and DeltaConvBlock are solid, but the headline 6929x/3.4x gains are module-level only, so the efficiency claim overstates end-to-end speedups.","tokens_in":15750,"tokens_out":4130,"would_cite":false,"duration_ms":41237,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that self-attention in diffusion models is predominantly local, and that a distilled convolutional block can replace it with comparable fidelity and drastically lower cost.","keywords":["self-attention","diffusion models","convolutional networks","knowledge distillation","local attention patterns","text-to-image generation","efficient inference","linear complexity"],"falsifier":"Measure total inference time and FLOPs for the complete denoising network, including frozen cross-attention, FFN, and ResNet blocks, at 512 by 512 and 1024 by 1024 pixels before and after the replacement on identical hardware. If the full-model speedup is close to 1x, the central efficiency claim fails; if it is a large multiple, the claim holds.","tokens_in":14648,"feed_emoji":"🖼️","tokens_out":11060,"duration_ms":105094,"temperature":0.7,"pith_summary":"This paper tries to establish that self-attention in pretrained text-to-image diffusion models is overwhelmingly local, not global, and that it can therefore be replaced by convolutional computation without losing generation quality. Layer-wise and spectral analyses of SD1.5, SDXL, and PixArt show attention mass concentrated near each query and decaying roughly quadratically with distance, with effective receptive fields mostly below 15 to 20 pixels. The proposed DeltaConvFusion replaces every self-attention block with a pyramid-convolution block plus an average-pooling bias branch, trained by distilling the frozen teacher's features and noise predictions. The paper reports that this substitution matches or improves the teacher's image quality while cutting reported self-attention cost by up to 6929 times and beating LinFusion in efficiency, a result that, if right, removes the main scaling bottleneck for high-resolution generation.","feed_headline":"Diffusion self-attention is mostly local; convolutions can replace it","feed_subtitle":"Distilling attention into pyramid convolutions keeps image quality while cutting self-attention cost up to 6,929 times.","key_machinery":"The load-bearing object is the DeltaConvBlock, a convolutional replacement for a self-attention block. It has two branches matched to the two observed components of attention maps: a pyramid-convolution branch, where the input is downsampled at several scales, passed through depthwise convolutions with a scaled gating nonlinearity, and upsampled back, so nearby pixels accumulate a hierarchy of receptive fields; and an average-pooling branch that produces a spatially invariant channel bias. The two branches are recombined with pointwise convolutions, and the whole module is trained with feature-level and output-level distillation while the pretrained model's other weights are frozen. Because the pyramid convolutions run at downsampled scales, the module's complexity is linear in the number of spatial positions rather than quadratic, which is the property that removes the self-attention bottleneck.","core_discovery":"The central claim, stated on the paper's own terms, is that global interactions are not the source of what self-attention contributes to current diffusion models. The attention map of a query pixel is concentrated in a small neighborhood; high-frequency structure decays quadratically with distance, and the remaining signal is a broad, spatially invariant bias. This pattern appears in both U-Net and DiT backbones, and replacing all self-attention layers with localized neighborhood attention with kernel size 13 already preserves coherent generation. DeltaConvFusion makes that observation operational: it distills the attention maps into DeltaConvBlocks, so the convolutional student reproduces the teacher's spatial behavior while the rest of the network stays frozen. On SD1.5, SDXL, and PixArt, the paper reports comparable or better DINOv2-based quality and lower FLOPs and latency than the attention-based originals.","pith_inferences":["A natural extension the authors do not pursue is training a diffusion model from scratch with DeltaConvBlocks instead of distilling an attention teacher; the locality evidence suggests such a backbone could work, but the paper only demonstrates the distillation route.","The reported efficiency gain should be read as module-level: frozen cross-attention, FFN, and ResNet blocks still dominate the full network, so an end-to-end latency comparison would likely show a much smaller speedup than the headline 6929 times.","The localization result may not transfer to tasks requiring genuine long-range dependency, such as video or multi-object scenes with repeated elements; a test would be replacing self-attention in those models and checking whether quality degrades.","A quick validation of the paper's core observation would be to run the same attention-map analysis on a non-diffusion generative model; if those maps are not local, the locality is a feature of this training regime rather than of attention in general."],"forward_implications":["If the locality finding holds, high-resolution generation can replace quadratic self-attention with linear-complexity local operators, removing the main scaling bottleneck at 4K and beyond.","Freezing the rest of the network and distilling only the replaced blocks is enough to keep the paper's image-quality, fidelity, and text-alignment metrics on par with or better than the teacher.","The same recipe applies to both U-Net and DiT backbones, indicating that the localized behavior is not specific to one architecture family.","A model trained only at 512 by 512 pixels can generate coherent 1024 by 1024 images after the replacement, whereas the self-attention baseline fragments.","The method is orthogonal to attention-sharing acceleration, so its efficiency gains can stack with caching and timestep-sharing approaches."],"supporting_citations":[{"why":"Supplies the pretrained SD1.5 U-Net whose self-attention maps are analyzed and whose self-attention blocks are replaced by DeltaConvBlocks.","marker":"[33]"},{"why":"Supplies the SDXL U-Net baseline used to test whether the localization finding and the replacement transfer to a larger latent diffusion model.","marker":"[31]"},{"why":"Supplies the PixArt DiT baseline that anchors the DiT experiments and the high-resolution FLOPs comparison.","marker":"[3]"},{"why":"Supplies LinFusion, the state-space efficient baseline that DeltaConvFusion compares against in quality, FLOPs, and latency.","marker":"[21]"},{"why":"Supplies Neighborhood Attention, used in the ablation showing that a localized attention mechanism alone preserves generation quality.","marker":"[12]"},{"why":"Supplies the Min-SNR loss weighting used in the output-level distillation objective.","marker":"[11]"},{"why":"Supplies DiTFastAttn, a baseline and an orthogonal acceleration method compared in latency and FLOPs.","marker":"[46]"},{"why":"Supplies DINOv2 embeddings underlying the DS and FDD metrics used to compare generation quality.","marker":"[29]"},{"why":"Provides the standard self-attention formulation that serves as the baseline for every replacement and FLOPs comparison.","marker":"[44]"}],"fun_headline_variants":["Self-attention in diffusion is actually local, so use convolutions","Diffusion: replace global attention with local conv, keep quality","Cut diffusion cost 6929x by distilling attention into convolutions","Attention is local: Convolutions can replace it in diffusion models","Diffusion attention is local: Convolutions do it 6929x cheaper"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline efficiency gains assume the FLOPs and latency figures in Tables 1 and 2 measure the same scope for every method; if they count only the replaced self-attention module, the reported thousands-fold reduction does not produce an equivalent speedup for the whole model.","fun_headline_variants_meta":{"raw":{"variants":["Self-attention in diffusion is actually local, so use convolutions","Diffusion: replace global attention with local conv, keep quality","Cut diffusion cost 6929x by distilling attention into convolutions","Attention is local: Convolutions can replace it in diffusion models","Diffusion attention is local: Convolutions do it 6929x cheaper"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000511,"raw_usage":{"total_tokens":2466,"prompt_tokens":909,"completion_tokens":1557,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":1463}},"tokens_in":525,"tokens_out":1557,"duration_ms":11327,"temperature":1.0,"reasoning_tokens":1463,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:08:03.299778+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure total inference time and FLOPs for the complete denoising network, including frozen cross-attention, FFN, and ResNet blocks, at 512 by 512 and 1024 by 1024 pixels before and after the replacement on identical hardware. If the full-model speedup is close to 1x, the central efficiency claim fails; if it is a large multiple, the claim holds.","supporting_citations":[{"cited_title":"High-resolution image synthesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained SD1.5 U-Net whose self-attention maps are analyzed and whose self-attention blocks are replaced by DeltaConvBlocks."},{"cited_title":"Pixart-Σ: Weak-to-strong training of diffusion transformer for 4k text-to-image generation","cited_arxiv_id":null,"evidence_quote":"Supplies the PixArt DiT baseline that anchors the DiT experiments and the high-resolution FLOPs comparison."},{"cited_title":"Neighborhood attention transformer","cited_arxiv_id":null,"evidence_quote":"Supplies Neighborhood Attention, used in the ablation showing that a localized attention mechanism alone preserves generation quality."},{"cited_title":"Efficient diffu- sion training via min-snr weighting strategy","cited_arxiv_id":null,"evidence_quote":"Supplies the Min-SNR loss weighting used in the output-level distillation objective."},{"cited_title":"Ditfastattn: Attention compression for diffusion transformer models","cited_arxiv_id":null,"evidence_quote":"Supplies DiTFastAttn, a baseline and an orthogonal acceleration method compared in latency and FLOPs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies DINOv2 embeddings underlying the DS and FDD metrics used to compare generation quality."},{"cited_title":"Attention is all you need","cited_arxiv_id":null,"evidence_quote":"Provides the standard self-attention formulation that serves as the baseline for every replacement and FLOPs comparison."}],"review_version":1}