{"id":"5f12af64-6f99-4440-928e-5ef714962b9b","arxiv_id":"2501.11211","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Diffusion model inference can be accelerated by computing only the small differences between adjacent time steps, with reduced bit-width and zero-skipping, at moderate accuracy loss.","lead":"Ditto is a hardware accelerator and algorithm that speeds up diffusion model image generation by reusing values that barely change between denoising steps, computing only the differences with fewer bits. A smart generalist would read this because it attacks the main cost of diffusion models, the repeated full-network forward passes, with a mix of exact math, quantization, and a specialized chip design.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Accuracy claim is unsubstantiated: Table II shows BED FID degrading from 2.962 to 5.897, yet no quantized baseline is reported to separate Ditto's effect from quantization.","rationale":"The reader's designated weakest assumption, the Defo policy of fixing execution types after the second time step, is substantially cushioned by the paper's own design-space exploration: Fig. 17 reports 92% prediction accuracy, Fig. 18 shows 98.8% of ideal performance, and Fig. 19, even under a synthetic dynamic-similarity distribution, shows 98.03% of ideal. That concern is real but not the most load-bearing. The Table II BED result is more concrete: a visible FID degradation from 2.962 to 5.897 sits in the main accuracy table under a claim that accuracy is preserved, and no quantized baseline is provided to establish that the degradation is only quantization rather than something introduced by Ditto. The paper deserves credit for a clean algorithmic idea, exact distributive rewriting, a broad seven-model evaluation, and a reasonable attempt to address memory overhead through Defo. The missing comparison is an addressable omission rather than a fatal flaw, and no artifact release or bit-exactness check is provided. A conditional verdict is therefore appropriate, pending a head-to-head Ditto-versus-quantized-baseline accuracy comparison and, ideally, an independent bit-exactness trace of the difference-processing path.","tokens_in":25225,"tokens_out":8958,"duration_ms":95013,"concrete_test":"Run the Q-Diffusion 8-bit quantized BED and CHUR models, and ideally all Table I models, with the same sampler, step count, and evaluation pipeline, and compare FID/IS against both the FP32 and Ditto columns of Table II. Additionally, instrument the Ditto simulator to compare layer outputs and final latents against the quantized baseline: if every activation produced by the Ditto path is bit-identical to the Q-Diffusion quantized baseline, the algorithm is lossless and the BED gap is a quantization artifact; if any activation diverges, trace the first divergent layer to determine whether the error comes from reduced-bit-width classification, Defo's nonlinear-function bypass, or the attention difference factorization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the accuracy claim, not the Defo policy. Section VI/Table II reports FP32 versus Ditto image quality, with BED FID rising from 2.962 to 5.897 and IS from 2.227 to 2.338, while the text states that 'the Ditto algorithm preserves the accuracy of all diffusion models compared to baseline FP32 models.' No Q-Diffusion 8-bit quantized baseline is reported anywhere in the evaluation, so the reader cannot separate quantization error from error introduced by the difference-processing path. The hardware section claims full-bit-width support 'ensuring numerical equivalent results with original operations' (Section V-A), and the distributive rewrite is exact algebraically; the strong version of the paper's claim is therefore that Ditto should be bit-identical to its quantized baseline. If that is true, Table II only measures quantization, and the 'preserves accuracy' wording is misleading. If it is false, then the difference path, including Encoding Unit classification, Defo's nonlinear-function bypasses, or the attention factorization, introduces extra error. Either way, the central assertion of near-baseline image quality is currently unsupported by the data shown.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Ditto, a software-hardware co-design to accelerate diffusion model inference by exploiting the high value similarity between adjacent time steps in the reverse diffusion process. The Ditto algorithm computes temporal differences between consecutive time-step activations, executes linear layers on these differences using reduced bit-width and zero-skipping, and adds the previous output back. A static/runtime execution-flow optimization (Defo) decides per layer whether difference processing or original-activation processing is more efficient, and Defo+ additionally applies spatial difference processing to layers that remain in the original-activation mode. The Ditto hardware combines adder-tree PEs with an encoding unit that reorders and classifies differences, supporting dynamic 4/8-bit operations and zero skipping. The evaluation covers seven diffusion models (DDPM, BED, CHUR, IMG, SDM, DiT, Latte), compares against GPU, ITC, Diffy, and Cambricon-D, and reports up to 1.5x speedup and 17.74% energy saving over ITC, with ablations for Defo and dynamic execution-flow exploration.","tokens_in":25461,"tokens_out":3491,"duration_ms":33616,"significance":"If the claims hold, this is a substantive contribution to efficient diffusion-model inference, as it shows how temporal similarity can be systematically exploited in both software and hardware. The distributive rewrite for linear layers and the Q/K split for attention are algebraically exact, the hardware is evaluated with a cycle-accurate simulator across seven diverse models, and the paper includes ablations (Fig. 16), decision-accuracy analysis (Fig. 17), and a design-space exploration of dynamic temporal similarity (Fig. 19). The main weakness is the accuracy evidence: Table II compares FP32 against Ditto with no quantized baseline, so the central claim of 'preserving accuracy' is not yet substantiated, and the Defo fixed-decision policy's generality is asserted rather than demonstrated across models with fluctuating temporal similarity. These issues are local and fixable, but they are load-bearing for the paper's main conclusions.","major_comments":[{"comment":"The claim that 'the Ditto algorithm preserves the accuracy of all diffusion models compared to baseline FP32 models' is not supported by the reported data, because no quantized baseline (e.g., Q-Diffusion 8-bit without Ditto) is reported. For BED, FID degrades from 2.962 to 5.897 while IS changes from 2.227 to 2.338; without the quantized baseline the reader cannot tell whether this degradation comes from quantization or from the difference-processing path (Encoding Unit classification, Defo nonlinear-function bypasses, or the attention factorization). If Ditto is bit-identical to its quantized baseline, as suggested by the 'numerical equivalent results' statement in Section V-A, then Table II measures only quantization error and the text overstates the algorithm's accuracy behavior; if Ditto is not bit-identical, the additional error must be quantified and attributed. Please report the quantized-model accuracy for the same 8-bit setting used by the hardware evaluation, and ideally an ablation that turns off Defo and the difference path to isolate their effects.","section":"Section VI-A, Table II"},{"comment":"The Defo policy fixes each layer's execution type at the second time step and applies it to all later time steps, and the paper supports this with 92% decision accuracy and 98.8% performance of an ideal per-step re-decision design. However, the main evaluation (Figs. 13 and Table II) only includes models whose temporal similarity is stable in the sense of Fig. 6b; the paper's own synthetic exploration (Fig. 19) shows that decision accuracy drops by 7% when temporal similarity fluctuates, and the authors note that 'some future models with high temporal similarity may exhibit dynamic temporal similarity across the time domain.' The claim in Section I that Ditto is effective 'regardless of the type of diffusion model' is therefore too broad, and the paper should either scope that claim to models with temporally stable similarity or include a real model or sampler chain where the fixed second-step decision is challenged.","section":"Section IV-B, Figs. 17 and 19"},{"comment":"The crossover experiment applies Ditto's software techniques (attention difference, Defo, Defo+) to Cambricon-D and sign-mask data flow to Ditto, and reports speedups, but it does not report the accuracy of these modified configurations. Because sign-mask data flow alters the treatment of nonlinear functions and Defo changes layer execution types, the numerical results of the modified baselines may differ from the unmodified ones, and the speedup comparison is only meaningful if these modifications are accuracy-neutral or if the accuracy of every configuration is reported. Please provide accuracy numbers for the configurations in Fig. 15, or a statement of numerical equivalence with the original quantized model.","section":"Section VI-B, Fig. 15"}],"minor_comments":[{"comment":"The abstract says 'up to 1.5x speedup and 17.74% energy saving', but Section VI-B reports these as averages over the benchmarks; please make the wording consistent (e.g., 'up to 1.5x average speedup' or provide the per-model maximum).","section":"Abstract and Section VI-B"},{"comment":"The caption states cosine-similarity values of 0.9997, 0.9972, 0.9934, and 0.948, while the text in Section II-B says the similarity 'exceeds 0.94' for these layers; the lower value 0.948 is inconsistent with the claim if it refers to the same layers, so please clarify which time-step pair each value corresponds to.","section":"Fig. 3a"},{"comment":"The key identity for attention difference processing, QtKt = Qt+1Kt+1 + QtΔK + ΔQKt+1, is stated in prose but not numbered; please number it as an equation, since it is central to the attention-layer optimization.","section":"Section IV-A, Attention Layers"},{"comment":"Table III lists Ditto as 'A4W8', but the hardware supports both 4-bit and 8-bit activations; to avoid confusion with the A4W8 configuration of Diffy, please label Ditto's bit-width as 'A4/8W8' and state in the text that the hardware is evaluated for support of both modes.","section":"Section V and Table III"},{"comment":"References [46] and [47] are the same paper (Shapeshifter) and should be merged or the duplicate removed.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The core difference-processing idea is sound and the hardware evaluation is thorough, but the accuracy table is the main obstacle: without a quantized baseline it is impossible to verify the 'preserves accuracy' claim, which is central to the paper's contribution. If the authors can show that Ditto is bit-identical to the quantized baseline, then the accuracy discussion must be reframed as 'Ditto adds no accuracy loss over the underlying quantization'; if it is not bit-identical, the source of additional error needs to be isolated. The fixed second-step Defo policy is a secondary concern, and the cross-over accuracy in Fig. 15 should also be addressed. I recommend major revision rather than rejection because the requested additions are within the scope of the manuscript's existing evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a serious referee. The core algorithm is mathematically clean: because reverse diffusion reuses the same weights, linear layers can be rewritten as operations on temporal differences, with the initial time step paying full cost. The attention decomposition (QtΔK + ΔQKt+1) is correct and sensible, and the Defo per-layer execution selection addresses the real memory-access overhead that naive difference processing creates. The zero-value skipping, mixed-precision PE design, and iso-area comparison against Cambricon-D are all described carefully. The hardware evaluation is broad—seven models, several baselines, ablations, design-space exploration—and the simulator is at least based on a published open-source one.\n\nThe soft spot is exactly where the stress-test note lands. Table II reports FP32 versus Ditto FID/IS, and BED goes from 2.962 to 5.897 FID. The text claims 'the Ditto algorithm preserves the accuracy of all diffusion models.' That claim is unjustified as presented. The evaluation uses Q-Diffusion for quantization, but no quantized baseline is reported, so the reader cannot tell whether the degradation comes from 8-bit quantization itself or from the difference-processing path, including the Encoding Unit's classification, Defo's bypassing, or the attention factorization. Since Section V-A claims full-bit-width support 'ensuring numerical equivalent results with original operations,' the strongest version of their argument implies Ditto should match the quantized baseline bit-for-bit; then Table II only measures quantization and the wording is misleading. If it doesn't, then the difference path adds error. Either way, the central accuracy claim is currently unsupported. This is fixable: add the Q-Diffusion 8-bit baseline to Table II, report per-model breakdown, and either show the difference path is bit-exact or quantify its extra error.\n\nThe Defo fixed-at-second-time-step assumption is a softer concern. It is a heuristic with acknowledged limits, and the Fig. 19 exploration shows 98% of ideal performance under synthetic dynamic similarity, so I would not call it a fatal flaw. The lack of released artifacts and the reliance on a modified simulator are real but secondary; they lower confidence in the exact speedup numbers without invalidating the approach.\n\nBottom line: this is a solid co-design paper with one load-bearing reporting gap. It deserves peer review, but the reviewers should insist on the quantized baseline before the accuracy claim is allowed to stand. I would bring it to a reading group, but I wouldn't cite the accuracy numbers yet.","headline":"Useful accelerator co-design, but the accuracy-preservation claim conflates quantization with the difference path and needs a quantized baseline before it can be believed.","tokens_in":25973,"tokens_out":1946,"would_cite":false,"duration_ms":19274,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Diffusion model inference can be accelerated by computing only the differences between consecutive denoising steps, with nearly all of those differences representable in 4 bits or as zeros.","keywords":["diffusion model acceleration","temporal value similarity","difference processing","post-training quantization","hardware accelerator","dynamic bit-width","zero skipping","image generation"],"falsifier":"Run a diffusion model with a high-order sampler (e.g., DPM-Solver or EDM) on the same seven benchmarks and record, for every layer and every step, whether difference processing or original-activation execution finishes in fewer cycles; if the per-layer winner changes substantially after step 2 in any model — as it would if temporal similarity surges near the final denoising steps — then the fixed Defo decision becomes suboptimal and the claimed 98.8% of ideal speedup would be violated in a directly measurable way.","tokens_in":25038,"feed_emoji":"⚡","tokens_out":7276,"duration_ms":62985,"temperature":0.7,"pith_summary":"Diffusion models generate images by repeatedly denoising, and each step recomputes the whole network on slightly different inputs. This paper argues that those inputs are so similar that the step-to-step difference is a much cheaper operand, and that computing linear layers on the difference (then adding the previous output) costs fewer bits and skips many zeros. Across seven diffusion models, about 96% of temporal differences fit in 4 bits or are zero, a pattern that holds at every time step. If the claim holds, accelerators built around this difference processing can reach about 1.5x speedup and 17.7% energy savings over an 8-bit baseline while keeping image quality statistically unchanged. The paper completes the argument with a hardware design whose processing elements natively handle both 4-bit and 8-bit data and skip zero differences.","feed_headline":"Diffusion AI speeds up 1.5x by computing only the change between steps","feed_subtitle":"96% of step-to-step differences are 4-bit or zero, so a new accelerator skips them and cuts energy 17.7%.","key_machinery":"The central identity is the distributive property of linear layers, $f(x_{t-1}) = f(x_t + \\Delta_t) = f(x_t) + f(\\Delta_t)$, where $\\Delta_t = x_{t-1} - x_t$ is the temporal difference between adjacent denoising steps. Because $f(\\Delta_t)$ is computed on data that is mostly zero and mostly 4-bit, an adder-tree MAC unit with 4-bit multipliers (and a shifter to compose 8-bit products from two 4-bit products) executes the bulk of the arithmetic at reduced cost, then a summation unit adds $f(\\Delta_t)$ back to the previous step's output. The second mechanism is Defo, which uses a layer table and cycle counters to freeze, at the second time step, whether each linear layer will run in difference mode or original-activation mode, and bypasses difference calculation and summation around non-linear functions based on static dependency analysis.","core_discovery":"On the paper's own terms, Ditto establishes that the reverse diffusion chain is temporally sparse: adjacent time steps share cosine similarity above 0.94 (0.98 on average), so the temporal difference between consecutive activations occupies a value range up to 8.96x narrower than the activations themselves. In an 8-bit quantized model, 44.48% of those differences are exactly zero and 96.01% require 4 bits or fewer, meaning only 3.99% need full bit-width. Ditto exploits this by running the first time step at full bit-width, then for each linear layer applying the distributive property to compute the layer output as the previous output plus the layer applied to the temporal difference, using 4-bit multipliers and zero skipping. A runtime and compile-time optimization (Defo) decides per layer, after observing the first two time steps, whether difference processing or original-activation processing is faster, and a companion accelerator (Ditto hardware) implements dynamic sparsity and mixed precision in a single adder-tree PE. Evaluated on seven diffusion models from DDPM to DiT, the hardware achieves up to 1.5x speedup and 17.74% energy savings over an iso-area 8-bit tensor-core baseline while matching FP32 FID/IS/CLIP metrics within noise.","pith_inferences":["The same temporal-difference trick should transfer to other iterative generative or inference loops where consecutive states are highly correlated, such as video diffusion, autoregressive token generation, and iterative refinement in super-resolution; the load-bearing condition is only that the difference signal is sparser in bit-width and zeros than the state itself.","The paper's own Fig. 19 suggests a stress test: with samplers that take larger or non-uniform jumps (e.g., DPM-Solver or EDM), temporal similarity may fluctuate near the final steps, and the fixed-at-step-2 Defo decision would become suboptimal; a dynamic re-evaluation every few steps would reclaim most of the gap at negligible control cost.","A testable extension is to measure the temporal-difference bit-width distribution per layer and per step for a broader sampler set; if 4-bit-or-zero fractions stay above roughly 90% throughout, Ditto's hardware case strengthens, and if they collapse, the accelerator's PE count and Defo policy would need to be re-tuned.","Because Ditto's benefit scales with the number of time steps, its relative advantage should shrink with fast samplers that use 10-20 steps; the paper's 20-step IMG and Latte benchmarks already show the trend, and the practical sweet spot is likely in high-step fidelity regimes."],"forward_implications":["If Ditto is right, a large fraction of diffusion inference work is redundant: roughly 44% of multiply-accumulates can be skipped as zero differences and most of the rest run at 4-bit precision, so the speedup does not rely on approximate computing — outputs are numerically equivalent to the 8-bit baseline.","The mechanism generalizes across model families: the paper evaluates pixel-space DDPM, latent-space conditional/unconditional LDM (BED, CHUR, IMG, SDM), and diffusion transformers (DiT, Latte), and reports consistent BOPs reduction at every adjacent time step, including the final cleaning steps.","Difference processing can be combined with existing time-step-aware quantization schemes such as Q-Diffusion, since Ditto operates on the quantized activations and does not require retraining.","Defo+ shows that spatial similarity inside a tensor can be harvested alongside temporal similarity using the same hardware, with only an offset register and a multiplexer, which is a direct extension the paper evaluates.","The hardware's uniform 4-bit PE design avoids the outlier-PE area penalty of mixed-precision accelerators like Cambricon-D, so the comparison runs at iso-area with more effective PEs."],"supporting_citations":[{"why":"Defines the DDPM reverse diffusion process whose step-to-step similarity Ditto exploits, and serves as the pixel-space benchmark.","marker":"[29]"},{"why":"The DDIM sampler that produces the adjacent-time-step chains for six of the seven evaluated models, making the temporal differences concrete.","marker":"[81]"},{"why":"The latent diffusion framework behind BED, CHUR, IMG, SDM and the block structures containing the attention and non-linear layers Defo must handle.","marker":"[68]"},{"why":"Q-Diffusion's time-step-aware quantization defines the 8-bit activation/weight representation inside which the bit-width requirement of temporal differences is measured.","marker":"[50]"},{"why":"Diffy is the spatial-difference baseline that Ditto extends and outperforms; its row-dimension adaptation is used as the spatial-difference reference in Fig. 5.","marker":"[58]"},{"why":"Cambricon-D is the temporal-difference accelerator baseline; its sign-mask data flow and outlier-PE design are the main comparison in the hardware evaluation.","marker":"[43]"},{"why":"Supplies the cycle-accurate simulator used for the hardware evaluation.","marker":"[14]"}],"fun_headline_variants":["Diffusion models skip similar steps, gain 1.5x speed and 17.7% energy","Temporal similarity in diffusion: 1.5x speedup by computing only deltas","Ditto: Cut diffusion compute by leveraging step-to-step value similarity","Exploit diffusion's temporal redundancy: 1.5x faster accelerator","Zero-skip temporal deltas give diffusion a 1.5x speedup, 17.7% energy cut"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Defo fixes each layer's execution type at the second time step and applies it to all later time steps, assuming the relative efficiency of difference processing versus original activations stays stable across the whole reverse diffusion chain.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion models skip similar steps, gain 1.5x speed and 17.7% energy","Temporal similarity in diffusion: 1.5x speedup by computing only deltas","Ditto: Cut diffusion compute by leveraging step-to-step value similarity","Exploit diffusion's temporal redundancy: 1.5x faster accelerator","Zero-skip temporal deltas give diffusion a 1.5x speedup, 17.7% energy cut"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000339,"raw_usage":{"total_tokens":1914,"prompt_tokens":1033,"completion_tokens":881,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":649,"completion_tokens_details":{"reasoning_tokens":765}},"tokens_in":649,"tokens_out":881,"duration_ms":7549,"temperature":1.0,"reasoning_tokens":765,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:30:50.489895+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a diffusion model with a high-order sampler (e.g., DPM-Solver or EDM) on the same seven benchmarks and record, for every layer and every step, whether difference processing or original-activation execution finishes in fewer cycles; if the per-layer winner changes substantially after step 2 in any model — as it would if temporal similarity surges near the final denoising steps — then the fixed Defo decision becomes suboptimal and the claimed 98.8% of ideal speedup would be violated in a directly measurable way.","supporting_citations":[{"cited_title":"Diffy: A d ´ej`a vu-free differ- ential deep neural network accelerator,","cited_arxiv_id":null,"evidence_quote":"Diffy is the spatial-difference baseline that Ditto extends and outperforms; its row-dimension adaptation is used as the spatial-difference reference in Fig. 5."},{"cited_title":"Cambricon-d: Full-network differential acceleration for diffusion models,","cited_arxiv_id":null,"evidence_quote":"Cambricon-D is the temporal-difference accelerator baseline; its sign-mask data flow and outlier-PE design are the main comparison in the hardware evaluation."}],"review_version":1}