Pith. sign in

REVIEW 4 major objections 5 minor 34 references

Local Representative Token Guided Merging for Text-to-Image Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A training-free token merging rule that selects a representative token per local window improves Stable Diffusion's FID from 37.02 to 34.89 while reducing generation time from 2.62 to 2.17 seconds per image.

desk verdict A plausible incremental token-merging scheme with a clean ablative story, but the headline FID gain rests on a small, unreported evaluation that needs to be fixed before the claim is credible. read the letter →

arxiv 2507.12771 v1 pith:GOCBYUOQ submitted 2025-07-17 cs.CV cs.AI

Min-Jeong Lee , Hee-Dong Kim , Seong-Whan Lee This is my paper · ORCID
classification cs.CVcs.AI
keywords stablediffusiontokenmergingtext-to-imagegenerationrepresentativeadaptivewindowtraining-freeaccelerationattentionmechanismFID
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes ReToM, a training-free token merging strategy for text-to-image diffusion models. It claims that grouping image tokens into local windows and merging each window into its most representative token improves image quality and speeds up generation at the same time. On Stable Diffusion, ReToM lowers FID from 37.02 to 34.89 and raises the CLIP score to 39.40, while inference time falls from 2.62 to 2.17 seconds per image. The authors argue that this comes from replacing random destination tokens with similarity-based representatives and from varying window sizes to match how different U-Net layers emphasize local detail or global context.

What carries the argument

The central object is the representative token: within each local window $W$, each token $x_i$ receives a score $\mathrm{sim}(x_i,W) = \frac{1}{N_W - 1} \sum_{j \in W, j \neq i} \cos\mathrm{Sim}(x_i, x_j)$, and the token with the maximum average similarity becomes the destination $D$; source tokens $S_i$ merge as $x_{\mathrm{merged}} = \alpha D + (1-\alpha) \frac{1}{r} \sum_i S_i$. The window acts as a local boundary, with sizes adapted to U-Net structure, and a caching strategy stores the similarity matrix for $p$ timesteps. This machinery carries the argument because it determines which tokens are discarded and which survives.

What would settle it

Run ReToM with period $p=1$ (recomputing similarity at every timestep) and compare FID and generation time against the default period; if the FID gap is large and $p=1$ is much slower, the caching assumption is doing little for quality. Alternatively, compute a rank correlation such as Kendall's tau between pairwise cosine similarity matrices at consecutive timesteps; if it drops sharply within $p$ steps, the representative token chosen from cache is likely stale.

Watch

Extended reading notes

Core claim

ReToM's central claim is that token merging in attention-based image generators should be guided by local representativeness rather than random or fixed-region matching. For each window of tokens, the method computes pairwise cosine similarities, averages them per token, and designates the token with the highest average as the representative destination; the top-$r$ similar tokens are then merged into it. Window sizes are small in U-Net downsampling and upsampling blocks, where local detail matters, and large in bottleneck layers, where global context matters. To avoid recomputing similarities at every denoising step, the pairwise similarities are cached for a period $p$, relying on the observation that consecutive timesteps change token similarity only gradually. The paper reports that this combination beats the Stable Diffusion baseline and prior token merging methods in FID and CLIP while keeping inference faster than the baseline.

Load-bearing premise

The method relies on the assumption that the relative similarity between tokens in a window stays roughly the same over several denoising steps, so cached similarity values remain valid for choosing representative tokens.

Editorial extensions

If this is right

  • Token merging can improve generation quality rather than merely preserve it, since ReToM reports a FID below the no-merging Stable Diffusion baseline.
  • The same merging rule applies to any attention mechanism without retraining, so it can be dropped into other diffusion architectures.
  • Choosing the merge destination by local representativeness is the decisive factor: fixed-window ReToM with representative selection (FID 35.00) already beats ToMeSD's random-selection FID of 37.20.
  • When similarity caching is valid, the overhead of computing pairwise similarities is amortized over several timesteps, so the representative-selection cost does not add wall-clock time.
  • The method offers a practical speed-quality trade-off for real-time or high-resolution text-to-image generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The caching assumption implies a testable rate: measuring how quickly the ranking of token similarities changes across timesteps would predict when the cache period $p$ must shrink, a quantity the paper does not quantify.
  • The representative-token idea could transfer to other iterative transformer workloads, such as video diffusion or autoregressive decoding, where consecutive hidden states also change gradually.
  • Because the reported experiments use ImageNet class-conditioned generation, the benefit may depend on class-structured local texture; testing on arbitrary text prompts would show whether the improvement generalizes.
  • If the FID gain holds at higher resolutions, ReToM might reduce memory pressure in $1024 \times 1024$ generators where quadratic attention is the bottleneck.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ReToM, a training-free token merging method for Stable Diffusion text-to-image generation. The method divides attention tokens into local windows with adaptively chosen sizes per U-Net layer, selects a 'representative token' per window as the token with highest average cosine similarity, merges the top-r similar tokens into it, and caches the pairwise similarities over a period p to avoid recomputation at every timestep. Experiments on ImageNet report that ReToM improves FID from 37.02 to 34.89 and CLIP score from 38.10 to 39.40 relative to the Stable Diffusion baseline, while reducing inference time from 2.62 s/im to 2.17 s/im, and that it outperforms ToMeSD and ATC in FID. The paper also presents ablations on window-size selection and destination-token selection.

Significance. If the reported results are reliable, ReToM would be a practically useful contribution: it is training-free, architecture-agnostic, and claims simultaneous quality improvement and speedup over the standard Stable Diffusion baseline. The paper also provides useful ablations (fixed vs. adaptive windows, most vs. least representative token) and includes qualitative comparisons. However, the central quality claim rests on a small FID evaluation with no uncertainty quantification, and several key hyperparameters are unspecified, so the significance is currently conditional on additional evidence. The method's derivation is not circular; the representative token is defined by the same cosine similarity used for merging, which is a design choice rather than a logical flaw.

major comments (4)
  1. [Sec. IV-B, Table I] The headline FID improvement (37.02 to 34.89) is computed on only 2,000 generated images and 5,000 real images, with no seeds, repeated runs, or confidence intervals reported. FID is known to have high variance at this sample size; the standard protocol uses 30,000-50,000 generated samples. The stress-test concern is valid: the reported gap may be sampling noise, and this is the primary evidence that token merging improves generation quality. The authors should either run the full FID evaluation or provide repeated subsampled evaluations with error bars.
  2. [Abstract vs. Sec. IV-F] The abstract states a 6.2% FID improvement, while Sec. IV-F states a 5.8% improvement for the same numbers (37.02 to 34.89). This internal inconsistency suggests the metric is not stable across runs or that a calculation error exists; it should be corrected and reconciled.
  3. [Sec. IV-A and Algorithm 1] The hyperparameters essential for reproducibility are not specified: the merging ratio R, the merging weight α, the similarity period p, and the exact adaptive window-size schedule per U-Net layer. Without these values, Table I cannot be reproduced, and claims that 'adaptive' windows are better than 'fixed' windows cannot be independently verified. Please report the exact settings used for all configurations in Table I.
  4. [Sec. III-C and Sec. IV-E] The caching strategy assumes that token-similarity rankings remain largely consistent across timesteps, but the only evidence is four qualitative heatmaps at timesteps 0, 300, 600, and 900 (Fig. 3). No quantitative measure of rank stability or correlation between consecutive timesteps is provided, and the value of p used in the experiments is not stated. This weakens the efficiency claim: if rankings change quickly, the cached representative token becomes stale and more frequent recomputation is needed. Please add a quantitative stability analysis (e.g., rank correlation or overlap of selected representative tokens between timesteps) and report p.
minor comments (5)
  1. [Sec. III-C] Typo: 'This means that that the relative similarity' should read 'This means that the relative similarity'.
  2. [Sec. IV-E] Typo: 'An visualization' should be 'A visualization'.
  3. [Algorithm 1] The variable t is initialized to 0 and incremented inside the window loop, but the algorithm does not show a loop over sampling timesteps; the intended control flow should be clarified. Also, the condition 't mod p = 0' is evaluated per window, which could cause within-timestep inconsistency if the loop is not structured correctly.
  4. [Sec. IV-A] The paper says ImageNet validation data is used, but it does not specify how class labels are converted into text prompts for text-to-image generation. Please state the prompt template and whether the same real images are used for both the FID reference set and the prompt set.
  5. [Table II] The SSIM metric is reported but its computation is not described (e.g., against which reference images, at what resolution, and with what window size). Please clarify.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: ReToM is an algorithmic construction, not a derived quantity fitted to its own evidence, and the quality/efficiency claims rest on external evaluation.

full rationale

The paper's central contribution is a token-merging algorithm: Eq. (1) defines the representative token as the token with the highest average cosine similarity to other tokens in a window, and Eq. (2) defines the merged token as a weighted mean of the representative and source tokens. These are design choices, not predictions derived from fitted parameters. The FID and CLIP improvements are measured against the Stable Diffusion baseline and existing methods (ToMeSD, ATC) using external ImageNet data; they are not constructed from the method's own inputs. The cached-similarity assumption in Sec. III-C is an empirical premise validated by heatmap visualization, not a circular step. The adaptive window sizes are assigned heuristically based on U-Net structure; selecting the best configuration from Table I is hyperparameter selection, which is a correctness/evaluation concern, not circularity. The self-citations in the reference list (e.g., [6], [7], [17], [32]) appear only in related-work context and do not support the load-bearing claims. No equation reduces to another equation by construction, no fitted parameter is renamed as a prediction, and no uniqueness theorem or author-imported ansatz is invoked to force the proposed choice. Thus, the derivation chain is self-contained and not circular.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method's central experimental comparison depends on four user-chosen quantities (R, alpha, p, window schedule) that are not reported, and on three domain assumptions about diffusion dynamics, U-Net structure, and similarity-based merging. No new physical entities are introduced.

free parameters (4)
  • Merging ratio R = not reported
    Algorithm 1 defines r as the number of tokens per window times R, but no value is given in Table I or the text.
  • Merging weight alpha = not reported
    Eq. (2) combines representative token D and source tokens with alpha, but alpha is never specified.
  • Similarity period p = not reported
    The caching strategy recomputes cosine similarity every p steps; p is not reported in experiments.
  • Adaptive window size schedule = not reported
    Section III-A says small windows for down/upsampling blocks and large for bottleneck, but the concrete sizes per layer are never listed.
assumptions (3)
  • domain assumption Token similarities evolve gradually across consecutive diffusion timesteps
    Invoked in Sec. III-C and Sec. IV-E to justify caching cosine similarities for period p; supported only by qualitative heatmaps, not quantitative rank stability.
  • domain assumption U-Net layer structure implies local details matter in down/upsampling blocks and global context in the bottleneck
    Invoked in Sec. III-A to motivate adaptive window sizes; no quantitative analysis links receptive fields to merging window sizes.
  • domain assumption Merging into the token with highest average cosine similarity preserves salient local features better than random merging
    Invoked in Sec. III-B; shown only through downstream FID/CLIP comparisons, not a proven property.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Local Representative Token Guided Merging for Text-to-Image Generation." pith.science (2026). https://pith.science/paper/GOCBYUOQ

@misc{pith2026250712771,
  author       = {Pith},
  title        = {Pith review of: Local Representative Token Guided Merging for Text-to-Image Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GOCBYUOQ}},
  note         = {Machine review of arXiv:2507.12771}
}
read the original abstract

Stable diffusion is an outstanding image generation model for text-to-image, but its time-consuming generation process remains a challenge due to the quadratic complexity of attention operations. Recent token merging methods improve efficiency by reducing the number of tokens during attention operations, but often overlook the characteristics of attention-based image generation models, limiting their effectiveness. In this paper, we propose local representative token guided merging (ReToM), a novel token merging strategy applicable to any attention mechanism in image generation. To merge tokens based on various contextual information, ReToM defines local boundaries as windows within attention inputs and adjusts window sizes. Furthermore, we introduce a representative token, which represents the most representative token per window by computing similarity at a specific timestep and selecting the token with the highest average similarity. This approach preserves the most salient local features while minimizing computational overhead. Experimental results show that ReToM achieves a 6.2% improvement in FID and higher CLIP scores compared to the baseline, while maintaining comparable inference time. We empirically demonstrate that ReToM is effective in balancing visual quality and computational efficiency.

Figures

Figures reproduced from arXiv: 2507.12771 by the authors.

Figure 1
Figure 1. Overview of local representative token guided merging. First, we adjust the window size for each transformer block as illustrated in the bottom left [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Qualitative results of ToMeSD and our ReToM applied to the stable diffusion model using the ImageNet validation dataset. While ToMeSD focuses [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. An visualization of changes in the cosine similarity of all tokens [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 28 canonical work pages

  1. [1]

    High-resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , 2022, pp. 10 684–10 695

    work page 2022

  2. [2]

    Generative adversarial networks,

    I. Goodfellow et al. , “Generative adversarial networks,” Commun. ACM, vol. 63, no. 11, pp. 139–144, 2020

    work page 2020

  3. [3]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” vol. 33, 2020, pp. 6840–6851

    work page 2020

  4. [4]

    Denoising diffusion implicit models,

    J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” Int. Conf. Learn. Represent. (ICLR) , 2021

    work page 2021

  5. [5]

    Abstract representations of associated emotions in the human brain,

    J. Kim et al. , “Abstract representations of associated emotions in the human brain,” J. Neurosci, vol. 35, no. 14, pp. 5655–5663, 2015

    work page 2015

  6. [6]

    Translation-, rotation- and scale-invariant recognition of hand-drawn symbols in schematic diagrams,

    S.-W. Lee, J. H. Kim, and F. C. Groen, “Translation-, rotation- and scale-invariant recognition of hand-drawn symbols in schematic diagrams,” Int. J. Pattern Recognit. Artif. Intell. , vol. 4, no. 01, pp. 1–25, 1990

    work page 1990

  7. [7]

    Accurate object contour tracking based on boundary edge selection,

    M.-C. Roh, T.-Y . Kim, J. Park, and S.-W. Lee, “Accurate object contour tracking based on boundary edge selection,” Pattern Recognit., vol. 40, no. 3, pp. 931–943, 2007

    2007

  8. [8]

    U-net: Convolutional networks for biomedical image segmentation,

    O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Med. Image Comput. Comput-assisted. Intervention. (MICCAI) , 2015, pp. 234–241

    work page 2015

Show all 34 references
  1. [9]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Adv. Neural Inf. Process. Syst. (NeurIPS) , vol. 30, 2017

    2017

  2. [10]

    On distillation of guided diffusion models,

    C. Meng et al. , “On distillation of guided diffusion models,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , 2023, pp. 14 297–14 306

    2023

  3. [11]

    Progressive distillation for fast sampling of diffusion models,

    T. Salimans and J. Ho, “Progressive distillation for fast sampling of diffusion models,” arXiv preprint arXiv:2202.00512 , 2022

    2022 arXiv

  4. [12]

    A-vit: Adaptive tokens for efficient vision transformer,

    H. Yin et al., “A-vit: Adaptive tokens for efficient vision transformer,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , 2022, pp. 10 809–10 818

    2022

  5. [13]

    Learned token pruning for transformers,

    S. Kim et al., “Learned token pruning for transformers,” in Proc. ACM SIGKDD Conf. Knowl. Discovery. Data Mining. (KDD) , 2022, pp. 784–794

    2022

  6. [14]

    Tokenlearner: Adaptive space-time tokenization for videos,

    M. Ryoo, A. Piergiovanni, A. Arnab, M. Dehghani, and A. Angelova, “Tokenlearner: Adaptive space-time tokenization for videos,” vol. 34, 2021, pp. 12 786–12 797

    2021

  7. [15]

    Tore: Token reduction for efficient human mesh recovery with transformer,

    Z. Dou et al. , “Tore: Token reduction for efficient human mesh recovery with transformer,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , 2023, pp. 15 143–15 155

    2023

  8. [16]

    Only train once: A one-shot neural network training and pruning framework,

    T. Chen et al. , “Only train once: A one-shot neural network training and pruning framework,” Adv. Neural Inf. Process. Syst. (NeurIPS) , 2021

    2021

  9. [17]

    Text extraction in mpeg compressed video for content-based indexing,

    Y .-K. Lim, S.-H. Choi, and S.-W. Lee, “Text extraction in mpeg compressed video for content-based indexing,” in Proc. Int. Conf. Pattern Recognit. (ICPR) , vol. 4, 2000, pp. 409–412

    2000

  10. [18]

    Structural pruning for diffusion models,

    G. Fang, X. Ma, and X. Wang, “Structural pruning for diffusion models,” in Adv. Neural Inf. Process. Syst. (NeurIPS) , vol. 36, 2023, pp. 16 716–16 728

    2023

  11. [19]

    Token merging: Your vit but faster,

    D. Bolya et al., “Token merging: Your vit but faster,”Int. Conf. Learn. Represent. (ICLR), 2023

    2023

  12. [20]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” Int. Conf. Learn. Represent. (ICLR) , 2021

    2021

  13. [21]

    Token merging for fast stable diffusion,

    D. Bolya and J. Hoffman, “Token merging for fast stable diffusion,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , 2023, pp. 4599–4603

    2023

  14. [22]

    Focal attention for long-range interactions in vision transformers,

    J. Yang et al. , “Focal attention for long-range interactions in vision transformers,” in Adv. Neural Inf. Process. Syst. (NeurIPS) , vol. 34, 2021, pp. 30 008–30 022

    2021

  15. [23]

    Understanding the effective receptive field in deep convolutional neural networks,

    W. Luo, Y . Li, R. Urtasun, and R. Zemel, “Understanding the effective receptive field in deep convolutional neural networks,” Adv. Neural Inf. Process. Syst. (NeurIPS) , vol. 29, 2016

    2016

  16. [24]

    Autodiffusion: Training-free optimization of time steps and architectures for automated diffusion model acceleration,

    L. Li et al. , “Autodiffusion: Training-free optimization of time steps and architectures for automated diffusion model acceleration,” Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV) , 2023

    2023

  17. [25]

    Dpm-solver++: Fast solver for guided sampling of dif- fusion probabilistic models,

    C. Lu et al. , “Dpm-solver++: Fast solver for guided sampling of dif- fusion probabilistic models,” arXiv preprint arXiv:2211.01095 , 2022

    2022 arXiv

  18. [26]

    Stand-alone self-attention in vision models,

    P. Ramachandran et al., “Stand-alone self-attention in vision models,” Adv. Neural Inf. Process. Syst. (NeurIPS) , vol. 32, 2019

    2019

  19. [27]

    Self-attention does not need o(n2) memory,

    M. N. Rabe and C. Staats, “Self-attention does not need o(n2) memory,” arXiv preprint arXiv:2112.05682 , 2021

    2021 arXiv

  20. [28]

    Not all patches are what you need: Expediting vision transformers via token reorganizations,

    Y . Liang et al., “Not all patches are what you need: Expediting vision transformers via token reorganizations,” Int. Conf. Learn. Represent. (ICLR), vol. abs/2202.07800, 2022

    2022 arXiv

  21. [29]

    Groupvit: Semantic segmentation emerges from text supervision,

    J. Xu et al. , “Groupvit: Semantic segmentation emerges from text supervision,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2022, pp. 18 134–18 144

    2022

  22. [30]

    Importance-based token merg- ing for diffusion models,

    H. Wu, J. Xu, H. Le, and D. Samaras, “Importance-based token merg- ing for diffusion models,” arXiv preprint arXiv:2411.16720 , 2024

    2024 arXiv

  23. [31]

    Agglomerative token clustering,

    J. B. Haurum, S. Escalera, G. W. Taylor, and T. B. Moeslund, “Agglomerative token clustering,” in Eur . Conf. on Comput. Vis. (ECCV), 2024, pp. 200–218

    2024

  24. [32]

    Multilayer cluster neural network for totally unconstrained handwritten numeral recognition,

    S.-W. Lee, “Multilayer cluster neural network for totally unconstrained handwritten numeral recognition,” Neural Networks, vol. 8, no. 5, pp. 783–792, 1995

    1995

  25. [33]

    Imagenet: A large-scale hierarchical image database,

    J. Deng et al., “Imagenet: A large-scale hierarchical image database,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) , 2009, pp. 248–255

    2009

  26. [34]

    Clip- score: A reference-free evaluation metric for image captioning,

    J. Hessel, A. Holtzman, M. Forbes, R. L. Bras, and Y . Choi, “Clip- score: A reference-free evaluation metric for image captioning,” in Conf. Empir . Methods Nat. Lang. Process. (EMNLP) , 2021

    2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.