REVIEW 4 major objections 4 minor 39 references
Q-Align: Alleviating Attention Leakage in Zero-Shot Appearance Transfer via Query-Query Alignment
T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Query-query alignment, not query-key alignment, stops attention leakage in zero-shot appearance transfer.
desk verdict A plausible, low-cost fix for Cross-Image attention leakage, but the headline gains rest on an unvalidated GPT metric and mixed old-metric results. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the query-query alignment matrix S = Q_app Q_str^T together with the aggregation matrix P' derived from its per-row top-k (k=1) selection. S states which appearance-image position corresponds to each structure-image position. P' encodes those matches as sparse weights, row-wise softmax reweights them, and applying P' to the appearance keys and values produces K* and V*. These rearranged key/value pairs are then consumed by the standard cross-image attention formula. The design choice of rearranging keys and values rather than queries is deliberate, since queries carry the spatial semantics of the output and should not be moved. The whole procedure runs in a sing
What would settle it
Take image pairs with ground-truth dense semantic correspondence and compare the accuracy of top-1 query-query matches against top-1 query-key matches; if query-query alignment is not systematically more accurate, the core premise fails. A complementary check: run Q-Align on pairs with extreme pose or category differences and look for outputs where the structure is preserved but the appearance identity lands on semantically wrong regions, which would indicate the alignment matrix is unreliable.
Extended reading notes
Core claim
Q-Align's central claim is that attention leakage in zero-shot appearance transfer comes from query-key misalignment: when the output image's queries are dotted against the appearance image's keys, attention spreads over semantically related but wrong regions. The paper replaces that signal with a query-query alignment matrix S = Q_app Q_str^T, keeps the top-1 match in each row, and turns those matches into a sparse aggregation matrix P' via row-wise softmax. Applying P' to the appearance keys and values yields K* = P'K_app and V* = P'V_app, and the final attention is softmax(Q_out(K*)^T / sqrt(d))V*. Because queries define spatial semantics and live in the same embedding space, the authors
Load-bearing premise
The method assumes that for every position in the structure image, the most similar position in the appearance image is the semantically correct one to copy appearance from; if that similarity ranking is wrong, the transferred colors and textures land on the wrong spots.
Editorial extensions
If this is right
- Q-Align gives zero-shot appearance transfer without optimization, training, or object masks, so it can be dropped into any pretrained diffusion model's self-attention layers.
- Attention leakage is measured at 52.2% of 180 test pairs for the Cross-Image baseline, and Q-Align improves 69.1% of those leaky cases, which should translate to fewer artifacts like background-colored holes or stray object features.
- Average appearance-fidelity and structural-consistency scores improve over Cross-Image and other zero-shot baselines across most of the six tested domains: animal faces, animals, cars, birds, buildings, and cakes.
- Since the method is layer-local and parameter-free, it can compose with other attention-control editing techniques that also swap or refine keys and values.
- The paper's evidence that query and key spaces are incomparable implies other key-value mixing attention methods face the same leakage risk and are candidates for the same fix.
Reading between the lines
- The matrix S is itself a dense semantic-correspondence map; it could be exported and evaluated directly against ground-truth correspondences, which would test the core assumption more directly than end-to-end image quality.
- The hard top-1 selection is the most brittle part: for appearance and structure images with large pose or viewpoint gaps, a top-k aggregation with a small soft window may preserve the sharpness benefit while adding robustness.
- The method is applied per self-attention layer, but leakage is likely scale-dependent; stacking query-query rearrangement across multiple layers might give a coarser-to-finer alignment than any single layer.
- The proposed vision-language-model scoring protocol could be reused to audit other generative-image tasks where Gram loss and IoU disagree with human perception.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper identifies 'attention leakage' in zero-shot appearance transfer using Cross-Image attention and proposes Q-Align to mitigate it. The method replaces query-key alignment with query-query alignment S = Q_app Q_str^T, constructs a top-1 rearrangement matrix P' from S, rearranges K_app and V_app into K* and V*, and applies the cross-image attention softmax(Q_out K*^T / sqrt(d)) V*. The authors claim improved appearance fidelity and competitive structure preservation over Cross-Image, DragonDiffusion, DiffuseIT, and SpliceViT, and introduce a GPT-4o-based evaluation protocol for appearance fidelity and structural consistency.
Significance. The problem is relevant: attention leakage is a real artifact in training-free appearance transfer, and the proposed mechanism is simple, zero-shot, and accompanied by public code. The paper also contributes a useful analysis of query-key versus query-query alignment in Stable Diffusion. However, the central claim rests on two currently weak pillars: (i) the assumption that S provides semantically correct dense correspondence is backed only by one qualitative example and not by any quantitative correspondence metric, and (ii) the main quantitative evidence is a new self-proposed GPT-4o metric that is not validated against human raters, while pre-existing Gram-loss and IoU metrics show mixed or inconclusive results. If these gaps are filled, the method would be a solid contribution; in its current form the evidence is not yet commensurate with the claims.
major comments (4)
- [§4.2, Eqs. (3)–(8)] The mechanism hinges on S = Q_app Q_str^T being a semantically correct dense correspondence. With k=1, each output location receives keys/values from exactly one appearance location, so any mis-assignment in S directly places incorrect appearance features with no averaging to dilute the error. The paper provides only the single 3×3 grid of Fig. 3 as evidence, and the cited [39] supports a different claim about query/key spaces, not that queries from two different images are directly comparable. Please report a quantitative correspondence accuracy (e.g., PCK on annotated pairs) and an ablation over k. Also, the fallback in Eq. (4), if triggered, reverts to the original diagonal K/V, which is exactly the leakage-prone configuration; the frequency of that fallback is not reported.
- [§5.1, Tables 2–3] The main quantitative advantage of Q-Align is measured by a new GPT-4o-based score that the paper itself proposes. The only support that this score matches human perception is Figure 5, which is anecdotal and reports no correlation, agreement, or inter-rater statistic. Since both the method and the metric are new, the reported gains (e.g., average 4.30 vs. 4.19 for Cross-Image) could reflect biases in the scoring procedure. Please validate the protocol against a human-rated sample quantitatively, or report results on established metrics that are known to correlate with perception.
- [§5.1, Table 1] On the pre-existing metrics, Q-Align is not consistently better: Gram loss is worse than Cross-Image in Animal (1.68 vs. 1.08) and Car (1.32 vs. 1.12), and the IoU improvements are small relative to the reported standard deviations (e.g., Animal 0.71/0.12 vs. 0.66/0.16). The paper dismisses Gram loss as unreliable, but the supporting argument is one illustrative example rather than a systematic analysis. Please provide statistical significance tests and, ideally, additional established perceptual metrics (e.g., LPIPS) to substantiate the claimed appearance-fidelity advantage.
- [§3, attention-leakage statistics] The leakage analysis reports that leakage occurs in 94/180 pairs and Q-Align improves 65 of those. This means 29 leakage cases are not improved, and there is no breakdown of how often Q-Align leaves leakage unchanged, worsens it, or introduces new artifacts in the 86 non-leakage or non-improved pairs. Since the method introduces a hard spatial selection, error analysis on the failure cases is needed to understand when the query-query alignment fails and whether the failures correlate with object size, background complexity, or image-pair similarity.
minor comments (4)
- [§4.2, Eq. (3)] The notation is confusing: (P^T) is defined but then P is used in Eqs. (4)–(7). Please clarify the orientation of the matrix and why the transpose form is introduced.
- [§5.1, Figure 5] The figure illustrates that Gram loss disagrees with the GPT score, but only for one selected example. A quantitative comparison on a larger sample would strengthen the argument for the proposed metric.
- [§5.2, Figure 6] The qualitative selection explicitly chooses samples that exhibit leakage in Cross-Image. Please include a set of randomly selected pairs to control for selection bias.
- [§5.3, Figure 7] The threshold 'values below 0.2 are masked as 0' is arbitrary. Please state the choice criterion or provide a sensitivity analysis.
Circularity Check
No significant circularity: the method is an empirical transformation evaluated against external baselines and independently defined metrics; no fitted parameter is later renamed as a prediction, and the paper's own partial-failure statistics rule out a construction-forced improvement.
full rationale
The paper's claimed derivation chain is not circular. Q-Align defines the alignment matrix S = Q_app Q_str^T (Eq. 2), builds a rearrangement matrix P with a top-1 hard selection (Eq. 3), falls back to the identity only when no match exists (Eq. 4), and applies P' to K_app and V_app (Eqs. 6-7) before computing attention (Eq. 8). No parameter is fitted from the evaluation data and then reported as a prediction; the method is zero-shot and parameter-free except for k=1, which is stated rather than fitted. The main experimental claims are evaluated with existing Gram Loss and IoU metrics (Table 1) and a separately proposed GPT-based protocol (Tables 2-3). The GPT score is not defined in terms of the method's equations, so an improvement on it is not true by construction. The paper explicitly reports that attention leakage occurs in 94 of 180 pairs and that Q-Align improves only 65 of those (Section 3), which would be impossible if the improvement were logically forced by the method's definition. The supporting citation [39] is to external work by different authors, not a self-citation chain, and while the correctness of the query-query alignment assumption may be debated, that is a validity or robustness concern, not a circularity. Concerns about metric choice or the lack of quantitative correspondence validation are substantive for correctness but do not make the derivation equivalent to its inputs.
Assumptions & free parameters
free parameters (1)
- k (top-k for key-value rearrangement) =
1
assumptions (3)
- domain assumption Queries of the structure and appearance images lie in the same embedding space and their similarity is semantically meaningful.
- domain assumption GPT-4o can reliably rate appearance fidelity and structural consistency of edited images on a 1-5 scale.
- domain assumption Self-attention layers of Stable Diffusion encode sufficient semantic and appearance information for this mapping.
Cite this review
Pith. "Pith review of Q-Align: Alleviating Attention Leakage in Zero-Shot Appearance Transfer via Query-Query Alignment." pith.science (2026). https://pith.science/paper/W5F4HYSB
@misc{pith2026250821090,
author = {Pith},
title = {Pith review of: Q-Align: Alleviating Attention Leakage in Zero-Shot Appearance Transfer via Query-Query Alignment},
year = {2026},
howpublished = {\url{https://pith.science/paper/W5F4HYSB}},
note = {Machine review of arXiv:2508.21090}
}
read the original abstract
We observe that zero-shot appearance transfer with large-scale image generation models faces a significant challenge: Attention Leakage. This challenge arises when the semantic mapping between two images is captured by the Query-Key alignment. To tackle this issue, we introduce Q-Align, utilizing Query-Query alignment to mitigate attention leakage and improve the semantic alignment in zero-shot appearance transfer. Q-Align incorporates three core contributions: (1) Query-Query alignment, facilitating the sophisticated spatial semantic mapping between two images; (2) Key-Value rearrangement, enhancing feature correspondence through realignment; and (3) Attention refinement using rearranged keys and values to maintain semantic consistency. We validate the effectiveness of Q-Align through extensive experiments and analysis, and Q-Align outperforms state-of-the-art methods in appearance fidelity while maintaining competitive structure preservation.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[39]
Alignment attention by matching key and query distributions
Shujian Zhang, XINJIE FAN, Huangjie Zheng, Korawat Tanwisuth, and Mingyuan Zhou. Alignment attention by matching key and query distributions. In NeurIPS, 2021
work page 2021
-
[1]
Cross-image attention for zero-shot appearance transfer
Yuval Alaluf, Daniel Garibi, Or Patashnik, Hadar Averbuch-Elor, and Daniel Cohen- Or. Cross-image attention for zero-shot appearance transfer. In SIGGRAPH, 2024
work page 2024
-
[2]
Surf: Speeded up robust features
Herbert Bay, Tinne Tuytelaars, and Luc Van Gool. Surf: Speeded up robust features. In ECCV, 2006
2006
-
[3]
Masactrl: Tuning-free mutual self-attention control for consistent image syn- thesis and editing
Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image syn- thesis and editing. In ICCV, 2023
work page 2023
-
[4]
Emerging properties in self-supervised vision trans- formers
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bo- janowski, and Armand Joulin. Emerging properties in self-supervised vision trans- formers. In ICCV, 2021
2021
-
[5]
Cats++: Boosting cost aggrega- tion with convolutions and transformers
Seokju Cho, Sunghwan Hong, and Seungryong Kim. Cats++: Boosting cost aggrega- tion with convolutions and transformers. IEEE TPAMI, 2023
work page 2023
-
[6]
Stargan v2: Diverse image synthesis for multiple domains
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In CVPR, 2020
work page 2020
-
[7]
Diffedit: Diffusion-based semantic image editing with mask guidance
Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion-based semantic image editing with mask guidance. In ICLR, 2023. N. KIM ET AL. : Q-ALIGN 11
work page 2023
Show all 39 references
-
[8]
An image is worth 16x16 words: Trans- formers for image recognition at scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Trans- formers for image recognitio...
2021
-
[9]
Gatys, Alexander S
Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. A neural algorithm of artistic style. arXiv preprint arXiv:1508.06576, 2015
2015 arXiv
-
[10]
Vector quantized diffusion model for text-to-image syn- thesis
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image syn- thesis. In CVPR, 2022
2022
-
[11]
Asic: Aligning sparse in-the-wild image collections
Kamal Gupta, Varun Jampani, Carlos Esteves, Abhinav Shrivastava, Ameesh Makadia, Noah Snavely, and Abhishek Kar. Asic: Aligning sparse in-the-wild image collections. In ICCV, 2023
2023
-
[12]
Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T. Freeman. Unsupervised semantic segmentation by distilling feature correspondences. In ICLR, 2022
2022
-
[13]
Unsupervised semantic correspondence using stable diffusion
Eric Hedlin, Gopal Sharma, Shweta Mahajan, Hossam Isack, Abhishek Kar, Andrea Tagliasacchi, and Kwang Moo Yi. Unsupervised semantic correspondence using stable diffusion. In NeurIPS, 2023
2023
-
[14]
Prompt-to-prompt image editing with cross attention control
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. In ICLR, 2023
2023
-
[15]
Gpt-4o system card
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024
2024 arXiv
-
[16]
Segment anything in high quality
Lei Ke, Mingqiao Ye, Martin Danelljan, Yifan Liu, Yu-Wing Tai, Chi-Keung Tang, and Fisher Yu. Segment anything in high quality. In NeurIPS, 2023
2023
-
[17]
Flexiedit: Frequency-aware latent refinement for enhanced non-rigid editing
Gwanhyeong Koo, Sunjae Yoon, Ji Woo Hong, and Chang D Yoo. Flexiedit: Frequency-aware latent refinement for enhanced non-rigid editing. arXiv preprint arXiv:2407.17850, 2024
2024 arXiv
-
[18]
Diffusion-based image translation using disentangled style and content representation
Gihyun Kwon and Jong Chul Ye. Diffusion-based image translation using disentangled style and content representation. In ICLR, 2023
2023
-
[19]
Jae Yong Lee, Joseph DeGol, Victor Fragoso, and Sudipta N. Sinha. Patchmatch-based neighborhood consensus for semantic correspondence. In CVPR, 2021
2021
-
[20]
Zone: Zero-shot instruction- guided local editing
Shanglin Li, Bohan Zeng, Yutang Feng, Sicheng Gao, Xiuhui Liu, Jiaming Liu, Lin Li, Xu Tang, Yao Hu, Jianzhuang Liu, and Baochang Zhang. Zone: Zero-shot instruction- guided local editing. In CVPR, 2024
2024
-
[21]
Towards under- standing cross and self-attention in stable diffusion for text-guided image editing
Bingyan Liu, Chengyu Wang, Tingfeng Cao, Kui Jia, and Jun Huang. Towards under- standing cross and self-attention in stable diffusion for text-guided image editing. In CVPR, 2024. 12 N. KIM ET AL. : Q-ALIGN
2024
-
[22]
G-eval: NLG evaluation using gpt-4 with better human alignment
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using gpt-4 with better human alignment. In ACL, 2023
2023
-
[23]
Distinctive image features from scale-invariant key points
DG Lowe. Distinctive image features from scale-invariant key points. Int. J. Comput. Vis., 2004
2004
-
[24]
Diffusion hyperfeatures: Searching through time and space for semantic correspon- dence
Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfeatures: Searching through time and space for semantic correspon- dence. In NeurIPS, 2023
2023
-
[25]
Dragondiffu- sion: Enabling drag-style manipulation on diffusion models
Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang. Dragondiffu- sion: Enabling drag-style manipulation on diffusion models. In ICLR, 2024
2024
-
[26]
Zero-shot image-to-image translation
Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun- Yan Zhu. Zero-shot image-to-image translation. In SIGGRAPH, 2023
2023
-
[27]
Hierarchical text-conditional image generation with clip latents
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022
2022 arXiv
-
[28]
High-resolution image synthesis with latent diffusion models
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Om- mer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022
2022
-
[29]
Very deep convolutional networks for large- scale image recognition
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large- scale image recognition. arXiv preprint arXiv:1409.1556, 2015
2015 arXiv
-
[30]
Emergent correspondence from image diffusion
Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariha- ran. Emergent correspondence from image diffusion. In NeurIPS, 2023
2023
-
[31]
Raft: Recurrent all-pairs field transforms for optical flow
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In ECCV, 2020
2020
-
[32]
Gocor: Bringing globally optimized correspondence volumes into your neural network
Prune Truong, Martin Danelljan, Luc V Gool, and Radu Timofte. Gocor: Bringing globally optimized correspondence volumes into your neural network. In NeurIPS, 2020
2020
-
[33]
Splicing vit features for semantic appearance transfer
Narek Tumanyan, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Splicing vit features for semantic appearance transfer. In CVPR, 2022
2022
-
[34]
Plug-and-play diffusion features for text-driven image-to-image translation
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In CVPR, 2023
2023
-
[35]
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017
2017
-
[36]
Dy- namic prompt learning: Addressing cross-attention leakage for text-based image edit- ing
Kai Wang, Fei Yang, Shiqi Yang, Muhammad Atif Butt, and Joost van de Weijer. Dy- namic prompt learning: Addressing cross-attention leakage for text-based image edit- ing. In NeurIPS, 2023
2023
-
[37]
Inversion-free image editing with language-guided diffusion models
Sihan Xu, Yidong Huang, Jiayi Pan, Ziqiao Ma, and Joyce Chai. Inversion-free image editing with language-guided diffusion models. In CVPR, 2024. N. KIM ET AL. : Q-ALIGN 13
2024
-
[38]
A tale of two features: Stable diffusion comple- ments dino for zero-shot semantic correspondence
Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. A tale of two features: Stable diffusion comple- ments dino for zero-shot semantic correspondence. In NeurIPS, 2023
2023
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.