Selective Coupling of Decoupled Informative Regions: Masked Attention Alignment for Data-Free Quantization of Vision Transformers

Biao Qian; Jungong Han; Yang Wang; Yong Wu

arxiv: 2606.04373 · v2 · pith:T2BKT6EInew · submitted 2026-06-03 · 💻 cs.CV · cs.AI

Selective Coupling of Decoupled Informative Regions: Masked Attention Alignment for Data-Free Quantization of Vision Transformers

Biao Qian , Yang Wang , Yong Wu , Jungong Han This is my paper

Pith reviewed 2026-06-28 06:59 UTC · model grok-4.3

classification 💻 cs.CV cs.AI

keywords data-free quantizationvision transformersmasked attention alignmentinformative regionssynthetic samplesself-attentionpatch similarity

0 comments

The pith

Vision transformers concentrate semantics in sparse attention patches that can be isolated to generate better synthetic data for quantization.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that self-attention semantics in vision transformers live mostly in a sparse subset of patches called informative regions, and that these regions carry the bulk of the mutual information between synthetic samples and a quantized model's outputs. It decouples the regions by maximizing differential entropy on patch similarities within the synthetic images, then selectively aligns the masked attention maps of a full-precision model and the quantized model to produce higher-quality synthetic samples. A periodic refreshing step keeps the samples matched to the quantized model as training evolves. This directly targets the distribution mismatch that hurt earlier data-free quantization methods for transformers. A sympathetic reader would care because it offers a way to compress vision transformers accurately when real training data cannot be used or shared.

Core claim

The semantics in the self-attention mechanism is predominantly localized to a sparse subset of patches, called informative regions; the informative regions dominate the mutual information between synthetic samples and Q's outputs. Maximizing differential entropy over patch similarity of synthetic samples decouples these informative regions from noisy background, after which the regions are selected to align full-precision models with the quantized model Q via a masked attention alignment objective, yielding high-quality synthetic samples that are refreshed periodically to adapt to the evolving state of Q.

What carries the argument

Masked attention alignment objective, which uses differential-entropy maximization on patch similarity to decouple informative regions and then selectively couples them to align full-precision and quantized ViT models.

If this is right

Synthetic samples exhibit reduced distribution mismatch with the input distribution expected by quantized ViTs.
Performance exceeds prior data-free quantization methods across multiple ViT backbones and downstream tasks.
Periodic sample refreshing allows the synthetic data to track changes in the quantized model throughout training.
Mutual information between synthetic samples and quantized outputs is preserved by focusing alignment on the decoupled informative regions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The sparsity finding may extend to other attention-heavy architectures such as language models or multimodal transformers.
If the informative regions prove stable across tasks, precomputed region masks could accelerate repeated quantization runs.
The entropy-based decoupling step could be tested with alternative similarity metrics to see whether region isolation improves.
The selective coupling idea suggests that attention sparsity itself could be exploited in other model-compression pipelines beyond quantization.

Load-bearing premise

Maximizing differential entropy over patch similarity of synthetic samples will reliably isolate the informative regions that actually drive the quantized model's outputs.

What would settle it

An experiment that masks the entropy-selected patches versus random patches and finds no larger change in the quantized model's output distribution or accuracy would falsify the claim that the selected regions dominate the relevant mutual information.

Figures

Figures reproduced from arXiv: 2606.04373 by Biao Qian, Jungong Han, Yang Wang, Yong Wu.

**Figure 1.** Figure 1: (a) Existing arts, e.g., MimiQ (Choi et al., 2025), suffer from a obvious performance gap compared to our MaskAQ, owing to the suboptimal quality of synthetic samples. (b) The synthetic samples from existing arts, e.g., PSAQ-ViT (Li et al., 2022) and MimiQ (Choi et al., 2025), usually exhibit two primary issues: semantic dispersion, where the semantics is distributed across the entire image, deviating from… view at source ↗

**Figure 2.** Figure 2: Illustration of our proposed MaskAQ, where our basic idea is to couple the selected informative regions of synthetic samples x with varying Q, yielding high-quality of synthetic samples. MaskAQ is achieved via two primary components: (a) informative region decoupling (Sec.2.2.1) to construct coherent image structure via differential entropy maximum (Lf b in Eq.(10)) over patch similarity; and (b) masked at… view at source ↗

**Figure 3.** Figure 3: Ablation study for classification accuracy (%) about the hyper-parameters (a) λf b and (b) λalign. in Eq.(10); w/o Lalign in Eq.(14); LQ w/o binary mask in Eq.(16); abandoning the periodic sample refreshing strategy during the training process; and our MaskAQ. The ablation experiments are conducted with ViT-T and DeiT-T on ImageNet. Tab.2 suggests the obvious accuracy advantages (at least 0.45% and 0.93%… view at source ↗

**Figure 4.** Figure 4: (a) Examples of synthetic samples from MimiQ (Choi et al., 2025) and our MaskAQ, apart from the corresponding real samples. (b) Visualization of attention maps (the brighter, the larger) from the intermediate layer of P and Q based on MaskAQ. λf b ∈ {0.1, 1, 10} and λalign = {0.01, 0.1, 1, 10}, thereby exploring a comprehensive grid of hyperparameter configurations. Fig.3 illustrates that the optimal perf… view at source ↗

**Figure 5.** Figure 5: More examples of synthetic samples from MimiQ (Choi et al., 2025) and our MaskAQ, apart from the corresponding real samples. C. Additional Discussions and Results As indicated in Sec.3.1 of the main body, we further offer additional discussions and more experimental results. C.1. Additional Results on Object Detection and Semantic Segmentation Tasks Apart from the image classification task in the main body… view at source ↗

**Figure 6.** Figure 6: Additional visualization of attention maps (the brighter, the larger) from the intermediate layer of P and Q based on MaskAQ. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗

read the original abstract

Data-Free Quantization (DFQ) addresses data security concerns by synthesizing samples, without accessing real data. It has garnered increasing attention in the context of Vision Transformers (ViTs), owing to the superiority of the self-attention mechanism compared to classical convolutional operation. However, previous DFQ arts for ViTs often suffer from a distribution mismatch between synthetic samples and input distribution expected by quantized models Q, resulting in the suboptimal performance. In this paper, we propose a novel Masked Attention Alignment approach for Data-Free Quantization of ViTs, named MaskAQ, revealing that: 1) the semantics in the self-attention mechanism is predominantly localized to a sparse subset of patches, called informative regions; 2) the informative regions dominate the mutual information between synthetic samples and Q's outputs. To these ends, we incorporate differential entropy maximum over patch similarity of synthetic samples, to decouple informative regions from noisy background. To couple with varied Q, the informative regions are selected to align full-precision models with Q via a masked attention alignment objective, thus yielding high-quality synthetic samples. Furthermore, a periodic sample refreshing strategy comes up to endow MaskAQ with the capacity to continually adapt to the evolving state of Q throughout the training process, to preserve desirable mutual information with synthetic samples. Extensive experiments verify the merits of MaskAQ over state-of-the-art approaches across multiple backbones and downstream tasks. Our code is available at https://github.com/hfutqian/MaskAQ.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

MaskAQ adds differential entropy decoupling of patches plus masked attention alignment to DFQ for ViTs, with released code and multi-backbone experiments, but the link from entropy max to dominant MI patches rests on an unproven assumption.

read the letter

The core contribution is a concrete method that first maximizes differential entropy over patch similarities in synthetic samples to isolate sparse informative regions, then uses those regions in a masked attention alignment loss between the full-precision model and the quantized target. A periodic sample refresh keeps the synthetic data in step with the evolving quantized model.

The work does a few things cleanly. It identifies the distribution mismatch issue in prior ViT DFQ and targets it with an attention-focused mechanism rather than generic regularization. Running on several backbones and tasks plus shipping the code at the GitHub link gives readers something they can actually test.

The soft spot is the load-bearing assumption. The paper states that semantics live mostly in those sparse patches and that they dominate mutual information with the quantized outputs, then treats max differential entropy on similarity as the way to find them. No derivation shows why that entropy criterion must select the patches that actually drive the quantized model's behavior across architectures or bit-widths. If the selected mask misses the high-MI regions, the alignment objective has nothing to align to. The stress-test note lands here; the proxy is indirect and the paper does not close the gap with math or targeted ablations.

This is for people already working on data-free quantization or efficient ViT deployment. A reader in that niche can extract the method and the empirical numbers. It is coherent enough on its own terms to deserve a serious referee, even with the justification gap.

Referee Report

2 major / 1 minor

Summary. The paper proposes MaskAQ, a data-free quantization method for Vision Transformers. It claims that self-attention semantics localize to sparse 'informative regions' that dominate mutual information between synthetic samples and quantized model outputs. The approach decouples these regions via differential entropy maximization over patch similarity statistics of synthetic samples, then couples them to the quantized model Q through a masked attention alignment objective; a periodic sample refreshing strategy adapts to Q's evolving state during training. Experiments reportedly demonstrate superiority over prior DFQ methods across multiple ViT backbones and tasks.

Significance. If the core assumptions hold and the method generalizes, MaskAQ could meaningfully advance data-free quantization for attention-based models by mitigating distribution mismatch through targeted region alignment rather than global statistics. The public code release supports reproducibility and allows direct verification of the reported gains.

major comments (2)

[Abstract / §3] Abstract and Method (no equation shown): the central claim that 'informative regions dominate the mutual information between synthetic samples and Q's outputs' is asserted without a derivation showing why differential entropy maximization on patch-similarity matrices must select patches that actually carry the dominant MI term with Q for arbitrary ViT architectures or bit-widths. This assumption is load-bearing for both the decoupling and the subsequent masked alignment steps.
[§3.2] Method description: the masked attention alignment objective assumes rather than demonstrates that the entropy-selected mask preserves the dominant MI term when coupled to Q; if the criterion selects generator-salient but low-attention patches in the target Q, the alignment cannot recover the claimed dominance. No cross-architecture or cross-bit-width analysis is referenced to test this.

minor comments (1)

[Abstract] The abstract is unusually long and contains multiple distinct claims; condensing the motivation and contributions would improve readability.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the thoughtful and detailed comments. Below we address each major comment directly, indicating where revisions will be made to strengthen the manuscript.

read point-by-point responses

Referee: [Abstract / §3] Abstract and Method (no equation shown): the central claim that 'informative regions dominate the mutual information between synthetic samples and Q's outputs' is asserted without a derivation showing why differential entropy maximization on patch-similarity matrices must select patches that actually carry the dominant MI term with Q for arbitrary ViT architectures or bit-widths. This assumption is load-bearing for both the decoupling and the subsequent masked alignment steps.

Authors: The claim originates from empirical observations that self-attention semantics in ViTs concentrate in sparse patches, which we support with attention visualizations in the manuscript. Differential entropy maximization over patch-similarity statistics is introduced to promote diversity among patches and thereby isolate regions with high semantic content. While a general closed-form derivation applicable to every architecture and bit-width is not provided, the approach is motivated by the information-theoretic intuition that maximizing entropy of similarity distributions reduces redundancy and highlights dominant contributors to model outputs. In the revised manuscript we will expand §3 with additional discussion of this rationale, including further visualizations that correlate the selected regions with attention weights in both the generator and Q. revision: partial
Referee: [§3.2] Method description: the masked attention alignment objective assumes rather than demonstrates that the entropy-selected mask preserves the dominant MI term when coupled to Q; if the criterion selects generator-salient but low-attention patches in the target Q, the alignment cannot recover the claimed dominance. No cross-architecture or cross-bit-width analysis is referenced to test this.

Authors: We agree that an explicit demonstration of MI preservation would improve clarity. The current experimental section already reports results across multiple ViT backbones (DeiT, PVT, Swin) and bit-width configurations (W4A4, W8A8), with consistent gains over prior DFQ methods. To directly address the concern, the revised version will include a new ablation subsection that quantifies the overlap between entropy-selected masks and high-attention regions in Q, together with a comparison of mutual-information estimates (or proxy metrics) before and after alignment. This analysis will be performed across the same set of architectures and bit-widths already evaluated. revision: yes

Circularity Check

0 steps flagged

No circularity; derivation rests on empirical assumptions validated externally

full rationale

The paper's central claims—that informative regions dominate mutual information and can be isolated via differential entropy maximization on patch similarity—are presented as observations motivating the MaskAQ method, with the masked attention alignment and periodic refreshing as practical steps. No equations, self-citations, or fitted parameters are shown in the provided text that reduce the claimed performance gains or the MI dominance to a quantity defined by construction from the same synthetic samples or outputs. The approach is framed as relying on extensive experiments across backbones and tasks for verification, making the derivation self-contained against external benchmarks rather than tautological.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only; no explicit free parameters, axioms, or invented entities are stated. The approach implicitly assumes that informative regions identified via entropy on synthetic samples transfer to real data distributions.

pith-pipeline@v0.9.1-grok · 5805 in / 1036 out tokens · 18405 ms · 2026-06-28T06:59:46.469983+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

12 extracted references · 1 canonical work pages

[1]

L., Kiros, J

Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv:1607.06450,

Pith/arXiv arXiv
[2]

An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint arXiv:2010.11929,

Dosovitskiy, A. An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint arXiv:2010.11929,

Pith/arXiv arXiv 2010
[3]

Crafting papers on machine learning

Langley, P. Crafting papers on machine learning. In Langley, P. (ed.),Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207–1216, Stan- ford, CA,

2000
[4]

10 Masked Attention Alignment for Data-Free Quantization of Vision Transformers Li, Y ., Kim, Y ., Lee, D., Kundu, S., and Panda, P

arXiv:2102.05426. 10 Masked Attention Alignment for Data-Free Quantization of Vision Transformers Li, Y ., Kim, Y ., Lee, D., Kundu, S., and Panda, P. Genq: Quantization in low data regimes with generative syn- thetic data. InEuropean Conference on Computer Vision, pp. 216–235. Springer,

arXiv
[5]

Fq-vit: Post-training quantization for fully quantized vision trans- former.arXiv preprint arXiv:2111.13824,

Lin, Y ., Zhang, T., Sun, P., Li, Z., and Zhou, S. Fq-vit: Post-training quantization for fully quantized vision trans- former.arXiv preprint arXiv:2111.13824,

arXiv
[6]

C., and Bialek, W

Tishby, N., Pereira, F. C., and Bialek, W. The informa- tion bottleneck method.arXiv preprint physics/0004057,

Pith/arXiv arXiv
[7]

12 Masked Attention Alignment for Data-Free Quantization of Vision Transformers A

doi: 10.1109/CVPR.2017.544. 12 Masked Attention Alignment for Data-Free Quantization of Vision Transformers A. Details of Related Work As indicated in Sec.1 of the main body, we provide the comprehensive review of related work in this section, which mainly encompasses Vision Transformer architectures, data-driven quantization and data-free quantization ap...

work page doi:10.1109/cvpr.2017.544 2017
[8]

Other lines of research modify the ViT architecture to better capture multi-scale and local visual structure

introduced a data-efficient training method with token-based distillation, achieving competitive performance on ImageNet without massive external datasets and thus facilitating practical ViT adoption. Other lines of research modify the ViT architecture to better capture multi-scale and local visual structure. For example, Swin Transformer (Liu et al., 202...

2020
[9]

and video recognition (Arnab et al., 2021; Neimark et al., 2021). Therefore, the DFQ for ViTs are directly influenced by their architectural distinctions from Convolutional Neural Networks (CNNs), including patch-based tokenization, LayerNorm usage, attention-driven operations, and the distribution sensitivity of multi-head activations. A.2. Data-Driven Q...

2021
[10]

However, both QAT and PTQ necessitate access to the original training data, which poses significant risks to data privacy and security in data-sensitive application scenarios

employs Hessian-aware reconstruction techniques with improved importance weighting to stabilize ultra-low precision (3-4 bit) quantization for ViTs. However, both QAT and PTQ necessitate access to the original training data, which poses significant risks to data privacy and security in data-sensitive application scenarios. A.3. Data-Free Quantization for ...

2020
[11]

and our MaskAQ, apart from the corresponding real samples. C. Additional Discussions and Results As indicated in Sec.3.1 of the main body, we further offer additional discussions and more experimental results. C.1. Additional Results on Object Detection and Semantic Segmentation Tasks Apart from the image classification task in the main body, we further r...

2014
[12]

Table 3 suggests that our MaskAQ achieves great improvements over other approaches

and PSAQ-ViT V2 (Li et al., 2023a), we adopt the results reported in MimiQ (Choi et al., 2025). Table 3 suggests that our MaskAQ achieves great improvements over other approaches. For example, MaskAQ exhibits at most 0.92 and 1.24 gains on the COCO and ADE20K datasets, compared to state-of-the-art approaches,e.g., MimiQ. The above fact further confirms th...

2025

[1] [1]

L., Kiros, J

Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv:1607.06450,

Pith/arXiv arXiv

[2] [2]

An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint arXiv:2010.11929,

Dosovitskiy, A. An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint arXiv:2010.11929,

Pith/arXiv arXiv 2010

[3] [3]

Crafting papers on machine learning

Langley, P. Crafting papers on machine learning. In Langley, P. (ed.),Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207–1216, Stan- ford, CA,

2000

[4] [4]

10 Masked Attention Alignment for Data-Free Quantization of Vision Transformers Li, Y ., Kim, Y ., Lee, D., Kundu, S., and Panda, P

arXiv:2102.05426. 10 Masked Attention Alignment for Data-Free Quantization of Vision Transformers Li, Y ., Kim, Y ., Lee, D., Kundu, S., and Panda, P. Genq: Quantization in low data regimes with generative syn- thetic data. InEuropean Conference on Computer Vision, pp. 216–235. Springer,

arXiv

[5] [5]

Fq-vit: Post-training quantization for fully quantized vision trans- former.arXiv preprint arXiv:2111.13824,

Lin, Y ., Zhang, T., Sun, P., Li, Z., and Zhou, S. Fq-vit: Post-training quantization for fully quantized vision trans- former.arXiv preprint arXiv:2111.13824,

arXiv

[6] [6]

C., and Bialek, W

Tishby, N., Pereira, F. C., and Bialek, W. The informa- tion bottleneck method.arXiv preprint physics/0004057,

Pith/arXiv arXiv

[7] [7]

12 Masked Attention Alignment for Data-Free Quantization of Vision Transformers A

doi: 10.1109/CVPR.2017.544. 12 Masked Attention Alignment for Data-Free Quantization of Vision Transformers A. Details of Related Work As indicated in Sec.1 of the main body, we provide the comprehensive review of related work in this section, which mainly encompasses Vision Transformer architectures, data-driven quantization and data-free quantization ap...

work page doi:10.1109/cvpr.2017.544 2017

[8] [8]

Other lines of research modify the ViT architecture to better capture multi-scale and local visual structure

introduced a data-efficient training method with token-based distillation, achieving competitive performance on ImageNet without massive external datasets and thus facilitating practical ViT adoption. Other lines of research modify the ViT architecture to better capture multi-scale and local visual structure. For example, Swin Transformer (Liu et al., 202...

2020

[9] [9]

and video recognition (Arnab et al., 2021; Neimark et al., 2021). Therefore, the DFQ for ViTs are directly influenced by their architectural distinctions from Convolutional Neural Networks (CNNs), including patch-based tokenization, LayerNorm usage, attention-driven operations, and the distribution sensitivity of multi-head activations. A.2. Data-Driven Q...

2021

[10] [10]

However, both QAT and PTQ necessitate access to the original training data, which poses significant risks to data privacy and security in data-sensitive application scenarios

employs Hessian-aware reconstruction techniques with improved importance weighting to stabilize ultra-low precision (3-4 bit) quantization for ViTs. However, both QAT and PTQ necessitate access to the original training data, which poses significant risks to data privacy and security in data-sensitive application scenarios. A.3. Data-Free Quantization for ...

2020

[11] [11]

and our MaskAQ, apart from the corresponding real samples. C. Additional Discussions and Results As indicated in Sec.3.1 of the main body, we further offer additional discussions and more experimental results. C.1. Additional Results on Object Detection and Semantic Segmentation Tasks Apart from the image classification task in the main body, we further r...

2014

[12] [12]

Table 3 suggests that our MaskAQ achieves great improvements over other approaches

and PSAQ-ViT V2 (Li et al., 2023a), we adopt the results reported in MimiQ (Choi et al., 2025). Table 3 suggests that our MaskAQ achieves great improvements over other approaches. For example, MaskAQ exhibits at most 0.92 and 1.24 gains on the COCO and ADE20K datasets, compared to state-of-the-art approaches,e.g., MimiQ. The above fact further confirms th...

2025