REVIEW 5 major objections 5 minor 1 cited by
Compress image to patches for Vision Transformer
T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper replaces the patch-embedding stage of a Vision Transformer with a frozen learned-compression encoder plus a convolutional reshape, cutting self-attention tokens to a quarter and FLOPs by 63.35%, while improving from-scratch…
desk verdict A genuinely new plug-in idea—frozen learned-compression encoder as ViT tokenizer—with real FLOPs savings, but the accuracy claim overreaches and the evidence is thinner than the abstract suggests. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the CI2P module: a frozen bmshj2018 factorized learned-compression encoder (a CNN trained to reconstruct images under an MSE-plus-bitrate loss) followed by a PatchReshape convolutional stack, which replaces the patch embedding. The encoder downsamples the image into a compact latent; the reshape stack converts that latent into the exact $C\times H\times W$ token grid ViT expects. Its work is to compress the spatial dimension fourfold so that self-attention sees 64 tokens instead of 256, while the frozen encoder's weights supply convolutional inductive bias without any task-specific fine-tuning.
What would settle it
Encode and decode the ImageNet validation set with the bmshj2018 factorized quality-5 encoder at 256×256, measure the PSNR/SSIM of the reconstructions, and train a ViT-B/16 on those decoded images. If the reconstructions are visibly lossy or the ViT trained on them loses more than roughly one point of top-1 accuracy relative to training on the originals, the claim that the compression step preserves the visual information needed for classification is falsified.
Extended reading notes
Core claim
CI2P-ViT's central discovery is that the fixed, lossy compression latent of a factorized learned-compression encoder (bmshj2018 factorized, quality 5) is a suitable input for self-attention: the encoder maps a $256\times256$ image to a $[192,16,16]$ tensor, and a PatchReshape stack of convolutions converts that into a $[768,8,8]$ tensor, i.e., 64 patches instead of ViT-B/16's 256. Because the encoder is trained in isolation to reconstruct images under an MSE-plus-bitrate loss and is frozen afterward, the ViT structure remains unchanged, and the token-count reduction directly lowers the quadratic part of self-attention cost. The paper reports 63.35% FLOPs reduction at 256×256 (more at higher resolutions), a 92.37% from-scratch accuracy on Animals-10 against an 89.0% ViT-B/16 baseline, and about a 2× training speedup. A dual-scale attention variant computes attention at both the 16×16 and 8×8 scales, reducing parameters to 49.7M and reaching 77% ImageNet top-1 from scratch.
Load-bearing premise
The argument rests on the premise that the frozen compression encoder, trained only to minimize reconstruction error and bitrate, keeps enough class-discriminative detail in its fourfold smaller latent that the ViT has no less useful information than it would from raw patches.
Editorial extensions
If this is right
- CI2P-ViT cuts the self-attention token count to a quarter, reducing FLOPs by 63.35% at 256×256, with larger savings (65.21% at 384, 67.47% at 512) as resolution grows.
- From-scratch training on Animals-10 reaches 92.37% versus 89.0% for ViT-B/16, with roughly 2× faster training on the same hardware.
- Because the encoder is frozen, the efficiency gain does not require end-to-end fine-tuning of the compression backbone, and the rest of the ViT architecture is untouched.
- The dual-scale CI2P-ViTds variant cuts parameters from 88.96M to 49.7M and FLOPs by 72.15% while reaching 77% ImageNet top-1 from scratch.
- The authors expect the same module to improve as better learned-compression encoders become available, since the front-end is task-independent.
Reading between the lines
- The same frozen-compression front-end could be transferred to other sequence-based vision models that use patch embedding, such as masked autoencoders, without changing their training objective; the paper only demonstrates classification.
- The ImageNet result (72.9% vs 77.91% for ViT-B/16) hints that the quality-5 encoder discards some class-relevant detail at scale; a natural extension is to re-train or fine-tune the encoder with a classification-aware loss and measure whether the accuracy gap closes while keeping the token reduction.
- If the latent preserves enough detail, CI2P-style token reduction could benefit dense prediction tasks like detection and segmentation, where the paper's argument about preserving visual information would matter most; this is not tested in the paper.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CI2P-ViT, a hybrid architecture that replaces the patch embedding of a Vision Transformer with a compression module containing a frozen learned image-compression encoder (bmshj2018 factorized, quality=5) followed by convolutional/inverted-residual reshape layers. For 256x256 inputs the module reduces the self-attention token sequence from 256 to 64, and the paper reports a 63.35% FLOPs reduction, a 2x training speedup, and a from-scratch Animals-10 accuracy improvement (92.37% vs 89.0%) over ViT-B/16. A dual-scale variant, CI2P-ViTds, is also introduced, with 49.7M parameters and 72.15% FLOPs reduction at 256x256. ImageNet experiments report 72.9% top-1 for CI2P-ViT and 77% for CI2P-ViTds, compared with 77.91% for ViT-B/16.
Significance. If the accuracy claim survives scrutiny, the paper would demonstrate a drop-in efficiency module that preserves the standard ViT structure while reducing self-attention cost and injecting CNN inductive bias. The FLOPs arithmetic is transparent and the CNN-ViT control experiment is a useful ablation. The dual-scale variant is a sensible extension. However, the central accuracy claim is currently supported mainly by a single small-dataset run, while the paper's own ImageNet results show a 5-point drop relative to ViT-B/16. The manuscript also lacks reconstruction-fidelity or information-retention measurements for its stated lossless-preservation assumption, and the tensor notation in Section 3.1 is internally inconsistent. These issues prevent the paper from being accepted in its current form, but they are addressable with additional experiments and clarification.
major comments (5)
- [Section 4.1 and Table 1] The central claim that CI2P 'markedly enhances' accuracy is supported only by a single Animals-10 run (92.37% vs 89.0%) with no standard deviation, no number of seeds, and no description of the train/test split. The same table reports ImageNet top-1 of 72.9% for CI2P-ViT versus 77.91% for ViT-B/16, a 4.99-point drop, which directly contradicts the general claim in the abstract. Please provide multi-seed results with confidence intervals, report the dataset split procedure, and either restrict the accuracy claim to the Animals-10 setting or explain why the ImageNet result does not refute the claimed general enhancement.
- [Section 3.1, Eqs. (2)-(7)] The assertion that the compression latent preserves all visual information is not supported by any measurement. Equation (7) optimizes pixel MSE plus bitrate, not class-discriminative information, and the paper reports no reconstruction fidelity, no information-retention metric, and no examples on the classification datasets. This premise is load-bearing because the 4x token reduction could lose task-relevant details while leaving reconstruction quality high; the ImageNet accuracy drop is consistent with such a lossy bottleneck. Please report reconstruction PSNR/SSIM on the evaluation datasets and a linear-probing or information-retention measure of the latent, and either remove or qualify the sentence 'ensuring that all visual information from the original is preserved' in Section 3.1.
- [Section 3.1, Eqs. (2)-(4)] The tensor notation in Eqs. (2)-(4) is internally inconsistent and prevents reproduction. The text states that d is set to 32 and s is 4, but the output tensor is later described as [192, 16x16] for a 256x256 input, and the PatchReshape step halves the spatial dimension to [768, 8x8]. If d=32, a 256x256 input would produce an 8x8 spatial latent, not 16x16. Please rewrite the equations in standard tensor notation with consistent symbols for the downsampling factor and the channel multiplier, and define the exact tensor shapes for a 256x256 input.
- [Table 1] Table 1 lists identical FLOPs (23.13 G) for ViT-B/32 and ViT-B/16. For the same input resolution, ViT-B/32 uses a 32x32 patch and therefore processes 64 tokens, whereas ViT-B/16 processes 256 tokens, so the two models cannot have equal computational cost. This suggests the FLOPs numbers are not computed with a reliable method. Please recompute all FLOPs with a standard profiling tool and provide a breakdown that includes the CI2P encoder and PatchReshape overhead, so that the claimed 63.35% reduction can be independently verified.
- [Section 4.1] The comparison between CI2P-ViT and ViT-B/16 is confounded by pretraining status: CI2P-ViT uses a frozen encoder pretrained on a large corpus of natural images, while the ViT-B/16 baseline is trained from scratch. The CNN-ViT control trains the CNN from scratch, so it does not isolate the effect of the compression pretraining. The observed Animals-10 gain could therefore be due to transfer from the pretrained encoder rather than to the information-preserving property of the compression objective. Please add an ablation with a randomly initialized but frozen bmshj2018 encoder, or otherwise disentangle pretraining transfer from the compression-induced inductive bias.
minor comments (5)
- [Section 2.2] The paragraph on CvT cites [11] ('Escaping the big data paradigm with compact transformers'), which is the Compact Convolutional Transformer paper, not the Context Vision Transformer (CvT) paper by Wu et al. Please correct the citation and the model name.
- [Table 2] The first row of Table 2 begins with a stray '2' before '23.127 G', and the ViT-B/16 FLOPs value differs from Table 1 (23.127 G vs 23.13 G). Please unify the formatting and numbers.
- [Abstract] The abstract states FLOPs are 'measured in floating-point operations per second'; FLOPs is a count of operations, not a rate. Please rephrase to 'floating-point operations'.
- [Section 4.2] The ImageNet training description is very brief: 300 epochs from scratch with only random flipping is reported, but no learning-rate schedule, warmup, batch size, or hardware details are given. This makes the ImageNet result difficult to interpret or reproduce; please add a full training configuration.
- [General] No code or model checkpoints are provided. Given the domain-specific details of the CI2P module and the need to verify the FLOPs and tensor shapes, releasing code would substantially improve reproducibility.
Circularity Check
No significant circularity: the CI2P-ViT gains are reported empirical measurements with an externally pretrained, frozen compression encoder, not quantities forced by construction.
full rationale
The paper's derivation chain does not reduce any reported prediction to a fitted input or a self-citation. The CI2P encoder is taken from the external CompressAI/bmshj2018 model, trained with the reconstruction loss in Eq. (7), and its parameters are frozen during ViT training; the ViT accuracy and FLOPs numbers in Tables 1 and 2 are empirical outcomes, not quantities implied by the loss. The 63.35% FLOPs reduction follows from counting operations on the 8x8 token grid rather than the 16x16 token grid, and the accuracy claims are measured on test sets, not derived from the encoder objective. No fitted parameter is renamed as a prediction, and no load-bearing argument rests on a self-citation or an imported uniqueness theorem. The paper's unsupported assertion that the lossy compression 'ensuring that all visual information from the original is preserved' is an empirical/correctness concern about information retention, not a circularity: the statement is an assumption about the pretrained encoder, and the accuracy results are still evaluated independently. Thus the central claims have independent empirical content and should not be scored for circularity.
Assumptions & free parameters
free parameters (3)
- bmshj2018 factorized quality level =
5
- latent downsampling factor =
16 in actual encoder (equations state d=32, s=4)
- PatchReshape channel and spatial configuration =
maps [192, 16, 16] to [768, 8, 8]
assumptions (4)
- domain assumption The lossy compression encoder's latent representation preserves all information needed for classification.
- domain assumption CNN inductive bias from the frozen compression encoder improves ViT accuracy on small datasets.
- domain assumption The ViT-B/16 implementation in the mmlab framework with identical training settings is a valid baseline.
- domain assumption Positional information is handled adequately after patch replacement.
Cite this review
Pith. "Pith review of Compress image to patches for Vision Transformer." pith.science (2026). https://pith.science/paper/IZGP5YLN
@misc{pith2026250210120,
author = {Pith},
title = {Pith review of: Compress image to patches for Vision Transformer},
year = {2026},
howpublished = {\url{https://pith.science/paper/IZGP5YLN}},
note = {Machine review of arXiv:2502.10120}
}
read the original abstract
The Vision Transformer (ViT) has made significant strides in the field of computer vision. However, as the depth of the model and the resolution of the input images increase, the computational cost associated with training and running ViT models has surged dramatically. This paper proposes a hybrid model based on CNN and Vision Transformer, named CI2P-ViT. The model incorporates a module called CI2P, which utilizes the CompressAI encoder to compress images and subsequently generates a sequence of patches through a series of convolutions. CI2P can replace the Patch Embedding component in the ViT model, enabling seamless integration into existing ViT models. Compared to ViT-B/16, CI2P-ViT has the number of patches input to the self-attention layer reduced to a quarter of the original. This design not only significantly reduces the computational cost of the ViT model but also effectively enhances the model's accuracy by introducing the inductive bias properties of CNN. The ViT model's precision is markedly enhanced. When trained from the ground up on the Animals-10 dataset, CI2P-ViT achieved an accuracy rate of 92.37%, representing a 3.3% improvement over the ViT-B/16 baseline. Additionally, the model's computational operations, measured in floating-point operations per second (FLOPs), were diminished by 63.35%, and it exhibited a 2-fold increase in training velocity on identical hardware configurations.
Forward citations
Cited by 1 Pith paper
-
FTCFormer: Fuzzy Token Clustering Transformer for Image Classification
FTCFormer replaces grid-based downsampling in vision transformers with fuzzy clustering, shared-neighbor assignment, and channel-wise token merging, reporting consistent but small accuracy improvements over TCFormer.
Reference graph
Works this paper leans on
-
[1]
Communications of the ACM 60(6):84–90
Krizhevsky A, Sutskever I, Hinton GE (2017) Imagenet classification with deep convolutional neural networks. Communications of the ACM 60(6):84–90
work page 2017
-
[2]
Neural computation 1(4):541–551
LeCun Y, Boser B, Denker JS, et al (1989) Backpropagation applied to handwritten zip code recognition. Neural computation 1(4):541–551
work page 1989
-
[3]
Advances in neural information processing systems 9199(10.5555):2969,239–2969,250
Faster R (2015) Towards real-time object detection with region pro- posal networks. Advances in neural information processing systems 9199(10.5555):2969,239–2969,250
work page 2015
-
[4]
In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431–3440
Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431–3440
work page 2015
-
[5]
Advances in Neural Information Processing Systems 30:5998–6008
Vaswani A, Shazeer N, Parmar N, et al (2017) Attention is all you need. Advances in Neural Information Processing Systems 30:5998–6008
work page 2017
-
[6]
arXiv preprint arXiv:1810.04805
Devlin J, Chang MW, Lee K, et al (2018) Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805
arXiv 2018
-
[7]
arXiv preprint arXiv:2010.11929 13
Dosovitskiy A, Beyer L, Kolesnikov A, et al (2020) An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 13
arXiv 2020
-
[8]
In: Proceedings of the IEEE/CVF international conference on computer vision, pp 10,012–10,022
Liu Z, Lin Y, Cao Y, et al (2021) Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 10,012–10,022
work page 2021
Show all 21 references
-
[9]
In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778
He K, Zhang X, Ren S, et al (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778
2016
-
[10]
In: International conference on machine learning, PMLR, pp 10,347–10,357
Touvron H, Cord M, Douze M, et al (2021) Training data-efficient image transformers & distillation through attention. In: International conference on machine learning, PMLR, pp 10,347–10,357
2021
-
[11]
arXiv preprint arXiv:2104.05704
Hassani A, Walton S, Shah N, et al (2021) Escaping the big data paradigm with compact transformers. arXiv preprint arXiv:2104.05704
2021 arXiv
-
[12]
In: Proceedings of the IEEE/CVF international conference on computer vision, pp 579–588
Yuan K, Guo S, Liu Z, et al (2021) Incorporating convolution designs into visual transformers. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 579–588
2021
-
[13]
arXiv preprint arXiv:2011.03029
B´ egaint J, Racap´ e F, Feltman S, et al (2020) Compressai: a pytorch library and evaluation platform for end-to-end compression research. arXiv preprint arXiv:2011.03029
2020 arXiv
-
[14]
arXiv preprint arXiv:1802.01436
Ball´ e J, Minnen D, Singh S, et al (2018) Variational image compression with a scale hyperprior. arXiv preprint arXiv:1802.01436
2018 arXiv
-
[15]
Advances in neural information processing systems 31
Minnen D, Ball´ e J, Toderici GD (2018) Joint autoregressive and hierarchi- cal priors for learned image compression. Advances in neural information processing systems 31
2018
-
[16]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 7939–7948
Cheng Z, Sun H, Takeuchi M, et al (2020) Learned image compression with discretized gaussian mixture likelihoods and attention modules. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 7939–7948
2020
-
[17]
Mobilenets 10:151
Andrew G, Menglong Z, et al (2017) Efficient convolutional neural networks for mobile vision applications. Mobilenets 10:151
2017
-
[18]
URL https://www.kaggle.com/datasets/ alessiocorrado99/animals10/data
Alessio C (2020) Animals-10. URL https://www.kaggle.com/datasets/ alessiocorrado99/animals10/data
2020
-
[19]
In: Proceedings of the IEEE/CVF international conference on computer vision, pp 558–567
Yuan L, Chen Y, Wang T, et al (2021) Tokens-to-token vit: Training vision transformers from scratch on imagenet. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 558–567
2021
-
[20]
Advances in neural information processing systems 34:15,908–15,919 14
Han K, Xiao A, Wu E, et al (2021) Transformer in transformer. Advances in neural information processing systems 34:15,908–15,919 14
2021
-
[21]
arXiv preprint arXiv:2102.10882 15
Chu X, Tian Z, Zhang B, et al (2021) Conditional positional encodings for vision transformers. arXiv preprint arXiv:2102.10882 15
2021 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.