Compact Vision Transformer by Reduction of Kernel Complexity

Yancheng Wang; Yingzhen Yang

REVIEW 4 major objections 6 minor 98 references

Compact Vision Transformer by Reduction of Kernel Complexity

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Guided by a two-sided kernel-complexity bound, KCR-Transformer prunes MLP channels and improves accuracy on ViT and Swin at lower FLOPs.

desk verdict The empirical block is promising, but Theorem 3.1 is false as written and the 'provably small generalization error' claim does not survive contact with the paper. read the letter →

arxiv 2507.12780 v1 pith:4P67UUO7 submitted 2025-07-17 cs.CV cs.LG

Yancheng Wang , Yingzhen Yang This is my paper

classification cs.CVcs.LG

keywords visiontransformerchannelpruningkernelcomplexitygeneralizationboundtruncatednuclearnormNyströmapproximationGumbel-Softmaxmodelcompression

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to make vision transformer compression a theorem-guided operation rather than an empirical one. It introduces the KCR-Transformer block, which selects input and output channels of each transformer MLP with a differentiable Gumbel-Softmax mask during an architecture search, then retrains the pruned network under a regularizer that lowers a quantity called kernel complexity. The theoretical engine is a two-sided generalization bound: with high probability the expected squared loss lies between the training loss plus or minus a kernel-complexity term scaled by sample size, so a network with small kernel complexity and a well-fitted training loss has an expected loss that is provably close to its training loss. The paper claims this makes channel pruning generalization-aware, and its experiments report that replacing every block in ViT-S, ViT-B, Swin-T, and Swin-B with KCR blocks reduces FLOPs and parameters while raising ImageNet top-1 accuracy, with further gains on ADE20K segmentation and COCO detection.

What carries the argument

The load-bearing object is the gradient-flow recursion on the final-layer residual, $F^{(t)} - Y = (I_n - \eta K_n)^t(F^{(0)} - Y) = -(I_n - \eta K_n)^t Y$, which expresses the training loss in terms of powers of the normalized gram matrix. A function-class complexity bound over the feature-backbone hypothesis class converts that recursion into the two-sided KCR bound, whose kernel complexity term $KC(K)$ measures how few full-rank directions the gram matrix needs. Since $KC(K)$ shrinks when the tail eigenvalues are small, the paper optimizes a separable surrogate, the truncated nuclear norm $\|K\|_r = \operatorname{tr}(K_n) - \operatorname{tr}(U_r^\top K_n U_r)$, computed from approximate top eigenvectors of $K_n$ via the Nyström method; this surrogate is added to the cross-entropy loss during retraining.

What would settle it

Compute the empirical KCR upper and lower bounds in Eq. (2) from the evolving gram matrix during end-to-end retraining of a KCR-Swin model; if the validation expected loss systematically lies outside the bounds at late epochs, the theorem does not apply to the actual training procedure. A complementary ablation is to retrain the same searched architecture without the kernel-complexity regularizer: if validation accuracy is unchanged while the measured kernel complexity climbs, the regularizer is not the operative cause of the reported gains.

Watch

Extended reading notes

Core claim

The central discovery, on the paper's own terms, is that the generalization error of a vision transformer is controlled from both sides by its training loss and the eigenvalue tail of the normalized gram matrix $K_n = FF^\top/n$ of its penultimate-layer features. Theorem 3.1 states that for every $x > 0$, with probability at least $1 - \exp(-x)$, $$\|(I_n - \eta K_n)^t Y\|$_F^{2}$ - KC(K) - x/n \lesssim L_D(\mathrm{NN}_W) \lesssim \|(I_n - \eta K_n)^t Y\|$_F^{2}$ + KC(K) + x/n,$$ where the kernel complexity is $KC(K) = \min_{h \in [0,r_0]} \left( h/n + \sqrt{n^{-1}\sum_{i=h+1}^{r_0} \hat\lambda_i} \right)$ with $\hat\lambda_i$ the eigenvalues of $K_n$. Because gradient descent drives the training-loss term small, a pruned network whose kernel complexity is also small inherits a small expected loss. The paper turns this into a pruning recipe: choose MLP channels with Gumbel-Softmax masks to reduce FLOPs, then retrain with a separable low-rank approximation of the truncated nuclear norm as a regularizer, so that optimization acts directly on the quantity controlling the bound.

Load-bearing premise

The load-bearing premise is that the two-sided bound proved for gradient descent on the final linear layer, with the feature backbone frozen and a squared loss, still describes the actual algorithm, which searches, prunes, and retrains all weights with cross-entropy; if full training leaves that linearized regime, the provable-generalization conclusion reduces to a heuristic.

Editorial extensions

If this is right

For all four tested backbones, KCR-Transformer blocks reduce ImageNet FLOPs by 0.4–3.1 G and parameters by 1.1–17.8 M while improving top-1 accuracy by 0.9–1.2 points (Swin-B: 83.5% at 15.4 G to 84.7% at 12.6 G).
The approximated KCR upper and lower bounds remain close to the validation expected loss across training, so the bound is tight enough to serve as an online generalization indicator during retraining.
The same block swap improves ADE20K semantic segmentation mIoU with a Swin-B backbone (51.6 to 52.4) and COCO instance segmentation box mAP (51.9 to 52.5), showing the recipe transfers beyond classification.
The kernel-complexity regularizer and the channel search together reduce the measured kernel complexity on ImageNet (e.g., ViT-B from 4.35 to 0.52), which the paper presents as direct evidence that the compressed network is better conditioned for generalization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the two-sided bound holds under the full end-to-end training the algorithm actually runs, kernel complexity becomes a cheap, computable proxy for generalization that could rank architectures or trigger early stopping without a validation set.
The same differentiable MLP channel mask applies in principle to transformers outside vision, since language-model MLPs spend most of their FLOPs in the same linear layers; the paper only tests vision backbones, so this is an extension rather than a claim.
The Nyström landmark set and rank ratio are chosen once by cross-validation; making the landmarks adapt to the evolving feature distribution during retraining could tighten the approximate truncated nuclear norm and the bound.
A direct ablation separating search from regularizer—randomly pruning the same channels while keeping the KCR loss—would show whether the accuracy gains come from lowering kernel complexity or from the retraining schedule.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the block itself is a reasonable contribution, but the theorem that is supposed to back it is false as printed, and the 'provably' language has to go.

What's actually new: a Gumbel-Softmax channel-selection search on the MLP channels of ViT/Swin blocks, with a truncated nuclear norm regularizer approximated by Nyström, plus a search-retrain pipeline. Reported numbers are good: 1–1.2 points accuracy gain with 10–20% FLOPs reduction across ViT-S/B and Swin-T/B on ImageNet, plus gains on COCO and ADE20K. If those reproduce, this is a useful drop-in block. The authors also give implementation and hyperparameter-selection details, which is more than many such papers.

Soft spots, in order of size:

1. Theorem 3.1 as stated is dimensionally wrong. Y has n rows, so the residual term ||(I − ηK_n)^t Y||_F^2 is O(n) — it is the unnormalized training error. The expected loss L_D is per-sample, O(1). For large n the lower bound fails and the upper bound is vacuous. The correct empirical term is (1/n) times that squared norm, which the proof in Appendix B should have produced but did not. This is not a nitpick; it invalidates the 'provably small generalization error' conclusion.

2. Even after rescaling, the proof covers only gradient descent on the last layer W1 with features frozen and squared loss, while Algorithm 1 trains all weights with cross-entropy after a Gumbel search. The theorem and the algorithm are different objects, and the paper does not bridge them.

3. The empirical validation is partly circular: Table 4's KCL reduction measures the approximate TNN that Eq. (4) directly minimizes, so a drop mostly confirms the optimizer worked rather than showing generalization.

4. There are no error bars, no code release, and the baseline numbers come from different training recipes than the 300-epoch EMA schedule used for the KCR models. Controlled same-recipe comparisons are needed before the accuracy claim is credible.

There is also a dimension typo in the gram matrix definition: K = F^T F should be F F^T.

Bottom line: the method is plausible and worth a proper referee, but the paper needs major revision: rescale the theorem, either extend the proof to the actual training procedure or honestly soften the claim, add same-recipe baselines, and release code. I would not want to see this desk-rejected; I would want to see it reviewed with these issues on the table.

Referee Report

4 major / 6 minor

Summary. The paper proposes KCR-Transformer, a compact vision transformer block that prunes input/output channels of the MLP layers in transformer blocks via differentiable Gumbel-Softmax channel selection. The pruning is guided by a claimed sharp generalization bound (Theorem 3.1) based on a kernel complexity (KC) term, which is minimized through an efficiently computable Nyström approximation of the truncated nuclear norm added as a separable regularizer to the cross-entropy loss. The training pipeline consists of an architecture-search stage followed by a retraining stage. Experiments on ImageNet-1k classification, ADE20K semantic/instance segmentation, and COCO object detection report reduced FLOPs and parameter counts while maintaining or improving accuracy compared with ViT and Swin baselines.

Significance. If Theorem 3.1 were correct and the experiments properly controlled, the paper would be a significant contribution: it offers a compact transformer building block, a theoretically motivated and computationally tractable regularizer, and broad empirical coverage across classification, segmentation, and detection. The Nyström-based approximation making the truncated nuclear norm separable and SGD-optimizable is a practical strength, as is the compatibility of the block with popular transformer backbones. However, the central theoretical claim is not established as stated: the bound in Theorem 3.1 mixes scales, and the proof applies only to a frozen-feature squared-loss setting that Algorithm 1 does not implement. The empirical evidence also lacks controlled baselines and error bars. With correction and rescoping, the method may still be of interest, but the current manuscript overclaims its theoretical guarantees.

major comments (4)

[Section 3.2, Theorem 3.1 and Appendix B, Eq. (8)] The displayed bound mixes an unnormalized residual with a per-sample loss. With Y in R^{n x C}, ||(I_n - eta K_n)^t Y||_F^2 is O(n), whereas L_D(NN_W) is a per-sample expected squared error of order 1 (equal to 1 at the initialization W1(0)=0 used in the proof). Taking eta -> 0 makes the lower bound of Eq. (2) of order n against a loss of order 1, which cannot hold for growing n regardless of the constant hidden in the symbol "≲". The proof should replace the residual by its normalized version (1/n)||(I_n - eta K_n)^t Y||_F^2; without this normalization Theorem 3.1 is false as stated, and the 'provably small generalization error' claim does not follow.
[Section 3.2, Algorithm 1, and Appendix B, Eqs. (5)-(7)] Theorem 3.1 is proved only for gradient descent on W1 with the feature backbone F frozen and a squared-loss recursion. Algorithm 1, however, trains all weights W including the feature backbone, optimizes cross-entropy after a warm-up phase, and in the search stage also optimizes architecture parameters through Gumbel-Softmax. The recursion leading to (I_n - eta K_n)^t Y breaks once F changes during training, and the actual training objective is not the squared error used in the theorem. Consequently, Theorem 3.1 does not certify the generalization of the networks produced by Algorithm 1; at best it motivates the TNN/KC regularizer as a heuristic. The claims in the abstract and introduction should be scoped accordingly, or a theorem covering the actual training algorithm should be supplied.
[Section 4.1, Table 1] The accuracy comparisons in Table 1 are against published baseline numbers for ViT-S, ViT-B, Swin-T, and Swin-B, not against those models retrained under the KCR pipeline (300-epoch AdamW training, EMA inference, data augmentation, etc.). Because KCR models are retrained from scratch with this strong recipe, the reported Top-1 gains of 0.7-1.2% and the FLOPs reductions cannot be unambiguously attributed to the KCR block. Please retrain all baseline models under the identical training recipe, report means and standard deviations over multiple seeds, and release code/checkpoints to make the empirical claims reproducible.
[Section 4.5, Table 4] The reported KCL reduction (e.g., 4.12 to 0.65 for ViT-S) is partly by construction: Eq. (4) explicitly minimizes an approximate truncated nuclear norm / kernel complexity, which is the same quantity that the KCL column in Table 4 reports. The table thus confirms that the optimizer reduces the regularized objective, not that the compression independently improves generalization. The generalization claim should rest on the accuracy/efficiency comparisons, which need the controlled baselines requested above; otherwise the ablation is circular as evidence for the generalization-aware claim.

minor comments (6)

[Abstract] The abstract refers to 'TCR-Transformers' in the final sentence; this should read 'KCR-Transformers'.
[Section 4.5] The section title says 'with three different vision transformers' but Table 4 compares four models (ViT-S, ViT-B, Swin-T, Swin-B).
[Section 3.2 and Section 4.2] The notation KCL is used in Section 4.2 and Section 4.5, but only KC(K) is defined in Section 3.2; the relationship between KCL, KC, and A-KC should be made explicit.
[Algorithm 1, line 12] The line 'Perform gradient descent with batch Bj using the loss CE(t)_j by Eq. (4)' is ambiguous because Eq. (4) includes the KCR term; the warm-up loss should be written as the plain cross-entropy loss.
[Figure 1] The caption mentions 'approximated KCL upper/lower bounds' while the text uses 'KCR upper/lower bounds'; the figure also lacks axis labels and a legend, making it hard to verify the claimed tight correlation.
[Section 2.2] There is a typo in 'components fo DNNs' in the first sentence of the related-work paragraph on kernel methods; it should read 'components of DNNs'.

Circularity Check

1 steps flagged · score 4.0 of 10

The core bound in Theorem 3.1 is an external Rademacher complexity result and is not circular; the main circular step is the KCL ablation, which reports the exact regularizer minimized during training as if it independently validated generalization.

fitted input called prediction [Section 4.5 (Ablation Study on the Effects of KCR-Transformer in Reducing the KCL), Eq. (4), Table 4]
"KCR (W) is a term in the training loss. ... L(t) train,j(W) =CE(t) j + ηKCRj(W), (4) ... It is observed in Table 4 that KCR models exhibit significantly reduced KCL compared to the baseline models with even less parameter size and FLOPs, demonstrating enhanced generalization of the image classification task with better top-1 classification accuracy."

The KCL column in Table 4 is the same approximate truncated nuclear norm / kernel complexity term that Eq. (4) directly minimizes through KCR_j(W) in the training loss L_train = CE + η KCR_j(W). The reduction from KCL roughly 4 to roughly 0.5 therefore only shows that gradient descent successfully minimized the regularizer; it is not independent evidence about generalization. Presenting that drop as 'demonstrating enhanced generalization' is reporting a fitted objective as an empirical prediction, i.e. the claimed validation is forced by construction.

full rationale

Theorem 3.1 is not circular: its proof invokes the external local Rademacher complexity results of Bartlett et al. (2005), and the claimed bound relates the expected loss to the squared training residual and a kernel-complexity term, neither of which is defined in terms of the theorem's conclusion. The ImageNet, COCO, and ADE20K comparisons are external benchmarks, and the FLOPs/parameter reductions are architectural decisions, not reductions to the paper's own fit. No load-bearing self-citation chain is present; the only same-author citation (Wang et al. 2022a) appears in related work and is not used as a premise of the main argument. The apparent scale inconsistency in Theorem 3.1 identified by the reviewer is a correctness concern, not a circularity, so it is not scored here. The one genuine circular step is the KCL ablation in Section 4.5: the reported metric is the very loss term being optimized in Eq. (4), so its decrease cannot serve as independent validation of improved generalization. Because this circularity affects an auxiliary empirical claim while the central derivation and external benchmarks retain independent content, the score is 4 rather than higher.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The method depends on several tuned constants and unproved bridges between theory and practice. The free parameters (eta, lambda, gamma, m, t_warm, Gumbel schedule) are chosen by cross-validation or by hand, and the selected values for lambda and gamma are not reported. The three key unproved links, the Rademacher-to-KC mapping, the frozen-feature recursion, and the Nyström fidelity, are load-bearing for the 'provably small generalization error' claim.

free parameters (6)

eta (KCR regularizer weight) = 1
Chosen by 5-fold cross-validation from {0.1, 0.5, 1, 5, 10, 50, 100}; controls how strongly the TNN/KC term is minimized during retraining.
lambda (search cost weight) = not reported
Selected by 5-fold CV from 0.1 to 0.5 with step 0.05; balances classification accuracy against FLOPs during the Gumbel search; chosen value is not stated.
gamma (rank ratio) = not reported
Defines the truncation rank r = ceil(gamma * min(n, d)) for the TNN; chosen from {0.05, ..., 0.5} by CV; determines which eigenvalues are penalized; chosen value is not stated.
m (Nystrom landmark count) = 100000
Number of landmark features for the Nyström approximation, selected from {10000, ..., 200000} by 5-fold CV; controls the fidelity of the approximate TNN and the cost of the algorithm.
t_warm = 90
Warm-up epochs before the KC regularizer is enabled; chosen from {0.1, ..., 0.6} * ttrain by CV; affects final accuracy and the reported KCL values.
Gumbel temperature schedule = tau_0 = 4.5, decay 0.95 per epoch
Hand-set annealing schedule for the Gumbel-Softmax temperature in the search stage; shapes which channels are selected; no ablation is provided.

assumptions (4)

domain assumption The local Rademacher complexity of the DNN function class is bounded by the kernel complexity KC defined on the feature gram matrix.
Invoked in Appendix B via (Bartlett et al., 2005, Theorem 3.3, Corollary 6.7). The paper does not prove that the transformer class satisfies the entropy conditions linking local Rademacher complexity to the eigenvalue tail of the feature gram matrix.
domain assumption Training dynamics are governed by the linearized recursion on W1 with fixed features F.
Appendix B Eq (5)-(7) assumes only W1 is updated and F is constant. Algorithm 1 trains all weights and the Gumbel parameters, so the theorem's premise does not hold during retraining.
domain assumption The Nyström approximation faithfully estimates the TNN and KC of the true large-scale gram matrix.
Sections 3.2 and 4.2 state the approximate TNN is 'expected to be close' to the true TNN for each h; no error bound is provided for m=100000 on ImageNet.
ad hoc to paper Cross-entropy training is compatible with the squared-loss bound in Theorem 3.1.
The bound in Eq (2) is stated for the squared loss ||(In - eta*Kn)^t Y||^2_F, while Eq (4) optimizes cross-entropy; the paper does not reconcile the two losses.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Compact Vision Transformer by Reduction of Kernel Complexity." pith.science (2026). https://pith.science/paper/4P67UUO7

@misc{pith2026250712780,
  author       = {Pith},
  title        = {Pith review of: Compact Vision Transformer by Reduction of Kernel Complexity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4P67UUO7}},
  note         = {Machine review of arXiv:2507.12780}
}

read the original abstract

Self-attention and transformer architectures have become foundational components in modern deep learning. Recent efforts have integrated transformer blocks into compact neural architectures for computer vision, giving rise to various efficient vision transformers. In this work, we introduce Transformer with Kernel Complexity Reduction, or KCR-Transformer, a compact transformer block equipped with differentiable channel selection, guided by a novel and sharp theoretical generalization bound. KCR-Transformer performs input/output channel selection in the MLP layers of transformer blocks to reduce the computational cost. Furthermore, we provide a rigorous theoretical analysis establishing a tight generalization bound for networks equipped with KCR-Transformer blocks. Leveraging such strong theoretical results, the channel pruning by KCR-Transformer is conducted in a generalization-aware manner, ensuring that the resulting network retains a provably small generalization error. Our KCR-Transformer is compatible with many popular and compact transformer networks, such as ViT and Swin, and it reduces the FLOPs of the vision transformers while maintaining or even improving the prediction accuracy. In the experiments, we replace all the transformer blocks in the vision transformers with KCR-Transformer blocks, leading to KCR-Transformer networks with different backbones. The resulting TCR-Transformers achieve superior performance on various computer vision tasks, achieving even better performance than the original models with even less FLOPs and parameters.

Figures

Figures reproduced from arXiv: 2507.12780 by the authors.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

98 extracted references · 70 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

Du, Wei Hu, Zhiyuan Li, and Ruosong Wang

Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning (ICML), volume 97 of Machine Learning Research, pp.\ 322--332. PMLR , 2019

2019
[3]

Implicit gradient regularization

David Barrett and Benoit Dherin. Implicit gradient regularization. In International Conference on Learning Representations, 2021

2021
[4]

Bartlett, Olivier Bousquet, and Shahar Mendelson

Peter L. Bartlett, Olivier Bousquet, and Shahar Mendelson. Local rademacher complexities. Ann. Statist., 33 0 (4): 0 1497--1537, 08 2005

2005
[5]

To understand deep learning we need to understand kernel learning

Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning we need to understand kernel learning. In International Conference on Machine Learning, pp.\ 541--549. PMLR, 2018

2018
[6]

Token merging: Your vit but faster

Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. ICLR, 2023

2023
[7]

Learned thresholds token merging and pruning for vision transformers

Maxim Bonnaerens and Joni Dambre. Learned thresholds token merging and pruning for vision transformers. Transactions on Machine Learning Research, 2023. ISSN 2835-8856

2023
[8]

Cvtstego-net: A convolutional vision transformer architecture for spatial image steganalysis

Mario Alejandro Bravo-Ortiz, Esteban Mercado-Ruiz, Juan Pablo Villa-Pulgarin, Carlos Angel Hormaza-Cardona, Sebastian Qui \ n ones-Arredondo, Harold Brayan Arteaga-Arteaga, Simon Orozco-Arias, Oscar Cardona-Morales, and Reinel Tabares-Soto. Cvtstego-net: A convolutional vision transformer architecture for spatial image steganalysis. Journal of Information...

2024

Show all 98 references

[9]

Efficientvit: Enhanced linear attention for high-resolution low-computation visual recognition

Han Cai, Chuang Gan, and Song Han. Efficientvit: Enhanced linear attention for high-resolution low-computation visual recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023

2023
[10]

Cascade R-CNN: high quality object detection and instance segmentation

Zhaowei Cai and Nuno Vasconcelos. Cascade R-CNN: high quality object detection and instance segmentation. IEEE Trans. Pattern Anal. Mach. Intell. , 43 0 (5): 0 1483--1498, 2021. doi:10.1109/TPAMI.2019.2956516. URL https://doi.org/10.1109/TPAMI.2019.2956516

2021
[11]

A kernel analysis of feature learning in deep neural networks

Abdulkadir Canatar and Cengiz Pehlevan. A kernel analysis of feature learning in deep neural networks. In 2022 58th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp.\ 1--8. IEEE, 2022

2022
[12]

Generalization bounds of stochastic gradient descent for wide and deep neural networks

Yuan Cao and Quanquan Gu. Generalization bounds of stochastic gradient descent for wide and deep neural networks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alch \' e - Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Process...

2019
[13]

End-to-end object detection with transformers

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part I 16, pp.\ ...

2020
[14]

Vision transformer slimming: Multi-dimension searching in continuous optimization space

Arnav Chavan, Zhiqiang Shen, Zhuang Liu, Zechun Liu, Kwang-Ting Cheng, and Eric P Xing. Vision transformer slimming: Multi-dimension searching in continuous optimization space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 4931--4941, 2022

2022
[15]

CrossVit: Cross-attention multi-scale vision transformer for image classification

Chun-Fu Chen, Quanfu Fan, and Rameswar Panda. CrossVit: Cross-attention multi-scale vision transformer for image classification . In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021 a

2021
[16]

Autoformer: Searching transformers for visual recognition

Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 12270--12280, 2021 b

2021
[17]

Chasing sparsity in vision transformers: An end-to-end exploration

Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transformers: An end-to-end exploration. Advances in Neural Information Processing Systems, 34: 0 19974--19988, 2021 c

2021
[18]

Calibrating transformers via sparse gaussian processes

Wenlong Chen and Yingzhen Li. Calibrating transformers via sparse gaussian processes. arXiv preprint arXiv:2303.02444, 2023

2023 arXiv
[19]

Neural architecture search on imagenet in four GPU hours: A theoretically inspired perspective

Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in four GPU hours: A theoretically inspired perspective. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021 d

2021
[20]

Primal-attention: Self-attention through asymmetric kernel svd in primal representation

Yingyi Chen, Qinghua Tao, Francesco Tonin, and Johan Suykens. Primal-attention: Self-attention through asymmetric kernel svd in primal representation. Advances in Neural Information Processing Systems, 36: 0 65088--65101, 2023

2023
[21]

Mobile-former: Bridging mobilenet and transformer

Yinpeng Chen, Xiyang Dai, Dongdong Chen, Mengchen Liu, Xiaoyi Dong, Lu Yuan, and Zicheng Liu. Mobile-former: Bridging mobilenet and transformer. arXiv preprint arXiv:2108.05895, 2021 e

2021 arXiv
[22]

Kerple: Kernelized relative positional embedding for length extrapolation

Ta-Chung Chi, Ting-Han Fan, Peter J Ramadge, and Alexander Rudnicky. Kerple: Kernelized relative positional embedding for length extrapolation. Advances in Neural Information Processing Systems, 35: 0 8386--8399, 2022

2022
[23]

Colwell, and Adrian Weller

Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tam \' a s Sarl \' o s, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J. Colwell, and Adrian Weller. Rethinking attention with perfo...

2021
[24]

Lee, and Mahdi Soltanolkotabi

Alexandru Damian, Jason D. Lee, and Mahdi Soltanolkotabi. Neural networks can learn representations with gradient descent. In Po - Ling Loh and Maxim Raginsky (eds.), Conference on Learning Theory, 2-5 July 2022, London, UK , volume 178 of Proceedings of Machine Learning Resea...

2022
[25]

Neuralef: Deconstructing kernels by deep neural networks

Zhijie Deng, Jiaxin Shi, and Jun Zhu. Neuralef: Deconstructing kernels by deep neural networks. In International Conference on Machine Learning, pp.\ 4976--4992. PMLR, 2022

2022
[26]

An image is worth 16x16 words: Transformers for image recognition at scale

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

2021
[27]

An image is worth 16x16 words: Transformers for image recognition at scale

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

2021
[28]

Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel

Stanislav Fort, Leo Scherlis, Surya Ganguli, and Jascha Sohl-Dickstein. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. In Advances in Neural Information Processing Systems, volume 33, pp....

2020
[29]

Linearized two-layers neural networks in high dimension

Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Linearized two-layers neural networks in high dimension . The Annals of Statistics, 49 0 (2): 0 1029 -- 1054, 2021

2021
[30]

NASV it: Neural architecture search for efficient vision transformers with gradient conflict aware supernet training

Chengyue Gong, Dilin Wang, Meng Li, Xinlei Chen, Zhicheng Yan, Yuandong Tian, qiang liu, and Vikas Chandra. NASV it: Neural architecture search for efficient vision transformers with gradient conflict aware supernet training. In International Conference on Learning Representat...

2022
[31]

Levit: a vision transformer in convnet's clothing for faster inference

Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Herv \'e J \'e gou, and Matthijs Douze. Levit: a vision transformer in convnet's clothing for faster inference. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 1225...

2021
[32]

lossless

Lingyu Gu, Yongqi Du, Yuan Zhang, Di Xie, Shiliang Pu, Robert C. Qiu, and Zhenyu Liao. "lossless" compression of deep neural networks: A high-dimensional neural tangent kernel approach. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advan...

2022
[33]

Deep learning with kernels through rkhm and the perron-frobenius operator

Yuka Hashimoto, Masahiro Ikeda, and Hachem Kadri. Deep learning with kernels through rkhm and the perron-frobenius operator. Advances in Neural Information Processing Systems, 36: 0 50677--50696, 2023

2023
[34]

Rethinking spatial dimensions of vision transformers

Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions of vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021

2021
[35]

Hierarchical kernels in deep kernel learning

Wentao Huang, Houbao Lu, and Haizhang Zhang. Hierarchical kernels in deep kernel learning. Journal of Machine Learning Research, 24 0 (391): 0 1--30, 2023

2023
[36]

Neural tangent kernel: Convergence and generalization in neural networks

Arthur Jacot, Franck Gabriel, and Cl \'e ment Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018 a

2018
[37]

Neural tangent kernel: Convergence and generalization in neural networks

Arthur Jacot, Cl \' e ment Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicol \` o Cesa - Bianchi, and Roman Garnett (eds.), Advances in Neural Informa...

2018
[38]

Token fusion: Bridging the gap between token pruning and token merging

Minchul Kim, Shangqian Gao, Yen-Chang Hsu, Yilin Shen, and Hongxia Jin. Token fusion: Bridging the gap between token pruning and token merging. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp.\ 1383--1392, 2024

2024
[39]

Local rademacher complexities and oracle inequalities in risk minimization

Vladimir Koltchinskii. Local rademacher complexities and oracle inequalities in risk minimization. Ann. Statist., 34 0 (6): 0 2593--2656, 12 2006

2006
[40]

Spvit: Enabling faster vision transformers via latency-aware soft token pruning

Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, et al. Spvit: Enabling faster vision transformers via latency-aware soft token pruning. In Computer Vision--ECCV 2022: 17th European Conference, Tel Aviv, Israel, ...

2022
[41]

Sampling methods for the nystr \" o m method

Sanjiv Kumar, Mehryar Mohri, and Ameet Talwalkar. Sampling methods for the nystr \" o m method. J. Mach. Learn. Res., 13: 0 981--1006, 2012. doi:10.5555/2503308.2343678. URL https://dl.acm.org/doi/10.5555/2503308.2343678

2012
[42]

Neural tangent kernel analysis of deep narrow neural networks

Jongmin Lee, Joo Young Choi, Ernest K Ryu, and Albert No. Neural tangent kernel analysis of deep narrow neural networks. In International Conference on Machine Learning, pp.\ 12282--12351. PMLR, 2022

2022
[43]

Q-vit: Accurate and fully quantized low-bit vision transformer

Yanjing Li, Sheng Xu, Baochang Zhang, Xianbin Cao, Peng Gao, and Guodong Guo. Q-vit: Accurate and fully quantized low-bit vision transformer. Advances in Neural Information Processing Systems (NeurIPS), 35: 0 34451--34463, 2022 a

2022
[44]

Efficientformer: Vision transformers at mobilenet speed

Yanyu Li, Geng Yuan, Yang Wen, Ju Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang, and Jian Ren. Efficientformer: Vision transformers at mobilenet speed. Advances in Neural Information Processing Systems, 35: 0 12934--12949, 2022 b

2022
[45]

Repq-vit: Scale reparameterization for post-training quantization of vision transformers

Zhikai Li, Junrui Xiao, Lianwei Yang, and Qingyi Gu. Repq-vit: Scale reparameterization for post-training quantization of vision transformers. In IEEE International Conference on Computer Vision (ICCV), pp.\ 17227--17236, 2023

2023
[46]

Swinir: Image restoration using swin transformer

Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. ICCV, 2021

2021
[47]

Microsoft coco: Common objects in context

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp.\ 740--755. Springer, 2014

2014
[48]

Fq-vit: Post-training quantization for fully quantized vision transformer

Yang Lin, Tianyu Zhang, Peiqin Sun, Zheng Li, and Shuchang Zhou. Fq-vit: Post-training quantization for fully quantized vision transformer. In International Joint Conference on Artificial Intelligence (IJCAI), pp.\ 1173--1179. ijcai.org, 2022

2022
[49]

Visual instruction tuning

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information ...

2023
[50]

Updp: A unified progressive depth pruner for cnn and vision transformer

Ji Liu, Dehua Tang, Yuanxian Huang, Li Zhang, Xiaocheng Zeng, Dong Li, Mingjie Lu, Jinzhang Peng, Yu Wang, Fan Jiang, et al. Updp: A unified progressive depth pruner for cnn and vision transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp...

2024
[51]

Efficientvit: Memory efficient vision transformer with cascaded group attention

Xinyu Liu, Houwen Peng, Ningxin Zheng, Yuqing Yang, Han Hu, and Yixuan Yuan. Efficientvit: Memory efficient vision transformer with cascaded group attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 14420--14430, 2023 b

2023
[52]

Revisiting token pruning for object detection and instance segmentation

Yifei Liu, Mathias Gehrig, Nico Messikommer, Marco Cannici, and Davide Scaramuzza. Revisiting token pruning for object detection and instance segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp.\ 2658--2668, 2024 b

2024
[53]

Swin transformer: Hierarchical vision transformer using shifted windows

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030, 2021 a

2021 arXiv
[54]

Post-training quantization for vision transformer

Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quantization for vision transformer. Advances in Neural Information Processing Systems (NeurIPS), 34: 0 28092--28103, 2021 b

2021
[55]

Prune and merge: Efficient token compression for vision transformer with spatial information preserved

Junzhu Mao, Yang Shen, Jinyang Guo, Yazhou Yao, Xiansheng Hua, and Hengtao Shen. Prune and merge: Efficient token compression for vision transformer with spatial information preserved. IEEE Transactions on Multimedia, 2025

2025
[56]

Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer

Sachin Mehta and Mohammad Rastegari. Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer. ICLR, 2022

2022
[57]

Geometric parameters of kernel machines

Shahar Mendelson. Geometric parameters of kernel machines. In Jyrki Kivinen and Robert H. Sloan (eds.), Conference on Computational Learning Theory, volume 2375 of Lecture Notes in Computer Science, pp.\ 29--43. Springer, 2002

2002
[58]

Jisoo Mok, Byunggook Na, Ji - Hoon Kim, Dongyoon Han, and Sungroh Yoon. Demystifying the neural tangent kernel from a practical perspective: Can it be trusted for neural architecture search without training? In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CV...

2022
[59]

Kernel analysis of deep networks

Gr \'e goire Montavon, Mikio L Braun, and Klaus-Robert M \"u ller. Kernel analysis of deep networks. Journal of Machine Learning Research, 12 0 (9), 2011

2011
[60]

Fourierformer: Transformer meets generalized fourier integral theorem

Tan Nguyen, Minh Pham, Tam Nguyen, Khai Nguyen, Stanley Osher, and Nhat Ho. Fourierformer: Transformer meets generalized fourier integral theorem. Advances in Neural Information Processing Systems, 35: 0 29319--29335, 2022

2022
[61]

Bertozzi, Richard G

Tan Minh Nguyen, Tam Minh Nguyen, Nhat Ho, Andrea L. Bertozzi, Richard G. Baraniuk, and Stanley J. Osher. A primal-dual framework for transformers and neural networks. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 202...

2023
[62]

Eshaan Nichani, Yu Bai, and Jason D. Lee. Identifying good directions to escape the NTK regime and efficiently learn low-degree plus sparse polynomials. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Process...

2022
[63]

A survey on efficient vision transformers: algorithms, techniques, and performance benchmarking

Lorenzo Papa, Paolo Russo, Irene Amerini, and Luping Zhou. A survey on efficient vision transformers: algorithms, techniques, and performance benchmarking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

2024
[64]

u gner, Bertrand Charpentier, Simon Geisler, Morgane Ayle, and Stephan G \

John Rachwan, Daniel Z \" u gner, Bertrand Charpentier, Simon Geisler, Morgane Ayle, and Stephan G \" u nnemann. Winning the lottery ahead of time: Efficient early network pruning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesv \' a ri, Gang Niu, and Sivan Sab...

2022
[65]

Designing network design spaces

Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Doll \'a r. Designing network design spaces. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020

2020
[66]

Dynamicvit: Efficient vision transformers with dynamic token sparsification

Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. arXiv preprint arXiv:2106.02034, 2021

2021 arXiv
[67]

Imagenet large scale visual recognition challenge

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115 0 (3): 0 211--252, 2015

2015
[68]

Analyzing finite neural networks: Can we trust neural tangent kernel theory? In Mathematical and Scientific Machine Learning, pp.\ 868--895

Mariia Seleznova and Gitta Kutyniok. Analyzing finite neural networks: Can we trust neural tangent kernel theory? In Mathematical and Scientific Machine Learning, pp.\ 868--895. PMLR, 2022

2022
[69]

FLAVA: A foundational language and vision alignment model

Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. FLAVA: A foundational language and vision alignment model. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, ...

2022
[70]

Implicit kernel attention

Kyungwoo Song, Yohan Jung, Dongjun Kim, and Il-Chul Moon. Implicit kernel attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.\ 9713--9721, 2021

2021
[71]

Vitas: Vision transformer architecture search

Xiu Su, Shan You, Jiyang Xie, Mingkai Zheng, Fei Wang, Chen Qian, Changshui Zhang, Xiao - Gang Wang, and Chang Xu. Vitas: Vision transformer architecture search. In European Conference on Computer Vision (ECCV), volume 13681 of Lecture Notes in Computer Science, pp.\ 139--157....

2022
[72]

Mean-field analysis on two-layer neural networks from a kernel perspective

Shokichi Takakura and Taiji Suzuki. Mean-field analysis on two-layer neural networks from a kernel perspective. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024

2024
[73]

Mnasnet: Platform-aware neural architecture search for mobile

Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 2820--2828, 2019

2019
[74]

Training data-efficient image transformers & distillation through attention

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv \'e J \'e gou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp.\ 10347--10357. PMLR, 2021

2021
[75]

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp.\ 5998--6008, 2017

2017
[76]

Adaptive cross-layer attention for image restoration

Yancheng Wang, Ning Xu, Chong Chen, and Yingzhen Yang. Adaptive cross-layer attention for image restoration. arXiv preprint arXiv:2203.03619, 2022 a

2022 arXiv
[77]

NTK-SAP: improving neural network pruning by aligning training dynamics

Yite Wang, Dawei Li, and Ruoyu Sun. NTK-SAP: improving neural network pruning by aligning training dynamics. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023

2023
[78]

Vtc-lfc: Vision transformer compression with low-frequency components

Zhenyu Wang, Hao Luo, Pichao Wang, Feng Ding, Fan Wang, and Hao Li. Vtc-lfc: Vision transformer compression with low-frequency components. Advances in Neural Information Processing Systems, 35: 0 13974--13988, 2022 b

2022
[79]

Ntk-approximating MLP fusion for efficient language model fine-tuning

Tianxin Wei, Zeming Guo, Yifan Chen, and Jingrui He. Ntk-approximating MLP fusion for efficient language model fine-tuning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Lea...

2023
[80]

Auto-prox: Training-free vision transformer architecture search via automatic proxy discovery

Zimian Wei, Peijie Dong, Zheng Hui, Anggeng Li, Lujun Li, Menglong Lu, Hengyue Pan, and Dongsheng Li. Auto-prox: Training-free vision transformer architecture search via automatic proxy discovery. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp....

2024
[81]

Kernel and rich regimes in overparametrized models

Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference on Learning Theory, pp.\ 3635--3673. PMLR, 2020

2020
[82]

Tinyvit: Fast pretraining distillation for small vision transformers

Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. In Computer Vision--ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23--27, 2022, Proceedings, Part XXI, p...

2022
[83]

Disentangling trainability and generalization in deep neural networks

Lechao Xiao, Jeffrey Pennington, and Samuel Schoenholz. Disentangling trainability and generalization in deep neural networks. In International Conference on Machine Learning, pp.\ 10462--10472. PMLR, 2020

2020
[84]

Unified perceptual parsing for scene understanding

Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, Sep...

2018
[85]

Lpvit: Low-power semi-structured pruning for vision transformers

Kaixin Xu, Zhe Wang, Chunyun Chen, Xue Geng, Jie Lin, Xulei Yang, Min Wu, Xiaoli Li, and Weisi Lin. Lpvit: Low-power semi-structured pruning for vision transformers. In European Conference on Computer Vision, pp.\ 269--287. Springer, 2024

2024
[86]

From knowledge distillation to self-knowledge distillation: A unified approach with normalized loss and customized soft labels

Zhendong Yang, Ailing Zeng, Chun Yuan, and Yu Li. From knowledge distillation to self-knowledge distillation: A unified approach with normalized loss and customized soft labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 17185--17194, 2023

2023
[87]

Vitkd: Feature-based knowledge distillation for vision transformers

Zhendong Yang, Zhe Li, Ailing Zeng, Zexian Li, Chun Yuan, and Yu Li. Vitkd: Feature-based knowledge distillation for vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 1379--1388, 2024

2024
[88]

Width & depth pruning for vision transformers

Fang Yu, Kun Huang, Meng Wang, Yuan Cheng, Wei Chu, and Li Cui. Width & depth pruning for vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.\ 3143--3151, 2022 a

2022
[89]

Unified visual transformer compression

Shixing Yu, Tianlong Chen, Jiayi Shen, Huan Yuan, Jianchao Tan, Sen Yang, Ji Liu, and Zhangyang Wang. Unified visual transformer compression. ICLR, 2022 b

2022
[90]

Tokens-to-token vit: Training vision transformers from scratch on imagenet

Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF international conference on computer vision, 2021

2021
[91]

Cumulative spatial knowledge distillation for vision transformers

Borui Zhao, Renjie Song, and Jiajun Liang. Cumulative spatial knowledge distillation for vision transformers. In IEEE International Conference on Computer Vision (ICCV), pp.\ 6146--6155, 2023

2023
[92]

Savit: Structure-aware vision transformer pruning via collaborative optimization

Chuanyang Zheng, Kai Zhang, Zhi Yang, Wenming Tan, Jun Xiao, Ye Ren, Shiliang Pu, et al. Savit: Structure-aware vision transformer pruning via collaborative optimization. Advances in Neural Information Processing Systems, 35: 0 9010--9023, 2022

2022
[93]

Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H. S. Torr, and Li Zhang. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In IEEE Conference on Computer Visi...

2021
[94]

Semantic understanding of scenes through the ADE20K dataset

Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ADE20K dataset. Int. J. Comput. Vis., 127 0 (3): 0 302--321, 2019

2019
[95]

Deformable detr: Deformable transformers for end-to-end object detection

Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. ICLR, 2021

2021
[96]

@esa (Ref

\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
[97]

\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
[98]

https://coremltools.readme.io/docs

@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

2017 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [2]

Du, Wei Hu, Zhiyuan Li, and Ruosong Wang

Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning (ICML), volume 97 of Machine Learning Research, pp.\ 322--332. PMLR , 2019

2019

[3] [3]

Implicit gradient regularization

David Barrett and Benoit Dherin. Implicit gradient regularization. In International Conference on Learning Representations, 2021

2021

[4] [4]

Bartlett, Olivier Bousquet, and Shahar Mendelson

Peter L. Bartlett, Olivier Bousquet, and Shahar Mendelson. Local rademacher complexities. Ann. Statist., 33 0 (4): 0 1497--1537, 08 2005

2005

[5] [5]

To understand deep learning we need to understand kernel learning

Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning we need to understand kernel learning. In International Conference on Machine Learning, pp.\ 541--549. PMLR, 2018

2018

[6] [6]

Token merging: Your vit but faster

Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. ICLR, 2023

2023

[7] [7]

Learned thresholds token merging and pruning for vision transformers

Maxim Bonnaerens and Joni Dambre. Learned thresholds token merging and pruning for vision transformers. Transactions on Machine Learning Research, 2023. ISSN 2835-8856

2023

[8] [8]

Cvtstego-net: A convolutional vision transformer architecture for spatial image steganalysis

Mario Alejandro Bravo-Ortiz, Esteban Mercado-Ruiz, Juan Pablo Villa-Pulgarin, Carlos Angel Hormaza-Cardona, Sebastian Qui \ n ones-Arredondo, Harold Brayan Arteaga-Arteaga, Simon Orozco-Arias, Oscar Cardona-Morales, and Reinel Tabares-Soto. Cvtstego-net: A convolutional vision transformer architecture for spatial image steganalysis. Journal of Information...

2024