Pith. sign in

REVIEW 21 cited by

How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2106.10270 v2 pith:2DZ4ZMWM submitted 2021-06-18 cs.CV cs.AIcs.LG

Andreas Steiner , Alexander Kolesnikov , Xiaohua Zhai , Ross Wightman , Jakob Uszkoreit , Lucas Beyer This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords datatrainingvisionaugregmodelsaugmentationcomputedataset
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Vision Transformers (ViT) have been shown to attain highly competitive performance for a wide range of vision applications, such as image classification, object detection and semantic image segmentation. In comparison to convolutional neural networks, the Vision Transformer's weaker inductive bias is generally found to cause an increased reliance on model regularization or data augmentation ("AugReg" for short) when training on smaller training datasets. We conduct a systematic empirical study in order to better understand the interplay between the amount of training data, AugReg, model size and compute budget. As one result of this study we find that the combination of increased compute and AugReg can yield models with the same performance as models trained on an order of magnitude more training data: we train ViT models of various sizes on the public ImageNet-21k dataset which either match or outperform their counterparts trained on the larger, but not publicly available JFT-300M dataset.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 21 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Human-like Object Grouping in Self-supervised Vision Transformers

    cs.CV 2026-03 conditional novelty 6.5 of 10

    DINO self-supervised transformers best predict human same/different object RTs; object-centric patch affinity and Gram-matrix distillation explain and transfer the alignment.

  2. Representation Trajectories Matters: Complementary Evidence for OOD Detection and Image Classification

    cs.CV 2026-07 accept novelty 6.0 of 10

    Recording how an image's representation evolves block-by-block, relative to learned class routes, improves OOD detection in 131/152 comparisons and clean classification in 71/72 model–dataset cases.

  3. Advancing Multimodal Fusion on Heterogeneous Medical Data with Hybrid Geometry Attention

    cs.CV 2026-07 conditional novelty 6.0 of 10

    CURE, a cascaded fusion framework with hybrid hyperbolic/quantum attention, reports state-of-the-art accuracy and lower compute on 16 medical datasets.

  4. Elastic ViTs from Pretrained Models without Retraining

    cs.CV 2025-10 conditional novelty 6.0 of 10

    A single-shot, label-free, retraining-free structured pruning method generates elastic ViTs at any sparsity by reweighting gradient-based importance scores with block correlations learned by an evolutionary strategy.

  5. Hidden in plain sight: VLMs overlook their visual representations

    cs.CV 2025-06 conditional novelty 6.0 of 10

    VLMs perform far worse than their own visual encoders on vision-centric tasks because the language model fails to use accessible visual information and instead follows its language priors.

  6. Moment Alignment: Unifying Gradient and Hessian Matching for Domain Generalization

    cs.LG 2025-06 reject novelty 6.0 of 10

    A unified moment-alignment theory bounds target-domain error by cross-domain differences in loss derivatives, and the new CMA algorithm implements exact gradient and Hessian matching in closed form.

  7. TDVE-Assessor: Benchmarking and Evaluating the Quality of Text-Driven Video Editing with LMMs

    cs.CV 2025-05 conditional novelty 6.0 of 10

    TDVE-DB and TDVE-Assessor deliver a large MOS-annotated benchmark and a Qwen2.5-VL-based assessor for text-driven video editing quality.

  8. Scaling Pre-training to One Hundred Billion Data for Vision Language Models

    cs.CV 2025-02 conditional novelty 6.0 of 10

    Scaling VLM pretraining from 10B to 100B image-text pairs yields saturation on standard benchmarks but large gains on cultural diversity, low-resource language retrieval, and subgroup disparity.

  9. TEMSET-24K: Densely Annotated Dataset for Indexing Multipart Endoscopic Videos using Surgical Timeline Segmentation

    cs.CV 2025-02 conditional novelty 6.0 of 10

    A new densely labeled dataset of 24,000+ TEMS surgical video clips with phase, task, and action labels, plus a benchmark model for automatic timeline indexing.

  10. Screening Is Effective for Visual Recognition

    cs.CV 2026-07 conditional novelty 5.0 of 10

    A vision transformer with thresholded, non-normalized patch relevance (Screening) outperforms a same-scale ViT on ImageNet-1k and CIFAR-100.

  11. Underwater Monocular Metric Depth Estimation: Real-World Benchmarks and Synthetic Fine-Tuning with Vision Foundation Models

    cs.CV 2025-07 conditional novelty 5.0 of 10

    Fine-tuning Depth Anything V2 on physics-based synthetic underwater versions of Hypersim improves metric depth accuracy on real underwater benchmarks like FLSea and SQUID, though one AbsRel number worsens slightly.

  12. InceptionMamba: An Efficient Hybrid Network with Large Band Convolution and Bottleneck Mamba

    cs.CV 2025-06 conditional novelty 5.0 of 10

    InceptionMamba, a hybrid CNN-Mamba backbone with band convolutions and a bottleneck state-space mixer, reaches 84.7% ImageNet top-1 at 14.3 GFLOPs.

  13. Leaner Transformers: More Heads, Less Depth

    cs.LG 2025-05 conditional novelty 5.0 of 10

    The authors claim that more attention heads improve transformer conditioning enough to replace depth with width, yielding 30-50% parameter reductions at matched accuracy.

  14. DiTASK: Multi-Task Fine-Tuning with Diffeomorphic Transformations

    cs.CV 2025-02 conditional novelty 5.0 of 10

    DiTASK fine-tunes vision transformers for multiple tasks by applying learnable diffeomorphic maps to the singular values of frozen weights, and reports state-of-the-art multi-task performance with fewer parameters.

  15. CoMViT: An Efficient Vision Backbone for Supervised Classification in Medical Imaging

    cs.CV 2025-10 conditional novelty 4.0 of 10

    A 4.5M-parameter ViT combining convolutional tokenization, diagonal masking, temperature scaling, and sequence pooling matches or beats larger models on MedMNIST.

  16. DeepTraverse: A Depth-First Search Inspired Network for Algorithmic Visual Understanding

    cs.CV 2025-06 reject novelty 4.0 of 10

    DeepTraverse is a weight-tied residual network plus squeeze-and-excitation attention, framed as depth-first search, with claimed efficiency gains that rest on a questionable ImageNet subset comparison.

  17. EMRA-proxy: Enhancing Multi-Class Region Semantic Segmentation in Remote Sensing Images with Attention Proxy

    cs.CV 2025-05 conditional novelty 4.0 of 10

    Combining region proxy tokens with class-token attention improves remote sensing segmentation, reporting top mIoU on LoveDA, Potsdam, and Vaihingen.

  18. Color Fundus Photography Analysis: Co-evolution of Data, Preprocessing, and Modeling toward Multimodal AI

    cs.CV 2026-07 conditional novelty 3.0 of 10

    A review that organizes color fundus photography AI as the co-evolution of datasets, preprocessing, and models, concluding that performance ceilings are set by joint optimization of data, hygiene, and multimodal context.

  19. Towards Cellular-Scale Interpretability in Pathology Foundation Models for Biomarker Assessment

    cs.CV 2025-11 reject novelty 3.0 of 10

    JWTH achieves modest tissue-classification gains by adding attention pooling and stain augmentation to a DINOv3 backbone, but the biomarker claims in the abstract are unsupported by the experiments.

  20. ReStNet: A Reusable & Stitchable Network for Dynamic Adaptation on IoT Devices

    cs.CV 2025-06 reject novelty 3.0 of 10

    ReStNet builds a hybrid model from two pre-trained networks by stitching at the most CKA-similar layer and fine-tuning only the stitching layer, claiming flexible runtime trade-offs for IoT devices.

  21. Pose Matters: Evaluating Vision Transformers and CNNs for Human Action Recognition on Small COCO Subsets

    cs.CV 2025-06 reject novelty 2.0 of 10

    A tiny COCO benchmark reports a 90% binary ViT accuracy, but the headline comparison is confounded by task difficulty and by missing code and data.

Pith tools