REVIEW 4 major objections 3 minor 1 cited by
Do Vision Transformers See Like Humans? Evaluating their Perceptual Alignment
T0 review · 4 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Larger Vision Transformers show lower perceptual alignment with human judgments on TID2013, and the training diet—not dataset diversity—drives the decline.
desk verdict Plausible and useful, but the size effect may be a training-duration confound—needs the full methods. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the perceptual alignment score computed between a ViT's internal representations and human quality judgments on the TID2013 dataset. The study systematically varies model size, dataset size and diversity, number of training epochs, data augmentation, and regularization strength to isolate which training choices move that alignment score. TID2013 supplies the human ground truth, and the alignment metric quantifies how closely model responses track those human scores.
What would settle it
Compute the same alignment measure on a different human-judgment dataset (e.g., pairwise similarity choices rather than quality ratings) across the same grid of model sizes and training schedules. If larger models do not show lower alignment there, the claimed trade-off is specific to TID2013 or to the chosen metric rather than a general property of ViT perception.
Extended reading notes
Core claim
The paper claims that perceptual alignment between Vision Transformers and human judgments on TID2013 systematically decreases as model size increases. It further claims that the number of times a model sees the same training image is a key factor: more repeated exposures lower alignment. Stronger data augmentation and regularization also lower alignment, and these effects compound when models are trained over many epochs. In contrast, increasing the diversity of the training dataset has a minimal impact on alignment. The authors interpret these results as a trade-off between model complexity, training strategies, and alignment with human perception, with implications for applications that r
Load-bearing premise
The results rest on the assumption that TID2013 human opinion scores and the chosen alignment metric faithfully represent human perceptual alignment; if that metric emphasizes certain distortions or a particular feature layer, the finding that bigger models align less may not generalize to human-like visual understanding in general.
Editorial extensions
If this is right
- Scaling up ViTs for better image recognition does not automatically bring their representations closer to human perception, and may move them further away.
- Training schedules that reuse images many times—longer training, repeated epochs, strong augmentation—risk reducing perceptual alignment even if they improve accuracy.
- Dataset diversity alone is not a sufficient lever; simply adding more distinct images will not fix misalignment if model scale and training pressure push in the other direction.
- Applications that depend on human-like judgments, such as perceptual image quality metrics, may need to favor smaller models or deliberately designed training regimens to preserve alignment.
- Reporting only accuracy is incomplete; perceptual alignment should be tracked as a separate axis of model behavior.
Reading between the lines
- If the alignment metric is sensitive mainly to low-level distortion judgments, the observed scale-alignment trade-off may not carry over to tasks requiring high-level semantic similarity; this is testable by measuring alignment on more semantic human-judgment tasks.
- One could test whether the repeated-exposure effect generalizes across model families: an editorial extension would be a matched comparison of CNNs and ViTs under identical training diets to see whether the degradation is specific to attention-based architectures.
- The findings hint at a possible design rule—protect perceptual alignment by limiting epochs or using milder augmentation—which could be validated by early-stopping or curriculum strategies that keep alignment high without sacrificing accuracy.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This abstract-only submission claims to systematically analyze how model size, dataset size, data augmentation, and regularization affect Vision Transformer (ViT) perceptual alignment with human judgments, using the TID2013 dataset. The headline findings are that larger models show lower perceptual alignment, dataset diversity has minimal impact, repeated exposure to the same images reduces alignment, and stronger augmentation/regularization further reduce alignment. The authors present these as evidence of a trade-off between model complexity/training strategy and human-like visual understanding. Since only the abstract was available, no technical derivation, experimental protocol, or quantitative evidence could be inspected.
Significance. If the reported findings are supported by the full analysis, the paper would provide a valuable empirical contribution: it would quantify a systematic relationship between ViT training choices and perceptual alignment, with concrete implications for applications in which model behavior should track human perception. The use of an external human-judgment benchmark (TID2013) is a strength, as it reduces the risk that the alignment scores are circularly derived from the model's own training objective. However, the abstract alone does not allow assessment of effect sizes, statistical reliability, or whether the causal attributions are identified from the experimental design. The significance is therefore conditional on the full text providing the missing details.
major comments (4)
- [Abstract (core size claim)] The claim that 'larger models exhibit lower perceptual alignment' may be confounded with training duration, augmentation, and regularization. In common ViT practice, larger models are trained with more epochs, heavier augmentation, and stronger regularization. If the comparison is between off-the-shelf checkpoints or model families that differ jointly in scale and training schedule, the size effect is not identifiable. The manuscript should specify whether model size was varied while holding epochs, augmentation, and regularization constant, or whether statistical controls were applied. Without this, the central causal attribution is not supported.
- [Abstract (metric operationalization)] The abstract reports results on TID2013 but does not state which alignment metric, which feature layer(s), or which similarity function were used. Perceptual alignment can vary substantially across layers and metrics. If the metric or hyperparameters were tuned on TID2013 itself, the findings would be subject to circularity. The manuscript should define the alignment metric precisely and state whether any choices were made with knowledge of the TID2013 human scores.
- [Abstract (statistical evidence)] The abstract states directional findings ('lower alignment', 'minimal impact', 'further decrease') without effect sizes, confidence intervals, or significance tests. For a systematic study, the abstract should at least report summary statistics or the number of models/configurations compared. This is load-bearing because the claims are quantitative and comparative; without error bars or inferential tests, the reader cannot distinguish true systematic effects from noise across model checkpoints.
- [Abstract (dataset diversity conclusion)] The claim that 'increasing dataset diversity has a minimal impact' requires a definition of dataset diversity and a description of what was varied: number of image classes, total dataset size, domain coverage, or something else. If only one diversity axis was tested, the conclusion is overgeneralized. The manuscript should specify the diversity manipulation and the range over which it was varied.
minor comments (3)
- [Abstract] The abstract uses phrases such as 'consistently with previous works' but provides no citation; the full text should place the finding in context with the claimed prior evidence.
- [Abstract] The sentence 'exposing models to the same images more times reduces alignment' should clarify whether this refers to number of epochs, number of gradient steps, or repeated passes over a fixed training set; these interpretations have different implications.
- [Abstract] The manuscript title asks whether ViTs 'see like humans', but the abstract only reports correlations with a specific distortion-rating benchmark; the full text should temper the generalization to human-like visual understanding or provide a broader suite of human-alignment tasks.
Circularity Check
No significant circularity in the abstract; the study is an empirical benchmark evaluation against external human judgments.
full rationale
This review is based on the abstract only. The central claims—that larger ViTs show lower perceptual alignment, that repeated exposure to the same images reduces alignment, and that augmentation/regularization further reduce alignment—are empirical findings evaluated against TID2013 human opinion scores. The outcome measure is an external human-judgment benchmark, not a quantity derived from the model or from the authors' own prior definitions. The independent variables (model size, dataset size, augmentation, regularization) are training configurations rather than parameters fitted to the benchmark. No equation in the abstract shows a fitted parameter renamed as a prediction, no self-citation is invoked as load-bearing, and no claim reduces by construction to its inputs. The skeptic's concern about confounding (e.g., model size covarying with training duration) is an internal-validity or identifiability issue, not circularity. The reader's note that the metric or hyperparameters might have been tuned on TID2013 is speculation without evidence from the available text; per the hard rules, circularity cannot be claimed without quotable reduction. Therefore the appropriate finding is no significant circularity, score 0.
Assumptions & free parameters
assumptions (2)
- domain assumption TID2013 human opinion scores are a valid reference for human visual perception
- domain assumption The chosen perceptual alignment metric meaningfully captures human-model agreement
Cite this review
Pith. "Pith review of Do Vision Transformers See Like Humans? Evaluating their Perceptual Alignment." pith.science (2026). https://pith.science/paper/TUMXHEZK
@misc{pith2026250809850,
author = {Pith},
title = {Pith review of: Do Vision Transformers See Like Humans? Evaluating their Perceptual Alignment},
year = {2026},
howpublished = {\url{https://pith.science/paper/TUMXHEZK}},
note = {Machine review of arXiv:2508.09850}
}
read the original abstract
Vision Transformers (ViTs) achieve remarkable performance in image recognition tasks, yet their alignment with human perception remains largely unexplored. This study systematically analyzes how model size, dataset size, data augmentation and regularization impact ViT perceptual alignment with human judgments on the TID2013 dataset. Our findings confirm that larger models exhibit lower perceptual alignment, consistent with previous works. Increasing dataset diversity has a minimal impact, but exposing models to the same images more times reduces alignment. Stronger data augmentation and regularization further decrease alignment, especially in models exposed to repeated training cycles. These results highlight a trade-off between model complexity, training strategies, and alignment with human perception, raising important considerations for applications requiring human-like visual understanding.
Forward citations
Cited by 1 Pith paper
-
Accuracy Does Not Guarantee Human-Likeness: Cross-Domain Human-Centered Benchmark in Monocular Depth Estimation
Across 69 monocular depth estimators, human-likeness of error patterns peaks near human-level accuracy and declines for the most accurate models: accuracy does not guarantee human-like depth perception.
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.