REVIEW 2 major objections 4 minor 45 references
The Impact of the Single-Label Assumption in Image Recognition Benchmarking
T0 review · 2 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read The reported ImageNetV2 accuracy drop is substantially a single-label measurement artifact.
desk verdict The paper's qualitative claim is plausible and important, but the ASMA metric as defined cannot produce the reported numbers, so the headline result is currently unverifiable. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Two evaluation instruments carry the argument. Variable top-k selects the top k predicted classes for each image, where k equals the number of valid labels in its ground truth, so a model is credited for each correct secondary label. ASMA divides images into subgroups by label count, computes per-label accuracy within each subgroup, and averages those subgroup accuracies, preventing the rare high-label-count images from dominating. PatchML is a synthetic diagnostic dataset built by cropping objects from ImageNet images and pasting them in random non-overlapping grid cells on black canvases, removing co-occurrence and context cues; it isolates whether multi-label recognition is genuine object-level behavior or a byproduct of scene statistics.
What would settle it
Take the ImageNetV2 images the paper counts as multi-label, have independent annotators list all visible objects, and recompute ASMA and the V1-to-V2 gap; if the gap returns above the 0–6% range or the multi-label prevalence estimate changes materially, the reported artifact is an artifact of the annotation protocol.
Extended reading notes
Core claim
The central discovery is that the ImageNetV2 degradation is substantially a measurement artifact: because ImageNetV2 has a higher incidence of multi-label images than ImageNetV1, top-1 accuracy penalizes models for predicting valid secondary labels. The authors quantify this by constructing variable top-k predictions, where k equals the number of ground-truth labels per image, and by introducing Aggregate Subgroup Model Accuracy (ASMA), which averages per-label accuracy across subgroups of images with the same label count. Across 315 ImageNet-pretrained models, the V1-to-V2 gap falls from 6–14% under top-1 to 0–11% under ReaL accuracy and to 0–6% under ASMA. PatchML, a synthetic dataset of randomly combined cropped objects on blank canvases, further shows that models trained with single-label supervision still rank multiple correct labels near the top, indicating latent multi-label capability beyond what standard metrics report.
Load-bearing premise
The load-bearing premise is that the list of valid objects used for ImageNetV2 is complete and correct for every image; if those annotations miss valid objects or include spurious ones, the measured gap reduction is an artifact of the annotation protocol.
Editorial extensions
If this is right
- The common 11–14% ImageNetV2 drop should not be read as pure generalization failure; multi-label-aware metrics cut it to 0–6%.
- Models with high top-1 accuracy are not necessarily the best at ranking multiple valid labels, so top-1 leaderboards can misrank models for multi-object scenes.
- Training signals that go beyond single-label targets (dense token labels, masked image modeling, distillation from web-scale teachers) predict stronger multi-label capability; scale alone is not enough.
- Evaluation protocols for ImageNet should adopt multi-label-aware metrics alongside top-1 to avoid underestimating DNN capabilities.
Reading between the lines
- A testable extension: apply the same variable top-k and ASMA protocol to other single-label benchmarks that likely contain multi-label images, such as Places or iNaturalist, to see whether their apparent domain gaps also narrow.
- Beyond the paper's evaluation-only scope, the PatchML construction could be turned into a training curriculum; if models fine-tuned on multi-object composites improve ASMA on natural images, that would confirm the latent-capability story causally.
- If the gap reduction is real, earlier conclusions that ImageNet classifiers fail under distribution shift may need to be re-stated as conclusions about single-label evaluation rather than about the models themselves.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that the commonly reported 11–14% top-1 accuracy drop on ImageNetV2 is largely an artifact of the single-label evaluation protocol. It proposes variable top-k evaluation with k equal to the number of valid labels per image, introduces Aggregate Subgroup Model Accuracy (ASMA) as a subgroup-averaged multi-label accuracy, and presents PatchML, a synthetic dataset of composed object patches. Across 315 ImageNet-pretrained models, the authors report that the V1–V2 gap narrows to 0–6% under ASMA and that many models rank multiple correct labels highly. The paper concludes that single-label metrics understate DNN multi-label capability.
Significance. If the quantitative claims could be verified, the work would be valuable: it addresses a widely discussed anomaly in benchmarking, uses a broad model pool (315 TIMM models), proposes a synthetic diagnostic that removes co-occurrence cues, and ships a code repository (claimed). The reframing of the ImageNetV2 gap as partly evaluation-induced is consistent with prior work on ReaL labels and would have implications for benchmark design. However, the central quantitative claim currently rests on an ASMA definition that appears inconsistent with the reported numbers, and on self-authored ImageNetV2 annotations that lack independent validation; both need to be resolved before the conclusions can be accepted.
major comments (2)
- [Section 3.3, Eq. (Ag) and Figure 4c/Figure 5b] The ASMA definition is internally inconsistent with the reported numbers. The label-wise accuracy formula (1/C) * sum_c I(y_gt_g,i,c = y_hat_g,i,c) with C=1000 and y_hat in {0,1}^C is a Hamming-style accuracy, but no rule is given for constructing the binary prediction vector y_hat from the variable top-k softmax outputs of Section 3.1. If y_hat is the indicator of the top-k predicted classes and k equals the number of true labels g, then the per-image score is 1 - (false positives)/1000, which is at least ~0.99 for any image with all true labels in the top-k and typically not much lower otherwise. This cannot produce the subgroup accuracies near 0.2–0.7 in Figure 5b or the 0–6% ASMA gap in Figure 4c. The authors must correct the formula (e.g., to example-based accuracy |Y ∩ Yhat| / |Y ∪ Yhat|, which they claim to extend) or specify the binarization and recompute all ASMA-based figures. Because the abstract and Section 1.1 base the central 'gap shrinks to 0–6%' claim on ASMA, this inconsistency is load-bearing.
- [Section 4.2.1, Table 1, Figure 4] The quantitative gap reduction on ImageNetV2 depends entirely on the multi-label annotations from Anzaku et al. [14], which are self-authored and not machine-checked, reproduced, or independently validated in this paper. The paper provides no inter-annotator agreement, error analysis, or third-party comparison for these labels. Section 5 acknowledges that the annotations 'may still miss plausible object categories,' but the main conclusion that the V2 gap is an evaluation artifact assumes the labels are complete and correct. I ask for a sensitivity analysis or an independent validation subset (e.g., labels from a different annotation protocol, or agreement rates on a sample) before the gap-reduction claim is accepted.
minor comments (4)
- [Figure 4 caption] The caption says 'all plotted against label count' but the x-axis is the index of pre-trained models; please correct the caption to match the actual axes.
- [Section 3.3] The ASMA sum runs over g = 0 to G-1, but no images have zero labels in Table 1; please specify how empty subgroups are handled or start the sum at g = 1.
- [Section 3.2, Algorithm 1] It is unclear whether the patch pool S is reset for each k value and for each of the five seeds; please clarify, as this affects the independence of the PatchML variants summarized in Table 2.
- [Abstract and Section 3.3] The metric is called 'Aggregate Subgroup Model Accuracy' in the abstract and 'Average Subgroup Multi-Label Accuracy' in Section 3.3; please use a single consistent name throughout.
Circularity Check
The central ImageNetV2 gap-reduction result is measured against the authors' own unvalidated multi-label annotations (Anzaku et al. [14]); the ASMA formula in Section 3.3 is internally inconsistent with Figure 4c, a verifiability problem that is not itself a circular reduction.
-
self citation load bearing
[Section 4.1 (Test Dataset Description) and Section 4.2.1 (Prevalence of Multi-label Images), Table 1 and Figure 3]
"For ImageNetV2, we used the refined labels generated by Anzaku et al. [14], which account for multi-label images. ... To do this, we utilize the re-assessed labels from ReaL [5] for ImageNetV1 and the refined labels introduced by Anzaku et al. [14] for ImageNetV2."
The paper's headline finding—that ImageNetV2 is approximately 48% multi-label versus 16% for ImageNetV1, and that the top-1 accuracy drop shrinks to 0–6% under ASMA—is computed against a ground-truth label set created by the same authors (Anzaku et al. [14]). No independent human agreement, machine check, or external validation of these labels is provided, and Section 5 concedes they 'may still miss plausible object categories.' The central quantitative claim therefore inherits its value from the authors' own annotation protocol rather than from an independently verified measurement.
full rationale
The paper is not globally circular: the ReaL labels for ImageNetV1 are external, the PatchML diagnostic is a freshly constructed controlled dataset with labels derived from object-detection annotations, and the ranking analysis of 315 TIMM models is self-contained. Those parts provide independent content. The main circularity concern is the load-bearing reliance on the authors' own ImageNetV2 annotations from Anzaku et al. [14] for both the multi-label prevalence claim and the headline 0–6% ASMA gap. Because these labels are neither machine-checked nor independently validated in the paper, the central V2-specific result is not externally anchored. Separately, the ASMA definition in Section 3.3 is inconsistent with Figure 4c: the stated Hamming-style label-wise accuracy over C=1000 classes, combined with the variable top-k prediction mechanism of Section 3.1, lacks a rule converting top-k indices to the binary vector y_hat, and under the natural conversion every model would score near (C-k)/C, making the plotted 0–0.9-scale values and 0–6% gap unreproducible. This is a definitional/verifiability flaw, not a circular reduction to the paper's own inputs, so it is flagged here rather than counted as a circular step. On balance the self-cited annotation input is substantial and load-bearing but not the whole derivation, yielding a score of 4.
Assumptions & free parameters
free parameters (2)
- PatchML object counts K =
{2, 3, 4, 6, 9}
- PatchML patch sizes P =
{256, 256, 256, 170, 128}
assumptions (4)
- domain assumption The ReaL multi-label annotations for ImageNetV1 are complete and correct for the 46,837 images they cover.
- domain assumption The ImageNetV2 multi-label annotations from Anzaku et al. [14] are complete and correct for the 9,858 annotated images.
- domain assumption A model that recognizes all valid objects in an image will place those objects among its top-k predictions, where k is the number of valid labels.
- domain assumption The PatchML labels, obtained by taking the union of source-patch labels, are the complete set of recognizable objects in each composite image.
Cite this review
Pith. "Pith review of The Impact of the Single-Label Assumption in Image Recognition Benchmarking." pith.science (2026). https://pith.science/paper/H5DXSS2V
@misc{pith2026241218409,
author = {Pith},
title = {Pith review of: The Impact of the Single-Label Assumption in Image Recognition Benchmarking},
year = {2026},
howpublished = {\url{https://pith.science/paper/H5DXSS2V}},
note = {Machine review of arXiv:2412.18409}
}
abstract
Deep neural networks (DNNs) are typically evaluated under the assumption that each image has a single correct label. However, many images in benchmarks like ImageNet contain multiple valid labels, creating a mismatch between evaluation protocols and the actual complexity of visual data. This mismatch can penalize DNNs for predicting correct but unannotated labels, which may partly explain reported accuracy drops, such as the widely cited 11 to 14 percent top-1 accuracy decline on ImageNetV2, a replication test set for ImageNet. This raises the question: do such drops reflect genuine generalization failures or artifacts of restrictive evaluation metrics? We rigorously assess the impact of multi-label characteristics on reported accuracy gaps. To evaluate the multi-label prediction capability (MLPC) of single-label-trained models, we introduce a variable top-$k$ evaluation, where $k$ matches the number of valid labels per image. Applied to 315 ImageNet-trained models, our analyses demonstrate that conventional top-1 accuracy disproportionately penalizes valid but secondary predictions. We also propose Aggregate Subgroup Model Accuracy (ASMA) to better capture multi-label performance across model subgroups. Our results reveal wide variability in MLPC, with some models consistently ranking multiple correct labels higher. Under this evaluation, the perceived gap between ImageNet and ImageNetV2 narrows substantially. To further isolate multi-label recognition performance from contextual cues, we introduce PatchML, a synthetic dataset containing systematically combined object patches. PatchML demonstrates that many models trained with single-label supervision nonetheless recognize multiple objects. Altogether, these findings highlight limitations in single-label evaluation and reveal that modern DNNs have stronger multi-label capabilities than standard metrics suggest.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[14]
E. T. Anzaku, H. Hong, J.-W. Park, W. Yang, K. Kim, J. Won, D. V . K. Herath, A. Van Messem, W. De Neve, Leveraging Human-Machine Interactions for Com- puter Vision Dataset Quality Enhancement, in: B. J. Choi, D. Singh, U. S. Tiwary, W.-Y . Chung (Eds.), Intelligent Human Computer Interaction, 2024, pp. 295–309. doi:10.1007/978-3-031-53827-8_27
-
[1]
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, Imagenet: A Large- scale Hierarchical Image Database, in: IEEE conference on computer vision and pattern recognition, 2009, pp. 248–255
work page 2009
-
[2]
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, L. Fei-Fei, ImageNet Large Scale Visual Recognition Challenge, International Journal of Computer Vision 115 (2015) 211–252. doi:10.1007/s11263-015-0816-y
- [3]
- [4]
- [5]
-
[6]
V . Shankar, R. Roelofs, H. Mania, A. Fang, B. Recht, L. Schmidt, Evaluating Ma- chine Accuracy on ImageNet, in: International Conference on Machine Learning, volume 37, 2020, pp. 8634–8644
work page 2020
-
[7]
D. Tsipras, S. Santurkar, L. Engstrom, A. Ilyas, A. Madry, From ImageNet to Image Classification: Contextualizing Progress on Benchmarks, in: International Conference on Machine Learning, 2020, pp. 9625–9635
work page 2020
Show all 45 references
-
[8]
Vasudevan, B
V . Vasudevan, B. Caine, R. Gontijo-Lopes, S. Fridovich-Keil, R. Roelofs, When does dough become a bagel? Analyzing the remaining mistakes on ImageNet, in: Conference on Neural Information Processing Systems, 2022
2022
- [9]
-
[10]
C. G. Northcutt, A. Athalye, J. Mueller, Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks, in: Thirty-fifth Conference on Neu- ral Information Processing Systems Datasets and Benchmarks Track, 2021. URL: https://openreview.net/forum?id=XccDXrDNLek
2021
-
[11]
A. S. Luccioni, D. Rolnick, Bugs in the data: how ImageNet misrepresents bio- diversity, in: Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium ...
2023 doi
-
[12]
B. Y . Idrissi, D. Bouchacourt, R. Balestriero, I. Evtimov, C. Hazirbas, N. Ballas, P. Vincent, M. Drozdzal, D. Lopez-Paz, M. Ibrahim, ImageNet-X: Understanding Model Mistakes with Factor of Variation Annotations, 2022
2022
-
[13]
Peychev, M
M. Peychev, M. N. M ¨uller, M. Fischer, M. Vechev, Automated Classification of Model Errors on ImageNet, Thirty-seventh Conference on Neural Information Processing Systems (2023)
2023
-
[15]
Stock, M
P. Stock, M. Cisse, ConvNets and ImageNet Beyond Accuracy: Understanding Mistakes and Uncovering Biases, in: The European Conference on Computer Vision, 2018, pp. 504–519. doi:10.1007/978-3-030-01231-1_31
2018 doi
-
[16]
Engstrom, A
L. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, J. Steinhardt, A. Madry, Identi- fying Statistical Bias in Dataset Replication, in: Proceedings of the 37th Interna- tional Conference on Machine Learning, volume 119, 2020, pp. 2922–2932
2020
-
[17]
E. T. Anzaku, H. Wang, A. Babalola, A. Van Messem, W. De Neve, Re- assessing accuracy degradation: a framework for understanding DNN be- havior on similar-but-non-identical test datasets, Machine Learning 114 (2025). URL: https://doi.org/10.1007/s10994-024-06693-x . doi: 10. 1...
2025 doi
-
[18]
M. R. Taesiri, G. Nguyen, S. Habchi, C.-P. Bezemer, A. Nguyen, ImageNet-Hard: The Hardest Images Remaining from a Study of the Power of Zoom and Spatial Biases in Image Classification, in: Conference on Neural Information Processing Systems, 2023
2023
-
[19]
E. Cole, O. M. Aodha, T. Lorieul, P. Perona, D. Morris, N. Jojic, Multi-Label Learning from Single Positive Labels, in: IEEE /CVF Conference on Computer 30 Vision and Pattern Recognition, 2021, pp. 933–942. doi: 10.1109/CVPR46437. 2021.00099
2021
-
[20]
Verelst, P
T. Verelst, P. K. Rubenstein, M. Eichner, T. Tuytelaars, M. Berman, Spatial Consistency Loss for Training Multi-Label Classifiers from Single-Label Anno- tations, in: IEEE /CVF Winter Conference on Applications of Computer Vision, 2023, pp. 3868–3878. doi:10.1109/WACV56688.2023.00387
2023
-
[21]
S. Yun, S. J. Oh, B. Heo, D. Han, J. Choe, S. Chun, Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels, in: The IEEE /CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 2340–2350. doi:10.1109/CVPR46437.2021.00237
2021
-
[22]
S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, Y . Yoo, CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features, in: International Conference on Computer Vision (ICCV), 2019
2019
-
[23]
Takahashi, T
R. Takahashi, T. Matsubara, K. Uehara, RICAP: Random Image Cropping and Patching Data Augmentation for Deep CNNs, in: Proceedings of The 10th Asian Conference on Machine Learning, volume 95, 2018, pp. 786–798
2018
-
[24]
L. Deng, Y . Liu, Y . Shi, W. Zhang, C. Yang, H. Liu, Deep neural networks for inferring binding sites of RNA-binding proteins by using distributed repre- sentations of RNA primary sequence and secondary structure, BMC Genomics 21 (2020) 866. URL: https://doi.org/10.1186/s1286...
2020 doi
-
[25]
Zhang, Z.-H
M.-L. Zhang, Z.-H. Zhou, A Review on Multi-Label Learning Algorithms, IEEE Transactions on Knowledge and Data Engineering 26 (2014) 1819–1837. doi:10. 1109/TKDE.2013.39
2014
-
[26]
Wightman, PyTorch Image Models, 2019
R. Wightman, PyTorch Image Models, 2019. URL: https://github.com/ rwightman/pytorch-image-models. doi:10.5281/zenodo.4414861, publi- cation Title: GitHub repository. 31
2019 doi
-
[27]
K. He, X. Zhang, S. Ren, J. Sun, Deep Residual Learning for Image Recognition, in: IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778. doi:10.1109/CVPR.2016.90
2016 doi
- [28]
-
[29]
M. Tan, Q. V . Le, EfficientNet: Rethinking Model Scaling for Convolutional Neu- ral Networks, in: Proceedings of the 36th International Conference on Machine Learning, volume 97, 2019, pp. 6105–6114
2019
-
[30]
S. Woo, S. Debnath, R. Hu, X. Chen, Z. Liu, I. S. Kweon, S. Xie, Con- vNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 16133–16142. doi:10.1109/CVPR52729.2023.01548
2023
-
[31]
Dosovitskiy, L
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, N. Houlsby, An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, in: Ninth International Conference on Learnin...
2021
-
[32]
Touvron, M
H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, H. J ´egou, Training Data-efficient Image Transformers & Distillation Through Attention, in: Pro- ceedings of the 38th International Conference on Machine Learning, volume 139, 2021, pp. 10347–10357
2021
-
[33]
L. Dong, S. Piao, F. Wei, BEiT: BERT Pre-Training of Image Transformers, in: The Tenth International Conference on Learning Representations, 2022
2022
-
[35]
Touvron, M
H. Touvron, M. Cord, A. Sablayrolles, G. Synnaeve, H. Jegou, Going deeper with Image Transformers, in: IEEE /CVF International Conference on Computer Vision (ICCV), IEEE, 2021, pp. 32–42. doi:10.1109/ICCV48922.2021.00010
2021
-
[36]
L. Yuan, Q. Hou, Z. Jiang, J. Feng, S. Yan, VOLO: Vision Outlooker for Visual Recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (2023) 6575–6586. URL: https://ieeexplore.ieee.org/ document/9888055. doi:10.1109/TPAMI.2022.3206108
2023
- [37]
-
[38]
Mahajan, K
D. Mahajan, K. He, M. Paluri, Y . Li, A. Bharambe, L. Van Der Maaten, Explor- ing the Limits of Weakly Supervised Pretraining, in: European Conference on Computer Vision, volume 11206, 2018, pp. 185–201
2018
-
[39]
Singh, L
M. Singh, L. Gustafson, A. Adcock, V . De Freitas Reis, B. Gedik, R. P. Kosaraju, D. Mahajan, R. Girshick, P. Dollar, L. Van Der Maaten, Revisiting Weakly Supervised Pre-Training of Visual Perception Models, in: IEEE /CVF Conference on Computer Vision and Pattern Recognition, ...
2022
-
[40]
Jiang, Q
Z.-H. Jiang, Q. Hou, L. Yuan, D. Zhou, Y . Shi, X. Jin, A. Wang, J. Feng, All Tokens Matter: Token Labeling for Training Better Vision Transformers, in: Ad- vances in Neural Information Processing Systems, volume 34, 2021, pp. 18590– 18602
2021
-
[41]
Z. Liu, H. Mao, C.-Y . Wu, C. Feichtenhofer, T. Darrell, S. Xie, A ConvNet for the 2020s, in: IEEE Conference on Computer Vision and Pattern Recognition, 2022, pp. 11976–11986
2022
- [42]
-
[43]
Radford, J
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, I. Sutskever, Learning Transferable Visual Models From Natural Language Supervision, in: Proceedings of the 38th International Conference on Machine Learni...
2021
-
[44]
Radosavovic, R
I. Radosavovic, R. P. Kosaraju, R. Girshick, K. He, P. Dollar, Designing Net- work Design Spaces, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 10425–10433. doi:10.1109/CVPR42600.2020. 01044
2020
- [45]
-
[46]
Jetley, N
S. Jetley, N. Murray, E. Vig, End-to-End Saliency Mapping via Probability Distri- bution Prediction, in: IEEE Conference on Computer Vision and Pattern Recog- nition, 2016, pp. 5753–5761. doi:10.1109/CVPR.2016.620. 34
2016 doi
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.