REVIEW 3 major objections 5 minor 1 cited by
I Am Big, You Are Little; I Am Right, You Are Wrong
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper argues that image classifiers can be compared by the smallest set of pixels that preserves their top classification, and that this minimal set differs systematically by architecture and grows when the model is wrong.
desk verdict The cross-architecture MPS size ordering is a real new empirical result, but the position claim is statistically unsupported and the central measure needs sensitivity analysis before being taken as ground truth. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the minimal sufficient pixel set (MPS): an approximately smallest subset of an image's pixels that, with all other pixels set to a baseline, still makes the model output the original top classification. The machinery that produces MPSs is ReX, a black-box causal-explanation algorithm that partitions the image, tests masked 'mutant' images against the model, assigns each pixel a causal responsibility score, ranks pixels by that score, and adds pixels until the classification is restored. Because exact minimality is DP-complete and the ranking step is NP-complete, ReX's output is approximately minimal and guaranteed sufficient, not minimal; the paper uses the highest-responsibility MPS per image and compares sets across models by size ratio, Sørensen-Dice coefficient, and Hausdorff distance.
What would settle it
Run ReX with a much larger number of random starting partitions, or an exact exhaustive search on small cropped images, and compare the resulting MPS sizes across architectures; if the ordering among ConvNext, EVA, Inception, ResNet, and ViT changes materially with search effort or seed, the claimed architecture-level concentration differences are an artifact of the approximation.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is a set of stable statistical differences in minimal sufficient pixel sets across models. For the same images, EVA and ConvNext architectures consistently yield small MPSs (5.4-8.1% of image area across the six models), Inception models yield large ones (23.0-25.0%), and ResNet and ViT lie in between; a Kruskal-Wallis test across architectures gives p<0.01, and a Friedman test shows intra-architecture differences for all families except Inception. Position also differs: average Sørensen-Dice overlaps between best-performing models are low, Hausdorff distances are high, and in some images the MPSs are entirely disjoint. On the validation set, a mixed linear model estimates that incorrect classifications are associated with a 2.6% increase in MPS area (standard error 0.4%, p<0.01). The authors interpret small MPSs as evidence that very large pretrained models are 'myopic' and may be overfit, and suggest MPS size could be used as a supplementary check on a model's output.
Load-bearing premise
The comparison collapses if ReX's approximate pixel sets do not faithfully track each model's true decision-relevant pixels, since exact minimality is computationally intractable and ReX uses only 20 iterations from random partitions; size and location differences could then be artifacts of the search rather than model behavior.
Editorial extensions
If this is right
- Model families can be told apart by where and how much they look: with identical hyperparameters and seed, ConvNext and EVA rely on roughly 5-8% of the image while Inception relies on roughly 23-25%, so accuracy tables alone under-describe a classifier.
- Small MPSs indicate that large pretrained models will happily classify heavily masked, out-of-distribution inputs, which the authors connect to overfitting and 'myopic' decision-making.
- The statistically significant 2.6% increase in MPS area for wrong answers supports using MPS size as a post-classification diagnostic.
- MPS overlap and distance measures show that architectures often base decisions on different image regions, so the visual evidence used by different models is genuinely different even when their classifications agree.
Reading between the lines
- An implicit extension is to treat MPS size as a rough robustness probe: a model that accepts a very small set of zero-masked pixels is accepting strongly out-of-distribution inputs, so the architecture ordering found here may predict relative tolerance to other masked or corrupted inputs.
- A natural testable extension is to vary the masking baseline: if the 5-8% versus 23-25% ordering persists under blur, mean-color, and noise baselines, the concentration differences are about the models themselves; if it is baseline-specific, the metric partly measures sensitivity to a particular perturbation family.
- Because the authors compare only the single highest-responsibility MPS, a further extension would measure the full multiplicity of explanations per image; architectures that offer several disjoint alternative pixel sets may be less brittle than their smallest MPS alone suggests.
- The 2.6% area increase for errors points to a confidence-free misclassification detector: in deployment, compute the MPS area and flag decisions whose area is far above the model's own correct-decision distribution, a test that does not require ground-truth labels.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes using minimal sufficient pixel sets (MPSs), computed by the black-box causal XAI tool ReX, as a measure of 'concentration' for image classifiers. It compares 15 ImageNet-fine-tuned models from five architectures (ConvNext, EVA, Inception, ResNet, ViT) on 1000 ImageNet validation/test images (plus 50 Caltech-256 images), measuring MPS size relative to image area, location via Dice and Hausdorff distances, and the association between misclassification and MPS size. The paper reports statistically significant architecture differences in MPS size (Kruskal-Wallis, Friedman), descriptive location differences, and a 2.6% increase in MPS area for incorrect classifications from a mixed linear model.
Significance. If the MPS estimates are faithful, the paper provides a novel and useful comparative methodology: moving beyond accuracy to measure how much input evidence each model family uses. The empirical pattern—ConvNext/EVA using 5–8% of image area versus Inception's 23–25%—is striking and could inform model selection and OOD robustness discussions. The paper's strengths include the use of non-parametric matched tests, a same-seed replication across models, a Caltech-256 sanity check, and explicit acknowledgment of ReX's approximation. However, the study's central claims depend on ReX's approximate search being unbiased across architectures and on the location claim being backed by inferential statistics, both of which need further support.
major comments (3)
- [§4.3, RQ2 (Tables 2–3)] The paper's abstract and introduction claim that different architectures have 'statistically different concentration, in both size and position,' but the location claim is supported only by average Dice and Hausdorff values with no significance test, confidence interval, or adjustment for multiple comparisons. Please add an inferential analysis (e.g., permutation tests on the per-image DC/Hausdorff distributions, or a mixed model with image and model as random effects) and report effect sizes with confidence intervals.
- [§3, Algorithmic Overview] ReX is approximate—20 random 4-part partitions, a responsibility ranking that is NP-complete, and the paper's own statement that the MPS is 'approximately minimal, but not necessarily the minimum.' Because the default baseline (0 after processing) makes virtually all mutants out-of-distribution, the large architecture differences in Table 1 could reflect different architectures' sensitivity to OOD inputs or different approximation-error profiles rather than true differences in minimal sufficient pixel sets. Please provide robustness checks (varying iteration count, baseline value, and random seed; or cross-validating against another sufficient-set method) to show the architecture ranking is stable.
- [§4.2–4.3, RQ3] The mixed linear model used to claim a 2.6% increase in MPS size for incorrect classifications is not described in enough detail to assess validity or reproduce. Please state the fixed and random effects, how model accuracy and architecture were controlled, whether the outcome was transformed, and provide a full coefficient table. Also check the Caltech-256 standard error, reported as '0.01%,' which appears implausible relative to the 3.3% estimate.
minor comments (5)
- [Table 1] Report standard deviations or confidence intervals for the mean MPS area ratios, and state the number of images per cell; the current table gives no sense of spread, and the caption's '3.6× larger' claim is not directly implied by the averaged entries (e.g., Inception-ResNet V2 0.254 vs ConvNext-V2 Large 0.089 is about 2.9×).
- [Table 3] The header calls the measure 'Hausdorff coefficient'; Hausdorff distance is a metric, not a coefficient, so please standardize the terminology.
- [§4.3, RQ1] The Friedman test reporting (ConvNext p<0.01, Inception p>0.01, etc.) is unclear; please report test statistics and whether p-values are adjusted (e.g., Bonferroni) for the multiple intra-architecture comparisons.
- [§4.3 and Table 4] The Caltech-256 validation uses only 50 images; the sentence 'evidence of inter-architecture differences are not found for the EVA, Inception or ResNet models' is ambiguous (intra-architecture?), and the small sample should be stated as a limitation in the main text.
- [Figure 3] Add an explicit y-axis label and units; the violin plot is central to the size comparison, and the caption's color coding should be repeated in the figure itself.
Circularity Check
No significant circularity: the architecture and correctness comparisons are empirical measurements of ReX outputs, not conclusions forced by definition or by fitted parameters.
full rationale
The paper's core claims (RQ1-RQ3) are statistical comparisons of ReX-produced minimal sufficient pixel sets across 15 models. The quantity being compared — MPS size as a ratio of image area, and MPS location via Dice/Hausdorff — is directly measured from model outputs; no equation defines the conclusion in terms of its own premises. The only self-citational element is the use of ReX [7] and related causal-explanation work [2,8], but ReX is an openly available, algorithmically described black-box tool (Section 3), and the paper explicitly treats its outputs as approximate ('not necessarily the minimum'). The 2.6% misclassification effect is a fitted mixed-model coefficient, but it is presented as an empirical association, not as a prediction derived from the model's own assumptions. No global constant, normalization, or uniqueness theorem forces the cross-architecture size ordering; the observed ordering is an experimental result. Potential concerns about ReX's approximation error being biased across architectures are validity threats, not circularity: they do not reduce the measured comparisons to the paper's inputs by construction.
Assumptions & free parameters
free parameters (4)
- ReX iteration count =
20 (default)
- ReX initial superpixel count =
4 parts per image
- Baseline masking value =
0 (applied after processing)
- ReX random seed =
same fixed seed for all runs
assumptions (5)
- domain assumption ReX's approximately minimal sufficient pixel set is a valid proxy for where a model concentrates its classification.
- domain assumption ReX's approximation error is not systematically different across architectures.
- domain assumption ImageNet-1k ground truth labels are reliable enough to define correctness.
- domain assumption Models are comparable despite different input sizes, pretraining datasets, and parameter counts.
- domain assumption ONNX conversion preserves the decision behavior of the original models.
Cite this review
Pith. "Pith review of I Am Big, You Are Little; I Am Right, You Are Wrong." pith.science (2026). https://pith.science/paper/LBVMABAF
@misc{pith2026250723509,
author = {Pith},
title = {Pith review of: I Am Big, You Are Little; I Am Right, You Are Wrong},
year = {2026},
howpublished = {\url{https://pith.science/paper/LBVMABAF}},
note = {Machine review of arXiv:2507.23509}
}
read the original abstract
Machine learning for image classification is an active and rapidly developing field. With the proliferation of classifiers of different sizes and different architectures, the problem of choosing the right model becomes more and more important. While we can assess a model's classification accuracy statistically, our understanding of the way these models work is unfortunately limited. In order to gain insight into the decision-making process of different vision models, we propose using minimal sufficient pixels sets to gauge a model's `concentration': the pixels that capture the essence of an image through the lens of the model. By comparing position, overlap, and size of sets of pixels, we identify that different architectures have statistically different concentration, in both size and position. In particular, ConvNext and EVA models differ markedly from the others. We also identify that images which are misclassified are associated with larger pixels sets than correct classifications.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 1 Pith paper
-
Activation-Deactivation: A General Framework for Robust Post-hoc Explainable AI
ConvAD replaces input occlusion in post-hoc explanation with neuron deactivation in a CNN forward pass, yielding more robust causal explanations with no retraining.
Reference graph
Works this paper leans on
-
[1]
Ayyar, Jenny Benois-Pineau, and Akka Zemmari
Meghna P. Ayyar, Jenny Benois-Pineau, and Akka Zemmari. Review of white box methods for explana- tions of convolutional neural networks in image clas- sification tasks. Journal of Electronic Imaging, 30(5): 050901, 2021. 2
work page 2021
-
[2]
MRxaI: Black-Box Explainability for Image Classifiers in a Medical Setting
Nathan Blake, Hana Chockler, David A Kelly, Santi- ago Calderon Pena, and Akchunya Chanchal. Mrxai: Black-box explainability for image classifiers in a medical setting. arXiv preprint arXiv:2311.14471 ,
-
[3]
C. E. Bonferroni. Teoria statistica delle classi e calcolo delle probabilita. Pubblicazioni del R Istituto Superi- ore di Scienze Economiche e Commerciali di Firenze, pages 13–60, 1936. 4
work page 1936
-
[4]
Grad- cam++: Generalized gradient-based visual explana- tions for deep convolutional networks
Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad- cam++: Generalized gradient-based visual explana- tions for deep convolutional networks. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 839–847, 2018. 2
work page 2018
-
[5]
Hana Chockler and Joseph Y . Halpern. Responsibil- ity and blame: A structural-model approach. J. Artif. Intell. Res., 22:93–115, 2004. 3
work page 2004
-
[6]
Hana Chockler and Joseph Y . Halpern. Explaining Image Classifiers. In Proceedings of the 21st Inter- national Conference on Principles of Knowledge Rep- resentation and Reasoning, pages 264–272, 2024. 3
work page 2024
-
[7]
Kelly, Daniel Kroening, and Youcheng Sun
Hana Chockler, David A. Kelly, Daniel Kroening, and Youcheng Sun. Causal explanations for image classi- fiers, 2024. 1, 2, 3
work page 2024
-
[8]
Multiple different explanations for image clas- sifiers
Hana Chockler, David A Kelly, and Daniel Kroen- ing. Multiple different explanations for image clas- sifiers. In ECAI European Conference on Artificial Intelligence, 2025. 1, 2, 3
work page 2025
Show all 42 references
-
[9]
Lee R. Dice. Measures of the amount of ecologic as- sociation between species. Ecology, 26:297—-302,
-
[10]
Kruskal-Wallis Test, pages 288–290
Yadolah Dodge. Kruskal-Wallis Test, pages 288–290. Springer New York, New York, NY , 2008. 4
2008
-
[11]
An image is worth 16x16 words: Transformers for image recognition at scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...
2010 arXiv
-
[12]
Complex- ity results for explanations in the structural-model ap- proach
Thomas Eiter and Thomas Lukasiewicz. Complex- ity results for explanations in the structural-model ap- proach. Artif. Intell., 154(1-2):145–198, 2004. 3
2004
-
[13]
Eva: Exploring the limits of masked visual representation learning at scale
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceed- ings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition ...
2023
-
[14]
Joseph Y . Halpern. Actual Causality. The MIT Press,
-
[15]
Halpern and Judea Pearl
Joseph Y . Halpern and Judea Pearl. Causes and expla- nations: A structural-model approach. Part I: Causes. British Journal for the Philosophy of Science , 56(4),
-
[16]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 770–778, 2016. 4
2016
-
[17]
Huttenlocher, G.A
D.P. Huttenlocher, G.A. Klanderman, and W.J. Ruck- lidge. Comparing images using the hausdorff distance. IEEE Transactions on Pattern Analysis and Machine Intelligence, 15(9):850–863, 1993. 3
1993
-
[18]
Compar- ing the decision-making mechanisms by transformers and cnns via explanation methods
Mingqi Jiang, Saeed Khorram, and Li Fuxin. Compar- ing the decision-making mechanisms by transformers and cnns via explanation methods. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9546–9555, 2024. 1, 2, 8
2024
-
[19]
Flaws of imagenet, com- puter vision’s favourite dataset
Nikita Kisel, Illia V olkov, Kateˇrina Hanzelkov´a, Klara Janouskova, and Jiri Matas. Flaws of imagenet, com- puter vision’s favourite dataset. In The Fourth Blog- post Track at ICLR 2025, 2021. 8
2025
-
[20]
Cap- tum: A unified and generic model interpretability li- brary for pytorch, 2020
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz-Richardson. Cap- tum: A unified and generic model interpretability li- brary for pytorch, 2020. 2
2020
-
[21]
A convnet for the 2020s
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 11976–11986, 2022. 4
2022
-
[22]
A unified approach to interpreting model predictions
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. InAdvances in Neu- ral Information Processing Systems 30 , pages 4765–
-
[23]
A com- parison between vgg16, vgg19 and resnet50 architec- ture frameworks for image classification
Sheldon Mascarenhas and Mukul Agarwal. A com- parison between vgg16, vgg19 and resnet50 architec- ture frameworks for image classification. In 2021 International Conference on Disruptive Technolo- gies for Multi-Disciplinary Research and Applications (CENTCON), pages 96–99, 2021. 2
2021
-
[24]
Compar- ison of neural network and maximum likelihood ap- proaches in image classification
MR Mustapha, HS Lim, and MZ Mat Jafri. Compar- ison of neural network and maximum likelihood ap- proaches in image classification. Journal of Applied Sciences(Faisalabad), 10(22):2847–2854, 2010. 1
2010
-
[25]
Intriguing prop- erties of vision transformers
Muhammad Muzammal Naseer, Kanchana Ranas- inghe, Salman H Khan, Munawar Hayat, Fahad Shah- baz Khan, and Ming-Hsuan Yang. Intriguing prop- erties of vision transformers. In Advances in Neural Information Processing Systems, pages 23296–23308. Curran Associates, Inc., 2021. 1
2021
-
[26]
Papadimitriou
C.H. Papadimitriou. The complexity of unique solu- tions. Journal of ACM, 31:492–500, 1984. 3
1984
-
[27]
Rise: Ran- domized input sampling for explanation of black-box models
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Ran- domized input sampling for explanation of black-box models. ArXiv, abs/1806.07421, 2018. 2
2018 arXiv
-
[28]
Do vision transformers see like convolutional neural net- works? In Advances in Neural Information Process- ing Systems, pages 12116–12128
Maithra Raghu, Thomas Unterthiner, Simon Korn- blith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural net- works? In Advances in Neural Information Process- ing Systems, pages 12116–12128. Curran Associates, Inc., 2021. 1
2021
-
[29]
Deep convo- lutional neural networks for image classification: A comprehensive review
Waseem Rawat and Zenghui Wang. Deep convo- lutional neural networks for image classification: A comprehensive review. Neural Computation, 29(9): 2352–2449, 2017. 2
2017
-
[30]
”why should i trust you?”: Explaining the predictions of any classifier
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. ”why should i trust you?”: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , page 1135–1144, New York, NY , USA, 2016. As...
2016
-
[31]
Grad-CAM: Visual explanations from deep networks via gradient-based localization
Ramprasaath R Selvaraju, Michael Cogswell, Ab- hishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual explanations from deep networks via gradient-based localization. In In- ternational Conference on Computer Vision (ICCV) , pages 618–626. IEEE, 2017. 2
2017
-
[32]
An anal- ysis of convolutional neural networks for image clas- sification
Neha Sharma, Vibhor Jain, and Anju Mishra. An anal- ysis of convolutional neural networks for image clas- sification. Procedia Computer Science, 132:377–384,
-
[33]
One explanation is not enough: Structured attention graphs for image clas- sification
Vivswan Shitole, Fuxin Li, Minsuk Kahng, Prasad Tadepalli, and Alan Fern. One explanation is not enough: Structured attention graphs for image clas- sification. In Neural Information Processing Systems (NeurIPS), pages 11352–11363, 2021. 1, 2
2021
-
[34]
Siegel and N.J
S. Siegel and N.J. Castellan. Nonparametric Statistics for the Behavioral Sciences. McGraw-Hill, 1988. 5
1988
-
[35]
Sørensen
T. Sørensen. A method of establishing groups of equal amplitude in plant sociology based on similarity of species and its application to analyses of the vegeta- tion on Danish commons. Kongelige Danske Vidensk- abernes Selskab., 5:1—-34, 1948. 3
1948
-
[36]
Is robustness the cost of accuracy? – a comprehensive study on the robustness of 18 deep image classification models
Dong Su, Huan Zhang, Hongge Chen, Jinfeng Yi, Pin- Yu Chen, and Yupeng Gao. Is robustness the cost of accuracy? – a comprehensive study on the robustness of 18 deep image classification models. In Proceed- ings of the European Conference on Computer Vision (ECCV), 2018. 1, 2
2018
-
[37]
Rethinking the in- ception architecture for computer vision
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the in- ception architecture for computer vision. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016. 4
2016
-
[38]
When does dough become a bagel? analyzing the re- maining mistakes on imagenet
Vijay Vasudevan, Benjamin Caine, Raphael Gontijo- Lopes, Sara Fridovich-Keil, and Rebecca Roelofs. When does dough become a bagel? analyzing the re- maining mistakes on imagenet. In Proceedings of the 36th International Conference on Neural Information Processing Systems, Red ...
2022
-
[39]
Comparative anal- ysis of image classification algorithms based on tradi- tional machine learning and deep learning
Pin Wang, En Fan, and Peng Wang. Comparative anal- ysis of image classification algorithms based on tradi- tional machine learning and deep learning. Pattern Recognition Letters, 141:61–67, 2021. 2
2021
-
[40]
Re-labeling imagenet: from single to multi-labels, from global to localized labels
Sangdoo Yun, Seong Joon Oh, Byeongho Heo, Dongyoon Han, Junsuk Choe, and Sanghyuk Chun. Re-labeling imagenet: from single to multi-labels, from global to localized labels. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), pages 2340–2350, 2021. 8
2021
-
[2018]
International Conference on Computational In- telligence and Data Science. 2
-
[4774]
Curran Associates, Inc., 2017. 2
2017
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.