Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Shortcut Learning Susceptibility in Vision Classifiers

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Vision transformers with positional encodings rely on injected shortcuts more than CNNs, which resist them best.

desk verdict Plausible architecture-comparison study undercut by missing clean-data baseline and absent variance estimates; the ranking is not yet established. read the letter →

arxiv 2502.09150 v2 pith:WTQQ5SRK submitted 2025-02-13 cs.LG cs.CV

Pirzada Suhail , Vrinda Goel , Amit Sethi This is my paper · ORCID
classification cs.LGcs.CV
keywords shortcutlearningspuriouscorrelationsvisiontransformersconvolutionalneuralnetworksmulti-layerperceptronsnetworkinversionratesensitivity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to rank how readily three vision architectures—CNNs, MLPs, and ViTs—fall back on injected shortcut cues instead of learning the actual class features. The authors insert positional and intensity-based shortcuts into four image datasets, train each architecture on the modified data, and measure the accuracy gap between a test set that keeps the shortcuts and one that removes them. A larger gap is read as stronger shortcut reliance. They report that CNNs show the smallest gaps, ViTs with positional encodings the largest, with MLPs in between; intensity-based shortcuts flip the ordering, making MLPs the worst. The same protocol with different learning rates shows that small learning rates favor learning meaningful features over shortcuts.

What carries the argument

The load-bearing mechanism is the Accuracy Difference: the absolute difference between a model's accuracy on the shortcut test set and its accuracy on the normal test set, with a parallel Loss Difference, used as the quantitative measure of shortcut reliance. The shortcut injection is the controlled manipulation: a $4\times4$ white patch in class-correlated positions (positional) or a class-correlated pixel-intensity alteration (intensity). Network inversion, based on a composite loss that reconstructs training-like images from the classifier's weights, supplies the qualitative evidence about what each architecture internalizes. Together these let the authors compare architectures under identical shortcut exposure and attribute performance gaps to shortcut use rather than to dataset difficulty.

What would settle it

Train each architecture on the unmodified dataset and evaluate it on the same shortcut and normal test sets; if the accuracy gap is comparable to the gap seen after shortcut training, the measured susceptibility is confounded by base feature-learning ability rather than shortcut reliance.

Watch

Extended reading notes

Core claim

The central discovery is a conditional ranking of shortcut susceptibility. For positional shortcuts—a $4\times4$ white patch placed at a class-correlated location—CNNs are the most resistant, with the smallest accuracy and loss differences between the shortcut and normal test sets; ViTs with positional encodings are the most susceptible, because the encoding makes the shortcut's spatial location directly readable; ViTs without positional encodings land near CNNs, and MLPs sit in between. For intensity-based shortcuts, CNNs are again most resistant but MLPs become the most susceptible, while ViTs without positional encodings track CNNs on natural-image datasets and MLPs on digit datasets. The paper further claims that training at a small learning rate ($10^{-5}$ or $10^{-4}$) produces gradual learning that favors class-relevant features, whereas a large learning rate ($10^{-3}$) latches onto shortcut cues. Network-inversion reconstructions are offered as qualitative evidence that CNNs preserve spatial, class-relevant structure while ViTs reconstruct shortcut-dominated artifacts.

Load-bearing premise

The comparison assumes the accuracy difference between shortcut and normal test sets is caused solely by the injected shortcut, but no clean-trained baseline is measured, so a model that learns real features poorly could look shortcut-susceptible for unrelated reasons.

Editorial extensions

If this is right

  • If the ranking holds, architecture choice is itself a shortcut-mitigation decision: CNNs' spatial priors make them the safer default when spurious positional cues are plausible.
  • For ViTs with positional encodings, the result implies that position leakage is a concrete failure mode, so removing, randomizing, or regularizing positional encodings should reduce shortcut reliance.
  • The learning-rate finding gives a cheap, architecture-agnostic intervention: training at small learning rates ($10^{-4}$ or below) should lower susceptibility without changing the model.
  • The two-test-set protocol offers a reusable benchmark: future models can be compared on the accuracy gap between shortcut and normal test sets under the same injected cues.
  • Because the ranking changes with shortcut type, evaluations that use only one cue type are incomplete; robustness claims should be stated per shortcut modality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested control—training each architecture on clean data and measuring the same gap—would separate shortcut reliance from general feature-learning capacity; without it, a model that simply learns real features poorly could look shortcut-prone.
  • The positional-encoding result suggests a direct experiment the paper did not run: retrain the same ViT with positional encodings shuffled or removed and compare the accuracy gap, which would isolate the encoding's causal contribution.
  • Network inversion could be turned quantitative by scoring reconstructed images for class-feature fidelity; the current qualitative claim would then be testable at scale.
  • If low learning rates generalize, learning-rate schedules become a practical debiasing knob for deployed ViTs, complementing architectural fixes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper studies how CNNs, MLPs, and ViTs (with and without positional encodings) learn when artificial shortcuts are injected into training data. The authors define Accuracy Difference (absolute difference between test accuracy on shortcut-modified and clean test sets) and Loss Difference, and report bar-chart comparisons across four datasets (MNIST, SVHN, CIFAR-10, Fashion-MNIST) for positional and intensity shortcuts, plus learning-rate variations and network-inversion reconstructions. They conclude that CNNs are the most resistant, ViTs with positional encodings the most susceptible, and MLPs intermediate, with low learning rates promoting feature learning.

Significance. If the quantitative ranking were statistically supported, the paper would provide a useful practical guide for architecture selection under known spurious correlations. The controlled shortcut design and multiple datasets are assets. However, the current evidence is not sufficient to establish the ranking: there are no error bars, seeds, or numerical tables, and no clean-data baseline, so the primary metric may conflate shortcut reliance with architecture-level feature-learning capability. The qualitative inversion results also lack validation. The contribution is therefore suggestive rather than conclusive.

major comments (4)
  1. [Section 4, Figures 1–3] The central ranking claims are read entirely from figures; no per-seed variation, error bars, or numerical tables are provided. Because the differences between architectures (e.g., MLPs vs. ViTs without positional encodings) may be small, the reader cannot determine whether the ordering is statistically reliable. Add mean and standard deviation over at least five seeds, with tables of accuracy, loss, and their differences for every dataset, shortcut type, and learning rate.
  2. [Section 3, Accuracy Difference] The metric compares accuracy on the shortcut test set with accuracy on the normal test set, but no model is trained on clean data. A model with weak spatial inductive biases (e.g., an MLP or a from-scratch ViT) may have low accuracy on the normal test set even without any shortcut, so its large Accuracy Difference could reflect poor feature learning rather than selective reliance on the injected shortcut. Add a clean-data baseline and report the drop normalized by clean-test accuracy, or otherwise show that the architecture ranking is invariant to this confound.
  3. [Abstract vs. Section 4] The abstract says ViTs without positional encodings "almost entirely ignore the distinctive image features in the presence of shortcuts," but Section 4 states that ViTs without positional encodings perform "comparably better, close to CNNs" and are less susceptible. These statements are contradictory; clarify the intended claim (e.g., reconstruction quality vs. accuracy drop) and make the abstract consistent with the reported results.
  4. [Section 4, Figure 3] The conclusion that a small learning rate favors "meaningful, class-relevant features" is based on a single unlabeled comparison with no quantitative values and no control for number of epochs or learning-rate schedule. At a small learning rate the model may simply underfit, and the lower Accuracy Difference may reflect under-training rather than a qualitative shift in feature acquisition. Report the actual accuracies and losses and, ideally, a training-time-matched comparison.
minor comments (5)
  1. [Page 2, Related Works] Several citations are missing surrounding spaces, for example "segmentation accuracyLin et al. (2024)" and "background artifacts."; please fix the LaTeX spacing.
  2. [Section 3] "Loss Difference" is used in Figures 1–3 but is never defined; please provide a formal definition in the methodology.
  3. [Section 4] It is unclear whether the CIFAR-10 results use the standard 32×32 resolution and the canonical train/test split; please specify the exact dataset setup.
  4. [Section 4, Figure 4] The qualitative reconstructions are presented without any quantitative evaluation, such as fidelity to original training samples or a classification-confidence measure; please add a quantitative support or a clear description of the inversion loss used.
  5. [Section 3] The paper uses the absolute difference for Accuracy Difference, which discards the sign; consider reporting a signed difference or explaining why the sign is irrelevant in the studied settings.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation in the central Accuracy Difference ranking; only a minor self-cited inversion method supports the qualitative reconstruction component.

full rationale

The paper's principal quantitative claims are direct measurements rather than derived predictions: models are trained on shortcut-modified data and evaluated on shortcut and clean test sets, and the Accuracy Difference is computed from those accuracies. No fitted parameter is later called a prediction, and no author-specific uniqueness theorem or ansatz is invoked to force the architecture ranking. The main methodological caveat—no clean-trained baseline, so Accuracy Difference may reflect baseline feature-learning ability rather than shortcut reliance—is a validity/confounding concern, not a circularity of the kind defined here. The only self-citation that matters is the network-inversion method from Suhail & Sethi (2024a), used for the qualitative Figure 4 reconstructions. That method is not validated in the present paper, so the qualitative interpretations lean on the authors' prior work; however, the central quantitative ranking in Figures 1–3 is independent of the inversion tool. This gives at most a minor self-citation note (score 2), not load-bearing circularity.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

No mathematical derivation is involved; the central claims rest on unverified experimental assumptions about the metric, the test-set equivalence, and the inversion method.

free parameters (4)
  • Shortcut intensity magnitude
    Used in intensity-based shortcuts; exact value not reported, and results may depend on it.
  • Shortcut patch location and size = 4x4 white patch at class-specific locations
    Exact positions are not specified; the positional-shortcut ranking could depend on the chosen positions.
  • Learning rate grid = 1e-5, 1e-4, 1e-3
    The learning-rate conclusion is based on three hand-selected rates, with no schedule details reported.
  • Model hyperparameters
    Architecture depth and width, patch size, training epochs, optimizer, and batch size are not reported; the comparison across architecture families depends on these unstated choices.
assumptions (3)
  • domain assumption Accuracy Difference between shortcut and normal test sets is a valid measure of shortcut reliance.
    Used in Section 4 to rank architectures; assumes no other distribution shift or baseline differences.
  • domain assumption The normal test set differs from the shortcut test set only by absence of the injected shortcut.
    Required for interpreting accuracy drops as shortcut reliance; not verified with a clean-training baseline.
  • domain assumption Network inversion reconstructions reflect the features the classifier internalized.
    Used for qualitative claims in Section 4, Figure 4; method inherited from the authors' prior work and not validated here.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Shortcut Learning Susceptibility in Vision Classifiers." pith.science (2026). https://pith.science/paper/WTQQ5SRK

@misc{pith2026250209150,
  author       = {Pith},
  title        = {Pith review of: Shortcut Learning Susceptibility in Vision Classifiers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WTQQ5SRK}},
  note         = {Machine review of arXiv:2502.09150}
}
read the original abstract

Shortcut learning, where machine learning models exploit spurious correlations in data instead of capturing meaningful features, poses a significant challenge to building robust and generalizable models. This phenomenon is prevalent across various machine learning applications, including vision, natural language processing, and speech recognition, where models may find unintended cues that minimize training loss but fail to capture the underlying structure of the data. Vision classifiers based on Convolutional Neural Networks (CNNs), Multi-Layer Perceptrons (MLPs), and Vision Transformers (ViTs) leverage distinct architectural principles to process spatial and structural information, making them differently susceptible to shortcut learning. In this study, we systematically evaluate these architectures by introducing deliberate shortcuts into the dataset that are correlated with class labels both positionally and via intensity, creating a controlled setup to assess whether models rely on these artificial cues or learn actual distinguishing features. We perform both quantitative evaluation by training on the shortcut-modified dataset and testing on two different test sets-one containing the same shortcuts and another without them-to determine the extent of reliance on shortcuts. Additionally, qualitative evaluation is performed using network inversion-based reconstruction techniques to analyze what the models internalize in their weights, aiming to reconstruct the training data as perceived by the classifiers. Further, we evaluate susceptibility to shortcut learning across different learning rates. Our analysis reveals that CNNs at lower learning rates tend to be more reserved against entirely picking up shortcut features, while ViTs, particularly those without positional encodings, almost entirely ignore the distinctive image features in the presence of shortcuts.

Figures

Figures reproduced from arXiv: 2502.09150 by the authors.

Figure 1
Figure 1. Accuracy Difference across architectures for positional shortcuts. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Accuracy Difference across architectures for intensity-based shortcuts. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Accuracy Difference across different learning rates ( [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Reconstructed training samples as perceived by the classifier. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. VEIL: How Visual Encoding Hijacking Induces Bias In Vision Models

    cs.CV 2026-07 conditional novelty 6.5 of 10

    Vision models trained on chart images of time series often latch onto rendering style rather than temporal class structure, an effect the authors call visual encoding hijacking.

Reference graph

Works this paper leans on

19 extracted references · 8 canonical work pages · cited by 1 Pith paper

  1. [1]

    Autoinverse: Uncertainty Aware Inversion of Neural Networks

    Navid Ansari, Hans-Peter Seidel, Nima Vahidi Ferdowsi, and Vahid Babaei. Autoinverse: Uncertainty aware inversion of neural networks, 2022. URL https://arxiv.org/abs/2208.13780

    work page Pith review arXiv 2022

  2. [2]

    Demonstrating and reducing shortcuts in vision-language representation learning

    Maurits Bleeker, Mariya Hendriksen, Andrew Yates, and Maarten de Rijke. Demonstrating and reducing shortcuts in vision-language representation learning. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=gfANevPraH

    work page 2024

  3. [3]

    Detecting shortcut learning for fair medical ai using shortcut testing

    Alexander Brown, Nenad Tomasev, Jan Freyberg, Yuan Liu, Alan Karthikesalingam, and Jessica Schrouff. Detecting shortcut learning for fair medical ai using shortcut testing. Nature Communications, 14 0 (1), July 2023. ISSN 2041-1723. doi:10.1038/s41467-023-39902-7. URL http://dx.doi.org/10.1038/s41467-023-39902-7

    doi:10.1038/s41467-023-39902-7 2023

  4. [4]

    Reconstructing training data from multiclass neural networks, 2023

    Gon Buzaglo, Niv Haim, Gilad Yehudai, Gal Vardi, and Michal Irani. Reconstructing training data from multiclass neural networks, 2023. URL https://arxiv.org/abs/2305.03350

    arXiv 2023

  5. [5]

    Wichmann

    Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2 0 (11): 0 665–673, November 2020. ISSN 2522-5839. doi:10.1038/s42256-020-00257-z. URL http://dx.doi.org/10.1038/s42256-020-00257-z

    doi:10.1038/s42256-020-00257-z 2020

  6. [6]

    Reconstructing training data from trained neural networks, 2022

    Niv Haim, Gal Vardi, Gilad Yehudai, Ohad Shamir, and Michal Irani. Reconstructing training data from trained neural networks, 2022. URL https://arxiv.org/abs/2206.07758

    arXiv 2022

  7. [7]

    On the foundations of shortcut learning

    Katherine Hermann, Hossein Mobahi, Thomas FEL, and Michael Curtis Mozer. On the foundations of shortcut learning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=Tj3xLVuE9f

    work page 2024

  8. [8]

    Jensen, R.D

    C.A. Jensen, R.D. Reed, R.J. Marks, M.A. El-Sharkawi, Jae-Byung Jung, R.T. Miyamoto, G.M. Anderson, and C.J. Eggen. Inversion of feedforward neural networks: algorithms and applications. Proceedings of the IEEE, 87 0 (9): 0 1536--1549, 1999. doi:10.1109/5.784232

    doi:10.1109/5.784232 1999

Show all 19 references
  1. [9]

    Inversion of neural networks by gradient descent

    J Kindermann and A Linden. Inversion of neural networks by gradient descent. Parallel Computing, 14 0 (3): 0 277--286, 1990. ISSN 0167-8191. doi:https://doi.org/10.1016/0167-8191(90)90081-J. URL https://www.sciencedirect.com/science/article/pii/016781919090081J

    1990

  2. [10]

    Manxi Lin, Nina Weng, Kamil Mikolaj, Zahra Bashir, Morten B. S. Svendsen, Martin G. Tolsgaard, Anders N. Christensen, and Aasa Feragen. Shortcut learning in medical image segmentation. In Marius George Linguraru, Qi Dou, Aasa Feragen, Stamatia Giannarou, Ben Glocker, Karim Lek...

    2024

  3. [11]

    Landscape learning for neural network inversion, 2022

    Ruoshi Liu, Chengzhi Mao, Purva Tendulkar, Hao Wang, and Carl Vondrick. Landscape learning for neural network inversion, 2022. URL https://arxiv.org/abs/2206.09027

    2022 arXiv

  4. [12]

    Rectify vit shortcut learning by visual saliency

    Chong Ma, Lin Zhao, Yuzhong Chen, Lei Guo, Tuo Zhang, Xintao Hu, Dinggang Shen, Xi Jiang, and Tianming Liu. Rectify vit shortcut learning by visual saliency. IEEE Transactions on Neural Networks and Learning Systems, 35 0 (12): 0 18013--18025, 2024. doi:10.1109/TNNLS.2023.3310531

    2024

  5. [13]

    Network inversion of binarised neural nets

    Pirzada Suhail. Network inversion of binarised neural nets. In The Second Tiny Papers Track at ICLR 2024, 2024. URL https://openreview.net/forum?id=zKcB0vb7qd

    2024

  6. [14]

    Network inversion for training-like data reconstruction

    Pirzada Suhail and Amit Sethi. Network inversion for training-like data reconstruction. In Neurips Safe Generative AI Workshop 2024, 2024 a . URL https://openreview.net/forum?id=rJy9ytRnep

    2024

  7. [15]

    Network inversion of convolutional neural nets

    Pirzada Suhail and Amit Sethi. Network inversion of convolutional neural nets. In Muslims in ML Workshop co-located with NeurIPS 2024, 2024 b . URL https://openreview.net/forum?id=f9sUu7U1Cp

    2024

  8. [16]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  9. [17]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  10. [18]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  11. [19]

    m @" 6fsviO(,_ ' ʭo lpw * r l/[gs bF> *?W x @ 9 ECm<.4 8 4 0 8 w [ s* d p 8 j p 6 G

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    1999

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.