Pith. sign in

REVIEW 4 major objections 4 minor 13 references

Corner Cases: How Size and Position of Objects Challenge ImageNet-Trained Models

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Small, off-center objects make ImageNet models trust backgrounds more.

desk verdict Useful benchmark and broad evaluation, but the central spurious-reliance claim outruns the evidence: the paper shows accuracy drops, not that errors track background class. read the letter →

arxiv 2505.03569 v2 pith:UL67CWOI submitted 2025-05-06 cs.CV

Mishal Fatima , Steffen Jung , Margret Keuper This is my paper · ORCID
classification cs.CV
keywords spuriouscorrelationsbackgroundrelianceobjectsizebiaspositionImageNet-1kgrouprobustnesssyntheticbenchmarkcompositionshift
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that two mundane properties of a photograph—how large the object is and where it sits in the frame—determine how much a trained classifier leans on the background rather than the object. Using ImageNet-1k's bounding boxes, the authors measure a center and size bias in the dataset and show that classes with smaller, less-centered objects are more often predicted correctly even after the object is inpainted away. To make the effect testable, they build Hard-Spurious-ImageNet, a synthetic benchmark in which foreground objects are resized to 56×56, 84×84, or 112×112 pixels and placed in the center or a corner against their original or a random inpainted background. Across many pretrained architectures, accuracy collapses when the object is small and off-center against an unfamiliar background, and standard spurious-feature fixes—data augmentation, DFR, and JTT—do not close the gap. If the claim holds, object size and position are not nuisance factors but first-order modulators of spurious background reliance that evaluation and mitigation methods must explicitly control.

What carries the argument

The load-bearing machinery is Hard-Spurious-ImageNet, a synthetic dataset built by cutting each ImageNet-1k ground-truth bounding box out of its image, inpainting the hole to produce a background-only image, then pasting the foreground back at controlled scales (56×56, 84×84, 112×112) and positions (center or corner) against either the original inpainted background or a random one. This yields four groups—CeO, CoO, CeR, and CoR—whose accuracy gaps isolate the contribution of size and position to background reliance. The paper also introduces two scalar class-level scores: a centeredness score based on the ℓ∞ distance between image and object centers, and a size score based on the fraction of image pixels covered by the object's bounding box; the negative correlation of these scores with inpainted-image accuracy is the paper's quantitative evidence that composition bias tracks spuriosity.

What would settle it

An experiment that would settle the claim: take the same core objects at 56×56 in the corner and place them on neutral, non-spurious backgrounds (e.g., flat gray or uniform noise) instead of natural inpainted scenes. If accuracy stays as low as on the random-background groups, the drop is caused by size and position alone rather than by background reliance; if accuracy recovers, the background is the decisive factor. A second check would compare inpainted accuracy against a control where the object is removed and the background is replaced by a class-independent texture, which would isolate inpainting artifacts from genuine background cues.

Watch

Extended reading notes

Core claim

The paper's central discovery is that spurious background reliance in ImageNet-trained models is systematically modulated by object size and position: the smaller the region-of-interest relative to the image and the farther the object is from the center, the more the model's prediction is carried by background cues. The evidence is the Hard-Spurious-ImageNet benchmark, where the hardest groups (small objects against random backgrounds in the corner) drop to single-digit or low-teens accuracy for most models, far below the same models' clean ImageNet accuracy, while the same object at 112×112 in the center keeps most of the performance. The authors further report that a low center/size score for a class correlates with higher accuracy on inpainted images (object removed), their proxy for learned background association. Finally, they show that neither data augmentation nor group-robustness retraining methods (DFR, JTT) recovers the hard-group accuracy, indicating that current mitigation approaches are not built for compositional shifts in size and location.

Load-bearing premise

The analysis assumes that a model's accuracy on inpainted images and the accuracy gaps between the four Hard-Spurious-ImageNet groups measure learned background associations; if inpainting artifacts, leftover object fragments, or the intrinsic difficulty of recognizing small off-center objects are what drive the numbers, the central claim weakens.

Editorial extensions

If this is right

  • Worst-group evaluation of spurious-feature robustness should include size and position as group-defining attributes, not just background; benchmarks that omit them overstate worst-group performance.
  • Data-augmentation recipes and group-robust methods (DFR, JTT) need to be retested or redesigned for compositional shifts; the paper's numbers show these methods leave the hardest size/location groups largely unrepaired.
  • Model comparisons on standard centered, large-crop ImageNet evaluation are optimistic for real deployment, where objects are often small and peripheral; accuracy in the wild should be measured with composition-controlled subsets.
  • Because size had a stronger effect than position across architectures, scale-invariant or multi-scale training may be a more promising direction than translation-invariance fixes alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the size-position mechanism generalizes beyond ImageNet, similar composition-controlled benchmarks in satellite imagery, surveillance, and medical imaging would likely expose the same background-reliance failure; this is a testable prediction, not a result of the paper.
  • The paper's proxy treats inpainted accuracy as spuriosity; a cleaner test would use human-verified 'object absent' pairs and measure whether model confidence tracks background class, which would separate genuine background association from inpainting artifacts.
  • The failure of DFR and JTT suggests that group definitions should be expanded from background type to the joint distribution of size, position, and background; adding size/location groups to existing robustness methods could be evaluated directly on the released dataset.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper studies how object size and position modulate spurious background reliance in ImageNet-trained classifiers. It introduces centeredness and size scores computed from ImageNet ground-truth bounding boxes, shows negative correlations between these scores and per-class accuracy on inpainted (object-removed) images, and constructs Hard-Spurious-ImageNet, a synthetic dataset that pastes ImageNet core objects at controlled sizes (56, 84, 112 px) and positions (center vs. top-right corner) against original or random inpainted backgrounds. It evaluates many pretrained models, plus ERM, DFR, and JTT training strategies, and finds that smaller, off-center objects yield the largest accuracy gaps and that existing debiasing methods provide only marginal gains on the hard groups. The authors also provide an aspect-ratio-preserving variant and a Hard-Spurious-ImageNet-10 subset with class-selected spurious backgrounds.

Significance. If the causal interpretation holds, this is a useful and timely benchmark contribution: it quantifies two understudied dataset biases and shows that standard group-robustness methods do not automatically address them. The paper has concrete strengths: the scores are defined directly from ground-truth boxes; the ablations (aspect-ratio preservation, SAM masks, Hard-Spurious-ImageNet-10, ERM variants) are sensible; the dataset and code are publicly released; and the evaluation spans many architectures and training recipes. The central claim, however, rests on interpreting accuracy gaps as evidence of spurious background reliance, and that inference needs a more direct test before the headline conclusion can be accepted.

major comments (4)
  1. [Section 5.7, Table 6; Section 3.3] The central claim that the CeR/CoR accuracy collapse is caused by spurious background reliance is not directly supported by the reported experiments. The CeR/CoR groups change the background and the object composition simultaneously: a small off-center object pasted into an unrelated scene is intrinsically harder to recognize, so a model with no learned background associations could show the same drop. The Appendix C example (Figure 10) notes that the Hard-Spurious-ImageNet-10 backgrounds contain cues that predict the background label, which is the right kind of evidence, but no aggregate predicted-label analysis is reported. I request a predicted-label confusion analysis on CeR/CoR groups (e.g., for Hard-Spurious-ImageNet-10, the fraction of errors whose predicted class equals the background class, and a comparison with a control where backgrounds are class-uncorrelated) to separate the spurious-reliance mechanism from generic difficulty of small off-center objects.
  2. [Section 4, Table 1; Section 6] The position-dependence claim rests on a single test location. The main test set places the core object only in the top-right corner, while the random-corner variant is introduced only together with aspect-ratio preservation, so position is confounded with resizing style. A position sweep (e.g., four corners and edge midpoints at a fixed object size) is needed to establish that 'far from the center' rather than 'in the top-right corner' is what drives the accuracy drop. The paper acknowledges this limitation in Section 6, but the limitation is load-bearing for the headline claim about off-center placement.
  3. [Section 5.7, Table 6] The claim that current mitigation methods 'fail to achieve considerable performance gains' is based on single runs without error bars or significance tests. ERM and DFR differ by fractions of a percent in average accuracy (65.94 vs. 65.97), and the Hard-group difference (57.56 vs. 59.79) could easily be within seed noise. Please report means and standard deviations over multiple seeds and a paired comparison for the Hard group, or soften the conclusion to a qualitative statement. Without this, the secondary contribution about method failure is not yet established.
  4. [Section 3.3, Figure 4] The spuriosity proxy used to motivate the dataset is per-class validation accuracy on inpainted ImageNet computed with a single model (ConvNext-Base), and the reported correlations with the center and size scores are weak (Spearman around -0.4). Class difficulty is an uncontrolled confound: harder classes may have low inpainted accuracy regardless of background dependence. Reporting the correlation averaged over several architectures and, where feasible, a difficulty-controlled partial correlation would substantially strengthen the motivating analysis.
minor comments (4)
  1. [Section 5.8] The text says 'ERMeasy ... consisting of subgroups CeO and CoO for size 54×54 and 112×112'; the first size should presumably be 56×56.
  2. [Appendix A and Table 8] There is a typo: 'Hard-Spurios-ImageNet-10' should be 'Hard-Spurious-ImageNet-10'.
  3. [Section 3.3] The sentence 'a higher spurious feature reliance is observed in case of non-centered large object sizes' appears to contradict the negative correlation direction; the intended meaning is likely smaller and less centered objects.
  4. [Section 5.4] The interpretation that augmented training 'may end up highlighting' spurious features is speculative; a concrete analysis of where the augmented model makes errors on CoR (e.g., saliency or predicted-label statistics) would be needed to support it.

Circularity Check

1 steps flagged · score 3.0 of 10

One partial self-selection step in Hard-Spurious-ImageNet-10; the main random-background benchmark is independent.

  1. fitted input called prediction [Section 4.1 (Hard-Spurious-ImageNet-10); evaluated in Appendix A, Table 8.]
    "To achieve this, we first analyze the level of spuriousity present in every class. We give inpainted images without the core objects, as input to the pretrained ConvNext-Base model , and record the accuracies of every class. The classes where accuracies are high indicate that the model has learnt to predict the class label without the presence of core objects. We choose 10 classes that are highly spurious"

    The ten background classes for Hard-Spurious-ImageNet-10 are selected using ConvNext-Base's own accuracy on inpainted images, which is the paper's operational measure of spurious reliance. The same ConvNext-Base is then evaluated on the resulting benchmark, and the CeR/CoR accuracy collapse in Table 8 is interpreted as evidence that 'the strength of spurious backgrounds is far greater than that of core features.' For ConvNext-Base specifically, this is partly self-confirming: the background set was chosen to maximize this model's tendency to predict the class from background alone.

full rationale

The paper's core object-size and centering scores are computed from ground-truth bounding boxes via Equations 1 and 2, independently of any model, so the correlations in Figure 4 are not circular. The main benchmark's CeO/CoO versus CeR/CoR comparison keeps the same cropped object and changes only the background from the original inpainted scene to a random inpainted scene, which is a clean intervention for detecting background dependence. The DFR, JTT, and data-augmentation experiments use standard published methods and are evaluated on the proposed groups, so no fitted parameter is renamed as a prediction. The only notable circularity is the Hard-Spurious-ImageNet-10 variant: its backgrounds are selected using ConvNext-Base's inpainted-image accuracy, and ConvNext-Base is then one of the models used to demonstrate that these backgrounds dominate small off-center objects. That step is self-referential for ConvNext-Base, but it does not drive the central random-background results, which hold across ResNet-50, CoAtNet, Hiera, and MViTv2. Section 6's stated limitations (one corner location, reliance on bounding boxes) concern scope and external validity rather than circularity. Overall circularity is low but not zero because of this one fitted selection step.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The paper is empirical and introduces no fitted mathematical constants. The listed free parameters are hand-chosen benchmark settings (object sizes, single corner location, training group proportions) and method hyperparameters (DFR lambda, JTT lambda_up) that the experiments depend on. The axioms are the measurement assumptions connecting the synthetic dataset and the inpainted-accuracy proxy to the claim about spurious background reliance.

free parameters (5)
  • Object size set = {56, 84, 112} pixels
    Hand-chosen core object resolutions for the benchmark; the monotonic trend is robust across these values, but the specific values are arbitrary design choices.
  • Object corner offset = top-right corner for main set; random corners for AR variant
    Position is only tested as a binary (center versus one corner) in the main dataset. The paper acknowledges only one location was used (Section 6), so the position axis is under-sampled.
  • Training group proportions = 320 easy, 80 medium/hard images per class
    Hand-chosen to mimic the long-tailed hardness distribution of ImageNet; affects absolute accuracies in the fine-tuning experiments but not the qualitative trend.
  • DFR l2 regularization lambda = 100
    Carried over from Kirichenko et al. for ImageNet-9; not tuned on this dataset (Supplementary F), so the DFR result may be pessimistic.
  • JTT upsampling factor lambda_up = 50
    Hand-chosen following prior work; not swept, so the JTT result may not reflect its optimal performance.
assumptions (3)
  • domain assumption ImageNet ground-truth bounding boxes correctly separate the core object from the background.
    Used to crop core features and to define backgrounds in Sections 3 and 4. Partially validated with Grounding DINO (average mIOU 0.8675), but 139 classes have mIOU below 0.8 (Supplementary D).
  • domain assumption Validation accuracy after inpainting the object measures spurious background reliance.
    Section 3.3 equates high inpainted accuracy with high spurious reliance; artifacts from LaMa or SAM, or residual object fragments, could inflate accuracy independently of true background associations.
  • domain assumption Accuracy gaps on the synthetic Hard-Spurious-ImageNet groups are driven by background reliance rather than generic out-of-distribution difficulty.
    The central interpretation in Sections 5 and 5.7; small and off-center objects are also simply harder, and the paper's GradCAM evidence is qualitative and not quantified across all groups.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Corner Cases: How Size and Position of Objects Challenge ImageNet-Trained Models." pith.science (2026). https://pith.science/paper/UL67CWOI

@misc{pith2026250503569,
  author       = {Pith},
  title        = {Pith review of: Corner Cases: How Size and Position of Objects Challenge ImageNet-Trained Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UL67CWOI}},
  note         = {Machine review of arXiv:2505.03569}
}
read the original abstract

Backgrounds in images play a major role in contributing to spurious correlations among different data points. Owing to aesthetic preferences of humans capturing the images, datasets can exhibit positional (location of the object within a given frame) and size (region-of-interest to image ratio) biases for different classes. In this paper, we show that these biases can impact how much a model relies on spurious features in the background to make its predictions. To better illustrate our findings, we propose a synthetic dataset derived from ImageNet-1k, Hard-Spurious-ImageNet, which contains images with various backgrounds, object positions, and object sizes. By evaluating the dataset on different pretrained models, we find that most models rely heavily on spurious features in the background when the region-of-interest (ROI) to image ratio is small and the object is far from the center of the image. Moreover, we also show that current methods that aim to mitigate harmful spurious features, do not take into account these factors, hence fail to achieve considerable performance gains for worst-group accuracies when the size and location of core features in an image change. The dataset and implementation code are available at https://github.com/Mishalfatima/Corner_Cases.

Figures

Figures reproduced from arXiv: 2505.03569 by the authors.

Figure 1
Figure 1. Gradcam visualizations for Pre-trained ConvNext-Base. a) Model predicts core class [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Left: ImageNet-1k classes and their center and size scores. Toyshop has largest center and size scores, whereas Volleyball has smallest center score and Balance Beam has smallest size score. Other classes are sampled randomly for visualization. Right: Counts in log scale of relative centers of ground truth bounding boxes containing the object corresponding to the image class (ImageNet-1k validation set). Most object… view at source ↗
Figure 3
Figure 3. Histograms showing distribution of scores in different classes of ImageNet-1k validation dataset. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Correlation between the validation accuracy on inpainted ImageNet (as explained in Subsection 3.3) [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Different samples from Hard-Spurious-ImageNet. Image size remains same in all images, i.e. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Benchmarking results of different models on Groups CeO (Core object in Center against its Original [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: (left) The effect of training epochs of ERM model on the performance of DFR. ERM model trained with 20 epochs gives the highest performance for DFR. (right) ERMall narrows the gap between easy, medium, and hard groups. on "hard" examples. First, a standard ERM model is…
Figure 8
Figure 8. Figure 8: Gradcam visualizations showing regions of the image the model pays attention to in order to make [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Effect of core feature size on model performance. Both the predictions are for the ERMall model. that most of the images in these classes are set in similar environments, hence the model has learnt to associate the given outdoor environment with these classes and are i…
Figure 10
Figure 10. Figure 10: left: Original images with their resized inpainted versions. right: Despite inpainting, the background (Bg) consists of cues that help the model predict the background label. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: (top): Class-wise mIOU scores between Grounding DINO predictions and ImageNet annotations on the validation set. Averaged mIOU is 0.875. (bottom):Histograms showing distribution of scores in different classes of train data in ImageNet1k dataset. In the main paper, we …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 9 canonical work pages

  1. [5]

    ImageNet- Hard: The Hardest Images Remaining from a Study of the Power of Zoom and Spatial Biases in Image Classification

    15 Published in Transactions on Machine Learning Research (08/2025) Mohammad Reza Taesiri, Giang Nguyen, Sarra Habchi, Cor-Paul Bezemer, and Anh Nguyen. ImageNet- Hard: The Hardest Images Remaining from a Study of the Power of Zoom and Spatial Biases in Image Classification. InAdvances in Neural Information Processing Systems (NeurIPS),

    work page 2025

  2. [7]

    Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo

    URLhttps://arxiv.org/abs/2304.06790. Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features. InInternational Conference on Computer Vision (ICCV),

    arXiv

  3. [8]

    All models are pretrained on ImageNet1k only

    Similar to main paper, We test the performance of the datasets on 5 different pre-trained architectures: ConvNext-Base (Liu et al., 2022), ResNet-50 (He et al., 2016), CoATNet (Dai et al., 2021), Hiera-Base with MAE (Ryali et al., 2023), and MVit2-small (Li et al., 2022). All models are pretrained on ImageNet1k only. We see that Hard-Spurious-ImageNet-10 ...

    work page 2022

  4. [9]

    Hence, we prefer human-annotated ImageNet bounding boxes

    Compared to the results in Table 1, the results with SAM are worse, mainly because the resized SAM object masks are not entirely accurate in cases where objects are small and thin, such as insects, etc. Hence, we prefer human-annotated ImageNet bounding boxes. 18 Published in Transactions on Machine Learning Research (08/2025) Table 9: Test Accuracies on ...

    work page 2025

  5. [10]

    17 Published in Transactions on Machine Learning Research (08/2025) 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 mIOU 0 50 100 150 200Frequency A vg

    Core: Japanese Spaniel Bg:Snorkel Core: Maltese Dog Bg: Potter’s Wheel Core: Shih-Tzu Bg: Busby Hat Figure 10:left: Original images with their resized inpainted versions.right: Despite inpainting, the background (Bg) consists of cues that help the model predict the background label. 17 Published in Transactions on Machine Learning Research (08/2025) 0.2 0...

    work page 2024

  6. [50]

    For the JTT model, the images are applied with random resized cropping followed by horizontal flipping

    After extracting the embeddings from the pre-trained ERM model, the embeddings are normalized using fit_transform()andtransform()functions ofsklearn.preprocessing.StandardScalerfor val and test data, respectively. For the JTT model, the images are applied with random resized cropping followed by horizontal flipping. No additional data augmentation is appl...

    work page 2025

  7. [100]

    (2023) for optimizing the last layer for ImageNet-9 dataset (Xiao et al., 2021)

    These hyperparamters are similar to the ones set by Kirichenko et al. (2023) for optimizing the last layer for ImageNet-9 dataset (Xiao et al., 2021). Since, the data distribution in the proposed dataset and ImageNet-9 is similar, we assumed the same hyperparamteres. In case of JTT, models have the same hyperparameters as the ERM trained model.λup is set to

    work page 2023

  8. [1991]

    C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011-001, California Institute of Technology,

    2011

Show all 13 references
  1. [2018]

    16 Published in Transactions on Machine Learning Research (08/2025) Table 8: Test Accuracies on Hard-Spurious-ImageNet-10 with highly spurious backgrounds. Model Clean Accuracy Object Resolution Group Accuracies CeO CoO CeR CoR Convnext-Base 85.8 562 57.4 38.6 8.2 1.0 842 79.0...

    2025

  2. [2021]

    Sahil Singla and Soheil Feizi

    URLhttps://arxiv.org/abs/2111.02114. Sahil Singla and Soheil Feizi. Salient ImageNet: How to discover spurious features in Deep Learning? In International Conference on Learning Representations (ICLR),

    arXiv

  3. [2022]

    Spawrious: A Benchmark for Fine Control of Spurious Correlation Biases

    14 Published in Transactions on Machine Learning Research (08/2025) Aengus Lynch, Gbetondji Jean-Sebastien Dovonon, Jean Kaddour, and Ricardo Silva. Spawrious: A Benchmark for Fine Control of Spurious Correlation Biases. InICLR Workshop on Spurious Correlation and Shortcut Lea...

    2025

  4. [2023]

    Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky

    URLhttps://arxiv.org/abs/2303.15389. Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution-Robust Large Mask Inpainting With Fourier Convolutions....

    arXiv

  5. [2025]

    Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz

    URLhttps://arxiv.org/abs/2508.11502. Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. InAdvances in Neur...

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.