{"id":"763331f0-5ee8-489c-8f56-f9e68f10c9a1","arxiv_id":"2505.03569","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Small or off-center objects make ImageNet-trained models depend more on background cues, and current debiasing methods do not fix this.","lead":"This paper builds a synthetic ImageNet variant in which objects are resized and moved to the corner, and shows that pretrained models increasingly rely on background cues exactly when the object is small and off-center. It then shows that existing spurious-correlation fixes like DFR and JTT do not recover the lost worst-group accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The CeR/CoR accuracy drops may reflect small-object composition difficulty rather than spurious background reliance; a predicted-label analysis on Hard-Spurious-ImageNet-10 would settle which mechanism holds.","rationale":"The reader's conditional verdict already identifies the proxy confound; this stress-test agrees and sharpens it. The most load-bearing condition for the abstract claim is that lower accuracy under background replacement measures learned reliance on class-correlated background cues. That condition is not established by group accuracies alone. The proposed predicted-label check is decisive: if predictions on CoR/CeR match the background class, the mechanism is confirmed; if not, the benchmark should be described as measuring context sensitivity or small-object difficulty rather than spurious feature reliance. The paper has merits: consistent trends across architectures, a controlled CeO/CeR comparison, released code, and validation of annotations. Those merits keep the verdict conditional; a failed label-tracking test would weaken the central claim but not the dataset's value as a robustness benchmark. Therefore no verdict change is needed.","tokens_in":17886,"tokens_out":6830,"duration_ms":74867,"concrete_test":"On Hard-Spurious-ImageNet-10, where each foreground class is paired with a fixed background class, take the CoR and CeR test images for each model in Table 8 and record the top-1 predicted class. Compute (i) accuracy, (ii) the fraction of all predictions equal to the paired background class, and (iii) among errors, the fraction equal to the paired background class. If, for small sizes (56x56 and 84x84), errors are dominated by the background class (e.g., more than 50% of errors), the drop is direct evidence of spurious background reliance. If errors spread across unrelated classes, the drop must be reinterpreted as generic small-object/composition difficulty. A useful secondary control is the same objects on a uniform gray background: if CeR/CoR accuracy is no better than on gray, the deficit is not specific to natural background cues.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (abstract; Section 5.7) is that low ROI-to-image ratio and off-center placement make models rely on spurious background features. The key evidence is the accuracy collapse in groups CeR and CoR relative to CeO and CoO (Tables 1 and 8). However, these group comparisons change the background and the image composition simultaneously. In CeR/CoR, a small object is pasted into an unrelated ImageNet scene; a model with no learned background correlations could show the same drop simply because small off-center objects are hard to recognize in novel contexts. In the non-10 dataset, random backgrounds are not class-correlated, so the test does not isolate 'spurious' background cues in the training-correlation sense. The paper validates bounding boxes (mIOU 0.875, Fig. 11), reports qualitative GradCAM examples (Fig. 8), and acknowledges testing only one corner location (Section 6), but it does not provide an aggregate check of whether errors on CeR/CoR actually track the background class. Without that check, the headline claim is an interpretation of accuracy gaps rather than a demonstrated mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how object size and position modulate spurious background reliance in ImageNet-trained classifiers. It introduces centeredness and size scores computed from ImageNet ground-truth bounding boxes, shows negative correlations between these scores and per-class accuracy on inpainted (object-removed) images, and constructs Hard-Spurious-ImageNet, a synthetic dataset that pastes ImageNet core objects at controlled sizes (56, 84, 112 px) and positions (center vs. top-right corner) against original or random inpainted backgrounds. It evaluates many pretrained models, plus ERM, DFR, and JTT training strategies, and finds that smaller, off-center objects yield the largest accuracy gaps and that existing debiasing methods provide only marginal gains on the hard groups. The authors also provide an aspect-ratio-preserving variant and a Hard-Spurious-ImageNet-10 subset with class-selected spurious backgrounds.","tokens_in":18084,"tokens_out":3793,"duration_ms":34623,"significance":"If the causal interpretation holds, this is a useful and timely benchmark contribution: it quantifies two understudied dataset biases and shows that standard group-robustness methods do not automatically address them. The paper has concrete strengths: the scores are defined directly from ground-truth boxes; the ablations (aspect-ratio preservation, SAM masks, Hard-Spurious-ImageNet-10, ERM variants) are sensible; the dataset and code are publicly released; and the evaluation spans many architectures and training recipes. The central claim, however, rests on interpreting accuracy gaps as evidence of spurious background reliance, and that inference needs a more direct test before the headline conclusion can be accepted.","major_comments":[{"comment":"The central claim that the CeR/CoR accuracy collapse is caused by spurious background reliance is not directly supported by the reported experiments. The CeR/CoR groups change the background and the object composition simultaneously: a small off-center object pasted into an unrelated scene is intrinsically harder to recognize, so a model with no learned background associations could show the same drop. The Appendix C example (Figure 10) notes that the Hard-Spurious-ImageNet-10 backgrounds contain cues that predict the background label, which is the right kind of evidence, but no aggregate predicted-label analysis is reported. I request a predicted-label confusion analysis on CeR/CoR groups (e.g., for Hard-Spurious-ImageNet-10, the fraction of errors whose predicted class equals the background class, and a comparison with a control where backgrounds are class-uncorrelated) to separate the spurious-reliance mechanism from generic difficulty of small off-center objects.","section":"Section 5.7, Table 6; Section 3.3"},{"comment":"The position-dependence claim rests on a single test location. The main test set places the core object only in the top-right corner, while the random-corner variant is introduced only together with aspect-ratio preservation, so position is confounded with resizing style. A position sweep (e.g., four corners and edge midpoints at a fixed object size) is needed to establish that 'far from the center' rather than 'in the top-right corner' is what drives the accuracy drop. The paper acknowledges this limitation in Section 6, but the limitation is load-bearing for the headline claim about off-center placement.","section":"Section 4, Table 1; Section 6"},{"comment":"The claim that current mitigation methods 'fail to achieve considerable performance gains' is based on single runs without error bars or significance tests. ERM and DFR differ by fractions of a percent in average accuracy (65.94 vs. 65.97), and the Hard-group difference (57.56 vs. 59.79) could easily be within seed noise. Please report means and standard deviations over multiple seeds and a paired comparison for the Hard group, or soften the conclusion to a qualitative statement. Without this, the secondary contribution about method failure is not yet established.","section":"Section 5.7, Table 6"},{"comment":"The spuriosity proxy used to motivate the dataset is per-class validation accuracy on inpainted ImageNet computed with a single model (ConvNext-Base), and the reported correlations with the center and size scores are weak (Spearman around -0.4). Class difficulty is an uncontrolled confound: harder classes may have low inpainted accuracy regardless of background dependence. Reporting the correlation averaged over several architectures and, where feasible, a difficulty-controlled partial correlation would substantially strengthen the motivating analysis.","section":"Section 3.3, Figure 4"}],"minor_comments":[{"comment":"The text says 'ERMeasy ... consisting of subgroups CeO and CoO for size 54×54 and 112×112'; the first size should presumably be 56×56.","section":"Section 5.8"},{"comment":"There is a typo: 'Hard-Spurios-ImageNet-10' should be 'Hard-Spurious-ImageNet-10'.","section":"Appendix A and Table 8"},{"comment":"The sentence 'a higher spurious feature reliance is observed in case of non-centered large object sizes' appears to contradict the negative correlation direction; the intended meaning is likely smaller and less centered objects.","section":"Section 3.3"},{"comment":"The interpretation that augmented training 'may end up highlighting' spurious features is speculative; a concrete analysis of where the augmented model makes errors on CoR (e.g., saliency or predicted-label statistics) would be needed to support it.","section":"Section 5.4"}],"recommendation":"major_revision","confidential_remarks":"This is a well-executed empirical study with a useful released benchmark. The core issue is that the headline causal claim about spurious background reliance is an interpretation of accuracy gaps, and the missing predicted-label analysis is within the scope of the manuscript to fix. I would be comfortable with acceptance after that evidence is added and the claims are scaled accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a genuinely useful empirical contribution: full ImageNet-1k coverage, controlled size/position/background manipulations, released dataset and code, and a wide model zoo. Second, the headline conclusion—that small, off-center objects make models lean harder on spurious background cues—is probably true, but the paper's own evidence does not isolate the \"spurious\" part as cleanly as the abstract implies.\n\nWhat's new: Hard-Spurious-ImageNet systematically varies object size (56/84/112), position (center vs corner), and background (original vs random) across all 1000 classes. The per-class center and size scores are simple, clearly defined, and correlate weakly but consistently with inpainted-image accuracy, a reasonable spuriosity proxy. The CeO/CeR comparison—same object size and position, only background swapped—is the right design, and the AR-preserving and SAM-mask ablations show the effect is not just a cropping artifact. The authors also verify annotations with Grounding DINO and give a responsible limitations section. The evaluation of DFR, JTT, and augmentation baselines is a fair attempt even if none of them fix the hard groups.\n\nThe soft spot is the inference from accuracy gaps to mechanism. The central claim is that models \"rely heavily on spurious features in the background\" when objects are small and off-center. What the experiments actually show is that accuracy drops when the background is swapped to a random one, and the drop is larger for small/off-center objects. That is consistent with spurious reliance, but also with a more mundane story: small objects are just harder to recognize in novel contexts, regardless of whether the background carries a class-correlated spurious cue. The CeR/CoR groups use random backgrounds, which are not class-correlated in the training sense, so the accuracy drop could reflect generic composition difficulty. The paper never checks, in aggregate, whether errors on CeR/CoR are background-class predictions; the evidence is a few qualitative GradCAM examples. A predicted-label analysis on Hard-Spurious-ImageNet-10 would settle it—if models output the background class on a substantial fraction of errors, the mechanism claim is confirmed. Without that, the strong reading of the abstract is an interpretation rather than a demonstrated result.\n\nOther issues are minor: only one corner location (acknowledged), no error bars or significance tests on method comparisons, and DFR/JTT hyperparameters are carried over from ImageNet-9 rather than tuned. None of this undermines the qualitative trend.\n\nWho this is for: people working on spurious correlations, robustness, or benchmark design. It deserves a serious referee; it is a solid contribution with a curable weakness in the interpretation. My recommendation: engage with it, cite the dataset, and treat the spurious-feature claim as conditional pending a mechanism check.","headline":"Useful benchmark and broad evaluation, but the central spurious-reliance claim outruns the evidence: the paper shows accuracy drops, not that errors track background class.","tokens_in":18647,"tokens_out":3243,"would_cite":true,"duration_ms":34043,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Small, off-center objects make ImageNet models trust backgrounds more.","keywords":["spurious correlations","background reliance","object size bias","object position bias","ImageNet-1k","group robustness","synthetic benchmark","composition shift"],"falsifier":"An experiment that would settle the claim: take the same core objects at 56×56 in the corner and place them on neutral, non-spurious backgrounds (e.g., flat gray or uniform noise) instead of natural inpainted scenes. If accuracy stays as low as on the random-background groups, the drop is caused by size and position alone rather than by background reliance; if accuracy recovers, the background is the decisive factor. A second check would compare inpainted accuracy against a control where the object is removed and the background is replaced by a class-independent texture, which would isolate inpainting artifacts from genuine background cues.","tokens_in":17655,"feed_emoji":"🎯","tokens_out":9338,"duration_ms":74913,"temperature":0.7,"pith_summary":"This paper argues that two mundane properties of a photograph—how large the object is and where it sits in the frame—determine how much a trained classifier leans on the background rather than the object. Using ImageNet-1k's bounding boxes, the authors measure a center and size bias in the dataset and show that classes with smaller, less-centered objects are more often predicted correctly even after the object is inpainted away. To make the effect testable, they build Hard-Spurious-ImageNet, a synthetic benchmark in which foreground objects are resized to 56×56, 84×84, or 112×112 pixels and placed in the center or a corner against their original or a random inpainted background. Across many pretrained architectures, accuracy collapses when the object is small and off-center against an unfamiliar background, and standard spurious-feature fixes—data augmentation, DFR, and JTT—do not close the gap. If the claim holds, object size and position are not nuisance factors but first-order modulators of spurious background reliance that evaluation and mitigation methods must explicitly control.","feed_headline":"Small, off-center objects push ImageNet models into background guessing","feed_subtitle":"New benchmark: accuracy collapses on small corner objects, and current fixes don't restore it.","key_machinery":"The load-bearing machinery is Hard-Spurious-ImageNet, a synthetic dataset built by cutting each ImageNet-1k ground-truth bounding box out of its image, inpainting the hole to produce a background-only image, then pasting the foreground back at controlled scales (56×56, 84×84, 112×112) and positions (center or corner) against either the original inpainted background or a random one. This yields four groups—CeO, CoO, CeR, and CoR—whose accuracy gaps isolate the contribution of size and position to background reliance. The paper also introduces two scalar class-level scores: a centeredness score based on the ℓ∞ distance between image and object centers, and a size score based on the fraction of image pixels covered by the object's bounding box; the negative correlation of these scores with inpainted-image accuracy is the paper's quantitative evidence that composition bias tracks spuriosity.","core_discovery":"The paper's central discovery is that spurious background reliance in ImageNet-trained models is systematically modulated by object size and position: the smaller the region-of-interest relative to the image and the farther the object is from the center, the more the model's prediction is carried by background cues. The evidence is the Hard-Spurious-ImageNet benchmark, where the hardest groups (small objects against random backgrounds in the corner) drop to single-digit or low-teens accuracy for most models, far below the same models' clean ImageNet accuracy, while the same object at 112×112 in the center keeps most of the performance. The authors further report that a low center/size score for a class correlates with higher accuracy on inpainted images (object removed), their proxy for learned background association. Finally, they show that neither data augmentation nor group-robustness retraining methods (DFR, JTT) recovers the hard-group accuracy, indicating that current mitigation approaches are not built for compositional shifts in size and location.","pith_inferences":["If the size-position mechanism generalizes beyond ImageNet, similar composition-controlled benchmarks in satellite imagery, surveillance, and medical imaging would likely expose the same background-reliance failure; this is a testable prediction, not a result of the paper.","The paper's proxy treats inpainted accuracy as spuriosity; a cleaner test would use human-verified 'object absent' pairs and measure whether model confidence tracks background class, which would separate genuine background association from inpainting artifacts.","The failure of DFR and JTT suggests that group definitions should be expanded from background type to the joint distribution of size, position, and background; adding size/location groups to existing robustness methods could be evaluated directly on the released dataset."],"forward_implications":["Worst-group evaluation of spurious-feature robustness should include size and position as group-defining attributes, not just background; benchmarks that omit them overstate worst-group performance.","Data-augmentation recipes and group-robust methods (DFR, JTT) need to be retested or redesigned for compositional shifts; the paper's numbers show these methods leave the hardest size/location groups largely unrepaired.","Model comparisons on standard centered, large-crop ImageNet evaluation are optimistic for real deployment, where objects are often small and peripheral; accuracy in the wild should be measured with composition-controlled subsets.","Because size had a stronger effect than position across architectures, scale-invariant or multi-scale training may be a more promising direction than translation-invariance fixes alone."],"supporting_citations":[{"why":"Supplies ImageNet-1k and its ground-truth object bounding boxes, from which the benchmark's core/background split is derived.","marker":"(Deng et al., 2009)"},{"why":"Defines the spurious-attribute group formalism and worst-group evaluation that Hard-Spurious-ImageNet adopts and extends with size and position.","marker":"(Sagawa et al., 2020)"},{"why":"Shows how to test background reliance by severing foreground from background; the paper generalizes this approach to all ImageNet classes with controlled size and location.","marker":"(Xiao et al., 2021)"},{"why":"Provides the Inpaint-Anything pipeline that removes objects so that background-only accuracy can serve as the spuriosity proxy.","marker":"(Yu et al., 2023)"},{"why":"LaMa fills the masked object regions, producing the inpainted backgrounds used in all four groups.","marker":"(Suvorov et al., 2022)"},{"why":"Segment Anything predicts object masks inside the bounding boxes, enabling the object-removal step in the dataset construction.","marker":"(Kirillov et al., 2023)"},{"why":"Supplies the Deep Feature Reweighting baseline whose hard-group performance is measured in the robustness experiments.","marker":"(Kirichenko et al., 2023)"},{"why":"Supplies the Just Train Twice baseline whose hard-group performance is measured in the robustness experiments.","marker":"(Liu et al., 2021)"},{"why":"Grounding DINO validates the ImageNet bounding boxes, supporting the claim that backgrounds are largely free of true objects.","marker":"(Liu et al., 2024)"}],"fun_headline_variants":["Small off-center objects flip ImageNet models to background guessing","Size and position of objects decide when ImageNet uses background","Hard-Spurious-ImageNet reveals background reliance for small corner objects","Current debiasing methods fail when object size and position shift"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The analysis assumes that a model's accuracy on inpainted images and the accuracy gaps between the four Hard-Spurious-ImageNet groups measure learned background associations; if inpainting artifacts, leftover object fragments, or the intrinsic difficulty of recognizing small off-center objects are what drive the numbers, the central claim weakens.","fun_headline_variants_meta":{"raw":{"variants":["Small off-center objects flip ImageNet models to background guessing","Size and position of objects decide when ImageNet uses background","Hard-Spurious-ImageNet reveals background reliance for small corner objects","Current debiasing methods fail when object size and position shift"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00086,"raw_usage":{"total_tokens":3744,"prompt_tokens":967,"completion_tokens":2777,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":2706}},"tokens_in":583,"tokens_out":2777,"duration_ms":20702,"temperature":1.0,"reasoning_tokens":2706,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:48:59.145403+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An experiment that would settle the claim: take the same core objects at 56×56 in the corner and place them on neutral, non-spurious backgrounds (e.g., flat gray or uniform noise) instead of natural inpainted scenes. If accuracy stays as low as on the random-background groups, the drop is caused by size and position alone rather than by background reliance; if accuracy recovers, the background is the decisive factor. A second check would compare inpainted accuracy against a control where the object is removed and the background is replaced by a class-independent texture, which would isolate inpainting artifacts from genuine background cues.","supporting_citations":[{"cited_title":"(2023) for optimizing the last layer for ImageNet-9 dataset (Xiao et al., 2021)","cited_arxiv_id":null,"evidence_quote":"Supplies the Deep Feature Reweighting baseline whose hard-group performance is measured in the robustness experiments."}],"review_version":1}