{"id":"adc5c562-357c-4ee0-8f70-eeafe411f1c8","arxiv_id":"2412.10597","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A new metric and texture-identification method show that ImageNet classifiers rely heavily on specific textures, but the claim that texture bias explains natural adversarial examples is largely a consequence of how textures are identified.","lead":"This paper introduces a metric and a texture-identification method to measure how much image classifiers rely on specific textures, then uses them to argue that texture bias explains why ImageNet-A images are confidently misclassified. It is a clear empirical study with a plausible new method, but the main explanatory claim rests on model-defined texture labels that may be circular.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ImageNet-A claim rests on a self-consistency loop: TID, dominant class textures, and the 'texture in the image' are all computed from the same model's softmax, so misclassified samples are nearly forced to mismatch their true label.","rationale":"The descriptive parts of the paper -- TAV construction, high-confidence texture classifications, and accuracy/confidence correlations across architectures -- are plausible and supported by multiple models and released code. The problem is specifically the causal claim in Section IV-D. I derived that the Fig. 12 statistics are almost tautological: since the same model produces (a) the TAV rows, (b) the TID label of each ImageNet-A image, and (c) the dominant textures of the prediction and label classes, a confidently misclassified sample will automatically have high prediction alignment and low label alignment. The human validation does not validate the ImageNet-A statistic itself. Therefore the central explanatory claim requires independent texture labels or a control group. This does not change the reader's conditional verdict: the paper is worth publishing as a method and descriptive study, but the causal interpretation should be revised or revalidated. The proposed test -- independent texture labels plus a misclassified-validation control -- would settle whether the concern lands.","tokens_in":29367,"tokens_out":5895,"duration_ms":54407,"concrete_test":"Recompute the Fig. 12 alignment analysis on ImageNet-A using an independent texture label source instead of TID. Concretely: (1) have human annotators (or a separate texture classifier trained on DTD/PTD with held-out validation) label the texture present in a random subset of ImageNet-A and in the ImageNet validation images; (2) define the dominant label/prediction textures from the independent labels; (3) measure the label-agreement and prediction-agreement rates. If the >90% label mismatch and high prediction match do not replicate with independent labels, the headline result is an artifact of the model-derived TID. A useful secondary control: run the same TID-based pipeline on ordinary misclassified ImageNet validation images; if they exhibit the same alignment pattern as ImageNet-A, the statistic does not identify anything specific to natural adversarial examples.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim -- that over 90% of ImageNet-A samples contain textures misaligned with their true label and that this explains confident mispredictions -- does not follow from the reported statistics because the statistics are constructed from the model's own outputs at every step. In Eq. 3, TID labels a real image by cosine similarity between softmax(f_theta(x)) and TAV rows; TAV rows themselves are built from the same model's softmax predictions on PTD textures (Eq. 2). The 'dominant texture for an object class' is then the TID mode over ImageNet validation images (Figs. 8-10), again using the same model. An ImageNet-A image that is confidently predicted as class j therefore has a softmax vector close to those of validation images predicted as j, so its TID label will tend to coincide with the dominant prediction texture; because it is misclassified, its softmax is not close to validation images of the true label, so its TID label will tend not to match the label texture. The >90% label mismatch and high prediction match in Fig. 12 are thus largely a formal consequence of how the variables are defined, and do not require any actual texture-content mismatch. The human evaluation (61% overall, with classes near chance) does not break this loop, because it was run on ImageNet validation images, not ImageNet-A, and did not independently verify the specific label/prediction alignment statistic. Without independent texture labels or a control set, the paper's explanatory conclusion conflates 'texture identified by the model's output geometry' with 'texture present in the image.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Texture Association Value (TAV), a metric computed from a model's softmax predictions on the Prompted Textures Dataset, and the Texture Identification (TID) procedure, which assigns a texture class to a real image by cosine similarity between the model's softmax output and TAV rows. The authors use TID to analyze texture bias on ImageNet validation images and to study ImageNet-A natural adversarial examples. They report that model accuracy and confidence on ImageNet validation depend strongly on whether the TID-identified texture is the dominant texture for the object class, and they claim that over 90% of ImageNet-A samples contain textures misaligned with their true label but aligned with the prediction, concluding that texture bias explains confident mispredictions. Experiments are run on eight torchvision models, with the main results reported for ResNet50.","tokens_in":29626,"tokens_out":3722,"duration_ms":35305,"significance":"If the central interpretive claims were supported, the paper would be a valuable extension of texture-bias research from synthetic cue-conflict data to real-world images, and it would offer a potential explanation for a robustness phenomenon. The paper has concrete strengths: the TAV/TID framework is clearly specified, the code is released, the evaluation spans several architectures, and the authors undertake a human validation study of TID. However, the headline result is that texture misalignment explains the existence of natural adversarial examples, and that claim is not supported by the current experimental design. The analysis depends on model-derived texture labels, and the human validation is too weak and too indirect to break this dependency. The paper is worth revising, but the causal and explanatory framing must be substantially reworked or supplemented with independent evidence.","major_comments":[{"comment":"The central claim that over 90% of ImageNet-A samples contain textures misaligned with the true label is largely a formal consequence of how TID, prediction textures, and label textures are constructed. TID labels an image by maximizing cosine similarity between the model's softmax output and TAV rows, and TAV rows are themselves built from the same model's softmax predictions (Eq. 2). The prediction texture for a class is the TID mode over validation images predicted as that class (Figure 10). An ImageNet-A image that is confidently predicted as class j therefore has a softmax vector similar to validation images predicted as j, so its TID label will tend to match the prediction texture and not the label texture. The human evaluation in Section III-B1 does not break this loop, because it was run on ImageNet validation images and did not independently verify the label/prediction alignment statistic. The explanatory conclusion 'Natural adversarial examples are a consequence of texture bias' (Section IV-D takeaway) and the Abstract's phrasing 'texture bias explains the existence' are therefore not supported by the statistics as reported. Independent texture annotations for ImageNet-A or a matched control group of correctly classified images is needed.","section":"Section III-B, Eq. (3); Section IV-D, Figures 12–13"},{"comment":"The human validation of TID reaches only 61% overall agreement, and the agreement varies widely by texture class, with classes near chance (the text reports 'cobwebbed' at 13%, and Figure 4 shows several classes below 40%). This level of accuracy is insufficient to establish that TID reliably identifies textures in the ImageNet-A analysis, especially because the alignment statistic depends on correct texture labels for individual images. The paper should report per-class agreement for all 56 texture classes with sample counts, and it should test whether the ImageNet-A alignment result in Figure 12 is robust when only high-agreement texture classes are used.","section":"Section III-B1, Figure 4"},{"comment":"The causal claim that texture misalignment explains confident mispredictions requires a control group, but the paper analyzes only ImageNet-A misclassified images. Without showing that correctly classified images overwhelmingly contain their label's dominant texture, and that the label-texture mismatch rate is significantly lower for correct predictions than for ImageNet-A mispredictions, the reported rates cannot distinguish a texture-misalignment cause from a generic property of the TID assignment. The paper also states that texture misalignment is a 'sufficient condition' (Section III-B and IV-D) for confident misprediction, but it never provides evidence of sufficiency, such as an intervention that changes a predicted class by changing texture while holding shape constant.","section":"Section IV-D, Figures 12–13"},{"comment":"The TID forces each image to a single texture class via argmax over 56 classes, but the paper acknowledges in Section III-B1 that natural images often contain multiple overlapping textures. This assumption could systematically bias the alignment statistic: an image whose model response is driven by a background texture will be assigned that texture even if the foreground object exhibits a different one. The authors should quantify the sensitivity of the ImageNet-A alignment results to this assumption, for example by thresholding the cosine similarity or reporting the full similarity distribution rather than only the argmax.","section":"Section III-B, Eq. (3)"}],"minor_comments":[{"comment":"The abstract contains a formatting artifact ('Texture Association V alue') that should be corrected.","section":"Abstract"},{"comment":"There is a duplicated word in the text: 'shows shows' appears in the discussion of Figure 8.","section":"Section IV-C"},{"comment":"The phrase 'the magitude of the similarity' contains a typo; it should be 'magnitude'.","section":"Section IV-D"},{"comment":"The text reports 'over 90% of the samples contain textures that disagree with the texture associated with their true label' and points to the blue and orange bars, but the decomposition of the four categories into the >90% figure is not stated explicitly. The authors should define the exact union of categories used in the claim.","section":"Section IV-D, Figure 12"},{"comment":"Accuracy and confidence differences between dominant and non-dominant textures are reported as point estimates without error bars or statistical significance tests; given the grouped structure of the data (images nested in labels), a mixed-effects or per-label paired analysis would strengthen these claims.","section":"Section IV-C, Figures 9 and 11"},{"comment":"Figure 13 reports per-class agreement rates but many classes contain very few ImageNet-A samples (the paper mentions a class with a single sample). Displaying sample counts or confidence intervals per class would prevent overinterpretation of individual bars.","section":"Section IV-D, Figure 13"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a revised arXiv preprint (v2). The main reason for major revision is the circularity in the ImageNet-A analysis: since TID, prediction textures, and label textures are all computed from the same model's softmax, the >90% label-mismatch statistic is partly an artifact of the construction. The human validation at 61% agreement, with some classes near chance, does not provide the needed independent grounding. I would advise the editor that the paper cannot be accepted in its present form, but the underlying idea (using model responses to texture data to probe real-world classification behavior) is worth pursuing if the authors add a control group, independent texture annotations on ImageNet-A, and a more cautious causal framing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a useful methods paper wearing an overconfident explanation on its sleeve. The TAV/TID pipeline is new, and the descriptive findings are solid; the claim that texture bias *explains* ImageNet-A is not supported by the reported stats, because the stats are largely a self-consistency loop.\n\nWhat's genuinely new: TAV extends the authors' earlier texture-object associations with entropy weighting and scales to PTD; TID is a model-output-based texture probe applied to real images; and the observation that models confidently classify isolated textures (over 25k PTD images at >96% confidence) is a clean, reproducible result. The accuracy/confidence differences between dominant and non-dominant texture groups across eight architectures are descriptive but real, and the correlation tables are consistent.\n\nThe soft spot is the leap from 'texture presence correlates with accuracy/confidence' to 'texture misalignment causes ImageNet-A failures.' The stress-test note is right: TID labels an image by cosine similarity to TAV rows, dominant textures are computed from those same TID labels on validation data, and the ImageNet-A 'prediction texture' is the dominant texture for the predicted class. So an ImageNet-A image that is confidently predicted as class j will, by construction, tend to have a TID label matching the dominant prediction texture and not the label texture. The >90% label mismatch in Figure 12 is largely a formal consequence of the definitions. The human evaluation (61% agreement, some classes at chance) was run on ImageNet validation, not ImageNet-A, and doesn't independently verify the label/prediction alignment statistic. There's also no control group of correctly classified images. The paper itself acknowledges the single-dominant-texture assumption is often false, which further weakens the causal reading.\n\nThat said, I don't think this is fatal to the descriptive contribution. The TAV/TID method is worth publishing, and the ImageNet-A observation could be reframed as a hypothesis rather than a conclusion. The authors release code and data. For peer review, I'd send it out, but I'd ask for (a) independent texture annotations on ImageNet-A, (b) a control group of correctly classified images, and (c) a caveat about the circularity in the causal language. If the authors redo the analysis with an independent texture classifier or human labels, the natural-adversarial-example claim could become interesting rather than tautological.","headline":"Useful TAV/TID methods paper whose central ImageNet-A explanation is a self-consistency loop; publish the method, not the causal claim.","tokens_in":30216,"tokens_out":1867,"would_cite":true,"duration_ms":16904,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Texture bias—models' reliance on repeated surface patterns rather than object shape—explains the confident mispredictions in ImageNet-A, where over 90% of these natural adversarial examples contain textures misaligned with their true label.","keywords":["texture bias","natural adversarial examples","Texture Association Value","texture identification","ImageNet-A","model robustness","model confidence","computer vision"],"falsifier":"A concrete test: take ImageNet-A images that the paper labels as misaligned (texture agrees with the prediction, disagrees with the label) and apply a style-transfer or texture-replacement method that changes the image's texture to the dominant texture of its true label while preserving the object's shape. If texture bias is the cause, these images should flip to correct predictions with high confidence. If a large fraction remain confidently misclassified after the texture is aligned, the texture-misalignment explanation is insufficient.","tokens_in":29094,"feed_emoji":"🧵","tokens_out":8297,"duration_ms":62045,"temperature":0.7,"pith_summary":"This paper sets out to show that texture bias—the tendency of image classifiers to rely on repeated surface patterns rather than object shape—is a dominant force in real-world behavior, and that it explains why natural adversarial examples exist. Natural adversarial examples are ordinary photographs that models confidently misclassify. The authors introduce the Texture Association Value (TAV), a score for how strongly a model links each of 56 texture classes to each of 1,000 object classes, and the Texture Identification (TID) method, which tags a real image with the texture whose expected model response it most resembles. They find that models confidently classify isolated textures, that accuracy and confidence on ImageNet validation images are dramatically higher when the image contains the dominant texture of the true label or predicted class, and that over 90% of ImageNet-A samples contain a texture that disagrees with the dominant texture of their true label. If correct, the work means texture misalignment is a sufficient cause of confident wrong predictions on real data, redirecting robustness research toward texture rather than only toward adversarial perturbations.","feed_headline":"Texture bias explains 90% of natural adversarial examples","feed_subtitle":"A new metric shows models classify by texture, and misaligned textures drive confident wrong predictions.","key_machinery":"The central object is the Texture Association Value (TAV), an $n \\times m$ matrix where $n=56$ texture classes and $m=1000$ ImageNet object classes, computed from model argmax predictions on the Prompted Textures Dataset. Each entry is the product of four factors: the probability that a texture is predicted as an object class, the complement of the entropy of that texture's prediction distribution, the probability that predictions of an object class come from that texture, and the complement of the entropy of that object class's texture distribution. The companion Texture Identification (TID) assigns a texture to a real image by cosine similarity between the image's softmax output and the TAV rows. This machinery lets the authors detect which texture a model 'sees' in a real image without additional training, and then relate that texture to the model's accuracy and confidence.","core_discovery":"The central discovery is that the confident, incorrect predictions collected in ImageNet-A are explained by texture bias. The paper argues that models learn strong associations between specific textures and object classes, that these associations drive predictions on real images, and that natural adversarial examples are precisely the cases where the image's dominant texture agrees with the predicted (wrong) class but disagrees with the true label. The authors demonstrate this by showing that models make confident predictions on isolated texture images, that accuracy and confidence on the ImageNet validation set are substantially higher when the dominant texture of the true label (or predicted class) is present, and that more than 90% of ImageNet-A samples have textures misaligned with their true label's dominant texture. They conclude that texture bias is a sufficient condition for the existence of natural adversarial examples.","pith_inferences":["Beyond the paper: the same TAV/TID machinery could be applied to other visual dimensions such as color, lighting, or background scene by substituting an appropriate dataset, as the authors themselves note for color.","The 61% human agreement rate on TID suggests texture identification is inherently ambiguous; a stricter test of the causal claim would manipulate the texture in an image while keeping shape fixed and show predictions flip accordingly.","If texture misalignment is the mechanism, then adversarial perturbations that inject the dominant texture of a wrong class should be especially effective, connecting texture bias to classical adversarial attacks in a testable way.","The paper's single-dominant-texture assumption may cause TID to mislabel images with mixed textures; a multi-label extension of TID could strengthen or qualify the ImageNet-A result."],"forward_implications":["If a model's accuracy on an object class depends on whether that class's dominant texture is present, then evaluation sets that happen to lack that texture will systematically understate the model's real-world accuracy.","Natural adversarial examples could be curated or generated by selecting images whose dominant texture matches a wrong class and mismatches the true class, giving a principled way to build robustness test sets.","Mitigation strategies for texture bias should target the learned texture-object associations themselves, rather than only adding shape cues during training.","The TAV metric can be used as a pre-deployment audit tool: list the textures each class relies on and flag classes where the dominant texture is an unexpected shortcut such as a grid for window screens.","Confident mispredictions on ordinary photographs may not require an adversary; many may be ordinary texture-driven errors."],"supporting_citations":[{"why":"Introduces the texture-shape cue conflict dataset and the finding that ImageNet-trained CNNs are biased toward texture; this is the phenomenon the paper extends to real data.","marker":"[1]"},{"why":"BagNet shows local patches containing texture-like information suffice for high-accuracy ImageNet classification, supporting the claim that texture alone can drive predictions.","marker":"[3]"},{"why":"Provides the ImageNet-A dataset of natural adversarial examples that the paper explains through texture misalignment.","marker":"[6]"},{"why":"Supplies the Prompted Textures Dataset, the 56-class texture corpus used to compute the TAV matrix.","marker":"[7]"},{"why":"Introduces texture-object associations from model predictions on texture data, the basis of the TAV formulation.","marker":"[8]"},{"why":"Provides ImageNet, the training and validation benchmark underlying all models and the ImageNet validation set used for accuracy and confidence analyses.","marker":"[19]"},{"why":"Defines the ResNet50 architecture that is the primary model for the reported experiments.","marker":"[21]"}],"fun_headline_variants":["Texture bias drives 90% of natural adversarial examples","Why AI fails on real images: texture bias","New metric ties texture bias to 90% of adversarial examples","Texture bias explains confident failures on natural images","90% of natural adversarial examples are texture-bias errors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Texture Identification (TID) step correctly labels the texture present in a real image, because the entire ImageNet-A analysis compares the texture the model responds to against the texture dominant for each class; if TID is wrong about what texture is present, the claim that over 90% of these images contain a misaligned texture collapses. The paper's own human validation reaches only 61% agreement, with some texture classes near chance levels, so this premise is weakly supported.","fun_headline_variants_meta":{"raw":{"variants":["Texture bias drives 90% of natural adversarial examples","Why AI fails on real images: texture bias","New metric ties texture bias to 90% of adversarial examples","Texture bias explains confident failures on natural images","90% of natural adversarial examples are texture-bias errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000352,"raw_usage":{"total_tokens":1865,"prompt_tokens":836,"completion_tokens":1029,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":452,"completion_tokens_details":{"reasoning_tokens":963}},"tokens_in":452,"tokens_out":1029,"duration_ms":7390,"temperature":1.0,"reasoning_tokens":963,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:48:28.632923+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: take ImageNet-A images that the paper labels as misaligned (texture agrees with the prediction, disagrees with the label) and apply a style-transfer or texture-replacement method that changes the image's texture to the dominant texture of its true label while preserving the object's shape. If texture bias is the cause, these images should flip to correct predictions with high confidence. If a large fraction remain confidently misclassified after the texture is aligned, the texture-misalignment explanation is insufficient.","supporting_citations":[{"cited_title":"ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness,","cited_arxiv_id":null,"evidence_quote":"Introduces the texture-shape cue conflict dataset and the finding that ImageNet-trained CNNs are biased toward texture; this is the phenomenon the paper extends to real data."},{"cited_title":"On Synthetic Texture Datasets: Challenges, Creation, and Curation","cited_arxiv_id":"2409.10297","evidence_quote":"Supplies the Prompted Textures Dataset, the 56-class texture corpus used to compute the TAV matrix."},{"cited_title":"Explorations in Texture Learning","cited_arxiv_id":"2403.09543","evidence_quote":"Introduces texture-object associations from model predictions on texture data, the basis of the TAV formulation."}],"review_version":1}