{"id":"d54db99c-2a81-49f6-a8c0-c053772fd28c","arxiv_id":"2506.03664","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Pre-trained ImageNet classifiers encode age more strongly than race or gender in their activations, but the evidence is limited because linear probes overfit and do not generalize to unseen faces.","lead":"This paper probes three pre-trained ImageNet image classifiers to see how much demographic information (age, race, gender) is embedded in their internal neural activations, using facial images with intersecting categories. It finds that age is the most separable attribute, but the linear classifiers it uses fail to generalize to held-out faces, so the headline result rests on training-set and visual-inspection evidence.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim rests on in-sample linear-probe accuracy that the paper itself shows does not generalize; no shuffled-label or held-out split control rules out memorization.","rationale":"The reader's REJECT verdict is well supported. The paper's central claim depends on a quantitative measure that, by the authors' own admission in Section 7, only separates the training data and fails on unseen data. A linear probe with millions of parameters trained on roughly 14.5k examples is expected to memorize even random labels, so high training accuracy is not evidence of a generalizable representation. The activation-map visualizations are suggestive but are averaged group patterns and can reflect correlated surface cues such as wrinkles, hairstyle, or background, exactly as the reader notes. I partially agree with the reader's weakest_assumption: the most decisive flaw is the missing generalization and control analysis, whereas the reader also emphasizes visual correlates. These are complementary, but the statistical gap is the more load-bearing issue because it directly invalidates the main quantitative support. The paper does contribute a reproducible methodology and an honest limitation statement, but the abstract and conclusion overstate the finding. A shuffled-label control and an official-split evaluation would settle whether any real age encoding exists beyond the training set. Until such evidence is supplied, the central claim should not be accepted as established.","tokens_in":16207,"tokens_out":5155,"duration_ms":53740,"concrete_test":"Retrain the linear probe with the identical architecture and data but with randomly permuted intersectional labels (a shuffled-label control), and also evaluate the unshuffled probe on FairFace's official validation split instead of the 10% random hold-out. If shuffled-label training accuracy matches the reported values, or if age-class validation accuracy on the official split is at chance, then in-sample separability cannot support 'strongly encode age information.'","verdict_should_be":"REJECT","load_bearing_attack":"The load-bearing step is the inference from high linear-probe accuracy to 'strongly encode age information' (Sections 6.4 and 7). The probes are trained on about 14.5k examples (roughly 115 per group times 126 groups) after downsampling activations to at most 8x8 spatial resolution, but the fully connected probe still has millions of parameters (e.g., InceptionV3 mixed7 has 768 channels, giving about 49k inputs times 126 outputs), so it can memorize the training partition. Figures 5, 7, and 9 show training accuracy near 1.0 while validation accuracy is far lower, and Section 7 explicitly concedes that the probes 'could distinguish training data, but did not generalize to unseen data.' The conclusion nevertheless claims that pre-trained ImageNet classifiers 'strongly encode age information in their representations.' No shuffled-label control is run, and no evaluation on FairFace's official held-out split is reported. Thus the observed in-sample separability is not established as encoding of age rather than an artifact of probe capacity and overfitting. This is an internal tension between the paper's own limitation statement and its central claim, not merely a disagreement with the field's consensus.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates intersectional bias in the internal representations of three pre-trained ImageNet CNNs (VGG16, ResNet50, InceptionV3) using the FairFace dataset. For each of 126 intersectional groups defined by race, age, and gender, the authors train linear classifier probes on layer activations and visualize average activations as topographic maps. The central claim is that these models, regardless of architecture, 'strongly encode age information' in their representations, with weaker encoding of ethnicity and of gender in middle-aged groups.","tokens_in":16425,"tokens_out":4308,"duration_ms":49813,"significance":"If the central claim were established, the paper would be a useful contribution to the transfer-learning fairness literature, because it would identify a generalizable demographic signal in widely used backbone representations. The paper also has concrete strengths: the code is publicly released, the evaluation uses an independent and relatively balanced face dataset, and the intersectional grouping across 126 categories is more granular than much prior work. However, the main evidence offered for the central claim is in-sample linear-probe accuracy, which the authors themselves report as failing to generalize, and the qualitative topographic-map analysis is not validated against a quantitative baseline. As a result, the significance of the paper is currently limited by the gap between its conclusions and the strength of the evidence.","major_comments":[{"comment":"The central claim that pre-trained ImageNet classifiers 'strongly encode age information' rests on training accuracy of linear probes, but the paper's own results show that validation accuracy is far lower and the authors state in Section 7 that the probes 'could distinguish training data, but did not generalize to unseen data.' In-sample separability by a high-capacity probe is not evidence of generalizable encoding; it is consistent with memorization or overfitting. No shuffled-label control or evaluation on FairFace's official held-out split is reported, so the abstract's conclusion is not supported by the probe experiments as presented.","section":"§6.4, §7, Figs. 5, 7, 9"},{"comment":"The argument that FairFace is an 'independent evaluation data set' because its labels are not ImageNet categories and its examples were not used in training does not rule out encoding of visual correlates of the labels. Age labels are strongly correlated with wrinkles, skin texture, hairstyle, facial shape, and other appearance cues; the paper itself says in Section 6.4 that age differences are 'driven by change of skin texture with age.' This undercuts the interpretation that the models encode 'age information' as a demographic concept rather than low-level visual attributes. The conclusion needs to either control for such correlates or be carefully reframed as a claim about visual appearance correlates.","section":"§5.2, §6.4"},{"comment":"The probe setup is not calibrated against a null model. After downsampling activations to at most 8x8 spatial resolution, InceptionV3's mixed7 layer has 768 channels, yielding about 49,152 input features to a fully connected layer with 126 outputs, trained on roughly 14.5k examples (90% of 126 batches of up to 128). With this parameter count and class count, high training accuracy can be achieved even for random labels. The paper reports no shuffled-label baseline, no repeated runs with different random splits, and no confidence intervals. Therefore the near-1.0 training accuracies in Figures 5, 7, and 9 do not by themselves demonstrate that the representations 'strongly encode' the intersectional groups.","section":"§5.3, Figs. 5, 7, 9"},{"comment":"The topographic-map interpretation relies on subjective visual comparison of average activation patterns, and the paper does not quantify the similarity or dissimilarity that drives the conclusions, such as 'activation differences between genders mostly emerge for middle ages' or 'age differences are most pronounced.' Without a quantitative similarity measure or a statistical test, these qualitative claims cannot be independently verified. They are used to reinforce the central claim and should either be backed by a quantitative analysis or be presented as hypotheses rather than findings.","section":"§5.4, §6.4"}],"minor_comments":[{"comment":"The phrase 'regardless of the architecture' is too strong given that only three CNN architectures, all trained on ImageNet with similar supervision, are examined; the paper itself lists Vision Transformers as future work.","section":"§1, §7"},{"comment":"The relationship between the 128-example batch per group used for probes and the 640-example cap in Figure 1b should be clarified; it is not clear whether the probe batches are drawn from the capped set or from the full set.","section":"§5.3"},{"comment":"The text says 'High training accuracy with low performance on the validation accuracy indicates overfitting,' but the paper then uses training accuracy to rank layers and identify biases; this inconsistency should be addressed by stating explicitly which quantity is intended to measure encoding.","section":"§6.2"},{"comment":"The statement that multi-layer non-linear classifiers 'would be more likely to learn generalized patterns from the activations' is speculative and not tested; it should be removed or supported by experiments.","section":"§6.4"},{"comment":"There are typographical errors, including 'missclassfications' and 'classfication' in the classification-errors paragraph.","section":"§6.3"}],"recommendation":"major_revision","confidential_remarks":"The reader's reject verdict is defensible: the paper contains an internal contradiction between the stated generalization failure of the probes and the strong conclusion about encoding age. I chose major_revision rather than reject because the load-bearing issues are addressable in principle: the authors could add shuffled-label controls, evaluate on a proper held-out split, and substantially soften or reframe the central claim to in-sample separability or to encoding of visual correlates. If the authors are unwilling or unable to provide such controls, the paper should not be accepted. I would also ask the editor to watch the heavy reliance on the authors' own NAP/topographic-map methodology (references [20]–[23]), which is used without external validation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is an incremental paper from a group that has been building the NAP/topographic-map toolset for a while. The new bit is applying it to intersections of race, age, and gender across three ImageNet CNNs, and adding linear probes. The empirical pattern — age differences dominate, race confusion mainly around Latino/Hispanic, gender effects in middle ages — is consistent across VGG16, ResNet50, and InceptionV3, and matches their earlier clustering study. The code is public and the methods are described clearly enough to reproduce.\n\nWhat I like: the authors are unusually candid about the probes not generalizing. Section 7 says it outright. The topographic maps are a nice descriptive device, and the qualitative reading of adjacent-age similarity lines up with the error tables.\n\nThe soft spot is exactly where the stress-test note lands. The abstract and conclusion claim that ImageNet classifiers “strongly encode age information,” but the linear probe evidence is in-sample: training accuracy near 1.0, validation accuracy far lower, and no shuffled-label control. With 126 output classes and a fully connected probe that can have millions of parameters, memorization is a real possibility. The validation accuracy is above chance and the trends are consistent, so the result is not empty, but it supports “some linearly decodable age signal in the training distribution” rather than “strongly encode” in a generalizable sense. The topographic maps are computed on the same training data and could reflect correlated visual cues (wrinkles, hairstyle) rather than demographic concepts. To be fair, the authors do flag that linear probes alone are insufficient, but then they use them as the basis for the headline claim anyway. That tension is the paper's main problem.\n\nIs it new? Only modestly. The age-dominance finding was already in their IJCNN clustering paper (ref 23). This adds multi-architecture and linear-probe evidence, which is useful confirmation but not a conceptual jump.\n\nBottom line: it's an honest, reproducible descriptive study useful to people working on bias in transfer learning, but the central claim needs to be dialed back to match the evidence. A serious referee could push them toward held-out evaluation or a more cautious interpretation. I'd send it to review, but I would not cite it as evidence of “strong encoding” without seeing a generalization check.","headline":"A transparent but overclaimed extension of the authors' own prior work: the age-differentiation result is plausible and consistent across architectures, but the headline 'strongly encode age' rests on in-sample probe accuracy the paper itself admits does not generalize.","tokens_in":16919,"tokens_out":3276,"would_cite":false,"duration_ms":36231,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pre-trained ImageNet classifiers, regardless of architecture, strongly encode age information in their representations of facial images, while race and gender signals are weaker.","keywords":["intersectional bias","pre-trained image classifiers","representation bias","linear classifier probes","Neuron Activation Profiles","topographic activation maps","FairFace","age bias"],"falsifier":"A decisive check would be to rerun the same probes on FairFace images whose demographic labels are kept but whose appearance correlates are removed or controlled, for example by normalizing skin texture to remove wrinkles or cropping out hair and clothing; if the age probe's accuracy collapses to chance while accuracy on the appearance controls stays high, the central claim is reduced to appearance encoding, and if the age accuracy survives such normalization, the claim is confirmed.","tokens_in":15998,"feed_emoji":"🧠","tokens_out":6098,"duration_ms":62142,"temperature":0.7,"pith_summary":"This paper asks whether commonly reused image-recognition backbones carry demographic bias inside their learned representations, not just in their final predictions. Using linear classifier probes and topographic visualizations of neuron activations on the FairFace face dataset, it finds that VGG16, ResNet50, and InceptionV3 all encode age strongly across their middle layers, so much that age groups are the easiest demographic division to recover linearly. Race is present but weaker, mostly in confusions among certain ethnicities, and gender is mostly separable only in middle-aged groups. The authors conclude that transfer-learning users inherit an age signal in the feature extractor regardless of architecture, and that linear probes alone underestimate how usable the bias is because non-linear classifiers would likely exploit it.","feed_headline":"ImageNet models encode age in face representations","feed_subtitle":"Age is the most separable demographic in their activations; race and gender signals are weaker.","key_machinery":"The argument is carried by two instruments: linear classifier probes, which train a linear readout on a layer's activations to predict the 126 intersectional groups and treat higher accuracy as evidence that the concept is encoded; and Neuron Activation Profiles (NAPs), which average activations per group, subtract the overall expected activation, and lay the resulting patterns out as topographic activation maps via a UMAP projection with a particle-repulsion layout. The probes give the quantitative separability curve across layers, while the maps give a visual counterpart in which similar group patterns predict which groups the model will confuse. A third, smaller mechanism is the error table analysis, which names the most frequent misclassifications and thereby locates the bias in particular intersections.","core_discovery":"The paper's central claim is that pre-trained ImageNet classifiers, regardless of architecture, strongly encode age information in their representations, with less pronounced encoding of race and gender. Across three architectures, central layers' activations allow a linear probe to separate the 126 intersectional groups in training data, but validation accuracy stays low, meaning the signal is real yet not general enough for a linear readout. Error analysis and topographic activation maps show age errors dominate and usually move to adjacent age groups; race errors concentrate on confusion with 'Latino_Hispanic', 'White', 'Indian', and 'Middle Eastern'; gender errors are rare and gender differences appear mainly for middle ages. The paper interprets this as evidence that the bias lives in the feature extractor itself and would be available to a non-linear classifier during transfer learning.","pith_inferences":["The authors leave open whether the effect is specific to ImageNet; a natural extension is to run the same probe and NAP pipeline on models trained on non-face and face-specific datasets, which would separate data-source effects from architecture effects.","A quantitative pairwise similarity index derived from the NAP maps could turn the visual confusions into a per-intersection audit metric, letting practitioners flag the most confusable demographic cells before deployment.","If the age signal is largely carried by skin-texture and face-shape cues, then the same measurement could be re-framed as a test of whether representations encode 'facial age appearance' rather than social age; the paper's method cannot distinguish these, and that distinction matters for deciding whether intervention is needed.","The failure of linear probes to generalize suggests that representation-bias audits should treat linear separability as a lower bound and include non-linear probes or few-shot readouts to estimate the bias a real downstream model would experience."],"forward_implications":["A downstream model built on any of these backbones can, in principle, read off age from the feature extractor, so age bias transfers even to tasks unrelated to age.","Because the age signal appears in all three architectures, switching backbone is not a remedy; mitigation would have to act on the representation itself.","Race bias shows up as systematic confusions among particular ethnicities, so auditing single-race accuracy is not enough.","Gender differences are concentrated in middle-aged groups, so gender-bias measurements that average across ages can hide the effect.","The gap between high training and low validation accuracy on the probes implies that a non-linear classifier would likely exploit the encoded bias more fully than a linear probe can."],"supporting_citations":[{"why":"Supplies the linear classifier probe method that quantifies how strongly a layer encodes the intersectional groups.","marker":"[2]"},{"why":"Supplies FairFace, the evaluation dataset whose age, race, and gender labels define the groups being probed.","marker":"[16]"},{"why":"Supplies ImageNet, the training data underlying all three pre-trained models under test.","marker":"[7]"},{"why":"Defines the VGG16 architecture that is one of the three backbones analyzed.","marker":"[34]"},{"why":"Defines the ResNet50 architecture that is one of the three backbones analyzed.","marker":"[13]"},{"why":"Defines the InceptionV3 architecture that is one of the three backbones analyzed.","marker":"[37]"},{"why":"Introduces Neuron Activation Profiles, the averaging-and-normalization technique used to build the topographic activation maps.","marker":"[20]"},{"why":"Prior work by the authors that introduced visualization of bias in activations as topographic maps; this paper extends it to intersectional groups.","marker":"[21]"},{"why":"A related clustering-based study whose conclusion that age is strongly encoded is cited as independent agreement.","marker":"[23]"}],"fun_headline_variants":["Age is the strongest hidden bias in ImageNet face encodings","ImageNet models leak age more than race or gender","Pre-trained models pick up age more than race or gender","Age dominates the hidden biases in ImageNet classifiers","ImageNet features betray age, not just race or gender"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument depends on treating FairFace's demographic labels as the thing the representations are encoding: if the models are actually encoding wrinkles, hairstyles, facial hair, or other appearance cues that merely line up with the labels, then the claimed 'age bias' is an observation about visual appearance rather than about a demographic concept.","fun_headline_variants_meta":{"raw":{"variants":["Age is the strongest hidden bias in ImageNet face encodings","ImageNet models leak age more than race or gender","Pre-trained models pick up age more than race or gender","Age dominates the hidden biases in ImageNet classifiers","ImageNet features betray age, not just race or gender"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1302,"prompt_tokens":781,"completion_tokens":521,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":397,"completion_tokens_details":{"reasoning_tokens":441}},"tokens_in":397,"tokens_out":521,"duration_ms":5643,"temperature":1.0,"reasoning_tokens":441,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:57:08.507237+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check would be to rerun the same probes on FairFace images whose demographic labels are kept but whose appearance correlates are removed or controlled, for example by normalizing skin texture to remove wrinkles or cropping out hair and clothing; if the age probe's accuracy collapses to chance while accuracy on the appearance controls stays high, the central claim is reduced to appearance encoding, and if the age accuracy survives such normalization, the claim is confirmed.","supporting_citations":[{"cited_title":"Alain and Y","cited_arxiv_id":null,"evidence_quote":"Supplies the linear classifier probe method that quantifies how strongly a layer encodes the intersectional groups."},{"cited_title":"Karkkainen and J","cited_arxiv_id":null,"evidence_quote":"Supplies FairFace, the evaluation dataset whose age, race, and gender labels define the groups being probed."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies ImageNet, the training data underlying all three pre-trained models under test."},{"cited_title":"Szegedy, V","cited_arxiv_id":null,"evidence_quote":"Defines the InceptionV3 architecture that is one of the three backbones analyzed."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces Neuron Activation Profiles, the averaging-and-normalization technique used to build the topographic activation maps."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior work by the authors that introduced visualization of bias in activations as topographic maps; this paper extends it to intersectional groups."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"A related clustering-based study whose conclusion that age is strongly encoded is cited as independent agreement."}],"review_version":1}