{"id":"e743cd45-cf2c-4156-84a1-ef02f95ace7f","arxiv_id":"2505.13281","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Transformers outperformed young children and matched their error profile on a geometry odd-one-out task, while vision-language models underperformed vision-only models.","lead":"The paper tests eight image recognition models on 43 geometry and topology puzzles and compares their performance with human children and adults. Transformer models scored highest, beating young children, while vision-language models scored lowest, suggesting that adding language training can hurt abstract geometric sensitivity.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline result rests on an unvalidated cosine-similarity rule for picking the odd-one-out; a re-scoring under alternative rules could overturn DINOv2's margin over children and the r>0.90 alignment.","rationale":"I considered four candidate concerns: the unvalidated cosine-similarity operationalization, the lack of a significance test for the transformer-versus-children gap, the fragility of correlations computed on only seven class means with unequal concept counts, and the confound between model architecture and training data size in the VLM comparison. The most load-bearing is the operationalization concern, because every quantified result in the paper depends on interpreting the smallest-average-cosine image as the model's odd-one-out choice. The paper follows the earlier CNN study by Upadhyay et al., but precedent does not establish validity for the specific human-comparison claim made here. The descriptive results are transparent and the stimuli are public, so the missing piece is a robustness check against an alternative scoring rule or against human item-level choices. The reader's CONDITIONAL verdict is appropriate and already captures this uncertainty; I do not recommend changing it. If the alternative-rule check shows that DINOv2's margin and correlations are stable, the paper's central claim would be substantially strengthened.","tokens_in":11773,"tokens_out":8673,"duration_ms":91193,"concrete_test":"Re-score all eight models on the full 43-stimulus battery using at least one independent decision rule that does not rely on the minimum-mean-cosine heuristic, for example the image whose representation is farthest from the centroid of the other five, or a linear probe trained to recognize GT concepts on held-out stimuli. Then compare DINOv2's accuracy against Izard and Spelke children and recompute the Figure 4 correlations. If DINOv2's roughly 11-point margin over children and the r>0.90 correlations do not survive the alternative rule, the headline result is an artifact of the chosen similarity heuristic. As a direct validation, if per-stimulus human odd-one-out choices are available, compute Cohen's kappa between the cosine-rule choice and the modal human choice for each stimulus; chance-level kappa would invalidate the proxy immediately.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's load-bearing empirical claim is that DINOv2 and ViT surpass the 37.72% accuracy of Izard and Spelke's 3-6-year-old children and align with children's class-difficulty profile (r>0.90 in Figure 4). The entire model evaluation in the section 'Evaluation on Neural Models' is determined by one fixed heuristic: represent each image by the final hidden layer, compute pairwise cosine similarities, average them per image, and label the lowest-average image as the odd-one-out. This heuristic is never validated on these 43 stimuli. Because the stimulus set deliberately varies low-level visual features across the six images, the lowest-similarity image may be selected for reasons unrelated to the intended geometric or topological concept, such as a color, texture, or orientation outlier. If the cosine rule does not track the conceptual odd-one-out, then the reported model accuracies are not measures of GT-concept sensitivity, and both the child-surpassing comparison and the alignment correlations lose their interpretation. This is the least secure condition for the central claim. Secondary issues such as the absence of a significance test for the model-child gap and the small number of class means entering the correlation are correctable; the validity of the decision rule is foundational.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates eight pretrained computer vision models (three CNNs, two transformers, three VLMs) on the 43-concept odd-one-out battery of Dehaene et al. (2006), using a deterministic decision rule: the image with the lowest average pairwise cosine similarity in the final hidden layer is chosen as the odd-one-out. It compares model accuracy and class-level accuracy profiles (seven GT classes) against published human data from Dehaene et al. (2006), Izard and Spelke (2009) children and adults. The main empirical claims are that DINOv2 (48.89%) and ViT (46.67%) surpass Izard-Spelke children (37.72%), that their class-accuracy profiles correlate with children's at r > 0.90, and that VLMs underperform both CNNs and transformers. These findings are interpreted as evidence that GT concepts can be learned from visual experience, contrary to the core-knowledge account, and that current text-image alignment harms abstract geometric sensitivity.","tokens_in":11978,"tokens_out":8836,"duration_ms":82892,"significance":"If the decision rule validly operationalizes human odd-one-out judgments, the results are theoretically significant: they provide a sufficiency argument that abstract geometric and topological sensitivity can arise from large-scale visual pretraining, without innate dedicated GT circuitry, and they extend prior CNN-only findings to transformer and VLM classes. The paper's strengths are its use of external human benchmarks, publicly available models, and a fully specified parameter-free evaluation protocol that is easy to reproduce. The strongest evidence is the transformer-children class-profile alignment, but its statistical basis is thin (seven class means, one with three concepts), and the VLM/architecture comparisons are confounded. The central claims therefore require additional validation and re-analysis rather than being unfixable.","major_comments":[{"comment":"The odd-one-out decision rule is never validated against human perceptual judgments on these stimuli. The rule selects the image with the lowest average pairwise cosine similarity in the final hidden layer; because the stimulus set deliberately varies low-level visual features across the six images, the selected image could be an outlier in color, texture, or orientation rather than in the intended geometric/topological concept. Since the overall accuracy comparison in Figure 2 and the class-profile alignment in Figure 4 are both computed from this rule, its construct validity is load-bearing for the paper's main claims. Please provide stimulus-level validation of the rule (e.g., human similarity ratings for the six-image sets, alternative similarity metrics, or an analysis of which images are selected and why), and report how the headline results change with the choice of representation layer or pooling.","section":"Evaluation on Neural Models"},{"comment":"The claim that transformers surpass young children is not significance-tested. DINOv2's 48.89% is 21/43 correct and ViT's 46.67% is 20/43; the Izard-Spelke children's 37.72% is an aggregate of participant-level accuracy. With 43 binary trials per model, the standard error of a model proportion near 0.5 is roughly 7.6 percentage points, so the reported gap of about 11 points between DINOv2 and the children is well within the range that could arise by chance when the variance of the child sample is also considered. Please report a statistical test that accounts for item-level and participant-level variation (e.g., bootstrap over stimuli and children, or a mixed-effects model), and adjust the abstract's 'surpassing' wording accordingly.","section":"Results, Overall sensitivity (Figure 2)"},{"comment":"The alignment correlations are computed on only seven class means, and one class (Symmetrical Figures) contains just three stimuli, a limitation the manuscript itself notes in 'Sensitivity by class.' With n=7, the 95% confidence interval for r=0.91 is wide, and a single noisy class mean can move the correlation substantially. The 'strong alignment' claim should be supported by per-concept correlations or by bootstrap/jackknife intervals over the 43 stimuli, and ideally by a weighting of classes by their item counts.","section":"Results, Alignment to human profiles (Figure 4)"},{"comment":"The attribution of model-class differences to architecture or to language supervision is confounded. The VLMs differ from vision-only models in architecture, parameter count, training dataset size, and supervised vs self-supervised objective, in addition to the contrastive text-image loss; the transformers differ from CNNs in architecture, dataset scale, and label/no-label training. The Discussion's claim that 'modality alignment with text reduces the sensitivity to GT concepts' is therefore not supported by the present design. Please either run controlled comparisons (e.g., a ViT with and without language supervision, trained on comparable data) or re-frame these conclusions explicitly as hypotheses.","section":"Table 1 and Discussion"},{"comment":"The inferential leap from 'trained on large image datasets' to 'learned for free through everyday interaction' is not justified. The CNNs were trained with ImageNet-1k human class labels, which are a form of explicit supervision rather than unsupervised everyday experience; even the self-supervised DINOv2 was trained on LVD-142M, a curated dataset of internet images with statistics very different from a child's visual environment. Because the paper's title and abstract frame the result as evidence against the core-knowledge account, the authors should state explicitly which training regimes they consider proxies for 'free' learning, or restrict the conclusion to 'can arise from large-scale visual pretraining.'","section":"Discussion, 'Transformer models show higher sensitivity to GT concepts'"}],"minor_comments":[{"comment":"Typo: 'computer visions models' should be 'computer vision models.'","section":"Abstract"},{"comment":"Typo: 'for two reason' should be 'for two reasons.'","section":"Discussion, 'Transformers show higher sensitivity to GT concepts'"},{"comment":"Typo: 'the Transformer models and and the young children' contains a duplicated 'and.'","section":"Discussion, 'From cognitive alignment to developmental alignment'"},{"comment":"In the text, 'CLIP (RN5-50)' should be 'CLIP (RN-50).'","section":"Results, 'Alignment to human profiles'"},{"comment":"The description of the final hidden layer is not architecture-specific enough for reproduction: for ViT/DINOv2, is the representation the [CLS] token, a mean over patch tokens, or the pre-logits pooled feature? For CNNs, is it the global average pooled feature map? Please specify the exact layer and pooling for each of the eight models.","section":"Evaluation on Neural Models"},{"comment":"The significance asterisks in the heatmap are uncorrected across 24 correlations (8 models x 3 human datasets); please report adjusted p-values or note that the pairwise p-values are uncorrected.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"No additional concerns beyond the report; the issues are empirical and addressable by additional analyses."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the quick take. This is a clearly written extension of the Upadhyay et al. CNN benchmark to transformers and vision-language models on the full 43-item Dehaene odd-one-out battery. The new empirical results are worth having: DINOv2 and ViT beat the reported accuracy of 3–6-year-old children overall, and the VLM family clearly underperforms vision-only models. If those rankings hold up, they're a useful input to the learned-not-innate debate and a reason to treat pretrained transformers as tractable cognitive models.\n\nCredit where it's due: the paper is honest about prior work, uses publicly released models, and reports all 43-item results. The by-class breakdown and the correlation heatmap are informative. There are no fitted parameters; this is measurement against published human data.\n\nSoft spots, in order of importance. First, the headline result—transformers surpassing children—rests on model accuracies in 43 binary items, with no significance test or confidence interval on the difference. 48.9% vs 37.7% is a real-looking gap, but it's about five items out of 43, and the human data come from a different study with different procedures. A permutation or bootstrap test over items would be straightforward and should be added. Second, the alignment correlations are computed on 7 class means, one with only 3 concepts. That's coarse; r=0.93 looks impressive but the effective sample is tiny. The authors flag this, but it should temper the 'strong alignment' language. Third, the VLM comparison is confounded with model size, training data, and objective. The post hoc 'text hard to describe' explanation is plausible but not tested, e.g., by comparing VLM performance on easy-to-describe vs hard-to-describe concepts within the battery. Fourth, and this is the one that matters most, the model evaluation uses a single decision rule—lowest average cosine similarity in the final hidden layer—and the rule is never validated against human odd-one-out choices on these stimuli. That's not a red flag by itself, since the same rule was used in the precursor CNN paper, but it is a load-bearing assumption.\n\nThat last point doesn't sink the paper. If the cosine rule were wildly off, you'd be unlikely to see class-level correlations with children above 0.9. But a robustness check with a few alternative similarity measures would strengthen the claims considerably.\n\nBottom line: this is a solid, incremental benchmark paper for cognitive science and AI evaluation. It deserves peer review. I'd send it out, with a request for the significance test, a robustness check on the decision rule, and toned-down causal language about multimodality. I'd bring it to a reading group if people care about whether pretrained models are human-like in geometric abstraction.","headline":"A useful benchmark extension—transformers beat kids, VLMs flop—but the headline gap needs a significance test and the cosine-similarity decision rule deserves a robustness check.","tokens_in":12543,"tokens_out":3362,"would_cite":true,"duration_ms":33649,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Transformer-based computer vision models outperform 3- to 6-year-old children on geometric and topological odd-one-out tasks while closely matching the children's pattern of easy and hard concepts.","keywords":["geometric reasoning","topological concepts","odd-one-out task","cognitive alignment","vision transformers","vision-language models","core knowledge","learning for free"],"falsifier":"Run a control study on the same 43 stimuli in which the cosine-similarity rule is replaced by a linear probe trained on human odd-one-out choices for half the concepts and tested on the remaining half; if the transformer accuracy advantage over children disappears, or if the correlation with human profiles is recomputed at the level of the 43 individual concepts and drops to near zero, the central claim is falsified.","tokens_in":11494,"feed_emoji":"🧩","tokens_out":9338,"duration_ms":80447,"temperature":0.7,"pith_summary":"The paper asks whether sensitivity to geometric and topological concepts — shape, angle, rotation, symmetry, and the like — must be innate, as the core-knowledge account claims, or can be learned 'for free' from ordinary visual experience. It answers by running eight pretrained computer vision models on the 43-concept odd-one-out test originally designed for the Mundurucu and later used with Western children and adults. Transformer-based models (ViT, DINOv2) achieve the highest accuracy, roughly 47–49 percent, which exceeds the 37.7 percent of 3- to 6-year-old children, while their pattern of which concept classes are easy versus hard correlates with children's at r>0.90. Vision-language models such as CLIP and ALIGN perform worst, even though they are larger and trained on more data, suggesting that naive image-text alignment can damage abstract geometric sensitivity. The authors take this as evidence that large-scale visual pretraining is sufficient to produce human-like geometric and topological sensitivity, undermining the necessity of innate geometric core knowledge.","feed_headline":"DINOv2 vision model beats 3-6 year olds at geometry odd-one-out","feed_subtitle":"Transformer scores 48.9% on 43 geometry concepts versus children's 37.7%, with matching difficulty profile.","key_machinery":"The evaluating device is a zero-shot odd-one-out probe on the penultimate layer: for each six-image stimulus, the model's six final-hidden-layer representations are compared by pairwise cosine similarity, and the image with the lowest average similarity to the other five is chosen as odd. This probe is applied to eight pretrained models spanning CNNs, transformers, and VLMs, and the resulting class-level accuracies are compared to three published human datasets through Pearson correlation of the seven-class profiles. The load-bearing identity is the assumption that lowest-average-cosine is the visual analogue of the human 'odd one out' judgment; all accuracy and alignment claims pass through it.","core_discovery":"On the 43-concept, seven-class odd-one-out stimuli originally designed to test non-schooled humans, the paper reports that transformer-based vision models, especially DINOv2 (48.89% accuracy) and ViT (46.67%), surpass the 37.72% overall accuracy of the published 3–6-year-old children sample, while remaining far below the adult samples (65.34% and 85.10%). The transformers' class-level accuracy profiles align with the children's profiles at r = 0.93 and r = 0.91, the highest alignment of any model class tested. Contrastively trained vision-language models — CLIP with ResNet or ViT backbones, and ALIGN — show lower accuracy (33.33–37.78% for CLIP, 24.44% for ALIGN, the latter not above chance) and weaker human alignment, which the paper interprets as evidence that text-image contrastive alignment can impair the formation of abstract geometric representations. These results are framed as supporting the 'learning for free' account of geometric and topological concepts: sensitivity need not be innate because it can be acquired through large-scale visual pretraining, and the transformer-child correlation points to these models as candidate cognitive and developmental models.","pith_inferences":["A stricter test of the learning-for-free claim would replace the zero-shot cosine probe with a linear readout trained on one set of GT concepts and tested on held-out concepts; if sensitivity dissolves under that probe, the current accuracy may reflect the probe rather than the representation.","The r-values are computed on only 7 class means; correlating model and child accuracy at the level of the 43 individual concepts would be a harder, more diagnostic alignment test.","The VLM deficit is confounded with training data: CLIP and ALIGN were trained on natural image-caption pairs, so a vision-only encoder trained on the same image collections would be needed to separate the effect of modality alignment from data distribution.","One testable extension is to track DINOv2's class-level accuracy during training; if Euclidean geometry sensitivity emerges before transformation sensitivity in the same order as in children, the developmental-alignment claim would gain direct support."],"forward_implications":["Transformer models clear the child benchmark for the first time in this paradigm: they are as accurate or more accurate than 3–6-year-olds on all 7 concept classes, not just overall.","The strong child-profile correlations (r > 0.90) make transformer encoders plausible computational models for studying how geometric sensitivity develops over training, through representation-level analyses rather than behavior alone.","The VLM result undercuts the naive expectation from dual-coding theory that adding language should help: contrastive image-text training at current scale appears to reduce sensitivity to abstract geometric and topological concepts.","Because transformers were trained on much larger image corpora than CNNs, the accuracy gain is consistent with a scale-driven learning account, though the paper does not isolate architecture from data size.","The common easy/hard ordering — Euclidean geometry easiest, transformations and symmetry hardest — suggests graded continuity between human and model GT sensitivity, with Symmetrical Figures as a clear remaining gap."],"supporting_citations":[{"why":"Supplies the 43-concept, seven-class odd-one-out stimulus set and the Mundurucu human baseline that anchors the whole comparison.","marker":"Dehaene et al. (2006)"},{"why":"Provides the 3–6-year-old children and Western adult accuracy datasets that set the benchmarks the models must beat or approach.","marker":"Izard and Spelke (2009)"},{"why":"Establishes the cosine-similarity evaluation protocol from the final hidden layer and the prior CNN results this study extends.","marker":"Upadhyay et al. (2025)"},{"why":"Frames the expectation that vision transformers and CLIP align with human geometric sensitivity and supplies the comparison point for those models.","marker":"Campbell et al. (2024)"},{"why":"Defines the DINOv2 architecture and its self-supervised training corpus, the best-performing model in the study.","marker":"Oquab et al. (2024)"},{"why":"Defines the CLIP vision-language model whose two variants are tested and found to underperform.","marker":"Radford et al. (2021)"},{"why":"Defines the ALIGN vision-language model, the worst-performing model in the study.","marker":"Jia et al. (2021)"},{"why":"Defines the ViT architecture that, along with DINOv2, constitutes the transformer class.","marker":"Dosovitskiy et al. (2021)"},{"why":"Supplies the ResNet-18 and ResNet-50 CNN architectures that anchor the CNN class and the shallower-deeper comparison.","marker":"K. He et al. (2015)"},{"why":"Articulates the core-knowledge account that the learning-for-free interpretation is designed to challenge.","marker":"Spelke & Kinzler (2007)"}],"fun_headline_variants":["Transformers surpass kids on geometry, CLIP doesn't","Vision transformers mirror child geometry intuition","CLIP's text-vision mix harms shape sense","DINOv2: geometry without innate bias","AI learns geometry from images, like kids do"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison rests on treating 'the image with the lowest average pairwise cosine similarity in the final hidden layer' as the model's odd-one-out answer; if that probe does not track perceptual oddity on these exact stimuli, the model accuracies, the child-surpassing result, and the alignment correlations lose their meaning.","fun_headline_variants_meta":{"raw":{"variants":["Transformers surpass kids on geometry, CLIP doesn't","Vision transformers mirror child geometry intuition","CLIP's text-vision mix harms shape sense","DINOv2: geometry without innate bias","AI learns geometry from images, like kids do"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000485,"raw_usage":{"total_tokens":2444,"prompt_tokens":1046,"completion_tokens":1398,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":662,"completion_tokens_details":{"reasoning_tokens":1327}},"tokens_in":662,"tokens_out":1398,"duration_ms":12933,"temperature":1.0,"reasoning_tokens":1327,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:16:07.183238+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a control study on the same 43 stimuli in which the cosine-similarity rule is replaced by a linear probe trained on human odd-one-out choices for half the concepts and tested on the remaining half; if the transformer accuracy advantage over children disappears, or if the correlation with human profiles is recomputed at the level of the 43 individual concepts and drops to near zero, the central claim is falsified.","supporting_citations":[{"cited_title":", Izard, V","cited_arxiv_id":null,"evidence_quote":"Supplies the 43-concept, seven-class odd-one-out stimulus set and the Mundurucu human baseline that anchors the whole comparison."},{"cited_title":"\\ Spelke, E S","cited_arxiv_id":null,"evidence_quote":"Provides the 3–6-year-old children and Western adult accuracy datasets that set the benchmarks the models must beat or approach."},{"cited_title":", Marupudi, V","cited_arxiv_id":null,"evidence_quote":"Establishes the cosine-similarity evaluation protocol from the final hidden layer and the prior CNN results this study extends."},{"cited_title":"Human-Like Geometric Abstraction in Large Pre-trained Neural Networks","cited_arxiv_id":"2402.04203","evidence_quote":"Frames the expectation that vision transformers and CLIP align with human geometric sensitivity and supplies the comparison point for those models."}],"review_version":1}