pith. machine review for the scientific record. sign in

arxiv: 2605.06420 · v1 · submitted 2026-05-07 · 🧬 q-bio.NC

Recognition: unknown

Beyond Object-Level Alignment: Do Brains and DNNs Preserve the Same Transformations?

Yukiyasu Kamitani

Pith reviewed 2026-05-08 03:12 UTC · model grok-4.3

classification 🧬 q-bio.NC
keywords brain-DNN alignmentnaturalitystimulus transformationsfMRIdeep neural networkshierarchyNaturality Violation ScoreGOD dataset
0
0 comments X

The pith

Brain and DNN alignment holds when they preserve the same stimulus transformations, with semantic ones matching higher brain areas to deeper layers.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Standard brain-DNN comparisons check if the two systems give similar responses to the same images or share similar overall patterns. This paper instead asks whether brains and models apply the same transformations when stimuli are changed in specific ways. It borrows the idea of naturality from category theory and defines a Naturality Violation Score to measure how well a change commutes through either system first. Tests on fMRI data from the GOD set and several vision models show that transformations involving meaning align best in higher visual cortex with deeper model layers, while basic visual feature changes align in early cortex with shallow layers. This reframes alignment as a test of jointly preserved operations rather than isolated stimulus matches.

Core claim

We formalize alignment as approximate naturality: propagating a proxy-defined stimulus change through the brain representation and then mapping to the model side should approximately equal mapping first and then propagating through the model. Deviations are quantified by a Naturality Violation Score normalized to a permutation null. Axis-resolved results on fMRI responses from five subjects, three DNNs, and three world-model embeddings reveal a hierarchy crossover where semantic axes such as animacy yield low NVS values toward higher visual cortex and deeper layers while low- and mid-level visual axes align toward earlier cortex and shallower layers.

What carries the argument

The Naturality Violation Score (NVS), a normalized measure of commutativity failure in the naturality square formed by a proxy stimulus transformation, brain-side propagation, model-side propagation, and an explicit comparison map between the two representation spaces.

If this is right

  • Alignment can be tested selectively for particular families of transformations rather than as a single overall similarity number.
  • Semantic axes align most strongly toward higher visual cortex and deeper DNN layers.
  • Low- and mid-level visual axes align most strongly toward earlier visual cortex and shallower layers.
  • Synthetic controlled settings confirm that NVS detects complementary failures missed by aggregate object- or geometry-level measures.
  • The alignment pattern is selective over candidate transformations rather than uniform across all possible maps.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Richer proxy spaces could be constructed to test preservation of more complex or naturalistic world transformations.
  • The method supplies a concrete way to compare candidate brain-like models by how well their internal transformations match measured brain ones.
  • Controlled experiments that vary the strength or type of proxy change could isolate which transformation families drive the observed hierarchy crossover.

Load-bearing premise

The chosen proxy-defined stimulus changes and the explicit comparison map between brain and model spaces accurately represent the transformations that the systems actually preserve.

What would settle it

Recomputing NVS on an independent stimulus set or with alternative proxies and finding that semantic axes no longer produce reliably lower scores in higher visual cortex and deeper layers than in mismatched regions or axes.

Figures

Figures reproduced from arXiv: 2605.06420 by Yukiyasu Kamitani.

Figure 1
Figure 1. Figure 1: Brain–DNN alignment as approximate naturality (motivational concept diagram). Left: the World Model W parameterizes which stimulus change r : s → s ′ is tested; the translators η : B → M and η ′ : M → B connect the two sides. Right: the naturality square (the analogous square is also asked of η ′ ). This is the motivational categorical reading; in practice FB(r), FM(r) are not estimated as endomorphisms of… view at source ↗
Figure 2
Figure 2. Figure 2: Dual-triangle decomposition of NVS. Each world morphism ∆W has two paths to a target space. Left: direct ΦM(∆W ) vs. brain-mediated η(ΦB(∆W )). Right: direct ΦB(∆W ) vs. DNN-mediated η ′ (ΦM(∆W )). NVS is the symmetric mean of the two per-direction permutation-normalized residuals (§3); used identically in the synthetic PoC and the empirical fMRI study. inside B or M — not strict functors, since identities… view at source ↗
Figure 3
Figure 3. Figure 3: 5-subject mean NVSa (6 scalar-axis rows) and NVSfull (full-FW row, bottom) × 3 FW × 3 vision DNNs. Each submap is a 5 ROI × 8 layer heatmap, ROI horizontal (V1→HVC), layer vertical (L1→L8). Color: deep blue = stronger preservation, white = permutation null (1.0), red = above null. A/R/V = AlexNet/ResNet/ViT-B. The axis-resolved rows show the hierarchy crossover (low-level axes → V1×shallow; semantic axes →… view at source ↗
read the original abstract

Brain-DNN alignment is usually assessed through stimulus-level correspondence or stimulus-set geometry. Inspired by category theory, we operationalize a different question: do brain and model preserve the same candidate transformations among stimuli? We formalize this as approximate naturality: if a proxy-defined stimulus change is propagated through the brain side and then translated to the model side, the result should match translating first and then propagating, so that the naturality square approximately commutes. We quantify deviations from commutativity by a Naturality Violation Score (NVS) normalized to a permutation null, shifting alignment from per-stimulus sameness to preservation of structure under an explicitly chosen comparison map. As a proof of concept, a controlled five-factor synthetic setting shows that NVS separates complementary alignment failures that aggregate object- and geometry-level scalars cannot resolve. Applied to fMRI responses from the GOD dataset (5 subjects), 3 vision DNNs, and 3 World-Model proxy embeddings, the axis-resolved analysis reveals a hierarchy crossover: semantic axes align most strongly toward HVC and deeper DNN layers (NVS^animacy = 0.39 vs 0.52 for the next-best axis and 1.0 for the permutation-null baseline), whereas low- and mid-level visual axes align toward earlier visual cortex and shallower layers. Supporting analyses (a 15-axis appendix atlas, dissociation tests against RSA/CKA and encoding/decoding accuracy, and a W-less anchor-ablation control) confirm that the alignment is selective over candidate morphism families rather than uniform. NVS thereby turns brain-DNN comparison into a test of jointly preserved candidate transformations, relative to an explicit proxy space and permutation null, and opens a path to richer proxy spaces and controlled world-side transformations.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces a category-theoretic framework for brain-DNN alignment that tests preservation of the same transformations among stimuli via approximate commutativity of naturality squares. Deviations are quantified by a Naturality Violation Score (NVS) normalized to a permutation null. A controlled five-factor synthetic validation shows NVS can separate complementary alignment failures unresolved by aggregate metrics. Applied to GOD fMRI data (5 subjects), 3 vision DNNs, and 3 World-Model proxy embeddings, the axis-resolved results report a hierarchy crossover: semantic axes (e.g., animacy) align most strongly toward HVC and deeper layers (NVS^animacy = 0.39 vs. 0.52 next-best axis and 1.0 null baseline), while low- and mid-level visual axes align toward earlier visual cortex and shallower layers. Supporting analyses include a 15-axis appendix atlas, dissociation tests vs. RSA/CKA and encoding/decoding accuracy, and a W-less anchor-ablation control.

Significance. If the proxy-defined axes and comparison map validly represent transformations actually preserved by the systems, the work provides a substantive advance by reframing alignment as a test of jointly preserved structure under morphisms rather than per-stimulus or geometry-level scalars. The synthetic case demonstrates resolution of distinct failure modes, the permutation null supplies an external baseline, and the dissociation tests plus anchor-ablation add controls that strengthen claims of selectivity over candidate morphism families. This opens a path to richer proxy spaces and controlled world-side transformations, with potential to complement existing metrics in computational neuroscience.

major comments (2)
  1. [Abstract] Abstract: The hierarchy-crossover claim interprets NVS^animacy = 0.39 (vs. 0.52 for the next-best axis and 1.0 permutation-null baseline) as stronger preservation of semantic transformations toward HVC and deeper DNN layers. This interpretation is load-bearing and rests on the assumption that the three World-Model proxy embeddings define stimulus changes whose naturality squares are meaningful for the actual brain and DNN representations; the GOD fMRI application provides no independent verification that the 15-axis atlas axes correspond to transformations the systems use, raising the possibility that low NVS reflects joint alignment to the external embedding rather than intrinsic preservation.
  2. [Formalization of approximate naturality and NVS] Formalization of approximate naturality and NVS (main text): The naturality square depends on an explicitly chosen comparison map between brain and model spaces. While a W-less anchor-ablation control is mentioned, the manuscript does not report tests of NVS invariance to alternative constructions of this map. Because the map is required to propagate stimulus changes in either order, lack of such tests undermines the claim that reported alignment reflects preserved transformations rather than map-specific artifacts.
minor comments (2)
  1. The abstract states that supporting analyses 'confirm that the alignment is selective over candidate morphism families rather than uniform,' but the main text should expand on the exact statistical criteria and effect sizes in the dissociation tests against RSA/CKA to allow readers to assess the strength of selectivity.
  2. Notation for NVS^animacy and the permutation-null baseline should be accompanied by the explicit formula (including any normalization details) in the main text or a dedicated methods subsection for reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive comments. We address each major point below, clarifying the scope of our claims and proposing targeted revisions to strengthen the manuscript.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The hierarchy-crossover claim interprets NVS^animacy = 0.39 (vs. 0.52 for the next-best axis and 1.0 permutation-null baseline) as stronger preservation of semantic transformations toward HVC and deeper DNN layers. This interpretation is load-bearing and rests on the assumption that the three World-Model proxy embeddings define stimulus changes whose naturality squares are meaningful for the actual brain and DNN representations; the GOD fMRI application provides no independent verification that the 15-axis atlas axes correspond to transformations the systems use, raising the possibility that low NVS reflects joint alignment to the external embedding rather than intrinsic preservation.

    Authors: We agree that the results are conditional on the chosen proxy axes representing candidate transformations. The manuscript explicitly frames NVS as measuring approximate preservation relative to an explicit proxy space and permutation null (see abstract and Section 3). The observed dissociation—semantic axes showing lower NVS in HVC/deeper layers while low-level axes align earlier—would be unlikely under uniform alignment to the external embedding, as all axes derive from the same proxies. The five-factor synthetic validation further demonstrates that NVS isolates transformation preservation beyond geometry-level alignment. We will revise the abstract and add a paragraph in the Discussion to emphasize that these are proxy-defined candidates and that independent validation of the axes (e.g., via behavioral or perturbation experiments) remains an important direction for future work. revision: partial

  2. Referee: [Formalization of approximate naturality and NVS] Formalization of approximate naturality and NVS (main text): The naturality square depends on an explicitly chosen comparison map between brain and model spaces. While a W-less anchor-ablation control is mentioned, the manuscript does not report tests of NVS invariance to alternative constructions of this map. Because the map is required to propagate stimulus changes in either order, lack of such tests undermines the claim that reported alignment reflects preserved transformations rather than map-specific artifacts.

    Authors: The referee correctly notes that we report only the W-less anchor-ablation. This ablation removes the anchor stimuli used to fit the linear comparison map and shows that NVS patterns persist, indicating the result is not driven by anchor-specific fitting. However, we did not systematically vary the map construction itself (e.g., orthogonal Procrustes, CCA, or nonlinear alternatives). We will add an appendix with these alternative maps on the GOD data, confirming that the hierarchy crossover for semantic vs. low-level axes remains stable. This will directly address invariance. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper defines NVS directly from the approximate commutativity of the naturality square (proxy stimulus change propagated through brain then translated vs. translated then propagated through model), using an explicitly chosen comparison map and World-Model proxy embeddings. It normalizes deviations to a permutation null that is external to the fitted data. The hierarchy-crossover observation is an empirical application of this metric to GOD fMRI, DNN layers, and axis-resolved proxies, not a reduction of the claimed result to its own inputs by construction. No self-definitional equations, fitted parameters renamed as predictions, or load-bearing self-citations appear in the derivation chain; the synthetic validation and supporting dissociation tests further keep the metric independent of the target claims.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Only the abstract is available; the ledger is therefore limited to elements explicitly named or implied. No free parameters are stated. The central claim rests on the assumption that proxy stimulus changes can stand in for the transformations of interest.

axioms (1)
  • domain assumption Approximate naturality (commutativity of the naturality square up to a chosen comparison map) is a meaningful test of jointly preserved transformations between brain and model representations.
    The paper treats this as the operational definition of the alignment question; it is invoked to justify shifting from stimulus-level to transformation-level comparison.

pith-pipeline@v0.9.0 · 5618 in / 1319 out tokens · 30603 ms · 2026-05-08T03:12:36.172235+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

48 extracted references · 1 canonical work pages

  1. [1]

    Nature Communications , volume =

    Horikawa, Tomoyasu and Kamitani, Yukiyasu , title =. Nature Communications , volume =. 2017 , doi =

    2017

  2. [2]

    Nature Neuroscience , volume =

    Kamitani, Yukiyasu and Tong, Frank , title =. Nature Neuroscience , volume =. 2005 , doi =

    2005

  3. [3]

    NeuroImage , volume =

    Yamashita, Okito and Sato, Masa-aki and Yoshioka, Taku and Tong, Frank and Kamitani, Yukiyasu , title =. NeuroImage , volume =. 2008 , doi =

    2008

  4. [4]

    Annual Review of Vision Science , volume =

    Kamitani, Yukiyasu and Tanaka, Misato and Shirakawa, Ken , title =. Annual Review of Vision Science , volume =. 2025 , doi =

    2025

  5. [5]

    and Muraki, Yusuke and Majima, Kei and Kamitani, Yukiyasu , title =

    Shirakawa, Ken and Nagano, Yoshihiro and Tanaka, Misato and Aoki, Shuntaro C. and Muraki, Yusuke and Majima, Kei and Kamitani, Yukiyasu , title =. Neural Networks , volume =. 2025 , doi =

    2025

  6. [6]

    and Rajalingham, Rishi and Issa, Elias B

    Schrimpf, Martin and Kubilius, Jonas and Hong, Ha and Majaj, Najib J. and Rajalingham, Rishi and Issa, Elias B. and Kar, Kohitij and Bashivan, Pouya and Prescott-Roy, Jonathan and Geiger, Franziska and Poggio, Tomaso and DiCarlo, James J. , title =. bioRxiv , year =

  7. [7]

    and Kamitani, Yukiyasu , title =

    Nonaka, Soma and Majima, Kei and Aoki, Shuntaro C. and Kamitani, Yukiyasu , title =. iScience , volume =. 2021 , doi =

    2021

  8. [8]

    Yamins, Daniel L. K. and Hong, Ha and Cadieu, Charles F. and Solomon, Ethan A. and Seibert, Darren and DiCarlo, James J. , title =. Proceedings of the National Academy of Sciences , volume =. 2014 , doi =

    2014

  9. [9]

    Frontiers in Systems Neuroscience , volume =

    Kriegeskorte, Nikolaus and Mur, Marieke and Bandettini, Peter , title =. Frontiers in Systems Neuroscience , volume =. 2008 , doi =

    2008

  10. [10]

    Annual Review of Vision Science , volume =

    Kriegeskorte, Nikolaus , title =. Annual Review of Vision Science , volume =. 2015 , doi =

    2015

  11. [11]

    Proceedings of the International Conference on Machine Learning (ICML) , year =

    Kornblith, Simon and Norouzi, Mohammad and Lee, Honglak and Hinton, Geoffrey , title =. Proceedings of the International Conference on Machine Learning (ICML) , year =

  12. [12]

    and Welling, Max , title =

    Cohen, Taco S. and Welling, Max , title =. Proceedings of the International Conference on Machine Learning (ICML) , year =

  13. [13]

    International Conference on Learning Representations (ICLR) , year =

    Sanborn, Sophia and Shewmake, Christian and Olshausen, Bruno and Hillar, Christopher , title =. International Conference on Learning Representations (ICLR) , year =

  14. [14]

    Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) , year =

    Mikolov, Tomas and Yih, Wen-tau and Zweig, Geoffrey , title =. Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) , year =

  15. [15]

    Proceedings of the International Conference on Machine Learning (ICML) , year =

    Park, Kiho and Choe, Yo Joong and Veitch, Victor , title =. Proceedings of the International Conference on Machine Learning (ICML) , year =

  16. [16]

    da Costa, Newton C. A. and French, Steven , title =

  17. [17]

    , title =

    van Fraassen, Bas C. , title =

  18. [18]

    Memory Evolutive Systems: Hierarchy, Emergence, Cognition , publisher =

    Ehresmann, Andr. Memory Evolutive Systems: Hierarchy, Emergence, Cognition , publisher =

  19. [19]

    Proceedings of the International Conference on Machine Learning (ICML) , year =

    Radford, Alec and Kim, Jong Wook and Hallacy, Chris and Ramesh, Aditya and Goh, Gabriel and Agarwal, Sandhini and Sastry, Girish and Askell, Amanda and Mishkin, Pamela and Clark, Jack and Krueger, Gretchen and Sutskever, Ilya , title =. Proceedings of the International Conference on Machine Learning (ICML) , year =

  20. [20]

    Transactions on Machine Learning Research , year =

    Oquab, Maxime and Darcet, Timoth. Transactions on Machine Learning Research , year =

  21. [21]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Fu, Stephanie and Tamir, Netanel and Sundaram, Shobhita and Chai, Lucy and Zhang, Richard and Dekel, Tali and Isola, Phillip , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  22. [22]

    Proceedings of the International Conference on Machine Learning (ICML) , year =

    Kim, Been and Wattenberg, Martin and Gilmer, Justin and Cai, Carrie and Wexler, James and Viegas, Fernanda and Sayres, Rory , title =. Proceedings of the International Conference on Machine Learning (ICML) , year =

  23. [23]

    Recurrent world models facilitate policy evolution , booktitle =

    Ha, David and Schmidhuber, J. Recurrent world models facilitate policy evolution , booktitle =. 2018 , volume =

    2018

  24. [24]

    Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , year =

    Bisk, Yonatan and Holtzman, Ari and Thomason, Jesse and Andreas, Jacob and Bengio, Yoshua and Chai, Joyce and Lapata, Mirella and Lazaridou, Angeliki and May, Jonathan and Nisnevich, Aleksandr and Pinto, Nicolas and Turian, Joseph , title =. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , year =. doi:10.18653/v1...

    work page doi:10.18653/v1/2020.emnlp-main.703 2020

  25. [25]

    Cognitive Systems Research , volume =

    Cao, Rosa and Yamins, Daniel , title =. Cognitive Systems Research , volume =. 2024 , doi =

    2024

  26. [26]

    Journal of Neuroscience , volume =

    Konkle, Talia and Caramazza, Alfonso , title =. Journal of Neuroscience , volume =. 2013 , doi =

    2013

  27. [27]

    Position: Categorical deep learning is an algebraic theory of all architectures , booktitle =

    Gavranovi. Position: Categorical deep learning is an algebraic theory of all architectures , booktitle =. 2024 , volume =

    2024

  28. [28]

    Frontiers in Psychology , volume =

    Phillips, Steven , title =. Frontiers in Psychology , volume =. 2022 , doi =

    2022

  29. [29]

    and Kay, Kendrick N

    Conwell, Colin and Prince, Jacob S. and Kay, Kendrick N. and Alvarez, George A. and Konkle, Talia , title =. Nature Communications , volume =. 2024 , doi =

    2024

  30. [30]

    and Grant, Erin and Groen, Iris and Achterberg, Jascha and Tenenbaum, Joshua B

    Sucholutsky, Ilia and Muttenthaler, Lukas and Weller, Adrian and Peng, Andi and Bobu, Andreea and Kim, Been and Love, Bradley C. and Grant, Erin and Groen, Iris and Achterberg, Jascha and Tenenbaum, Joshua B. and Collins, Katherine M. and Hermann, Katherine L. and Oktar, Kerem and Greff, Klaus and Hebart, Martin N. and Cloos, Nathan and Kriegeskorte, Niko...

    2025

  31. [31]

    , title =

    Bao, Pinglei and She, Liang and McGill, Mason and Tsao, Doris Y. , title =. Nature , volume =. 2020 , doi =

    2020

  32. [32]

    and Epstein, Russell A

    Bonner, Michael F. and Epstein, Russell A. , title =. Proceedings of the National Academy of Sciences , volume =. 2017 , doi =

    2017

  33. [33]

    , title =

    Conway, Bevil R. , title =. The Neuroscientist , volume =. 2009 , doi =

    2009

  34. [34]

    WordNet: An Electronic Lexical Database , publisher =

  35. [35]

    and Oliva, Aude , title =

    Greene, Michelle R. and Oliva, Aude , title =. Cognitive Psychology , volume =. 2009 , doi =

    2009

  36. [36]

    Neuron , volume =

    Konkle, Talia and Oliva, Aude , title =. Neuron , volume =. 2012 , doi =

    2012

  37. [37]

    International Journal of Computer Vision , volume =

    Oliva, Aude and Torralba, Antonio , title =. International Journal of Computer Vision , volume =. 2001 , doi =

    2001

  38. [38]

    and Field, David J

    Olshausen, Bruno A. and Field, David J. , title =. Nature , volume =. 1996 , doi =

    1996

  39. [39]

    , title =

    Portilla, Javier and Simoncelli, Eero P. , title =. International Journal of Computer Vision , volume =. 2000 , doi =

    2000

  40. [40]

    , title =

    Sharan, Lavanya and Rosenholtz, Ruth and Adelson, Edward H. , title =. Journal of Vision , volume =. 2014 , doi =

    2014

  41. [41]

    and Ratan Murty, N

    Schrimpf, Martin and Kubilius, Jonas and Lee, Michael J. and Ratan Murty, N. Apurva and Ajemian, Robert and DiCarlo, James J. , title =. Neuron , volume =. 2020 , doi =

    2020

  42. [42]

    and Nishimoto, Shinji and Gallant, Jack L

    Naselaris, Thomas and Kay, Kendrick N. and Nishimoto, Shinji and Gallant, Jack L. , title =. NeuroImage , volume =. 2011 , doi =

    2011

  43. [43]

    Proceedings of the National Academy of Sciences , volume =

    Long, Bria and Yu, Chen-Ping and Konkle, Talia , title =. Proceedings of the National Academy of Sciences , volume =. 2018 , doi =

    2018

  44. [44]

    and Szedm

    Hardoon, David R. and Szedm. Canonical correlation analysis: An overview with application to learning methods , journal =. 2004 , doi =

    2004

  45. [45]

    A generalized solution of the orthogonal

    Sch. A generalized solution of the orthogonal. Psychometrika , volume =. 1966 , doi =

    1966

  46. [46]

    , booktitle =

    Krizhevsky, Alex and Sutskever, Ilya and Hinton, Geoffrey E. , booktitle =. 2012 , volume =

    2012

  47. [47]

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages =

    Deep residual learning for image recognition , author =. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages =

  48. [48]

    International Conference on Learning Representations (ICLR) , year =

    An image is worth 16x16 words: Transformers for image recognition at scale , author =. International Conference on Learning Representations (ICLR) , year =