Similarity-based representation factorization for revealing interpretable dimensions in representational data
Pith reviewed 2026-06-29 17:54 UTC · model grok-4.3
The pith
SRF recovers low-dimensional interpretable dimensions from similarity matrices of representational data even when measurements are sparse or incomplete.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
SRF is a computational method that takes similarity matrices derived from measured data and recovers low-dimensional non-negative embeddings whose factors are interpretable as the dimensions underlying the representations. Across simulations and real neural, behavioral, and computational datasets the method extracts these dimensions even from very sparsely sampled data, and the dimensions match task-specific models while predicting independent behavioral properties.
What carries the argument
Similarity-Based Representation Factorization (SRF), a factorization procedure that decomposes similarity matrices into low-dimensional non-negative factors.
If this is right
- Recovered dimensions match those obtained by task-specific models.
- Dimensions predict independent behavioral properties.
- Method improves exploratory analysis of representations.
- Method offers higher power for confirmatory hypothesis testing than direct comparison of similarity matrices.
Where Pith is reading between the lines
- The approach could enable side-by-side comparison of representational dimensions across different recording modalities or model architectures without building new task models each time.
- Dimensions recovered this way might be used to test whether specific axes of variation remain stable across individuals or learning stages.
- Because the factors are non-negative and low-dimensional, they could serve as a compact basis for generating new stimuli that isolate individual dimensions.
Load-bearing premise
Similarity matrices derived from measured data contain recoverable low-dimensional non-negative structure that factorization can extract without task-specific constraints or post-hoc adjustments.
What would settle it
Apply SRF to a dataset whose true underlying dimensions are independently known from a task-specific model and observe whether the recovered factors fail to match those dimensions or lose predictive power for held-out behavioral measures.
read the original abstract
The study of representations is widespread across fields, including neuroscience, psychology, and artificial intelligence. While representations are often studied and compared through similarities between stimuli, current methods provide only limited access to the dimensions that shape these representations and are often limited in interpretability. To overcome these challenges, here we introduce Similarity-Based Representation Factorization (SRF), a general computational method for recovering low-dimensional, non-negative, interpretable embeddings from similarity matrices derived from measured data. Across simulations and many neural, behavioral, and computational datasets, SRF recovers interpretable dimensions from diverse forms of representational data, even for very sparsely sampled, incomplete data. The dimensions derived from these datasets match those obtained by task-specific models, predict independent behavioral properties, improve exploratory analysis, and offer higher power for confirmatory hypothesis testing than comparing similarity matrices. Together, these results establish SRF as a general-purpose method with broad applications for uncovering, understanding, and using the dimensions underlying representations.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Similarity-Based Representation Factorization (SRF), a general method to recover low-dimensional non-negative interpretable embeddings from similarity matrices derived from measured data. It claims validation across simulations and diverse neural, behavioral, and computational datasets (including sparse/incomplete cases), where the recovered dimensions match those from task-specific models, predict independent behavioral properties, improve exploratory analysis, and yield higher power for confirmatory hypothesis testing than direct similarity-matrix comparisons.
Significance. If the central recoverability claim holds without task-specific constraints or post-hoc adjustments, SRF would offer a broadly applicable tool for interpreting representations across neuroscience, psychology, and AI, enabling more powerful hypothesis testing and unifying exploratory approaches; the paper's emphasis on simulations plus multiple real datasets is a strength in this regard.
major comments (2)
- [Abstract] Abstract: the central claim that SRF recovers the 'true' interpretable dimensions directly from S (even when sparse/incomplete) requires that S admits a consistently recoverable low-rank non-negative factorization S ≈ XX^T aligned with the generative process; the provided text gives no equations, uniqueness proof, or ablation on initialization/regularization sensitivity, leaving open whether outputs are data-determined or method-dependent.
- The skeptic concern on recoverability is load-bearing: without explicit demonstration (e.g., via multiple random initializations or comparison to alternative factorizations) that the non-negative low-dimensional structure is uniquely recoverable rather than one of many possible factorizations, the assertion that dimensions 'match task-specific models' and 'predict independent behavioral properties' cannot be distinguished from algorithmic bias.
minor comments (1)
- [Abstract] Abstract: the phrase 'higher power for confirmatory hypothesis testing' would benefit from a brief parenthetical on the statistical test used and how power was quantified.
Simulated Author's Rebuttal
We thank the referee for the constructive comments on recoverability and the need for supporting demonstrations. We address each point below and outline planned revisions to strengthen the manuscript.
read point-by-point responses
-
Referee: [Abstract] Abstract: the central claim that SRF recovers the 'true' interpretable dimensions directly from S (even when sparse/incomplete) requires that S admits a consistently recoverable low-rank non-negative factorization S ≈ XX^T aligned with the generative process; the provided text gives no equations, uniqueness proof, or ablation on initialization/regularization sensitivity, leaving open whether outputs are data-determined or method-dependent.
Authors: The full manuscript (Methods, Section 2) presents the SRF objective and update rules as a constrained optimization problem minimizing ||S - XX^T|| with X ≥ 0. While a general uniqueness theorem does not exist for this factorization (consistent with the broader NMF literature), the simulation section reports recovery performance across 50 random initializations per condition and varying regularization strengths, with consistent alignment to ground-truth dimensions. We will add a dedicated ablation subsection quantifying sensitivity to initialization and regularization parameters. revision: yes
-
Referee: The skeptic concern on recoverability is load-bearing: without explicit demonstration (e.g., via multiple random initializations or comparison to alternative factorizations) that the non-negative low-dimensional structure is uniquely recoverable rather than one of many possible factorizations, the assertion that dimensions 'match task-specific models' and 'predict independent behavioral properties' cannot be distinguished from algorithmic bias.
Authors: The current simulations already include multiple random initializations and report low variance in recovered dimensions; real-data sections also benchmark SRF against PCA and standard NMF, showing superior alignment with task-specific models. To further address the concern we will expand the main text with explicit variance-across-initializations statistics and additional baseline comparisons, while noting that the method's value is demonstrated empirically rather than via uniqueness guarantees. revision: partial
- Formal mathematical uniqueness proof for SRF under arbitrary similarity matrices and sparsity patterns
Circularity Check
No circularity in SRF derivation or claims
full rationale
The paper introduces SRF as an algorithmic method to factorize similarity matrices into low-dimensional non-negative embeddings. Claims rest on empirical applications to simulations and real datasets, with comparisons to independent task-specific models and behavioral measures. No equations, self-citations, or steps are shown that reduce the recovered dimensions to fitted parameters by construction or to self-referential definitions. The central recoverability claim is presented as a testable property of the data and method rather than an input assumption renamed as output.
Axiom & Free-Parameter Ledger
Reference graph
Works this paper leans on
-
[1]
E. J. Allen, G. St-Yves, Y. Wu, J. L. Breedlove, J. S. Prince, L. T. Dowdle, M. Nau, B. Caron, F. Pestilli, I. Charest, J. B. Hutchinson, T. Naselaris, and K. Kay. A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nat. Neurosci., 25: 0 116--126, 2022
2022
-
[2]
P. Bao, L. She, M. McGill, and D. Y. Tsao. A map of object space in primate inferotemporal cortex. Nature, 583: 0 103--108, 2020
2020
-
[3]
D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba. Network dissection: quantifying interpretability of deep visual representations. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3319--3327. IEEE, 2017
2017
-
[4]
Bockes, M
A. Bockes, M. N. Hebart, and A. Lingnau. Revealing key dimensions underlying the recognition of dynamic human actions. Commun. Psychol., 3: 0 149, 2025
2025
-
[5]
S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn., 3: 0 1--122, 2011
2011
-
[6]
Bracci and H
S. Bracci and H. P. Op de Beeck. Understanding human object vision: A picture is worth a thousand representations. Annu. Rev. Psychol., 74: 0 113--135, 2023
2023
-
[7]
Bricken, A
T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, R. Lasenby, Y. Wu, S. Kravec, N. Schiefer, T. Maxwell, N. Joseph, Z. Hatfield-Dodds, A. Tamkin, K. Nguyen, B. McLean, J. E. Burke, T. Hume, S. Carter, T. Henighan, and C. Olah. Towards monosemanticity: Decomposing language models with dictionary...
2023
-
[8]
E. J. Cand \`e s and B. Recht. Exact matrix completion via convex optimization. Found. Comput. Math., 9 0 (6): 0 717--772, 2009
2009
-
[9]
K. W. Church and P. Hanks. Word association norms, mutual information, and lexicography. Comput. Linguist., 16 0 (1): 0 22--29, 1990
1990
-
[10]
Contier, C
O. Contier, C. I. Baker, and M. N. Hebart. Distributed representations of behaviour-derived object dimensions in the human visual system. Nat. Hum. Behav., 8: 0 2179--2193, 2024
2024
-
[11]
Costa, T
V. Costa, T. Fel, E. S. Lubana, B. Tolooshams, and D. Ba. From flat to hierarchical: extracting sparse representations with matching pursuit. Adv. Neural Inf. Process. Syst., 38, 2025
2025
-
[12]
small world of words
S. De Deyne, D. J. Navarro, A. Perfors, M. Brysbaert, and G. Storms. The "small world of words" english word association norms for over 12,000 cue words. Behav. Res. Methods, 51: 0 987--1006, 2019
2019
-
[13]
J. J. DiCarlo, D. Zoccolan, and N. C. Rust. How does the brain solve visual object recognition? Neuron, 73: 0 415--434, 2012
2012
-
[14]
C. Ding, X. He, and H. D. Simon. On the equivalence of nonnegative matrix factorization and spectral clustering. In Proc. SIAM International Conference on Data Mining, pages 606--610, 2005
2005
-
[15]
C. Du, K. Fu, B. Wen, Y. Sun, J. Peng, W. Wei, Y. Gao, S. Wang, C. Zhang, J. Li, S. Qiu, L. Chang, and H. He. Human-like object concept representations emerge naturally in multimodal large language models. Nat. Mach. Intell., 7: 0 860--875, 2025
2025
-
[16]
Finkelstein, E
L. Finkelstein, E. Gabrilovich, Y. Matias, E. Rivlin, Z. Solan, G. Wolfman, and E. Ruppin. Placing search in context: The concept revisited. ACM Trans. Inf. Syst., 20: 0 116--131, 2002
2002
-
[17]
Fyshe, L
A. Fyshe, L. Wehbe, P. P. Talukdar, B. Murphy, and T. M. Mitchell. A compositional and interpretable semantic space. In Proc. Conference of the North American Chapter of the ACL, pages 32--41. Association for Computational Linguistics, 2015
2015
-
[18]
G \"a rdenfors
P. G \"a rdenfors. Conceptual spaces: the geometry of thought. MIT Press, Cambridge, MA, 2000
2000
-
[19]
Geirhos, K
R. Geirhos, K. Narayanappa, B. Mitzkus, T. Thieringer, M. Bethge, F. A. Wichmann, and W. Brendel. Partial success in closing the gap between human and machine vision. Adv. Neural Inf. Process. Syst., 34: 0 23885--23899, 2021
2021
-
[20]
Geirhos, R
R. Geirhos, R. S. Zimmermann, B. Bilodeau, W. Brendel, and B. Kim. Don't trust your eyes: on the (un)reliability of feature visualizations. Proc. Mach. Learn. Res., 235: 0 15294--15330, 2024
2024
-
[21]
D. Gross. Recovering low-rank matrices from few coefficients in any basis. IEEE Trans. Inf. Theory, 57 0 (3): 0 1548--1566, 2011
2011
-
[22]
M. N. Hebart, A. H. Dickter, A. Kidder, W. Y. Kwok, A. Corriveau, C. Van Wicklin, and C. I. Baker. THINGS : A database of 1,854 object concepts and more than 26,000 naturalistic object images. PLoS One, 14: 0 e0223792, 2019
2019
-
[23]
M. N. Hebart, C. Y. Zheng, F. Pereira, and C. I. Baker. Revealing the multidimensional mental representations of natural objects underlying human similarity judgements. Nat. Hum. Behav., 4: 0 1173--1185, 2020
2020
-
[24]
M. N. Hebart, O. Contier, L. Teichmann, A. H. Rockter, C. Y. Zheng, A. Kidder, A. Corriveau, M. Vaziri-Pashkam, and C. I. Baker. THINGS -data, a multimodal collection of large-scale datasets for investigating object representations in human brain and behavior. eLife, 12: 0 e82580, 2023
2023
-
[25]
F. Hill, R. Reichart, and A. Korhonen. SimLex -999: Evaluating semantic models with (genuine) similarity estimation. Comput. Linguist., 41: 0 665--695, 2015
2015
-
[26]
Huben, H
R. Huben, H. Cunningham, L. R. Smith, A. Ewart, and L. Sharkey. Sparse autoencoders find highly interpretable features in language models. In Proc. International Conference on Learning Representations, 2024
2024
-
[27]
A. V. Jagadeesh and J. L. Gardner. Texture-like representation of objects in human visual cortex. Proc. Natl. Acad. Sci. U. S. A., 119: 0 e2115302119, 2022
2022
-
[28]
I. T. Jolliffe. Principal components in regression analysis. In Principal Component Analysis, pages 129--155. Springer, 1986
1986
-
[29]
E. L. Josephs, M. N. Hebart, and T. Konkle. Dimensions underlying human understanding of the reachable world. Cognition, 234: 0 105368, 2023
2023
-
[30]
Khosla, N
M. Khosla, N. A. Ratan Murty, and N. Kanwisher. A highly selective response to food in human visual cortex revealed by hypothesis-free voxel decomposition. Curr. Biol., 32 0 (19): 0 4159--4171.e9, 2022
2022
-
[31]
Khrulkov, L
V. Khrulkov, L. Mirvakhabova, E. Ustinova, I. Oseledets, and V. Lempitsky. Hyperbolic image embeddings. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6418--6428. IEEE, 2020
2020
-
[32]
Superposition is compressed sensing
D. Klindt, C. O'Neill, P. Reizinger, H. Maurer, and N. Miolane. From superposition to sparse codes: interpretable representations in neural networks. Preprint at arXiv, 2503.01824, 2025
-
[33]
Konkle and A
T. Konkle and A. Caramazza. Tripartite organization of the ventral stream by animacy and object size. J. Neurosci., 33 0 (25): 0 10235--10242, 2013
2013
-
[34]
Kornblith, M
S. Kornblith, M. Norouzi, H. Lee, and G. Hinton. Similarity of neural network representations revisited. Proc. Mach. Learn. Res., 97: 0 3519--3529, 2019
2019
-
[35]
Kriegeskorte and R
N. Kriegeskorte and R. A. Kievit. Representational geometry: integrating cognition, computation, and the brain. Trends Cogn. Sci., 17: 0 401--412, 2013
2013
-
[36]
Kriegeskorte and X.-X
N. Kriegeskorte and X.-X. Wei. Neural tuning and representational geometry. Nat. Rev. Neurosci., 22 0 (11): 0 703--718, 2021
2021
-
[37]
Kriegeskorte, M
N. Kriegeskorte, M. Mur, and P. Bandettini. Representational similarity analysis - connecting the branches of systems neuroscience. Front. Syst. Neurosci., 2: 0 4, 2008
2008
-
[38]
Kuang, C
D. Kuang, C. Ding, and H. Park. Symmetric nonnegative matrix factorization for graph clustering. In Proc. SIAM International Conference on Data Mining, pages 106--117. SIAM, 2012
2012
-
[39]
D. D. Lee and H. S. Seung. Learning the parts of objects by non-negative matrix factorization. Nature, 401: 0 788--791, 1999
1999
-
[40]
F. P. Mahner, L. Muttenthaler, U. Güçlü, and M. N. Hebart. Dimensions underlying the representational alignment of deep neural networks with humans. Nat. Mach. Intell., 7: 0 848--859, 2025
2025
-
[41]
F. P. Mahner, J. Roth, K. C. Lam, M. F. Bonner, F. Pereira, and M. N. Hebart. Characterizing universal object representations across vision models. Preprint at arXiv, 2605.13675, 2026
work page internal anchor Pith review Pith/arXiv arXiv 2026
-
[42]
N. J. Majaj, H. Hong, E. A. Solomon, and J. J. DiCarlo. Simple learned weighted sums of inferior temporal neuronal firing rates accurately predict human core object recognition performance. J. Neurosci., 35 0 (39): 0 13402--13418, 2015
2015
-
[43]
M. Mur, M. Meys, J. Bodurka, R. Goebel, P. A. Bandettini, and N. Kriegeskorte. Human object-similarity judgments reflect and transcend the primate- IT object representation. Front. Psychol., 4: 0 128, 2013
2013
-
[44]
Muttenthaler, K
L. Muttenthaler, K. Greff, F. Born, B. Spitzer, S. Kornblith, M. C. Mozer, K.-R. Müller, T. Unterthiner, and A. K. Lampinen. Aligning machine and human visual representations across abstraction levels. Nature, 647: 0 349--355, 2025
2025
-
[45]
Naselaris, K
T. Naselaris, K. N. Kay, S. Nishimoto, and J. L. Gallant. Encoding and decoding in fMRI . Neuroimage, 56: 0 400--410, 2011
2011
-
[46]
Nickel and D
M. Nickel and D. Kiela. Poincaré embeddings for learning hierarchical representations. Adv. Neural Inf. Process. Syst., 30: 0 6338--6347, 2017
2017
-
[47]
R. M. Nosofsky. Attention, similarity, and the identification-categorization relationship. J. Exp. Psychol. Gen., 115: 0 39--61, 1986
1986
-
[48]
C. Olah, A. Mordvintsev, and L. Schubert. Feature visualization. Distill, 2: 0 e7, 2017
2017
-
[49]
Papale, F
P. Papale, F. Wang, M. W. Self, and P. R. Roelfsema. An extensive dataset of spiking activity to reveal the syntax of the ventral stream. Neuron, 113: 0 539--553.e5, 2025
2025
-
[50]
J. C. Peterson, J. T. Abbott, and T. L. Griffiths. Evaluating (and improving) the correspondence between deep neural networks and human representations. Cogn. Sci., 42: 0 2648--2669, 2018
2018
-
[51]
Radford, J
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. Proc. Mach. Learn. Res., 139: 0 8748--8763, 2021
2021
-
[52]
B. D. Roads and B. C. Love. Modeling similarity and psychological space. Annu. Rev. Psychol., 75 0 (1): 0 215--240, 2024
2024
-
[53]
Schölkopf and A
B. Schölkopf and A. J. Smola. Learning with kernels: Support vector machines, regularization, optimization, and beyond. MIT Press, 2002
2002
-
[54]
H. H. Schütt, A. D. Kipnis, J. Diedrichsen, and N. Kriegeskorte. Statistical inference on representational geometries. eLife, 12: 0 e82566, 2023
2023
-
[55]
G. G. Scott, A. Keitel, M. Becirspahic, B. Yao, and S. C. Sereno. The glasgow norms: Ratings of 5,500 words on nine scales. Behav. Res. Methods, 51: 0 1258--1270, 2019
2019
-
[56]
R. N. Shepard. Stimulus and response generalization: a stochastic model relating generalization to distance in psychological space. Psychometrika, 22: 0 325--345, 1957
1957
-
[57]
R. N. Shepard. The analysis of proximities: Multidimensional scaling with an unknown distance function. I . Psychometrika, 27: 0 125--140, 1962
1962
-
[58]
Q. Shi, H. Sun, S. Lu, M. Hong, and M. Razaviyayn. Inexact block coordinate descent methods for symmetric nonnegative matrix factorization. IEEE Trans. Signal Process., 65: 0 5995--6008, 2017
2017
-
[59]
Sorscher, S
B. Sorscher, S. Ganguli, and H. Sompolinsky. Neural representational geometry underlies few-shot concept learning. Proc. Natl. Acad. Sci. U. S. A., 119 0 (43): 0 e2200800119, 2022
2022
-
[60]
L. M. Stoinski, J. Perkuhn, and M. N. Hebart. THINGSplus : new norms and metadata for the THINGS database of 1854 object concepts and 26,107 natural object images. Behav. Res. Methods, 56: 0 1583--1603, 2024
2024
-
[61]
Sucholutsky, L
I. Sucholutsky, L. Muttenthaler, A. Weller, A. Peng, A. Bobu, B. Kim, B. C. Love, C. J. Cueva, E. Grant, I. Groen, J. Achterberg, J. B. Tenenbaum, K. M. Collins, K. L. Hermann, K. Oktar, K. Greff, M. N. Hebart, N. Cloos, N. Kriegeskorte, N. Jacoby, Q. Zhang, R. Marjieh, R. Geirhos, S. Chen, S. Kornblith, S. Rane, T. Konkle, T. P. O'Connell, T. Unterthiner...
2025
-
[62]
Teichmann, M
L. Teichmann, M. N. Hebart, and C. I. Baker. Dynamic representation of multidimensional object properties in the human brain. J. Neurosci., 46: 0 e1057252026, 2026
2026
-
[63]
Thasarathan, J
H. Thasarathan, J. Forsyth, T. Fel, M. Kowal, and K. G. Derpanis. Universal sparse autoencoders: Interpretable cross-model concept alignment. Proc. Mach. Learn. Res., 267: 0 59304--59325, 2025
2025
-
[64]
W. S. Torgerson. Multidimensional scaling: I . theory and method. Psychometrika, 17: 0 401--419, 1952
1952
-
[65]
van Bree and M
S. van Bree and M. N. Hebart. Shared and distinct object spaces in human and macaque inferotemporal cortex. Preprint at bioRxiv, 2026.05.20.724014, 2026
2026
-
[66]
van der Maaten and G
L. van der Maaten and G. Hinton. Visualizing data using t- SNE . J. Mach. Learn. Res., 9: 0 2579--2605, 2008
2008
-
[67]
Zeng and J
A. Zeng and J. L. Gallant. Disentangling superpositions: interpretable brain encoding model with sparse concept atoms. Preprint at bioRxiv, 2025.11.29.691321, 2025
2025
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.