Generalizing Psychological Similarity Spaces to Unseen Stimuli

Kai-Uwe K\"uhnberger; Lucas Bechberger

REVIEW 3 major objections 5 minor 12 references

Generalizing Psychological Similarity Spaces to Unseen Stimuli

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A simple regression can carry a human similarity space to unseen images

desk verdict A candid feasibility study with a real load-bearing assumption: the augmentation labels make the psychological generalization claim shakier than the R2 numbers suggest, but the paper is honest, reproducible, and deserves a serious referee. read the letter →

arxiv 1908.09260 v2 pith:HNRJDJLU submitted 2019-08-25 cs.LG stat.ML

Lucas Bechberger , Kai-Uwe Kühnberger This is my paper · ORCID

classification cs.LGstat.ML

keywords multidimensionalscalingpsychologicalsimilarityspaceconceptualspacesspatialarrangementmethodneuralnetworkregressiondataaugmentationgeneralizationtounseenstimuliNOUNdataset

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that a psychological similarity space, built from human dissimilarity ratings for a small fixed set of images, can be extended to new images by learning a mapping from image features into that space. On the 64-image NOUN set, the authors fit linear and lasso regressors from the activation vectors of a pretrained convolutional network into MDS-derived target spaces, using 1,000 augmented copies per image to enlarge the training data. The regressors beat a zero baseline on the real targets and fail when the target coordinates are shuffled, leading the authors to conclude that the semantic structure of the similarity space is what makes generalization possible. A separate comparison shows metric and nonmetric MDS produce nearly indistinguishable spaces for SpAM dissimilarities on this data, so the stronger ratio-scale assumption buys nothing. The overall verdict is that the hybrid approach works in principle, with best lasso $R^2\approx0.42$ in four dimensions and $R^2\approx0.52$ in two, but with strong overfitting.

What carries the argument

The load-bearing machinery is a two-stage pipeline. First, human dissimilarity ratings from the Spatial Arrangement Method are converted into point coordinates by minimizing MDS stress, here with the SMACOF algorithm; stress measures how far Euclidean distances $d_{ij}$ are from a (linear or monotone) transformation of dissimilarities $\delta_{ij}$. Second, those coordinates become regression targets: a linear or lasso model maps the second-to-last-layer activations of a pretrained convolutional network to points in the same space, so new images receive coordinates by interpolation. Data augmentation expands the 64 labeled images to 64,000, with all variants of one image sharing that image's target point; shuffled-target controls in the same setup show that exploiting the semantic layout of the space, not memorization, is what transfers to unseen originals.

What would settle it

Collect human dissimilarity ratings between original NOUN images and their augmented variants; if heavily blurred, cropped, or noised versions are judged about as dissimilar to their source as to other originals, then assigning each augmented image the source's MDS coordinates is wrong and the reported generalization is not a clean estimate about psychological similarity. A simpler check: train only on the 64 original images without augmentation and see whether the regression still beats the baseline; if it does not, the positive result depends entirely on the augmentation assumption.

Watch

Extended reading notes

Core claim

The central claim is that a simple linear mapping from a pretrained network's image features can carry an MDS-derived psychological similarity space to unseen stimuli, and that this works because the target space has semantic geometry rather than arbitrary coordinates. With genuine targets, linear regression on 2048-dimensional Inception-v3 activations reaches test $R^2\approx0.37$ in the four-dimensional NOUN space and lasso reaches $R^2\approx0.42$; in two dimensions the lasso reaches $R^2\approx0.52$, the best configuration tested. With shuffled target coordinates the same regression yields negative $R^2$ in the four-dimensional case, which the authors take as evidence that the correct mapping is learnable while a structureless mapping is not. The paper further claims that on the NOUN data metric and nonmetric MDS give comparable solutions under both metric and nonmetric stress, so the additional ratio-scale assumption of metric MDS is neither beneficial nor harmful.

Load-bearing premise

Every augmented copy of an image is labeled with that image's human-rated similarity coordinates, so the whole generalization result assumes random crops, blur, noise, affine changes, and contrast or brightness edits leave the object's identity and its psychological similarity to other objects unchanged.

Editorial extensions

If this is right

A small human rating experiment plus a pretrained network is enough to place a much larger image collection in a psychological similarity space; no additional human judgments are needed for new images.
Because shuffled targets fail to generalize, any target space claimed to be psychological must show this semantic-structure effect; a regression that performs equally on shuffled coordinates is not capturing human similarity.
Target-space dimensionality is a real design choice: two dimensions outperformed all others here, larger spaces increased overfitting, and one dimension was too poor to beat baseline, so deployments should tune dimension empirically.
Metric versus nonmetric MDS is not a critical choice on SpAM-style data; nonmetric MDS is the safer default because it achieves the same results with weaker assumptions.
The persistent overfitting with only 64 unique targets points to the number of distinct stimuli, not the regressor class, as the limiting factor; the comparison with the 360-image study suggests larger rating sets should sharply improve generalization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct test of the augmentation assumption would be to collect human ratings between original NOUN images and their augmented variants; if transformed versions are not reliably closer to their source than to other originals, the current labels are noisy and $R^2$ overstates true psychological generalization.
The two-dimensional optimum suggests the NOUN similarity domain may have intrinsically low dimensionality; comparing stress and regression performance across other stimulus sets would separate a property of the domain from an artifact of the method.
Residuals of the trained regression identify images whose predicted similarity coordinates deviate most from human placements; those residuals could guide which new stimulus pairs most need additional human ratings.
An end-to-end variant that fine-tunes the network against the similarity-space objective, rather than freezing features, is a natural next step; the paper's overfitting analysis implies much of the remaining gap is data-driven, so this extension is testable on the same NOUN space.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing to know: this is a pair of honest feasibility studies, not a breakthrough. The genuinely new bits are the metric-versus-nonmetric MDS comparison on the NOUN SpAM data and the target-dimensionality sweep in the regression study. The core hybrid idea—regressing CNN features into MDS-derived spaces—was already in the authors' 2018 workshop paper and is close to Sanders and Nosofsky; the paper is upfront about that.

Credit where due: the writing is plain about overfitting, the shuffled-target control is a good check, and the code is public. The qualitative conclusions hold up: ANN features beat pixels and the zero baseline, shuffled targets fail, and two-dimensional targets give the best regression. That is all plausible and supported by the tables.

The stress-test concern about augmentation labels is the real soft spot. Every augmented version of a NOUN image inherits the original's MDS coordinates. For familiar objects, random crops and affine shears mostly preserve identity, but NOUN stimuli are deliberately novel; a crop that removes a salient part or a shear that changes shape can change perceived similarity. If that happens often, the regression targets are partially pseudo-labels, and R2≈0.42/0.49 measures learnability of the assigned mapping rather than clean psychological generalization. The shuffled-target control does not rescue the claim because it preserves the augmentation-to-original assignment scheme. The headline claim should be softened to "conditional on augmentation preserving similarity."

Two smaller issues. Beta is selected on the test set, so the reported lasso numbers are optimistic; nested validation would tighten them. There are no error bars or repeated MDS/regression runs, so the two-dimensional optimum could be noise. The best distance function in Study 1 is chosen post hoc, which is normal in that literature but still worth noting. None of these are fatal; they make the paper a solid proof-of-concept rather than a strong quantitative claim.

The paper deserves a serious referee. It is not high-impact, but it is honest, connected to prior work, and ships reproducible code. A referee should ask for either a small human-rating check of the augmentation invariance or an explicit reframing of the claim as mapping to identity-based pseudo-coordinates.

Referee Report

3 major / 5 minor

Summary. The paper proposes a hybrid approach for generalizing psychological similarity spaces to unseen stimuli: build a similarity space via multidimensional scaling (MDS) from human dissimilarity ratings, then train a regression model from image features to that space. The first study compares metric and nonmetric MDS on the NOUN data set of 64 novel objects, finding that the two MDS variants produce comparable solutions and concluding that the ratio-scale assumption of metric MDS is neither beneficial nor harmful. The second study regresses from pixel features or pretrained Inception-v3 activations into MDS-derived target spaces, using data augmentation to expand the 64 images to 64,000. The main reported results are that ANN-based features beat pixel features, linear and lasso regressions beat a zero baseline for correct targets, shuffled targets are much harder to learn, and a two-dimensional target space yields the best regression performance (linear R2≈0.49, lasso R2≈0.52). The paper is framed as a feasibility study, with the authors acknowledging strong overfitting and modest performance.

Significance. If the results hold, the paper makes a useful contribution by demonstrating a concrete pipeline that maps raw stimuli into MDS-derived psychological spaces, a step toward scaling conceptual-space representations beyond the fixed stimulus sets used in psychological experiments. The paper has real strengths: it uses an independent human-rated similarity target from the NOUN data set, it includes a shuffled-target control that shows semantic structure is learnable, it compares ANN features with pixel features, and it provides code for reproduction. The qualitative conclusions—ANN features help, shuffled targets do not generalize, two-dimensional targets are easiest—are supported by the reported numbers. However, the quantitative claims are weakened by two load-bearing methodological issues: the validity of the augmentation-based labels and the selection of the regularization parameter on the test set. The contribution is therefore promising but needs strengthening or reframing before publication.

major comments (3)

[Section 5.1, Tables 1 and 3] The central generalization claim rests on assigning the MDS coordinates of each original NOUN image to all 1,000 augmented versions of that image. This assumes that random crops, affine transformations, blur, noise, and contrast/brightness changes preserve the psychological similarity of the stimulus. For novel, unfamiliar objects this is a strong assumption: a crop that removes a salient part or a shear that changes the overall shape can plausibly alter perceived similarity. Because each test fold in the eight-fold cross-validation consists of augmented versions of held-out original identities, the reported R2 values could reflect learning an augmentation-invariant mapping to pseudo-labels rather than generalizing a psychological similarity space. The shuffled-target control does not resolve this issue, since the shuffled condition uses the same augmentation-to-coordinate assignment scheme. Please provide evidence that human similarity ratings are stable under the applied augmentations, or re-evaluate the model on the 64 original images only and discuss how the results change.
[Section 5.1 and Section 5.2, Table 1] The lasso regularization parameter β appears to be selected on the test set. Section 5.2 states that 'The best results on the test set are achieved for β = 0.01', and the tables report 'Test Set Performance' for each β without describing any inner validation loop. With 14 candidate β values and a single eight-fold split, selecting β on the test data optimistically biases the reported R2 values. This is load-bearing for the quantitative claim that lasso improves over linear regression and that R2≈0.42 in the four-dimensional space. Please use nested cross-validation or a pre-specified β, and report the selection procedure explicitly.
[Section 5.1 and Table 3] All experiments appear to use a single augmentation realization and a single eight-fold split, with no confidence intervals or significance tests. The paper draws conclusions from relatively small differences—for example, the comparison between metric and nonmetric MDS target spaces and the ranking of target dimensionalities in Experiment 3. Without repeated cross-validation with different augmentation seeds and split seeds, the reader cannot assess whether these differences are stable or due to chance. Please add error bars or repeated evaluations to support the qualitative conclusions.

minor comments (5)

[Section 2.1] The text says 'each single modification by the user updates N distance values', but moving one stimulus changes its distance to the other N−1 stimuli, so it updates N−1 distance values. Please correct this.
[Section 5.1, Tables 1 and 3] The zero baseline yields MSE = 1.0000 and R2 = 0.0000 for every target space, which suggests that the target coordinates are normalized, but this normalization is not described. Please state explicitly how the target coordinates were scaled or standardized.
[Tables 1–3] The reported 'degree of overfitting' for R2 is computed by dividing training R2 by test R2, which is not meaningful when R2 is negative or close to zero. Please report the difference between training and test R2 instead.
[Section 5.1] There is a typo: 'Secton 3.3' should be 'Section 3.3'.
[References] The code repository link in the introduction contains a trailing '(?)' and the software reference to Bechberger (2019) is incomplete; please provide the full URL and version information.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: regression targets come from independent human dissimilarity ratings, features from a pretrained network, and the shuffled-target control keeps the central mapping non-tautological.

full rationale

The paper's derivation chain is self-contained against external data: the MDS target spaces are derived from human dissimilarity ratings (Horst and Hout 2016, or SMACOF fits to the same NOUN ratings), while the regression features are activation vectors from a pretrained Inception-v3 network. The mapping from features to MDS coordinates is therefore not derived from the quantity it predicts. The augmented images inherit the coordinates of their source image, but this is an empirical invariance assumption, not a circular reduction: test folds consist of augmentations of held-out original identities, and the shuffled-target control preserves the same augmentation-to-target assignment scheme, so it tests whether the assigned mapping is learnable beyond an arbitrary permutation. The comparison in Section 3.3 showing that MDS distances reflect dissimilarity ratings better than image-based baselines is explicitly acknowledged by the authors as 'no surprise' because MDS is directly based on the dissimilarity ratings; it is a sanity check, not a hidden prediction. The only self-citation, to Bechberger and Kypridemou (2018), is disclosed as provenance for an updated and extended version and is not load-bearing for any of the paper's conclusions. No equation or definition in the paper reduces a predicted quantity to its own input, and no fitted parameter is renamed as a prediction. The central claims remain conditional on the augmentation-preserves-similarity assumption, but that is a validity concern, not circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central regression result depends on two tuned parameters, beta and target dimensionality t, plus a distance-weight optimization in the first study. The axioms are standard MDS mathematics plus domain assumptions about SpAM scale, pretrained features, and augmentation preserving identity. No new theoretical entities are introduced. The largest burden is beta tuning on test data.

free parameters (3)

lasso regularization beta = 0.01 for ANN features in 4D; 0.2 and 0.5 for pixel features; varies with target space in Table 3
Selected by comparing test-set performance across a grid of 14 values, so the reported R2 values are post-selection and likely optimistic.
target dimensionality t = 2 chosen as the best regression target; 4 used for the main comparison
The paper varies t from 1 to 10 and selects 2 as optimal on the same evaluation data, making the headline comparison a model-selection result rather than a parameter-free prediction.
per-dimension distance weights for baselines = not reported; estimated by non-negative least squares in five-fold cross-validation
Used in Study 1 to compute baseline correlations; the weights are fit to the dissimilarity data, so the baseline comparisons are partially optimized.

assumptions (5)

standard math SMACOF iteratively maximizes a majorizing function and converges monotonically to a local stress minimum.
Invoked in Section 2.2 to justify using the smacof package; standard results from de Leeuw (1977) and Borg and Groenen (2005).
domain assumption SpAM dissimilarity ratings for the NOUN stimuli are at least ordinally valid as psychological dissimilarities.
The whole MDS extraction in Section 3 rests on this; the paper explicitly leaves open whether the ratings are ratio scaled.
domain assumption Inception-v3 second-to-last layer activations contain visual features relevant to judging novel object similarity.
Used as the only ANN feature set in Section 5.1 without an independent check that these features align with the psychological dimensions.
domain assumption Random augmentation preserves each NOUN object's identity and psychological similarity, so augmented copies can inherit the original MDS coordinate.
Assumed in Section 5.1 when assigning targets to 1,000 augmented versions; this is load-bearing for the generalization measurement.
domain assumption The four-dimensional Horst and Hout (2016) MDS space is a valid psychological target space for the regression.
Serves as the reference target in Experiment 1 without independent validation beyond the original publication.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Generalizing Psychological Similarity Spaces to Unseen Stimuli." pith.science (2026). https://pith.science/paper/HNRJDJLU

@misc{pith2026190809260,
  author       = {Pith},
  title        = {Pith review of: Generalizing Psychological Similarity Spaces to Unseen Stimuli},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HNRJDJLU}},
  note         = {Machine review of arXiv:1908.09260}
}

read the original abstract

The cognitive framework of conceptual spaces proposes to represent concepts as regions in psychological similarity spaces. These similarity spaces are typically obtained through multidimensional scaling (MDS), which converts human dissimilarity ratings for a fixed set of stimuli into a spatial representation. One can distinguish metric MDS (which assumes that the dissimilarity ratings are interval or ratio scaled) from nonmetric MDS (which only assumes an ordinal scale). In our first study, we show that despite its additional assumptions, metric MDS does not necessarily yield better solutions than nonmetric MDS. In this chapter, we furthermore propose to learn a mapping from raw stimuli into the similarity space using artificial neural networks (ANNs) in order to generalize the similarity space to unseen inputs. In our second study, we show that a linear regression from the activation vectors of a convolutional ANN to similarity spaces obtained by MDS can be successful and that the results are sensitive to the number of dimensions of the similarity space.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 12 canonical work pages

[1]

Bechberger, L. (2019). lbechberger/LearningPsychologicalSpaces: Study on Multi- dimensional Scaling and Neural Networks on the NOUN Dataset

work page 2019
[2]

and Kypridemou, E

Bechberger, L. and Kypridemou, E. (2018). Mapping Images to Psychological SimilaritySpacesUsingNeuralNetworks. In Proceedingsofthe6thInternational Workshop on Artiﬁcial Intelligence and Cognition

work page 2018
[3]

and Groenen, J

Borg, I. and Groenen, J. F. (2005).Modern Multidimensional Scaling: Theory and Applications. Springer Series in Statistics. Springer-Verlag New York, 2 edition. de Leeuw, J. (1977). Recent Development in Statistics, chapter Applications of Convex Analysis to Multidimensional Scaling, pages 133–146. North Holland Publishing. Gärdenfors, P. (2000).Conceptua...

work page 2005
[4]

Horst, J. S. and Hout, M. C. (2016). The Novel Object and Unusual Name (NOUN) Database: A Collection of Novel Images for Use in Experimental Research.Be- havior Research Methods, 48(4):1393–1409

work page 2016
[5]

C., Goldinger, S

Hout, M. C., Goldinger, S. D., and Brady, K. J. (2014). MM-MDS: A Multidimen- sional Scaling Database with Similarity Ratings for 240 Object Categories from the Massive Memory Picture Database.PLOS ONE, 9(11):1–11

work page 2014
[6]

C., Goldinger, S

Hout, M. C., Goldinger, S. D., and Ferguson, R. W. (2013). The Versatility of SpAM:AFast,Eﬃcient,SpatialMethodofDataCollectionforMultidimensional Scaling. Journal of Experimental Psychology: General, 142(1):256. Kruskal,J.B.(1964a). MultidimensionalScalingbyOptimizingGoodnessofFitto a Nonmetric Hypothesis.Psychometrika, 29(1):1–27. Kruskal,J.B.(1964b). No...

work page 2013
[7]

C., Abbott, J

Peterson, J. C., Abbott, J. T., and Griﬃths, T. L. (2017). Adapting Deep Net- work Features to Capture Psychological Representations: An Abridged Report. In Proceedings of the Twenty-Sixth International Joint Conference on Artiﬁcial

work page 2017
[8]

Peterson,J.C.,Abbott,J.T.,andGriﬃths,T.L.(2018)

Intelligence, IJCAI-17, pages 4934–4938. Peterson,J.C.,Abbott,J.T.,andGriﬃths,T.L.(2018). Evaluating(andImproving) theCorrespondenceBetweenDeepNeuralNetworksandHumanRepresentations. Cognitive Science, 42(8):2648–2669

work page 2018

Show all 12 references

[9]

Sanders, C. A. and Nosofsky, R. M. (2018). Using Deep-Learning Representations ofComplexNaturalStimuliasInputtoPsychologicalModelsofClassiﬁcation. In Proceedings of the 2018 Conference of the Cognitive Science Society, Madison

2018
[10]

Spearman, C. (1904). The Proof and Measurement of Association between Two Things. The American Journal of Psychology, 15(1):72–101

1904
[11]

Szegedy, C., Vanhoucke, V., Ioﬀe, S., Shlens, J., and Wojna, Z. (2016). Rethinking the Inception Architecture for Computer Vision. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2818–2826

2016
[12]

Wickelmaier, F. (2003). An Introduction to MDS.Sound Quality Research Unit, Aalborg University, Denmark, 46(5)

2003

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Bechberger, L. (2019). lbechberger/LearningPsychologicalSpaces: Study on Multi- dimensional Scaling and Neural Networks on the NOUN Dataset

work page 2019

[2] [2]

and Kypridemou, E

Bechberger, L. and Kypridemou, E. (2018). Mapping Images to Psychological SimilaritySpacesUsingNeuralNetworks. In Proceedingsofthe6thInternational Workshop on Artiﬁcial Intelligence and Cognition

work page 2018

[3] [3]

and Groenen, J

Borg, I. and Groenen, J. F. (2005).Modern Multidimensional Scaling: Theory and Applications. Springer Series in Statistics. Springer-Verlag New York, 2 edition. de Leeuw, J. (1977). Recent Development in Statistics, chapter Applications of Convex Analysis to Multidimensional Scaling, pages 133–146. North Holland Publishing. Gärdenfors, P. (2000).Conceptua...

work page 2005

[4] [4]

Horst, J. S. and Hout, M. C. (2016). The Novel Object and Unusual Name (NOUN) Database: A Collection of Novel Images for Use in Experimental Research.Be- havior Research Methods, 48(4):1393–1409

work page 2016

[5] [5]

C., Goldinger, S

Hout, M. C., Goldinger, S. D., and Brady, K. J. (2014). MM-MDS: A Multidimen- sional Scaling Database with Similarity Ratings for 240 Object Categories from the Massive Memory Picture Database.PLOS ONE, 9(11):1–11

work page 2014

[6] [6]

C., Goldinger, S

Hout, M. C., Goldinger, S. D., and Ferguson, R. W. (2013). The Versatility of SpAM:AFast,Eﬃcient,SpatialMethodofDataCollectionforMultidimensional Scaling. Journal of Experimental Psychology: General, 142(1):256. Kruskal,J.B.(1964a). MultidimensionalScalingbyOptimizingGoodnessofFitto a Nonmetric Hypothesis.Psychometrika, 29(1):1–27. Kruskal,J.B.(1964b). No...

work page 2013

[7] [7]

C., Abbott, J

Peterson, J. C., Abbott, J. T., and Griﬃths, T. L. (2017). Adapting Deep Net- work Features to Capture Psychological Representations: An Abridged Report. In Proceedings of the Twenty-Sixth International Joint Conference on Artiﬁcial

work page 2017

[8] [8]

Peterson,J.C.,Abbott,J.T.,andGriﬃths,T.L.(2018)

Intelligence, IJCAI-17, pages 4934–4938. Peterson,J.C.,Abbott,J.T.,andGriﬃths,T.L.(2018). Evaluating(andImproving) theCorrespondenceBetweenDeepNeuralNetworksandHumanRepresentations. Cognitive Science, 42(8):2648–2669

work page 2018