Pith. sign in

REVIEW 4 major objections 5 minor 52 references

How do Transformer Embeddings Represent Compositions? A Functional Analysis

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Transformer embedding models build compound-word representations from their parts in a largely linear way, and BERT is the outlier.

desk verdict A useful empirical benchmark showing modern embedding APIs behave linearly for compound words, but the 'architecture vs. data' conclusion overreaches the evidence. read the letter →

arxiv 2506.00914 v1 pith:6XLFGXXM submitted 2025-06-01 cs.CL cs.AI

Aishik Nagar , Ishaan Singh Rawal , Mansi Dhanania , Cheston Tan This is my paper · ORCID
classification cs.CLcs.AI
keywords compositionalitytransformerembeddingscompoundwordsvectoradditionridgeregressionBERTembeddingarithmeticsemantictransparency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether transformer-based embedding models build a compound word's representation from the representations of its parts, and by what function. Across six candidate functions, ridge regression — a linear model — best predicts compound embeddings from constituent embeddings, and plain vector addition is almost as accurate. The linear behavior also appears for novel compounds the models have likely never seen, which the authors take as evidence that compositionality is a property of the model architecture rather than of memorized training data. The notable outlier is BERT, whose embeddings are far less compositional under the same tests.

What carries the argument

The central object is a composition function $f(u,v)=\hat{w}$ that maps two constituent embeddings $u$ and $v$ to a predicted embedding $\hat{w}$ for the compound. The paper evaluates six concrete functions — simple addition, weighted addition, element-wise multiplication, dilation, ridge regression, and a two-layer MLP — and measures each by the cosine similarity between $\hat{w}$ and the true compound embedding $w$, benchmarked by Jensen-Shannon divergence between that similarity distribution and a random-pair baseline. Ridge regression on the concatenated pair is the best linear fit, and the near-tie with simple addition is what carries the paper's claim that composite embeddings are approximately additive in the embedding space.

What would settle it

Searching the pretraining corpora of Google, Mistral, and OpenAI embeddings for any of the LADEC-NC compounds (e.g., 'zoodough' or 'hemnut') would settle the architectural claim: if such compounds occur, the equal LADEC vs. LADEC-NC performance no longer demonstrates out-of-distribution compositionality.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that modern transformer embedding models are largely compositional, and that their composition is linear. Testing six composition functions on the LADEC compound database, the authors find that ridge regression on the concatenated constituent embeddings yields the highest cosine similarity to actual compound embeddings and the largest separation from a random-pair baseline, followed closely by simple addition, weighted addition, dilation, and an MLP; element-wise multiplication fails. Performance on novel, synthetically generated compounds is nearly identical to performance on existing compounds, so the authors conclude that the capacity to compose constituents linearly is built into the architecture rather than learned from encountering specific compounds. BERT is the exception: every composition model separates poorly from baseline on its embeddings, a difference the paper attributes to BERT's bidirectional masking objective, subword tokenization, and scale.

Load-bearing premise

The claim that compositionality is architectural rather than learned assumes that the novel compounds in the LADEC-NC set never appeared in the models' training corpora, but the paper does not verify this.

Editorial extensions

If this is right

  • New compounds that a model has never seen can be predicted from their parts, so systems built on these embeddings — retrieval, analogy, RAG — can in principle compose unseen terms without retraining.
  • Because a linear function beats a non-linear MLP at predicting compound embeddings, the end-to-end mapping from constituents to compound appears nearly linear despite the transformer's internal nonlinearity.
  • BERT's poor compositionality implies that architecture and training objective, not model scale alone, determine whether an embedding space supports linear composition.
  • The correlation between embedding-compositionality scores and human semantic-transparency ratings suggests these scores can serve as an automatic proxy for how transparent a compound is.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The architectural conclusion rests on the untested assumption that the LADEC-NC compounds never appeared in the models' pretraining corpora; auditing those corpora (or testing on truly held-out compounds with dated first occurrence) would settle it.
  • The near-tie between ridge regression and addition suggests the additional learned weights in ridge regression may be capturing little beyond the average of the two constituents; an ablation that compares ridge on $[u,v]$ with ridge on $u+v$ would quantify the true gain from concatenation.
  • The BERT gap might be driven by its WordPiece subword tokenization fragmenting compounds, rather than by bidirectionality per se; a transformer with whole-word tokenization but the same MLM objective would isolate the cause.
  • If linear composition is a general property, then tasks like few-shot concept learning or compositional generalization benchmarks could be re-expressed as simple embedding arithmetic, making them transparent and easy to debug.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper tests six composition functions (simple addition, weighted addition, multiplication, dilation, ridge regression, and an MLP) for predicting transformer embedding vectors of compound words from embeddings of their constituents, across four embedding models (Google, Mistral, OpenAI Large, and BERT). It uses the LADEC compound dataset, a newly constructed LADEC-NC set of supposedly novel compounds, and a synthetic adjective-noun SANC dataset, evaluating with cosine similarity and Jensen-Shannon divergence against a random-pair baseline. The paper reports that ridge regression best accounts for compound embeddings, that simple addition performs almost as well, that modern embedding models are far more compositional than BERT, and that similar performance on LADEC and LADEC-NC implies compositionality arises from architecture rather than training data.

Significance. If the empirical results hold, the paper offers useful evidence that several widely used embedding spaces are approximately linearly compositional and that simple addition is a strong, parameter-free baseline. The SANC dataset is a transparent resource, and the correlations with human semantic transparency ratings in Appendix A.8 provide an independent sanity check. However, the headline architecture-versus-data conclusion is not established by the experimental design, and several of the quantitative claims rest on very small metric differences without uncertainty quantification.

major comments (4)
  1. [Section 3.3.4, Table 2, Section 8] The conclusion that compositionality 'arises from the architecture of embedding models, rather than their training data or recipe' is not supported by the LADEC versus LADEC-NC comparison. Matching prediction quality on novel surface forms demonstrates generalization beyond memorizing those exact compounds, but it does not distinguish an architectural origin from a learned compositional rule that transfers to unseen strings. The paper's own Section 8 states that the role of specific training data was not analyzed, which directly contradicts the Section 3.3.4 claim. In addition, LADEC-NC novelty was checked only against existing dictionary words, not against the training corpora of the models, and for the proprietary APIs this cannot be verified.
  2. [Section 3.2.4 and Table 1] The central ranking of composition models rests on extremely small JS divergence differences without any uncertainty quantification. For example, Google gives 0.824 for simple addition versus 0.829 for ridge regression, and Mistral gives 0.829 versus 0.831; these differences are far smaller than sampling noise would plausibly be. The claim that ridge regression 'best accounts for compositionality' should be supported by bootstrap confidence intervals, per-item variance, or significance tests. Additionally, JS divergence measures separation from the random baseline rather than direct prediction accuracy, so the reported cosine similarities and JS divergences should be interpreted together with this caveat.
  3. [Section 3.1.2 and Section 3.3.2] The evaluation protocol for ridge regression is underspecified. Unlike the MLP, which is described with an 80-20 train-test split, ridge regression is described only as regressing target values against concatenated constituent embeddings, with no stated held-out split. If ridge coefficients are fit and evaluated on the same compounds, its comparison with the parameter-free simple addition model is an in-sample comparison and is not fair. The paper should report held-out evaluation for ridge regression and clarify whether the Table 1 MLP numbers are test-set numbers.
  4. [Section 3.3.1, Table 1, and Appendix B] The comparison between BERT and the modern models is confounded by multiple factors. BERT embeddings are extracted from the CLS token of layer 12, while Google, Mistral, and OpenAI embeddings are obtained through proprietary APIs, and the models differ in scale, tokenization, and training data. Therefore the claim that BERT's poorer compositionality is due to architectural choices such as bidirectionality, masked language modeling, or subword tokenization is speculative; Appendix B acknowledges several alternative explanations, including scale and training data. This should be presented as an observation about these specific systems, not as evidence about architecture.
minor comments (5)
  1. [Section 3.3.3] The text refers to 'NLP' where 'MLP' is clearly intended; this should be corrected.
  2. [Table 1] The reason for reporting 'n/a' for multiplication JS divergence is not explained. Since multiplication produces negative mean cosine similarities, the interaction between the metric and the baseline distribution should be discussed.
  3. [Appendix A.10 and Table 11] The additional models (OpenAI Small, CLIP, FastText, Comp-CLIP) are introduced without model versions, extraction details, or a description of how their embeddings were obtained, which limits reproducibility.
  4. [Section 2 and Appendix A.2] The paper mentions a code submission but does not provide a URL, and several proprietary models are referenced without API model identifiers or access dates; this makes exact replication difficult.
  5. [Appendix A.7] The grid-search results in Table 4 report 'Cos. Dist.' while the main text uses cosine similarity; the sign convention should be stated to avoid confusion.

Circularity Check

2 steps flagged · score 6.0 of 10

Ridge 'best predictor' ranking and the LADEC-NC learned-model metrics are in-sample fits; parameter-free simple addition provides independent support, so circularity is partial.

  1. fitted input called prediction [Section 3.1.2 (Methods), Section 3.3.2 (Results), Table 1]
    "Ridge Regression: The target values w are regressed against the concatenation of u and v (Baroni and Zamparelli, 2010; Vecchi et al., 2017), with parameter α = 1.0. ... From Table 1, for all embedding models, Ridge Regression results in the highest JS Divergence values (by a small margin) and highest cosine similarities. ... The train-test split was 80-20."

    Ridge regression selects its coefficient matrix by minimizing regression error to the same target embeddings w that are later scored in Table 1; the paper states no held-out split for ridge (the 80-20 split sentence appears only in the MLP bullet). Ranking ridge as 'best predicts compound embeddings' on the fitting data is therefore an in-sample goodness-of-fit, not a prediction: the high cosine/JS values are forced by the least-squares objective rather than by out-of-sample compositional structure.

  2. fitted input called prediction [Section 3.3.4, Table 2, Appendix A.6]
    "From Table 2, the metrics for LADEC versus LADEC-NC are very similar. This means that embeddings of never-seen-before compounds (e.g. 'zoodough') can be predicted from the embeddings of their constituents ('zoo' and 'dough') to the same extent as existing compounds. The results suggest that compositionality arises from the architecture of embedding models, rather than their training data or recipe. ... Mistral Novel Compound Words: Weights = [4.3603,3.7515]"

    Appendix A.6 shows additive/dilation models were re-trained on the novel set ('Mistral Novel Compound Words'), so their Table 2 LADEC-NC metrics are fits on the target set. For ridge and MLP, Table 2 reports LADEC-NC numbers without stating a train-on-LADEC/evaluate-on-LADEC-NC protocol, and the MLP split is on the same dataset; thus the 'can be predicted' language for learned models is either a refit or unshown. Only parameter-free simple addition is a genuine out-of-sample predictor, so the architecture-vs-training-data conclusion is not established by the learned-model comparisons.

full rationale

Most of the paper is an empirical comparison rather than a derivation, and no load-bearing self-citation chain was found (Rawal et al. 2024 is cited only for the general point that benchmark scores can be misleading). The core observation that vector addition comes close to ridge regression is parameter-free and thus not circular. The circularity is partial and located in two 'prediction' steps: ridge regression is declared the best predictor from what appears to be its own training data, since the only stated train-test split (80-20) appears in the MLP bullet and no held-out protocol is given for ridge; and the LADEC-NC comparison re-fits learned composition models on the novel set (Appendix A.6) before reporting near-identical metrics, so those metrics are fits rather than predictions of never-seen compounds. The paper's Section 8 limitation ('We do not analyze the role of specific training data in enabling compositionality') further concedes the training-data question, but that is a validity gap, not an additional by-construction circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on several fitted parameters (weighted addition, dilation, ridge alpha, MLP architecture) and on domain assumptions about the validity of cosine similarity and JS divergence as compositionality measures. The most fragile premise is the unverified novelty and transparency of LADEC-NC compounds, which underpins the architecture-vs-data conclusion.

free parameters (4)
  • Weighted Addition weights (alpha, beta) = Google [4.068, 2.992]; Mistral [5.331, 3.991]; OpenAI [4.384, 3.998]; BERT [5.088, 5.524]
    Learned via gradient descent on LADEC and used to evaluate compositionality.
  • Dilation parameter lambda = Google 3.106; Mistral 2.984; OpenAI 4.538; BERT 2.061
    Learned via gradient descent on LADEC.
  • Ridge regression regularization alpha = 1.0
    Set by hand, not tuned; affects the ridge fit.
  • MLP hidden layer sizes = 256 and 128
    Chosen by hand; MLP trained with 80-20 split and cosine embedding loss.
assumptions (4)
  • domain assumption Cosine similarity is a valid measure of semantic similarity between embeddings.
    Used to evaluate predicted vs actual compound embeddings (Section 3.2.4).
  • domain assumption JS divergence between the predicted-similarity distribution and a random baseline measures how well a composition model captures semantic composition.
    Used as the primary compositionality metric (Section 3.2.4).
  • ad hoc to paper The LADEC-NC novel compounds are semantically transparent and were unseen during model training.
    Needed to support the architecture-vs-data conclusion (Section 3.2.2 and 3.3.4).
  • domain assumption Constituent embeddings (u and v) are obtained without context and represent the words' meanings as used in the compound.
    The paper does not specify how constituent embeddings are extracted for proprietary models; this affects all composition functions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of How do Transformer Embeddings Represent Compositions? A Functional Analysis." pith.science (2026). https://pith.science/paper/6XLFGXXM

@misc{pith2026250600914,
  author       = {Pith},
  title        = {Pith review of: How do Transformer Embeddings Represent Compositions? A Functional Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6XLFGXXM}},
  note         = {Machine review of arXiv:2506.00914}
}
read the original abstract

Compositionality is a key aspect of human intelligence, essential for reasoning and generalization. While transformer-based models have become the de facto standard for many language modeling tasks, little is known about how they represent compound words, and whether these representations are compositional. In this study, we test compositionality in Mistral, OpenAI Large, and Google embedding models, and compare them with BERT. First, we evaluate compositionality in the representations by examining six diverse models of compositionality (addition, multiplication, dilation, regression, etc.). We find that ridge regression, albeit linear, best accounts for compositionality. Surprisingly, we find that the classic vector addition model performs almost as well as any other model. Next, we verify that most embedding models are highly compositional, while BERT shows much poorer compositionality. We verify and visualize our findings with a synthetic dataset consisting of fully transparent adjective-noun compositions. Overall, we present a thorough investigation of compositionality.

Figures

Figures reproduced from arXiv: 2506.00914 by the authors.

Figure 1
Figure 1. Linear concept transformation functions (f [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison across compositionality models (rows) and embedding models (columns). [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. KDEs across models for the SANC dataset. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Compositionality of Simple Addition in rela [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Constituent dominance of adjectives and nouns, ordered by Mistral dominance rank. 7 Conclusion In this study, we explored the compositionality of various transformer-based embedding models. For the first time, we comprehensively analyse modern embedding models and eval…
Figure 6
Figure 6. Figure 6: UMAP plot of Mistral embeddings using Ridge Regression. The top figure shows the UMAP color-coded by adjective. The most dominant adjec￾tives are ‘beige’, ‘maroon’ and ‘magenta’; least are ‘brown’, ‘green’ and ‘black’. The middle figures shows the UMAP colored by noun.…
Figure 7
Figure 7. Figure 7: Cosine similarity distribution densities across different composition models. Each subfigure shows the [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 31 canonical work pages

  1. [1]

    Mart\' i n Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Man\' e , Rajat Monga, Sherry Moore, Der...

    work page 2015

  2. [2]

    Marco Baroni and Roberto Zamparelli. 2010. https://aclanthology.org/D10-1115 Nouns are vectors, adjectives are matrices: Representing adjective-noun constructions in semantic space . In Empirical Methods in Natural Language Processing, pages 1183--1193

    work page 2010

  3. [3]

    Douglas Biber, Stig Johansson, Geoffrey Leech, Susan Conrad, and Edward Finegan. 1999. Longman of spoken and written english. Harlow: Pearson Education

    work page 1999

  4. [4]

    Marc Brysbaert and Boris New. 2009. https://doi.org/10.3758/BRM.41.4.978 Subtlex-us: A new measure for american english word frequency based on film subtitles . Behavior Research Methods, 41(4):978--990

    work page doi:10.3758/brm.41.4.978 2009

  5. [5]

    Marc Brysbaert, Amy Beth Warriner, and Victor Kuperman. 2014. https://doi.org/10.3758/s13428-013-0403-5 Concreteness ratings for 40 thousand generally known english word lemmas . Behavior Research Methods, 46(3):904--911

    doi:10.3758/s13428-013-0403-5 2014

  6. [6]

    Lars Buijtelaar and Sandro Pezzelle. 2023. https://doi.org/10.18653/v1/2023.eacl-main.163 A psycholinguistic analysis of BERT ' s representations of compounds . In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2230--2241

    doi:10.18653/v1/2023.eacl-main.163 2023

  7. [7]

    John A Bullinaria and Joseph P Levy. 2012. Extracting semantic representations from word co-occurrence statistics: stop-lists, stemming, and svd. Behavior Research Methods , 44:890--907

    work page 2012

  8. [8]

    Danilo S Carvalho, Edoardo Manino, Julia Rozanova, Lucas Cordeiro, and Andr \'e Freitas. 2022. Montague semantics and modifier consistency measurement in neural language models. arXiv preprint arXiv:2212.04310

    work page Pith review arXiv 2022

Show all 52 references
  1. [9]

    Albert Coil and Vered Shwartz. 2023. https://doi.org/10.18653/v1/2023.findings-acl.169 From chocolate bunny to chocolate crocodile: Do language models understand noun compounds? In Findings of the Association for Computational Linguistics: ACL 2023, pages 2698--2710

    2023 doi

  2. [10]

    Silvio Cordeiro, Carlos Ramisch, Marco Idiart, and Aline Villavicencio. 2016. Predicting the compositionality of nominal compounds: Giving word embeddings a hard time. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa...

    2016

  3. [12]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 b . https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Assoc...

    2019 doi

  4. [13]

    Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. 2019. https://doi.org/10.18653/v1/P19-1315 Towards understanding linear word analogies . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3253--3262

    2019 doi

  5. [14]

    Charles J Fillmore et al. 2006. Frame semantics. Cognitive Linguistics: Basic Readings , 34:373--400

    2006

  6. [15]

    Jr Firth. 1957. A synopsis of linguistic theory, 1930-1955. Studies in Linguistic Analysis

    1957

  7. [16]

    Jerry A Fodor and Ernest Lepore. 2002. The compositionality papers. Oxford University Press

    2002

  8. [17]

    Christina L Gagn \'e , Thomas L Spalding, and Daniel Schmidtke. 2019 a . Ladec: The large database of english compounds. Behavior Research Methods , 51:2152--2179

    2019

  9. [18]

    Christina L Gagn \'e , Thomas L Spalding, and Daniel Schmidtke. 2019 b . LADEC : The large database of english compounds. Behavior Research Methods, 51(5):2152--2179

    2019

  10. [19]

    Jumbly Grindrod. 2024. Transformers, contextualism, and polysemy. arXiv preprint arXiv:2404.09577

    2024 arXiv

  11. [20]

    Janosch Haber and Massimo Poesio. 2024. https://doi.org/10.1162/coli_a_00500 P olysemy --- E vidence from linguistics, behavioral science, and contextualized language models . Computational Linguistics, 50(1):351--417

    2024 doi

  12. [21]

    Yikun Han, Chunjiang Liu, and Pengfei Wang. 2023. A comprehensive survey on vector database: Storage and retrieval technique, challenge. arXiv preprint arXiv:2310.11703

    2023

  13. [22]

    Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. CLEVR : A diagnostic dataset for compositional language and elementary visual reasoning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , page...

    2017

  14. [23]

    Barbara J Juhasz, Yun-Hsuan Lai, and Michelle L Woodcock. 2015 a . A database of 629 english compound words: ratings of familiarity, lexeme meaning dominance, semantic transparency, age of acquisition, imageability, and sensory experience. Behavior research methods, 47:1004--1019

    2015

  15. [24]

    Juhasz, Yun-Hsuan Lai, and Michelle L

    Barbara J. Juhasz, Yun-Hsuan Lai, and Michelle L. Woodcock. 2015 b . https://doi.org/10.3758/s13428-014-0523-6 A database of 629 english compound words: Ratings of familiarity, lexeme meaning dominance, semantic transparency, age of acquisition, imageability, and sensory exper...

    2015 doi

  16. [25]

    Yap, and Winston D

    Say Young Kim, Melvin J. Yap, and Winston D. Goh. 2019. https://doi.org/10.3758/s13428-018-1143-3 The role of semantic transparency in visual word recognition of compound words: A megastudy approach . Behavior Research Methods, 51(6):2722--2732

    2019 doi

  17. [26]

    Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114

    2013 arXiv

  18. [27]

    Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. 2017. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253

    2017

  19. [28]

    Landauer and Susan T

    Thomas K. Landauer and Susan T. Dumais. 1997. https://doi.org/10.1037/0033-295X.104.2.211 A solution to plato's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge . Psychological Review, 104(2):211--240

    1997 doi

  20. [29]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Proc...

    2020

  21. [30]

    Gary Libben. 1998. Semantic transparency in the processing of compounds: Consequences for representation, processing, and impairment. Brain and Language , 61(1):30--44

    1998

  22. [31]

    Jianhua Lin. 1991. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory, 37(1):145--151

    1991

  23. [32]

    Leland McInnes, John Healy, and James Melville. 2018. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426

    2018 arXiv

  24. [33]

    Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781

    2013 arXiv

  25. [34]

    Jeff Mitchell and Mirella Lapata. 2008. Vector-based models of semantic composition. In Proceedings of ACL-08: HLT , pages 236--244

    2008

  26. [35]

    Jeff Mitchell and Mirella Lapata. 2009. Language models based on semantic composition. In Empirical Methods in Natural Language Processing , pages 430--439

    2009

  27. [36]

    Jeff Mitchell and Mirella Lapata. 2010. Composition in distributional models of semantics. Cognitive Science , 34(8):1388--1429

    2010

  28. [37]

    Hosein Mohebbi, Ali Modarressi, and Mohammad Taher Pilehvar. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.61 Exploring the role of BERT token representations to explain sentence probing results . In Empirical Methods in Natural Language Processing, pages 792--806

    2021 doi

  29. [38]

    Allan Paivio. 1990. Mental Representations: A Dual Coding Approach. Oxford University Press

    1990

  30. [39]

    Kiho Park, Yo Joong Choe, and Victor Veitch. 2023. https://openreview.net/forum?id=T0PoOJg8cK The linear representation hypothesis and the geometry of large language models . In Causal Representation Learning Workshop at NeurIPS 2023

    2023

  31. [40]

    James Pustejovsky. 1998. The Generative Lexicon . MIT Press

    1998

  32. [41]

    Ishaan Singh Rawal, Alexander Matyasko, Shantanu Jaiswal, Basura Fernando, and Cheston Tan. 2024. https://proceedings.mlr.press/v235/rawal24a.html Dissecting multimodality in V ideo QA transformer models by impairing modality fusion . In Proceedings of the 41st International C...

    2024

  33. [42]

    Fran c ois Recanati. 2004. Literal Meaning . Cambridge University Press

    2004

  34. [43]

    Siva Reddy, Diana McCarthy, and Suresh Manandhar. 2011. An empirical study on compositionality in compound nouns. In Proceedings of 5th International Joint Conference on Natural Language Processing , pages 210--218

    2011

  35. [44]

    Turner, Callum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, Tristan Hume, Francesco Mosconi, C

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L. Turner, Callum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, Tristan Hume, Francesco Mosconi,...

    2024

  36. [45]

    Peter D Turney and Patrick Pantel. 2010. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research , 37:141--188

    2010

  37. [46]

    Walter J. B. van Heuven, Paweł Mandera, Emmanuel Keuleers, and Marc Brysbaert. 2014. https://doi.org/10.1080/17470218.2013.850521 Subtlex-uk: A new and improved word frequency database for british english . Quarterly Journal of Experimental Psychology, 67(6):1176--1190

    2014

  38. [47]

    Eva M Vecchi, Marco Marelli, Roberto Zamparelli, and Marco Baroni. 2017. Spicy adjectives and nominal donkeys: Capturing semantic deviance using compositionality in distributional spaces. Cognitive Science , 41(1):102--136

    2017

  39. [48]

    Amy Beth Warriner, Victor Kuperman, and Marc Brysbaert. 2013. https://doi.org/10.3758/s13428-012-0314-x Norms of valence, arousal, and dominance for 13 , 915 english lemmas . Behavior Research Methods, 45(4):1191--1207

    2013 doi

  40. [49]

    Xinyan Velocity Yu, Sewon Min, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2022. https://arxiv.org/abs/2211.17257 Crepe: Open-domain question answering with false presuppositions . arXiv preprint arXiv:2211.17257

    2022 arXiv

  41. [50]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675

    2019 arXiv

  42. [51]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, et al. 2023. http://arxiv.org/abs/2303.18223 A survey of large language models . arXiv preprint arXiv:2303.18223

    2023 arXiv

  43. [52]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  44. [53]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.