REVIEW 3 major objections 5 minor 49 references
Towards Universal Semantics With Large Language Models
T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Fine-tuned 1B and 8B language models generate accurate, cross-translatable NSM explications and outperform GPT-4o on the paper's benchmark.
desk verdict First LLM-NSM paper: real contribution, but the accuracy claim is entangled in a circular evaluation loop. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the NSM explication: a paraphrase of a target word's meaning composed of semantic primes, a fixed inventory of roughly 65 primitive word-meanings claimed to be lexicalized across most languages, with occasional non-prime 'semantic molecules' allowed for brevity. The argument runs through three automatic metrics. The Legality Score is $\alpha \cdot (\text{primes}-\text{molecules})/\text{total words}$, rewarding prime-heavy, non-circular text. The Substitutability Score averages, over three grader LLMs and four ambiguous passages, the capped log-probability gain $\min(\beta, \Delta_{\mathrm{baseline}}-\Delta_{\mathrm{min}}+\Delta_{\mathrm{ent}})$ for recovering a masked target word when the explication is supplied, with a cap $\beta=40$. Cross-translatability is measured by round-tripping text through a machine translation service into five low-resource languages and comparing back-translations to originals with BLEU and embedding similarity. These metrics select the highest-scoring candidates from hundreds of thousands of LLM-generated explications to form the training dataset, and the same metrics are then used to compare DeepNSM to general-purpose baselines.
What would settle it
An independent blind evaluation by human linguistic experts on the same 149 benchmark words would settle the claim: if native-speaker experts rank GPT-4o's explications as equal to or more accurate than DeepNSM's, or if a held-out set of expert-authored NSM explications scores lower on the substitutability metric than obviously wrong paraphrases, the central performance claim is refuted.
Extended reading notes
Core claim
The paper claims that NSM explication generation is not only automatable but works best in small, task-specific models: DeepNSM-1B and DeepNSM-8B achieve the highest overall Explication Scores on a hand-curated 149-word benchmark, beating GPT-4o and Gemini-2.0-Flash while also using more semantic primes, fewer non-prime molecules, and less circularity. The same explications are shown to be more robust than WordNet dictionary definitions when translated into five low-resource languages and back, supporting the framework's universality premise. The authors also show that filtering the training dataset by their automatic quality scores is what drives much of the gain: unfiltered fine-tuned models score noticeably worse.
Load-bearing premise
The substitutability score, which uses grader language models to measure whether an explication helps recover a masked target word, is assumed to track true descriptive accuracy; because the same score filters the training data and then evaluates the final models, the headline performance gap would be an artifact of the metric rather than a real semantic improvement if that assumption fails.
Editorial extensions
If this is right
- Dedicated fine-tuning can substitute for model scale in a linguistically constrained generation task: 1B- and 8B-parameter models match or beat far larger general models on NSM explication quality.
- NSM explications could serve as an intermediate representation in low-resource translation pipelines, since they survive round-trip translation with less semantic drift than dictionary-style definitions.
- Automated legality, substitutability, and cross-translatability scores make it feasible to screen hundreds of thousands of candidate explications, enabling the first large-scale NSM dataset.
- Because the filtered dataset is public, future work can fine-tune other model families or sizes without repeating the expensive candidate-generation and scoring pipeline.
Reading between the lines
- The riskiest spot is the evaluation loop: the same substitutability metric filters the training data and then judges the final models, so an independent measure of meaning preservation, such as human paraphrase acceptability testing, would be needed to confirm the metric is not rewarding a proxy that only looks like accuracy.
- If explication generation generalizes beyond English, semantic primes could become an interlingua for machine translation into low-resource languages, but that would require testing the prime inventory directly in each target language rather than only through round-trip machine translation.
- A testable extension is to use DeepNSM-generated prime paraphrases as an auxiliary training signal for low-resource language models, letting those models learn word meanings by their distribution near known prime expressions, as the paper sketches in its appendix.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents DeepNSM, the first attempt to adapt large language models to generate Natural Semantic Metalanguage (NSM) explications. The authors propose three automatic evaluation methods: a legality score based on prime/molecule counts (Eq. 1), a substitutability score using LLM log-probability recovery of masked target words (Eq. 5), and a round-trip cross-translatability test. They build a dataset of approximately 44,000 entries by having Gemini-2.0-Flash generate candidate explications for WordNet senses and filtering them with their Explication Score (Eq. 6), then fine-tune 1B and 8B Llama models. On a hand-curated 149-word benchmark, the filtered DeepNSM models achieve the highest Explication Score and competitive cross-translatability across five low-resource languages; a brief human ranking is also reported. The central claim is that 1B and 8B models outperform GPT-4o in producing accurate, cross-translatable explications.
Significance. If the central claim held, this would be a valuable contribution: it is the first open-sourced pipeline for NSM explication generation, with code, models, and dataset released; it introduces automatic metrics that, if validated, would enable scalable research on NSM; and the cross-translatability results support the practical relevance of NSM-style representations for low-resource translation. The paper also makes concrete falsifiable predictions, such as the ill/sick analysis in Appendix B. However, the significance is conditional: because the main evaluation metric is also the data-filtering criterion, the headline accuracy advantage over GPT-4o needs an independent validation before the contribution can be accepted as stated.
major comments (3)
- [§4, §5.2, Table 1] The Explication Score (Eq. 6) is used both to select training data in §4 (threshold ≥35, passed by fewer than 15% of candidates) and to evaluate models in Table 1. Fine-tuning on data selected to maximize a metric is expected to improve that metric even when the metric does not track real semantic quality, so the reported accuracy advantage of DeepNSM over GPT-4o is partly self-fulfilling. The filtered-versus-unfiltered comparison (DeepNSM-* vs DeepNSM-*†) shows the filter moves the metric, but it does not show that the movement reflects genuine accuracy; the human ranking that could break this loop is reported only as aggregate percentages with no rater count, item count, expertise, or inter-annotator agreement. Please break the circularity, for example by validating the substitutability score against expert human judgments on a held-out set, or by reporting a fully specified human evaluation with agreement statistics.
- [§5.2, 'Metrics Align with Qualitative Judgements'] The human evaluation is too underspecified to serve as an independent validation. The text states that DeepNSM explications received top rankings 46% of the time, versus 28% for GPT, 21% for Gemini, and 5% for Llama, but it does not report how many raters judged how many items, what expertise the raters had, what instructions or rubrics were used, or any inter-annotator agreement statistic. Without this information the claimed alignment between automatic metrics and human judgment is not verifiable, and the human evaluation cannot break the circularity of using the same metric for filtering and evaluation.
- [§3.2, §5.2, Limitations] The substitutability score (Eq. 5) is an unvalidated LLM-based heuristic: the Limitations section concedes that the grader LLMs' alignment with human judgments is unverified. Section 5.2 further shows that dictionary definitions score highest on substitutability (12.14) yet are dismissed as 'cheating' because they reuse the target word; this demonstrates that the raw substitutability metric does not enforce non-circularity, and only the explicit zeroing in Eq. 6 suppresses this artifact. The central 'accurate explications' claim therefore depends on a metric that is known to be gameable in a direction that favors definitions over NSM explications; a validation study against expert-authored explications is needed.
minor comments (5)
- [Table 2, §5.2] The claim that NSM explications 'consistently achieve the highest cross-translatability scores across all languages tested' is not supported: DeepNSM-8B obtains BLEU 23.3 for Dzongkha, below Llama-3.1-8B-it's 23.8, and embedding similarity 91.7 for Kinyarwanda, below Gemini-2.0-Flash's 93.0. Please qualify this statement.
- [Eqs. 3–4, Appendix C] Equations 3 and 4 are described as averages but contain no division by k, whereas Algorithm 1 in Appendix C divides by k after the loop; please make the notation consistent.
- [Eq. 5] The subscripts and superscripts in Eq. 5 (e.g., Δ(g,p) baseline− Δ(g,p) min + Δ(g,p) ent) obscure the intended grouping; please rewrite with a clearer notation such as Δ_base(g,p).
- [§3.1] The legality scoring counts 'words' without defining tokenization or handling of inflected and multiword exponents, and it does not address the polysemy of English prime exponents (e.g., 'above' in a non-spatial sense, as noted in Appendix A); this makes the legality component of the Explication Score potentially noisy.
- [Appendix Table 3] The reported standard errors in Table 3 (e.g., 0.0008 for several proportions) appear implausibly small; please specify how they were computed and what resampling procedure was used.
Circularity Check
The headline accuracy advantage is partly circular: the same Explication Score (Eq. 6) used to filter the training data is also the main metric used to show DeepNSM outperforms GPT-4o; the paper's own filtered-vs-unfiltered comparison shows filtering drives the metric.
-
fitted input called prediction
[Section 4 (Quality Filtering and Final Dataset) and Section 5.1/5.2 (Evaluation)]
"Legality and substitutability scores are then combined to compute the overall Explication Score (Equation 6). This score serves as a quality indicator that can be used to filter out low-quality entries from the final dataset. ... After generating and scoring candidate explications, we filter out those with scores below 35, a quality threshold met by fewer than 15% of candidates ... The generated explications are evaluated using the legality, substitutability, and cross-translatability tests described in Section 3, as well as qualitative human ranking."
The Explication Score (Eq. 6 = γ(Substitutability + Legality)) is used twice: first as the training-data quality filter (threshold ≥ 35, passed by fewer than 15% of candidates), then as the principal evaluation metric in Table 1 supporting the claim that DeepNSM 1B/8B 'consistently outperform or match' GPT-4o. Fine-tuning a model on data pre-selected to score highly on a metric is expected to raise that same metric even when the metric does not track semantic fidelity. The paper itself shows the filter changes exactly the reported quantity: quality-filtered DeepNSM-1B and 8B achieve a 34% and 15% relative increase in substitutability over their unfiltered counterparts.
full rationale
This paper is not a self-citation chain: NSM theory is imported from external linguistics literature, and the automatic metrics are newly defined in the paper itself. The main circularity is metric double-use. Eq. 6 is computed from Eq. 1 (legality) and Eq. 5 (substitutability); Section 4 uses Eq. 6 to filter which Gemini-generated explications enter the 43,000-entry training set, and Section 5 uses Eq. 6 to rank DeepNSM against GPT-4o in Table 1. Selecting training data to maximize a metric and then evaluating on that metric partially forces the result. The paper's own comparison to unfiltered models quantifies the effect (34%/15% relative substitutability gains), confirming the loop is load-bearing. Independent evidence partially mitigates the circularity: cross-translatability (Table 2) does not use Eq. 6 for filtering and favors DeepNSM on most low-resource languages, and the blind human ranking gives DeepNSM 46% top rankings vs GPT-4o's 28%. However, the human ranking is reported only as aggregate percentages with no annotator-expertise or inter-annotator detail, and the Limitations section concedes that substitutability relies on 7-12B grader models whose alignment with human judgments is unverified. Thus the accuracy half of the central claim is partially circular; the cross-translatability half is independent. Overall score 6.
Assumptions & free parameters
free parameters (5)
- alpha (legality weighting) =
10
- beta (substitutability cap) =
40
- gamma (overall score scaling) =
2
- k (truncation depth) =
2
- quality filtering threshold =
35
assumptions (4)
- domain assumption Semantic primes are universal and lexicalized across languages
- ad hoc to paper LLM log-probability substitutability is a valid measure of descriptive accuracy
- domain assumption Round-trip translation with Google Translate measures cross-translatability
- domain assumption WordNet sense inventory provides a representative distribution of word meanings
Cite this review
Pith. "Pith review of Towards Universal Semantics With Large Language Models." pith.science (2026). https://pith.science/paper/2F2EZZFI
@misc{pith2026250511764,
author = {Pith},
title = {Pith review of: Towards Universal Semantics With Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/2F2EZZFI}},
note = {Machine review of arXiv:2505.11764}
}
read the original abstract
The Natural Semantic Metalanguage (NSM) is a linguistic theory based on a universal set of semantic primes: simple, primitive word-meanings that have been shown to exist in most, if not all, languages of the world. According to this framework, any word, regardless of complexity, can be paraphrased using these primes, revealing a clear and universally translatable meaning. These paraphrases, known as explications, can offer valuable applications for many natural language processing (NLP) tasks, but producing them has traditionally been a slow, manual process. In this work, we present the first study of using large language models (LLMs) to generate NSM explications. We introduce automatic evaluation methods, a tailored dataset for training and evaluation, and fine-tuned models for this task. Our 1B and 8B models outperform GPT-4o in producing accurate, cross-translatable explications, marking a significant step toward universal semantic representation with LLMs and opening up new possibilities for applications in semantic analysis, translation, and beyond. Our code is available at https://github.com/OSU-STARLAB/DeepNSM.
Figures
Figures from the paper (12 more)
Reference graph
Works this paper leans on
-
[1]
A. B. Abacha, W.-w. Yim, Y . Fu, Z. Sun, M. Yetisgen, F. Xia, and T. Lin. Medec: A benchmark for medical error detection and correction in clinical notes. arXiv preprint arXiv:2412.19260, 2024
arXiv 2024
- [2]
-
[3]
K. Allan. Linguistic Semantics, Vols I and II. London: Routledge & Kegan Paul, 1986
work page 1986
-
[4]
N. Arnawa. The implementation of natural semantic metalanguage and semantic field in language teaching: A case study. Journal of Language Teaching and Research, 8(3):507, 2017
work page 2017
-
[5]
J. T. Ball, S. Rodgers, R. Schvaneveldt, and A. Ball. Creating meaningful word vectors and examining their use as representations of word meaning. Proceedings of the Annual Meeting of the Cognitive Science Society, 46, 2024
work page 2024
-
[6]
S. Bird. Nltk: the natural language toolkit. In Proceedings of the COLING/ACL 2006 interactive presentation sessions, pages 69–72, 2006. 11
work page 2006
- [7]
- [8]
Show all 49 references
-
[9]
Bromhead and C
H. Bromhead and C. Goddard. Applied semantics and climate communication. Australian Review of Applied Linguistics, July 2023
2023
-
[10]
J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, and Z. Liu. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation.arXiv preprint arXiv:2402.03216, 2024
2024 arXiv
-
[11]
Conneau, K
A. Conneau, K. Khandelwal, N. Goyal, V . Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V . Stoyanov. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116, 2019
1911 arXiv
-
[12]
Dettmers, A
T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314, 36, 2023
2023 arXiv
-
[13]
Dimakis, S
A. Dimakis, S. Markantonatou, and A. Anastasopoulos. Dictionary-aided translation for handling multi-word expressions in low-resource languages. In L.-W. Ku, A. Martins, and V . Srikumar, editors,Findings of the Association for Computational Linguistics: ACL 2024 , pages 2588–...
2024
-
[14]
Elsner and J
M. Elsner and J. Needle. Translating a low-resource language using gpt-3 and a human-readable dictionary. In Proceedings of the 20th SIGMORPHON workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 1–13, 2023
2023
-
[15]
Fähndrich
J. Fähndrich. Semantic decomposition and marker passing in an artificial representation of meaning. PhD Thesis, Technische Universität Berlin, 2018
2018
-
[16]
Giulianelli, I
M. Giulianelli, I. Luden, R. Fernandez, and A. Kutuzov. Interpretable Word Sense Repre- sentations via Definition Generation: The Case of Semantic Change Analysis, July 2023. arXiv:2305.11993 [cs]
2023 arXiv
-
[17]
C. Goddard. Testing the translatability of semantic primitives into an australian aboriginal language. Anthropological Linguistics, 33:31–56, 1991
1991
-
[18]
C. Goddard. Semantic Analysis: A Practical Introduction. Oxford University Press, Oxford, first edition edition, 1998
1998
-
[19]
C. Goddard. 1. Natural Semantic Metalanguage: The state of the art. In C. Goddard, editor, Studies in Language Companion Series, volume 102, pages 1–34. John Benjamins Publishing Company, Amsterdam, Apr. 2008
2008
-
[20]
Goddard, editor
C. Goddard, editor. Cross-linguistic semantics. Number v. 102 in Studies in language companion series. John Benjamins Pub. Co, Amsterdam ; Philadelphia, 2008. OCLC: ocn192134491
2008
-
[21]
C. Goddard. The natural semantic metalanguage approach. In B. Heine and H. Narrog, editors, The Oxford Handbook of Linguistic Analysis, pages 459–484. Oxford University Press, Oxford, 2009
2009
-
[22]
Goddard and A
C. Goddard and A. Wierzbicka. Cultural scripts: What are they and what are they good for? Intercultural Pragmatics, 1(2), Jan. 2004
2004
-
[23]
Goddard and A
C. Goddard and A. Wierzbicka, editors. Semantic and lexical universals: theory and empir- ical findings. Number v. 25 in Studies in language companion series (SLCS) 0165-7763. J. Benjamins, Amsterdam Philadelphia, 2010
2010
-
[24]
Goddard and A
C. Goddard and A. Wierzbicka. Semantic fieldwork and lexical universals. Studies in Language, 38(1):80–127, May 2014. 12
2014
-
[25]
Goddard and A
C. Goddard and A. Wierzbicka. Words and meanings: lexical semantics across domains, languages, and cultures. Oxford linguistics. Oxford University Press, Oxford, first edition edition, 2014. OCLC: ocn830367893
2014
-
[26]
Virus”, “bacteria
C. Goddard and A. Wierzbicka. Semantics in the time of coronavirus: “Virus”, “bacteria”, “germs”, “disease” and related concepts. Russian Journal of Linguistics, 25(1):7–23, Dec. 2021
2021
-
[27]
Goddard, A
C. Goddard, A. Wierzbicka, and H. Fabréga. Evolutionary semantics: using NSM to model stages in human cognitive evolution. Language Sciences, 42:60–79, Mar. 2014
2014
-
[28]
Litaina, A
T. Litaina, A. Soularidis, G. Bouchouras, K. Kotis, and E. Kavakli. Towards llm-based semantic analysis of historical legal documents. 2024
2024
-
[29]
H. Man, N. T. Ngo, V . D. Lai, R. A. Rossi, F. Dernoncourt, and T. H. Nguyen. LUSIFER: Language Universal Space Integration for Enhanced Multilingual Embeddings with Large Language Models, Jan. 2025. arXiv:2501.00874 [cs]
2025 arXiv
-
[30]
S. Maskova. Semantic Analysis of Snow-related Words in Danish and Kalaallisut (West Greenlandic). Scandinavian Studies in Language, 13(1):225–248, Dec. 2022
2022
-
[31]
J. D. McCawley. A program for logic. In Semantics of natural language, volume 40, pages 498–544. Springer, Dordrecht, 1972
1972
-
[32]
R. Merx, A. Mahmudi, K. Langford, L. A. de Araujo, and E. Vylomova. Low-resource machine translation through retrieval-augmented llm prompting: a study on the mambai language. arXiv preprint arXiv:2404.04809, 2024
2024 arXiv
-
[33]
G. A. Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39– 41, 1995
1995
-
[34]
Otomo and A
A. Otomo and A. Torii. An nsm approach to the meaning of tear and its japanese equivalents. In Selected Papers from the 2005 Conference of the Australian Linguistic Society. Citeseer, 2006
2005
-
[35]
B. Peeters. Nsm approach: Natural semantic metalanguage resource base. https://www. nsm-approach.net, 2025. Accessed: 2025-05-15
2025
-
[36]
Periti and S
F. Periti and S. Montanelli. Lexical Semantic Change through Large Language Models: a Survey. ACM Computing Surveys, 56(11):1–38, Nov. 2024
2024
-
[37]
Reimers and I
N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019
2019
-
[38]
Sadow and S
L. Sadow and S. S. Fernández. Pedagogical pragmatics: Natural semantic metalanguage applications to language learning and teaching. Scandinavian Studies in Language, 13(1):53–66, 2022
2022
-
[39]
Siino, M
M. Siino, M. Falco, D. Croce, and P. Rosso. Exploring llms applications in law: A literature review on current legal nlp approaches. IEEE Access, 2025
2025
-
[40]
Y . Song, L. Li, C. Lothritz, S. Ezzini, L. Sleem, N. Gentile, R. State, T. F. Bissyandé, and J. Klein. Is llm the silver bullet to low-resource languages machine translation? arXiv preprint arXiv:2503.24102, 2025
2025 arXiv
-
[41]
Touvron, T
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023
2023 arXiv
-
[42]
Wierzbicka
A. Wierzbicka. Understanding Cultures through Their Key Words English, Russian, Polish, German, and Japanese. Oxford University Press, Oxford, first edition edition, 1997
1997
-
[43]
Wierzbicka
A. Wierzbicka. Why there are no ‘colour universals’ in language and thought. Journal of the Royal Anthropological Institute, 14(2):407–425, June 2008. 13
2008
-
[44]
Wierzbicka
A. Wierzbicka. Experience, evidence, and sense: the hidden cultural legacy of English. Oxford university press, Oxford, 2010
2010
-
[45]
semantic primitives
A. Wierzbicka. “semantic primitives”, fifty years later.Russian Journal of Linguistics, 25(2):317– 342, 2021
2021
-
[46]
B. J. Wilson and G. M. Farese. What did adam smith mean? the semantics of the opening key principles in the wealth of nations. In P. Sagar, editor,Interpreting Adam Smith: Critical Essays, pages 77–95. Cambridge University Press, Cambridge, 2023
2023
-
[47]
Yu, C.-H
Y . Yu, C.-H. H. Yang, J. Kolehmainen, P. G. Shivakumar, Y . Gu, S. R. R. Ren, Q. Luo, A. Gourav, I.-F. Chen, Y .-C. Liu, T. Dinh, A. G. D. Filimonov, S. Ghosh, A. Stolcke, A. Rastow, and I. Bulyko. Low-rank adaptation of large language model rescoring for parameter-efficient ...
2023
-
[48]
Zamblera
F. Zamblera. Computational nsm: a prolog-based notation. Online]. SÍNTESIS CURRICULAR, 2010
2010
-
[49]
above” is a valid prime in the spatial sense (“the sky is above the ground
M. Zeifert. Natural semantic (legal?) metalanguage. what can legal theory learn from anna wierzbicka? Between Text, Meaning and Legal Languages, page 173, 2023. 14 Appendix This appendix offers supplementary materials, background information, and extended discussions that coul...
2023
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.