REVIEW 4 major objections 5 minor 1 cited by
Exploring the Structure of AI-Induced Language Change in Scientific English
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read AI-spiked words carry whole synonym groups upward
desk verdict A useful, readable test of whether ChatGPT-era word spikes replace synonyms or come with them, but the main result is partly built into the method because ChatGPT picks the synonyms and validates the declining words. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the replacement hypothesis, treated as the null: because relative frequencies are zero-sum, a spiking word should push its semantic neighbours down. The test bed is a pipeline that takes 32 focal words from the AI-overuse literature, lemmatizes and part-of-speech-tags them, pulls candidate synonyms from a thesaurus, filters to the five most relevant for academic abstracts with an LLM assistant, and tracks occurrences per million in PubMed abstracts from 2020 to 2024 with chi-square significance tests, backed by a 1975–2024 corpus of 6.94 billion tokens for long-term trends. Part-of-speech tagging does real work here: it separates "underscore VERB" from "underscore NOUN" and shows that the shifts concentrate in adjectives, verbs, and adverbs, with nouns barely participating outside "potential."
What would settle it
A matched comparison of verified human-written pre-2022 PubMed abstracts with LLM-assisted post-2022 abstracts, controlling for topic and author discipline, could test the additive claim: if "crucial" rises while its synonyms fall in the LLM-assisted set, the replacement hypothesis survives; if the entire cluster rises, the paper's conclusion is supported.
Extended reading notes
Core claim
The paper's central claim is that, in most cases, whole synonym groups move together, so the replacement hypothesis is invalid. A spiked word like "crucial" is followed, less sharply, by its thesaurus neighbours "critical," "essential," "significant," and "noteworthy"; "underscore" rises together with "emphasize" and "highlight." Words such as "advancement," "emphasize," and "meticulous" therefore do not simply replace similar terms, but are used as semantic and pragmatic additions. The decreasing side is different: "important" falls, but of 72 candidate decreasing items only 13 also fall in LLM-written abstracts, and their trajectories are gradual and fluctuating, not mirror-image spikes. Taken together, the paper concludes that LLM influence on scientific English is additive and concentrated in low-content style words, while losses look like ordinary organic language change.
Load-bearing premise
The load-bearing premise is that the frequency spikes are caused by LLM adoption, inferred from timing and prior literature rather than direct causal evidence; if topic shifts or unrelated style drift explain the spikes, the conclusion that LLMs add rather than replace synonyms collapses.
Editorial extensions
If this is right
- If collective rises hold, LLM-assisted abstracts should be detectably denser in entire style-word clusters, not just in a single focal word.
- The lack of mirror-image declines means AI usage is inflating the total volume of qualifying vocabulary rather than reshuffling a fixed lexical budget.
- Because nouns barely participate outside "potential," monitoring that ignores part of speech will miss where the change is happening.
- Gradual, fluctuating declines imply that searching for steep negative spikes will under-detect LLM influence on vocabulary loss.
- If the trend continues, scientific writing may homogenize around a shared AI style, as the paper's broader-impact discussion notes.
Reading between the lines
- Beyond the paper's explicit claims, single-word spike detectors will undercount LLM influence whenever the whole synonym cluster rises together; a cluster-level metric should be more sensitive.
- A direct test of the additive view: rewrite a fixed set of human abstracts with an LLM and compare cluster frequencies to originals; the additive view predicts higher total frequency across the synonym group, not just the focal word.
- The paper calls for spoken-language data; an extension is that AI-style words should enter spontaneous speech with a lag behind written science, which would support indirect cultural spread rather than only direct copy-paste use.
- Because the observation window (2020–2024) is short, a gradual decline like "important" could be the start of a long replacement trajectory; a 10-year follow-up could distinguish additive coexistence from slow organic decline.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper analyzes frequency changes of 'focal words' (e.g., 'delve', 'crucial') and their synonyms in PubMed abstracts from 1975 to 2024, using part-of-speech tagging and a comparison of 2020 vs. 2024 occurrences per million. The authors report that for most focal words, several of their ChatGPT-selected synonyms also increase in frequency, which they interpret as evidence against the 'replacement hypothesis' (that a spiking word replaces its semantic neighbors). They also identify decreasing words by requiring a decline both in PubMed and in ChatGPT-generated abstracts, and find that these decreases exhibit more complex, gradual patterns. The paper concludes that LLM-induced changes are primarily semantic and pragmatic additions rather than lexical replacements.
Significance. If the central claim is robust, the paper makes a useful contribution to the study of AI-induced language change by moving beyond single-word frequency spikes and by using POS tagging to separate word forms. The open GitHub repository, the reproducible pipeline, and the explicit falsifiable prediction ('abstracts processed through LLMs contain higher frequencies of AI-overused words') are strengths. However, the current significance is conditional: the main empirical claim is weakened by a selection procedure that relies on the very model class suspected to cause the changes, and by the absence of a control group or aggregate statistical summary. The conclusion about 'semantic and pragmatic additions' also depends on an unverified causal attribution to LLM use, which the authors acknowledge in the Limitations. With the methodological gaps addressed, the findings would be a meaningful step for the field.
major comments (4)
- [Section 2, Procedure] The synonym-selection step uses ChatGPT 4o-mini to choose the five most relevant synonyms per focal word from Merriam-Webster. Because the paper's own hypothesis is that ChatGPT overuses the focal words, using the same class of model to define the semantic neighborhoods risks circularity: the chosen synonyms may be precisely the words that ChatGPT also favors, so the observed collective increases may reflect the model's lexical preferences rather than the behavior of semantic neighborhoods generally. The replacement hypothesis concerns all (or most) semantically related words; testing it with model-selected neighbors cannot invalidate it. The authors should re-run the analysis with the full Merriam-Webster synonym sets, or with a human-selected or random subset, and show that the 'collective movement' pattern persists. Table 1 already illustrates the sensitivity: several ChatGPT-selected synonyms decrease (e.g., 'carefully' -9.83%, 'illustrate' -13.88%, 'important' -23.43%, 'stress' -3.3%).
- [Section 2, Analysis and Results] The central claim that 'in most cases, entire synonym groups tend to move collectively' is not supported by any aggregate statistic in the text. Figure 3 is a distribution plot, but the manuscript does not report the fraction of focal words with, say, at least three of four or five synonyms increasing, nor does it compare this fraction to a null expectation. A control group of frequency- and POS-matched non-synonymous words, or a permutation test, is needed to show that the collective increases are not just a general upward drift of academic vocabulary in this period. Without such a baseline, 'most cases' remains an informal impression rather than a quantitative finding.
- [Section 3, Methodology] The procedure for identifying decreasing focal words is circular: a candidate word is included only if it experienced a decrease in both the PubMed corpus and the ChatGPT-generated abstracts ('If a word found on our original list experienced a decrease in the AI authored abstracts, it became a focal word'). Because the ChatGPT-generated abstracts are used as the selection criterion, the conclusion that the resulting words are 'underused by both AI and human authors' is true by construction, not discovered. To make the finding non-tautological, the authors should use a held-out or independent LLM corpus for validation, or report the analysis for all 72 candidate words rather than only the 13 that survive the filter.
- [Discussion and Limitations] The paper's title and main interpretation ('AI-induced language change', 'semantic and pragmatic additions') depend on the claim that the observed frequency spikes are caused by LLM usage, but the only evidence offered is the timing of the spikes. The authors are transparent about this in the Limitations ('the frequency shifts observed are often attributed, by conjecture, to the influence of LLM usage'), but the central conclusion that the replacement hypothesis is invalidated would lose its footing if the spikes were caused by topic shifts or general style drift. A concrete falsifiable test should be added, such as comparing trajectories in a non-LLM-affected corpus or examining whether the spikes are concentrated in fields with higher LLM adoption. Without such a test, the causal framing is premature, and the title should be conditional or the contribution reframed as descriptive.
minor comments (5)
- [Section 2, Procedure] The spaCy model is written as 'encore web sm'; it should be 'en_core_web_sm', and the claimed 97% accuracy should be attributed to a specific evaluation source.
- [Section 3, Analysis and Results] In the sentence about 'fringeNOUN', the POS tag should be separated as 'fringe NOUN' for consistency with the notation used elsewhere in the paper.
- [Section 2, Analysis and Results] The text contains a duplicated possessive at 'our our anonymous GitHub repository'; this should be corrected to 'our anonymous GitHub repository'.
- [General] The paper uses both 'COVID-19' and 'Covid-19' in different places; please standardize the spelling.
- [Section 3, Methodology] The choice of the 2020-2024 comparison window is not justified; since ChatGPT was released in late 2022, alternative windows such as 2021-2023 or 2022-2024 should be explored to ensure that the classification of increases/decreases is not an artifact of the chosen endpoints.
Circularity Check
No significant circularity; the derivation is self-contained, with ChatGPT used transparently as a data source rather than as a fitted input.
full rationale
The paper's central claims rest on frequency trends measured in PubMed abstracts, which are independent of the ChatGPT-based synonym selection. Section 2 uses Merriam-Webster as the initial source of synonyms and ChatGPT only to rank them for academic relevance; the subsequent frequency analysis is conducted on external PubMed data. The finding that many synonyms also increase is an empirical observation, not a consequence of how the synonym sets were constructed. Section 3 uses ChatGPT-generated abstracts as a filter to identify words that are underused by AI, but the paper explicitly presents this as a selection criterion rather than as a predicted outcome, and the reported long-term trend analysis of the resulting words is independent of that filter. Citations to Juzek and Ward (2025) are descriptive and supportive, not load-bearing; no uniqueness theorem or unverified result from that work is used to force the paper's conclusions. The paper also candidly acknowledges that the causal attribution to LLMs is an assumption. None of the derivation steps reduces to its own inputs by construction, so the circularity score is 0.
Assumptions & free parameters
free parameters (3)
- Number of synonyms per focal word (k = 5) =
5
- Annotator stopping threshold for decreasing words =
60 words
- Comparison time window =
2020 to 2024
assumptions (4)
- domain assumption PubMed abstracts are a valid proxy for Scientific English.
- standard math The chi-square test's assumptions hold for temporally ordered word-frequency data.
- ad hoc to paper The ChatGPT-generated abstracts accurately represent LLM lexical preferences.
- ad hoc to paper The timing of the spikes implies LLM causation.
Cite this review
Pith. "Pith review of Exploring the Structure of AI-Induced Language Change in Scientific English." pith.science (2026). https://pith.science/paper/WJUBUBEQ
@misc{pith2026250621817,
author = {Pith},
title = {Pith review of: Exploring the Structure of AI-Induced Language Change in Scientific English},
year = {2026},
howpublished = {\url{https://pith.science/paper/WJUBUBEQ}},
note = {Machine review of arXiv:2506.21817}
}
read the original abstract
Scientific English has undergone rapid and unprecedented changes in recent years, with words such as "delve," "intricate," and "crucial" showing significant spikes in frequency since around 2022. These changes are widely attributed to the growing influence of Large Language Models like ChatGPT in the discourse surrounding bias and misalignment. However, apart from changes in frequency, the exact structure of these linguistic shifts has remained unclear. The present study addresses this and investigates whether these changes involve the replacement of synonyms by suddenly 'spiking words,' for example, "crucial" replacing "essential" and "key," or whether they reflect broader semantic and pragmatic qualifications. To further investigate structural changes, we include part of speech tagging in our analysis to quantify linguistic shifts over grammatical categories and differentiate between word forms, like "potential" as a noun vs. as an adjective. We systematically analyze synonym groups for widely discussed 'spiking words' based on frequency trends in scientific abstracts from PubMed. We find that entire semantic clusters often shift together, with most or all words in a group increasing in usage. This pattern suggests that changes induced by Large Language Models are primarily semantic and pragmatic rather than purely lexical. Notably, the adjective "important" shows a significant decline, which prompted us to systematically analyze decreasing lexical items. Our analysis of "collapsing" words reveals a more complex picture, which is consistent with organic language change and contrasts with the patterns of the abrupt spikes. These insights into the structure of language change contribute to our understanding of how language technology continues to shape human language.
Figures
Forward citations
Cited by 1 Pith paper
-
From Context Shift to Stylistic Collapse: Why Training Objectives Matter More Than Scale
Stylistic collapse in LLMs is largely pre-alignment and scale-independent; strong entropy regularization reduces it far more than weak smoothing or RLHF.
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...
-
[2]
Aitchison, J. 2005. Language change. In Cobley, P., ed., The Routledge Companion to Semiotics and Linguistics . London: Routledge. chapter 7
work page 2005
-
[3]
I.; Babaei, H.; LeJeune, D.; Siahkoohi, A.; and Baraniuk, R
Alemohammad, S.; Casco-Rodriguez, J.; Luzi, L.; Humayun, A. I.; Babaei, H.; LeJeune, D.; Siahkoohi, A.; and Baraniuk, R. G. 2023. Self-consuming generative models go mad. arXiv preprint arXiv:2307.01850
arXiv 2023
-
[4]
Altman, D. G., and Bland, J. M. 1995. Statistics notes: Absence of evidence is not evidence of absence. Bmj 311(7003):485
work page 1995
-
[5]
Baron, N. S. 2010. Always on: Language in an online and mobile world . Oxford University Press
work page 2010
-
[6]
Bizzoni, Y.; Degaetano-Ortlieb, S.; Fankhauser, P.; and Teich, E. 2020. Linguistic variation and change in 250 years of english scientific writing: A data-driven approach. Frontiers in Artificial Intelligence 3:73
work page 2020
-
[7]
Briesch, M.; Sobania, D.; and Rothlauf, F. 2023. Large language models suffer from their own output: An analysis of the self-consuming training loop. arXiv preprint arXiv:2311.16822
arXiv 2023
-
[8]
Bybee, J. L. 2006. From usage to grammar: The mind's response to repetition. Language 82(4):711--733
work page 2006
Show all 40 references
-
[9]
Crystal, D. 2008. Txtng: The gr8 db8 . OUP Oxford
2008
-
[10]
Degaetano-Ortlieb, S., and Teich, E. 2018. Using relative entropy for detection and analysis of periods of diachronic linguistic change. In Proceedings of the second joint SIGHUM workshop on computational linguistics for cultural heritage, social sciences, humanities and liter...
2018
-
[11]
Geng, M.; Chen, C.; Wu, Y.; Chen, D.; Wan, Y.; and Zhou, P. 2024. The impact of large language models in academia: from writing to speaking. arXiv preprint arXiv:2409.13686
2024 arXiv
-
[12]
contamination
Gray, A. 2024. Chatgpt" contamination": estimating the prevalence of llms in the scholarly literature. arXiv preprint arXiv:2403.16887
2024 arXiv
-
[13]
Haslwanter, T. 2016. An introduction to statistics with python. With applications in the life sciences. Switzerland: Springer International Publishing
2016
-
[14]
Haspelmath, M. 2000. Why is grammaticalization irreversible? Linguistics 37:1043--68
2000
-
[15]
Hataya, R.; Bao, H.; and Arai, H. 2023. Will large-scale generative models corrupt future datasets? In Proceedings of the IEEE/CVF International Conference on Computer Vision , 20555--20565
2023
-
[16]
Heine, B. 2017. Grammaticalization. The handbook of historical linguistics 573--601
2017
-
[17]
J., and Thompson, S
Hopper, P. J., and Thompson, S. A. 1984. The discourse basis for lexical categories in universal grammar. Language 60(4):703--752
1984
-
[18]
S., and Ward, Z
Juzek, T. S., and Ward, Z. B. 2025. Why does chatgpt" delve" so much? exploring the sources of lexical overrepresentation in large language models. In Proceedings of the 31st International Conference on Computational Linguistics (COLING 2025)
2025
-
[19]
G.; Horv \'a t, E.- \'A .; and Lause, J
Kobak, D.; M \'a rquez, R. G.; Horv \'a t, E.- \'A .; and Lause, J. 2024. Delving into chatgpt usage in academic writing through excess vocabulary. arXiv preprint arXiv:2406.07016
2024 arXiv
-
[20]
Koppenburg, P. 2024. Tweet on 01 april 2024. https://x.com/PKoppenburg/status/1774757167045788010. Accessed: 2025-01-23
2024
-
[21]
Krielke, M.-P. 2024. Cross-linguistic dependency length minimization in scientific language: Syntactic complexity reduction in english and german in the late modern period. Languages in Contrast 24(1):133--163
2024
-
[22]
Labov, W. 1973. Sociolinguistic patterns . University of Pennsylvania press
1973
-
[23]
Liang, W.; Izzo, Z.; Zhang, Y.; Lepp, H.; Cao, H.; Zhao, X.; Chen, L.; Ye, H.; Liu, S.; Huang, Z.; et al. 2024a. Monitoring ai-modified content at scale: A case study on the impact of chatgpt on ai conference peer reviews. arXiv preprint arXiv:2403.07183
-
[24]
Liang, W.; Zhang, Y.; Wu, Z.; Lepp, H.; Ji, W.; Zhao, X.; Cao, H.; Liu, S.; He, S.; Huang, Z.; et al. 2024b. Mapping the increasing use of llms in scientific papers. arXiv preprint arXiv:2404.01268
-
[25]
Liu, J., and Bu, Y. 2024. Towards the relationship between aigc in manuscript writing and author profiles: evidence from preprints in llms. arXiv preprint arXiv:2404.15799
2024 arXiv
-
[26]
Mahajan, S. 2023. Artificial intelligence and its impacts on the society. Contemporary Social Sciences 32(4):135--151
2023
-
[27]
Makridakis, S. 2017. The forthcoming artificial intelligence (ai) revolution: Its impact on society and firms. Futures 90:46--60
2017
-
[28]
Matsui, K. 2024. Delving into pubmed records: Some terms in medical writing have drastically changed after the arrival of chatgpt. medRxiv 2024--05
2024
-
[29]
Menzel, K. 2022. Medical discourse in late modern english: Insights from a multidisciplinary corpus of scientific journal articles. In Corpus pragmatic studies on the history of medical discourse . John Benjamins. 79--104
2022
-
[30]
Merriam-Webster . 2025. Merriam-Webster Thesaurus . https://www.merriam-webster.com/thesaurus. Accessed: 2025-01-11
2025
-
[31]
Milroy, J., and Milroy, L. 1985. Linguistic change, social network and speaker innovation1. Journal of linguistics 21(2):339--384
1985
-
[32]
V.; and Peters, H
Montani, I.; Honnibal, M.; Boyd, A.; Landeghem, S. V.; and Peters, H. 2023. explosion/spacy: v3.7.2: Fixes for apis and requirements. Version v3.7.2, Zenodo
2023
-
[33]
National Library of Medicine . 2024. PubMed Database . https://pubmed.ncbi.nlm.nih.gov/. Accessed: 2024-12-06
2024
-
[34]
Pachegowda, C. 2023. The global impact of ai-artificial intelligence: Recent advances and future directions, a review. arXiv preprint arXiv:2401.12223
2023 arXiv
-
[35]
Shapira, P. 2024. Delving into "delve". Accessed: 2025-01-23
2024
-
[36]
G., and Kaufman, T
Thomason, S. G., and Kaufman, T. 2023. Language contact, creolization, and genetic linguistics . Univ of California Press
2023
-
[37]
E.; Haberland, M.; Reddy, T.; Cournapeau, D.; Burovski, E.; Peterson, P.; Weckesser, W.; Bright, J.; van der Walt , S
Virtanen, P.; Gommers, R.; Oliphant, T. E.; Haberland, M.; Reddy, T.; Cournapeau, D.; Burovski, E.; Peterson, P.; Weckesser, W.; Bright, J.; van der Walt , S. J.; Brett, M.; Wilson, J.; Millman, K. J.; Mayorov, N.; Nelson, A. R. J.; Jones, E.; Kern, R.; Larson, E.; Carey, C. J...
2020
-
[38]
Winford, D. 2003. An introduction to contact linguistics. (No Title)
2003
-
[39]
Yakura, H.; Lopez-Lopez, E.; Brinkmann, L.; Serna, I.; Gupta, P.; and Rahwan, I. 2024. Empirical evidence of large language model's influence on human spoken communication. arXiv preprint arXiv:2409.01754
2024 arXiv
-
[40]
Zhang, X.; Xiong, W.; Chen, L.; Zhou, T.; Huang, H.; and Zhang, T. 2024. From lists to emojis: How format bias affects model alignment. arXiv preprint arXiv:2409.11704
2024 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.