REVIEW 4 major objections 4 minor 2 cited by
Human-LLM Coevolution: Evidence from Academic Writing
T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Academic writers began avoiding ChatGPT's telltale words within weeks of public criticism.
desk verdict A useful descriptive update on LLM-word frequencies, but the coevolution claim rests on a timing correlation rather than a tested causal effect. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a monthly word-frequency time series: each target word's occurrences are normalized per 10,000 abstracts and tracked from 2018 through 2024. The paper also defines a ratio $R_{ij}(T_1,T_2)=f_{ij}(T_1)/f_{ij}(T_2)$, the frequency of word $i$ in category $j$ between two periods, used to compare computer-science abstracts with other disciplines and pre/post-2024 windows. Around this core sit two control experiments: abstracts rewritten by GPT-4o-mini with and without a prompt prohibiting the flagged words, and detection scores from the Binoculars detector, a zero-shot machine-generated-text detector, which show that revised abstracts are not reliably distinguished from human ones. The mechanism the argument rests on is temporal alignment: the drop begins in April 2024, when the words became publicly visible as LLM giveaways, rather than in May 2024, when a new model version appeared.
What would settle it
If the April 2024 decline is driven by deliberate author adaptation, then tracking individual authors who published abstracts both before and after April 2024 should show those authors reducing their own use of flagged words after the public discussion, while authors with no known LLM exposure should not. A controlled generation test would also work: if new LLM versions rather than human avoidance caused the drop, then prompting an early-2024 model and a late-2024 model on the same abstract-writing task should reproduce the word-frequency decline without any human editing step.
Extended reading notes
Core claim
Analyzing monthly word frequencies in 1,294,653 preprint abstracts submitted between January 2018 and December 2024, the paper finds that the words most strongly associated with ChatGPT—'delve', 'intricate', 'pivotal', 'realm', 'showcasing', and the six peer-review words 'commendable', 'innovative', 'meticulous', 'notable', 'versatile'—reached a peak in early 2024 and then began to decline from April 2024 onward, the same period in which these words were publicly singled out as LLM markers. At the same time, words the authors call relatively common, such as 'significant' and 'additionally', continued to rise, and the long-term decline of 'is' and 'are' continued. The paper interprets this split as evidence that researchers who use LLMs are selectively avoiding vocabulary that would mark their text as machine-generated, while remaining insensitive to less conspicuous LLM-favored words. That asymmetry, the authors argue, is a concrete sign of human-LLM coevolution rather than one-way contamination.
Load-bearing premise
The load-bearing premise is that the timing of the word-frequency drop is caused by researchers noticing public discussion of LLM-favored words and deliberately changing their writing, rather than by alternative causes such as new model versions, topic shifts, seasonal patterns, editorial policy, or changes in who submits abstracts—the paper does not test these alternatives statistically.
Editorial extensions
If this is right
- Flagged-vocabulary detectors will lose accuracy over time, because the very signal they key on is being actively suppressed by writers.
- Corpus-level word-frequency measurement remains a viable way to estimate LLM impact, but it should weight common, less conspicuous words such as 'significant' rather than distinctive words such as 'delve'.
- The divergent trajectories of flagged words (down) and common LLM-favored words (up) are themselves evidence that awareness and adaptation, not just model exposure, shape academic language.
- Real-world machine-generated text detection is harder than benchmark evaluations suggest, since authors mix, edit, and selectively avoid telltale phrasing.
- Prompt-level choices that forbid certain words reduce but do not eliminate LLM-style vocabulary, so even 'cleaned' outputs retain measurable traces.
Reading between the lines
- Editorial inference: the same adaptation loop should be visible outside academia wherever LLM output is publicly discussed—for example, in blog posts, workplace writing, or social media—so the paper's timeline method could be reapplied to other public corpora.
- Editorial inference: the paper's causal story could be sharpened by correlating the speed and size of each word's decline with the citation counts or online attention of the papers that flagged it; if attention drives adaptation, more publicized words should fall faster.
- Editorial inference: author-level panel data identifying repeat preprint authors could distinguish genuine behavioural change from compositional shifts in who submits, a test that aggregate monthly frequencies cannot perform.
- Editorial inference: if detectors are retrained periodically on fresh human text, they may track coevolution, but this creates an arms race in which the definition of 'human-like' keeps moving; the paper points toward continuous monitoring rather than static benchmarks.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper analyzes monthly word frequencies in arXiv abstracts from 2018 to 2024, focusing on words that earlier studies had flagged as characteristic of ChatGPT output. It reports that words such as "delve" and "intricate" declined after April 2024, shortly after those words were publicly highlighted, while common words such as "significant" and "additionally" continued to rise. The authors interpret this divergence as evidence that researchers have noticed the flagged vocabulary and now avoid it, which they take as evidence of human-LLM coevolution and as a challenge for machine-generated-text detectors. Supporting analyses compare flagged words with withdrawn-paper abstracts, contrast computer science with other disciplines, and test a small detector experiment using GPT-4o-mini revisions and the Binoculars detector.
Significance. The descriptive frequency curves are a useful and timely documentation of shifts in academic vocabulary after LLM usage became widespread, and the paper makes a plausible, falsifiable suggestion that vocabulary-based signals of LLM use are not static. A particular strength is the prompt experiment in Section 4: using a suppression prompt (P2) demonstrably reduces the frequency of flagged words, which serves as a positive control for the mechanism the authors invoke. If the causal interpretation were secured, the paper would have broader implications for LLM-text detection and for measuring LLM influence in scholarly corpora. However, the causal claim that researchers adapted because the words were publicly flagged is not yet supported by the statistical evidence presented.
major comments (4)
- [Section 3, Figure 2] The load-bearing claim that flagged words began to decrease from April 2024 because researchers noticed public discussion is not backed by any statistical test or control series. The words were selected because they rose sharply in 2023, so a decline after the peak is also what pure mean reversion would produce; the paper needs change-point tests, confidence bands, and comparison with a matched set of non-flagged control words before the causal interpretation can be supported.
- [Section 3, paragraph contrasting with Leiter et al.] The rebuttal that a model-release explanation "would have been delayed" is not a tested prediction. GPT-4o was released in May 2024 and could immediately change outputs if users switched to it or if system prompts changed; without adoption data or a placebo analysis around release dates, the timing argument does not discriminate between author adaptation and model change.
- [Section 3, Figures 2e-f] The withdrawn-paper comparison is not a counterfactual for the main corpus. Withdrawn papers are a small, topically and quality-selected subset, the curves are smoothed with 12-month rolling averages, and there are no error bars, so the comparison cannot rule out demographic, topic, or policy confounds.
- [Section 4, Figure 5] The conclusion that Binoculars "on average does not return any difference" and that true-positive differences are "not significant" is stated without reporting the score distributions, sample sizes, or a significance test. Since the detector experiment is the paper's main evidence for the MGT-detection challenge, these quantities need to be reported.
minor comments (4)
- [Table 1 caption] The citation counts are dated January 16, 2022, but the cited papers were published in 2024; this appears to be a typo (likely 2025) and should be corrected.
- [Section 3, Figure 3b] The text says the ratio uses the frequency in the first quarter of 2024 divided by the frequency from January 2023 to December 2024; because the numerator and denominator cover periods of different lengths, the definition of R in the displayed equation should be clarified.
- [Section 4, Section 1] There are small language errors: "state-of-art" should be "state-of-the-art" in Section 4, and "scribed to LLMs usage" in Section 1 should presumably be "ascribed to LLM usage."
- [Figure 2 caption] The caption refers to "Word groups a and b" but the panels are labeled (a)-(f); using consistent panel labels would improve readability.
Circularity Check
No significant circularity: the word-frequency analyses are self-contained empirical measurements, and the causal interpretation, while open to alternative explanations, is not built into the definitions or equations.
full rationale
The paper tracks the monthly frequency of words that were identified as LLM-associated in earlier work, including Geng and Trotta (2024) by the same first author. This is not circular: the earlier papers provide the word lists, but the present analysis measures those words' frequencies in arXiv abstracts over 2018-2024 independently. No parameter is fitted to make the post-April 2024 decline emerge, and no equation equals a predicted quantity to an input by construction. The claim that researchers intentionally avoided LLM-typical words is an interpretation of a timing correlation, not an algebraic consequence of the method. The self-citations are used to identify which words to track, not to prove the central empirical claim; the current paper's own data and figures carry the analysis. The P2 prompt experiment deliberately suppresses the flagged words and demonstrates that prompting can reduce them, which is an experimental manipulation rather than a hidden circular prediction. Therefore, no load-bearing step reduces to its own inputs.
Assumptions & free parameters
free parameters (2)
- Minimum word frequency inclusion threshold =
20 occurrences per 10,000 abstracts
- Rolling average window for withdrawn papers =
12 months
assumptions (4)
- domain assumption The Kaggle arXiv metadata accurately represents arXiv submissions and their abstract text.
- domain assumption Normalizing word frequency per 10,000 abstracts is a valid way to compare writing style over time.
- ad hoc to paper The word lists from Liang et al. (2024a,b) and Geng and Trotta (2024) accurately characterize words favored or disfavored by ChatGPT.
- ad hoc to paper Temporal correlation between public discussion and word-frequency decline indicates author adaptation.
Cite this review
Pith. "Pith review of Human-LLM Coevolution: Evidence from Academic Writing." pith.science (2026). https://pith.science/paper/BWFKL6GT
@misc{pith2026250209606,
author = {Pith},
title = {Pith review of: Human-LLM Coevolution: Evidence from Academic Writing},
year = {2026},
howpublished = {\url{https://pith.science/paper/BWFKL6GT}},
note = {Machine review of arXiv:2502.09606}
}
read the original abstract
With a statistical analysis of arXiv paper abstracts, we report a marked drop in the frequency of several words previously identified as overused by ChatGPT, such as "delve", starting soon after they were pointed out in early 2024. The frequency of certain other words favored by ChatGPT, such as "significant", has instead kept increasing. These phenomena suggest that some authors of academic papers have adapted their use of large language models (LLMs), for example, by selecting outputs or applying modifications to the LLM-generated content. Such coevolution and cooperation of humans and LLMs thus introduce additional challenges to the detection of machine-generated text in real-world scenarios. Estimating the impact of LLMs on academic writing by examining word frequency remains feasible, and more attention should be paid to words that were already frequently employed, including those that have decreased in frequency due to LLMs' disfavor.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 2 Pith papers
-
Who Gets Seen in the Age of AI? Adoption Patterns of Large Language Models in Scholarly Writing and Citation Outcomes
Analyzing 98,000 Scopus computer science papers, the paper finds a global rise in AI-like writing after ChatGPT and reports regional differences in citation returns, but the key differential-gain result is statistical...
-
AI4Research: A Survey of Artificial Intelligence for Scientific Research
A survey that organizes AI-for-research work into five tasks, comprehension, survey, discovery, writing, and peer review, and compiles associated tools and benchmarks.
Reference graph
Works this paper leans on
-
[1]
Shammur Absar Chowdhury, Hind Almerekhi, Mucahid Kutlu, Kaan Efe Keles, Fatema Ahmad, Tasnim Mohiuddin, George Mikros, and Firoj Alam. 2024. Genai content detection task 2: Ai vs. human--academic essay authenticity challenge. arXiv preprint arXiv:2412.18274
work page Pith review arXiv 2024
-
[2]
Aldan Creo and Shushanta Pudasaini. 2025. Silverspeak: Evading ai-generated text detectors using homoglyphs. In Proceedings of the 1stWorkshop on GenAI Content Detection (GenAIDetect), pages 1--46
work page 2025
-
[3]
Jad Doughman, Osama Mohammed Afzal, Hawau Olamide Toyin, Shady Shehata, Preslav Nakov, and Zeerak Talat. 2024. Exploring the limitations of detecting machine-generated text. arXiv preprint arXiv:2406.11073
work page Pith review arXiv 2024
-
[4]
Mingmeng Geng, Caixi Chen, Yanru Wu, Dongping Chen, Yao Wan, and Pan Zhou. 2024. The impact of large language models in academia: from writing to speaking. arXiv preprint arXiv:2409.13686
arXiv 2024
-
[5]
Mingmeng Geng and Roberto Trotta. 2024. Is chatgpt transforming academics' writing style? arXiv preprint arXiv:2404.08627
arXiv 2024
-
[6]
Soumya Suvra Ghosal, Souradip Chakraborty, Jonas Geiping, Furong Huang, Dinesh Manocha, and Amrit Singh Bedi. 2023. Towards possibilities & impossibilities of ai-generated text detection: A survey. arXiv preprint arXiv:2310.15264
arXiv 2023
-
[7]
Andrew Gray. 2024. Chatgpt" contamination": estimating the prevalence of llms in the scholarly literature. arXiv preprint arXiv:2403.16887
arXiv 2024
-
[8]
Yanzhu Guo, Guokan Shang, and Chlo \'e Clavel. 2024. Benchmarking linguistic diversity of large language models. arXiv preprint arXiv:2412.10271
arXiv 2024
Show all 30 references
-
[9]
Abhimanyu Hans, Avi Schwarzschild, Valeriia Cherepanova, Hamid Kazemi, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. 2024. Spotting llms with binoculars: Zero-shot detection of machine-generated text. arXiv preprint arXiv:2401.12070
2024 arXiv
-
[10]
Dmitry Kobak, Rita Gonz \'a lez-M \'a rquez, Em o ke- \'A gnes Horv \'a t, and Jan Lause. 2024. Delving into chatgpt usage in academic writing through excess vocabulary. arXiv preprint arXiv:2406.07016
2024 arXiv
-
[11]
Christoph Leiter, Jonas Belouadi, Yanran Chen, Ran Zhang, Daniil Larionov, Aida Kostikova, and Steffen Eger. 2024. Nllg quarterly arxiv report 09/24: What are the most influential current ai papers? arXiv preprint arXiv:2412.12121
2024 arXiv
-
[12]
Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, et al. 2024 a . Monitoring ai-modified content at scale: A case study on the impact of chatgpt on ai conference peer reviews. arXiv preprint arXi...
2024 arXiv
-
[13]
Weixin Liang, Yaohui Zhang, Zhengxuan Wu, Haley Lepp, Wenlong Ji, Xuandong Zhao, Hancheng Cao, Sheng Liu, Siyu He, Zhi Huang, et al. 2024 b . Mapping the increasing use of llms in scientific papers. arXiv preprint arXiv:2404.01268
2024 arXiv
-
[14]
Zhehui Liao, Maria Antoniak, Inyoung Cheong, Evie Yu-Yen Cheng, Ai-Heng Lee, Kyle Lo, Joseph Chee Chang, and Amy X Zhang. 2024. Llms as research tools: A large scale survey of researchers' usage and perceptions. arXiv preprint arXiv:2411.05025
2024 arXiv
-
[15]
Jialin Liu and Yi Bu. 2024. Towards the relationship between aigc in manuscript writing and author profiles: evidence from preprints in llms. arXiv preprint arXiv:2404.15799
2024 arXiv
-
[16]
Yule Liu, Zhiyuan Zhong, Yifan Liao, Zhen Sun, Jingyi Zheng, Jiaheng Wei, Qingyuan Gong, Fenghua Tong, Yang Chen, Yang Zhang, et al. 2024. On the generalization ability of machine-generated text detectors. arXiv preprint arXiv:2412.17242
2024 arXiv
-
[17]
Dino Pedreschi, Luca Pappalardo, Emanuele Ferragina, Ricardo Baeza-Yates, Albert-L \'a szl \'o Barab \'a si, Frank Dignum, Virginia Dignum, Tina Eliassi-Rad, Fosca Giannotti, J \'a nos Kert \'e sz, et al. 2024. Human-ai coevolution. Artificial Intelligence, page 104244
2024
-
[18]
Delip Rao, Jonathan Young, Thomas Dietterich, and Chris Callison-Burch. 2024. Withdrarxiv: A large-scale dataset for retraction study. arXiv preprint arXiv:2412.03775
2024 arXiv
-
[19]
Pedro Reviriego, Javier Conde, Elena Merino-G \'o mez, Gonzalo Mart \' nez, and Jos \'e Alberto Hern \'a ndez. 2024. Playing with words: Comparing the vocabulary and lexical diversity of chatgpt and humans. Machine Learning with Applications, 18:100602
2024
-
[20]
Karolina Rudnicka. 2023. Can grammarly and chatgpt accelerate language change? ai-powered technologies and their impact on the english language: wordiness vs. conciseness. Procesamiento de Lenguaje Natural, 71
2023
-
[21]
Jenna Russell, Marzena Karpinska, and Mohit Iyyer. 2025. People who frequently use chatgpt for writing tasks are accurate and robust detectors of ai-generated text. arXiv preprint arXiv:2501.15654
2025 arXiv
-
[22]
Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. 2023. Can ai-generated text be reliably detected? arXiv preprint arXiv:2303.11156
2023 arXiv
-
[23]
Ruixiang Tang, Yu-Neng Chuang, and Xia Hu. 2024. The science of detecting llm-generated text. Communications of the ACM, 67(4):50--59
2024
-
[24]
Yichen Wang, Shangbin Feng, Abe Bohan Hou, Xiao Pu, Chao Shen, Xiaoming Liu, Yulia Tsvetkov, and Tianxing He. 2024. Stumbling blocks: Stress testing the robustness of machine-generated text detectors under attacks. arXiv preprint arXiv:2402.11638
2024 arXiv
-
[25]
Yuxia Wang, Artem Shelmanov, Jonibek Mansurov, Akim Tsvigun, Vladislav Mikhailov, Rui Xing, Zhuohan Xie, Jiahui Geng, Giovanni Puccetti, Ekaterina Artemova, et al. 2025. Genai content detection task 1: English and multilingual machine-generated text detection: Ai vs. human. ar...
2025 arXiv
-
[26]
Debora Weber-Wulff, Alla Anohina-Naumeca, Sonja Bjelobaba, Tom \'a s Folt \`y nek, Jean Guerrero-Dib, Olumide Popoola, Petr S igut, and Lorna Waddington. 2023. Testing of detection tools for ai-generated text. International Journal for Educational Integrity, 19(1):26
2023
-
[27]
Qihui Zhang, Chujie Gao, Dongping Chen, Yue Huang, Yixin Huang, Zhenyang Sun, Shilin Zhang, Weiye Li, Zhengyan Fu, Yao Wan, et al. 2024 a . Llm-as-a-coauthor: Can mixed human-written and machine-generated text be detected? In Findings of the Association for Computational Lingu...
2024
-
[28]
Yuehan Zhang, Yongqiang Ma, Jiawei Liu, Xiaozhong Liu, Xiaofeng Wang, and Wei Lu. 2024 b . Detection vs. anti-detection: Is text generated by ai detectable? In International Conference on Information, pages 209--222. Springer
2024
-
[29]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[30]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.