A Penalty Goes a Long Way: Measuring Lexical Diversity in Synthetic Texts Under Prompt-Influenced Length Variations

Anna Rumshisky; Ishita Dasgupta; Saayan Mitra; Somdeb Sarkhel; Uttaran Bhattacharya; Vijeta Deshpande

REVIEW 4 major objections 6 minor 40 references

A Penalty Goes a Long Way: Measuring Lexical Diversity in Synthetic Texts Under Prompt-Influenced Length Variations

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Penalty-Adjusted Type-Token Ratio (PATTR) folds a task's target response length into the diversity score and, on synthetic LLM video scripts, selects more diverse outputs than MATTR or compression ratio while suppressing the usual…

desk verdict PATTR is a straightforward, interpretable length-penalized TTR worth knowing, but the empirical case for its superiority is weakened by a mismatched filtering comparison and by length bias that depends heavily on LT. read the letter →

arxiv 2507.15092 v1 pith:VSU7VYZI submitted 2025-07-20 cs.CL

Vijeta Deshpande , Ishita Dasgupta , Uttaran Bhattacharya , Somdeb Sarkhel , Saayan Mitra , Anna Rumshisky This is my paper

classification cs.CL

keywords lexicaldiversitytype-tokenratiolengthbiassynthetictextdatafilteringpromptengineeringPATTR

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Conventional lexical diversity metrics such as Type-Token Ratio, Moving-Average TTR, and Compression Ratio systematically reward short texts, so when prompt changes alter response length they rank the least informative outputs as the most diverse. This paper argues that the bias can be corrected by folding the task's target response length directly into the metric: Penalty-Adjusted Type-Token Ratio (PATTR) divides the number of unique words by the word count plus the absolute deviation from the target length. On a large synthetic corpus of video scripts generated from seven instruction-tuned language models, the authors show that PATTR's correlation with length can be tuned from negative to near-zero to positive by choosing $L_T$, and that top-10/100/1,000 selections made with PATTR are on par or better by ROUGE, BLEU, entropy, n-gram diversity, and Wasserstein distance than selections made with MATTR or CR. If the claim holds, practitioners filtering synthetic data for model training can choose diversity scores that respect the length a task actually requires instead of inheriting a hidden preference for short text.

What carries the argument

The load-bearing object is the penalty-adjusted denominator $|w| + \|\,|w|-L_T\,\|_1$, where $L_T$ is a user-specified target length for the task. Replacing the plain TTR denominator with this term makes the score depend on absolute distance from the target rather than on raw length, so the bias toward short responses is controlled by one tunable number. The paper's empirical machinery is a structured prompting setup: a fixed task description, incrementally added formatting instructions, varied style inputs, and varied user topics generate 12,000 responses per model, making response length and diversity observable under controlled prompt perturbation. The metric is evaluated by ranking and filtering responses, then measuring homogenization with pairwise similarity and distributional diversity.

What would settle it

Take a corpus with human-annotated lexical diversity and a known target length. If, with $L_T$ set correctly, PATTR's top-100 selections do not beat MATTR and CR on pairwise-similarity and entropy measures under equal length constraints, the central filtering claim fails. A sharper check: on a pool where all responses share the true target length but lengths vary, PATTR's rank order should stay stable as lengths deviate; if changing $L_T$ by a small amount reverses which responses rank top, the metric is not robust to the parameter it depends on.

Watch

Extended reading notes

Core claim

The central discovery is that a single penalty term $\|\,|w|-L_T\,\|_1$ in the denominator of TTR turns a length-biased score into a length-aware one. PATTR is defined as $\mathrm{PATTR}(w, L_T) = |\mathrm{set}(w)| / (|w| + \|\,|w| - L_T\,\|_1)$, where $L_T$ is the task's target response length. Shorter-than-target and longer-than-target responses are penalized symmetrically, so the metric no longer automatically crowns the shortest script; the paper shows empirically that varying $L_T$ from 100 to 275 to 400 flips PATTR's Spearman correlation with response length from $-0.42$ to $+0.03$ to $+0.91$. In filtering experiments, PATTR with $L_T=400$ selects top-10/100/1,000 video scripts that have lower pairwise ROUGE and BLEU similarity, higher per-token entropy, higher n-gram diversity, and a smaller Wasserstein distance to a uniform vocabulary distribution than selections made with MATTR or compression ratio, while staying closer to the target length. The authors also show PATTR's selected outputs are both longer and significantly more diverse than MATTR's, which they take as evidence that the improvement is not a shortcut through short-text bias.

Load-bearing premise

The whole correction rests on choosing a target response length $L_T$ that is genuinely appropriate for the task and fair across all models and prompts; if that number is wrong, the penalty simply re-injects length bias in the opposite or an unintended direction.

Editorial extensions

If this is right

Data curation pipelines that filter synthetic text by diversity can set $L_T$ to the desired response length, so short filler and overlong rambling outputs are both downweighted instead of the short ones being automatically rewarded.
Analyses of prompt engineering effects on diversity become interpretable: a change in measured diversity can be separated from a change in length once length bias is explicitly parameterized.
For tasks with known length constraints, such as essay or short-story generation, PATTR offers a per-response ranking that does not need pairwise corpus comparisons, so it scales linearly with the number of strings.
Because PATTR remains a per-response score, it can rank large synthetic corpora in linear time, unlike pairwise similarity methods whose cost grows quadratically with corpus size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension is to use PATTR as the filter inside synthetic-data distillation loops, checking whether training on PATTR-filtered data slows the diversity collapse that repeated self-training induces.
The same penalty idea could be transferred to syntactic and semantic diversity measures; if those metrics also have a length component, a deviation penalty of the same form may remove bias without re-tuning.
For open-ended tasks where no natural $L_T$ exists, one could set $L_T$ from the model family's own length distribution, which would make PATTR a normalization rather than a task constraint; the paper does not explore this.
The direction of the bias is not fixed: the paper's correlation results imply a user could deliberately set $L_T$ low or high to favor short or long responses, which may be undesirable in evaluation but useful for controlled data augmentation.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know. First, PATTR is a clean and simple idea: add |L - LT| to the denominator of TTR so that deviations from a task-specific target length are penalized. That is genuinely new relative to MATTR, MTLD, vocd, and the other length-correction heuristics in the related work. Second, the paper's central claim that PATTR is 'robust to length variations' is not actually supported by its own evidence. The metric does not remove length dependence; it swaps the sign of that dependence based on LT. Table 1 shows this directly: Spearman correlation with response length goes from -0.42 at LT=100 to +0.91 at LT=400. So robustness is inherited entirely from the user's choice of LT, a point the Limitations section does concede. The empirical sweep is broad and that should be credited. Seven models, more than 100k scripts (the abstract says 20M words, Section 3.1 says over 50M - an inconsistency worth fixing), and several external diversity measures. The sensitivity analysis in Table 4 also shows that with a sensible LT, PATTR can select longer and more diverse responses on ROUGE-L. There is real potential here for synthetic-data practitioners. The soft spots are proportionate to the overclaim. The filtering comparison in Figures 3 and E.1 is confounded: MATTR and CR are evaluated after length-binning the corpus, while PATTR is evaluated without any length filter. Since ROUGE, BLEU, and entropy are length-sensitive, this can make PATTR look better simply because its selected set has more length spread, not because it is lexically better. When the comparison is forced onto a common 350-450 word band (Table 3), PATTR is at parity or slightly below MATTR on 4/6-gram diversity - undercutting the 'consistently outperforms' claim. The win-rate results in Table 2 are also near-tautological for large LT: a metric that heavily penalizes short responses will naturally avoid ranking short responses first, so the low win rates do not independently demonstrate diversity. There are also smaller issues: no released code or data, despite the dataset being a stated contribution; no error bars in the main figures; and an internal inconsistency where the text says 11 models but lists only 7. None of this is fatal. The metric is well-motivated, the evaluation strategy is fixable, and the core idea is transparent enough to be useful. A serious referee could push for a matched-length headline comparison, a systematic LT sensitivity analysis, and released artifacts. I would engage with it: send to peer review, expect conditional acceptance after revision, and possibly bring it up in a reading group to discuss evaluation design for length-sensitive metrics. For my own work, I probably wouldn't cite it until the empirical case is tightened.

Referee Report

4 major / 6 minor

Summary. The paper introduces Penalty-Adjusted Type-Token Ratio (PATTR), defined as PATTR(w, L_T) = |set(w)| / (|w| + ||w| - L_T|), where L_T is a user-specified target response length. On a synthetic corpus of video scripts generated by seven LLaMA, OLMo, and Phi instruct models, the authors compare PATTR with Moving-Average TTR (MATTR) and Compression Ratio (CR) in two settings: measuring bias toward short responses (win rates) and filtering top-10/100/1,000 responses for corpus diversity (ROUGE, BLEU, entropy, n-gram diversity, Wasserstein distance). The central claim is that PATTR is robust to prompt-induced length variations and yields on-par or better filtered diversity than MATTR and CR.

Significance. If the central claim were established, PATTR would be a simple, practical addition to the toolbox of lexical diversity metrics for synthetic-text curation, and the released corpus would be a useful resource. The paper has real strengths: the data generation is systematic (12,000 prompts per model, controlled variations in instructions, style, and topic), the evaluation is broad (five diversity measures, multiple model families, sensitivity analyses in Table 4 and Appendix F), and the limitations section is candid about the need to specify L_T and about the lack of human-judgment validation. However, the load-bearing claim of robustness is not supported by the evidence as presented: the sign of PATTR's length correlation changes with L_T (Table 1), and the headline filtering comparison is confounded by asymmetric length filtering between PATTR and the baseline metrics. These issues affect the main conclusions and require a substantive revision.

major comments (4)

[§3.2, Eq. (6); §4.1, Table 1] The claim that PATTR is 'robust to text length variations' (Abstract, §4.2) is not a property of the metric alone; it is inherited from the user's choice of L_T. Equation (6) is piecewise: for |w| < L_T the denominator equals L_T, so PATTR increases with vocabulary size and therefore with length; for |w| > L_T the denominator is 2|w| - L_T, so PATTR decreases with length. Table 1 confirms this: the Spearman correlation between PATTR and response length is -0.42 at L_T=100, near 0 at L_T=275, and +0.91 at L_T=400. Thus a misspecified L_T re-injects length bias in either direction. The paper should either reframe the contribution as 'a tunable length-penalized TTR that lets practitioners control length bias via L_T' or provide a principled procedure for setting L_T that does not depend on the corpus median. As written, the robustness claim is not established.
[§4.3, Fig. 3 and Fig. E.1] The main filtering comparison is confounded. In Figures 3 and E.1, MATTR and CR are applied only after restricting responses to length bins (0-2K, 200-600, 300-500, 350-450), while PATTR is evaluated on the unfiltered corpus (horizontal orange line). ROUGE, BLEU, and entropy are length-sensitive: selecting responses clustered near L_T=400 can lower pairwise similarity scores even if the ranking is not lexically better. To support the 'consistently outperforms' claim, PATTR must be compared with MATTR and CR under identical length constraints, with the length distributions of the selected sets reported for all metrics.
[§4.3, Evaluation-3, Table 3] When all three metrics are forced onto the same 350-450 word band, PATTR's 4-gram and 6-gram diversity (3.65 and 5.64) are below MATTR (3.66 and 5.65) and only slightly above CR (3.53 and 5.52). This directly undercuts the claim that PATTR provides consistently better or on-par diversity, and it suggests that PATTR's apparent advantage in Figure 3 is substantially driven by length selection rather than lexical ranking. The paper should report whether the differences are statistically significant and discuss this table more prominently, since it is the only matched-length comparison.
[§4.2, Tables 2 and D.1] The win-rate results for PATTR at L_T=400 and L_T=600 are near-tautological. Since the penalty term ||w|-L_T| grows with deviation from L_T, and most responses in each model are shorter than 400 or 600 words (Table B.1), the metric by construction penalizes short responses. Low win rates for short sequences therefore do not independently demonstrate that PATTR identifies lexically diverse texts; they demonstrate that the penalty discourages short selections. A more informative control would compare PATTR against a baseline that applies the same length penalty to an otherwise length-neutral lexical score (e.g., TTR with the same denominator), or would report the lexical-diversity gain per unit of length difference.

minor comments (6)

[Abstract and §3.1] The Abstract states the corpus contains 'over 20M words,' while §3.1 says 'more than 50 million words (measured with whitespace-separated words).' These numbers should be reconciled.
[§3.1] The text says 'we use this set of prompts to generate video-scripts from 11 language models' but then lists seven models (OLMo-2 7B/13B, Llama-3.1 8B, Llama-3.2 1B/3B, Phi-3 Mini/Medium). The count '11' appears to be a typo.
[§4.1, Table 1] The table header includes a 'p-value' column, but the body reports only significance stars; either report actual p-values or change the column header.
[§3.2, Eq. (6)] The piecewise behavior of PATTR is not discussed explicitly. For |w| < L_T, PATTR = |set(w)| / L_T; for |w| > L_T, PATTR = |set(w)| / (2|w| - L_T). Stating this would make the length-dependence transparent and would help readers calibrate the metric's behavior.
[§4.3, Fig. 3 caption] The caption says 'Except for ROUGE-1 with 0-2K filter, PATTR consistently outperforms MATTR and CR,' but the main text reports that PATTR outperforms in 14 of 16 scenarios. These statements should be aligned so the reader knows which scenarios are exceptions.
[Appendix F, Table F.1] For Phi-3-med, several ROUGE-1 and ROUGE-2 t-statistics are positive (e.g., +16.08 at L_T=400, W=128), and the surrounding text explains this as likely due to length bias in the metrics. Since this is an interpretation rather than a demonstrated fact, the paper should report the actual length distributions and, where possible, a length-matched comparison to support the claim.

Circularity Check

1 steps flagged · score 3.0 of 10

PATTR's length-bias behavior is built into its own definition, while the headline filtering claim is tested against external metrics; partial by-construction demonstration, not full circularity.

self definitional [Section 3.2, Eq. (6); Section 4.1, Table 1; Section 4.2, Table 2]
"P AT T R(w, LT ) = |set (w)| / (|w| + P (|w| , LT )) ... For PATTR, we show that its correlation with text length varies based on the target response length LT. Using responses generated by the Llama-3.1-8B model as an example, we observe that setting LT to 100, 275, and 400 respectively yields a strong negative, near-zero, and a strong positive correlation with the response length (Table 1)."

The sign and magnitude of the PATTR-length correlation are imposed by Eq. (6). For |w| < LT, the denominator is constant at LT, so PATTR = |set(w)|/LT and therefore grows with vocabulary size, which itself grows with length; for |w| > LT, the denominator is 2|w|-LT, which penalizes length. Hence the 'negative, neutral, positive' pattern in Table 1 is an analytic restatement of the formula, not an independent empirical discovery. Likewise, Table 2's near-zero win rates for LT=600 follow mechanically from the penalty excluding short responses regardless of their lexical content, making the 'length-bias mitigation' demonstration partly definitional.

full rationale

The paper proposes a new metric, PATTR, instead of deriving a prediction from existing theory, so most of the derivation chain is self-contained. The central claim that PATTR yields on-par-or-better diversity when filtering top-k responses is evaluated with external, independently defined measures: ROUGE-1/2/L, BLEU, token entropy, n-gram diversity, and Wasserstein distance against a uniform-vocabulary reference. Those measures are not defined in terms of PATTR, so the top-k filtering claim is not circular. There are no load-bearing self-citations and no imported uniqueness theorems. The main circularity-adjacent issue is that the metric's headline property, 'robustness to length variations,' is engineered through the penalty term | |w| - LT | in Eq. (6); Table 1's correlation sign flips and Table 2's win-rate changes are direct consequences of the formula, so these demonstrations are self-definitional rather than independent confirmation. The paper's own Limitations section acknowledges that LT must be supplied by the user and may be unsuitable for open-ended tasks, which confirms that the robustness benefit is conditional on the user's choice rather than emergent. A separate, non-circular validity concern is that Figure 3/E.1 compare unfiltered PATTR against length-filtered MATTR/CR using length-sensitive pairwise metrics, and the appendix footnote concedes that ROUGE/BLEU have length bias; this is a fairness confound, not a circular derivation. Weighing these, the score is 3: one by-construction demonstration weakens the 'length-bias mitigation' evidence, but the central filtering evaluation retains independent content.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central metric rests on one user-chosen free parameter, LT, and on domain assumptions that the evaluation metrics are valid diversity ground truth. No new physical or ontological entities are introduced. The comparison also depends on baseline hyperparameters (MATTR window and CR truncation length).

free parameters (3)

Target response length LT = 400 for main experiments; 100, 200, 275, 600 in sensitivity analyses
User-supplied parameter in Eq. (5)-(6). The paper sets LT=400 for video scripts, which is close to the median response length of several tested models (Table B.1). PATTR's correlation with length and its filtering behavior depend strongly on this value.
MATTR window size W = 128 in most comparisons; 32 in Table 2
Baseline hyperparameter chosen for Moving-Average TTR. The comparison outcome can shift with W, and the paper does not tune baselines.
CR truncation length L = 128
Baseline hyperparameter for Compression Ratio. Treating each response as a single-document corpus and truncating to 128 words is a design choice that affects the comparison.

assumptions (4)

standard math Vocabulary size grows sublinearly with text length (Herdan-Heaps law)
Invoked in Section 2 and Section 4.2 to justify why TTR and CR favor shorter texts.
domain assumption ROUGE, BLEU, per-token entropy, n-gram diversity, and Wasserstein distance to a uniform vocabulary distribution are valid proxies for lexical diversity
Section 3.3 and Evaluations 1-4 use these as ground truth for judging which selection is more diverse; no human annotation or independent validation is provided.
domain assumption A fixed target response length LT is available for the task and can be set before generation
Eq. (5)-(6) require LT; the paper assumes LT=400 for video scripts, and the Limitations section admits this may not hold for open-ended tasks.
ad hoc to paper Uniform vocabulary usage (each word appearing exactly once) is the maximally diverse distribution
Evaluation-4 defines the reference distribution this way to compute the Wasserstein distance; this is a paper-specific definition of ideal diversity.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Penalty Goes a Long Way: Measuring Lexical Diversity in Synthetic Texts Under Prompt-Influenced Length Variations." pith.science (2026). https://pith.science/paper/VSU7VYZI

@misc{pith2026250715092,
  author       = {Pith},
  title        = {Pith review of: A Penalty Goes a Long Way: Measuring Lexical Diversity in Synthetic Texts Under Prompt-Influenced Length Variations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VSU7VYZI}},
  note         = {Machine review of arXiv:2507.15092}
}

abstract

Synthetic text generated by Large Language Models (LLMs) is increasingly used for further training and improvement of LLMs. Diversity is crucial for the effectiveness of synthetic data, and researchers rely on prompt engineering to improve diversity. However, the impact of prompt variations on response text length, and, more importantly, the consequential effect on lexical diversity measurements, remain underexplored. In this work, we propose Penalty-Adjusted Type-Token Ratio (PATTR), a diversity metric robust to length variations. We generate a large synthetic corpus of over 20M words using seven models from the LLaMA, OLMo, and Phi families, focusing on a creative writing task of video script generation, where diversity is crucial. We evaluate per-response lexical diversity using PATTR and compare it against existing metrics of Moving-Average TTR (MATTR) and Compression Ratio (CR). Our analysis highlights how text length variations introduce biases favoring shorter responses. Unlike existing metrics, PATTR explicitly considers the task-specific target response length ($L_T$) to effectively mitigate length biases. We further demonstrate the utility of PATTR in filtering the top-10/100/1,000 most lexically diverse responses, showing that it consistently outperforms MATTR and CR by yielding on par or better diversity with high adherence to $L_T$.

Figures

Figures reproduced from arXiv: 2507.15092 by the authors.

**Figure 1.** Task example and synthetic data generation. We show an example of the video script generation task and highlight key aspects of the synthetic dataset we generate based on this task. counteract these effects, significant prompt engineering efforts are employed during synthetic data curation to maintain a desired level of diversity (Long et al., 2024; Face, 2024). However, modifying prompts to enhance diversity can … view at source ↗

**Figure 2.** Variations in response length and diversity scores w.r.t. the number of instructions to LMs. Left to right y-axis: response length variations, diversity scores using Compression Ratio (CR), using Type-Token Ratio (TTR), and using PATTR. CR (lower ⇒ more diverse) and TTR (higher ⇒ more diverse) favor the shortest responses (gray dot, corresponding to ∼ 4 instructions in this experiment) but PATTR (higher ⇒ more diver… view at source ↗

**Figure 3.** Evaluation of top-10 diverse examples with pairwise similarity scores. Average pairwise similarity scores (ROUGE-1/2/L, BLEU) for the top-10 diverse examples selected by PATTR (LT = 400), MATTR (window length of 128 words), and CR (truncation length of 128 words). The x-axis represents different length constraints (e.g., 200-600: 200 ≤ word count ≤ 600). The y-axis shows similarity scores (lower values indicate grea… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Evaluation of top-1,000 diverse examples with entropy. We measure the diversity of top-1,000 examples selected by PATTR (LT ∈ {200, 400, 600}), MATTR (32-word window), and CR (first 128 words) with entropy (based on SmolLM2-1.7B). Higher values of entropy represent a m…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 16 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, and 1 others. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

arXiv 2024
[4]

Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Mart \' n Bl \'a zquez, Guilherme Penedo, Lewis Tunstall, Andr \'e s Marafioti, Hynek Kydl \' c ek, Agust \' n Piqueres Lajar \' n, Vaibhav Srivastav, and 1 others. 2025. Smollm2: When smol goes big--data-centric training of a small language model. arXiv preprint arXiv:2502.02737

arXiv 2025
[5]

Alexander Bukharin and Tuo Zhao. 2023. Data diversity matters for robust instruction tuning. arXiv preprint arXiv:2311.14736

arXiv 2023
[6]

Tuhin Chakrabarty, Philippe Laban, and Chien-Sheng Wu. 2024. Can ai writing be salvaged? mitigating idiosyncrasies and improving human-ai alignment in the writing process through edits. arXiv preprint arXiv:2409.14509

arXiv 2024
[7]

Tuhin Chakrabarty, Vishakh Padmakumar, Faeze Brahman, and Smaranda Muresan. 2023. Creativity support in the age of large language models: An empirical study involving emerging writers. arXiv preprint arXiv:2309.12570

arXiv 2023
[8]

Michael A Covington and Joe D McFall. 2010 a . Cutting the gordian knot: The moving-average type--token ratio (mattr). Journal of quantitative linguistics, 17(2):94--100

work page 2010

Show all 40 references

[9]

Michael A Covington and Joe D McFall. 2010 b . Cutting the gordian knot: The moving-average type--token ratio (mattr). Journal of quantitative linguistics, 17(2):94--100

2010
[10]

Anil R Doshi and Oliver P Hauser. 2024. Generative ai enhances individual creativity but reduces the collective diversity of novel content. Science Advances, 10(28):eadn5290

2024
[11]

Hugging Face. 2024. Cosmopedia: A knowledge base for open llms. https://huggingface.co/blog/cosmopedia. Accessed: 2025-02-14

2024
[12]

Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. 2024. Scaling synthetic data creation with 1,000,000,000 personas. arXiv preprint arXiv:2406.20094

2024 arXiv
[13]

Carlos G \'o mez-Rodr \' guez and Paul Williams. 2023. A confederacy of models: A comprehensive evaluation of llms on creative writing. arXiv preprint arXiv:2310.08433

2023 arXiv
[14]

P Guiraud. 1960. Problems and methods of linguistic statistics. Dordrecht, the Netherlands: D. Reidel

1960
[15]

Yanzhu Guo, Guokan Shang, Michalis Vazirgiannis, and Chlo \'e Clavel. 2023. The curious decline of linguistic diversity: Training language models on synthetic text. arXiv preprint arXiv:2311.09807

2023 arXiv
[16]

Gustav Herdan. 1960. Type-token mathematics: A textbook of mathematical linguistics.'s-gravenhage: Mouton. Journal of Quantitative Linguistics. London/New York: Routledge

1960
[17]

David Herel and Tomas Mikolov. 2024. Collapse of self-trained language models. arXiv preprint arXiv:2404.02305

2024 arXiv
[18]

Wendell Johnson. 1944. Studies in language behavior: A program of research. Psychological Monographs, 56(2):1--15

1944
[19]

Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. 2023. Understanding the effects of rlhf on llm generalisation and diversity. arXiv preprint arXiv:2310.06452

2023 arXiv
[20]

Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, and 1 others. 2024. Synthetic data (almost) from scratch: Generalized instruction tuning for language models. arXiv preprint arXi...

2024 arXiv
[21]

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81

2004
[22]

Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, and 1 others. 2024. Best practices and lessons learned on synthetic data for language models. arXiv preprint arXiv:2404.07503

2024 arXiv
[23]

Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. 2024. On llms-driven synthetic data generation, curation, and evaluation: A survey. arXiv preprint arXiv:2406.15126

2024 arXiv
[24]

U ber den zusammenhang zwischen wortschatzumfang und l \

Heinz-Dieter Mass. 1972. \"U ber den zusammenhang zwischen wortschatzumfang und l \"a nge eines textes. Zeitschrift f \"u r Literaturwissenschaft und Linguistik , 2(8):73

1972
[25]

Philip M McCarthy. 2005. An assessment of the range and usefulness of lexical diversity measures and the potential of the measure of textual, lexical diversity (MTLD). Ph.D. thesis, The University of Memphis

2005
[26]

Philip M McCarthy and Scott Jarvis. 2007. vocd: A theoretical and empirical evaluation. Language Testing, 24(4):459--488

2007
[27]

Philip M McCarthy and Scott Jarvis. 2010. Mtld, vocd-d, and hd-d: A validation study of sophisticated approaches to lexical diversity assessment. Behavior research methods, 42(2):381--392

2010
[28]

William Orwig, Emma R Edenbaum, Joshua D Greene, and Daniel L Schacter. 2024. The language of creativity: Evidence from humans and large language models. The Journal of creative behavior, 58(1):128--136

2024
[29]

Vishakh Padmakumar and He He. 2023. Does writing with language models reduce content diversity? arXiv preprint arXiv:2309.05196

2023 arXiv
[30]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

2002
[31]

Nikita Salkar, Thomas Trikalinos, Byron C Wallace, and Ani Nenkova. 2022. Self-repetition in abstractive neural summarizers. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2022, page 341. NIH Public Access

2022
[32]

Mohamed El Amine Seddik, Suei-Wen Chen, Soufiane Hayou, Pierre Youssef, and Merouane Debbah. 2024. How bad is training on synthetic data? a statistical analysis of language model collapse. arXiv preprint arXiv:2404.05090

2024 arXiv
[33]

Chantal Shaib, Joe Barrow, Jiuding Sun, Alexa F Siu, Byron C Wallace, and Ani Nenkova. 2024 a . Standardizing the measurement of text diversity: A tool and a comparative analysis of scores. arXiv preprint arXiv:2403.00553

2024
[34]

Chantal Shaib, Yanai Elazar, Junyi Jessy Li, and Byron C Wallace. 2024 b . Detection and measurement of syntactic templates in generated text. arXiv preprint arXiv:2407.00211

2024 arXiv
[35]

Leonid Nisonovich Vaserstein. 1969. Markov processes over denumerable products of spaces, describing large systems of automata. Problemy Peredachi Informatsii, 5(3):64--72

1969
[36]

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560

2022 arXiv
[37]

Junchao Wu, Shu Yang, Runzhe Zhan, Yulin Yuan, Lidia Sam Chao, and Derek Fai Wong. 2025. A survey on llm-generated text detection: Necessity, methods, and future directions. Computational Linguistics, pages 1--65

2025
[38]

Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. 2024. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. arXiv preprint arXiv:2406.08464

2024 arXiv
[39]

Simon Yu, Liangyu Chen, Sara Ahmadian, and Marzieh Fadaee. 2024. Diversify and conquer: Diversity-centric data selection with iterative refinement. arXiv preprint arXiv:2409.11378

2024 arXiv
[40]

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675

2019 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, and 1 others. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

arXiv 2024

[4] [4]

Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Mart \' n Bl \'a zquez, Guilherme Penedo, Lewis Tunstall, Andr \'e s Marafioti, Hynek Kydl \' c ek, Agust \' n Piqueres Lajar \' n, Vaibhav Srivastav, and 1 others. 2025. Smollm2: When smol goes big--data-centric training of a small language model. arXiv preprint arXiv:2502.02737

arXiv 2025

[5] [5]

Alexander Bukharin and Tuo Zhao. 2023. Data diversity matters for robust instruction tuning. arXiv preprint arXiv:2311.14736

arXiv 2023

[6] [6]

Tuhin Chakrabarty, Philippe Laban, and Chien-Sheng Wu. 2024. Can ai writing be salvaged? mitigating idiosyncrasies and improving human-ai alignment in the writing process through edits. arXiv preprint arXiv:2409.14509

arXiv 2024

[7] [7]

Tuhin Chakrabarty, Vishakh Padmakumar, Faeze Brahman, and Smaranda Muresan. 2023. Creativity support in the age of large language models: An empirical study involving emerging writers. arXiv preprint arXiv:2309.12570

arXiv 2023

[8] [8]

Michael A Covington and Joe D McFall. 2010 a . Cutting the gordian knot: The moving-average type--token ratio (mattr). Journal of quantitative linguistics, 17(2):94--100

work page 2010