Pith. sign in

REVIEW 4 major objections 4 minor 30 references

Formalising lexical and syntactic diversity for data sampling in French

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper establishes that a tractable entropy-guided sampling heuristic markedly increases lexical diversity, but that lexical diversity cannot reliably proxy syntactic diversity in French corpus construction.

desk verdict Useful cautionary study of diversity measures, but the headline Q1 result rests on an uncontrolled baseline and the Q2 negative result is underdetermined by aggregation choices. read the letter →

arxiv 2501.08003 v1 pith:RUSJGXEM submitted 2025-01-14 cs.CL

Louis Estève , Manon Scholivet , Agata Savary This is my paper · ORCID
classification cs.CL MSC 68T5094A17
keywords lexicaldiversitysyntacticdatasamplingRényientropyShannonFrenchcorpusdependencytreesconstruction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks how to build a large, diverse French corpus without paying for exhaustive search or full parsing. It proposes a greedy sampling algorithm that keeps documents whose addition raises Rényi entropy over word forms, and shows that the selected corpus reaches lexical entropy 7.98, far above 7.656 for random selection under the normal model (about 347 standard deviations). It then asks whether cheap lexical diversity can substitute for expensive syntactic diversity, where syntactic diversity counts whole dependency subtrees predicted by a parser. Across 705 blocks of an already-parsed French corpus, correlations between lexical and syntactic entropy fluctuate with the Rényi order $\alpha$, with the corpus genre, and even with whether subcorpora are pooled, so the authors conclude that using lexical diversity does not increase syntactic diversity and that an arbitrary diversity measure may miss diversity-related properties. A sympathetic reader takes from this that corpus builders should parse at least a sample whenever syntactic diversity is the actual target.

What carries the argument

The central objects are formal diversity measures imported from ecology, applied to two category systems. For lexical diversity, categories are unique word forms and elements are token occurrences; for syntactic diversity, categories are complete dependency subtrees (POS labels plus dependency relations under the FTB-dep schema) and elements are subtree instances. The paper uses Shannon–Weaver entropy $H$ and its Rényi generalization $H_\alpha$, where low $\alpha$ emphasizes variety (the number of categories) and higher $\alpha$ emphasizes balance of category frequencies. The sampling mechanism is Algorithm 1: starting from an initial corpus, scan candidate documents from a massive raw corpus, keep the candidate that most increases entropy when added, append it to the working corpus, and lower the exhaustivity level if the target size is not reached. The evaluation machinery is correlation analysis: compute lexical and syntactic $H_\alpha$ on 705 blocks of 100K sentences and measure Spearman and Pearson correlations across $\alpha$ values and across subcorpora.

What would settle it

Take the same 705 blocks, obtain manually validated gold parses for a random subset, and recompute the lexical–syntactic correlations; if correlations stabilize at high positive values once parser errors are removed, the claim that lexical diversity cannot proxy syntactic diversity would fail. Alternatively, find a dataset and $\alpha$ where the correlation remains stably above 0.9 across resamples and genres.

Watch

Extended reading notes

Core claim

In the paper's own terms, the discovery is two-sided. For Q1, the answer is yes: the tractable heuristic, which scans candidate documents at decreasing exhaustivity levels and greedily appends the document that most increases $H_\alpha$, produces a corpus with lexical entropy 7.98, while twenty random samples of the same target size follow a normal distribution with mean $\approx 7.656$ and $\sigma \approx 9.027\times 10^{-4}$; the heuristic result is about 347 standard deviations from the mean, so the gain is statistically overwhelming. For Q2, the answer is negative: correlation between lexical and syntactic $H_\alpha$ (Pearson and Spearman) is not stable; it changes with $\alpha$ (0 to 5), with dataset, and within genre, and pooling three subcorpora produces a negative Pearson correlation down to $-0.43$ near $\alpha = 1$ even where individual subcorpora hover around zero. The authors state that the use of lexical diversity did not increase syntactic diversity, meaning a corpus sampled to maximize lexical entropy is not thereby syntactically diverse.

Load-bearing premise

The Q2 conclusion rests on parser-predicted dependency subtrees being accurate enough that fluctuating correlations reflect real language properties; Section 7 concedes that predicted syntax contains errors, especially for rare phenomena.

Editorial extensions

If this is right

  • A corpus sampled by this greedy entropy heuristic will be lexically far more diverse than a random sample of equal size, with no exhaustive power-set search; the mechanism can be applied to new raw corpora.
  • Because lexical–syntactic correlations are unstable across datasets and $\alpha$, a lexical-only sampling step cannot guarantee long-tail syntactic coverage; some parsing is unavoidable when syntactic diversity is the goal.
  • The Rényi order $\alpha$ is not a free parameter to ignore: shifting from variety-oriented to balance-oriented diversity changes conclusions about how lexical and syntactic diversity relate.
  • Genre-specific validation is required: two parliamentary corpora show little resemblance in lexical–syntactic correlation, so a proxy tuned on one genre should not be transferred without checking.
  • Pooling heterogeneous data can produce correlation artifacts (e.g., $-0.43$ at $\alpha \approx 1$), so correlation estimates should be reported per source as well as for the union.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A practical design rule the authors gesture at: use a small parsed validation set to measure the lexical–syntactic correlation for the exact target genre and $\alpha$ before deciding whether lexical sampling suffices; if the correlation is weak, parse directly.
  • The instability is consistent with lexical and syntactic diversity being driven by different statistical regimes—vocabulary growth and hapax distributions for the lexicon, structural recombination for syntax—so a single scalar entropy is unlikely to bridge them.
  • A testable extension would redo the correlation on manually corrected parses or a different dependency annotation scheme; if the instability persists, it is a property of French text rather than parser noise, which would generalize the paper's conclusion.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper formalizes lexical and syntactic diversity as Shannon and Rényi entropies computed over token forms and dependency subtrees, and proposes a greedy heuristic (Algorithm 1) for sampling a large French corpus so as to increase lexical diversity. The authors evaluate the heuristic (Q1) by comparing the entropy of the heuristic-selected corpus with random samples, and study (Q2) whether lexical diversity correlates with syntactic diversity across 705 blocks of 100K sentences from three subcorpora. They find a very large apparent increase over random for Q1 and report that lexical–syntactic correlations fluctuate with α, dataset, and aggregation, concluding that lexical diversity is not a reliable proxy for syntactic diversity.

Significance. The paper tackles a practically important question for large corpus construction: can cheap lexical diversity stand in for expensive syntactic diversity? Its formal framing via ecological diversity measures is appropriate, and the idea of varying the Rényi order α to interpolate between variety and balance is sound. The main empirical finding, if robust, would be a useful negative result: corpus builders should not assume that maximizing lexical diversity automatically yields syntactic diversity. The paper is also transparent in listing limitations. However, the two central empirical claims rest on experiments with testable flaws: the Q1 comparison uses an uncontrolled baseline, and the Q2 conclusion is based on a single arbitrary aggregation level with acknowledged parser noise but no sensitivity analysis. These issues are load-bearing, not cosmetic.

major comments (4)
  1. [§5, Q1] The 347σ claim is based on an uncontrolled baseline comparison. The heuristic output TOTAL is BASE plus HPLTdiverse, but the random baseline Nrandom is sampled from HPLTsmall alone. The entropy Hdiverse = 7.98 for TOTAL is therefore compared with Hrandom ≈ 7.656 for a corpus that does not contain BASE, so the difference conflates the contribution of BASE with the effect of the heuristic. To support the claim, the authors should compare TOTAL with BASE ∪ Nrandom (or, equivalently, compare the marginal increase H(TOTAL) − H(BASE) with H(BASE ∪ Nrandom) − H(BASE) over random seeds). Without this control, the reported significance level is not a valid test of the heuristic's effectiveness.
  2. [§5, Q1 and §4] The heuristic optimizes exactly the lexical entropy used for evaluation, so the increase in lexical diversity is partly self-fulfilling. A proper random-selection baseline would address this, but the current baseline does not, because of the issue in the previous comment. The claim that the heuristic 'significantly increases diversity relative to random sampling' would be more convincing if the authors reported the distribution of H(BASE ∪ Nrandom) and showed that Hdiverse lies far from it; the current 347σ figure does not establish this.
  3. [§5, Q2 and §7] The central negative finding—that lexical and syntactic diversity do not stably correlate—is computed over 705 blocks of 100K sentences each, and Section 7 concedes that the block size is arbitrary and that another value might change the results. Correlations over aggregate units are sensitive to the unit of aggregation (ecological correlation), and the sampling heuristic in Algorithm 1 operates on documents, not on 100K-sentence blocks. The observed instability, including the union's negative Pearson dip near α≈1, could be an artifact of mixing heterogeneous documents within blocks. The paper should report sensitivity to block size (e.g., document-level, sentence-level, and several intermediate sizes) or at least show that the qualitative pattern of correlations is stable. Without this, the conclusion that 'the use of lexical diversity did not increase syntactic diversity' is underdetermined.
  4. [§7 (Limitations) and §5, Q2] Section 7 states that the syntax used to compute syntactic diversity is parser-predicted and contains errors, and that rare phenomena may be badly predicted. Since rare syntactic phenomena are precisely the long-tail items that motivate diversity sampling, and since parser errors are plausibly correlated with lexical frequency, the Q2 correlations may reflect parser noise rather than genuine linguistic properties. The manuscript acknowledges this risk but presents no analysis of its magnitude. At minimum, the paper should quantify parser accuracy on the relevant rare constructions (e.g., by hand-checking a sample of rare dependency subtrees) or discuss why parser noise cannot explain the observed fluctuation pattern. This is a specific, testable concern that affects the validity of the central claim.
minor comments (4)
  1. [Throughout] There are several typos and inconsistent notations: 'datsets' in the Introduction, 'subtress' in §2, 'HPLT small' vs 'HPLTsmall' in §4, and 'T OT AL' with irregular spacing in §4. The authors should also spell out the acronym CLSD when first used in §5.
  2. [Algorithm 1] The exhaustivity array E is never instantiated with concrete values, and the target size S is described as 'here 3.1 billion' while earlier the goal is stated as 'around 1.5 billion' tokens similar to BigScience; this discrepancy should be clarified. The initial comparison H(W ∪ d) with d = ∅ (line 9) also needs a convention for the entropy of W ∪ ∅.
  3. [§2] The formal definitions would benefit from stating explicitly that the entropy is computed on the empirical distribution of categories (maximum-likelihood estimates) and from giving the normalization step for token forms (footnote 3) more prominence, since it affects all later lexical diversity numbers.
  4. [§5, Q2] Figure 2 plots correlations for α from 0 to 5, but the paper does not report whether the trends converge for higher α or how the choice b = e influences the numbers; a sentence clarifying the role of the log base would help readers interpret the absolute entropy values.

Circularity Check

1 steps flagged · score 3.0 of 10

Q1's diversity gain is measured with the same entropy that Algorithm 1 maximizes, making the 'significant increase' partly self-fulfilling; the Q2 lexical-vs-syntactic correlation study is independent and not circular.

  1. self definitional [§4 Algorithm 1 (lines 7-10) and §5 Q1 evaluation]
    "7: if H (W ∪ n) > H(W ) then 8: f ← f + 1 9: if H (W ∪ n) > H(W ∪ d) then 10: d ← n ... The value of Hdiverse is away from µ by ≈ 347σ. The p-value is ≈ 0.5 Thus, Ndiverse is highly significantly more diverse than at random, which confirms the effectiveness of Algorithm 1."

    Algorithm 1 accepts a document only when it increases the Shannon entropy H of the working corpus (line 7), and the Q1 evaluation measures the entropy H of the resulting sample against random samples. The reported 'highly significant' increase in diversity is therefore a restatement of the selection criterion: any greedy procedure that repeatedly adds entropy-increasing documents will, by construction, end with a corpus whose H exceeds the expected H of random selection.

full rationale

The paper's central contribution — the Q2 finding that lexical and syntactic entropies do not correlate stably across datasets, α values, and the union of corpora — is not circular: lexical Hα and syntactic Hα are computed over different category systems (token forms vs. dependency subtrees), and the paper does not predict one from the other; it reports measured correlations and honestly concludes that 'these results do not provide a definitive conclusion for Q2.' No load-bearing self-citation or imported uniqueness theorem appears; the ecological diversity background (Hill, Rényi, Shannon-Weaver) is standard external mathematics. Two acknowledged limitations — the arbitrary 100K-sentence block size and parser prediction errors in the syntactic categories — are empirical validity threats, not definitional circularity: they weaken the support for the negative Q2 claim but do not make its inputs equal to its outputs. The only genuine circular step is Q1's self-evaluation, where the entropy objective of Algorithm 1 is also the evaluation metric; the paper even calls this a 'verification' of effectiveness, though success is guaranteed by the selection rule. Because the central Q2 result remains independent and the random baseline provides partial grounding, the overall circularity is limited, giving score 3 rather than a higher value.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new theoretical entities; its free parameters are engineering choices (E, S, normalization categories, block size), and its axioms are standard domain assumptions from ecology-based diversity measurement plus dataset-quality assumptions.

free parameters (5)
  • Exhaustivity array E
    Controls how many documents are examined before one is appended; the paper never gives the specific values used, so the greedy search depth is an unreported hand choice.
  • Target corpus size S = 3.1 billion tokens (reported; earlier text says around 1.5 billion)
    Sets the stopping criterion for the sampler; the paper contains two different values, making the actual target ambiguous.
  • Normalization token categories
    Numbers, URLs, tags, etc. are collapsed into single categories like [NUMBER]; this changes the entropy values and is a design choice not varied or justified.
  • Block size for Q2 = 100K sentences
    Used to split BASE into 705 blocks for correlation estimation; the authors admit in Section 7 that the value is arbitrary.
  • Random subset HPLTsmall size = described as 'around 6 billion documents', which exceeds HPLT's 99.59M documents; likely a typo
    The initial random draw from HPLT determines the pool for both heuristic and random samples; its true size is unclear.
assumptions (5)
  • domain assumption Shannon and Rényi entropy are appropriate formal measures of lexical and syntactic diversity for NLP datasets.
    Adopted from ecology and information theory (Hill 1973, Chao et al. 2014, Morales et al. 2020); the paper does not validate that these entropies track the diversity properties that matter for downstream NLP tasks.
  • domain assumption Complete syntactic subtrees containing only POS labels and dependency relations are valid categories for syntactic diversity.
    Introduced in Section 2; no empirical justification is given for this particular category definition.
  • domain assumption The FTB-dep parser's predicted syntax is accurate enough for diversity scoring.
    Used throughout Q2; the authors acknowledge in Section 7 that parser errors, especially on rare phenomena, may impact diversity scores.
  • domain assumption HPLT's French-language assignment by majority vote over language predictors is correct for the sampled documents.
    Section 3 states the texts are divided into languages by majority vote; errors in this assignment could mix other languages into the French corpus and inflate apparent diversity.
  • domain assumption The random subset HPLTsmall is representative of HPLT for sampling purposes.
    The algorithm starts from a random draw; if this draw is biased, both the heuristic and random baselines inherit the bias.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Formalising lexical and syntactic diversity for data sampling in French." pith.science (2026). https://pith.science/paper/RUSJGXEM

@misc{pith2026250108003,
  author       = {Pith},
  title        = {Pith review of: Formalising lexical and syntactic diversity for data sampling in French},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RUSJGXEM}},
  note         = {Machine review of arXiv:2501.08003}
}
read the original abstract

Diversity is an important property of datasets and sampling data for diversity is useful in dataset creation. Finding the optimally diverse sample is expensive, we therefore present a heuristic significantly increasing diversity relative to random sampling. We also explore whether different kinds of diversity -- lexical and syntactic -- correlate, with the purpose of sampling for expensive syntactic diversity through inexpensive lexical diversity. We find that correlations fluctuate with different datasets and versions of diversity measures. This shows that an arbitrarily chosen measure may fall short of capturing diversity-related properties of datasets.

Figures

Figures reproduced from arXiv: 2501.08003 by the authors.

Figure 1
Figure 1. Two toy datasets with sample syntactic categories (on the right) and elements (inside the sentences). [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Correlation between lexical and syntactic [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 18 canonical work pages

  1. [1]

    Danial Alihosseini, Ehsan Montahaei, and Mahdieh Soleymani Baghshah. 2019. https://doi.org/10.18653/v1/W19-2311 Jointly measuring diversity and quality in text generation models . In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pages 90--98, Minneapolis, Minnesota. Association for Computational Linguistics

    doi:10.18653/v1/w19-2311 2019

  2. [2]

    Marie Candito, Beno \^ t Crabb \'e , and Pascal Denis. 2010. http://www.lrec-conf.org/proceedings/lrec2010/pdf/392_Paper.pdf Statistical F rench dependency parsing: Treebank conversion and first results . In Proceedings of the Seventh International Conference on Language Resources and Evaluation ( LREC '10) , Valletta, Malta. European Language Resources A...

    work page 2010

  3. [3]

    Anne Chao, Chun-Huo Chiu, and Lou Jost. 2014. https://www.jstor.org/stable/24810182 Unifying Species Diversity , Phylogenetic Diversity , Functional Diversity , and Related Similarity and Differentiation Measures Through Hill Numbers . Annual Review of Ecology, Evolution, and Systematics, 45:297--324. Publisher: Annual Reviews

    work page arXiv 2014

  4. [4]

    Ona De Gibert, Graeme Nail, Nikolay Arefyev, Marta Ba \ n \'o n, Jelmer Van Der Linde, Shaoxiong Ji, Jaume Zaragoza-Bernabeu, Mikko Aulamo, Gema Ram \' rez-S \'a nchez, Andrey Kutuzov, et al. 2024. https://aclanthology.org/2024.lrec-main.100.pdf A new massive multilingual dataset for high-performance language technologies . In Proceedings of the 2024 Join...

    work page 2024

  5. [5]

    Xiang Gao, Sungjin Lee, Yizhe Zhang, Chris Brockett, Michel Galley, Jianfeng Gao, and Bill Dolan. 2019. https://doi.org/10.18653/v1/N19-1125 Jointly optimizing diversity and relevance in neural response generation . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technol...

    work page doi:10.18653/v1/n19-1125 2019

  6. [6]

    Shester Gueuwou, Sophie Siake, Colin Leong, and Mathias M \"u ller. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.664 JWS ign: A highly multilingual corpus of B ible translations for more diversity in sign language processing . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9907--9927, Singapore. Association for Co...

    doi:10.18653/v1/2023.findings-emnlp.664 2023

  7. [7]

    M. O. Hill. 1973. https://doi.org/10.2307/1934352 Diversity and Evenness : A Unifying Notation and Its Consequences . Ecology, 54(2):427--432. Number: 2 Publisher: Ecological Society of America

    doi:10.2307/1934352 1973

  8. [8]

    Po-Yao Huang, Xiaojun Chang, and Alexander Hauptmann. 2019. https://doi.org/10.18653/v1/D19-1154 Multi-head attention with diversity for learning grounded multilingual multimodal representations . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process...

    work page doi:10.18653/v1/d19-1154 2019

Show all 30 references
  1. [9]

    Yadav, Maria Peifer, Sardar Hamidian, and H

    Yejin Kim, Scott Rome, Kevin Foley, Mayur Nankani, Rimon Melamed, Javier Morales, Abhay K. Yadav, Maria Peifer, Sardar Hamidian, and H. Howie Huang. 2024. https://aclanthology.org/2024.lrec-main.766 Improving content recommendation: Knowledge graph-based semantic contrastive l...

    2024

  2. [10]

    Yekyung Kim. 2020. https://aclanthology.org/2020.lifelongnlp-1.1 Deep active learning for sequence labeling based on diversity and uncertainty in gradient . In Proceedings of the 2nd Workshop on Life-long Learning for Spoken Language Systems, pages 1--8, Suzhou, China. Associa...

    2020

  3. [11]

    Kummerfeld, Parker Hill, Michael A

    Stefan Larson, Anish Mahendran, Andrew Lee, Jonathan K. Kummerfeld, Parker Hill, Michael A. Laurenzano, Johann Hauswald, Lingjia Tang, and Jason Mars. 2019. https://doi.org/10.18653/v1/N19-1051 Outlier detection for improved data quality and diversity in dialog systems . In Pr...

    2019 doi

  4. [12]

    Hugo Lauren c on, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo Gonz\' a lez Ponferrada, Huu Nguyen, J\" o rg Frohberg, Mario S a s ko, Quentin Lhoest, Angelina McMillan-Major, Gerard Dupont...

    2022

  5. [13]

    Chen Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. 2024 a . https://doi.org/10.18653/v1/2024.naacl-long.112 Are multilingual LLM s culturally-diverse reasoners? an investigation into multicultural proverbs and sayings . In Proceedings of the 2024 Conference of the Nort...

    2024 doi

  6. [14]

    Guisheng Liu, Yi Li, Zhengcong Fei, Haiyan Fu, Xiangyang Luo, and Yanqing Guo. 2024 b . https://aclanthology.org/2024.lrec-main.1134 Prefix-diffusion: A lightweight diffusion model for diverse image captioning . In Proceedings of the 2024 Joint International Conference on Comp...

    2024

  7. [15]

    Yang Janet Liu and Amir Zeldes. 2023. https://doi.org/10.18653/v1/2023.eacl-main.227 Why can ' t discourse parsing generalize? a thorough investigation of the impact of data diversity . In Proceedings of the 17th Conference of the European Chapter of the Association for Comput...

    2023 doi

  8. [16]

    Kanishka Misra and Kyle Mahowald. 2024. https://arxiv.org/abs/2403.19827 Language models learn rare phenomena from less rare phenomena: The case of the missing aanns . Preprint, arXiv:2403.19827

    2024 arXiv

  9. [17]

    Pedro Ramaciotti Morales, Robin Lamarche-Perrin, Raphael Fournier-S'niehotta, Remy Poulain, Lionel Tabourier, and Fabien Tarissan. 2020. https://doi.org/10.48550/arXiv.2001.01296 Measuring Diversity in Heterogeneous Information Networks . arXiv preprint. Issue: arXiv:2001.0129...

    2020 doi arXiv

  10. [18]

    Behrang QasemiZadeh, Miriam R. L. Petruck, Regina Stodden, Laura Kallmeyer, and Marie Candito. 2019. https://doi.org/10.18653/v1/S19-2003 S em E val-2019 task 2: Unsupervised lexical frame induction . In Proceedings of the 13th International Workshop on Semantic Evaluation, pa...

    2019 doi

  11. [19]

    Carlos Ramisch, Agata Savary, Bruno Guillaume, Jakub Waszczuk, Marie Candito, Ashwini Vaidya, Verginica Barbu Mititelu, Archna Bhatia, Uxoa I \ n urrieta, Voula Giouli, Tunga G \"u ng \"o r, Menghan Jiang, Timm Lichte, Chaya Liebeskind, Johanna Monti, Renata Ramisch, Sara Stym...

    2020

  12. [20]

    Alfréd Rényi. 1961. https://projecteuclid.org/ebooks/berkeley-symposium-on-mathematical-statistics-and-probability/Proceedings-of-the-Fourth-Berkeley-Symposium-on-Mathematical-Statistics-and/chapter/On-Measures-of-Entropy-and-Information/bsmsp/1200512181 On Measures of Entropy...

    1961

  13. [21]

    Claude Elwood Shannon and Warren Weaver. 1949. A Mathematical Theory of Communication . University of Illinois Press, Urbana

    1949

  14. [22]

    Bastow Wilson

    Benjamin Smith and J. Bastow Wilson. 1996. https://doi.org/10.2307/3545749 A Consumer 's Guide to Evenness Indices . Oikos, 76(1):70--82. Number: 1 Publisher: [Nordic Society Oikos, Wiley]

    1996 doi

  15. [23]

    Leonie Weissweiler, Abdullatif Köksal, and Hinrich Schütze. 2024. https://arxiv.org/abs/2403.06965 Hybrid human-llm corpus construction and llm evaluation for rare linguistic phenomena . Preprint, arXiv:2403.06965

    2024 arXiv

  16. [24]

    Yu Xia, Xu Liu, Tong Yu, Sungchul Kim, Ryan Rossi, Anup Rao, Tung Mai, and Shuai Li. 2024. https://doi.org/10.18653/v1/2024.naacl-long.479 Hallucination diversity-aware active learning for text summarization . In Proceedings of the 2024 Conference of the North American Chapter...

    2024 doi

  17. [25]

    Jianxiang Xiang, Zhenhua Liu, Haodong Liu, Yin Bai, Jia Cheng, and Wenliang Chen. 2024. https://aclanthology.org/2024.lrec-main.440 D iffusion D ialog: A diffusion model for diverse dialog generation with latent space . In Proceedings of the 2024 Joint International Conference...

    2024

  18. [26]

    Vikas Yadav, Hyuk joon Kwon, Vijay Srinivasan, and Hongxia Jin. 2024. https://aclanthology.org/2024.lrec-main.601 Explicit over implict: Explicit diversity conditions for effective question answer generation . In Proceedings of the 2024 Joint International Conference on Comput...

    2024

  19. [27]

    Yuting Yang, Pei Huang, Feifei Ma, Juan Cao, and Jintao Li. 2024. https://aclanthology.org/2024.lrec-main.1100 PAD : A robustness enhancement ensemble method via promoting attention diversity . In Proceedings of the 2024 Joint International Conference on Computational Linguist...

    2024

  20. [28]

    Yusen Zhang, Nan Zhang, Yixin Liu, Alexander Fabbri, Junru Liu, Ryo Kamoi, Xiaoxin Lu, Caiming Xiong, Jieyu Zhao, Dragomir Radev, Kathleen McKeown, and Rui Zhang. 2024. https://doi.org/10.18653/v1/2024.naacl-long.187 Fair abstractive summarization of diverse perspectives . In ...

    2024 doi

  21. [29]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  22. [30]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.