Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Measuring Diversity in Synthetic Datasets

T0 review · 3 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Synthetic-data diversity can be scored by the trace of a softmax matrix.

desk verdict A new and useful diversity score that tracks pseudo-truths well, but the axiom proofs are incorrect under the default temperature and the abstract overstates the gains. read the letter →

arxiv 2502.08512 v3 pith:YJ4Q74EB submitted 2025-02-12 cs.CL cs.AI

Yuchang Zhu , Huizhe Zhang , Bingzhe Wu , Jintang Li , Zibin Zheng , Peilin Zhao , Liang Chen , Yatao Bian This is my paper · ORCID
classification cs.CLcs.AI
keywords syntheticdatasetsdiversityevaluationLLM-generateddatasoftmaxclassificationeffectivenumbertraceofsimilaritymatrixVendiScorecomparisondatasetquality
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that diversity in LLM-generated datasets can be measured by a single trace: embed the samples, form their pairwise similarity matrix, softmax each row, and add the diagonal entries. This score, called DCScore, is interpreted as the effective number of distinct samples, with n identical copies scoring 1 and n fully distinguishable samples scoring n. The authors argue it satisfies four standard diversity axioms, runs in $O(n^2)$ time at the summarization stage for general kernels, and correlates with generation temperature, human diversity rankings, and GPT-4 rankings. A correct version of the claim would give synthetic-data users a reference-free, axiom-grounded way to compare generation settings, select data, and monitor mode collapse.

What carries the argument

The central object is the row-softmax of the kernel matrix $K$ at temperature $\tau$, giving the probability matrix $P$, and DCScore($D$) is the trace of $P$. The classification reading is what carries the argument: P[i,i] is the probability that sample i is classified into its own category, so the trace converts a matrix of pairwise similarities into a count of how many samples are confidently self-identifying. The same matrix supports the axioms, since the proofs track how identical rows, sample permutations, and newly added similar samples change the diagonal of the softmax.

What would settle it

Take n clearly distinct sentences, embed them, use the default inner-product kernel and temperature $\tau = 1$, and compute DCScore; if the result is below n, the effective-number axiom holds only in the limit described by the paper's appendix, not as an exact range. A second direct check is to add one sample that is more similar to dataset B than to dataset A and confirm that the merged B dataset has the lower trace, as monotonicity requires.

Watch

Extended reading notes

Core claim

DCScore treats each sample as its own category in an n-class classification problem. After mapping samples to embeddings, a kernel function produces a similarity matrix K; a softmax with temperature $\tau$ converts each row of K into a probability distribution over the n categories; the diversity score is the trace of the resulting probability matrix P, i.e., the sum of diagonal entries P[i,i]. A sample that is easy to distinguish from the others is classified into its own category with high probability and contributes close to 1, while a dataset of near-duplicates spreads probability mass and contributes little. The paper proves that this trace satisfies the four diversity axioms from the ecological diversity literature—effective number, identical samples, symmetry, and monotonicity—and reports that it tracks three diversity pseudo-truths (generation temperature, human judgment, and an LLM evaluator) with Spearman rho at or above 0.6, while remaining stable across embedding models and kernels.

Load-bearing premise

The axiom proof assumes that when all samples are distinct, every sample is classified into its own category with probability exactly 1; with the default temperature and ordinary embedding similarities, distinct samples still share positive similarity, so the score remains below n and the stated [1,n] effective-number range is a limit rather than an exact equality.

Editorial extensions

If this is right

  • Synthetic-data users can compare generation temperatures, prompts, and models on a single diversity number without collecting a human reference set.
  • The quadratic softmax summarization makes diversity checks feasible for datasets of tens of thousands of samples under general kernels, where spectral methods cost $O(n^3)$.
  • Because DCScore tracks downstream classifier accuracy in the paper's experiments, it can serve as an early indicator of whether additional generation effort is adding training value.
  • The duplicate-sample stability result means augmented datasets with repeated entries are not penalized disproportionately, which matters for LLM augmentation pipelines.
  • The four axioms give a common checklist for future diversity metrics, allowing a direct comparison of DCScore with any alternative.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the effective-number claim is exact only in the limit where distinct samples have zero mutual similarity; at finite temperature with inner-product embeddings the trace is literally below n, so the [1,n] range is an idealization rather than a sharp guarantee.
  • Editorial extension: because the score inherits the choice of embedding and kernel, it measures diversity relative to an encoder; the same text corpus can be scored differently under different representations, which is useful for aspect-sensitive evaluation but means there is no single intrinsic diversity value.
  • Editorial extension: DCScore could be inverted into a generation objective—sampling to maximize or match a target trace—and the resulting datasets tested on downstream tasks against temperature-based baselines.
  • Editorial extension: the trace of a softmax kernel is structurally close to spectral diversity measures such as VendiScore; checking whether DCScore emerges as a limiting case of those spectral scores could unify the two families and explain when their rankings disagree.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes DCScore, a reference-free diversity metric for synthetic datasets. DCScore embeds the diversity-sensitive components of a dataset, computes a pairwise kernel matrix K, applies a row-wise softmax to obtain a probability matrix P, and returns the trace of P. The authors claim that DCScore satisfies four diversity axioms from Leinster and Cobbold (2012) — effective number, identical samples, symmetry, and monotonicity — and that it correlates more strongly than baselines with generation temperature, human judgment, and GPT-4 judgments, while being cheaper than eigenvalue-based methods such as VendiScore. The paper includes correlation experiments, computational cost experiments, and additional probes on downstream training and image data.

Significance. If the paper's claims were fully supported, DCScore would be a simple, reference-free, and computationally efficient diversity measure with a principled axiomatic foundation, which would be useful for evaluating LLM-generated datasets. The empirical study is broad: it covers generation-temperature correlation, human evaluation, GPT-4 evaluation, multiple embedding and kernel functions, duplicate-sample stability, and downstream task accuracy, and the code is released. The O(n^2) summarization cost relative to VendiScore's O(n^3) is a genuine practical advantage for non-linear kernels. However, the central theoretical claim — that DCScore satisfies the Leinster-Cobbold effective-number axiom under the default configuration used in the experiments — is false, and the empirical 'stronger correlation' claim is not supported by the reported point estimates, which are largely ties with VendiScore and carry no confidence intervals.

major comments (3)
  1. [Section 4.2, Eq. (4), Appendix B] The effective-number axiom is not satisfied under the paper's default hyperparameters. With the default inner-product kernel and tau=1, take n distinct normalized one-hot vectors: K[i,i]=1 and K[i,j]=0 for i != j. Then Eq. (4) gives P[i,i] = e/(e+n-1), so DCScore = n*e/(e+n-1), which is about 2.67 for n=100 and tends to e (about 2.72) as n grows, not to n. Appendix B asserts 'P[i,i] = 1' for all distinct samples, which does not follow from Eq. (4); the hedge 'DCScore tending to n' is a different statement and is still unsupported for tau=1. Consequently, the stated range [1,n], the monotonicity premise DCScore(D1)=DCScore(D2)=n, and the interpretation in Table 8 of DCScore values as effective sample counts are not established. This is load-bearing because the paper's main theoretical contribution is the claim that DCScore satisfies the Leinster-Cobbold axioms. The authors should either prove the axioms in a clearly stated regime (e.g., tau -> 0 or a kernel construction that makes self-similarity dominate) and use that regime in all experiments, or weaken the theoretical claims to an approximation and revise the abstract, Section 4.2, and the wording around Table 8 accordingly.
  2. [Abstract, Section 5.2.1, Table 3] The abstract's claim that DCScore enjoys a 'stronger correlation' with diversity pseudo-truths is not supported by Table 3. Across the eight settings, DCScore ties VendiScore in three columns, is worse in three, and is better in only two. The text statement that 'DCScore outperforms all baseline methods under the few-shot setting across all datasets' is also inaccurate: in the few-shot 13B text-classification and story-completion columns DCScore is tied with VendiScore (0.9909 vs. 0.9909 and 0.9857 vs. 0.9857). No confidence intervals or significance tests are reported for these Spearman values, so 'stronger' is not established. Please report bootstrap intervals or paired significance tests for the Table 3 comparisons, and revise the abstract and Section 5.2.1 to describe the result accurately, e.g., 'on par with VendiScore in most settings while offering lower computational cost.'
  3. [Appendix B, Monotonicity proof] The monotonicity proof is not a derivation. It states that if the new sample is more similar to D2 than to D1, then 'the classification function f(.) exhibits lower confidence,' so P_D2'[i,i] < P_D1'[i,i], and concludes Eq. (17). This is an intuitive claim, not a proof: the softmax self-probability is not monotone in every off-diagonal similarity in an obvious way, and no argument is given for why the trace inequality follows for every row. The proof also assumes the premise DCScore(D1)=DCScore(D2)=n, which, as noted above, is false under the default tau=1 and inner-product kernel. Please provide a complete proof under stated conditions or remove the monotonicity axiom claim.
minor comments (6)
  1. [Throughout] There are many missing spaces around command names, e.g., 'DCScoresatisfies', 'ofDCScore', 'DCScoreand', and 'A.P.' abbreviations in Table 8; these should be fixed in the final manuscript.
  2. [Section 4.1, Eq. (4)] The definition of the classification function f_K is indirect: f_K(K[i,j]) is written as a function of a single scalar, but the denominator sums over all j. Please make explicit that the softmax is over the full row of K, e.g., by writing P[i,:] = softmax(K[i,:]/tau).
  3. [Section 5.2.1, Paragraph after Table 3] The sentence 'DCScore performs on par with VendiScore' is accurate, but the following sentence 'DCScore outperforms all baseline methods under the few-shot setting' is not; please align the text with the actual ties and losses in Table 3.
  4. [Appendix C.1.1] The bullet 'Datasets on Appendix 5.5.1' should refer to Section 5.5.1, not Appendix 5.5.1.
  5. [Section 5.2.3, Table 5] The GPT-4 correlation results have large standard deviations in the few-shot settings (e.g., 0.6057±0.30 and 0.6131±0.18). The paper calls these 'strong correlations' but does not report the number of pairwise comparisons or any test against chance-level agreement; please add this information.
  6. [Section 5.4, Figure 5] The hyperparameter sensitivity analysis shows that Spearman's rho depends strongly on tau, with low tau yielding negative correlations. Since tau=1 is used in all other experiments, please justify this choice explicitly and discuss the risk that the reported correlations are partly an artifact of tau selection.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DCScore is defined independently, the axioms are proved from Eq. (4), and the correlation experiments use external pseudo-truths; the effective-number proof gap is a correctness issue, not a circular reduction.

full rationale

DCScore is defined in Definition 4.1 as tr(P), where P is the row-softmax of a kernel matrix (Eq. 4). The four Leinster-Cobbold axioms are proved in Appendix B from this definition, not imported from the authors' prior work, and no parameter is fitted to the diversity pseudo-truths before correlation. The correlation experiments against generation temperature, human judgment, and GPT-4 are external benchmarks, so there is no fitted-input-called-prediction pattern. The one same-author citation (Leng et al. 2025, Appendix A.2) is background material about applications of diversity metrics and does not support the definition, the axioms, or the complexity claims. The main defect in the paper is the effective-number proof: it asserts P[i,i]=1 for 'all distinct' samples and then hedges to 'DCScore tending to n,' but under the default inner-product kernel with tau=1, orthogonal distinct samples give DCScore = n*e/(e+n-1), which tends to e, not n. This is a quantitative and axiomatic correctness gap, not circularity: DCScore is not defined in terms of the effective-number conclusion, and the axioms are not used to construct the score. Accordingly, no circular step is identified.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the softmax trace formula plus three domain assumptions: pairwise similarity captures diversity, duplicate rows behave as copies, and self-similarity dominates for distinct samples. The temperature tau and the embedding/kernel configuration are hand-set and affect results; no new physical entities are introduced.

free parameters (2)
  • Softmax temperature tau = 1 (default in all experiments)
    Hand-set; Figure 5 shows correlations vary strongly with tau, with higher tau giving higher Spearman rho, so the default is a modeling choice, not a derived value.
  • Embedding function and kernel = SimCSE (unsup-simcse-bert-base-uncased) with inner-product kernel
    Chosen by hand; scores change with different embeddings (Table 13) and kernels (Table 14), so the configuration is part of the method's definition.
assumptions (4)
  • domain assumption Self-similarity dominates cross-similarity for 'distinct' samples.
    Appendix B effective-number proof sets P[i,i]=1 for distinct samples; with inner-product embeddings this requires near-orthogonality and is not guaranteed.
  • domain assumption Duplicate samples produce exact row duplicates in the kernel matrix.
    Appendix B Eq. (12) for the identical-samples axiom assumes merging two identical datasets simply repeats each row; this holds for translation-invariant kernels but is stated as general.
  • domain assumption Diversity of a synthetic dataset is fully determined by pairwise similarities of diversity-sensitive components.
    Section 3.2 defines the evaluation problem this way; it excludes aspects like label distribution or global coverage that pairwise similarities may not capture.
  • ad hoc to paper Row-softmax is the correct classification function and tau=1 is a meaningful scale.
    Section 4.1 introduces softmax as 'a natural option'; no external justification, and sensitivity analysis shows performance depends on tau.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Measuring Diversity in Synthetic Datasets." pith.science (2026). https://pith.science/paper/YJ4Q74EB

@misc{pith2026250208512,
  author       = {Pith},
  title        = {Pith review of: Measuring Diversity in Synthetic Datasets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YJ4Q74EB}},
  note         = {Machine review of arXiv:2502.08512}
}
read the original abstract

Large language models (LLMs) are widely adopted to generate synthetic datasets for various natural language processing (NLP) tasks, such as text classification and summarization. However, accurately measuring the diversity of these synthetic datasets-an aspect crucial for robust model performance-remains a significant challenge. In this paper, we introduce DCScore, a novel method for measuring synthetic dataset diversity from a classification perspective. Specifically, DCScore formulates diversity evaluation as a sample classification task, leveraging mutual relationships among samples. We further provide theoretical verification of the diversity-related axioms satisfied by DCScore, highlighting its role as a principled diversity evaluation method. Experimental results on synthetic datasets reveal that DCScore enjoys a stronger correlation with multiple diversity pseudo-truths of evaluated datasets, underscoring its effectiveness. Moreover, both empirical and theoretical evidence demonstrate that DCScore substantially reduces computational costs compared to existing methods. Code is available at: https://github.com/bluewhalelab/dcscore.

Figures

Figures reproduced from arXiv: 2502.08512 by the authors.

Figure 1
Figure 1. The overview of DCScore. DCScore consists of text representation, pairwise similarity, and diversity summarization stages. • Effective number: Diversity should be defined as the effective number of samples in a dataset, ranging from 1 to n. DCScore meets this axiom, as evidenced by its behavior: DCScore equals 1 when all samples in D are identical and equals n when all samples are distinct. • Identical samples: Give… view at source ↗
Figure 2
Figure 2. Experimental settings of correlation evaluation. computational complexity across multiple kernels is more advantageous than achieving it with a single kernel. 5. Experiments We conduct experiments to verify the effectiveness of DCScore by examining correlation, computational cost, hyperparameter sensitivity, and further probing. Limited by space, we provide additional experiments in Appendix D. 5.1. Experimental Set… view at source ↗
Figure 3
Figure 3. Diversity evaluation of DCScore on datasets generated using varying τg. DCScore shows a strong correlation with τg, indicating its effectiveness in evaluating the dataset diversity. Based on the above-mentioned settings, a completely di￾verse dataset may yield a diversity score of 10. As shown in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Computation times under different sample sizes. DCScore outperforms all baselines in computational cost [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 6
Figure 6. Figure 6: Diversity evaluation on colored MNIST using two differ￾ent embedding functions (Inception V3 and Dino V2). A higher label number signifies greater dataset diversity. performance in the zero-shot setting, further demonstrat￾ing the effectiveness of DCScore. Detailed exp…
Figure 7
Figure 7. Figure 7: Prompt settings for GPT-4 evaluations. classifier on self-generated datasets. Specifically, we fix the LoRA scaling factor to 32 and the rank of the update matrices to 8. We use AdamW (Loshchilov, 2017) with an initial learning rate of 5e-5 and linear learning rate dec…
Figure 8
Figure 8. Figure 8: Loss curves of the downstream task training. experimental settings described in Appendix D.1 and leverage different random seeds for data sampling [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9 [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Exploring More to Solve More: Boosting Diversity in Text Diffusion Models via Entropy-Based Guidance

    cs.CL 2026-07 conditional novelty 5.0 of 10

    A kernel-entropy guidance signal, linearized into logit space, shifts the fidelity-diversity frontier of text diffusion models and lifts LLaDA-8B pass@32 on HumanEval and MBPP by 8-15 absolute points.

Reference graph

Works this paper leans on

94 extracted references · 30 canonical work pages · cited by 1 Pith paper

  1. [1]

    Synthetic dialogue dataset generation using llm agents

    Abdullin, Y., Molla-Aliod, D., Ofoghi, B., Yearwood, J., and Li, Q. Synthetic dialogue dataset generation using llm agents. arXiv preprint arXiv:2401.17461, 2024

    arXiv 2024

  2. [2]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    User's guide to correlation coefficients

    Akoglu, H. User's guide to correlation coefficients. Turkish journal of emergency medicine, 18 0 (3): 0 91--93, 2018

    2018

  4. [4]

    J., Kragic, D., and Kjellstrom, H

    Butepage, J., Black, M. J., Kragic, D., and Kjellstrom, H. Deep representation learning for human motion prediction and classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 6158--6166, 2017

    2017

  5. [5]

    Language gans falling short

    Caccia, M., Caccia, L., Fedus, W., Larochelle, H., Pineau, J., and Charlin, L. Language gans falling short. arXiv preprint arXiv:1811.02549, 2018

    arXiv 2018

  6. [6]

    Instruction mining: Instruction data selection for tuning large language models

    Cao, Y., Kang, Y., Wang, C., and Sun, L. Instruction mining: Instruction data selection for tuning large language models. arXiv preprint arXiv:2307.06290, 2023

    arXiv 2023

  7. [7]

    Mixture of soft prompts for controllable data generation

    Chen, D., Lee, C., Lu, Y., Rosati, D., and Yu, Z. Mixture of soft prompts for controllable data generation. arXiv preprint arXiv:2303.01580, 2023

    arXiv 2023

  8. [8]

    Chung, J. J. Y., Kamar, E., and Amershi, S. Increasing diversity while maintaining accuracy: Text data generation with large language models and human interventions. arXiv preprint arXiv:2306.04140, 2023

    arXiv 2023

Show all 94 references
  1. [9]

    Eval all, trust a few, do wrong to none: Comparing sentence generation models

    C \' fka, O., Severyn, A., Alfonseca, E., and Filippova, K. Eval all, trust a few, do wrong to none: Comparing sentence generation models. arXiv preprint arXiv:1804.07972, 2018

    2018 arXiv

  2. [10]

    D., Zoph, B., Shlens, J., and Le, Q

    Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V. Randaugment: Practical data augmentation with no separate search. arXiv preprint arXiv:1909.13719, 2 0 (4): 0 7, 2019

    1909 arXiv

  3. [11]

    Auggpt: Leveraging chatgpt for text data augmentation

    Dai, H., Liu, Z., Liao, W., Huang, X., Cao, Y., Wu, Z., Zhao, L., Xu, S., Liu, W., Liu, N., et al. Auggpt: Leveraging chatgpt for text data augmentation. arXiv preprint arXiv:2302.13007, 2023

    2023 arXiv

  4. [12]

    and Dieng, A

    Dan Friedman, D. and Dieng, A. B. The vendi score: A diversity evaluation metric for machine learning. Transactions on machine learning research, 2023

    2023

  5. [13]

    The mnist database of handwritten digit images for machine learning research [best of the web]

    Deng, L. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 29 0 (6): 0 141--142, 2012

    2012

  6. [14]

    B., Ruiz, F

    Dieng, A. B., Ruiz, F. J., Blei, D. M., and Titsias, M. K. Prescribed generative adversarial networks. arXiv preprint arXiv:1910.04302, 2019

    1910 arXiv

  7. [15]

    K., Joty, S., Li, B., and Bing, L

    Ding, B., Qin, C., Liu, L., Chia, Y. K., Joty, S., Li, B., and Bing, L. Is gpt-3 a good data annotator? arXiv preprint arXiv:2212.10450, 2022

    2022 arXiv

  8. [16]

    T., and Joty, S

    Ding, B., Qin, C., Zhao, R., Luo, T., Li, X., Chen, G., Xia, W., Hu, J., Luu, A. T., and Joty, S. Data augmentation using llms: Data perspectives, learning paradigms and challenges. arXiv preprint arXiv:2403.02990, 2024

    2024 arXiv

  9. [17]

    G., Santos, G

    dos Santos, V. G., Santos, G. L., Lynn, T., and Benatallah, B. Identifying citizen-related issues from social media using llm-based data augmentation. In International Conference on Advanced Information Systems Engineering, pp.\ 531--546. Springer, 2024

    2024

  10. [18]

    and Black, A

    Du, W. and Black, A. W. Boosting dialog response generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    2019

  11. [19]

    Evuru, C. K. R., Ghosh, S., Kumar, S., Tyagi, U., Manocha, D., et al. Coda: Constrained generation based data augmentation for low-resource nlp. arXiv preprint arXiv:2404.00415, 2024

    2024 arXiv

  12. [20]

    Simcse: Simple contrastive learning of sentence embeddings

    Gao, T., Yao, X., and Chen, D. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821, 2021

    2021 arXiv

  13. [21]

    Chatgpt outperforms crowd workers for text-annotation tasks

    Gilardi, F., Alizadeh, M., and Kubli, M. Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120 0 (30): 0 e2305016120, 2023

    2023

  14. [22]

    J., Cubuk, E

    Gontijo-Lopes, R., Smullin, S. J., Cubuk, E. D., and Dyer, E. Affinity and diversity: Quantifying mechanisms of data augmentation. arXiv preprint arXiv:2002.08973, 2020

    2002 arXiv

  15. [23]

    Generative adversarial networks

    Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial networks. Communications of the ACM, 63 0 (11): 0 139--144, 2020

    2020

  16. [24]

    Llm-based code generation method for golang compiler testing

    Gu, Q. Llm-based code generation method for golang compiler testing. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp.\ 2201--2203, 2023

    2023

  17. [25]

    A., Mishra, S., and Baral, C

    Gupta, H., Scaria, K., Anantheswaran, U., Verma, S., Parmar, M., Sawant, S. A., Mishra, S., and Baral, C. Targen: Targeted data generation with large language models. arXiv preprint arXiv:2310.17876, 2023

    2023 arXiv

  18. [26]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017

    2017

  19. [27]

    The curious case of neural text degeneration

    Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751, 2019

    1904 arXiv

  20. [28]

    J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  21. [29]

    Learning preference model for llms via automatic preference data generation

    Huang, S., Zhao, J., Li, Y., and Wang, L. Learning preference model for llms via automatic preference data generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 9187--9199, 2023

    2023

  22. [30]

    T., Boutros, F., Kuijper, A., and Damer, N

    Huber, M., Luu, A. T., Boutros, F., Kuijper, A., and Damer, N. Bias and diversity in synthetic-based face recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp.\ 6215--6226, 2024

    2024

  23. [31]

    T., and Farnia, F

    Jalali, M., Li, C. T., and Farnia, F. An information-theoretic evaluation of generative models in learning multi-modal distributions. Advances in Neural Information Processing Systems, 36: 0 9931--9943, 2023

    2023

  24. [32]

    Jordan, M. I. and Mitchell, T. M. Machine learning: Trends, perspectives, and prospects. Science, 349 0 (6245): 0 255--260, 2015

    2015

  25. [33]

    Natural language processing: state of the art, current trends and challenges

    Khurana, D., Koli, A., Khatter, K., and Singh, S. Natural language processing: state of the art, current trends and challenges. Multimedia tools and applications, 82 0 (3): 0 3713--3744, 2023

    2023

  26. [34]

    Improved precision and recall metric for assessing generative models

    Kynk \"a \"a nniemi, T., Karras, T., Laine, S., Lehtinen, J., and Aila, T. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32, 2019

    2019

  27. [35]

    Diversity, density, and homogeneity: Quantitative characteristic metrics for text collections

    Lai, Y.-A., Zhu, X., Zhang, Y., and Diab, M. Diversity, density, and homogeneity: Quantitative characteristic metrics for text collections. arXiv preprint arXiv:2003.08529, 2020

    2003 arXiv

  28. [36]

    Exploring precision and recall to assess the quality and diversity of llms

    Le Bronnec, F., V \'e rine, A., Negrevergne, B., Chevaleyre, Y., and Allauzen, A. Exploring precision and recall to assess the quality and diversity of llms. In 62nd Annual Meeting of the Association for Computational Linguistics, 2024

    2024

  29. [37]

    Beyond scale: the diversity coefficient as a data quality metric demonstrates llms are pre-trained on formally diverse data

    Lee, A., Miranda, B., Sundar, S., and Koyejo, S. Beyond scale: the diversity coefficient as a data quality metric demonstrates llms are pre-trained on formally diverse data. arXiv preprint arXiv:2306.13840, 2023

    2023 arXiv

  30. [38]

    Graddiv: Adversarial robustness of randomized neural networks via gradient diversity regularization

    Lee, S., Kim, H., and Lee, J. Graddiv: Adversarial robustness of randomized neural networks via gradient diversity regularization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45 0 (2): 0 2645--2651, 2022

    2022

  31. [39]

    and Cobbold, C

    Leinster, T. and Cobbold, C. A. Measuring diversity: the importance of species similarity. Ecology, 93 0 (3): 0 477--489, 2012

    2012

  32. [40]

    Rich feature learning via diversification

    Leng, X., Chen, Y., Tang, X., and Bian, Y. Rich feature learning via diversification. In Workshop on Spurious Correlation and Shortcut Learning: Foundations and Solutions, 2025. URL https://openreview.net/forum?id=dv23kah60r

    2025

  33. [41]

    A diversity-promoting objective function for neural conversation models

    Li, J., Galley, M., Brockett, C., Gao, J., and Dolan, B. A diversity-promoting objective function for neural conversation models. arXiv preprint arXiv:1510.03055, 2015

    2015 arXiv

  34. [42]

    Empowering large language models for textual data augmentation

    Li, Y., Ding, K., Wang, J., and Lee, K. Empowering large language models for textual data augmentation. arXiv preprint arXiv:2404.17642, 2024 a

    2024 arXiv

  35. [43]

    Synthetic data generation with large language models for text classification: Potential and limitations

    Li, Z., Zhu, H., Lu, Z., and Yin, M. Synthetic data generation with large language models for text classification: Potential and limitations. arXiv preprint arXiv:2310.07849, 2023

    2023 arXiv

  36. [44]

    Data augmentation for text-based person retrieval using large language models

    Li, Z., Si, L., Guo, C., Yang, Y., and Cao, Q. Data augmentation for text-based person retrieval using large language models. arXiv preprint arXiv:2405.11971, 2024 b

    2024 arXiv

  37. [45]

    Holistic evaluation of language models

    Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022

    2022 arXiv

  38. [46]

    Diverse image generation via self-conditioned gans

    Liu, S., Wang, T., Bau, D., Zhu, J.-Y., and Torralba, A. Diverse image generation via self-conditioned gans. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 14286--14295, 2020

    2020

  39. [47]

    Roberta: A robustly optimized bert pretraining approach

    Liu, Y. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  40. [48]

    On llms-driven synthetic data generation, curation, and evaluation: A survey

    Long, L., Wang, R., Xiao, R., Zhao, J., Ding, X., Chen, G., and Wang, H. On llms-driven synthetic data generation, curation, and evaluation: A survey. arXiv preprint arXiv:2406.15126, 2024

    2024 arXiv

  41. [49]

    Decoupled weight decay regularization

    Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  42. [50]

    L., Daly, R

    Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., and Potts, C. Learning word vectors for sentiment analysis. In Lin, D., Matsumoto, Y., and Mihalcea, R. (eds.), Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language ...

    2011

  43. [51]

    Zero-shot stance detection using contextual data generation with llms

    Mahmoudi, G., Behkamkia, B., and Eetemadi, S. Zero-shot stance detection using contextual data generation with llms. arXiv preprint arXiv:2405.11637, 2024

    2024 arXiv

  44. [52]

    and Gimelshein, N

    Milakov, M. and Gimelshein, N. Online normalizer calculation for softmax. arXiv preprint arXiv:1805.02867, 2018

    2018 arXiv

  45. [53]

    Dqi: Measuring data quality in nlp

    Mishra, S., Arunkumar, A., Sachdeva, B., Bryan, C., and Baral, C. Dqi: Measuring data quality in nlp. arXiv preprint arXiv:2005.00816, 2020

    2005 arXiv

  46. [54]

    A corpus and evaluation framework for deeper understanding of commonsense stories

    Mostafazadeh, N., Chambers, N., He, X., Parikh, D., Batra, D., Vanderwende, L., Kohli, P., and Allen, J. A corpus and evaluation framework for deeper understanding of commonsense stories. arXiv preprint arXiv:1604.01696, 2016

    2016 arXiv

  47. [55]

    F., Oh, S

    Naeem, M. F., Oh, S. J., Uh, Y., Choi, Y., and Yoo, J. Reliable fidelity and diversity metrics for generative models. In International conference on machine learning, pp.\ 7176--7185. PMLR, 2020

    2020

  48. [56]

    Towards a scalable reference-free evaluation of generative models

    Ospanov, A., Zhang, J., Jalali, M., Cao, X., Bogdanov, A., and Farnia, F. Towards a scalable reference-free evaluation of generative models. arXiv preprint arXiv:2407.02961, 2024

    2024 arXiv

  49. [57]

    and He, H

    Padmakumar, V. and He, H. Does writing with language models reduce content diversity? arXiv preprint arXiv:2309.05196, 2023

    2023 arXiv

  50. [58]

    S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E

    Park, D. S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E. D., and Le, Q. V. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:1904.08779, 2019

    1904 arXiv

  51. [59]

    Mauve: Measuring the gap between neural text and human text using divergence frontiers

    Pillutla, K., Swayamdipta, S., Zellers, R., Thickstun, J., Welleck, S., Choi, Y., and Harchaoui, Z. Mauve: Measuring the gap between neural text and human text using divergence frontiers. Advances in Neural Information Processing Systems, 34: 0 4816--4828, 2021

    2021

  52. [60]

    unsup-simcse-bert-base-uncased, 2021

    princeton nlp. unsup-simcse-bert-base-uncased, 2021. URL https://huggingface.co/princeton-nlp/unsup-simcse-bert-base-uncased

    2021

  53. [61]

    Quine, W. V. Ontological relativity and other essays, 1969

    1969

  54. [62]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Reimers, N. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019

    1908 arXiv

  55. [63]

    T., Wu, T., Guestrin, C., and Singh, S

    Ribeiro, M. T., Wu, T., Guestrin, C., and Singh, S. Beyond accuracy: Behavioral testing of nlp models with checklist. arXiv preprint arXiv:2005.04118, 2020

    2005 arXiv

  56. [64]

    and Laradji, I

    Sahu, G. and Laradji, I. H. Mixsumm: Topic-based data augmentation using llms for low-resource extractive text summarization. arXiv preprint arXiv:2407.07341, 2024

    2024 arXiv

  57. [65]

    G., Ramanan, K

    Samuel, V., Aynaou, H., Chowdhury, A. G., Ramanan, K. V., and Chadha, A. Can llms augment low-resource reading comprehension datasets? opportunities and challenges. arXiv preprint arXiv:2309.12426, 2023

    2023 arXiv

  58. [66]

    Gaussian processes for machine learning

    Seeger, M. Gaussian processes for machine learning. International journal of neural systems, 14 0 (02): 0 69--106, 2004

    2004

  59. [67]

    Svd-softmax: Fast softmax approximation on large vocabulary neural networks

    Shim, K., Lee, M., Choi, I., Boo, Y., and Sung, W. Svd-softmax: Fast softmax approximation on large vocabulary neural networks. Advances in neural information processing systems, 30, 2017

    2017

  60. [68]

    Generating diverse translations with sentence codes

    Shu, R., Nakayama, H., and Cho, K. Generating diverse translations with sentence codes. In Proceedings of the 57th annual meeting of the association for computational linguistics, pp.\ 1823--1827, 2019

    2019

  61. [69]

    D., Ng, A

    Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A. Y., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp.\ 1631--1642, 2013

    2013

  62. [70]

    Scaling data diversity for fine-tuning language models in human alignment

    Song, F., Yu, B., Lang, H., Yu, H., Huang, F., Wang, H., and Li, Y. Scaling data diversity for fine-tuning language models in human alignment. arXiv preprint arXiv:2403.11124, 2024

    2024 arXiv

  63. [71]

    The proof and measurement of association between two things

    Spearman, C. The proof and measurement of association between two things. 1961

    1961

  64. [72]

    and Hearst, M

    Stasaski, K. and Hearst, M. A. Semantic diversity in dialogue with natural language inference. arXiv preprint arXiv:2205.01497, 2022

    2022 arXiv

  65. [73]

    Large language models for data annotation: A survey

    Tan, Z., Beigi, A., Wang, S., Guo, R., Bhattacharjee, A., Jiang, B., Karami, M., Li, J., Cheng, L., and Liu, H. Large language models for data annotation: A survey. arXiv preprint arXiv:2402.13446, 2024

    2024 arXiv

  66. [74]

    Alpacaeval : An automatic evaluator for instruction-following language models, 2023

    tatsu lab. Alpacaeval : An automatic evaluator for instruction-following language models, 2023. URL https://github.com/tatsu-lab/alpaca_eval?tab=readme-ov-file#evaluators

    2023

  67. [75]

    and Berant, J

    Tevet, G. and Berant, J. Evaluating the evaluation of diversity in natural language generation. arXiv preprint arXiv:2004.02990, 2020

    2004 arXiv

  68. [76]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  69. [77]

    A., Khashabi, D., and Hajishirzi, H

    Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560, 2022

    2022 arXiv

  70. [78]

    J., and Sch \"o lkopf, B

    Wen, Y., Liu, W., Feng, Y., Raj, B., Singh, R., Weller, A., Black, M. J., and Sch \"o lkopf, B. Pairwise similarity learning is simple. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 5308--5318, 2023

    2023

  71. [79]

    White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., and Schmidt, D. C. A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382, 2023

    2023 arXiv

  72. [80]

    Investigating the effectiveness of data augmentation from similarity and diversity: An empirical study

    Yang, S., Guo, S., Zhao, J., and Shen, F. Investigating the effectiveness of data augmentation from similarity and diversity: An empirical study. Pattern Recognition, 148: 0 110204, 2024 a

    2024

  73. [81]

    Mini-da: Improving your model performance through minimal data augmentation using llm

    Yang, S., Liu, X., Dong, X., and Fu, B. Mini-da: Improving your model performance through minimal data augmentation using llm. In Proceedings of the Fifth Workshop on Data Science with Human-in-the-Loop (DaSH 2024), pp.\ 25--30, 2024 b

    2024

  74. [82]

    Zerogen: Efficient zero-shot learning via dataset generation

    Ye, J., Gao, J., Li, Q., Xu, H., Feng, J., Wu, Z., Yu, T., and Kong, L. Zerogen: Efficient zero-shot learning via dataset generation. arXiv preprint arXiv:2202.07922, 2022

    2022 arXiv

  75. [83]

    Llm-da: Data augmentation via large language models for few-shot named entity recognition

    Ye, J., Xu, N., Wang, Y., Zhou, J., Zhang, Q., Gui, T., and Huang, X. Llm-da: Data augmentation via large language models for few-shot named entity recognition. arXiv preprint arXiv:2402.14568, 2024

    2024 arXiv

  76. [84]

    M., Park, D., Kang, J., Lee, S.-W., and Park, W

    Yoo, K. M., Park, D., Kang, J., Lee, S.-W., and Park, W. Gpt3mix: Leveraging large-scale language models for text augmentation. arXiv preprint arXiv:2104.08826, 2021

    2021 arXiv

  77. [85]

    Seqgan: Sequence generative adversarial nets with policy gradient

    Yu, L., Zhang, W., Wang, J., and Yu, Y. Seqgan: Sequence generative adversarial nets with policy gradient. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017

    2017

  78. [86]

    J., Krishna, R., Shen, J., and Zhang, C

    Yu, Y., Zhuang, Y., Zhang, J., Meng, Y., Ratner, A. J., Krishna, R., Shen, J., and Zhang, C. Large language model as attributed training data generator: A tale of diversity and bias. Advances in Neural Information Processing Systems, 36, 2024

    2024

  79. [87]

    Large language models for healthcare data augmentation: An example on patient-trial matching

    Yuan, J., Tang, R., Jiang, X., and Hu, X. Large language models for healthcare data augmentation: An example on patient-trial matching. In AMIA Annual Symposium Proceedings, volume 2023, pp.\ 1324. American Medical Informatics Association, 2023

    2023

  80. [88]

    Advancing llm reasoning generalists with preference trees

    Yuan, L., Cui, G., Wang, H., Ding, N., Wang, X., Deng, J., Shan, B., Chen, H., Xie, R., Lin, Y., et al. Advancing llm reasoning generalists with preference trees. arXiv preprint arXiv:2404.02078, 2024

    2024 arXiv

  81. [89]

    mixup: Beyond empirical risk minimization

    Zhang, H. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017

    2017 arXiv

  82. [90]

    Improving diversity of commonsense generation by large language models via in-context learning

    Zhang, T., Peng, B., and Bollegala, D. Improving diversity of commonsense generation by large language models via in-context learning. arXiv preprint arXiv:2404.16807, 2024

    2024 arXiv

  83. [91]

    Character-level convolutional networks for text classification

    Zhang, X., Zhao, J., and LeCun, Y. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015

    2015

  84. [92]

    Bootstrapped unsupervised sentence representation learning

    Zhang, Y., He, R., Liu, Z., Bing, L., and Li, H. Bootstrapped unsupervised sentence representation learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (V...

    2021

  85. [93]

    Texygen: A benchmarking platform for text generation models

    Zhu, Y., Lu, S., Zheng, L., Guo, J., Zhang, W., Wang, J., and Yu, Y. Texygen: A benchmarking platform for text generation models. In The 41st international ACM SIGIR conference on research & development in information retrieval, pp.\ 1097--1100, 2018

    2018

  86. [94]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.