Pith. sign in

REVIEW 4 major objections 5 minor 39 references

Evaluation of LLMs in Speech is Often Flawed: Test Set Contamination in Large Language Models for Speech Recognition

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper establishes that large fractions of the LibriSpeech and Common Voice evaluation sets appear in public LLM pretraining corpora, and that even tiny amounts of this contamination bias the probabilities LLMs assign to test…

desk verdict Controlled contamination experiments are solid and the extent estimates are useful but approximate; worth serious peer review. read the letter →

arxiv 2505.22251 v2 pith:MMTRES5X submitted 2025-05-28 eess.AS cs.CL

Yuan Tseng , Titouan Parcollet , Rogier van Dalen , Shucong Zhang , Sourav Bhattacharya This is my paper · ORCID
classification eess.AScs.CL
keywords testsetcontaminationLLM-basedspeechrecognitionLibriCommonVoiceperplexitybiasdataleakageASRevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the standard benchmarks used to evaluate LLM-based speech systems are not clean: a large fraction of LibriSpeech and Common Voice evaluation transcripts already appear in public pretraining corpora. To show this matters, the authors train billion-parameter language models from scratch that differ only in whether a handful of LibriSpeech books were mixed into pretraining. The contaminated models are more likely to generate the leaked transcripts and less likely to generate non-leaked ones, and the same bias persists when the models are wired into speech recognisers: perplexities shift significantly while error rates change only subtly. A sympathetic reader should care because it means reported gains on these benchmarks may partly reflect memorisation rather than capability, and because likelihood-based metrics can reveal contamination that word-error-rate numbers hide.

What carries the argument

The load-bearing object is the distinction between leaked and non-leaked evaluation sentences, measured by word-level perplexity with bootstrapped 95% confidence intervals. Two mechanisms carry the argument: (1) a MinHash LSH book-level duplicate search that flags LibriSpeech and Common Voice transcripts whose source documents appear in the Pile; (2) controlled from-scratch pretraining of LLMs that differ only in the presence of a tiny number of LibriSpeech books, which isolates the memorisation effect from the fact that leaked and non-leaked sentences differ inherently in difficulty (an n-gram control shows leaked sentences are naturally more probable). The perplexity gap between contaminated and uncontaminated models is the evidence that contamination, not distributional difference, causes the bias.

What would settle it

Search a random sample of the 6,873 LibriSpeech utterances counted as leaked for their exact sentence text inside the corresponding Pile documents (rather than matching books); if most of the sample cannot be found verbatim, the two-thirds contamination figure falls and the claimed scale of exposure weakens, whereas finding them confirms the paper's estimate.

Watch

Extended reading notes

Core claim

The central claim is that test set contamination during LLM pretraining biases the output probabilities of LLM-based speech systems, even though the amount of contaminated text is minuscule and standard error metrics may not catch it. Concretely: the Pile contains near-duplicates of the Project Gutenberg books behind 6,873 of 11,126 LibriSpeech dev/test utterances and Wikipedia articles containing 10,388 of 32,796 Common Voice English dev/test utterances verbatim. When the authors pretrain Pythia-style LLMs from scratch with 82 LibriSpeech books mixed into 15B to 60B tokens of otherwise identical data, leaked sentences receive consistently lower perplexity (higher probability) than in uncontaminated twins, non-leaked sentences receive no comparable benefit, larger models show a stronger effect, and repeating the books in training strengthens it. In ASR systems built on these LLMs, contamination significantly lowers perplexity on leaked transcripts while changing WER/CER only slightly and inconsistently.

Load-bearing premise

The contamination counts rely on a book-level proxy: an utterance is called leaked if its source Project Gutenberg book or Wikipedia article is found in the Pile, without verifying that the exact sentence appears in the Pile document, and the Pile is treated as representative of public LLM pretraining corpora as a whole.

Editorial extensions

If this is right

  • Findings from LLM-based speech systems evaluated on LibriSpeech and Common Voice may partly reflect memorisation of evaluation transcripts, so gains quoted on these benchmarks should be re-verified on genuinely held-out data.
  • Contamination both inflates the probability of generating test-set sentences and slightly reduces the probability of generating unseen sentences, meaning it can simultaneously flatter leaked-data results and underestimate true capability on new data.
  • The bias is not diluted by training on more tokens as long as the leaked text stays in the corpus, and it grows with model size and with repeated exposure to the same source books.
  • Error-rate metrics such as WER and CER can miss the effect entirely; the contamination signal shows up in the probabilities, so likelihood-based evaluation (perplexity) over contaminated benchmarks is the unreliable part.
  • Speech-conditioned text generation inherits the same bias as unconditional generation, so the issue extends to LLM-based ASR, speech translation, and dialog systems that condition on acoustic input.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the same source materials (Project Gutenberg, Wikipedia) underpin many other ASR datasets, a similar contamination audit would likely flag parts of other audiobook- and wiki-sourced benchmarks whenever they share pretraining corpora.
  • A low-cost test for the paper's mechanism: paraphrase or re-tokenise evaluation transcripts so that no memorised surface form is present; the perplexity gap between 'contaminated' and 'clean' evaluations should shrink if memorisation is the driver.
  • The paper's book-level proxy suggests the true utterance-level contamination of LibriSpeech could be lower than two-thirds; conversely, the authors note their Common Voice count is conservative, so the actual exposure across both datasets may differ from the headline numbers in either direction.
  • Beyond speech, the same failure mode argues that any likelihood- or probability-based report on a benchmark that shares source documents with pretraining data should include a contamination scan as standard practice.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that evaluations of LLM-based speech systems on LibriSpeech and Common Voice are often invalid because large fractions of those evaluation sets already appear in public LLM pretraining corpora. The authors report that roughly two-thirds of LibriSpeech dev/test utterances and one-third of Common Voice English dev/test utterances are in the Pile. To establish the causal mechanism, they train Pythia-style LLMs from scratch with and without injected LibriSpeech books, showing that contaminated models assign higher perplexity-based likelihood to leaked sentences and lower likelihood to non-leaked sentences, with effects that persist across model size, training tokens, and contamination multiplicity. They then build LLM-based ASR systems with these models and show that perplexity is biased toward leaked transcriptions even though error rates change little. The conclusion is that likelihood-based evaluation of LLM speech systems is vulnerable to contamination and that held-out data are necessary.

Significance. If the extent estimates are correct, this is an important negative result for the speech-LLM community: two of the most heavily used ASR benchmarks are partly memorized by pretrained LLMs, so score gains on them may be inflated. The controlled experiments are a real strength: training LLMs from scratch with matched step counts, bootstrap confidence intervals, an n-gram baseline that separates distributional from contamination effects, and a LibriSpeech-training-set control (Figure 3) provide a credible demonstration that tiny amounts of exact-sentence contamination shift model probability. The paper is also honest about one open direction, namely that Section III-A cannot distinguish contamination from inherent sentence difficulty. The main weaknesses are in the extent-estimation leg: the LibriSpeech estimate is book-level rather than utterance-level, the Common Voice estimate matches article titles rather than exact sentences, and only the Pile is measured despite the broad title and abstract claims. These issues do not undermine the controlled experimental result, but they do mean the benchmark-level threat is quantified as an upper bound, not as a precise contamination rate.

major comments (4)
  1. [Section II-A] The LibriSpeech contamination estimate counts an utterance as leaked whenever its source Project Gutenberg book matches a Pile document at Jaccard similarity > 0.7. This book-level proxy does not verify that the specific utterance string appears in the Pile document; preprocessing, sentence segmentation, truncation, or OCR variants can drop individual utterances from a duplicated book. The 6,873/11,126 figure is therefore an upper bound on utterance-level leakage. The authors should either verify exact utterance strings in matched Pile documents for a sample of utterances and report precision/recall for the 0.7 threshold, or explicitly rephrase 'effectively contaminated with nearly two-thirds' as 'as many as two-thirds.' Without this fix, the headline extent claim is stronger than the evidence.
  2. [Section II-B] The Common Voice estimate matches Wikipedia article titles to Pile documents, not the exact sentences inside those documents. Finding a duplicate article title does not guarantee that the specific transcript appears in the Pile text, so the 10,388/32,796 number can overstate contamination from false-positive title matches. The paper discusses only the possibility that the true number is larger (due to wording differences), but the estimate could also be smaller; the authors should report direct sentence-level matching or a conservative analysis that acknowledges both directions of error.
  3. [Abstract and Title] The title and abstract generalize from the Pile to 'public LLM pretraining corpora' and to 'LLMs' broadly, but the extent analysis is performed only on the Pile. Other corpora that contain Project Gutenberg or Wikipedia (ROOTS, RedPajama, Dolma, TxT360) are listed but not measured. The paper should either extend the contamination measurement to at least one or two additional corpora or qualify the claims to 'the Pile and models trained on it.' Otherwise the benchmark-level conclusion is only as broad as the corpora actually studied.
  4. [Section IV-C and Table I] Table I reports averages over five random seeds and states that contaminated results are bolded when significantly better, but no confidence intervals, p-values, or test descriptions are provided for any entry. The differences are extremely small (for example, 1.610 vs. 1.603 for 6.9B punctuated ASR dev-clean leaked perplexity), so without error bars or a significance test the claim that ASR systems 'assign significantly higher probabilities' to leaked sentences is not fully supported. Please report the confidence intervals or the full statistical test for Table I, in the same style as Figure 2.
minor comments (5)
  1. [Section III, Probability calculation] The NLL formula normalizes by nwords, the number of words, while p(y_t) refers to tokens; please clarify whether token-level probabilities are averaged over words or whether the notation should be token count.
  2. [Section II-A, manual verification] The statement that 'we manually fix all sentences that do not match due to text preprocessing errors' should include the number of affected sentences or a brief description of the manual procedure for reproducibility.
  3. [Section II-B] The note that actual Common Voice contamination 'may be larger' is asymmetric; the title-matching procedure can also produce false positives, so the discussion should mention that the estimate could be either larger or smaller than the true utterance-level rate.
  4. [Figure 1] The caption and text do not specify the n-gram smoothing method or the exact training data split for the LibriSpeech language modeling baseline; please add these details.
  5. [Section III-B] The text says '82 books' when describing the contamination injection, but the exact number of books that are not originally in the Pile should be checked for consistency with Figure 3 and Section II-A; a small discrepancy would confuse readers.

Circularity Check

1 steps flagged · score 2.0 of 10

No significant circularity: the controlled contamination experiments are genuine measurements with appropriate controls, though the definition of 'leaked' sentences partly builds the measured effect into the experimental setup.

  1. self definitional [Section III-B, 'Comparing LLMs trained with/without contamination']
    "Hence, leaked sentences refer to LibriSpeech sentences not originally in the Pile from now on. ... we simulate a realistic ratio of leaked data to total pretraining data, by randomly mixing the 82 books containing LibriSpeech sentences not originally in the Pile into the pretraining data of contaminated models."

    In the controlled experiments, the category 'leaked' is defined as exactly those sentences that are deliberately inserted into the contaminated model's training data. Reporting that contaminated LLMs assign higher average probability to leaked than to non-leaked sentences is therefore, in part, a restatement of the training exposure rather than an independent prediction. This is a mild by-construction element.

full rationale

The paper's central claim—that tiny amounts of test-set contamination bias LLM likelihoods—is supported by two independent legs. The controlled experiments in Section III-B are self-contained: the authors train Pythia models from scratch with and without injected LibriSpeech books, compare perplexities under identical step counts and hyperparameters, and include a control condition (Figure 3) that distinguishes memorization of specific test sentences from mere exposure to similar-distribution books. These experiments are measurements, not derivations, and they hold independent content beyond the definitions used. The only mild by-construction element is that 'leaked' in Section III-B is defined as the injected set, so the direction of the effect is partly tautological; however, the paper's contribution is the controlled quantification, which is legitimate. The extent estimates in Section II rely on book-level MinHash matching and Wikipedia-article-title matching as proxies for utterance-level presence, which is a measurement-validity concern rather than circularity, and the authors explicitly acknowledge the Common Voice estimate may undercount. Self-citations (e.g., following the ASR architecture of [18], whose author list may overlap with the present authors) are methodological borrowings, not load-bearing evidence for the contamination claim. The open-source Pythia analysis in Section III-A honestly reports that contamination effects are confounded by distributional differences and draws no strong conclusion from it. Overall, no claim reduces by construction to its inputs, no fitted parameter is renamed as a prediction, and no uniqueness or ansatz is imported from self-citation. The appropriate finding is a low circularity score of 2, reflecting only the mild definitional element in the controlled leaked/non-leaked comparison.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central bias claim is empirically measured rather than derived from a fitted law. The only fitted number feeding the headline contamination statistics is the MinHash Jaccard threshold. The main domain assumptions are the Pile-as-proxy for all pretraining corpora and the book-level-to-utterance-level inference. No invented entities are introduced.

free parameters (1)
  • MinHash Jaccard duplicate threshold = 0.7
    Chosen by manual verification of candidate MinHash pairs from the same LibriSpeech/Pile data being classified; determines the reported contamination fraction but is not fitted to the probability-bias result.
assumptions (4)
  • domain assumption The Pile is representative of public LLM pretraining corpora
    The paper generalizes its conclusions from Pile-based contamination (Section II) to 'public LLM pretraining corpora' in the abstract and title, but only checks the Pile directly.
  • domain assumption Book-level duplication implies utterance-level leakage
    Section II-A counts all utterances from a book as leaked when the book is found in the Pile via MinHash, without per-utterance verification of the exact text.
  • domain assumption Bootstrapping treats sentences as independent
    The confidence intervals in Figure 2 resample sentences independently, ignoring book-level clustering that could broaden the intervals.
  • domain assumption Verbatim Wikipedia matching is a lower bound for Common Voice
    Acknowledged in Section II-B; reworded Common Voice sentences that do not match Wikipedia verbatim are not counted as contaminated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluation of LLMs in Speech is Often Flawed: Test Set Contamination in Large Language Models for Speech Recognition." pith.science (2026). https://pith.science/paper/MMTRES5X

@misc{pith2026250522251,
  author       = {Pith},
  title        = {Pith review of: Evaluation of LLMs in Speech is Often Flawed: Test Set Contamination in Large Language Models for Speech Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MMTRES5X}},
  note         = {Machine review of arXiv:2505.22251}
}
read the original abstract

Recent work suggests that large language models (LLMs) can improve performance of speech tasks compared to existing systems. To support their claims, results on LibriSpeech and Common Voice are often quoted. However, this work finds that a substantial amount of the LibriSpeech and Common Voice evaluation sets appear in public LLM pretraining corpora. This calls into question the reliability of findings drawn from these two datasets. To measure contamination impact, LLMs trained with/without contamination are compared. A contaminated LLM is more likely to generate test sentences it has seen during training. Then, speech recognisers based on LLMs are compared. They show only subtle error rate differences if the LLM is contaminated, but assign significantly higher probabilities to transcriptions seen during LLM training. Results show that LLM outputs can be biased by tiny amounts of data contamination, highlighting the importance of evaluating LLM-based speech systems with held-out data.

Figures

Figures reproduced from arXiv: 2505.22251 by the authors.

Figure 1
Figure 1. Perplexities of leaked and non-leaked sentences calculated [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Comparison of LLMs contaminated with data from the [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 2
Figure 2. Comparing perplexity differences between contaminated and [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: A LLM-based speech system for ASR, with speech embed [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 24 canonical work pages

  1. [1]

    Prompting large language models with speech recognition abilities,

    Y . Fathullah, C. Wu, E. Lakomkin, J. Jia, Y . Shangguan, K. Li, J. Guo, W. Xiong, J. Mahadeokar, O. Kalinli, C. Fuegen, and M. Seltzer, “Prompting large language models with speech recognition abilities,” in Proceedings of International Confer- ence on Acoustics, Speech, and Signal Processing , 2024

    work page 2024

  2. [2]

    Salsa: Speedy asr-llm synchronous aggregation,

    A. Mittal, D. Prabhu, S. Sarawagi, and P. Jyothi, “Salsa: Speedy asr-llm synchronous aggregation,” in Proceedings of Interspeech, 2024, pp. 3485–3489

    work page 2024

  3. [3]

    Delayed fusion: Integrating large language models into first- pass decoding in end-to-end speech recognition,

    T. Hori, M. Kocour, A. Haider, E. McDermott, and X. Zhuang, “Delayed fusion: Integrating large language models into first- pass decoding in end-to-end speech recognition,” in Proceed- ings of International Conference on Acoustics, Speech, and Signal Processing, 2025

    work page 2025

  4. [4]

    Let’s fuse step by step: A generative fusion decoding algorithm with llms for multi-modal text recog- nition,

    C.-J. Hsu, Y .-C. Chen, F.-T. Liao, P.-C. Ho, Y .-H. Wang, P.- C. Hsu, and D.-s. Shiu, “Let’s fuse step by step: A generative fusion decoding algorithm with llms for multi-modal text recog- nition,” arXiv preprint arXiv:2405.14259 , 2024

    arXiv 2024

  5. [5]

    COSMIC: Data efficient instruction-tuning for speech in-context learning,

    J. Pan, J. Wu, Y . Gaur, S. Sivasankaran, Z. Chen, S. Liu, and J. Li, “COSMIC: Data efficient instruction-tuning for speech in-context learning,” arXiv preprint arXiv:2311.02248 , 2023

    arXiv 2023

  6. [6]

    On decoder-only architec- ture for speech-to-text and large language model integration,

    J. Wu, Y . Gaur, Z. Chen, L. Zhou, Y . Zhu, T. Wang, J. Li, S. Liu, B. Ren, L. Liu, and Y . Wu, “On decoder-only architec- ture for speech-to-text and large language model integration,” in Proceedings of IEEE Automatic Speech Recognition and Understanding Workshop, 2023

    work page 2023

  7. [7]

    Can generative large language models perform asr error correction?

    R. Ma, M. Qian, P. Manakul, M. Gales, and K. Knill, “Can generative large language models perform asr error correction?” arXiv preprint arXiv:2307.04172 , 2023

    arXiv 2023

  8. [8]

    Contextual spelling correction with large language models,

    G. Song, Z. Wu, G. Pundak, A. Chandorkar, K. Joshi, X. Velez, D. Caseiro, B. Haynor, W. Wang, N. Siddhartha, P. Rondon, and K. C. Sim, “Contextual spelling correction with large language models,” in Proceedings of IEEE Automatic Speech Recognition and Understanding Workshop , 2023

    work page 2023

Show all 39 references
  1. [9]

    Denoising LM: Pushing the limits of error correction models for speech recognition,

    Z. Gu, T. Likhomanenko, H. Bai, E. McDermott, R. Col- lobert, and N. Jaitly, “Denoising LM: Pushing the limits of error correction models for speech recognition,” arXiv preprint arXiv:2405.15216, 2024

    2024

  2. [10]

    NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark,

    O. Sainz, J. A. Campos, I. Garc ´ıa-Ferrero, J. Etxaniz, O. L. de Lacalle, and E. Agirre, “NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark,” in Findings of Empirical Methods in Natural Language Process- ing, 2023

    2023

  3. [11]

    Data contamination: From mem- orization to exploitation,

    I. Magar and R. Schwartz, “Data contamination: From mem- orization to exploitation,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics , 2022

    2022

  4. [12]

    Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLMs,

    S. Balloccu, P. Schmidtov ´a, M. Lango, and O. Dusek, “Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLMs,” in Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, 2024

    2024

  5. [13]

    Lib- rispeech: An ASR corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: An ASR corpus based on public domain audio books,” in Proceedings of International Conference on Acous- tics, Speech, and Signal Processing , 2015

    2015

  6. [14]

    Common voice: A massively-multilingual speech corpus,

    R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Hen- retty, R. Morais, L. Saunders, F. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,” in Proceedings of Language Resources and Evaluation Conference , 2020

    2020

  7. [15]

    The Pile: An 800GB dataset of diverse text for language modeling,

    L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy, “The Pile: An 800GB dataset of diverse text for language modeling,” arXiv preprint arXiv:2101.00027 , 2020

    2020 arXiv

  8. [16]

    Comparing discrete and continuous space llms for speech recognition,

    Y . Xu, S.-X. Zhang, J. Yu, Z. Wu, and D. Yu, “Comparing discrete and continuous space llms for speech recognition,” in Proceedings of Interspeech , 2024

    2024

  9. [17]

    Connecting speech encoder and large language model for ASR,

    W. Yu, C. Tang, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang, “Connecting speech encoder and large language model for ASR,” in Proceedings of International Conference on Acoustics, Speech, and Signal Processing , 2024

    2024

  10. [18]

    An embarrassingly simple approach for LLM with strong ASR capacity,

    Z. Ma, G. Yang, Y . Yang, Z. Gao, J. Wang, Z. Du, F. Yu, Q. Chen, S. Zheng, S. Zhang, and X. Chen, “An embarrassingly simple approach for LLM with strong ASR capacity,” arXiv preprint arXiv:2402.08846, 2024

    2024 arXiv

  11. [19]

    WavLLM: Towards robust and adaptive speech large language model,

    S. Hu, L. Zhou, S. Liu, S. Chen, L. Meng, H. Hao, J. Pan, X. Liu, J. Li, S. Sivasankaran, L. Liu, and F. Wei, “WavLLM: Towards robust and adaptive speech large language model,” arXiv preprint arXiv:2404.00656 , 2024

    2024 arXiv

  12. [20]

    Efficient streaming llm for speech recognition,

    J. Jia, G. Keren, W. Zhou, E. Lakomkin, X. Zhang, C. Wu, F. Seide, J. Mahadeokar, and O. Kalinli, “Efficient streaming llm for speech recognition,” arXiv preprint arXiv:2410.03752 , 2024

    2024 arXiv

  13. [21]

    Ctc-assisted llm-based contextual asr,

    G. Yang, Z. Ma, Z. Gao, S. Zhang, and X. Chen, “Ctc-assisted llm-based contextual asr,” in Proceedings of Spoken Language Technology Workshop, 2024, pp. 126–131

    2024

  14. [22]

    The bigscience ROOTS corpus: A 1.6TB composite multilingual dataset,

    H. Laurenc ¸on, L. Saulnier, T. Wang, C. Akiki, A. V . del Moral, T. L. Scao, L. V . Werra, C. Mou, E. G. Ponferrada, H. Nguyen, J. Frohberg, M. ˇSaˇsko, Q. Lhoest, A. McMillan- Major, G. Dupont, S. Biderman, A. Rogers, L. B. allal, F. D. Toni, G. Pistilli, O. Nguyen, S. Nikpo...

    2022

  15. [23]

    RedPajama: an open dataset for training large language models,

    T. Computer, “RedPajama: an open dataset for training large language models,” 2023. [Online]. Available: https: //github.com/togethercomputer/RedPajama-Data

    2023

  16. [24]

    Dolma: an open corpus of three trillion tokens for language model pretraining research,

    L. Soldaini, R. Kinney, A. Bhagia, D. Schwenk, D. Atkin- son, R. Authur, B. Bogin, K. Chandu, J. Dumas, Y . Elazar, V . Hofmann, A. Jha, S. Kumar, L. Lucy, X. Lyu, N. Lambert, I. Magnusson, J. Morrison, N. Muennighoff, A. Naik, C. Nam, M. Peters, A. Ravichander, K. Richardson,...

    2024

  17. [25]

    LLM360 K2: Building a 65B 360-open-source large language model from scratch,

    Z. Liu, B. Tan, H. Wang, W. Neiswanger, T. Tao, H. Li, F. Koto, Y . Wang, S. Sun, O. Pangarkar, R. Fan, Y . Gu, V . Miller, L. Ma, L. Tang, N. Ranjan, Y . Zhuang, G. He, R. Wang, M. Deng, R. Algayres, Y . Li, Z. Shen, P. Nakov, and E. Xing, “LLM360 K2: Building a 65B 360-open-...

    2025 arXiv

  18. [26]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V . Goswami, N. Goyal, A. Hartshorn, S. Ho...

    2023 arXiv

  19. [27]

    The Llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru,...

    2024 arXiv

  20. [28]

    LLaMA: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “LLaMA: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  21. [29]

    Leskovec, A

    J. Leskovec, A. Rajaraman, and J. D. Ullman, Mining of Massive Datasets, 2nd ed. Cambridge University Press, 2014

    2014

  22. [30]

    Benchmarking non- parametric statistical tests,

    M. Keller, S. Bengio, and S. Wong, “Benchmarking non- parametric statistical tests,” 2005

    2005

  23. [31]

    Confidence intervals for evaluation in machine learning

    L. Ferrer and P. Riera, “Confidence intervals for evaluation in machine learning.” [Online]. Available: https://github.com/ luferrer/ConfidenceIntervals

  24. [32]

    Pythia: A suite for analyzing large language models across training and scaling,

    S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. Van Der Wal, “Pythia: A suite for analyzing large language models across training and scaling,” in Proceedings of I...

    2023

  25. [33]

    GPT-NeoX-20B: An open-source autoregressive language model,

    S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. McDonell, J. Phang, M. Pieler, U. S. Prashanth, S. Purohit, L. Reynolds, J. Tow, B. Wang, and S. Weinbach, “GPT-NeoX-20B: An open-source autoregressive language model,” in Proceedings of Bi...

    2022

  26. [34]

    OPT: Open pre-trained trans- former language models,

    S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V . Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer, “OPT: Open pre-trained trans- former language models,” arXiv preprin...

    2022 arXiv

  27. [35]

    OLMo: Accelerating the science of language models,

    D. Groeneveld, I. Beltagy, E. Walsh, A. Bhagia, R. Kin- ney, O. Tafjord, A. Jha, H. Ivison, I. Magnusson, Y . Wang, S. Arora, D. Atkinson, R. Authur, K. Chandu, A. Cohan, J. Dumas, Y . Elazar, Y . Gu, J. Hessel, T. Khot, W. Merrill, J. Morrison, N. Muennighoff, A. Naik, C. Nam...

    2024

  28. [36]

    The secret sharer: Evaluating and testing unintended memorization in neural networks,

    N. Carlini, C. Liu, ´U. Erlingsson, J. Kos, and D. Song, “The secret sharer: Evaluating and testing unintended memorization in neural networks,” in Proceedings of the 28th USENIX Con- ference on Security Symposium , 2019, pp. 267–284

    2019

  29. [37]

    Open- source conversational AI with SpeechBrain 1.0,

    M. Ravanelli, T. Parcollet, A. Moumen, S. de Langen, C. Sub- akan, P. Plantinga, Y . Wang, P. Mousavi, L. D. Libera, A. Ploujnikov, F. Paissan, D. Borra, S. Zaiem, Z. Zhao, S. Zhang, G. Karakasidis, S.-L. Yeh, P. Champion, A. Rouhe, R. Braun, F. Mai, J. Zuluaga-Gomez, S. M. Mo...

    2024

  30. [38]

    WavLM: Large-scale self-supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y . Qian, Y . Qian, J. Wu, M. Zeng, X. Yu, and F. Wei, “WavLM: Large-scale self-supervised pre-training for full stack speech processing,” IEEE Journal of Selecte...

    2022

  31. [39]

    SpecAugment: A simple data augmen- tation method for automatic speech recognition,

    D. S. Park, W. Chan, Y . Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V . Le, “SpecAugment: A simple data augmen- tation method for automatic speech recognition,” in Proceedings of Interspeech, 2019, pp. 2613–2617

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.