REVIEW 4 major objections 6 minor 23 references
Loquacious Set: 25,000 Hours of Transcribed and Diverse English Speech Recognition Data for Research and Commercial Use
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A 25,000-hour English speech corpus built from six open datasets gives researchers and industry a common, commercially usable benchmark.
desk verdict Useful curated 25k-hour commercial ASR dataset with sensible splits and recipes, but the headline benchmark claim rests on unaddressed train/eval overlap between Libriheavy/LibriSpeech and YODAS en003. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the curated corpus itself: six permissively licensed source collections are combined into nested training splits (small, medium, large, clean) and two evaluation sets, with all audio resampled to 16 kHz, segmented to 1–40 seconds (3–40 seconds for YODAS), and filtered by language and duration. Transcription normalization is the main curation step: numerals are converted to words, sentences containing non-English characters or more than four symbols are removed, punctuation and YouTube-style descriptors are stripped, and the remaining text is upper-cased and restricted to the 26 Latin letters plus the apostrophe. This machinery makes the dataset reproducible and gives every sample a standard audio-label pair, which is what allows a single training recipe to be applied across heterogeneous sources.
What would settle it
Compute the overlap at the segment, speaker, and book level between the training splits and the evaluation splits: check whether any Libriheavy training segment shares an audiobook or speaker with LibriSpeech test-other, and whether any YODAS training segment matches the YODAS-based dev/test segments. If overlap exists, rerun the reported word error rate table on a strictly disjoint split; a large increase in word error rate would show that the published numbers partly reflect in-domain test material.
Extended reading notes
Core claim
The central claim is that the Loquacious Set is the first large-scale English speech recognition corpus to combine a permissive commercial license, acoustic and linguistic diversity, normalized transcripts, defined training subsets, and dedicated evaluation sets. It pools audio from six existing datasets—Common Voice, VoxPopuli, Libriheavy, People's Speech, YODAS, and LibriSpeech—and applies a five-step text-normalization pipeline, duration filtering, and language identification, yielding 25,000 hours of curated speech in about 9.4 million segments with an average duration of 9.6 seconds. The paper reports that conformer models trained on this corpus achieve a 4.6% word error rate on the LibriSpeech test-other set with a 480M-parameter model, beating a widely used 1.5B-parameter open speech recognition system that scores 5.1%; the same model reaches 6.9% on the VoxPopuli test set and 12.0% on the Common Voice test set. The smaller subsets are designed so that 250-hour and 2,500-hour training runs can fit on four GPUs, making the benchmark accessible to groups without large compute budgets.
Load-bearing premise
The reported benchmark results assume that the audio in the evaluation sets is not drawn from the same books, speakers, or segments as the training data: the Libriheavy training data and the LibriSpeech test-other set both derive from the same public-domain audiobook collection, and the YODAS-based dev/test segments come from a source subset that is also used in training, and the paper does not state that overlapping content was removed.
Editorial extensions
If this is right
- Groups with four GPUs can train competitive models on the 250-hour or 2,500-hour subsets, while the full 25,000-hour set is designed for larger laboratories.
- Researchers in industry can legally use the corpus for commercial speech recognition development, closing the licensing gap that blocks industrial use of several existing large datasets.
- Models trained on the Loquacious Set beat a widely used 1.5B-parameter open speech recognition system on LibriSpeech test-other (4.6% versus 5.1% word error rate) and on VoxPopuli test (6.9% versus 7.5%) with fewer parameters, according to the paper's measurements.
- Adding related but not identical data helps cross-domain performance: a 100M model trained on the corpus gets a 15.9% word error rate on the Common Voice test set versus 19.5% when trained on Common Voice alone.
- The clean 13,000-hour subset matches the full 25,000-hour model on LibriSpeech test-other (6.0% word error rate with a 250M model), suggesting a cheaper compute path for clean-speech research.
Reading between the lines
- A natural next step is to publish the full per-utterance language-identification decisions and the exact filtering statistics for each source; that would let other builders reproduce the curation and quantify how much of YODAS and People's Speech survives normalization.
- The same curation recipe—normalization, duration bounds, language filtering, and nested subsets—could be ported to other permissively licensed multilingual corpora, though the language-identification step would need per-language validation.
- Versioning will determine whether the Loquacious Set becomes a lasting benchmark: since Common Voice releases change over time, freezing a snapshot and releasing hashes of audio segments would preserve comparability across future work.
- A stricter evaluation protocol would exclude any audiobook or YODAS source that appears in training; such a split would test generalization to unseen speakers and books more cleanly.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces the Loquacious Set, a curated English ASR corpus of roughly 25,000 hours built from six existing datasets (VoxPopuli, CommonVoice, People's Speech, YODAS, Libriheavy, and LibriSpeech), with text normalization, audio filtering, and defined small/medium/large/clean training splits plus dev/test sets. It ships SpeechBrain recipes and reports conformer encoder-decoder WERs on LibriSpeech test-other, VoxPopuli, CommonVoice, and its own dev/test, claiming state-of-the-art results that beat Whisper models at comparable or smaller parameter counts.
Significance. If the held-out evaluation is confirmed and the normalization quality is demonstrated, the dataset would be a valuable common resource for academic and industrial ASR: it has a permissive licensing profile, defined subsets of different sizes, and covers diverse acoustic conditions. The release of code and recipes is a concrete reproducibility strength. The main claims, however, depend on the evaluation sets being genuinely out-of-domain, which is not currently established.
major comments (4)
- [3.1, Table 1] The paper does not address training/evaluation overlap. The large split includes 11,000 hours of Libriheavy, which is derived from LibriVox audiobooks, while the evaluation sets include LibriSpeech test-other (also LibriVox-derived); moreover, the YODAS dev/test segments are extracted from en003, which is among the YODAS subsets used to form the 6,100-hour training portion. The manuscript never states whether book-, speaker-, or segment-level decontamination was performed. Please state the overlap-removal procedure and report statistics on remaining overlap, or explicitly justify why overlap is absent. Without this, the reported WERs in Table 2 on LibriSpeech test-other and the Loquacious dev/test are not established as held-out, which weakens the comparison with Whisper and the claim of state-of-the-art performance.
- [3.1] The selection rule for the large split is given only as 'up to a certain threshold to reach 25,000 hours of speech.' The per-source caps and any quality thresholds should be stated explicitly (e.g., a table of the exact hours taken from each source and the random seed). While the code may contain these details, the paper should provide them for reproducibility, especially because Table 1 reports 25,150 hours rather than 25,000.
- [3.2] The normalization process is described but its resulting transcription quality is not quantified. The paper claims 'normalised transcriptions with few errors' but provides no measurement of residual errors after normalization (e.g., character error rate against a held-out reference subset, or the number of manually verified samples for YODAS and People's Speech). Since unreliable transcriptions are a stated limitation of the source corpora, please report a concrete accuracy figure for the normalized text, including for the manually verified YODAS dev/test segments.
- [4.2] The Whisper comparison contains a factual error: the text states 'our 100M models obtains 5.7%' on LibriSpeech test-other, but Table 2 shows that the 250M model achieves 5.7%, while the 100M model achieves 6.0%. Please correct the parameter count in the comparison (and adjust the following sentence about '3 times more parameters' accordingly). This correction is needed to support the claim that the Loquacious recipes beat Whisper at smaller model sizes.
minor comments (6)
- [4.1] The phrase 'height of them are used' should be 'eight of them are used'.
- [3.1] The sentence 'most of the samples below this threshold correspond are just noises' contains a grammatical error; it should read 'correspond to noise' or 'are just noise.'
- [Throughout] The dataset name appears as 'CommonV oice' in several places due to a LaTeX artifact; please correct to 'CommonVoice.'
- [Table 1, 3.1] Table 1 gives the large split as 25,150 hours and the clean split as 13,150 hours, while the text in Section 3.1 says 25,000 and 13,000 hours; please harmonize these numbers.
- [3.1] The text mentions 'YoutuBe videos' in the Acoustic conditions paragraph; this should be 'YouTube videos.'
- [4.2] The literature numbers for Whisper (e.g., 6.7% on LibriSpeech test-other for Small.en) are cited without a reference; please add a source for these numbers.
Circularity Check
No circular derivation: the paper's claims are empirical dataset construction and reproducible ASR benchmarks; the evaluation-overlap concern is a data-hygiene issue, not a circularity step.
full rationale
The Loquacious Set paper contains no derivation chain in which an output quantity is defined in terms of the quantity it claims to predict. The central outputs are (i) a curated 25,000-hour corpus assembled by subsampling six existing datasets and (ii) measured WERs from SpeechBrain conformer recipes. These are empirical measurements, not predictions derived from a fitted parameter. The SpeechBrain citation [14] is a toolkit reference whose authors overlap with the present paper, but the experimental hyperparameters and table of results are fully specified in the paper, so the citation is not load-bearing for the WER claims. The language-identification filtering of YODAS uses a published SpeechBrain model as an external tool; this is a data-cleaning step, not an argument whose conclusion is assumed. The strongest integrity concern is that Loquacious dev/test are drawn from YODAS en003 while the large training split includes YODAS audio, and that Libriheavy training and LibriSpeech test-other share the LibriVox domain; without an explicit decontamination statement the reported WERs could be optimistic. However, this is potential train/evaluation leakage, not circularity: the WER numbers are not equivalent by construction to the training data, and the paper does not claim a derivation that reduces to its inputs. No fitted input is renamed as a prediction, no uniqueness theorem is imported, and no ansatz is smuggled in via citation. Under the stated criteria, the appropriate finding is no significant circularity (score 0).
Assumptions & free parameters
free parameters (3)
- Large split size caps =
Libriheavy 11,000h, YODAS 6,100h, People's Speech 5,900h, CommonVoice 1,600h, VoxPopuli 550h
- YODAS minimum segment duration =
3 seconds
- Symbol removal threshold =
more than 4 symbols triggers removal
assumptions (4)
- domain assumption Source datasets' licenses permit commercial combination
- domain assumption SpeechBrain VoxLingua107 language ID correctly identifies English in YODAS
- domain assumption Libriheavy training data do not overlap LibriSpeech test-other
- domain assumption YODAS dev and test segments are disjoint from the large training split
Cite this review
Pith. "Pith review of Loquacious Set: 25,000 Hours of Transcribed and Diverse English Speech Recognition Data for Research and Commercial Use." pith.science (2026). https://pith.science/paper/JD75TWRR
@misc{pith2026250521578,
author = {Pith},
title = {Pith review of: Loquacious Set: 25,000 Hours of Transcribed and Diverse English Speech Recognition Data for Research and Commercial Use},
year = {2026},
howpublished = {\url{https://pith.science/paper/JD75TWRR}},
note = {Machine review of arXiv:2505.21578}
}
read the original abstract
Automatic speech recognition (ASR) research is driven by the availability of common datasets between industrial researchers and academics, encouraging comparisons and evaluations. LibriSpeech, despite its long success as an ASR benchmark, is now limited by its size and focus on clean, read speech, leading to near-zero word error rates. More recent datasets, including MOSEL, YODAS, Gigaspeech, OWSM, Libriheavy or People's Speech suffer from major limitations including licenses that researchers in the industry cannot use, unreliable transcriptions, incorrect audio data, or the lack of evaluation sets. This work presents the Loquacious Set, a 25,000-hour curated collection of commercially usable English speech. Featuring hundreds of thousands of speakers with diverse accents and a wide range of speech types (read, spontaneous, talks, clean, noisy), the Loquacious Set is designed to work for academics and researchers in the industry to build ASR systems in real-world scenarios.
Figures
Reference graph
Works this paper leans on
-
[1]
Introduction The success of deep learning in automatic speech recognition (ASR) hinges on access to standardised datasets. Corpus like the Wall Street Journal (WSJ) [1] and TIMIT [2] were crucial for early ASR research, but their limited size and price rapidly became limiting issues. The advent of LibriSpeech [3], with its open CC-BY license and 1,000 hou...
-
[2]
Large Scale English ASR Datasets The Loquacious Set is not the first attempt to create a standard- ised large-scale ASR dataset for the English language. This sec- tion discusses others relevant initiatives. The CommonV oice dataset [13] is based on crowd- sourcing, and can theoretically leverage millions of users recording each a few samples on their dev...
work page Pith review arXiv 2025
-
[3]
The Loquacious Set This section describes the data selection and corpus structure of the Loquacious Set (section 3.1) as well as the process used to normalise the transcriptions (section 3.2). License considerations. The Loquacious dataset uses existing data and does not impose any new licensing restrictions. Each component of the original datasets keeps ...
-
[4]
Speech Recognition Experiments This section details the experimental protocol (section 4.1) as well as the speech recognition results (section 4.2) ob- tained with the Loquacious Set on various benchmarking speech recognition datasets. 4.1. Experimental protocol Models are trained on the three subsets of the Loquacious Set, hence containing either 250, 25...
-
[5]
Conclusion The Loquacious Set offers 25,000 hours of normalised tran- scribed and diverse English speech recognition data available for academics, researchers in the industry and commercial use. The dataset is easy to reproduce thanks to the SpeechBrain re- leased source code and can be loaded in a single line of code. It introduces a new common ground fo...
-
[6]
M. Gaido, S. Papi, L. Bentivogli, A. Brutti, M. Cettolo, R. Gretter, M. Matassoni, M. Nabih, and M. Negri, “Mosel: 950,000 hours of speech data for open-source speech foundation model train- ing on eu languages,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , 2024, pp. 13 934–13 947
work page 2024
-
[7]
The design for the wall street journal- based csr corpus,
D. B. Paul and J. Baker, “The design for the wall street journal- based csr corpus,” in Speech and Natural Language: Proceed- ings of a Workshop Held at Harriman, New York, February 23-26, 1992, 1992
work page 1992
-
[8]
Darpa timit acoustic-phonetic continous speech corpus cd-rom. nist speech disc 1-1.1,
J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, and D. S. Pallett, “Darpa timit acoustic-phonetic continous speech corpus cd-rom. nist speech disc 1-1.1,”NASA STI/Recon technical report n, vol. 93, p. 27403, 1993
work page 1993
Show all 23 references
-
[9]
Lib- rispeech: an asr corpus based on public domain audio books,
V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: an asr corpus based on public domain audio books,” in 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 5206–5210
2015
-
[10]
Robust speech recognition via large-scale weak supervision,
A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in International conference on machine learning . PMLR, 2023, pp. 28 492–28 518
2023
-
[11]
Conformer-1: Robust asr via large-scale semisupervised boot- strapping,
K. Zhang, L. Chkhetiani, F. M. Ramirez, Y . Khare, A. Vanzo, M. Liang, S. R. Martin, G. Oexle, R. Bousbib, T. Peyash et al., “Conformer-1: Robust asr via large-scale semisupervised boot- strapping,” arXiv preprint arXiv:2404.07341, 2024
2024 arXiv
-
[12]
Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio,
G. Chen, S. Chai, G. Wang, J. Du, W.-Q. Zhang, C. Weng, D. Su, D. Povey, J. Trmal, J. Zhang et al., “Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio,” arXiv preprint arXiv:2106.06909, 2021
2021 arXiv
-
[13]
Yodas: Youtube-oriented dataset for audio and speech,
X. Li, S. Takamichi, T. Saeki, W. Chen, S. Shiota, and S. Watan- abe, “Yodas: Youtube-oriented dataset for audio and speech,” in 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2023, pp. 1–8
2023
-
[14]
Libriheavy: a 50,000 hours asr corpus with punc- tuation casing and context,
W. Kang, X. Yang, Z. Yao, F. Kuang, Y . Yang, L. Guo, L. Lin, and D. Povey, “Libriheavy: a 50,000 hours asr corpus with punc- tuation casing and context,” in ICASSP 2024-2024 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 1...
2024
-
[15]
People’s Speech [9] and YODAS [7] are the most recent attempts at overcoming the read versus spontaneous speech issue at large scale
(MLS) is very similar to Libriheavy as it comes from the same source but is multilingual. People’s Speech [9] and YODAS [7] are the most recent attempts at overcoming the read versus spontaneous speech issue at large scale. The People’s Speech provides 30,000 hours of mostly A...
-
[16]
The people’s speech: A large-scale diverse english speech recognition dataset for commercial usage,
D. Galvez, G. Diamos, J. Ciro, J. F. Cer ´on, K. Achorn, A. Gopi, D. Kanter, M. Lam, M. Mazumder, and V . J. Reddi, “The people’s speech: A large-scale diverse english speech recognition dataset for commercial usage,” arXiv preprint arXiv:2111.09344, 2021
2021 arXiv
-
[17]
Speechstew: Simply mix all available speech recognition data to train one large neural network,
W. Chan, D. Park, C. Lee, Y . Zhang, Q. Le, and M. Norouzi, “Speechstew: Simply mix all available speech recognition data to train one large neural network,”arXiv preprint arXiv:2104.02133, 2021
2021 arXiv
-
[18]
Reproducing whisper-style training using an open-source toolkit and publicly available data,
Y . Peng, J. Tian, B. Yan, D. Berrebbi, X. Chang, X. Li, J. Shi, S. Arora, W. Chen, R. Sharma et al., “Reproducing whisper-style training using an open-source toolkit and publicly available data,” in 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEE...
2023
-
[19]
Common voice: A massively-multilingual speech corpus,
R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,”arXiv preprint arXiv:1912.06670, 2019
1912 arXiv
-
[20]
Open-source conversational ai with speechbrain 1.0,
M. Ravanelli, T. Parcollet, A. Moumen, S. de Langen, C. Subakan, P. Plantinga, Y . Wang, P. Mousavi, L. D. Libera, A. Ploujnikov, F. Paissan, D. Borra, S. Zaiem, Z. Zhao, S. Zhang, G. Karakasidis, S.-L. Yeh, P. Champion, A. Rouhe, R. Braun, F. Mai, J. Zuluaga- Gomez, S. M. Mou...
2024
-
[21]
Mls: A large-scale multilingual dataset for speech research,
V . Pratap, Q. Xu, A. Sriram, G. Synnaeve, and R. Collobert, “Mls: A large-scale multilingual dataset for speech research,” arXiv preprint arXiv:2012.03411, 2020
2012 arXiv
-
[22]
V oxpopuli: A large-scale multilingual speech corpus for representation learn- ing, semi-supervised learning and interpretation,
C. Wang, M. Riviere, A. Lee, A. Wu, C. Talnikar, D. Haz- iza, M. Williamson, J. Pino, and E. Dupoux, “V oxpopuli: A large-scale multilingual speech corpus for representation learn- ing, semi-supervised learning and interpretation,” in Proceedings of the 59th Annual Meeting of ...
2021
-
[23]
Joint ctc-attention based end-to-end speech recognition using multi-task learning,
S. Kim, T. Hori, and S. Watanabe, “Joint ctc-attention based end-to-end speech recognition using multi-task learning,” in2017 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2017, pp. 4835–4839
2017
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.