A Speech Test Set of Practice Business Presentations with Additional Relevant Texts

Dominik Mach\'a\v{c}ek; Jon\'a\v{s} Kratochv\'il; Ond\v{r}ej Bojar; Tereza Vojt\v{e}chov\'a

REVIEW 2 major objections 6 minor 20 references

A Speech Test Set of Practice Business Presentations with Additional Relevant Texts

T0 review · 2 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper presents a publicly released test corpus of 39 noisy, non-native English business presentations with slides and webpages, and reports that three baseline ASR systems all fail on it, with the best system's mean word error rate…

desk verdict Worth a look: a small but real L2 English ASR test set with noisy audio and aligned slides/web texts, though its headline WERs rest on unverified self-transcriptions and should be treated as approximate. read the letter →

arxiv 1908.00916 v1 pith:DICTA5V7 submitted 2019-08-02 cs.CL

Dominik Macháček , Jonáš Kratochvíl , Tereza Vojtěchová , Ondřej Bojar This is my paper · ORCID

classification cs.CL

keywords automaticspeechrecognitionASRevaluationcorpusnon-nativeEnglishL2businesspresentationsnoisydomainadaptation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's aim is to give the speech-recognition community a test set that matches a real, difficult setting: short English business pitches given by non-native speakers in a noisy hall. The corpus contains 39 presentations, 61 speakers, about one hour of audio, and reference transcriptions written by the speakers and then edited by the authors. For 36 of the firms, it also includes the presentation slides and web pages, so a system can be tested with and without access to in-domain vocabulary and named entities before hearing the talk. The authors benchmark three ASR systems and show that none handles the data well: the best mean word error rate is 45.63%.

What carries the argument

The load-bearing object is the test corpus itself, built as triplets: audio recordings, speaker-produced reference transcripts, and in-domain texts (slides, web pages) for 36 of 39 firms. The authors define the evaluation setup through word error rate (WER)—the minimum number of insertions, deletions, and substitutions needed to edit the ASR output into the reference, divided by reference word count—computed with case and punctuation ignored. The baselines are defined by their training resources (TED-LIUM 3 and Broadcast News for JRTk; 1600 hours of BBC audio and subtitle text for the Kaldi model), so the corpus's reported failure rates are tied to those configurations. The additional texts come in three formats (original, XLIFF, plaintext), which the authors suggest makes them easy to use for vocabulary extraction or adaptation experiments.

What would settle it

Have an independent professional transcriber re-transcribe a sample of the recordings, and compute the word error rate between the corpus reference and the professional transcript; if that disagreement is as large as the ASR error rates (tens of percent), then the ground-truth assumption fails and the benchmark cannot separate ASR errors from transcript errors.

Watch

Extended reading notes

Core claim

The central claim is that a one-hour corpus of 39 recordings of student-run business presentations, spoken in L2 English by 61 European high-school students and captured with headset microphones in a noisy trade-fair environment, is a usable and challenging public test set for ASR. Its special feature is that extra relevant texts—slides and web pages of the fictional companies—are packaged in original, XLIFF, and plaintext forms, giving evaluators a way to study whether supplying in-domain vocabulary and named entities before recognition improves output. The paper establishes the difficulty by evaluating three baselines: a JRTk system trained on TED talks and Broadcast News, a Kaldi model trained on BBC broadcast data, and Google Cloud Speech-to-Text. On all 39 recordings, mean word error rates are 45.63% for JRTk, 89.32% for Google, and 87.47% for the Kaldi-BBC system, with JRTk's individual scores ranging from 25% to over 99%; the authors take this as evidence that current systems are far from robust on accented, noisy, spontaneous speech.

Load-bearing premise

The reported word error rates are only meaningful if the reference transcriptions, written by the student speakers and lightly edited by the authors, accurately represent what was said; if those transcripts are systematically incomplete or paraphrased, every WER in the paper would be off.

Editorial extensions

If this is right

The corpus offers a ready-made benchmark for noisy, non-native, spontaneous English, with per-recording WER spread wide enough to distinguish robust systems from brittle ones.
Because slides and web pages contain the same named entities and domain words as the talks, the corpus allows a controlled test of whether exposing an ASR system to that text beforehand reduces errors.
The reported failure of a cloud ASR system on some recordings (100% WER from empty outputs) indicates that the corpus can also stress-test a system's noise robustness, not just its language model.
The corpus gives a way to evaluate whether ASR models trained on L1 English speech, such as TED talks or broadcast audio, transfer to the European L2 English found in international business settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension not reported in the paper is a controlled adaptation experiment: take the slides and web pages, extract their named entities, bias the ASR language model, and measure the change in WER specifically on entity words; the corpus's design supports exactly this comparison.
Because the reference transcripts deliberately preserve non-standard learner grammar and vocabulary, the corpus measures ASR against authentic L2 speech, not idealized English; systems fine-tuned on corrected transcripts may look worse on it than they would in a deployment where speakers' errors are accepted.
The one-hour size makes the corpus unsuitable for training, which is likely the authors' intent; its value is as a targeted test set, and combining it with larger training corpora in a multi-task setting is a plausible use.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is a modest but genuinely useful corpus paper. The new thing is the package: L2 English from eight European L1s, realistic noisy fairground audio, and aligned slides and web pages for 36 of the 39 firms. No prior speech corpus I know of combines those three elements. The release itself is the contribution, and it is a reasonable one for ASR evaluation of non-native speech in noisy conditions.

The paper also does several things well. The collection methodology is clearly described, including the incentive structure for the transcription competition. The authors are transparent about the messy parts: one ASR system failed completely, Google returned empty outputs on some recordings, and the recordings contain live music and announcements. They preserved non-standard grammar in the transcripts instead of silently "correcting" it, which is the right choice for authenticity. The baseline WERs are presented as descriptive statistics, not as a scientific claim about ASR limitations. For a corpus paper, that is appropriate.

The soft spot is exactly where the stress-test note points. The ground-truth transcripts were produced by the student speakers themselves and then edited by the authors. There is no independent transcription, no inter-annotator agreement, and no quantitative check of reference accuracy. Section 4.2 simply takes the participant transcripts as ground truth. Self-transcription can systematically normalize speech: speakers omit fillers, false starts, and repeated words, or write what they intended to say rather than what they actually said. If that happened here, every WER in Table 7 is biased, and the benchmark is less reliable for comparing ASR systems. I think the qualitative picture would survive—this audio is clearly hard for current ASR—but the precise numbers should not be treated as a rigorous benchmark until the reference is verified.

A second, minor point: the paper claims the additional texts could help domain adaptation, but does not demonstrate that they actually do. That is fine for a resource paper, as long as the claim stays a suggestion rather than a result.

Who is this for? ASR researchers working on non-native English, noisy conditions, or domain adaptation. They will get value from the corpus itself and from the honest baseline documentation. The paper deserves a serious referee, but the authors should be asked to either add some independent verification of a sample of transcriptions or explicitly label the WERs as approximate and frame the corpus as a challenge set rather than a precisely labeled benchmark.

Referee Report

2 major / 6 minor

Summary. The manuscript presents a new, publicly released test corpus composed of 39 audio recordings of practice business presentations given by high-school students in English as a second language, together with manual transcriptions, presentation slides, and web pages of the students' fictional firms. The authors describe the collection methodology, including an incentive scheme in which participants transcribed their own speech, and report baseline word error rates for three ASR systems (Google Cloud Speech-to-Text, Kaldi BBC, and JRTk). The reported WERs are high, e.g., a mean WER of 45.63% for JRTk on all recordings, and the paper positions the corpus as a challenging test set for ASR and domain adaptation.

Significance. If the resource is reliable, it fills a genuine gap: a publicly available, noisy L2-English speech test set with in-domain texts and named entities, collected under realistic trade-fair conditions with eight European L1 backgrounds. The authors are careful about ethical compliance and data formats, and the corpus is released with a persistent handle, which is commendable. The baseline WER measurements, however, are trustworthy only to the extent that the reference transcriptions are verbatim and accurate; the paper currently provides no independent verification of transcription quality. The resource itself has value independent of the baseline numbers, but the benchmark claims in Section 4 need strengthening before the numbers can be used as a reliable comparison point.

major comments (2)

[Section 2.3 and Section 4.2] The WER benchmark in Section 4 is scored against reference transcripts that were produced by the speakers themselves and then informally edited by the authors (Section 2.3), with no independent transcription, inter-annotator agreement, or quantitative verification of verbatim accuracy. Because speakers may systematically normalize their own speech (e.g., omitting fillers, false starts, or repairs, or writing what they intended rather than what was actually uttered), the reference may be non-verbatim in ways that bias the reported WERs. Section 4.2 states that the participant transcripts were taken as ground truth without further validation. The paper should either report an independent verification (e.g., re-transcribe a sample of the recordings and provide agreement statistics) or explicitly qualify the reported WER values as preliminary and state this limitation as a caveat for users of the corpus.
[Section 4.3, Table 7] In Table 7, the Kaldi BBC model's WER has an implausibly narrow spread across the corpus (standard deviation 2.29 on the 'Recognized by all' subset, min/max 83.96/91.03 on all recordings), while Google and JRTk vary by tens of percentage points. This pattern suggests the Kaldi outputs may be dominated by a systematic artifact, such as a constant misrecognition or a decoding failure that still returns text. The authors should provide a few representative Kaldi transcripts alongside their references to rule out a degenerate decoding path, and they should report the number of recordings in the 'Recognized by all' subset so the reader can assess how much data the left-hand columns of Table 7 represent.

minor comments (6)

[Table 3] The single-speaker row of Table 3 lists counts that sum to 15 rather than the stated total of 17; the per-language counts (cs, es, ro, sk, hu) should be checked and corrected.
[Table 6] The Web row of Table 6 sums to 24 rather than the listed total of 23; this should be reconciled with Table 5, which implies 24 firms with web pages (20 with both slides and web plus 4 with web only).
[Abstract and Section 2.2] The abstract contains the non-standard word 'beneﬁtable', presumably a typo for 'beneficial', and Section 2.2 says winners 'were awarded prices for their performances', which should read 'prizes'.
[Section 4.1] The sentence 'We also tried Microsoft Cloud ASR but it failed for all our recordings' should specify the failure mode (empty outputs, server errors, or other) and the number of recordings attempted.
[Section 4.3] The paper should report how many recordings fall in the 'Recognized by all' subset, since the left-hand columns of Table 7 and Figure 1 are based on an unreported number of data points.
[References] Reference [17] contains typographical errors in the author names ('Mller', 'Stker', 'Zenkel') and should be corrected to the proper spellings (Müller, Stüker, Zenkel).

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper presents a corpus and reports measured WER of external ASR systems, with no derivation or fitted prediction that reduces to its inputs.

full rationale

This paper does not derive any formal result from its own assumptions. It describes the collection of a speech corpus and reports word error rates of three external ASR systems (Google Cloud Speech-to-Text, a Kaldi BBC model, and JRTk) against human-produced reference transcriptions. The WER numbers in Table 7 are measurements of black-box systems, not outputs of a model that the paper fits or defines. Consequently, none of the circularity patterns apply: there is no self-definitional relationship, no fitted parameter later renamed as a prediction, and no load-bearing self-citation chain. The one substantive validity concern is the accuracy of the ground-truth transcriptions, which were produced by the student speakers themselves and only informally edited by the authors (Sections 2.3 and 4.2). That concern is a data-quality or correctness risk, not a circularity: the corpus and the WER evaluation are not defined in terms of each other, and the reported numbers are empirical observations rather than consequences of an assumed model. Self-citations to external ASR toolkits are normal references to public systems and do not carry the argument. Therefore the honest finding is no significant circularity, with score 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters or invented entities. The central claims depend on domain assumptions about transcription quality, metric validity, and the usefulness of the additional materials, none of which are independently proven in the paper.

assumptions (4)

domain assumption Student self-transcriptions, after author review, are sufficiently accurate to serve as ASR ground truth.
Section 2.3 describes the transcription process but provides no inter-annotator agreement or independent verification.
domain assumption WER is an appropriate metric for evaluating ASR output on this corpus.
Section 4.2 defines WER as the standard metric; this is conventional but still an assumption about what the evaluation should measure.
domain assumption The additional slides and web pages are relevant enough to the spoken content to be useful for domain adaptation.
Section 3.3 claims this utility, but the paper does not run any adaptation experiment to verify it.
domain assumption The noisy recording conditions at the fair are representative of real-world environments for ASR deployment.
Section 3 asserts this belief; no comparison to other noisy conditions is provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Speech Test Set of Practice Business Presentations with Additional Relevant Texts." pith.science (2026). https://pith.science/paper/DICTA5V7

@misc{pith2026190800916,
  author       = {Pith},
  title        = {Pith review of: A Speech Test Set of Practice Business Presentations with Additional Relevant Texts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DICTA5V7}},
  note         = {Machine review of arXiv:1908.00916}
}

read the original abstract

We present a test corpus of audio recordings and transcriptions of presentations of students' enterprises together with their slides and web-pages. The corpus is intended for evaluation of automatic speech recognition (ASR) systems, especially in conditions where the prior availability of in-domain vocabulary and named entities is benefitable. The corpus consists of 39 presentations in English, each up to 90 seconds long. The speakers are high school students from European countries with English as their second language. We benchmark three baseline ASR systems on the corpus and show their imperfection.

Figures

Figures reproduced from arXiv: 1908.00916 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 17 canonical work pages

[1]

In: Linguistic Data Consortium (LDC)

Abdulaziz, A., Kepuska, V .: Noisy TIMIT Speech LDC2017S 04. In: Linguistic Data Consortium (LDC). Linguistic Data Consortium (LDC), U niversity of Penn- sylvania (2017)

work page 2017
[2]

In: Proc

Bell, P ., Gales, M., Hain, T., Kilgour, J., Lanchantin, P ., Liu, X., McParland, A., Renals, S., Saz, O., Wester, M., Woodland, P .: The MGB Challe nge: Evaluating Multi-Genre Broadcast Media Recognition. In: Proc. ASRU (2 015)

work page
[3]

In: 2017 IEEE Automatic Speech Recognition and Understanding Works hop (ASRU)

Ghahremani, P ., Manohar, V ., Hadian, H., Povey, D., Khudanpur, S.: Investigation of Transfer Learning for ASR Using LF-MMI Trained Neural Net works. In: 2017 IEEE Automatic Speech Recognition and Understanding Works hop (ASRU). pp. 279–286 (Dec 2017)

work page 2017
[4]

In: Proceedings of the 1997 DARP A Speech Recognition Worksh op

Graff, D.: The 1996 Broadcast News Speech And Language-M odel Corpus. In: Proceedings of the 1997 DARP A Speech Recognition Worksh op. pp. 11–14 (1996)

work page 1996
[5]

In: Proceedings of the Ninth International Conference on Language Resource s and Evaluation (LREC’14)

Gretter, R.: Euronews: a Multilingual Speech Corpus for ASR. In: Proceedings of the Ninth International Conference on Language Resource s and Evaluation (LREC’14). pp. 2635–2638. European Language Resources Ass ociation (ELRA), Reykjavik, Iceland (May 2014)

work page 2014
[6]

CoRR abs/1805.04699 (2 018), http://arxiv.org/abs/1805.04699

Hernandez, F., Nguyen, V ., Ghannay, S., Tomashenko, N.A ., Est` eve, Y .: TED-LIUM 3: Twice as Much Data and Corpus Repartition for Experiments on Speaker Adaptation. CoRR abs/1805.04699 (2 018), http://arxiv.org/abs/1805.04699

arXiv
[7]

In: Proc

Hu, Y ., Loizou, P .: Subjective Comparison of Speech Enha ncement Algorithms. In: Proc. of ICASSP. vol. 1 (Jun 2006)

work page 2006
[8]

In: Proc

Kim, C., Misra, A., Chin, K., Hughes, T., Narayanan, A., S ainath, T., Bacchiani, M.: Generation of large-scale simulated utterances in virt ual rooms to train deep- neural networks for far-ﬁeld speech recognition in Google H ome. In: Proc. of INTERSPEECH (Aug 2017)

work page 2017

Show all 20 references

[9]

In: P roceedings of ICASSP 97 (Jan 1997)

Lavie, A., Waibel, A., Levin, L., , Gates, D., , Zeppenfel d, T., Zhan, P .: JANUS III: Speech-to-speech Translation in Multiple Languages. In: P roceedings of ICASSP 97 (Jan 1997)

1997
[10]

In: 2018 IEEE Spoken L anguage Technol- ogy Workshop, SLT 2018, Athens, Greece, December 18-21, 201 8

Narayanan, A., Misra, A., Sim, K.C., Pundak, G., Tripat hi, A., Elfeky, M., Haghani, P ., Strohman, T., Bacchiani, M.: Toward Domain-In variant Speech Recognition via Large Scale Training. In: 2018 IEEE Spoken L anguage Technol- ogy Workshop, SLT 2018, Athens, Greece, December...

2018
[11]

In: 2015 IEEE Interna tional Confer- ence on Acoustics, Speech and Signal Processing (ICASSP)

Panayotov, V ., Chen, G., Povey, D., Khudanpur, S.: Libr ispeech: An ASR Cor- pus Based on Public Domain Audio Books. In: 2015 IEEE Interna tional Confer- ence on Acoustics, Speech and Signal Processing (ICASSP). p p. 5206–5210 (Apr 2015)

2015
[12]

In: IEEE 2011 Work shop on Auto- matic Speech Recognition and Understanding

Povey, D., Ghoshal, A., Boulianne, G., Burget, L., Glem bek, O., Goel, N., Hanne- mann, M., Motl´ ıˇ cek, P ., Qian, Y ., Schwarz, P ., Silovsk´ y,J., Stemmer, G., V esel´ y, A Speech Test Set of Practice Business Presentations with Ad ditional Relevant Texts 11 K.: The Kaldi ...

2011
[13]

In: Interspeech 2016

Povey, D., Peddinti, V ., Galvez, D., Ghahremani, P ., Ma nohar, V ., Na, X., Wang, Y ., Khudanpur, S.: Purely Sequence-Trained Neural Ne tworks for ASR Based on Lattice-Free MMI. In: Interspeech 2016. pp. 2751–2 755 (2016), http://dx.doi.org/10.21437/Interspeech.2016-595

2016 doi
[14]

In: Linguistic Data Consortium (LDC)

Schmidt-Nielsen, A., Marsh, E., Tardelli, J., Gatewoo d, P ., Kreamer, E., Tremain, T., Cieri, C., Wright, J.: Speech in Noisy Environments (SPI NE) Training Audio LDC2000S87. In: Linguistic Data Consortium (LDC). Linguis tic Data Consor- tium (LDC), University of Pennsylvania (2000)

2000
[15]

In: IEEE Works hop on Automatic Speech Recognition and Understanding, 2001

Soltau, H., Metze, F., Fugen, C., Waibel, A.: A One-Pass Decoder Based on Polymorphic Linguistic Context Assignment. In: IEEE Works hop on Automatic Speech Recognition and Understanding, 2001. ASRU ’01. pp. 2 14–217 (Dec 2001)

2001
[16]

master thesis, Universit¨ at des Saarlandes, Universit` a degli studi di Trento (2018)

Tchistiakova, S.: Acoustic Models for Second Language Learners. master thesis, Universit¨ at des Saarlandes, Universit` a degli studi di Trento (2018)

2018
[17]

In: The International Workshop on Spoken Lan- guage Translation (IWSLT)

Thai-Son Nguyen and Markus Mller and Sebastian Sperber and Thomas Zenkel and Sebastian Stker and Alex Waibel: The 2017 KIT IWSLT Speec h-to-Text Sys- tems for English and German. In: The International Workshop on Spoken Lan- guage Translation (IWSLT). Tokyo, Japan (December, 1...

2017
[18]

In: 2013 IEEE Workshop on Automa tic Speech Recognition and Understanding

Vincent, E., Barker, J., Watanabe, S., Le Roux, J., Nest a, F., Matassoni, M.: The second CHiME speech separation and recognition challenge: An overview of chal- lenge systems and outcomes. In: 2013 IEEE Workshop on Automa tic Speech Recognition and Understanding. pp. 162–167 ...

2013
[19]

In: ICASSP , IEEE Internat ional Conference on Acoustics, Speech and Signal Processing - Proceedings

Wang, Z., Schultz, T., Waibel, A.: Comparison of Acoust ic Model Adaptation Techniques on Non-Native Speech. In: ICASSP , IEEE Internat ional Conference on Acoustics, Speech and Signal Processing - Proceedings. v ol. 1 (May 2003)

2003
[20]

In: Proc

Zhao, G., Sonsaat, S., Silpachai, A., Lucic, I., Chukha rev-Hudilainen, E., Levis, J., Gutierrez-Osuna, R.: L2-ARCTIC: A Non-Nativ e En- glish Speech Corpus. In: Proc. Interspeech 2018. pp. 2783–2 787 (2018), http://dx.doi.org/10.21437/Interspeech.2018-1110

2018 doi

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

In: Linguistic Data Consortium (LDC)

Abdulaziz, A., Kepuska, V .: Noisy TIMIT Speech LDC2017S 04. In: Linguistic Data Consortium (LDC). Linguistic Data Consortium (LDC), U niversity of Penn- sylvania (2017)

work page 2017

[2] [2]

In: Proc

Bell, P ., Gales, M., Hain, T., Kilgour, J., Lanchantin, P ., Liu, X., McParland, A., Renals, S., Saz, O., Wester, M., Woodland, P .: The MGB Challe nge: Evaluating Multi-Genre Broadcast Media Recognition. In: Proc. ASRU (2 015)

work page

[3] [3]

In: 2017 IEEE Automatic Speech Recognition and Understanding Works hop (ASRU)

Ghahremani, P ., Manohar, V ., Hadian, H., Povey, D., Khudanpur, S.: Investigation of Transfer Learning for ASR Using LF-MMI Trained Neural Net works. In: 2017 IEEE Automatic Speech Recognition and Understanding Works hop (ASRU). pp. 279–286 (Dec 2017)

work page 2017

[4] [4]

In: Proceedings of the 1997 DARP A Speech Recognition Worksh op

Graff, D.: The 1996 Broadcast News Speech And Language-M odel Corpus. In: Proceedings of the 1997 DARP A Speech Recognition Worksh op. pp. 11–14 (1996)

work page 1996

[5] [5]

In: Proceedings of the Ninth International Conference on Language Resource s and Evaluation (LREC’14)

Gretter, R.: Euronews: a Multilingual Speech Corpus for ASR. In: Proceedings of the Ninth International Conference on Language Resource s and Evaluation (LREC’14). pp. 2635–2638. European Language Resources Ass ociation (ELRA), Reykjavik, Iceland (May 2014)

work page 2014

[6] [6]

CoRR abs/1805.04699 (2 018), http://arxiv.org/abs/1805.04699

Hernandez, F., Nguyen, V ., Ghannay, S., Tomashenko, N.A ., Est` eve, Y .: TED-LIUM 3: Twice as Much Data and Corpus Repartition for Experiments on Speaker Adaptation. CoRR abs/1805.04699 (2 018), http://arxiv.org/abs/1805.04699

arXiv

[7] [7]

In: Proc

Hu, Y ., Loizou, P .: Subjective Comparison of Speech Enha ncement Algorithms. In: Proc. of ICASSP. vol. 1 (Jun 2006)

work page 2006

[8] [8]

In: Proc

Kim, C., Misra, A., Chin, K., Hughes, T., Narayanan, A., S ainath, T., Bacchiani, M.: Generation of large-scale simulated utterances in virt ual rooms to train deep- neural networks for far-ﬁeld speech recognition in Google H ome. In: Proc. of INTERSPEECH (Aug 2017)

work page 2017