Pith. sign in

REVIEW 4 major objections 3 minor 50 references

How to Evaluate Automatic Speech Recognition: Comparing Different Performance and Bias Measures

T0 review · 4 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Averaged word error rates are not enough: ASR evaluation should also report the median, spread, and relative bias across speaker groups.

desk verdict A useful empirical comparison of ASR bias measures with a real finding about mitigation increasing group bias, but the specific numbers need error bars and one Table 5 value is a typo. read the letter →

arxiv 2507.05885 v1 pith:MQ75BQNV submitted 2025-07-08 cs.CL cs.SDeess.AS

Tanvina Patel , Wiebke Hutiri , Aaron Yi Ding , Odette Scharenborg This is my paper · ORCID
classification cs.CLcs.SDeess.AS
keywords ASRbiasworderrorrateperformancemeasuresfairnessDutchspeechrecognitionspeakerdiversityend-to-end
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the standard practice of reporting a single average word error rate (WER) for an automatic speech recognition system is insufficient for evaluating performance and bias across diverse speaker groups. Using Dutch end-to-end ASR systems trained on adult native speech and tested on children, teenagers, non-native speakers, and older adults, it compares averaged, median, standard deviation, and range of WER against several bias measures. The central finding is that bias-mitigation techniques that improve average WER—such as speed perturbation, spectral augmentation, and fine-tuning—can increase bias against non-native speakers. The paper therefore recommends reporting the median, standard deviation, and range alongside the average, and measuring bias as a relative difference with the minimum-WER group as reference. If adopted, ASR evaluation would more honestly reflect how well systems serve diverse users rather than just the 'average' user.

What carries the argument

The central objects are the performance measures (average, median, standard deviation, range of WER) and the bias measures: group-to-min and group-to-norm differences (absolute and relative), plus an overall-bias meta-measure that averages these differences across speaker groups. These measures are computed per speaker group (children, teenagers, non-native teens/adults, older adults) and per speech type (read vs. human-machine interaction). The argument is carried by the comparison of these measures across five ASR systems—a conformer trained without augmentation, with speed perturbation, with speed and spectral augmentation, a pre-trained model, and that model fine-tuned on the training corpus—showing that average WER and bias measures diverge, particularly after mitigation.

What would settle it

Re-compute the bias rankings with bootstrap confidence intervals or significance tests on the per-group WERs; if the differences between systems (e.g., the non-native read WER of 52.2% vs. 52.8% for NoAug vs. SpAug) overlap within uncertainty, the conclusion that mitigation increases bias would be unsupported. Alternatively, re-run the evaluation on a read test set larger than 0.45 hours per group; if the bias rankings change, the small test set is the culprit.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that averaged WER is a poor summary statistic for ASR systems evaluated across diverse speakers, because it hides large performance spreads (e.g., non-native read speech ranging from 21.5% to 57.2% WER while the average is 40.0%). More specifically, it shows that several bias mitigation strategies—speed perturbation, spectral augmentation, and fine-tuning on the norm corpus—while improving average WER, mostly increase bias against non-native speakers relative to the best-performing group or the norm group. The paper demonstrates that different bias measures (difference vs. relative difference, group-to-min vs. group-to-norm) can produce different or even contradictory rankings, so the choice of measure matters. It concludes that ASR performance should be reported with the median and standard deviation, and system bias with a relative measure that takes all speaker groups into account using the minimum-WER group as the reference. The paper also notes that unlike WER, bias has no ground truth, so user feedback should complement quantitative measures.

Load-bearing premise

The analysis treats the per-group word error rates as stable, real differences, but some groups are tested on very small sets (the read CGN set is only 0.45 hours) and no confidence intervals or significance tests are reported, so a difference of one to three WER points could reflect sampling noise.

Editorial extensions

If this is right

  • ASR model cards and leaderboards should report median, standard deviation, and range of WER in addition to the average, broken down by speaker group and speech type.
  • Bias should be reported as a relative difference to the minimum-WER group (or to the norm group when training data is available), not only as an absolute difference.
  • A mitigation strategy that improves average WER should not be assumed to reduce bias; it can increase bias against specific groups such as non-native speakers.
  • When training data is unavailable for a pre-trained model, the minimum-WER group is the appropriate reference, because the norm group may not be identifiable.
  • Overall bias should be computed as an average of group-level biases, while also reporting the standard deviation across groups to capture variability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The central mechanism—that averaging hides spread—is language- and architecture-independent, so the reporting recommendation likely transfers to ASR in other languages and to newer end-to-end models; a replication on English or multilingual test suites would test this.
  • Because the read-speech norm test set is very small (0.45 h), some of the exact bias rankings may be fragile; a re-analysis with confidence intervals could overturn specific conclusions, though the qualitative finding that average WER can improve while bias worsens would probably survive.
  • A practical extension is to report a 'bias delta' per mitigation: the change in overall relative bias alongside the change in average WER, so developers can see the trade-off directly.
  • The paper's final note that bias lacks ground truth suggests that quantitative bias measures should be paired with user studies; a concrete next step is collecting subjective intelligibility ratings from the same speaker groups.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper compares performance and bias measures for evaluating automatic speech recognition systems, using Dutch end-to-end models trained on CGN and evaluated on CGN and the Jasmin corpus with speaker groups distinguished by age and nativeness. The authors compute WER averages, medians, standard deviations, and ranges, as well as group-based bias measures (G2min and G2norm, in difference and relative forms) and an aggregate Overall Bias. Their main finding is that averaged WER hides large disparities across speaker groups, that the median is consistently below the mean, and that bias mitigation approaches (SpAug, SpSpecAug, fine-tuning) improve average WER but often increase bias against non-native speakers. They close with recommendations to report median, standard deviation, and a relative bias measure using the minimum-WER group as reference.

Significance. If the results are reliable, the paper provides a useful, concrete evaluation protocol for ASR fairness and addresses a gap in the literature on how to measure bias when many speaker groups are involved. The central descriptive finding--that WER spread across the Jasmin groups is large and that the median sits below the mean--is well supported by Tables 2 and 4 and is robust to the issues I raise. The paper also makes a clear methodological contribution by systematically comparing bias measures that are often used inconsistently in the literature. However, the specific conclusions about bias mitigation increasing bias and about which model is least biased depend on point estimates without uncertainty quantification and on Table 5, which contains a concrete arithmetic error. The contribution is empirical rather than theoretical, and its actionable force hinges on correcting these issues.

major comments (4)
  1. [Section 4.2 / Table 5] The W_sFTcgn row of Table 5 reports G2normdiff Read = 24.9, but recomputing from Table 2 with Eq. 1 gives a different value. With bnorm = CGN Read = 7.8 and the five read-group WERs (DC 40.9, DT 22.4, NnT 57.7, NnA 60.8, DOA 28.2), the group differences are 33.1, 14.6, 49.9, 53.0, and 20.4, whose mean is 34.2. The same row's relative value (4.38) equals 34.2/7.8, confirming that the tabulated 24.9 is an error. Since this value is used in the cross-model comparison and in the claim that fine-tuning increases bias relative to Whisper, all entries of Table 5 should be recomputed and verified before the bias rankings are relied upon.
  2. [Section 4.2 / Tables 2, 4, 5] No confidence intervals, bootstrap resampling, or significance tests are reported, yet the narrative that 'bias mitigation approaches, despite improving average WER, mostly show increased bias against non-native speakers' rests on one-to-three-point WER differences on small test sets (e.g., CGN read is 0.45 h and DT HMI is 0.94 h per Table 1). For instance, SpAug changes NnA read WER from 52.2% to 52.8% (Table 2), and several fine-tuning differences are of similar magnitude. A cluster-by-speaker bootstrap or a pairwise significance test could plausibly reverse the sign of these deltas. I ask that the authors provide uncertainty estimates or explicitly temper the causal claims in Section 4.2.
  3. [Section 4.2 / Table 5] The statement 'Overall, W_s exhibits the lowest bias across all measures' is not supported by Table 5 as printed. For HMI G2mindiff, W_s has the highest value (18.85) among all models, not the lowest (NoAug: 13.20; SpAug: 16.13; SpSpecAug: 18.73; W_sFTcgn: 13.73). Additionally, as printed, W_sFTcgn's G2normdiff Read (24.9) is lower than W_s's (25.26), so even under the published numbers the claim is not strictly true; after the arithmetic correction it becomes true for that cell, but the blanket 'across all measures' statement still needs qualification.
  4. [Section 4.2 / Figure 1 and Table 5] The comparison of bias between Read and HMI speech uses different norm references: CGN Read for Read speech and CGN CTS for HMI speech. The observation that 'HMI speech shows lower bias than read speech, despite higher WERs for HMI' is partly an artifact of the higher WER of the CTS reference group (e.g., 54.1% for W_s versus 17.1% for CGN Read). This confound is not discussed and limits the interpretability of cross-speech-type bias comparisons. Please either match the reference groups across speech types or analyze the sensitivity of the bias measures to the choice of norm.
minor comments (3)
  1. [Figure 1] The legend contains a typo, 'G2min-NoAaug' should be 'G2min-NoAug', and the many colored series make the figure difficult to read; a tabular version or separate panels would improve clarity.
  2. [Section 3.2] The phrase 'Unigram model with 5k byte pair tokens' is unclear; it should probably read 'a unigram language model with a 5k byte-pair-encoded subword vocabulary.'
  3. [Table 1] The header 'Test-Read | Test-CTS/HMI' combined with the 'Style' column entries like 'Read |CTS' is confusing; the two test components (Read and CTS/HMI) should be presented as separate columns with clear labels.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: bias measures are explicitly re-derived arithmetic functions of directly measured WER on fresh public-corpus test sets; the sole self-citation is non-load-bearing.

full rationale

The paper's derivation chain is self-contained and non-circular. The bias measures G2mindiff, G2normdiff, G2minreldiff, G2normreldiff, and the Overall Bias meta-measure are simple arithmetic transformations of WER, defined explicitly in Eqs. 1-2 of Section 3.3, and computed on WER values measured from fresh test data from the public CGN and Jasmin corpora (Tables 1 and 2). The central claim that averaged error rates are not sufficient is an empirical observation grounded in the large spread of the per-group WER values, e.g., NoAug read speech ranges from 21.5% to 57.2% across the five Jasmin groups while the average is 40.0% (Tables 2 and 4). That observation does not follow from the definitions of the bias measures by construction, since the spread could in principle have been small. The only self-citation is the attribution of G2norm and the Overall Bias meta-measure to the first author's prior work (ref [33]); this attribution is not load-bearing because the formulas are re-derived explicitly in the present paper (Eqs. 1-2) and their numerical values are recomputed from the new test data, so the conclusions do not depend on [33] as an unexamined premise. No uniqueness theorem is imported, no fitted parameter is relabeled as a prediction, and no known empirical result is renamed. The internal inconsistency flagged in review, in which Table 5 reports G2normdiff Read = 24.9 for WsFTcgn while recomputation from Table 2 via Eq. 1 yields 34.2, is an arithmetic-consistency defect that weakens the reliability of the cross-model bias rankings, but it is not circularity: the measure remains a defined function of the data rather than a restatement of the paper's conclusion. Because the analysis is self-contained against external benchmark data and the measures are parameter-free, the appropriate finding is no significant circularity, with the score reflecting only the presence of a minor, non-load-bearing self-citation.

Assumptions & free parameters 0 free parameters · 5 assumptions · 1 invented entities

The central analysis adds no free parameters and no physical entities; it takes WER, the group definitions, and the chosen reference groups from the corpora and prior work. The main load-bearing choices are the domain assumptions listed above.

assumptions (5)
  • domain assumption WER is a valid base metric for ASR performance and bias assessment.
    Section 3.3 defines all measures on WER; the paper does not test whether conclusions change under CER or PER.
  • domain assumption Jasmin speaker-group labels (age group, native vs non-native) are accurate ground-truth metadata.
    Section 3.1 relies on these labels to define groups; errors here would shift bias measures.
  • domain assumption CGN adult native test sets are an appropriate norm reference for both read and HMI conditions.
    Section 4.2 uses CGN read as norm for read speech and CGN CTS as norm for HMI speech, despite CTS being human-human telephone speech rather than human-machine speech.
  • domain assumption Error-rate parity is the operative fairness notion.
    Section 2.2 states most ASR fairness work implicitly assumes error-rate parity; the paper inherits it without defending it.
  • domain assumption Reported point estimates are stable enough to rank systems and measures.
    Section 4.1 draws conclusions from 1-3 point WER differences on test sets as small as 0.45 h, with no uncertainty quantification.
invented entities (1)
  • G2normreldiff (relative group-to-norm bias measure)
    purpose: Measures each group's WER gap relative to the norm group's WER, normalized by the norm WER; proposed in this paper (Table 3).
    A rescaling of existing quantities; it is a definition, not an entity with a falsifiable prediction outside this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of How to Evaluate Automatic Speech Recognition: Comparing Different Performance and Bias Measures." pith.science (2026). https://pith.science/paper/MQ75BQNV

@misc{pith2026250705885,
  author       = {Pith},
  title        = {Pith review of: How to Evaluate Automatic Speech Recognition: Comparing Different Performance and Bias Measures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MQ75BQNV}},
  note         = {Machine review of arXiv:2507.05885}
}
read the original abstract

There is increasingly more evidence that automatic speech recognition (ASR) systems are biased against different speakers and speaker groups, e.g., due to gender, age, or accent. Research on bias in ASR has so far primarily focused on detecting and quantifying bias, and developing mitigation approaches. Despite this progress, the open question is how to measure the performance and bias of a system. In this study, we compare different performance and bias measures, from literature and proposed, to evaluate state-of-the-art end-to-end ASR systems for Dutch. Our experiments use several bias mitigation strategies to address bias against different speaker groups. The findings reveal that averaged error rates, a standard in ASR research, alone is not sufficient and should be supplemented by other measures. The paper ends with recommendations for reporting ASR performance and bias to better represent a system's performance for diverse speaker groups, and overall system bias.

Figures

Figures reproduced from arXiv: 2507.05885 by the authors.

Figure 1
Figure 1. Bias estimated for the Read and HMI test sets using the difference measure (top panel) and relative difference (bottom panel) for both G2min and G2norm using the three conformer models and two Whisper models. values are higher for W s than the other models indicating larger WER variations even within speaker groups (and least for DT), W s is the only model that was not trained on CGN, leading to a training/test set … view at source ↗
Figure 2
Figure 2. Performance Measures: Avg ± Stdev (•), M d (▲), Rg:min (⋆) and max (■) for the individual speaker groups; (top): Read speech, (bottom): HMI speech, for all models. System Bias: The Overall Bias in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 46 canonical work pages

  1. [1]

    Large datasets and advances in deep learning have significantly improved the per- formance of speech technologies [2, 3]

    Introduction Automatic speech processing technologies enhance digital in- clusion with user-friendly interfaces, ensuring accessibility re- gardless of sight, mobility, or literacy [1]. Large datasets and advances in deep learning have significantly improved the per- formance of speech technologies [2, 3]. Despite improved per- formance observed for stand...

    work page

  2. [2]

    Background and Related Work 2.1. Diversity of Speech Human speech is highly variable, and the variability is caused by several factors: speaker’s physiology, socio-linguistic back- ground, and speech and language development or proficiency. The anatomical structure of the speech production organs and their changes due to aging impact the speech production...

    arXiv 2025

  3. [3]

    The Dutch Corpora We use the Corpus Gesproken Nederlands (CGN) [38], which consists of speech spoken by Dutch adult, native speakers

    Methodology 3.1. The Dutch Corpora We use the Corpus Gesproken Nederlands (CGN) [38], which consists of speech spoken by Dutch adult, native speakers. The type of speech data includes lecture recordings, broadcast news, and spontaneous conversations. The training data is∼430 h; the test sets consist of CGN read (Rd) broadcast news and Conver- sational Tel...

    work page

  4. [4]

    4.1), followed by bias measures evalua- tion (Sec

    Experimental Results In this section, we first evaluate our models in terms of recogni- tion performance (Sec. 4.1), followed by bias measures evalua- tion (Sec. 4.2) and recommendations (Sec. 4.3). 4.1. ASR System Performance Table 2 presents the WER results for the Dutch ASR system, trained on CGN (adult speech) and tested on various speaker groups. The...

    work page

  5. [5]

    In line with the potential pitfalls, there is a clear need for performance and bias measures to capture performance variation, and this paper gives recommendations on it

    Conclusion The error rate is a fundamental metric offering a holistic system evaluation, however, our granular, intersectional analysis of dif- ferent demographic speaker groups showed that error rate does not reflect the performance and bias within and across speaker groups well. In line with the potential pitfalls, there is a clear need for performance ...

    work page

  6. [6]

    The accent gap,

    D. Harwell, “The accent gap,” Washington Post, 2018

    work page 2018

  7. [7]

    Also the overall bias mea- sures capture the earlier findings that the mitigation approaches do not reduce bias despite improving performance

    This is due to Stdev being measured with respect to mean WER, while bias is measured with respect to the group with the minimum WER or the norm group. Also the overall bias mea- sures capture the earlier findings that the mitigation approaches do not reduce bias despite improving performance. G2min and G2norm do not show similar trends and hence, do not c...

    work page

  8. [8]

    Google’s speech recognition has a gender bias,

    R. Tatman, “Google’s speech recognition has a gender bias,” Lin- guistics and Data Science , 2016

    work page 2016

Show all 50 references
  1. [9]

    V oice in human-agent interaction: A survey,

    K. Seaborn, N. P. Miyake, P. Pennefather, and M. Otake- Matsuura, “V oice in human-agent interaction: A survey,” ACM Computing Surveys, vol. 54, no. 4, 2021

    2021

  2. [10]

    Speech recognition in our every- day life,

    ClickWorker, “Speech recognition in our every- day life,” https://www.clickworker.com/customer-blog/ speech-recognition-in-everyday-life/, 2021

    2021

  3. [11]

    A review of deep learning techniques for speech processing,

    A. Mehrish, N. Majumder, R. Bharadwaj, R. Mihalcea, and S. Po- ria, “A review of deep learning techniques for speech processing,” Information Fusion, vol. 99, p. 101869, 2023

    2023

  4. [12]

    SUPERB: Speech Processing Universal PER- formance Benchmark,

    S. wen Yang et al., “SUPERB: Speech Processing Universal PER- formance Benchmark,” in Proc. Interspeech , 2021, pp. 1194– 1198

    2021

  5. [13]

    To- wards inclusive automatic speech recognition,

    S. Feng, B. M. Halpern, O. Kudina, and O. Scharenborg, “To- wards inclusive automatic speech recognition,” Computer Speech & Language, vol. 84, p. 101567, 2024

    2024

  6. [14]

    A survey on bias and fairness in machine learning,

    N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, and A. Gal- styan, “A survey on bias and fairness in machine learning,” arXiv preprint arXiv: 1908.09635, 2019

    1908 arXiv

  7. [15]

    Racial disparities in automated speech recog- nition,

    A. Koenecke et al., “Racial disparities in automated speech recog- nition,” PNAS, vol. 117, no. 14, pp. 7684–7689, 2020

    2020

  8. [16]

    De-biasing “bias

    K. Lum, Y . Zhang, and A. Bower, “De-biasing “bias” measure- ment,” in ACM Conf. on Fairness, Accountability, and Trans- parency (F AccT), 2022, pp. 379–389

    2022

  9. [17]

    V oice recognition still has signifi- cant race and gender biases,

    J. P. Bajorek, “V oice recognition still has signifi- cant race and gender biases,” https://hbr.org/2019/05/ voice-recognition-still-has-significant-race-and-gender-biases

    2019

  10. [18]

    Speech recognition tech is yet another example of bias,

    C. L. Lloreda, “Speech recognition tech is yet another example of bias,” Washington Post, 2020

    2020

  11. [19]

    Toward Fairness in Speech Recognition: Dis- covery and mitigation of performance disparities,

    P. Dheram et al., “Toward Fairness in Speech Recognition: Dis- covery and mitigation of performance disparities,” in Proc. Inter- speech, 2022, pp. 1268–1272

    2022

  12. [20]

    An overview of noise-robust automatic speech recognition,

    J. Li, L. Deng, Y . Gong, and R. Haeb-Umbach, “An overview of noise-robust automatic speech recognition,” IEEE/ACM Transac- tions on Audio, Speech, and Language Processing , vol. 22, no. 4, pp. 745–777, 2014

    2014

  13. [21]

    Effects of talker dialect, gender & race on accuracy of bing speech and youtube automatic captions,

    R. Tatman and C. Kasten, “Effects of talker dialect, gender & race on accuracy of bing speech and youtube automatic captions,” in Proc. Interspeech, 2017, pp. 934–938

    2017

  14. [22]

    Disorders of communication: Dysarthria,

    P. Enderby, “Disorders of communication: Dysarthria,” Hand- book of clinical neurology / edited by P .J. Vinken and G.W. Bruyn, vol. 110, pp. 273–81, 01 2013

    2013

  15. [23]

    Quan- tifying bias in automatic speech recognition,

    S. Feng, O. Kudina, B. M. Halpern, and O. Scharenborg, “Quan- tifying bias in automatic speech recognition,” arXiv preprint arXiv:2103.15122, 2021

    2021 arXiv

  16. [24]

    The ordering of milestones in language development for children from 1 to 6 years of age,

    M. R. Luinge, W. J. Post, H. P. Wit, and S. M. Goorhuis-Brouwer, “The ordering of milestones in language development for children from 1 to 6 years of age,” Jour . of speech, lang., and hearing re- search (JSLHR), vol. 49, p. 923–940, 2006

    2006

  17. [25]

    The development of gen- dered speech in children: Insights from adult L1 and L2 percep- tions,

    P. Fung, J. Schertz, and E. K. Johnson, “The development of gen- dered speech in children: Insights from adult L1 and L2 percep- tions,” JASA Express Letters, vol. 1, no. 1, p. 014407, 2021

    2021

  18. [26]

    Whats special in a child’s larynx?

    M. Prakash and J. C. Johnny, “Whats special in a child’s larynx?” Jour . of Pharmacy & Bioallied Sciences , vol. 7, pp. S55 – S58, 2015

    2015

  19. [27]

    Acoustics of children’s speech: Developmental changes of temporal and spectral parame- ters,

    S. Lee, A. Potamianos, and S. Narayanan, “Acoustics of children’s speech: Developmental changes of temporal and spectral parame- ters,” The Jour . of the Acous. Soc. of America, vol. 105, pp. 1455– 1468, 1999

    1999

  20. [28]

    Male and female speech: a study of mean f0, f0 range, phonation type and speech rate in Parisian French and Ameri- can English speakers,

    E. P ´epiot, “Male and female speech: a study of mean f0, f0 range, phonation type and speech rate in Parisian French and Ameri- can English speakers,” inSpeech Prosody 7, Dublin, Ireland, May 2014, pp. 305–309

    2014

  21. [29]

    Differences in voice quality between men and women: Use of the long-term average spectrum (LTAS),

    E. Mendoza, N. Valencia, J. Mu ˜noz, and H. Trujillo, “Differences in voice quality between men and women: Use of the long-term average spectrum (LTAS),”Jour . of V oice, vol. 10, no. 1, pp. 59– 66, 1996

    1996

  22. [30]

    Both [5, 27] found speech type to impact ASR per- formance with read speech being favored over non-read speech

    found that non-native speakers with a non-tonal first lan- guage were better recognized than those with a tonal first lan- guage, indicating potential bias towards inherent linguistic dif- ferences. Both [5, 27] found speech type to impact ASR per- formance with read speech be...

  23. [31]

    Casual Conversations (CC)

    showing the impact of race and skin types. Final notes: Only a few studies consider intersectional de- mographic groups [5, 15, 26, 27, 33]. Apart from gender which is typically evaluated as binary (unfortunately), the number and type of groups for a particular factor varies a...

  24. [32]

    Studying language, culture, and society: Sociolinguistics or linguistic anthropology,

    J. J. Gumperz and J. Cook-Gumperz, “Studying language, culture, and society: Sociolinguistics or linguistic anthropology,” Jour . of Sociolinguistics, vol. 12, pp. 532–545, 2008

    2008

  25. [33]

    The production of “new

    J. E. Flege, “The production of “new” and “similar” phones in a foreign language: Evidence for the effect of equivalence classifi- cation,” Jour . of Phonetics, vol. 15, no. 1, pp. 47–65, 1987

    1987

  26. [34]

    Gender and Dialect Bias in YouTube’s Automatic Captions,

    R. Tatman, “Gender and Dialect Bias in YouTube’s Automatic Captions,” ACL Workshop on Ethics in Natural Lang. Process. , pp. 53–59, 2017

    2017

  27. [35]

    Gender representation in French broadcast corpora and its impact on ASR performance,

    M. Garnerin, S. Rossato, and L. Besacier, “Gender representation in French broadcast corpora and its impact on ASR performance,” Workshop on AI for Smart TV Content Production, Access and Delivery (AI4TV), pp. 3–9, 2019

    2019

  28. [36]

    Investigating the Impact of Gender Representation in ASR Training Data: a Case Study on Librispeech,

    ——, “Investigating the Impact of Gender Representation in ASR Training Data: a Case Study on Librispeech,” in Workshop on Gender Bias in Natural Lang. Process. , 2021, pp. 86–92

    2021

  29. [37]

    Seamless equal accuracy ratio for inclusive CTC speech recognition,

    H. Gao et al., “Seamless equal accuracy ratio for inclusive CTC speech recognition,” Speech Comm., vol. 136, pp. 76–83, 2022

    2022

  30. [38]

    Training and typological bias in ASR performance for world Englishes,

    Y . Chan et al., “Training and typological bias in ASR performance for world Englishes,” in Proc. Interspeech, 2022, pp. 1273–1277

    2022

  31. [39]

    Towards measuring fairness in speech recognition: Casual Conversations dataset transcriptions,

    C. Liu et al., “Towards measuring fairness in speech recognition: Casual Conversations dataset transcriptions,” in IEEE Int. Conf. on Acoustics, Speech and Signal Process. (ICASSP) , 2022, pp. 6162–6166

    2022

  32. [40]

    Model-based approach for measuring the fairness in ASR,

    Z. Liu, I. E. Veliche, and F. Peng, “Model-based approach for measuring the fairness in ASR,” in IEEE Int. Conf. on Acoustics, Speech and Signal Process. (ICASSP) , 2022, pp. 6532–6536

    2022

  33. [41]

    Using Data Augmentations and VTLN to Reduce Bias in Dutch End-to-End Speech Recognition Systems,

    T. Patel and O. Scharenborg, “Using Data Augmentations and VTLN to Reduce Bias in Dutch End-to-End Speech Recognition Systems,” arXiv preprint arXiv:2307.02009, 2023

    2023 arXiv

  34. [42]

    Mitigating bias against non-native accents,

    Y . Zhang, Y . Zhang, B. M. Halpern, T. Patel, and O. Scharenborg, “Mitigating bias against non-native accents,” inProc. Interspeech, 2022, pp. 3168–3172

    2022

  35. [43]

    Mitigating regional accent bias in asr systems,

    Z. Li, “Mitigating regional accent bias in asr systems,” in Masters Thesis. TU Delft, 2023

    2023

  36. [44]

    Compar- ing data augmentation and training techniques to reduce bias against non-native accents in hybrid speech recognition systems,

    Y . Zhang, Y . Zhang, T. Patel, and O. Scharenborg, “Compar- ing data augmentation and training techniques to reduce bias against non-native accents in hybrid speech recognition systems,” in Speech for Social Good Workshop, 2022

    2022

  37. [45]

    Exploring data augmentation in bias mitigation against non- native-accented speech,

    Y . Zhang, A. Herygers, T. B. Patel, Z. Yue, and O. Scharenborg, “Exploring data augmentation in bias mitigation against non- native-accented speech,” Workshop on Automatic Speech Recog- nition and Understanding (ASRU), pp. 1–8, 2023

    2023

  38. [46]

    The Spoken Dutch Corpus. Overview and First Evaluation,

    N. Oostdijk, “The Spoken Dutch Corpus. Overview and First Evaluation,” in Proc. of the Lang. Resources and Eval. (LREC) . Athens, Greece, 2000, pp. 887–894

    2000

  39. [47]

    Jasmin-CGN: Extension of the spoken Dutch corpus with speech of elderly people, children and non-natives in the human-machine interaction modality,

    C. Cucchiarini, H. V . Hamme, O. van Herwijnen, and F. Smits, “Jasmin-CGN: Extension of the spoken Dutch corpus with speech of elderly people, children and non-natives in the human-machine interaction modality,” in Proc. of the Lang. Resources and Eval. (LREC). Genova, Italy:[...

    2006

  40. [48]

    Conformer: Convolution-augmented transformer for speech recognition,

    A. Gulati et al., “Conformer: Convolution-augmented transformer for speech recognition,” in Proc. Interspeech, Shanghai, China, 2020, pp. 5036–5040

    2020

  41. [49]

    Robust speech recognition via large-scale weak su- pervision,

    A. Radford, J. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak su- pervision,” in Int. Conf. on Machine Learning. PMLR, 2023, pp. 28 492–28 518

    2023

  42. [50]

    ESPnet: End-to-End speech processing toolkit,

    S. Watanabe et al. , “ESPnet: End-to-End speech processing toolkit,” in Proc. Interspeech, 2018, pp. 2207–2211

    2018

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.