SHEET: A Multi-purpose Open-source Speech Human Evaluation Estimation Toolkit

Erica Cooper; Tomoki Toda; Wen-Chin Huang

REVIEW 3 major objections 5 minor 1 cited by

SHEET: A Multi-purpose Open-source Speech Human Evaluation Estimation Toolkit

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SHEET, an open-source toolkit for MOS prediction, retrains SSL-MOS across 22 self-supervised speech models and shows that larger and multilingual backbones beat the original predictor.

desk verdict Solid engineering contribution with a useful 22-backbone SSL-MOS benchmark; the headline 'optimal SSL model' claim is real but only descriptive, and the NISQA comparison is partly confounded by in-domain training. read the letter →

arxiv 2505.15061 v1 pith:C7XR2HGU submitted 2025-05-21 cs.SD eess.AS

Wen-Chin Huang , Erica Cooper , Tomoki Toda This is my paper · ORCID

classification cs.SDeess.AS

keywords speechqualityassessmentMOSpredictionself-supervisedlearningSSL-MOSopen-sourcetoolkitBVCCNISQAobjectiveevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces SHEET, an open-source toolkit with training and evaluation recipes for speech quality prediction models that estimate human mean opinion scores (MOS). To demonstrate the toolkit, the authors retrain SSL-MOS, a widely used MOS predictor built on a self-supervised speech model, using 22 different SSL backbones on the BVCC and NISQA datasets. They report that the resulting predictors with HuBERT large and data2vec large outperform the original SSL-MOS and UTMOS on BVCC, and that WavLM large and XLS-R 1b surpass all five comparing systems on NISQA. The paper treats these results as evidence that SHEET can reproduce and extend SSL-MOS training, and that the choice of SSL backbone matters as much as the prediction architecture.

What carries the argument

The central object is SSL-MOS: an encoder-decoder in which a speech self-supervised model produces frame-wise hidden states, a decoder maps those states to frame-wise quality scores, time pooling collapses them to one predicted score, and the training objective is the L1 distance $\|y - \hat{y}\|_1$ between that pooled score and the human MOS. SHEET packages this recipe in a library-plus-recipe structure with per-dataset configuration files, automatic early stopping, and evaluation across multiple test sets, and it adds features previously reported to help MOS prediction: range clipping, repetitive padding, clipped L1 loss, listener modeling, contrastive loss, multi-dataset training, and k-nearest-neighbor retrieval augmentation. The experiment holds this recipe fixed and swaps only the SSL encoder, drawing the 22 encoders from a public collection of speech SSL models; output from the last layer of each encoder feeds the decoder.

What would settle it

A paired experiment in which the same SHEET training configuration is run with only the SSL backbone changed between wav2vec 2.0 base and WavLM large would settle the cause: if the performance gap disappears or reverses, the backbone is not the driver.

Watch

Extended reading notes

Core claim

SHEET's central demonstration is that SSL-MOS, the model that takes frame-wise hidden states from a speech SSL encoder and feeds them to a decoder trained with an L1 loss after time pooling, is not tied to any particular SSL backbone. Re-running the recipe over 22 SSL models, the authors find the best backbones are data2vec large and HuBERT large on BVCC, and WavLM large and XLS-R 1b on NISQA, with the latter two beating all five compared systems on the NISQA test sets. The best BVCC results are comparable to but not better than RAMP+, the current state of the art, and reproducing RAMP+ is named as future work. The paper also observes that SSL-MOS is sensitive to the backbone choice, that large models usually but not always beat base models, and that cross-lingual pretraining helps on a German live-call test set.

Load-bearing premise

The claim that larger or multilingual SSL backbones cause the quality-prediction gains depends on the assumption that the extra training features and hyperparameters built into SHEET are not the real cause; the authors explicitly leave that possibility open.

Editorial extensions

If this is right

MOS predictors built on self-supervised speech models should be benchmarked per dataset, since the best backbone on BVCC differs from the best on NISQA.
The provided checkpoints give speech-generation and enhancement labs an off-the-shelf alternative to the original SSL-MOS that is stronger on at least two common test suites.
On NISQA, WavLM large and XLS-R 1b establish competitive baselines for future SSQA systems to beat.
BVCC remains a case where RAMP+ leads, so reproducing RAMP+ inside SHEET is the stated next step.
SSQA models trained on only one dataset generalize poorly out of domain, so a general-purpose MOS predictor remains an open problem.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A controlled ablation keeping all SHEET training features fixed and swapping only the SSL backbone would isolate whether the reported gains come from the backbone or from SHEET's extra training tricks; the paper does not run that ablation.
If the backbone is indeed the driver, then published results that simply say 'SSL-MOS' should name the SSL encoder, because the same architecture with different encoders can rank differently across datasets.
A natural next experiment is to take the WavLM large and XLS-R 1b checkpoints to newer MOS challenge test sets outside BVCC and NISQA to check whether the advantage persists.
Using SHEET's multi-dataset training feature, combining BVCC and NISQA training data may improve out-of-domain generalization, a direction the paper flags but does not test.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The toolkit is the real contribution here. SHEET gives the SSQA community something it genuinely lacked: a single codebase with training recipes, evaluation scripts, and pre-trained models across multiple datasets, with torch.hub and HuggingFace access. That is a useful engineering artifact, and the authors deserve credit for shipping it. The empirical sweep is also a step beyond the original SSL-MOS paper: 22 SSL backbones on BVCC and NISQA, with checkpoints for the comparing systems taken from official sources. The finding that WavLM large and XLS-R 1b work well on NISQA is new and plausibly useful for practitioners.

The soft spots are real but not fatal, and the paper is largely honest about them. The biggest issue is the NISQA comparison. The new SSL-MOS rows were trained on the NISQA training split, while SSL-MOS, UTMOS, and RAMP+ were evaluated zero-shot on NISQA after training only on BVCC. Saying these new variants 'surpassed all five comparing systems' overstates the case: the gain could come from in-domain training rather than backbone choice. The paper does not flag this asymmetry in Section 3.3, and the abstract's wording ('identified the optimal speech SSL model') inherits the ambiguity. On BVCC, the comparison to the original SSL-MOS is cleaner but still not strictly controlled, since the new variants also use range clipping, repetitive padding, and other recipe features. The authors actually concede this in Section 3.3, which counts in their favor.

A second, more minor issue: the headline ranking is selected on the test set. There are no error bars or multiple-comparison correction, and each configuration appears to be a single run. For a leaderboard meant to guide backbone choice, that is acceptable as descriptive guidance, but the paper should say so explicitly and avoid causal language like 'the optimal SSL model.' The conclusion section does partially reframe it as identifying optimal models for each dataset, which helps.

The reader's stress-test concern about the NISQA confound holds up on reading. I would push back, however, on any implication that the toolkit is weak: the central reusable artifact and the 22-backbone comparison are independently valuable regardless of the causal attribution. The authors also tried to reproduce RAMP and transparently reported that they could not, which is the right behavior.

Verdict: worth a serious referee. The paper should be accepted after revision if the authors either add held-out selection and uncertainty estimates, or explicitly frame the leaderboard as descriptive and fix the NISQA comparison wording. I would bring this to a reading group for the practical benchmark data, and I would cite it when comparing SSL backbones for MOS prediction.

Referee Report

3 major / 5 minor

Summary. The paper introduces SHEET, an open-source toolkit for subjective speech quality assessment (SSQA) that provides training and evaluation recipes for multiple datasets, supports a wide range of SSL backbones and modeling features, and ships pre-trained models via Torch Hub and HuggingFace. To demonstrate the toolkit, the authors re-evaluate SSL-MOS across 22 SSL models on the BVCC and NISQA datasets, report that different backbones are optimal for each dataset (e.g., data2vec large and HuBERT large on BVCC; WavLM large and XLS-R 1b on NISQA), and claim that these variants surpass the original SSL-MOS and are comparable to or better than several existing systems.

Significance. If its empirical claims are supported, SHEET would be a genuinely useful community resource: it is one of the first SSQA toolkits to combine model training, multi-dataset evaluation, and easy inference in one package, and it provides a systematic, transparent comparison of 22 SSL backbones. The paper ships reproducible training recipes, pre-trained checkpoints, and a public table of results, which are concrete strengths. However, the headline empirical conclusions are currently weakened by two confounds (in-domain versus zero-shot comparison on NISQA, and recipe differences versus backbone differences on BVCC) and by selecting the best model on the test sets used for the final comparison. These issues affect the central claim that an 'optimal speech SSL model' was identified and that the gains over SSL-MOS are attributable to the backbone choice.

major comments (3)

[Section 3.3, Table 2] The NISQA comparison is confounded by training data. The new SSL-MOS variants were trained on the NISQA TRAIN SIM and NISQA TRAIN LIVE splits (Section 3.1.1), while SSL-MOS, UTMOS, and RAMP+ were trained only on BVCC and are evaluated on NISQA zero-shot (Section 3.1.3). Therefore the claim that 'SSL-MOS models with WavLM large and XLS-R 1b surpassed all five comparing systems' may reflect in-domain training rather than the SSL backbone. Please retrain the comparing systems on the same NISQA training data, evaluate the new variants zero-shot (trained on BVCC only) on NISQA, or explicitly reframe the result as an in-domain versus zero-shot comparison and soften the 'surpassed' claim accordingly.
[Section 3.3] The BVCC gains over the original SSL-MOS are not attributable to the SSL backbone alone. The authors concede that the improvement 'could be because of our hyperparameter choices, as well as the use of the features described in Section 2.3' (range clipping, repetitive padding, clipped L1 loss, listener modeling). These features are part of SHEET but not of the original SSL-MOS, so the comparison isolates neither the backbone nor the recipe. Please include an ablation (e.g., the new backbones with the original SSL-MOS training settings, or the original wav2vec 2.0 base with SHEET's full recipe) to separate the backbone effect, or limit the claim to the full SHEET pipeline.
[Section 3.2, Table 2] The optimal SSL model is selected based on performance on the same test sets used for the final comparison, with no held-out validation set and no correction for testing 22 models. This selection-on-the-test-set procedure inflates the apparent advantage of the chosen backbones and makes the word 'optimal' statistically unsupported. Please perform model selection on the validation splits (e.g., NISQA VAL SIM/LIVE or a held-out portion of BVCC) and then evaluate on the test sets, or explicitly discuss this limitation in the text.

minor comments (5)

[Section 3.1.2] The sentence 'we refer authors to the S3PRL codebase' should be 'we refer readers to the S3PRL codebase'.
[References] In references [2] and [17], 'Mller' appears to be a rendering issue for 'Möller'; please fix the encoding. In reference [35], 'NeruIPS' should read 'NeurIPS'.
[Table 2] In the NISQA FOR column, some entries are missing spaces (e.g., '0.140 0.9450.177' for WavLM large); please ensure consistent formatting so that the MSE and LCC values are clearly separated.
[Section 2.5] The torch.hub example contains an anonymized repository placeholder ('<anonymized>/sheet:v0.1.0'); the final version should include the actual repository and tag.
[Section 3.1.2] The paper does not state whether each configuration was trained with multiple random seeds or report variance across runs; since the comparisons are based on single numbers, a brief statement on seed variability (or its absence) would aid interpretation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central comparisons are empirical, against official external checkpoints, and no claim reduces to a fit or to the authors' own definitions.

full rationale

The paper's load-bearing claims are empirical rankings from Table 2, produced by training SSL-MOS variants on public BVCC and NISQA training splits and evaluating on official test sets. The comparing systems (UTMOS, RAMP+, NISQA, DNSMOS, and the original SSL-MOS checkpoint) are official released checkpoints, so the comparisons are not definitional equivalences. The 'optimal SSL model' is the best-performing row in a fixed experimental protocol, not a parameter fitted to produce that ranking. The acknowledged confounds — NISQA-trained SSL-MOS rows having in-domain training while SSL-MOS/UTMOS/RAMP+ are BVCC-only, and BVCC improvements possibly stemming from SHEET's additional training features rather than the SSL backbone — are interpretability and experimental-design limitations, not circular reductions: no equation or construction forces the reported numbers. Self-citations (SSL-MOS, VoiceMOS) provide the framework and baseline but are not used as an unverified authority to forbid alternatives or to justify the outcome. Because the central derivation is self-contained against external benchmarks, the circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper is an empirical toolkit and benchmark paper. It introduces no new scientific entities, but it relies on standard assumptions about human labels, SSL representations, and the SSL-MOS architecture. The free parameters are the test-selected backbone choice and the shared training hyperparameters.

free parameters (2)

Optimal SSL backbone per dataset = BVCC: data2vec large; NISQA: WavLM large
Chosen as the best-performing model on the test sets among 22 candidates; this is a model selection on test data, so reported scores are optimistic.
Shared training hyperparameters = batch 16; SGD lr 0.001; momentum 0.9; max 100k steps; patience 2k
Hand-chosen and shared across all SSL-MOS variants; no sensitivity analysis, so rankings could shift under different settings.

assumptions (4)

domain assumption Human MOS ratings in BVCC and NISQA are treated as ground truth labels.
The toolkit and experiments optimize and evaluate against these listener scores; if labels are noisy or biased, all results inherit that. Invoked in Section 3.1.1.
domain assumption SSL representations from S3PRL are meaningful features for MOS prediction.
The SSL-MOS framework and the comparison assume the last-layer outputs of 22 SSL models carry quality-relevant information. Invoked in Sections 2.3 and 3.1.2.
domain assumption The SSL-MOS architecture (SSL encoder plus decoder, L1 loss, time pooling) is a valid SSQA model.
This is the baseline model used throughout; improvements are measured within this architecture. Defined in Section 2.3, Eq. (1).
domain assumption Evaluation metrics (Sys MSE, Sys SRCC, Utt MSE, Utt LCC) reflect prediction quality.
These metrics are standard in the VoiceMOS and NISQA communities and are used without justification in the paper. Invoked in Section 3.1.4.

how reviews work

0 comments

Cite this review

Pith. "Pith review of SHEET: A Multi-purpose Open-source Speech Human Evaluation Estimation Toolkit." pith.science (2026). https://pith.science/paper/C7XR2HGU

@misc{pith2026250515061,
  author       = {Pith},
  title        = {Pith review of: SHEET: A Multi-purpose Open-source Speech Human Evaluation Estimation Toolkit},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C7XR2HGU}},
  note         = {Machine review of arXiv:2505.15061}
}

read the original abstract

We introduce SHEET, a multi-purpose open-source toolkit designed to accelerate subjective speech quality assessment (SSQA) research. SHEET stands for the Speech Human Evaluation Estimation Toolkit, which focuses on data-driven deep neural network-based models trained to predict human-labeled quality scores of speech samples. SHEET provides comprehensive training and evaluation scripts, multi-dataset and multi-model support, as well as pre-trained models accessible via Torch Hub and HuggingFace Spaces. To demonstrate its capabilities, we re-evaluated SSL-MOS, a speech self-supervised learning (SSL)-based SSQA model widely used in recent scientific papers, on an extensive list of speech SSL models. Experiments were conducted on two representative SSQA datasets named BVCC and NISQA, and we identified the optimal speech SSL model, whose performance surpassed the original SSL-MOS implementation and was comparable to state-of-the-art methods.

Figures

Figures reproduced from arXiv: 2505.15061 by the authors.

**Figure 1.** Illustration of SSL-MOS. result surpassed the original SSL-MOS model and is comparable with the state-of-the-art method. Such results underscore the usefulness of the proposed toolkit and its adaptability to different datasets. 2. SHEET design and supported models 2.1. Overview The structure of SHEET follows the design of popular speech processing toolkits like Kaldi [21] and ESPnet [22]. Specifically, in such a de… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Improving Speech Enhancement with Multi-Metric Supervision from Learned Quality Assessment
cs.SD 2025-06 conditional novelty 6.0 of 10

A multi-metric learned quality model (Uni-VERSA-Ext) is used as a differentiable training loss for speech enhancement, with a regularization term to prevent adversarial exploitation.

Reference graph

Works this paper leans on

44 extracted references · 38 canonical work pages · cited by 1 Pith paper

[1]

<anonymized>/sheet:v0.1.0

Introduction Speech quality assessment (SQA) refers to the task of evaluating the quality of speech signals [1–3], and is an essential compo- nent to various applications, including telecommunications and speech generation tasks, from text-to-speech (TTS), voice con- version (VC) to speech enhancement. The gold standard for evaluating speech signals is to...

arXiv 2022
[2]

Speech Quality Estimation: Models and Trends,

S. Mller, W.-Y. Chan, N. Ct, T. H. Falk, A. Raake, and M. Wl- termann, “Speech Quality Estimation: Models and Trends,”IEEE Signal Processing Magazine, vol. 28, no. 6, pp. 18–28, 2011

work page 2011
[3]

Experiments To demonstrate the capabilities ofSHEET, in this section, we conduct an extensive experiment to re-evaluate the effectiveness of existing SSL models in the SSL-MOS framework. 3.1. Experimental setting 3.1.1. Dataset The following two datasets are used in the experiments. •TheBVCCdataset [20] was used in the main track of VMC

work page
[4]

Conclusions In this work, we introducedSHEET, an open-source toolkit designed to facilitate research in SSQA. By addressing the limi- tations of existing toolkits,SHEETprovides a standardized and flexible framework for conducting SSQA experiments, evalu- ating existing models across multiple datasets, or simply em- ploying the provided pre-trained models....

work page
[5]

Acknowledgements This work was partly supported by JSPS KAKENHI Grant Number 25K00143 and JST AIP Acceleration Research JP- MJCR25U5, Japan

work page
[6]

Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs,

A. W. Rix, J. G. Beerends, M. P. Hollier, and A. P. Hekstra, “Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs,” inProc. ICASSP, vol. 2, 2001, pp. 749–752

work page 2001
[7]

Speech quality assessment,

P. C. Loizou, “Speech quality assessment,” inMultimedia Analysis, Processing and Communications, W. Lin, D. Tao, J. Kacprzyk, Z. Li, E. Izquierdo, and H. Wang, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2011, pp. 623–654

work page 2011
[8]

MOSNet: Deep Learning-Based Objective Assessment for Voice Conversion,

C.-C. Lo, S.-W. Fu, W.-C. Huang, X. Wang, J. Yamagishi, Y. Tsao, and H.-M. Wang, “MOSNet: Deep Learning-Based Objective Assessment for Voice Conversion,” inProc. Interspeech, 2019, pp. 1541–1545

work page 2019

Show all 44 references

[9]

A review on subjective and objective evaluation of synthetic speech,

E. Cooper, W.-C. Huang, Y. Tsao, H.-M. Wang, T. Toda, and J. Yamagishi, “A review on subjective and objective evaluation of synthetic speech,”Acoustical Science and Technology, vol. 45, no. 4, pp. 161–183, 2024

2024
[10]

An Al- gorithm for Intelligibility Prediction of TimeFrequency Weighted Noisy Speech,

C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, “An Al- gorithm for Intelligibility Prediction of TimeFrequency Weighted Noisy Speech,”IEEE/ACM TASLP, vol. 19, no. 7, pp. 2125–2136, 2011

2011
[11]

Mel-cepstral distance measure for objective speech quality assessment,

R. Kubichek, “Mel-cepstral distance measure for objective speech quality assessment,” inProceedings of IEEE Pacific Rim Con- ference on Communications Computers and Signal Processing, vol. 1, 1993, pp. 125–128 vol.1

1993
[12]

The Voicemos Challenge 2024: Beyond Speech Quality Prediction,

W.-C. Huang, S.-W. Fu, E. Cooper, R. Zezario, T. Toda, H.-M. Wang, J. Yamagishi, and Y. Tsao, “The Voicemos Challenge 2024: Beyond Speech Quality Prediction,” inProc. SLT, 2024

2024
[13]

AutoMOS: Learning a non- intrusive assessor of naturalness-of-speech,

B. Patton, Y. Agiomyrgiannakis, M. Terry, K. Wilson, R. A. Saurous, and D. Sculley, “AutoMOS: Learning a non- intrusive assessor of naturalness-of-speech,”arXiv preprint arXiv:1611.09207, 2016

2016 arXiv
[14]

Torchaudio-squim: Reference-less speech quality and intelligibility measures in torchaudio,

A. Kumar, K. Tan, Z. Ni, P. Manocha, X. Zhang, E. Henderson, and B. Xu, “Torchaudio-squim: Reference-less speech quality and intelligibility measures in torchaudio,” inProc. ICASSP, 2023, pp. 1–5

2023
[15]

DNSMOS: A Non- Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors,

C. K. A. Reddy, V. Gopal, and R. Cutler, “DNSMOS: A Non- Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors,” inProc. ICASSP, 2021, pp. 6493–6497

2021
[16]

The VoiceMOS Challenge 2022,

W.-C. Huang, E. Cooper, Y. Tsao, H.-M. Wang, T. Toda, and J. Ya- magishi, “The VoiceMOS Challenge 2022,” inProc. Interspeech, 2022, pp. 4536–4540

2022
[17]

The Voicemos Challenge 2023: Zero-Shot Subjective Speech Quality Prediction for Multiple Domains,

E. Cooper, W.-C. Huang, Y. Tsao, H.-M. Wang, T. Toda, and J. Ya- magishi, “The Voicemos Challenge 2023: Zero-Shot Subjective Speech Quality Prediction for Multiple Domains,” inProc. ASRU, 2023, pp. 1–7

2023
[18]

Generaliza- tion ability of MOS prediction networks,

E. Cooper, W.-C. Huang, T. Toda, and J. Yamagishi, “Generaliza- tion ability of MOS prediction networks,” inProc. ICASSP, 2022, pp. 8442–8446

2022
[19]

UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022,

T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari, “UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022,” inProc. Interspeech, 2022, pp. 4521–4525

2022
[20]

How do voices from past speech synthesis challenges compare today?

E. Cooper and J. Yamagishi, “How do voices from past speech synthesis challenges compare today?” inProc. 11th ISCA Speech Synthesis Workshop (SSW 11), 2021, pp. 183–188

2021
[21]

SpeechBERTScore: Reference-Aware Automatic Evaluation of Speech Generation Leveraging NLP Evaluation Metrics,

T. Saeki, S. Maiti, S. Takamichi, S. Watanabe, and H. Saruwatari, “SpeechBERTScore: Reference-Aware Automatic Evaluation of Speech Generation Leveraging NLP Evaluation Metrics,” inProc. Interspeech, 2024, pp. 4943–4947

2024
[22]

Versa: A versatile evaluation toolkit for speech, audio, and music,

J. Shi, H. jin Shim, J. Tian, S. Arora, H. Wu, D. Petermann, J. Q. Yip, Y. Zhang, Y. Tang, W. Zhang, D. S. Alharthi, Y. Huang, K. Saito, J. Han, Y. Zhao, C. Donahue, and S. Watanabe, “Versa: A versatile evaluation toolkit for speech, audio, and music,” 2024. [Online]. Availabl...

2024 arXiv
[23]

NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets,

G. Mittag, B. Naderi, A. Chehadi, and S. Mller, “NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets,” inProc. Interspeech, 2021, pp. 2127–2131

2021
[24]

Self-supervised speech representation learning: A review,

A. Mohamed, H.-y. Lee, L. Borgholt, J. D. Havtorn, J. Edin, C. Igel, K. Kirchhoff, S.-W. Li, K. Livescu, L. Maaløeet al., “Self-supervised speech representation learning: A review,”IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1179–1210, 2022

2022
[25]

RAMP: Retrieval- Augmented MOS Prediction via Confidence-based Dynamic Weighting,

H. Wang, S. Zhao, X. Zheng, and Y. Qin, “RAMP: Retrieval- Augmented MOS Prediction via Confidence-based Dynamic Weighting,” inProc. Interspeech, 2023, pp. 1095–1099

2023
[26]

The Singing Voice Conversion Challenge 2023,

W.-C. Huang, L. P. Violeta, S. Liu, J. Shi, and T. Toda, “The Singing Voice Conversion Challenge 2023,” inProc. ASRU, 2023, pp. 1–8

2023
[27]

The Kaldi Speech Recognition Toolkit,

D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarzet al., “The Kaldi Speech Recognition Toolkit,” inProc. ASRU, 2011

2011
[28]

ESPnet: End-to-End Speech Processing Toolkit,

S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y. Unno, N. E. Y. Soplin, J. Heymann, M. Wiesner, N. Chen, A. Renduch- intala, and T. Ochiai, “ESPnet: End-to-End Speech Processing Toolkit,” inProc. Interspeech, 2018, pp. 2207–2211

2018
[29]

An End- To-End Non-Intrusive Model for Subjective and Objective Real- World Speech Assessment Using a Multi-Task Framework,

Z. Zhang, P. Vyas, X. Dong, and D. S. Williamson, “An End- To-End Non-Intrusive Model for Subjective and Objective Real- World Speech Assessment Using a Multi-Task Framework,” in Proc. ICASSP, 2021, pp. 316–320

2021
[30]

Espnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to- Speech Toolkit,

T. Hayashi, R. Yamamoto, K. Inoue, T. Yoshimura, S. Watanabe, T. Toda, K. Takeda, Y. Zhang, and X. Tan, “Espnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to- Speech Toolkit,” inProc. ICASSP, 2020, pp. 7654–7658

2020
[31]

Utilizing Self-Supervised Representations for MOS Prediction,

W.-C. Tseng, C. yu Huang, W.-T. Kao, Y. Y. Lin, and H. yi Lee, “Utilizing Self-Supervised Representations for MOS Prediction,” inProc. Interspeech, 2021, pp. 2781–2785

2021
[32]

On the NISQA dataset, on average, the WavLM large model [33] and the XLS-R 1b model [34] achieved the best and second best scores on the Sys MSE and Sys SRCC metrics, respectively

and the HuBERT large model achieved the best Sys MSE and Sys SRCC scores, respectively. On the NISQA dataset, on average, the WavLM large model [33] and the XLS-R 1b model [34] achieved the best and second best scores on the Sys MSE and Sys SRCC metrics, respectively. The fact...

2024
[33]

MB- NET: MOS Prediction for Synthesized Speech with Mean-Bias Network,

Y. Leng, X. Tan, S. Zhao, F. Soong, X.-Y. Li, and T. Qin, “MB- NET: MOS Prediction for Synthesized Speech with Mean-Bias Network,” inProc. ICASSP, 2021, pp. 391–395

2021
[34]

LDNet: uni- fied listener dependent modeling in MOS prediction for synthetic speech,

W.-C. Huang, E. Cooper, J. Yamagishi, and T. Toda, “LDNet: uni- fied listener dependent modeling in MOS prediction for synthetic speech,” inProc. ICASSP, 2022, pp. 896–900

2022
[35]

Alignnet: Learning dataset score align- ment functions to enable better training of speech quality estima- tors,

J. Pieper and S. Voran, “Alignnet: Learning dataset score align- ment functions to enable better training of speech quality estima- tors,” inProc. Interspeech, 2024, pp. 82–86

2024
[36]

VoxSim: A perceptual voice similarity dataset,

J. Ahn, Y. Kim, Y. Choi, D. Kwak, J.-H. Kim, S. Mun, and J. S. Chung, “VoxSim: A perceptual voice similarity dataset,” inProc. Interspeech, 2024, pp. 2580–2584

2024
[37]

A Large- Scale Evaluation of Speech Foundation Models,

S.-w. Yang, H.-J. Chang, Z. Huang, A. T. Liu, C.-I. Lai, H. Wu, J. Shi, X. Chang, H.-S. Tsai, W.-C. Huang, T.-h. Feng, P.-H. Chi, Y. Y. Lin, Y.-S. Chuang, T.-H. Huang, W.-C. Tseng, K. Lakhotia, S.-W. Li, A. Mohamed, S. Watanabe, and H.-y. Lee, “A Large- Scale Evaluation of Spe...

2024
[38]

data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language,

A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli, “data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language,” inProc. ICML, 2022, pp. 1298– 1312

2022
[39]

WavLM: Large-scale self-supervised pre-training for full stack speech processing,

S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “WavLM: Large-scale self-supervised pre-training for full stack speech processing,”IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505– 1518, 2022

2022
[40]

XLS-R: Self-supervised Cross-lingual Speech Rep- resentation Learning at Scale,

A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y. Saraf, J. Pino, A. Baevski, A. Conneau, and M. Auli, “XLS-R: Self-supervised Cross-lingual Speech Rep- resentation Learning at Scale,” inProc. Interspeech, 2022, pp. 2278–2282

2022
[41]

wav2vec 2.0: A framework for self-supervised learning of speech representations,

A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” inProc. NeruIPS, 2020

2020
[43]

PAM: Prompting Audio- Language Models for Audio Quality Assessment,

S. Deshmukh, D. Alharthi, B. Elizalde, H. Gamper, M. Al Is- mail, R. Singh, B. Raj, and H. Wang, “PAM: Prompting Audio- Language Models for Audio Quality Assessment,” inProc. Inter- speech, 2024, pp. 3320–3324

2024
[44]

Audio Large Language Models Can Be Descriptive Speech Quality Evaluators,

C. Chen, Y. Hu, S. Wang, H. Wang, Z. Chen, C. Zhang, C.- H. H. Yang, and E. Chng, “Audio Large Language Models Can Be Descriptive Speech Quality Evaluators,” inProc. ICLR, 2025

2025
[2022]

Each sample was rated by 8 distinct listeners

It contains English speech samples in 16 kHz and their MOS ratings from 187 different TTS and VC systems, which mainly come from past years of the Blizzard Challenges (BC) and Voice Conversion Challenges (VCC), as well as published samples from ESPnet-TTS [30]. Each sample was...

2000

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

<anonymized>/sheet:v0.1.0

Introduction Speech quality assessment (SQA) refers to the task of evaluating the quality of speech signals [1–3], and is an essential compo- nent to various applications, including telecommunications and speech generation tasks, from text-to-speech (TTS), voice con- version (VC) to speech enhancement. The gold standard for evaluating speech signals is to...

arXiv 2022

[2] [2]

Speech Quality Estimation: Models and Trends,

S. Mller, W.-Y. Chan, N. Ct, T. H. Falk, A. Raake, and M. Wl- termann, “Speech Quality Estimation: Models and Trends,”IEEE Signal Processing Magazine, vol. 28, no. 6, pp. 18–28, 2011

work page 2011

[3] [3]

Experiments To demonstrate the capabilities ofSHEET, in this section, we conduct an extensive experiment to re-evaluate the effectiveness of existing SSL models in the SSL-MOS framework. 3.1. Experimental setting 3.1.1. Dataset The following two datasets are used in the experiments. •TheBVCCdataset [20] was used in the main track of VMC

work page

[4] [4]

Conclusions In this work, we introducedSHEET, an open-source toolkit designed to facilitate research in SSQA. By addressing the limi- tations of existing toolkits,SHEETprovides a standardized and flexible framework for conducting SSQA experiments, evalu- ating existing models across multiple datasets, or simply em- ploying the provided pre-trained models....

work page

[5] [5]

Acknowledgements This work was partly supported by JSPS KAKENHI Grant Number 25K00143 and JST AIP Acceleration Research JP- MJCR25U5, Japan

work page

[6] [6]

Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs,

A. W. Rix, J. G. Beerends, M. P. Hollier, and A. P. Hekstra, “Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs,” inProc. ICASSP, vol. 2, 2001, pp. 749–752

work page 2001

[7] [7]

Speech quality assessment,

P. C. Loizou, “Speech quality assessment,” inMultimedia Analysis, Processing and Communications, W. Lin, D. Tao, J. Kacprzyk, Z. Li, E. Izquierdo, and H. Wang, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2011, pp. 623–654

work page 2011

[8] [8]

MOSNet: Deep Learning-Based Objective Assessment for Voice Conversion,

C.-C. Lo, S.-W. Fu, W.-C. Huang, X. Wang, J. Yamagishi, Y. Tsao, and H.-M. Wang, “MOSNet: Deep Learning-Based Objective Assessment for Voice Conversion,” inProc. Interspeech, 2019, pp. 1541–1545

work page 2019