REVIEW 2 major objections 7 minor 25 references
Simple text cleanup beats model fine-tuning on messy neurological Spanish speech, and a new wild corpus makes that gap visible.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-12 04:07 UTC pith:E4EODSWQ
load-bearing objection Usable multi-pathology Spanish in-the-wild ASR benchmark; PP beats the FT recipes they actually ran, and that result is real for those setups. the 2 major comments →
S-DiverSe: Spanish Diverse Speech
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
On the new in-the-wild Spanish neurological corpus S-DiverSe, rule-based text post-processing that removes repeated characters, words, and phrases lowers word error rate more reliably than full or LoRA fine-tuning of Whisper or Voxtral, which improve in-domain TORGO and NeuroVoz scores but fail to transfer and can push error rates above 100 percent.
What carries the argument
S-DiverSe itself—the 3.2-hour, 444-segment evaluation set with pathology, sex, and intelligibility labels—plus the direct head-to-head of the three-step heuristic post-processor against the four fine-tuning recipes (full, full-LoRA, encoder-only, encoder-LoRA) trained on TORGO, NeuroVoz, and Common Voice combinations.
Load-bearing premise
The particular fine-tuning recipes, learning rates, epochs, and data mixes the authors tried are representative enough of what parameter adaptation can do, so that post-processing's win can be blamed on a real domain gap rather than under-tuned adaptation.
What would settle it
A carefully tuned or larger-scale fine-tuning run on the same or expanded pathological Spanish data that beats the post-processing-only baseline on the full S-DiverSe set without collapsing on TORGO or NeuroVoz.
If this is right
- S-DiverSe becomes a public Spanish evaluation benchmark that exposes where current ASR systems fail on uncontrolled dysarthric speech.
- Practitioners should prefer lightweight text cleanup over fine-tuning when the target is wild neurological Spanish outside the training distribution.
- Future data collection must deliberately cover more stroke and female speakers, because the present wild-sourced mix is male- and ALS-heavy.
- Clean read speech and hospital monologues cannot substitute for in-the-wild pathological Spanish when measuring real generalization.
Where Pith is reading between the lines
- If the domain gap is mainly acoustic and stylistic rather than linguistic, multi-condition data augmentation or self-supervised adaptation on unlabeled wild pathological audio may succeed where supervised fine-tuning failed.
- The same post-processing-versus-fine-tuning comparison could be run on other low-resource languages that currently lack wild neurological corpora, testing whether the robustness pattern is language-independent.
- Commercial black-box systems that already show lower error on S-DiverSe may already contain the kind of diverse training data the open models lack; releasing even partial metadata about that data would accelerate open progress.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces S-DiverSe, a 3.2-hour in-the-wild Spanish ASR evaluation corpus of 444 manually transcribed segments from 22 speakers with ALS, Parkinson’s disease, and stroke, with metadata for sex, condition, and intelligibility. It reports baselines for Whisper-large-v3, Voxtral-Mini, omniASR CTC 1B v2, and ElevenLabs Scribe v2, and compares heuristic text post-processing against several fine-tuning recipes (full FT, encoder FT, and LoRA variants) trained on TORGO, NeuroVoz, and Spanish Common Voice. The central empirical claim is that rule-based post-processing is more robust than the tested parameter-updating methods for out-of-domain neurological Spanish speech: PP lowers S-DiverSe WER without harming in-domain scores, while fine-tuning improves TORGO/NeuroVoz but fails to generalize (and can catastrophically degrade) on S-DiverSe.
Significance. The work addresses a clear resource gap: publicly available Spanish pathological speech is scarce and mostly controlled/clinical, so an in-the-wild multi-condition evaluation set with human transcripts is a useful community contribution even at modest scale. Strengths include transparent composition analysis (sex/condition/intelligibility/duration), multi-system baselines with error-type breakdowns, explicit comparison to TORGO and NeuroVoz, and open release of annotations and video links. The PP-versus-FT contrast in Table 2 is concrete and actionable for practitioners. The finding is scoped to the tested adaptation recipes rather than a universal proof that fine-tuning cannot work, but it still motivates dedicated in-the-wild Spanish pathological benchmarks and more careful domain-adaptation practice.
major comments (2)
- §3.3–3.4 and §4.2/§5: The conclusion that heuristic post-processing is more robust than fine-tuning, and that parameter updates “fail to generalize … regardless of language composition,” rests on a narrow adaptation search (fixed LRs/epochs, LoRA r=8/α=16, three data mixes, NeuroVoz oversampled ×3). Table 2 supports the claim for these recipes, but the manuscript should more tightly scope the claim to the tested strategies and state that broader adaptation (e.g., LR/rank sweeps, longer training, multi-task or continued pretraining objectives) remains unexplored. Without that qualification, the domain-gap interpretation is stronger than the experimental design strictly warrants.
- Table 2 and §4.1–4.2: WER differences that drive the central PP-vs-FT claim (e.g., Whisper PP 22.01% vs FFT+PP 125.68% on S-DiverSe; Voxtral PP 23.73% vs several FT variants) are reported as point estimates only. With 444 segments and highly skewed pathology/intelligibility strata, bootstrap CIs or paired significance tests on total and per-pathology WER would make the robustness claim more rigorous and show whether smaller gaps (e.g., E-LoRA vs PP-only) are stable.
minor comments (7)
- §2.2: Weighted Cohen’s κ=0.38 (fair) for intelligibility is reported with 74% adjacent agreement; consider adding a short note on how this uncertainty should affect interpretation of Figure 4 (WER by intelligibility), or reporting results with collapsed adjacent bins as a sensitivity check.
- Table 1 / Figure 1: The strong male (87.4%) and ALS (78.1%) imbalance is acknowledged as reflecting in-the-wild availability; a one-sentence caution that per-pathology and per-sex WERs (especially stroke and female) are underpowered would help readers avoid over-reading those columns.
- §3.5 / evaluation protocol: Clarify whether filled-pause tokens and <unk> stripping are applied symmetrically to all systems (including Scribe v2 API output), and whether any system-specific normalization was needed for commercial transcripts.
- Figure 4 caption/text: The note that “High” is PD-only and “High/Medium” is ALS-only is important; move it into the figure caption so the plot is self-contained.
- §2.1 perplexity analysis: Median perplexity 33 (S-DiverSe) vs 49 (Common Voice) is used later to argue domain mismatch; briefly define the salamandra-2b prompting/tokenization setup so the comparison is reproducible.
- Presentation: Normalize author-name accents and model names consistently (e.g., Voxtral vs V oxtral, NeuroVoz vs NeuroV oz) across text and Table 2; fix minor spacing/typos in the arXiv rendering of compound names.
- Data release: State more explicitly what a third party must do to reconstruct the audio (YouTube availability risk, segment time stamps if any) so the benchmark’s long-term reproducibility expectations are clear.
Circularity Check
No circularity: empirical dataset/evaluation paper; WER and adaptation comparisons are measured, not derived by construction from fitted inputs.
full rationale
S-DiverSe is a corpus-release and ASR benchmarking paper. Its central claims (baseline WERs; heuristic post-processing more robust than the reported fine-tuning recipes on out-of-domain S-DiverSe) rest on direct measurement against held-out human orthographic transcripts, not on a derivation chain. Post-processing rules (character/word/phrase deduplication) are fixed heuristics independent of the S-DiverSe score; fine-tuning uses external corpora (TORGO, NeuroVoz, Common Voice) with stated hyperparameters and is evaluated, not defined, on S-DiverSe. No parameter is fitted to a subset of S-DiverSe and then re-reported as a prediction; no uniqueness theorem or ansatz is imported via self-citation to force the result; no known empirical pattern is merely renamed. Using the new corpus as the OOD test set is the purpose of a benchmark paper and does not constitute circular derivation. Score 0 is therefore appropriate.
Axiom & Free-Parameter Ledger
free parameters (4)
- NeuroVoz oversampling factor =
3
- LoRA rank r and alpha =
r=8, α=16, dropout=0.1
- Fine-tuning learning rates and epochs =
strategy-specific (1e-5/3, 2e-5/5, 3e-4/10)
- PP max word length and dedup rules =
15-char threshold; consecutive word/phrase collapse
axioms (4)
- domain assumption Self-reported diagnoses plus video metadata and perceptual non-normative speech suffice to label ALS/PD/stroke for ASR evaluation (not clinical inference).
- domain assumption A 1–5 intelligibility scale based on perceived transcription effort is a valid analysis axis despite fair IAA (κ=0.38).
- ad hoc to paper S-DiverSe must not be split for training because strata would be too skewed; it is evaluation-only.
- domain assumption Standard WER after lowercasing, punctuation strip, <unk> removal, and digit-to-word conversion is the right success metric, with filled pauses counted as tokens.
invented entities (1)
-
S-DiverSe corpus
independent evidence
read the original abstract
Automatic speech recognition (ASR) has advanced remarkably for standard speech, yet speech affected by neurological conditions remains a challenge. We present S-DiverSe (Spanish Diverse Speech), a corpus of 3.2 hours of in-the-wild Spanish speech from 22 speakers with amyotrophic lateral sclerosis, Parkinson's disease, and stroke. The dataset contains 444 manually transcribed audio segments with metadata on speaker sex, disease type, and intelligibility. S-DiverSe is designed to support ASR evaluation and development for neurologically affected Spanish speech. We describe the dataset, analyze its composition, and report baseline ASR results alongside initial adaptation experiments. Our findings reveal that heuristic text post-processing is more robust than fine-tuning for out-of-domain neurological Spanish speech. This underscores the need for dedicated in-the-wild Spanish benchmarks.
Figures
Reference graph
Works this paper leans on
-
[1]
However, they con- tinue to face substantial challenges when applied to real-world speech scenarios [2]
Introduction Automatic speech recognition (ASR) has improved markedly in recent years, with state-of-the-art models achieving low word error rates on standard benchmarks [1]. However, they con- tinue to face substantial challenges when applied to real-world speech scenarios [2]. One of these challenges is the recogni- tion of speech produced by individual...
-
[2]
S-DiverSe: Spanish Diverse Speech S-DiverSe is designated for speech recognition tasks. It com- prises 3.2 hours of human-transcribed speech, extracted from in-the-wild recordings. It features 22 unique speakers with ALS, PD, and post-stroke aftereffects. The corpus contains 444 audio segments of variable duration, and provides metadata for speaker sex, c...
Pith/arXiv arXiv 2026
-
[3]
Experimental setup S-DiverSe is used exclusively for ASR evaluation. The fac- tor preventing further partitioning is not its modest size but its variability: it spans three pathologies, multiple intelligibil- ity levels, and heterogeneous acoustic conditions, so any train- ing split would be too skewed per stratum for reliable adapta- tion. Following the ...
2026
-
[4]
High” bin comprises PD speech exclusively, while the “High/Medium
Results and discussion Table 2 summarizes WER on NeuroV oz, TORGO, and S- DiverSe for all systems and adaptation settings. 4.1. Baseline results Among open-weight models,omniASR CTC 1B v2achieves the lowest WER on S-DiverSe, while V oxtral-Mini leads on NeuroV oz and Whisper-large-v3 on TORGO. No single open- weight model dominates across all corpora, sug...
-
[5]
The dataset contains 3.2 hours of in-the-wild segments from 22 speakers with ALS, Parkinson’s disease, and stroke
Conclusion We present S-DiverSe, the first Spanish corpus of neurologi- cally affected speech spanning multiple diseases. The dataset contains 3.2 hours of in-the-wild segments from 22 speakers with ALS, Parkinson’s disease, and stroke. Our experiments re- veal that current ASR systems struggle under the heterogeneous captured conditions. Heuristic text p...
-
[6]
Generative AI Use Disclosure We used a generative AI tool to paraphrase and polish portions of the manuscript to improve readability and grammar
-
[7]
Linguistics, Artificial Intelligence and Lan- guage and Speech Technologies: from Research to Applica- tions
Acknowledgments We would like to thank Irene Gordo Bernat, speech-language pathologist, for her valuable insights and assistance with speech analysis and data selection. This project has been partially funded by the European Union’s Horizon 2020 RIA ELO- QUENCE project (Grant Agreement No. 101135916). Views and opinions expressed are, however, those of th...
2020
-
[8]
Automatic speech recognition: A survey of deep learning techniques and ap- proaches,
H. Ahlawat, N. Aggarwal, and D. Gupta, “Automatic speech recognition: A survey of deep learning techniques and ap- proaches,”International Journal of Cognitive Computing in En- gineering, 2025
2025
-
[9]
Asr benchmarking: Need for a more representative conversational dataset,
G. Maheshwari, D. Ivanov, T. Johannet, and K. El Haddad, “Asr benchmarking: Need for a more representative conversational dataset,” inICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5
2025
-
[10]
The torgo database of acoustic and articulatory speech from speakers with dysarthria,
F. Rudzicz, A. K. Namasivayam, and T. Wolff, “The torgo database of acoustic and articulatory speech from speakers with dysarthria,”Language resources and evaluation, vol. 46, no. 4, pp. 523–541, 2012
2012
-
[11]
Dysarthric speech database for universal access research
H. Kim, M. Hasegawa-Johnson, A. Perlman, J. R. Gunderson, T. S. Huang, K. L. Watkin, S. Frameet al., “Dysarthric speech database for universal access research.” inInterspeech, vol. 2008, 2008, pp. 1741–1744
2008
-
[12]
Community-supported shared infrastructure in support of speech accessibility,
M. Hasegawa-Johnson, X. Zheng, H. Kim, C. Mendes, M. Dickin- son, E. Hege, C. Zwilling, M. M. Channell, L. Mattie, H. Hodges et al., “Community-supported shared infrastructure in support of speech accessibility,”Journal of Speech, Language, and Hearing Research, vol. 67, no. 11, pp. 4162–4175, 2024
2024
-
[13]
The Interspeech 2025 Speech Accessibility Project Challenge,
X. Zheng, B. Phukon, J. Na, E. Cutrell, K. J. Han, M. Hasegawa- Johnson, P.-P. Jiang, A. Kuila, C. Lea, B. MacDonald, G. Man- tena, V . Ravichandran, L. Sari, K. Tomanek, C. D. Yoo, and C. Zwilling, “The Interspeech 2025 Speech Accessibility Project Challenge,” inInterspeech 2025, 2025, pp. 3269–3273
2025
-
[14]
New spanish speech cor- pus database for the analysis of people suffering from parkinson’s disease
J. R. Orozco-Arroyave, J. D. Arias-Londo ˜no, J. F. Vargas-Bonilla, M. C. Gonzalez-R ´ativa, and E. N ¨oth, “New spanish speech cor- pus database for the analysis of people suffering from parkinson’s disease.” inLrec, 2014, pp. 342–347
2014
-
[15]
Neurovoz: a castillian spanish corpus of parkinsonian speech,
J. Mendes-Laureano, J. A. G ´omez-Garc´ıa, A. Guerrero-L ´opez, E. Luque-Buzo, J. D. Arias-Londo ˜no, F. J. Grandas-P ´erez, and J. I. Godino-Llorente, “Neurovoz: a castillian spanish corpus of parkinsonian speech,”Scientific Data, vol. 11, no. 1, p. 1367, 2024
2024
-
[16]
Neurovoz: a castillian spanish corpus of parkinsonian speech,
J. Mendes-Laureano, J. A. G ´omez-Garc´ıa, A. Guerrero-L ´opez, E. Luque-Buzo, J. D. Arias-Londo ˜no, F. J. Grandas-P ´erez, and J. I. Godino Llorente, “Neurovoz: a castillian spanish corpus of parkinsonian speech,” Mar. 2024. [Online]. Available: https://doi.org/10.5281/zenodo.10777657
-
[17]
A. Gonzalez-Agirre, M. P `amies, J. Llop, I. Baucells, S. Da Dalt, D. Tamayo, J. J. Saiz, F. Espu ˜na, J. Prats, J. Aula-Blascoet al., “Salamandra technical report,”arXiv preprint arXiv:2502.08489, 2025
Pith/arXiv arXiv 2025
-
[18]
Com- mon voice: A massively-multilingual speech corpus,
R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Hen- retty, R. Morais, L. Saunders, F. Tyers, and G. Weber, “Com- mon voice: A massively-multilingual speech corpus,” inProceed- ings of the twelfth language resources and evaluation conference, 2020, pp. 4218–4222
2020
-
[19]
Robust speech recognition via large-scale weak supervision,
A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” inInternational conference on machine learning. PMLR, 2023, pp. 28 492–28 518
2023
-
[20]
A. H. Liu, A. Ehrenberg, A. Lo, C. Denoix, C. Barreau, G. Lam- ple, J.-M. Delignon, K. R. Chandu, P. von Platen, P. R. Mud- direddyet al., “V oxtral,”arXiv preprint arXiv:2507.13264, 2025
Pith/arXiv arXiv 2025
-
[21]
A. H. Liu, K. Khandelwal, S. Subramanian, V . Jouault, A. Rastogiet al., “Ministral 3,” 2026. [Online]. Available: https://arxiv.org/abs/2601.08584
Pith/arXiv arXiv 2026
-
[22]
Omnilingual asr: Open-source multilingual speech recognition for 1600+ languages,
A. Omnilingual, G. Keren, A. Kozhevnikov, Y . Meng, C. Rop- ers, M. Setzler, S. Wang, I. Adebara, M. Auli, C. Baliogluet al., “Omnilingual asr: Open-source multilingual speech recognition for 1600+ languages,”arXiv preprint arXiv:2511.09690, 2025
arXiv 2025
-
[23]
wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,
A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,”Advances in neural information processing systems, vol. 33, pp. 12 449–12 460, 2020
2020
-
[24]
Scribe v2,
ElevenLabs, “Scribe v2,” Speech-to-Text model, 2026, https: //elevenlabs.io/docs/overview/models
2026
-
[25]
Cba-whisper: Curriculum learning-based adalora fine-tuning on whisper for low-resource dysarthric speech recognition,
T. Tan, X. Chen, X. Le, W. Fan, X. Xia, C. Huang, and J. Lu, “Cba-whisper: Curriculum learning-based adalora fine-tuning on whisper for low-resource dysarthric speech recognition,” inProc. Interspeech 2025, 2025, pp. 3309–3313
2025
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.