Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Speechless: Speech Instruction Training Without Speech for Low Resource Languages

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Speechless proposes replacing speech instruction data with semantic tokens generated from text, so an LLM can be fine-tuned on text and still hear speech at inference.

desk verdict A useful TTS-free recipe for speech instruction tuning, but the paper's own numbers reveal a train/inference token gap that the evaluation never closes. read the letter →

arxiv 2505.17417 v1 pith:UKYVFGNP submitted 2025-05-23 eess.AS cs.CLcs.SD

Alan Dao (Gia Tuan Dao) , Dinh Bach Vu , Huy Hoang Ha , Tuan Le Duc Anh , Shreyas Gopal , Yue Heng Yeo , Warren Keng Hoong Low , Eng Siong Chng
show 1 more author
Jia Qi Yip
This is my paper
classification eess.AScs.CLcs.SD
keywords speechinstructiontuninglow-resourcelanguagestext-to-semanticssemantictokensWhisperencoderalignmentresidualvectorquantizationvoiceassistantsVietnamesemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that an early-fusion voice assistant can be instruction-tuned using only text instructions, with no spoken instruction recordings and no text-to-speech system. The method converts text into discrete semantic tokens that mimic the output of a frozen speech encoder, so the language model is fine-tuned on tokenized text but sees real speech tokens at inference. If the alignment is faithful, the speech instruction bottleneck for low-resource languages reduces to having an ASR dataset and a text instruction corpus, which are far more common than high-quality TTS.

What carries the argument

The load-bearing object is Speechless, a one-billion-parameter decoder-only model that translates text into the discrete token language of a quantized Whisper encoder. A residual vector quantizer with an enlarged codebook converts Whisper's continuous encoder embeddings into semantic tokens, stripping away acoustic detail while keeping meaning; Speechless is trained on paired ASR transcripts and those tokens, with a duration token compressing repeated codebook entries. This creates an offline 'text-to-semantics' translation step, so instruction tuning of the LLM happens entirely in token space and the frozen Whisper encoder supplies speaker invariance and noise robustness at inference.

What would settle it

Record a set of Vietnamese spoken questions and commands, pass them through the frozen Whisper encoder into the fine-tuned LLM, and score the answers; if the accuracy is substantially below the same questions given as text, the text-to-semantics alignment has not transferred to the instruction domain.

Watch

Extended reading notes

Core claim

The central claim is that synthetic speech tokens can be produced from text alone and used in place of real speech for instruction tuning. The authors build a residual vector quantizer over the Whisper encoder's representations, then train Speechless, a decoder-only text-to-semantics model, to map ASR transcripts to those quantized tokens. An LLM is then fine-tuned on instruction text whose user turns have been converted offline by Speechless into Whisper-style semantic tokens. At inference, real audio is passed through the same frozen Whisper encoder, and the LLM accepts the resulting tokens as if they had come from the synthetic training data. The paper reports competitive ASR decoding from the generated tokens and English spoken-instruction scores on par with a model trained on 200k real spoken instructions.

Load-bearing premise

The method stands or falls on whether the sequences of semantic tokens generated from text are close enough to the sequences Whisper produces from real speech, for questions and commands never seen during training.

Editorial extensions

If this is right

  • Any language with an ASR dataset and a text instruction corpus can get a speech-instruction-tuned assistant without recording or synthesizing spoken commands.
  • TTS quality and speaker diversity stop being constraints on synthetic speech instruction data, since no waveform is ever produced.
  • Because the speech encoder stays frozen and training happens at token level, the fine-tuning cost is close to text-only instruction tuning rather than speech model training.
  • The observed drop on text benchmarks after speech instruction tuning indicates a modality-sharing cost that future work will need to manage, and the method inherits Whisper's weaknesses on noisy input.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves the speech-to-text alignment transfer unverified in Vietnamese: VoiceBench is English-only and the final model's Vietnamese speech understanding is never measured, so a direct Vietnamese spoken-command evaluation is the natural next experiment.
  • Because the quantizer and Speechless are trained on clean read speech only, the authors' own results suggest that noisy real-world instructions are a likely failure mode until noisy ASR data is added to Stage 1.
  • If the alignment holds, the method should extend to any language Whisper already recognizes, since the only language-specific resource needed is ASR transcripts—a claim that a multilingual spoken-instruction test could confirm or refute.
  • A cheap internal check would be to compare, for the same Vietnamese text, the LLM's answers when given Speechless-generated tokens versus tokens from real recorded speech; high agreement would isolate alignment quality from downstream LLM quality.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. Speechless proposes a three-stage pipeline for training early-fusion speech-language models without speech instruction data or a TTS system. Stage 1 trains a residual vector quantizer (RVQ) on Whisper encoder features; Stage 2 trains a 1B decoder-only model, also called Speechless, to map text (plus duration tokens) into those semantic tokens; Stage 3 uses that model to convert text instruction datasets into semantic-token sequences and fine-tunes an LLM on them. At inference, audio is encoded by Whisper, quantized, and fed to the LLM. The paper reports ASR-style comparisons in Table 1, VoiceBench results in Table 2, and MMLU/VMLU results in Table 3, and it releases code and a tokenized Vietnamese instruction dataset.

Significance. The core idea is attractive and potentially valuable: if text-to-semantic-token alignment can make an LLM robust to quantized Whisper encoder tokens, then low-resource languages that have ASR corpora but no good TTS could obtain speech-instruction tuning. The paper releases code and a tokenized dataset, and the three-stage design is clearly described. However, the current evidence does not substantiate the headline claims. The Speechless rows in Table 1 are text-to-semantic reconstructions, not speech recognition, and no experiment measures whether the final LLM understands real speech instructions in Vietnamese. The main instruction-following results come from an 8B model whose configuration is not described in the methodology. These gaps are central to the claimed contribution.

major comments (4)
  1. [Section 4.1, Table 1] The rows labeled 'Speechless' are not automatic speech recognition results: they are produced by decoding semantic tokens that the Speechless model generated from text with the Whisper decoder. The actual inference path for the final assistant is audio -> Whisper encoder -> RVQ -> tokens -> LLM, and Table 1 shows that this path has much higher WER than the Speechless path, e.g., 34.84% versus 7.08% with beam search on Common Voice Vietnamese. Since Stage 3 fine-tuning used Speechless-generated tokens, the model is trained on a substantially cleaner token distribution than it receives from real speech at inference. The paper provides no Vietnamese (or other low-resource) spoken-instruction evaluation showing that the LLM copes with this distribution shift, and Section 5 concedes that robustness in diverse linguistic contexts needs further exploration. This gap is load-bearing for the central claim that the method enables voice assistants for low-resource languages.
  2. [Section 3.2 vs Tables 2 and 3] The methodology states that the authors chose LLaMA 3.2 1B Base for most experiments and that the 3B model performed similarly, yet Tables 2 and 3 report results for 'Speechless-llama3.1-8B-instruct'. The paper does not describe how this 8B model was trained, whether it uses the same Stage 1 and Stage 2 components, or why the 1B and 3B models were not evaluated on these benchmarks. The main instruction-following evidence therefore comes from an unstated configuration, making the results difficult to interpret and reproduce.
  3. [Section 4.2, Tables 2 and 3] VoiceBench is an English-only benchmark, and most of its spoken inputs are generated by Google TTS rather than real human speech. Therefore Table 2 cannot certify the paper's low-resource claim: it does not test whether the model understands Vietnamese speech instructions, nor whether it handles the noisier token stream produced by the quantized Whisper pipeline on real Vietnamese audio. A minimal addition would be a Vietnamese spoken-questions evaluation with audio passed through the actual Whisper + RVQ pipeline, with either transcripts or human judgments as the metric.
  4. [Table 1 caption and Section 4.1] Table 1 mixes three different evaluation objects under the heading 'ASR': direct Whisper zero-shot, the quantized Whisper pipeline, and Speechless text-to-semantic generation. The caption says 'Comparative analysis of model performance for general, noisy, and multilingual ASR', but the Speechless rows do not consume audio at all. This is misleading; the table should be reframed as a semantic-token reconstruction comparison, and the quantized Whisper pipeline should be identified as the relevant speech-recognition baseline for the final end-to-end system.
minor comments (6)
  1. [Section 2.2] There is a typo in 'sematic tokens'; it should be 'semantic tokens'.
  2. [Section 4.1] The sentence 'This is also clear when see that with added noise ...' is grammatically incomplete and should be rephrased.
  3. [Section 3.1] The phrase 'MLS Eng 10k' is ambiguous; please specify whether it refers to 10k hours, 10k utterances, or some other quantity.
  4. [Section 5] The sentence 'Nevertheless, Speechlessthe methods described in this paper can in principle be applied to noisy data' contains a typo and should be corrected.
  5. [Table 2 caption] The note that 'Experiments other than ours were performed by the VoiceBench authors' means the baselines were not reproduced locally; this should be stated in the main text so readers can calibrate the comparison.
  6. [Abstract and Section 1] The phrase 'the first pre-tokenized Vietnamese instruction dataset' is a strong claim; it should be substantiated with a comparison to prior released datasets or softened.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the derivation chain is an empirical training pipeline validated on held-out benchmarks.

full rationale

The claimed derivation is a three-stage training pipeline: (1) an RVQ quantizer is learned from ASR audio to map Whisper encoder outputs to discrete tokens; (2) Speechless is trained to map text transcripts to those quantized tokens using paired ASR data; (3) an LLM is instruction-tuned on tokens generated by Speechless from text instruction datasets. Each stage is fitted to its own target, and the central claim (that the resulting LLM can process spoken instructions) is tested against external benchmarks (VoiceBench, MMLU, VMLU) using held-out data. No equation or parameter is defined in terms of the quantity it is used to predict, and no reported result is a renamed fit. The paper's reliance on Ichigo [1] for the tokenizer and instruction data is self-citation, but it is not load-bearing for the core reduction: the same authors' Ichigo work supplies infrastructure, while the evaluation is independent and the WER comparisons in Table 1 are held-out measurements. The paper's Vietnamese effectiveness claim is weakened by the absence of a Vietnamese speech-instruction evaluation and by the Table 1 gap between Speechless-generated tokens (5.90-7.08% WER on CV Vi) and quantized Whisper tokens (34.84-36.53% WER on CV Vi), but this is a train/inference distribution gap and an unvalidated transfer assumption, not a circular derivation. Section 5's concession that robustness in noisy and diverse linguistic contexts requires further exploration is a limitation statement, not evidence that the method reduces to its inputs. The derivation is therefore self-contained.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The method does not introduce new physical entities. It builds on Whisper's existing representations and adds a learned text-to-semantic-token model. The key assumptions are about distribution transfer and the adequacy of the frozen Whisper decoder, plus an unanalyzed data filtering step.

free parameters (1)
  • RVQ codebook size = 2048
    Chosen by hand after Kaiming initialization failed; not fitted to data, but it is a design choice that affects the token space and is not inherited from prior work.
assumptions (4)
  • domain assumption Whisper encoder representations are semantic, speaker-invariant, and sufficiently language-agnostic to be shared between text-generated and speech-derived tokens.
    The whole method rests on Whisper's latent space being stable enough that tokens synthesized from text can stand in for tokens obtained from audio. Invoked in Sections 2.1 and 2.2.
  • domain assumption Speechless generalizes from read-speech ASR transcripts to instruction-style text (questions, imperatives) not seen in training.
    Speechless is trained on ViVoice and LibriTTS-R, which are read speech; instruction data has a different style. No direct evaluation of generated instruction tokens against Whisper tokens for the same instructions is provided. Invoked in Section 2.3.
  • domain assumption The frozen Whisper decoder can decode RVQ-quantized tokens adequately for evaluation and downstream use, even though it was not trained with quantization.
    Table 1 shows quantization alone degrades Whisper WER; the authors acknowledge 'information lost during the residual vector quantization operation' but assume the degradation is acceptable. Invoked in Section 4.1.
  • domain assumption The instruction data filtering and response curation do not introduce bias that materially changes the benchmark outcomes.
    Samples are removed and responses rewritten with Qwen2.5-32B; the effect of these choices on VoiceBench scores is not analyzed. Invoked in Section 2.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Speechless: Speech Instruction Training Without Speech for Low Resource Languages." pith.science (2026). https://pith.science/paper/UKYVFGNP

@misc{pith2026250517417,
  author       = {Pith},
  title        = {Pith review of: Speechless: Speech Instruction Training Without Speech for Low Resource Languages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UKYVFGNP}},
  note         = {Machine review of arXiv:2505.17417}
}
read the original abstract

The rapid growth of voice assistants powered by large language models (LLM) has highlighted a need for speech instruction data to train these systems. Despite the abundance of speech recognition data, there is a notable scarcity of speech instruction data, which is essential for fine-tuning models to understand and execute spoken commands. Generating high-quality synthetic speech requires a good text-to-speech (TTS) model, which may not be available to low resource languages. Our novel approach addresses this challenge by halting synthesis at the semantic representation level, bypassing the need for TTS. We achieve this by aligning synthetic semantic representations with the pre-trained Whisper encoder, enabling an LLM to be fine-tuned on text instructions while maintaining the ability to understand spoken instructions during inference. This simplified training process is a promising approach to building voice assistant for low-resource languages.

Figures

Figures reproduced from arXiv: 2505.17417 by the authors.

Figure 1
Figure 1. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TESU-LLM: Training Speech-LLMs Without Speech via Unified Encoder Alignment

    cs.CL 2025-06 conditional novelty 5.0 of 10

    TESU-LLM shows that a frozen LLM can answer spoken queries after training only a 13M-parameter projector on text, using SeamlessM4T's shared speech-text encoder.

Reference graph

Works this paper leans on

45 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    Introduction V oice assistants have become an integral part of modern tech- nology, providing users with the ability to interact with de- vices through natural language [1]. These voice assistants can be achieved through a cascade of automatic speech recogni- tion (ASR) which transcribes instructions which are then pro- cessed by an instruction-tuned larg...

    arXiv 2025

  2. [2]

    First, we train a residual vector quantizer (RVQ) to en- code speech into discrete semantic tokens that align with Whis- per’s encoder representations

    Methodology Our method consists of three main stages, as illustrated in Fig- ure 1. First, we train a residual vector quantizer (RVQ) to en- code speech into discrete semantic tokens that align with Whis- per’s encoder representations. Second, we develop Speechless, a decoder-only language model that learns to generate these se- mantic tokens directly fro...

    work page

  3. [3]

    Datasets For Stage 1, we utilized two automatic speech recognition (ASR) datasets: viV oice (Vietnamese) and LibriTTS-R[22] (English)

    Experiments 3.1. Datasets For Stage 1, we utilized two automatic speech recognition (ASR) datasets: viV oice (Vietnamese) and LibriTTS-R[22] (English). The ViV oice dataset consists of 868k utterances for training, 10k for validation, and 10k for testing, while the LibriTTS-R dataset contains 112k training samples, 5.6k vali- dation samples, and 4.6k test...

    work page

  4. [4]

    ASR and Speechless Comparisons To evaluate the performance of the Speechless model alone, we make use of ASR test sets

    Results 4.1. ASR and Speechless Comparisons To evaluate the performance of the Speechless model alone, we make use of ASR test sets. To do this evaluation, we compare semantic tokens generated by Whisper Encoder from speech with the semantic tokens generated by Speechless from text. In both cases the semantic tokens are decoded by the same Whis- per Decod...

    work page

  5. [5]

    By lever- aging a quantized Whisper encoder, Speechless generates se- mantic speech tokens, effectively addressing challenges in low- resource languages

    Conclusion This paper introducedSpeechless, a novel method for gener- ating synthetic training data for early-fusion speech language models without traditional text-to-speech systems. By lever- aging a quantized Whisper encoder, Speechless generates se- mantic speech tokens, effectively addressing challenges in low- resource languages. Our experiments dem...

    work page

  6. [6]

    This is also clear when see that with added noise (VBD noisy), the Whisper encoder starts to generate tokens that show poorer WER in comparison

    This shows that Speechless is able to map raw text informa- tion to clean speech in the latent space. This is also clear when see that with added noise (VBD noisy), the Whisper encoder starts to generate tokens that show poorer WER in comparison. We can also observe that once quantized, the Whisper en- coder’s performance declines in both noisy and multil...

    work page

  7. [7]

    SALMONN: Towards generic hearing abilities for large language models,

    C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. MA, and C. Zhang, “SALMONN: Towards generic hearing abilities for large language models,” inThe Twelfth International Conference on Learning Representations, 2024

    2024

  8. [8]

    Ichigo: Mixed-Modal Early-Fusion Realtime Voice Assistant

    A. Dao, D. B. Vu, and H. H. Ha, “Ichigo: Mixed-modal early- fusion realtime voice assistant,”arXiv preprint arXiv:2410.15316, 2024

    work page Pith review arXiv 2024

Show all 45 references
  1. [9]

    Wavchat: A survey of spoken dialogue models,

    S. Ji, Y . Chen, M. Fang, J. Zuo, J. Lu, H. Wang, Z. Jiang, L. Zhou, S. Liu, X. Chenget al., “Wavchat: A survey of spoken dialogue models,”arXiv preprint arXiv:2411.13577, 2024

    2024 arXiv

  2. [10]

    Recent advances in speech language models: A survey,

    W. Cui, D. Yu, X. Jiao, Z. Meng, G. Zhang, Q. Wang, Y . Guo, and I. King, “Recent advances in speech language models: A survey,” arXiv preprint arXiv:2410.03751, 2024

    2024 arXiv

  3. [11]

    Llama- omni: Seamless speech interaction with large language models,

    Q. Fang, S. Guo, Y . Zhou, Z. Ma, S. Zhang, and Y . Feng, “Llama- omni: Seamless speech interaction with large language models,” arXiv preprint arXiv:2409.06666, 2024

    2024 arXiv

  4. [12]

    Instruction data generation and unsupervised adap- tation for speech language models,

    V . Noroozi, Z. Chen, S. Majumdar, S. Huang, J. Balam, and B. Ginsburg, “Instruction data generation and unsupervised adap- tation for speech language models,” inInterspeech 2024, 2024, pp. 4049–4053

    2024

  5. [13]

    Tango 2: Aligning diffusion-based text-to-audio generations through direct preference optimization,

    N. Majumder, C.-Y . Hung, D. Ghosal, W.-N. Hsu, R. Mihalcea, and S. Poria, “Tango 2: Aligning diffusion-based text-to-audio generations through direct preference optimization,” inProceed- ings of the 32nd ACM International Conference on Multimedia, 2024, pp. 564–572

    2024

  6. [14]

    Dis- tilling an end-to-end voice assistant without instruction training data,

    W. Held, E. Li, M. Ryan, W. Shi, Y . Zhang, and D. Yang, “Dis- tilling an end-to-end voice assistant without instruction training data,”arXiv preprint arXiv:2410.02678, 2024

    2024 arXiv

  7. [15]

    Cosmic: Data efficient instruction-tuning for speech in-context learning,

    J. Pan, J. Wu, Y . Gaur, S. Sivasankaran, Z. Chen, S. Liu, and J. Li, “Cosmic: Data efficient instruction-tuning for speech in-context learning,”CoRR, vol. abs/2311.02248, 2023

    2023 arXiv

  8. [16]

    Lib- risqa: Pioneering free-form and open-ended spoken question an- swering with a novel dataset and framework,

    Z. Zhao, Y . Jiang, H. Liu, Y . Wang, and Y . Wang, “Lib- risqa: Pioneering free-form and open-ended spoken question an- swering with a novel dataset and framework,”arXiv preprint arXiv:2308.10390, 2023

    2023 arXiv

  9. [17]

    An efficient and high fidelity vietnamese streaming end-to-end speech synthesis,

    T. N. D. Tran, T. C. Chu, V . Hoang, T. H. Bui, and H. Q. Truong, “An efficient and high fidelity vietnamese streaming end-to-end speech synthesis,” inInterspeech 2022, 2022, pp. 466–470

    2022

  10. [18]

    Low-resource multilingual and zero-shot multispeaker TTS,

    F. Lux, J. Koch, and N. T. Vu, “Low-resource multilingual and zero-shot multispeaker TTS,” inProceedings of the 2nd Confer- ence of the Asia-Pacific Chapter of the Association for Computa- tional Linguistics and the 12th International Joint Conference on Natural Language Proce...

    2022

  11. [19]

    Enhancing low-resource language and in- struction following capabilities of audio language models,

    P. Manakul, G. Sun, W. Sirichotedumrong, K. Tharnpipitchai, and K. Pipatanakul, “Enhancing low-resource language and in- struction following capabilities of audio language models,”arXiv preprint arXiv:2409.10999, 2024

    2024 arXiv

  12. [20]

    Unsupervised cross-modal alignment of speech and text embedding spaces,

    Y .-A. Chung, W.-H. Weng, S. Tong, and J. Glass, “Unsupervised cross-modal alignment of speech and text embedding spaces,”Ad- vances in neural information processing systems, vol. 31, 2018

    2018

  13. [21]

    Alpaca: A strong, replicable instruction-following model,

    R. Taori, I. Gulrajani, T. Zhang, Y . Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, “Alpaca: A strong, replicable instruction-following model,” March 2023, stanford Center for Research on Foundation Models (CRFM)

    2023

  14. [22]

    An analysis of semantically-aligned speech-text embeddings,

    M. Huzaifah and I. Kukanov, “An analysis of semantically-aligned speech-text embeddings,” in2022 IEEE Spoken Language Tech- nology Workshop (SLT). IEEE, 2023, pp. 747–754

    2023

  15. [23]

    Astra: Aligning speech and text representa- tions for asr without sampling,

    N. Gaur, R. Agrawal, G. Wang, P. Haghani, A. Rosenberg, and B. Ramabhadran, “Astra: Aligning speech and text representa- tions for asr without sampling,” inInterspeech 2024, 2024, pp. 3904–3908

    2024

  16. [24]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” inInternational conference on machine learning. PMLR, 2023, pp. 28 492–28 518

    2023

  17. [25]

    Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing,

    J. Ao, R. Wang, L. Zhou, C. Wang, S. Ren, Y . Wu, S. Liu, T. Ko, Q. Li, Y . Zhang, Z. Wei, Y . Qian, J. Li, and F. Wei, “Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing,” 2021

    2021

  18. [26]

    Sailor 2 dataset,

    Sailor2, “Sailor 2 dataset,” https://huggingface.co/datasets/ sailor2/sailor2-sft-stage1, 2024, accessed on February 18, 2025

    2024

  19. [27]

    Vtsnlp instruct general dataset,

    VTSNLP, “Vtsnlp instruct general dataset,” https://huggingface. co/datasets/VTSNLP/instruct general dataset, 2024, accessed on February 18, 2025

    2024

  20. [28]

    V oicebench: Benchmarking llm-based voice assistants,

    Y . Chen, X. Yue, C. Zhang, X. Gao, R. T. Tan, and H. Li, “V oicebench: Benchmarking llm-based voice assistants,”arXiv preprint arXiv:2410.17196, 2024

    2024 arXiv

  21. [29]

    Libritts: A corpus derived from librispeech for text- to-speech,

    H. Zen, V . Dang, R. Clark, Y . Zhang, R. J. Weiss, Y . Jia, Z. Chen, and Y . Wu, “Libritts: A corpus derived from librispeech for text- to-speech,”arXiv preprint arXiv:1904.02882, 2019

    1904 arXiv

  22. [30]

    vivoice: Enabling vietnamese multi-speaker speech synthesis,

    Capleaf, “vivoice: Enabling vietnamese multi-speaker speech synthesis,” https://huggingface.co/datasets/capleaf/viV oice, 2024, accessed on February 18, 2025

    2024

  23. [31]

    noise classes at varying signal-to-noise ratios. The tran- scripts from these datasets were used to evaluate the seman- tic token quality produced by the Speechless model after de- quantization and decoding via the Whisper decoder. For multilingual ASR evaluation, we utilized ...

  24. [32]

    Libritts-p: A corpus with speaking style and speaker identity prompts for text-to-speech and style captioning,

    M. Kawamura, R. Yamamoto, Y . Shirahata, T. Hasumi, and K. Tachibana, “Libritts-p: A corpus with speaking style and speaker identity prompts for text-to-speech and style captioning,” 2024

    2024

  25. [33]

    Mls: A large-scale multilingual dataset for speech research,

    V . Pratap, Q. Xu, A. Sriram, G. Synnaeve, and R. Collobert, “Mls: A large-scale multilingual dataset for speech research,”ArXiv, vol. abs/2012.03411, 2020

    2012 arXiv

  26. [34]

    Efficient memory man- agement for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory man- agement for large language model serving with pagedattention,” arXiv preprint arXiv:2309.06180, 2023

    2023 arXiv

  27. [35]

    Ray: A distributed framework for emerging ai applications,

    P. Moritz, R. Nishihara, S. Wang, A. Tumanov, R. Liaw, E. Liang, M. Elibol, Z. Yang, W. Paul, M. I. Jordan, and I. Stoica, “Ray: A distributed framework for emerging ai applications,”arXiv preprint arXiv:1712.05889, 2018

    2018 arXiv

  28. [36]

    Lib- rispeech: an asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: an asr corpus based on public domain audio books,” in2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 5206–5210

    2015

  29. [37]

    Speech enhancement for a noise-robust text-to-speech synthe- sis system using deep recurrent neural networks,

    C. Valentini-Botinhao, X. Wang, S. Takaki, and J. Yamagishi, “Speech enhancement for a noise-robust text-to-speech synthe- sis system using deep recurrent neural networks,” inInterspeech 2016, 2016, pp. 352–356

    2016

  30. [38]

    The diverse environments multi-channel acoustic noise database (demand): A database of multichannel environmental noise recordings,

    J. Thiemann, N. Ito, and E. Vincent, “The diverse environments multi-channel acoustic noise database (demand): A database of multichannel environmental noise recordings,” inProceedings of Meetings on Acoustics, vol. 19, no. 1. AIP Publishing, 2013

    2013

  31. [39]

    Com- mon voice: A massively-multilingual speech corpus,

    R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Hen- retty, A. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Com- mon voice: A massively-multilingual speech corpus,” 2020

    2020

  32. [40]

    Length- controlled alpacaeval: A simple way to debias automatic evalua- tors,

    Y . Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto, “Length- controlled alpacaeval: A simple way to debias automatic evalua- tors,”arXiv preprint arXiv:2404.04475, 2024

    2024 arXiv

  33. [41]

    SD-QA: Spoken dialectal question answering for the real world,

    F. Faisal, S. Keshava, M. M. I. Alam, and A. Anastasopoulos, “SD-QA: Spoken dialectal question answering for the real world,” inFindings of the Association for Computational Linguistics: EMNLP 2021, M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, Eds., pp. 3296–3315

    2021

  34. [42]

    Can a suit of armor conduct electricity? a new dataset for open book question answering,

    T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal, “Can a suit of armor conduct electricity? a new dataset for open book question answering,”arXiv preprint arXiv:1809.02789, 2018

    2018 arXiv

  35. [43]

    Universal and transferable adversarial attacks on aligned language models,

    A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrik- son, “Universal and transferable adversarial attacks on aligned language models,”arXiv preprint arXiv:2307.15043, 2023

    2023 arXiv

  36. [44]

    Moshi: a speech-text foundation model for real-time dialogue,

    A. D ´efossez, L. Mazar´e, M. Orsini, A. Royer, P. P´erez, H. J´egou, E. Grave, and N. Zeghidour, “Moshi: a speech-text foundation model for real-time dialogue,”arXiv preprint arXiv:2410.00037, 2024

    2024 arXiv

  37. [45]

    BLSP: Bootstrapping language-speech pre-training via behavior alignment of continuation writing,

    C. Wang, M. Liao, Z. Huang, J. Lu, J. Wu, Y . Liu, J. Zhang, and C. Zong, “BLSP: Bootstrapping language-speech pre-training via behavior alignment of continuation writing,” 2024

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.