Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Voila: Voice-Language Foundation Models for Real-Time Autonomous Interaction and Voice Role-Play

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Voila presents a full-duplex voice-language model family that answers in 195 milliseconds, clones voices from ten-second samples, and unifies ASR, TTS, and conversation in one open-source architecture.

desk verdict The open-source model and LibriSpeech numbers are real, but the paper's headline 195 ms latency and full-duplex autonomy claims have no experimental support anywhere in the manuscript. read the letter →

arxiv 2505.02707 v1 pith:5QZ2R3MV submitted 2025-05-05 cs.AI cs.CLcs.SD

Yemin Shi , Yu Shu , Siwei Dong , Guangyi Liu , Jaward Sesay , Jingwen Li , Zhiting Hu This is my paper · ORCID
classification cs.AIcs.CLcs.SD
keywords voice-languagefoundationmodelsfull-duplexconversationend-to-endspeechmodelinglow-latencyinteractionvoicecloningspeakerpersonalizationaudiotokenizationbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Voila, a family of end-to-end voice-language foundation models designed to move spoken AI from turn-based, text-mediated pipelines to autonomous full-duplex conversation. The central claims are three: the model responds in 195 milliseconds, faster than the average human turn-taking gap of under 300 milliseconds; it preserves and controls vocal qualities such as tone, rhythm, and emotion through a hierarchical multi-scale Transformer that pairs an LLM backbone with a dedicated audio transformer; and it supports over a million pre-built voices plus new voices from ten-second audio samples, steered by text personas. Voila is also presented as a unified model that handles spoken dialogue, ASR, TTS, and, with minimal finetuning, multilingual speech translation, and it is released open-source. If the claims hold, a single model can listen, reason, and speak concurrently while remaining steerable by text and personalizable by voice, which is the missing piece for proactive AI companions.

What carries the argument

The central object is the hierarchical multi-scale Transformer: a voice-language LLM backbone that handles semantic content and an audio Transformer that predicts acoustic tokens from the backbone's output. Its companion pieces are the Voila tokenizer—a four-layer residual-vector-quantization neural codec with semantic information distilled into the first level—and the word-level text-audio interleaved alignment, in which each text token is paired with its own audio tokens and then embedded, repeated, and averaged before entering the backbone. For full-duplex interaction, Voila-autonomous fuses two streams by tokenizing and embedding both the user's audio and its own audio and averaging the embeddings before the backbone. This machinery lets the pretrained LLM reason in a familiar symbolic space while a separate module handles fine acoustic detail, and it is what turns voice role-play, million-voice libraries, and 195-millisecond latency into properties of a single open-source model rather than an assembled pipeline.

What would settle it

Play the 1,580 spoken benchmark questions to human participants and score their answers with the same automated rubric used for the model outputs, then compare with published accuracy on the original text items; if spoken accuracy falls materially below text accuracy, the difficulty-preservation assumption fails and the reported leaderboard gaps would need rechecking.

Watch

Extended reading notes

Core claim

Voila's central discovery is that the reasoning power of a pretrained LLM and the acoustic richness of speech can be combined in one end-to-end architecture, without a text bottleneck, by separating semantic and acoustic modeling across a multi-scale Transformer. The model uses a neural tokenizer whose first residual-vector-quantization level carries semantic content and whose remaining levels carry acoustic detail; the backbone LLM consumes semantic tokens interleaved with text tokens at word level, and a second audio Transformer predicts the acoustic tokens that the tokenizer decodes into waveform. Voila-autonomous, the full-duplex variant, fuses two simultaneous input streams—the user's audio and its own audio—so it can hear itself while speaking and decide to interrupt, backchannel, or stay silent. Trained on speech recognition, synthesis, and instruction-following tasks under a chat format, the same model reaches 30.56 on the new Voila Benchmark (versus 13.29 for SpeechGPT and 11.45 for Moshi), competitive ASR word error rates on LibriSpeech (2.7% when trained on its training split), and a TTS word error rate of 3.2%. The design also treats speaker identity as a plug-in embedding, making voice cloning from ten seconds of audio and persona steering by text direct corollaries of the architecture rather than add-on modules.

Load-bearing premise

The load-bearing premise is that turning text questions into speech and scoring machine transcripts of the answers preserves the difficulty of the original tasks and measures voice-language ability rather than the side effects of transcription and judging.

Editorial extensions

If this is right

  • A spoken assistant built on Voila could interject, acknowledge, and respond mid-utterance, matching the timing and overlap of human conversation rather than waiting for a full turn.
  • Voice cloning and persona creation would no longer require separate TTS modules or fine-tuning per speaker: one ten-second sample plus a text prompt defines a new character.
  • Because ASR, TTS, and dialogue share one backbone, downstream tasks such as multilingual speech translation and emotion-aware response can be added with minimal finetuning, and the same model serves all of them.
  • Releasing the models, tokenizer, benchmark, and a million-voice library would give the field a common test bed for measuring whether speech-to-speech models preserve LLM reasoning, not just acoustic fidelity.
  • The 195-millisecond latency figure, if reproducible in real deployments, would push voice AI below a human-visible threshold, making full-duplex conversation the default interface for wearable and ambient AI.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test the paper does not run: feed the same questions to the model as text and as speech; if accuracy drops sharply on spoken input, the bottleneck is audio understanding rather than reasoning, which would change how the architecture should be judged.
  • The benchmark relies on a text-to-speech rewrite of math and code and on an automated judge scoring speech-recognition transcripts, so the numbers likely understate acoustic capability and overstate reasoning under noisy conditions; a human-listener evaluation on the same 1,580 items would settle which.
  • If the word-level text-audio interleaved alignment holds up, it suggests that explicit sub-word synchronization, not just a streaming encoder, is the active ingredient for stable full-duplex training—an insight transferable to other speech-language models.
  • The million-voice library, built from an embedding extractor, implies that voice identity may be separable from linguistic content in the learned representation; that hypothesis could be tested by cross-lingual voice cloning and by removing the speaker embedding at inference.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces Voila, a family of end-to-end voice-language models built on a hierarchical multi-scale Transformer. The architecture combines a voice tokenizer with semantic and acoustic RVQ levels, a text-audio interleaved alignment strategy, and a two-stream full-duplex variant (Voila-autonomous) that averages embeddings of the user and system audio streams. The authors claim 195 ms response latency, support for over a million voices, and unified ASR/TTS/dialogue capabilities in a single model. The experimental section describes a new audio benchmark (Voila Benchmark) constructed by rewriting text questions with GPT-4o and synthesizing them with Google TTS, with scoring by GPT-4o on Whisper transcriptions; it also reports LibriSpeech ASR and TTS WER numbers. Voila outperforms SpeechGPT and Moshi on the new benchmark and is competitive with strong ASR/TTS baselines.

Significance. The paper has tangible strengths: the models, code, tokenizer, benchmark, and a million-voice library are released open-source; the LibriSpeech ASR and TTS results are concrete and competitive; the proposed interleaved text-audio alignment with a four-level RVQ tokenizer is a plausible architectural contribution. If the headline claims of 195 ms latency and full-duplex autonomous interaction were properly measured and validated, the work would be significant for real-time voice AI. However, the current manuscript does not provide such measurements, and the new benchmark's scoring protocol is unvalidated, so the benchmark numbers cannot substitute for evidence of the paper's central claims. The value of the contribution is therefore primarily in the released artifacts and the component-level results, not in the demonstrated real-time autonomous capabilities featured in the title and abstract.

major comments (4)
  1. [Abstract and Section 1] The abstract's headline quantitative claim, 'response latency of just 195 milliseconds, surpassing the average human response time,' is never defined or measured in the paper. Section 4 contains no latency experiment, no definition of the latency interval (e.g., end of user speech to start of system audio, or time-to-first-token), no hardware or streaming configuration, and no comparison against pipeline or other end-to-end baselines. As the central quantitative promise of the paper, its complete absence from the evaluation is a load-bearing omission.
  2. [Section 3.2 and Figure 5] The 'full-duplex' and 'autonomous interaction' claims are not supported by the presented mechanism or by any experiment. The description only states that the user audio stream and Voila's own audio stream are tokenized, embedded, averaged, and fed into the backbone LLM; it does not explain or evaluate how the model decides when to speak, how it handles barge-in, whether it produces backchannels, or how it avoids self-interference. No results on overlapping speech, interruption handling, or proactive initiation are provided, despite the title and abstract promising 'real-time autonomous interaction.'
  3. [Sections 4.1 and 4.2] The Voila Benchmark's validity is not established. The benchmark is constructed by rewriting text questions with GPT-4o and synthesizing speech with Google TTS, and responses are scored by GPT-4o on Whisper transcriptions. No evidence is given that TTS rendering preserves the difficulty of the original tasks, that GPT-4o scoring correlates with human judgment, or that Whisper transcription errors affect all compared models equally. Without such validation, the numerical comparisons in Tables 1 and 2 cannot be interpreted as reliable measurements of voice-language capability.
  4. [Section 3.3] The paper claims 'over one million pre-built voices' and customization from audio samples 'as short as 10 seconds,' but it provides no experimental evaluation of voice similarity, naturalness, or intelligibility of the cloned voices. This is a stated key feature of the system, and its absence from the experimental section weakens the claim.
minor comments (4)
  1. [Throughout] The text contains repeated formatting artifacts such as 'V oila' with a space; these should be fixed in the final version.
  2. [References] The same work by Maiti et al. appears as both 2024a and 2024b in the references and citations; please consolidate.
  3. [Abstract vs. Section 3.3] The abstract says voice cloning from samples 'as short as 10 seconds,' while Section 3.3 says the audio clip can be 'any length—ranging from a few seconds to several hours.' Please make the statements consistent.
  4. [Section 3 and 4] The paper does not report model sizes, number of parameters, training data composition beyond 'extensive audio-text data,' or compute budget. For a foundation-model paper, at least the parameter count per model variant should be stated in the main text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Voila's claims are measured on external benchmarks or are unsupported assertions, and the only self-citation is descriptive and not load-bearing.

full rationale

Voila's central evaluations are conducted against external benchmarks and external baselines. The ASR results use LibriSpeech test-clean with standard WER, and the TTS results follow the VALL-E protocol with HuBERT transcription; these are independent, parameter-free measurements with no fitted quantity renamed as a prediction. The Voila Benchmark is assembled from external text datasets (MMLU, MATH, HumanEval, NQ-Open, GSM8K), converted to speech by GPT-4o rewriting plus Google TTS, and scored by GPT-4o on Whisper transcriptions. This protocol raises validity questions about whether spoken rewrites preserve difficulty and whether GPT-4o scores match human judgment, but it does not reduce Voila's accuracy to its training inputs or to any fitted parameter: the reference answers come from the original datasets, and the model being evaluated is distinct from the grading model. The abstract's headline numbers—195 ms latency and full-duplex autonomous interaction—are never defined or measured in the experimental section, but an unsupported claim is an evidentiary gap, not a circular derivation. The only self-citation is LLaSM (Shu et al., 2023) in the related-work survey of Whisper-encoder connector models; it is descriptive, not load-bearing for Voila's architecture or for any result. No equation, benchmark score, or design choice can be shown to be equivalent to its own inputs by construction, so no significant circularity is present.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No explicit fitted parameters are reported. The central claims depend on several unvalidated design choices and evaluation assumptions, and the 195 ms latency value is asserted without a measurement protocol.

assumptions (4)
  • domain assumption Whisper transcription followed by GPT-4o scoring is a valid measure of spoken-response correctness.
    The Voila Benchmark evaluation (Section 4.2) uses this protocol without comparing against human judgments or alternative metrics.
  • ad hoc to paper Converting text benchmarks to speech with GPT-4o rewriting and Google TTS preserves the reasoning difficulty of the original tasks.
    The benchmark construction (Section 4.1) assumes TTS rendering does not alter problem content or difficulty, which is not validated.
  • ad hoc to paper The hierarchical multi-scale architecture with interleaved text-audio tokens is effective for voice-language modeling.
    Section 3 presents this design as the core contribution, but no ablations isolate its effect on the reported results.
  • ad hoc to paper Averaging the embeddings of the user and Voila audio streams in Voila-autonomous enables full-duplex interaction without self-interference.
    Figure 5 and Section 3.2 describe the fusion, but the paper provides no evaluation of simultaneous listening and speaking.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Voila: Voice-Language Foundation Models for Real-Time Autonomous Interaction and Voice Role-Play." pith.science (2026). https://pith.science/paper/5QZ2R3MV

@misc{pith2026250502707,
  author       = {Pith},
  title        = {Pith review of: Voila: Voice-Language Foundation Models for Real-Time Autonomous Interaction and Voice Role-Play},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5QZ2R3MV}},
  note         = {Machine review of arXiv:2505.02707}
}
read the original abstract

A voice AI agent that blends seamlessly into daily life would interact with humans in an autonomous, real-time, and emotionally expressive manner. Rather than merely reacting to commands, it would continuously listen, reason, and respond proactively, fostering fluid, dynamic, and emotionally resonant interactions. We introduce Voila, a family of large voice-language foundation models that make a step towards this vision. Voila moves beyond traditional pipeline systems by adopting a new end-to-end architecture that enables full-duplex, low-latency conversations while preserving rich vocal nuances such as tone, rhythm, and emotion. It achieves a response latency of just 195 milliseconds, surpassing the average human response time. Its hierarchical multi-scale Transformer integrates the reasoning capabilities of large language models (LLMs) with powerful acoustic modeling, enabling natural, persona-aware voice generation -- where users can simply write text instructions to define the speaker's identity, tone, and other characteristics. Moreover, Voila supports over one million pre-built voices and efficient customization of new ones from brief audio samples as short as 10 seconds. Beyond spoken dialogue, Voila is designed as a unified model for a wide range of voice-based applications, including automatic speech recognition (ASR), Text-to-Speech (TTS), and, with minimal adaptation, multilingual speech translation. Voila is fully open-sourced to support open research and accelerate progress toward next-generation human-machine interactions.

Figures

Figures reproduced from arXiv: 2505.02707 by the authors.

Figure 1
Figure 1. Different paradigms of voice conversation systems: [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Voila models: (a) Voila-e2e for end-to-end voice conversation, (b) Voila-autonomous for autonomous interaction. Both models allow easy customization of speaker characteristics and voice via text instructions and audio samples. 3 Voila: Voice-Language Foundation Models As shown in [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Text and audio interleaved alignment. generated by neural codec models (Zeghidour et al., 2021; Défossez et al., 2022) with residual vector quantization (RVQ) can effectively restore sound, but they have weak semantic dependency, making it difficult for LLM training/finetuning to converge. We rather extend the approach of Zhang et al. (2023b) by distilling semantic information into the first level of tokens with RVQ… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Input embedding and output decoding in Voila. embeddings (top-left panel), whose mean is computed and fed into the backbone LLM (bottom-right panel). Finally, the audio transformer takes the output from the backbone LLM as input to predict the corresponding audio token…
Figure 5
Figure 5. Figure 5: Voila-autonomous two-stream inputs, including user’s audio stream and Voila ’s own audio stream. system prompt: one each to indicate the start, reference point, and end of the voice embedding segment. The extracted speaker voice embedding is added to the embedding of t…
Figure 6
Figure 6. Figure 6: Domain distribution in Voila Benchmark [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Performance comparison across the diverse domains in [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DialectS2S: End-to-End Speech Dialogue Modeling for Low-Resource Chinese Dialects

    cs.CL 2026-08 conditional novelty 5.0 of 10

    DialectS2S adapts an end-to-end speech dialogue model to Chinese dialects by synthesizing dialect training data and adding a self-aligned speech supervision stage that uses the model's own text predictions to build TT...

Reference graph

Works this paper leans on

69 extracted references · 29 canonical work pages · cited by 1 Pith paper

  1. [1]

    A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. Von Platen, Y. Saraf, J. Pino, et al. Xls-r: Self-supervised cross-lingual speech representation learning at scale. arXiv preprint arXiv:2111.09296, 2021

    arXiv 2021

  2. [2]

    B \'e rub \'e , M

    C. B \'e rub \'e , M. Ni en, R. Vinay, A. Geiger, T. Budig, A. Bhandari, C. R. P. Benito, N. Ibarcena, O. Pistolese, P. Li, et al. Proactive behavior in voice assistants: A systematic review and conceptual model. Computers in Human Behavior Reports, page 100411, 2024

    work page 2024

  3. [3]

    S. Bora. Breaking the silence: How voice AI is shaping the future of human-machine interactions, 2024. Medium, accessed November 8, 2024

    work page 2024

  4. [4]

    Borsos, R

    Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi, et al. Audiolm: a language modeling approach to audio generation. IEEE/ACM transactions on audio, speech, and language processing, 31: 0 2523--2533, 2023

    2023

  5. [5]

    S. Buss. Autonomous action: Self-determination in the passive mode. Ethics, 122 0 (4): 0 647--691, 2012

    work page 2012

  6. [6]

    Buyukgoz, J

    S. Buyukgoz, J. Grosinger, M. Chetouani, and A. Saffiotti. Two ways to make your robot proactive: Reasoning about human intentions or reasoning about possible futures. Frontiers in Robotics and AI, 9: 0 929267, 2022

    work page 2022

  7. [7]

    Casanova, J

    E. Casanova, J. Weber, C. D. Shulby, A. C. Junior, E. G \"o lge, and M. A. Ponti. Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone. In International Conference on Machine Learning, pages 2709--2720. PMLR, 2022

    work page 2022

  8. [8]

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herb...

    work page 2021

Show all 69 references
  1. [9]

    Y. Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919, 2023

    2023 arXiv

  2. [10]

    Y. Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y. Leng, Y. Lv, J. He, J. Lin, C. Zhou, and J. Zhou. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759, 2024

    2024 arXiv

  3. [11]

    Cobbe, V

    K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    2021 arXiv

  4. [12]

    D \'e fossez, L

    A. D \'e fossez, L. Mazar \'e , M. Orsini, A. Royer, P. P \'e rez, H. J \'e gou, E. Grave, and N. Zeghidour. Moshi: a speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00037, 2024

    2024 arXiv

  5. [13]

    Défossez, J

    A. Défossez, J. Copet, G. Synnaeve, and Y. Adi. High fidelity neural audio compression, 2022. URL https://arxiv.org/abs/2210.13438

    2022 arXiv

  6. [14]

    Faruqui and D

    M. Faruqui and D. Hakkani-T \"u r. Revisiting the boundary between asr and nlu in the age of conversational dialog systems. Computational Linguistics, 48 0 (1): 0 221--232, 2022

    2022

  7. [15]

    J. L. Flanagan. Speech analysis synthesis and perception. Springer Berlin, Heidelberg, 1972

    1972

  8. [16]

    Grosinger

    J. Grosinger. On proactive human-ai systems. In AIC, pages 140--146, 2022

    2022

  9. [17]

    S. Hao, Y. Gu, H. Ma, J. Hong, Z. Wang, D. Wang, and Z. Hu. Reasoning with language model is planning with world model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8154--8173, 2023 a

    2023

  10. [18]

    S. Hao, T. Liu, Z. Wang, and Z. Hu. ToolkenGPT : Augmenting frozen language models with massive tools via tool embeddings. arXiv preprint arXiv:2305.11554, 2023 b

    2023 arXiv

  11. [19]

    Hassid, T

    M. Hassid, T. Remez, T. A. Nguyen, I. Gat, A. Conneau, F. Kreuk, J. Copet, A. Defossez, G. Synnaeve, E. Dupoux, et al. Textually pretrained speech language models. Advances in Neural Information Processing Systems, 36, 2024

    2024

  12. [20]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021 a

    2021

  13. [21]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021 b

    2021 arXiv

  14. [22]

    N. Hoke. Digital assistants vs digital companions: What's the difference?, 2021. URL https://blog.intuitionrobotics.com/digital-assistants-vs-digital-companions-whats-the-difference. Blog post

    2021

  15. [23]

    W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units, 2021. URL https://arxiv.org/abs/2106.07447

    2021 arXiv

  16. [24]

    Huang, B

    A. Huang, B. Wu, B. Wang, C. Yan, C. Hu, C. Feng, F. Tian, F. Shen, J. Li, M. Chen, P. Liu, R. Miao, W. You, X. Chen, X. Yang, Y. Huang, Y. Zhang, Z. Gong, Z. Zhang, H. Zhou, J. Sun, B. Li, C. Feng, C. Wan, H. Hu, J. Wu, J. Zhen, R. Ming, S. Yuan, X. Zhang, Y. Zhou, B. Li, B. ...

    2025 arXiv

  17. [25]

    Huang, M

    R. Huang, M. Li, D. Yang, J. Shi, X. Chang, Z. Ye, Y. Wu, Z. Hong, J. Huang, J. Liu, Y. Ren, Z. Zhao, and S. Watanabe. Audiogpt: Understanding and generating speech, music, sound, and talking head, 2023. URL https://arxiv.org/abs/2304.12995

    2023 arXiv

  18. [26]

    Hurst, A

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  19. [27]

    H. Kim, S. Seo, K. Jeong, O. Kwon, S. Kim, J. Kim, J. Lee, E. Song, M. Oh, J.-W. Ha, S. Yoon, and K. M. Yoo. Integrating paralinguistics in speech-empowered large language models for natural conversation, 2024. URL https://arxiv.org/abs/2402.05706

    2024 arXiv

  20. [28]

    Kumar, P

    R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar. High-fidelity audio compression with improved rvqgan. Advances in Neural Information Processing Systems, 36, 2024

    2024

  21. [29]

    Kwiatkowski, J

    T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M.-W. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: A benchmark for question answering resea...

    2019 doi

  22. [30]

    Lakhotia, E

    K. Lakhotia, E. Kharitonov, W.-N. Hsu, Y. Adi, A. Polyak, B. Bolte, T.-A. Nguyen, J. Copet, A. Baevski, A. Mohamed, and E. Dupoux. Generative spoken language modeling from raw audio, 2021. URL https://arxiv.org/abs/2102.01192

    2021 arXiv

  23. [31]

    Lebourdais, M

    M. Lebourdais, M. Tahon, A. Laurent, and S. Meignier. Automatic speech interruption detection: Analysis, corpus, and system. In Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-Coling 2024), pages \`a --para \^ tre, 2024

    2024

  24. [32]

    Y. Li, H. Sun, M. Lin, T. Li, G. Dong, T. Zhang, B. Ding, W. Song, Z. Cheng, Y. Huo, S. Chen, X. Li, D. Pan, S. Zhang, X. Wu, Z. Liang, J. Liu, T. Zhang, K. Lu, Y. Zhao, Y. Shen, F. Yang, K. Yu, T. Lin, J. Xu, Z. Zhou, and W. Chen. Baichuan-omni technical report. arXiv preprin...

  25. [33]

    T.-E. Lin, Y. Wu, F. Huang, L. Si, J. Sun, and Y. Li. Duplex conversation: Towards human-like interaction in spoken dialogue systems. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3299--3308, 2022

    2022

  26. [34]

    A. H. Liu, H.-J. Chang, M. Auli, W.-N. Hsu, and J. Glass. Dinosr: Self-distillation and online clustering for self-supervised speech representation learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  27. [35]

    Maiti, Y

    S. Maiti, Y. Peng, S. Choi, J.-w. Jung, X. Chang, and S. Watanabe. Voxtlm: Unified decoder-only models for consolidating speech recognition, synthesis and speech, text continuation tasks. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Process...

    2024

  28. [36]

    Maiti, Y

    S. Maiti, Y. Peng, S. Choi, J. weon Jung, X. Chang, and S. Watanabe. Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation tasks, 2024 b . URL https://arxiv.org/abs/2309.07937

    2024 arXiv

  29. [37]

    A. S. Meyer. Timing in conversation. Journal of Cognition, 6 0 (1), 2023

    2023

  30. [38]

    Mitsui, K

    K. Mitsui, K. Mitsuda, T. Wakatsuki, Y. Hono, and K. Sawada. Pslm: Parallel generation of text and speech with llms for low-latency spoken dialogue systems. arXiv preprint arXiv:2406.12428, 2024

    2024 arXiv

  31. [39]

    Nachmani, A

    E. Nachmani, A. Levkovitch, R. Hirsch, J. Salazar, C. Asawaroengchai, S. Mariooryad, E. Rivlin, R. Skerry-Ryan, and M. T. Ramanovich. Spoken question answering and speech continuation using spectrogram-powered llm. arXiv preprint arXiv:2305.15255, 2023

    2023 arXiv

  32. [40]

    T. A. Nguyen, B. Muller, B. Yu, M. R. Costa-Jussa, M. Elbayad, S. Popuri, P.-A. Duquenne, R. Algayres, R. Mavlyutov, I. Gat, et al. Spirit-lm: Interleaved spoken and written language model. arXiv preprint arXiv:2402.05755, 2024

    2024 arXiv

  33. [41]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022

    2022

  34. [42]

    Panayotov, G

    V. Panayotov, G. Chen, D. Povey, and S. Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206--5210. IEEE, 2015

    2015

  35. [43]

    Polyak, Y

    A. Polyak, Y. Adi, J. Copet, E. Kharitonov, K. Lakhotia, W.-N. Hsu, A. Mohamed, and E. Dupoux. Speech resynthesis from discrete disentangled self-supervised representations, 2021. URL https://arxiv.org/abs/2104.00355

    2021 arXiv

  36. [44]

    Radford, J

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. Robust speech recognition via large-scale weak supervision, 2022. URL https://arxiv.org/abs/2212.04356

    2022 arXiv

  37. [45]

    Rekesh, N

    D. Rekesh, N. R. Koluguri, S. Kriman, S. Majumdar, V. Noroozi, H. Huang, O. Hrinchuk, K. Puvvada, A. Kumar, J. Balam, et al. Fast conformer with linearly scalable attention for efficient speech recognition. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (...

    2023

  38. [46]

    P. K. Rubenstein, C. Asawaroengchai, D. D. Nguyen, A. Bapna, Z. Borsos, F. de Chaumont Quitry, P. Chen, D. E. Badawy, W. Han, E. Kharitonov, H. Muckenhirn, D. Padfield, J. Qin, D. Rozenberg, T. Sainath, J. Schalkwyk, M. Sharifi, M. T. Ramanovich, M. Tagliasacchi, A. Tudor, M. ...

    2023 arXiv

  39. [47]

    R. Schafer. Scientific bases of human-machine communication by voice. Proceedings of the National Academy of Sciences of the United States of America, 92 0 (22): 0 9914--9920, 1995

    1995

  40. [48]

    Schneider, A

    S. Schneider, A. Baevski, R. Collobert, and M. Auli. wav2vec: Unsupervised pre-training for speech recognition. arXiv preprint arXiv:1904.05862, 2019

    1904 arXiv

  41. [49]

    Schroeder and N

    J. Schroeder and N. Epley. Mistaking minds and machines: How speech affects dehumanization and anthropomorphism. Journal of Experimental Psychology: General, 145 0 (11): 0 1427, 2016

    2016

  42. [50]

    Y. Shao, L. Li, J. Dai, and X. Qiu. Character-llm: A trainable agent for role-playing. arXiv preprint arXiv:2310.10158, 2023

    2023 arXiv

  43. [51]

    Y. Shen, K. Song, X. Tan, D. Li, W. Lu, and Y. Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face, 2023. URL https://arxiv.org/abs/2303.17580

    2023 arXiv

  44. [52]

    Y. Shu, S. Dong, G. Chen, W. Huang, R. Zhang, D. Shi, Q. Xiang, and Y. Shi. Llasm: Large language and speech model, 2023

    2023

  45. [53]

    G. Skantze. Turn-taking in conversational systems and human-robot interaction: a review. Computer Speech & Language, 67: 0 101178, 2021

    2021

  46. [54]

    Introducing hertz-dev, the first open-source base model for conversational audio generation, 2024

    Standard Intelligence . Introducing hertz-dev, the first open-source base model for conversational audio generation, 2024. URL https://si.inc/hertz-dev/. Blog post

    2024

  47. [55]

    Stivers, N

    T. Stivers, N. J. Enfield, P. Brown, C. Englert, M. Hayashi, T. Heinemann, G. Hoymann, F. Rossano, J. P. De Ruiter, K.-E. Yoon, et al. Universals and cultural variation in turn-taking in conversation. Proceedings of the National Academy of Sciences, 106 0 (26): 0 10587--10592, 2009

    2009

  48. [56]

    W. Tan, H. Inaguma, N. Dong, P. Tomasello, and X. Ma. Ssr: Alignment-aware modality connector for speech language models. arXiv preprint arXiv:2410.00168, 2024

    2024 arXiv

  49. [57]

    C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang. Salmonn: Towards generic hearing abilities for large language models, 2024. URL https://arxiv.org/abs/2310.13289

    2024 arXiv

  50. [58]

    C. Wang, S. Chen, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111, 2023 a

    2023 arXiv

  51. [59]

    H. Wang, C. Liang, S. Wang, Z. Chen, B. Zhang, X. Xiang, Y. Deng, and Y. Qian. Wespeaker: A research and production oriented speaker embedding learning toolkit. In ICASSP 2023, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1--5. IEEE, 2023 b

    2023

  52. [60]

    Z. M. Wang, Z. Peng, H. Que, J. Liu, W. Zhou, Y. Wu, H. Guo, R. Gan, Z. Ni, J. Yang, et al. Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. arXiv preprint arXiv:2310.00746, 2023 c

    arXiv

  53. [61]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    2022

  54. [62]

    W. Wu, H. Wu, L. Jiang, X. Liu, J. Hong, H. Zhao, and M. Zhang. From role-play to drama-interaction: An llm solution. arXiv preprint arXiv:2405.14231, 2024

    2024 arXiv

  55. [63]

    D. Yang, J. Tian, X. Tan, R. Huang, S. Liu, X. Chang, J. Shi, S. Zhao, J. Bian, X. Wu, Z. Zhao, S. Watanabe, and H. Meng. Uniaudio: An audio foundation model toward universal audio generation, 2023. URL https://arxiv.org/abs/2310.00704

    2023 arXiv

  56. [64]

    L. Yang, C. Achard, and C. Pelachaud. Multimodal analysis of interruptions. In International Conference on Human-Computer Interaction, pages 306--325. Springer, 2022

    2022

  57. [65]

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022

    2022 arXiv

  58. [66]

    Zeghidour, A

    N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30: 0 495--507, 2021

    2021

  59. [67]

    Zhang, S

    D. Zhang, S. Li, X. Zhang, J. Zhan, P. Wang, Y. Zhou, and X. Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. arXiv preprint arXiv:2305.11000, 2023 a

    2023 arXiv

  60. [68]

    Zhang, D

    X. Zhang, D. Zhang, S. Li, Y. Zhou, and X. Qiu. Speechtokenizer: Unified speech tokenizer for speech large language models. arXiv preprint arXiv:2308.16692, 2023 b

    2023 arXiv

  61. [69]

    Y. Zhu, D. Su, L. He, L. Xu, and D. Yu. Generative pre-trained speech language model with efficient hierarchical transformer, 2024. URL https://arxiv.org/abs/2406.00976

    2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.