Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Chain-of-Thought Training for Open E2E Spoken Dialogue Systems

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Chain-of-thought post-training makes end-to-end spoken dialogue trainable on 300 hours of public data, matching a text-LLM baseline.

desk verdict A useful, reproducible CoT post-training recipe for open E2E spoken dialogue, but best-of-10 Whisper-criterion selection inflates the headline speech-quality numbers. read the letter →

arxiv 2506.00722 v1 pith:DNYLA6JJ submitted 2025-05-31 cs.CL cs.SDeess.AS

Siddhant Arora , Jinchuan Tian , Hayato Futami , Jee-weon Jung , Jiatong Shi , Yosuke Kashiwagi , Emiru Tsunoo , Shinji Watanabe This is my paper
classification cs.CLcs.SDeess.AS
keywords spokendialoguesystemschain-of-thoughtend-to-endspeechmodelsmultimodallanguagepost-trainingfoundationdataefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Speech-to-speech dialogue models that skip intermediate text usually demand enormous training corpora and still reply incoherently. The paper claims that restructuring the task into a three-step chain—transcribe the user's speech, write a text response, then speak it—removes that barrier when applied as post-training to an open multimodal speech-language model already pre-trained on recognition, synthesis, and text tasks. On the public 300-hour Switchboard corpus the staged model beats a direct speech-to-speech baseline by more than 1.5 ROUGE-1 points, matches a text-language-model baseline that is given ground-truth transcripts, and does so with a single 1.7B-parameter model that is about three times smaller than the cascaded system it rivals. The authors also report higher emotional similarity to human references and synthesized speech close in quality to a dedicated single-speaker text-to-speech system. If the result holds, spoken dialogue systems become trainable on modest public datasets rather than millions of hours of proprietary audio.

What carries the argument

The mechanism is Eq. (1), the chain-of-thought (CoT) factorization: a Viterbi approximation joined with a conditional-independence assumption that decomposes speech-to-speech generation into three sequential autoregressive stages—ASR, text-response generation, and TTS—conditioned on the previous stages' outputs. Inference follows the same order: greedily decode the transcript, sample the text response, then sample the speech output using the generated transcript and response, all in a shared discrete token space so a single decoder can execute the whole chain. This is what aligns post-training with the model's pre-training objectives and is what the paper credits for faster convergence, lower compute, and better data efficiency.

What would settle it

Fine-tune the same pre-trained modules as a cascaded ASR–text-LLM–TTS pipeline on the same Switchboard data and compare semantic and audio-quality scores with the CoT E2E model; if the fine-tuned cascade matches the CoT model, the gains come from fine-tuning itself, not from the chain-of-thought structure. A second check: feed increasingly corrupted ASR transcripts into the text and speech stages and see whether response quality tracks WER—if it degrades proportionally, the later stages do not repair ASR errors as claimed.

Watch

Extended reading notes

Core claim

The core claim is that the posterior probability of a spoken response given the user's speech and a speaker prompt, $P(Y|X,X_{\mathrm{spk}})$, can be factored through two intermediate discrete variables—the ASR transcript and the text response—using a Viterbi-style argmax and a conditional-independence assumption, giving $P(Y|X,X_{\mathrm{spk}}) \approx P(Y|X,X_{\mathrm{spk}}, \hat{S}_{\mathrm{res}}, \hat{S}_{\mathrm{asr}})$ with $\hat{S}_{\mathrm{asr}} = \arg\max P(S_{\mathrm{asr}}|X)$ and $\hat{S}_{\mathrm{res}} = \arg\max P(S_{\mathrm{res}}|X,X_{\mathrm{spk}},S_{\mathrm{asr}})$. Training a single decoder-only SpeechLM on real human-human conversations with this chain, teacher-forcing the intermediate targets and computing loss only on targets, yields semantically coherent responses that match a strong text-LLM baseline while keeping full end-to-end differentiability. The paper shows the approach works with as little as 300 hours of public Switchboard data, with particularly large coherence gains over direct speech-to-speech training, and argues the efficiency comes from each stage mirroring the model's pre-training tasks.

Load-bearing premise

The premise that carries the argument is that the single best ASR transcript, chosen by stage-wise argmax, preserves the information the later stages need from the user's speech, so any recognition error that the text and speech stages cannot repair is information lost compared with direct speech-to-speech modelling.

Editorial extensions

If this is right

  • A spoken dialogue system with text-LLM-level response quality can be trained on roughly 300 hours of public conversational speech, not the millions of hours typically assumed.
  • The same pre-trained speech-language model handles recognition, response generation, and synthesis in one 1.7B-parameter network, roughly a third the size of the cascaded systems it matches.
  • Training on real human-human dialogues yields responses more emotionally similar to human references, suggesting the chain preserves paralinguistic cues that text-only cascades drop.
  • Because each stage is an explicit intermediate, errors can be monitored and post-processed stage by stage, which improves final speech quality.
  • The alignment of each stage with the model's pre-training tasks makes the recipe transferable to future architectures without new task tokens.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A fine-tuned cascaded baseline—the paper compares only against zero-shot task-specific modules—would separate the benefit of fine-tuning from the benefit of the CoT structure itself.
  • The three-stage factorization suggests a natural extension to multi-turn dialogue: keep the conversation history in the prompt and check whether the conditional-independence assumption still holds when context accumulates.
  • The same intermediate-token recipe could be applied to other speech-to-text-to-speech tasks such as speech translation or summarization by changing the middle stage's prompt.
  • Since the ablation shows ASR quality drops slightly after CoT training while response quality stays stable, deliberately corrupting the transcript input would test how much error repair the text stage genuinely performs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a chain-of-thought (CoT) post-training formulation for open end-to-end (E2E) spoken dialogue systems. Instead of directly predicting the output speech from input speech and a speaker prompt, the system is trained to first generate an ASR transcript, then a text response, and finally the speech response, with all stages sharing one SpeechLM. The method is evaluated on Switchboard and Fisher human-human conversational data, and the paper reports that the CoT model improves semantic coherence (ROUGE, METEOR, perplexity) over a one-step E2E baseline, matches a text-only SmolLM baseline and a VITS TTS system in quality, captures emotion better in conversation-level analysis, and is parameter-efficient. The authors state they will release models and code.

Significance. If the reported results are accepted, the contribution is practically significant: it offers a simple, compute-efficient recipe for making open E2E spoken dialogue trainable on public data (about 300 hours of Switchboard) with a 1.7B-parameter model, and it connects spoken dialogue post-training to standard ASR/LLM/TTS task formats. The paper's strengths include the use of real human-human conversations, ablations for the intermediate ASR target, speaker prompt, and post-processing, statistical significance tests, and a clearly stated plan to release code and models. The principal weakness is that the headline speech-output metrics are obtained under a best-of-10 selection criterion that is the same Whisper-based intelligibility criterion later used for evaluation, so the evidence for the central speech-quality claim is currently partly circular.

major comments (3)
  1. [Sec. 4.2, Tables 2-3] The reported speech-output numbers are produced by an oracle selection step that optimizes the evaluation criterion. Section 4.2 states that after sampling, the decoder performs 'post-process the outputs, computing top-10 samples and selecting the speech with the highest intelligibility to generated text response Ŝ_res (Eq. 1)', and intelligibility is measured through Whisper. Tables 2 and 3 then evaluate the selected speech by transcribing it with Whisper and computing ROUGE/METEOR/WER, with UTMOS computed on the same Whisper-selected samples. The selected sample is therefore not a typical model output; the 'w/o Post-process' row in Table 3 (SWBD WER 42.0, UTMOS 2.10 versus 13.6/3.55 with post-processing) shows the gap is large. Because the SpeechLM E2E baseline is decoded without this selection, the ROUGE-1 improvement (14.2 vs 10.5 in Table 2) and the VITS-matching audio claim in Table 3 do not establish that CoT training itself improves the final spoken output. Please report unselected single-sample outputs, decouple the selection criterion from the evaluation metric, and apply the same selection procedure to all baselines if selection is kept.
  2. [Sec. 4.2 / Sec. 5, Table 3] The claim that intelligibility cannot be computed for the E2E baseline is inconsistent with the paper's own evaluation protocol. Section 4.1 says semantic quality is evaluated using human references as ground truth and Whisper transcriptions for E2E models; Section 5 then says 'For the E2E SpeechLM, we cannot compute intelligibility due to the absence of ground-truth text references.' If human reference responses exist for ROUGE/METEOR, they can also serve as WER references; if they do not, the semantic evaluation of the E2E baseline in Table 2 is itself unexplained. The table's ✗ entries should be justified or the WER comparison should be reported with a clearly defined reference.
  3. [Sec. 3.1, Eq. (1); Sec. 5, Table 2] The robustness of the CoT factorization to ASR errors is only demonstrated for the intermediate text response, not for the final speech output. The ablation 'using GT Transcript Ŝ_asr' in Table 2 is reported only for Ŝ_res; since the final speech decoder conditions on Ŝ_asr and Ŝ_res, ASR hallucination could propagate differently into the speech channel. Reporting the final speech WER and semantic metrics with ground-truth versus predicted Ŝ_asr would close this gap and directly support the claim that the staged decoding reduces error propagation.
minor comments (4)
  1. [Sec. 3.1, Eq. (1)] The notation uses both P and p for the same distributions (e.g., P(Y|...) vs p(S_res|...)) and it is not clear whether X_spk conditions the ASR stage; please make the notation uniform and explicit.
  2. [Sec. 4.2] The post-processing that 'removes hallucinations' is not described; please provide the exact rule or cite the code, since it affects all reported metrics.
  3. [Table 3] Format the E2E rows as '✗ / 2.03' rather than '✗2.03' to avoid ambiguity.
  4. [Abstract] The sentence 'existing E2E approaches often require large-scale training data and generates responses lacking semantic coherence' has a subject-verb agreement error; also, the abstract should make clear which results use 300 hours of Switchboard alone versus the combined Switchboard+Fisher training set.

Circularity Check

1 steps flagged · score 6.0 of 10

Best-of-10 speech selection (Sec. 4.2) optimizes the same Whisper-based intelligibility criterion used to score the speech output (Sec. 4.1), making the reported WER 13.6/12.5, UTMOS 3.55/3.32, and speech ROUGE rows selected-sample statistics; the w/o Post-process ablation (WER 42.0, UTMOS 2.10) shows typical outputs are far worse, confounding headline gains over the unselected E2E baseline.

  1. fitted input called prediction [Sec. 3.4/4.2 (inference post-processing), Sec. 4.1 (evaluation protocol), Tables 2-3 and Sec. 5 (results)]
    "For speech response generation, we apply top-k sampling (same as text response), and further post-process the outputs, computing top-10 samples and selecting the speech with the highest intelligibility to generated text response S_hat_res (Eq. 1). ... For E2E models, semantic quality is evaluated by transcribing synthesized speech Y_hat using Whisper large [6]."

    The reported metrics for the final speech output are computed on Whisper transcriptions of Y_hat (Sec. 4.1: ROUGE/METEOR via 'transcribing synthesized speech Y_hat using Whisper large'; WER via 'measuring intelligibility similarly through Whisper hypotheses'). Sec. 4.2's post-processing selects, from 10 samples, the speech with highest Whisper-based intelligibility to the predicted text S_hat_res. Hence the reported WER (13.6/12.5) is the best-of-10 value of the very criterion being reported, and the ROUGE/METEOR of Y_hat is forced to track the text-response scores: Table 2's speech and text rows are nearly identical (14.2/1.3/10.3 vs 14.2/1.3/10.2; 17.6/3.2/12.3 vs 17.6/3.2/12.2). The 'w/o Post-process' ablation (WER 42.0, UTMOS 2.10) shows unselected outputs are far worse.

full rationale

Eq. (1) is a legitimate Viterbi/C.I. decomposition of P(Y|X,Xspk) and is not circular. The text-response rows (Table 2, S_hat_res) are genuine model outputs and show a real semantic improvement over the SpeechLM E2E baseline even in the unselected 'w/o Post-process' row (12.3 vs 10.5 ROUGE-1 on SWBD). Citations to the authors' own toolkits ([22] ESPnet-SpeechLM, [44] ESPnet, [49] ESPnet-SDS) are infrastructure references, not load-bearing theoretical claims. However, one load-bearing evaluation choice is circular: the decoding-time selection (Sec. 4.2) picks the best of 10 speech samples by Whisper-based intelligibility to the predicted text, while the reported speech-output quality/intelligibility (Sec. 4.1: ROUGE/METEOR via Whisper transcription, WER via Whisper hypotheses, UTMOS on the selected sample) characterizes only that selected sample. The selection directly optimizes the reported criterion, so Table 3's WER 13.6/12.5 and the speech-semantic rows of Table 2 are best-of-10 statistics. The ablation shows this is decisive: without post-processing, WER rises to 42.0/36.5 and UTMOS drops to 2.10, near the E2E baseline (2.03). Consequently the headline comparisons — 'over 1.5 ROUGE-1 improvement over the baseline' and matching LJSpeech VITS — conflate oracle-style sample selection with the CoT training contribution. This is partial circularity: the speech-output predictions reduce by construction to the selection rule, while the CoT formulation and text-intermediate results retain independent content.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical, architectural, or data entities; its load-bearing assumptions are the factorization in Eq. 1, alignment with pre-training, and Whisper-based content evaluation. The listed free parameters are decoding and preprocessing choices rather than fitted model parameters, but they directly affect the reported results.

free parameters (4)
  • top-k response sampling (k=30, temperature=0.8) = k=30, temperature=0.8
    Chosen for text response decoding in Sec 4.2; affects ROUGE, METEOR, and perplexity; no tuning curves reported.
  • top-10 speech sample selection by intelligibility = 10 samples, Whisper-based selection
    Post-processing in Sec 4.2 selects the generated speech with highest intelligibility to the text response; this inflates reported WER and UTMOS.
  • data preprocessing thresholds = remove utterances under 5 words; truncate to 30 seconds
    Applied before training in Sec 4.1; changes content distribution and affects response quality; chosen by hand.
  • high-quality speaker prompt = LibriSpeech prompt with UTMOS approximately 4.5
    Replacing Switchboard speaker prompts with a cleaner prompt during inference yields substantial gains in Table 3; this is a prompt choice, not a model parameter.
assumptions (3)
  • domain assumption Sequential Viterbi approximation and conditional independence justify factorizing P(Y|X,Xspk) into ASR, text response, and TTS stages as in Eq. 1.
    Invoked in Sec 3.1. If greedy stage-wise argmax is a poor approximation, CoT inference may lose information that direct E2E modeling preserves.
  • domain assumption SpeechLM pre-training tasks (ASR, TTS, text LM) align with the CoT stage prompts, enabling efficient post-training.
    Central efficiency claim in Sec 3.1 and Sec 3.3; not independently measured except through final task metrics.
  • domain assumption Whisper Large transcriptions are a valid proxy for semantic content of synthesized speech.
    Used in Sec 4.1 to evaluate semantic quality of E2E and CoT speech outputs; Whisper errors propagate into ROUGE and METEOR scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Chain-of-Thought Training for Open E2E Spoken Dialogue Systems." pith.science (2026). https://pith.science/paper/DNYLA6JJ

@misc{pith2026250600722,
  author       = {Pith},
  title        = {Pith review of: Chain-of-Thought Training for Open E2E Spoken Dialogue Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DNYLA6JJ}},
  note         = {Machine review of arXiv:2506.00722}
}
read the original abstract

Unlike traditional cascaded pipelines, end-to-end (E2E) spoken dialogue systems preserve full differentiability and capture non-phonemic information, making them well-suited for modeling spoken interactions. However, existing E2E approaches often require large-scale training data and generates responses lacking semantic coherence. We propose a simple yet effective strategy leveraging a chain-of-thought (CoT) formulation, ensuring that training on conversational data remains closely aligned with the multimodal language model (LM)'s pre-training on speech recognition~(ASR), text-to-speech synthesis (TTS), and text LM tasks. Our method achieves over 1.5 ROUGE-1 improvement over the baseline, successfully training spoken dialogue systems on publicly available human-human conversation datasets, while being compute-efficient enough to train on just 300 hours of public human-human conversation data, such as the Switchboard. We will publicly release our models and training code.

Figures

Figures reproduced from arXiv: 2506.00722 by the authors.

Figure 1
Figure 1. Training schematic for our CoT-based E2E spoken dialogue system, compared with cascaded and conventional E2E systems. Our approach employs multi-stage reasoning, improving semantic coherence and speech quality while preserving E2E differentiability. We use teacher forcing during training, and multi-stage decoding using generated intermediate outputs for inference (Sec. 3.4). training process. While OMNI-Flatten [32]… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Efficient Chain-of-Modality Reasoning via Progressive Compression for Spoken Language Models

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Spoken math models that emit a 40%-compressed reasoning trace between question and answer beat full-reasoning baselines by ~3 accuracy points while using roughly one third of the text tokens.

Reference graph

Works this paper leans on

56 extracted references · 42 canonical work pages · cited by 1 Pith paper

  1. [1]

    De- spite their growing importance, building effective spoken dia- logue systems remains a challenging task due to the complexity of human communication

    Introduction Spoken dialogue systems [1, 2] are designed to engage in natu- ral and interactive conversations with end users, playing a crit- ical role in voice assistants and intelligent home devices. De- spite their growing importance, building effective spoken dia- logue systems remains a challenging task due to the complexity of human communication. T...

    work page

  2. [2]

    Chain-of-Thought Training for Open E2E Spoken Dialogue Systems

    Related studies Recently, “Chat” SpeechLMs [23] have gained attention for their ability to engage in natural and interactive conversations. Early efforts [24–26] developed speech-aware LMs that gener- ated text responses from spoken input but relied on external TTS and V AD systems for spoken dialogue. More recently, speech- to-speech LMs [27–31] have eme...

    work page Pith review arXiv 2025

  3. [3]

    tar- get

    Method We begin our problem formulation with a single-turn spoken di- alog system, whereXandYrepresent the user’s and system’s speech feature sequences. Traditional cascaded spoken dia- logue systems employ separate modules for sub-tasks, as shown in Fig. 1a. These systems typically consist of an ASR model, which optimizesP(S asr|X)to produce the ASR tran...

    work page

  4. [4]

    Spk prompt

    Experiments 4.1. Datasets For our experiments, we focus exclusively onrealhuman- human conversation datasets, selecting 2 widely used corpora: Switchboard [11] (≈300 hours) and Fisher [34] (≈2000 hours). For Switchboard, we use the Eval2000 dataset for eval- uation, while for Fisher, we follow Dialog GSLM [28], split- ting the dataset into 98:1:1 for trai...

    work page 2000

  5. [5]

    SpeechLM E2E

    Results and Discussion Semantic Coherence and Audio Quality Results: Tab. 2 presents the semantic quality of responses. Even state-of-the- art LLM SmolLM2 struggles to achieve high ROUGE and ME- TEOR scores (Tab. 2), reflecting the spontaneity and unpre- dictability of human-human conversations. Despite this, our CoT E2E model generates semantically coher...

    work page

  6. [6]

    speaking while listening

    Conclusion We propose a CoT-based formulation for E2E spoken dialogue systems, achieving competitive performance with task-specific baselines, while producing more coherent responses with su- perior audio quality than conventional E2E models. Our CoT- based approach also surpasses cascaded systems in parame- ter efficiency and emotional expressiveness. Fu...

    work page

  7. [7]

    Acknowledgement Experiments of this work used the Bridges2 system at PSC and Delta system at NCSA through allocations CIS210014 and IRI120008P from the Advanced Cyberinfrastructure Co- ordination Ecosystem: Services & Support (ACCESS) pro- gram, supported by National Science Foundation grants #2138259,#:2138286, #:2138307, #:2137603, and #:2138296

    work page

  8. [8]

    Jokinen et al.,Spoken dialogue systems

    K. Jokinen et al.,Spoken dialogue systems. Morgan & Claypool Publishers, 2009

    work page 2009

Show all 56 references
  1. [9]

    Social robots that interact with people,

    C. Breazeal et al., “Social robots that interact with people,” Springer handbook of robotics, pp. 1349–1369, 2008

    2008

  2. [10]

    Challenges for spoken dialogue systems,

    J. Glass, “Challenges for spoken dialogue systems,” inProceed- ings of the 1999 IEEE ASRU Workshop, MIT Laboratory for Computer Science Cambridge, vol. 696, 1999

    1999

  3. [11]

    Audiogpt: Understanding and generating speech, music, sound, and talking head,

    R. Huang et al., “Audiogpt: Understanding and generating speech, music, sound, and talking head,” inProceedings of the AAAI, 2024, pp. 23 802–23 804

    2024

  4. [12]

    Wiseman,Py-webrtcvad, Accessed: 2024-12-10, 2024

    J. Wiseman,Py-webrtcvad, Accessed: 2024-12-10, 2024

    2024

  5. [13]

    Robust speech recognition via large-scale weak supervision,

    A. Radford et al., “Robust speech recognition via large-scale weak supervision,” inProc. ICML, 2023

    2023

  6. [14]

    Reproducing whisper-style training using an open-source toolkit and publicly available data,

    Y . Peng et al., “Reproducing whisper-style training using an open-source toolkit and publicly available data,” inProc. ASRU, 2023

    2023

  7. [15]

    Dialoglue: A natural language understand- ing benchmark for task-oriented dialogue,

    S. Mehri et al., “Dialoglue: A natural language understand- ing benchmark for task-oriented dialogue,”arXiv preprint arXiv:2009.13570, 2020

    2009 arXiv

  8. [16]

    Few-shot natural language generation for task- oriented dialog,

    B. Peng et al., “Few-shot natural language generation for task- oriented dialog,” inFindings of EMNLP 2020, T. Cohn et al., Eds., Online: Association for Computational Linguistics, Nov. 2020, pp. 172–182

    2020

  9. [17]

    Coqui,Introducing open-xtts: An open-source toolkit for tts, Ac- cessed: 2024-12-10, 2024

    2024

  10. [18]

    Switchboard: Telephone speech corpus for research and development,

    J. Godfrey et al., “Switchboard: Telephone speech corpus for research and development,” in[Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Sig- nal Processing, vol. 1, 1992, 517–520 vol.1

    1992

  11. [19]

    Towards empathetic open-domain conversa- tion models: A new benchmark and dataset,

    H. Rashkin, “Towards empathetic open-domain conversa- tion models: A new benchmark and dataset,”arXiv preprint arXiv:1811.00207, 2018

    2018 arXiv

  12. [20]

    Prediction of turn-taking using multitask learn- ing with prediction of backchannels and fillers,

    K. Hara et al., “Prediction of turn-taking using multitask learn- ing with prediction of backchannels and fillers,”Listener, vol. 162, p. 364, 2018

    2018

  13. [21]

    Prosodic features which cue back-channel re- sponses in english and japanese,

    N. Ward et al., “Prosodic features which cue back-channel re- sponses in english and japanese,”Journal of pragmatics, vol. 32, no. 8, pp. 1177–1207, 2000

    2000

  14. [22]

    Automatic acoustic synthesis of human- like laughter,

    S. Sundaram et al., “Automatic acoustic synthesis of human- like laughter,”The Journal of the Acoustical Society of America, vol. 121, no. 1, pp. 527–535, 2007

    2007

  15. [23]

    A conversation robot with back-channel feed- back function based on linguistic and nonlinguistic informa- tion,

    S. Fujie et al., “A conversation robot with back-channel feed- back function based on linguistic and nonlinguistic informa- tion,” inProc. ICARA Int. Conference on Autonomous Robots and Agents, Citeseer, 2004, pp. 379–384

    2004

  16. [24]

    Speechgpt: Empowering large language mod- els with intrinsic cross-modal conversational abilities,

    D. Zhang et al., “Speechgpt: Empowering large language mod- els with intrinsic cross-modal conversational abilities,”arXiv preprint arXiv:2305.11000, 2023

    2023 arXiv

  17. [25]

    Xie et al.,Mini-omni: Language models can hear, talk while thinking in streaming, 2024

    Z. Xie et al.,Mini-omni: Language models can hear, talk while thinking in streaming, 2024

    2024

  18. [26]

    Moshi: A speech-text foundation model for real-time dialogue,

    A. D ´efossez et al., “Moshi: A speech-text foundation model for real-time dialogue,” Kyutai, Tech. Rep., 2024

    2024

  19. [27]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei et al., “Chain-of-thought prompting elicits reasoning in large language models,”Advances in neural information pro- cessing systems, vol. 35, pp. 24 824–24 837, 2022

    2022

  20. [28]

    Automatic chain of thought prompting in large language models,

    Z. Zhang et al., “Automatic chain of thought prompting in large language models,”arXiv preprint arXiv:2210.03493, 2022

    2022 arXiv

  21. [29]

    ESPnet-SpeechLM: An open speech language model toolkit,

    J. Tian et al., “ESPnet-SpeechLM: An open speech language model toolkit,”arXiv, 2024

    2024

  22. [30]

    Ji et al.,Wavchat: A survey of spoken dialogue models, 2024

    S. Ji et al.,Wavchat: A survey of spoken dialogue models, 2024

    2024

  23. [31]

    Qwen-audio: Advancing universal audio under- standing via unified large-scale audio-language models,

    Y . Chu et al., “Qwen-audio: Advancing universal audio under- standing via unified large-scale audio-language models,”arXiv preprint arXiv:2311.07919, 2023

    2023 arXiv

  24. [32]

    Vita: Towards open-source interactive omni multi- modal llm,

    C. Fu et al., “Vita: Towards open-source interactive omni multi- modal llm,”arXiv preprint arXiv:2408.05211, 2024

    2024 arXiv

  25. [33]

    Distilling an end-to-end voice assistant without instruction training data,

    W. Held et al., “Distilling an end-to-end voice assistant without instruction training data,”arXiv:2410.02678, 2024

    2024 arXiv

  26. [34]

    Speechgpt-gen: Scaling chain-of-information speech generation,

    D. Zhang et al., “Speechgpt-gen: Scaling chain-of-information speech generation,”arXiv preprint arXiv:2401.13527, 2024

    2024 arXiv

  27. [35]

    Generative spoken dialogue language mod- eling,

    T. A. Nguyen et al., “Generative spoken dialogue language mod- eling,”Trans. Assoc. Comput. Linguistics, vol. 11, pp. 250–266, 2023

    2023

  28. [36]

    Beyond turn-based interfaces: Syn- chronous llms as full-duplex dialogue agents,

    B. Veluri et al., “Beyond turn-based interfaces: Syn- chronous llms as full-duplex dialogue agents,”arXiv preprint arXiv:2409.15594, 2024

    2024 arXiv

  29. [37]

    Llama-omni: Seamless speech interaction with large language models,

    Q. Fang et al., “Llama-omni: Seamless speech interaction with large language models,”arXiv preprint arXiv:2409.06666, 2024

    2024 arXiv

  30. [38]

    Parrot: Autoregressive spoken dialogue lan- guage modeling with decoder-only transformers,

    Z. Meng et al., “Parrot: Autoregressive spoken dialogue lan- guage modeling with decoder-only transformers,” inAudio Imagination: NeurIPS 2024 Workshop AI-Driven Speech, Mu- sic, and Sound Generation

    2024

  31. [39]

    Omniflatten: An end-to-end gpt model for seamless voice conversation,

    Q. Zhang et al., “Omniflatten: An end-to-end gpt model for seamless voice conversation,”arXiv preprint arXiv:2410.17799, 2024

    2024 arXiv

  32. [40]

    Audiolm: A language modeling approach to audio generation,

    Z. Borsos et al., “Audiolm: A language modeling approach to audio generation,”IEEE/ACM TASLP, vol. 31, pp. 2523–2533, 2023

    2023

  33. [41]

    The fisher corpus: A resource for the next gener- ations of speech-to-text.,

    C. Cieri et al., “The fisher corpus: A resource for the next gener- ations of speech-to-text.,” inLREC, vol. 4, 2004, pp. 69–71

    2004

  34. [42]

    Rouge: A package for automatic evaluation of sum- maries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of sum- maries,” inText summarization branches out, 2004, pp. 74–81

    2004

  35. [43]

    Meteor: An automatic metric for mt evalua- tion with improved correlation with human judgments,

    S. Banerjee et al., “Meteor: An automatic metric for mt evalua- tion with improved correlation with human judgments,” inACL workshop on intrinsic and extrinsic evaluation measures for ma- chine translation and/or summarization, 2005, pp. 65–72

    2005

  36. [44]

    Perplexity—a measure of the difficulty of speech recognition tasks,

    F. Jelinek et al., “Perplexity—a measure of the difficulty of speech recognition tasks,”The Journal of the Acoustical Soci- ety of America, vol. 62, no. S1, S63–S63, 1977

    1977

  37. [45]

    Language models are unsupervised multitask learners,

    A. Radford et al., “Language models are unsupervised multitask learners,”OpenAI, 2019

    2019

  38. [46]

    Versa: A versatile evaluation toolkit for speech, audio, and music,

    J. Shi et al., “Versa: A versatile evaluation toolkit for speech, audio, and music,”arXiv preprint arXiv:2412.17667, 2024

    2024 arXiv

  39. [47]

    UTMOS: UTokyo-SaruLab system for voice- MOS challenge 2022,

    T. Saeki et al., “UTMOS: UTokyo-SaruLab system for voice- MOS challenge 2022,” inInterspeech, 2022, pp. 4521–4525

    2022

  40. [48]

    Emotion2vec: Self-supervised pre-training for speech emotion representation,

    Z. Ma et al., “Emotion2vec: Self-supervised pre-training for speech emotion representation,”Proc. ACL 2024 Findings, 2024

    2024

  41. [49]

    L. B. Allal et al.,Smollm2: When smol goes big – data-centric training of a small language model, 2025

    2025

  42. [50]

    Espnet-tts: Unified, reproducible, and in- tegratable open source end-to-end text-to-speech toolkit,

    T. Hayashi et al., “Espnet-tts: Unified, reproducible, and in- tegratable open source end-to-end text-to-speech toolkit,” in ICASSP, IEEE, 2020, pp. 7654–7658

    2020

  43. [51]

    ESPnet: End-to-end speech processing toolkit,

    S. Watanabe et al., “ESPnet: End-to-end speech processing toolkit,” inProceedings of Interspeech, 2018, pp. 2207–2211

    2018

  44. [52]

    Espnet-slu: Advancing spoken language under- standing through espnet,

    S. Arora et al., “Espnet-slu: Advancing spoken language under- standing through espnet,” inICASSP, IEEE, 2022, pp. 7167– 7171

    2022

  45. [53]

    Simple and controllable music generation,

    J. Copet et al., “Simple and controllable music generation,”Ad- vances in Neural Information Processing Systems, vol. 36, 2024

    2024

  46. [54]

    Espnet-codec: Comprehensive training and eval- uation of neural codecs for audio, music, and speech,

    J. Shi et al., “Espnet-codec: Comprehensive training and eval- uation of neural codecs for audio, music, and speech,”arXiv preprint arXiv:2409.15897, 2024

    2024 arXiv

  47. [55]

    Towards robust speech representation learning for thousands of languages,

    W. Chen et al., “Towards robust speech representation learning for thousands of languages,”arXiv preprint arXiv:2407.00837, 2024

    2024 arXiv

  48. [56]

    ESPnet-SDS: Unified toolkit and demo for spoken dialogue systems,

    S. Arora et al., “ESPnet-SDS: Unified toolkit and demo for spoken dialogue systems,” inNAACL (System Demonstrations), Apr. 2025, pp. 248–259

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.