Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Speech model learns when to listen and speak without audio codecs

desk verdict SALMONN-omni's codec-free full-duplex architecture is a genuinely new idea worth referee time, but the headline 35.9% gain is not yet interpretable because the QA evaluation sets overlap with the training data. read the letter →

arxiv 2505.17060 v1 pith:4WFUSRLD submitted 2025-05-17 cs.CL cs.AI

Wenyi Yu , Siyin Wang , Xiaoyu Yang , Xianzhao Chen , Xiaohai Tian , Jun Zhang , Guangzhi Sun , Lu Lu
show 2 more authors
Yuxuan Wang Chao Zhang
This is my paper · ORCID
classification cs.CLcs.AI
keywords full-duplexspeechLLMcodec-freeinteractionturn-takingpredictionbarge-inbackchannelingdynamicthinkingtokensreinforcementlearningfordialoguestreamingsynthesis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces SALMONN-omni, a speech-enabled large language model built to hold a full-duplex conversation: it can listen to a user, decide when to speak, generate speech, and keep taking in sound while its own voice is playing. The central claim is that no audio codec tokens need to be inserted into the LLM's vocabulary to do this. Instead, a streaming speech encoder and a streaming speech synthesizer are attached to a single LLM through embeddings, and the model is trained to emit two special tokens, and , as ordinary text tokens to mark listening, speaking, and transitions between them. On spoken question-answering and open-domain dialogue benchmarks, the authors report an average 35.9% relative improvement over prior open-source full-duplex models when the model must predict its own turn-taking timing, and performance competitive with half-duplex models trained on far more data.

What carries the argument

The central mechanism is the explicit 'thinking' strategy, a pair of special tokens <think> and <shift> generated by the LLM as ordinary output tokens, synchronized to 80 ms time blocks of audio. <think> stands for the listening state (or the silence while the model waits after finishing a response), and <shift> marks both listening-to-speaking and speaking-to-listening transitions. Because these tokens are generated and fed back into the LLM's input sequence, the model learns turn-taking, barge-in acceptance, and backchanneling as autoregressive text prediction, which aligns with the LLM's native operation and avoids the modality gap and data hunger of codec-injection approaches.

What would settle it

Construct a spoken QA evaluation set from questions published after the model's training data was collected, and re-run the predicted turn-taking comparison against Moshi and Freeze-Omni; if the 35.9% average relative improvement disappears or shrinks sharply, memorization rather than full-duplex capability explains the result. A cheaper check: search the training QA sets for exact or near-duplicate strings of the evaluation questions and measure the overlap rate.

Watch

Extended reading notes

Core claim

The paper's discovery is that a single, standalone LLM can serve as a full-duplex conversational agent if it is trained to treat dialogue-state transitions as part of its own token generation. SALMONN-omni interleaves environment-stream speech embeddings, assistant-stream (own output) speech embeddings, and text tokens into one autoregressive sequence, divided into fixed 80 ms blocks. In each block the LLM either predicts the special token <think> while listening, generates a <shift> to mark a state change, or produces answer tokens while speaking; the streaming synthesizer converts the LLM's hidden embeddings into speech. This explicit 'thinking' strategy outperforms an implicit variant, and incorporating the assistant stream raises turn-taking success from roughly 70% to 90%. The result, in the authors' phrase, is that the LLM is 'secretly a full-duplex predictor': no external voice-activity detector, interruption predictor, or second LLM process is needed.

Load-bearing premise

The reported gains assume the evaluation benchmarks are not contaminated by training data: the QA training set includes Web Questions, TriviaQA, and Alpaca-52k, and evaluation uses Web Questions, a TriviaQA subset, and AlpacaEval, with no deduplication or held-out split reported; if questions overlap, the accuracy and 35.9% improvement could reflect memorization rather than full-duplex skill.

Editorial extensions

If this is right

  • In predicted turn-taking mode, the reported numbers put SALMONN-omni ahead of Moshi and Freeze-Omni on all four benchmarks, with an average 35.9% relative gain.
  • Under oracle turn-taking, SALMONN-omni is competitive with turn-based speech LLMs such as Qwen2.5-Omni and miniCPM-o while using less training data.
  • Explicit thinking beats implicit thinking, and adding the assistant stream is what pushes turn-taking success from roughly 70% to 90%.
  • Applying DPO after supervised training raises the overall barge-in/backchanneling F1 from 0.86 to 0.90, at the cost of an early conservative phase where the model almost never allows itself to be interrupted.
  • The three-stage training recipe—encoder connection, synthesizer connection, then RL—shows that full-duplex behavior can be learned without codec tokens in the LLM vocabulary.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 'LLM as full-duplex predictor' idea should generalize: any autoregressive model with access to stream-aligned timestamps could learn state transitions the same way, including text-only agents that need to decide when to yield the floor.
  • Because the model hears its own output through the assistant stream, the same training signal could make echo cancellation and self-speech recognition emergent rather than engineered; this is a testable extension the paper partially demonstrates.
  • The reported DPO dynamics—precision rising while recall collapses, then recovering—suggest a reward-design hazard: naive interruption penalties teach silence first. Fine-grained rewards that separately credit appropriate barge-in and backchannel responses may train faster.
  • A direct stress test would be to vary the 80 ms block size and the 4-token-to-12-speech-token ratio, since the claimed naturalness depends on this synchronization granularity.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces SALMONN-omni, a full-duplex speech LLM that avoids injecting audio codec tokens into the LLM vocabulary by combining a streaming Mamba speech encoder, a frozen Llama-3-8B-Instruct backbone with LoRA, and a streaming CosyVoice2-based synthesizer. A periodic synchronization mechanism and special tokens <think> and <shift> are introduced to let the LLM learn when to start and stop speaking. The model is trained in three stages: encoder connection, end-to-end SFT with synthesizer, and DPO for dialogue dynamics. The authors report a 35.9% average relative improvement over Moshi and Freeze-Omni in predicted turn-taking, competitiveness with half-duplex models, and strong performance in barge-in/backchanneling evaluations.

Significance. If the empirical claims survive a decontamination check, the paper makes a valuable architectural contribution: it demonstrates that a single LLM can perform full-duplex conversation without codec tokens, using a lightweight 'thinking' mechanism for state transitions, and it is the first to apply DPO to full-duplex dialogue dynamics. The method is clearly described and the ablation study in Appendix C comparing thinking-strategy variants is informative. However, the central quantitative claim is currently not supported because the evaluation benchmarks overlap with the training data and because the comparison omits some systems it categorizes as full-duplex.

major comments (3)
  1. [Section 4.2/Table 7 and Section 4.3] Web Questions (4k samples) and TriviaQA (58k samples) are listed among the QA training data, while Web Questions and a 1000-sample TriviaQA subset from OpenAudioBench are used as evaluation benchmarks in Section 4.3. The manuscript does not report deduplication, held-out splits, or overlap statistics for these sets; the sentence at the end of Section 4.3 stating that evaluation sentences differ from training sentences refers only to the barge-in/backchanneling evaluation, not to the QA benchmarks. Because the headline 35.9% relative improvement is computed on these benchmarks, the result may reflect memorization of the training questions rather than full-duplex competence. The authors should quantify the overlap, exclude overlapping items, or re-evaluate on non-overlapping benchmarks before the SOTA claim can be accepted.
  2. [Table 3 and Section 4.3] The predicted-turn-taking comparison includes only Moshi and Freeze-Omni, yet Table 1 classifies VITA-1.5 and MinMo as full-duplex models as well; the paper does not explain why they are excluded from this setting. Additionally, the reported 35.9% average relative improvement is the mean over eight cells (four datasets by two metrics) with individual relative gains ranging from 2.8% (AlpacaEval S2T) to 96.5% (TriviaQA S2S); this averaging masks large heterogeneity and should be reported with per-cell values and a clear statement of the averaging procedure.
  3. [Section 5.3 and Section 4.3] None of the reported numbers in Tables 3, 4, 5, and 6 include error bars, confidence intervals, or significance tests. The differences on some cells are small (e.g., AlpacaEval S2T 4.01 vs 3.90 for Freeze-Omni), and the barge-in/backchanneling F1 evaluations rely on self-constructed evaluation sets whose construction, speaker composition, and acoustic conditions are not described in enough detail to assess generalizability or reproducibility. The authors should report variance estimates and either release the barge-in evaluation sets or evaluate on a publicly available benchmark.
minor comments (4)
  1. [Section 5.2 heading] The heading "capailities" should be corrected to "capabilities."
  2. [Section 5.3.1] The phrase "difference full-duplex speech interaction LLMs" should be "different full-duplex speech interaction LLMs."
  3. [Table 5] The "Echo factor" rows labeled ×0.0 and ×0.1 should be defined explicitly; the current phrasing "the model can hear×n its own echo" leaves ambiguous whether ×0.0 means no echo or a different scaling, especially for Freeze-Omni's rows.
  4. [Appendix F] The emotion-intensity evaluation using gpt-4o-audio-preview is not mentioned in the main experimental setup; consider moving it into the main text or clearly labeling it as a separate evaluation to avoid confusion.

Circularity Check

1 steps flagged · score 6.0 of 10

The headline QA benchmark gains are fitted-input predictions because training and evaluation share the same QA sources; the architectural and internal-ablation claims are not circular.

  1. fitted input called prediction [Sec. 4.2 'Training tasks' and Table 7; Sec. 4.3 'Evaluation datasets and metrics'; headline results in Sec. 5.2 / Table 3 and Abstract/Conclusion.]
    "For spoken QA task, we gather questions from a variety of sources, including Alpaca-52k [49], Web Questions [50], TriviaQA [51], SQuAD [52], Natural Questions [53], V oiceAssistant-400K from Mini-Omni [14] and UltraChat from SLAM-Omni [54]. ... We evaluate the speech interaction ability on four commonly used datasets, Llama Questions [55], Web Questions [50], TriviaQA [51] and AlpacaEval from V oicebench [56]. Noted that the sentences used for evaluation are different from those during training."

    The central SOTA claim (an average 35.9% relative improvement over Moshi and Freeze-Omni, Table 3) is measured on Web Questions, TriviaQA, and AlpacaEval. The same sources appear in the training mixture: Table 7 lists Web Questions (4k) and TriviaQA (58k), and Alpaca-52k (39k) is the Alpaca-family instruction set from which AlpacaEval is derived, with no deduplication reported. No held-out split or contamination check is described; the disjointness sentence in Sec. 4.3 refers only to the barge-in evaluation sentences, not to the QA evaluation sets. Since the model was fine-tuned with LoRA on these very QA benchmarks, the reported S2T/S2S accuracies and the 35.9% margin can be explained by memorized answer text rather than by full-duplex conversational ability.

full rationale

The architectural derivation is self-contained: the streaming-encoder, interleaved-stream, explicit-thinking, and three-stage-training components are defined in the paper and supported by internal ablations (implicit vs. explicit thinking, embedding-layer choice, DPO batch size, and comparison of thinking variants). The barge-in/backchannel and turn-taking evaluations also use evaluation sentences that the paper explicitly says differ from training. Citations to prior work by the same authors (SALMONN, MT2KD, CosyVoice2) provide components and distillation recipes but are not the load-bearing justification for the full-duplex contribution. The circularity is limited to the external-benchmark centerpiece: training and evaluation draw from the same QA sources (Web Questions and TriviaQA explicitly; Alpaca-52k vs. AlpacaEval in the same family), with no reported decontamination. Consequently the headline 35.9% relative improvement and the S2T/S2S accuracy numbers in Table 3 are fitted-input results, not independent predictions. This warrants a 6: one major empirical claim reduces by construction to its training input, while the architecture and internal ablations retain independent content.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central design rests on the assumption that a pretrained text LLM can absorb continuous speech embeddings from two streams and learn turn-taking from special tokens, that the Whisper-distilled encoder preserves QA-relevant information, that CosyVoice2 can be driven from layer-24 LLM embeddings, and that the self-built barge-in sets represent real conversation dynamics. No free parameters are fitted in a mathematical derivation; the hand-chosen timing constants and the layer-24 selection are the closest to fitted values. The paper introduces special tokens and a dual-stream interleaving mechanism, but no new physical entities, forces, or conserved quantities.

free parameters (5)
  • Time block duration = 80 ms
    Chosen by hand in Section 4.1 to balance latency and speech quality; each block yields one text token.
  • Speech-to-text token ratio = 12 speech tokens per 4 text tokens
    Determines 480 ms of speech per LLM block and 320 ms delay before audio output.
  • LLM embedding layer for synthesizer = layer 24
    Selected via ablation in Appendix E (Table 12) as best averaged S2T/S2S performance.
  • DPO batch size = 256
    Selected via Figure 4 and Table 10 as the training setup that reaches final F1 0.90.
  • LoRA rank = 32
    Set in Section 4.1 for LLM fine-tuning.
assumptions (5)
  • domain assumption A pretrained text LLM (Llama-3-8B) can learn to attend to continuous speech embeddings interleaved with text and maintain dialogue state without codec tokens.
    Core premise of the architecture; validated only by the paper's own experiments (Sections 3.3, 5).
  • domain assumption The Mamba streaming encoder, distilled from Whisper-large-v3 with L1 loss, preserves enough semantic content for QA and dialogue.
    Section 3.1; relies on knowledge distillation from a non-streaming teacher.
  • domain assumption CosyVoice2 can generate fluent speech when conditioned on LLM layer-24 hidden embeddings instead of its original text input.
    Section 3.2; requires linear adaptation of embeddings.
  • ad hoc to paper The special tokens <think> and <shift> are a sufficient representation for learning listening/speaking transitions.
    Introduced in Section 3.3; ablation (Table 2) supports it only on the model's own data.
  • ad hoc to paper The self-constructed barge-in and backchanneling evaluation sets are representative of real full-duplex conversation dynamics.
    Section 4.3; no external benchmark is used for this capability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation." pith.science (2026). https://pith.science/paper/4WFUSRLD

@misc{pith2026250517060,
  author       = {Pith},
  title        = {Pith review of: SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4WFUSRLD}},
  note         = {Machine review of arXiv:2505.17060}
}
read the original abstract

In order to enable fluid and natural human-machine speech interaction, existing full-duplex conversational systems often adopt modular architectures with auxiliary components such as voice activity detectors, interrupters, conversation state predictors, or multiple LLMs. These systems, however, suffer from error accumulation across modules and struggle with key challenges such as context-dependent barge-in and echo cancellation. Recent approaches, most notably Moshi, simplify the pipeline by injecting audio codecs into the token space of a single LLM. However, such methods still incur significant performance degradation when operating on the speech rather than text modality. In this paper, we introduce SALMONN-omni, the first single, standalone full-duplex speech LLM that operates without audio codecs in its token space. It features a novel dynamic thinking mechanism within the LLM backbone, enabling the model to learn when to transition between speaking and listening states. Experiments on widely used benchmarks for spoken question answering and open-domain dialogue show that SALMONN-omni achieves at least 30\% relative performance improvement over existing open-source full-duplex models and performs highly competitively to half-duplex and turn-based systems, despite using substantially less training data. Moreover, SALMONN-omni demonstrates strong performance in complex conversational scenarios, including turn-taking, backchanneling, echo cancellation and context-dependent barge-in, with further improvements achieved through reinforcement learning. Some demo conversations between user and SALMONN-omni are provided in the following repository https://github.com/bytedance/SALMONN.

Figures

Figures reproduced from arXiv: 2505.17060 by the authors.

Figure 1
Figure 1. The architecture of SALMONN-omni. Two input streams, the environment stream and [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Illustration of “Implicit” and “Explicit” thinking strategies. The tokens on the top of the [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Three-stage training strategy for SALMONN-omni [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The overall F1 score of SALMONN-omni when trained with dif￾ferent batch sizes during the DPO stage. DPO further enhances model’s full-duplex modelling capabilities We train SALMONN-omni on both context￾independent and context-dependent settings, and it turns out that t…
Figure 5
Figure 5. Figure 5: Spoken QA: SALMONN-omni can handle turn-taking in spoken question answering [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: Open-domain dialogue: SALMONN-omni can handle turn-taking in spoken question [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Context-dependent barge-in: When the user is quick to buzz in with the response, [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Distinguishing backchanneling. When the user interjects with a backchannel cue, [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Silent Thought: Modeling Internal Cognition in Full-Duplex Spoken Dialogue Models via Latent Reasoning

    eess.AS 2026-03 unverdicted novelty 7.0 of 10

    FLAIR enables spoken dialogue AI to conduct continuous latent reasoning while perceiving speech through recursive latent embeddings and an ELBO-based finetuning objective.

  2. Adaptive Perturbation Selection for Contrastive Audio Decoding

    cs.SD 2026-06 unverdicted novelty 6.0 of 10

    A learned per-example router over a 105-perturbation audio library improves contrastive decoding for audio-LLM hallucination, with task-dependent best distortions (e.g., reverse audio for temporal order).

Reference graph

Works this paper leans on

57 extracted references · 29 canonical work pages · cited by 2 Pith papers

  1. [1]

    GPT-4 technical report

    OpenAI. GPT-4 technical report. 2024. URL: https://arxiv.org/abs/2303.08774

    arXiv 2024

  2. [2]

    LLaMA: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, et al. “LLaMA: Open and efficient foundation language models”. In: arXiv preprint arXiv:2302.13971 (2023)

    arXiv 2023

  3. [3]

    DeepSeek-R1: Incentivizing reasoning capabil- ity in LLMs via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, et al. “DeepSeek-R1: Incentivizing reasoning capabil- ity in LLMs via reinforcement learning”. In: arXiv preprint arXiv:2501.12948 (2025)

    arXiv 2025

  4. [4]

    SALMONN: Towards generic hearing abilities for large language models

    Changli Tang, Wenyi Yu, Guangzhi Sun, et al. “SALMONN: Towards generic hearing abilities for large language models”. In: Proc. ICLR. Vienna, 2024

    work page 2024

  5. [5]

    Listen, think, and understand

    Yuan Gong, Hongyin Luo, Alexander H. Liu, et al. “Listen, think, and understand”. In:Proc. ICLR. Vienna, 2024

    work page 2024

  6. [6]

    Qwen2-Audio technical report

    Yunfei Chu, Jin Xu, Qian Yang, et al. “Qwen2-Audio technical report”. In: arXiv preprint arXiv:2407.10759 (2024)

    arXiv 2024

  7. [7]

    Boosting large language model for speech synthesis: An empirical study

    Hongkun Hao, Long Zhou, Shujie Liu, et al. “Boosting large language model for speech synthesis: An empirical study”. In: arXiv preprint arXiv:2401.00246 (2023)

    arXiv 2023

  8. [8]

    The Llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, et al. “The Llama 3 herd of models”. In: arXiv preprint arXiv:2407.21783 (2024)

    arXiv 2024

Show all 57 references
  1. [10]

    SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities

    Dong Zhang, Shimin Li, Xin Zhang, et al. “SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities”. In: Proc. EMNLP Findings. Singapore, 2023

    2023

  2. [11]

    Moshi: A speech-text foundation model for real-time dialogue

    Alexandre Défossez, Laurent Mazaré, Manu Orsini, et al. “Moshi: A speech-text foundation model for real-time dialogue”. In: arXiv preprint arXiv:2410.00037 (2024)

    2024 arXiv

  3. [12]

    LLaMA-Omni: Seamless speech interaction with large language models

    Qingkai Fang, Shoutao Guo, Yan Zhou, et al. “LLaMA-Omni: Seamless speech interaction with large language models”. In: Proc. ICLR. Singapore, 2025

    2025

  4. [13]

    Spirit-LM: Interleaved spoken and written language model

    Tu Anh Nguyen, Benjamin Muller, Bokai Yu, et al. “Spirit-LM: Interleaved spoken and written language model”. In: TACL 13 (2025), pp. 30–52

    2025

  5. [14]

    Mini-Omni: Language models can hear, talk while thinking in streaming

    Zhifei Xie and Changqiao Wu. “Mini-Omni: Language models can hear, talk while thinking in streaming”. In: arXiv preprint arXiv:2408.16725 (2024)

    2024 arXiv

  6. [15]

    GLM-4-V oice: Towards intelligent and human-like end-to-end spoken chatbot

    Aohan Zeng, Zhengxiao Du, Mingdao Liu, et al. “GLM-4-V oice: Towards intelligent and human-like end-to-end spoken chatbot”. In: arXiv preprint arXiv:2412.02612 (2024)

    2024 arXiv

  7. [16]

    Beyond turn-based interfaces: Syn- chronous LLMs as full-duplex dialogue agents

    Bandhav Veluri, Benjamin Peloquin, Bokai Yu, et al. “Beyond turn-based interfaces: Syn- chronous LLMs as full-duplex dialogue agents”. In: Proc. EMNLP. Miami, 2024

    2024

  8. [17]

    OmniFlatten: An end-to-end GPT model for seamless voice conversation

    Qinglin Zhang, Luyao Cheng, Chong Deng, et al. “OmniFlatten: An end-to-end GPT model for seamless voice conversation”. In: arXiv preprint arXiv:2410.17799 (2024)

    2024 arXiv

  9. [18]

    Talking turns: Benchmarking audio foundation models on turn-taking dynamics

    Siddhant Arora, Zhiyun Lu, Chung-Cheng Chiu, et al. “Talking turns: Benchmarking audio foundation models on turn-taking dynamics”. In: Proc. ICLR. Singapore, 2025

    2025

  10. [19]

    VITA-1.5: Towards GPT-4o level real-time vision and speech interaction

    Chaoyou Fu, Haojia Lin, Xiong Wang, et al. “VITA-1.5: Towards GPT-4o level real-time vision and speech interaction”. In: arXiv preprint arXiv:2501.01957 (2025)

    2025 arXiv

  11. [20]

    Freeze-Omni: A smart and low latency speech- to-speech dialogue model with frozen LLM

    Xiong Wang, Yangze Li, Chaoyou Fu, et al. “Freeze-Omni: A smart and low latency speech- to-speech dialogue model with frozen LLM”. In: arXiv preprint arXiv:2411.00774 (2024)

    2024 arXiv

  12. [21]

    Kimi-Audio technical report

    Ding Ding, Zeqian Ju, Yichong Leng, et al. “Kimi-Audio technical report”. In: arXiv preprint arXiv:2504.18425 (2025)

    2025 arXiv

  13. [22]

    Qwen2.5-Omni technical report

    Jin Xu, Zhifang Guo, Jinzheng He, et al. “Qwen2.5-Omni technical report”. In: arXiv preprint arXiv:2503.20215 (2025)

    2025 arXiv

  14. [23]

    POMDP-based statistical spoken dialog systems: A review

    Steve Young, Milica Gaši´c, Blaise Thomson, et al. “POMDP-based statistical spoken dialog systems: A review”. In: Proceedings of the IEEE 101.5 (2013), pp. 1160–1179

    2013

  15. [24]

    A neural conversational model

    Oriol Vinyals and Quoc V . Le. “A neural conversational model”. In:Proc. ICML Deep Learning Workshop. Lille, 2015

    2015

  16. [25]

    Building end-to-end dialogue systems using generative hierarchical neural network models

    Iulian V . Serban, Alessandro Sordoni, Yoshua Bengio, et al. “Building end-to-end dialogue systems using generative hierarchical neural network models”. In: Proc. AAAI. Phoenix, 2016

    2016

  17. [26]

    MiniCPM-O 2.6: A GPT-4o level MLLM for vision, speech, and mul- timodal live streaming on your phone

    OpenBMB. MiniCPM-O 2.6: A GPT-4o level MLLM for vision, speech, and mul- timodal live streaming on your phone . https : / / openbmb . notion . site / 185ede1b7a558042b5d5e45e6b237da9. 2025. 11

    2025

  18. [27]

    Baichuan-Audio: A unified framework for end-to-end speech interaction

    Tianpeng Li, Jun Liu, Tao Zhang, et al. “Baichuan-Audio: A unified framework for end-to-end speech interaction”. In: arXiv preprint arXiv:2502.17239 (2025)

    2025 arXiv

  19. [28]

    MinMo: A multimodal large language model for seamless voice interaction

    Qian Chen, Yafeng Chen, Yanni Chen, et al. “MinMo: A multimodal large language model for seamless voice interaction”. In: arXiv preprint arXiv:2501.06282 (2025)

    2025 arXiv

  20. [29]

    Generative spoken dialogue language modeling

    Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, et al. “Generative spoken dialogue language modeling”. In: TACL 11 (2023), pp. 250–266

    2023

  21. [30]

    Optimizing expected word error rate via sampling for speech recognition

    Matt Shannon. “Optimizing expected word error rate via sampling for speech recognition”. In: Proc. Interspeech. Stockholm, 2017

    2017

  22. [31]

    Variable Frame Rate Acoustic Models Using Minimum Error Reinforcement Learning

    Dongcheng Jiang, Chao Zhang, and Philip C Woodland. “Variable Frame Rate Acoustic Models Using Minimum Error Reinforcement Learning.” In: Interspeech. 2021, pp. 2601– 2605

    2021

  23. [32]

    Seed-ASR: Understanding diverse speech and contexts with LLM-based speech recognition

    Ye Bai, Jingping Chen, Jitong Chen, et al. “Seed-ASR: Understanding diverse speech and contexts with LLM-based speech recognition”. In: arXiv preprint arXiv:2407.04675 (2024)

    2024 arXiv

  24. [33]

    Speech recognition with LLMs adapted to disordered speech using reinforcement learning

    Chirag Nagpal, Subhashini Venugopalan, Jimmy Tobin, et al. “Speech recognition with LLMs adapted to disordered speech using reinforcement learning”. In: Proc. ICASSP. Hyderabad, 2025

    2025

  25. [34]

    Reinforcement learning for spoken dialogue systems

    Satinder Singh, Michael Kearns, Diane Litman, et al. “Reinforcement learning for spoken dialogue systems”. In: Proc. NIPS. Denver, 1999

    1999

  26. [35]

    Automatic learning of dialogue strategy using dialogue simulation and reinforcement learning

    Konrad Scheffler and Steve Young. “Automatic learning of dialogue strategy using dialogue simulation and reinforcement learning”. In: Proceedings of HLT. V ol. 2. 2002, p. 0

    2002

  27. [36]

    SpeechAlign: Aligning speech generation to human preferences

    Dong Zhang, Zhaowei Li, Shimin Li, et al. “SpeechAlign: Aligning speech generation to human preferences”. In: Proc. NeurIPS. Vancouver, 2024

    2024

  28. [37]

    Enhancing zero-shot text-to-speech synthesis with human feedback

    Chen Chen, Yuchen Hu, Wen Wu, et al. “Enhancing zero-shot text-to-speech synthesis with human feedback”. In: arXiv preprint arXiv:2406.00654 (2024)

    2024 arXiv

  29. [38]

    Seed-TTS: A family of high-quality versatile speech generation models

    Philip Anastassiou, Jiawei Chen, Jitong Chen, et al. “Seed-TTS: A family of high-quality versatile speech generation models”. In: arXiv preprint arXiv:2406.02430 (2024)

    2024 arXiv

  30. [39]

    Proximal policy optimization algo- rithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, et al. “Proximal policy optimization algo- rithms”. In: arXiv preprint arXiv:1707.06347 (2017)

    2017 arXiv

  31. [40]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, et al. “Direct preference optimization: Your language model is secretly a reward model”. In: NIPS. New Orleans, 2023

    2023

  32. [41]

    Step-Audio: Unified understanding and genera- tion in intelligent speech interaction

    Ailin Huang, Boyong Wu, Bruce Wang, et al. “Step-Audio: Unified understanding and genera- tion in intelligent speech interaction”. In: arXiv preprint arXiv:2502.11946 (2025)

    2025 arXiv

  33. [42]

    MT2KD: Towards a general-purpose encoder for speech, speaker, and audio events

    Xiaoyu Yang, Qiujia Li, Chao Zhang, et al. “MT2KD: Towards a general-purpose encoder for speech, speaker, and audio events”. In: arXiv preprint arXiv:2409.17010 (2024)

    2024 arXiv

  34. [43]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, et al. “Robust speech recognition via large-scale weak supervision”. In: Proc. ICML. Honolulu, 2023

    2023

  35. [44]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. “Mamba: Linear-time sequence modeling with selective state spaces”. In: Proc. COLM. Montreal, 2024

    2024

  36. [45]

    CosyV oice 2: Scalable streaming speech synthesis with large language models

    Zhihao Du, Yuxuan Wang, Qian Chen, et al. “CosyV oice 2: Scalable streaming speech synthesis with large language models”. In: arXiv preprint arXiv:2412.10117 (2024)

    2024 arXiv

  37. [46]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, et al. “LoRA: Low-rank adaptation of large language models”. In: Proc. ICLR. 2022

    2022

  38. [47]

    Librispeech: An ASR corpus based on public domain audio books

    Vassil Panayotov, Guoguo Chen, Daniel Povey, et al. “Librispeech: An ASR corpus based on public domain audio books”. In: Proc. ICASSP. South Brisbane, 2015

    2015

  39. [48]

    GigaSpeech: An evolving, multi-domain ASR corpus with 10,000 hours of transcribed audio

    Guoguo Chen, Shuzhou Chai, Guanbo Wang, et al. “GigaSpeech: An evolving, multi-domain ASR corpus with 10,000 hours of transcribed audio”. In: Proc. Interspeech. Brno, 2021

    2021

  40. [49]

    Stanford Alpaca: An Instruction-following LLaMA model

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, et al. Stanford Alpaca: An Instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca. 2023

    2023

  41. [50]

    Semantic parsing on freebase from question-answer pairs

    Jonathan Berant, Andrew Chou, Roy Frostig, et al. “Semantic parsing on freebase from question-answer pairs”. In: Proc. EMNLP. Seattle, 2013

    2013

  42. [51]

    TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension

    Mandar Joshi, Eunsol Choi, Daniel Weld, et al. “TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension”. In: Proc. ACL. Vancouver, 2017

    2017

  43. [52]

    SQuAD: 100,000+ Questions for Machine Comprehension of Text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, et al. “SQuAD: 100,000+ Questions for Machine Comprehension of Text”. In: Proc. EMNLP. Austin, 2016. 12

    2016

  44. [53]

    Natural questions: a bench- mark for question answering research

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, et al. “Natural questions: a bench- mark for question answering research”. In: Transactions of the Association for Computational Linguistics 7 (2019), pp. 453–466

    2019

  45. [54]

    SLAM-Omni: Timbre-Controllable V oice Interac- tion System with Single-Stage Training

    Wenxi Chen, Ziyang Ma, Ruiqi Yan, et al. “SLAM-Omni: Timbre-Controllable V oice Interac- tion System with Single-Stage Training”. In: arXiv preprint arXiv:2412.15649 (2024)

    2024 arXiv

  46. [55]

    Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLM

    Eliya Nachmani, Alon Levkovitch, Roy Hirsch, et al. “Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLM”. In: Proc. ICLR. Vienna, 2024

    2024

  47. [56]

    V oicebench: Benchmarking llm-based voice assistants

    Yiming Chen, Xianghu Yue, Chen Zhang, et al. “V oicebench: Benchmarking llm-based voice assistants”. In: arXiv preprint arXiv:2410.17196 (2024)

    2024 arXiv

  48. [57]

    Libriheavy: A 50,000 hours ASR corpus with punctuation casing and context

    Wei Kang, Xiaoyu Yang, Zengwei Yao, et al. “Libriheavy: A 50,000 hours ASR corpus with punctuation casing and context”. In: Proc. ICASSP. Seoul, 2024

    2024

  49. [58]

    Decoupled Weight Decay Regularization

    Ilya Loshchilov and Frank Hutter. “Decoupled Weight Decay Regularization”. In: Proc. ICLR. New Orleans, 2019. 13 A Further Experimental details A.1 Details for training data For V oiceAssistant-400K and UltraChat, we only select the first-round QA and filter repeated questions...

    2019

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.