{"id":"ecea12d6-1b21-40f3-acca-82ad4e265772","arxiv_id":"2505.17060","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SALMONN-omni is a standalone full-duplex speech LLM that interleaves continuous speech and text embeddings in one model and uses special thinking tokens to learn turn-taking and barge-in behavior.","lead":"This paper introduces SALMONN-omni, a voice chatbot that can listen and speak simultaneously with a single language model, without converting speech into discrete code tokens. It reports large gains over prior open-source full-duplex models on spoken question answering and conversation benchmarks, with a new training method for deciding when to talk or stay silent.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation sets overlap with QA training data (Web Questions, TriviaQA, Alpaca-52k vs. the same eval benchmarks) and no deduplication or held-out split is reported, so the headline 35.9% relative improvement is not yet decontaminated.","rationale":"I read the paper as making two intertwined claims: an architectural contribution (a standalone full-duplex speech LLM without codec tokens in the LLM vocabulary, with a learned thinking mechanism for state transitions) and a quantitative claim (at least 30% relative improvement over existing open-source full-duplex models in predicted turn-taking). The architecture is described in enough detail that the core idea is plausible, and the ablations in Section 5.1 and Appendix C give some internal support for the explicit thinking strategy. The DPO results in Section 5.3.2 also provide a coherent story independent of the QA benchmarks. However, the headline number depends entirely on Table 3, and that table is scored on benchmarks whose names appear in the training data list with no stated disjointness guarantee. The reader's weakest assumption correctly identifies this as the most load-bearing risk. I considered other concerns, such as sparse details on exactly how predicted turn-taking accuracy is scored and the absence of released code, but those affect reproducibility rather than directly undermining the central quantitative claim. The contamination risk is the one that could change the conclusion if it lands, so it is the appropriate focus. The appropriate verdict remains CONDITIONAL: the architecture deserves follow-up, but the performance margin must be verified on a decontaminated evaluation before the SOTA claim is accepted.","tokens_in":16281,"tokens_out":5949,"duration_ms":61235,"concrete_test":"Compute normalized exact-match and near-duplicate overlap between the training QA items (Web Questions, TriviaQA, Alpaca-52k) and the evaluation items (Web Questions, the OpenAudioBench TriviaQA subset, AlpacaEval). If any overlap exists, retrain or re-run evaluation on a disjoint split with all overlapping question texts removed, then recompute Table 3 and the average relative improvement over Moshi and Freeze-Omni. If the margin falls below the claimed 30% or the ranking changes, the headline empirical claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The empirical centerpiece is the 35.9% average relative improvement over Moshi and Freeze-Omni in predicted turn-taking (Table 3). That number is only interpretable if the evaluation questions are not part of the training corpus, and the paper does not establish this. Section 4.2 and Table 7 list Web Questions (4k), TriviaQA (58k), and Alpaca-52k (39k) among the QA training data; Section 4.3 evaluates on Web Questions, a 1000-sample TriviaQA subset from OpenAudioBench, and AlpacaEval. No deduplication, split, or holdout procedure is reported, and the only disjointness statement in Section 4.3 concerns the barge-in sentences, not the QA sets. Because the model is trained with LoRA on the same general benchmarks it is scored on, the S2T and S2S gains and the 35.9% relative margin could reflect memorized answer text rather than full-duplex competence. This does not invalidate the architectural contribution or the turn-taking and barge-in ablations, but it removes the paper's main quantitative SOTA claim until contamination is ruled out.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SALMONN-omni, a full-duplex speech LLM that avoids injecting audio codec tokens into the LLM vocabulary by combining a streaming Mamba speech encoder, a frozen Llama-3-8B-Instruct backbone with LoRA, and a streaming CosyVoice2-based synthesizer. A periodic synchronization mechanism and special tokens <think> and <shift> are introduced to let the LLM learn when to start and stop speaking. The model is trained in three stages: encoder connection, end-to-end SFT with synthesizer, and DPO for dialogue dynamics. The authors report a 35.9% average relative improvement over Moshi and Freeze-Omni in predicted turn-taking, competitiveness with half-duplex models, and strong performance in barge-in/backchanneling evaluations.","tokens_in":16567,"tokens_out":6233,"duration_ms":55893,"significance":"If the empirical claims survive a decontamination check, the paper makes a valuable architectural contribution: it demonstrates that a single LLM can perform full-duplex conversation without codec tokens, using a lightweight 'thinking' mechanism for state transitions, and it is the first to apply DPO to full-duplex dialogue dynamics. The method is clearly described and the ablation study in Appendix C comparing thinking-strategy variants is informative. However, the central quantitative claim is currently not supported because the evaluation benchmarks overlap with the training data and because the comparison omits some systems it categorizes as full-duplex.","major_comments":[{"comment":"Web Questions (4k samples) and TriviaQA (58k samples) are listed among the QA training data, while Web Questions and a 1000-sample TriviaQA subset from OpenAudioBench are used as evaluation benchmarks in Section 4.3. The manuscript does not report deduplication, held-out splits, or overlap statistics for these sets; the sentence at the end of Section 4.3 stating that evaluation sentences differ from training sentences refers only to the barge-in/backchanneling evaluation, not to the QA benchmarks. Because the headline 35.9% relative improvement is computed on these benchmarks, the result may reflect memorization of the training questions rather than full-duplex competence. The authors should quantify the overlap, exclude overlapping items, or re-evaluate on non-overlapping benchmarks before the SOTA claim can be accepted.","section":"Section 4.2/Table 7 and Section 4.3"},{"comment":"The predicted-turn-taking comparison includes only Moshi and Freeze-Omni, yet Table 1 classifies VITA-1.5 and MinMo as full-duplex models as well; the paper does not explain why they are excluded from this setting. Additionally, the reported 35.9% average relative improvement is the mean over eight cells (four datasets by two metrics) with individual relative gains ranging from 2.8% (AlpacaEval S2T) to 96.5% (TriviaQA S2S); this averaging masks large heterogeneity and should be reported with per-cell values and a clear statement of the averaging procedure.","section":"Table 3 and Section 4.3"},{"comment":"None of the reported numbers in Tables 3, 4, 5, and 6 include error bars, confidence intervals, or significance tests. The differences on some cells are small (e.g., AlpacaEval S2T 4.01 vs 3.90 for Freeze-Omni), and the barge-in/backchanneling F1 evaluations rely on self-constructed evaluation sets whose construction, speaker composition, and acoustic conditions are not described in enough detail to assess generalizability or reproducibility. The authors should report variance estimates and either release the barge-in evaluation sets or evaluate on a publicly available benchmark.","section":"Section 5.3 and Section 4.3"}],"minor_comments":[{"comment":"The heading \"capailities\" should be corrected to \"capabilities.\"","section":"Section 5.2 heading"},{"comment":"The phrase \"difference full-duplex speech interaction LLMs\" should be \"different full-duplex speech interaction LLMs.\"","section":"Section 5.3.1"},{"comment":"The \"Echo factor\" rows labeled ×0.0 and ×0.1 should be defined explicitly; the current phrasing \"the model can hear×n its own echo\" leaves ambiguous whether ×0.0 means no echo or a different scaling, especially for Freeze-Omni's rows.","section":"Table 5"},{"comment":"The emotion-intensity evaluation using gpt-4o-audio-preview is not mentioned in the main experimental setup; consider moving it into the main text or clearly labeling it as a separate evaluation to avoid confusion.","section":"Appendix F"}],"recommendation":"major_revision","confidential_remarks":"The benchmark overlap issue in Section 4.2/Table 7 versus Section 4.3 is the primary barrier to accepting the empirical claims; if the authors can demonstrate non-contamination or re-evaluate on held-out data, the paper would be a strong candidate. The architectural contribution is interesting and the ablation study is solid, but the current SOTA claim is not yet decontaminated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real architectural contribution that should be taken seriously, but the paper's central quantitative claim—35.9% average relative improvement over Moshi and Freeze-Omni in predicted turn-taking—is currently not decontaminated. The training set (Section 4.2, Table 7) includes Web Questions (4k), TriviaQA (58k), and Alpaca-52k (39k); evaluation (Section 4.3) uses Web Questions, a 1000-sample TriviaQA subset, and AlpacaEval. No deduplication or held-out split is reported. That means the S2T/S2S gains and the 30%+ margin could partly reflect memorized answers rather than full-duplex competence. The stress-test note is right on this point. The paper's own sentence that evaluation sentences differ from training applies only to the barge-in sentences, not the QA sets.\n\nWhat is actually new: interleaving continuous environment-stream and assistant-stream embeddings with text in a single LLM, no codec tokens in vocabulary, plus <think>/<shift> transitions and DPO for dialogue dynamics. The ablations comparing implicit vs explicit thinking, the layer choice, and the DPO batch-size behavior are informative and internally consistent. The architecture contrasts cleanly with Moshi (codec injection) and Freeze-Omni/VITA (two LLM processes). That is a real design space contribution.\n\nSoft spots, in proportion: (1) the training/eval overlap above is load-bearing for the headline claim; (2) all main metrics lack error bars, significance tests, and the barge-in F1 numbers are self-constructed and small; (3) no code or model release, though the demo repo exists. None of this invalidates the architecture or the ablations, but it caps what the paper can currently claim.\n\nWho it's for: people working on end-to-end spoken dialogue systems and full-duplex modeling. A serious referee should engage, and the paper deserves a revise-and-resubmit track with decontaminated evaluation, more eval detail, and ideally release of code/data.","headline":"SALMONN-omni's codec-free full-duplex architecture is a genuinely new idea worth referee time, but the headline 35.9% gain is not yet interpretable because the QA evaluation sets overlap with the training data.","tokens_in":17105,"tokens_out":2676,"would_cite":true,"duration_ms":26178,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Speech model learns when to listen and speak without audio codecs","keywords":["full-duplex speech LLM","codec-free speech interaction","turn-taking prediction","barge-in","backchanneling","dynamic thinking tokens","reinforcement learning for dialogue","streaming speech synthesis"],"falsifier":"Construct a spoken QA evaluation set from questions published after the model's training data was collected, and re-run the predicted turn-taking comparison against Moshi and Freeze-Omni; if the 35.9% average relative improvement disappears or shrinks sharply, memorization rather than full-duplex capability explains the result. A cheaper check: search the training QA sets for exact or near-duplicate strings of the evaluation questions and measure the overlap rate.","tokens_in":16074,"feed_emoji":"🎙️","tokens_out":5204,"duration_ms":43999,"temperature":0.7,"pith_summary":"This paper introduces SALMONN-omni, a speech-enabled large language model built to hold a full-duplex conversation: it can listen to a user, decide when to speak, generate speech, and keep taking in sound while its own voice is playing. The central claim is that no audio codec tokens need to be inserted into the LLM's vocabulary to do this. Instead, a streaming speech encoder and a streaming speech synthesizer are attached to a single LLM through embeddings, and the model is trained to emit two special tokens, <think> and <shift>, as ordinary text tokens to mark listening, speaking, and transitions between them. On spoken question-answering and open-domain dialogue benchmarks, the authors report an average 35.9% relative improvement over prior open-source full-duplex models when the model must predict its own turn-taking timing, and performance competitive with half-duplex models trained on far more data.","feed_headline":"Speech model learns when to listen and speak without audio codecs","feed_subtitle":"A single LLM handles full-duplex chat, beating codec-based systems by 35.9% on average.","key_machinery":"The central mechanism is the explicit 'thinking' strategy, a pair of special tokens <think> and <shift> generated by the LLM as ordinary output tokens, synchronized to 80 ms time blocks of audio. <think> stands for the listening state (or the silence while the model waits after finishing a response), and <shift> marks both listening-to-speaking and speaking-to-listening transitions. Because these tokens are generated and fed back into the LLM's input sequence, the model learns turn-taking, barge-in acceptance, and backchanneling as autoregressive text prediction, which aligns with the LLM's native operation and avoids the modality gap and data hunger of codec-injection approaches.","core_discovery":"The paper's discovery is that a single, standalone LLM can serve as a full-duplex conversational agent if it is trained to treat dialogue-state transitions as part of its own token generation. SALMONN-omni interleaves environment-stream speech embeddings, assistant-stream (own output) speech embeddings, and text tokens into one autoregressive sequence, divided into fixed 80 ms blocks. In each block the LLM either predicts the special token <think> while listening, generates a <shift> to mark a state change, or produces answer tokens while speaking; the streaming synthesizer converts the LLM's hidden embeddings into speech. This explicit 'thinking' strategy outperforms an implicit variant, and incorporating the assistant stream raises turn-taking success from roughly 70% to 90%. The result, in the authors' phrase, is that the LLM is 'secretly a full-duplex predictor': no external voice-activity detector, interruption predictor, or second LLM process is needed.","pith_inferences":["The 'LLM as full-duplex predictor' idea should generalize: any autoregressive model with access to stream-aligned timestamps could learn state transitions the same way, including text-only agents that need to decide when to yield the floor.","Because the model hears its own output through the assistant stream, the same training signal could make echo cancellation and self-speech recognition emergent rather than engineered; this is a testable extension the paper partially demonstrates.","The reported DPO dynamics—precision rising while recall collapses, then recovering—suggest a reward-design hazard: naive interruption penalties teach silence first. Fine-grained rewards that separately credit appropriate barge-in and backchannel responses may train faster.","A direct stress test would be to vary the 80 ms block size and the 4-token-to-12-speech-token ratio, since the claimed naturalness depends on this synchronization granularity."],"forward_implications":["In predicted turn-taking mode, the reported numbers put SALMONN-omni ahead of Moshi and Freeze-Omni on all four benchmarks, with an average 35.9% relative gain.","Under oracle turn-taking, SALMONN-omni is competitive with turn-based speech LLMs such as Qwen2.5-Omni and miniCPM-o while using less training data.","Explicit thinking beats implicit thinking, and adding the assistant stream is what pushes turn-taking success from roughly 70% to 90%.","Applying DPO after supervised training raises the overall barge-in/backchanneling F1 from 0.86 to 0.90, at the cost of an early conservative phase where the model almost never allows itself to be interrupted.","The three-stage training recipe—encoder connection, synthesizer connection, then RL—shows that full-duplex behavior can be learned without codec tokens in the LLM vocabulary."],"supporting_citations":[{"why":"Moshi is the main codec-injection full-duplex baseline that SALMONN-omni is compared against in predicted turn-taking.","marker":"[11]"},{"why":"Freeze-Omni is the non-standalone full-duplex baseline that requires two LLM processes and serves as a comparison.","marker":"[20]"},{"why":"CosyVoice 2 is the streaming speech synthesizer that SALMONN-omni finetunes and integrates through LLM embeddings.","marker":"[45]"},{"why":"Whisper-large-v3 is the teacher model for the streaming speech encoder and is also used to transcribe speech for S2S evaluation.","marker":"[43]"},{"why":"Mamba provides the selective state-space blocks used in the streaming speech encoder.","marker":"[44]"},{"why":"Direct Preference Optimization is the reinforcement learning method used in stage 3 to improve barge-in and backchanneling behavior.","marker":"[40]"},{"why":"MT2KD supplies the knowledge-distillation approach for training the streaming encoder from non-streaming encoders.","marker":"[42]"},{"why":"Llama 3 is the LLM backbone that SALMONN-omni finetunes with LoRA.","marker":"[8]"},{"why":"Qwen2.5-Omni is a half-duplex turn-based baseline used in the oracle turn-taking comparison.","marker":"[22]"}],"fun_headline_variants":["One LLM, zero codecs: full-duplex chat via dynamic thinking","LLM secretly predicts full-duplex turns without codec injection","Standalone speech LLM: no codecs, learns when to speak and listen","Single LLM handles barge-in and echo, no extra modules needed","Full-duplex speech without codec injection: it's all in the LLM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains assume the evaluation benchmarks are not contaminated by training data: the QA training set includes Web Questions, TriviaQA, and Alpaca-52k, and evaluation uses Web Questions, a TriviaQA subset, and AlpacaEval, with no deduplication or held-out split reported; if questions overlap, the accuracy and 35.9% improvement could reflect memorization rather than full-duplex skill.","fun_headline_variants_meta":{"raw":{"variants":["One LLM, zero codecs: full-duplex chat via dynamic thinking","LLM secretly predicts full-duplex turns without codec injection","Standalone speech LLM: no codecs, learns when to speak and listen","Single LLM handles barge-in and echo, no extra modules needed","Full-duplex speech without codec injection: it's all in the LLM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00131,"raw_usage":{"total_tokens":5382,"prompt_tokens":1029,"completion_tokens":4353,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":645,"completion_tokens_details":{"reasoning_tokens":4254}},"tokens_in":645,"tokens_out":4353,"duration_ms":26488,"temperature":1.0,"reasoning_tokens":4254,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:45:08.976769+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a spoken QA evaluation set from questions published after the model's training data was collected, and re-run the predicted turn-taking comparison against Moshi and Freeze-Omni; if the 35.9% average relative improvement disappears or shrinks sharply, memorization rather than full-duplex capability explains the result. A cheaper check: search the training QA sets for exact or near-duplicate strings of the evaluation questions and measure the overlap rate.","supporting_citations":[{"cited_title":"Robust speech recognition via large-scale weak supervision","cited_arxiv_id":null,"evidence_quote":"Whisper-large-v3 is the teacher model for the streaming speech encoder and is also used to transcribe speech for S2S evaluation."},{"cited_title":"Mamba: Linear-time sequence modeling with selective state spaces","cited_arxiv_id":null,"evidence_quote":"Mamba provides the selective state-space blocks used in the streaming speech encoder."},{"cited_title":"Direct preference optimization: Your language model is secretly a reward model","cited_arxiv_id":null,"evidence_quote":"Direct Preference Optimization is the reinforcement learning method used in stage 3 to improve barge-in and backchanneling behavior."},{"cited_title":"MT2KD: Towards A General-Purpose Encoder for Speech, Speaker, and Audio Events","cited_arxiv_id":"2409.17010","evidence_quote":"MT2KD supplies the knowledge-distillation approach for training the streaming encoder from non-streaming encoders."}],"review_version":1}