The reviewed record of science sign in
Pith

arxiv: 2607.06540 · v1 · pith:Z23FHT2H · submitted 2026-07-07 · cs.CL

Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs

Zhenyu Liu , Yunxin Li , Xuanyu Zhang , Qixun Teng , Shenyuan Jiang , Haolan Chen , Minjun Zhao , Fanbo Meng
show 5 more authors
Yu Xu Yancheng He Baotian Hu Haizhou Li Min Zhang
This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-08 02:18 UTCglm-5.2pith:Z23FHT2Hrecord.jsonopen to challenge →

classification cs.CL
keywords full-duplexmodalityslmsinterferencesemantichierarchicalnativeadvances
0
0 comments X

The pith

Splitting deep layers lets spoken AI talk and think at once

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that the modality interference degrading full-duplex spoken language models stems from a specific, identifiable cause: gradient vectors for acoustic and semantic objectives diverge into conflict in deep shared layers, while sparse text-token alignment dilutes semantic gradient magnitude throughout. The authors measure this by computing layer-wise cosine similarity between text and speech loss gradients (which turns negative in deep layers) and gradient magnitude ratios (which show semantic suppression under padding-based alignment). They then build Lychee-FD, which splits the deep portion of a shared transformer backbone into three parallel heads (text, acoustic, control) operating at the same depth as the original model, and adds a dense semantic alignment channel that generates continuous internal text monologues to maintain strong semantic gradients during training. The combination yields +7.4% on Spoken QA and +28.5% on FullDuplexBench 1.5 over prior baselines, while matching the half-duplex backbone's knowledge retention at no additional inference depth.

Core claim

The central discovery is a diagnostic claim paired with an architectural response. The diagnostic: when a single transformer must simultaneously optimize text and speech objectives, the gradient vectors for these two objectives are synergistic in shallow layers but become orthogonal or negatively correlated in deep layers, producing destructive cross-task parameter updates. A secondary diagnostic is that padding-based temporal alignment of sparse text tokens (3 Hz) with dense audio frames (25 Hz) suppresses semantic gradient magnitudes, letting acoustic reconstruction dominate optimization. The architectural response: physically separating deep-layer parameters into parallel modality heads (

What carries the argument

Hierarchical Parameter Separation (splitting deep transformer layers into parallel text/acoustic/control heads) and the Semantic Alignment Channel (dense continuous text supervision generating internal monologues to counteract gradient dilution from sparse padding-based alignment).

If this is right

  • Full-duplex spoken AI can match or exceed half-duplex knowledge retention without added inference depth, removing a key barrier to deploying simultaneous listen-and-speak systems.
  • The gradient-conflict diagnostic method (layer-wise cosine similarity between modality-specific loss gradients) becomes a general tool for diagnosing interference in any multi-modal model sharing parameters across objectives.
  • Other multi-modal architectures (vision-language, audio-visual) may exhibit the same shallow-synergy / deep-conflict pattern, suggesting that late-layer parameter separation is a broadly applicable design principle.
  • The semantic alignment channel concept (dense internal text generation as a gradient anchor) could apply to any setting where sparse supervision from one modality is diluted by dense frames from another.
  • The DAG-based parallel inference approach (broadcasting hidden states to multiple GPU heads) provides a deployment template for any architecture with parallel output heads.

Load-bearing premise

The paper observes that gradient cosine similarity between text and speech objectives turns negative in deep layers and then builds an architecture to separate those parameters, but the causal claim that gradient conflict is the root cause rests on a single-step parameter-update approximation of what is actually a multi-step optimization process, so the observed interference could alternatively stem from loss weighting, learning-rate balancing, or data distribution mismatch.

What would settle it

If the same performance gains could be achieved by simply rebalancing loss weights or adjusting learning rates per modality in a fully shared architecture (without any parameter separation), then gradient conflict in shared deep layers would be a symptom rather than the root cause, and the hierarchical separation would be an unnecessary complication.

Figures

Figures reproduced from arXiv: 2607.06540 by Baotian Hu, Fanbo Meng, Haizhou Li, Haolan Chen, Minjun Zhao, Min Zhang, Qixun Teng, Shenyuan Jiang, Xuanyu Zhang, Yancheng He, Yunxin Li, Yu Xu, Zhenyu Liu.

Figure 1
Figure 1. Figure 1: Visualization of the efficiency and intelli￾gence trade-off. Existing paradigms face a dilemma when extending a half-duplex SLM (black star) to a full-duplex one. Native End-to-End models (blue diamond) sacrifice accuracy for efficiency, while Thinker-Talker models (red diamond) pre￾serve knowledge but incur prohibitive inference costs. In contrast, our proposed Hierarchical framework (green diamond) combi… view at source ↗
Figure 2
Figure 2. Figure 2: Optimization Dynamics Visualization. (a) Gradient Cosine Similarity: [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Two mainstream architecture paradigms of SLMs and our proposed Lychee-FD. Our design features a hierarchical parameter separation strategy to resolve deep-layer modality conflicts and a semantic alignment channel to enforce robust knowledge retention without sacrificing inference efficiency. Semantic Dilution. Prevalent Full-Duplex SLMs typically address the frequency mismatch between text (approximately 3… view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of speech synthesis quality via UTMOS. Lychee-FD (w/o Param-Sep) denotes the variant without hierarchical parameter separation strategy. 0 4 8 12 16 20 24 28 Layer Index −0.075 −0.050 −0.025 0.000 0.025 0.050 0.075 G r a die n t C o sin e Simila rit y () Task Synergy Task Conflict Per-Layer Gradient Cosine Similarity [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Layer ablation study on model perfor￾mance and computation cost. Accuracy on LlamaQ saturates after 4 separated layers, while computa￾tional overhead grows linearly. This indicates that 4 layers represent the optimal trade-off between perfor￾mance gains and inference efficiency. To explore the impact of the number of separated lay￾ers, we conduct a layer ablation study on both model performance on LlamaQ a… view at source ↗
Figure 7
Figure 7. Figure 7: A case study demonstrating Lychee-FD’s capability in handling complex turn-taking dynamics. Lychee-FD successfully generates backchannels, halts immediately upon interruption, and provides a contex￾tually accurate response to the user’s specific query without misinterpreting passive feedback. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: An error analysis illustrating a common limitation in handling side-talk during full-duplex interaction. While the model correctly detects voice activity and halts its speech, it fails to recognize that the user’s utterance is directed at a third party, highlighting a key area for future community research. 8.1 Case Study: Interruption and Backchannel In [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Global gradient influence scores among Text and Speech tasks. [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗
read the original abstract

Developing seamless, high-performance, native intelligent full-duplex Spoken Language Models (SLMs) remains a critical challenge and long-standing goal for the speech and NLP community. Despite notable progress, recent endeavors are fundamentally constrained by severe modality interference, which causes substantial knowledge degradation and compromises semantic integrity -- ultimately making full-duplex SLMs feel unnatural and unintelligent. In this paper, through an exhaustive fine-grained analysis of model optimization dynamics, we uncover the root cause of such performance degradation, revealing that modality interference arises from inherent gradient conflicts between acoustic and semantic modeling when the two modalities are forced to share a deep parameter space. Guided by this key insight, we introduce Lychee-FD, a native end-to-end full-duplex framework designed to mitigate modality interference. Importantly, we propose a hierarchical parameter separation strategy that decouples conflicting modalities in deep layers while preserving cross-modality coherence via a dedicated semantic alignment channel. Extensive experiments on multiple full-duplex benchmarks demonstrate that our method significantly advances the state of the art, yielding substantial improvements in both speech intelligence (+7.4% on Spoken QA) and full-duplex interaction fluidity (+28.5% on FullDuplexBench 1.5) without compromising inference efficiency. To the best of our knowledge, this work is the first to achieve two key advances: 1) uncovering and elucidating the root cause of modality interference in full-duplex SLMs, and 2) designing an elegant hierarchical model together with a practical solution for seamless, high-performance, native intelligent full-duplex SLMs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 7 minor

Summary. This paper introduces Lychee-FD, a native end-to-end full-duplex spoken language model. The authors first analyze gradient dynamics in a shared-parameter CDM architecture, reporting that gradient cosine similarity between text and speech objectives turns negative in deep layers (optimization divergence) and that temporal alignment with padding tokens suppresses semantic gradient magnitude (semantic dilution). Based on this analysis, they propose (1) a hierarchical parameter separation strategy that splits deep layers into parallel text, acoustic, and control heads, and (2) a semantic alignment channel that provides dense textual supervision. Experiments on Spoken QA benchmarks (LlamaQ, WebQ, TriviaQA) and full-duplex interaction benchmarks (FDBench, FullDuplexBench 1.0/1.5) show Lychee-FD achieving state-of-the-art results, including +7.4% on Spoken QA and +28.5% on FullDuplexBench 1.5 over prior baselines, while matching the half-duplex backbone's knowledge retention without increasing inference depth.

Significance. The paper makes a practical contribution to full-duplex SLMs: the hierarchical parameter separation architecture is well-motivated, and the empirical results across multiple benchmarks are strong, particularly the recovery and surpassing of the half-duplex backbone's QA accuracy (Table 1) and the low latency figures (Table 2). The open-sourcing of model weights, training pipeline, and code is a notable strength. The cross-architecture replication of the gradient cosine similarity pattern on Moshi (Figure 5) adds value by suggesting the phenomenon is not architecture-specific. The DAG-PP inference algorithm (Appendix 10) addresses a real deployment concern. However, the strength of the causal claim about gradient conflict being the 'root cause' of modality interference is not fully supported by the current experimental design, as detailed below.

major comments (3)
  1. §4.4, Table 1 (w/o Param-Sep ablation): The primary ablation comparing the full model against 'w/o Param-Sep' confounds architectural separation with total parameter count. The full model uses 24 shared + 4 text + 4 acoustic + 2 control = 34 layer-blocks (~10B params, per §4.3), while the 'fully shared architecture' appears to revert to the original 28-layer backbone. This means the separated model has roughly 21% more total parameters. The catastrophic S→S accuracy drop to 27.6% in the ablation could therefore be driven by reduced capacity rather than unresolved gradient conflict. The paper does not include a parameter-matched shared control (e.g., 34 shared layers, or a single shared 10-layer head replacing the three separated heads). Without such a control, the ablation cannot isolate the effect of parameter separation from the effect of added capacity, which is load-bearing for the中央
  2. Appendix 9, Eq. (9): The global gradient influence analysis is presented as causal validation that gradient conflict was the root cause of the original degradation. However, once parameters are separated into disjoint heads, the head-specific gradients operate on non-overlapping parameter sets and cannot conflict by construction. The shift from negative to positive influence scores (Figure 9, right) is therefore partly mechanical: the remaining influence flows only through the shared backbone gradients, which Figure 2a already shows are synergistic in shallow layers. This analysis does not independently distinguish between 'gradient conflict was the root cause' and 'removing deep-layer shared parameters eliminates any possibility of conflict regardless of cause.' The paper should either acknowledge this limitation or provide a control that tests the causal claim more directly (e.g., a PC
  3. §3.1: The gradient analysis (Figure 2) uses only 1K samples from the training set without parameter updates. While this is sufficient to show a correlation between negative cosine similarity and the known performance degradation, the paper repeatedly frames this as uncovering the 'root cause' (Abstract, §1, §3.1, §6). The correlation between gradient conflict and degradation does not establish causation; alternative explanations (e.g., suboptimal loss weighting, learning-rate imbalance, data distribution mismatch) are not ruled out. The paper should either moderate the causal language or provide an intervention that directly tests whether resolving gradient conflict (as opposed to simply adding capacity) is what drives the improvement.
minor comments (7)
  1. Figure 6 (Appendix 7): The layer ablation does not control for total parameter count—adding more separated layers always increases total parameters. The saturation at 4 layers is therefore ambiguous between 'enough separation to resolve conflict' and 'enough capacity.' A note acknowledging this would strengthen the analysis.
  2. §4.3: The paper states the model has 'approximately 10B total parameters' but does not report the parameter count of the w/o Param-Sep baseline or the half-duplex backbone. Including these numbers would make the capacity confound more transparent.
  3. Table 1: The 'w/o Param-Sep' row shows S→T accuracy of 67.0 on LlamaQ, which is only slightly below the full model's 73.7, yet S→S drops catastrophically to 36.0. The paper attributes this to acoustic modeling being suppressed, but this asymmetry (text survives, speech collapses) is not fully explained. A brief discussion of why text generation is more robust would help.
  4. §3.2: The loss function in Eq. (7) sums cross-entropy losses across heads without weighting. The paper does not discuss whether loss weighting was tuned or whether equal weighting is optimal. Given the gradient magnitude ratio analysis (Figure 2b), some discussion of whether loss reweighting was considered as an alternative to architectural separation would be informative.
  5. The paper uses 'root cause' language extensively (Abstract, §1, §3.1, §6). Given that the evidence is correlational and the ablation is confounded, this language should be moderated to 'a significant contributing factor' or similar.
  6. References: Lei et al. (2026) is cited for the global gradient influence metric (Appendix 9) but does not appear in the reference list. Please add the full citation.
  7. §4.5, Table 2: The 'Lat.' column for Lychee-FD on FullDuplexBench 1.5 is 826ms, which is described as 'lowest latency.' It would help to clarify whether this includes the DAG-PP parallel inference overhead or reflects single-GPU inference.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and substantive review. The comments correctly identify that our causal claims about gradient conflict are not fully supported by the current experimental design. We agree to moderate the causal language and add controls where feasible. Below we address each comment point by point.

read point-by-point responses

  1. Referee: §4.4, Table 1 (w/o Param-Sep ablation): The primary ablation comparing the full model against 'w/o Param-Sep' confounds architectural separation with total parameter count... Without such a control, the ablation cannot isolate the effect of parameter separation from the effect of added capacity.

    Authors: The referee is correct that the current ablation confounds parameter separation with added capacity. The full model uses 24 shared + 10 specialized = 34 layer-blocks, while the 'w/o Param-Sep' variant reverts to the original 28-layer backbone, creating an approximately 21% parameter difference that is not controlled for. We will address this in revision in two ways. First, we will add a parameter-matched shared control: a 34-layer fully shared architecture (28 original + 6 additional shared layers) trained with identical data and hyperparameters. This isolates the effect of separation from the effect of capacity. Second, we note that our layer ablation study (Appendix 7, Figure 6) already provides partial evidence against a pure capacity explanation: accuracy jumps from 36.0% to 65.4% as separated layers increase from 0 to 4, but saturates beyond that point. If added capacity were the primary driver, we would expect a more monotonic relationship rather than the sharp threshold we observe. That said, we agree this is not a substitute for a direct parameter-matched control, and we will include one in the revised manuscript. We will also explicitly acknowledge the confound in the discussion of the original ablation. revision: yes

  2. Referee: Appendix 9, Eq. (9): The global gradient influence analysis is presented as causal validation that gradient conflict was the root cause... once parameters are separated into disjoint heads, the head-specific gradients operate on non-overlapping parameter sets and cannot conflict by construction. The shift from negative to positive influence scores is therefore partly mechanical... This analysis does not independently distinguish between 'gradient conflict was the root cause' and 'removing deep-layer shared parameters eliminates any possibility of conflict regardless of cause.'

    Authors: This is a valid logical objection. Once parameters are separated into disjoint heads, direct gradient conflict on those head-specific parameters is eliminated by construction, so the improvement in influence scores is at least partly mechanical rather than evidence that conflict was the original cause. We concede this point. In the revision, we will reframe the Appendix 9 analysis as what it actually demonstrates: that the shared baseline exhibits destructive cross-task influence (negative scores) on overlapping parameters, and that separation eliminates this interference. We will explicitly state that this analysis cannot independently distinguish between 'gradient conflict was the root cause' and 'separation removes the possibility of conflict regardless of cause.' We will remove language characterizing this analysis as 'causal validation' and instead describe it as corroborating evidence consistent with the gradient conflict hypothesis. To more directly test the causal claim, we will add an intervention experiment: gradient surgery (PCGrad-style projection) applied to the shared architecture, which resolves gradient conflict without adding capacity or separating parameters. If gradient conflict is indeed the primary driver, gradient surgery on the shared model should recover a substantial portion of the performance gap. This provides a cleaner causal test. revision: yes

  3. Referee: §3.1: The gradient analysis uses only 1K samples without parameter updates... The paper repeatedly frames this as uncovering the 'root cause'... The correlation between gradient conflict and degradation does not establish causation; alternative explanations (e.g., suboptimal loss weighting, learning-rate imbalance, data distribution mismatch) are not ruled out.

    Authors: We agree that the gradient cosine similarity analysis on 1K samples without parameter updates establishes a correlation, not a causal relationship, and that alternative explanations (loss weighting, learning-rate imbalance, data distribution mismatch) are not ruled out by this analysis alone. The referee is correct that our language is too strong. In the revision, we will moderate the causal claims throughout the paper. Specifically: (1) In the Abstract, Introduction, and Conclusion, we will replace 'root cause' with 'a key contributing factor' or 'a significant source of' modality interference. (2) In §3.1, we will explicitly acknowledge that the analysis is correlational and list alternative explanations that are not ruled out. (3) We will note that the cross-architecture replication on Moshi (Figure 5) strengthens the correlational evidence by showing the pattern generalizes, but does not establish causation. (4) The gradient surgery intervention mentioned in our response to Comment 2, combined with the parameter-matched control from Comment 1, will together provide a more rigorous test of whether resolving gradient conflict specifically (as opposed to adding capacity) drives the improvement. If these additional experiments do not fully support the causal claim, we will adjust the framing accordingly. revision: yes

Circularity Check

2 steps flagged

Global gradient influence analysis is near-tautological after parameter separation, but the paper's main empirical claims rest on independent benchmarks

specific steps
  1. fitted input called prediction [Appendix 9, Eq. 9 and surrounding text]
    "Inspired by Lei et al. (2026), we quantify how updating parameters for one task causally affects the performance of another. Specifically, we calculate the normalized global Influence Score I_{m→i} of task m on task i (m, i ∈ {T, A}): I_{m→i} = [L_i(θ) − L_i(θ − ηg_m)] / [L_i(θ) − L_i(θ − ηg_i)] ... In contrast, under our Hierarchical Parameter Separation (right), this destructive interference is entirely resolved. Notably, the influence score of text on speech shifts to a positive value (0.205)."

    The influence score I_{m→i} measures whether applying task m's gradient to the global parameters θ helps or hurts task i. In the fully shared baseline, θ is shared, so g_m (gradient from task m) directly modifies parameters used by task i — a negative score is meaningful. But after Hierarchical Parameter Separation, the text and speech heads have disjoint parameter sets θ_T and θ_A. When computing I_{T→A}, the gradient g_T is taken w.r.t. θ_T, so θ − ηg_T only modifies text-head parameters. The speech loss L_A(θ − ηg_T) is evaluated on parameters that were not changed by g_T (except possibly shared backbone). The positive influence score (0.205) is therefore largely an artifact of the architecture: once parameters are separated, the text gradient cannot directly corrupt speech parameters,,

  2. self citation load bearing [Appendix 9, Eq. 9]
    "Inspired by Lei et al. (2026), we quantify how updating parameters for one task causally affects the performance of another."

    The influence score formula (Eq. 9) is attributed to Lei et al. (2026). Checking the reference list: 'Yishu Lei, Shuwei He, Jing Hu, Dan Zhang, Xianlong Luo, Danxiang Zhu, Shikun Feng, Rui Liu, Jingzhou He, Yu Sun, Hua Wu, and Haifeng Wang. Moe adapter for large audio language models: Sparsity, disentanglement, and gradient-conflict-free. CoRR, abs/2601.02967, 2026.' The title itself contains 'gradient-conflict-free,' suggesting the cited work already assumes gradient conflict is the problem and proposes a solution. The present paper uses this formula to validate that its own separation strategy resolves gradient conflict. However, the formula is a single-step loss-difference proxy, not a multi-step causal proof. The citation provides the metric, and the metric is then applied to show the,

full rationale

The paper's core empirical results (Table 1, Table 2) are evaluated on independent external benchmarks (LlamaQ, WebQ, TriviaQA, FullDuplexBench 1.0/1.5, FDBench) against external baselines (Moshi, VITA-1.5, Freeze-Omni, etc.), so the main performance claims are not circular. The gradient cosine similarity analysis (Section 3.1, Eqs. 1-4) is a standard first-principles diagnostic applied to a pre-existing model (StepAudio-2-mini), not a self-referential construction. The two flagged items are secondary: (1) the global gradient influence analysis (Appendix 9) is near-tautological after parameter separation — disjoint parameter sets cannot exhibit direct gradient conflict by construction — but this is used as supplementary validation, not as the primary evidence for the paper's claims; (2) the Lei et al. (2026) citation provides the influence score formula, and while that work's title suggests thematic overlap ('gradient-conflict-free'), the formula itself is a generic loss-difference metric, not a result that assumes the present paper's conclusion. The primary ablation (w/o Param-Sep) has a confound with parameter count (34 vs. 28 layers), but that is a correctness/experimental design concern, not circularity. Overall, the derivation chain is largely self-contained against external benchmarks, with one near-tautological supplementary analysis and one minor self-citation that is not load-bearing for the central claims.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The paper does not invent new entities (particles, forces, dimensions). It introduces architectural components (Semantic Head, Acoustic Head, Control Head, Semantic Alignment Channel) and a training pipeline, but these are engineering constructs, not postulated physical or mathematical objects requiring independent evidence.

free parameters (4)
  • Number of shared backbone layers = 24
    Chosen based on the StepAudio-2-mini architecture; the layer ablation (Appendix 7, Figure 6) shows 4 separated layers as optimal but the 24-layer split point for shared vs. specialized is a design choice.
  • Specialized head layer counts (Text/Acoustic/Control) = 4/4/2
    Selected via ablation in Appendix 7; the text and acoustic heads get 4 layers each, control gets 2.
  • Training data size = 140K
    Filtered from a larger synthesized set; the filtering criteria and final size are design choices.
  • Backchannel injection probability = 0.5
    Set in the data synthesis pipeline (Appendix 11.2) for user backchannel injection during post-processing.
axioms (4)
  • domain assumption Gradient cosine similarity and magnitude ratio between per-layer parameters are sufficient statistics to diagnose modality interference in transformer-based SLMs.
    Section 3.1. The paper reduces the optimization dynamics analysis to these two geometric measures, assuming they capture the essential conflict. This is a modeling assumption about what 'root cause' means.
  • domain assumption A one-step parameter update influence score (Eq. 9) adequately approximates the causal effect of one task's gradient on another task's loss across multi-step training.
    Appendix 9. The influence score I_{m→i} uses a single gradient step to quantify cross-task interference, but actual training involves thousands of steps with momentum, scheduling, and non-linear interactions.
  • domain assumption Synthesized full-duplex dialogue data generated by LLM agents with rule-based constraints is representative enough of real human conversation to train a deployable full-duplex SLM.
    Section 4.2 and Appendix 11. The entire training set is synthetic, and the paper acknowledges in Limitations that side-talk discrimination remains unsolved due to data coverage gaps.
  • domain assumption StepAudio-2-mini is a representative and sufficient backbone for demonstrating that gradient conflicts are a universal phenomenon in FDSLMs.
    Section 3.1 and Section 5. The primary analysis uses StepAudio-2-mini; Moshi is used as secondary validation. Only two architectures are tested.

pith-pipeline@v1.1.0-glm · 27667 in / 3134 out tokens · 542082 ms · 2026-07-08T02:18:27.152060+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

55 extracted references · 55 canonical work pages · 21 internal anchors

  1. [1]

    Qwen2.5-Omni Technical Report

    Jin Xu and Zhifang Guo and Jinzheng He and Hangrui Hu and Ting He and Shuai Bai and Keqin Chen and Jialin Wang and Yang Fan and Kai Dang and Bin Zhang and Xiong Wang and Yunfei Chu and Junyang Lin , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2503.20215 , eprinttype =. 2503.20215 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2503.20215 2025

  2. [2]

    Step-Audio 2 Technical Report

    Boyong Wu and Chao Yan and Chen Hu and Cheng Yi and Chengli Feng and Fei Tian and Feiyu Shen and Gang Yu and Haoyang Zhang and Jingbei Li and Mingrui Chen and Peng Liu and Wang You and Xiangyu Tony Zhang and Xingyuan Li and Xuerui Yang and Yayue Deng and Yechang Huang and Yuxin Li and Yuxin Zhang and Zhao You and Brian Li and Changyi Wan and Hanpeng Hu an...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2507.16632 2025

  3. [3]

    AnyGPT: Unified Multimodal

    Jun Zhan and Junqi Dai and Jiasheng Ye and Yunhua Zhou and Dong Zhang and Zhigeng Liu and Xin Zhang and Ruibin Yuan and Ge Zhang and Linyang Li and Hang Yan and Jie Fu and Tao Gui and Tianxiang Sun and Yu. AnyGPT: Unified Multimodal. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),. 2024 , ur...

    work page doi:10.18653/v1/2024.acl-long.521 2024

  4. [4]

    2024 , url=

    GPT-4o System Card , author=. 2024 , url=

    work page 2024

  5. [5]

    Forty-second International Conference on Machine Learning,

    Xiong Wang and Yangze Li and Chaoyou Fu and Yike Zhang and Yunhang Shen and Lei Xie and Ke Li and Xing Sun and Long Ma , title =. Forty-second International Conference on Machine Learning,. 2025 , url =

    work page 2025

  6. [6]

    arXiv preprint arXiv:2511.12609 , year=

    Uni-moe-2.0-omni: Scaling language-centric omnimodal large model with advanced moe, training and data , author=. arXiv preprint arXiv:2511.12609 , year=

    work page arXiv

  7. [7]

    Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models

    Perception, reason, think, and plan: A survey on large multimodal reasoning models , author=. arXiv preprint arXiv:2505.04921 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    Moshi: a speech-text foundation model for real-time dialogue

    Alexandre D. Moshi: a speech-text foundation model for real-time dialogue , journal =. 2024 , url =. doi:10.48550/ARXIV.2410.00037 , eprinttype =. 2410.00037 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2410.00037 2024

  9. [9]

    VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction

    Chaoyou Fu and Haojia Lin and Xiong Wang and Yifan Zhang and Yunhang Shen and Xiaoyu Liu and Haoyu Cao and Zuwei Long and Heting Gao and Ke Li and Long Ma and Xiawu Zheng and Rongrong Ji and Xing Sun and Caifeng Shan and Ran He , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2501.01957 , eprinttype =. 2501.01957 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2501.01957 2025

  10. [10]

    Beyond the Turn-Based Game: Enabling Real-Time Conversations with Duplex Models , booktitle =

    Xinrong Zhang and Yingfa Chen and Shengding Hu and Xu Han and Zihang Xu and Yuanwei Xu and Weilin Zhao and Maosong Sun and Zhiyuan Liu , editor =. Beyond the Turn-Based Game: Enabling Real-Time Conversations with Duplex Models , booktitle =. 2024 , url =. doi:10.18653/V1/2024.EMNLP-MAIN.644 , timestamp =

    work page doi:10.18653/v1/2024.emnlp-main.644 2024

  11. [11]

    OmniFlatten: An End-to-end

    Qinglin Zhang and Luyao Cheng and Chong Deng and Qian Chen and Wen Wang and Siqi Zheng and Jiaqing Liu and Hai Yu and Chao. OmniFlatten: An End-to-end. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),. 2025 , url =

    work page 2025

  12. [12]

    Peloquin and Bokai Yu and Hongyu Gong and Shyamnath Gollakota , editor =

    Bandhav Veluri and Benjamin N. Peloquin and Bokai Yu and Hongyu Gong and Shyamnath Gollakota , editor =. Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents , booktitle =. 2024 , url =. doi:10.18653/V1/2024.EMNLP-MAIN.1192 , timestamp =

    work page doi:10.18653/v1/2024.emnlp-main.1192 2024

  13. [13]

    Lost in the Middle: How Language Models Use Long Contexts

    Tu Anh Nguyen and Eugene Kharitonov and Jade Copet and Yossi Adi and Wei. Generative Spoken Dialogue Language Modeling , journal =. 2023 , url =. doi:10.1162/TACL\_A\_00545 , timestamp =

    work page internal anchor Pith review doi:10.1162/tacl 2023

  14. [14]

    CoRR , volume =

    Yiqun Yao and Xiang Li and Xin Jiang and Xuezhi Fang and Naitong Yu and Wenjia Ma and Aixin Sun and Yequan Wang , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2509.02521 , eprinttype =. 2509.02521 , timestamp =

    work page doi:10.48550/arxiv.2509.02521 2025

  15. [15]

    FireRedChat: A Pluggable, Full-Duplex Voice Interaction System with Cascaded and Semi-Cascaded Implementations

    Junjie Chen and Yao Hu and Junjie Li and Kangyue Li and Kun Liu and Wenpeng Li and Xu Li and Ziyuan Li and Feiyu Shen and Xu Tang and Manzhen Wei and Yichen Wu and Fenglong Xie and Kaituo Xu and Kun Xie , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2509.06502 , eprinttype =. 2509.06502 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2509.06502 2025

  16. [16]

    FlexDuo: A Pluggable System for Enabling Full-Duplex Capabilities in Speech Dialogue Systems

    Borui Liao and Yulong Xu and Jiao Ou and Kaiyuan Yang and Weihua Jian and Pengfei Wan and Di Zhang , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2502.13472 , eprinttype =. 2502.13472 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2502.13472 2025

  17. [17]

    CoRR , volume =

    Guojian Li and Chengyou Wang and Hongfei Xue and Shuiyuan Wang and Dehui Gao and Zihan Zhang and Yuke Lin and Wenjie Li and Longshuai Xiao and Zhonghua Fu and Lei Xie , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2509.23938 , eprinttype =. 2509.23938 , timestamp =

    work page doi:10.48550/arxiv.2509.23938 2025

  18. [18]

    SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities , booktitle =

    Dong Zhang and Shimin Li and Xin Zhang and Jun Zhan and Pengyu Wang and Yaqian Zhou and Xipeng Qiu , editor =. SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities , booktitle =. 2023 , url =. doi:10.18653/V1/2023.FINDINGS-EMNLP.1055 , timestamp =

    work page doi:10.18653/v1/2023.findings-emnlp.1055 2023

  19. [19]

    The Thirteenth International Conference on Learning Representations,

    Qingkai Fang and Shoutao Guo and Yan Zhou and Zhengrui Ma and Shaolei Zhang and Yang Feng , title =. The Thirteenth International Conference on Learning Representations,. 2025 , url =

    work page 2025

  20. [20]

    The Thirteenth International Conference on Learning Representations,

    Aohan Zeng and Zhengxiao Du and Mingdao Liu and Lei Zhang and Shengmin Jiang and Yuxiao Dong and Jie Tang , title =. The Thirteenth International Conference on Learning Representations,. 2025 , url =

    work page 2025

  21. [21]

    2025 , eprint=

    Fun-Audio-Chat Technical Report , author=. 2025 , eprint=

    work page 2025

  22. [22]

    CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models

    Zhihao Du and Yuxuan Wang and Qian Chen and Xian Shi and Xiang Lv and Tianyu Zhao and Zhifu Gao and Yexin Yang and Changfeng Gao and Hui Wang and Fan Yu and Huadai Liu and Zhengyan Sheng and Yue Gu and Chong Deng and Wen Wang and Shiliang Zhang and Zhijie Yan and Jingren Zhou , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2412.10117 , eprint...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2412.10117 2024

  23. [23]

    7th International Conference on Learning Representations,

    Ilya Loshchilov and Frank Hutter , title =. 7th International Conference on Learning Representations,. 2019 , url =

    work page 2019

  24. [24]

    Findings of the Association for Computational Linguistics:

    Kentaro Mitsui and Koh Mitsuda and Toshiaki Wakatsuki and Yukiya Hono and Kei Sawada , editor =. Findings of the Association for Computational Linguistics:. 2024 , url =. doi:10.18653/V1/2024.FINDINGS-EMNLP.151 , timestamp =

    work page doi:10.18653/v1/2024.findings-emnlp.151 2024

  25. [25]

    Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming

    Zhifei Xie and Changqiao Wu , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2408.16725 , eprinttype =. 2408.16725 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2408.16725 2024

  26. [26]

    Robust Speech Recognition via Large-Scale Weak Supervision , booktitle =

    Alec Radford and Jong Wook Kim and Tao Xu and Greg Brockman and Christine McLeavey and Ilya Sutskever , editor =. Robust Speech Recognition via Large-Scale Weak Supervision , booktitle =. 2023 , url =

    work page 2023

  27. [27]

    FD-Bench:

    Yizhou Peng and Yi. FD-Bench:. 26th Annual Conference of the International Speech Communication Association, Interspeech 2025, Rotterdam, The Netherlands, 17-21 August 2025 , publisher =. 2025 , url =. doi:10.21437/INTERSPEECH.2025-739 , timestamp =

    work page doi:10.21437/interspeech.2025-739 2025

  28. [28]

    Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities

    Guan. Full-Duplex-Bench:. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2503.04721 , eprinttype =. 2503.04721 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2503.04721 2025

  29. [29]

    23rd Annual Conference of the International Speech Communication Association, Interspeech 2022, Incheon, Korea, September 18-22, 2022 , pages =

    Takaaki Saeki and Detai Xin and Wataru Nakata and Tomoki Koriyama and Shinnosuke Takamichi and Hiroshi Saruwatari , editor =. 23rd Annual Conference of the International Speech Communication Association, Interspeech 2022, Incheon, Korea, September 18-22, 2022 , pages =. 2022 , url =. doi:10.21437/INTERSPEECH.2022-439 , timestamp =

    work page doi:10.21437/interspeech.2022-439 2022

  30. [30]

    FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs

    Keyu An and Qian Chen and Chong Deng and Zhihao Du and Changfeng Gao and Zhifu Gao and Yue Gu and Ting He and Hangrui Hu and Kai Hu and Shengpeng Ji and Yabin Li and Zerui Li and Heng Lu and Haoneng Luo and Xiang Lv and Bin Ma and Ziyang Ma and Chongjia Ni and Changhe Song and Jiaqi Shi and Xian Shi and Hao Wang and Wen Wang and Yuxuan Wang and Zhangyu Xi...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2407.04051 2024

  31. [31]

    ArXiv , year=

    MiMo-Audio: Audio Language Models are Few-Shot Learners , author=. ArXiv , year=

    work page

  32. [32]

    CoRR , volume =

    Yishu Lei and Shuwei He and Jing Hu and Dan Zhang and Xianlong Luo and Danxiang Zhu and Shikun Feng and Rui Liu and Jingzhou He and Yu Sun and Hua Wu and Haifeng Wang , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2601.02967 , eprinttype =. 2601.02967 , timestamp =

    work page doi:10.48550/arxiv.2601.02967 2026

  33. [33]

    WavChat: A Survey of Spoken Dialogue Models

    Shengpeng Ji and Yifu Chen and Minghui Fang and Jialong Zuo and Jingyu Lu and Hanting Wang and Ziyue Jiang and Long Zhou and Shujie Liu and Xize Cheng and Xiaoda Yang and Zehan Wang and Qian Yang and Jian Li and Yidi Jiang and Jingzhen He and Yunfei Chu and Jin Xu and Zhou Zhao , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2411.13577 , epri...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2411.13577 2024

  34. [34]

    On The Landscape of Spoken Language Models:

    Siddhant Arora and Kai. On The Landscape of Spoken Language Models:. Trans. Mach. Learn. Res. , volume =. 2025 , url =

    work page 2025

  35. [35]

    Duplex Conversation: Towards Human-like Interaction in Spoken Dialogue Systems , booktitle =

    Ting. Duplex Conversation: Towards Human-like Interaction in Spoken Dialogue Systems , booktitle =. 2022 , url =. doi:10.1145/3534678.3539209 , timestamp =

    work page doi:10.1145/3534678.3539209 2022

  36. [36]

    Efficient and Direct Duplex Modeling for Speech-to-Speech Language Model , booktitle =

    Ke Hu and Ehsan Hosseini. Efficient and Direct Duplex Modeling for Speech-to-Speech Language Model , booktitle =. 2025 , url =. doi:10.21437/INTERSPEECH.2025-874 , timestamp =

    work page doi:10.21437/interspeech.2025-874 2025

  37. [37]

    Toward Enabling Natural Conversation with Older Adults via the Design of LLM-Powered Voice Agents that Support Interruptions and Backchannels , booktitle =

    Chao Liu and Mingyang Su and Yan Xiang and Yuru Huang and Yiqian Yang and Kang Zhang and Mingming Fan , editor =. Toward Enabling Natural Conversation with Older Adults via the Design of LLM-Powered Voice Agents that Support Interruptions and Backchannels , booktitle =. 2025 , url =. doi:10.1145/3706598.3714228 , timestamp =

    work page doi:10.1145/3706598.3714228 2025

  38. [38]

    A Preliminary Exploration with GPT-4o Voice Mode

    Yu. A Preliminary Exploration with GPT-4o Voice Mode , journal =. 2025 , url =. doi:10.48550/ARXIV.2502.09940 , eprinttype =. 2502.09940 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2502.09940 2025

  39. [39]

    Advances in Neural Information Processing Systems , year=

    SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation , author=. Advances in Neural Information Processing Systems , year=

    work page

  40. [40]

    Schegloff and Gail Jefferson , journal =

    Harvey Sacks and Emanuel A. Schegloff and Gail Jefferson , journal =. A Simplest Systematics for the Organization of Turn-Taking for Conversation , urldate =

    work page

  41. [41]

    Scaling Laws for Neural Language Models

    Jared Kaplan and Sam McCandlish and Tom Henighan and Tom B. Brown and Benjamin Chess and Rewon Child and Scott Gray and Alec Radford and Jeffrey Wu and Dario Amodei , title =. CoRR , volume =. 2020 , url =. 2001.08361 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  42. [42]

    and Sifre, Laurent , title =

    Hoffmann, Jordan and Borgeaud, Sebastian and Mensch, Arthur and Buchatskaya, Elena and Cai, Trevor and Rutherford, Eliza and de Las Casas, Diego and Hendricks, Lisa Anne and Welbl, Johannes and Clark, Aidan and Hennigan, Tom and Noland, Eric and Millican, Katie and van den Driessche, George and Damoc, Bogdan and Guy, Aurelia and Osindero, Simon and Simony...

    work page 2022

  43. [43]

    An Overview of Multi-Task Learning in Deep Neural Networks

    Sebastian Ruder , title =. CoRR , volume =. 2017 , url =. 1706.05098 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  44. [44]

    Gradient Surgery for Multi-Task Learning , url =

    Yu, Tianhe and Kumar, Saurabh and Gupta, Abhishek and Levine, Sergey and Hausman, Karol and Finn, Chelsea , booktitle =. Gradient Surgery for Multi-Task Learning , url =

    work page

  45. [45]

    Multi-Task Learning as Multi-Objective Optimization , url =

    Sener, Ozan and Koltun, Vladlen , booktitle =. Multi-Task Learning as Multi-Objective Optimization , url =

    work page

  46. [46]

    AudioLM:

    Zal. AudioLM:. 2023 , url =. doi:10.1109/TASLP.2023.3288409 , timestamp =

    work page doi:10.1109/taslp.2023.3288409 2023

  47. [47]

    Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers , year=

    Chen, Sanyuan and Wang, Chengyi and Wu, Yu and Zhang, Ziqiang and Zhou, Long and Liu, Shujie and Chen, Zhuo and Liu, Yanqing and Wang, Huaming and Li, Jinyu and He, Lei and Zhao, Sheng and Wei, Furu , journal=. Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers , year=

    work page

  48. [48]

    2022 , url =

    Neil Zeghidour and Alejandro Luebs and Ahmed Omran and Jan Skoglund and Marco Tagliasacchi , title =. 2022 , url =. doi:10.1109/TASLP.2021.3129994 , timestamp =

    work page doi:10.1109/taslp.2021.3129994 2022

  49. [49]

    Clifton , title =

    Peng Xu and Xiatian Zhu and David A. Clifton , title =. 2023 , url =. doi:10.1109/TPAMI.2023.3275156 , timestamp =

    work page doi:10.1109/tpami.2023.3275156 2023

  50. [50]

    Machine Learning , year =

    Caruana, Rich , title =. Machine Learning , year =. doi:10.1023/A:1007379606734 , url =

    work page doi:10.1023/a:1007379606734

  51. [51]

    VITA: Towards Open-Source Interactive Omni Multimodal LLM

    Chaoyou Fu and Haojia Lin and Zuwei Long and Yunhang Shen and Meng Zhao and Yifan Zhang and Xiong Wang and Di Yin and Long Ma and Xiawu Zheng and Ran He and Rongrong Ji and Yunsheng Wu and Caifeng Shan and Xing Sun , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2408.05211 , eprinttype =. 2408.05211 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2408.05211 2024

  52. [52]

    Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities

    Zhifei Xie and Changqiao Wu , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2410.11190 , eprinttype =. 2410.11190 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2410.11190 2024

  53. [53]

    LLM-Enhanced Dialogue Management for Full-Duplex Spoken Dialogue Systems

    Hao Zhang and Weiwei Li and Rilin Chen and Vinay Kothapally and Meng Yu and Dong Yu , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2502.14145 , eprinttype =. 2502.14145 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2502.14145 2025

  54. [54]

    GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot

    Aohan Zeng and Zhengxiao Du and Mingdao Liu and Kedong Wang and Shengmin Jiang and Lei Zhao and Yuxiao Dong and Jie Tang , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2412.02612 , eprinttype =. 2412.02612 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2412.02612 2024

  55. [55]

    LUCY: Linguistic Understanding and Control Yielding Early Stage of Her

    Heting Gao and Hang Shao and Xiong Wang and Chaofan Qiu and Yunhang Shen and Siqi Cai and Yuchen Shi and Zihan Xu and Zuwei Long and Yike Zhang and Shaoqi Dong and Chaoyou Fu and Ke Li and Long Ma and Xing Sun , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2501.16327 , eprinttype =. 2501.16327 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2501.16327 2025