Pith. sign in

REVIEW 4 major objections 6 minor 4 cited by

BoSS: Beyond-Semantic Speech

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Current open spoken language models largely fail to interpret emotion, dialect, age, and non-verbal signals, a gap the paper names Beyond-Semantic Speech (BoSS).

desk verdict A readable position-plus-probe paper that introduces BoSS and L1–L5 and reports consistently low SLM scores on beyond-semantic tasks, but the central empirical claim is undercut by missing evaluation artifacts, an unreported scaling parameter, and no human baseline. read the letter →

arxiv 2507.17563 v1 pith:XDYNSEVQ submitted 2025-07-23 cs.SD cs.CLeess.AS

Qing Wang , Zehan Li , Hang Lv , Hongjie Chen , Yaodong Song , Jian Kang , Jie Lian , Jie Li
show 3 more authors
Yongxiang Li Zhongjiang He Xuelong Li
This is my paper
classification cs.SDcs.CLeess.AS
keywords Beyond-SemanticSpeechSpokenInteractionSystemCapabilityLevelsParalinguisticsLanguageModelsEmotionPerceptionandResponseDialectComprehensionNon-verbalvocalsignalsAge
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that current end-to-end spoken language models (SLMs) are good at words but largely deaf to everything in speech that is not words. It names that extra layer Beyond-Semantic Speech (BoSS)—affective cues, contextual dynamics, and implicit semantics layered on top of explicit semantics—and organizes it into a five-level capability ladder for spoken interaction systems. Across five probe tasks (dialect comprehension and following, context memory, emotion perception and response, age perception and response, non-verbal vocal signals), seven open-source SLMs score well only on context memory and show consistently low performance on the other four. The paper's point is that these beyond-semantic capabilities are missing from current models and need to be modelled explicitly on the way to human-level conversational intelligence. If the finding is right, speech-AI benchmarks must start measuring the non-literal layer of communication, not just word accuracy.

What carries the argument

The load-bearing mechanism is the BoSS framework's Optimal Meaning Hypothesis: choose, at each time step, the interpretation $H$ with the highest ratio of cognitive effect $E(H, O_t, C_t)$ to processing effort $P(H, O_t, C_t)$, where both are learned neural-network scorers and temporal dynamics are carried by a hidden Markov model whose emission probability is a softmax over those relevance ratios. This turns Relevance Theory's formula for communicative relevance into a decision rule for machines, and the observation vector $O_t$ plus context vector $C_t$ specify what beyond-semantic information must enter that rule. The L1–L5 capability ladder does the organizational work, placing dialect, emotion, age, and non-verbal responsiveness at the levels (L4–L5) where current models fall short.

What would settle it

Run the paper's five test sets with human raters in place of the LLM judge: if humans judge most current SLM responses as emotionally and age appropriate and dialect-consistent, or if a different judge model materially changes the rankings, the claim that SLMs are hard to fully interpret beyond-semantic signals is refuted.

Watch

Extended reading notes

Core claim

The paper reports that the information which "encompasses but transcends explicit semantics"—emotional tone, accent, speaker age, pauses, coughs, laughter, and the way these reshape literal meaning—is not being recovered by today's end-to-end SLMs. The evidence is a set of five evaluations: dialect comprehension varies by model but dialect following is poor across the board; multi-turn context memory is the only dimension where most models score well; emotion perception and response, age-appropriate response, and responsiveness to non-verbal vocal signals all fall well below what human interaction would require. The paper formalizes the missing layer as an optimization problem: at each time step the system should select the meaning hypothesis $H^*$ that maximizes cognitive effect over processing effort, $H^*_t = \arg\max_{H} E(H)/P(H)$, with the observation vector $O_t = [V_{L,t}, V_{AC,t}, V_{CD,t}, V_{IS,t}]$ combining lexical, affective, contextual, and implicit semantic latents. The demonstrated gap is the paper's central claim: current SLMs are hard to fully interpret beyond-semantic signals.

Load-bearing premise

The whole finding depends on the assumption that the new test sets—synthesized dialect audio, LLM-judged emotion and age responses, and spliced non-verbal sounds—actually measure beyond-semantic understanding rather than quirks of the probes and the judge.

Editorial extensions

If this is right

  • If current SLMs really do miss BoSS, then high scores on ASR/TTS-style benchmarks do not imply conversational competence; the two must be measured separately.
  • Reaching the upper rungs of the L1–L5 ladder requires engineering BoSS dimensions into the model—affective, contextual, and implicit-semantic latents in the encoder and decoder—rather than scaling the language model alone.
  • Dialect generation appears to be the most tractable BoSS dimension, since the paper's own generation experiment cuts ASR word error rates by more than 70 percent for several dialects when dialect data is added.
  • The five evaluation sets provide a reusable baseline: future SLMs can be compared against the reported scores for dialect following, emotion response, age adaptation, and non-verbal responsiveness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test the paper leaves implicit: keep the text of a user utterance fixed while varying synthesized prosody, speaker age, or emotion; if an SLM's response does not change accordingly, the model is ignoring BoSS, and this could be used as a lightweight diagnostic.
  • The low scores partly reflect the judges: using an LLM judge with a power-scaled score and a dialect classifier may understate what models would show with human raters, so a human-judge replication would separate model failure from measurement failure.
  • A curriculum hinted at by the results: because context memory is the one dimension where models already succeed, future training could inject affective and contextual latents into the speech encoder first, then train response adaptation, rather than attempting all BoSS dimensions at once.
  • The dialect-generation result suggests generation-to-understanding bootstrapping: synthetic dialect audio from a controllable TTS could be used to create training data that teaches comprehension models the same accent cues, an avenue the paper does not pursue.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Beyond-Semantic Speech (BoSS) as a category of speech information beyond literal semantics, together with a five-level Spoken Interaction System Capability framework (L1–L5). It formalizes an "optimal meaning hypothesis" using a relevance ratio, softmax HMM emissions, and a variational mutual-information bound (Section 4), then evaluates seven open end-to-end spoken language models on Chinese dialect comprehension/following, multi-turn context memory, emotion perception/response, age perception/response, and non-verbal vocal signal responsiveness. The headline empirical finding is that current SLMs perform poorly on most beyond-semantic dimensions, especially dialect generation, age-aware response, and non-verbal responsiveness, while performing better on context memory. The paper concludes that current SLMs are "hard to fully interpret beyond-semantic signals."

Significance. The proposed BoSS taxonomy and L1–L5 levels could serve as a useful organizing frame for future spoken-dialogue benchmarks, and the paper's empirical study has real strengths: it uses publicly released models and datasets (ESD, Emotion2Vec, MagicData), uses objective string matching for the dialect-comprehension task to avoid LLM-judge artifacts, and the low scores are consistent across several independent models, making a coarse capability gap plausible. However, the headline conclusion is only as strong as the measurement protocols for emotion, age, and non-verbal responsiveness, and those protocols are underspecified and unvalidated against human judgment. The formal framework in Section 4 is not instantiated in any experiment, so the contribution at this stage is primarily a conceptual taxonomy plus a preliminary, partially validated benchmark rather than a fully supported computational framework.

major comments (4)
  1. [5.3.2, 5.4.2, Tables 3 and 5] The central claim that SLMs "are hard to fully interpret beyond-semantic signals" is carried by the low scores in Tables 3 and 5, but these scores are produced by GPT-4o with no human baseline, no released evaluation prompts, and no inter-annotator agreement. The text says only that "carefully designed evaluation prompts" were used (Section 5.4.2) and that each response was evaluated three times, but no prompt text, no human-judge correlation, and no agreement statistic are reported. Without evidence that GPT-4o's ratings track human judgments of age-appropriateness or emotional appropriateness, the uniformly low scores may reflect judge-model biases (e.g., toward length, politeness, or template phrasing) rather than missing BoSS capability. This is a load-bearing validity gap for the paper's main empirical conclusion; the objective dialect and context-memory results alone do not support the broad claim.
  2. [Eq. (10), Section 5.4.2, Table 5] Equation (10) defines the final age-task score with a "scaling factor p", but p is never reported anywhere in the manuscript and no sensitivity analysis is given. Because p is a free parameter, the absolute scores in Table 5 are not reproducible and are not comparable across settings unless p is fixed. At minimum the authors must report p, justify the choice, and show that the qualitative ranking of models is stable across a reasonable range of p. Additionally, no confidence intervals or standard errors are reported for any table, despite small samples (e.g., 150 age samples, 50 samples per emotion state in ESD-zh); this makes it impossible to tell which cross-model differences are meaningful.
  3. [5.5.2, Table 6] Table 6 (non-verbal responsiveness) is central to the claim that SLMs cannot respond appropriately to coughs, laughter, throat-clearing, and sneezing, yet the scoring procedure for this table is not described at all. The reader is not told what the score measures, whether it is produced by a model judge or by rule-based matching, how "attentiveness or concern" is operationalized, or how the 3,500 segments per category are combined with the AQA test set. Without this protocol, Table 6 cannot support any statement about model capability, and the difference between Kimi-Audio (9.19) and the remaining models (approximately 1.5–2.2) cannot be interpreted.
  4. [Section 4, Eqs. (2)–(9)] The formal framework—the optimal meaning hypothesis, the HMM emission in Eq. (7), and the variational mutual-information bound in Eq. (9)—is never instantiated or tested in Section 5. No system implements NNE, NNP, the HMM, or the Viterbi decoding mentioned in Section 4.1, and no experiment uses the quantities defined there. As a result, the paper's stated contribution of a "formalized framework" is currently a proposal only; the connection between the theory and the empirical benchmark is absent. The authors should either implement or simulate the framework, or clearly reframe Section 4 as a conceptual model and avoid implying that it has been validated.
minor comments (6)
  1. [Abstract] The abstract contains grammatical slips: "a hierarchical framework illustrated the evolution" should be "illustrating the evolution", and "We evaluate ... reveals" should be "reveals" or "and it reveals".
  2. [5.1.2 and Figure 3] There is a typo "we selecte" (should be "we selected"), and "the table 1" should be "Table 1"; also, the caption of Figure 3 labels panel (b) "Dialect Following" while the text refers to "dialectal chit-chat" and "dialect generation"—please align the terminology.
  3. [4.2, Eqs. (8)–(9)] Notation in Eqs. (8)–(9) is confusingly overloaded: E is used both for cognitive effect (Eq. 5) and for expectation, and the distributions P(ZL), V(ZL), and the conditioning on Zw are not fully defined before the KL expansion. Please define all distributions and use a distinct symbol for expectation.
  4. [Table 4] The audio-level emotion score in Table 4 says "scoring them with the Emotion2Vec model" but does not specify whether the score is top-1 classification accuracy, a similarity metric, or a probability; please state the exact metric and the reference labels used.
  5. [5.2.1 and Table 2] The context-memory test is described as having 50 samples per turn setting, but no test statistics or standard errors are given; also, Kimi-Audio is absent from Table 2 without explanation. Please report sample-level uncertainty and clarify the model coverage.
  6. [5.1.1] The dialect datasets are validated by ten native speakers per dialect, but no inter-rater agreement or inclusion criterion is reported; please quantify the validation.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; the only self-citation (GOAT-TTS) appears in a side sub-experiment and is not load-bearing for the central claim.

full rationale

The central empirical claim that current spoken language models underperform on beyond-semantic signals is supported by measuring publicly released SLMs (GLM-4-Voice, MiniCPM-o, Baichuan-Omni, SpeechGPT, Freeze-Omni, Qwen2.5-Omni, Kimi-Audio) on constructed and public datasets, including ESD for emotion, MagicData-derived text for dialect, and Emotion2Vec for audio scoring. These measurements are external to the paper's framework, so the conclusion is not derived from the framework's own definitions or fitted variables. The formal derivation in Section 4 uses a standard variational upper bound on mutual information (Eq. 8-9) and does not inject the target result as an assumption. The only explicit self-citation is GOAT-TTS, which the paper calls 'our previous work' and selects as the evaluation target for the dialect-generation sub-experiment (Section 5.1.2, Table 1); this supports a secondary capability demonstration and is not load-bearing for the main finding. Unspecified evaluation details, such as the hidden GPT-4o prompts, the unreported scaling factor p in Eq. 10, and the absent scoring description for Table 6, raise reproducibility and measurement-validity concerns but do not constitute circular reasoning because no fitted parameter is renamed as a prediction and no equation reduces to its own input. Accordingly, no circular steps are identified, and the score is 2 solely to reflect the one minor, non-load-bearing self-citation.

Assumptions & free parameters 1 free parameters · 4 assumptions · 2 invented entities

The paper's contribution rests on accepted cognitive theories and a standard variational information bound, plus the unverified validity of its own evaluation protocol. The theoretical part has no fitted parameters; the experimental part has one unspecified scaling parameter p and several unstated modeling choices.

free parameters (1)
  • p (power scaling factor in Eq 10) = not specified
    Rescales averaged 0-5 judge scores into final percentages; different p values change absolute scores and rankings, and the paper never gives the value used.
assumptions (4)
  • domain assumption Relevance of a message equals cognitive effect divided by processing effort, R = E/P (Eq 1).
    Adopted from Sperber and Wilson's Relevance Theory in Section 2.2 and used as the objective in Eq 2; the paper does not validate this ratio empirically.
  • domain assumption The best interpretation H* maximizes E(H, O_t, C_t)/P(H, O_t, C_t) (Eq 2).
    Converts cognitive relevance into an optimization criterion without evidence that human or machine communication obeys this argmax.
  • domain assumption A lossless speech encoder satisfies P(Z_L | U) = P(Z_L | Z_w) (Section 4.2).
    Needed to turn Eq 8 into the mutual-information identity in Eq 9; the paper calls it 'ideally' and never verifies it for real encoders.
  • ad hoc to paper Speech beyond-semantic states can be modeled as a hidden Markov process with softmax emissions over E/P (Eq 7).
    Assumed without fitting or evaluation; no HMM is trained.
invented entities (2)
  • Beyond-Semantic Speech (BoSS)
    purpose: Name for the set of emotional, contextual, and implicit information beyond literal word meaning; organizes the paper's four dimensions.
    A definitional construct with no external measurement outside the paper's own tasks; its boundaries (explicit semantics included inside 'beyond-semantic') are inconsistent.
  • Spoken Interaction System Capability Levels L1-L5
    purpose: Taxonomy for ranking spoken dialogue systems from command execution to human-level social interaction.
    Analogous to SAE autonomous-driving levels; no independent dataset or task suite is tied to the levels.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BoSS: Beyond-Semantic Speech." pith.science (2026). https://pith.science/paper/XDYNSEVQ

@misc{pith2026250717563,
  author       = {Pith},
  title        = {Pith review of: BoSS: Beyond-Semantic Speech},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XDYNSEVQ}},
  note         = {Machine review of arXiv:2507.17563}
}
read the original abstract

Human communication involves more than explicit semantics, with implicit signals and contextual cues playing a critical role in shaping meaning. However, modern speech technologies, such as Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) often fail to capture these beyond-semantic dimensions. To better characterize and benchmark the progression of speech intelligence, we introduce Spoken Interaction System Capability Levels (L1-L5), a hierarchical framework illustrated the evolution of spoken dialogue systems from basic command recognition to human-like social interaction. To support these advanced capabilities, we propose Beyond-Semantic Speech (BoSS), which refers to the set of information in speech communication that encompasses but transcends explicit semantics. It conveys emotions, contexts, and modifies or extends meanings through multidimensional features such as affective cues, contextual dynamics, and implicit semantics, thereby enhancing the understanding of communicative intentions and scenarios. We present a formalized framework for BoSS, leveraging cognitive relevance theories and machine learning models to analyze temporal and contextual speech dynamics. We evaluate BoSS-related attributes across five different dimensions, reveals that current spoken language models (SLMs) are hard to fully interpret beyond-semantic signals. These findings highlight the need for advancing BoSS research to enable richer, more context-aware human-machine communication.

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialects

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    PolySpeech-100 is a new benchmark for native-level speech comprehension across 110 linguistic variants that evaluates 22 models and reports E2E advantages on dialects, robustness gaps on low-resource languages, and de...

  2. DBHN-Net: Dual-Branch Hybrid Neural Network For Low-Complexity Monaural Speech Enhancement

    cs.SD 2026-06 unverdicted novelty 5.0 of 10

    DBHN-Net combines ANN and SNN branches with BandSplit, TF-Mamba, SFEG, ITB, and cross-attention fusion modules to deliver speech enhancement at 7.5x lower computational complexity on three public datasets.

  3. Towards High-Level Semantic Intelligence

    cs.AI 2026-07 conditional novelty 4.0 of 10

    A survey proposing that AI's next stage should be understood as High-Level Semantic Intelligence: mastering humor, sarcasm, metaphor, empathy, persuasion, and narrative across modalities.

  4. Afrispeech Semantics: Evaluating Audio Semantic Reasoning in Spoken Language Models Across Domains and Accents

    cs.CL 2026-05 unverdicted novelty 4.0 of 10

    Audio language models are benchmarked on five semantic and paralinguistic reasoning tasks to reveal limitations in handling spoken audio evidence, accent variation, and domain shifts.

Reference graph

Works this paper leans on

58 extracted references · 28 canonical work pages · cited by 4 Pith papers

  1. [1]

    SAE International, Warrendale, PA (2021)

    On-Road Automated Driving (ORAD) Committee: Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles. SAE International, Warrendale, PA (2021)

    work page 2021

  2. [2]

    The philosophical review 66(3), 377–388 (1957)

    Grice, H.P.: Meaning. The philosophical review 66(3), 377–388 (1957)

    work page 1957

  3. [3]

    Wadsworth, Belmont, CA (1971)

    Mehrabian, A.: Silent Messages. Wadsworth, Belmont, CA (1971)

    work page 1971

  4. [4]

    The body as a medium of expression 162, 174 (1975) 19

    Crystal, D.: Paralinguistics. The body as a medium of expression 162, 174 (1975) 19

    work page 1975

  5. [5]

    Trager, G.L.: Paralanguage: A first approximation. Stud. Linguist. 13, 1–12 (1958)

    work page 1958

  6. [6]

    Crystal, D.: Prosodic systems and intonation in english (1969)

    work page 1969

  7. [7]

    Syntax and semantics 3, 43–58 (1975)

    Grice, H.P.: Logic and conversation. Syntax and semantics 3, 43–58 (1975)

    work page 1975

  8. [8]

    PhD thesis, Massachusetts Institute of Technology (1980)

    Pierrehumbert, J.B.: The phonology and phonetics of english intonation. PhD thesis, Massachusetts Institute of Technology (1980)

    work page 1980

Show all 58 references
  1. [9]

    Language and speech40(2), 141–201 (1997)

    Cutler, A., Dahan, D., Van Donselaar, W.: Prosody in the comprehension of spoken language: A literature review. Language and speech40(2), 141–201 (1997)

    1997

  2. [10]

    Harvard University Press, Cambridge, MA (1986)

    Sperber, D., Wilson, D.: Relevance: Communication and Cognition. Harvard University Press, Cambridge, MA (1986)

    1986

  3. [11]

    MIT press, Cambridge, MA (2000)

    Picard, R.W.: Affective Computing. MIT press, Cambridge, MA (2000)

    2000

  4. [12]

    Computer Speech & Language 27(1), 4–39 (2013)

    Schuller, B., Steidl, S., Batliner, A., Burkhardt, F., Devillers, L., M¨ uLler, C., Narayanan, S.: Paralinguistics in speech and language—state-of-the-art and the challenge. Computer Speech & Language 27(1), 4–39 (2013)

    2013

  5. [13]

    In: International Conference on Machine Learning (2022)

    Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C., Sutskever, I.: Robust speech recognition via large-scale weak supervision. In: International Conference on Machine Learning (2022)

    2022

  6. [14]

    ArXiv abs/2305.19269 (2023)

    Huang, R., Zhang, C., Wang, Y., Yang, D., Liu, L., Ye, Z., Jiang, Z., Weng, C., Zhao, Z., Yu, D.: Make-a-voice: Unified voice synthesis with discrete representa- tion. ArXiv abs/2305.19269 (2023)

    2023 arXiv

  7. [15]

    ArXiv abs/2411.01156 (2024)

    Liao, S., Wang, Y., Li, T., Cheng, Y., Zhang, R., Zhou, R., Xing, Y.: Fish-speech: Leveraging large language models for advanced multilingual text-to-speech syn- thesis. ArXiv abs/2411.01156 (2024)

    2024 arXiv

  8. [16]

    ArXiv abs/2310.00704 (2023)

    Yang, D., Tian, J., Tan, X., Huang, R., Liu, S., Chang, X., Shi, J., Zhao, S., Bian, J., Wu, X., Zhao, Z., Meng, H.: Uniaudio: An audio foundation model toward universal audio generation. ArXiv abs/2310.00704 (2023)

    2023 arXiv

  9. [17]

    In: Annual Meeting of the Association for Computational Linguistics (2024)

    Zhan, J., Dai, J., Ye, J., Zhou, Y., Zhang, D., Liu, Z., Zhang, X., Yuan, R., Zhang, G., Li, L., Yan, H., Fu, J., Gui, T., Sun, T., Jiang, Y., Qiu, X.: Anygpt: Unified multimodal llm with discrete sequence modeling. In: Annual Meeting of the Association for Computational Lingu...

    2024

  10. [18]

    ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 11341–11345 (2023) 20

    Mehta, S., Tu, R., Beskow, J., Sz´ ekely, ´E., Henter, G.E.: Matcha-tts: A fast tts architecture with conditional flow matching. ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 11341–11345 (2023) 20

    2023

  11. [19]

    ArXiv abs/2406.02430 (2024)

    Anastassiou, P., Chen, J., Chen, J., Chen, Y., Chen, Z., Chen, Z., Cong, J., Deng, L., Ding, C., Gao, L., Gong, M., Huang, P., Huang, Q., Huang, Z., Huo, Y., Jia, D., Li, C., Li, F., Li, H., Li, J., Li, X., Li, X., Liu, L., Liu, S., Liu, S., Liu, X., Liu, Y., Liu, Z., Lu, L., ...

    2024 arXiv

  12. [20]

    ArXiv abs/2410.06885 (2024)

    Chen, Y., Niu, Z., Ma, Z., Deng, K., Wang, C., Zhao, J., Yu, K., Chen, X.: F5- tts: A fairytaler that fakes fluent and faithful speech with flow matching. ArXiv abs/2410.06885 (2024)

    2024 arXiv

  13. [21]

    ArXiv abs/2402.01912 (2024)

    Lyth, D., King, S.: Natural language guidance of high-fidelity text-to-speech with synthetic annotations. ArXiv abs/2402.01912 (2024)

    2024 arXiv

  14. [22]

    ArXiv abs/2301.11325 (2023)

    Agostinelli, A., Denk, T.I., Borsos, Z., Engel, J., Verzetti, M., Caillon, A., Huang, Q., Jansen, A., Roberts, A., Tagliasacchi, M., Sharifi, M., Zeghidour, N., Frank, C.H.: Musiclm: Generating music from text. ArXiv abs/2301.11325 (2023)

    2023 arXiv

  15. [23]

    IEEE/ACM Transactions on Audio, Speech, and Language Processing 31, 1720–1733 (2022)

    Yang, D., Yu, J., Wang, H., Wang, W., Weng, C., Zou, Y., Yu, D.: Diffsound: Discrete diffusion model for text-to-sound generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing 31, 1720–1733 (2022)

    2022

  16. [24]

    ArXiv abs/2312.15185 (2023)

    Ma, Z., Zheng, Z., Ye, J., Li, J., Gao, Z., Zhang, S., Chen, X.: emo- tion2vec: Self-supervised pre-training for speech emotion representation. ArXiv abs/2312.15185 (2023)

    2023 arXiv

  17. [25]

    ArXiv abs/2308.16692 (2023)

    Zhang, X., Zhang, D., Li, S., Zhou, Y., Qiu, X.: Speechtokenizer: Unified speech tokenizer for speech large language models. ArXiv abs/2308.16692 (2023)

    2023 arXiv

  18. [26]

    ArXiv abs/2310.13289 (2023)

    Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., Zhang, C.: Salmonn: Towards generic hearing abilities for large language models. ArXiv abs/2310.13289 (2023)

    2023 arXiv

  19. [27]

    ArXiv abs/2304.12995 (2023)

    Huang, R., Li, M., Yang, D., Shi, J., Chang, X., Ye, Z., Wu, Y., Hong, Z., Huang, J.-B., Liu, J., Ren, Y., Zhao, Z., Watanabe, S.: Audiogpt: Understanding and generating speech, music, sound, and talking head. ArXiv abs/2304.12995 (2023)

    2023 arXiv

  20. [28]

    ArXiv abs/2305.06355 (2023)

    Li, K., He, Y., Wang, Y., Li, Y., Wang, W., Luo, P., Wang, Y., Wang, L., Qiao, Y.: Videochat: Chat-centric video understanding. ArXiv abs/2305.06355 (2023)

    2023 arXiv

  21. [29]

    ArXiv abs/2306.12925 (2023)

    Rubenstein, P.K., Asawaroengchai, C., Nguyen, D.D., Bapna, A., Borsos, Z., Chaumont Quitry, F., Chen, P., Badawy, D.E., Han, W., Kharitonov, E., Muckenhirn, H., Padfield, D.R., Qin, J., Rozenberg, D., Sainath, T.N., Schalk- wyk, J., Sharifi, M., Tadmor, M.D., Ramanovich, Tagli...

    2023 arXiv

  22. [30]

    2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 1–8 (2023)

    Gong, Y., Liu, A.H., Luo, H., Karlinsky, L., Glass, J.R.: Joint audio and speech understanding. 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 1–8 (2023)

    2023

  23. [31]

    ArXiv abs/2310.04673 (2023)

    Wang, J., Du, Z., Chen, Q., Chu, Y., Gao, Z., Li, Z., Hu, K., Zhou, X., Xu, J., Ma, Z., Wang, W., Zheng, S., Zhou, C., Yan, Z., Zhang, S.: Lauragpt: Listen, attend, understand, and regenerate audio with gpt. ArXiv abs/2310.04673 (2023)

    2023 arXiv

  24. [32]

    In: Conference on Empirical Methods in Natural Language Processing (2024)

    Hu, S., Zhou, L., Liu, S., Chen, S., Hao, H., Pan, J., Liu, X., Li, J., Sivasankaran, S., Liu, L., Wei, F.: Wavllm: Towards robust and adaptive speech large language model. In: Conference on Empirical Methods in Natural Language Processing (2024)

    2024

  25. [33]

    ArXiv abs/2406.11768 (2024)

    Ghosh, S., Kumar, S., Seth, A., Evuru, C.K.R., Tyagi, U., Sakshi, S., Nieto, O., Duraiswami, R., Manocha, D.: Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities. ArXiv abs/2406.11768 (2024)

    2024 arXiv

  26. [34]

    ArXiv abs/2402.01831 (2024)

    Kong, Z., Goel, A., Badlani, R., Ping, W., Valle, R., Catanzaro, B.: Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. ArXiv abs/2402.01831 (2024)

    2024 arXiv

  27. [35]

    ArXiv abs/2503.03983 (2025)

    Ghosh, S., Kong, Z., Kumar, S., Sakshi, S., Kim, J., Ping, W., Valle, R., Manocha, D., Catanzaro, B., Loss, A.-C.C.: Audio flamingo 2: An audio-language model with long-audio understanding and expert reasoning abilities. ArXiv abs/2503.03983 (2025)

    2025 arXiv

  28. [36]

    ArXiv abs/2503.02318 (2025)

    Xie, Z., Lin, M., Liu, Z., Wu, P., Yan, S., Miao, C.: Audio-reasoner: Improving reasoning capability in large audio language models. ArXiv abs/2503.02318 (2025)

    2025

  29. [37]

    In: North American Chapter of the Association for Computational Linguistics (2024)

    Wang, B., Zou, X., Lin, G., Sun, S., Liu, Z., Zhang, W., Liu, Z., Aw, A., Chen, N.F.: Audiobench: A universal benchmark for audio large language models. In: North American Chapter of the Association for Computational Linguistics (2024)

    2024

  30. [38]

    ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 12136–12140 (2023)

    Huang, C., Lu, K.-H., Wang, S., Hsiao, C.-Y., Kuan, C.-Y., Wu, H., Arora, S., Chang, K.-W., Shi, J., Peng, Y., Sharma, R., Watanabe, S., Ramakrishnan, B., Shehata, S., Lee, H.-y.: Dynamic-superb: Towards a dynamic, collaborative, and comprehensive instruction-tuning benchmark ...

    2023

  31. [39]

    In: Annual Meeting of the Association for Computational Linguistics (2024)

    Yang, Q., Xu, J., Liu, W., Chu, Y., Jiang, Z., Zhou, X., Leng, Y., Lv, Y., Zhao, Z., Zhou, C., Zhou, J.: Air-bench: Benchmarking large audio-language 22 models via generative comprehension. In: Annual Meeting of the Association for Computational Linguistics (2024)

    2024

  32. [40]

    IEEE/ACM Transactions on Audio, Speech, and Language Processing 31, 775–788 (2023)

    Chen, W., Xing, X., Xu, X., Pang, J., Du, L.: Speechformer++: A hierarchical efficient framework for paralinguistic speech processing. IEEE/ACM Transactions on Audio, Speech, and Language Processing 31, 775–788 (2023)

    2023

  33. [41]

    ArXiv abs/2311.07919 (2023)

    Chu, Y., Xu, J., Zhou, X., Yang, Q., Zhang, S., Yan, Z., Zhou, C., Zhou, J.: Qwen- audio: Advancing universal audio understanding via unified large-scale audio- language models. ArXiv abs/2311.07919 (2023)

    2023 arXiv

  34. [42]

    ArXiv abs/2407.10759 (2024)

    Chu, Y., Xu, J., Yang, Q., Wei, H., Wei, X., Guo, Z., Leng, Y., Lv, Y., He, J., Lin, J., Zhou, C., Zhou, J.: Qwen2-audio technical report. ArXiv abs/2407.10759 (2024)

    2024 arXiv

  35. [43]

    ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 10316–10320 (2023)

    Lin, G.-T., Shivakumar, P.G., Gandhe, A., Yang, C.-H.H., Gu, Y., Ghosh, S., Stolcke, A., Lee, H.-y., Bulyko, I.: Paralinguistics-enhanced large language mod- eling of spoken dialogue. ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (...

    2023

  36. [44]

    2024 IEEE 14th International Symposium on Chinese Spoken Language Processing (ISCSLP), 586–590 (2023)

    Xue, H., Liang, Y., Mu, B., Zhang, S., Chen, M., Chen, Q., Xie, L.: E-chat: Emotion-sensitive spoken dialogue system with large language models. 2024 IEEE 14th International Symposium on Chinese Spoken Language Processing (ISCSLP), 586–590 (2023)

    2023

  37. [45]

    ArXiv abs/2409.06666 (2024)

    Fang, Q., Guo, S., Zhou, Y., Ma, Z., Zhang, S., Feng, Y.: Llama-omni: Seamless speech interaction with large language models. ArXiv abs/2409.06666 (2024)

    2024 arXiv

  38. [46]

    arXiv preprint arXiv:2412.02612 (2024)

    Zeng, A., Du, Z., Liu, M., Wang, K., Jiang, S., Zhao, L., Dong, Y., Tang, J.: Glm- 4-voice: Towards intelligent and human-like end-to-end spoken chatbot. arXiv preprint arXiv:2412.02612 (2024)

    2024 arXiv

  39. [47]

    arXiv preprint arXiv:2408.01800 (2024)

    Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Li, H., Zhao, W., He, Z., et al.: Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800 (2024)

    2024 arXiv

  40. [48]

    arXiv preprint arXiv:2501.15368 (2025)

    Li, Y., Liu, J., Zhang, T., Chen, S., Li, T., Li, Z., Liu, L., Ming, L., Dong, G., Pan, D., et al.: Baichuan-omni-1.5 technical report. arXiv preprint arXiv:2501.15368 (2025)

    2025

  41. [49]

    GitHub (2025)

    Open-Moss: SpeechGPT 2.0-preview. GitHub (2025)

    2025

  42. [50]

    arXiv preprint arXiv:2411.00774 (2024)

    Wang, X., Li, Y., Fu, C., Shen, Y., Xie, L., Li, K., Sun, X., Ma, L.: Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm. arXiv preprint arXiv:2411.00774 (2024)

    2024 arXiv

  43. [51]

    5-omni technical report

    Xu, J., Guo, Z., He, J., Hu, H., He, T., Bai, S., Chen, K., Wang, J., Fan, Y., 23 Dang, K., et al.: Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20215 (2025)

    2025 arXiv

  44. [52]

    arXiv preprint arXiv:2504.18425 (2025)

    Ding, D., Ju, Z., Leng, Y., Liu, S., Liu, T., Shang, Z., Shen, K., Song, W., Tan, X., Tang, H., et al.: Kimi-audio technical report. arXiv preprint arXiv:2504.18425 (2025)

    2025 arXiv

  45. [53]

    arXiv preprint arXiv:2203.16844 (2022)

    Yang, Z., Chen, Y., Luo, L., Yang, R., Ye, L., Cheng, G., Xu, J., Jin, Y., Zhang, Q., Zhang, P., et al.: Open source magicdata-ramc: A rich annotated mandarin conversational (ramc) speech dataset. arXiv preprint arXiv:2203.16844 (2022)

    2022 arXiv

  46. [54]

    arXiv preprint arXiv:2501.04962 (2025)

    Cui, W., Jiao, X., Meng, Z., King, I.: Voxeval: Benchmarking the knowledge understanding capabilities of end-to-end spoken language models. arXiv preprint arXiv:2501.04962 (2025)

    2025 arXiv

  47. [55]

    arXiv preprint arXiv:2504.12339 (2025)

    Song, Y., Chen, H., Lian, J., Zhang, Y., Xia, G., Li, Z., Zhao, G., Kang, J., Li, J., Li, Y., et al.: Goat-tts: Expressive and realistic speech generation via a dual-branch llm. arXiv preprint arXiv:2504.12339 (2025)

    2025 arXiv

  48. [56]

    arXiv preprint arXiv:2310.11230 (2023)

    Yao, Z., Guo, L., Yang, X., Kang, W., Kuang, F., Yang, Y., Jin, Z., Lin, L., Povey, D.: Zipformer: A faster and better encoder for automatic speech recognition. arXiv preprint arXiv:2310.11230 (2023)

    2023 arXiv

  49. [57]

    In: International Conference on Machine Learning, pp

    Skerry-Ryan, R., Battenberg, E., Xiao, Y., Wang, Y., Stanton, D., Shor, J., Weiss, R., Clark, R., Saurous, R.A.: Towards end-to-end prosody transfer for expres- sive speech synthesis with tacotron. In: International Conference on Machine Learning, pp. 4693–4702 (2018). PMLR

    2018

  50. [58]

    IEEE Signal processing magazine 18(1), 32–80 (2001) 24

    Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., Taylor, J.G.: Emotion recognition in human-computer interaction. IEEE Signal processing magazine 18(1), 32–80 (2001) 24

    2001

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.