Pith. sign in

REVIEW 4 major objections 5 minor 55 references

Step-Audio-AQAA: a Fully End-to-End Expressive Large Audio Language Model

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A fully end-to-end large audio-language model can answer spoken queries with direct speech and, on its own benchmark, beats state-of-the-art rivals in expressive control.

desk verdict A real 130B end-to-end speech model with a useful recipe, but the SOTA claim rests on an unreported in-house MOS study. read the letter →

arxiv 2506.08967 v2 pith:Y32ZSMFP submitted 2025-06-10 cs.SD cs.CLeess.AS

Ailin Huang , Bingxin Li , Bruce Wang , Boyong Wu , Chao Yan , Chengli Feng , Heng Wang , Hongyu Zhou
show 68 more authors
Hongyuan Wang Jingbei Li Jianjian Sun Joanna Wang Mingrui Chen Peng Liu Ruihang Miao Shilei Jiang Tian Fei Wang You Xi Chen Xuerui Yang Yechang Huang Yuxiang Zhang Zheng Ge Zheng Gong Zhewei Huang Zixin Zhang Bin Wang Bo Li Buyun Ma Changxin Miao Changyi Wan Chen Xu Dapeng Shi Dingyuan Hu Enle Liu Guanzhe Huang Gulin Yan Hanpeng Hu Haonan Jia Jiahao Gong Jiaoren Wu Jie Wu Jie Yang Junzhe Lin Kaixiang Li Lei Xia Longlong Gu Ming Li Nie Hao Ranchen Ming Shaoliang Pang Siqi Liu Song Yuan Tiancheng Cao Wen Li Wenqing He Xu Zhao Xuelin Zhang Yanbo Yu Yinmin Zhong Yu Zhou Yuanwei Liang Yuanwei Lu Yuxiang Yang Zidong Yang Zili Zhang Binxing Jiao Heung-Yeung Shum Jiansheng Chen Jing Li Xiangyu Zhang Xinhao Zhang Yibo Zhu Daxin Jiang Shuchang Zhou Chen Hu
This is my paper
classification cs.SDcs.CLeess.AS
keywords largeaudiolanguagemodelqueryanswerend-to-endspeechinteractiondual-codebooktokenizerdirectpreferenceoptimizationmergingemotioncontrolinterleavedtext-audiotokens
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to establish that a large audio-language model trained end to end can take a spoken question and reply directly with synthesized speech, skipping the usual speech-to-text and text-to-speech cascade. It claims that Step-Audio-AQAA, built on a 130-billion-parameter backbone and a dual-codebook audio tokenizer, achieves this with fine-grained voice control, letting the model alter emotion and speaking rate sentence by sentence. The paper reports that the model outperforms the current state-of-the-art end-to-end LALMs, Kimi-Audio and Qwen-Omni, on the StepEval-Audio-360 benchmark, particularly in speech emotion control, creativity, role-playing, and language ability. A sympathetic reader would care because this points toward voice assistants that speak with natural prosody and emotional range rather than reading a text reply.

What carries the argument

The load-bearing mechanism is the dual-codebook audio tokenizer combined with interleaved token output. The linguistic tokenizer (Paraformer encoder, 16.7 Hz, codebook size 1,024) captures phonemic structure; the semantic tokenizer (CosyVoice-style, 25 Hz, codebook size 4,096) captures acoustic detail. The two token streams are interleaved 2:3 for input and joined with text tokens at a 10:15 ratio for output. The 130-billion-parameter decoder-only backbone (Step-Omni) had its vocabulary extended with 5,120 audio tokens and is post-trained in two SFT stages, then aligned by masked DPO in which audio tokens are masked out of the preference loss, and finally merged as $(5 \cdot W_{\text{SFT-1st}} + 5 \cdot W_{\text{SFT-2nd}} + 1 \cdot W_{\text{DPO}})/11$. A flow-matching vocoder conditioned only on audio tokens renders the final waveform.

What would settle it

Run a blinded, preregistered human evaluation of Step-Audio-AQAA, Kimi-Audio, and Qwen-Omni on the nine StepEval-Audio-360 dimensions with a fresh panel of raters and report per-dimension means, confidence intervals, and inter-annotator agreement; if the emotion-control advantage shrinks to noise or reverses, the central claim fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a fully end-to-end Audio Query-Audio Answer model is practical: the same neural network that understands an audio question can directly emit audio tokens that a vocoder turns into a natural spoken answer. The design choices that make this work are a dual-codebook tokenizer (linguistic plus semantic), interleaved text/audio token output at a 10:15 ratio, a masked DPO objective that avoids applying the preference loss to audio tokens, and a 5:5:1 weight merge of the first-stage SFT, second-stage SFT, and DPO models. The paper further claims that this recipe yields the best scores among the compared LALMs on most of the nine evaluated dimensions, with speech emotion control the largest edge, while conceding weaknesses in singing and voice instruction following.

Load-bearing premise

The load-bearing assumption is that the in-house StepEval-Audio-360 evaluation is an unbiased, precise measure of LALM quality; the paper reports no rater count, inter-annotator agreement, confidence intervals, or significance tests, and it gives no sizes for the compared baselines.

Editorial extensions

If this is right

  • Fully end-to-end audio query-audio answer removes the ASR/TTS cascade and its error accumulation, simplifying spoken-interaction systems.
  • Fine-grained voice control becomes possible at the sentence level, allowing emotion and speech rate to be modified without a separate TTS stage.
  • Interleaving text and audio tokens at a 10:15 ratio improves semantic coherence over audio-only output, as shown by the ablation results.
  • Masking audio tokens out of the DPO loss preserves voice-token generation while still aligning content with human preferences.
  • Marker-preserving concatenation of single-label audio segments lets the model switch speech states within one turn, enabling multi-label speech.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the masked-DPO finding generalizes, preference alignment in any multimodal generator that mixes discrete tokens from different codebooks may need to protect perceptual codebooks from the reward gradient, a design rule the paper only states for audio.
  • The in-house nature of StepEval-Audio-360 means the headline comparison should be treated as a claim to verify, not an established ranking; independent benchmarks and blind listening tests would settle it.
  • The 130B scale of Step-Audio-AQAA is not controlled against baseline sizes, so part of the reported gap could be scale rather than the proposed recipe; a smaller variant of the same pipeline would test this.
  • Sentence-level voice control suggests the same architecture could be steered by explicit emotion and speech-rate annotations to become a controllable spoken assistant, an extension the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents Step-Audio-AQAA, a fully end-to-end large audio-language model (LALM) that accepts raw audio input and directly generates interleaved text and audio tokens, which are then rendered as speech by a vocoder. The model builds on a 130-billion-parameter backbone (Step-Omni), a dual-codebook tokenizer that combines linguistic and semantic tokens, and a post-training recipe consisting of two-stage supervised fine-tuning, a masked Direct Preference Optimization objective, and weight merging with a 5:5:1 ratio. The authors claim that Step-Audio-AQAA outperforms existing LALMs, specifically Kimi-Audio and Qwen-Omni, on the StepEval-Audio-360 benchmark, particularly in speech emotion control, creativity, role-playing, and voice understanding. The paper also reports ablations on text-audio interleaving ratios and interleaving methods. The model and benchmark are released on Hugging Face.

Significance. If the central comparative claim were fully supported, the paper would make a useful contribution: it describes a complete end-to-end AQAA architecture, releases a 130B-scale model and a benchmark, and proposes a concrete post-training recipe (dual-codebook tokenization, masked DPO, model merging) that could inform future work on expressive spoken dialogue systems. The open release of the model and the StepEval-Audio-360 dataset is a strength that helps reproducibility. However, the significance is currently conditional on the credibility of the evaluation. The evaluation evidence is almost entirely self-generated: the backbone, the benchmark, the TTS data synthesis, and the human evaluation all originate from the same team, and the reported quantitative results lack the statistical detail needed to verify the headline claim. The paper's main contribution is a performance claim, so this evidential gap is load-bearing rather than cosmetic.

major comments (4)
  1. [§4.2, §5.1, Figure 3] The central claim that Step-Audio-AQAA outperforms Kimi-Audio and Qwen-Omni rests on a human MOS evaluation that is not statistically reported. Section 4.2 says expert evaluators used a 1-5 MOS scale, but the paper gives no rater count, no number of evaluated sessions, no confidence intervals, no significance tests, and no inter-annotator agreement measure. Figure 3 is a radar chart without numeric scores, making the margins of difference unreadable. Because the abstract and Section 5.1 directly assert superiority over state-of-the-art LALMs, this missing statistical grounding is a load-bearing flaw. The authors should report the exact MOS values with uncertainty estimates and perform appropriate pairwise significance tests, or temper the comparative claims to what the evidence supports.
  2. [§2.2, §4.2] The comparison is potentially confounded by model scale. Section 2.2 states that Step-Audio-AQAA uses a 130-billion-parameter backbone, but Sections 4.2 and 5.1 provide no parameter counts or compute descriptions for the baselines Kimi-Audio and Qwen-Omni. If those baselines are substantially smaller, the observed advantages may reflect scale rather than the proposed dual-codebook and post-training recipe. The authors should report or cite parameter counts for the baselines and discuss whether the comparison is scale-controlled, or explicitly acknowledge that the comparison is a system-level one in which scale is not held fixed.
  3. [§3.2, §4.1, §5.1] The evaluation chain is substantially self-referential. The AQTAA training data is generated by Step-Audio-TTS-3B (Section 3.2), the benchmark StepEval-Audio-360 is released by the same team (Section 4.1), and the human evaluation in Section 5.1 is conducted by the authors. This does not automatically invalidate the results, but it creates a clear risk of experimenter bias and makes the headline comparative claim difficult to assess from the outside. A concrete mitigation would be an independent or blinded human evaluation, an external benchmark, or at a minimum a very detailed protocol description that includes rater recruitment, instructions, and item sampling. Without such mitigation, the claim that Step-Audio-AQAA 'excels especially in speech control' remains unverified.
  4. [§5.2, Tables 1 and 2] The ablation results that motivate key design choices (interleaving ratios and interleaving methods) use GPT-4o as an automated judge, but the paper does not specify the prompt, the scoring scale, or the validation of this judge against human ratings. Several reported numbers are surprisingly low (e.g., Relevance = 0.0526 and Factuality = 0.0316 for audio_only in Table 1), and the table does not explain whether these are averages, success rates, or something else. Since these ablations are used to support the 10:15 ratio and marker-preserving concatenation, they are load-bearing for the proposed recipe. The authors should provide the evaluation protocol, define the metrics, and ideally show correlation with human judgments.
minor comments (5)
  1. [§2.2, Figure 1, Figure 2, §5.2] The output interleaving ratio is reported inconsistently: the text says '10:15 ratio' in Section 2.2 and Section 5.2, Figure 1's caption says '10:6:9', and Figure 2 says '3:2:3'. The authors should reconcile these numbers or explain what each ratio refers to.
  2. [§5.2, Table 1] The list of mixing proportions in the text is garbled: item (2) and item (3) both read 'ratio_6_50', and the table lists 'ratio_3_5' instead of a correctly labeled third condition. This should be corrected.
  3. [§2.1] The claim that dual-codebook training reduces next-token prediction perplexity is stated as an observation without any supporting numbers, comparison, or reference. Please provide the empirical evidence or mark this as a hypothesis.
  4. [§5.1] The paper acknowledges that Step-Audio-AQAA underperforms in Singing and Voice Instruction Following, but the abstract and conclusion state the model 'outperforms' state-of-the-art LALMs without these caveats. The claims should be qualified accordingly.
  5. [Throughout] There are several typos and stylistic errors, including 'Superivsed fine-turning' (Figure 2), 'emothion' (Section 6), 'V ALL-E' (Section 1), 'state-of-art' (Abstract), and 'the linguistic tokenizater' (Section 2.1). A careful proofreading pass is needed.

Circularity Check

1 steps flagged · score 4.0 of 10

Headline superiority claim rests on a same-team benchmark, but the technical derivation is self-contained.

  1. self citation load bearing [Abstract; Section 4.1; Section 4.2; Section 5.1, Figure 3]
    "Evaluations on the StepEval-Audio-360 benchmark demonstrate that Step-Audio-AQAA excels especially in speech control, outperforming the state-of-art LALMs in key areas. ... StepEval-Audio-360 [37] is a comprehensive benchmark dataset designed to evaluate the capabilities of LALMs in human-AI audio interaction. ... Comprehensive human evaluations were conducted to compare Step-Audio-AQAA with Kimi-Audio and Qwen-Omni across the nine critical dimensions of StepEval-Audio-360 outlined above."

    The paper's only reported evidence for its central claim (Section 5.1/Figure 3) comes from human MOS ratings on StepEval-Audio-360, which is cited as [37] and released by the same StepFun/Step-Audio organization that built Step-Audio-AQAA. No independent benchmark or external evaluation protocol is used; no rater count, inter-annotator agreement, confidence intervals, or significance tests are reported. The comparison to Kimi-Audio and Qwen-Omni provides some independent content, but the yardstick itself is a load-bearing self-citation: the 'state-of-the-art' claim is validated on a test set authored by the model's own team, so the evidence chain reduces to self-assessment.

full rationale

The architecture and training recipe (dual-codebook tokenizer, interleaved text/audio output, masked DPO, weight merge) are described concretely and do not reduce by construction to their inputs. The ablations use GPT-4o as a judge and report numeric scores, and the comparison includes external baselines. The main circularity concern is evaluative rather than derivational: the only benchmark supporting the headline 'outperforms SOTA' claim is StepEval-Audio-360 [37], created and released by the same team, with no rater counts, confidence intervals, significance tests, or external validation. This makes the headline evidence self-referential in source, though not mathematically forced. Because the baselines are real external models and the benchmark is publicly released, the central claim retains some independent content; hence score 4 rather than higher.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim depends on many hand-set hyperparameters (interleaving ratios, merge weights) and on unverified self-created infrastructure: the Step-Omni backbone, the StepEval-Audio-360 benchmark, the Step-Audio-TTS data generator, and the internal human evaluation. No external calibration or cross-check is provided.

free parameters (3)
  • Input token interleaving ratio (linguistic:semantic) = 2:3
    Chosen to match the 16.7 Hz vs 25 Hz token rates of the two tokenizers; not fitted to the evaluation data, but a design decision the model's behavior depends on.
  • Output text-audio interleaving ratio = 10:15 (per Section 2; 3:2:3 in Figure 2)
    Selected after the ablation in Table 1, where ratio_10_15 gave the highest GPT-4o-judged scores. The paper does not report a held-out confirmation, and the value appears inconsistently in the text.
  • Weight-merge coefficients = 5, 5, 1
    Chosen by hand in Section 3.4 (Equation 3) with no ablation or sensitivity analysis; the final model's behavior depends on these weights.
assumptions (5)
  • domain assumption Step-Omni's 130B pretraining provides a sufficiently strong multimodal backbone
    The paper builds entirely on the authors' prior Step-Omni [19] and does not independently verify its capability distribution.
  • domain assumption StepEval-Audio-360 is a valid and representative measure of LALM skill
    The benchmark is a self-released dataset (Hugging Face [37]); its construction, prompt set, and scoring rubric are not described in this paper.
  • domain assumption Step-Audio-TTS-3B synthesized answers are high enough quality for supervised tuning
    Section 3.2 states AQTAA data was generated by the team's own TTS model; no quality checks or manual verification are reported.
  • domain assumption The two baselines (Kimi-Audio, Qwen-Omni) were evaluated fairly at comparable scale and configuration
    Section 4.2 lists the baselines but gives no parameter counts, inference settings, or prompt details, so the comparison may be confounded by scale.
  • ad hoc to paper Dual-codebook tokens lower next-token perplexity
    Section 2.1 asserts that dual-codebook training reduced perplexity relative to the single codebook in [19], but no numbers or experiments are shown in this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Step-Audio-AQAA: a Fully End-to-End Expressive Large Audio Language Model." pith.science (2026). https://pith.science/paper/Y32ZSMFP

@misc{pith2026250608967,
  author       = {Pith},
  title        = {Pith review of: Step-Audio-AQAA: a Fully End-to-End Expressive Large Audio Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y32ZSMFP}},
  note         = {Machine review of arXiv:2506.08967}
}
read the original abstract

Large Audio-Language Models (LALMs) have significantly advanced intelligent human-computer interaction, yet their reliance on text-based outputs limits their ability to generate natural speech responses directly, hindering seamless audio interactions. To address this, we introduce Step-Audio-AQAA, a fully end-to-end LALM designed for Audio Query-Audio Answer (AQAA) tasks. The model integrates a dual-codebook audio tokenizer for linguistic and semantic feature extraction, a 130-billion-parameter backbone LLM and a neural vocoder for high-fidelity speech synthesis. Our post-training approach employs interleaved token-output of text and audio to enhance semantic coherence and combines Direct Preference Optimization (DPO) with model merge to improve performance. Evaluations on the StepEval-Audio-360 benchmark demonstrate that Step-Audio-AQAA excels especially in speech control, outperforming the state-of-art LALMs in key areas. This work contributes a promising solution for end-to-end LALMs and highlights the critical role of token-based vocoder in enhancing overall performance for AQAA tasks.

Figures

Figures reproduced from arXiv: 2506.08967 by the authors.

Figure 1
Figure 1. Model architecture of Step-Audio-AQAA. The backbone of Step-Audio-AQAA is a [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Illustration of (A) tokenized AQTA data pairs and tokenized AQTAA data pairs utilized in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Human evaluation of the end-to-end speech interactions on StepEval-Audio-360 benchmark. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

55 extracted references · 18 canonical work pages

  1. [1]

    Claude 3.5 sonnet

    Anthropic. Claude 3.5 sonnet. 2024. URL https://www.anthropic.com/news/ claude-3-5-sonnet

    work page 2024

  2. [2]

    Ml4co-kida: Knowledge inheritance in dataset aggregation.arXiv preprint arXiv:2201.10328, 2022

    Zixuan Cao, Yang Xu, Zhewei Huang, and Shuchang Zhou. Ml4co-kida: Knowledge inheritance in dataset aggregation.arXiv preprint arXiv:2201.10328, 2022

    arXiv 2022

  3. [3]

    Vall-e 2: Neural codec language models are human parity zero-shot text to speech synthesizers.arXiv preprint arXiv:2406.05370, 2024

    Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, and Furu Wei. Vall-e 2: Neural codec language models are human parity zero-shot text to speech synthesizers.arXiv preprint arXiv:2406.05370, 2024

    arXiv 2024

  4. [4]

    Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms.arXiv preprint arXiv:2406.07476, 2024

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms.arXiv preprint arXiv:2406.07476, 2024

    arXiv 2024

  5. [5]

    Qwen2-audio technical report.arXiv preprint arXiv:2407.10759, 2024

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audio technical report.arXiv preprint arXiv:2407.10759, 2024

    arXiv 2024

  6. [6]

    Simple and controllable music generation

    Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation. InThirty-seventh Conference on Neural Information Processing Systems, 2023

    work page 2023

  7. [7]

    Recent advances in speech language models: A survey.arXiv preprint arXiv:2410.03751, 2024

    Wenqian Cui, Dianzhi Yu, Xiaoqi Jiao, Ziqiao Meng, Guangyan Zhang, Qichao Wang, Yiwen Guo, and Irwin King. Recent advances in speech language models: A survey.arXiv preprint arXiv:2410.03751, 2024

    arXiv 2024

  8. [8]

    Pengi: An audio language model for audio tasks.Advances in Neural Information Processing Systems, 36:18090–18108, 2023

    Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. Pengi: An audio language model for audio tasks.Advances in Neural Information Processing Systems, 36:18090–18108, 2023

    work page 2023

Show all 55 references
  1. [9]

    Kimi-audio technical report.arXiv preprint arXiv:2504.18425, 2025

    Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, et al. Kimi-audio technical report.arXiv preprint arXiv:2504.18425, 2025

    2025 arXiv

  2. [10]

    Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens.arXiv preprint arXiv:2407.05407, 2024

    Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens.arXiv preprint arXiv:2407.05407, 2024

    2024 arXiv

  3. [11]

    Emo-dpo: Controllable emo- tional speech synthesis through direct preference optimization

    Xiaoxue Gao, Chen Zhang, Yiming Chen, Huayun Zhang, and Nancy F Chen. Emo-dpo: Controllable emo- tional speech synthesis through direct preference optimization. InICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2025

    2025

  4. [12]

    Paraformer: Fast and accurate parallel transformer for non-autoregressive end-to-end speech recognition.arXiv preprint arXiv:2206.08317, 2022

    Zhifu Gao, Shiliang Zhang, Ian McLoughlin, and Zhijie Yan. Paraformer: Fast and accurate parallel transformer for non-autoregressive end-to-end speech recognition.arXiv preprint arXiv:2206.08317, 2022

    2022 arXiv

  5. [13]

    Joint audio and speech understanding

    Yuan Gong, Alexander H Liu, Hongyin Luo, Leonid Karlinsky, and James Glass. Joint audio and speech understanding. In2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1–8. IEEE, 2023

    2023

  6. [14]

    Gemini 2.0 pro

    Google. Gemini 2.0 pro. 2025. URLhttps://deepmind.google/technologies/gemini/pro/

    2025

  7. [15]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  8. [16]

    Vall-e r: Robust and efficient zero-shot text-to-speech synthesis via monotonic alignment.arXiv preprint arXiv:2406.07855, 2024

    Bing Han, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Yanming Qian, Yanqing Liu, Sheng Zhao, Jinyu Li, and Furu Wei. Vall-e r: Robust and efficient zero-shot text-to-speech synthesis via monotonic alignment.arXiv preprint arXiv:2406.07855, 2024

    2024 arXiv

  9. [17]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 9

    2016

  10. [18]

    Drivingworld: Constructingworld model for autonomous driving via video gpt.arXiv preprint arXiv:2412.19505, 2024

    Xiaotao Hu, Wei Yin, Mingkai Jia, Junyuan Deng, Xiaoyang Guo, Qian Zhang, Xiaoxiao Long, and Ping Tan. Drivingworld: Constructingworld model for autonomous driving via video gpt.arXiv preprint arXiv:2412.19505, 2024

    2024 arXiv

  11. [19]

    Step-audio: Unified understanding and generation in intelligent speech interaction

    Ailin Huang, Boyong Wu, Bruce Wang, Chao Yan, Chen Hu, Chengli Feng, Fei Tian, Feiyu Shen, Jingbei Li, Mingrui Chen, et al. Step-audio: Unified understanding and generation in intelligent speech interaction. arXiv preprint arXiv:2502.11946, 2025

    2025 arXiv

  12. [20]

    Audiogpt: Understanding and generating speech, music, sound, and talking head

    Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, et al. Audiogpt: Understanding and generating speech, music, sound, and talking head. InProceedings of the AAAI Conference on Artificial Intelli...

    2024

  13. [21]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  14. [22]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Hel- yar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  15. [23]

    Wavchat: A survey of spoken dialogue models.arXiv preprint arXiv:2411.13577, 2024

    Shengpeng Ji, Yifu Chen, Minghui Fang, Jialong Zuo, Jingyu Lu, Hanting Wang, Ziyue Jiang, Long Zhou, Shujie Liu, Xize Cheng, et al. Wavchat: A survey of spoken dialogue models.arXiv preprint arXiv:2411.13577, 2024

    2024 arXiv

  16. [24]

    An llm compiler for parallel function calling

    Sehoon Kim, Suhong Moon, Ryan Tabrizi, Nicholas Lee, Michael W Mahoney, Kurt Keutzer, and Amir Gholami. An llm compiler for parallel function calling. InForty-first International Conference on Machine Learning, 2024

    2024

  17. [25]

    Understanding the effects of rlhf on llm generalisation and diversity.arXiv preprint arXiv:2310.06452, 2023

    Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefen- stette, and Roberta Raileanu. Understanding the effects of rlhf on llm generalisation and diversity.arXiv preprint arXiv:2310.06452, 2023

    arXiv

  18. [26]

    Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities.arXiv preprint arXiv:2402.01831, 2024

    Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities.arXiv preprint arXiv:2402.01831, 2024

    2024 arXiv

  19. [27]

    Flexkbqa: A flexible llm-powered framework for few-shot knowledge base question answering

    Zhenyu Li, Sunqi Fan, Yu Gu, Xiuxing Li, Zhichao Duan, Bowen Dong, Ning Liu, and Jianyong Wang. Flexkbqa: A flexible llm-powered framework for few-shot knowledge base question answering. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 18608–1...

    2024

  20. [28]

    Merging models with fisher-weighted averaging.Advances in Neural Information Processing Systems, 35:17703–17716, 2022

    Michael S Matena and Colin A Raffel. Merging models with fisher-weighted averaging.Advances in Neural Information Processing Systems, 35:17703–17716, 2022

    2022

  21. [29]

    Using an llm to help with code understanding

    Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. Using an llm to help with code understanding. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–13, 2024

    2024

  22. [30]

    The role of paralinguistic cues in social life.ANALYSIS OF MODERN SCIENCE AND INNOVATION, 1(2):215–218, 2024

    Umarova Lobar Nematullayevna. The role of paralinguistic cues in social life.ANALYSIS OF MODERN SCIENCE AND INNOVATION, 1(2):215–218, 2024

    2024

  23. [31]

    A survey on speech large language models.arXiv preprint arXiv:2410.18908, 2024

    Jing Peng, Yucheng Wang, Yu Xi, Xu Li, Xizhuo Zhang, and Kai Yu. A survey on speech large language models.arXiv preprint arXiv:2410.18908, 2024

    2024

  24. [32]

    Direct preference optimization: Your language model is secretly a reward model.Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model.Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    2023

  25. [33]

    Audiopalm: A large language model that can speak and listen.arXiv preprint arXiv:2306.12925, 2023

    Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. Audiopalm: A large language model that can speak and listen.arXiv preprint arXiv:2306.12925, 2023

    2023 arXiv

  26. [34]

    How to debug code with github copilot

    Jeimy Ruiz. How to debug code with github copilot. 2025. URL https://www.anthropic.com/news/ claude-3-5-sonnet

    2025

  27. [35]

    Paralinguistics in speech and language—state-of-the-art and the challenge

    Björn Schuller, Stefan Steidl, Anton Batliner, Felix Burkhardt, Laurence Devillers, Christian MüLler, and Shrikanth Narayanan. Paralinguistics in speech and language—state-of-the-art and the challenge. Computer Speech & Language, 27(1):4–39, 2013

    2013

  28. [36]

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face, 2023

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face, 2023. URL https://arxiv.org/abs/ 2303.17580

    2023 arXiv

  29. [37]

    Stepeval-audio-360

    StepFun. Stepeval-audio-360. 2025. URL https://huggingface.co/datasets/stepfun-ai/ StepEval-Audio-360. 10

    2025

  30. [38]

    Salmonn: Towards generic hearing abilities for large language models.arXiv preprint arXiv:2310.13289, 2023

    Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models.arXiv preprint arXiv:2310.13289, 2023

    2023 arXiv

  31. [39]

    Preference alignment improves language model-based tts

    Jinchuan Tian, Chunlei Zhang, Jiatong Shi, Hao Zhang, Jianwei Yu, Shinji Watanabe, and Dong Yu. Preference alignment improves language model-based tts. InICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2025

    2025

  32. [40]

    Lami: Large language models for multi-modal human-robot interaction

    Chao Wang, Stephan Hasler, Daniel Tanneberg, Felix Ocker, Frank Joublin, Antonello Ceravola, Joerg Deigmoeller, and Michael Gienger. Lami: Large language models for multi-modal human-robot interaction. InExtended Abstracts of the CHI Conference on Human Factors in Computing Sy...

    2024

  33. [41]

    Neural codec language models are zero-shot text to speech synthesizers

    Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111, 2023

    2023 arXiv

  34. [42]

    A comprehensive survey of llm alignment techniques: Rlhf, rlaif, ppo, dpo and more.arXiv preprint arXiv:2407.16216, 2024

    Zhichao Wang, Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mehrotra, Xiang-Bo Mao, Sitaram Asur, et al. A comprehensive survey of llm alignment techniques: Rlhf, rlaif, ppo, dpo and more.arXiv preprint arXiv:2407.16216, 2024

    2024 arXiv

  35. [43]

    Model soups: aver- aging weights of multiple fine-tuned models improves accuracy without increasing inference time

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: aver- aging weights of multiple fine-tuned models improves accuracy without increasing in...

    2022

  36. [44]

    Codec-superb: An in-depth analysis of sound codec models.arXiv preprint arXiv:2402.13071, 2024

    Haibin Wu, Ho-Lam Chung, Yi-Cheng Lin, Yuan-Kuei Wu, Xuanjun Chen, Yu-Chi Pai, Hsiu-Hsuan Wang, Kai-Wei Chang, Alexander H Liu, and Hung-yi Lee. Codec-superb: An in-depth analysis of sound codec models.arXiv preprint arXiv:2402.13071, 2024

    2024 arXiv

  37. [45]

    When search engine services meet large language models: visions and challenges.IEEE Transactions on Services Computing, 2024

    Haoyi Xiong, Jiang Bian, Yuchen Li, Xuhong Li, Mengnan Du, Shuaiqiang Wang, Dawei Yin, and Sumi Helal. When search engine services meet large language models: visions and challenges.IEEE Transactions on Services Computing, 2024

    2024

  38. [46]

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report.arXiv preprint arXiv:2503.20215, 2025

    2025 arXiv

  39. [47]

    Uniaudio: Towards universal audio generation with large language models

    Dongchao Yang, Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Haohan Guo, Xuankai Chang, Jiatong Shi, Jiang Bian, Zhou Zhao, et al. Uniaudio: Towards universal audio generation with large language models. InForty-first International Conference on Machine Learning, 2024

    2024

  40. [48]

    Soundstream: An end-to-end neural audio codec.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495–507, 2021

    Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495–507, 2021

    2021

  41. [49]

    Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot.arXiv preprint arXiv:2412.02612, 2024

    Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot.arXiv preprint arXiv:2412.02612, 2024

    2024 arXiv

  42. [50]

    Root mean square layer normalization.Advances in Neural Information Processing Systems, 32, 2019

    Biao Zhang and Rico Sennrich. Root mean square layer normalization.Advances in Neural Information Processing Systems, 32, 2019

    2019

  43. [51]

    Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities.arXiv preprint arXiv:2305.11000, 2023

    Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities.arXiv preprint arXiv:2305.11000, 2023

    2023 arXiv

  44. [52]

    Speechalign: Aligning speech generation to human preferences.arXiv preprint arXiv:2404.05600, 2024

    Dong Zhang, Zhaowei Li, Shimin Li, Xin Zhang, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechalign: Aligning speech generation to human preferences.arXiv preprint arXiv:2404.05600, 2024

    2024 arXiv

  45. [53]

    Vistorybench: Comprehensive benchmark suite for story visualiza- tion.arXiv preprint arXiv:2505.24862, 2025

    Cailin Zhuang, Ailin Huang, Wei Cheng, Jingwei Wu, Yaoqi Hu, Jiaqi Liao, Zhewei Huang, Hongyuan Wang, Xinyao Liao, Weiwei Cai, et al. Vistorybench: Comprehensive benchmark suite for story visualiza- tion.arXiv preprint arXiv:2505.24862, 2025

    2025

  46. [54]

    Toolqa: A dataset for llm question answering with external tools.Advances in Neural Information Processing Systems, 36:50117–50143, 2023

    Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. Toolqa: A dataset for llm question answering with external tools.Advances in Neural Information Processing Systems, 36:50117–50143, 2023

    2023

  47. [55]

    Pre-trained language model based ranking in baidu search

    Lixin Zou, Shengqiang Zhang, Hengyi Cai, Dehong Ma, Suqi Cheng, Shuaiqiang Wang, Daiting Shi, Zhicong Cheng, and Dawei Yin. Pre-trained language model based ranking in baidu search. InProceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 401...

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.