Pith

open record

sign in

arxiv: 2501.13306 · v2 · pith:3D7VU4LA · submitted 2025-01-23 · cs.SD · cs.CL· eess.AS

OSUM: Advancing Open Speech Understanding Models with Limited Resources in Academia

Xuelong Geng , Kun Wei , Qijie Shao , Shuiyun Liu , Zhennan Lin , Zhixian Zhao , Guojian Li , Wenjie Tian
show 13 more authors
Peikun Chen Yangze Li Pengcheng Guo Mingchen Shao Shuiyuan Wang Yuang Cao Chengyou Wang Tianyi Xu Yuhang Dai Xinfa Zhu Yue Li Li Zhang Lei Xie
This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:3D7VU4LArecord.jsonopen to challenge →

classification cs.SD cs.CLeess.AS
keywords speechosumtrainingrecognitionacademicmodelsresourcestasks
0
0 comments X
read the original abstract

Large Language Models (LLMs) have made significant progress in various downstream tasks, inspiring the development of Speech Understanding Language Models (SULMs) to enable comprehensive speech-based interactions. However, most advanced SULMs are developed by the industry, leveraging large-scale datasets and computational resources that are not readily available to the academic community. Moreover, the lack of transparency in training details creates additional barriers to further innovation. In this study, we present OSUM, an Open Speech Understanding Model designed to explore the potential of training SLUMs under constrained academic resources. The OSUM model combines a Whisper encoder with a Qwen2 LLM and supports a wide range of speech tasks, including speech recognition (ASR), speech recognition with timestamps (SRWT), vocal event detection (VED), speech emotion recognition (SER), speaking style recognition (SSR), speaker gender classification (SGC), speaker age prediction (SAP), and speech-to-text chat (STTC). By employing an ASR+X training strategy, OSUM achieves efficient and stable multi-task training by simultaneously optimizing ASR alongside target tasks. Beyond delivering strong performance, OSUM emphasizes transparency by providing openly available data preparation and training methodologies, offering valuable insights and practical guidance for the academic community. By doing so, we aim to accelerate research and innovation in advanced SULM technologies.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. HumDial-EIBench: A Human-Recorded Multi-Turn Emotional Intelligence Benchmark for Audio Language Models

    eess.AS 2026-04 unverdicted novelty 7.0

    HumDial-EIBench is a new benchmark using real human dialogues to evaluate audio language models on emotional intelligence tasks including multi-turn tracking, causal reasoning, empathy generation, and acoustic-semanti...

  2. MSU-Bench: Towards Speaker-Centric Understanding in Conversational Multi-Speaker Scenarios

    eess.AS 2026-06 unverdicted novelty 6.0

    MSU-Bench is a new two-tier benchmark covering speaker grounding to dialogue reasoning in multi-speaker conversations, with Gemini-assisted annotation and human verification.

  3. Hard to Be Heard: Phoneme-Level ASR Analysis of Phonologically Complex, Low-Resource Endangered Languages

    cs.CL 2026-04 conditional novelty 6.0

    Phoneme-level analysis of ASR on Archi and Rutul shows data scarcity explains recognition errors better than phonological complexity, with language-specific adaptations improving wav2vec2 performance.

  4. Audio-Cogito: Towards Deep Audio Reasoning in Large Audio Language Models

    eess.AS 2026-04 unverdicted novelty 5.0

    Audio-Cogito is an open-source LALM using Cogito-pipe data curation and self-distillation to achieve leading open-source performance on audio reasoning benchmarks.

  5. Towards Building Speech Large Language Models for Multitask Understanding in Low-Resource Languages

    cs.SD 2025-09 unverdicted novelty 5.0

    Introduces XLSR-Thai encoder, U-Align alignment, and Thai-SUP data pipeline to enable multitask speech understanding SLLMs for Thai.

  6. Kimi-Audio Technical Report

    eess.AS 2025-04 unverdicted novelty 5.0

    Kimi-Audio is an open-source audio foundation model that achieves state-of-the-art results on speech recognition, audio understanding, question answering, and conversation after pre-training on more than 13 million ho...