pith. sign in

arxiv: 2411.05361 · v2 · pith:MNANA4JGnew · submitted 2024-11-08 · 💻 cs.CL · eess.AS

Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks

Chien-yu Huang , Wei-Chih Chen , Shu-wen Yang , Andy T. Liu , Chen-An Li , Yu-Xiang Lin , Wei-Cheng Tseng , Anuj Diwan
show 72 more authors
Yi-Jen Shih Jiatong Shi William Chen Chih-Kai Yang Wenze Ren Xuanjun Chen Chi-Yuan Hsiao Puyuan Peng Shih-Heng Wang Chun-Yi Kuan Ke-Han Lu Kai-Wei Chang Fabian Ritter-Gutierrez Kuan-Po Huang Siddhant Arora You-Kuan Lin Ming To Chuang Eunjung Yeo Kalvin Chang Chung-Ming Chien Kwanghee Choi Jun-You Wang Cheng-Hsiu Hsieh Yi-Cheng Lin Chee-En Yu I-Hsiang Chiu Heitor R. Guimar\~aes Jionghao Han Tzu-Quan Lin Tzu-Yuan Lin Homu Chang Ting-Wu Chang Chun Wei Chen Shou-Jen Chen Yu-Hua Chen Hsi-Chun Cheng Kunal Dhawan Jia-Lin Fang Shi-Xin Fang Kuan-Yu Fang Chiang Chi An Fu Hsien-Fu Hsiao Ching Yu Hsu Shao-Syuan Huang Lee Chen Wei Hsi-Che Lin Hsuan-Hao Lin Hsuan-Ting Lin Jian-Ren Lin Ting-Chun Liu Li-Chun Lu Tsung-Min Pai Ankita Pasad Shih-Yun Shan Kuan Suwon Shon Yuxun Tang Yun-Shao Tsai Jui-Chiang Wei Tzu-Chieh Wei Chengxi Wu Dien-Ruei Wu Chao-Han Huck Yang Chieh-Chi Yang Jia Qi Yip Shao-Xiang Yuan Vahid Noroozi Zhehuai Chen Haibin Wu Karen Livescu David Harwath Shinji Watanabe Hung-yi Lee
This is my paper
classification 💻 cs.CL eess.AS
keywords dynamic-superbevaluationtasksbenchmarkmodelsgenerationlanguagephase-2
0
0 comments X
read the original abstract

Multimodal foundation models, such as Gemini and ChatGPT, have revolutionized human-machine interactions by seamlessly integrating various forms of data. Developing a universal spoken language model that comprehends a wide range of natural language instructions is critical for bridging communication gaps and facilitating more intuitive interactions. However, the absence of a comprehensive evaluation benchmark poses a significant challenge. We present Dynamic-SUPERB Phase-2, an open and evolving benchmark for the comprehensive evaluation of instruction-based universal speech models. Building upon the first generation, this second version incorporates 125 new tasks contributed collaboratively by the global research community, expanding the benchmark to a total of 180 tasks, making it the largest benchmark for speech and audio evaluation. While the first generation of Dynamic-SUPERB was limited to classification tasks, Dynamic-SUPERB Phase-2 broadens its evaluation capabilities by introducing a wide array of novel and diverse tasks, including regression and sequence generation, across speech, music, and environmental audio. Evaluation results show that no model performed well universally. SALMONN-13B excelled in English ASR and Qwen2-Audio-7B-Instruct showed high accuracy in emotion recognition, but current models still require further innovations to handle a broader range of tasks. We open-source all task data and the evaluation pipeline at https://github.com/dynamic-superb/dynamic-superb.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Putting HUMANS first: Efficient LAM Evaluation with Human Preference Alignment

    cs.CL 2026-04 unverdicted novelty 7.0

    Curated 50-example subsets of LAM benchmarks, via regression, predict human preferences at 0.98 correlation, outperforming the full benchmark and yielding the open-sourced HUMANS proxy.