Pith. sign in

REVIEW 3 major objections 2 minor 2 cited by

LLaSO: A Foundational Framework for Reproducible Research in Large Language and Speech Model

T0 review · 3 major / 2 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLaSO claims to be the first fully open, end-to-end framework for speech-language modeling, releasing data, a benchmark, and a 3.8B-parameter baseline that reaches a normalized score of 0.72.

desk verdict The file we got is a physics paper, not LLaSO, so the actual claims are unverifiable as submitted; the abstract alone describes a potentially valuable open LSLM stack that deserves a real look once the correct manuscript is provided. read the letter →

arxiv 2508.15418 v1 pith:QFMR4T4N submitted 2025-08-21 cs.CL cs.AIcs.LGcs.MMcs.SD

Yirong Sun , Yizhong Geng , Peidong Wei , Yanjun Chen , Jinghan Yang , Rongfei Chen , Wei Zhang , Xiaoyu Shen This is my paper
classification cs.CLcs.AIcs.LGcs.MMcs.SD
keywords largespeech-languagemodelsopen-sourceframeworkspeech-textalignmentinstructiontuningevaluationbenchmarkreproducibilitypublicdataLLaSO
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to solve the reproducibility crisis in large speech-language models, where weights are often released without corresponding training data or configurations. It introduces LLaSO, an open end-to-end framework built from three resources: a 12M-instance speech-text alignment corpus, a 13.5M-instance multi-task instruction-tuning dataset, and a reproducible evaluation benchmark. To validate the framework, the authors train and release LLaSO-Base, a 3.8B-parameter model trained exclusively on their public data, reporting a normalized score of 0.72 that they say surpasses comparable models. The paper also reports that broader training coverage improves performance while notable generalization gaps remain on unseen tasks, especially pure-audio scenarios. If the claims hold, the field gains a complete public stack for building, comparing, and reproducing speech-language models.

What carries the argument

The load-bearing mechanism is the complete release triad: LLaSO-Align (12M speech-text alignment instances), LLaSO-Instruct (13.5M multi-task instruction instances), and LLaSO-Eval (a standardized, reproducible benchmark), together with the LLaSO-Base model trained only on these public resources. The normalized score aggregates per-task results into a single comparable number, and it is the measuring instrument that makes the baseline reproducible.

What would settle it

Compute the overlap between LLaSO-Instruct and LLaSO-Eval, for example by checking n-gram or task-prompt similarity, or by deleting and rewriting the answers to any overlapping eval tasks; if the normalized score drops well below 0.72, the reported baseline was inflated by training/eval overlap.

Watch

Extended reading notes

Core claim

The central claim is that a fully open, end-to-end stack can replace the fragmented and opaque pipelines common in speech-language modeling, and that a reference model trained only on released public data can be competitive. LLaSO-Base, at 3.8B parameters, achieves a normalized score of 0.72 on LLaSO-Eval, which the paper presents as a strong, reproducible baseline surpassing comparable models. The analysis further shows that broader training coverage helps performance but leaves gaps on unseen tasks, particularly pure-audio ones.

Load-bearing premise

The 0.72 score is a true measure of capability only if the LLaSO-Eval tasks are not substantially overlapping with the LLaSO-Instruct training data, and the abstract does not state that they are disjoint.

Editorial extensions

If this is right

  • Any lab can reconstruct LLaSO-Base from the released data and check the 0.72 score, since training data and configurations are public.
  • Future speech-language models can be compared on LLaSO-Eval without reimplementing private pipelines.
  • The 3.8B public-data baseline gives smaller labs a reference point for what is achievable without proprietary data.
  • The observed generalization gaps mark pure-audio tasks as a distinct weakness that future work should target.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If LLaSO-Eval tasks overlap substantially with LLaSO-Instruct training data, the 0.72 score could partly reflect memorization; a contamination audit would settle how much is genuine capability.
  • An open stack like LLaSO could make speech-language scaling-law studies possible, because data composition and training configurations would be fully known.
  • The pure-audio gap hints that speech-text alignment alone does not confer strong audio-only understanding, suggesting a testable extension: adding audio-only instruction data and measuring whether the gap closes.
  • If the baseline holds up, the release pattern could shift field norms, encouraging other groups to publish training data and configurations alongside model weights.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 2 minor

Summary. The paper claims to introduce LLaSO, a fully open, end-to-end framework for large speech-language modeling, comprising LLaSO-Align (12M instances), LLaSO-Instruct (13.5M instances), LLaSO-Eval, and LLaSO-Base (3.8B parameters). The abstract states that LLaSO-Base achieves a normalized score of 0.72, establishing a reproducible baseline that surpasses comparable models, and that code, datasets, and weights will be released. However, the supplied full text is an unrelated condensed-matter paper, 'Atomically thin silver films for enhanced nanoscale nonlinear optics' (arXiv:2508.15417), and contains no description of the LLaSO framework, its training data, evaluation protocol, or results. The central claims are therefore unsupported by the manuscript as submitted.

Significance. If the claims were accompanied by the promised open stack and a precise evaluation protocol, the contribution would be significant: large-scale public speech-text alignment and instruction-tuning datasets, a 3.8B reference model trained only on public data, and a standardized benchmark would give the LSLM community concrete reproducibility infrastructure. The abstract's numbers are concrete and falsifiable. However, the submitted manuscript contains none of this material. The architecture, dataset construction, training configuration, normalization formula, and benchmark design are all absent. Because the full text is a different paper, the significance of the LLaSO contribution cannot be assessed from the submitted artifact.

major comments (3)
  1. [Full text] The entire submitted full text is an unrelated physics paper titled 'Atomically thin silver films for enhanced nanoscale nonlinear optics' (arXiv:2508.15417). There are no methods, equations, architecture details, or evaluation results for any LLaSO component. The central claim of a reproducible open LSLM framework therefore has no supporting material in the manuscript. This is a load-bearing mismatch, not a presentational issue.
  2. [Abstract] The abstract reports a 'normalized score of 0.72' but does not define the normalization procedure, the evaluation set, or the comparison models. No evidence is provided anywhere in the manuscript to support the assertion that this score 'surpasses comparable models.' The reader cannot verify whether this is an external benchmark, an internal metric, or a self-defined score.
  3. [Abstract] The manuscript does not state whether the LLaSO-Eval tasks are disjoint from the LLaSO-Instruct and LLaSO-Align training corpora. The abstract itself acknowledges 'generalization gaps ... on unseen tasks,' which implies that the main evaluation may include seen or overlapping tasks. Without an explicit data-leakage statement or benchmark-card, the 0.72 score cannot be interpreted as a measure of generalization rather than memorization.
minor comments (2)
  1. [Abstract] The abstract points to a GitHub repository for code and data, but the full text contains no repository pointer, license information, dataset documentation, or model card. The reproducibility promise is therefore not yet verifiable.
  2. [Abstract] The phrase 'first fully open, end-to-end framework' is asserted without comparative context. Even if the full manuscript were present, it should cite and discuss existing open speech-language modeling efforts to justify the priority claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity identified; supplied full text is an unrelated physics paper, so the LLaSO derivation chain cannot be checked.

full rationale

The target paper (arXiv:2508.15418, LLaSO) is represented only by its abstract, while the supplied full text is a different manuscript: 'Atomically thin silver films for enhanced nanoscale nonlinear optics' (arXiv:2508.15417). The abstract makes claims about LLaSO-Align (12M instances), LLaSO-Instruct (13.5M instances), LLaSO-Eval, a 3.8B-parameter model, and a normalized score of 0.72, but none of these claims are accompanied by any derivation, evaluation protocol, normalization formula, benchmark construction, or comparison set in the material available. Without the actual manuscript, no specific reduction can be exhibited (no equation equating a 'prediction' to a fitted input, no benchmark that is constructed from training data by the paper's own text, no self-citation chain that forces the result). The reader's speculation that LLaSO-Eval tasks may overlap LLaSO-Instruct is not asserted by the paper and cannot be verified from the supplied text; per the hard rules, an unsupported overlap hypothesis is not a demonstrated circularity. Therefore the honest finding is that no circularity can be established from the evidence provided, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 2 assumptions · 4 invented entities

No numeric fitting parameters are visible in the abstract. The main non-empirical inputs are the design choices of the datasets, benchmark, and model. The first appears as a set of named resources that the paper contributes, so they are listed as introduced entities rather than hidden parameters.

assumptions (2)
  • domain assumption Publicly released training data is sufficient to train a competitive speech-language model without proprietary data.
    The abstract states LLaSO-Base is 'trained exclusively on our public data.' This assumes the public corpora are representative and large enough for the reported performance.
  • domain assumption Standardized evaluation on LLaSO-Eval is a valid proxy for LSLM capability.
    The normalized score is treated as a meaningful measure; the benchmark design is not described in the abstract.
invented entities (4)
  • LLaSO-Align independent evidence
    purpose: 12M-instance speech-text alignment corpus
    The corpus is released publicly, so others can inspect and reuse it; its usefulness can be tested independently.
  • LLaSO-Instruct independent evidence
    purpose: 13.5M-instance multi-task instruction-tuning dataset
    Publicly released; can be used independently.
  • LLaSO-Eval independent evidence
    purpose: Benchmark for standardized evaluation
    A benchmark provides a measurable external handle; if released, others can run it.
  • LLaSO-Base independent evidence
    purpose: 3.8B-parameter reference model trained on the public data
    Model weights are promised for release; its outputs can be evaluated by third parties.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLaSO: A Foundational Framework for Reproducible Research in Large Language and Speech Model." pith.science (2026). https://pith.science/paper/QFMR4T4N

@misc{pith2026250815418,
  author       = {Pith},
  title        = {Pith review of: LLaSO: A Foundational Framework for Reproducible Research in Large Language and Speech Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QFMR4T4N}},
  note         = {Machine review of arXiv:2508.15418}
}
read the original abstract

The development of Large Speech-Language Models (LSLMs) has been slowed by fragmented architectures and a lack of transparency, hindering the systematic comparison and reproducibility of research. Unlike in the vision-language domain, the LSLM field suffers from the common practice of releasing model weights without their corresponding training data and configurations. To address these critical gaps, we introduce LLaSO, the first fully open, end-to-end framework for large-scale speech-language modeling. LLaSO provides the community with three essential resources: (1) LLaSO-Align, a 12M-instance speech-text alignment corpus; (2) LLaSO-Instruct, a 13.5M-instance multi-task instruction-tuning dataset; and (3) LLaSO-Eval, a reproducible benchmark for standardized evaluation. To validate our framework, we build and release LLaSO-Base, a 3.8B-parameter reference model trained exclusively on our public data. It achieves a normalized score of 0.72, establishing a strong, reproducible baseline that surpasses comparable models. Our analysis reveals that while broader training coverage enhances performance, significant generalization gaps persist on unseen tasks, particularly in pure audio scenarios. By releasing the complete stack of data, benchmarks, and models, LLaSO establishes a foundational open standard to unify research efforts and accelerate community-driven progress in LSLMs. We release the code, dataset, pretrained models, and results in https://github.com/EIT-NLP/LLaSO.

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. FlexiSLM: A Dynamic and Controllable Frame Rate Spoken Language Model

    cs.SD 2026-06 unverdicted novelty 7.0 of 10

    FlexiSLM is the first spoken language model supporting dynamic and controllable frame rates on speech input and output, outperforming fixed-rate 7B models at high quality and enabling faster inference at lower rates l...

  2. Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    C-Gate represents speech frames as convex combinations of LLM token embeddings to enforce manifold compatibility, delivering up to 48.7% relative WER reduction on LibriSpeech while preserving emotion recognition accuracy.

Reference graph

Works this paper leans on

95 extracted references · 13 canonical work pages · cited by 2 Pith papers

  1. [1]

    Vaibhav Adlakha, Parishad BehnamGhader, Xing Han Lu, Nicholas Meade, and Siva Reddy. 2024. https://arxiv.org/abs/2307.16877 Evaluating correctness and faithfulness of instruction-following models for question answering . Preprint, arXiv:2307.16877

    arXiv 2024

  2. [2]

    Afroz Ahamad, Ankit Anand, and Pranesh Bhargava. 2020. Accentdb: A database of non-native english accents to assist neural speech recognition. arXiv preprint arXiv:2005.07973

    work page Pith review arXiv 2020

  3. [3]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, and 8 others. 2022. https://arxiv.org/abs/2204.14198 Flamingo: a visu...

    arXiv 2022

  4. [4]

    Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. 2019. Common voice: A massively-multilingual speech corpus. arXiv preprint arXiv:1912.06670

    arXiv 2019

  5. [5]

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. 2023. https://arxiv.org/abs/2308.01390 Openflamingo: An open-source framework for training large autoregressive vis...

    arXiv 2023

  6. [6]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. https://arxiv.org/abs/2308.12966 Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond . Preprint, arXiv:2308.12966

    arXiv 2023

  7. [7]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. https://arxiv.org/abs/2502.13923 Qwen2.5-vl technical report . Preprint, arXiv:2502.13923

    arXiv 2025

  8. [8]

    Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. 2020. Slurp: A spoken language understanding resource package. arXiv preprint arXiv:2011.13205

    arXiv 2020

Show all 95 references
  1. [9]

    Mathieu Bernard and Hadrien Titeux. 2021. https://doi.org/10.21105/joss.03958 Phonemizer: Text to phones transcription for multiple languages in python . Journal of Open Source Software, 6(68):3958

    2021 doi

  2. [10]

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, and 1 others. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258

    2021 arXiv

  3. [11]

    brianlan. 2017. T his script calculates P honeme E rror R ate using package leven (edit distance algorithm). U sers can also choose whether to merge phonemes (refer to the paper S peaker-independent phone recognition using hidden M arkov models) during calculation. --- gist.gi...

    2017

  4. [12]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  5. [13]

    Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. 2014. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing, 5(4):377--390

    2014

  6. [14]

    Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. 2024. https://arxiv.org/abs/2312.06742 Honeybee: Locality-enhanced projector for multimodal llm . Preprint, arXiv:2312.06742

    2024 arXiv

  7. [15]

    Sahil Chaudhary. 2023. Code alpaca: An instruction-following llama model for code generation. https://github.com/sahil280114/codealpaca

    2023

  8. [16]

    Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, and 1 others. 2021. Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio. arXiv preprint arXiv:2106.06909

    2021 arXiv

  9. [17]

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. 2023 a . https://arxiv.org/abs/2310.09478 Minigpt-v2: large language model as a unified interface for vision-language mult...

    2023 arXiv

  10. [18]

    Stanley F Chen, Douglas Beeferman, and Roni Rosenfeld. 1998. Evaluation metrics for language models

    1998

  11. [19]

    Ye Chen, Wei Cai, Liangmin Wu, Xiaowei Li, Zhanxuan Xin, and Cong Fu. 2023 b . Tigerbot: An open multilingual multitask llm. arXiv preprint arXiv:2312.08688

    2023 arXiv

  12. [20]

    Tan, and Haizhou Li

    Yiming Chen, Xianghu Yue, Chen Zhang, Xiaoxue Gao, Robby T. Tan, and Haizhou Li. 2024. https://arxiv.org/abs/2410.17196 Voicebench: Benchmarking llm-based voice assistants . Preprint, arXiv:2410.17196

    2024 arXiv

  13. [21]

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. 2024. https://arxiv.org/abs/2407.10759 Qwen2-audio technical report . Preprint, arXiv:2407.10759

    2024 arXiv

  14. [22]

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. https://arxiv.org/abs/2311.07919 Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models . Preprint, arXiv:2311.07919

    2023 arXiv

  15. [23]

    Federico Cocchi, Nicholas Moratelli, Davide Caffagni, Sara Sarto, Lorenzo Baraldi, Marcella Cornia, and Rita Cucchiara. 2025. https://arxiv.org/abs/2503.15621 Llava-more: A comparative study of llms and visual backbones for enhanced visual instruction tuning . Preprint, arXiv:...

    2025 arXiv

  16. [24]

    Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world’s first truly open instruction-tuned llm

    2023

  17. [25]

    Alexandre D \'e fossez, Laurent Mazar \'e , Manu Orsini, Am \'e lie Royer, Patrick P \'e rez, Herv \'e J \'e gou, Edouard Grave, and Neil Zeghidour. 2024. Moshi: a speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00037

    2024 arXiv

  18. [26]

    Zhihao Du, Jiaming Wang, Qian Chen, Yunfei Chu, Zhifu Gao, Zerui Li, Kai Hu, Xiaohuan Zhou, Jin Xu, Ziyang Ma, and 1 others. 2023. Lauragpt: Listen, attend, understand, and regenerate audio with gpt. arXiv preprint arXiv:2310.04673

    2023 arXiv

  19. [27]

    Benjamin Elizalde, Soham Deshmukh, and Huaming Wang. 2024. https://arxiv.org/abs/2309.05767 Natural language supervision for general-purpose audio representations . Preprint, arXiv:2309.05767

    2024 arXiv

  20. [28]

    Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Douglas Eck, Karen Simonyan, and Mohammad Norouzi. 2017. https://arxiv.org/abs/arXiv:1704.01279 Neural audio synthesis of musical notes with wavenet autoencoders

    2017 arXiv

  21. [29]

    Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. 2025. https://arxiv.org/abs/2409.06666 Llama-omni: Seamless speech interaction with large language models . Preprint, arXiv:2409.06666

    2025 arXiv

  22. [30]

    Chaoyou Fu, Yi-Fan Zhang, Shukang Yin, Bo Li, Xinyu Fang, Sirui Zhao, Haodong Duan, Xing Sun, Ziwei Liu, Liang Wang, and 1 others. 2024. Mme-survey: A comprehensive survey on evaluation of multimodal llms. arXiv preprint arXiv:2411.15296

    2024 arXiv

  23. [31]

    Sreyan Ghosh, Zhifeng Kong, Sonal Kumar, S Sakshi, Jaehyeon Kim, Wei Ping, Rafael Valle, Dinesh Manocha, and Bryan Catanzaro. 2025. Audio flamingo 2: An audio-language model with long-audio understanding and expert reasoning abilities. arXiv preprint arXiv:2503.03983

    2025 arXiv

  24. [32]

    Yuan Gong, Sameer Khurana, Leonid Karlinsky, and James Glass. 2023 a . Whisper-at: Noise-robust automatic speech recognizers are also strong general audio event taggers. arXiv preprint arXiv:2307.03183

    2023 arXiv

  25. [33]

    Yuan Gong, Alexander H Liu, Hongyin Luo, Leonid Karlinsky, and James Glass. 2023 b . Joint audio and speech understanding. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1--8. IEEE

    2023

  26. [34]

    Yuan Gong, Jin Yu, and James Glass. 2022. Vocalsound: A dataset for improving human vocal sounds recognition. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 151--155. IEEE

    2022

  27. [35]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  28. [36]

    Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2022. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:2212.09689

    2022 arXiv

  29. [37]

    Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. 2023. https://arxiv.org/abs/2306.14610 Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality . Preprint, arXiv:2306.14610

    2023 arXiv

  30. [38]

    Kaiyi Huang, Chengqi Duan, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. 2025. https://arxiv.org/abs/2307.06350 T2i-compbench++: An enhanced and comprehensive benchmark for compositional text-to-image generation . Preprint, arXiv:2307.06350

    2025 arXiv

  31. [39]

    huggingface. 2023. E valuation metrics for A S R - H ugging F ace A udio C ourse --- huggingface.co. https://huggingface.co/learn/audio-course/chapter5/evaluation. [Accessed 15-07-2025]

    2023

  32. [40]

    Keith Ito and Linda Johnson. 2017. The lj speech dataset. https://keithito.com/LJ-Speech-Dataset/

    2017

  33. [41]

    Feng Jiang, Zhiyu Lin, Fan Bu, Yuhao Du, Benyou Wang, and Haizhou Li. 2025. https://arxiv.org/abs/2503.05085 S2s-arena, evaluating speech2speech protocols on instruction following with paralinguistic information . Preprint, arXiv:2503.05085

    2025 arXiv

  34. [42]

    Jordan. 2023. https://huggingface.co/datasets/whitefox44/AlpacaGPT3.5Customized Whitefox44/alpacagpt3.5customized · datasets at hugging face

    2023

  35. [43]

    Charles, and 21 others

    KimiTeam, Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, Zhengtao Wang, Chu Wei, Yifei Xin, Xinran Xu, Jianwei Yu, Yutao Zhang, Xinyu Zhou, Y. Charles, and 21 others. 2025. https://arxiv.org/abs/2504.18425 Kimi-a...

    2025 arXiv

  36. [45]

    Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. 2024 b . Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. arXiv preprint arXiv:2402.01831

    2024 arXiv

  37. [46]

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2023 a . https://arxiv.org/abs/2306.00890 Llava-med: Training a large language-and-vision assistant for biomedicine in one day . Preprint, arXiv:2306.00890

    2023 arXiv

  38. [47]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023 b . https://arxiv.org/abs/2301.12597 Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models . Preprint, arXiv:2301.12597

    2023 arXiv

  39. [48]

    Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, Lingpeng Kong, and Qi Liu. 2023 c . https://arxiv.org/abs/2306.04387 M ^3 it: A large-scale dataset towards multi-modal multilingual instruction tuning . Preprint,...

    2023 arXiv

  40. [49]

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2023. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122

    2023 arXiv

  41. [50]

    Junyan Lin, Haoran Chen, Dawei Zhu, and Xiaoyu Shen. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.325 To preserve or to compress: An in-depth study of connector selection in multimodal large language models . In Proceedings of the 2024 Conference on Empirical Methods in N...

    2024 doi

  42. [51]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023 a . Visual instruction tuning. Advances in neural information processing systems, 36:34892--34916

    2023

  43. [52]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023 b . https://arxiv.org/abs/2304.08485 Visual instruction tuning . Preprint, arXiv:2304.08485

    2023 arXiv

  44. [53]

    Yinghao Ma, Anders land, Anton Ragni, Bleiz MacSen Del Sette, Charalampos Saitis, Chris Donahue, Chenghua Lin, Christos Plachouras, Emmanouil Benetos, Elona Shatri, and 1 others. 2024. Foundation models for music: A survey. arXiv preprint arXiv:2408.14340

    2024 arXiv

  45. [54]

    Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. 2023. https://arxiv.org/abs/2212.07796 Crepe: Can vision-language foundation models reason compositionally? Preprint, arXiv:2212.07796

    2023 arXiv

  46. [55]

    Potsawee Manakul, Guangzhi Sun, Warit Sirichotedumrong, Kasima Tharnpipitchai, and Kunat Pipatanakul. 2024. https://arxiv.org/abs/2409.10999 Enhancing low-resource language and instruction following capabilities of audio language models . Preprint, arXiv:2409.10999

    2024 arXiv

  47. [56]

    Plumbley, Yuexian Zou, and Wenwu Wang

    Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D. Plumbley, Yuexian Zou, and Wenwu Wang. 2024. https://doi.org/10.1109/taslp.2024.3419446 Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research...

    2024

  48. [57]

    Andrew Morris, Viktoria Maier, and Phil Green. 2004. From wer and ril to mer and wil: improved evaluation measures for connected speech recognition

    2004

  49. [58]

    Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.02707

    2023 arXiv

  50. [59]

    Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. 2017. Voxceleb: a large-scale speaker identification dataset. arXiv preprint arXiv:1706.08612

    2017 arXiv

  51. [60]

    Gonzalo Navarro. 2001. A guided tour to approximate string matching. ACM computing surveys (CSUR), 33(1):31--88

    2001

  52. [61]

    OpenAI. 2024. https://openai.com/index/hello-gpt-4o Hello gpt-4o

    2024

  53. [62]

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206--5210. IEEE

    2015

  54. [63]

    Prabhat Pandey, Rupak Vignesh Swaminathan, KV Girish, Arunasish Sen, Jian Xie, Grant P Strimel, and Andreas Schwarz. 2025. Sift-50m: A large-scale multilingual dataset for speech instruction fine-tuning. arXiv preprint arXiv:2504.09081

    2025 arXiv

  55. [64]

    Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. 2019. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:1904.08779

    2019 arXiv

  56. [65]

    Jing Peng, Yucheng Wang, Bohan Li, Yiwei Guo, Hankun Wang, Yangui Fang, Yu Xi, Haoyu Li, Xu Li, Ke Zhang, Shuai Wang, and Kai Yu. 2025. https://arxiv.org/abs/2410.18908 A survey on speech large language models for understanding . Preprint, arXiv:2410.18908

    2025

  57. [66]

    Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2018. Meld: A multimodal multi-party dataset for emotion recognition in conversations. arXiv preprint arXiv:1810.02508

    2018 arXiv

  58. [67]

    Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. 2020. Mls: A large-scale multilingual dataset for speech research. ArXiv, abs/2012.03411

    2020 arXiv

  59. [68]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In International conference on machine...

    2021

  60. [69]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. https://arxiv.org/abs/2212.04356 Robust speech recognition via large-scale weak supervision . Preprint, arXiv:2212.04356

    2022 arXiv

  61. [70]

    Ricardo Reimao and Vassilios Tzerpos. 2019. For: A dataset for synthetic speech detection. In 2019 International Conference on Speech Technology and Human-Computer Dialogue (SpeD), pages 1--10. IEEE

    2019

  62. [71]

    Tiger Research. 2023. https://huggingface.co/datasets/TigerResearch/tigerbot-alpaca-en-50k Tigerresearch/tigerbot-alpaca-en-50k · datasets at hugging face

    2023

  63. [72]

    Andrew Rouditchenko, Saurabhchand Bhati, Edson Araujo, Samuel Thomas, Hilde Kuehne, Rogerio Feris, and James Glass. 2025. Omni-r1: Do you really need audio to fine-tune your audio llm? arXiv preprint arXiv:2505.09439

    2025

  64. [73]

    RuterNorway. 2023. R uter N orway/ O pen O rca N o-15k · D atasets at H ugging F ace --- huggingface.co. https://huggingface.co/datasets/RuterNorway/OpenOrcaNo-15k. [Accessed 17-04-2025]

    2023

  65. [74]

    Yi Su, Jisheng Bai, Qisheng Xu, Kele Xu, and Yong Dou. 2025. Audio-language models for audio-centric tasks: A survey. arXiv preprint arXiv:2501.15177

    2025 arXiv

  66. [75]

    Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2024. https://arxiv.org/abs/2310.13289 Salmonn: Towards generic hearing abilities for large language models . Preprint, arXiv:2310.13289

    2024 arXiv

  67. [76]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca

    2023

  68. [77]

    Christophe Veaux, Junichi Yamagishi, Kirsten MacDonald, and 1 others. 2016. Superseded-cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit.(2016). URL http://datashare. is. ed. ac. uk/handle/10283/2651

    2016

  69. [78]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. https://arxiv.org/abs/2409.12191 Qwen2-v...

    2024 arXiv

  70. [79]

    Pete Warden. 1804. Speech commands: A dataset for limited-vocabulary speech recognition. arxiv 2018. arXiv preprint arXiv:1804.03209

    2018 arXiv

  71. [80]

    Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Marianna Nezhurina, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2024. https://arxiv.org/abs/2211.06687 Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation . Preprint, arXiv:2211.06687

    2024 arXiv

  72. [81]

    Zhifei Xie, Mingbao Lin, Zihang Liu, Pengcheng Wu, Shuicheng Yan, and Chunyan Miao. 2025. https://arxiv.org/abs/2503.02318 Audio-reasoner: Improving reasoning capability in large audio language models . Preprint, arXiv:2503.02318

    2025

  73. [82]

    Zhifei Xie and Changqiao Wu. 2024 a . https://arxiv.org/abs/2408.16725 Mini-omni: Language models can hear, talk while thinking in streaming . Preprint, arXiv:2408.16725

    2024 arXiv

  74. [83]

    Zhifei Xie and Changqiao Wu. 2024 b . https://arxiv.org/abs/2410.11190 Mini-omni2: Towards open-source gpt-4o with vision, speech and duplex capabilities . Preprint, arXiv:2410.11190

    2024 arXiv

  75. [84]

    Xinchen-ai. 2024. G it H ub - xinchen-ai/ W estlake- O mni --- github.com. https://github.com/xinchen-ai/Westlake-Omni. [Accessed 21-06-2025]

    2024

  76. [85]

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. 2025. https://arxiv.org/abs/2503.20215 Qwen2.5-omni technical report . Preprint, arXiv:2503.20215

    2025 arXiv

  77. [86]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. https://arxiv.org/abs/2505.09388 Qw...

    2025 arXiv

  78. [87]

    Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. https://doi.org/10.18653/v1/D15-1237 W iki QA : A challenge dataset for open-domain question answering . In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2013--2018, Lisbon, Portu...

    2015 doi

  79. [88]

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2024. https://doi.org/10.1093/nsr/nwae403 A survey on multimodal large language models . National Science Review, 11(12)

    2024 doi

  80. [89]

    Chien yu Huang, Ke-Han Lu, Shih-Heng Wang, Chi-Yuan Hsiao, Chun-Yi Kuan, Haibin Wu, Siddhant Arora, Kai-Wei Chang, Jiatong Shi, Yifan Peng, Roshan Sharma, Shinji Watanabe, Bhiksha Ramakrishnan, Shady Shehata, and Hung yi Lee. 2024. https://arxiv.org/abs/2309.09510 Dynamic-supe...

    2024 arXiv

  81. [90]

    Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. 2024 a . https://arxiv.org/abs/2412.02612 Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot . Preprint, arXiv:2412.02612

    2024 arXiv

  82. [91]

    Yunan Zeng, Yan Huang, Jinjin Zhang, Zequn Jie, Zhenhua Chai, and Liang Wang. 2024 b . Investigating compositional challenges in vision-language models for visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14141--14151

    2024

  83. [92]

    Junbo Zhang, Zhiwen Zhang, Yongqing Wang, Zhiyong Yan, Qiong Song, Yukai Huang, Ke Li, Daniel Povey, and Yujun Wang. 2021. speechocean762: An open-source non-native english speech corpus for pronunciation assessment. In Proc. Interspeech 2021

    2021

  84. [93]

    Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. 2024. https://arxiv.org/abs/2308.16692 Speechtokenizer: Unified speech tokenizer for speech large language models . Preprint, arXiv:2308.16692

    2024 arXiv

  85. [94]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. https://arxiv.org/abs/2304.10592 Minigpt-4: Enhancing vision-language understanding with advanced large language models . Preprint, arXiv:2304.10592

    2023 arXiv

  86. [95]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  87. [96]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.