Pith. sign in

REVIEW 3 major objections 5 minor 57 references

Towards Reliable Large Audio Language Model

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a large audio language model can learn to say 'I don't know' on one audio modality and that this refusal ability transfers to the other two audio modalities.

desk verdict First LALM reliability study with a genuinely interesting cross-modal rejection result, but the RGI metric conflates answer corrections with refusals, so the 'meta ability' claim is not yet supported. read the letter →

arxiv 2505.19294 v1 pith:ZYYPIFZM submitted 2025-05-25 cs.SD cs.CLcs.HCcs.MMeess.AS

Ziyang Ma , Xiquan Li , Yakun Song , Wenxi Chen , Chenpeng Du , Jian Wu , Yuanzhe Chen , Zhuo Chen
show 3 more authors
Yuping Wang Yuxuan Wang Xie Chen
This is my paper · ORCID
classification cs.SDcs.CLcs.HCcs.MMeess.AS
keywords largeaudiolanguagemodelsreliabilityIdon'tknowrefusalGainIndexknowledgeboundarycross-modaltransferLoRAfine-tuningmulti-modalchain-of-thought
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether large audio language models can be made reliable in a specific sense: refusing to answer questions that fall outside what they know. It compares training-free methods (IDK prompting, multi-modal chain-of-thought, and a task-agent pipeline) with training-based post-training, where the model is fine-tuned on a dataset of questions it failed to answer consistently. To judge which method genuinely helps, the paper introduces the Reliability Gain Index (RGI), the log ratio of the increase in rejecting wrong answers to the increase in rejecting correct ones. The paper's central finding is that reliability awareness is a 'meta ability': after LoRA fine-tuning on one audio modality, refusal behavior transfers to the other two, with RGI greater than zero on every cross-modal pair. If that is right, one targeted fine-tuning step could make a universal audio assistant appropriately humble on speech, music, and sound without per-modality data.

What carries the argument

The central instrument is the Reliability Gain Index, $RGI = \log(\Delta_{\mathrm{Hum}}/\Delta_{\mathrm{Con}})$, where $\Delta_{\mathrm{Hum}}$ is the fraction of originally wrong answers that a method converts into refusals and $\Delta_{\mathrm{Con}}$ is the fraction of originally correct answers it converts into refusals. Because RGI is a log ratio, a positive value means the method rejects more unknown questions than known ones, a negative value means it is mostly throwing away correct answers, and a method that rejects both classes at the same rate gets RGI 0 instead of an inflated reliability score. The training-based arm is carried by a model-specific IDK dataset made with the 5@5 rule: the model samples five answers per question, the question keeps its ground-truth label only if all five samples are correct, and otherwise it is relabelled IDK. LoRA fine-tuning on that dataset produces the cross-modal models whose refusal behavior the RGI heatmap measures.

What would settle it

Rebuild the IDK training dataset with a different label rule, for example 1@5, labels supplied by an independent model, or human-verified known/unknown judgements, and recompute the cross-modal RGI heatmap; if any target modality drops to $\mathrm{RGI} \le 0$, the claimed meta-ability is an artifact of the 5@5 rule rather than a transferable refusal skill.

Watch

Extended reading notes

Core claim

The central claim is that a large audio language model can be taught to recognize its knowledge boundary and say 'I don't know', and that this ability is not tied to the audio modality it was learned on. Concretely, the authors LoRA-fine-tune Qwen2-Audio-7B-Instruct on a model-specific IDK dataset built from one modality of the MMAU benchmark, then evaluate on the other two. The reported result is that every cross-modal entry has RGI greater than zero, for example training on sound and testing on music gives RGI 0.19, training on sound and testing on speech gives RGI 0.23, and training on speech and testing on sound gives RGI 0.35. The authors read these positive values as evidence that 'awareness of reliability' is a meta ability that transfers across speech, music, and general sound despite the structural and content differences among them. The same tables show that all tested reliable methods improve truthfulness, while the training-based method achieves the best trade-off between accuracy and truthfulness, with an overall RGI of 0.26.

Load-bearing premise

The load-bearing premise is that asking the model the same question five times and labelling it 'unknown' whenever any of the five answers is wrong gives an accurate map of what the model actually knows, so every cross-modal RGI number inherits the quality of those self-made labels.

Editorial extensions

If this is right

  • Both training-free and training-based methods raise truthfulness and reliability relative to the MMAU baseline, so some reliability gain is available without any model update.
  • Training-based LoRA fine-tuning has the best RGI, 0.26 overall, while losing less accuracy than the prompting methods, making post-training the more balanced route to reliable LALMs.
  • With all cross-modal RGI values greater than zero, a single IDK fine-tune on one audio modality appears sufficient to produce refusal behavior on all three modalities.
  • The RGI metric corrects a flaw in previous reliability scores, which can increase even when a method rejects correct and wrong answers at the same rate and is therefore not genuinely improving reliability.
  • The authors frame the result as a foundation for omni-modal reliability, expecting the same meta-ability to be transferable to modalities beyond audio, such as video.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the IDK labels come from the model's own 5@5 self-consistency, the cross-modal transfer may be a transfer of a sampling-based refusal heuristic rather than a model-internal epistemic state; replacing the labels with an independent judge's known/unknown decisions would test this directly.
  • A natural follow-up is to hold the task type fixed while changing the audio content, or vice versa, to determine whether the transferred RGI comes from task-level knowledge boundaries or from the specific sound, music, or speech surface.
  • The log-ratio form of RGI is not defined when a method rejects zero correct answers; a smoothed or bounded variant would make the metric usable in deployments where over-conservatism is completely absent.
  • The same cross-modal test could be run in image-language models, fine-tuning on one visual domain and measuring whether refusal of unknown visual questions transfers to unrelated domains, which would show whether this meta-ability is audio-specific.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper investigates methods for making large audio language models (LALMs) refuse to answer questions outside their knowledge, using Qwen2-Audio-7B-Instruct on the MMAU benchmark across speech, music, and environmental sound. It compares training-free methods (IDK prompting, multimodal chain-of-thought, and a task agent) with training-based LoRA fine-tuning on a model-generated IDK dataset. The authors propose a new metric, the Reliability Gain Index (RGI), defined as the log ratio of relative humbleness increase to relative conservativeness increase, and argue that RGI better captures the trade-off between rejecting unknown questions and not rejecting known ones. The central claim is that reliability awareness is a 'meta ability' that transfers across audio modalities, supported by cross-modal RGI values greater than zero in Figure 3 and Appendix F.1.

Significance. If the main claim were established, this would be a useful contribution: it is one of the first systematic studies of reliability for LALMs, it compares several practical methods, and the idea that refusal behavior can transfer across audio modalities would be of interest to the broader audio-language-model community. The paper ships useful experimental details: open prompts, GPT-4o-mini answer normalization, LoRA hyperparameters, and evaluations on multiple baseline models in the appendix. However, the proposed RGI metric, as defined, conflates ordinary correctness changes with refusal behavior, and the cross-modal transfer claim rests on this metric. The same-modality training-based results also lack a stated train/test split. These issues are load-bearing for the headline conclusions, so the paper requires substantial revision before the claims can be accepted.

major comments (3)
  1. [Section 4.2, Eqs. (7)-(10)] The definitions of Delta_Con and Delta_Hum do not match the prose. Eq. (8) gives Delta_Con = (Nc - Ncc)/Nc = (Ncr + Ncw)/Nc, so correct answers that become wrong (Ncw) are counted as conservativeness, even though the text says conservativeness is the proportion of correct answers that were rejected. Similarly, Eq. (9) gives Delta_Hum = (Nw - Nww)/Nw = (Nwc + Nwr)/Nw, so wrong answers that become correct (Nwc) are counted as humbleness, even though the text says humbleness is the proportion of wrong answers that were correctly rejected. Consequently RGI rewards ordinary accuracy gains even if the model never produces an 'I don't know' response, and penalizes accuracy losses even if rejection behavior is unchanged. In the cross-modal experiments the accuracy shifts are material (Table 10: the Music-trained model reaches 62.76% on Sound versus a 60.96% baseline; the Sound-trained model drops to 48.95% on Speech versus 50.75%), while rejection rates are modest (Table 12: the Music-trained model rejects only 7.81% on Sound). A small number of wrong-to-correct flips can therefore make RGI > 0 without any transferred IDK refusal. The claim in Section 4.3 and Figure 3 that RGI > 0 demonstrates transfer of reliability awareness is not supported by the metric as defined. Please redefine the indices using Nwr/Nw and Ncr/Nc (or an equivalent decomposition), recompute Tables 2 and 11 and Figure 3, and report the Nwc/Nwr and Ncr/Ncw breakdowns.
  2. [Section 5.1, IDK dataset construction] The IDK training labels are generated by the model itself: a question is treated as known only if all five self-sampled answers are correct (the 5@5 rule). This makes the 'knowledge boundary' depend on the model's own decoding behavior rather than on an independent ground truth. If sampling is near-deterministic, if temperature is not controlled, or if five samples are not diverse, the IDK labels will misstate the model's actual knowledge distribution. This does not by itself invalidate the empirical cross-modal result, but it means the transfer claim can be interpreted as transfer of the 5@5 labeling rule rather than of a genuine 'meta ability' to know when to refuse. Please report the sampling temperature and variance, compare results under different K@N thresholds, and ideally validate the labels against human or external judgments. At minimum, show that the cross-modal RGI pattern in Figure 3 is robust to the label-construction rule.
  3. [Section 5.1 and Appendix A, evaluation protocol] The experiments are conducted on the MMAU Test-mini set, and Section 5.1 describes constructing the IDK dataset from the same benchmark, but the paper never states that the LoRA training questions are disjoint from the evaluation questions. If the models in Table 1 and Table 2 are trained on IDK labels derived from the same Test-mini questions on which they are then evaluated, the training-based gains are contaminated by memorization and cannot validate the method. Please specify the exact train/evaluation split or cross-validation procedure. This issue does not directly invalidate the cross-modal heatmap (where training and test modalities differ), but it is essential for interpreting the same-modality results and the 'cross-validation' row in Tables 1 and 2.
minor comments (5)
  1. [Appendix E, Eqs. (17)-(18)] The worked example is arithmetically inconsistent: if 10% of both correct and incorrect answers are converted into rejections starting from 50%/50%, the resulting distribution is 45% correct, 10% rejected, and 45% wrong, not 40%/20%/40%. Either the percentages or the resulting numbers should be corrected.
  2. [Section 4.2, Eq. (10)] RGI is undefined when Delta_Con = 0, which can occur in principle; the paper should state how zero denominators are handled or add a small smoothing term.
  3. [Section 5.3, Figure 3] The heatmap in Figure 3 reports point estimates without confidence intervals. With roughly 333 questions per modality, the differences among RGI values may be within sampling noise; bootstrap intervals would strengthen the transfer claim.
  4. [Table 1 and Section 5.2] The row labeled 'Ours (Normalized)' is confusing; clarify that it is Qwen2-Audio-Instruct evaluated with GPT-4o-mini answer normalization, to distinguish it from the raw MMAU baseline.
  5. [Throughout] There are several typos and phrasing issues: 'donate' in Eq. (1) should be 'denote'; 'konw' in the Introduction; 'greater than0' in Section 5.2; 'cross-model SFT' in Section 5.3 should be 'cross-modal SFT'; and 'tables bellow' in Appendix C.1.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: cross-modal RGI>0 is an empirical outcome, not a definitionally forced result; the noted RGI metric conflation is a validity concern, not a circular step.

full rationale

The paper's derivation chain is not circular under the strict criteria. The IDK training labels are model-specific, generated by a 5@5 self-consistency rule on Qwen2-Audio (Section 5.1). This is self-referential as a label-construction choice, but the central claim—that LoRA fine-tuning on one audio modality yields RGI>0 on the other two modalities (Figure 3, Tables 10–11)—is an empirical outcome measured on the external MMAU benchmark. Nothing in the definitions forces the transfer result; it must be observed. The RGI metric in Section 4.2 does have an internal-validity weakness: Eq. 9 defines ΔHum = (Nw − Nww)/Nw = (Nwc + Nwr)/Nw, so wrong-to-correct flips are algebraically counted as humbleness even without any rejection, and symmetrically Eq. 8 counts correct-to-wrong flips as conservativeness. Thus RGI>0 can in principle be driven by ordinary accuracy transfer rather than learned IDK refusal. That is a correctness/validity concern about what the metric isolates, not a circularity: the paper does not assume RGI>0, and the sign of RGI is not forced by the metric's construction. No load-bearing self-citation, imported uniqueness theorem, ansatz-smuggling citation, fitted-parameter-renamed-as-prediction, or renaming of a known result is present. The honest finding is therefore no significant circularity (score 0).

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The central results rest on three hand-chosen ingredients: the 5@5 self-consistency rule for IDK labels, the per-modality LoRA alpha values, and a 1000-question subset of one benchmark. The RGI metric is the paper's own construct and carries the cross-modal 'meta ability' claim, so its definitional quirk is a free-standing limitation rather than an inherited one.

free parameters (3)
  • K@N IDK labeling threshold = 5@5
    Defines which MMAU Test-mini questions become IDK training labels; Figure 5 shows IDK share changes with the threshold (50.2% at 1@5 to 63.5% at 5@5), so the training signal depends on this hand choice.
  • LoRA alpha = sound: 32, music: 16, speech: 16
    Hyperparameter chosen per modality without a described validation split; Figure 4 shows RGI decreases with alpha and becomes negative, so headline results depend on this choice.
  • Number of decoding samples N = 5
    Each question is sampled five times to decide IDK labels; sampling temperature and decoding parameters are not reported, so label stability is unverified.
assumptions (4)
  • domain assumption MMAU Test-mini (1000 questions) is representative of the full MMAU test set.
    The paper uses only Test-mini because the main test set is closed; per-modality cells contain on the order of 300 questions, so RGI heatmaps are point estimates on small samples. (Appendix A)
  • ad hoc to paper A question is within the model's knowledge iff all 5 self-sampled answers are correct (5@5).
    This rule converts the model's own outputs into ground-truth IDK labels with no calibration against human labels or an external judge. (Section 5.1)
  • domain assumption GPT-4o-mini normalization correctly maps raw model outputs to answer choices or IDK.
    Every metric in Tables 1, 2, 10, and 11 is computed after this external API normalizes answers; systematic normalization errors would propagate into all comparisons. (Section 5.2, Appendix C.2)
  • domain assumption The baseline model never rejects, so N = Nc + Nw before applying a reliable method.
    Equation 6 partitions baseline answers only into correct and wrong; if the baseline spontaneously refuses, the denominators of Delta-Con and Delta-Hum are mis-specified. (Section 4.2, Eq. 6)
invented entities (1)
  • Reliability Gain Index (RGI)
    purpose: Quantify whether a reliability method increases rejection of wrong answers more than rejection of correct answers.
    Defined in Eq. 10 as log(Delta-Hum / Delta-Con). It has no external validation, and Delta-Hum as defined in Eq. 9 counts wrong answers that become correct as if they were rejected, so the metric does not cleanly measure what its name claims.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Reliable Large Audio Language Model." pith.science (2026). https://pith.science/paper/ZYYPIFZM

@misc{pith2026250519294,
  author       = {Pith},
  title        = {Pith review of: Towards Reliable Large Audio Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZYYPIFZM}},
  note         = {Machine review of arXiv:2505.19294}
}
read the original abstract

Recent advancements in large audio language models (LALMs) have demonstrated impressive results and promising prospects in universal understanding and reasoning across speech, music, and general sound. However, these models still lack the ability to recognize their knowledge boundaries and refuse to answer questions they don't know proactively. While there have been successful attempts to enhance the reliability of LLMs, reliable LALMs remain largely unexplored. In this paper, we systematically investigate various approaches towards reliable LALMs, including training-free methods such as multi-modal chain-of-thought (MCoT), and training-based methods such as supervised fine-tuning (SFT). Besides, we identify the limitations of previous evaluation metrics and propose a new metric, the Reliability Gain Index (RGI), to assess the effectiveness of different reliable methods. Our findings suggest that both training-free and training-based methods enhance the reliability of LALMs to different extents. Moreover, we find that awareness of reliability is a "meta ability", which can be transferred across different audio modalities, although significant structural and content differences exist among sound, music, and speech.

Figures

Figures reproduced from arXiv: 2505.19294 by the authors.

Figure 1
Figure 1. Illustration of Unreliable vs. Reliable LALMs. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of Reliability Awareness Transferability in LALM. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Heatmap for cross-modal SFT results. The [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Rejection Rate(%), Reliability(%), and Reliability Gain Index (RGI) performance for different LoRA [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: IDK percentage for constructing IDK dataset [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 40 canonical work pages

  1. [1]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

    arXiv 2023

  2. [2]

    Saurabhchand Bhati, Yuan Gong, Leonid Karlinsky, Hilde Kuehne, Rogerio Feris, and James Glass. 2024. State-space large audio language models. arXiv preprint arXiv:2411.15685

    work page Pith review arXiv 2024

  3. [3]

    Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei. 2023. BEATs : Audio pre-training with acoustic tokenizers. Proc. ICML

    work page 2023

  4. [4]

    Wenxi Chen, Ziyang Ma, Xiquan Li, Xuenan Xu, Yuzhe Liang, Zhisheng Zheng, Kai Yu, and Xie Chen. 2025. SLAM-AAC : Enhancing audio captioning with paraphrasing augmentation and CLAP-Refine through llms. Proc. ICASSP

    work page 2025

  5. [5]

    Qinyuan Cheng, Tianxiang Sun, Xiangyang Liu, Wenwei Zhang, Zhangyue Yin, Shimin Li, Linyang Li, Zhengfu He, Kai Chen, and Xipeng Qiu. 2024 a . Can ai assistants know what they don't know? Proc. ICML

    work page 2024

  6. [6]

    Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Jingdong Sun, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, and Alexander Hauptmann. 2024 b . Emotion-LLaMA : Multimodal emotion recognition and reasoning with instruction tuning. Proc. NeurIPS

    work page 2024

  7. [7]

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna. lmsys. org (accessed 14 April 2023)

    2023

  8. [8]

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. 2024. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759

    arXiv 2024

Show all 57 references
  1. [9]

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919

    2023 arXiv

  2. [10]

    Keqi Deng, Guangzhi Sun, and Philip C Woodland. 2024 a . Wav2prompt: End-to-end speech prompt generation and tuning for llm in zero and few-shot learning. arXiv preprint arXiv:2406.00522

    2024 arXiv

  3. [11]

    Zihao Deng, Yinghao Ma, Yudong Liu, Rongchen Guo, Ge Zhang, Wenhu Chen, Wenhao Huang, and Emmanouil Benetos. 2024 b . Musilingo: Bridging music and text with pre-trained language models for music captioning and query response. Proc. NAACL

    2024

  4. [12]

    Xuelong Geng, Tianyi Xu, Kun Wei, Bingshen Mu, Hongfei Xue, He Wang, Yangze Li, Pengcheng Guo, Yuhang Dai, Longhao Li, et al. 2024. Unveiling the potential of llm-based asr on chinese open-source datasets. arXiv preprint arXiv:2405.02132

    2024 arXiv

  5. [13]

    Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. 2024. GAMA : A large audio-language model with advanced audio understanding and complex reasoning abilities. Proc. EMNLP

    2024

  6. [14]

    Yuan Gong, Alexander H Liu, Hongyin Luo, Leonid Karlinsky, and James Glass. 2023. Joint audio and speech understanding. Proc. ASRU

    2023

  7. [15]

    Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James Glass. 2024. Listen, think, and understand. Proc. ICLR

    2024

  8. [16]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

    2021 arXiv

  9. [17]

    Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, et al. 2024. Wavllm: Towards robust and adaptive speech large language model. Proc. EMNLP

    2024

  10. [18]

    Shuyue Jia, Subhrangshu Bit, Edward Searls, Lindsey Claus, Pengrui Fan, Varuna H Jasodanand, Meagan V Lauber, Divya Veerapaneni, William M Wang, Rhoda Au, et al. 2024. MedPodGPT : A multilingual audio-augmented large language model for medical research and education. medRxiv

    2024

  11. [19]

    Wonjune Kang, Junteng Jia, Chunyang Wu, Wei Zhou, Egor Lakomkin, Yashesh Gaur, Leda Sari, Suyoun Kim, Ke Li, Jay Mahadeokar, et al. 2024. Frozen large language models can perceive paralinguistic aspects of speech. arXiv preprint arXiv:2410.01162

    2024 arXiv

  12. [20]

    Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. 2024. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. Proc. ICML

    2024

  13. [21]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023 a . Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. Proc. ICML

    2023

  14. [22]

    Xiquan Li, Wenxi Chen, Ziyang Ma, Xuenan Xu, Yuzhe Liang, Zhisheng Zheng, Qiuqiang Kong, and Xie Chen. 2025. DRCap : Decoding CLAP latents with retrieval-augmented generation for zero-shot audio captioning. Proc. ICASSP

    2025

  15. [23]

    Yizhi Li, Ge Zhang, Yinghao Ma, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Zekun Wang, Jian Yang, et al. 2024. Omnibench: Towards the future of universal omni-language models. arXiv preprint arXiv:2409.15272

    2024

  16. [24]

    Yuang Li, Yu Wu, Jinyu Li, and Shujie Liu. 2023 b . Prompting large language models for zero-shot domain adaptation in speech recognition. Proc. ASRU

    2023

  17. [25]

    Guan-Ting Lin, Prashanth Gurunath Shivakumar, Ankur Gandhe, Chao-Han Huck Yang, Yile Gu, Shalini Ghosh, Andreas Stolcke, Hung-yi Lee, and Ivan Bulyko. 2024. Paralinguistics-enhanced large language modeling of spoken dialogue. Proc. ICASSP

    2024

  18. [26]

    Jizhong Liu, Gang Li, Junbo Zhang, Chenyu Liu, Heinrich Dinkel, Yongqing Wang, Zhiyong Yan, Yujun Wang, and Bin Wang. 2024 a . Leveraging ced encoder and large language models for automated audio captioning. Proc. DCASE Challenge

    2024

  19. [27]

    Shansong Liu, Atin Sakkeer Hussain, Chenshuo Sun, and Ying Shan. 2024 b . Music understanding llama: Advancing text-to-music generation with question answering and captioning. Proc. ICASSP

    2024

  20. [28]

    Ke-Han Lu, Zhehuai Chen, Szu-Wei Fu, He Huang, Boris Ginsburg, Yu-Chiang Frank Wang, and Hung-yi Lee. 2024 a . Desta: Enhancing speech language models through descriptive speech-text alignment. Proc. Interspeech

    2024

  21. [29]

    Ke-Han Lu, Zhehuai Chen, Szu-Wei Fu, Chao-Han Huck Yang, Jagadeesh Balam, Boris Ginsburg, Yu-Chiang Frank Wang, and Hung-yi Lee. 2024 b . Developing instruction-following speech language model without speech instruction-tuning data. arXiv preprint arXiv:2409.20007

    2024 arXiv

  22. [30]

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Proc. NeurIPS

    2022

  23. [31]

    Ziyang Ma, Yakun Song, Chenpeng Du, Jian Cong, Zhuo Chen, Yuping Wang, Yuxuan Wang, and Xie Chen. 2025 a . Language model can listen while speaking. Proc. AAAI

    2025

  24. [32]

    Ziyang Ma, Guanrou Yang, Yifan Yang, Zhifu Gao, Jiaming Wang, Zhihao Du, Fan Yu, Qian Chen, Siqi Zheng, Shiliang Zhang, et al. 2024. An embarrassingly simple approach for LLM with strong ASR capacity. arXiv preprint arXiv:2402.08846

    2024 arXiv

  25. [33]

    Ziyang Ma, Guanrou Yang, Yifan Yang, Zhifu Gao, Jiaming Wang, Zhihao Du, Fan Yu, Qian Chen, Siqi Zheng, Shiliang Zhang, et al. 2025 b . Speech recognition meets large language model: Benchmarking, models, and exploration. In Proc. AAAI

    2025

  26. [34]

    Lingwei Meng, Shujie Hu, Jiawen Kang, Zhaoqing Li, Yuejiao Wang, Wenxuan Wu, Xixin Wu, Xunying Liu, and Helen Meng. 2024. Large language model can transcribe speech in multi-talker scenarios with versatile instructions. arXiv preprint arXiv:2409.08596

    2024 arXiv

  27. [35]

    Xingwei Qu, Yuelin Bai, Yinghao Ma, Ziya Zhou, Ka Man Lo, Jiaheng Liu, Ruibin Yuan, Lejun Min, Xueling Liu, Tianyu Zhang, et al. 2024. Mupt: A generative symbolic music pretrained transformer. arXiv preprint arXiv:2404.06393

    2024 arXiv

  28. [36]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. Proc. ICML

    2023

  29. [37]

    S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. 2024. Mmau: A massive multi-task audio understanding and reasoning benchmark. arXiv preprint arXiv:2410.19168

    2024 arXiv

  30. [38]

    Mohan Shi, Zengrui Jin, Yaoxun Xu, Yong Xu, Shi-Xiong Zhang, Kun Wei, Yiwen Shao, Chunlei Zhang, and Dong Yu. 2024. Advancing multi-talker asr performance with large language models. Proc. SLT

    2024

  31. [39]

    Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2024. SALMONN : Towards generic hearing abilities for large language models. Proc. ICLR

    2024

  32. [40]

    Chen Wang, Minpeng Liao, Zhongqiang Huang, Jinliang Lu, Junhong Wu, Yuchen Liu, Chengqing Zong, and Jiajun Zhang. 2023 a . Blsp: Bootstrapping language-speech pre-training via behavior alignment of continuation writing. arXiv preprint arXiv:2309.00916

    2023 arXiv

  33. [41]

    Jiaming Wang, Zhihao Du, Qian Chen, Yunfei Chu, Zhifu Gao, Zerui Li, Kai Hu, Xiaohuan Zhou, Jin Xu, Ziyang Ma, et al. 2023 b . LauraGPT : Listen, attend, understand, and regenerate audio with GPT . arXiv preprint arXiv:2310.04673

    2023 arXiv

  34. [42]

    Jian Wu, Yashesh Gaur, Zhuo Chen, Long Zhou, Yimeng Zhu, Tianrui Wang, Jinyu Li, Shujie Liu, Bo Ren, Linquan Liu, et al. 2023 a . On decoder-only architecture for speech-to-text and large language model integration. Proc. ASRU

    2023

  35. [43]

    Shih-Lun Wu, Xuankai Chang, Gordon Wichern, Jee-weon Jung, Fran c ois Germain, Jonathan Le Roux, and Shinji Watanabe. 2023 b . BEATs -based audio captioning model with INSTRUCTOR embedding supervision and ChatGPT mix-up. Proc. DCASE Challenge

    2023

  36. [44]

    Hongshen Xu, Zichen Zhu, Situo Zhang, Da Ma, Shuai Fan, Lu Chen, and Kai Yu. 2024 a . Rejection improves reliability: Training llms to refuse unknown questions using rl from knowledge feedback. Proc. COLM

    2024

  37. [45]

    Yaoxun Xu, Hangting Chen, Jianwei Yu, Qiaochu Huang, Zhiyong Wu, Shi-Xiong Zhang, Guangzhi Li, Yi Luo, and Rongzhi Gu. 2024 b . SECap : Speech emotion captioning with large language model. Proc. AAAI

    2024

  38. [46]

    Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee K Wong, Zhenguo Li, and Hengshuang Zhao. 2024 c . Drivegpt4: Interpretable end-to-end autonomous driving via large language model. Proc. RA-L

    2024

  39. [47]

    Guanrou Yang, Ziyang Ma, Zhifu Gao, Shiliang Zhang, and Xie Chen. 2024 a . Ctc-assisted llm-based contextual asr. Proc. SLT

    2024

  40. [48]

    Guanrou Yang, Ziyang Ma, Fan Yu, Zhifu Gao, Shiliang Zhang, and Xie Chen. 2024 b . Mala-asr: Multimedia-assisted llm-based asr. Proc. Interspeech

    2024

  41. [49]

    Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, et al. 2024 c . Air-bench: Benchmarking large audio-language models via generative comprehension. Proc. ACL

    2024

  42. [50]

    Yuqing Yang, Ethan Chern, Xipeng Qiu, Graham Neubig, and Pengfei Liu. 2024 d . Alignment for honesty. Proc. NeurIPS

    2024

  43. [51]

    Gal Yona, Roee Aharoni, and Mor Geva. 2024. Can large language models faithfully express their intrinsic uncertainty in words? Proc. EMNLP

    2024

  44. [52]

    Wenyi Yu, Changli Tang, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2024. Connecting speech encoder and large language model for ASR . Proc. ICASSP

    2024

  45. [53]

    Ruibin Yuan, Hanfeng Lin, Yi Wang, Zeyue Tian, Shangda Wu, Tianhao Shen, Ge Zhang, Yuhang Wu, Cong Liu, Ziya Zhou, et al. 2024. Chatmusician: Understanding and generating music intrinsically with llm. Proc. ACL

    2024

  46. [54]

    Hanning Zhang, Shizhe Diao, Yong Lin, Yi R Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. 2024. R-tuning: Teaching large language models to refuse unknown questions. Proc. NAACL

    2024

  47. [55]

    Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. 2023. Multimodal chain-of-thought reasoning in language models. Proc. TMLR

    2023

  48. [56]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  49. [57]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.