Pith. sign in

REVIEW 4 major objections 5 minor 58 references

Teaching Audio-Aware Large Language Models What Does Not Hear: Mitigating Hallucinations through Synthesized Negative Samples

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training on self-generated absent-sound examples teaches audio large language models to stop reporting sounds that are not in the audio, with no change to the language model itself.

desk verdict A clean, data-efficient training recipe with a plausible mechanism, but the headline gain rests on an unpublished self-authored benchmark, so the size of the claimed improvement is not yet interpretable. read the letter →

arxiv 2505.14518 v2 pith:AXUIASNA submitted 2025-05-20 eess.AS cs.CLcs.SD

Chun-Yi Kuan , Hung-yi Lee This is my paper · ORCID
classification eess.AScs.CLcs.SD
keywords audiohallucinationaudio-awarelargelanguagemodelsnegativesamplescontrastive-liketrainingsynthesizeddatalightweightadapterfrozenbackbonequestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that audio-aware large language models can be taught to hear absence: by training on synthesized examples of sounds that are not present, the model learns to say 'no' rather than inventing sound events. The authors show that a small trainable adapter placed between a frozen audio encoder and a frozen language model, trained on self-generated positive and negative captions, reaches 77.5 percent accuracy on an audio hallucination benchmark, comfortably above the best prior model at 70.8 percent. Because the language model is never modified and the training data totals about 316 hours, the recipe offers a data- and computation-efficient path to more reliable audio AI. If correct, this makes 'negative knowledge'—explicit examples of what is not there—a practical training signal for multimodal systems.

What carries the argument

The load-bearing mechanism is the synthesized negative sample: the backbone LLM is prompted with an audio dataset's text metadata (captions or tags) to generate lists of sound events that are absent from the clip, and these lists are used as training targets alongside positive descriptions. A frozen Whisper encoder supplies audio representations, a randomly initialized Qformer-and-linear adapter projects them into the LLM's input space, and the whole system is trained with a next-token prediction loss. The contrastive-like signal comes from explicitly teaching with both 'present' and 'absent' labels, with the separated positive/negative variant outperforming a combined-prompt variant.

What would settle it

Run the trained 'Positive + Negative' model on a newly built audio hallucination dataset whose audio clips and yes/no questions are guaranteed disjoint from AudioCaps, Clotho, FSD50K, MACS, ESC50, UrbanSound8K, VocalSound, and AudioSet-20K; if accuracy drops to baseline levels, the effect is benchmark-specific.

Watch

Extended reading notes

Core claim

The central discovery is that contrastive-like training with synthesized negative samples—descriptions of sound events the backbone LLM says are absent from the audio—substantially reduces object hallucination in audio-aware LLMs, and that separating positive and negative samples into different training instances works better than combining them. The authors report that the 'Positive + Negative' variant achieves 77.5 percent accuracy and 77.1 weighted F1 on the audio hallucination benchmark (versus 70.8 and 68.5 for the strongest baseline), while also staying competitive on audio question answering and a synonym/hypernym reasoning test. The method requires no parameter changes to the Whisper encoder or the LLaMA-3.1-8B backbone; only the audio modality adapter is trained. This is the first proposal, on the authors' account, to attack ALLM hallucination from the training side.

Load-bearing premise

The hallucination benchmark is introduced only by citation to the authors' own prior papers (Section 4.2.1) with no description of its construction, and the training corpora are not shown to be disjoint from it, so the reported gains could be optimistic.

Editorial extensions

If this is right

  • Audio hallucination benchmarks can be improved without retraining or LoRA-tuning the language model, hence with minimal risk of degrading text abilities.
  • Training data requirements drop to roughly 3–30 percent of the hours used by large baselines such as Qwen-Audio and SALMONN, making hallucination mitigation accessible with modest compute.
  • The benefit transfers to conventional audio QA and semantic-reasoning benchmarks, so the negative-sample signal does not trade away general understanding.
  • Separating positive and negative samples into distinct examples beats merging them into a single combined prompt, guiding how future contrastive data should be structured.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same 'teach absence' recipe could plausibly transfer to speech hallucination in spoken-dialogue models, since the mechanism is about contrastive targets rather than audio-specific features.
  • Because negative examples are drawn from text metadata alone, the method likely works best when the metadata is rich; for sparsely labeled clips, mining negatives from a learned sound-event vocabulary might strengthen the effect.
  • A natural testable extension is to choose absent sounds adaptively—picking near-miss or confusable sounds that the model tends to hallucinate—rather than sampling them from the LLM's free association.
  • The reported benchmark is the authors' own; an independent evaluation with disjoint clips would clarify how much of the gain is due to negative training versus curation of the test set.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes LISTEN, a training method that freezes a Whisper encoder and a LLaMA-3.1-8B backbone and trains only a Qformer-based adapter on synthetic audio-text pairs. The synthetic data is generated by the backbone LLM from caption/tag metadata, using separate positive, negative, and combined prompts. The central claim is that adding negative samples markedly reduces audio hallucination (accuracy 77.5 vs 70.8 for SALMONN-13B, weighted F1 77.1 vs 68.5) while retaining competitive performance on Clotho-AQA and on a synonym/hypernym reasoning test, using far fewer training hours than prior ALLMs.

Significance. If the results hold, LISTEN offers a lightweight, data-efficient recipe for suppressing audio hallucinations without modifying LLM parameters, with a clear ablation isolating the contribution of negative samples and a comparison showing that self-generated data from the backbone is sufficient relative to a stronger generator (Gemini-1.5-Pro). The architecture is simple and reproducible, and the comparison to several strong baselines is useful. However, the headline evaluation rests on an unpublished benchmark from the authors' own prior work with no documented provenance or split analysis, and the reported numbers are not accompanied by error bars or significance tests. The central claim therefore needs stronger external validation before it can be accepted at face value.

major comments (4)
  1. [§4.2.1, Table 3] The Audio Hallucination benchmark is cited only to Refs. [1,2] and is not described: there is no dataset size, audio source, question distribution, construction procedure, or split information. Because Table 2 shows that the training corpora include AudioCaps, Clotho, FSD50K, MACS, and UrbanSound8K, and because the benchmark questions ask about sound events in exactly the kind of caption/tag metadata used to construct the training data, the reported gains (77.5 accuracy / 77.1 weighted F1 vs 70.8 / 68.5 for SALMONN-13B) are uninterpretable without an explicit statement that the benchmark clips and questions are disjoint from all training data, including the synthetic targets. Please release the benchmark or evaluate on an independently constructed external set, and report the benchmark's yes/no prior and question source.
  2. [§3.1, §4.1] Negative samples are generated by the backbone LLM from text metadata alone. For datasets with partial annotations such as AudioSet-20K and FSD50K, an unlisted sound event is not guaranteed to be absent from the audio; an LLM-generated 'absent' sound may therefore actually be present, creating false-negative training labels. This could teach the model to over-answer 'no' and inflate F1(N). Please quantify this risk (e.g., by listening to a random subset of negative samples, or by checking negative tags against a stronger audio tagger) and, if the risk is nonzero, describe how it is mitigated or filtered.
  3. [Table 3, §5.1] The winning model's Yes rate is 63.4%, while all strong baselines have Yes rates between 76.8% and 92.1%. The large improvement in F1(N) (74.0 vs 60.1 for SALMONN-13B) may therefore reflect a global shift toward 'no' answers rather than improved discrimination between present and absent sounds. Please report the benchmark's yes/no prior and add a threshold-invariant analysis (e.g., ROC/AUC or balanced accuracy) to separate calibration/prior effects from genuine hallucination suppression.
  4. [Table 3 and §4.3] All results in Table 3 appear to be from a single training run with no error bars, confidence intervals, or significance tests. Several comparisons are close (e.g., 77.5 vs 74.5 between 'Positive + Negative' and 'Combined'), so the reader cannot tell whether the reported differences are stable. Please provide multiple seeds, bootstrapped intervals, or a paired significance test for the main hallucination and QA comparisons.
minor comments (5)
  1. [Table 3] The table contains typos: 'Audio Hallcination' should be 'Audio Hallucination', and 'Y esshows' should be 'Yes shows'.
  2. [§4.1, Table 2] The table header renders 'VocalSound' with an anomalous space ('V ocalSound'); also, the text says duration is calculated per unique audio excluding duplicates, but it is not clear whether the reported 'Duration' is raw audio length or processed/trimmed duration used in training.
  3. [§3.1, Eq. (1)] The notation P_seed ∈ {D_caption, D_tag} conflates the annotation content with the prompt template; renaming the seed content (e.g., T_seed) would make the distinction between annotation and prompt clearer.
  4. [§1, §2] The Introduction's claim of being 'the first to address hallucination in audio-aware large language models' is broader than what the Related Work section supports; 'the first to address it from a model-training perspective' is the defensible version.
  5. [§3.2] The method is called 'contrastive-like', but no contrastive loss is used; please specify explicitly that the contrastive element comes from paired positive/negative training targets rather than a contrastive objective, to avoid confusion with standard contrastive learning.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the training objectives and evaluation metrics are not defined in terms of each other, and no fitted parameter is relabeled as a prediction.

full rationale

The paper is an empirical training/evaluation study, not a formal derivation. The method trains a lightweight adapter using next-token prediction on LLM-synthesized positive and negative descriptions (§3), with the backbone LLM and audio encoder frozen. The hallucination evaluation (§4.2.1, Table 3) uses a benchmark introduced in the authors' prior work [1,2], which is a self-citation and a reproducibility concern because the benchmark is not described and no train/eval disjointness is stated. However, this is not circularity in the derivation-chain sense: the benchmark is not an input to the training objective, no hyperparameter or adapter weight is fitted to its labels, and the claims do not reduce to an equation that defines the output in terms of the evaluation. Independent grounding is provided by Clotho-AQA and the Synonym/Hypernym Test, which are external benchmarks and show competitive or superior performance. The method's use of self-generated negative samples is the intended training signal, not a hidden equivalence to the test metric. Therefore no step qualifies as self-definitional, fitted-input-called-prediction, or self-citation load-bearing circularity under the required evidence standard.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method adds no new physical or conceptual entities; its contributions rest on a set of domain assumptions about data validity, plus unreported hyperparameters.

free parameters (3)
  • Training hyperparameters (learning rate, batch size, epochs) = not reported
    Not disclosed in the paper; these affect adapter convergence and final performance.
  • Negative sampling count and prompt variants = not reported (examples only)
    The paper gives example prompts but no specification of how many negative samples per audio or how they are sampled; this directly shapes the contrastive signal.
  • Qformer query count and projection dimensions = not reported
    The adapter architecture (Section 3.2) is described qualitatively; specific dimensions are omitted, preventing exact reproduction.
assumptions (4)
  • domain assumption The hallucination benchmark from Refs [1,2] validly measures audio hallucination.
    Section 4.2.1 relies on this benchmark without describing its construction or data.
  • domain assumption Sounds generated as 'absent' by the backbone LLM from text metadata are truly absent from the corresponding audio.
    Section 3.1 uses these as training targets; if a generated absent sound is actually present but unannotated, the model is trained to produce false negatives.
  • domain assumption Whisper-small encoder representations retain sufficient audio-event information for the adapter to align with the LLM.
    Section 3.2 chooses Whisper-small based on references [24,25], but no analysis in this paper checks that its features capture the relevant events for the benchmark.
  • domain assumption The metadata captions/tags are complete enough to serve as the seed for positive sample generation.
    Section 3.1 builds all training targets on D_caption or D_tag; incomplete or noisy annotations would bias both positive and negative samples.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Teaching Audio-Aware Large Language Models What Does Not Hear: Mitigating Hallucinations through Synthesized Negative Samples." pith.science (2026). https://pith.science/paper/AXUIASNA

@misc{pith2026250514518,
  author       = {Pith},
  title        = {Pith review of: Teaching Audio-Aware Large Language Models What Does Not Hear: Mitigating Hallucinations through Synthesized Negative Samples},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AXUIASNA}},
  note         = {Machine review of arXiv:2505.14518}
}
read the original abstract

Recent advancements in audio-aware large language models (ALLMs) enable them to process and understand audio inputs. However, these models often hallucinate non-existent sound events, reducing their reliability in real-world applications. To address this, we propose LISTEN (Learning to Identify Sounds Through Extended Negative Samples), a contrastive-like training method that enhances ALLMs' ability to distinguish between present and absent sounds using synthesized data from the backbone LLM. Unlike prior approaches, our method requires no modification to LLM parameters and efficiently integrates audio representations via a lightweight adapter. Experiments show that LISTEN effectively mitigates hallucinations while maintaining impressive performance on existing audio question and reasoning benchmarks. At the same time, it is more efficient in both data and computation.

Figures

Figures reproduced from arXiv: 2505.14518 by the authors.

Figure 1
Figure 1. Data Construction Stage uses a backbone large language model to generate audio-text aligned descriptions. Training Stage focuses on audio-text alignment by training an audio modality adapter, while the backbone large language model remains frozen. dio datasets. While many approaches rely on external LLMs for data generation, our method takes a different approach by utilizing the backbone LLM of the ALLM itself, simi… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 34 canonical work pages

  1. [1]

    These models can process audio, speech, and text in- puts at the same time, using text prompts to extract relevant information from audio and speech

    Introduction Audio-aware large language models (ALLMs) expand on tradi- tional text-based LLMs by incorporating the ability to process audio. These models can process audio, speech, and text in- puts at the same time, using text prompts to extract relevant information from audio and speech. This advancement enables large language models to understand mult...

    work page

  2. [2]

    Hallucination Mitigation: This work is the first to ad- dress hallucination in audio-aware large language models. To tackle this issue, we propose LISTEN (Learning to Identify Sounds Through Extended Negative Samples), a contrastive- like method that helps models distinguish between present and absent sounds, effectively reducing hallucination and im- pro...

    work page

  3. [3]

    This is achieved by leveraging a backbone-LLM- synthesized dataset, which automatically generates audio- text pairs and contrastive data across general audio scenarios

    Efficient Data Utilization and Training: Our method in- tegrates general audio representations into LLMs using a lightweight adapter without modifying any LLM parameters, requiring only 3% to 30% of the dataset used by baseline models. This is achieved by leveraging a backbone-LLM- synthesized dataset, which automatically generates audio- text pairs and c...

    work page

  4. [4]

    Related Work Audio-aware large language models (ALLMs) extend tradi- tional LLMs by incorporating audio perception, enabling them to process both audio and text inputs for a wider range of audio- related tasks. Recent advancements [8–10, 12–15, 18–21] have focused on bridging modality gaps by generating paired audio- text data, often leveraging powerful l...

    work page Pith review arXiv 2025

  5. [5]

    Method 3.1. Data Construction During the data construction stage, inspired by [8, 9], our goal is to create audio-text pair data with minimal textual discrepan- cies between the underlying LLM used in the ALLM and the constructed training data. While LLMs cannot directly process audio, they excel at understanding and interpreting textual meta- data [2]. T...

    work page

  6. [6]

    For example, Replay the audio

    Positive Samples Generation Prompt ( Ppos) aims to gen- erate descriptions of sound events that actually occur in the audio. For example, Replay the audio

    work page

  7. [7]

    For example, Identify sounds that are absent as con- trasting examples

    Negative Samples Generation Prompt (Pneg) aims to gen- erate descriptions of sound events that are not present in the audio. For example, Identify sounds that are absent as con- trasting examples

    work page

  8. [8]

    It aims to generate descriptions of both the sound events that are present and those that are ab- sent in the audio

    Combined Samples Generation Prompt (Pcomb) integrates both of the above prompts. It aims to generate descriptions of both the sound events that are present and those that are ab- sent in the audio. For example, Replay the audio and identify sounds that are absent as contrasting examples . In summary, the final input prompt (Pf inal) to the LLM can be expr...

    work page

Show all 58 references
  1. [9]

    We utilize the foundation model Whisper 2

    Audio Encoder. We utilize the foundation model Whisper 2

  2. [10]

    Blsp: Bootstrapping language-speech pre- training via behavior alignment of continuation writing,

    C. Wang et al. , “Blsp: Bootstrapping language-speech pre- training via behavior alignment of continuation writing,” arXiv preprint arXiv:2309.00916, 2023

    2023 arXiv

  3. [11]

    Birds chirping 3

    A car driving by 2. Birds chirping 3. A dog barking Combined Samples Generation Prompt (Pcomb): Replay the audio and identify sounds that are absent as contrasting examples. Response: Specific sound events detected in the provided audio:

  4. [12]

    Water pouring Contrastive examples of specific sound events not present in the provided audio:

    A woman’s voice 2. Water pouring Contrastive examples of specific sound events not present in the provided audio:

  5. [13]

    A dog barking 3

    A car driving by 2. A dog barking 3. The door bang recognition and speech translation tasks. As Whisper fol- lows an encoder-decoder architecture, we utilize only its en- coder component. To retain the advantages of the pre-trained model, the audio encoder’s parameters remain frozen

  6. [14]

    This study employs the instruction-tuned LLaMA-3.1-8B 3 [26] as the core large language model

    Backbone Large Language Model. This study employs the instruction-tuned LLaMA-3.1-8B 3 [26] as the core large language model. To preserve the model’s original text- processing capabilities, we opted not to apply LoRA [16] or fine-tune any of its parameters

  7. [15]

    The only trainable component is the audio modality adapter, which is randomly initialized

    Audio Modality Adapter. The only trainable component is the audio modality adapter, which is randomly initialized. This adapter projects the output representations extracted by the audio encoder into the input dimension of the backbone large language model. In detail, we emplo...

  8. [16]

    yes” and “no

    Experiment 4.1. Training Datasets The training datasets we used include AudioSet-20K [28], Au- dioCaps [29], FSD50K [30], MACS [31], ESC50 [32], Ur- banSound8K [33], Clotho [34], and V ocalSound [35]. Among these original datasets, AudioCaps, Clotho, and MACS contain ground tr...

  9. [17]

    Table 3: Evaluation results of our proposed models and other baseline models

    Positive-only Training Data: The training set contains only positive samples, with a total of 2N data points. Table 3: Evaluation results of our proposed models and other baseline models. Acc denotes accuracy, F1 (Y) and F1 (N) are F1 scores for yes and no answers, respectivel...

  10. [18]

    Positive and Negative Training Data: The training set in- cludes both positive and negative samples, each with N data points, resulting in a total of 2N data points

  11. [19]

    A combined sam- ple includes both sound events that are present and those that are absent within a single sample

    Combined Training Data: The training set consists of com- bined samples, as described in Section 3.1. A combined sam- ple includes both sound events that are present and those that are absent within a single sample. In contrast, positive sam- ples only contain present sound ev...

  12. [20]

    Results 5.1. Performance on Audio Hallucination In Table 3, compared to previous baselines, our proposed mod- els achieve the best performance on the audio hallucination benchmark, both in terms of accuracy and F1 score. For ques- tions where the ground truth is “no”, which ar...

  13. [21]

    Additionally, we achieve impressive results on audio un- derstanding and reasoning benchmarks, demonstrating the ro- bustness and versatility of this approach

    Conclusion, Future work, and Limitation This paper proposes a contrastive-like training method that en- ables audio-aware large language models to recognize both present and absent sounds in audio by self-generated data, effec- tively reducing hallucination issues and improvin...

  14. [22]

    Can large audio-language models truly hear? tackling hallucinations with multi-task assessment and stepwise audio reasoning,

    C.-Y . Kuan and H.-y. Lee, “Can large audio-language models truly hear? tackling hallucinations with multi-task assessment and stepwise audio reasoning,” inICASSP 2025-2025 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025

    2025

  15. [23]

    as our audio encoder. Prior studies [24, 25] have high- lighted Whisper’s impressive performance on various audio- related tasks, despite its original design for automatic speech 2huggingface.co/openai/whisper-small Table 1: Examples generated under different sampling prompts....

  16. [24]

    Understanding sounds, missing the questions: The challenge of object hallucination in large audio-language models,

    C.-Y . Kuan, W.-P. Huang, and H.-y. Lee, “Understanding sounds, missing the questions: The challenge of object hallucination in large audio-language models,” Interspeech 2024, 2024

    2024

  17. [25]

    A survey of hallucination in large foundation models,

    V . Rawte, A. Sheth, and A. Das, “A survey of hallucination in large foundation models,” arXiv:2309.05922, 2023

    2023 arXiv

  18. [26]

    Siren’s song in the ai ocean: a survey on halluci- nation in large language models,

    Y . Zhang et al., “Siren’s song in the ai ocean: a survey on halluci- nation in large language models,” arXiv:2309.01219, 2023

    2023 arXiv

  19. [27]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,

    L. Huang et al. , “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,” arXiv:2311.05232, 2023

    2023 arXiv

  20. [28]

    Chainpoll: A high efficacy method for llm hallucination detection,

    R. Friel and A. Sanyal, “Chainpoll: A high efficacy method for llm hallucination detection,” arXiv preprint arXiv:2310.18344 , 2023

    2023 arXiv

  21. [29]

    A survey on hallucination in large vision- language models,

    H. Liu, W. Xue, Y . Chen, D. Chen, X. Zhao, K. Wang, L. Hou, R. Li, and W. Peng, “A survey on hallucination in large vision- language models,” arXiv preprint arXiv:2402.00253, 2024

    2024 arXiv

  22. [30]

    Towards general-purpose speech abilities for large language models using unpaired data,

    Y . Fathullah et al. , “Towards general-purpose speech abilities for large language models using unpaired data,” arXiv preprint arXiv:2311.06753, 2023

    2023 arXiv

  23. [31]

    Developing instruction-following speech lan- guage model without speech instruction-tuning data,

    K.-H. Lu et al. , “Developing instruction-following speech lan- guage model without speech instruction-tuning data,” arXiv preprint arXiv:2409.20007, 2024

    2024 arXiv

  24. [32]

    Blsp-emo: Towards empathetic large speech-language models,

    ——, “Blsp-emo: Towards empathetic large speech-language models,” arXiv preprint arXiv:2406.03872, 2024

    2024 arXiv

  25. [33]

    Qwen-audio: Advancing universal audio understand- ing via unified large-scale audio-language models,

    Y . Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou, “Qwen-audio: Advancing universal audio understand- ing via unified large-scale audio-language models,”arXiv preprint arXiv:2311.07919, 2023

    2023 arXiv

  26. [34]

    Qwen2-audio technical report,

    Y . Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y . Leng, Y . Lv, J. He, J. Linet al., “Qwen2-audio technical report,”arXiv preprint arXiv:2407.10759, 2024

    2024 arXiv

  27. [35]

    Salmonn: Towards generic hearing abilities for large language models,

    C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang, “Salmonn: Towards generic hearing abilities for large language models,” arXiv preprint arXiv:2310.13289, 2023

    2023 arXiv

  28. [36]

    Joint audio and speech understanding,

    Y . Gong, A. H. Liu, H. Luo, L. Karlinsky, and J. Glass, “Joint audio and speech understanding,” in2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2023, pp. 1–8

    2023

  29. [37]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, W. Chen et al. , “Lora: Low-rank adaptation of large language models,” in International Conference on Learning Representa- tions, 2021

    2021

  30. [38]

    Minigpt- 4: Enhancing vision-language understanding with advanced large language models,

    D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “Minigpt- 4: Enhancing vision-language understanding with advanced large language models,” in The Twelfth International Conference on Learning Representations, 2023

    2023

  31. [39]

    Desta: Enhancing speech language mod- els through descriptive speech-text alignment,

    K.-H. Lu et al. , “Desta: Enhancing speech language mod- els through descriptive speech-text alignment,” arXiv preprint arXiv:2406.18871, 2024

    2024 arXiv

  32. [40]

    Speech-copilot: Leveraging large language models for speech processing via task decomposition, modularization, and program generation,

    C.-Y . Kuan, C.-K. Yang, W.-P. Huang, K.-H. Lu, and H.-y. Lee, “Speech-copilot: Leveraging large language models for speech processing via task decomposition, modularization, and program generation,” arXiv preprint arXiv:2407.09886, 2024

    2024 arXiv

  33. [41]

    Dynamic-superb: Towards a dynamic, col- laborative, and comprehensive instruction-tuning benchmark for speech,

    C.-y. Huang et al. , “Dynamic-superb: Towards a dynamic, col- laborative, and comprehensive instruction-tuning benchmark for speech,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 12 136–12 140

    2024

  34. [42]

    Dynamic-superb phase-2: A collaboratively expanding benchmark for measuring the capabil- ities of spoken language models with 180 tasks,

    C.-y. Huang, W.-C. Chen et al. , “Dynamic-superb phase-2: A collaboratively expanding benchmark for measuring the capabil- ities of spoken language models with 180 tasks,” arXiv preprint arXiv:2411.05361, 2024

    2024 arXiv

  35. [43]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Ale- man, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  36. [44]

    Robust speech recognition via large-scale weak supervision,

    A. Radford et al., “Robust speech recognition via large-scale weak supervision,” in International conference on machine learning . PMLR, 2023, pp. 28 492–28 518

    2023

  37. [45]

    Whisper-at: Noise-robust automatic speech recognizers are also strong audio event taggers,

    Y . Gong, S. Khurana, L. Karlinsky, and J. Glass, “Whisper-at: Noise-robust automatic speech recognizers are also strong audio event taggers,” in Proc. Interspeech 2023, 2023

    2023

  38. [46]

    Investigating the emergent audio classification ability of asr foundation models,

    R. Ma, A. Liusie, M. J. Gales, and K. M. Knill, “Investigating the emergent audio classification ability of asr foundation models,” arXiv preprint arXiv:2311.09363, 2023

    2023 arXiv

  39. [47]

    The llama 3 herd of models,

    A. Dubey et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  40. [48]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” in International conference on machine learning. PMLR, 2023, pp. 19 730–19 742

    2023

  41. [49]

    Audio set: An ontology and human-labeled dataset for audio events,

    J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017, pp. 776–780

    2017

  42. [50]

    Audiocaps: Generat- ing captions for audios in the wild,

    C. D. Kim, B. Kim, H. Lee, and G. Kim, “Audiocaps: Generat- ing captions for audios in the wild,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, V ol- ume 1 (Long and Short Pape...

    2019

  43. [51]

    Fsd50k: an open dataset of human-labeled sound events,

    E. Fonseca, X. Favory, J. Pons, F. Font, and X. Serra, “Fsd50k: an open dataset of human-labeled sound events,” IEEE/ACM Trans- actions on Audio, Speech, and Language Processing , vol. 30, pp. 829–852, 2021

    2021

  44. [52]

    Sound event envelope estimation in polyphonic mixtures,

    I. Mart ´ın-Morat´o, A. Mesaros, T. Heittola, T. Virtanen, M. Cobos, and F. J. Ferri, “Sound event envelope estimation in polyphonic mixtures,” in ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2019, pp. 935–939

    2019

  45. [53]

    ESC: Dataset for Environmental Sound Classi- fication,

    K. J. Piczak, “ESC: Dataset for Environmental Sound Classi- fication,” in Proceedings of the 23rd Annual ACM Conference on Multimedia . ACM Press, 2015, pp. 1015–1018. [Online]. Available: http://dl.acm.org/citation.cfm?doid=2733373.2806390

    2015

  46. [54]

    A dataset and taxonomy for urban sound research,

    J. Salamon, C. Jacoby, and J. P. Bello, “A dataset and taxonomy for urban sound research,” in22nd ACM International Conference on Multimedia (ACM-MM’14), Orlando, FL, USA, Nov. 2014, pp. 1041–1044

    2014

  47. [55]

    Clotho- aqa: A crowdsourced dataset for audio question answering,

    S. Lipping, P. Sudarsanam, K. Drossos, and T. Virtanen, “Clotho- aqa: A crowdsourced dataset for audio question answering,” in 2022 30th European Signal Processing Conference (EUSIPCO) . IEEE, 2022, pp. 1140–1144

    2022

  48. [56]

    V ocalsound: A dataset for improv- ing human vocal sounds recognition,

    Y . Gong, J. Yu, and J. Glass, “V ocalsound: A dataset for improv- ing human vocal sounds recognition,” in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 151–155

    2022

  49. [57]

    Gemini 1.5: Un- locking multimodal understanding across millions of tokens of context,

    G. Team, P. Georgiev, V . I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang et al., “Gemini 1.5: Un- locking multimodal understanding across millions of tokens of context,” arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  50. [58]

    What do mllms hear? examining the interaction between llm and audio encoder com- ponents in multimodal large language models,

    E. B. C ¸ oban, M. I. Mandel, and J. Devaney, “What do mllms hear? examining the interaction between llm and audio encoder com- ponents in multimodal large language models,” in Audio Imagi- nation: NeurIPS 2024 Workshop AI-Driven Speech, Music, and Sound Generation, 2024

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.