Pith. sign in

REVIEW 4 major objections 3 minor 2 cited by

FusionAudio-1.2M: Towards Fine-grained Audio Captioning with Multimodal Contextual Fusion

T0 review · 4 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FusionAudio-1.2M claims that fusing speech, music, general-sound, and visual cues into LLM-written captions yields a dataset that improves audio-text retrieval and audio understanding, and that models trained on it outperform those…

desk verdict A genuinely large and useful audio captioning dataset, but the main superiority claim is not actually controlled. read the letter →

arxiv 2506.01111 v1 pith:A52ZRHYO submitted 2025-06-01 cs.SD cs.AIeess.AS

Shunian Chen , Xinyuan Xie , Zheshu Chen , Liyan Zhao , Owen Lee , Zhan Su , Qilin Sun , Benyou Wang This is my paper
classification cs.SDcs.AIeess.AS
keywords audiocaptioningmultimodalfusiondatasetgenerationlargelanguagemodelaudio-textretrievalunderstandingsourceseparationqualityfiltering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that an automated, scalable pipeline can produce audio captions as detailed and reliable as manually annotated ones, and that the scale and quality of the resulting dataset carry over to downstream models. The central claim is that fusing several specialized audio cues—speech content, music attributes, general sound events, and visual context from the accompanying video—and having a large language model resolve conflicts and synthesize a single caption yields captions that are longer, more accurate, and less prone to hallucination than those in existing automated audio-caption datasets. The paper further claims that models pre-trained or fine-tuned on this dataset outperform models trained on prior datasets on audio-text retrieval and on a battery of audio understanding tasks, even when the comparison is restricted to the same number of training QA pairs. If true, this matters because high-quality caption data is a bottleneck for training better audio-language models, and the pipeline offers a path to scale that data without relying on costly human annotation.

What carries the argument

The load-bearing mechanism is a two-stage pipeline. In the first stage, 'Multimodal Contextual Cue Extraction,' the audio is split with a source-separation model into vocal and non-vocal tracks, and then four specialized models each produce a distinct description: Whisper transcribes speech, OpenMu describes music (after YamNet confirms music is present), GAMA describes general sound events, and Qwen2.5-VL produces a timestamped visual description of the video. In the second stage, 'LLM-Driven Contextual Synthesis,' the QwQ-32B LLM receives these four streams plus human-annotated AudioSet tags. The prompt instructs the model to cross-validate the sources, to use video information only for conditional correction of ambiguous audible events, to flag conflicts internally, and to generate a final caption that is strictly about audio, with a separate list of purely auditory ambiguities. A CLAP-based filter, calibrated against human scores on a 300-caption sample, then discards captions whose audio-text similarity falls below a threshold. This entire apparatus is what converts scattered multimodal cues into the fine-grained captions that drive the downstream gains.

What would settle it

Re-run the understanding benchmark normalizing by unique clips instead of QA pairs: take 25,000 unique FusionAudio clips (using one QA pair per clip to match baseline diversity) and fine-tune GAMA on that subset alongside the same model fine-tuned on 25,000 unique clips of AudioCaps, WavCaps, AudioSetCaps and Auto-ACD; if the FusionAudio advantage shrinks or vanishes, the reported gain is an artifact of data diversity rather than caption quality.

Watch

Extended reading notes

Core claim

The central claim, on the paper's own terms, is that multimodal contextual fusion is the key to fine-grained audio captioning at scale. The paper proposes a two-stage automated pipeline that first isolates a vocal track and then runs specialized models to extract separate contextual cues: an ASR model for speech, a music-understanding model for musical attributes, an audio-understanding model for general sound events, and a vision-language model to extract timestamped visual context from the associated video. A large language model then acts as an integration engine, synthesizing these streams into a single, purely auditory caption and using the video information only to disambiguate sounds when the audio alone is ambiguous. The result is FusionAudio-1.2M, a dataset of 1.2 million captions and 6 million question-answer pairs derived from AudioSet clips. The paper claims that this dataset, and especially a high-quality subset selected by a CLAP-based quality filter, gives models the best fine-grained audio understanding, and that the advantage holds even when all training data is normalized to 25,000 QA pairs.

Load-bearing premise

The claim that FusionAudio's captions are intrinsically higher quality rests on comparing models fine-tuned on the same number of 25,000 QA pairs, but for the baselines those pairs come from 25,000 unique clips while FusionAudio's come from only about 9,000 unique clips, so if clip diversity rather than caption quality explains the measured gains, the paper's central superiority claim would not be established.

Editorial extensions

If this is right

  • If the central claim holds, automated audio captioning no longer needs manual annotation for scale: the same pipeline can generate detailed captions for millions of clips at a fraction of the cost.
  • Models pre-trained on FusionAudio-1.2M should achieve higher recall on audio-text retrieval, particularly on fine-grained queries where small acoustic differences matter.
  • Audio-language models fine-tuned on FusionAudio or FusionAudio-high should generalize better to unseen understanding tasks across adverse acoustic conditions, high-level semantics, and fine-grained recognition.
  • Because the dataset generates multiple question-answer pairs from a single 10-second clip, the same number of training pairs can be produced from fewer unique audio clips, making data acquisition more efficient.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's ablation shows that removing the video stream hurts performance most, which suggests the pipeline's gains depend heavily on reliable visual context; for audio-only sources the approach would need a fallback.
  • The 25,000-QA-pair comparison uses only about 9,000 unique clips for FusionAudio versus 25,000 for the baselines, so if clip diversity itself drives some of the gains, the claimed quality advantage could shrink under an equal-diversity comparison.
  • The quality filter relies on a CLAP similarity threshold calibrated on only 300 human-rated captions, so the resulting 7.3% filter rate may not transfer to other domains, audio lengths, or model families.
  • The paper does not test whether the generated captions benefit audio generation or other downstream tasks beyond retrieval and understanding, leaving open the possibility that the captions' style is tuned specifically for those benchmarks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes FusionAudio-1.2M, a large-scale audio captioning dataset produced by a two-stage automated pipeline that first extracts context from general audio, speech, music, and video using specialized pretrained models, then uses an LLM to synthesize these cues into detailed captions. The dataset contains 1.2 million captions and 6 million QA pairs built from AudioSet. The authors report human evaluation on 300 samples, a CLAP-based automatic quality filter, and evaluate the dataset by pre-training an HTSAT-BERT model for audio-text retrieval and fine-tuning GAMA for a range of audio understanding tasks. They claim that models trained on FusionAudio outperform those trained on existing audio captioning datasets, including with fewer unique audio clips, and they provide ablations for modality contributions and data scaling.

Significance. If the claims hold, FusionAudio-1.2M would be a valuable resource for audio-language research: it addresses a real need for fine-grained, large-scale captions, incorporates multimodal context in a scalable way, and includes both captions and QA pairs. The paper's strengths include the scale of the released dataset, the use of human verification to calibrate the automatic filter, the systematic ablations of modality contributions, the data-scaling study, and the commitment to release code and data. However, the central empirical claim of dataset superiority is not yet convincingly established because the main comparisons contain confounds in text encoder architecture and in the number of unique audio clips. The dataset contribution itself is likely useful regardless of the outcome of those controls.

major comments (4)
  1. [§5.2.1, Table 5] The normalization to 25,000 QA pairs is incomplete. Section 5.2.1 says training data are normalized, but FusionAudio supplies these QA pairs from about 9,000 unique audio clips while each baseline uses about 25,000 unique clips. This varies both caption quality and the number/distribution of underlying audio scenes, so the gains in Table 5 cannot be attributed specifically to intrinsic caption quality; more annotations per clip or a quality-selected subset could produce the same result. Section 5.2.2's statement that 'dominant performance was achieved using substantially fewer unique audio clips' is the same confound rather than independent evidence of data efficiency. A matched comparison holding unique clip count fixed is needed.
  2. [§5.1.1, Table 4] Table 4 is not architecture-controlled. Section 5.1.1 states that all experiments use the HTSAT-BERT model, but the Auto-ACD and Sound-VECaps rows in Table 4 use HTSAT+RoBERTa while the other rows use HTSAT+BERT. Since the text encoder differs across baselines, the retrieval comparisons cannot be attributed solely to dataset quality. These two baselines should be re-run with HTSAT+BERT before concluding that FusionAudio captions improve retrieval performance.
  3. [§3.3, Figure 3d] The CLAP-score distribution in Figure 3d is partly circular as evidence of audio-text alignment. Section 3.3 filters the dataset using a CLAP audio-text cosine similarity threshold of 0.08, and Figure 3d then presents the CLAP score distribution of the resulting dataset as evidence of alignment. Filtering on the same metric mechanically shifts the distribution toward higher scores. The alignment claim should be validated with an independent measure or with explicit acknowledgment that the distribution is selection-induced.
  4. [§5.2.2, Limitation] The paper reports no error bars or significance tests, and the Limitation section acknowledges this. Several margins in Table 5 are small relative to what single-run comparisons can support (e.g., FA vs CompA-R on the overall average is 56.7 vs 56.6). Without multiple seeds or significance testing, the claim that FusionAudio-high 'achieving the highest average scores in all scenarios' is not statistically supported. At minimum, variance across seeds should be reported for the main comparisons.
minor comments (3)
  1. [§5.2.2] The text says FusionAudio 'consistently outperformed models trained on all benchmarked datasets across the majority of the 13 tasks,' but Table 5 contains 15 evaluation columns; the task count should be made consistent.
  2. [Table 4] The Auto-ACD row reports R@5 values as dashes and leaves the Average column empty, which makes the comparison table incomplete and harder to interpret.
  3. [Figure 3c] The caption 'Diversity of Object Types' is unclear; the text describes instruments, emotions, and music genres, so the figure label should match the actual content being measured.

Circularity Check

1 steps flagged · score 4.0 of 10

CLAP-based filtering makes the CLAP alignment statistic self-selected, but the core dataset-quality claim rests on independent downstream benchmarks.

  1. fitted input called prediction [Sec. 3.3 (Automatic Filtering) and Sec. 4.1 (Audio-Text Alignment, Fig. 3d)]
    "We leveraged the CLAP model to automatically filter low-quality captions. We computed cosine similarity between CLAP-generated audio and caption embeddings as a quality indicator. ... Audio-Text Alignment: Figure 3d shows the distribution of cosine similarity between audio and text embeddings calculated by CLAP."

    The same CLAP cosine-similarity score is used twice: first as the automatic filter (with a threshold of 0.08 tuned on the same 300-sample human evaluation), and later as evidence of audio-text alignment. Because every caption retained in FusionAudio-1.2M was selected to pass the CLAP similarity threshold, the plotted CLAP-score distribution in Figure 3d is shifted by construction and cannot independently validate alignment. This is a fitted input renamed as an alignment statistic. The circularity is localized: the paper's main superiority claims use independent metrics (HTSAT-BERT retrieval in Table 4 and GAMA understanding benchmarks in Table 5), so the central result does not reduce to the filter.

full rationale

The manuscript's derivation chain is mostly not circular. The dataset is built by an explicit multimodal pipeline (ASR, music captioning, GAMA sound captions, video captions, and LLM synthesis), and the claimed downstream improvements are measured on AudioCaps retrieval and 15 audio-understanding benchmarks with architectures (HTSAT-BERT, GAMA) that are independent of the CLAP filter used for curation. The clearest circularity is in Section 4.1: Figure 3d presents the CLAP audio-text similarity distribution of FusionAudio-1.2M as evidence of alignment, but the dataset was filtered by thresholding exactly this CLAP score, so the distribution is self-selected by construction. The threshold was also tuned on the same 300 human-evaluated samples, strengthening the self-referential character of that statistic. Additional concerns, such as the 25,000-QA-pair comparison in Table 5 using only 9,000 unique clips for FusionAudio versus 25,000 for baselines, and the HTSAT+RoBERTa architecture mismatch for two baselines in Table 4, are experimental confounds rather than definitional circularity: the downstream numbers are not forced by the dataset construction. Because the main performance claims rest on independent evaluations, the paper has substantial non-circular content; the score of 4 reflects one genuine circular statistic without implicating the central empirical finding.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new theoretical entities are postulated. The dataset is an empirical artifact. The main load-bearing choices are the CLAP filtering threshold (a fitted parameter) and several assumptions about the reliability of external models and the video-text-audio correspondence.

free parameters (2)
  • CLAP filtering threshold = 0.08
    Chosen by optimizing F1.05 on 300 human-rated samples to maximize agreement with hallucination labels; applied to filter the entire dataset.
  • F1.05 beta = 1.05
    A metric design choice that slightly weights recall; affects which threshold is selected.
assumptions (4)
  • domain assumption AudioSet clips have associated video streams that provide semantically relevant visual context for the audio.
    The pipeline relies on video as a corrective modality (Section 3.1); if many videos are unrelated to the audio, the visual cues could mislead rather than help.
  • domain assumption Specialized expert models (Whisper, GAMA, OpenMu, Qwen2.5-VL-72B) produce sufficiently accurate outputs to serve as reliable cues.
    Stage 1 assumes these models provide correct speech, audio, music, and visual descriptions; errors propagate to the final captions.
  • domain assumption CLAP cosine similarity between audio and caption is a valid proxy for caption quality and hallucination.
    The automatic filter (Section 3.3) assumes captions with higher CLAP similarity are more accurate; this is validated only on 300 samples.
  • domain assumption The LLM (QwQ-32B) follows the long integration prompt and avoids introducing visual information or speech content into the final audio caption.
    The integration prompt (Appendix B.4) explicitly prohibits visual/speech content, but the paper provides no automatic check that the model complies at scale.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FusionAudio-1.2M: Towards Fine-grained Audio Captioning with Multimodal Contextual Fusion." pith.science (2026). https://pith.science/paper/A52ZRHYO

@misc{pith2026250601111,
  author       = {Pith},
  title        = {Pith review of: FusionAudio-1.2M: Towards Fine-grained Audio Captioning with Multimodal Contextual Fusion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A52ZRHYO}},
  note         = {Machine review of arXiv:2506.01111}
}
read the original abstract

High-quality, large-scale audio captioning is crucial for advancing audio understanding, yet current automated methods often generate captions that lack fine-grained detail and contextual accuracy, primarily due to their reliance on limited unimodal or superficial multimodal information. Drawing inspiration from human auditory perception, which adeptly integrates cross-modal cues and performs sophisticated auditory scene analysis, we introduce a novel two-stage automated pipeline. This pipeline first employs specialized pretrained models to extract diverse contextual cues (e.g., speech, music, general sounds, and visual information from associated video). A large language model (LLM) then synthesizes these rich, multimodal inputs to generate detailed and context-aware audio captions. Key contributions of this work include: (1) the proposed scalable method for fine-grained audio caption generation; (2) FusionAudio, a new large-scale dataset comprising 1.2 million such detailed captions, combined with 6 million QA pairs; and (3) enhanced audio models developed using FusionAudio, specifically a CLAP-based audio encoder with superior audio-text alignment and instruction following. This paper paves the way for more nuanced and accurate automated understanding of complex audio environments. Code and data can be found in https://github.com/satsuki2486441738/FusionAudio.

Figures

Figures reproduced from arXiv: 2506.01111 by the authors.

Figure 1
Figure 1. Human auditory percep￾tion integrates multisensory cues. Inspired by these principles, we introduce a two-stage pipeline for enhanced automated audio captioning. First, specialized pre￾trained models extract diverse contextual cues: an Automatic Speech Recognition (ASR) model [17] for speech, a music under￾standing model [18] for musical attributes, an audio understand￾ing model [2] for general sounds, and a visual … view at source ↗
Figure 2
Figure 2. Overview of our proposed multimodal audio captioning pipeline. The process involves [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Key statistics of FusionAudio-1.2M: (a) Proportion of top 5 audio labels from AudioSet; [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: T-SNE Embedding of popular categories between different datasets [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Scaling result of understanding and retrieval tasks. Details of the legend in (b): [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Instruction for Human Evaluation. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Detailness and Hallucination Rates Distribution of Human Rating [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Prompt for video caption. C.3 Situations where multimodal contextual cues work Our multimodal approach is designed to excel in challenging audio understanding scenarios ( [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: An example prompt for audio caption generation [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: An example prompt for extracting objects from audio. [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Concise prompt for modal info check Examples for audios with different clap scores. Here we show the severity of hallucinations in audio captions under different clap similarity intervals. The red - marked parts are the hallucinatory parts of the audio captions [PITH…
Figure 12
Figure 12. Figure 12: Examples for audios with different clap scores. [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: An example prompt for multi-choice questions. [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: Prompt for integration. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_14.png]
Figure 15
Figure 15. Figure 15: Prompt for integration Cont. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_15.png]
Figure 16
Figure 16. Figure 16: Prompt for integration Cont. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_16.png]
Figure 17
Figure 17. Figure 17: Prompt for integration Cont. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_17.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EvA: An Evidence-First Audio Understanding Paradigm for LALMs

    cs.SD 2026-03 conditional novelty 6.0 of 10

    Preserving multi-scale non-speech evidence via hierarchical aggregation and non-compressive time-aligned fusion measurably lifts LALM perception more than reasoning.

  2. Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation

    eess.AS 2025-11 conditional novelty 6.0 of 10

    A 10.7M-pair audio-caption corpus and systematic comparison show contrastive pretraining is more data-efficient while captioning scales better, and supervised initialization yields diminishing returns.

Reference graph

Works this paper leans on

48 extracted references · 35 canonical work pages · cited by 2 Pith papers

  1. [1]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword- to-caption augmentation

    Yusong Wu*, Ke Chen*, Tianyu Zhang*, Yuchen Hui*, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword- to-caption augmentation. InIEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, 2023

    2023

  2. [2]

    GAMA: A large audio-language model with advanced audio understanding and complex reasoning abilities

    Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. GAMA: A large audio-language model with advanced audio understanding and complex reasoning abilities. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conference on Empirical M...

    work page 2024

  3. [3]

    Qwen2-audio technical report

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759, 2024

    arXiv 2024

  4. [4]

    Clotho: An audio captioning dataset, 2019

    Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset, 2019

    work page 2019

  5. [5]

    AudioCaps: Gen- erating captions for audios in the wild

    Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. AudioCaps: Gen- erating captions for audios in the wild. In Jill Burstein, Christy Doran, and Thamar Solorio, editors,Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, V olume 1 (Long and Short Pape...

    work page 2019

  6. [6]

    Laion-audio-630k dataset, 2023

    LAION-AI. Laion-audio-630k dataset, 2023. Accessed: 2024-04-16

    work page 2023

  7. [7]

    Plumbley, Yuexian Zou, and Wenwu Wang

    Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D. Plumbley, Yuexian Zou, and Wenwu Wang. WavCaps: A ChatGPT-assisted weakly-labelled audio captioning dataset for audio-language multimodal research.IEEE/ACM Transactions on Audio, Speech, and Language Processing, pages 1–15, 2024

    2024

  8. [8]

    Plumbley, Woon- Seng Gan, and Jianfeng Chen

    Jisheng Bai, Haohe Liu, Mou Wang, Dongyuan Shi, Wenwu Wang, Mark D. Plumbley, Woon- Seng Gan, and Jianfeng Chen. Audiosetcaps: An enriched audio-caption dataset using automated generation pipeline with large audio and language models, 2024

    work page 2024

Show all 48 references
  1. [9]

    Auto-acd: A large-scale dataset for audio-language representation learning, 2024

    Luoyi Sun, Xuenan Xu, Mengyue Wu, and Weidi Xie. Auto-acd: A large-scale dataset for audio-language representation learning, 2024

    2024

  2. [10]

    Plumbley, and Wenwu Wang

    Yi Yuan, Dongya Jia, Xiaobin Zhuang, Yuanzhe Chen, Zhengxi Liu, Zhuo Chen, Yuping Wang, Yuxuan Wang, Xubo Liu, Xiyuan Kang, Mark D. Plumbley, and Wenwu Wang. Sound-vecaps: Improving audio generation with visual enhanced captions, 2025

    2025

  3. [11]

    Air-bench: Benchmarking large audio-language models via generative comprehension, 2024

    Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, and Jingren Zhou. Air-bench: Benchmarking large audio-language models via generative comprehension, 2024

    2024

  4. [12]

    W. H. Sumby and I. Pollack. Visual contribution to speech intelligibility in noise.Journal of the Acoustical Society of America, 26:212–215, 1954

    1954

  5. [13]

    Logothetis, and Stefano Panzeri

    Christoph Kayser, Nikos K. Logothetis, and Stefano Panzeri. Visual enhancement of the information representation in auditory cortex.Current Biology, 20(1):19–24, 2010

    2010

  6. [14]

    Ernst and Heinrich H

    Marc O. Ernst and Heinrich H. Bülthoff. Merging the senses into a robust percept.Trends in Cognitive Sciences, 8(4):162–169, 2004

    2004

  7. [15]

    Bregman.Auditory scene analysis: The perceptual organization of sound

    Albert S. Bregman.Auditory scene analysis: The perceptual organization of sound. The MIT Press, 1990

    1990

  8. [16]

    S. A. Shamma, M. Elhilali, and C. Micheyl. Temporal coherence and attention in auditory scene analysis.Trends in Neurosciences, 34(3):114–123, Mar 2011. Epub 2010 Dec 31. 11

    2011

  9. [17]

    Robust speech recognition via large-scale weak supervision, 2022

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision, 2022

    2022

  10. [18]

    Openmu: Your swiss army knife for music understanding

    Mengjie Zhao, Zhi Zhong, Zhuoyuan Mao, Shiqi Yang, Wei-Hsiang Liao, Shusuke Takahashi, Hiromi Wakaki, and Yuki Mitsufuji. Openmu: Your swiss army knife for music understanding. arXiv preprint arXiv:2410.15573, 2024

    2024 arXiv

  11. [19]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Han...

    2025 arXiv

  12. [20]

    Qwq-32b: Embracing the power of reinforcement learning, March 2025

    Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025

    2025

  13. [21]

    Elizalde, S

    B. Elizalde, S. Deshmukh, M. Al Ismail, and H. Wang. Clap learning audio concepts from natural language supervision. InIEEE International Conference on Acoustics, Speech and Signal Processing, pages 1–5. IEEE, 2023

    2023

  14. [22]

    H. Li, Y . Kang, T. Liu, W. Ding, and Z. Liu. Ctal: Pre-training crossmodal transformer for audio-and-language representations.arXiv preprint arXiv:2109.00181, 2021

    2021 arXiv

  15. [23]

    Yeh, P.-Y

    C.-F. Yeh, P.-Y . Huang, V . Sharma, S.-W. Li, and G. Gosh. Flap: Fast language-audio pre- training. In2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1–8. IEEE, 2023

    2023

  16. [24]

    J. Tian, H. Dongchao Yang, et al. Uniaudio: An audio foundation model toward universal audio generation.arXiv preprint arXiv:2310.00704, 2023

    2023 arXiv

  17. [25]

    X. Wang, M. Thakker, Z. Chen, N. Kanda, S. Eskimez, M. Chen, S. Tang, J. Liu, T. Li, and T. Yoshioka. Speechx: Neural codec language model as a versatile speech transformer. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024

    2024

  18. [26]

    Pengi: An audio language model for audio tasks

    Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. Pengi: An audio language model for audio tasks. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems, volume 36, pages 18090–18108. Cur...

    2023

  19. [27]

    Y . Chu, J. Xu, X. Zhou, S. Yang, Z. Zhang, C. Yan, and J. Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models.arXiv preprint arXiv:2311.07919, 2023

    2023 arXiv

  20. [28]

    Z. Kong, A. Goel, R. Badlani, W. Ping, R. Valle, and B. Catanzaro. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities.arXiv preprint arXiv:2402.01831, 2024

    2024 arXiv

  21. [29]

    Hybrid transformers for music source separation

    Simon Rouard, Francisco Massa, and Alexandre Défossez. Hybrid transformers for music source separation. InICASSP 23, 2023

    2023

  22. [30]

    Yamnet: Audio event classification

    TensorFlow. Yamnet: Audio event classification. https://github.com/tensorflow/ models/tree/master/research/audioset/yamnet, n.d. Accessed: 2025-04-19

    2025

  23. [31]

    Gemmeke, Daniel P

    Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Chan- ning Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In2017 IEEE International Conference on Acoustics, Speech and Signal Proces...

    2017

  24. [32]

    Visualizing data using t-sne.Journal of Machine Learning Research, 9(86):2579–2605, 2008

    Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne.Journal of Machine Learning Research, 9(86):2579–2605, 2008

    2008

  25. [33]

    Hts- at: A hierarchical token-semantic audio transformer for sound classification and detection

    Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Hts- at: A hierarchical token-semantic audio transformer for sound classification and detection. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing ...

    2022

  26. [34]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human langua...

    2019

  27. [36]

    • If the intensity or emotional context of the sound is conveyed (e.g., the dog barking intensely or the doorbell ringing in a rapid succession)

    Detailing Key Things to Look For: • Whether the caption captures all major sounds and events in the audio (e.g., dog barking, doorbell ringing, etc.). • If the intensity or emotional context of the sound is conveyed (e.g., the dog barking intensely or the doorbell ringing in a...

  28. [37]

    trumpets,

    Hallucinations You will be given the highlighted words or phrases marked by DeepSeek-V3 that need to be verified in the original caption: 2 A [male voice] delivers a [scripted narration] [in Polish], likely from a [recorded radio or podcast segment], accompanied by [subtle stu...

  29. [38]

    Extract instruments and return as a list

  30. [39]

    Extract emotions and return as a list

  31. [40]

    Extract music genres and return as a list

  32. [41]

    Extract scenes and return as a list

  33. [42]

    All words must be found in the sentence

  34. [43]

    Return a JSON format without any other words

  35. [44]

    instrument

    Words must be extracted from the corresponding caption. The return format should only be like this: { "instrument": [], "emotion": [], "music genre": [], "scene": [] } Figure 10: An example prompt for extracting objects from audio. D More on Dataset Statistics D.1 Embedding Sp...

  36. [45]

    List contributing caption types

  37. [46]

    Return as string keys list

  38. [47]

    A car’s engine roars as it accelerates

    Format: [’type1’, ’type2’]" Figure 11: Concise prompt for modal info check Examples for audios with different clap scores. Here we show the severity of hallucinations in audio captions under different clap similarity intervals. The red - marked parts are the hallucinatory part...

  39. [48]

    Audio Description

    •Sources of Ambiguity: – Auditory Similarity or Vagueness of the Sound Itself: Some sounds may be auditorily similar to others and easily confused (e.g., vehicle sound vs. airplane sound, typing sound vs. light tapping sound). The sound’s own quality, distance, or reverberatio...

  40. [2024]

    Association for Computational Linguistics

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.