Pith. sign in

REVIEW 5 major objections 6 minor 66 references

AudioGenie: A Training-Free Multi-Agent Framework for Diverse Multimodality-to-Multiaudio Generation

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AudioGenie claims that multimodality-to-multiaudio generation can be solved without training by coordinating off-the-shelf models in a plan-and-verify multi-agent loop.

desk verdict A genuinely new task framing and a coherent training-free multi-agent system, but the headline SOTA claim is inflated by an asymmetric comparison and missing releases. read the letter →

arxiv 2505.22053 v2 pith:4DZXKYCD submitted 2025-05-28 cs.SD cs.MAcs.MMeess.AS

Yan Rong , Jinting Wang , Guangzhi Lei , Shan Yang , Li Liu This is my paper
classification cs.SDcs.MAcs.MMeess.AS
keywords multimodality-to-multiaudiogenerationmulti-agentsystemtraining-freeframeworkself-correctiontree-of-thoughtvideo-to-audioaudiobenchmarkmixture-of-experts
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AudioGenie is a training-free multi-agent system that claims to generate multiple audio types—sound effects, speech, music, and songs—from video, text, image, or combined inputs in a single unified pipeline. Instead of training one model, it coordinates a generation team and a supervisor team: the generation team decomposes a scene into timestamped audio sub-events, assigns each event to a specialist model, and mixes the results, while the supervisor team verifies every plan and output and drives retries. To measure this, the paper introduces MA-Bench, a benchmark of 198 annotated videos in which each video contains several audio events with detailed timestamps and descriptions. Across eight generation tasks and nine metrics, the paper reports state-of-the-art or comparable results, including higher human ratings on accuracy and alignment than single-pass baselines. The reason this matters is that it suggests complex audio production can be assembled from off-the-shelf components rather than custom-trained systems.

What carries the argument

The load-bearing mechanism is the 'plan-and-verify' loop built around three modules. Fine-grained task decomposition turns the input into a structured JSON list of sub-audio events, which is what makes multiple simultaneous events visible and separately generatable. The adaptive Mixture-of-Experts collaborative entity gives each audio type a dedicated expert that selects two candidate models from a tool library and refines plans through intra-expert self-reflection and inter-expert collaborative refinement. The trial-and-error iterative refinement module organizes each event's generation as a tree-of-thought with initial, generation, and refinement nodes, using the supervisor team's verdicts to decide between targeted post-processing and full retries. The supervisor team is the arbiter that connects all stages, evaluating plans for timing and content and evaluating audio for quality, alignment, and aesthetics.

What would settle it

Run MA-Bench twice through AudioGenie, once with the real supervisor team and once with random or inverted supervisor verdicts while keeping the same tree-of-thought search, and compare the mean opinion scores. If random feedback matches the real supervisor, the reported gains come from retrying rather than from evaluation quality; if real feedback clearly wins, the supervisor's judgment is the load-bearing component.

Watch

Extended reading notes

Core claim

The paper's central claim is that multimodality-to-multiaudio generation does not require a single unified trained model. AudioGenie is a dual-layer system: the generation team runs three stages—fine-grained task decomposition, adaptive Mixture-of-Experts selection, and trial-and-error iterative refinement—while the supervisor team checks each stage by acting as an audio planning expert, a domain expert, and a professional audio evaluator. Each audio event is planned as a JSON entry with type, object, timestamps, description, and volume; the expert layer picks two candidate generators from a tool library, refines the event plan, and generates the event; the supervisor scores output on quality, alignment, and aesthetics. If the output is fixable, refinement nodes post-process it; if it fails core requirements, sibling generation nodes retry with adjusted prompts or another model. On MA-Bench, the authors report state-of-the-art or comparable performance on 9 metrics in 8 tasks, and ablations show that removing the supervisor team and the tree-of-thought self-correction causes the largest drops.

Load-bearing premise

The self-correction loop assumes the LLM-based supervisor team can reliably judge generated audio on quality, alignment, and aesthetics, and that each retry or refinement moves the output closer to the supervisor's verdict.

Editorial extensions

If this is right

  • Complex videos with several simultaneous sounds should yield complete, mixed audio tracks instead of a single dominant event, because each sub-event is generated and mixed separately.
  • New input modalities or audio types can be added without retraining by placing a new model in the tool library and giving a domain expert its prior knowledge.
  • Unreliable single-pass outputs can be improved by spending more compute: the tree-of-thought loop retries failed events with adjusted prompts or alternative models.
  • The MA-Bench dataset of 198 annotated multi-audio videos provides a common test bed for future multimodality-to-multiaudio systems, since no comparable benchmark existed before.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the supervisor's verdicts are the real engine of quality, then swapping in a stronger audio-language evaluator or a learned audio-quality metric should lift results further; this is directly testable.
  • The architecture suggests a scaling pattern for tool libraries: adding more candidate models per audio type should improve the best-branch outcome at the cost of more trial-and-error calls, which could be measured as a compute-versus-quality curve.
  • A natural follow-up the paper does not explore is distilling the accepted branches into a single trained model, which would trade the inference-time search for a cheaper one-pass generator with similar event coverage.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This paper proposes AudioGenie, a training-free multi-agent framework for multimodal-to-multiaudio (MM2MA) generation, i.e., producing sound effects, speech, music, and songs from video, text, image, or combinations of these inputs. The system consists of a generation team that decomposes the input into audio events, selects domain-specific experts and models from a tool library, and applies a tree-of-thought trial-and-error refinement loop, plus a supervisor team that evaluates plans and generated audio and provides feedback. The authors also introduce MA-Bench, a benchmark of 198 curated videos with multi-type audio annotations and comprehensive event-level annotations. Experiments on MA-Bench report state-of-the-art or comparable performance across nine metrics in eight tasks, with ablations and additional evaluations on VGGSound, V2M, and with different MLLMs in the appendices.

Significance. The proposed system is relevant to the multimedia community and, if the results are validated, would be a useful demonstration that a training-free orchestration of existing audio generators can handle multi-type audio generation in a unified framework. The paper has several strengths: it is training-free and builds on existing models; it includes a new benchmark that could facilitate future research; it provides ablations for each major component; and it reports generalization experiments on public datasets and with different MLLMs, which partially address the 'self-built benchmark' concern. However, the central SOTA claim currently rests on an asymmetric comparison in Table 3, on objective metrics that the authors themselves identify as not yet established for MM2MA (Section 5), and on a subjective study that lacks statistical rigor. These issues need to be addressed before the claimed superiority can be accepted.

major comments (5)
  1. [Section 4.4.1, Table 3] The comparison in Table 3 is structurally unequal. AudioGenie is allowed to produce a mixed output with multiple audio events of different types, while each baseline produces a single audio category (AudioX produces two). Since MOS-Accuracy explicitly asks whether all audio events in the input are produced, and AES Production Complexity rewards richness and complexity, the large gains on these metrics for AudioGenie (e.g., MOS-Acc 3.54 vs 2.90 for video input, PC 3.99 vs 3.02 for AudioX) may reflect the greater number of output tracks rather than the quality of the proposed decomposition, MoE selection, or self-correction mechanisms. To support the SOTA claim, the authors should report an additional comparison in which the baselines are given the same decomposed event list or in which the evaluation is done per audio event, so that track count is not a confounding factor.
  2. [Table 4, Section 4.4.2] In the SISO setting, which is more comparable to the baselines, the advantages of AudioGenie are smaller and in some cases negative. For audio effect generation, AudioGenie is below AudioX on PAM (0.750 vs 0.786) and CU (6.043 vs 6.209); for song generation, it is below DiffRhythm on PQ (8.212 vs 8.328). The paper describes these results as 'SOTA or comparable', but these cases below SOTA should be discussed explicitly in the text, and the abstract's universal 9-metric superiority claim should be qualified accordingly.
  3. [Section 5] The paper states that establishing automatic MM2MA evaluation metrics is future work, which is an explicit admission that the objective metrics used in Tables 3 and 4 (PAM and AES) are not established for the MM2MA setting. Since these metrics are the quantitative basis of the SOTA claim, the authors should validate them for this new setting (e.g., by reporting correlation with human judgments on MA-Bench) or explicitly label the results as preliminary. Without such validation, the claimed SOTA on 9 objective/subjective metrics is not adequately supported.
  4. [Section 4.3.2] The subjective MOS protocol is under-specified. The paper reports mean scores and standard deviations from 'at least 20 participants', but does not state whether raters were blind to the system identity, how many ratings were obtained per audio clip, whether raters rated independently, or any inter-rater agreement statistic (e.g., ICC or Krippendorff's alpha). In addition, no significance tests are reported, so differences such as the 3.54 vs 2.90 MOS-Accuracy gap cannot be assessed for reliability. The authors should provide these details and apply appropriate statistical tests to support the subjective claims.
  5. [Reproducibility] The central evaluation is conducted on the authors' self-built MA-Bench, but the benchmark annotations, the implementation code, and the exact agent prompts are not released. The paper only mentions a project website with audio samples. Since the framework is training-free and composed of existing models, releasing the prompts and the benchmark would allow the results to be independently reproduced and would substantially strengthen the SOTA claim. I recommend making these resources available.
minor comments (6)
  1. [Section 3.2] There is a typo in 'Each aduio event' which should be 'Each audio event'.
  2. [Abstract] The abstract mentions '9 metrics in 8 tasks', but the paper does not explicitly enumerate these 8 tasks; Table 3 has four input configurations and Table 4 has four generation tasks. Please clarify what the 8 tasks are.
  3. [Section 3.3] The term 'adaptive Mixture-of-Experts' is used, but the implementation is a discrete model selection from a fixed tool library based on expert knowledge. Since 'MoE' normally implies a learned or trainable routing, a brief clarification or a different term would avoid confusion.
  4. [Table 4] In the Song Generation row, AudioGenie is below DiffRhythm on PQ; the paper should add a sentence explaining this specific case, as it is the only objective metric where AudioGenie is not best among the SOTA baselines.
  5. [Section 4.5] The ablations in Table 5 report differences (e.g., PAM 0.966 vs 0.888 for w/o supervisor team) without significance tests; given the small size of MA-Bench, confidence intervals or at least per-sample variability should be reported.
  6. [Conclusion] The conclusion says 'SOTA or comparable performance across several metrics' while the abstract claims '9 metrics in 8 tasks'. These statements should be aligned.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the system is training-free, no fitted parameter is renamed as a prediction, the central SOTA claims are checked on external metrics and partly on public datasets, and the paper's few self-citations are only related-work positioning.

full rationale

AudioGenie contains no mathematical derivation chain whose output could equal its input by construction; the paper's claims are empirical evaluations of an agent pipeline that composes existing generation models (MMAudio, Auffusion, CosyVoice 2, MusicGen, DiffRhythm, etc.). The system is explicitly training-free, so no parameter is fitted to MA-Bench or to any test set and then renamed as a prediction; the 'fitted input called prediction' pattern does not apply. The evaluation uses external metrics (PAM, AES) applied to outputs plus a MOS user study; the only author-built evaluation artifact is MA-Bench itself. Building one's own benchmark is a self-selection risk, but it is not circularity: the benchmark is not used to fit the model, results are also reported on public VGGSound and V2M (Appendix B) and with three different backbone MLLMs (Appendix C), and the ablation study (Table 5) shows the modules contribute on the paper's own metrics rather than being decorative. The paper also concedes in Section 5 that 'Establishing automatic MM2MA evaluation metrics' is future work, an honest limitation on metric maturity rather than a circular derivation. The self-citations (refs [41], [42] as speech examples, [49] as a music example, [60] as a prior single-event agent) are related-work positioning and do not carry the load of the central claim; none is invoked as a uniqueness theorem or as justification for an ansatz. The Table 3 comparison is structurally unequal because AudioGenie emits multiple mixed tracks while most baselines emit one audio category, and the MOS-Accuracy item explicitly asks whether all input events were produced; that is an experimental-fairness concern about what the comparison proves, not a reduction of the prediction to its inputs, and under the controlled SISO setting (Table 4) the margins shrink accordingly. No claimed result was found that is equivalent to its input by definition.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

No new physical entities are introduced. The only ad hoc components are hand-crafted prompts and early-stopping hyperparameters.

free parameters (1)
  • tree depth and retry limits = not specified
    Early stopping limits are chosen by hand to prevent infinite loops, as described in Section 3.4.
assumptions (3)
  • domain assumption LLMs and audio language models can reliably evaluate generated audio for quality, alignment, and aesthetics, and provide useful corrective feedback.
    The entire self-correction loop in Section 3.4 depends on the supervisor's assessments being accurate enough to guide improvements.
  • domain assumption Existing single-type audio generation models can be combined with refined prompts to produce high-quality individual events without retraining.
    The tool library (Table 2) is used as-is, and the framework assumes their outputs can be fixed or improved via prompt or model switching.
  • domain assumption The MLLM (Qwen2.5-VL-72B) can accurately parse multimodal input into structured audio events with correct timestamps and descriptions.
    Stage 1 (Section 3.2) relies on the LLM's video and image understanding to generate the JSON event plan.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AudioGenie: A Training-Free Multi-Agent Framework for Diverse Multimodality-to-Multiaudio Generation." pith.science (2026). https://pith.science/paper/4DZXKYCD

@misc{pith2026250522053,
  author       = {Pith},
  title        = {Pith review of: AudioGenie: A Training-Free Multi-Agent Framework for Diverse Multimodality-to-Multiaudio Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4DZXKYCD}},
  note         = {Machine review of arXiv:2505.22053}
}
read the original abstract

Multimodality-to-Multiaudio (MM2MA) generation faces significant challenges in synthesizing diverse and contextually aligned audio types (e.g., sound effects, speech, music, and songs) from multimodal inputs (e.g., video, text, images), owing to the scarcity of high-quality paired datasets and the lack of robust multi-task learning frameworks. Recently, multi-agent system shows great potential in tackling the above issues. However, directly applying it to MM2MA task presents three critical challenges: (1) inadequate fine-grained understanding of multimodal inputs (especially for video), (2) the inability of single models to handle diverse audio events, and (3) the absence of self-correction mechanisms for reliable outputs. To this end, we propose AudioGenie, a novel training-free multi-agent system featuring a dual-layer architecture with a generation team and a supervisor team. For the generation team, a fine-grained task decomposition and an adaptive Mixture-of-Experts (MoE) collaborative entity are designed for detailed comprehensive multimodal understanding and dynamic model selection, and a trial-and-error iterative refinement module is designed for self-correction. The supervisor team ensures temporal-spatial consistency and verifies outputs through feedback loops. Moreover, we build MA-Bench, the first benchmark for MM2MA tasks, comprising 198 annotated videos with multi-type audios. Experiments demonstrate that our AudioGenie achieves state-of-the-art (SOTA) or comparable performance across 9 metrics in 8 tasks. User study further validates the effectiveness of our method in terms of quality, accuracy, alignment, and aesthetic. The project website with audio samples can be found at https://audiogenie.github.io/.

Figures

Figures reproduced from arXiv: 2505.22053 by the authors.

Figure 1
Figure 1. Overview of the proposed AudioGenie, which supports various input and multiple types of audio outputs. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Statistics of video categories within our MABench. (a) provides statistical distributions of video categories and [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. A visualization example to demonstrate multi-event sound effect generation. The inputs include both video and [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

66 extracted references · 36 canonical work pages

  1. [1]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al . 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 (2025)

    arXiv 2025

  2. [2]

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. 2024. Video generation models as world simulators. OpenAI Blog 1, 8 (2024), 1

    work page 2024

  3. [3]

    Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Börje Karlsson, Jie Fu, and Yemin Shi. 2024. AutoAgents: a framework for automatic agent generation. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence. 22–30

    work page 2024

  4. [4]

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. 2020. Vg- gsound: A large-scale audio-visual dataset. In IEEE International Conference on Acoustics, Speech and Signal Processing . 721–725

    work page 2020

  5. [5]

    Ke Chen, Yusong Wu, Haohe Liu, Marianna Nezhurina, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2024. Musicldm: Enhancing novelty in text-to-music gener- ation using beat-synchronous mixup strategies. In IEEE International Conference on Acoustics, Speech and Signal Processing . 1206–1210

    work page 2024

  6. [6]

    Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, and Xie Chen. 2024. F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching. arXiv preprint arXiv:2410.06885 (2024)

    arXiv 2024

  7. [7]

    Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, and Yuki Mitsufuji. 2024. Taming multimodal joint training for high- quality video-to-audio synthesis. arXiv preprint arXiv:2412.15322 (2024)

    arXiv 2024

  8. [8]

    Jeongsoo Choi, Ji-Hoon Kim, Jinyu Li, Joon Son Chung, and Shujie Liu. 2025. V2SFlow: Video-to-Speech Generation with Speech Decomposition and Rectified Flow. In IEEE International Conference on Acoustics, Speech and Signal Processing . 1–5

    work page 2025

Show all 66 references
  1. [9]

    Marco Comunità, Riccardo F Gramaccioni, Emilian Postolache, Emanuele Rodolà, Danilo Comminiello, and Joshua D Reiss. 2024. Syncfusion: Multimodal Onset- Synchronized Video-to-Audio Foley Synthesis. In IEEE International Conference on Acoustics, Speech and Signal Processing . 936–940

    2024

  2. [10]

    Gaoxiang Cong, Yuankai Qi, Liang Li, Amin Beheshti, Zhedong Zhang, Anton Hengel, Ming-Hsuan Yang, Chenggang Yan, and Qingming Huang. 2024. Style- Dubber: Towards Multi-Scale Style Learning for Movie Dubbing. In Findings of the Association for Computational Linguistics . 6767–6779

    2024

  3. [11]

    Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. 2023. Simple and controllable music generation. Advances in Neural Information Processing Systems 36 (2023), 47704–47720

    2023

  4. [12]

    Rishit Dagli, Shivesh Prakash, Robert Wu, and Houman Khosravani. 2024. SEE- 2-SOUND: Zero-Shot Spatial Environment-to-Spatial Sound. In ICML Workshop on Foundation Models in the Wild

    2024

  5. [13]

    Google DeepMind. 2025. Gemini 2.5 Pro. https://cloud.google.com/vertex- ai/generative-ai/docs/models/gemini/2-5-pro

    2025

  6. [14]

    Soham Deshmukh, Dareen Alharthi, Benjamin Elizalde, Hannes Gamper, Mah- moud Al Ismail, Rita Singh, Bhiksha Raj, and Huaming Wang. 2024. PAM: Prompt- ing Audio-Language Models for Audio Quality Assessment. In Proc. Interspeech. 3320–3324

    2024

  7. [15]

    Shuangrui Ding, Zihan Liu, Xiaoyi Dong, Pan Zhang, Rui Qian, Conghui He, Dahua Lin, and Jiaqi Wang. 2024. Songcomposer: A large language model for lyric and melody composition in song generation. arXiv preprint arXiv:2402.17645 (2024)

    2024 arXiv

  8. [16]

    Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, et al . 2024. Cosyvoice 2: Scal- able streaming speech synthesis with large language models. arXiv preprint arXiv:2412.10117 (2024)

    2024 arXiv

  9. [17]

    Johnny Friberg and Dan Gärdenfors. 2004. Audio games: new perspectives on game audio. In Proceedings of the ACM SIGCHI International Conference on Advances in computer entertainment technology . 148–154

    2004

  10. [18]

    Hao-Han Guo, Kun Liu, Fei-Yu Shen, Yi-Chen Wu, Feng-Long Xie, Kun Xie, and Kai-Tuo Xu. 2024. Fireredtts: A foundation text-to-speech framework for industry-level generative speech applications. arXiv preprint arXiv:2409.03283 (2024)

    2024 arXiv

  11. [19]

    Wei Guo, Heng Wang, Jianbo Ma, and Weidong Cai. 2024. Gotta hear them all: Sound source aware vision to audio generation. arXiv preprint arXiv:2411.15447 (2024)

    2024 arXiv

  12. [20]

    Akshita Gupta, Tatiana Likhomanenko, Karren Dai Yang, Richard He Bai, Zakaria Aldeneh, and Navdeep Jaitly. 2024. Visatronic: A Multimodal Decoder-Only Model for Speech Synthesis. arXiv preprint arXiv:2411.17690 (2024)

    2024 arXiv

  13. [21]

    Zhiqing Hong, Rongjie Huang, Xize Cheng, Yongqi Wang, Ruiqi Li, Fuming You, Zhou Zhao, and Zhimeng Zhang. 2024. Text-to-Song: Towards Controllable Music Generation Incorporating Vocals and Accompaniment. arXiv preprint arXiv:2404.09313 (2024)

    2024 arXiv

  14. [22]

    Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. 2023. Make-an-audio: Text- to-audio generation with prompt-enhanced diffusion models. In International Conference on Machine Learning . 13916–13932

    2023

  15. [23]

    Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, et al. 2024. Audiogpt: Understanding and generating speech, music, sound, and talking head. In Pro- ceedings of the AAAI Conference on Artificia...

    2024

  16. [24]

    Ji-Hoon Kim, Jeongsoo Choi, Jaehun Kim, Chaeyoung Jung, and Joon Son Chung

  17. [25]

    Songju Lei, Xize Cheng, Mengjiao Lyu, Jianqiao Hu, Jintao Tan, Runlin Liu, Lingyu Xiong, Tao Jin, Xiandong Li, and Zhou Zhao. 2024. Uni-Dubbing: Zero- Shot Speech Synthesis from Visual Articulation. InProceedings of the 62nd Annual Meeting of the Association for Computational ...

    2024

  18. [26]

    Shun Lei, Yixuan Zhou, Boshi Tang, Max WY Lam, Hangyu Liu, Jingcheng Wu, Shiyin Kang, Zhiyong Wu, Helen Meng, et al. 2024. Songcreator: Lyrics-based universal song generation. Advances in Neural Information Processing Systems 37 (2024), 80107–80140

    2024

  19. [27]

    Ruiqi Li, Siqi Zheng, Xize Cheng, Ziang Zhang, Shengpeng Ji, and Zhou Zhao

  20. [28]

    Yunxin Li, Haoyuan Shi, Baotian Hu, Longyue Wang, Jiashun Zhu, Jinyi Xu, Zhen Zhao, and Min Zhang. 2024. Anim-director: A large multimodal model powered agent for controllable animation video generation. In SIGGRAPH Asia 2024 Conference Papers. 1–11

    2024

  21. [29]

    Jinhua Liang, Huan Zhang, Haohe Liu, Yin Cao, Qiuqiang Kong, Xubo Liu, Wenwu Wang, Mark D Plumbley, Huy Phan, and Emmanouil Benetos. 2024. Wavcraft: Audio editing and generation with natural language prompts. ICLR Workshop on LLM Agents

    2024

  22. [30]

    Haohe Liu, Ke Chen, Qiao Tian, Wenwu Wang, and Mark D Plumbley. 2024. Au- dioSR: Versatile audio super-resolution at scale. In IEEE International Conference on Acoustics, Speech and Signal Processing . 1076–1080

    2024

  23. [31]

    Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. 2023. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.12503 (2023)

    2023 arXiv

  24. [32]

    Shansong Liu, Atin Sakkeer Hussain, Chenshuo Sun, and Ying Shan. 2024. Music understanding llama: Advancing text-to-music generation with question answer- ing and captioning. In IEEE International Conference on Acoustics, Speech and Signal Processing. 286–290

    2024

  25. [33]

    Xubo Liu, Qiuqiang Kong, Yan Zhao, Haohe Liu, Yi Yuan, Yuzhuo Liu, Rui Xia, Yuxuan Wang, Mark D Plumbley, and Wenwu Wang. 2024. Separate anything you describe. IEEE/ACM Transactions on Audio, Speech, and Language Processing (2024)

    2024

  26. [34]

    Zihan Liu, Shuangrui Ding, Zhixiong Zhang, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. 2025. SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation. arXiv preprint arXiv:2502.13128 (2025)

    2025 arXiv

  27. [35]

    Navonil Majumder, Chia-Yu Hung, Deepanway Ghosal, Wei-Ning Hsu, Rada Mihalcea, and Soujanya Poria. 2024. Tango 2: Aligning diffusion-based text-to- audio generations through direct preference optimization. In Proceedings of the 32nd ACM International Conference on Multimedia . 564–572

    2024

  28. [36]

    Jan Melechovsky, Zixun Guo, Deepanway Ghosal, Navonil Majumder, Dorien Herremans, and Soujanya Poria. 2023. Mustango: Toward controllable text-to- music generation. arXiv preprint arXiv:2311.08355 (2023)

    2023 arXiv

  29. [37]

    Ziqian Ning, Huakang Chen, Yuepeng Jiang, Chunbo Hao, Guobin Ma, Shuai Wang, Jixun Yao, and Lei Xie. 2025. DiffRhythm: Blazingly Fast and Embarrass- ingly Simple End-to-End Full-Length Song Generation with Latent Diffusion. arXiv preprint arXiv:2503.01183 (2025)

    2025 arXiv

  30. [38]

    OpenAI. 2024. GPT-4o. https://openai.com/index/hello-gpt-4o

    2024

  31. [39]

    Chen Qian and Xin Cong. 2023. Communicative agents for software development. arXiv preprint arXiv:2307.07924 6, 3 (2023)

    2023 arXiv

  32. [40]

    Yong Ren, Chenxing Li, Manjie Xu, Wei Liang, Yu Gu, Rilin Chen, and Dong Yu

  33. [41]

    Yan Rong and Li Liu. 2025. Seeing your speech style: A novel zero-shot identity- disentanglement face-based voice conversion. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, Vol. 39. 25092–25100

    2025

  34. [42]

    Yan Rong, Shan Yang, Guangzhi Lei, and Li Liu. 2025. Dopamine Audiobook: A Training-free MLLM Agent for Emotional and Human-like Audiobook Genera- tion. arXiv preprint arXiv:2504.11002 (2025)

    2025 arXiv

  35. [43]

    In IEEE International Conference on Acoustics, Speech and Signal Processing

    STA-V2A: Video-to-audio generation with semantic and temporal align- ment. In IEEE International Conference on Acoustics, Speech and Signal Processing . 1–5

  36. [44]

    Zeyue Tian, Yizhu Jin, Zhaoyang Liu, Ruibin Yuan, Xu Tan, Qifeng Chen, Wei Xue, and Yike Guo. 2025. AudioX: Diffusion Transformer for Anything-to-Audio Generation. arXiv preprint arXiv:2503.10522 (2025)

    2025 arXiv

  37. [45]

    Zeyue Tian, Zhaoyang Liu, Ruibin Yuan, Jiahao Pan, Qifeng Liu, Xu Tan, Qifeng Chen, Wei Xue, and Yike Guo. 2024. Vidmuse: A simple video-to-music generation framework with long-short-term modeling. arXiv preprint arXiv:2406.04321 (2024)

    2024 arXiv

  38. [46]

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously Large Neural Networks: The MM ’25, October 27–31, 2025, Dublin, Ireland Rong et al. Sparsely-Gated Mixture-of-Experts Layer. InInternational Conference o...

    2017

  39. [47]

    Rong-Cheng Tu, Wenhao Sun, Zhao Jin, Jingyi Liao, Jiaxing Huang, and Dacheng Tao. 2024. SPAgent: Adaptive Task Decomposition and Model Selection for General Video Generation and Editing. arXiv preprint arXiv:2411.18983 (2024)

    2024 arXiv

  40. [48]

    Heng Wang, Jianbo Ma, Santiago Pascual, Richard Cartwright, and Weidong Cai. 2024. V2a-mapper: A lightweight solution for vision-to-audio generation by connecting foundation models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 15492–15501

    2024

  41. [49]

    Andros Tjandra, Yi-Chiao Wu, Baishan Guo, John Hoffman, Brian Ellis, Apoorv Vyas, Bowen Shi, Sanyuan Chen, Matt Le, Nick Zacharov, et al . 2025. Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Sound. arXiv preprint arXiv:2502.05139 (2025)

    2025 arXiv

  42. [50]

    Zhenyu Wang, Aoxue Li, Zhenguo Li, and Xihui Liu. 2024. Genartist: Multimodal llm as an agent for unified image generation and editing. Advances in Neural Information Processing Systems 37 (2024), 128374–128395

    2024

  43. [51]

    Tianxin Xie, Yan Rong, Pengfei Zhang, and Li Liu. 2024. Towards Controllable Speech Synthesis in the Era of Large Language Models: A Survey. arXiv preprint arXiv:2412.06602 (2024)

    2024 arXiv

  44. [52]

    Jinting Wang, Li Liu, and Jun Wang. 2025. MotionComposer: Enhancing Rhyth- mic Music Generation with Adaptive Retrieval Reference. In IEEE International Conference on Acoustics, Speech and Signal Processing . 1–5

    2025

  45. [53]

    Zhifei Xie, Mingbao Lin, Zihang Liu, Pengcheng Wu, Shuicheng Yan, and Chun- yan Miao. 2025. Audio-reasoner: Improving reasoning capability in large audio language models. arXiv preprint arXiv:2503.02318 (2025)

    2025

  46. [54]

    Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. 2024. Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7151–7161

    2024

  47. [55]

    Zhifeng Xie, Qile He, Youjia Zhu, Qiwei He, and Mengtian Li. 2025. Film- Composer: LLM-Driven Music Production for Silent Film Clips. arXiv preprint arXiv:2503.08147 (2025)

    2025 arXiv

  48. [56]

    Jing Yang, Amit Barde, and Mark Billinghurst. 2022. Audio augmented reality: A systematic review of technologies, applications, and future research directions. journal of the audio engineering society 70, 10 (2022), 788–809

    2022

  49. [57]

    Yi Yuan, Haohe Liu, Xubo Liu, Qiushi Huang, Mark D Plumbley, and Wenwu Wang. 2024. Retrieval-augmented text-to-audio generation. In IEEE International Conference on Acoustics, Speech and Signal Processing . 581–585

    2024

  50. [58]

    Jinlong Xue, Yayue Deng, Yingming Gao, and Ya Li. 2024. Auffusion: Leveraging the power of diffusion and large language models for text-to-audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing (2024)

    2024

  51. [59]

    Yiming Zhang, Yicheng Gu, Yanhong Zeng, Zhening Xing, Yuancheng Wang, Zhizheng Wu, and Kai Chen. 2024. Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds. arXiv preprint arXiv:2407.01494 (2024)

    2024 arXiv

  52. [60]

    Yehang Zhang, Xinli Xu, Xiaojie Xu, Li Liu, and Yingcong Chen. 2025. Long-Video Audio Synthesis with Multi-Agent Collaboration. arXiv preprint arXiv:2503.10719 (2025)

    2025 arXiv

  53. [61]

    Chong Zhang, Yukun Ma, Qian Chen, Wen Wang, Shengkui Zhao, Zexu Pan, Hao Wang, Chongjia Ni, Trung Hieu Nguyen, Kun Zhou, et al. 2025. InspireMusic: Integrating Super Resolution and Large Language Model for High-Fidelity Long- Form Music Generation. arXiv preprint arXiv:2503.00...

    2025 arXiv

  54. [62]

    Yixuan Zhou, Xiaoyu Qin, Zeyu Jin, Shuoyi Zhou, Shun Lei, Songtao Zhou, Zhiyong Wu, and Jia Jia. 2024. Voxinstruct: Expressive human instruction- to-speech generation with unified multilingual codec language modelling. In Proceedings of the 32nd ACM International Conference on...

    2024

  55. [63]

    Heda Zuo, Weitao You, Junxian Wu, Shihong Ren, Pei Chen, Mingxu Zhou, Yujia Lu, and Lingyun Sun. 2025. GVMGen: A General Video-to-Music Generation Model with Hierarchical Attentions. arXiv preprint arXiv:2501.09972 (2025). A Additional SOTA Comparisons on Short Videos Given th...

    2025 arXiv

  56. [64]

    Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023. Automatic Chain of Thought Prompting in Large Language Models. In The Eleventh International Conference on Learning Representations

    2023

  57. [2024]

    arXiv preprint arXiv:2410.12957 (2024)

    MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic Synchronization. arXiv preprint arXiv:2410.12957 (2024)

    2024 arXiv

  58. [2025]

    arXiv preprint arXiv:2503.16956 (2025)

    From Faces to Voices: Learning Hierarchical Representations for High- quality Video-to-Speech. arXiv preprint arXiv:2503.16956 (2025)

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.