MusFlow: Multimodal Music Generation via Conditional Flow Matching

Jiahao Song; Yuzhao Wang

REVIEW 5 major objections 4 minor 1 cited by

MusFlow: Multimodal Music Generation via Conditional Flow Matching

T0 review · 5 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MusFlow claims that small MLP adapters mapping images, stories, and captions into the CLAP audio space, plus conditional flow matching in a VAE latent, are enough to generate music that matches any subset of those conditions.

desk verdict A competent, lightweight multimodal music generation system with a useful new dataset; the dataset construction loop and missing error bars need scrutiny before the claims fully land. read the letter →

arxiv 2504.13535 v1 pith:U2XOFEH6 submitted 2025-04-18 cs.SD cs.MMeess.AS

Jiahao Song , Yuzhao Wang This is my paper · ORCID

classification cs.SDcs.MMeess.AS

keywords MultimodalmusicgenerationFlowmatchingCLAPembeddingalignmentImage-to-musicStory-to-musicMulti-agentdatasetannotationMMusSetText-to-music

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that high-quality music can be generated from conditions that carry no explicit musical instructions—images, story texts, and ordinary captions—using a single lightweight model rather than a large language model as a reasoning bridge. It claims that aligning each condition into the CLAP audio embedding space with small MLP adapters, then reconstructing a compressed Mel-spectrogram by conditional flow matching, is enough to make unimodal and multimodal conditions drive semantically appropriate music. If true, this would lower the expertise barrier for music creation: a user could supply a scene image or a narrative and get matching background music without writing instrument-and-genre prompts. The paper also introduces a dataset, MMusSet, built by an automated multi-agent annotation workflow, and reports that MusFlow matches or beats existing text-to-music and multimodal baselines on objective and subjective metrics.

What carries the argument

The load-bearing mechanism is feature alignment by MLP adapters into the CLAP audio space: each adapter distills a condition (image via CLIP, story via CLIP, caption via CLAP) into the embedding of the music it should evoke, and the generation is carried by Conditional Flow Matching in the FM-OT formulation, which learns a straight optimal-transport path from Gaussian noise to the compressed Mel-spectrogram in the MusicLDM VAE latent. The same alignment objective is reused during joint training, and random condition masking forces the model to work with arbitrary subsets of inputs.

What would settle it

A direct test is to take MMusSet pairs and ask human raters whether each matched music clip plausibly fits its image and story; if agreement does not exceed chance for pairs passing the 0.3 CLAP threshold, the dataset's semantic fit and the resulting quality gains are artifacts of the matching criterion. A second check is to train MusFlow on randomly paired image-story-caption-music quadruples with the same pipeline; if FAD and CLAP scores barely drop, the alignment loss and generation stage were not actually learning condition-to-music correspondence.

Watch

Extended reading notes

Core claim

MusFlow's central claim is that the hard part of multimodal music generation is not reasoning about content but aligning representations: if the image, story, or caption embedding can be mapped into the same space as the music's own CLAP embedding, a conditional flow-matching model trained in a pretrained VAE latent space can generate music consistent with any subset of those conditions. The mapping is learned by three separate MLP adapters, one per modality, trained with an MSE loss against the target music's CLAP embedding, and the aligned embeddings are averaged into one fused condition. During inference the model samples a latent trajectory from Gaussian noise to the data distribution and decodes it to a Mel-spectrogram and then a waveform. The paper reports that this architecture outperforms MusicGen, MusicLDM, AudioLDM2, CoDi, and M2UGen on most FAD, KL, CLAP, ImageBind, and human-evaluation metrics across caption-, story-, image-, and multimodal-to-music tasks.

Load-bearing premise

The whole pipeline rests on trusting that a CLAP similarity score of 0.3 between an automatically written caption and a music clip means the music truly fits the image and story; if that threshold admits mismatched pairs, the dataset and the reported alignment scores are inflated.

Editorial extensions

If this is right

If the claim holds, text-to-music models do not need an LLM-based reasoning module; a frozen encoder plus a trained adapter suffices, cutting memory and compute.
Image-to-music and story-to-music become viable on a single consumer GPU, which could make background-music creation practical for films, audiobooks, and podcasts without musical prompt expertise.
The multi-agent annotation workflow could be reused to build multimodal training sets for other generation tasks, including video-to-music once the video modality is added.
Because random condition masking is used, one trained model covers all seven input combinations, so a deployed system does not need separate models for each modality mix.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The authors leave implicit that the model's ceiling is set by how well CLAP captures musical semantics: if CLAP is blind to dimensions such as tempo or loudness, the adapters cannot recover them, and generated music will be correspondingly limited.
A testable extension is to replace the simple averaging of aligned embeddings with learned weights or cross-attention, then measure whether conflicting conditions, such as a bright image paired with a sad caption, are resolved better.
The 0.3 CLAP threshold in the dataset pipeline is a hidden hyperparameter; one could probe it by training the same model on pairs filtered at higher thresholds and checking whether quality metrics improve further.
The paper's stated limitations of 16 kHz sampling and 10-second clips bound the current practical deployment, so extending the vocoder stage to 44.1 kHz and longer durations is a natural next step the authors do not pursue.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about MusFlow. First, it is a sensible systems paper: it combines CLAP-space MLP adapters with conditional flow matching to generate music from images, story texts, or captions, and it ships a new multimodal dataset plus trained model. Second, the strongest claim—that MMusSet contains genuine image–story–caption–music quadruples—is not actually verified, and this is the load-bearing stone of the whole paper.

What is new and good: the specific pipeline of mapping CLIP/CLAP embeddings into the target audio's CLAP space via separate MLPs, then fusing them for a flow-matching decoder, is not in the prior literature. The multi-agent annotation workflow (BLIP captioning, CLIP filtering, fine-tuned Qwen2-VL for music captions, CLAP matching to FMA-Medium) is a pragmatic way to build a multimodal dataset at scale. The experiments cover four tasks, and MusFlow beats the baselines on most objective and subjective metrics. The limitations section is honest about sampling rate, missing video, and QLoRA rank.

Now the soft spots, in proportional order. The stress-test note is on target: the Music Filtering Agent picks the highest-CLAP-scoring clip among ~25k FMA tracks and then checks a 0.3 threshold. The maximum of 25k cosine similarities can easily exceed 0.3 by chance, so the paper must report the distribution of selected scores and a shuffled/null baseline to show the threshold is selective. If the pairs are arbitrary, the alignment adapters learn a spurious mapping and the CLAP/ImageBind scores in Table 1 inherit that bias. This is not a fatal flaw but it is an unverified assumption in the central claim, and the authors can fix it with a few lines. Second, the evaluation lacks error bars or significance tests throughout; the caption-to-music CLAP score is actually below MusicLDM (0.32 vs 0.34), so the “best” claim is already qualified. Third, the text-only baselines (MusicGen, AudioLDM2, MusicLDM) are adapted to story and image inputs, but the adaptation is not described, making those comparisons hard to interpret. Fourth, no comparison to MusicFlow, despite citing it as the closest flow-matching text-to-music system. The subjective test (20 participants, 50 samples) is small but typical for the area.

The circularity concern—aligning to CLAP embeddings and then evaluating with CLAP score—is real but not damning; it is standard practice in music generation, and the FAD/KL and human ratings partially break the loop.

Who is this for? Anyone working on text-to-music or multimodal generation, and anyone building datasets via automated pipelines. It deserves a serious referee, not a desk rejection. I would send it out with a request for the null-distribution analysis, error bars, and baseline adaptation details, and expect a major revision.

Referee Report

5 major / 4 minor

Summary. The paper proposes MusFlow, a flow-matching-based music generation model that accepts images, story texts, and music captions as conditions, either singly or in combination. Conditions are encoded with frozen CLIP and CLAP encoders, then mapped into the CLAP audio embedding space by three MLP adapters trained with the MSE loss of Eq. (3); a conditional flow-matching module (Eq. (8)) generates a compressed VAE latent of a mel-spectrogram, which is decoded with a pretrained HiFi-GAN vocoder. To obtain training data, the authors design a multi-agent workflow (Sec. 3.2) with BLIP story generation, CLIP-based script review, a QLoRA-fine-tuned Qwen2-VL music-caption composer, and CLAP-based music selection from FMA-Medium, producing the 33.3k-sample MMusSet dataset. Experiments cover caption-to-music, story-to-music, image-to-music, and multimodal generation, with objective (FAD, KL, CLAP, ImageBind) and subjective (OVL, REL) metrics.

Significance. If the results hold, MusFlow would provide a lightweight alternative to LLM-bridged multimodal music generators, and MMusSet would be a useful public resource. The paper is the first to apply conditional flow matching to multimodal music generation, and the multi-agent dataset-construction pipeline is a reasonable response to the lack of image-and-story-conditioned music data. The authors commit to releasing code, data, and samples. However, the evidence as presented is not yet proportionate to the headline claim: the semantic metrics are partly circular with the training and data-construction objectives, the music-filtering step is not validated against a null distribution, and the adaptation of text-only baselines to story and image inputs is undocumented. These issues are fixable within the scope of a revision, but they currently weaken the central claim.

major comments (5)

[Sec. 3.2, Music Filtering Agent] The Music Filtering Agent selects, for each generated caption, the highest-CLAP-similarity clip from FMA-Medium (~25k tracks) and accepts it only if the similarity exceeds 0.3. Because the maximum of many cosine similarities can exceed any fixed threshold even when most pairs are unrelated, the reported acceptance rule does not by itself establish that the selected clips are semantically matched to the captions. The paper reports neither the distribution of selected CLAP scores nor a shuffled/null baseline. If most accepted pairs are arbitrary, the MLP adapters (Eq. (3)) learn a spurious image/story-to-CLAP mapping, and the CLAP/ImageBind scores in Table 1 inherit that bias. Please report the score distribution, a null baseline (e.g., caption-to-random-clip max similarities), and ideally a human validation of a random subset of MMusSet quadruples.
[Eq. (3) and Table 1] The alignment loss in Eq. (3) trains the adapters to reproduce the CLAP embedding of the target audio, and the CLAP score in Table 1 evaluates the generated audio in the same embedding space; moreover, the dataset itself was filtered by CLAP similarity during construction. Consequently, the reported CLAP gains may reflect the model learning to match the retrieval criterion rather than genuinely improved semantic alignment with images or stories. The authors should add evaluation with a metric not used in training or data construction (e.g., per-condition human REL judgments, or retrieval with a different audio-text encoder), and should report ImageBind scores separately for each input modality rather than only for the image and multimodal conditions.
[Sec. 4.2.2 and 4.2.3, Table 1] Table 1 reports story-to-music and image-to-music results for text-only baselines (MusicGen, AudioLDM2-Music, MusicLDM), but the paper never explains how these models were given story or image inputs. No prompt template, fixed mapping, or fine-tuning procedure is described, so the comparison is not reproducible and the baseline numbers may not reflect the models' actual capabilities under these conditions. Please provide the exact adaptation protocol, or restrict the comparison to models that natively accept the tested modalities.
[Sec. 3.2 vs. Sec. 4.4] There is an inconsistency about the data used to fine-tune Qwen2-VL. Section 3.2 states that the Script Writing Agent generates story texts for Music2Image, extended-MusicCaps, MelBench, and MUImage, that these form the 10k pretraining samples, and that 'with this pretraining data' Qwen2-VL is fine-tuned. Section 4.4 instead states that fine-tuning is performed on MusicCaps, MelBench, and MUImage and validated on Music2Image. If Music2Image was used to fine-tune Qwen2-VL, then all evaluation on Music2Image (including the MusFlow experiments) is contaminated; if it was not, the text should be corrected. Please clarify the data split and, if needed, re-run the evaluation with a properly held-out set.
[Sec. 4.2, Tables 1 and 3] No error bars, confidence intervals, or significance tests are reported for any objective or subjective metric. The subjective evaluation uses 20 participants and 50 samples per model, and the objective differences in Table 1 are often small (e.g., caption-to-music CLAP 0.32 vs. 0.34). Without variance estimates, it is impossible to determine which differences are meaningful, especially where MusFlow is not the best (CLAP score). Please report standard deviations over evaluation seeds or bootstrapped confidence intervals, and a significance test for the subjective comparison.

minor comments (4)

[Sec. 4.3] The title contains a typo: 'Feature Aligenment Evaluation' should be 'Feature Alignment Evaluation'.
[Sec. 3.1.2] There are typos: 'vector filed' should be 'vector field' and 'Guassian' should be 'Gaussian'.
[Table 1] The table header contains a garbled run of Unicode glyphs (the '/uni00000018/...' sequence); the table should be regenerated with clean labels so that the metric names are readable.
[Sec. 3.2] The description of the Music Composition Agent says the fine-tuning data consists of 'image-story-caption triplets', but the preceding sentences describe generating stories and captions from images. Please specify exactly which fields are used as inputs and targets for the Qwen2-VL fine-tuning, and how the Music2Image validation set is kept out of that fine-tuning.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: MusFlow's central quality claim rests on independent FAD/KL/human evaluations; the CLAP-space overlap is a metric caveat, not a load-bearing circular step.

full rationale

MusFlow is an empirical system paper rather than a theorem derivation. The central claim—that MusFlow generates high-quality music from unimodal or multimodal conditions—is supported by Frechet Audio Distance, KL divergence, and human OVL/REL ratings, none of which are part of the training objectives or dataset-construction filter. The alignment loss in Eq. (3) trains the MLP adapters to map condition embeddings to e_M, the CLAP embedding of the paired audio, and the dataset's Music Filtering Agent used CLAP similarity with threshold 0.3; consequently the CLAP Score column in Table 1 is not fully independent evidence of semantic alignment. However, the paper does not rely on CLAP Score alone: ImageBind Score, FAD, KL, and human evaluations provide external support, and the caption-to-music CLAP Score of MusFlow (0.32) is actually lower than MusicLDM's (0.34), which is inconsistent with a by-construction inflation. No load-bearing step is justified by self-citation, no uniqueness theorem is imported, and no fitted parameter is renamed as a prediction. The dataset-pairing validity concern about the 0.3 threshold and max-over-FMA selection is a correctness-risk issue, not a circularity of the derivation.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on trained neural networks and hyperparameters rather than new physical or mathematical entities. The key free parameters are the joint loss weight, the two filtering thresholds, and the QLoRA rank; the axioms are the soundness of flow matching theory, the reliability of pretrained encoders, and the assumption that CLAP embeddings and automated threshold filtering capture the real semantic link between conditions and music.

free parameters (4)

lambda in joint loss = not specified
Controls the balance between generation loss and alignment loss in Eq. (9); set by hand but its value is not reported.
Music Filtering Agent threshold = 0.3
CLAP score threshold used to accept or reject music-caption matches in dataset construction (Sec. 3.2); directly shapes MMusSet composition.
Script Review Agent threshold = 0.3
CLIP score threshold for filtering image-story pairs (Sec. 3.2); affects which pairs survive into training.
QLoRA rank for Qwen2-VL = 8
Rank of the low-rank adapter used to fine-tune the vision-language model; Table 4 shows higher rank improves caption quality.

assumptions (4)

standard math Flow matching theory (CFM with FM-OT) produces a valid training objective for generative modeling.
The derivation in Sec. 3.1.2 relies on results from Lipman et al. [27] and Liu et al. [32].
domain assumption Pretrained CLIP, CLAP, VAE, and HiFi-GAN models provide adequate semantic and acoustic representations.
The pipeline treats these encoders and decoders as frozen and reliable enough for music semantics, following MusicLDM.
ad hoc to paper The CLAP embedding of the target audio is a sufficient supervisory target for aligning image, story, and caption conditions.
The alignment loss in Eq. (3) minimizes distance between adapter outputs and the ground-truth music CLAP embedding; no independent evidence is given that this space preserves all needed condition semantics.
domain assumption Automatic CLAP-based matching in dataset construction yields semantically valid quadruples.
The Music Filtering Agent selects music by CLAP score with threshold 0.3 (Sec. 3.2), assuming high CLAP similarity implies true semantic fit.

how reviews work

0 comments

Cite this review

Pith. "Pith review of MusFlow: Multimodal Music Generation via Conditional Flow Matching." pith.science (2026). https://pith.science/paper/U2XOFEH6

@misc{pith2026250413535,
  author       = {Pith},
  title        = {Pith review of: MusFlow: Multimodal Music Generation via Conditional Flow Matching},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U2XOFEH6}},
  note         = {Machine review of arXiv:2504.13535}
}

read the original abstract

Music generation aims to create music segments that align with human aesthetics based on diverse conditional information. Despite advancements in generating music from specific textual descriptions (e.g., style, genre, instruments), the practical application is still hindered by ordinary users' limited expertise or time to write accurate prompts. To bridge this application gap, this paper introduces MusFlow, a novel multimodal music generation model using Conditional Flow Matching. We employ multiple Multi-Layer Perceptrons (MLPs) to align multimodal conditional information into the audio's CLAP embedding space. Conditional flow matching is trained to reconstruct the compressed Mel-spectrogram in the pretrained VAE latent space guided by aligned feature embedding. MusFlow can generate music from images, story texts, and music captions. To collect data for model training, inspired by multi-agent collaboration, we construct an intelligent data annotation workflow centered around a fine-tuned Qwen2-VL model. Using this workflow, we build a new multimodal music dataset, MMusSet, with each sample containing a quadruple of image, story text, music caption, and music piece. We conduct four sets of experiments: image-to-music, story-to-music, caption-to-music, and multimodal music generation. Experimental results demonstrate that MusFlow can generate high-quality music pieces whether the input conditions are unimodal or multimodal. We hope this work can advance the application of music generation in multimedia field, making music creation more accessible. Our generated samples, code and dataset are available at musflow.github.io.

Figures

Figures reproduced from arXiv: 2504.13535 by the authors.

**Figure 2.** An illustration of our MusFlow framework for multimodal music generation. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** An illustration of our proposed Multi-Agent Workflow for dataset creation. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Ablation study results of the Alignment Module. CLAP [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

FlowSonic: Stable Zero-Shot Music Editing via High-Order Trajectory Integration
cs.SD 2026-07 reject novelty 4.0 of 10

FlowSonic combines deterministic rectified-flow inversion, cached cross-attention injection, and a 'seeded' third-order Adams-Bashforth solver to report better timbre and genre edits on small datasets.

Reference graph

Works this paper leans on

57 extracted references · 20 canonical work pages · cited by 1 Pith paper

[1]

Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, An- toine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matt Sharifi, Neil Zeghidour, and Christian Frank

Andrea Agostinelli, Timo I. Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, An- toine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matt Sharifi, Neil Zeghidour, and Christian Frank. 2023. MusicLM: Generating Music From Text. arXiv preprint arXiv:2301.11325 (2023)

arXiv 2023
[2]

Michael Samuel Albergo and Eric Vanden-Eijnden. 2023. Building Normalizing Flows with Stochastic Interpolants. In The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=li7qeBbCR1t

work page 2023
[3]

Jean-Pierre Briot and François Pachet. 2020. Deep learning for music generation: challenges and directions. Neural Computing and Applications 32, 4 (2020), 981– 993

work page 2020
[4]

Giovanna Castellano, Vincenzo Digeno, Giovanni Sansaro, and Gennaro Vessio

work page
[5]

Ke Chen*, Yusong Wu*, Haohe Liu*, Marianna Nezhurina, Taylor Berg- Kirkpatrick, and Shlomo Dubnov. 2024. MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategies. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP

work page 2024
[6]

Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. 2018. Neural ordinary differential equations. Advances in neural information processing systems 31 (2018)

2018
[7]

Sanjoy Chowdhury, Sayan Nag, K J Joseph, Balaji Vasan Srinivasan, and Dinesh Manocha. 2024. MELFuSION: Synthesizing Music from Image and Language Cues Using Diffusion Models. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 26816–26825. doi:10.1109/CVPR52733.2024.02533

arXiv 2024
[8]

Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Defossez. 2023. Simple and Controllable Music Genera- tion. In Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Asso- ciates, Inc., 47704–47720. https://proceedi...

2023

Show all 57 references

[9]

Marco Cuturi. 2013. Sinkhorn Distances: Lightspeed Computation of Optimal Transport. In Advances in Neural Information Processing Systems , C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger (Eds.), Vol. 26. Cur- ran Associates, Inc. https://proceedings.neu...

2013
[10]

Michaël Defferrard, Kirell Benzi, Pierre Vandergheynst, and Xavier Bresson
[11]

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems 36 (2023), 10088–10115

2023
[12]

Shangzhe Di, Zeren Jiang, Si Liu, Zhaokai Wang, Leyan Zhu, Zexin He, Hong- ming Liu, and Shuicheng Yan. 2021. Video Background Music Generation with Controllable Music Transformer. In Proceedings of the 29th ACM International Conference on Multimedia (Virtual Event, China) (MM...

2021
[13]

Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, Zhifu Gao, and Zhijie Yan. 2024. CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens. CoRR abs/2407.05407 (2024...

2024 doi arXiv
[14]

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. 2024. Scaling rectified flow transformers for high-resolution image s...

2024
[15]

Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. 2023. ImageBind One Embedding Space to Bind Them All. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 15180–15190. doi:10.1109/CVP...

2023
[17]

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. CLIPScore: A Reference-free Evaluation Metric for Image Captioning. In EMNLP (1). 7514–7528. https://doi.org/10.18653/v1/2021.emnlp-main.595

2021 doi
[18]

Tanisha Hisariya, Huan Zhang, and Jinhua Liang. 2024. Bridging Paintings and Music – Exploring Emotion based Music Generation through Paintings. arXiv:2409.07827 [cs.SD] https://arxiv.org/abs/2409.07827

2024 arXiv
[19]

Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. Advances in neural information processing systems 33 (2020), 6840–6851

2020
[20]

Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber
[21]

Qingqing Huang, Daniel S Park, Tao Wang, Timo I Denk, Andy Ly, Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Frank, et al. 2023. Noise2music: Text-conditioned music generation with diffusion models. arXiv preprint arXiv:2302.03917 (2023)

2023 arXiv
[22]

Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Baner- jee, and Fillia Makedon. 2020. A survey on contrastive self-supervised learning. Technologies 9, 1 (2020), 2

2020
[23]

Jaeyong Kang, Soujanya Poria, and Dorien Herremans. 2024. Video2Music: Suit- able music generation from videos using an Affective Multimodal Transformer model. Expert Syst. Appl. 249, PC (Sept. 2024), 17 pages. doi:10.1016/j.eswa.2024. 123640

2024 doi
[24]

Kingma and Max Welling

Diederik P. Kingma and Max Welling. 2014. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings

2014
[25]

Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020. HiFi-GAN: generative adversarial networks for efficient and high fidelity speech synthesis. In Proceed- ings of the 34th International Conference on Neural Information Processing Systems (Vancouver, BC, Canada) (NIPS ’20). Cu...

2020
[26]

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning . PMLR, 12888–12900

2022
[27]

Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. 2023. Flow Matching for Generative Modeling. In The Eleventh International Conference on Learning Representations . https://openreview.net/ forum?id=PqvMRDCJT9t

2023
[28]

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual In- struction Tuning. In Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., 34892–34916. https:/...

2023
[29]

Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D Plumbley. 2024. Audioldm 2: Learning holistic audio generation with self-supervised pretraining. IEEE/ACM Transactions on Audio, Speech, and Language Processing (2024)

2024
[30]

Qiang Liu. 2022. Rectified flow: A marginal preserving approach to optimal transport. arXiv preprint arXiv:2209.14577 (2022)

2022 arXiv
[31]

Shansong Liu, Atin Sakkeer Hussain, Qilong Wu, Chenshuo Sun, and Ying Shan
[32]

Xingchao Liu, Chengyue Gong, and qiang liu. 2023. Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. In The Eleventh International Conference on Learning Representations . https://openreview.net/ forum?id=XVjTT1nw5z

2023
[33]

Robert J. McCann. 1997. A Convexity Principle for Interacting Gases. Advances in Mathematics 128, 1 (1997), 153–179. doi:10.1006/aima.1997.1634

1997
[34]

Shivam Mehta, Ruibo Tu, Jonas Beskow, Éva Székely, and Gustav Eje Henter
[35]

arXiv preprint arXiv:2412.06660 (2024)

MuMu-LLaMA: Multi-modal Music Understanding and Generation via Large Language Models. arXiv preprint arXiv:2412.06660 (2024)

2024 arXiv
[36]

Gabriel Peyré and Marco Cuturi. 2019. Computational Optimal Transport. Foundations and Trends in Machine Learning 11 (5-6) (2019), 355–602. https: //arxiv.org/abs/1803.00567

2019 arXiv
[37]

K R Prajwal, Bowen Shi, Matthew Lee, Apoorv Vyas, Andros Tjandra, Mahi Luthra, Baishan Guo, Huiyu Wang, Triantafyllos Afouras, David Kant, and Wei- Ning Hsu. 2024. MusicFlow: cascaded flow matching for text guided music generation. In Proceedings of the 41st International Conf...

2024
[38]

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

2021
[39]

In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Matcha-TTS: A Fast TTS Architecture with Conditional Flow Matching. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 11341–11345. doi:10.1109/ICASSP48485.2024.10448291

2024
[40]

Jan Melechovsky, Zixun Guo, Deepanway Ghosal, Navonil Majumder, Dorien Her- remans, and Soujanya Poria. 2024. Mustango: Toward Controllable Text-to-Music Generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lingui...

2024
[41]

Siu-Lan Tan, Annabel Cohen, Annabel J Cohen, Scott D Lipscomb, and Roger A Kendall. 2013. The psychology of music in multimedia. (2013)

2013
[42]

Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. 2023. Any-to-Any Generation via Composable Diffusion. In Thirty-seventh Conference on Neural Information Processing Systems . https://openreview.net/forum?id= 2EDqbSCnmF

2023
[43]

Alexander Tong, Kilian FATRAS, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector-Brooks, Guy Wolf, and Yoshua Bengio. 2024. Improving and generalizing flow-based generative models with minibatch optimal transport. Transactions on Machine Learning Research(2024). ht...

2024
[44]

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 10684–10695. ACM Multimedia 2025, Octobe...

2022
[45]

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. 2021. Score-Based Generative Modeling through Stochas- tic Differential Equations. In International Conference on Learning Representations. https://openreview.net/forum?id=PxTIG12RRHS

2021
[46]

Lei Wang, Ziyi Zhao, Hanwei Liu, Junwei Pang, Yi Qin, and Qidi Wu. 2024. A re- view of intelligent music generation systems. Neural Computing and Applications 36, 12 (2024), 6381–6401

2024
[47]

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191 (2024)

2024 arXiv
[48]

Yajie Wang, Mulin Chen, and Xuelong Li. 2024. Continuous Emotion-Based Image-to-Music Generation. Trans. Multi. 26 (Jan. 2024), 5670–5679. doi:10.1109/ TMM.2023.3338089

2024
[49]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)

2017
[50]

Bao Hangbo Huang Shaohan Dong Li Wei Furu Wang, Wenhui. 2021. MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021". Association for Computational Lingu...

2021 doi
[51]

Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. BARTSCORE: evaluating generated text as text generation. In Proceedings of the 35th International Confer- ence on Neural Information Processing Systems (NIPS ’21) . Curran Associates Inc., Red Hook, NY, USA, Article 2088, 15 pages

2021
[52]

Yueyue Zhu, Jared Baca, Banafsheh Rekabdar, and Reza Rawassizadeh. 2023. A survey of ai music generation tools and models. arXiv preprint arXiv:2308.12982 (2023)

2023 arXiv
[53]

Le Zhuo, Zhaokai Wang, Baisen Wang, Yue Liao, Chenxi Bao, Stanley Peng, Songhao Han, Aixi Zhang, Fei Fang, and Si Liu. 2023. Video background music generation: Dataset, method and evaluation. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 15637–15647

2023
[54]

Yusong Wu*, Ke Chen*, Tianyu Zhang*, Yuchen Hui*, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2023. Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation. InIEEE International Conference on Acoustics, Speech and Signal Proce...

2023
[55]

Jixun Yao, Yuguang Yang, Yu Pan, Ziqian Ning, Jiaohao Ye, Hongbin Zhou, and Lei Xie. 2024. Stablevc: Style controllable zero-shot voice conversion with conditional flow matching. arXiv preprint arXiv:2412.04724 (2024)

2024 arXiv
[2017]

arXiv:1612.01840 [cs.SD] https: //arxiv.org/abs/1612.01840

FMA: A Dataset For Music Analysis. arXiv:1612.01840 [cs.SD] https: //arxiv.org/abs/1612.01840

arXiv
[2022]

Knowledge-Based Systems 248 (2022), 108859

Leveraging Knowledge Graphs and Deep Learning for automatic art analysis. Knowledge-Based Systems 248 (2022), 108859. doi:10.1016/j.knosys.2022. 108859

2022 doi
[2024]

In The Twelfth International Conference on Learning Representations

MetaGPT: Meta Programming for A Multi-Agent Collaborative Frame- work. In The Twelfth International Conference on Learning Representations . https: //openreview.net/forum?id=VtmBAGCN7o

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, An- toine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matt Sharifi, Neil Zeghidour, and Christian Frank

Andrea Agostinelli, Timo I. Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, An- toine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matt Sharifi, Neil Zeghidour, and Christian Frank. 2023. MusicLM: Generating Music From Text. arXiv preprint arXiv:2301.11325 (2023)

arXiv 2023

[2] [2]

Michael Samuel Albergo and Eric Vanden-Eijnden. 2023. Building Normalizing Flows with Stochastic Interpolants. In The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=li7qeBbCR1t

work page 2023

[3] [3]

Jean-Pierre Briot and François Pachet. 2020. Deep learning for music generation: challenges and directions. Neural Computing and Applications 32, 4 (2020), 981– 993

work page 2020

[4] [4]

Giovanna Castellano, Vincenzo Digeno, Giovanni Sansaro, and Gennaro Vessio

work page

[5] [5]

Ke Chen*, Yusong Wu*, Haohe Liu*, Marianna Nezhurina, Taylor Berg- Kirkpatrick, and Shlomo Dubnov. 2024. MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategies. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP

work page 2024

[6] [6]

Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. 2018. Neural ordinary differential equations. Advances in neural information processing systems 31 (2018)

2018

[7] [7]

Sanjoy Chowdhury, Sayan Nag, K J Joseph, Balaji Vasan Srinivasan, and Dinesh Manocha. 2024. MELFuSION: Synthesizing Music from Image and Language Cues Using Diffusion Models. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 26816–26825. doi:10.1109/CVPR52733.2024.02533

arXiv 2024

[8] [8]

Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Defossez. 2023. Simple and Controllable Music Genera- tion. In Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Asso- ciates, Inc., 47704–47720. https://proceedi...

2023