REVIEW 3 major objections 5 minor 13 references
MusiScene: Leveraging MU-LLaMA for Scene Imagination and Enhanced Video Background Music Generation
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims that a music-only captioning model can imagine the video scenes a piece of music fits, making the visual stream unnecessary for context-aware video background music generation.
desk verdict A useful new dataset and task framing, but the main evaluation leans on the same pseudo-labels used to train the model, so the headline gains are not yet convincing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the MSI caption: a short text that describes the scene, atmosphere, or setting a piece of music is suitable for, produced as the answer to a scene question. The machinery is a caption-generation pipeline in which SwinBERT produces video captions, MU-LLaMA produces music captions, and Mixtral fuses them into MSI pseudo-labels; MusiScene learns to produce those labels from music alone, and MusicGen turns them into a soundtrack. The MSI caption is the bottleneck that converts audio into the scene information needed for downstream generation.
What would settle it
Have human annotators write or select scene descriptions for a sample of the video clips, then measure how often MusiScene's captions match those human descriptions; if MusiScene shows no advantage over MU-LLaMA against human-written scene captions, the headline claim about scene imagination would be refuted.
Extended reading notes
Core claim
The central claim is that scene relevance can be captured entirely in text generated from music, without any visual input at test time. MusiScene is created by freezing the language backbone and finetuning MU-LLaMA's music-understanding adapter on pseudo-labels produced by fusing SwinBERT video captions and MU-LLaMA music captions through Mixtral, using prompts that ask what type of scene the music suits. On a held-out 20% of the video-audio dataset, MusiScene improves over MU-LLaMA on BLEU, METEOR, ROUGE-L, and BERTScore against those MSI references. In the downstream task, MSI captions are fed to MusicGen; subjective evaluations rate the resulting music at 74.2 versus 73.5 for music captions and 61.4 for video captions, and in a second comparison at 78.4 versus 76.6 for a fusion caption. The authors conclude that MSI captions are sufficient to convey scene-related information for video background music generation.
Load-bearing premise
The results rest on the assumption that automatically fusing the video caption and the music caption produces a correct description of the scene the music suits; if those fused captions are systematically wrong, the comparisons in the paper measure agreement with a flawed model rather than true scene imagination.
Editorial extensions
If this is right
- Video background music generation can be driven by music-only captions, removing the need to run a video captioning model at inference time.
- Music captioning can be extended beyond low-level musical attributes to imagined scenes, giving text-to-music systems richer prompts.
- The released 3,371-pair video-audio caption dataset can support future work on both video captioning and music captioning.
- Listener ratings suggest scene-aware captions improve perceived coherence between a video and its generated soundtrack compared with video-only captions.
- Because MSI beats the verbose fusion caption in the second subjective study, longer or combined captions are not automatically better for soundtrack fit.
Reading between the lines
- An extension not tested in the paper: check whether MusiScene captions generalize across videos, meaning one music track's scene caption can soundtrack a different but semantically related video.
- A natural next experiment is human validation of Mixtral's fused captions; if human annotators disagree with the pseudo-labels, the MSI objective metrics would need to be re-read as measuring agreement with the oracle, not with true scene fit.
- The scene-imagination bottleneck could transfer to other music-language models and other text-to-music generators; nothing in the method is tied to the specific finetuned checkpoint.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces MusiScene, a music captioning model fine-tuned from MU-LLaMA to perform Music Scene Imagination (MSI): given a music input, the model produces captions describing the type of scene the music would suit. To train it, the authors construct VACAD, a dataset of 3,371 video-audio pairs sampled from AudioSet, and generate MSI labels by prompting Mixtral with SwinBERT video captions and MU-LLaMA music captions. The paper reports that MusiScene improves over MU-LLaMA on BLEU, METEOR, ROUGE-L, and BERTScore, and that using MSI captions as text inputs to MusicGen yields background music that is subjectively preferred over music generated from video captions, music captions, or a fusion of video and music captions. The central claimed contribution is that a music-only captioning model can replace explicit video analysis in video background music generation.
Significance. If the main claims were established, the paper would contribute a novel task formulation, a new cross-modal dataset, and a practical pipeline that reduces video background music generation to text-to-music generation from music-only captions. The authors release code and data, which is a helpful contribution for reproducibility. However, the current evidence does not support the central claims. The MSI evaluation in Table 2 is circular because the evaluation references are the same Mixtral-generated labels used for training. The subjective downstream results are small differences without significance tests, and the objective FAD result favors the video-caption baseline, contradicting the paper's narrative. The paper therefore needs substantial additional validation before its claims can be accepted.
major comments (3)
- [Section 3.1, Table 2] The evaluation of MusiScene's MSI capability is circular. The MSI references in Table 2 are exactly the Mixtral-generated captions produced in Section 3.1 by prompting Mixtral with SwinBERT video captions and MU-LLaMA music captions. MusiScene was fine-tuned on these same labels, so the BLEU, METEOR, ROUGE-L, and BERTScore gains in Table 2 measure how well MusiScene imitates the Mixtral teacher, not how well it performs genuine scene imagination. Moreover, the baseline MU-LLaMA was never trained to answer the MSI prompt, so its low scores may partly reflect a format mismatch. No human annotation, external reference, or inter-annotator agreement is reported to validate that the Mixtral-generated MSI labels are correct ground truth. Without independent evaluation, the abstract's claim that MusiScene is 'more capable of generating contextually relevant captions' is not established.
- [Section 5.2, Tables 4 and 5] The subjective evaluation results are too weak to support the paper's claim that MSI 'outperforms' all other captioning strategies. The reported mean differences are 74.2 vs 73.5 for MSI versus music captions and 78.4 vs 76.6 for MSI versus fusion, with 68 and 50 respondents respectively, but no significance tests, confidence intervals, effect sizes, or variance information are provided. Differences of 0.7 and 0.8 points are easily within sampling noise at these sample sizes. The authors should report paired significance tests (e.g., Wilcoxon signed-rank), score distributions, and ideally per-item or per-participant variability.
- [Section 5.1, Table 3] The objective downstream results are inconsistent with the paper's narrative. Table 3 shows FAD of 5.78 for MSI captions versus 4.46 for video captions, and since the paper defines a lower FAD as indicating generated audio closer to the target distribution, this result favors the video-caption baseline. The text dismisses this by saying lower FAD does not necessarily imply better quality, but this undermines the use of FAD as evidence for the MSI advantage. The KL divergence differences are also tiny (2.05-2.13) and likely within metric noise. The paper needs a clear pre-specified primary evaluation metric and should either reconcile the FAD result or explicitly soften the downstream claim.
minor comments (5)
- [Abstract and Section 1] The phrase 'prove that our MusiScene is more capable' is too strong for the empirical evidence presented; 'indicate' or 'suggest' would be more appropriate.
- [Section 2.1] 'Herrmann-1' appears to be an incomplete citation or placeholder for a system name; it should be corrected to a proper reference.
- [Section 3.1] The dataset is described as 'large-scale' with 3,371 pairs, which is modest relative to AudioSet's scale; consider rephrasing. Also, 'V ACAD' should be 'VACAD'.
- [Throughout] There are several typos and inconsistencies, including 'finetune' versus 'fine-tune', 'fintune' in Section 4.1, and 'focusing solely' in the abstract.
- [Section 4.2] Details of the MusicGen evaluation are missing, including the MusicGen model variant, hyperparameters, the number of generated samples per caption, and how the reference audio for FAD was selected.
Circularity Check
MSI pseudo-labels serve as both the training target and the evaluation reference, so Table 2 mainly measures fit to a Mixtral-generated teacher distribution rather than independent contextual relevance.
-
fitted input called prediction
[Section 3.1 (Dataset Creation) and Section 5.1 (Objective Evaluation, MSI), Tables 1 and 2]
"To combine cross-modal captions and to generate ground truth for MSI, we use Mixtral of Experts (Jiang et al., 2024) LLM with different prompts each (see Appendix A). ... These metrics measure how close the generated captions are to the referenced captions created in Section 3. The metrics shown in Table 2 suggests that MusiScene produces captions that are more context-related and capable of scene imagination than MU-LLaMA."
The evaluation references in Table 2 are the same Mixtral-generated MSI captions that define the training target. Those references were produced by prompting Mixtral with SwinBERT video captions and MU-LLaMA music captions, and MusiScene is a fine-tuned MU-LLaMA trained to imitate those Mixtral outputs. The base MU-LLaMA was never trained to emit MSI-style scene sentences such as 'The music is suitable for...', so its lower BLEU/METEOR/ROUGE/BERTScore largely reflects format mismatch. The reported gains therefore reduce to how well MusiScene reproduces its own teacher distribution, not to an independently established increase in contextual relevance.
full rationale
The main derivation chain is: (i) construct MSI ground truth by prompting Mixtral with SwinBERT video captions and MU-LLaMA music captions; (ii) fine-tune MU-LLaMA on 80% of these pseudo-labels to obtain MusiScene; (iii) evaluate MusiScene against the same Mixtral-generated labels on the remaining 20%; (iv) use MusiScene captions to generate music and compare with other caption strategies. Step (iii) is the circular point: the evaluation references are generated by the same pipeline that produced the training targets, and the inputs to that pipeline include outputs of the baseline model MU-LLaMA. The base model was never trained to write MSI-style 'The music is suitable for...' sentences, so the large BLEU/ROUGE/BERTScore gains in Table 2 largely measure format imitation rather than independent contextual relevance. The paper's conclusion that MusiScene is 'more capable of generating contextually relevant captions' is therefore defined and measured by the self-generated labels. This is partial, not total, circularity because the 20% held-out split avoids direct memorization and because the subjective listening test is an independent, human-based check. That said, the subjective test evaluates the downstream music rather than caption relevance, and the reported differences (74.2 vs 73.5; 78.4 vs 76.6) are small and lack significance testing. The objective FAD/KL results do not consistently favor MSI. Thus the central caption-quality claim rests primarily on a self-referential pseudo-label evaluation.
Assumptions & free parameters
free parameters (2)
- Training epochs =
20
- Dataset subset size =
3371 AudioSet clips
assumptions (4)
- domain assumption The AudioSet class label 'Music' identifies usable background-music videos for constructing the MSI training set.
- ad hoc to paper Mixtral-generated captions from SwinBERT and MU-LLaMA outputs constitute valid ground truth for music scene imagination.
- domain assumption SwinBERT video captions and MU-LLaMA music captions are accurate enough for the fusion step.
- domain assumption FAD and KL divergence on MusicGen outputs are meaningful proxies for the contextual fit of generated music.
Cite this review
Pith. "Pith review of MusiScene: Leveraging MU-LLaMA for Scene Imagination and Enhanced Video Background Music Generation." pith.science (2026). https://pith.science/paper/LOFHDLRZ
@misc{pith2026250705894,
author = {Pith},
title = {Pith review of: MusiScene: Leveraging MU-LLaMA for Scene Imagination and Enhanced Video Background Music Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/LOFHDLRZ}},
note = {Machine review of arXiv:2507.05894}
}
read the original abstract
Humans can imagine various atmospheres and settings when listening to music, envisioning movie scenes that complement each piece. For example, slow, melancholic music might evoke scenes of heartbreak, while upbeat melodies suggest celebration. This paper explores whether a Music Language Model, e.g. MU-LLaMA, can perform a similar task, called Music Scene Imagination (MSI), which requires cross-modal information from video and music to train. To improve upon existing music captioning models which focusing solely on musical elements, we introduce MusiScene, a music captioning model designed to imagine scenes that complement each music. In this paper, (1) we construct a large-scale video-audio caption dataset with 3,371 pairs, (2) we finetune Music Understanding LLaMA for the MSI task to create MusiScene, and (3) we conduct comprehensive evaluations and prove that our MusiScene is more capable of generating contextually relevant captions compared to MU-LLaMA. We leverage the generated MSI captions to enhance Video Background Music Generation (VBMG) from text.
Figures
Reference graph
Works this paper leans on
-
[1]
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. 2023. https://arxiv.org/abs/2306.05284 Simple and controllable music generation
arXiv 2023
-
[2]
Shangzhe Di, Zeren Jiang, Si Liu, Zhaokai Wang, Leyan Zhu, Zexin He, Hongming Liu, and Shuicheng Yan. 2021. https://doi.org/10.1145/3474085.3475195 Video background music generation with controllable music transformer . In Proceedings of the 29th ACM International Conference on Multimedia, MM ’21. ACM
-
[3]
SeungHeon Doh, Keunwoo Choi, Jongpil Lee, and Juhan Nam. 2023. https://arxiv.org/abs/2307.16372 Lp-musiccaps: Llm-based pseudo music captioning . Preprint, arXiv:2307.16372
arXiv 2023
-
[4]
Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. 2017. https://doi.org/10.1109/ICASSP.2017.7952261 Audio set: An ontology and human-labeled dataset for audio events . In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 776--780
arXiv 2017
-
[5]
Muhammad Taimoor Haseeb, Ahmad Hammoudeh, and Gus Xia. 2024. https://audiomatic-research.github.io/herrmann-1/ Gpt-4 driven cinematic music generation through text processing
work page 2024
-
[6]
Hugo Touvron et al. 2023. https://arxiv.org/abs/2307.09288 Llama 2: Open foundation and fine-tuned chat models . Preprint, arXiv:2307.09288
arXiv 2023
-
[7]
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven...
arXiv 2024
-
[8]
Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. 2018. https://arxiv.org/abs/1812.08466 Fréchet audio distance: A metric for evaluating music enhancement algorithms
arXiv 2018
Show all 13 references
-
[9]
Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. 2022. https://arxiv.org/abs/2209.15352 Audiogen: Textually guided audio generation
2022 arXiv
-
[10]
Yizhi Li, Ruibin Yuan, Ge Zhang, Yinghao Ma, Xingran Chen, Hanzhi Yin, Chenghao Xiao, Chenghua Lin, Anton Ragni, Emmanouil Benetos, Norbert Gyenge, Roger Dannenberg, Ruibo Liu, Wenhu Chen, Gus Xia, Yemin Shi, Wenhao Huang, Zili Wang, Yike Guo, and Jie Fu. 2023. https://arxiv.o...
2023 arXiv
-
[11]
Kevin Lin, Linjie Li, Chung-Ching Lin, Faisal Ahmed, Zhe Gan, Zicheng Liu, Yumao Lu, and Lijuan Wang. 2022. https://arxiv.org/abs/2111.13196 Swinbert: End-to-end transformers with sparse attention for video captioning . Preprint, arXiv:2111.13196
2022 arXiv
-
[12]
Shansong Liu, Atin Sakkeer Hussain, Chenshuo Sun, and Ying Shan. 2023. https://arxiv.org/abs/2308.11276 Music understanding llama: Advancing text-to-music generation with question answering and captioning . Preprint, arXiv:2308.11276
2023 arXiv
-
[13]
Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. 2022. https://arxiv.org/abs/2207.09983 Diffsound: Discrete diffusion model for text-to-sound generation
2022 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.