Pith. sign in

REVIEW 4 major objections 6 minor 54 references

EXPOTION: Facial Expression and Motion Control for Multimodal Music Generation

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Integrating facial-expression and upper-body motion features with text prompts improves generated music's musicality, creativity, beat-tempo consistency, temporal alignment, and text adherence, surpassing text-only and video-to-music…

desk verdict Useful dataset and a plausible PEFT recipe, but the central claim about visual conditioning is not supported because the ablation lacks a text-only fine-tuned control. read the letter →

arxiv 2507.04955 v1 pith:OYPEZI54 submitted 2025-07-07 cs.SD cs.AIcs.CVcs.MMeess.AS

Fathinah Izzati , Xinyue Li , Gus Xia This is my paper · ORCID
classification cs.SDcs.AIcs.CVcs.MMeess.AS
keywords expressivemusicgenerationfacialexpressioncontrolmotionmultimodalparameter-efficientfine-tuningvideo-to-musictemporalalignmentGenadapters
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Expotion is a music-generation system that adds two visual control signals—facial expressions and upper-body motion from a video—on top of a text prompt, and claims that this combination yields music that is more musical, creative, beat-consistent, and temporally aligned with the video than text-only or video-only baselines. The paper introduces a new dataset of seven hours of synchronized expressive video–music pairs and shows that a pretrained text-to-music transformer can be adapted to these multimodal controls by fine-tuning under 4% of its parameters on only 130 clips. If the claim holds, it means gesture and expression can act as practical, fine-grained creative controls for music generation without training a large model from scratch.

What carries the argument

The load-bearing mechanism is a joint visual-embedding encoder feeding a condition adaptor: MARLIN features for facial expression and Synchformer/RAFT features for motion are temporally interpolated (temporal smoothing), projected to low rank, fused with learnable positional embeddings, and inserted as a prefix into the last layers of the frozen MusicGen decoder. A zero-initialized learnable gate per layer lets the visual condition start as a no-op and gradually blend in. This lets a 5 fps visual stream control 50 Hz EnCodec audio tokens while preserving the base model's text understanding.

What would settle it

Run the same training and evaluation with text prompts written by humans who have not heard the target audio (or with text conditioning removed entirely); if the visual-conditioned model no longer beats the text-only baseline on video–music consistency and musical quality, the reported improvement depends on the leaked captions rather than on the video features.

Watch

Extended reading notes

Core claim

The central claim is that visual features of a performer—facial expressions encoded by MARLIN and body motion encoded by Synchformer or RAFT optical flow—can be injected into a frozen MusicGen decoder through a learned condition-prefix adaptor, and that this improves generation quality, rhythmic accuracy, and video–music alignment compared with using text alone. The paper further claims that the two modalities are complementary: text supplies semantic grounding, while visual motion, especially Synchformer-based motion, supplies expressive timing. On subjective ratings and objective metrics (FAD, KL, tempo error, beat F1, CLAP and LanguageBind similarity), the best Expotion configuration outperforms MusicGen text-only, VidMuse, and Video2Music. The authors also contribute a 7-hour dataset of 10-second synchronized clips of expressive facial and upper-body gestures paired with instrumental music.

Load-bearing premise

The text prompts are machine-generated captions of the very audio clips the model is trained to reproduce, and the same prompts are reused at inference, so the visual modality's measured benefit could be inflated by a text summary that already leaks the target music's character.

Editorial extensions

If this is right

  • Visual control of music generation can be added to a frozen text-to-music model by training only a small adaptor, so new control modalities do not require retraining the base generator.
  • Temporal smoothing makes fine-grained synchronization feasible: a low-frame-rate visual feature stream can steer high-rate audio code generation without explicit beat detection.
  • Text and vision are complementary in this setup: text provides semantic intent, motion provides expressive timing, and removing either degrades the output.
  • The proposed dataset of synchronized expressive video–music pairs enables future work on gesture- and expression-conditioned music, though its scale (130 training clips) limits genre coverage.
  • Motion features carry more expressive musical information than facial expressions for the collected data, so models tuned on this data favor motion-based control.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the caption confound is real, a stronger test would replace SALMONN captions with human-written prompts that do not describe the target audio; the visual advantage might shrink or disappear.
  • The same prefix-adaptor recipe could plausibly extend to other time-varying controls—conducting gestures, dance, or instrument motion—without retraining the base model.
  • The small dataset (130 clips) and 30-second source audio mean the claimed expressiveness may not transfer to unseen genres or unusual performance styles until the dataset is enlarged.
  • Because generic text prompts alone hurt performance, real deployments would need a robust captioning or user-prompt strategy; visual input alone is not sufficient semantic grounding.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Expotion, a video-and-text-conditioned music generation system built on MusicGen with parameter-efficient fine-tuning. It extracts facial-expression features with MARLIN and motion features with Synchformer or RAFT, temporally interpolates them, projects them to low-rank embeddings, and injects the joint embeddings as condition prefixes in the decoder's last layers. A new dataset of about 7 hours of synchronized facial/upper-body video and music is introduced, and the method is evaluated against vanilla MusicGen, VidMuse, and Video2Music using objective metrics (FAD, KL, IS, tempo error, beat F1, CLAP, LanguageBind) and a subjective study. The paper claims that integrating visual features alongside text improves musical quality, creativity, beat-tempo consistency, temporal alignment, and text adherence.

Significance. If supported, Expotion would be a useful contribution: it is, to my knowledge, the first work to condition music generation on synchronized facial expressions and upper-body gestures; the PEFT setup (about 4% of parameters) and low-rank projections make the approach practical with limited data; and the released dataset, code, and demo are valuable resources for multimodal music generation research. However, the evaluation as presented does not yet establish the central claim that visual features improve music quality. The missing text-only fine-tuned control and the construction of text prompts from the ground-truth audio are serious confounds, and Table 1 itself contradicts the abstract's emphasis on the face+motion configuration. The paper therefore needs a substantive revision of the experimental design and claims rather than cosmetic changes.

major comments (4)
  1. [§4.3, Table 1] Every Expotion row is obtained by fine-tuning MusicGen on the same 130 training clips with SALMONN-generated prompts, whereas the text-only baseline is vanilla MusicGen with no fine-tuning. The observed improvements of Expotion variants over MusicGen (e.g., FAD 1.93 vs 2.76, KL 0.65 vs 0.79) could therefore be due to dataset-specific adaptation rather than to the visual embeddings. A text-only condition using the same PEFT adapter, condition-prefix mechanism, and low-rank projections but without the visual embeddings zf/zm is needed to isolate the visual contribution; without it, the abstract's claim that visual features 'enhance' music quality is not supported.
  2. [§4.1] Text prompts are generated from the ground-truth audio clips with SALMONN and are used both for training and inference. Since the prompt is a machine summary of the target audio, the text path can leak information about the very music the model must generate, inflating text-audio similarity and general-quality metrics. The paper should either condition on captions obtained without access to the target audio (e.g., derived from the video only) or quantify the leakage by comparing against a model trained and tested with human-written or video-derived prompts.
  3. [Table 1, §5.1] The face+motion Syncformer configuration is not the best Expotion variant on the general-quality metrics; motion-only Syncformer with generated prompts achieves FAD 1.93 vs 2.55, KL 0.65 vs 0.67, and IS 1.57 vs 1.49. This contradicts the abstract's and conclusion's emphasis on face+motion superiority. Moreover, no error bars, confidence intervals, or significance tests are reported, so even the larger differences between Expotion and the baselines cannot be assessed, and the small differences between Expotion variants are not meaningful as presented.
  4. [§5.2, Figure 3] The subjective results are described only qualitatively ('all of our models—except for one—outperform the baseline'), with no numerical ratings, error bars, or statistical tests, and the supporting figure data are not included in the manuscript as supplied. Since the conclusion relies on 'subjective studies confirm that the combination of facial and motion features yields superior performance,' this evidence is missing as presented.
minor comments (6)
  1. [Eq. (6)] Equation (6) contains mismatched delimiters: the expression starts with a parenthesis and ends with a square bracket, making the concatenation operation unclear; please rewrite it as z_i = W_e^T [z_{f,i}; z_{m,i}] + z_pos,i.
  2. [§3.1.2] There is a typo in the third paragraph: 'Sychformer' should be 'Synchformer'.
  3. [§5.3.1] The word 'featrues' should be 'features' in the sentence about the two motion feature extractors.
  4. [§3] The first sentence of the methodology section contains a duplicated 'of of'.
  5. [§4.4.1] The objective metrics need precise definitions: FAD should specify the reference audio set and the number of generated samples used, and the KL divergence should specify the label distribution or feature space over which it is computed.
  6. [Table 1] The tempo-error comparison with Video2Music may not be apples-to-apples because Video2Music generates symbolic music that is then rendered, while the remaining systems generate audio directly; this should be acknowledged when interpreting the rhythm-alignment rows.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central visual-conditioning claim is evaluated against external baselines and metrics, and the cited prior architecture is not load-bearing.

full rationale

The paper's derivation chain is not circular. The condition-adapter design is adopted from CoCoMulla [20], which shares an author with the present paper, but the method is restated in full in Section 3.2 and the evaluation is conducted against external baselines (vanilla MusicGen, VidMuse, Video2Music) and external metrics (FAD, KL, IS, CLAP, LanguageBind). No uniqueness theorem is invoked, and no claim is justified solely by the self-citation. The SALMONN-generated text prompts are derived from the ground-truth audio (Section 4.1), which is a legitimate data-leakage concern that can inflate absolute text-audio consistency scores, but it is applied to the text-only MusicGen baseline as well (Section 4.4), so it does not reduce the visual-feature comparison to a tautology. The absence of a text-only fine-tuned control is a genuine experimental confound: every Expotion row in Table 1 is fine-tuned on the same 130 training clips, while the vanilla MusicGen baseline is not, so part of the reported gain may come from dataset adaptation rather than from the visual channel. This is a validity threat to the causal claim, but it is not a circular reduction: the visual features are neither fitted to the evaluation metrics nor defined in terms of the generated audio. The temporal-smoothing procedure is standard linear interpolation (Eqs. 1 and 3) presented with new terminology, but it is a preprocessing step rather than a renamed known result carrying the paper's central claim. In summary, the paper's core derivation and comparisons are self-contained against external benchmarks; the identified weaknesses are experimental-design confounds, not circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim relies on standard pretrained models and manual hyperparameters; no new physical entities are introduced.

free parameters (4)
  • low-rank projection dimensions d1, d2 = 12
    Hand-chosen in Section 4.2; controls the capacity of the visual adaptors.
  • training epochs = 40
    Stopped at 40 to prevent overfitting (Section 4.2); not tuned by validation.
  • video sampling rate for MARLIN features = 5 fps
    Chosen by resampling to 80 fps then striding 16 frames (Section 3.1.1); affects temporal granularity.
  • initial learning rate = 1e-02
    Reported as training hyperparameter in Section 4.2.
assumptions (4)
  • domain assumption MusicGen is a strong, controllable text-to-music base model
    The entire method fine-tunes MusicGen's decoder; if MusicGen were weak, the improvements would not transfer.
  • domain assumption MARLIN, Synchformer, and RAFT features capture expressive information needed for music alignment
    The paper relies on these pretrained encoders without validating their sufficiency for music-related cues.
  • domain assumption Volunteer recordings are causally and temporally aligned with the music
    The dataset was collected by having people listen to music and move; the paper assumes these movements encode musical structure.
  • domain assumption SALMONN captions are faithful text descriptions of the audio
    Captions generated by SALMONN are used as prompts for all text-conditioned models; if they are inaccurate or contain leakage, results are affected.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EXPOTION: Facial Expression and Motion Control for Multimodal Music Generation." pith.science (2026). https://pith.science/paper/OYPEZI54

@misc{pith2026250704955,
  author       = {Pith},
  title        = {Pith review of: EXPOTION: Facial Expression and Motion Control for Multimodal Music Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OYPEZI54}},
  note         = {Machine review of arXiv:2507.04955}
}
read the original abstract

We propose Expotion (Facial Expression and Motion Control for Multimodal Music Generation), a generative model leveraging multimodal visual controls - specifically, human facial expressions and upper-body motion - as well as text prompts to produce expressive and temporally accurate music. We adopt parameter-efficient fine-tuning (PEFT) on the pretrained text-to-music generation model, enabling fine-grained adaptation to the multimodal controls using a small dataset. To ensure precise synchronization between video and music, we introduce a temporal smoothing strategy to align multiple modalities. Experiments demonstrate that integrating visual features alongside textual descriptions enhances the overall quality of generated music in terms of musicality, creativity, beat-tempo consistency, temporal alignment with the video, and text adherence, surpassing both proposed baselines and existing state-of-the-art video-to-music generation models. Additionally, we introduce a novel dataset consisting of 7 hours of synchronized video recordings capturing expressive facial and upper-body gestures aligned with corresponding music, providing significant potential for future research in multimodal and interactive music generation.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 45 canonical work pages

  1. [1]

    EXPOTION: Facial Expression and Motion Control for Multimodal Music Generation

    INTRODUCTION Music generation models have become increasingly versa- tile and interactive, capable of integrating control signals from various modalities, including audio, text, and symbolic representations (such as MIDI or musical scores). These conditions act as control to guide the model toward produc- ing more precise, and targeted outputs aligned wit...

    work page Pith review arXiv 2025

  2. [2]

    Visual and Motion-Based Control Early interactive sys- tems mapped facial or bodily features directly to sound

    RELATED WORK We review three key paradigms for controllable music generation: visual and motion-based control, textual and symbolic conditioning, and training and adaptation strategies. Visual and Motion-Based Control Early interactive sys- tems mapped facial or bodily features directly to sound. Valenti et al.’s Sonify Your Face modulated audio via Bayes...

    work page

  3. [3]

    We froze the parame- ters of the vanilla Musicgen during training to preserve its text understanding ability

    METHODOLOGY Our approach consists of of 1) a joint embedding encoder to integrate temporally aligned video-based controls, and 2) a condition adaptor to fine-tune MusicGen by incorporating the learned joint visual embeddings. We froze the parame- ters of the vanilla Musicgen during training to preserve its text understanding ability. 3.1 Joint Visual Embe...

    work page

  4. [4]

    We recruited volunteers to record their facial expressions and upper body movements while listening to 30-second audio clips

    EXPERIMENTS 4.1 Dataset Due to the lack of sufficient paired video-audio data with clear facial features, we curated our own dataset by collect- ing the data manually. We recruited volunteers to record their facial expressions and upper body movements while listening to 30-second audio clips. Before starting the recording, the volunteers were asked to lis...

    work page

  5. [5]

    RESULTS 5.1 Objective Evaluations General Music Quality. The results in Table 1 demon- strate that models incorporating motion information, par- ticularly those using the Syncformer features with gener- ated prompts, consistently outperform others across gen- eral music quality metrics, especially the baselines. This configuration performs best overall, w...

    work page

  6. [6]

    CONCLUSION Expotion demonstrates that visual cues—specifically, body movements and facial expressions—can effectively serve as expressive controls for music generation. By leverag- ing a pretrained text-to-music model [19] and applying parameter-efficient fine-tuning, our approach achieves no- table improvements from the original text-only condition- ing ...

    work page

  7. [7]

    Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthe- sis,

    H. K. Cheng, M. Ishii, A. Hayakawa, T. Shibuya, A. Schwing, and Y . Mitsufuji, “Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthe- sis,” CVPR, 2025

    work page 2025

  8. [8]

    Diff-Foley: Syn- chronized Video-to-Audio Synthesis with Latent Dif- fusion Models,

    S. Luo, C. Yan, C. Hu, and H. Zhao, “Diff-Foley: Syn- chronized Video-to-Audio Synthesis with Latent Dif- fusion Models,” in Advances in Neural Information Processing Systems (NeurIPS), 2023

    work page 2023

Show all 54 references
  1. [9]

    Temporally Aligned Audio for Video with Autoregression,

    I. Viertola, V . Iashin, and E. Rahtu, “Temporally Aligned Audio for Video with Autoregression,”IEEE International Conference on Acoustics, Speech and Sig- nal Processing (ICASSP), 2025

    2025

  2. [10]

    FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds,

    Y . Zhang, Y . Gu, Y . Zeng, Z. Xing, Y . Wang, Z. Wu, and K. Chen, “FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds,”arXiv preprint arXiv:2407.01494, 2024

    2024 arXiv

  3. [11]

    Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matching,

    Y . Wang, W. Guo, R. Huang, J. Huang, Z. Wang, F. You, R. Li, and Z. Zhao, “Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matching,” in Advances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  4. [12]

    MM-LDM: Multi-Modal Latent Diffusion Model for Sounding Video Generation,

    M. Sun, W. Wang, Y . Qiao, J. Sun, Z. Qin, L. Guo, X. Zhu, and J. Liu, “MM-LDM: Multi-Modal Latent Diffusion Model for Sounding Video Generation,” in Proceedings of the 32nd ACM International Conference on Multimedia (ACM MM), 2024

    2024

  5. [13]

    Visual Echoes: A Sim- ple Unified Transformer for Audio-Visual Generation,

    S. Yang, Z. Zhong, M. Zhao, S. Takahashi, M. Ishii, T. Shibuya, and Y . Mitsufuji, “Visual Echoes: A Sim- ple Unified Transformer for Audio-Visual Generation,” arXiv preprint arXiv:2405.14598, 2024

    2024 arXiv

  6. [14]

    Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event Condition For Foley Sound,

    J. Lee, J. Im, D. Kim, and J. Nam, “Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event Condition For Foley Sound,” arXiv preprint arXiv:2408.11915, 2024

    2024

  7. [15]

    Tell what you hear from what you see: Video to audio generation through text,

    X. Liu, K. Su, and E. Shlizerman, “Tell what you hear from what you see: Video to audio generation through text,” arXiv preprint arXiv:2402.05937, 2024

    2024 arXiv

  8. [16]

    Text-to-audio gen- eration synchronized with videos,

    S. Mo, J. Shi, and Y . Tian, “Text-to-audio gen- eration synchronized with videos,” arXiv preprint arXiv:2403.07055, 2024

    2024 arXiv

  9. [17]

    Conditional Generation of Audio from Video via Fo- ley Analogies,

    Y . Du, Z. Chen, J. Salamon, B. Russell, and A. Owens, “Conditional Generation of Audio from Video via Fo- ley Analogies,” in Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 2426–2436

    2023

  10. [18]

    MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic Synchronization,

    R. Li, S. Zheng, X. Cheng, Z. Zhang, S. Ji, and Z. Zhao, “MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic Synchronization,” arXiv preprint arXiv:2410.12957, 2024

    2024 arXiv

  11. [19]

    VMAs: Video-to-Music Generation via Semantic Alignment in Web Music Videos,

    Y .-B. Lin, Y . Tian, L. Yang, G. Bertasius, and H. Wang, “VMAs: Video-to-Music Generation via Semantic Alignment in Web Music Videos,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2025

    2025

  12. [20]

    Video2Music: Suitable Music Generation from Videos using an Affec- tive Multimodal Transformer model,

    J. Kang, S. Poria, and D. Herremans, “Video2Music: Suitable Music Generation from Videos using an Affec- tive Multimodal Transformer model,” Expert Systems with Applications, vol. 249, p. 123640, 2024

    2024

  13. [21]

    V2Meow: Meowing to the Visual Beat via Video-to-Music Generation,

    K. Su, J. Y . Li, Q. Huang, D. Kuzmin, J. Lee, C. Don- ahue, F. Sha, A. Jansen, Y . Wang, M. Verzetti, and T. I. Denk, “V2Meow: Meowing to the Visual Beat via Video-to-Music Generation,” in Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2024, pp. 4952–4960

    2024

  14. [22]

    DanceCom- poser: Dance-to-Music Generation Using a Progressive Conditional Music Generator,

    X. Liang, W. Li, L. Huang, and C. Gao, “DanceCom- poser: Dance-to-Music Generation Using a Progressive Conditional Music Generator,” IEEE Transactions on Multimedia, 2024

    2024

  15. [23]

    Discrete Contrastive Diffusion for Cross- Modal Music and Image Generation,

    Y . Zhu, Y . Wu, K. Olszewski, J. Ren, S. Tulyakov, and Y . Yan, “Discrete Contrastive Diffusion for Cross- Modal Music and Image Generation,” in Proceedings of the International Conference on Learning Represen- tations (ICLR), 2023

    2023

  16. [24]

    Long- Term Rhythmic Video Soundtracker,

    J. Yu, Y . Wang, X. Chen, X. Sun, and Y . Qiao, “Long- Term Rhythmic Video Soundtracker,” inProceedings of the 40th International Conference on Machine Learning (ICML), 2023

    2023

  17. [25]

    Simple and control- lable music generation,

    J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Syn- naeve, Y . Adi, and A. Défossez, “Simple and control- lable music generation,” in Advances in Neural Infor- mation Processing Systems (NeurIPS), 2024

    2024

  18. [26]

    Content-based controls for music large language modeling,

    L. Lin, G. Xia, J. Jiang, and Y . Zhang, “Content-based controls for music large language modeling,” in Proceedings of the International Society for Music Information Retrieval Conference (ISMIR) , 2024. [Online]. Available: https://arxiv.org/abs/2310.17162

    2024 arXiv

  19. [27]

    Sketch2sound: Controllable audio generation via time-varying signals and sonic imita- tions,

    H. F. Garcia, O. Nieto, J. Salamon, B. Pardo, and P. Seetharaman, “Sketch2sound: Controllable audio generation via time-varying signals and sonic imita- tions,” arXiv preprint arXiv:2402.13253, 2024

    2024 arXiv

  20. [28]

    Audioclip: Extending clip to image, text and audio,

    A. Guzhov, F. Raue, J. Hees, and A. Dengel, “Audioclip: Extending clip to image, text and audio,” in ICASSP 2022 - IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 976–980

    2022

  21. [29]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of Machine Learning Research, vol. 21, no. 140, pp. 1–67, 2020

    2020

  22. [30]

    Vidmuse: A simple video-to- music generation framework with long-short-term mod- eling,

    Z. Tian, Z. Liu, R. Yuan, J. Pan, Q. Liu, X. Tan, Q. Chen, W. Xue, and Y . Guo, “Vidmuse: A simple video-to- music generation framework with long-short-term mod- eling,” CVPR, 2025

    2025

  23. [31]

    Sonify Your Face: Facial Expressions for Sound Generation,

    R. Valenti, A. Jaimes, and N. Sebe, “Sonify Your Face: Facial Expressions for Sound Generation,” in Proceed- ings of the 2010 ACM Multimedia Workshop on Visual Media Interpretation and Understanding (VMIU), 2010

    2010

  24. [32]

    Movement to emotions to music: using whole body emotional expression as an interaction for electronic music genera- tion,

    A. Clay, N. Couture, E. Decarsin, M. Desainte- Catherine, P.-H. Vulliard, and J. Larralde, “Movement to emotions to music: using whole body emotional expression as an interaction for electronic music genera- tion,” inProceedings of the International Conference on New Interface...

    2012

  25. [33]

    D2MNet for music generation jointly driven by facial expressions and dance movements,

    J. Huang, X. Huang, L. Yang, and Z. Tao, “D2MNet for music generation jointly driven by facial expressions and dance movements,” Array, 2024

    2024

  26. [34]

    Deep- Tunes: Music Generation based on Facial Emotions using Deep Learning,

    V . P, P. A, S. Vasist, S. Rao, and K. Srinivas, “Deep- Tunes: Music Generation based on Facial Emotions using Deep Learning,” in International Conference on Intelligent Computing and Technology (I2CT), 2022

    2022

  27. [35]

    A Contin- uous Emotional Music Generation System Based on Facial Expressions,

    J. Huang, X. Huang, L. Yang, and Z. Tao, “A Contin- uous Emotional Music Generation System Based on Facial Expressions,” inProceedings of the International Conference on Intelligent Data (ICID), 2022

    2022

  28. [36]

    Musiclm: Generating music from text,

    A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi, M. Sharifi, N. Zeghidour, and C. Frank, “Musiclm: Generating music from text,” 2023

    2023

  29. [37]

    Audio set: An on- tology and human-labeled dataset for audio events,

    J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, and R. C. Moore, “Audio set: An on- tology and human-labeled dataset for audio events,” in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2017, pp. 776–780

    2017

  30. [38]

    Vg- gsound: A large-scale audio-visual dataset,

    H. Chen, W. Xie, A. Vedaldi, and A. Zisserman, “Vg- gsound: A large-scale audio-visual dataset,” in Proceed- ings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 721– 725

    2020

  31. [39]

    Arrange, inpaint, and refine: Steerable long-term music audio generation and editing via content-based controls,

    L. Lin, G. Xia, Y . Zhang, and J. Jiang, “Arrange, inpaint, and refine: Steerable long-term music audio generation and editing via content-based controls,” in Proceedings of the 32nd International Joint Conference on Artificial Intelligence (IJCAI). IJCAI, 2024

    2024

  32. [40]

    Marlin: Masked autoencoder for facial video representation learning,

    Z. Cai, S. Ghosh, K. Stefanov, A. Dhall, J. Cai, H. Rezatofighi, R. Haffari, and M. Hayat, “Marlin: Masked autoencoder for facial video representation learning,” in CVPR. CVPR, 2023

    2023

  33. [41]

    Synch- former: Efficient synchronization from sparse cues,

    V . Iashin, W. Xie, E. Rahtu, and A. Zisserman, “Synch- former: Efficient synchronization from sparse cues,” in ICASSP 2024 – 2024 IEEE Int. Conf. on Acoustics, Speech and Signal Processing. IEEE, 2024

    2024

  34. [42]

    Raft: Recurrent all-pairs field transforms for optical flow,

    Z. Teed and J. Deng, “Raft: Recurrent all-pairs field transforms for optical flow,” in European Conference on Computer Vision (ECCV). Springer, 2020, pp. 402– 419

    2020

  35. [43]

    Salmonn: Towards generic hear- ing abilities for large language models,

    C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang, “Salmonn: Towards generic hear- ing abilities for large language models,” in Proceedings of the International Conference on Learning Represen- tations (ICLR), 2024

    2024

  36. [44]

    High fidelity neural audio compression,

    A. Défossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,” 2022. [Online]. Available: https://arxiv.org/abs/2210.13438

    2022 arXiv

  37. [45]

    Video2music: Suitable music generation from videos using an affective multimodal transformer model,

    J. Kang, S. Poria, and D. Herremans, “Video2music: Suitable music generation from videos using an affective multimodal transformer model,” Expert Systems with Applications , vol. 249, p. 123640, Sep

  38. [46]

    Languagebind: Extending video-language pretraining to n-modality by language- based semantic alignment,

    B. Zhu, B. Lin, M. Ning, Y . Yan, J. Cui, H. Wang, Y . Pang, W. Jiang, J. Zhang, Z. Li, W. Zhang, Z. Li, W. Liu, and L. Yuan, “Languagebind: Extending video-language pretraining to n-modality by language- based semantic alignment,” 2024. [Online]. Available: https://arxiv.org/...

    2024 arXiv

  39. [47]

    Frechet audio distance: A reference-free metric for evaluating music enhancement algorithms,

    K. Kilgour, R. Clark, K. Simonyan, and M. Sharifi, “Frechet audio distance: A reference-free metric for evaluating music enhancement algorithms,” in Inter- speech, 2019, pp. 2350–2354

    2019

  40. [48]

    Cnn architectures for large-scale audio classification,

    S. Hershey, S. Chaudhuri, D. P. W. Ellis, J. F. Gemmeke, A. Jansen, R. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, M. Slaney, R. J. Weiss, and K. Wilson, “Cnn architectures for large-scale audio classification,” in IEEE International Conference on Acoustics, Speech ...

    2017

  41. [49]

    Panns: Large-scale pretrained audio neural networks for audio pattern recognition,

    Q. Kong, Y . Cao, T. Iqbal, Y . Wang, W. Wang, and M. D. Plumbley, “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,” IEEE/ACM Transactions on Audio, Speech, and Language Process- ing, vol. 28, pp. 2880–2894, 2020

    2020

  42. [50]

    Efficient training of audio transform- ers with patchout,

    K. Koutini, H. Eghbal-zadeh, M. Widrich, J. Brandstet- ter, A. Thakur, V . Berenz, T. Mörwald, S. Hochreiter, and B. Hammer, “Efficient training of audio transform- ers with patchout,” in Proceedings of the IEEE Inter- national Conference on Acoustics, Speech and Signal Proces...

    2022

  43. [51]

    Improved techniques for training gans,

    T. Salimans, I. Goodfellow, W. Zaremba, V . Cheung, A. Radford, and X. Chen, “Improved techniques for training gans,” in Advances in Neural Information Pro- cessing Systems (NeurIPS), 2016, pp. 2234–2242

    2016

  44. [52]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,

    Y . Wu, K. Chen, T. Zhang, Y . Hui, M. Nezhurina, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Sign...

  45. [2023]

    Available: https://arxiv.org/abs/2211

    [Online]. Available: https://arxiv.org/abs/2211. 06687

  46. [2024]

    Available: http://dx.doi.org/10.1016/j

    [Online]. Available: http://dx.doi.org/10.1016/j. eswa.2024.123640

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.