Pith. sign in

REVIEW 2 major objections 5 minor 46 references

Diffusion based Text-to-Music Generation with Global and Local Text based Conditioning

T0 review · 2 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Global plus local text conditioning improves text adherence in a diffusion text-to-music model, and a mean-pooled T5 alone achieves better audio quality at lower cost.

desk verdict A useful empirical map of global/local text conditioning for TTM, but the headline CLAP claim is not properly isolated from architecture and parameter count. read the letter →

arxiv 2501.14680 v2 pith:PQHVUYBL submitted 2025-01-24 eess.AS cs.SD

Jisi Zhang , Pablo Peso Parada , Md Asif Jalal , Karthikeyan Saravanan This is my paper · ORCID
classification eess.AScs.SD
keywords text-to-musicgenerationlatentdiffusionglobaltextembeddinglocalmeanpoolingself-attentionCLAPT5
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Text-to-music diffusion models rely on text encoders to condition generation, but the type of text embedding matters more than the number of encoders. This paper proposes a latent diffusion UNet conditioned on both a local word-level embedding from T5 (via cross-attention) and a global sentence-level embedding from CLAP (via FiLM), and reports that this pairing improves text adherence over T5 alone: KL=1.47 versus 1.54. The paper then shows that a global embedding extracted from T5's own token embeddings by simple mean pooling yields FAD=1.89 and KL=1.51, better audio quality and nearly the same adherence as the dual-encoder setup (FAD=1.94, KL=1.47), while saving about 124 million parameters. The claim is that fusing global and local text representations helps, and that the global representation can come from the same frozen language model.

What carries the argument

The load-bearing mechanism is the placement of the two conditioning signals. The global text embedding $G_y$ is injected through Feature-wise Linear Modulation (FiLM), which applies a learned scale and shift to the UNet's intermediate features, alongside the time embedding; the local text embedding $F_y$ enters through standard cross-attention. The paper's proposed mean pooling, $G_{\text{mean}} = \frac{1}{M}\sum_{i=1}^{M} F^{(i)}_y$, derives the global embedding from the same T5 token embeddings, and self-attention pooling $G_{\text{SAP}} = \mathrm{Softmax}(W_{\text{SAP}} F_y^T) F_y$ is a trainable alternative. The diffusion model is trained with the v-objective and classifier-free guidance with 10 percent unconditional dropout, which lets a single model produce both conditional and unconditional outputs.

What would settle it

Re-train the T5-local baseline with the same 606M parameter budget as the CLAP+T5 model (for example, using a larger T5) across at least three seeds; if that baseline reaches FAD at or below 1.89 and KL at or below 1.47, the reported advantages of global conditioning and mean pooling would not be established.

Watch

Extended reading notes

Core claim

The central claim is that conditioning a latent diffusion UNet on both local and global text representations improves text-to-music generation, and that the global representation can be obtained without a separate cross-modal encoder. The paper reports that CLAP global plus T5 local conditioning achieves KL=1.47 versus KL=1.54 with T5 local embeddings alone, showing better text adherence. It also reports that mean pooling of the T5 token embeddings, which adds no parameters, achieves FAD=1.89 and KL=1.51, whereas the dual-encoder model achieves FAD=1.94 and KL=1.47. In the paper's telling, the discovery is that a parameter-free pooling operation on a frozen T5 encoder produces a global text embedding that matches or exceeds the quality of a dedicated CLAP encoder.

Load-bearing premise

The paper assumes the single-run differences in FAD and KL arise from the conditioning method, even though the compared systems differ in parameter count and show no error bars, so the gains could instead come from extra capacity or training luck.

Editorial extensions

If this is right

  • If the results generalize, text-to-music systems can drop the CLAP encoder and use a single frozen T5 with mean pooling, cutting about 124 million parameters without losing audio quality.
  • The FiLM-for-global and cross-attention-for-local split is a reusable architecture pattern for conditioning diffusion models in other modalities.
  • The failure of Sentence-T5 and SimCSE as global conditioners suggests the global embedding must be aligned with the generation task, not just semantically strong.
  • Scaling the language model (FLANT5-large) further improves the pooled approach, pointing to language-model capacity as the main lever for better music generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported FAD and KL gaps come from single runs without error bars, and compared configurations differ in parameter count; a matched-parameter, multi-seed re-run would be needed to confirm that the improvements arise from the conditioning mechanism rather than capacity or variance.
  • A natural testable extension is to fine-tune the text encoder on music-related instructions and pair it with mean pooling; since FLANT5-large already outperforms T5-base under mean pooling, such tuning may close the remaining KL gap entirely.
  • The self-attention pooling collapse the authors observed (attention concentrating on a few frequent tokens) could be addressed by isotropic regularization or principal-component whitening; a controlled comparison would show whether SAP becomes competitive once that collapse is prevented.
  • The evaluation is on instrument-only MusicCaps audio; the conclusions may shift for vocal music or other domains because FAD and KL are computed on a specific feature distribution.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes a diffusion-based text-to-music generation system in which a UNet is conditioned on both local text embeddings from T5 (through cross-attention) and global text embeddings either from CLAP or from pooling the T5 local embeddings (through FiLM). The authors evaluate the proposed architecture against retrained AudioLDM baselines and publicly available systems on MusicCaps, reporting FAD and KL scores. They claim that adding CLAP global embeddings to T5 local embeddings improves text adherence (KL 1.47 vs 1.54) and that a proposed mean-pooling variant is more parameter-efficient while achieving competitive or slightly better FAD. The paper includes ablations over global text conditioners (CLAP, Sentence-T5, SimCSE, mean pooling, self-attention pooling) and over local text encoders (T5-base, FLANT5-large).

Significance. If the central attribution claim is confirmed, the paper offers a useful and practical study: it retrains AudioLDM baselines on the same data, reports parameter counts for each configuration, and includes a focused ablation over global conditioners. The honest reporting of the self-attention pooling failure is a further strength. The proposed mean-pooling variant, if it holds with proper uncertainty quantification, would be a simple parameter-efficient way to obtain global text conditioning. However, the headline claim that CLAP's semantic content is responsible for the text-adherence gain is not yet supported by the experiments, because the comparison baseline differs in both architecture and parameter count. The reported differences are also small and are presented without error bars or repeated runs.

major comments (2)
  1. [Section IV, Table II; abstract] The central claim that adding CLAP global embeddings to T5 local embeddings improves text adherence (KL 1.47 vs 1.54) is not a controlled ablation. The row labelled '- T5-base' in Table II corresponds to 'AudioLDM (retrain) - T5-base' from Table I, which is a different architecture: it lacks the proposed FiLM-based global-conditioning branch and has 482M parameters versus 606M for the CLAP+T5 model. The KL gain could therefore be caused by the additional 124M parameters, the extra conditioning pathway, or the architectural differences between AudioLDM and the proposed LDM, rather than by the semantic content of the CLAP global embedding. The authors should include a matched-capacity T5-only variant of the proposed LDM architecture, ideally with an uninformative global vector of the same dimension injected through the same FiLM path, to attribute the improvement to CLAP semantics.
  2. [Tables I-III and Section III-A] All quantitative claims rest on a single run and a single checkpoint. The checkpoint is selected on the validation set as the one with the best FAD and KL scores, and no confidence intervals, error bars, multiple seeds, or repeated inference draws are reported. The differences interpreted as meaningful are small (e.g., KL 1.47 vs 1.51, FAD 1.89 vs 1.94), and such differences could plausibly arise from run-to-run variance or selection effects. The paper should report at least multiple training runs with standard deviations, or, failing that, bootstrap confidence intervals over evaluation draws, and should disclose whether the same checkpoint is used for all metrics.
minor comments (5)
  1. [Section II (Conditioning)] The classifier-free guidance formula appears to have the conditional and unconditional terms reversed: the equation gives \hat{v} = \omega v_\theta(z_t,t,\emptyset,\emptyset) + (1-\omega) v_\theta(z_t,t,G_y,F_y), which with \omega = 9 gives a negative coefficient to the conditional prediction. If this is a typo, the corrected formula should be provided; if implemented as written, it is anti-guidance rather than standard classifier-free guidance.
  2. [Section II, Eqs. (2)-(3)] The pooling equations are underspecified: the summation index i over F_y^{(i)} is not defined in terms of the number of tokens M, and the dimensions of G_y and F_y are not stated. Please clarify the notation for reproducibility.
  3. [Section II, Eq. (4)] The v-objective in Eq. (4) uses \alpha_t and \sigma_t without defining the noise schedule or their dependence on t. A brief definition or reference to a specific schedule would help.
  4. [Section IV-A] The text contains a typo and an incomplete sentence: 'mapping emebeddings to an isotropic distribution' should read 'embeddings', and the sentence beginning 'This may require adding regularisation...' should be completed.
  5. [Section III-A] The validation procedure on MusicBench is only described as checkpoint selection based on 'best FAD and KL scores'. Please specify the number of generated samples, prompt set, and whether FAD and KL are computed on the same audio set that is later used for testing.

Circularity Check

0 steps flagged · score 1.0 of 10

No circularity found: the reported FAD/KL gains are external benchmark measurements, not quantities defined by the model's fitted parameters; the only caveat is a confounded baseline, which is a correctness concern, not circular reasoning.

full rationale

This is a measurement paper. The claimed results (KL=1.47 vs 1.54, FAD=1.89 vs 1.94) are computed by the AudioLDM evaluation toolkit on MusicCaps against a processed reference set, so the reported scores are not defined in terms of the diffusion model's fitted parameters or the proposed pooling equations. Mean pooling is a deterministic average of T5 local embeddings, but the claimed benefit is an observed empirical outcome of injecting that summary into FiLM, not an identity: Equation (2) defines Gmean but does not by itself predict FAD/KL, and the comparison with the T5-only row is an external measurement. No load-bearing self-citation appears: the cited prior work (AudioLDM, T5, CLAP, etc.) is external and independently published; no uniqueness theorem or ansatz is imported from the present authors' earlier work. The mildest issue is that the T5-only baseline in Tables I-II is an AudioLDM retrain with T5-base rather than the proposed LDM architecture without CLAP, so the abstract's attribution of the KL gain to CLAP semantics is partly confounded with parameter count and architecture; that is a model-comparison/ablation weakness, not a circular derivation. Checkpoint selection by validation FAD/KL is standard model selection, not a fitted input renamed as a prediction.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper is an empirical system paper, not a derivation. Its central claims rest on standard diffusion equations, frozen pretrained text encoders, automatic metrics, and the particular training data. The main burden is experimental control, which is weakened by no error bars and unmatched parameter counts.

free parameters (4)
  • Classifier-free guidance scale omega = 9.0
    Set by hand in Section III.B and used for all generations; FAD/KL values in Tables I-II depend on this value and it was not swept.
  • Conditioning dropout probability = 0.1
    Random discard probability for global and local embeddings during CFG training, Section II.C; chosen by hand and affects the unconditional term in generation.
  • Self-attention pooling weight vector W_SAP = not reported (trained)
    The trainable vector in Eq. (3) used to compute attention weights for the SAP global embedding; the paper reports it collapses to a narrow token range, so the method underperforms.
  • UNet and conditioning projection weights = unknown (trained on MTG/FMA/pond5)
    Approximately 300M diffusion UNet parameters plus the W_Q, W_K, W_V projection matrices in Eq. (1) are fit to the training data; comparisons therefore include capacity differences, e.g. 606M vs 482M in Table II.
assumptions (5)
  • standard math The latent diffusion v-objective (Eq. 5) is an unbiased training objective for the reverse process.
    Invoked in Section II.A-C via AudioLDM-style latent diffusion and v-prediction; the paper does not re-derive it.
  • domain assumption Frozen T5 local embeddings and CLAP global embeddings provide complementary information for music generation.
    Central premise of the conditioning mechanism in Sections II.B-II.C; if the embeddings were redundant or misaligned, the reported gains would not follow.
  • domain assumption FAD and KL computed with the AudioLDM evaluation toolkit on the HT-Demucs processed MusicCaps set faithfully measure generation quality and text adherence.
    All reported comparisons in Tables I-III rest on these metrics; no human listening test or confidence intervals are provided.
  • domain assumption Selecting checkpoints by best FAD/KL on MusicBench every 10k steps transfers to MusicCaps.
    Section III.A states validation is performed on MusicBench; the final numbers are on MusicCaps, so the selected checkpoint is not fully held out.
  • domain assumption HT-Demucs instrumental-only filtering does not remove information relevant to the text prompts.
    Section III.A filters all training and evaluation audio to instrumental tracks, so models cannot generate vocals and evaluations may not reflect full MusicCaps captions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Diffusion based Text-to-Music Generation with Global and Local Text based Conditioning." pith.science (2026). https://pith.science/paper/PQHVUYBL

@misc{pith2026250114680,
  author       = {Pith},
  title        = {Pith review of: Diffusion based Text-to-Music Generation with Global and Local Text based Conditioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PQHVUYBL}},
  note         = {Machine review of arXiv:2501.14680}
}
read the original abstract

Diffusion based Text-To-Music (TTM) models generate music corresponding to text descriptions. Typically UNet based diffusion models condition on text embeddings generated from a pre-trained large language model or from a cross-modality audio-language representation model. This work proposes a diffusion based TTM, in which the UNet is conditioned on both (i) a uni-modal language model (e.g., T5) via cross-attention and (ii) a cross-modal audio-language representation model (e.g., CLAP) via Feature-wise Linear Modulation (FiLM). The diffusion model is trained to exploit both a local text representation from the T5 and a global representation from the CLAP. Furthermore, we propose modifications that extract both global and local representations from the T5 through pooling mechanisms that we call mean pooling and self-attention pooling. This approach mitigates the need for an additional encoder (e.g., CLAP) to extract a global representation, thereby reducing the number of model parameters. Our results show that incorporating the CLAP global embeddings to the T5 local embeddings enhances text adherence (KL=1.47) compared to a baseline model solely relying on the T5 local embeddings (KL=1.54). Alternatively, extracting global text embeddings directly from the T5 local embeddings through the proposed mean pooling approach yields superior generation quality (FAD=1.89) while exhibiting marginally inferior text adherence (KL=1.51) against the model conditioned on both CLAP and T5 text embeddings (FAD=1.94 and KL=1.47). Our proposed solution is not only efficient but also compact in terms of the number of parameters required.

Figures

Figures reproduced from arXiv: 2501.14680 by the authors.

Figure 1
Figure 1. Overview of our text-to-music diffusion model architecture conditioned over multiple text encoders. (a) A continuous latent space [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 31 canonical work pages

  1. [1]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in neural information processing systems , vol. 33, pp. 6840– 6851, 2020

    2020

  2. [2]

    Attention is all you need,

    A. Vaswani, N. M. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Neural Information Processing Systems , 2017

    2017

  3. [3]

    High-resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10674–10685, 2021

    work page 2022

  4. [4]

    AudioLDM: Text-to-audio generation with latent diffusion models,

    H. Liu, Z. Chen, Y . Yuan, X. Mei, X. Liu, D. P. Mandic, W. Wang, and M. . Plumbley, “AudioLDM: Text-to-audio generation with latent diffusion models,” in International Conference on Machine Learning , 2023

    work page 2023

  5. [5]

    AudioLDM 2: Learning holistic audio generation with self-supervised pretraining,

    H. Liu, Y . Yuan, X. Liu, X. Mei, Q. Kong, Q. Tian, Y . Wang, W. Wang, Y . Wang, and M. D. Plumbley, “AudioLDM 2: Learning holistic audio generation with self-supervised pretraining,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 2871–2883, 2024

    work page 2024

  6. [6]

    Fast timing- conditioned latent audio diffusion,

    Z. Evans, C. Carr, J. Taylor, S. H. Hawley, and J. Pons, “Fast timing- conditioned latent audio diffusion,” ArXiv, vol. abs/2402.04825, 2024

    arXiv 2024

  7. [7]

    Grad- TTS: A diffusion probabilistic model for text-to-speech,

    V . Popov, I. V ovk, V . Gogoryan, T. Sadekova, and M. A. Kudinov, “Grad- TTS: A diffusion probabilistic model for text-to-speech,” inInternational Conference on Machine Learning , 2021

    work page 2021

  8. [8]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” in International Conference on Machine Learning , 2021

    work page 2021

Show all 46 references
  1. [9]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. M. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” J. Mach. Learn. Res. , vol. 21, pp. 140:1–140:67, 2019

    2019

  2. [10]

    MuLan: A joint embedding of music audio and natural language,

    Q. Huang, A. Jansen, J. Lee, R. Ganti, J. Y . Li, and D. P. W. Ellis, “MuLan: A joint embedding of music audio and natural language,” in International Society for Music Information Retrieval Conference , 2022

    2022

  3. [11]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,

    Y . Wu*, K. Chen*, T. Zhang*, Y . Hui*, T. Berg-Kirkpatrick, and S. Dub- nov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023

    2023

  4. [12]

    BERT: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in North American Chapter of the Association for Computational Linguistics , 2019

    2019

  5. [13]

    MusicLM: Generating music from text,

    A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi, M. Sharifi, N. Zeghi- dour, and C. H. Frank, “MusicLM: Generating music from text,” ArXiv, vol. abs/2301.11325, 2023

    2023 arXiv

  6. [14]

    Audio-Text models do not yet leverage natural language,

    H.-H. Wu, O. Nieto, J. P. Bello, and J. Salamon, “Audio-Text models do not yet leverage natural language,” ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5, 2023

    2023

  7. [15]

    Simple and controllable music generation,

    J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y . Adi, and A. D ´efossez, “Simple and controllable music generation,” Advances in Neural Information Processing Systems , vol. 36, 2023

    2023

  8. [16]

    T-CLAP: Temporal-enhanced contrastive language-audio pretraining,

    Y . Yuan, Z. Chen, X. Liu, H. Liu, X. Xu, D. Jia, Y . Chen, M. . Plumbley, and W. Wang, “T-CLAP: Temporal-enhanced contrastive language-audio pretraining,” ArXiv, vol. abs/2404.17806, 2024

    2024 arXiv

  9. [17]

    eDiff-I: Text-to-image diffusion models with an ensemble of expert denoisers,

    Y . Balaji, S. Nah, X. Huang, A. Vahdat, J. Song, Q. Zhang, K. Kreis, M. Aittala, T. Aila, S. Laine, B. Catanzaro, T. Karras, and M.-Y . Liu, “eDiff-I: Text-to-image diffusion models with an ensemble of expert denoisers,” ArXiv, vol. abs/2211.01324, 2022

    2022 arXiv

  10. [18]

    Auffusion: Leveraging the power of diffusion and large language models for text-to-audio generation,

    J. Xue, Y . Deng, Y . Gao, and Y . Li, “Auffusion: Leveraging the power of diffusion and large language models for text-to-audio generation,” ArXiv, vol. abs/2401.01044, 2024

    2024 arXiv

  11. [19]

    Scaling instruction-finetuned language models,

    H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y . Tay, W. Fedus, Y . Li, X. Wang, M. Dehghani, S. Brahma, et al., “Scaling instruction-finetuned language models,” Journal of Machine Learning Research , vol. 25, no. 70, pp. 1–53, 2024

    2024

  12. [20]

    Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,

    J. Kong, J. Kim, and J. Bae, “Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,” Advances in neural information processing systems , vol. 33, pp. 17022–17033, 2020

    2020

  13. [21]

    Sentence-T5: Scalable sentence encoders from pre-trained text-to-text models,

    J. Ni, G. H. Abrego, N. Constant, J. Ma, K. Hall, D. Cer, and Y . Yang, “Sentence-T5: Scalable sentence encoders from pre-trained text-to-text models,” in Findings of the Association for Computational Linguistics: ACL 2022, pp. 1864–1874, 2022

    2022

  14. [22]

    RoBERTa: A robustly optimized bert pretraining approach,

    Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “RoBERTa: A robustly optimized bert pretraining approach,” ArXiv, vol. abs/1907.11692, 2019

    1907 arXiv

  15. [23]

    Whitening sentence representations for better semantics and faster retrieval,

    J. Su, J. Cao, W. Liu, and Y . Ou, “Whitening sentence representations for better semantics and faster retrieval,” ArXiv, vol. abs/2103.15316, 2021

    2021 arXiv

  16. [24]

    On the sentence embeddings from pre-trained language models,

    B. Li, H. Zhou, J. He, M. Wang, Y . Yang, and L. Li, “On the sentence embeddings from pre-trained language models,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9119–9130, 2020

    2020

  17. [25]

    SimCSE: Simple contrastive learning of sentence embeddings,

    T. Gao, X. Yao, and D. Chen, “SimCSE: Simple contrastive learning of sentence embeddings,” in Conference on Empirical Methods in Natural Language Processing, 2021

    2021

  18. [26]

    FiLM: Visual reasoning with a general conditioning layer,

    E. Perez, F. Strub, H. de Vries, V . Dumoulin, and A. C. Courville, “FiLM: Visual reasoning with a general conditioning layer,” in AAAI Conference on Artificial Intelligence , 2017

    2017

  19. [27]

    Self-attention encoding and pooling for speaker recognition,

    P. Safari, M. India, and J. Hernando, “Self-attention encoding and pooling for speaker recognition,” in Interspeech, 2020

    2020

  20. [28]

    Progressive distillation for fast sampling of diffusion models,

    T. Salimans and J. Ho, “Progressive distillation for fast sampling of diffusion models,” arXiv preprint arXiv:2202.00512 , 2022

    2022 arXiv

  21. [29]

    Variational diffusion models,

    D. Kingma, T. Salimans, B. Poole, and J. Ho, “Variational diffusion models,” Advances in neural information processing systems , vol. 34, pp. 21696–21707, 2021

    2021

  22. [30]

    Classifier-free diffusion guidance,

    J. Ho, “Classifier-free diffusion guidance,” ArXiv, vol. abs/2207.12598, 2022

    2022 arXiv

  23. [31]

    The MTG-Jamendo dataset for automatic music tagging,

    D. Bogdanov, M. Won, P. Tovstogan, A. Porter, and X. Serra, “The MTG-Jamendo dataset for automatic music tagging,” in International Conference on Machine Learning , 2019

    2019

  24. [32]

    FMA: A dataset for music analysis,

    M. Defferrard, K. Benzi, P. Vandergheynst, and X. Bresson, “FMA: A dataset for music analysis,” in International Society for Music Informa- tion Retrieval Conference , 2016

    2016

  25. [33]

    “Pond5.” https://www.pond5.com/, 2024

    2024

  26. [34]

    Hybrid transformers for music source separation,

    S. Rouard, F. Massa, and A. D ´efossez, “Hybrid transformers for music source separation,” in ICASSP 23, 2023

    2023

  27. [35]

    Mustango: Toward controllable text-to-music generation,

    J. Melechovsk ´y, Z. Guo, D. Ghosal, N. Majumder, D. Herremans, and S. Poria, “Mustango: Toward controllable text-to-music generation,” in North American Chapter of the Association for Computational Linguis- tics, 2023

    2023

  28. [36]

    AudioLDM training, finetuning, inference and evaluation

    “AudioLDM training, finetuning, inference and evaluation.” https://github.com/haoheliu/AudioLDM-training-finetuning, 2023

    2023

  29. [37]

    LAION-AI/CLAP

    “LAION-AI/CLAP.” https://github.com/LAION-AI/CLAP, 2023

    2023

  30. [38]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in International Conference on Learning Representations , 2017

    2017

  31. [39]

    Denoising diffusion implicit models,

    J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” in International Conference on Learning Representations , 2020

    2020

  32. [40]

    MusicLDM: Enhancing novelty in text-to-music generation using beat-synchronous mixup strategies,

    K. Chen, Y . Wu, H. Liu, M. Nezhurina, T. Berg-Kirkpatrick, and S. Dubnov, “MusicLDM: Enhancing novelty in text-to-music generation using beat-synchronous mixup strategies,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), ...

    2024

  33. [41]

    Stable Audio Open,

    Z. Evans, J. Parker, C. Carr, Z. Zukowski, J. Taylor, and J. Pons, “Stable Audio Open,” ArXiv, vol. abs/2407.14358, 2024

    2024 arXiv

  34. [42]

    Fr ´echet audio distance: A reference-free metric for evaluating music enhancement algorithms,

    D. Roblek, K. Kilgour, M. Sharifi, and M. Zuluaga, “Fr ´echet audio distance: A reference-free metric for evaluating music enhancement algorithms,” in Proc. Interspeech, pp. 2350–2354, 2019

    2019

  35. [43]

    Audio generation evaluation

    “Audio generation evaluation.” https://github.com/haoheliu/audioldm eval, 2023

    2023

  36. [44]

    CNN architectures for large-scale audio classification,

    S. Hershey, S. Chaudhuri, D. P. W. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, M. Slaney, R. J. Weiss, and K. W. Wilson, “CNN architectures for large-scale audio classification,” 2017 IEEE International Conference on Acoustics,...

    2017

  37. [45]

    PANNs: Large-scale pretrained audio neural networks for audio pattern recognition,

    Q. Kong, Y . Cao, T. Iqbal, Y . Wang, W. Wang, and M. D. Plumbley, “PANNs: Large-scale pretrained audio neural networks for audio pattern recognition,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 2880–2894, 2019

    2019

  38. [46]

    CoLLAT: On adding fine-grained audio understanding to language models using token-level locked-language tuning,

    D. A. R. Silva, S. Whitehead, C. T. Lengerich, and H. Leather, “CoLLAT: On adding fine-grained audio understanding to language models using token-level locked-language tuning,” in Neural Information Processing Systems, 2023

    2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.