Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

ETTA: Elucidating the Design Space of Text-to-Audio Models

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A systematic sweep of the text-to-audio design space finds synthetic captions are the dominant quality lever, and the resulting model, ETTA, rivals proprietary-data systems.

desk verdict The design-space study is the real contribution; the SOTA claim leans on unverified cross-paper numbers, so read it for the recipe, not the headline. read the letter →

arxiv 2412.19351 v2 pith:RELS3FEN submitted 2024-12-26 cs.SD cs.CLcs.LGeess.AS

Sang-gil Lee , Zhifeng Kong , Arushi Goel , Sungwon Kim , Rafael Valle , Bryan Catanzaro This is my paper · ORCID
classification cs.SDcs.CLcs.LGeess.AS
keywords text-to-audiogenerationsyntheticcaptionsAF-Syntheticdiffusiontransformerflowmatchinglatentclassifier-freeguidancecreativeaudio
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish which design choices actually drive text-to-audio (TTA) quality, by running a controlled sweep over training data, model architecture, training objective, and sampling. Its central empirical claim is that the single largest gain comes from replacing noisy or weakly related captions with 1.35 million synthetic captions that are strongly correlated with their audio, generated by an audio understanding model and filtered by a learned audio-text similarity score. A stabilised Diffusion Transformer implementation, an optimal-transport conditional flow matching objective, and logit-normal timestep sampling add further, smaller gains. The resulting model, ETTA, is claimed to be state of the art among models trained on publicly available data on AudioCaps and MusicCaps, and competitive with models trained on proprietary or licensed data. If correct, this identifies the bottleneck of current TTA systems in caption quality rather than in model capacity, and shows that a public-data system can approach closed-data performance.

What carries the argument

The object that carries the argument is AF-Synthetic, a dataset of 1.35 million audio-caption pairs built by having an audio understanding model propose captions for ten-second audio segments and then keeping, for each segment, the caption with the highest similarity to the audio as measured by a learned audio-text scoring model, rejecting pairs below a similarity threshold of 0.45 and keyword-filtering low-quality audio. This dataset is what converts the paper's other components into benchmark gains. The second mechanism is ETTA-DiT, a Diffusion Transformer whose changes, including adaptive layer norm applied to every block, a zero-initialised final projection, rotary position embeddings run in FP32, and dropout, make training more stable and improve convergence. The remaining machinery is the training objective, optimal-transport conditional flow matching with logit-normal timestep sampling, and an inference recipe chosen from Pareto sweeps over solver, number of function evaluations, and classifier-free guidance scale.

What would settle it

Run a matched evaluation in one shared harness: take ETTA and the leading baselines from Tables 2 and 3, and score all of them with identical 10-second preprocessing, identical implementations of the Fréchet distance, KL divergence, inception score, and CLAP metrics, and identical audio and text encoders. If several baselines match or beat ETTA's numbers under those conditions, the state-of-the-art-on-public-data claim would not survive.

Watch

Extended reading notes

Core claim

The paper's central claim is that ETTA, a latent diffusion model trained with an optimal-transport conditional flow matching objective on a synthetic-caption dataset, achieves state-of-the-art text-to-audio and text-to-music results among models trained only on publicly available data, and is comparable to models trained with proprietary or licensed data. The claim is built from a sequence of ablations: adding AF-Synthetic captions to the reference setup is the largest single improvement; replacing the reference DiT with ETTA-DiT gives the next step; switching from v-prediction diffusion to OT-CFM improves training stability and robustness at low sampling budgets; and logit-normal t-sampling adds a small final gain. The paper also claims that ETTA follows complex and imaginative prompts better than baselines in a human evaluation, suggesting that benchmark-style captions understate its ability. The authors present these results as an elucidation of the TTA design space, with data quality as the dominant factor and model capacity, text encoder choice, and sampling strategies as secondary levers.

Load-bearing premise

The headline comparisons assume the published scores for other models were measured with the same evaluation protocol as ETTA's, meaning identical audio preprocessing, identical metric implementations, and identical audio-text scoring models, so that numbers taken from different papers can be read as directly comparable.

Editorial extensions

If this is right

  • If the central claim holds, the practical gap between public-data and proprietary-data text-to-audio systems is largely a caption-quality problem, not an architecture or compute problem.
  • Future TTA research can prioritise building larger pools of strongly audio-correlated synthetic captions over adding model capacity, since the ablations show data quality outranks dataset size.
  • Flow matching with OT-CFM is a safer default than v-prediction diffusion for long TTA training runs, because it stays stable past 500k steps where v-diffusion diverges in the paper's loss curves.
  • Sampling choices should be tuned per objective, because the paper's Pareto curves show the Fréchet distance behaves as a convex function of the guidance scale while KL, IS, and CLAP scores keep improving, so the best guidance scale depends on whether diversity or prompt adherence is being optimised.
  • The creative-caption human evaluation suggests standard benchmarks underestimate model ability, implying that TTA progress should also be measured on compositional and imaginative prompts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If caption quality is the dominant bottleneck, a natural extension is to test whether scaling AF-Synthetic-style captions beyond 1.35 million continues to improve benchmarks or whether returns flatten, given the paper's data comparison suggests quality rather than sheer volume is what matters.
  • The 'comparable to proprietary models' result would be on firmer footing with a shared evaluation harness, so re-scoring all baselines under identical preprocessing and identical audio-text scoring models is the test most likely to revise the ranking.
  • The creative-caption protocol of twenty imaginative prompts with human OVL and REL ratings could be expanded into a public benchmark that tracks compositional understanding separately from standard AudioCaps and MusicCaps scores.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents a large-scale empirical study of design choices for text-to-audio diffusion and flow-matching models. It introduces AF-Synthetic, a 1.35M-caption synthetic dataset generated with Audio Flamingo and filtered by CLAP similarity; proposes ETTA-DiT, an improved DiT implementation; and systematically ablates training data, model architecture and size, training objective, and sampling strategy. The resulting model, ETTA, is evaluated on AudioCaps and MusicCaps and claimed to be state-of-the-art among models trained on public data, while also being competitive with models trained on proprietary data. The paper additionally documents mixed and negative results in Appendix F, releases code, and provides a demo page.

Significance. If the headline claim is supported, the paper would be a valuable reference for TTA design: it isolates data, architecture, objective, and sampler contributions more systematically than prior work, and the AF-Synthetic dataset and NFE/CFG Pareto analyses are useful contributions independent of the SOTA claim. The documentation of negative results (Appendix F) and the release of code are strengths. However, the SOTA claim currently rests on cross-paper metric comparisons that are not protocol-matched, and several design conclusions lack variance estimates or are weakened by internal inconsistencies. These issues are fixable but require additional experiments or careful reformulation.

major comments (4)
  1. [Section 4.3, Tables 2 and 3] The SOTA claim is not supported by the evidence as presented because 'best reported' baseline numbers are compared with the authors' own measurements without protocol matching. Table 2 itself reveals the problem: the published FDO for Stable Audio Open is 78.24, while the paper's reproduction of the same model with the same metric list reports 105.88. A 27.6-point gap on the same model and benchmark implies differences in evaluation pipeline (audio duration, resampling, metric implementation, or checkpoint). Consequently, ETTA's FDO of 80.13 cannot be directly compared with the published 78.24. Please either reproduce all competing baselines with the exact evaluation pipeline used for ETTA, or change the claim to 'competitive with published numbers' and explicitly state the protocol caveat. The same issue affects Table 3 and the SongDescriber results in Table 23.
  2. [Section 4.2, Tables 4-6, 17-18] All objective metrics are reported as single point estimates without variance, multiple seeds, or significance tests. Several design conclusions are drawn from differences comparable to typical run-to-run noise: in Table 4, the switch to logit-normal time sampling changes FDP by 0.57 and KLS by 0.06 in opposite directions, and in Table 5 AF-AudioSet vs AF-Synthetic on MusicCaps differ by 0.19 in FDP and 0.04 in KLS while the text claims AF-Synthetic is 'consistently better'. Without error bars or repeated runs, these attributions are not established. At minimum, report multiple seeds for the key ablation chains (Tables 4 and 5) or state explicitly which differences are within expected noise.
  3. [Section 3.1 and Section 4.3, metrics] The AF-Synthetic construction filters captions using CLAP similarity from the 'Laion-CLAP 630k-audioset-fusion-best' checkpoint, and the evaluation suite includes CLL from 'LAION's 630k-best' checkpoint plus CLM from MS-CLAP. Please state clearly whether the filtering checkpoint and the CLL evaluation checkpoint are the same; the naming suggests they may be different. Even if they differ, because both dataset selection and a headline text-adherence metric use CLAP-family embeddings, the paper should include a robustness check showing that ETTA's CLAP gains are not an artifact of training on captions pre-selected by CLAP. Concretely, evaluate with a CLAP checkpoint not used anywhere in the pipeline and show that the relative ranking over baselines is preserved; the human REL scores and KL/FD gains are helpful but do not fully address this point.
  4. [Section 4.2 and Appendix D (Figure 4), Table 5] The data-quality conclusion ('quality matters more') is compromised because the AF-AudioSet baseline appears to be evaluated at the onset of training instability. Appendix D states that AF-AudioSet 'quickly diverges around 250K steps and is unable to continue its training', and Table 5 reports results from models trained for 250K steps. The comparison between AF-AudioSet and AF-Synthetic in Table 5 may therefore conflate data quality with training stability. Please either use a checkpoint taken before divergence (e.g., 200K steps) or otherwise show that the comparison is unaffected by the divergence.
minor comments (4)
  1. [Section 4.3, Figure 3 and Table 19] The text states 'All metrics improve as the number of function evaluations (NFE) increases', but Table 19 shows Euler FDP worsens from 11.83 at NFE=50 to 12.10 at NFE=100; the statement should be qualified or removed.
  2. [Section 4.2, Table 5] The table header says 'million captions' and lists TangoPromptBank as 1.21, while the text says 2.33M audio-caption pairs were used; please align the table and text.
  3. [Section 4.3 and Table 23] The text says ETTA achieves the 'highest CLAP scores' on SongDescriber, but CLL for ETTA (0.44) is tied with AudioLDM2-large (0.44); please say 'highest CLM' or 'tied for highest CLL'.
  4. [Throughout] There are typos such as 'sigificaly' and 'simillar'; Table 11 captions contain 'sniping' and 'footnote', which appear to be errors; and the reference for Schneider et al. (2024) contains the garbled word 'moˆusai'.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: ETTA's benchmark results are empirical and use non-CLAP text conditioning plus distinct evaluation CLAP checkpoints; the one self-citation (AF-AudioSet's 0.45 threshold) is not load-bearing, while baseline protocol mismatch is a correctness concern, not circularity.

full rationale

ETTA is an empirical, systems-style paper: its contributions are a synthetic caption dataset, DiT implementation changes, and training/sampling choices, all validated by benchmark measurements. Walking the derivation chain for equation-level reduction, I find no step where a prediction is equivalent to an input by construction. (1) AF-Synthetic filtering uses cos(CLAP_audio(a), CLAP_text(c)) >= 0.45 with the LAION '630k-audioset-fusion-best' checkpoint, while the reported CLAP metrics use different checkpoints (CLL: LAION '630k-best'; CLM: MS-CLAP 2023). Moreover, the text encoder used for generation is T5, not CLAP, so the CLAP benchmark numbers are not the filter function applied to the model's own inputs. (2) The only load-adjacent self-citation is taking the 0.45 threshold as 'optimal according to AF-AudioSet (Kong et al., 2024b)'; this is a data-preprocessing choice inherited from the authors' own prior work, but it does not determine or construct the reported FDP, FDO, KL, IS, or CL scores, which are independent measurements of a trained model. (3) The SOTA claim relies on 'best reported' baseline numbers from different papers; this is a protocol-matching and correctness risk, not a circular derivation, and I do not count it as circularity. (4) The CFG and NFE sweep selects hyperparameters on the same benchmark used for reporting; this is test-set tuning and a validity caveat, but the reported scores are measured outputs of a trained model rather than the fitted values themselves, so it does not meet the bar of equation-level circularity. Net result: no circular step; score 2 reflects only the minor, non-load-bearing self-citation of the AF-AudioSet threshold.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claims rest on standard generative-modeling theory, plus domain assumptions about CLAP and benchmark metrics as reliable proxies for audio-text quality. The free parameters are experimental hyperparameters selected by sweeps; they are not fitted constants in a derivation but do directly influence the reported results.

free parameters (5)
  • CLAP similarity threshold for caption filtering = 0.45
    Captions with CLAP similarity below 0.45 are discarded when building AF-Synthetic (Section 3.1); the threshold is inherited from AF-AudioSet and controls the size/quality trade-off of the dataset.
  • Classifier-free guidance scale w_cfg = 3.5 (main results), 1 (ablations)
    Selected from a sweep over w_cfg (Section 4.2 and Figure 3); directly controls the fidelity-diversity trade-off in the reported benchmark scores.
  • Number of function evaluations (NFE) for sampling = 100 Euler steps
    Selected from the Pareto analysis as the operating point for main results; benchmark numbers change materially at lower NFE.
  • Dropout probability p_dropout = 0.1
    Applied to all modules in ETTA-DiT; the paper notes that p_dropout=0.0 gives slightly better 250k-step benchmark scores, but 0.1 is kept for generalization, affecting the reported main results.
  • Logit-normal timestep sampling scale = sigma = 1 (N(0,1))
    Used for OT-CFM training following prior work (Esser et al., 2024); marginally improves FDP over uniform timestep sampling (Table 4).
assumptions (4)
  • standard math Diffusion and flow-matching objectives define valid generative models for the VAE latent space.
    Section 3.2 and Appendix B.2-B.3 rely on these as known theory.
  • domain assumption CLAP embeddings are a reliable measure of audio-caption alignment for filtering and evaluation.
    Section 3.1 uses CLAP similarity to filter AF-Synthetic and Section 4.3 uses CLAP as an evaluation metric; if CLAP is biased, dataset quality and benchmark scores are affected.
  • domain assumption The objective metrics (FD, KL, IS, CLAP) are valid proxies for perceived text-to-audio quality.
    Section 4.3 uses these as primary evidence; the paper also runs human studies but only for a subset of conditions.
  • domain assumption Audio Flamingo generates accurate captions for 10-second audio segments across diverse sound categories.
    Section 3.1 uses Audio Flamingo to produce the synthetic captions that constitute AF-Synthetic; if the model is inaccurate, the dataset benefit is reduced.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ETTA: Elucidating the Design Space of Text-to-Audio Models." pith.science (2026). https://pith.science/paper/RELS3FEN

@misc{pith2026241219351,
  author       = {Pith},
  title        = {Pith review of: ETTA: Elucidating the Design Space of Text-to-Audio Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RELS3FEN}},
  note         = {Machine review of arXiv:2412.19351}
}
read the original abstract

Recent years have seen significant progress in Text-To-Audio (TTA) synthesis, enabling users to enrich their creative workflows with synthetic audio generated from natural language prompts. Despite this progress, the effects of data, model architecture, training objective functions, and sampling strategies on target benchmarks are not well understood. With the purpose of providing a holistic understanding of the design space of TTA models, we set up a large-scale empirical experiment focused on diffusion and flow matching models. Our contributions include: 1) AF-Synthetic, a large dataset of high quality synthetic captions obtained from an audio understanding model; 2) a systematic comparison of different architectural, training, and inference design choices for TTA models; 3) an analysis of sampling methods and their Pareto curves with respect to generation quality and inference speed. We leverage the knowledge obtained from this extensive analysis to propose our best model dubbed Elucidated Text-To-Audio (ETTA). When evaluated on AudioCaps and MusicCaps, ETTA provides improvements over the baselines trained on publicly available data, while being competitive with models trained on proprietary data. Finally, we show ETTA's improved ability to generate creative audio following complex and imaginative captions -- a task that is more challenging than current benchmarks.

Figures

Figures reproduced from arXiv: 2412.19351 by the authors.

Figure 1
Figure 1. Distributions of CLAP similarities between au [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 3
Figure 3. The effect of different sampling methods on the [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Training loss comparison across three setups: (left) Stable Audio Open-DiT vs. ETTA-DiT, using the same [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: The effect of different sampling methods on the generation quality of ETTA on AudioCaps and MusicCaps. We [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A2SB: Audio-to-Audio Schrodinger Bridges

    cs.SD 2025-01 conditional novelty 6.0 of 10

    A2SB applies Schrödinger bridges to music restoration, achieving state-of-the-art bandwidth extension and inpainting at 44.1kHz in a single vocoder-free model.

Reference graph

Works this paper leans on

81 extracted references · 18 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Musiclm: Generating music from text

    Andrea Agostinelli, Timo I Denk, Zal \'a n Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325, 2023

    arXiv 2023

  3. [3]

    Improving image generation with better captions

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2 0 (3): 0 8, 2023

    2023

  4. [4]

    Audiolm: a language modeling approach to audio generation

    Zal \'a n Borsos, Rapha \"e l Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: a language modeling approach to audio generation. IEEE/ACM transactions on audio, speech, and language processing, 31: 0 2523--2533, 2023

    work page 2023

  5. [5]

    Vggsound: A large-scale audio-visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 721--725. IEEE, 2020

    2020

  6. [6]

    Scaling instruction-finetuned language models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25 0 (70): 0 1--53, 2024

    2024

  7. [7]

    Simple and controllable music generation

    Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre D \'e fossez. Simple and controllable music generation. Advances in Neural Information Processing Systems, 36, 2024

    2024

  8. [8]

    Look, listen, and learn more: Design choices for deep audio embeddings

    Aurora Linh Cramer, Ho-Hsiang Wu, Justin Salamon, and Juan Pablo Bello. Look, listen, and learn more: Design choices for deep audio embeddings. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 3852--3856. IEEE, 2019

    work page 2019

Show all 81 references
  1. [9]

    Flashattention: Fast and memory-efficient exact attention with io-awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e . Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35: 0 16344--16359, 2022

    2022

  2. [10]

    High fidelity neural audio compression

    Alexandre D \'e fossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. Transactions on Machine Learning Research, 2023

    2023

  3. [11]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34: 0 8780--8794, 2021

    2021

  4. [12]

    Natural language supervision for general-purpose audio representations, 2023

    Benjamin Elizalde, Soham Deshmukh, and Huaming Wang. Natural language supervision for general-purpose audio representations, 2023. URL https://arxiv.org/abs/2309.05767

    2023 arXiv

  5. [13]

    Scaling rectified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M \"u ller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Mach...

    2024

  6. [14]

    Fast timing-conditioned latent audio diffusion

    Zach Evans, CJ Carr, Josiah Taylor, Scott H Hawley, and Jordi Pons. Fast timing-conditioned latent audio diffusion. In Forty-first International Conference on Machine Learning

  7. [15]

    Long-form music generation with latent diffusion

    Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Long-form music generation with latent diffusion. arXiv preprint arXiv:2404.10301, 2024 a

    2024 arXiv

  8. [16]

    Stable audio open

    Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Stable audio open. arXiv preprint arXiv:2407.14358, 2024 b

    2024 arXiv

  9. [17]

    Flux that plays music

    Zhengcong Fei, Mingyuan Fan, Changqian Yu, and Junshi Huang. Flux that plays music. arXiv preprint arXiv:2409.00587, 2024

    2024 arXiv

  10. [18]

    Audio set: An ontology and human-labeled dataset for audio events

    Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (...

    2017

  11. [19]

    Text-to-audio generation using instruction guided latent diffusion model

    Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. Text-to-audio generation using instruction guided latent diffusion model. In Proceedings of the 31st ACM International Conference on Multimedia, pp.\ 3590--3598, 2023

    2023

  12. [20]

    Ezaudio: Enhancing text-to-audio generation with efficient diffusion transformer

    Jiarui Hai, Yong Xu, Hao Zhang, Chenxing Li, Helin Wang, Mounya Elhilali, and Dong Yu. Ezaudio: Enhancing text-to-audio generation with efficient diffusion transformer. arXiv preprint arXiv:2409.10819, 2024

    2024 arXiv

  13. [21]

    Taming data and transformers for audio generation

    Moayed Haji-Ali, Willi Menapace, Aliaksandr Siarohin, Guha Balakrishnan, Sergey Tulyakov, and Vicente Ordonez. Taming data and transformers for audio generation. arXiv preprint arXiv:2406.19388, 2024

    2024 arXiv

  14. [22]

    Efficient diffusion training via min-snr weighting strategy

    Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 7441--7451, 2023

    2023

  15. [23]

    Gaussian error linear units (gelus)

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016

    2016 arXiv

  16. [24]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022

    2022 arXiv

  17. [25]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33: 0 6840--6851, 2020

    2020

  18. [26]

    Make-an-audio 2: Temporal-enhanced text-to-audio generation

    Jiawei Huang, Yi Ren, Rongjie Huang, Dongchao Yang, Zhenhui Ye, Chen Zhang, Jinglin Liu, Xiang Yin, Zejun Ma, and Zhou Zhao. Make-an-audio 2: Temporal-enhanced text-to-audio generation. arXiv preprint arXiv:2305.18474, 2023 a

    2023 arXiv

  19. [27]

    Noise2music: Text-conditioned music generation with diffusion models

    Qingqing Huang, Daniel S Park, Tao Wang, Timo I Denk, Andy Ly, Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Frank, et al. Noise2music: Text-conditioned music generation with diffusion models. arXiv preprint arXiv:2302.03917, 2023 b

    2023 arXiv

  20. [28]

    Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models

    Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. In International Conference on Machine Learning, pp.\ 13916--13932. PMLR, 2023 c

    2023

  21. [29]

    Elucidating the design space of diffusion-based generative models

    Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35: 0 26565--26577, 2022

    2022

  22. [30]

    Guiding a diffusion model with a bad version of itself

    Tero Karras, Miika Aittala, Tuomas Kynk \"a \"a nniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine. Guiding a diffusion model with a bad version of itself. arXiv preprint arXiv:2406.02507, 2024

    2024 arXiv

  23. [31]

    Fr 'echet audio distance: A metric for evaluating music enhancement algorithms

    Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fr 'echet audio distance: A metric for evaluating music enhancement algorithms. arXiv preprint arXiv:1812.08466, 2018

    2018 arXiv

  24. [32]

    Audiocaps: Generating captions for audios in the wild

    Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (...

    2019

  25. [33]

    Enclap: Combining neural audio codec and audio-text joint embedding for automated audio captioning

    Jaeyeon Kim, Jaeyoon Jung, Jinjoo Lee, and Sang Hoon Woo. Enclap: Combining neural audio codec and audio-text joint embedding for automated audio captioning. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 6735--6739....

    2024

  26. [34]

    Variational diffusion models

    Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34: 0 21696--21707, 2021

    2021

  27. [35]

    Kingma and Max Welling

    Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes . In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings , 2014

    2014

  28. [36]

    Panns: Large-scale pretrained audio neural networks for audio pattern recognition

    Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28: 0 2880--2894, 2020

    2020

  29. [37]

    Diffwave: A versatile diffusion model for audio synthesis

    Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021

    2021

  30. [38]

    Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities

    Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. International Conference on Machine Learning, 2024 a

    2024

  31. [39]

    Improving text-to-audio models with synthetic captions

    Zhifeng Kong, Sang-gil Lee, Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, Rafael Valle, Soujanya Poria, and Bryan Catanzaro. Improving text-to-audio models with synthetic captions. arXiv preprint arXiv:2406.15487, 2024 b

    2024 arXiv

  32. [40]

    Efficient training of audio transformers with patchout

    Khaled Koutini, Jan Schl \" u ter, Hamid Eghbal - zadeh, and Gerhard Widmer. Efficient training of audio transformers with patchout. In Interspeech 2022, 23rd Annual Conference of the International Speech Communication Association, Incheon, Korea, 18-22 September 2022, pp.\ 27...

    2022 doi

  33. [41]

    Audiogen: Textually guided audio generation

    Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre D \'e fossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. arXiv preprint arXiv:2209.15352, 2022

    2022 arXiv

  34. [42]

    Applying guidance in a limited interval improves sample and distribution quality in diffusion models

    Tuomas Kynk \"a \"a nniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in a limited interval improves sample and distribution quality in diffusion models. arXiv preprint arXiv:2404.07724, 2024

    2024 arXiv

  35. [43]

    Efficient neural music generation

    Max WY Lam, Qiao Tian, Tang Li, Zongyu Yin, Siyuan Feng, Ming Tu, Yuliang Ji, Rui Xia, Mingbo Ma, Xuchen Song, et al. Efficient neural music generation. Advances in Neural Information Processing Systems, 36, 2024

    2024

  36. [44]

    High fidelity text-guided music generation and editing via single-stage flow matching

    Gael Le Lan, Bowen Shi, Zhaoheng Ni, Sidd Srinivasan, Anurag Kumar, Brian Ellis, David Kant, Varun Nagaraja, Ernie Chang, Wei-Ning Hsu, et al. High fidelity text-guided music generation and editing via single-stage flow matching. arXiv preprint arXiv:2407.03648, 2024

    2024 arXiv

  37. [45]

    Bigvgan: A universal neural vocoder with large-scale training

    Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. Bigvgan: A universal neural vocoder with large-scale training. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=iTtGCMDEzS_

    2023

  38. [46]

    Quality-aware masked diffusion transformer for enhanced music generation

    Chang Li, Ruoyu Wang, Lijuan Liu, Jun Du, Yixuan Sun, Zilu Guo, Zhenrong Zhang, and Yuan Jiang. Quality-aware masked diffusion transformer for enhanced music generation. arXiv preprint arXiv:2405.15863, 2024 a

    2024 arXiv

  39. [47]

    Jen-1: Text-guided universal music generation with omnidirectional diffusion models

    Peike Patrick Li, Boyu Chen, Yao Yao, Yikai Wang, Allen Wang, and Alex Wang. Jen-1: Text-guided universal music generation with omnidirectional diffusion models. In 2024 IEEE Conference on Artificial Intelligence (CAI), pp.\ 762--769. IEEE, 2024 b

    2024

  40. [48]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022

    2022 arXiv

  41. [49]

    Generative pre-training for speech with flow matching

    Alexander H Liu, Matt Le, Apoorv Vyas, Bowen Shi, Andros Tjandra, and Wei-Ning Hsu. Generative pre-training for speech with flow matching. arXiv preprint arXiv:2310.16338, 2023 a

    2023 arXiv

  42. [50]

    Audioldm: Text-to-audio generation with latent diffusion models

    Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. In International Conference on Machine Learning, pp.\ 21450--21474. PMLR, 2023 b

    2023

  43. [51]

    Audioldm 2: Learning holistic audio generation with self-supervised pretraining

    Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D Plumbley. Audioldm 2: Learning holistic audio generation with self-supervised pretraining. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024

    2024

  44. [52]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  45. [53]

    Tango 2: Aligning diffusion-based text-to-audio generations through direct preference optimization

    Navonil Majumder, Chia-Yu Hung, Deepanway Ghosal, Wei-Ning Hsu, Rada Mihalcea, and Soujanya Poria. Tango 2: Aligning diffusion-based text-to-audio generations through direct preference optimization. arXiv preprint arXiv:2404.09956, 2024

    2024 arXiv

  46. [54]

    The song describer dataset: a corpus of audio captions for music-and-language evaluation

    Ilaria Manco, Benno Weck, Seungheon Doh, Minz Won, Yixiao Zhang, Dmitry Bogdanov, Yusong Wu, Ke Chen, Philip Tovstogan, Emmanouil Benetos, Elio Quinton, György Fazekas, and Juhan Nam. The song describer dataset: a corpus of audio captions for music-and-language evaluation. In ...

    2023

  47. [55]

    Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research

    Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D Plumbley, Yuexian Zou, and Wenwu Wang. Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. IEEE/ACM Transactions on Audio, Speech, and Lang...

    2024

  48. [56]

    Mustango: Toward controllable text-to-music generation

    Jan Melechovsky, Zixun Guo, Deepanway Ghosal, Navonil Majumder, Dorien Herremans, and Soujanya Poria. Mustango: Toward controllable text-to-music generation. arXiv preprint arXiv:2311.08355, 2023

    2023 arXiv

  49. [57]

    Mixed precision training

    Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXiv:1710.03740, 2017

    arXiv

  50. [58]

    Improving multimodal datasets with image captioning

    Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco, Sewoong Oh, and Ludwig Schmidt. Improving multimodal datasets with image captioning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  51. [59]

    Gpt-4o: A powerful multimodal language model

    OpenAI. Gpt-4o: A powerful multimodal language model. https://openai.com/research/hello-gpt-4o, 2024. Accessed: 2024-09-21

    2024

  52. [60]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 4195--4205, 2023

    2023

  53. [61]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  54. [62]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  55. [63]

    The musdb18 corpus for music separation

    Zafar Rafii, Antoine Liutkus, Fabian-Robert St \"o ter, Stylianos Ioannis Mimilakis, and Rachel Bittner. The musdb18 corpus for music separation. 2017

    2017

  56. [64]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj \"o rn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 10684--10695, 2022

    2022

  57. [65]

    Progressive distillation for fast sampling of diffusion models

    Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022

    2022 arXiv

  58. [66]

    Mo \^u sai: Efficient text-to-music diffusion models

    Flavio Schneider, Ojasv Kamal, Zhijing Jin, and Bernhard Sch \"o lkopf. Mo \^u sai: Efficient text-to-music diffusion models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 8050--8068, 2024

    2024

  59. [67]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021

    2021

  60. [68]

    auraloss: Audio focused loss functions in pytorch

    Christian J Steinmetz and Joshua D Reiss. auraloss: Audio focused loss functions in pytorch. In Digital music research network one-day workshop (DMRN+ 15), 2020

    2020

  61. [69]

    Automatic multitrack mixing with a differentiable mixing console of neural audio effects

    Christian J Steinmetz, Jordi Pons, Santiago Pascual, and Joan Serra. Automatic multitrack mixing with a differentiable mixing console of neural audio effects. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 71--75. IEEE, 2021

    2021

  62. [70]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

    2024

  63. [71]

    Conditional flow matching: Simulation-free dynamic optimal transport

    Alexander Tong, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector-Brooks, Kilian Fatras, Guy Wolf, and Yoshua Bengio. Conditional flow matching: Simulation-free dynamic optimal transport. arXiv preprint arXiv:2302.00482, 2023

    2023 arXiv

  64. [72]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  65. [73]

    Audiobox: Unified audio generation with natural language prompts

    Apoorv Vyas, Bowen Shi, Matthew Le, Andros Tjandra, Yi-Chiao Wu, Baishan Guo, Jiemin Zhang, Xinyue Zhang, Robert Adkins, William Ngan, et al. Audiobox: Unified audio generation with natural language prompts. arXiv preprint arXiv:2312.15821, 2023

    2023 arXiv

  66. [74]

    Cogvlm: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079, 2023

    2023 arXiv

  67. [75]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation

    Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Sign...

    2023

  68. [76]

    Auffusion: Leveraging the power of diffusion and large language models for text-to-audio generation

    Jinlong Xue, Yayue Deng, Yingming Gao, and Ya Li. Auffusion: Leveraging the power of diffusion and large language models for text-to-audio generation. arXiv preprint arXiv:2401.01044, 2024

    2024 arXiv

  69. [77]

    Improving audio generation with visual enhanced caption

    Yi Yuan, Dongya Jia, Xiaobin Zhuang, Yuanzhe Chen, Zhengxi Liu, Zhuo Chen, Yuping Wang, Yuxuan Wang, Xubo Liu, Mark D Plumbley, et al. Improving audio generation with visual enhanced caption. arXiv preprint arXiv:2407.04416, 2024

    2024 arXiv

  70. [78]

    Libritts: A corpus derived from librispeech for text-to-speech

    Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech. arXiv preprint arXiv:1904.02882, 2019

    1904 arXiv

  71. [79]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  72. [80]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  73. [81]

    Spotify", title =

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2009 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.