Pith. sign in

REVIEW 4 major objections 4 minor 46 references

IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read IMPACT is a text-to-audio framework that runs mask-based parallel decoding on continuous latent representations and claims the best fidelity-latency trade-off on AudioCaps.

desk verdict A competent MAR-to-audio transfer with strong but test-set-tuned speed/quality results; read it for the method, not for the claimed SOTA. read the letter →

arxiv 2506.00736 v1 pith:Y6MSWVSQ submitted 2025-05-31 eess.AS cs.SD

Kuan-Po Huang , Shu-wen Yang , Huy Phan , Bo-Ru Lu , Byeonggeun Kim , Sashank Macha , Qingming Tang , Shalini Ghosh
show 3 more authors
Hung-yi Lee Chieh-Chi Kao Chao Wang
This is my paper
classification eess.AScs.SD
keywords text-to-audiogenerationiterativemask-basedparalleldecodinglatentdiffusionmodelscontinuousaudiolatentsmaskedgenerativemodelingunconditionalpre-trainingCapsinferencespeed
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

IMPACT is a text-to-audio generation framework aimed at removing the usual trade-off between audio fidelity and inference speed. It keeps audio in the continuous latent space used by diffusion models, but generates the latent sequence by iterative mask-based parallel decoding, so many positions are predicted at each step. A lightweight MLP diffusion head, rather than a heavy attention transformer, performs the denoising. On AudioCaps the paper reports the best Fréchet Distance and Fréchet Audio Distance among the compared models, competitive subjective relevance and quality ratings, and latency close to MAGNET-S, the fastest discrete-token baseline. The paper also argues that an unconditional pre-training phase on unlabelled audio is necessary for these results.

What carries the argument

The load-bearing object is iterative mask-based parallel decoding in a continuous latent space, powered by a latent diffusion model. Where discrete-token models such as MAGNET choose which tokens to reveal by confidence score, IMPACT randomly selects which previously masked latent positions to generate at each decoding iteration, guided by a cosine masking schedule that keeps more positions hidden early and reveals more as context accumulates. The heavy Transformer encoder runs once per decoding iteration on the unmasked positions; the actual denoising is done by a lightweight MLP diffusion head adapted from MAR, which avoids attention layers in the diffusion loop. The Transformer latent encoder follows the MAE two-subencoder design, and its weights are initialized from pre-trained MAR image checkpoints, which Appendix D.3 calls indispensable for performance. Unconditional pre-training on unlabelled audio before text-conditional training is the supporting training mechanism that the paper credits for the fidelity gains.

What would settle it

Train an IMPACT model with the same architecture, data, and unconditional pre-training but initialize the latent encoder from random weights; if the FAD and KL results on AudioCaps degrade to roughly the level described in Appendix D.3's failure case, then the pretrained-image transfer, not the mask-parallel diffusion design, is responsible for the gains.

Watch

Extended reading notes

Core claim

The central claim is that discrete token representations are the bottleneck that keeps mask-based audio generators like MAGNET behind diffusion models, and that running mask-based parallel decoding directly on continuous latents removes that bottleneck. IMPACT starts from a fully masked latent sequence, and at each decoding iteration randomly selects a subset of remaining positions to predict; the masking fraction follows a cosine schedule. The selected latent positions are corrupted with noise and a small MLP diffusion head, conditioned on hidden states from a Transformer latent encoder, predicts the noise, following the standard reverse diffusion update. The paper reports that with 16 decoding iterations IMPACT already surpasses all publicly available baselines on FAD and KL on AudioCaps, and that a base model with 32 iterations reaches FAD 1.07 while taking 11.2 seconds to generate a batch of eight 10-second audios on a V100. The claim is that this combination makes IMPACT the first text-to-audio model to be both best in reported fidelity and competitive with the fastest existing generator in speed.

Load-bearing premise

The whole result rests on initializing the Transformer latent encoder from image-pretrained MAR checkpoints: Appendix D.3 says that without that initialization training produces poor performance, so if those pretrained weights are unavailable or unsuitable, the method's advantages may vanish.

Editorial extensions

If this is right

  • With 16 decoding iterations, IMPACT exceeds every public baseline in FAD and KL on AudioCaps within 5.7 seconds per batch of eight, so fidelity no longer requires diffusion-model-scale latency.
  • At batch size 64, IMPACT base with four decoding iterations generates 8.26 samples per second compared with MAGNET-S's 1.23, while posting better FAD, KL, and CLAP, indicating the method scales with batch size.
  • Unconditional pre-training on 5,500 hours of unlabelled AudioSet raises subjective relevance and quality scores even when some objective metrics barely move, making unlabelled audio data usable for text-to-audio training.
  • Removing the CLAP text encoder leaves CLAP scores essentially unchanged, so Flan-T5 text embeddings alone are sufficient for conditioning in this framework.
  • Mask parallelism is the source of speed: the single-pass variant scores FAD 12.26 and CLAP 0.125, versus FAD 1.07 and CLAP 0.364 with 32 decoding iterations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension the paper does not run would start the latent encoder from random weights while keeping unconditional pre-training, isolating whether the MAR image-pretrained initialization or the mask-parallel diffusion recipe deserves the credit.
  • Because IMPACT replaces confidence-based token selection with random position selection, it surrenders the ability to focus decoding on low-confidence regions; a continuous-space confidence proxy is an untested way to make decoding more efficient.
  • The same continuous mask-parallel diffusion recipe is a plausible fit for music generation and sound-event captioning, where discrete token bottlenecks are also present, but the paper evaluates only AudioCaps.
  • The near-zero effect of removing CLAP suggests future versions can simplify the conditioning stack to a single text encoder, which is not a claim the paper makes beyond its ablation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes IMPACT, a text-to-audio generation framework that applies iterative mask-based parallel decoding to continuous audio latents. A Transformer latent encoder is trained with a masking objective, and a lightweight MLP-based diffusion head predicts noise for the masked latents; an unconditional pre-training phase on unlabeled audio precedes text-conditional training. On AudioCaps, the authors report state-of-the-art FD and FAD scores while achieving latency competitive with MAGNET-S, with ablations over pre-training data, text-conditional data, decoding iterations, diffusion steps, and classifier-free guidance.

Significance. If the empirical claims hold, the paper makes a useful contribution: it adapts MAR-style continuous-latent mask decoding to audio, demonstrating a favorable fidelity-latency trade-off and providing evidence that unconditional pre-training helps. The ablations are informative and the derivations in Sections 3.1-3.2 follow standard DDPM/MGM machinery. However, the headline SOTA claim is currently conditional on evaluation-set hyperparameter selection and on initialization from image-pretrained MAR checkpoints, and several dataset-labeling inconsistencies undermine reproducibility. The method is promising, but the reported numbers should not be taken at face value until these issues are resolved.

major comments (4)
  1. [Section 5.1, 5.3.3, Appendix A] The leading SOTA claim is obtained by selecting hyperparameters on the AudioCaps evaluation set, with no validation split. Section 5.1 states that baselines are 'configured using hyperparameters that yield their overall optimal performance on objective metrics,' and Appendix A Table 7 shows FAD varying from 3.36 (beta_cfg_max=1.0) to 1.13 (beta_cfg_max=5.0) purely by choosing the max CFG scaler on the evaluation set. Similarly, Section 5.3.2 concludes that 100 diffusion steps is optimal from the eval-set results, and Section 5.3.3 identifies the 'optimal range' from the same results. This is a selection problem: the reported gap between IMPACT (FAD 1.07) and MAGNET-S (FAD 3.22) may be inflated by test-set tuning. Please report results with hyperparameters chosen on a validation split, or otherwise demonstrate that the selected settings generalize.
  2. [Appendix D.3] The method depends on initializing the Transformer latent encoder from pre-trained MAR image checkpoints, with the statement 'failing to do so results in poor performance.' This is load-bearing: if most of the generation quality comes from transferred image representations, then the proposed unconditional pre-training and mask-parallel decoding are not self-contained audio methods. The paper does not ablate random initialization with the same training budget, nor does it test audio-domain pretraining. Please provide such an ablation or an explicit analysis of how much of the reported performance is attributable to the MAR initialization.
  3. [Table 2 vs Table 9] The training-data configurations are internally inconsistent across tables. Table 2 model (b) is listed with 1200 h of pre-training data, but Table 9 lists model (b) as pre-trained on AudioSet, which is 5500 h. Table 2 model (d) is listed with 5500 h pre-training and 145 h text-conditional training, but Table 9 lists model (d) as AC+WC for both pre-training and fine-tuning. Table 1's 'IMPACT base, dec iter 32' row matches Table 2 model (b'), but Table 9 does not list (b') at all. These discrepancies affect the interpretation of the ablations in Section 5.2 and must be corrected with a single consistent dataset table.
  4. [Section 3.2.3] The diffusion sampling procedure is underspecified. Equation (3) gives one reverse step, but the text does not state how the initial noisy latent \(\hat z_T^i\) is obtained for the positions selected at each decoding iteration, nor how many reverse steps are run per decoding iteration. Since the default is \(\hat T=100\) diffusion steps, it is unclear whether each decoding iteration runs 100 denoising steps or a single step. This is a core reproducibility gap in the proposed method; please specify the initialization and the full reverse-process loop.
minor comments (4)
  1. [Section 4.2 vs Table 4] Section 4.2 states that the masking percentage factor is set to q=0.7 for all model training, but Table 4 reports that q=1.0 was used for the single-pass model 'to yield the best performance.' Please reconcile this contradiction and state the exact training configurations for each model.
  2. [Section 5.4 / Figure 4] Figure 4 compares MAGNET-S at 50 iterations with IMPACT at 4 decoding iterations. While Table 6 does report quality at these settings, the figure would be clearer if it also included a matched-iteration or matched-latency comparison, since MAGNET-S's 50 iterations are a consequence of its four-level codebook design rather than a freely tunable parameter.
  3. [Section 4.3] The subjective evaluation uses 30 generated samples rated by at least 10 participants each, but no inter-rater agreement or significance testing is reported. Given the relatively small sample size, please add confidence intervals or a significance test for the REL and OVL comparisons.
  4. [Appendix D.1] The subset property in Eq. (6) correctly states that once a position is unmasked it remains unmasked, but the surrounding text could more clearly distinguish the per-iteration predictor set \(\mathcal{M}^{(t)}_{\mathrm{pred}}\) from the full mask \(M^{(t)}\). A short pseudocode block would improve implementability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation is present: the reported SOTA numbers are externally benchmarked empirical results, and the eval-set hyperparameter choices are a selection concern rather than circularity.

full rationale

IMPACT's central claim (Section 1: 'Our model achieves state-of-the-art performance on objective metrics FD and FAD...') is an empirical benchmark result measured with external metrics (FD/FAD/KL/IS/CLAP; Appendix F). No equation in Sections 3.1-3.2 reduces a predicted quantity to an input fitted on that same quantity. The training objective in Eq. (2) is the standard noise-prediction loss, Eq. (3) is the standard DDPM reverse step, and Eqs. (4)-(5) define classifier-free guidance with a scalar beta_cfg_max. That scalar is selected by looking at eval-set performance (Appendix A, Table 7), as are the number of diffusion steps (Table 5) and decoding iterations (Section 5.3). This is hyperparameter selection on the test set, which is a reproducibility and overfitting concern, not a circular derivation, because the reported numbers are produced by a fixed trained model and are not the optimizer of the selection criterion by construction. The reliance on MAR image-checkpoint initialization (Appendix D.3) is load-bearing but not circular: it is an empirical transfer choice, cited to an external repository, and it does not define IMPACT's outputs in terms of its own results. There is no self-citation chain carrying the main claim; the citations to AudioLDM, MAR, MAGNET, Tango, and similar works are external baselines or components. The subjective evaluations (REL/OVL) are independent human ratings, and the latency measurements are direct wall-clock comparisons. No step in the paper's derivation reduces to its own inputs, so the appropriate circularity finding is none.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central result rests on standard DDPM math, borrowed MAR machinery, a borrowed VAE, and several empirical choices (q, cfg, schedule, pretraining scale). The most fragile unpaid assumption is the dependency on MAR image checkpoints, explicitly stated in Appendix D.3. No evidence of circularity: benchmarks are external, and none of the 'predictions' are defined in terms of fitted parameters. No new physical or conceptual entities are introduced.

free parameters (6)
  • masking percentage q = 0.7
    Training-time masking fraction (Section 4.2); chosen by hand, no ablation shown.
  • max classifier-free guidance beta_cfg_max = 5.0
    Appendix A and Table 7; selected among 1.0 to 5.0 using AudioCaps eval metrics.
  • decoding iterations T = 64 default; 4, 8, 16, 32 considered
    Inference trade-off parameter (Section 5.3.1); final SOTA numbers use 32 or 64, selected by eval-set performance.
  • diffusion steps T_hat = 100
    Selected from 50, 100, 150, 200 using AudioCaps FAD/KL/IS/CLAP (Table 5).
  • mask schedule gamma(t) = cos(pi/2 * t/T)
    Scheduling choice from prior MGM literature; no objective comparison to linear or alternative schedules.
  • unconditional pretraining data = 1200 h or 5500 h AudioSet
    Data scale selected by availability; model (b) vs (c) shows nontrivial metric differences.
assumptions (6)
  • standard math DDPM closed-form forward and reverse diffusion equations (Eq. 2-3) are assumed correct and applicable to AudioLDM VAE latents.
    Used in Section 3.1.1 and 3.2.3 without modification; standard results from Ho et al., 2020 and Nichol & Dhariwal, 2021.
  • domain assumption MAR's small MLP diffusion head with AdaLN and its training procedure transfer from images to audio latents.
    The diffusion head is 'directly adopted from MAR' (Appendix D.4); no audio-specific redesign or theoretical justification is given.
  • domain assumption Pre-trained MAR image-transformer checkpoints must initialize the latent encoder; poor performance otherwise.
    Appendix D.3 states 'failing to do so results in poor performance'; the method's success depends on this external transfer.
  • domain assumption AudioLDM VAE provides a continuous latent space in which patch-wise mask completion and diffusion denoising are valid operations.
    The VAE is adopted without modification (Section 4.2) and all masking and denoising happens on its patched latents; no analysis of the latent distribution is provided.
  • domain assumption Random selection of positions to unmask converges to high-quality outputs within 4 to 64 decoding iterations.
    Section 3.2.1 replaces confidence-based scheduling with random selection because continuous latents lack confidence scores; the convergence is only demonstrated empirically.
  • domain assumption Unconditional pretraining on unpaired AudioSet data transfers to text-conditional downstream training.
    Section 5.2.1 shows gains on most metrics, but the mechanism is not examined and the benefit depends on data scale.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling." pith.science (2026). https://pith.science/paper/Y6MSWVSQ

@misc{pith2026250600736,
  author       = {Pith},
  title        = {Pith review of: IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y6MSWVSQ}},
  note         = {Machine review of arXiv:2506.00736}
}
read the original abstract

Text-to-audio generation synthesizes realistic sounds or music given a natural language prompt. Diffusion-based frameworks, including the Tango and the AudioLDM series, represent the state-of-the-art in text-to-audio generation. Despite achieving high audio fidelity, they incur significant inference latency due to the slow diffusion sampling process. MAGNET, a mask-based model operating on discrete tokens, addresses slow inference through iterative mask-based parallel decoding. However, its audio quality still lags behind that of diffusion-based models. In this work, we introduce IMPACT, a text-to-audio generation framework that achieves high performance in audio quality and fidelity while ensuring fast inference. IMPACT utilizes iterative mask-based parallel decoding in a continuous latent space powered by diffusion modeling. This approach eliminates the fidelity constraints of discrete tokens while maintaining competitive inference speed. Results on AudioCaps demonstrate that IMPACT achieves state-of-the-art performance on key metrics including Fr\'echet Distance (FD) and Fr\'echet Audio Distance (FAD) while significantly reducing latency compared to prior models. The project website is available at https://audio-impact.github.io/.

Figures

Figures reproduced from arXiv: 2506.00736 by the authors.

Figure 1
Figure 1. Diagram of our IMPACT framework for the training phase and inference phase. ing with a fully empty sequence and gradually generating latents at each iteration with the small diffusion head, saving much time since the diffusion sampling loop operates on a lightweight model. 3.1. Training Phase 3.1.1. TEXT-CONDITIONAL TRAINING WITH MGM [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Performance of the baseline models and our proposed IMPACT model (b) with varying decoding iterations (dec iter), visualized by plotting objective metrics (FAD, IS, KL, and CLAP) against latency. Data points of the IMPACT model are plotted in red curves [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Heatmap visualizing latency measurements for IMPACT model (b) under varying decoding iterations and diffusion steps at a batch size of 8. Latency is depicted by color intensity, with brighter colors indicating higher values. Detailed objective performance values are provided in Appendix C. 5.3.3. OPTIMAL PARAMETERS [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Batch latency and throughput comparisons between MAGNET-S and our IMPACT base model (b) with 4 decoding iterations. than MAGNET-S, even with looping through the diffusion sampling process, due to its efficiency in reaching high objective performance with fewer decoding…
Figure 5
Figure 5. Figure 5: The cfg scaler β (t) cfg at each decoding iteration during a 64-step iterative mask-based parallel decoding process [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Heatmap visualizing FAD scores for IMPACT model (b) under varying decoding iterations and diffusion steps at a batch size of 8. FAD score is depicted by color intensity, with brighter colors indicating higher values. 1 https://github.com/facebookresearch/audiocraft/blo…
Figure 7
Figure 7. Figure 7: Heatmap visualizing KL scores for IMPACT model (b) under varying decoding iterations and diffusion steps at a batch size of 8. KL score is depicted by color intensity, with brighter colors indicating higher values. C.3. IS [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Heatmap visualizing IS scores for IMPACT model (b) under varying decoding iterations and diffusion steps at a batch size of 8. IS score is depicted by color intensity, with brighter colors indicating higher values. C.4. CLAP [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Heatmap visualizing CLAP scores for IMPACT model (b) under varying decoding iterations and diffusion steps at a batch size of 8. CLAP score is depicted by color intensity, with brighter colors indicating higher values. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: REL rating platform [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: OVL rating platform. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 15 canonical work pages

  1. [1]

    P., Whitman, B., and Lamere, P

    Bertin-Mahieux, T., Ellis, D. P., Whitman, B., and Lamere, P. The million song dataset. 2011

    work page 2011

  2. [2]

    Soundstorm: Efficient parallel audio generation, 2024

    Borsos, Z., Sharifi, M., Vincent, D., Kharitonov, E., Zeghidour, N., and Tagliasacchi, M. Soundstorm: Efficient parallel audio generation, 2024. URL https://openreview.net/forum?id=KknWbD5j95

    work page 2024

  3. [3]

    Chang, H., Zhang, H., Jiang, L., Liu, C., and Freeman, W. T. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 11315--11325, 2022

    work page 2022

  4. [4]

    T., Rubinstein, M., et al

    Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, L., Yang, M.-H., Murphy, K., Freeman, W. T., Rubinstein, M., et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704, 2023

    arXiv 2023

  5. [5]

    Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio

    Chen, G., Chai, S., Wang, G., Du, J., Zhang, W.-Q., Weng, C., Su, D., Povey, D., Trmal, J., Zhang, J., et al. Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio. arXiv preprint arXiv:2106.06909, 2021

    arXiv 2021

  6. [6]

    W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al

    Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25 0 (70): 0 1--53, 2024

    2024

  7. [7]

    Simple and controllable music generation

    Copet, J., Kreuk, F., Gat, I., Remez, T., Kant, D., Synnaeve, G., Adi, Y., and D \'e fossez, A. Simple and controllable music generation. Advances in Neural Information Processing Systems, 36, 2024

    2024

  8. [8]

    Fma: A dataset for music analysis

    Defferrard, M., Benzi, K., Vandergheynst, P., and Bresson, X. Fma: A dataset for music analysis. arXiv preprint arXiv:1612.01840, 2016

    arXiv 2016

Show all 46 references
  1. [9]

    Audio retrieval with wavtext5k and clap training

    Deshmukh, S., Elizalde, B., and Wang, H. Audio retrieval with wavtext5k and clap training. arXiv preprint arXiv:2209.14275, 2022

    2022 arXiv

  2. [10]

    Clotho: An audio captioning dataset

    Drossos, K., Lipping, S., and Virtanen, T. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 736--740. IEEE, 2020

    2020

  3. [11]

    Fluid: Scaling autoregressive text-to-image generative models with continuous tokens

    Fan, L., Li, T., Qin, S., Li, Y., Sun, C., Rubinstein, M., Sun, D., He, K., and Tian, Y. Fluid: Scaling autoregressive text-to-image generative models with continuous tokens. arXiv preprint arXiv:2410.13863, 2024

    2024 arXiv

  4. [12]

    F., Seetharaman, P., Kumar, R., and Pardo, B

    Garcia, H. F., Seetharaman, P., Kumar, R., and Pardo, B. Vampnet: Music generation via masked acoustic token modeling. arXiv preprint arXiv:2307.04686, 2023

    2023 arXiv

  5. [13]

    F., Ellis, D

    Gemmeke, J. F., Ellis, D. P. W., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., and Ritter, M. Audio set: An ontology and human-labeled dataset for audio events. In Proc. IEEE ICASSP 2017, New Orleans, LA, 2017

    2017

  6. [14]

    Text-to-audio generation using instruction-tuned llm and latent diffusion model

    Ghosal, D., Majumder, N., Mehrish, A., and Poria, S. Text-to-audio generation using instruction-tuned llm and latent diffusion model. arXiv preprint arXiv:2304.13731, 2023

    2023 arXiv

  7. [15]

    Ezaudio: Enhancing text-to-audio generation with efficient diffusion transformer

    Hai, J., Xu, Y., Zhang, H., Li, C., Wang, H., Elhilali, M., and Yu, D. Ezaudio: Enhancing text-to-audio generation with efficient diffusion transformer. arXiv preprint arXiv:2409.10819, 2024

    2024 arXiv

  8. [16]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

    2016

  9. [17]

    Masked autoencoders are scalable vision learners

    He, K., Chen, X., Xie, S., Li, Y., Doll \'a r, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 16000--16009, 2022

    2022

  10. [18]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017

    2017

  11. [19]

    and Salimans, T

    Ho, J. and Salimans, T. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. URL https://openreview.net/forum?id=qw8AKxfYbI

    2021

  12. [20]

    Denoising diffusion probabilistic models

    Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33: 0 6840--6851, 2020

    2020

  13. [21]

    Make-an-audio 2: Temporal-enhanced text-to-audio generation

    Huang, J., Ren, Y., Huang, R., Yang, D., Ye, Z., Zhang, C., Liu, J., Yin, X., Ma, Z., and Zhao, Z. Make-an-audio 2: Temporal-enhanced text-to-audio generation. arXiv preprint arXiv:2305.18474, 2023

    2023 arXiv

  14. [22]

    Fréchet audio distance: A reference-free metric for evaluating music enhancement algorithms

    Kilgour, K., Zuluaga, M., Roblek, D., and Sharifi, M. Fréchet audio distance: A reference-free metric for evaluating music enhancement algorithms. In Interspeech 2019, pp.\ 2350--2354, 2019. doi:10.21437/Interspeech.2019-2219

    2019 doi

  15. [23]

    D., Kim, B., Lee, H., and Kim, G

    Kim, C. D., Kim, B., Lee, H., and Kim, G. A udio C aps: Generating captions for audios in the wild. In Burstein, J., Doran, C., and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Lan...

    2019 doi

  16. [24]

    Improving text-to-audio models with synthetic captions

    Kong, Z., gil Lee, S., Ghosal, D., Majumder, N., Mehrish, A., Valle, R., Poria, S., and Catanzaro, B. Improving text-to-audio models with synthetic captions. In Synthetic Data’s Transformative Role in Foundational Speech Models, pp.\ 1--5, 2024. doi:10.21437/SynData4GenAI.2024-1

    2024 doi

  17. [25]

    Audiogen: Textually guided audio generation

    Kreuk, F., Synnaeve, G., Polyak, A., Singer, U., D \'e fossez, A., Copet, J., Parikh, D., Taigman, Y., and Adi, Y. Audiogen: Textually guided audio generation. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=CYK7RfcOzQ4

    2023

  18. [26]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pp.\ 12888--12900. PMLR, 2022

    2022

  19. [27]

    Mage: Masked generative encoder to unify representation learning and image synthesis

    Li, T., Chang, H., Mishra, S., Zhang, H., Katabi, D., and Krishnan, D. Mage: Masked generative encoder to unify representation learning and image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 2142--2152, 2023

    2023

  20. [28]

    Autoregressive image generation without vector quantization

    Li, T., Tian, Y., Li, H., Deng, M., and He, K. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406.11838, 2024

    2024 arXiv

  21. [29]

    Liu, H., Chen, Z., Yuan, Y., Mei, X., Liu, X., Mandic, D., Wang, W., and Plumbley, M. D. A udio LDM : Text-to-audio generation with latent diffusion models. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th Inte...

    2023

  22. [30]

    Liu, H., Yuan, Y., Liu, X., Mei, X., Kong, Q., Tian, Q., Wang, Y., Wang, W., Wang, Y., and Plumbley, M. D. Audioldm 2: Learning holistic audio generation with self-supervised pretraining. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024

    2024

  23. [31]

    Tango 2: Aligning diffusion-based text-to-audio generative models through direct preference optimization

    Majumder, N., Hung, C.-Y., Ghosal, D., Hsu, W.-N., Mihalcea, R., and Poria, S. Tango 2: Aligning diffusion-based text-to-audio generative models through direct preference optimization. In ACM Multimedia 2024, 2024. URL https://openreview.net/forum?id=7lqptq5dLG

    2024

  24. [32]

    and Mesaros, A

    Mart \' n-Morat \'o , I. and Mesaros, A. What is the ground truth? reliability of multi-annotator data for audio tagging. In 2021 29th European Signal Processing Conference (EUSIPCO), pp.\ 76--80. IEEE, 2021

    2021

  25. [33]

    D., Zou, Y., and Wang, W

    Mei, X., Meng, C., Liu, H., Kong, Q., Ko, T., Zhao, C., Plumbley, M. D., Zou, Y., and Wang, W. Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32: 0 33...

    2024

  26. [34]

    Tut database for acoustic scene classification and sound event detection

    Mesaros, A., Heittola, T., and Virtanen, T. Tut database for acoustic scene classification and sound event detection. In 2016 24th European Signal Processing Conference (EUSIPCO), pp.\ 1128--1132. IEEE, 2016

    2016

  27. [35]

    Nichol, A. Q. and Dhariwal, P. Improved denoising diffusion probabilistic models. In International conference on machine learning, pp.\ 8162--8171. PMLR, 2021

    2021

  28. [36]

    and Xie, S

    Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 4195--4205, 2023

    2023

  29. [37]

    Piczak, K. J. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, pp.\ 1015--1018, 2015

    2015

  30. [38]

    D., Ermon, S., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=HPuSIXJaa9

    2023

  31. [39]

    Salamon, J., Jacoby, C., and Bello, J. P. A dataset and taxonomy for urban sound research. In Proceedings of the 22nd ACM international conference on Multimedia, pp.\ 1041--1044, 2014

    2014

  32. [40]

    Improved techniques for training gans

    Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016

    2016

  33. [41]

    Auto-acd: A large-scale dataset for audio-language representation learning

    Sun, L., Xu, X., Wu, M., and Xie, W. Auto-acd: A large-scale dataset for audio-language representation learning. In Proceedings of the 32nd ACM International Conference on Multimedia, pp.\ 5025--5034, 2024

    2024

  34. [42]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation

    Wu, Y., Chen, K., Zhang, T., Hui, Y., Berg-Kirkpatrick, T., and Dubnov, S. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (...

    2023

  35. [43]

    Comparing discrete and continuous space llms for speech recognition

    Xu, Y., Zhang, S.-X., Yu, J., Wu, Z., and Yu, D. Comparing discrete and continuous space llms for speech recognition. In Interspeech 2024, pp.\ 2509--2513, 2024. doi:10.21437/Interspeech.2024-1533

    2024 doi

  36. [44]

    Continuous speech tokens makes llms robust multi-modality learners

    Yuan, Z., Liu, Y., Liu, S., and Zhao, S. Continuous speech tokens makes llms robust multi-modality learners. arXiv preprint arXiv:2412.04917, 2024

    2024 arXiv

  37. [45]

    L., Remez, T., Kreuk, F., Copet, J., D \'e fossez, A., Synnaeve, G., and Adi, Y

    Ziv, A., Gat, I., Lan, G. L., Remez, T., Kreuk, F., Copet, J., D \'e fossez, A., Synnaeve, G., and Adi, Y. Masked audio generation using a single non-autoregressive transformer. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.n...

    2024

  38. [46]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.