REVIEW 35 cited by
AudioLDM: Text-to-Audio Generation with Latent Diffusion Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Text-to-audio (TTA) system has recently gained attention for its ability to synthesize general audio based on text descriptions. However, previous studies in TTA have limited generation quality with high computational costs. In this study, we propose AudioLDM, a TTA system that is built on a latent space to learn the continuous audio representations from contrastive language-audio pretraining (CLAP) latents. The pretrained CLAP models enable us to train LDMs with audio embedding while providing text embedding as a condition during sampling. By learning the latent representations of audio signals and their compositions without modeling the cross-modal relationship, AudioLDM is advantageous in both generation quality and computational efficiency. Trained on AudioCaps with a single GPU, AudioLDM achieves state-of-the-art TTA performance measured by both objective and subjective metrics (e.g., frechet distance). Moreover, AudioLDM is the first TTA system that enables various text-guided audio manipulations (e.g., style transfer) in a zero-shot fashion. Our implementation and demos are available at https://audioldm.github.io.
Forward citations
Cited by 35 Pith papers
-
Analytic Distribution of Classifier-Free Guidance for Schedule Design
Deterministic CFG samples from p_t0 times an exponential path integral of the score discrepancy, and the resulting schedule DG-CFG reduces sampling steps at high guidance.
-
MAVIN: Multi-Shot Audio-Visual Generation with Customized Narrative Control
MAVIN proposes boundary-aware attention, ID-aware propagation, a multi-agent scripting pipeline, and the MAVINSet dataset as the first framework for multi-shot audio-visual generation with narrative control, claiming ...
-
A Sharp KL-Convergence Analysis for Diffusion Models under Minimal Assumptions
A new analysis shows O~(d/epsilon) steps suffice for KL-close diffusion sampling under only L2 score error and finite second moment assumptions, improving the known O~(d/epsilon^2).
-
Flow Matching Policy Gradients
FPO trains flow-based policies with PPO by replacing the likelihood ratio with an exponentiated flow matching loss difference.
-
Amortized Moment Matching for Visual Generation
Amortized Fréchet Distance uses neural nets to match conditional means and covariances, yielding stronger one-step visual generators than explicit FD-loss or multi-step teachers.
-
RPPNet: Perceptually-Grouped Rhythm-Pitch Primitives for Long-Term Structure Melody Generation via Boundary-Aware Modeling
RPPNet generates melodies by planning variable-length perceptually grouped rhythm-pitch primitives first and then decoding them into notes, beating bar-level baselines in subjective structure and musicality ratings.
-
FlashDiff: Efficient Regional Execution and Scheduling for Diffusion Model Serving
FlashDiff reduces diffusion serving latency by 30–97% and raises throughput 1.2–2.2× by adaptively skipping refinement of latent regions that no longer need it.
-
Dance to Music Generation leveraging Pre-training with Unpaired data and Contrastive Alignment
Beat-guided contrastive alignment of pretrained MotionBERT/MERT features plus ControlNet conditioning of AudioLDM improves dance–music alignment on AIST++ over a MusicGen textual-inversion baseline while remaining com...
-
SynSFX: Multi-Model Sound Effects Synthesis Dataset for Deepfake Detection and Evaluation
SynSFX provides a multi-generator sound-effect deepfake corpus showing speech detectors fail, joint training mitigates forgetting, but generalization to unseen generators remains poor due to artifact overfitting.
-
Making Separation-First Multi-Stream Audio Watermarking Feasible via Joint Training
Jointly training the watermark embedder/detector with the source separator enables ~1% bit-error-rate recovery of per-stem watermarks after mixing and separation, where independent training yields 15–35%.
-
Dual-End Consistency Model
DE-CM trains a flow-map consistency model on three sub-trajectories (coupling, instantaneous, noise-to-noisy) and reports 1.70 FID one-step on ImageNet 256.
-
UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Models
A single LLM can do ASR and zero-shot TTS on continuous speech features by switching between causal and bidirectional attention, reaching competitive but not state-of-the-art results.
-
Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction
Separating the text condition into a video caption and a visually grounded audio caption, and fusing the diffusion towers with dual cross-attention, gives the reported-best text-to-sounding-video quality and synchroni...
-
Audio-Guided Visual Editing with Complex Multi-Modal Prompts
A training-free framework that maps audio embeddings into Stable Diffusion's text space and fuses multiple audio/text prompts via per-patch residual noise selection, outperforming text-only editors on new audio-visual...
-
Hear-Your-Click: Interactive Object-Specific Video-to-Audio Generation
A new interactive method lets users click on an object in a video and generates audio for just that object, using mask-conditioned contrastive fine-tuning and latent diffusion.
-
Diff-TONE: Timestep Optimization for iNstrument Editing in Text-to-Music Diffusion Models
Diff-TONE selects the prompt-swap timestep using a distilled latent instrument classifier, improving content preservation over fixed timestep baselines in text-to-music diffusion editing.
-
In-the-wild Audio Spatialization with Flexible Text-guided Localization
A text-guided latent diffusion model converts monaural audio into binaural audio whose perceived directions and distances follow user-specified text prompts.
-
AudioGenie: A Training-Free Multi-Agent Framework for Diverse Multimodality-to-Multiaudio Generation
A training-free multi-agent framework that decomposes multimodal inputs into audio events, selects specialized generators, and self-corrects outputs to produce multiple audio types.
-
SpecMaskFoley: Steering Pretrained Spectral Masked Generative Transformer Toward Synchronized Video-to-audio Synthesis via ControlNet
A ControlNet branch plus a frequency-aware feature aligner lets a pretrained masked generative TTA model produce video-synchronized foley, beating several from-scratch models on VGGSound.
-
Exploring Efficient Waveform Diffusion Models for Foley Sound Generation
Dual-path attention over time-frequency representations lets a 3.26M-parameter waveform diffusion model match the quality of 50M+ parameter Foley generators.
-
Cross-modal Consistency Guidance for Robust Emotion Control in Auto-Regressive TTS Models
Introduces CCG-CFG with inconsistency-based dynamic scales and hard-sample mining distillation to boost emotional alignment in auto-regressive TTS, reporting up to 12% absolute gains in emotion recognition accuracy.
-
Via Score to Performance: Efficient Human-Controllable Long Song Generation with Bar-Level Symbolic Notation
A bar-level symbolic-score song generator (BACH) is claimed to beat published systems and commercial Suno on human-rated quality, duration, and efficiency, but the supporting full text is corrupted and unverifiable.
-
AudioGen-Omni: A Unified Multimodal Diffusion Transformer for Video-Synchronized Audio, Speech, and Song Generation
A single multimodal diffusion transformer generates video-synchronized general audio, speech, and song from flexible combinations of video, text, and lyrics inputs.
-
Balancing Information Preservation and Disentanglement in Self-Supervised Music Representation Learning
A multi-view SSL framework with combined reconstruction and separation-based contrastive losses obtains disentangled pitch and instrument subspaces without the accuracy loss seen with contrastive-only training on NSynth.
-
Aether Weaver: Multimodal Affective Narrative Co-Generation with Dynamic Scene Graphs
An integrated storytelling framework that generates text, scene graphs, images, and sound together reports higher expert-rated coherence than a sequential baseline, but the evaluation is small and qualitative.
-
SonicGauss: Position-Aware Physical Sound Synthesis for 3D Gaussian Representations
A three-stage diffusion pipeline maps 3D Gaussian Splatting object representations to position-dependent impact sounds, trained first on text captions and then on real recordings.
-
CHORDS: Diffusion Sampling Accelerator with Multi-core Hierarchical ODE Solvers
CHORDS accelerates diffusion sampling by running hierarchical ODE solvers on multiple cores, with slower solvers rectifying faster ones, achieving up to 2.9x speedup without retraining.
-
ADMC: Attention-based Diffusion Model for Missing Modalities Feature Completion
An attention-based diffusion model that generates missing modality features, combined with independently trained extractors, achieves state-of-the-art emotion and intent recognition on IEMOCAP and MIntRec.
-
UmbraTTS: Adapting Text-to-Speech to Environmental Contexts with Flow Matching
UmbraTTS jointly synthesizes speech and environmental audio via conditional flow matching, conditioned on text and acoustic context, with controllable background volume.
-
Inference-time Scaling for Diffusion-based Audio Super-resolution
Generating 120 candidate super-resolved audios and choosing the best by task-specific verifiers improves speech, music, and sound effects over single-sample diffusion output, at 120x compute.
-
Diffusion Models for Time Series Forecasting: A Survey
A survey classifies diffusion-based time series forecasting models into a two-axis taxonomy by conditioning source and integration method.
-
WhisQ: Cross-Modal Representation Learning for Text-to-Music MOS Prediction
WhisQ uses Whisper and Qwen with co-attention and optimal transport to predict music quality and text-alignment scores, but its reported improvements do not match its own data.
-
AI-Based Sound Effect Generation: A Narrative Review of Generative Models Across Input Modalities
A narrative review of 30 recent papers classifies AI sound-effect generators by input modality and summarizes reported progress and remaining limitations in temporal sync, evaluation, and controllability.
-
WaveLLDM: Design and Development of a Lightweight Latent Diffusion Model for Speech Enhancement and Restoration
WaveLLDM, a lightweight latent diffusion model with a neural codec, achieves low spectral distortion (LSD 0.48-0.60) on speech restoration but scores far below SOTA on PESQ and STOI.
-
ASAudio: A Survey of Advanced Spatial Audio Research
A comprehensive survey that systematically categorizes spatial audio research by representation, task, dataset, and evaluation.
Discussion (0). Sign in to comment.