Pith. sign in

REVIEW 12 cited by

Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2304.13731 v2 pith:UFAP3XQO submitted 2023-04-24 eess.AS cs.AIcs.CLcs.SD

Deepanway Ghosal , Navonil Majumder , Ambuj Mehrish , Soujanya Poria This is my paper · ORCID
classification eess.AScs.AIcs.CLcs.SD
keywords encodermodelaudiodiffusiongenerationinstruction-tunedlanguagelatent
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The immense scale of the recent large language models (LLM) allows many interesting properties, such as, instruction- and chain-of-thought-based fine-tuning, that has significantly improved zero- and few-shot performance in many natural language processing (NLP) tasks. Inspired by such successes, we adopt such an instruction-tuned LLM Flan-T5 as the text encoder for text-to-audio (TTA) generation -- a task where the goal is to generate an audio from its textual description. The prior works on TTA either pre-trained a joint text-audio encoder or used a non-instruction-tuned model, such as, T5. Consequently, our latent diffusion model (LDM)-based approach TANGO outperforms the state-of-the-art AudioLDM on most metrics and stays comparable on the rest on AudioCaps test set, despite training the LDM on a 63 times smaller dataset and keeping the text encoder frozen. This improvement might also be attributed to the adoption of audio pressure level-based sound mixing for training set augmentation, whereas the prior methods take a random mix.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 15 citations worldwide. Full citation record

  1. Latent Swap Joint Diffusion for 2D Long-Form Latent Generation

    cs.SD 2025-02 conditional novelty 7.0 of 10

    A training-free latent swap method that replaces averaging with binary swapping in joint diffusion, improving long-form audio spectrum and panorama generation.

  2. Diff-Symbo: Text-Controlled Long-Duration Symbolic Music Generation Using Autoregressive Latent Diffusion Model

    cs.SD 2026-08 conditional novelty 6.0 of 10

    Diff-Symbo generates long, text-controlled symbolic music by autoregressively extending 8-bar latent diffusion segments conditioned on the previous segment's latent.

  3. Improving Text-to-Audio Instruction Following via Fine-Grained Feedback from Audio-Aware Large Language Models

    eess.AS 2026-07 conditional novelty 6.0 of 10

    Using audio-aware LLMs to judge event presence and temporal order as DPO rewards improves multi-event text-to-audio instruction following.

  4. UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Models

    eess.AS 2025-10 conditional novelty 6.0 of 10

    A single LLM can do ASR and zero-shot TTS on continuous speech features by switching between causal and bidirectional attention, reaching competitive but not state-of-the-art results.

  5. TTA-Bench: A Comprehensive Benchmark for Evaluating Text-to-Audio Models

    cs.SD 2025-09 conditional novelty 6.0 of 10

    TTA-Bench offers a seven-dimension, 2,999-prompt evaluation of ten text-to-audio models with 118,000 human ratings, covering quality, robustness, fairness, bias, and toxicity.

  6. Ego-centric Predictive Model Conditioned on Hand Trajectories

    cs.CV 2025-08 conditional novelty 6.0 of 10

    Ego-PM predicts future hand trajectories and then uses them to condition latent diffusion video generation, jointly outputting actions and future frames in egocentric and robotic scenes.

  7. JAM: A Tiny Flow-based Song Generator with Fine-grained Controllability and Aesthetic Alignment

    cs.SD 2025-07 conditional novelty 6.0 of 10

    JAM is a 530M-parameter flow-matching song generator that adds word- and phoneme-level timing control and duration control, achieving strong lyric fidelity and musicality scores when ground-truth timings are provided.

  8. Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction

    eess.AS 2025-07 conditional novelty 6.0 of 10

    A causal audio language model with continuous-valued tokens and masked next-token prediction matches diffusion-based text-to-audio quality with smaller, streamable models.

  9. IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling

    eess.AS 2025-05 conditional novelty 5.0 of 10

    On AudioCaps, IMPACT reports the best Fréchet Distance and Fréchet Audio Distance among the compared systems while generating audio faster than diffusion baselines.

  10. Insect-Foundation: A Foundation Model and Large Multimodal Dataset for Vision-Language Insect Understanding

    cs.CV 2025-02 conditional novelty 5.0 of 10

    An insect-specific vision-language assistant trained on a new 1M-image multimodal insect dataset with patch-matching self-supervision reports improved insect classification and VQA accuracy over LLaVA and prior self-s...

  11. FlowSonic: Stable Zero-Shot Music Editing via High-Order Trajectory Integration

    cs.SD 2026-07 reject novelty 4.0 of 10

    FlowSonic combines deterministic rectified-flow inversion, cached cross-attention injection, and a 'seeded' third-order Adams-Bashforth solver to report better timbre and genre edits on small datasets.

  12. Survey on AI-Generated Media Detection: From Non-MLLM to MLLM

    cs.CV 2025-02 unverdicted novelty 3.0 of 10

    A survey organizing AI-generated media detection into Non-MLLM and MLLM based methods, with task and benchmark taxonomies.

Pith tools