Pith. sign in

REVIEW 4 major objections 7 minor 66 references

VoxAudio: Vocalized Audio Synthesis via Multi-Reward Autoregressive Flow Matching

T0 review · 4 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read VoxAudio generates intelligible quoted speech inside environmental soundscapes in real time.

desk verdict A serious, well-executed systems paper that makes a real advance in unified vocalized audio synthesis, with a genuine but partial evaluator-overlap concern that should be addressed before publication. read the letter →

arxiv 2608.12951 v1 pith:QDG5SBON submitted 2026-08-13 cs.SD

Wenxiang Guo , Changhao Pan , Ziyue Jiang , Fei Wu , Zhou Zhao This is my paper · ORCID
classification cs.SD
keywords vocalizedaudiosynthesistext-to-audiospeech-in-soundscapeautoregressiveflowmatchingstreaminggenerationmulti-rewardfine-tuningtemporalgroundingcaptioningdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VoxAudio aims to make quoted speech intelligible inside a generated environmental soundscape, so a caption like "a boy exclaims, 'What a lovely cat!' over jingling bells" yields audio in which those exact words are audible at the right moment, not vocal murmur. The paper argues that this becomes possible by treating speech-in-scene as one causal generative process: a chunk-wise autoregressive flow-matching model that streams audio chunk by chunk while later chunks are still denoising. It claims that training with randomized chunk boundaries makes the streaming granularity a free choice at inference, and that a multi-reward post-training stage, combining semantic fidelity, word accuracy, aesthetics, and temporal grounding, is what turns a sound-effects generator into a speech-capable one. If true, unified text-to-audio systems could replace the current two-model pipeline of TTS plus post-hoc mixing, with direct control over when speech starts, how long it lasts, and how loud it sits in the scene.

What carries the argument

The load-bearing object is the chunk-wise causal factorization of the flow-matching objective: the latent audio sequence is split into chunks, each with an independent noise level $t_i$, and the vector-field prediction for chunk $i$ is conditioned on all preceding chunks at their own noise levels ($\{(z^{(j)}_{t_j})\}_{j<i}$ in Eq. 2). This makes the model autoregressive across chunks and continuous within each chunk. Three mechanisms hang off it: randomized chunk boundaries during pretraining, where every latent frame opens a new chunk with probability $p_b$, so the inference chunk size is arbitrary; chunk-wise causal attention that allows full visibility inside a chunk but strict causality across chunks; and a lag-$\Delta$ sliding-window schedule (Eq. 5, Algorithm 1) that keeps a fixed denoising step difference between adjacent chunks so streaming emits audio while later chunks refine. The multi-reward Negative-aware FineTuning (NFT) loss (Eq. 4) replays exactly these asynchronous noise snapshots and steers the vector field toward implicit positive and negative targets weighted by group-normalized rewards.

What would settle it

Run the same checkpoint under two noise-level schedules: the training-time independent per-chunk sampling and the exact lag-$\Delta$ sliding-window schedule. If the lag-$\Delta$ stream yields substantially worse word error rate or temporal grounding than the independently sampled schedule on the same captions, the train-test gap the paper claims to close is not closed. A sharper test is to compare the default randomized-boundary checkpoint against a fixed-boundary checkpoint at mismatched inference chunk sizes: the mechanism predicts a sharp degradation for the fixed-boundary model and none for the randomized one.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that vocalized audio generation can be cast as causal autoregressive flow matching over latent chunks, where each chunk carries its own noise level, so the model learns the joint distribution of speech and scene as a single streaming process rather than as two separately generated tracks. The corollary claims are: independent per-chunk noise levels during training put the asynchronous lag-\$\Delta$ schedule used by sliding-window inference inside the training distribution, closing the train-test gap; randomizing chunk boundaries during pretraining lets the inference chunk size be chosen freely without retraining; and optimizing a weighted combination of semantic, linguistic, aesthetic, and temporal rewards via negative-aware fine-tuning is what makes the embedded speech intelligible and correctly placed. The paper reports word error rates of 0.038 on its 10-second vocalized-audio benchmark versus 0.264 for the strongest prior unified baseline, 1.61% on a standard English TTS benchmark, and streaming at a 0.32 real-time factor.

Load-bearing premise

The load-bearing premise is that training with independently sampled per-chunk noise levels puts the lag-$\Delta$ asynchronous schedule used at streaming inference inside the training distribution, so streamed chunks are not out-of-distribution for the model.

Editorial extensions

If this is right

  • A single text-to-audio model can now render quoted speech verbatim inside a scene, cutting word error rate from 0.264 to 0.038 on 10-second vocalized audio relative to the prior unified baseline.
  • Streaming generation with key-value caching makes first audio available in about 1.4 seconds and sustains a 0.32 real-time factor on one GPU, so interactive and variable-duration use becomes practical.
  • Because chunk boundaries are randomized at training, the operator can pick any inference chunk size or step lag; the ablations show small chunks and small lag favor latency, while larger values favor linguistic accuracy, with both metrics degrading at the extremes.
  • On pure speech benchmarks the same model reaches 1.61% word error rate, competitive with dedicated TTS systems even without reference-speaker prompts, so vocalized-audio training transfers to clean speech synthesis.
  • The new VoxCorpus and VoxBench, with quoted, time-stamped transcripts and a temporal-grounding metric, give the field a way to measure whether speech content and placement are actually correct.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's claims, the same chunk-agnostic causal flow-matching design suggests a natural upgrade path for live dubbing and podcast tools: a first-chunk latency near 1.4 seconds already approaches interactive use, and the lag-$\Delta$ parameter is an explicit knob to trade response time against intelligibility, which I predict will need per-application tuning.
  • The multi-reward objective is generic enough that speaker-timbre and prosody rewards, named as future work in the paper's limitation section, should be a direct addition rather than a redesign, potentially giving the same architecture fine-grained voice control.
  • A testable consequence of the paper's mechanism claim: a model trained with fixed chunk boundaries should show a sharp word-error-rate penalty when evaluated at a mismatched chunk size, whereas the randomized-boundary model should stay flat; the paper's ablation hints at this, but a systematic sweep would confirm it.
  • Because word error rate is computed through an external recognizer, the linguistic reward could in principle be gamed by producing easy-to-recognize but unnatural speech; the paper guards against this with the aesthetic and semantic rewards, but the interaction between those rewards deserves direct study.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. VoxAudio is a causal autoregressive flow-matching model for generating audio in which intelligible, quoted speech is embedded in an environmental soundscape. The paper contributes (i) a chunk-wise causal factorization with independent per-chunk noise levels and randomized chunk boundaries during pretraining, enabling sliding-window streaming inference with KV caching and variable target durations; (ii) a multi-reward Negative-aware FineTuning (NFT) stage that jointly optimizes semantic, linguistic (Whisper WER), aesthetic, and temporal-grounding (PE-A-Frame TG-IoU) rewards; and (iii) two new resources, VoxCorpus and VoxBench. Experiments compare VoxAudio with general T2A, unified audio, and TTS baselines on VoxBench, MECAT-en, AudioCaps, and SEED-TTS-EVAL, reporting large WER improvements over Dasheng-AudioGen (0.038 vs 0.264 on VoxBench-10s), competitive SEED-TTS-EVAL WER (1.61%), and 0.32 RTF streaming.

Significance. If the reported results hold, VoxAudio would establish a plausible path to end-to-end vocalized audio synthesis with lexical and temporal control, a genuinely under-served task. The strengths are substantial: the chunk-agnostic causal AR-FM formulation is conceptually clean and addresses a real limitation of fixed-length non-causal generators; the two-stage training and the ablation set are internally consistent; the use of independent per-chunk noise levels followed by reward-based alignment is a reasonable design; and the efficiency measurements (RTF, latency, first-chunk latency) are reported carefully. The main reservations are evaluative and presentational: the headline WER and TG-IoU metrics overlap with the training rewards, the streaming pseudocode does not transparently match the training objective, and the dataset and benchmark are not yet released, which limits external verification. These issues are fixable with additional experiments and clarification rather than requiring a change in the core approach.

major comments (4)
  1. [III-C, V-B, Table I] Section III-C defines reward terms using Whisper WER (linguistic accuracy) and PE-A-Frame TG-IoU (temporal grounding), and Section V-B evaluates WER with Whisper-large-v3 and TG-IoU with PE-A-Frame at threshold 0.3. The only stated separation between RL and evaluation is for the CLAP checkpoint; no such separation is reported for Whisper or PE-A-Frame. Since the headline VoxBench-10s claims (WER 0.038, TG-IoU 0.654 in Table I) are measured with the same learned functions that were optimized during NFT, the improvement over Dasheng-AudioGen may partly reflect scorer overfitting rather than human-perceived intelligibility and timing. SEED-TTS-EVAL and MOS-C mitigate but do not eliminate this concern. Please add an independent evaluation: a different ASR not used in training, human transcription WER, and human interval annotations or a temporal detector not used as a reward.
  2. [III-A, III-E, Algorithm 1] The training objective in Eq. (2) conditions the prediction for chunk i on all preceding chunks at their own noise levels. In Algorithm 1, the KV cache K is updated only when a chunk is fully decoded (lines 7-9), so active chunks that are still noisy are not available as context. For lag Δ>0, a newly activated chunk must attend to an earlier chunk that is still being denoised (e.g., with Δ=5 and N=25, chunk 2 becomes active when chunk 1 is at t=5δ). As written, the per-chunk loop cannot provide that context, making the inference procedure inconsistent with Eq. (2). Please clarify whether the implementation performs a joint forward pass over the active window, stores current noisy latents in K, or otherwise restores the training-time conditioning; this is load-bearing for the streaming claims in Section III-E and Table IV.
  3. [III-C, V-A, Table I] Section III-C first names LAION-CLAP as the semantic reward and then says "we adopt PEAV as the semantic metric in our experiments"; Section V-A lists the reward weights as (w_CLAP, w_Aes, w_WER, w_TG), so it is unclear which model is actually the semantic reward. This ambiguity matters because Table I shows the 'w/o Semantic Reward' row improving TG-IoU from 0.654 to 0.825 and nearly matching the full model's WER (0.042 vs 0.038), while only PEAV and CLAP drop. That pattern is difficult to reconcile with Section V-D's claim that the semantic reward is an essential anchor that prevents collapse. Please state the reward definition explicitly and explain the ablation behavior.
  4. [III-A, V-D] Section III-A argues that independent per-chunk noise sampling places the asynchronous states visited by streaming inference inside the training distribution. The specific 'under-represented' phrasing is not literally correct if per-chunk noise levels are iid with full support, since every tuple then has the same density. The load-bearing issue is instead that the iid training distribution and the lag-Δ schedule of Eq. (5)/Algorithm 1 have different joint structure, and the model may underweight the correlated trajectories used at inference. Please either train with a mixture that contains Eq. (5) schedules or add an experiment that varies the training noise-level distribution while holding the inference schedule fixed; the current Table IV varies only inference-time parameters and does not isolate this factor.
minor comments (7)
  1. [II-C] In Section II-C, "Flow-GRPO [36] applys" should read "applies".
  2. [Table IV] Table IV is difficult to parse: the "Default (C=16, Δ=5)" row is not clearly part of any labeled ablation group, and its WER of 0.076 differs from Table I's VoxAudio WER of 0.038 without explanation. Please clarify the evaluation subset and the relationship between the two tables.
  3. [V-B, Appendix C-B] Section V-B says MOS tests use 5 listeners, while Appendix C-B says a pool of 16 independent evaluators ensures each clip receives at least five ratings; please reconcile these descriptions.
  4. [V-C, Table I] The claim that MOS-C is "significantly higher" for VoxAudio is not supported by the reported confidence intervals (4.51±0.26 vs 4.16±0.34 overlap); please rephrase or provide a formal significance test.
  5. [V-D] In Section V-D, "Dasheng-AudioGe" is missing the final 'n'; please correct the typo.
  6. [IV-A, Appendix A] The abstract and main text say code and demos are available, but Appendix A says VoxCorpus and VoxBench "will be released" under CC BY-NC-SA 4.0; please state the actual availability of the data and benchmark for reproducibility.
  7. [VI] In Section VI, "mono-lingual" should be "monolingual".

Circularity Check

1 steps flagged · score 5.0 of 10

Whisper and PE-A-Frame are used both as NFT rewards and as the headline WER/TG-IoU metrics, so the main intelligibility and timing gains are partly measured against the optimization target.

  1. fitted input called prediction [Section III-C.1 (multi-reward objective) and Section V-B / Appendix C-B (evaluation protocol)]
    "Linguistic accuracy: Whisper [44] computes the Word Error Rate (WER). ... Temporal grounding: the intersection-over-union between the speech spans detected by PE-A-Frame [46] and the interval annotated in the caption. ... Speech metrics are quantified by word error rate (WER) computed with Whisper-large-v3 ... Temporal grounding (TG-IoU) measures the interval IoU, |I_det∩I_inst|/|I_det∪I_inst|, between the instructed interval I_inst and the speech span I_det detected by PE-A-Frame [46] (threshold 0.3)."

    The multi-reward NFT objective (Eq. 4) uses r assembled from Whisper WER and PE-A-Frame TG-IoU, with weights (w_CLAP,w_Aes,w_WER,w_TG)=(1.0,0.005,0.06,0.05) (Sec. V-A). The headline VoxBench-10s results (WER 0.038; TG-IoU 0.654) and MECAT WER are then measured with the identical scorers: Whisper-large-v3 and PE-A-Frame at threshold 0.3 (Sec. V-B). The paper explicitly separates only the CLAP checkpoint between training and evaluation, and no analogous separation is reported for Whisper or PE-A-Frame.

full rationale

The core technical derivation is self-contained: AR-FM with randomized chunk boundaries, the flow-matching objective (Eqs. 1-2), the causal attention mask (Eq. 3), the NFT loss (Eq. 4), the asynchronous schedule (Eq. 5), and the sliding-window algorithm are all explicitly specified and do not reduce to their own inputs. The dataset construction is also independent of the evaluation claims. The circularity concern is confined to the preference-alignment evaluation: the same Whisper and PE-A-Frame models that appear as reward functions in Eq. (4) are reused as the headline WER and TG-IoU metrics in Section V-B, and the paper only separates the CLAP checkpoint between training and evaluation. The strong WER/TG improvements on VoxBench are therefore in part optimization against the measurement itself. This is partially mitigated by external benchmarks (SEED-TTS-EVAL, MECAT, AudioCaps), human MOS-C ratings, and ablations showing the reward terms move the metrics, but the lack of an independent transcription check or human interval-annotation check for the two headline quantities keeps the validation from being fully external. No load-bearing self-citation or imported uniqueness theorem was found.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The method rests on standard flow matching plus several hand-chosen hyperparameters. The load-bearing domain assumptions are the frozen VAE's phonetic fidelity, the reliability of four proxy reward models, and the coverage of the inference noise schedule by training; these are plausible but not independently demonstrated in the preprint.

free parameters (6)
  • Chunk-boundary probability p_b = 0.15
    Sets the geometric chunk length distribution in supervised training (Section V-A); selected by hand and not swept in the paper.
  • Multi-reward weights (w_CLAP, w_Aes, w_WER, w_TG) = (1.0, 0.005, 0.06, 0.05)
    Manual balance across semantic, aesthetic, linguistic, and temporal rewards (Section V-A); no sensitivity analysis is reported.
  • Inference chunk size C = 16 (swept 8-20)
    Streaming granularity at inference; the paper shows C=8 and C=20 degrade some metrics in Table IV.
  • Streaming step lag Delta = 5 (swept 0-25)
    Latency-quality trade-off; Delta=0 collapses WER and Delta=25 raises RTF above 1 in Tables IV and VI.
  • Text classifier-free guidance scale s = 5.0
    Used at inference and in RL rollouts; no sensitivity scan is shown for this value.
  • PE-A-Frame detection threshold = 0.3
    Converts frame scores into speech spans for the TG-IoU metric and is therefore central to the temporal-grounding claims (Appendix C-B).
assumptions (5)
  • domain assumption The frozen Universe VAE maps speech and sound into a shared latent space without losing the lexical content needed for WER evaluation.
    Invoked in Section III-B; if the VAE discarded phonetic detail, no downstream objective could recover intelligible quoted speech.
  • domain assumption Independent per-chunk noise levels in supervised training cover the asynchronous sliding-window schedule used at inference.
    Argued in Sections III-A and III-E; if the training distribution under-weights lag-Delta states, streaming rollouts become out-of-distribution.
  • standard math The linear optimal-transport flow matching path with Gaussian endpoints is a valid generative model for audio latents.
    Standard flow matching background invoked in Equations (1) and (2).
  • domain assumption Whisper, PE-A-Frame, PEAV, LAION-CLAP, and Audiobox Aesthetics are valid proxies for linguistic accuracy, temporal placement, semantic fidelity, and acoustic quality.
    Used as rewards in Section III-C and as evaluation metrics in Section V-B; optimizing these proxies is assumed to transfer to human judgment.
  • domain assumption The real narrative corpus annotations produced by the Qwen and Gemini transcribe-then-fuse pipeline are accurate enough to provide verbatim transcript supervision.
    Section IV-A and Appendix A; captioning errors would inject wrong speech targets into training.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VoxAudio: Vocalized Audio Synthesis via Multi-Reward Autoregressive Flow Matching." pith.science (2026). https://pith.science/paper/QDG5SBON

@misc{pith2026260812951,
  author       = {Pith},
  title        = {Pith review of: VoxAudio: Vocalized Audio Synthesis via Multi-Reward Autoregressive Flow Matching},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QDG5SBON}},
  note         = {Machine review of arXiv:2608.12951}
}
read the original abstract

Vocalized audio synthesis, the task of generating audio in which intelligible speech is embedded within an environmental soundscape, underpins applications such as podcast production and video dubbing. Existing Text-to-Audio (T2A) systems either reduce quoted speech to unintelligible vocal murmur or delegate it to a separate TTS model with post-hoc mixing, which forfeits control over when speech occurs and how it interacts with the scene. We present VoxAudio, a causal autoregressive flow matching model that addresses this problem from three complementary aspects. At the architecture level, chunk-wise causal factorization with independent per-chunk noise levels lets audio be emitted through sliding-window streaming inference with KV caching at variable target durations; to enable inference at arbitrary chunk granularities, we further pretrain the model with randomized chunk boundaries. At the preference level, multi-reward Negative-aware FineTuning (NFT) jointly optimizes semantic fidelity, linguistic accuracy, aesthetic quality, and temporal grounding At the data level, to supply the missing supervision for vocal content, we build VoxCorpus, a large-scale corpus whose captions quote the verbatim transcript of embedded speech with time intervals, and VoxBench, an interval-annotated benchmark with a temporal-grounding metric. Experiments on four benchmarks spanning general audio, speech, and unified vocalized audio validate the effectiveness and efficiency of VoxAudio. Our code and demos are available at https://voxaudio.github.io.

Figures

Figures reproduced from arXiv: 2608.12951 by the authors.

Figure 1
Figure 1. Comparison of Audio Generation Paradigms. Decoupled Pipeline: Conventional methods synthesize soundscapes and speech independently, requiring post-hoc mixing. VoxAudio: A unified system for vocalized audio synthesis from single prompts. effects as mutually exclusive tasks or produce clips of a fixed length, offering little control over when events occur. Third, as illustrated in [PITH_FULL_IMAGE:figures/full_fig_p0… view at source ↗
Figure 2
Figure 2. Overview of the VoxAudio Framework. Top: The workflow for VoxAudio dataset construction. Middle: The training pipeline, comprising chunk-agnostic autoregressive flow-matching supervised training and preference alignment NFT with multi-dimensional rewards. Bottom: The sliding-window streaming inference mechanism, which facilitates real-time, continuous audio synthesis. optimization to denoising trajectories, while Fl… view at source ↗
Figure 3
Figure 3. Detailed architecture and attention mechanism of the Causal Diffusion Transformer. (a) The causal joint block facilitates interaction between audio latents and conditioning features; (b) The causal fused block refines integrated representations through deep self-attention; (c) The chunk-wise causal attention mask, where shaded regions indicate masked-out tokens, ensuring that each temporal chunk only attends to curr… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The Generated Mel-spectrograms of VoxAudio (top) and Dasheng-AudioGen (bottom). [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Comprehensive statistical analysis of VoxCorpus. (a) Histogram of audio duration. (b) Distribution of caption character lengths for Short and Complex [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Prompt template for generating Short and Complex structured audio descriptions based on raw captions and transcriptions. [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Prompt template used for generating diverse, time-aligned audio scene candidates with mandatory vocal content from keywords. [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 17 canonical work pages

  1. [1]

    Make-an-audio: Text-to-audio generation with prompt- enhanced diffusion models,

    R. Huang, J. Huang, D. Yang, Y . Ren, L. Liu, M. Li, Z. Ye, J. Liu, X. Yin, and Z. Zhao, “Make-an-audio: Text-to-audio generation with prompt- enhanced diffusion models,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 13 916–13 932

    2023

  2. [2]

    Audioldm: Text-to-audio generation with latent diffusion models,

    H. Liu, Z. Chen, Y . Yuan, X. Mei, X. Liu, D. Mandic, W. Wang, and M. D. Plumbley, “Audioldm: Text-to-audio generation with latent diffusion models,”arXiv preprint arXiv:2301.12503, 2023

    arXiv 2023

  3. [3]

    Audiogen: Textually guided audio generation,

    F. Kreuk, G. Synnaeve, A. Polyak, U. Singer, A. D ´efossez, J. Copet, D. Parikh, Y . Taigman, and Y . Adi, “Audiogen: Textually guided audio generation,”arXiv preprint arXiv:2209.15352, 2022

    arXiv 2022

  4. [4]

    Diffsound: Discrete diffusion model for text-to-sound generation,

    D. Yang, J. Yu, H. Wang, W. Wang, C. Weng, Y . Zou, and D. Yu, “Diffsound: Discrete diffusion model for text-to-sound generation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 1720–1733, 2023

    2023

  5. [6]

    Uniaudio: An audio foundation model toward universal audio generation,

    D. Yang, J. Tian, X. Tan, R. Huang, S. Liu, X. Chang, J. Shi, S. Zhao, J. Bian, X. Wuet al., “Uniaudio: An audio foundation model toward universal audio generation,”arXiv preprint arXiv:2310.00704, 2023

    arXiv 2023

  6. [7]

    Audiobox: Unified audio gener- ation with natural language prompts,

    A. Vyas, B. Shi, M. Le, A. Tjandra, Y .-C. Wu, B. Guo, J. Zhang, X. Zhang, R. Adkins, W. Nganet al., “Audiobox: Unified audio gener- ation with natural language prompts,”arXiv preprint arXiv:2312.15821, 2023

    arXiv 2023

  7. [8]

    Dasheng audiogen: A unified model for generating coherent audio scenes from text,

    J. Mei, H. Dinkel, Y . Niu, X. Sun, G. Li, Y . Liao, J. Zhou, J. Zhang, J. Luan, and M. Wu, “Dasheng audiogen: A unified model for generating coherent audio scenes from text,”arXiv preprint arXiv:2605.27838, 2026

    arXiv 2026

  8. [9]

    Diffusionnft: Online diffusion reinforcement with forward process,

    K. Zheng, H. Chen, H. Ye, H. Wang, Q. Zhang, K. Jiang, H. Su, S. Ermon, J. Zhu, and M.-Y . Liu, “Diffusionnft: Online diffusion reinforcement with forward process,”arXiv preprint arXiv:2509.16117, 2025

    arXiv 2025

Show all 66 references
  1. [10]

    Make-an-audio 2: Temporal-enhanced text-to- audio generation,

    J. Huang, Y . Ren, R. Huang, D. Yang, Z. Ye, C. Zhang, J. Liu, X. Yin, Z. Ma, and Z. Zhao, “Make-an-audio 2: Temporal-enhanced text-to- audio generation,”arXiv preprint arXiv:2305.18474, 2023

    2023 arXiv

  2. [11]

    Audioldm 2: Learning holistic audio generation with self-supervised pretraining,

    H. Liu, Y . Yuan, X. Liu, X. Mei, Q. Kong, Q. Tian, Y . Wang, W. Wang, Y . Wang, and M. D. Plumbley, “Audioldm 2: Learning holistic audio generation with self-supervised pretraining,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 2871–2883, 2024

    2024

  3. [12]

    Taming data and transformers for audio generation,

    M. Haji-Ali, W. Menapace, A. Siarohin, G. Balakrishnan, S. Tulyakov, and V . Ordonez, “Taming data and transformers for audio generation,” arXiv preprint arXiv:2406.19388, 2024

    2024 arXiv

  4. [13]

    Ezaudio: Enhancing text-to-audio generation with efficient diffusion transformer,

    J. Hai, Y . Xu, H. Zhang, C. Li, H. Wang, M. Elhilali, and D. Yu, “Ezaudio: Enhancing text-to-audio generation with efficient diffusion transformer,”arXiv preprint arXiv:2409.10819, 2024

    2024 arXiv

  5. [14]

    Stable audio open,

    Z. Evans, J. D. Parker, C. Carr, Z. Zukowski, J. Taylor, and J. Pons, “Stable audio open,” inICASSP 2025-2025 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5. IEEE TRANSACTIONS ON MULTIMEDIA 10

    2025

  6. [15]

    Mmaudio: Taming multimodal joint training for high- quality video-to-audio synthesis,

    H. K. Cheng, M. Ishii, A. Hayakawa, T. Shibuya, A. Schwing, and Y . Mitsufuji, “Mmaudio: Taming multimodal joint training for high- quality video-to-audio synthesis,” inProceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 28 901–28 911

    2025

  7. [16]

    Audiox: Diffusion transformer for anything-to-audio generation,

    Z. Tian, Y . Jin, Z. Liu, R. Yuan, X. Tan, Q. Chen, W. Xue, and Y . Guo, “Audiox: Diffusion transformer for anything-to-audio generation,”arXiv preprint arXiv:2503.10522, 2025

    2025 arXiv

  8. [17]

    Neural codec language models are zero-shot text to speech synthesizers,

    C. Wang, S. Chen, Y . Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Liet al., “Neural codec language models are zero-shot text to speech synthesizers,”arXiv preprint arXiv:2301.02111, 2023

    2023 arXiv

  9. [18]

    V oicebox: Text-guided multilingual universal speech generation at scale,

    M. Le, A. Vyas, B. Shi, B. Karrer, L. Sari, R. Moritz, M. Williamson, V . Manohar, Y . Adi, J. Mahadeokaret al., “V oicebox: Text-guided multilingual universal speech generation at scale,”Advances in neural information processing systems, vol. 36, pp. 14 005–14 034, 2023

    2023

  10. [19]

    Naturalspeech 3: Zero-shot speech syn- thesis with factorized codec and diffusion models,

    Z. Ju, Y . Wang, K. Shen, X. Tan, D. Xin, D. Yang, Y . Liu, Y . Leng, K. Song, S. Tanget al., “Naturalspeech 3: Zero-shot speech syn- thesis with factorized codec and diffusion models,”arXiv preprint arXiv:2403.03100, 2024

    2024 arXiv

  11. [20]

    F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching,

    Y . Chen, Z. Niu, Z. Ma, K. Deng, C. Wang, J. JianZhao, K. Yu, and X. Chen, “F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025,...

    2025

  12. [21]

    Cosyvoice 2: Scalable streaming speech synthesis with large language models,

    Z. Du, Y . Wang, Q. Chen, X. Shi, X. Lv, T. Zhao, Z. Gao, Y . Yang, C. Gao, H. Wanget al., “Cosyvoice 2: Scalable streaming speech synthesis with large language models,”arXiv preprint arXiv:2412.10117, 2024

    2024 arXiv

  13. [22]

    Cosyvoice 3: Towards in-the-wild speech generation via scaling-up and post-training,

    Z. Du, C. Gao, Y . Wang, F. Yu, T. Zhao, H. Wang, X. Lv, H. Wang, C. Ni, X. Shiet al., “Cosyvoice 3: Towards in-the-wild speech generation via scaling-up and post-training,”arXiv preprint arXiv:2505.17589, 2025

    2025 arXiv

  14. [23]

    Seed-tts: A family of high-quality versatile speech generation models,

    P. Anastassiou, J. Chen, J. Chen, Y . Chen, Z. Chen, Z. Chen, J. Cong, L. Deng, C. Ding, L. Gaoet al., “Seed-tts: A family of high-quality versatile speech generation models,”arXiv preprint arXiv:2406.02430, 2024

    2024 arXiv

  15. [24]

    Ditar: Diffusion transformer autoregressive modeling for speech generation,

    D. Jia, Z. Chen, J. Chen, C. Du, J. Wu, J. Cong, X. Zhuang, C. Li, Z. Wei, Y . Wanget al., “Ditar: Diffusion transformer autoregressive modeling for speech generation,”arXiv preprint arXiv:2502.03930, 2025

    2025

  16. [25]

    Vibevoice technical report,

    Z. Peng, J. Yu, W. Wang, Y . Chang, Y . Sun, L. Dong, Y . Zhu, W. Xu, H. Bao, Z. Wanget al., “Vibevoice technical report,”arXiv preprint arXiv:2508.19205, 2025

    2025 arXiv

  17. [26]

    Autoregressive image generation without vector quantization,

    T. Li, Y . Tian, H. Li, M. Deng, and K. He, “Autoregressive image generation without vector quantization,”Advances in Neural Information Processing Systems, vol. 37, pp. 56 424–56 445, 2024

    2024

  18. [27]

    Multimodal latent language modeling with next-token diffusion,

    Y . Sun, H. Bao, W. Wang, Z. Peng, L. Dong, S. Huang, J. Wang, and F. Wei, “Multimodal latent language modeling with next-token diffusion,”arXiv preprint arXiv:2412.08635, 2024

    2024 arXiv

  19. [28]

    Diffusion forcing: Next-token prediction meets full- sequence diffusion,

    B. Chen, D. Mart ´ı Mons ´o, Y . Du, M. Simchowitz, R. Tedrake, and V . Sitzmann, “Diffusion forcing: Next-token prediction meets full- sequence diffusion,”Advances in Neural Information Processing Sys- tems, vol. 37, pp. 24 081–24 125, 2024

    2024

  20. [29]

    Ar-diffusion: Asynchronous video generation with auto-regressive diffusion,

    M. Sun, W. Wang, G. Li, J. Liu, J. Sun, W. Feng, S. Lao, S. Zhou, Q. He, and J. Liu, “Ar-diffusion: Asynchronous video generation with auto-regressive diffusion,” inProceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 7364–7373

    2025

  21. [30]

    From slow bidirectional to fast autoregressive video diffusion models,

    T. Yin, Q. Zhang, R. Zhang, W. T. Freeman, F. Durand, E. Shechtman, and X. Huang, “From slow bidirectional to fast autoregressive video diffusion models,” inProceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 22 963–22 974

    2025

  22. [31]

    Autoregressive diffusion transformer for text-to-speech synthesis,

    Z. Liu, S. Wang, S. Inoue, Q. Bai, and H. Li, “Autoregressive diffusion transformer for text-to-speech synthesis,”arXiv preprint arXiv:2406.05551, 2024

    2024 arXiv

  23. [32]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Rayet al., “Training language models to follow instructions with human feedback,”Advances in neural information processing systems, vol. 35, pp. 27 730–27 744, 2022

    2022

  24. [33]

    Direct preference optimization: Your language model is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,”Advances in neural information processing systems, vol. 36, pp. 53 728–53 741, 2023

    2023

  25. [34]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models,

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y . Li, Y . Wuet al., “Deepseekmath: Pushing the limits of mathematical reasoning in open language models,”arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  26. [35]

    Diffusion model alignment using direct preference optimization,

    B. Wallace, M. Dang, R. Rafailov, L. Zhou, A. Lou, S. Purushwalkam, S. Ermon, C. Xiong, S. Joty, and N. Naik, “Diffusion model alignment using direct preference optimization,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 8228–8238

    2024

  27. [36]

    Flow-grpo: Training flow matching models via online rl,

    J. Liu, G. Liu, J. Liang, Y . Li, J. Liu, X. Wang, P. Wan, D. Zhang, and W. Ouyang, “Flow-grpo: Training flow matching models via online rl,” arXiv preprint arXiv:2505.05470, 2025

    2025 arXiv

  28. [37]

    Tango 2: Aligning diffusion-based text-to-audio generations through direct preference optimization,

    N. Majumder, C.-Y . Hung, D. Ghosal, W.-N. Hsu, R. Mihalcea, and S. Poria, “Tango 2: Aligning diffusion-based text-to-audio generations through direct preference optimization,” inProceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 564–572

    2024

  29. [38]

    Tangoflux: Super fast and faithful text to audio generation with flow matching and clap-ranked preference optimization,

    C.-Y . Hung, N. Majumder, Z. Kong, A. Mehrish, A. A. Bagherzadeh, C. Li, R. Valle, B. Catanzaro, and S. Poria, “Tangoflux: Super fast and faithful text to audio generation with flow matching and clap-ranked preference optimization,”arXiv preprint arXiv:2412.21037, 2024

    2024 arXiv

  30. [39]

    Prismaudio: Decomposed chain-of-thoughts and multi- dimensional rewards for video-to-audio generation,

    H. Liu, K. Luo, W. Wang, Q. Chen, P. Sun, R. Huang, X. Li, J. Ye, and W. Xue, “Prismaudio: Decomposed chain-of-thoughts and multi- dimensional rewards for video-to-audio generation,”arXiv preprint arXiv:2511.18833, 2025

    2025

  31. [40]

    Encoder-decoder gemma: Improving the quality-efficiency trade-off via adaptation,

    B. Zhang, F. Moiseev, J. Ainslie, P. Suganthan, M. Ma, S. Bhupatiraju, F. Lebron, O. Firat, A. Joulin, and Z. Dong, “Encoder-decoder gemma: Improving the quality-efficiency trade-off via adaptation,”arXiv preprint arXiv:2504.06225, 2025

    2025 arXiv

  32. [41]

    Scaling rectified flow transformers for high-resolution image synthesis,

    P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. M ¨uller, H. Saini, Y . Levi, D. Lorenz, A. Sauer, F. Boeselet al., “Scaling rectified flow transformers for high-resolution image synthesis,” inForty-first international conference on machine learning, 2024

    2024

  33. [42]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,

    Y . Wu*, K. Chen*, T. Zhang*, Y . Hui*, T. Berg-Kirkpatrick, and S. Dub- nov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” inIEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, 2023

    2023

  34. [43]

    Pushing the frontier of audiovisual perception with large- scale multimodal correspondence learning,

    A. Vyas, H.-J. Chang, C.-F. Yang, P.-Y . Huang, L. Gao, J. Richter, S. Chen, M. Le, P. Doll ´ar, C. Feichtenhofer, A. Lee, and W.-N. Hsu, “Pushing the frontier of audiovisual perception with large- scale multimodal correspondence learning,” 2025. [Online]. Available: https://a...

    2025

  35. [44]

    Robust speech recognition via large-scale weak supervi- sion,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervi- sion,” inInternational conference on machine learning. PMLR, 2023, pp. 28 492–28 518

    2023

  36. [45]

    Meta audiobox aesthetics: Unified automatic quality assessment for speech, music, and sound,

    A. Tjandra, Y .-C. Wu, B. Guo, J. Hoffman, B. Ellis, A. Vyas, B. Shi, S. Chen, M. Le, N. Zacharov, C. Wood, A. Lee, and W.-N. Hsu, “Meta audiobox aesthetics: Unified automatic quality assessment for speech, music, and sound,” 2025. [Online]. Available: https://arxiv.org/abs/2502.05139

    2025 arXiv

  37. [46]

    Perception encoder: The best visual embeddings are not at the output of the network,

    D. Bolya, P.-Y . Huang, P. Sun, J. H. Cho, A. Madotto, C. Wei, T. Ma, J. Zhi, J. Rajasegaran, H. Rasheed, J. Wang, M. Monteiro, H. Xu, S. Dong, N. Ravi, D. Li, P. Doll ´ar, and C. Feichtenhofer, “Perception encoder: The best visual embeddings are not at the output of the netwo...

    2025 arXiv

  38. [47]

    Audiocaps: Generating captions for audios in the wild,

    C. D. Kim, B. Kim, H. Lee, and G. Kim, “Audiocaps: Generating captions for audios in the wild,” inProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2...

    2019

  39. [48]

    Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research,

    X. Mei, C. Meng, H. Liu, Q. Kong, T. Ko, C. Zhao, M. D. Plumbley, Y . Zou, and W. Wang, “Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. ...

    2024

  40. [49]

    Qwen3-omni technical report,

    J. Xu, Z. Guo, H. Hu, Y . Chu, X. Wang, J. He, Y . Wang, X. Shi, T. He, X. Zhu, Y . Lv, Y . Wang, D. Guo, H. Wang, L. Ma, P. Zhang, X. Zhang, H. Hao, Z. Guo, B. Yang, B. Zhang, Z. Ma, X. Wei, S. Bai, K. Chen, X. Liu, P. Wang, M. Yang, D. Liu, X. Ren, B. Zheng, R. Men, F. Zhou,...

    2025 arXiv

  41. [50]

    Gem- ini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,

    G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosenet al., “Gem- ini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,”arXiv preprint arXiv...

    2025 arXiv

  42. [51]

    Text-to-audio generation using instruction tuned llm and latent diffusion model,

    D. Ghosal, N. Majumder, A. Mehrish, and S. Poria, “Text-to-audio generation using instruction tuned llm and latent diffusion model,”arXiv preprint arXiv:2304.13731, 2023

    2023 arXiv

  43. [52]

    Stable audio 3,

    Z. Evans, J. D. Parker, M. Rice, C. Carr, Z. Zukowski, J. Taylor, and J. Pons, “Stable audio 3,”arXiv preprint arXiv:2605.17991, 2026

    2026 arXiv

  44. [53]

    Qwen3-tts technical report,

    H. Hu, X. Zhu, T. He, D. Guo, B. Zhang, X. Wang, Z. Guo, Z. Jiang, H. Hao, Z. Guoet al., “Qwen3-tts technical report,”arXiv preprint arXiv:2601.15621, 2026. IEEE TRANSACTIONS ON MULTIMEDIA 11 2 4 6 8 10 12 14 Seconds 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0Count (1e6) 0 500 1000 1500 2...

    2026 arXiv

  45. [54]

    Audio set: An ontology and human- labeled dataset for audio events,

    J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human- labeled dataset for audio events,” in2017 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2017, p...

    2017

  46. [55]

    Vggsound: A large- scale audio-visual dataset,

    H. Chen, W. Xie, A. Vedaldi, and A. Zisserman, “Vggsound: A large- scale audio-visual dataset,” inICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 721–725. APPENDIXA DETAILS OFVOXCORPUS ANDVOXBENCH A. Real Narr...

    2020

  47. [56]

    Do not imagine or add details not mentioned in the source

    Strict Adherence:Descriptions must be extracted directly from thecaption. Do not imagine or add details not mentioned in the source

  48. [57]

    Remove Technical/Electronic Noise:Prohibit any mention of recording quality, electrical noise, or studio environments (e.g., high-fidelity, reverb, hiss, sample rate)

  49. [58]

    Preserve timing and dynamic changes

    Narrative Sound Effects:Retain and describe all meaningful sound effects, ambient sounds, and musical instruments (e.g., violin, piano). Preserve timing and dynamic changes

  50. [59]

    [SPEECH CONTENT](Transcription)

    Speech Logic: •If Transcription is[NONSPEECH], remove all vocal descriptions. •If speech exists, replace described content with the exact transcription using the format: “[SPEECH CONTENT](Transcription)”. Maintain tone and emotions

  51. [60]

    caption short

    Output Constraints:caption_shortmust be under 200 characters. All placeholder structures must be enclosed in double quotes. JSON Schema: { “caption short”: “Concise English description (max 200 chars)”, “caption complex”: “Complete and detailed English description” } Fig. 6. P...

  52. [61]

    Diversity:Design 5 different scenarios or perspectives based on the same keyword

  53. [62]

    Mandatory Vocals:Each candidate MUST include clear human dialogue or monologue logical to the scene

  54. [63]

    Caption Consistency:Theoverall_captionMUST explicitly mention the specific lines spoken by characters (e.g., ...someone says: “Quote”)

  55. [64]

    Duration:Each scene’s total duration must be between 0-10 seconds

  56. [65]

    Detailed Events:For vocal events,voice_detailsMUST include gender, tone, and emotion

  57. [66]

    candidates

    JSON Format:Return ONLY a JSON object with a root key “candidates” containing the list. JSON Schema & Example: { “candidates”: [ { “id”: 1, “overall caption”: “In a forest at dawn... a woman shouts: ‘Look over there!’, followed by rustling leaves.”, “audio event timeline”: [ {...

  58. [67]

    TABLE VII PER-CATEGORY RESULTS ONMECAT-EN

    Third, the single-chunk variant has the lowest total compute but must synthesize the entire clip before emitting any audio, and the RTF of the default configuration remains stable at longer target durations (0.34 at 20 s, 0.33 at 30 s), reflecting the constant per-chunk cost o...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.