Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

DualDub: Video-to-Soundtrack Generation via Joint Speech and Background Audio Synthesis

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proposes video-to-soundtrack generation, in which a single multimodal language model produces both intelligible speech and background audio from a video, and reports that this joint approach yields markedly more harmonious and…

desk verdict A credible new task and a well-built unified model, but the harmony metric that carries the headline claim needs external validation before the superiority result lands. read the letter →

arxiv 2507.10109 v1 pith:6KEIHP2M submitted 2025-07-14 cs.MM cs.SDeess.AS

Wenjie Tian , Xinfa Zhu , Haohe Liu , Zhixian Zhao , Zihao Chen , Chaofan Ding , Xinhan Di , Junjie Zheng
show 1 more author
Lei Xie
This is my paper
classification cs.MMcs.SDeess.AS
keywords video-to-soundtrackgenerationjointspeechandaudiosynthesiscross-modalalignmentcurriculumlearningmultimodallanguagemodelaudio-speechharmonydubbingvideo-to-audio
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Existing video-to-audio systems can synthesize background effects but treat speech as just another sound, producing unintelligible vocal noise. This paper proposes video-to-soundtrack generation as a distinct task: producing intelligible, speaker-controlled speech and background audio together from a video, a transcript, and a reference voice. It introduces DualDub, a multimodal language model with a cross-modal aligner that predicts audio and speech tokens in a single autoregressive pass, trained with a curriculum schedule that sidesteps the scarcity of fully paired data. On the new DualBench benchmark, DualDub reports substantially higher audio-speech harmony than concatenating separate state-of-the-art video-to-audio and video-to-speech systems. The claim matters because it suggests single-pass joint generation, rather than post-hoc mixing, is the right architecture for automated dubbing and narration.

What carries the argument

The load-bearing components are three. The cross-modal aligner combines two causal cross-attention heads (audio attends to past speech, speech attends to past audio) with two non-causal heads (audio and speech each attend to the full video), so rhythm and timing cohere without leaking future information during generation. The curriculum schedule trains the model first on video-to-audio, then adds text-to-speech, then fine-tunes on a small amount of fully paired video-text-speech-audio data, which is what makes joint training feasible given data scarcity. The generation head is a multimodal language model with two output heads predicting audio and speech tokens simultaneously, decoded through a flow-matching network rather than the codec's native decoder to recover waveform quality. On the evaluation side, the paper introduces CASP, a contrastive audio-speech pretraining model whose cosine similarity between the audio and speech embeddings defines the DualScore harmony metric.

What would settle it

An audit of the test set would settle it: if listeners can hear intelligible speech leaking into the separated 'background' tracks, or if a blind forced-choice test finds the stitched baselines as harmonious as DualDub as often as not, the reported harmony advantage is not established.

Watch

Extended reading notes

Core claim

The central claim is that generating background audio and speech in one autoregressive pass, conditioned on the same video, transcript, and speaker reference, yields soundtracks that are more synchronized and more acoustically harmonious than anything obtainable by running a separate video-to-audio model and a separate video-to-speech model and mixing their outputs. DualDub encodes video with a frozen visual encoder, tokenizes both tracks with a unified codec, lets causal cross-attention pass history between the audio and speech streams while non-causal attention aligns both with the video, and decodes tokens to waveforms through a flow-matching network into a pre-trained audio VAE latent space. The paper reports DualScore 0.59 versus 0.19-0.30 for concatenated baselines and subjective audio-speech correspondence and harmony scores of 3.44 versus 1.44-1.69, while remaining competitive or superior on individual audio and speech quality metrics. DualBench supplies 1,319 filtered test clips and a contrastive audio-speech pretraining metric, DualScore, for measuring harmony between the two generated tracks.

Load-bearing premise

The evaluation's load-bearing premise is that the 1,319 test clips produced by automatic separation and a -40 dB energy cutoff contain clean, correctly paired speech and background audio, and that the learned DualScore matches what humans mean by harmony.

Editorial extensions

If this is right

  • For automated dubbing, the result implies a single model can replace separate audio-effects and speech models, removing the timing and level conflicts that appear when two independently generated tracks are mixed.
  • Because the curriculum learns from large existing video-audio and text-speech corpora plus a small paired set, other multimodal generation tasks facing the same data scarcity could adopt the same staged recipe.
  • DualBench gives later work a common yardstick, so quality, video synchronization, and audio-speech harmony can be measured on the same 1,319 clips.
  • Joint conditioning on video means speech prosody and background audio are shaped by the same visual events, enabling scene-driven narration to be produced end-to-end.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension the paper leaves implicit is that if DualScore tracks perceived harmony, it could be reused as a training reward for joint audio-speech models rather than only a test-time metric, since it is computed from frozen embeddings.
  • The paper's separation-and-filter pipeline suggests the quality ceiling of V2ST evaluation depends on source separation; stronger separation than the current model might enlarge the usable training set and change the measured harmony gap.
  • The stated bottleneck of discrete 40-token-per-second codec representation points to a concrete next experiment: replacing discrete tokens with continuous representations should improve speech naturalness and audio fidelity under the same curriculum.
  • A testable extension is to vary the speaker reference and transcript while fixing the video, checking whether the generated background audio adapts to the changed speech content rather than staying tied only to the visual scene.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a new task, video-to-soundtrack (V2ST) generation, and presents DualDub, a multimodal language model that jointly generates background audio and speech from video, transcript, and a reference speech clip. The model consists of a multimodal encoder, a cross-modal aligner with causal and non-causal attention, dual decoding heads, and a curriculum learning strategy. The authors also introduce DualBench, a benchmark with a curated test set built from V2C-Animation via Mel-RoFormer separation and energy filtering, plus metrics including the newly proposed DualScore for audio-speech harmony. Experiments compare DualDub against concatenated V2A and V2S baselines, reporting higher objective harmony scores and higher subjective audio-speech correspondence and harmony (ASCH), along with V2A/V2S quality comparisons and ablations.

Significance. If the central empirical claim is accepted, the paper makes a useful contribution by defining a practical and underexplored task, proposing a coherent architecture for joint speech and background audio generation, and releasing a benchmark with code. The consistent subjective trend (Table 5, ASCH 3.44 vs 1.44-1.69) is a genuine strength, as is the open benchmark and the detailed curriculum learning design. However, the main quantitative evidence for the 'harmony' advantage is DualScore, a metric introduced and trained in this paper on private data processed by the same separation pipeline used to build the benchmark test set; no human correlation validation is provided. This makes the objective pillar of the central claim currently unverified, although the issue is addressable with additional evaluation.

major comments (3)
  1. [§4.3, §5.1, Table 5] The load-bearing objective evidence for the harmony advantage is DualScore, but its validity as a measure of perceived audio-speech harmony is not established. The CASP model is trained on 1,500 hours of private audio-speech pairs created with the same Mel-RoFormer separation and -40 dB filtering pipeline used to build the V2ST test set (§4.2, §5.1), and its only reported validation is retrieval accuracy on that same V2ST test set. No per-clip correlation between DualScore and human ASCH ratings is reported, so the large DualScore gap (0.59 vs 0.19-0.30) could reflect separator-specific artifacts, speech bleed, or other correlated features rather than harmony that humans perceive. I ask the authors to validate DualScore against human harmony judgments on a held-out set (including per-item correlation), to release the CASP training data or otherwise remove the training/evaluation overlap, and to re-report the system comparison using an independently separated or manually verified test set.
  2. [Table 3, §8] The ground-truth word error rate on the DualBench test set is 35.62%, while DualDub's generated speech reaches 12.74%; this indicates that the separated speech references in the test set are heavily corrupted or mislabeled, contradicting the assumption of clean, correctly paired ground-truth speech. Section 8 itself concedes that Mel-RoFormer 'sometimes separates empty audio or speech segments.' This does not necessarily invalidate the comparison across systems, but it undermines the reliability of ground-truth-anchored metrics and makes the test set unsuitable as a benchmark reference without further quality control. The authors should quantify the proportion of corrupted clips, provide a manual verification subset, and report the main results on that subset.
  3. [Tables 2-5] Objective metrics are reported as point estimates without error bars, confidence intervals, or significance tests. For instance, the DualScore differences in Table 5 (0.59 vs 0.19-0.30) and the FD/FAD/KLD differences in Tables 2 and 4 are presented without any measure of variance, so the reader cannot assess whether the differences are within noise. The subjective ASCH comparison is based on only 16 clips, even though 1,319 test clips are available. I request bootstrap confidence intervals or significance tests on all objective metrics and a subjective evaluation on a larger, diverse sample, or at least a justification for the small subjective sample size.
minor comments (6)
  1. [Figure 2] The label 'Casual Non-Casual' should be 'Causal Non-Causal'.
  2. [§4.3] The phrase 'we access audio and speech quality' should be 'we assess audio and speech quality'.
  3. [References] The reference list appears twice in the manuscript, duplicating references [1] through [58] verbatim; this should be corrected.
  4. [Table 4] The column header 'SSIM↑' under 'Audio-Video alignment' appears to be a typo for 'SIM↑' (speaker similarity), since the value 0.84 matches the SIM column in Table 3 and SSIM is not defined in the text.
  5. [Eq. (6)] Equation (6) contains a stray period and line break after the differential equation; the equation and surrounding punctuation should be cleaned up.
  6. [§5.1] The sentence 'Notably, following the data process of DualBench, we separate all audio-speech streams to avoid speech leaks into the background audio' is grammatically awkward and should be revised for clarity.

Circularity Check

0 steps flagged · score 2.0 of 10

No derivation-level circularity: DualDub's generation is not fitted to DualScore and the harmony ordering is independently supported by the human ASCH test; the in-paper DualScore sharing the Mel-RoFormer separation pipeline with training/test data is a metric-validity risk, not a circular reduction.

full rationale

The paper's central claim is an empirical model comparison rather than a derived prediction. DualDub is trained with cross-entropy token prediction (Eq. 5) and flow-matching loss (Eq. 7); it is never optimized against DualScore, and the generated samples are not in CASP's training set. The Table 5 DualScore margin (0.59 vs. 0.19-0.30) is therefore an out-of-sample evaluation, not a fitted value. The objective ordering is also reproduced by the independent 20-listener ASCH test (3.44 vs. 1.44-1.69), so the central claim does not reduce to the authors' own metric. The only self-referential element is the proposed DualScore: CASP is trained on 1,500 h of private audio-speech pairs that, like the V2ST test set, are produced by the same Mel-RoFormer separation and -40 dB energy filtering (Sections 4.2, 5.1, Table 1). This shared pipeline is a genuine metric-validity limitation: if separation leaves correlated artifacts, DualScore could reward those artifacts rather than perceived harmony, and Section 8 concedes that Mel-RoFormer 'sometimes separates empty audio or speech segments.' But this is a benchmark-quality/correctness risk, not a circular derivation: no equation equates DualScore to a fitted parameter of DualDub, and no proposed result is true by definition. Self-citations ([52], [57], [58]) appear only as context or as alternatives in the Limitations section and are not load-bearing for the main result. Accordingly, no circular step is exhibited, and the paper is best assessed as largely self-contained, with the DualScore pipeline flagged as an external-validity concern.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The central empirical claims rest less on mathematical axioms than on measurement assumptions. The main dependencies are the validity of the data separation and filtering pipeline, the accuracy of transcripts, and the new DualScore metric. The modeling itself uses standard LM and flow-matching assumptions.

free parameters (4)
  • Energy filtering threshold = -40 dB
    Clips with either separated speech or audio energy below -40 dB are discarded from both the DualBench test set and the training data. The threshold is described as empirically determined in Section 4.2, and it directly shapes which videos the benchmark and the model see.
  • Curriculum training schedule = 3 stages, 40 epochs each, lr ranges 2e-6 to 2e-4, 2e-7 to 2e-4, 2e-7 to 2e-5
    The three-stage curriculum and per-stage learning rate ranges are hand-designed choices from Section 5.2. No sensitivity analysis is provided, so the reported results depend on this schedule.
  • Speaker reference duration = 3 seconds
    Section 3.2 randomly intercepts 3 seconds of target speech for the speaker embedding. This choice affects speaker similarity and prosody conditioning, and the paper gives no analysis of its sensitivity.
  • CASP segment length = 5 seconds
    Section 4.3 samples 5-second segments from audio-speech pairs when training the CASP model that produces DualScore. The segment length affects the alignment learned by the harmony metric.
assumptions (5)
  • standard math Autoregressive factorization of the joint speech-audio token distribution (Eq. 5)
    The paper defines p(A_speech_t, A_audio_t | ...) as a product of per-timestep conditional probabilities over past tokens. This is the standard next-token assumption for language models; the two heads predict audio and speech tokens at the same time step, so the factorization is plausible but not formally proven.
  • domain assumption Mel-RoFormer accurately separates speech from background audio
    Used to construct both the DualBench test set and the filtered training data (Section 4.2 and Table 1). If separation is imperfect and speech leaks into the background track, the model may be rewarded for generating speech-like sounds in the audio stream, inflating harmony scores.
  • domain assumption V2C-Animation transcripts match the spoken content
    Table 3 computes WER against these transcripts for all models. The ground-truth WER is 35.62%, far higher than any model, which indicates the transcripts or reference speech are noisy; this complicates the speech-quality comparison.
  • domain assumption CLIP ViT-B/32 features are sufficient for video-to-audio synchronization
    Section 3.2 uses CLIP visual features on frames sampled one in three. This provides semantic content but limited temporal detail, so the synchronization claim depends on this representation being adequate.
  • ad hoc to paper DualScore (CASP embedding cosine similarity) measures audio-speech harmony
    Section 4.3 introduces CASP and DualScore. The metric is validated only by retrieval accuracy and a 16-sample listening test, so the central harmony comparison in Table 5 depends on this new, unestablished metric being a faithful proxy for human perception.
invented entities (1)
  • CASP audio-speech embedding space and DualScore metric independent evidence
    purpose: To quantify the harmony between generated speech and background audio in the DualBench benchmark; it is the primary objective evidence for the central harmony claim.
    CASP is trained on 1,500 hours of private audio-speech pairs and evaluated by retrieval on 1,319 real pairs (Top-1 70%, Top-3 90%, Top-5 95%), which is a falsifiable external handle. The correlation with human harmony judgments is only checked on 16 samples, so the evidence is present but thin.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DualDub: Video-to-Soundtrack Generation via Joint Speech and Background Audio Synthesis." pith.science (2026). https://pith.science/paper/6KEIHP2M

@misc{pith2026250710109,
  author       = {Pith},
  title        = {Pith review of: DualDub: Video-to-Soundtrack Generation via Joint Speech and Background Audio Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6KEIHP2M}},
  note         = {Machine review of arXiv:2507.10109}
}
read the original abstract

While recent video-to-audio (V2A) models can generate realistic background audio from visual input, they largely overlook speech, an essential part of many video soundtracks. This paper proposes a new task, video-to-soundtrack (V2ST) generation, which aims to jointly produce synchronized background audio and speech within a unified framework. To tackle V2ST, we introduce DualDub, a unified framework built on a multimodal language model that integrates a multimodal encoder, a cross-modal aligner, and dual decoding heads for simultaneous background audio and speech generation. Specifically, our proposed cross-modal aligner employs causal and non-causal attention mechanisms to improve synchronization and acoustic harmony. Besides, to handle data scarcity, we design a curriculum learning strategy that progressively builds the multimodal capability. Finally, we introduce DualBench, the first benchmark for V2ST evaluation with a carefully curated test set and comprehensive metrics. Experimental results demonstrate that DualDub achieves state-of-the-art performance, generating high-quality and well-synchronized soundtracks with both speech and background audio.

Figures

Figures reproduced from arXiv: 2507.10109 by the authors.

Figure 1
Figure 1. Overview of the proposed DualDub. DualDub fo [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Architecture of the proposed DualDub and Contrastive Audio-Speech Pretraining (CASP). (a) illustrates the overall [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Heatmap of top-1 retrieval results from contrastive [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AudioGen-Omni: A Unified Multimodal Diffusion Transformer for Video-Synchronized Audio, Speech, and Song Generation

    cs.SD 2025-08 conditional novelty 5.0 of 10

    A single multimodal diffusion transformer generates video-synchronized general audio, speech, and song from flexible combinations of video, text, and lyrics inputs.

Reference graph

Works this paper leans on

62 extracted references · 37 canonical work pages · cited by 1 Pith paper

  1. [1]

    Kevin Cai, Chonghua Liu, and David M Chan. 2024. Anim-400K: A Large-Scale Dataset for Automated End to End Dubbing of Video. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 11796–11800

    work page 2024

  2. [2]

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. 2020. Vg- gsound: A large-scale audio-visual dataset. InICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 721–725

    work page 2020

  3. [3]

    Qi Chen, Mingkui Tan, Yuankai Qi, Jiaqiu Zhou, Yuanqing Li, and Qi Wu. 2022. V2C: Visual voice cloning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 21242–21251

    work page 2022

  4. [4]

    Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei. 2022. Beats: Audio pre-training with acoustic tokenizers. arXiv preprint arXiv:2212.09058 (2022)

    arXiv 2022

  5. [5]

    Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, and Xie Chen. 2024. F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching. CoRR abs/2410.06885 (2024)

    arXiv 2024

  6. [6]

    Zhengyang Chen, Sanyuan Chen, Yu Wu, Yao Qian, Chengyi Wang, Shujie Liu, Yanmin Qian, and Michael Zeng. 2022. Large-scale self-supervised speech representation learning for automatic speaker verification. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 6147–6151

    work page 2022

  7. [7]

    Schwing, and Yuki Mitsufuji

    Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander G. Schwing, and Yuki Mitsufuji. 2024. Taming Multimodal Joint Training for High- Quality Video-to-Audio Synthesis. CoRR abs/2412.15322 (2024)

    arXiv 2024

  8. [8]

    Jeongsoo Choi, Joanna Hong, and Yong Man Ro. 2023. Diffv2s: Diffusion-based video-to-speech synthesis with vision-guided speaker embedding. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 7812–7821

    work page 2023

Show all 62 references
  1. [9]

    Jeongsoo Choi, Ji-Hoon Kim, Jinyu Li, Joon Son Chung, and Shujie Liu. 2024. V2SFlow: Video-to-Speech Generation with Speech Decomposition and Rectified Flow. CoRR abs/2411.19486 (2024)

    2024 arXiv

  2. [10]

    Gaoxiang Cong, Liang Li, Yuankai Qi, Zheng-Jun Zha, Qi Wu, Wenyu Wang, Bin Jiang, Ming-Hsuan Yang, and Qingming Huang. 2023. Learning to dub movies via hierarchical prosody models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 14687–14697

    2023

  3. [11]

    Gaoxiang Cong, Yuankai Qi, Liang Li, Amin Beheshti, Zhedong Zhang, Anton van den Hengel, Ming-Hsuan Yang, Chenggang Yan, and Qingming Huang. 2024. StyleDubber: Towards Multi-Scale Style Learning for Movie Dubbing. InFindings of the Association for Computational Linguistics, AC...

    2024

  4. [12]

    Russell, and Andrew Owens

    Yuexi Du, Ziyang Chen, Justin Salamon, Bryan C. Russell, and Andrew Owens

  5. [13]

    Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, Fan Yu, Huadai Liu, Zhengyan Sheng, Yue Gu, Chong Deng, Wen Wang, Shiliang Zhang, Zhijie Yan, and Jingren Zhou. 2024. CosyVoice 2: Scalable Streaming Speech Synt...

    2024 arXiv

  6. [14]

    Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang

  7. [15]

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. [n. d.]. Scaling rectified flow transformers for high-resolution image synthesis. In Forty- first international conference o...

  8. [16]

    In IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP 2023, Rhodes Island, Greece, June 4-10, 2023

    CLAP Learning Audio Concepts from Natural Language Supervision. In IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP 2023, Rhodes Island, Greece, June 4-10, 2023 . IEEE, 1–5

    2023

  9. [17]

    Ruibo Fu, Shuchen Shi, Hongming Guo, Tao Wang, Chunyu Qiang, Zhengqi Wen, Jianhua Tao, Xin Qi, Yi Lu, Xiaopeng Wang, Zhiyong Wang, Yukun Liu, Xuefei Liu, Shuai Zhang, and Guanjun Li. 2024. MINT: a Multi-modal Image and Narrative Text Dubbing Dataset for Foley Audio Content Pla...

    2024 arXiv

  10. [18]

    Hawley, and Jordi Pons

    Zach Evans, CJ Carr, Josiah Taylor, Scott H. Hawley, and Jordi Pons. 2024. Fast Timing-Conditioned Latent Audio Diffusion. In Forty-first International Confer- ence on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenRe- view.net

    2024

  11. [19]

    Yuan Gong, Andrew Rouditchenko, Alexander H Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James Glass. 2022. Contrastive audio-visual masked autoencoder. arXiv preprint arXiv:2210.07839 (2022)

    2022 arXiv

  12. [20]

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. 2023. ImageBind One Embedding Space to Bind Them All. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 1...

    2023

  13. [21]

    Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. 2017. CNN architectures for large-scale audio classification. In2017 ieee international conference on acoustics, sp...

    2017

  14. [22]

    Michael Hassid, Michelle Tadmor Ramanovich, Brendan Shillingford, Miaosen Wang, Ye Jia, and Tal Remez. 2022. More than Words: In-the-Wild Visually- Driven Prosody for Text-to-Speech. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA,...

    2022

  15. [23]

    Vladimir Iashin and Esa Rahtu. 2021. Taming Visually Guided Sound Generation. In British Machine Vision Conference

    2021

  16. [24]

    Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhen- hui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, Yi Ren, Yuexian Zou, Zhou Zhao, and Shinji Watanabe. 2024. AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head. In T...

    2024

  17. [25]

    Ziyue Jiang, Jinglin Liu, Yi Ren, Jinzheng He, Chen Zhang, Zhenhui Ye, Pengfei Wei, Chunfeng Wang, Xiang Yin, Zejun Ma, and Zhou Zhao. 2023. Mega-TTS 2: Zero-Shot Text-to-Speech with Arbitrary Length Speech Prompts. CoRR abs/2307.07218 (2023)

    2023 arXiv

  18. [26]

    Shengpeng Ji, Ziyue Jiang, Xize Cheng, Yifu Chen, Minghui Fang, Jialong Zuo, Qian Yang, Ruiqi Li, Ziang Zhang, Xiaoda Yang, Rongjie Huang, Yidi Jiang, Qian Chen, Siqi Zheng, Wen Wang, and Zhou Zhao. 2024. WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio L...

    2024 arXiv

  19. [27]

    Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. 2018. Fr\’echet audio distance: A metric for evaluating music enhancement algorithms. arXiv preprint arXiv:1812.08466 (2018)

    2018 arXiv

  20. [28]

    Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Eric Liu, Yichong Leng, Kaitao Song, Siliang Tang, Zhizheng Wu, Tao Qin, Xiangyang Li, Wei Ye, Shikun Zhang, Jiang Bian, Lei He, Jinyu Li, and Sheng Zhao. 2024. NaturalSpeech 3: Zero-Shot Speech Synthesis w...

    2024

  21. [29]

    Salik, Rajiv Ratn Shah, Yifang Yin, and Roger Zimmermann

    Yaman Kumar, Rohit Jain, Khwaja Mohd. Salik, Rajiv Ratn Shah, Yifang Yin, and Roger Zimmermann. 2019. Lipper: Synthesizing Thy Speech Using Multi-View Lipreading. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Application...

    2019

  22. [30]

    Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley. 2020. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing 28 (2020), 2880–2894

    2020

  23. [31]

    Shun Lei, Yixuan Zhou, Boshi Tang, Max WY Lam, Hangyu Liu, Jingcheng Wu, Shiyin Kang, Zhiyong Wu, Helen Meng, et al. 2024. Songcreator: Lyrics-based universal song generation. Advances in Neural Information Processing Systems 37 (2024), 80107–80140

    2024

  24. [32]

    Jiyoung Lee, Joon Son Chung, and Soo-Whan Chung. 2023. Imaginary Voice: Face-Styled Diffusion Model for Text-to-Speech. In IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP 2023, Rhodes Island, Greece, June 4-10, 2023. IEEE, 1–5

    2023

  25. [33]

    Mandic, Wenwu Wang, and Mark D

    Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo P. Mandic, Wenwu Wang, and Mark D. Plumbley. 2023. AudioLDM: Text-to-Audio Gen- eration with Latent Diffusion Models. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA...

    2023

  26. [34]

    Raghavan, Gavin Mischler, and Nima Mesgarani

    Yinghao Aaron Li, Cong Han, Vinay S. Raghavan, Gavin Mischler, and Nima Mesgarani. 2023. StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models. In Advances in Neural Information Processing Systems 36: ...

    2023

  27. [35]

    Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu, Yanqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu, et al. 2024. Autoregressive speech synthesis without vector quantization. arXiv preprint arXiv:2407.08551 (2024)

    2024 arXiv

  28. [36]

    Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. 2023. Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models. Advances in Neural Information Processing Systems 36 (2023), 48855–48876

    2023

  29. [37]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning . PMLR, 28492–28518

    2023

  30. [38]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Mod- els From Natural Language Supervision. In Proceedin...

    2021

  31. [39]

    Roy Sheffer and Yossi Adi. 2023. I hear your true colors: Image guided audio gen- eration. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 1–5

    2023

  32. [40]

    Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. 2022. Utmos: Utokyo-sarulab system for voicemos challenge 2022. arXiv preprint arXiv:2204.02152 (2022)

    2022 arXiv

  33. [41]

    Zineng Tang, Ziyi Yang, Mahmoud Khademi, Yang Liu, Chenguang Zhu, and Mohit Bansal. 2024. CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, W A, USA, June 16-22, 2024. ...

    2024

  34. [42]

    Kai Shen, Zeqian Ju, Xu Tan, Eric Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, and Jiang Bian. 2024. NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers. In The Twelfth International Conference on Learning Representations, ICLR...

    2024

  35. [43]

    Hui Wang, Shujie Liu, Lingwei Meng, Jinyu Li, Yifan Yang, Shiwan Zhao, Haiyang Sun, Yanqing Liu, Haoqin Sun, Jiaming Zhou, Yan Lu, and Yong Qin. 2025. FELLE: Autoregressive Speech Synthesis with Token-Wise Coarse-to-Fine Flow Match- ing. CoRR abs/2502.11128 (2025)

    2025 arXiv

  36. [44]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lam- ple. 2023. LLaMA: Open and Efficient Foundation ...

    2023 arXiv

  37. [45]

    Ju-Chiang Wang, Wei-Tsung Lu, and Minz Won. 2023. Mel-Band RoFormer for Music Source Separation. arXiv preprint arXiv:2310.01809 (2023)

    2023 arXiv

  38. [46]

    Heng Wang, Jianbo Ma, Santiago Pascual, Richard Cartwright, and Weidong Cai

  39. [47]

    Zhifeng Xie, Shengye Yu, Qile He, and Mengtian Li. 2024. Sonicvisionlm: Playing sound with vision language models. In Proceedings of the IEEE/CVF Conference Wenjie Tian, Xinfa Zhu, Haohe Liu, Zhixian Zhao, Zihao Chen, Chaofan Ding, Xinhan Di, Junjie Zheng, and Lei Xie on Compu...

    2024

  40. [48]

    Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. 2024. Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion La- tent Aligners. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, W A, USA, June 16-22, 2024....

    2024

  41. [49]

    Yongqi Wang, Wenxiang Guo, Rongjie Huang, Jiawei Huang, Zehan Wang, Fuming You, Ruiqi Li, and Zhou Zhao. 2024. Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matching. In Advances in Neural Infor- mation Processing Systems 38: Annual Conference on Neu...

    2024

  42. [50]

    Dongchao Yang, Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Xixin Wu, Zhou Zhao, Shinji Watan- abe, and Helen Meng. 2023. UniAudio: An Audio Foundation Model Toward Universal Audio Generation. CoRR abs/2310.00704 (2023)

    2023 arXiv

  43. [51]

    Guy Yariv, Itai Gat, Sagie Benaim, Lior Wolf, Idan Schwartz, and Yossi Adi

  44. [52]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Cheng- peng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jin...

    2024 arXiv

  45. [53]

    Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. 2019. Libritts: A corpus derived from librispeech for text-to- speech. arXiv preprint arXiv:1904.02882 (2019)

    2019 arXiv

  46. [54]

    Yiming Zhang, Yicheng Gu, Yanhong Zeng, Zhening Xing, Yuancheng Wang, Zhizheng Wu, and Kai Chen. 2024. FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds. CoRR abs/2407.01494 (2024)

    2024 arXiv

  47. [55]

    arXiv:2309.16429 [cs.LG]

    Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation. arXiv:2309.16429 [cs.LG]

    arXiv

  48. [56]

    Zhen Ye, Xinfa Zhu, Chi-Min Chan, Xinsheng Wang, Xu Tan, Jiahe Lei, Yi Peng, Haohe Liu, Yizhu Jin, Zheqi DAI, et al . 2025. Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis. arXiv preprint arXiv:2502.04128 (2025)

    2025 arXiv

  49. [57]

    Xinfa Zhu, Wenjie Tian, Xinsheng Wang, Lei He, Xi Wang, Sheng Zhao, and Lei Xie. 2025. CosyAudio: Improving Audio Generation with Confidence Scores and Synthetic Captions. CoRR abs/2501.16761 (2025)

    2025 arXiv

  50. [58]

    Xinfa Zhu, Wenjie Tian, and Lei Xie. 2024. Autoregressive Speech Synthesis with Next-Distribution Prediction. arXiv preprint arXiv:2412.16846 (2024)

    2024

  51. [59]

    Zhedong Zhang, Liang Li, Gaoxiang Cong, Haibing Yin, Yuhan Gao, Chenggang Yan, Anton van den Hengel, and Yuankai Qi. 2024. From Speaker to Dubber: Movie Dubbing with Prosody and Duration Consistency Learning. InProceedings of the 32nd ACM International Conference on Multimedia...

    2024

  52. [60]

    Yuan Zhao, Zhenqi Jia, Rui Liu, De Hu, Feilong Bao, and Guanglai Gao. 2024. Mcdubber: Multimodal context-aware expressive video dubbing. In National Conference on Man-Machine Speech Communication . Springer, 168–182

    2024

  53. [2023]

    In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023

    Conditional Generation of Audio from Video via Foley Analogies. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023. IEEE, 2426–2436

    2023

  54. [2024]

    V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models. In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Sympo...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.