Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

EmoDubber: Towards High Quality and Emotion Controllable Movie Dubbing

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Movie dubbing gains user-controlled emotion and intensity without losing lip sync or pronunciation.

desk verdict Solid lip-sync and pronunciation work; the emotion-controllability claim is undercut by a type-incorrect guidance equation and a self-confirming intensity metric. read the letter →

arxiv 2412.08988 v3 pith:QAHL7CQG submitted 2024-12-12 cs.SD cs.MMeess.AS

Gaoxiang Cong , Jiadong Pan , Liang Li , Yuankai Qi , Yuxin Peng , Anton van den Hengel , Jian Yang , Qingming Huang This is my paper · ORCID
classification cs.SDcs.MMeess.AS
keywords moviedubbingvisualvoicecloningemotion-controllablespeechsynthesisflowmatchingclassifierguidancelip-syncalignmentpronunciationenhancementcontrastivelearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EmoDubber makes the case that a dubbing system can satisfy three requirements at once: lip movements match the spoken audio, words remain intelligible, and the emotional colour of the delivery is chosen by the user rather than fixed by the source recording. The paper's central claim is that a user can supply an emotion label and two intensity weights, and the system will steer a flow-matching speech generator toward that emotion (and away from the others) while a duration-level contrastive alignment keeps the prosody locked to the video and a phoneme-enhancement path keeps pronunciation clear. A sympathetic reader would care because this moves dubbing from re-recording or limited transfer toward direct, director-style control: pick the emotion, set how strong it is, and keep the performance in sync.

What carries the argument

The carrying mechanism is the positive-and-negative guidance (PNGM) applied inside the flow-matching prediction network. Given an acoustics prior $\mu$ and the user instruction $E=\{c,\alpha,\beta\}$, the predicted velocity field is altered by adding $\gamma(\alpha\nabla\log p_{\psi}(c\mid x) - \beta\nabla\log p_{\psi}(\sum_{j\neq i} l_j c_j \mid x))$, so that the noise-to-mel-spectrogram trajectory is pushed toward the chosen emotion and pulled away from all others, with $\alpha$ and $\beta$ acting as the intensity controls. The rest of the architecture supplies the conditions for that guidance: duration-level contrastive learning produces a monotonic lip-to-phoneme alignment, the pronunciation-enhancing path expands phoneme sequences and fuses them with lip context, and the speaker-identity module converts the fused sequence into the acoustics prior $\mu$.

What would settle it

A listener study in which humans rate the emotional intensity of outputs generated with increasing positive guidance (and fixed negative guidance), using an emotion recognizer trained independently of the steering classifier, would settle whether the control is perceptual rather than self-confirming; the claim fails if human ratings do not track the guidance weights or if the independent recognizer shows no consistent emotion shift.

Watch

Extended reading notes

Core claim

EmoDubber's central claim is that emotion-controllable dubbing is achieved by combining four modules: Lip-related Prosody Aligning uses duration-level contrastive learning to enforce a monotonic correspondence between phoneme prosody and lip motion; Pronunciation Enhancing expands phoneme sequences to video length with monotonic alignment search and fuses them with the lip context in an efficient conformer; Speaker Identity Adapting up-samples the fused features and injects the reference speaker's style to form an acoustics prior; and Flow-based User Emotion Controlling modifies the flow-matching vector field by positive and negative classifier guidance, adding a scaled gradient toward the requested emotion and subtracting a scaled gradient toward the mixture of all other emotions. On the Chem and GRID benchmarks, the paper reports near-ground-truth lip-sync metrics (for example, LSE-C 8.11 against ground truth 8.12 on Chem setting 1), word error rates comparable to or better than prior dubbing systems, and an intensity score that rises with the positive guidance weight and falls with the negative weight. The paper also reports that adding emotion control leaves lip sync, pronunciation, and speaker similarity essentially unchanged across seven emotion categories.

Load-bearing premise

The load-bearing premise is that the emotion classifier used to steer generation is a trustworthy measure of real human-perceived emotion, meaning its gradients point toward speech humans hear as more emotional and its softmax logits track felt intensity, so if that classifier is miscalibrated or domain-shifted the emotion-control claim is not demonstrated.

Editorial extensions

If this is right

  • A director or user could author a dub's emotional performance directly, choosing for example 'angry' or 'sad' and setting how strong it should be, without asking an actor to re-record.
  • Lip sync and intelligibility can be improved together: the ablation study shows that removing the lip-prosody aligning module hurts sync most and removing the pronunciation-enhancing module hurts word error rate most, so the two goals are not a simple trade-off.
  • Emotion instructions transfer to videos whose original speech carries no emotion label: the paper's zero-shot emotion conversion on Chem shows distinct t-SNE clusters for happy, sad, surprised, and angry speech.
  • Dual positive-and-negative guidance gives a wider and finer intensity range than single-direction guidance, because users can amplify the target emotion and suppress competing ones separately.
  • Applying emotion control leaves the other dubbing qualities intact: across seven emotion categories, lip-sync confidence and distance, word error rate, and speaker similarity stay close to the emotion-free output, with sadness the partial exception.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An inference beyond the paper: because the same emotion classifier both steers generation and scores the output's intensity, the reported monotonic intensity curves may partly reflect the classifier's own preferences; an independent listener study or a different emotion recognizer would tell whether the perceived emotion actually scales with $\alpha$ and $\beta$.
  • A testable extension the paper leaves implicit: positive-and-negative guidance should transfer to any flow-matching or diffusion generator with a differentiable attribute classifier, so the same dual-gradient recipe could give fine-grained control of speaking style, accent, or even non-speech acoustic attributes without retraining.
  • A practical limitation worth checking: the alignment path depends on Montreal Forced Aligner and phoneme-level supervision, so extending the method to languages or casual video without reliable forced aligners would need a self-supervised duration-alignment substitute.
  • The intensity knob may enable emotion interpolation, generating mixed emotions by tuning $\alpha$ for one label and $\beta$ for another, which the paper gestures at with its mixture-of-logits negative term but does not systematically evaluate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes EmoDubber, an end-to-end movie dubbing system that takes text, a silent video, a reference audio, and a user-specified emotion instruction E = {c, alpha, beta} as inputs. The architecture combines a Lip-related Prosody Aligning (LPA) module with duration-level contrastive learning, a Pronunciation Enhancing (PE) conformer that fuses phoneme and lip-motion features, a Speaker Identity Adapting (SIA) module that injects speaker style into acoustics priors, and a Flow-based User Emotion Controlling (FUEC) decoder that applies positive and negative classifier guidance (PNGM) for emotion intensity control. Experiments on Chem, GRID, and a zero-shot speaker setting compare against dubbing baselines using SyncNet lip-sync metrics, Whisper word error rate, speaker cosine similarity, MCD, and subjective MOS, together with an emotion intensity analysis.

Significance. If the results hold, the paper would make a useful contribution to controllable movie dubbing: the lip-sync and pronunciation claims are supported by external metrics (SyncNet and Whisper), and the ablation in Table 3 cleanly attributes LSE gains to LPA and WER gains to PE. The code and trained models are promised, which would increase reproducibility. However, the headline emotion-controllability claim is not currently established: the negative-guidance term in Eq. (11) is not computable as written, and the Intensity Score in Section 4.5 is a self-confirming metric that uses the same emotion classifier that drives the guidance. The paper would be substantially strengthened by a corrected formulation of Eq. (11) and by independent human or held-out emotion perception tests.

major comments (5)
  1. [§3.5, Eq. (11)] The negative-guidance term is not well defined. In Eq. (11), the second gradient is taken over log p_psi(sum_{j != i} l_j c_j | phi_t(x)), but p_psi is described as a classifier that takes a mel-spectrogram and returns a distribution over emotion classes; the first argument is therefore a convex combination of one-hot emotion vectors, not an acoustic feature. As written, p_psi(...) is type-incorrect, so the expression cannot be computed or reproduced from the paper alone. The authors should define the suppressed-emotion term explicitly, for example using log(1 - p_psi(c_i | x)), log sum_{j != i} p_psi(c_j | x), or a properly defined soft-label likelihood, and state how gradients are taken with respect to x.
  2. [§4.5, Figure 3] The Intensity Score is the average softmax logit of the target emotion computed by the same emotion classifier psi that supplies the PNGM gradients in Eq. (11). Because the guidance mechanism explicitly increases log p_psi(c_i | phi_t(x)), Figure 3 mostly demonstrates that the optimizer follows the classifier gradient; it does not independently establish that listeners perceive stronger or more controllable emotion. Please report a human emotion-rating study (e.g., perceived emotion type and intensity) or evaluate with a held-out speech emotion recognition model that was not used during generation, and provide confidence intervals for each alpha/beta condition.
  3. [§4.4, Table 4] The GRID WER result needs discussion: generated speech achieves WER 18.53% while ground-truth audio achieves 22.41%. If an ASR model finds synthesized speech more intelligible than the original recordings, this may reflect ASR bias or over-articulation rather than pronunciation quality. The authors should report WER with confidence intervals, verify with a second ASR system, and explain why synthesized speech is expected to beat the ground-truth WER before using this result to support the generalizable pronunciation claim.
  4. [§4.4 and §4.5, Tables 1, 3, 4, 5 and Figure 3] None of the quantitative tables report variance, confidence intervals, or significance tests, although several comparisons are decided by small margins (e.g., Table 1, Setting 1.0, MCD 5.87 for Ours versus 5.41 for StyleDubber). Because the paper makes comparative claims of state-of-the-art performance, the authors should report standard deviations across multiple generation runs or test subsamples and, where feasible, statistical significance. This is especially important for the emotion intensity curves in Figure 3, which currently have no error bars.
  5. [§4.5, emotional controlling evaluation] There is no quantitative comparison with an existing intensity-controllable emotional TTS system. The only emotion baseline, GenerSpeech, is evaluated through a t-SNE visualization, which does not measure intensity controllability or perceived emotion quality. To support the claim of flexible intensity control, the authors should compare with an intensity-controllable emotional TTS method such as EmoDiff under matched emotion classes and report both objective and perceptual emotion metrics.
minor comments (5)
  1. [§3.2, Eq. (5)] The denominator in Lcl appears to be missing the temperature tau and the negative-sample structure of a standard contrastive loss; as written, the expression is not a recognizable InfoNCE loss. Please clarify or correct the equation.
  2. [§3.5, after Eq. (11)] The phrase 'We can set emotion classes as {c0, ..., cN-1} one-hot vector' should be reworded to 'each c_i is a one-hot vector', because the current wording is ambiguous about whether c_i is a class index or a vector.
  3. [§4.4, Chem results] The sentence reporting 'absolute improvements of 4.24% and 4.0%' should specify whether these are relative or absolute improvements in LSE-C and LSE-D, and against which baseline.
  4. [Figure 3 caption] The caption does not state the evaluation protocol for the Intensity Score, such as the number of test samples and whether the score is averaged over entire utterances or frames. Please add this information.
  5. [§4.1, Implementation Details] The STFT configuration lists window length 640, frame size 1,024, and hop length 160; this is unusual because the window length is smaller than the frame size. Please clarify whether 'frame size' refers to the FFT size and correct the terminology.

Circularity Check

1 steps flagged · score 7.0 of 10

Emotion intensity control is evaluated with the same emotion classifier that supplies the PNGM control gradient, making the reported intensity curves self-confirming; lip-sync and pronunciation results are independently grounded.

  1. self definitional [Section 3.5 Eq. (11) (PNGM) and Section 4.3 'Emotional Evaluation' / Fig. 3]
    "Emotional Evaluation. We use the Intensity Score, the average softmax logit of the target emotion, which ranges from 0 to 1, to measure the emotional intensity of generated audio. ... we can use positive guidance to guide M toward the direction of ci and use negative guidance to suppress others, which can be formulated as: ˜vt,i = vt(ϕt(x)|µ, θ) + γ(α∇logpψ(ci|ϕt(x)) − β∇logpψ(...)) (11)."

    The PNGM update in Eq. (11) performs gradient ascent along α∇log pψ(ci|φt(x)), i.e., it explicitly increases the emotion classifier's posterior for the chosen emotion at every flow-matching ODE step. The only quantitative emotion metric, the Intensity Score, is the average softmax logit of the target emotion computed by the same classifier ψ. Thus raising α must mechanically raise the reported Intensity Score; the monotonic curves in Fig. 3 are a consequence of the optimization objective itself, not independent evidence that listeners perceive stronger emotion. No human emotion-rating study is reported (MOS-N and MOS-S measure naturalness and similarity only), so the emotion-controllability claim rests on this self-referential measure.

full rationale

The paper's dubbing-quality results are externally grounded: LSE-C/LSE-D use SyncNet, WER uses Whisper-V3, and the ablations show LPA/PE/SIA affect those metrics in sensible directions. The circularity is confined to the emotion-control half of the contribution. The intensity evaluation is not an independent test: the PNGM gradient is the derivative of the log of the same classifier whose average posterior defines the Intensity Score, so increasing α increases the metric by construction. The t-SNE separation and spectrogram visualizations are suggestive but are not quantified against an independent emotion perceptron or human labels and do not break the circularity. I also note, as a reproducibility issue rather than a circularity issue, that the negative-guidance term in Eq. (11) writes pψ(Σ_{j≠i} lj cj | φt(x)) with a weighted sum of one-hot emotion labels as the acoustic argument, which is type-incorrect for a mel-spectrogram classifier and cannot be computed as printed. Overall score 7: one central 'prediction' (emotion intensity control) reduces by construction, while the dubbing-quality claims remain independent.

Assumptions & free parameters 2 free parameters · 5 assumptions · 1 invented entities

The central claims rest on external alignment supervision (MFA), an emotion classifier used both for steering and scoring, and the standard flow-matching/classifier-guidance machinery. The only new postulated quantity is the mixed-emotion soft label used in the negative guidance term, which is not independently validated.

free parameters (2)
  • PNGM guidance scales (gamma, alpha, beta) = gamma=15; alpha in [0,9]; beta in [0,2]; quality tests use alpha=3.5
    Hand-chosen controls of emotion strength; the reported intensity curves and quality results depend directly on these values.
  • Contrastive temperature tau = 0.1
    Set by hand for the DLCL loss in Eq. 5; no sensitivity analysis is reported.
assumptions (5)
  • domain assumption MFA forced alignment gives correct phoneme-to-lip correspondence, used to construct the ground-truth mask M^{gt}_{lip,pho}.
    The DLCL loss in Eqs. 5 and 6 assumes the MFA alignments (with FPS/SR conversion) are reliable enough to supervise the attention matrix.
  • domain assumption The emotion classifier psi trained on Emobox provides gradients that point toward perceptually more emotional speech and is a valid proxy for human emotion perception.
    PNGM in Eq. 11 and the Intensity Score in Section 4.5 both assume psi's logits are meaningful for mel-spectrogram guidance and evaluation.
  • standard math OT-CFM with linear interpolation can represent the mel-spectrogram distribution conditioned on acoustics prior mu.
    The flow matching training objective Eq. 10 assumes the linear OT path is an adequate probability path.
  • standard math Applying classifier guidance to a flow-matching vector field preserves the conditional sampling distribution.
    The method extends Classifier Guidance [11] to flow matching; the corrected vector field is assumed rather than derived.
  • ad hoc to paper The negative-guidance term in Eq. 11 is a well-defined probability of a mixed emotion label.
    As written, p_psi(sum_{j!=i} l_j c_j | phi_t(x)) applies a classifier to a weighted sum of one-hot vectors, which is not a standard class label; this is an unstated modeling choice.
invented entities (1)
  • Mixed emotion soft label c_M = sum_i l_i c_i
    purpose: Defines the negative guidance target in Eq. 11, representing the current emotional mixture to be suppressed.
    The paper introduces this weighted sum of one-hot emotion vectors without showing that it corresponds to a well-defined probability or to a perceptually meaningful emotion mixture.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EmoDubber: Towards High Quality and Emotion Controllable Movie Dubbing." pith.science (2026). https://pith.science/paper/QAHL7CQG

@misc{pith2026241208988,
  author       = {Pith},
  title        = {Pith review of: EmoDubber: Towards High Quality and Emotion Controllable Movie Dubbing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QAHL7CQG}},
  note         = {Machine review of arXiv:2412.08988}
}
read the original abstract

Given a piece of text, a video clip, and a reference audio, the movie dubbing task aims to generate speech that aligns with the video while cloning the desired voice. The existing methods have two primary deficiencies: (1) They struggle to simultaneously hold audio-visual sync and achieve clear pronunciation; (2) They lack the capacity to express user-defined emotions. To address these problems, we propose EmoDubber, an emotion-controllable dubbing architecture that allows users to specify emotion type and emotional intensity while satisfying high-quality lip sync and pronunciation. Specifically, we first design Lip-related Prosody Aligning (LPA), which focuses on learning the inherent consistency between lip motion and prosody variation by duration level contrastive learning to incorporate reasonable alignment. Then, we design Pronunciation Enhancing (PE) strategy to fuse the video-level phoneme sequences by efficient conformer to improve speech intelligibility. Next, the speaker identity adapting module aims to decode acoustics prior and inject the speaker style embedding. After that, the proposed Flow-based User Emotion Controlling (FUEC) is used to synthesize waveform by flow matching prediction network conditioned on acoustics prior. In this process, the FUEC determines the gradient direction and guidance scale based on the user's emotion instructions by the positive and negative guidance mechanism, which focuses on amplifying the desired emotion while suppressing others. Extensive experimental results on three benchmark datasets demonstrate favorable performance compared to several state-of-the-art methods.

Figures

Figures reproduced from arXiv: 2412.08988 by the authors.

Figure 1
Figure 1. (a) Illustration of the V2C tasks. (b) EmoDubber can [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Intensity performance of EmoDubber on Chem. The horizontal axis shows the positive guidance [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Visualization of audio samples generated by EmoDubber: one uses the proposed FUEC to guide emotions by users, and the other [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Visual results of emotional audio features by t-SNE, the [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Towards Expressive Video Dubbing with Multiscale Multimodal Context Interaction

    cs.MM 2024-12 conditional novelty 6.0 of 10

    M2CI-Dubber improves dubbing prosody by extracting global sentence-level and local phoneme-level features from multimodal context and fusing them with the current text through attention and graph interaction.

Reference graph

Works this paper leans on

69 extracted references · 57 canonical work pages · cited by 1 Pith paper

  1. [1]

    Eric Battenberg, R. J. Skerry-Ryan, Soroosh Mariooryad, Daisy Stanton, David Kao, Matt Shannon, and Tom Bagby. Location-relative attention mechanisms for robust long-form speech synthesis. In ICASSP, pages 6194–6198, 2020. 6

    work page 2020

  2. [2]

    Audio-visual efficient conformer for robust speech recognition

    Maxime Burchi and Radu Timofte. Audio-visual efficient conformer for robust speech recognition. In WACV, pages 2257–2266, 2023. 4

    work page 2023

  3. [3]

    Prompt-enhanced multiple instance learning for weakly supervised video anomaly detection

    Junxi Chen, Liang Li, Li Su, Zheng-Jun Zha, and Qing- ming Huang. Prompt-enhanced multiple instance learning for weakly supervised video anomaly detection. In CVPR, pages 18319–18329, 2024. 2

    work page 2024

  4. [4]

    V2C: visual voice cloning

    Qi Chen, Mingkui Tan, Yuankai Qi, Jiaqiu Zhou, Yuanqing Li, and Qi Wu. V2C: visual voice cloning. In CVPR, pages 21210–21219, 2022. 1, 2, 6, 7

    work page 2022

  5. [5]

    Neural ordinary differential equa- tions

    Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equa- tions. Advances in neural information processing systems , 31, 2018. 2

    work page 2018

  6. [6]

    Out of time: Auto- mated lip sync in the wild

    Joon Son Chung and Andrew Zisserman. Out of time: Auto- mated lip sync in the wild. In ACCV Workshop, pages 251– 263, 2016. 6

    work page 2016

  7. [7]

    Learning to dub movies via hierarchical prosody models

    Gaoxiang Cong, Liang Li, Yuankai Qi, Zheng-Jun Zha, Qi Wu, Wenyu Wang, Bin Jiang, Ming-Hsuan Yang, and Qingming Huang. Learning to dub movies via hierarchical prosody models. In CVPR, pages 14687–14697, 2023. 1, 2, 3, 4, 6, 7

    work page 2023

  8. [8]

    Styledubber: To- wards multi-scale style learning for movie dubbing

    Gaoxiang Cong, Yuankai Qi, Liang Li, Amin Beheshti, Zhe- dong Zhang, Anton van den Hengel, Ming-Hsuan Yang, Chenggang Yan, and Qingming Huang. Styledubber: To- wards multi-scale style learning for movie dubbing. In Find- ings of ACL, pages 6767–6779, 2024. 2, 3, 4, 5, 6, 7

    work page 2024

Show all 69 references
  1. [9]

    An audio-visual corpus for speech perception and au- tomatic speech recognition

    Martin Cooke, Jon Barker, Stuart Cunningham, and Xu Shao. An audio-visual corpus for speech perception and au- tomatic speech recognition. The Journal of the Acoustical Society of America, 120(5):2421–2424, 2006. 5

    2006

  2. [10]

    Stochas- tic context consistency reasoning for domain adaptive object detection

    Yiming Cui, Liang Li, Jiehua Zhang, Chenggang Yan, Hongkui Wang, Shuai Wang, Heng Jin, and Li Wu. Stochas- tic context consistency reasoning for domain adaptive object detection. In ACM MM, pages 1331–1340, 2024. 2

    2024

  3. [11]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural informa- tion processing systems, 34:8780–8794, 2021. 2, 5

    2021

  4. [12]

    Cosyvoice 2: Scalable streaming speech synthesis with large language models

    Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xi- ang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, et al. Cosyvoice 2: Scalable streaming speech synthesis with large language models. arXiv preprint arXiv:2412.10117, 2024. 2

    2024 arXiv

  5. [13]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In ICML, 2024. 2

    2024

  6. [14]

    Lumina-t2x: Transforming text into any modality, reso- lution, and duration via flow-based large diffusion transform- ers

    Peng Gao, Le Zhuo, Ziyi Lin, Chris Liu, Junsong Chen, Ruoyi Du, Enze Xie, Xu Luo, Longtian Qiu, Yuhang Zhang, et al. Lumina-t2x: Transforming text into any modality, reso- lution, and duration via flow-based large diffusion transform- ers. arXiv, 2024. 2

    2024

  7. [15]

    Gomez, and J¨urgen Schmidhuber

    Alex Graves, Santiago Fern ´andez, Faustino J. Gomez, and J¨urgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In ICML, pages 369–376, 2006. 4

    2006

  8. [16]

    Emod- iff: Intensity controllable emotional text-to-speech with soft- label guidance

    Yiwei Guo, Chenpeng Du, Xie Chen, and Kai Yu. Emod- iff: Intensity controllable emotional text-to-speech with soft- label guidance. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023. 2, 5, 6

    2023

  9. [17]

    V oiceflow: Efficient text-to-speech with rectified flow matching

    Yiwei Guo, Chenpeng Du, Ziyang Ma, Xie Chen, and Kai Yu. V oiceflow: Efficient text-to-speech with rectified flow matching. In ICASSP, pages 11121–11125, 2024. 2

    2024

  10. [18]

    More than words: In-the-wild visually-driven prosody for text-to- speech

    Michael Hassid, Michelle Tadmor Ramanovich, Brendan Shillingford, Miaosen Wang, Ye Jia, and Tal Remez. More than words: In-the-wild visually-driven prosody for text-to- speech. In CVPR, pages 10577–10587, 2022. 1, 2

    2022

  11. [19]

    Neural dubber: Dubbing for videos according to scripts

    Chenxu Hu, Qiao Tian, Tingle Li, Yuping Wang, Yuxuan Wang, and Hang Zhao. Neural dubber: Dubbing for videos according to scripts. In NeurIPS, pages 16582–16595, 2021. 1, 2, 4, 5, 6

    2021

  12. [20]

    Generspeech: Towards style transfer for generalizable out-of-domain text-to-speech

    Rongjie Huang, Yi Ren, Jinglin Liu, Chenye Cui, and Zhou Zhao. Generspeech: Towards style transfer for generalizable out-of-domain text-to-speech. In NeurIPS, 2022. 8

    2022

  13. [21]

    Audio sub- titling: Dubbing and voice-over effects and their impact on user experience

    Gonzalo Iturregui-Gallardo and Anna Matamala. Audio sub- titling: Dubbing and voice-over effects and their impact on user experience. Perspectives, 29(1):64–83, 2021. 2

    2021

  14. [22]

    Faces that speak: Jointly syn- thesising talking face and speech from text

    Youngjoon Jang, Ji-Hoon Kim, Junseok Ahn, Doyeop Kwak, Hongsun Yang, Yooncheol Ju, Ilhwan Kim, Byeong-Yeol Kim, and Joon Son Chung. Faces that speak: Jointly syn- thesising talking face and speech from text. In CVPR, pages 8818–8828, 2024. 6

    2024

  15. [23]

    Glow-tts: A generative flow for text-to-speech via monotonic alignment search

    Jaehyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. In NeurIPS, 2020. 4

    2020

  16. [24]

    Lip-to-speech synthesis in the wild with multi-task learning

    Minsu Kim, Joanna Hong, and Yong Man Ro. Lip-to-speech synthesis in the wild with multi-task learning. In ICASSP, pages 1–5, 2023. 4

    2023

  17. [25]

    Deep vi- sual forced alignment: Learning to align transcription with talking face video

    Minsu Kim, Chae Won Kim, and Yong Man Ro. Deep vi- sual forced alignment: Learning to align transcription with talking face video. In AAAI, pages 8273–8281, 2023. 4

    2023

  18. [26]

    Shih, Rohan Badlani, Jo ˜ao Felipe Santos, Evelina Bakhturina, Mikyas Desta, Rafael Valle, Sungroh Yoon, and Bryan Catanzaro

    Sungwon Kim, Kevin J. Shih, Rohan Badlani, Jo ˜ao Felipe Santos, Evelina Bakhturina, Mikyas Desta, Rafael Valle, Sungroh Yoon, and Bryan Catanzaro. P-flow: A fast and data-efficient zero-shot TTS through speech prompting. In NeurIPS, 2023. 2

    2023

  19. [27]

    Imag- inary voice: Face-styled diffusion model for text-to-speech

    Jiyoung Lee, Joon Son Chung, and Soo-Whan Chung. Imag- inary voice: Face-styled diffusion model for text-to-speech. In ICASSP, pages 1–5, 2023. 1, 2, 6, 7

    2023

  20. [28]

    Bigvgan: A universal neural vocoder with large-scale training

    Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. Bigvgan: A universal neural vocoder with large-scale training. In ICLR, 2023. 5

    2023

  21. [29]

    Mm-tts: A unified framework for multimodal, prompt-induced emotional text-to-speech syn- thesis

    Xiang Li, Zhi-Qi Cheng, Jun-Yan He, Xiaojiang Peng, and Alexander G Hauptmann. Mm-tts: A unified framework for multimodal, prompt-induced emotional text-to-speech syn- thesis. arXiv preprint arXiv:2404.18398, 2024. 2

    2024 arXiv

  22. [30]

    Flow matching for generative mod- eling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximil- ian Nickel, and Matt Le. Flow matching for generative mod- eling. arXiv preprint arXiv:2210.02747, 2022. 2

    2022 arXiv

  23. [31]

    Entity-enhanced adaptive reconstruction network for weakly supervised refer- ring expression grounding

    Xuejing Liu, Liang Li, Shuhui Wang, Zheng-Jun Zha, Zechao Li, Qi Tian, and Qingming Huang. Entity-enhanced adaptive reconstruction network for weakly supervised refer- ring expression grounding. IEEE PAMI, 45(3):3003–3018,

  24. [32]

    Visualtts: TTS with accurate lip-speech syn- chronization for automatic voice over

    Junchen Lu, Berrak Sisman, Rui Liu, Mingyang Zhang, and Haizhou Li. Visualtts: TTS with accurate lip-speech syn- chronization for automatic voice over. In ICASSP, pages 8032–8036, 2022. 2, 6

    2022

  25. [33]

    Towards practical lipreading with distilled and effi- cient models

    Pingchuan Ma, Brais Martinez, Stavros Petridis, and Maja Pantic. Towards practical lipreading with distilled and effi- cient models. In ICASSP, pages 7608–7612, 2021. 5

    2021

  26. [34]

    End-to- end audio-visual speech recognition with conformers

    Pingchuan Ma, Stavros Petridis, and Maja Pantic. End-to- end audio-visual speech recognition with conformers. In ICASSP, pages 7613–7617, 2021. 4

    2021

  27. [35]

    Emobox: Multilingual multi-corpus speech emo- tion recognition toolkit and benchmark

    Ziyang Ma, Mingjie Chen, Hezhao Zhang, Zhisheng Zheng, Wenxi Chen, Xiquan Li, Jiaxin Ye, Xie Chen, and Thomas Hain. Emobox: Multilingual multi-corpus speech emo- tion recognition toolkit and benchmark. arXiv preprint arXiv:2406.07162, 2024. 5

    2024 arXiv

  28. [36]

    Lipreading using temporal convolutional networks

    Brais Martinez, Pingchuan Ma, Stavros Petridis, and Maja Pantic. Lipreading using temporal convolutional networks. In ICASSP, pages 6319–6323, 2020. 5

    2020

  29. [37]

    Montreal forced aligner: Trainable text-speech alignment using kaldi

    Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal forced aligner: Trainable text-speech alignment using kaldi. In In- terspeech, pages 498–502, 2017. 4

    2017

  30. [38]

    Matcha-tts: A fast tts architecture with conditional flow matching

    Shivam Mehta, Ruibo Tu, Jonas Beskow, ´Eva Sz´ekely, and Gustav Eje Henter. Matcha-tts: A fast tts architecture with conditional flow matching. In ICASSP, pages 11341–11345,

  31. [39]

    Meta-stylespeech : Multi-speaker adaptive text-to- speech generation

    Dongchan Min, Dong Bok Lee, Eunho Yang, and Sung Ju Hwang. Meta-stylespeech : Multi-speaker adaptive text-to- speech generation. In ICML, pages 7748–7759, 2021. 6, 7

    2021

  32. [40]

    Andrew Cameron Morris, Viktoria Maier, and Phil D. Green. From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition. In Interspeech, pages 2765–2768, 2004. 6

    2004

  33. [41]

    Dynamic time warping

    Meinard M ¨uller. Dynamic time warping. Information Re- trieval for Music and Motion, pages 69–84, 2007. 6

    2007

  34. [42]

    Librispeech: An ASR corpus based on public domain audio books

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ASR corpus based on public domain audio books. In ICASSP, pages 5206–5210, 2015. 5

    2015

  35. [43]

    K. R. Prajwal, Rudrabha Mukhopadhyay, Vinay P. Nam- boodiri, and C. V . Jawahar. Learning individual speaking styles for accurate lip to speech synthesis. In CVPR, pages 13793–13802, 2020. 5

    2020

  36. [44]

    K. R. Prajwal, Rudrabha Mukhopadhyay, Vinay P. Nam- boodiri, and C. V . Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In ACM MM, pages 484–492, 2020. 6

    2020

  37. [45]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In ICML, pages 28492–28518, 2023. 6

    2023

  38. [46]

    Fastspeech 2: Fast and high-quality end-to-end text to speech

    Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. In ICLR, 2021. 6, 7

    2021

  39. [47]

    Maximum likelihood training of score-based diffusion mod- els

    Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion mod- els. Advances in neural information processing systems, 34: 1415–1428, 2021. 2

    2021

  40. [48]

    Equivariant flow matching with hybrid probability transport for 3d molecule generation

    Yuxuan Song, Jingjing Gong, Minkai Xu, Ziyao Cao, Yanyan Lan, Stefano Ermon, Hao Zhou, and Wei-Ying Ma. Equivariant flow matching with hybrid probability transport for 3d molecule generation. In NeurIPS, 2023. 2

    2023

  41. [49]

    Emomix: Emotion mixing via diffusion models for emotional speech synthesis

    Haobin Tang, Xulong Zhang, Jianzong Wang, Ning Cheng, and Jing Xiao. Emomix: Emotion mixing via diffusion models for emotional speech synthesis. arXiv preprint arXiv:2306.00648, 2023. 2

    2023 arXiv

  42. [50]

    Context disentangling and prototype inheriting for robust visual grounding

    Wei Tang, Liang Li, Xuejing Liu, Lu Jin, Jinhui Tang, and Zechao Li. Context disentangling and prototype inheriting for robust visual grounding. IEEE PAMI, 46(5):3213–3229,

  43. [51]

    Synchronous multi-modal semantic communica- tion system with packet-level coding

    Yun Tian, Jingkai Ying, Zhijin Qin, Ye Jin, and Xiaom- ing Tao. Synchronous multi-modal semantic communica- tion system with packet-level coding. IEEE Transactions on Wireless Communications, pages 1–1, 2025

    2025

  44. [52]

    Self-supervised cross-view represen- tation reconstruction for change captioning

    Yunbin Tu, Liang Li, Li Su, Zheng-Jun Zha, Chenggang Yan, and Qingming Huang. Self-supervised cross-view represen- tation reconstruction for change captioning. In ICCV, pages 2805–2815, 2023

    2023

  45. [53]

    Distractors-immune representation learning with cross-modal contrastive regularization for change cap- tioning

    Yunbin Tu, Liang Li, Li Su, Chenggang Yan, and Qing- ming Huang. Distractors-immune representation learning with cross-modal contrastive regularization for change cap- tioning. In ECCV, pages 311–328, 2024

    2024

  46. [54]

    SMART: syntax-calibrated multi-aspect rela- tion transformer for change captioning

    Yunbin Tu, Liang Li, Li Su, Zheng-Jun Zha, and Qing- ming Huang. SMART: syntax-calibrated multi-aspect rela- tion transformer for change captioning. IEEE PAMI, 46(7): 4926–4943, 2024. 2

    2024

  47. [55]

    Visualizing data using t-sne

    Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9 (11), 2008. 8

    2008

  48. [56]

    Generalized end-to-end loss for speaker verifica- tion

    Li Wan, Quan Wang, Alan Papir, and Ignacio L ´opez- Moreno. Generalized end-to-end loss for speaker verifica- tion. In ICASSP, pages 4879–4883, 2018. 1

    2018

  49. [57]

    Tan, and Haizhou Li

    Jiadong Wang, Xinyuan Qian, Malu Zhang, Robby T. Tan, and Haizhou Li. Seeing what you said: Talking face genera- tion guided by a lip reading expert. In CVPR, pages 14653– 14662, 2023. 6

    2023

  50. [58]

    Maskgct: Zero-shot text- to-speech with masked generative codec transformer

    Yuancheng Wang, Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Xueyao Zhang, Shunsi Zhang, and Zhizheng Wu. Maskgct: Zero-shot text- to-speech with masked generative codec transformer. arXiv preprint arXiv:2409.00750, 2024. 2

    2024 arXiv

  51. [59]

    Emotional face-to-speech

    Jiaxin Ye, Boyuan Cao, and Hongming Shan. Emotional face-to-speech. arXiv preprint arXiv:2502.01046, 2025. 2

    2025 arXiv

  52. [60]

    Unsupervised cross-media hashing learning via knowledge graph

    Zhaoda Ye, Xiangteng He, and Yuxin Peng. Unsupervised cross-media hashing learning via knowledge graph. Chinese Journal of Electronics, 31(6):1081–1091, 2022. 2

    2022

  53. [61]

    Lipvoicer: Generating speech from silent videos guided by lip reading

    Yochai Yemini, Aviv Shamsian, Lior Bracha, Sharon Gan- not, and Ethan Fetaya. Lipvoicer: Generating speech from silent videos guided by lip reading. In ICLR, 2024. 4, 6

    2024

  54. [62]

    Inductive state- relabeling adversarial active learning with heuristic clique rescaling

    Beichen Zhang, Liang Li, Shuhui Wang, Shaofei Cai, Zheng- Jun Zha, Qi Tian, and Qingming Huang. Inductive state- relabeling adversarial active learning with heuristic clique rescaling. IEEE PAMI, 46(12):9780–9796, 2024. 2

    2024

  55. [63]

    From speaker to dubber: Movie dubbing with prosody and duration consistency learning

    Zhedong Zhang, Liang Li, Gaoxiang Cong, YIN Haibing, Yuhan Gao, Chenggang Yan, Anton van den Hengel, and Yuankai Qi. From speaker to dubber: Movie dubbing with prosody and duration consistency learning. In ACM MM,

  56. [64]

    Generating high-quality symbolic music using fine-grained discriminators

    Zhedong Zhang, Liang Li, Jiehua Zhang, Zhenghui Hu, Hongkui Wang, Chenggang Yan, Jian Yang, and Yuankai Qi. Generating high-quality symbolic music using fine-grained discriminators. In ICPR, pages 332–344, 2024. 2

    2024

  57. [65]

    Prosody- enhanced acoustic pre-training and acoustic-disentangled prosody adapting for movie dubbing

    Zhedong Zhang, Liang Li, Chenggang Yan, Chunshan Liu, Anton van den Hengel, and Yuankai Qi. Prosody- enhanced acoustic pre-training and acoustic-disentangled prosody adapting for movie dubbing. arXiv preprint arXiv:2503.12042, 2025. 2

    2025 arXiv

  58. [66]

    Mcdubber: Multimodal context-aware ex- pressive video dubbing

    Yuan Zhao, Zhenqi Jia, Rui Liu, De Hu, Feilong Bao, and Guanglai Gao. Mcdubber: Multimodal context-aware ex- pressive video dubbing. arXiv preprint arXiv:2408.11593 ,

    arXiv

  59. [67]

    Towards expres- sive video dubbing with multiscale multimodal context inter- action

    Yuan Zhao, Rui Liu, and Gaoxiang Cong. Towards expres- sive video dubbing with multiscale multimodal context inter- action. In ICASSP, pages 1–5, 2025. 2

    2025

  60. [68]

    Schuller, and Haizhou Li

    Kun Zhou, Berrak Sisman, Rajib Rana, Bj ¨orn W. Schuller, and Haizhou Li. Speech synthesis with mixed emotions. IEEE Trans. Affect. Comput., 14(4):3120–3134, 2023. 5, 6

    2023

  61. [69]

    Content-dependent fine-grained speaker embedding for zero- shot speaker adaptation in text-to-speech synthesis

    Yixuan Zhou, Changhe Song, Xiang Li, Luwen Zhang, Zhiyong Wu, Yanyao Bian, Dan Su, and Helen Meng. Content-dependent fine-grained speaker embedding for zero- shot speaker adaptation in text-to-speech synthesis. In Inter- speech, pages 2573–2577, 2022. 7

    2022

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.