{"id":"b87d46f7-06bc-4256-b2d4-5fc9967b9389","arxiv_id":"2412.08988","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An emotion-controllable movie dubbing model uses supervised lip-prosody alignment, phoneme enhancement, and flow matching with positive/negative classifier guidance to synthesize speech with user-chosen emotion type and intensity.","lead":"EmoDubber is a movie dubbing system that generates speech matching a silent video while letting users choose an emotion and its strength. It reports large gains in lip-sync and pronunciation quality over prior dubbing systems on two benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Emotion-controllability claim rests on an undefined PNGM term and a self-confirming metric; keep CONDITIONAL pending corrected Eq. 11 and human intensity ratings.","rationale":"The reader's weakest assumption identified the emotion classifier as both steering and measurement instrument, which is exactly the soft spot I find. The paper's central claim is user-controllable emotion type and intensity, and the evidence for that claim depends on (a) Eq. 11 being a well-defined gradient update and (b) the Intensity Score measuring perceived emotion rather than the optimization target. The first condition fails on its face: p_ψ is defined on acoustic features, but the negative-guidance term evaluates it on a weighted sum of one-hot class vectors, which is not a valid input. The second condition is structurally self-confirming: Figure 3 plots the classifier's own posterior against the guidance scale that maximizes that posterior, so monotonicity is expected even if listeners perceive no change. The paper does report human MOS for naturalness and similarity, but no human emotion-intensity ratings, so the half of the contribution most central to the title lacks independent validation. I would therefore keep the reader's CONDITIONAL verdict rather than ACCEPT: the lip-sync and pronunciation results may be solid, but the emotion-controllability claim should be accepted only after the PNGM formula is corrected or clarified and confirmed by a human listening test. No change to the reader's verdict is needed beyond sharpening those conditions.","tokens_in":16483,"tokens_out":5286,"duration_ms":56477,"concrete_test":"Obtain the released implementation from the stated GitHub link and trace the PNGM update at inference. If the negative-guidance term literally calls p_ψ on Σ_{j≠i} l_j c_j, the formula is undefined and the method must be corrected; if it uses ∇log(1−p_ψ(c_i|x)) or another valid expression, then Eq. 11 must be revised to match the code. Independently, run a 20+ listener forced-choice or rating test on Chem samples synthesized with α∈{0,2,5}, fixed β, asking which sample sounds more strongly of the target emotion; perceived intensity must rise with α for the controllability claim to hold.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing point is the emotion-control mechanism, not lip-sync or pronunciation. Equation 11's negative-guidance term is β∇log p_ψ(Σ_{j≠i} l_j c_j | φ_t(x)), but p_ψ is a classifier on mel-spectrograms; its first argument here is a weighted sum of one-hot emotion labels, not an acoustic feature, so the expression is type-incorrect and not computable as written. A valid suppression term would be ∇log(1−p_ψ(c_i|x)) or ∇logΣ_{j≠i}p_ψ(c_j|x). Thus the central algorithmic description cannot be reproduced from the paper alone. Separately, the intensity evaluation is circular: Section 4.5's 'Intensity Score' is the average softmax output of the same classifier ψ that supplies the PNGM gradient, so Figure 3 mostly shows that increasing α raises ψ's own posterior, not that listeners hear stronger emotion. No human emotion-rating study is reported; MOS-N and MOS-S measure naturalness and similarity only. The GRID WER result (generated 18.53 vs ground truth 22.41) is an additional sign of evaluation-side artifacts. The lip-sync and pronunciation numbers are strong and may stand, but the emotion-controllability half of the title is not yet supported as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EmoDubber, an end-to-end movie dubbing system that takes text, a silent video, a reference audio, and a user-specified emotion instruction E = {c, alpha, beta} as inputs. The architecture combines a Lip-related Prosody Aligning (LPA) module with duration-level contrastive learning, a Pronunciation Enhancing (PE) conformer that fuses phoneme and lip-motion features, a Speaker Identity Adapting (SIA) module that injects speaker style into acoustics priors, and a Flow-based User Emotion Controlling (FUEC) decoder that applies positive and negative classifier guidance (PNGM) for emotion intensity control. Experiments on Chem, GRID, and a zero-shot speaker setting compare against dubbing baselines using SyncNet lip-sync metrics, Whisper word error rate, speaker cosine similarity, MCD, and subjective MOS, together with an emotion intensity analysis.","tokens_in":16719,"tokens_out":5857,"duration_ms":62799,"significance":"If the results hold, the paper would make a useful contribution to controllable movie dubbing: the lip-sync and pronunciation claims are supported by external metrics (SyncNet and Whisper), and the ablation in Table 3 cleanly attributes LSE gains to LPA and WER gains to PE. The code and trained models are promised, which would increase reproducibility. However, the headline emotion-controllability claim is not currently established: the negative-guidance term in Eq. (11) is not computable as written, and the Intensity Score in Section 4.5 is a self-confirming metric that uses the same emotion classifier that drives the guidance. The paper would be substantially strengthened by a corrected formulation of Eq. (11) and by independent human or held-out emotion perception tests.","major_comments":[{"comment":"The negative-guidance term is not well defined. In Eq. (11), the second gradient is taken over log p_psi(sum_{j != i} l_j c_j | phi_t(x)), but p_psi is described as a classifier that takes a mel-spectrogram and returns a distribution over emotion classes; the first argument is therefore a convex combination of one-hot emotion vectors, not an acoustic feature. As written, p_psi(...) is type-incorrect, so the expression cannot be computed or reproduced from the paper alone. The authors should define the suppressed-emotion term explicitly, for example using log(1 - p_psi(c_i | x)), log sum_{j != i} p_psi(c_j | x), or a properly defined soft-label likelihood, and state how gradients are taken with respect to x.","section":"§3.5, Eq. (11)"},{"comment":"The Intensity Score is the average softmax logit of the target emotion computed by the same emotion classifier psi that supplies the PNGM gradients in Eq. (11). Because the guidance mechanism explicitly increases log p_psi(c_i | phi_t(x)), Figure 3 mostly demonstrates that the optimizer follows the classifier gradient; it does not independently establish that listeners perceive stronger or more controllable emotion. Please report a human emotion-rating study (e.g., perceived emotion type and intensity) or evaluate with a held-out speech emotion recognition model that was not used during generation, and provide confidence intervals for each alpha/beta condition.","section":"§4.5, Figure 3"},{"comment":"The GRID WER result needs discussion: generated speech achieves WER 18.53% while ground-truth audio achieves 22.41%. If an ASR model finds synthesized speech more intelligible than the original recordings, this may reflect ASR bias or over-articulation rather than pronunciation quality. The authors should report WER with confidence intervals, verify with a second ASR system, and explain why synthesized speech is expected to beat the ground-truth WER before using this result to support the generalizable pronunciation claim.","section":"§4.4, Table 4"},{"comment":"None of the quantitative tables report variance, confidence intervals, or significance tests, although several comparisons are decided by small margins (e.g., Table 1, Setting 1.0, MCD 5.87 for Ours versus 5.41 for StyleDubber). Because the paper makes comparative claims of state-of-the-art performance, the authors should report standard deviations across multiple generation runs or test subsamples and, where feasible, statistical significance. This is especially important for the emotion intensity curves in Figure 3, which currently have no error bars.","section":"§4.4 and §4.5, Tables 1, 3, 4, 5 and Figure 3"},{"comment":"There is no quantitative comparison with an existing intensity-controllable emotional TTS system. The only emotion baseline, GenerSpeech, is evaluated through a t-SNE visualization, which does not measure intensity controllability or perceived emotion quality. To support the claim of flexible intensity control, the authors should compare with an intensity-controllable emotional TTS method such as EmoDiff under matched emotion classes and report both objective and perceptual emotion metrics.","section":"§4.5, emotional controlling evaluation"}],"minor_comments":[{"comment":"The denominator in Lcl appears to be missing the temperature tau and the negative-sample structure of a standard contrastive loss; as written, the expression is not a recognizable InfoNCE loss. Please clarify or correct the equation.","section":"§3.2, Eq. (5)"},{"comment":"The phrase 'We can set emotion classes as {c0, ..., cN-1} one-hot vector' should be reworded to 'each c_i is a one-hot vector', because the current wording is ambiguous about whether c_i is a class index or a vector.","section":"§3.5, after Eq. (11)"},{"comment":"The sentence reporting 'absolute improvements of 4.24% and 4.0%' should specify whether these are relative or absolute improvements in LSE-C and LSE-D, and against which baseline.","section":"§4.4, Chem results"},{"comment":"The caption does not state the evaluation protocol for the Intensity Score, such as the number of test samples and whether the score is averaged over entire utterances or frames. Please add this information.","section":"Figure 3 caption"},{"comment":"The STFT configuration lists window length 640, frame size 1,024, and hop length 160; this is unusual because the window length is smaller than the frame size. Please clarify whether 'frame size' refers to the FFT size and correct the terminology.","section":"§4.1, Implementation Details"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's strongest and most credible part is the lip-sync and pronunciation pipeline, supported by external SyncNet and Whisper metrics and a clean ablation. The emotion-controllability half of the title is currently supported by an uncomputable equation and a circular evaluation metric, but both issues appear fixable in revision. I would not reject the paper, but I would not accept it before the authors define Eq. (11) in a computable form, add independent perceptual emotion evaluation, and address the WER interpretation and lack of error bars."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the lip-sync and pronunciation half of this paper is credible and worth a look. The emotion-control half, which the title promises, is not yet demonstrated. If you cite this, cite it for the alignment and pronunciation results, not for the emotion claims.\n\nWhat's actually new: a duration-level contrastive loss (DLCL) that ties phoneme prosody to lip motion, a conformer-based pronunciation enhancer, and the idea of applying positive/negative classifier guidance to flow matching for emotion in dubbing. The ablation in Table 3 shows each module matters, and the LSE-C/LSE-D numbers on Chem and GRID are close to ground truth and clearly better than the baselines. That part is a real engineering contribution.\n\nThe soft spots are concentrated where the paper makes its headline claim. Equation 11 defines the negative guidance term as the gradient of log p_psi(sum of one-hot labels), but p_psi is a classifier on mel-spectrograms, and its first argument here is a weighted sum of one-hot emotion labels. That expression is type-incorrect; it cannot be computed as written. The intensity score in Section 4.5 is the average softmax output of the same classifier psi that supplies the guidance gradients, so Figure 3 mostly shows that increasing alpha raises psi's own posterior, not that listeners hear stronger emotion. No human emotion-rating study is reported—MOS-N and MOS-S only cover naturalness and similarity. And on GRID, the generated WER (18.53) is lower than the ground truth WER (22.41), which suggests the evaluation setup favors the generator; that needs explanation.\n\nNone of this kills the lip-sync and pronunciation results—those are grounded in external SyncNet and Whisper metrics and a clean ablation. But the emotion-controllability claim needs a corrected Eq. 11, an independent intensity metric (e.g., human ratings or a held-out classifier), and significance testing on the tables.\n\nBottom line: this deserves a serious referee, but the referee should require major revision before acceptance. If you are in the dubbing/TTS space, the alignment and pronunciation modules are worth reading carefully; the emotion part is a cautionary example of circular evaluation.","headline":"Solid lip-sync and pronunciation work; the emotion-controllability claim is undercut by a type-incorrect guidance equation and a self-confirming intensity metric.","tokens_in":17291,"tokens_out":2837,"would_cite":false,"duration_ms":27555,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Movie dubbing gains user-controlled emotion and intensity without losing lip sync or pronunciation.","keywords":["movie dubbing","visual voice cloning","emotion-controllable speech synthesis","flow matching","classifier guidance","lip-sync alignment","pronunciation enhancement","contrastive learning"],"falsifier":"A listener study in which humans rate the emotional intensity of outputs generated with increasing positive guidance (and fixed negative guidance), using an emotion recognizer trained independently of the steering classifier, would settle whether the control is perceptual rather than self-confirming; the claim fails if human ratings do not track the guidance weights or if the independent recognizer shows no consistent emotion shift.","tokens_in":16242,"feed_emoji":"🎬","tokens_out":9340,"duration_ms":90426,"temperature":0.7,"pith_summary":"EmoDubber makes the case that a dubbing system can satisfy three requirements at once: lip movements match the spoken audio, words remain intelligible, and the emotional colour of the delivery is chosen by the user rather than fixed by the source recording. The paper's central claim is that a user can supply an emotion label and two intensity weights, and the system will steer a flow-matching speech generator toward that emotion (and away from the others) while a duration-level contrastive alignment keeps the prosody locked to the video and a phoneme-enhancement path keeps pronunciation clear. A sympathetic reader would care because this moves dubbing from re-recording or limited transfer toward direct, director-style control: pick the emotion, set how strong it is, and keep the performance in sync.","feed_headline":"Dubbing gets an emotion dial: type and intensity on demand","feed_subtitle":"EmoDubber syncs dubbed speech to lip motion, keeps words clear, and lets users set how strong the emotion is.","key_machinery":"The carrying mechanism is the positive-and-negative guidance (PNGM) applied inside the flow-matching prediction network. Given an acoustics prior $\\mu$ and the user instruction $E=\\{c,\\alpha,\\beta\\}$, the predicted velocity field is altered by adding $\\gamma(\\alpha\\nabla\\log p_{\\psi}(c\\mid x) - \\beta\\nabla\\log p_{\\psi}(\\sum_{j\\neq i} l_j c_j \\mid x))$, so that the noise-to-mel-spectrogram trajectory is pushed toward the chosen emotion and pulled away from all others, with $\\alpha$ and $\\beta$ acting as the intensity controls. The rest of the architecture supplies the conditions for that guidance: duration-level contrastive learning produces a monotonic lip-to-phoneme alignment, the pronunciation-enhancing path expands phoneme sequences and fuses them with lip context, and the speaker-identity module converts the fused sequence into the acoustics prior $\\mu$.","core_discovery":"EmoDubber's central claim is that emotion-controllable dubbing is achieved by combining four modules: Lip-related Prosody Aligning uses duration-level contrastive learning to enforce a monotonic correspondence between phoneme prosody and lip motion; Pronunciation Enhancing expands phoneme sequences to video length with monotonic alignment search and fuses them with the lip context in an efficient conformer; Speaker Identity Adapting up-samples the fused features and injects the reference speaker's style to form an acoustics prior; and Flow-based User Emotion Controlling modifies the flow-matching vector field by positive and negative classifier guidance, adding a scaled gradient toward the requested emotion and subtracting a scaled gradient toward the mixture of all other emotions. On the Chem and GRID benchmarks, the paper reports near-ground-truth lip-sync metrics (for example, LSE-C 8.11 against ground truth 8.12 on Chem setting 1), word error rates comparable to or better than prior dubbing systems, and an intensity score that rises with the positive guidance weight and falls with the negative weight. The paper also reports that adding emotion control leaves lip sync, pronunciation, and speaker similarity essentially unchanged across seven emotion categories.","pith_inferences":["An inference beyond the paper: because the same emotion classifier both steers generation and scores the output's intensity, the reported monotonic intensity curves may partly reflect the classifier's own preferences; an independent listener study or a different emotion recognizer would tell whether the perceived emotion actually scales with $\\alpha$ and $\\beta$.","A testable extension the paper leaves implicit: positive-and-negative guidance should transfer to any flow-matching or diffusion generator with a differentiable attribute classifier, so the same dual-gradient recipe could give fine-grained control of speaking style, accent, or even non-speech acoustic attributes without retraining.","A practical limitation worth checking: the alignment path depends on Montreal Forced Aligner and phoneme-level supervision, so extending the method to languages or casual video without reliable forced aligners would need a self-supervised duration-alignment substitute.","The intensity knob may enable emotion interpolation, generating mixed emotions by tuning $\\alpha$ for one label and $\\beta$ for another, which the paper gestures at with its mixture-of-logits negative term but does not systematically evaluate."],"forward_implications":["A director or user could author a dub's emotional performance directly, choosing for example 'angry' or 'sad' and setting how strong it should be, without asking an actor to re-record.","Lip sync and intelligibility can be improved together: the ablation study shows that removing the lip-prosody aligning module hurts sync most and removing the pronunciation-enhancing module hurts word error rate most, so the two goals are not a simple trade-off.","Emotion instructions transfer to videos whose original speech carries no emotion label: the paper's zero-shot emotion conversion on Chem shows distinct t-SNE clusters for happy, sad, surprised, and angry speech.","Dual positive-and-negative guidance gives a wider and finer intensity range than single-direction guidance, because users can amplify the target emotion and suppress competing ones separately.","Applying emotion control leaves the other dubbing qualities intact: across seven emotion categories, lip-sync confidence and distance, word error rate, and speaker similarity stay close to the emotion-free output, with sadness the partial exception."],"supporting_citations":[{"why":"Supplies the lip-motion ROI extraction and prosody-adaptor pipeline that LPA extends with duration-level contrastive learning.","marker":"[7]"},{"why":"Provides the style phoneme encoder, utterance-level style learning, and a multi-scale style dubbing baseline that SIA builds on.","marker":"[8]"},{"why":"Pre-trains phoneme encoder, USL, and flow decoder on LibriSpeech and is the zero-shot dubbing baseline with prosody and duration consistency learning.","marker":"[63]"},{"why":"Defines the visual voice cloning task and baseline that EmoDubber compares against on speaker similarity and acoustic distance.","marker":"[4]"},{"why":"Introduces classifier guidance, the mechanism PNGM adapts with separate positive and negative gradient terms.","marker":"[11]"},{"why":"Supplies monotonic alignment search used in Pronunciation Enhancing to expand phoneme sequences to video length.","marker":"[23]"},{"why":"Introduces optimal-transport conditional flow matching, the training objective of the flow matching prediction network.","marker":"[38]"},{"why":"Provides the pooled emotion recognition datasets used to train the emotion expert classifier that steers and evaluates emotion.","marker":"[35]"},{"why":"Provides the forced alignment used to build the ground-truth lip-to-phoneme correspondence matrix for duration-level contrastive learning.","marker":"[37]"}],"fun_headline_variants":["EmoDubber: emotion control for dubbing, on demand","Dubbing with emotion dial: specify type and intensity","User-set emotions in movie dubbing, with clear sync","Flow guidance gives dubbing emotional intensity control","Pick the emotion: EmoDubber dubs with your intensity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the emotion classifier used to steer generation is a trustworthy measure of real human-perceived emotion, meaning its gradients point toward speech humans hear as more emotional and its softmax logits track felt intensity, so if that classifier is miscalibrated or domain-shifted the emotion-control claim is not demonstrated.","fun_headline_variants_meta":{"raw":{"variants":["EmoDubber: emotion control for dubbing, on demand","Dubbing with emotion dial: specify type and intensity","User-set emotions in movie dubbing, with clear sync","Flow guidance gives dubbing emotional intensity control","Pick the emotion: EmoDubber dubs with your intensity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000232,"raw_usage":{"total_tokens":1539,"prompt_tokens":1048,"completion_tokens":491,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":664,"completion_tokens_details":{"reasoning_tokens":410}},"tokens_in":664,"tokens_out":491,"duration_ms":5239,"temperature":1.0,"reasoning_tokens":410,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:22:00.583240+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A listener study in which humans rate the emotional intensity of outputs generated with increasing positive guidance (and fixed negative guidance), using an emotion recognizer trained independently of the steering classifier, would settle whether the control is perceptual rather than self-confirming; the claim fails if human ratings do not track the guidance weights or if the independent recognizer shows no consistent emotion shift.","supporting_citations":[{"cited_title":"Learning to dub movies via hierarchical prosody models","cited_arxiv_id":null,"evidence_quote":"Supplies the lip-motion ROI extraction and prosody-adaptor pipeline that LPA extends with duration-level contrastive learning."},{"cited_title":"Styledubber: To- wards multi-scale style learning for movie dubbing","cited_arxiv_id":null,"evidence_quote":"Provides the style phoneme encoder, utterance-level style learning, and a multi-scale style dubbing baseline that SIA builds on."},{"cited_title":"From speaker to dubber: Movie dubbing with prosody and duration consistency learning","cited_arxiv_id":null,"evidence_quote":"Pre-trains phoneme encoder, USL, and flow decoder on LibriSpeech and is the zero-shot dubbing baseline with prosody and duration consistency learning."},{"cited_title":"V2C: visual voice cloning","cited_arxiv_id":null,"evidence_quote":"Defines the visual voice cloning task and baseline that EmoDubber compares against on speaker similarity and acoustic distance."},{"cited_title":"Diffusion models beat gans on image synthesis","cited_arxiv_id":null,"evidence_quote":"Introduces classifier guidance, the mechanism PNGM adapts with separate positive and negative gradient terms."},{"cited_title":"Glow-tts: A generative flow for text-to-speech via monotonic alignment search","cited_arxiv_id":null,"evidence_quote":"Supplies monotonic alignment search used in Pronunciation Enhancing to expand phoneme sequences to video length."},{"cited_title":"Matcha-tts: A fast tts architecture with conditional flow matching","cited_arxiv_id":null,"evidence_quote":"Introduces optimal-transport conditional flow matching, the training objective of the flow matching prediction network."},{"cited_title":"Montreal forced aligner: Trainable text-speech alignment using kaldi","cited_arxiv_id":null,"evidence_quote":"Provides the forced alignment used to build the ground-truth lip-to-phoneme correspondence matrix for duration-level contrastive learning."}],"review_version":1}