Pith. sign in

REVIEW 5 major objections 5 minor 36 references

VisualSpeech: Enhancing Prosody Modeling in TTS Using Video

T0 review · 5 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VisualSpeech claims that adding video features to a TTS model improves prosody modeling, cutting pitch MSE by a third and duration MSE by half relative to FastSpeech2.

desk verdict A genuinely novel attempt to condition TTS prosody on video, but the evidence is under-powered and the lack of a movie-disjoint split leaves the core result vulnerable. read the letter →

arxiv 2501.19258 v2 pith:2TNAM7YS submitted 2025-01-31 cs.CL

Shumin Que , Anton Ragni This is my paper · ORCID
classification cs.CL
keywords text-to-speechprosodyvisualfeaturescross-attentionvideo-conditionedspeechsynthesisone-to-manymappingFastSpeech2CondensedMoviesDataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Text-to-speech must choose one prosodic rendering among many for the same sentence; this one-to-many problem is why generated speech can sound flat or contextually wrong. The paper argues that video of the speaker or scene contains information—expression, gesture, environment—that can guide that choice, and that this channel has been largely unused in prosody modeling. To test this, the authors extend FastSpeech2 with a visual encoder and a visual-text fusion module and train on a filtered film dataset. They report that the visual-conditioned model reduces pitch MSE from 0.27 to 0.18, duration MSE from 0.41 to 0.21, and Mel-cepstral distance from 7.64 to 7.38 relative to the text-only baseline, with higher UTMOS listening scores. If correct, this shows that readily available video context can carry prosody into TTS systems, complementing text.

What carries the argument

The load-bearing mechanism is the visual-text fusion module, which uses a single cross-attention layer $\delta(V,T)=\operatorname{Softmax}\left(\frac{T V^\top}{\sqrt{d_V}}\right)V$ in which text encoder outputs $T$ act as queries and visual encoder outputs $V$ as keys and values. Because the queries come from the text stream, the fusion output has the same length as the phone sequence and can be passed directly into the variance adaptor that predicts pitch, energy, and duration. Visual features are frame-level embeddings extracted offline by either the Omnivore video model or ResNet50, and the text side is the phone-embedding encoder of FastSpeech2. The paper also uses a simpler feed-forward predictor in a preliminary study to show that visual features alone encode prosody.

What would settle it

Shuffle the visual frames within each video clip and retrain VisualSpeech; if pitch and duration MSE stay close to the reported values, then time-aligned visual information is not what produces the improvement, and the cross-attention is acting as global pooling.

Watch

Extended reading notes

Core claim

The central claim is that visual context—facial expressions, gestures, and speaking environment visible in video frames—carries prosodic information that text alone does not, and that a TTS model can exploit it through cross-attention between text and video features. The paper supports this with two preliminary studies: a feed-forward network predicts pitch and energy from Omnivore video features with lower mean squared error than a mean predictor, and a text-plus-vision prosody predictor beats a text-only predictor on pitch and duration. The full VisualSpeech model then shows consistent gains over FastSpeech2 on all reported prosody metrics, and the gains persist when ground-truth pitch, energy, and duration are provided to both models, which the authors interpret as evidence that the visual stream also shapes mel-spectrogram prediction beyond those three targets. They conclude that visual information is predictive of prosody, complements textual features, and improves the expressiveness of synthesized speech.

Load-bearing premise

The load-bearing premise is that a frame-level visual feature sequence can supply phone-level prosodic information through learned cross-attention without explicit time alignment; if the attention only captures global video style, the reported gains could reflect speaker or scene correlations rather than aligned visual cues.

Editorial extensions

If this is right

  • If the reported gains hold, video-conditioned TTS becomes a practical route to expressive speech in applications where video is already present, such as game dialogue, dubbing, or narrated film content.
  • Visual features improve pitch and duration prediction beyond text alone—about 33% and 49% relative reduction in MSE on the test set—while energy prediction improves less, so the visual channel is informative but uneven across prosodic attributes.
  • The gain in Mel-cepstral distance persists when ground-truth pitch, energy, and duration are supplied to both models, suggesting the visual stream influences acoustic generation beyond the three explicit prosody targets.
  • The filtered CMD2 dataset, about 33 hours of film clips with aligned audio and text, gives later work a public resource for studying visual context in speech synthesis.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of whether the fusion uses temporal alignment is to replace the cross-attention with global average pooling of the visual features; if the performance gap mostly disappears, the reported gains come from global video style rather than moment-by-moment visual cues.
  • Because videos come from films, speaker identity, genre, and scene type are correlated with visual content, so a held-out evaluation across unseen films or speakers is needed before claiming the visual signal transfers.
  • The same text-as-query, video-as-context fusion could be extended to other contextual streams, such as speaker pose or listener gaze, or combined with reference-speech style tokens for finer prosodic control.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes VisualSpeech, a FastSpeech2-based TTS model that incorporates visual features extracted from video via a cross-attention fusion module, with the aim of improving prosody prediction (pitch, energy, duration). The authors report three experiments: (1) a feedforward network predicts pitch and energy from visual features alone, outperforming a mean predictor; (2) a text-plus-visual PED predictor yields small gains in pitch and duration but a loss in energy compared with text alone; and (3) full TTS experiments show improvements in pitch, energy, duration, MCD, and log-F0 over FastSpeech2. The paper also introduces the CMD2 dataset, a filtered subset of the Condensed Movies Dataset, and provides audio samples.

Significance. If the empirical claims hold, the paper would open a new conditioning modality for expressive TTS and contribute a valuable dataset (CMD2) for audiovisual speech research. The authors are to be credited for testing the contribution of visual features in a staged way (from isolated prosody prediction to full TTS), for comparing against a non-learned baseline (mean predictor) and a text-only model, and for making audio samples publicly available. However, the current evidence is not yet convincing: several load-bearing aspects—data split, statistical reliability, fusion ablations, and consistency between the two studies—limit the support for the central claim that visual context improves prosody through aligned visual cues.

major comments (5)
  1. [Section 3.1] The train/test split is random at the clip level with no movie-disjoint or speaker-disjoint partition. CMD2 is drawn from movies, and Omnivore/ResNet50 features can encode scene, speaker identity, and environment. If the same movies or speakers appear in both training and test, the improvements in Tables 4 and 5 could result from dataset-level correlations (e.g., a global style or speaker embedding) rather than from phone-aligned visual cues. This directly threatens the central claim, and I request a movie-disjoint split to assess generalization.
  2. [Tables 2–5] No error bars, confidence intervals, or significance tests are reported anywhere, and the test set contains only 200 samples. The text repeatedly uses terms like 'significantly' and 'substantial' (e.g., Section 3.4, 'the proposed models significantly outperform the baseline'), but these claims are unsupported. Given the small test set and the likely correlation among clips from the same movie, the observed differences (e.g., pitch MSE 0.27 vs. 0.18 in Table 4) may be within noise. Please provide variance estimates and significance tests.
  3. [Section 2.3] The proposed visual-text fusion module is never ablated against simpler alternatives. The authors motivate cross-attention over average pooling by arguing that 'local information might be lost' with a global vector (Section 2.3), yet no experiment compares the full fusion module to average-pooling or other fusion baselines. Without such an ablation, the reported gains cannot be attributed to the cross-attention mechanism; they may simply come from adding any visual conditioning signal, including a global visual vector that captures dataset-level style.
  4. [Section 3.3 vs. Section 3.4] Table 3 shows that adding visual features to the PED predictor worsens energy prediction (Text: 0.50; Text+VF-Omnivore: 0.59; Text+VF-ResNet50: 0.58), but Table 4 shows that the full TTS model improves energy prediction (FastSpeech2: 0.39; VisualSpeech Omnivore: 0.37; VisualSpeech ResNet50: 0.34). The paper does not explain this reversal. This inconsistency is important because it undermines the claim that visual features complement text for prosody prediction, and it suggests the TTS gains may arise from interactions outside the PED predictor.
  5. [Equation (1), Section 2.3] Section 2.3 claims that the fusion module enables 'bidirectional information flow between the two modalities,' but Equation (1) defines only a text-to-visual attention in which text features T serve as queries and visual features V as keys/values. No visual-to-text attention is specified. The proposed mechanism is therefore underspecified, and the bidirectional claim is inaccurate. Please either provide the full bidirectional formulation or revise the description to match the unidirectional equation.
minor comments (5)
  1. [Section 4] The statement 'we introduce VisualSpeech, the first visual text-to-speech synthesis model' is contradicted by the citation of VIT-TTS [10] in the same paper, which also uses video for TTS. Please temper this novelty claim.
  2. [Table 5 and footnote 4] The UTMOS scores (2.91, 3.06, 3.13) appear only in a footnote, while Table 5 lists only MCD and Log F0. Please integrate the UTMOS results into the table or clearly state in the main text that UTMOS was evaluated, and indicate whether higher UTMOS is better.
  3. [Section 3.4.1] The sentence 'the pitch contours generated by the two models incorporating additive visual features are closer to those produced by the text-only model' is missing a comparison to the ground truth; as written, it suggests the visual models diverge from the ground truth toward the text-only baseline, which contradicts the claimed improvement. Please revise to clarify.
  4. [Table 2 footnote] The footnote 'The results in the accepted paper contain a typographical error' is cryptic. Please specify which values were erroneous and how they were corrected in this version.
  5. [Section 3.1] The filtering criteria (pitch > 500 Hz, 2.5σ outlier removal in pitch, energy, and duration) are ad-hoc free parameters. Please justify these thresholds or provide a sensitivity analysis.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the paper's claims rest on external baselines and held-out test evaluation, not on self-referential definitions or fitted inputs.

full rationale

VisualSpeech evaluates its central claim against external, non-self-referential baselines. In Preliminary Study I, visual features are used to predict prosody and are compared with a mean predictor on a test set; the MSE values come from trained models generalizing to held-out clips, not from quantities defined by the fitted parameters. In Preliminary Study II and the TTS experiments, the proposed visual-text models are compared with the text-only FastSpeech2 baseline on the same test split, and the reported pitch, duration, energy, MCD, and log-F0 metrics are measured on generated outputs, not reconstructed from training targets. No equation defines the prediction in terms of the fitted parameter, and no load-bearing claim is justified solely by a self-citation. The reliance on the unpublished CMD2 dataset, the lack of a movie-disjoint or speaker-disjoint split, and the underspecified cross-attention mechanism are experimental validity and generalization concerns rather than circularity. The footnote about the accepted paper is self-referential but not load-bearing. Because the derivation chain remains anchored to external comparisons and independent feature extractors, no circular step can be exhibited.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on hand-chosen data-cleaning thresholds that shape the dataset, on the domain assumption that pretrained visual features encode prosodically relevant information, and on the unverified belief that cross-attention can align frame-level visual content to phone-level prosody. No new physical or mathematical entities are introduced.

free parameters (3)
  • Pitch threshold for singing removal = 500 Hz
    Speech samples with pitch values exceeding 500 Hz are discarded during dataset cleaning (Section 3.1). This hand-chosen threshold shapes the training set and therefore the measured prosody errors.
  • Outlier threshold for pitch, energy, duration statistics = 2.5 standard deviations
    Samples with pitch or duration outside 2.5 times the standard deviation are considered outliers and removed (Section 3.1). This post-hoc filtering affects the difficulty of the prosody prediction task.
  • Prosody averaging window in Preliminary Study I = t/n
    In Section 3.2, every t/n prosody features are averaged to align with each visual feature. This smoothing reduces target variance and affects the magnitude of the reported MSE; the baseline mean predictor uses the same averaged target.
assumptions (4)
  • domain assumption Softmax cross-attention reliably maps text queries to relevant visual frames
    The fusion module in Section 2.3 assumes that phone-level text features as queries can retrieve useful phone-aligned visual cues from frame-level visual features, which is not verified by any alignment or attention visualization.
  • domain assumption Omnivore and ResNet50 features are sufficient representations of video context for prosody
    The paper relies on these pretrained feature extractors to encode facial expression, gesture, and environment, but does not analyze which visual attributes carry the prosodic signal.
  • domain assumption CMD2 after vocal separation, denoising, and filtering is a valid proxy for clean expressive speech
    The preprocessing pipeline (MVSEP, Resemble-Enhance, outlier removal) may introduce artifacts or biases; the paper shows objective quality improvements but no listening test on the cleaned data alone.
  • standard math Standard backpropagation and MSE loss are accepted learning procedures
    All models are trained with Adam and MSE losses following FastSpeech2; no new mathematical principles are introduced.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VisualSpeech: Enhancing Prosody Modeling in TTS Using Video." pith.science (2026). https://pith.science/paper/2TNAM7YS

@misc{pith2026250119258,
  author       = {Pith},
  title        = {Pith review of: VisualSpeech: Enhancing Prosody Modeling in TTS Using Video},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2TNAM7YS}},
  note         = {Machine review of arXiv:2501.19258}
}
read the original abstract

Text-to-Speech (TTS) synthesis faces the inherent challenge of producing multiple speech outputs with varying prosody given a single text input. While previous research has addressed this by predicting prosodic information from both text and speech, additional contextual information, such as video, remains under-utilized despite being available in many applications. This paper investigates the potential of integrating visual context to enhance prosody prediction. We propose a novel model, VisualSpeech, which incorporates visual and textual information for improving prosody generation in TTS. Empirical results indicate that incorporating visual features improves prosodic modeling, enhancing the expressiveness of the synthesized speech. Audio samples are available at https://ariameetgit.github.io/VISUALSPEECH-SAMPLES/.

Figures

Figures reproduced from arXiv: 2501.19258 by the authors.

Figure 1
Figure 1. The proposed multi-modal TTS model: VisualSpeech. are correlated to the text. The proposed visual-text fusion mod￾ule is illustrated in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The Pitch Contour [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The Energy Contour by the text-only model. A similar trend is observed in energy prediction, as illustrated in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 25 canonical work pages

  1. [1]

    Despite the high-quality output, the prosody of the generated speech sometimes be- comes inappropriate for the context

    Introduction With advancements in deep learning [1, 2, 3], text-to-speech (TTS) [4, 5] models have become capable of generating speech that closely mimics human speech. Despite the high-quality output, the prosody of the generated speech sometimes be- comes inappropriate for the context. For example, most TTS approaches would fail to convey a different de...

    work page

  2. [2]

    VisualSpeech: Enhancing Prosody Modeling in TTS Using Video

    The Proposed Method 2.1. Overview This paper first verifies that visual information is predictive of prosody. It then demonstrates that the prosodic information de- rived from visual features complements that extracted from text. Lastly, the visual features are integrated into a modern TTS model, as shown in Figure 1. The architecture of the proposed mode...

    work page Pith review arXiv 2025

  3. [3]

    Experimental Setup To assess the impact of visual information on speech generation performance, a dataset containing both diverse prosodic and vi- sual data is essential

    Experiments 3.1. Experimental Setup To assess the impact of visual information on speech generation performance, a dataset containing both diverse prosodic and vi- sual data is essential. However, research exploring the role of visual features in speech synthesis from a prosody perspective is still scarce, and existing datasets are limited in scope. While...

    work page

  4. [4]

    Using two dis- tinct video feature extractors, we demonstrate that these visual features encapsulate prosodic information

    Conclusion In this work, we introduce VisualSpeech, the first visual text-to- speech synthesis model that incorporates visual features from corresponding videos to complement text features, significantly enhancing the prosody of the generated speech. Using two dis- tinct video feature extractors, we demonstrate that these visual features encapsulate proso...

    work page

  5. [5]

    FastDiff: A fast conditional diffusion model for high-quality speech synthesis,

    R. Huang, M. W. Lam, J. Wang, D. Su, D. Yu, Y . Ren, and Z. Zhao, “FastDiff: A fast conditional diffusion model for high-quality speech synthesis,” arXiv preprint arXiv:2204.09934, 2022

    arXiv 2022

  6. [6]

    Deep mixture density networks for acous- tic modeling in statistical parametric speech synthesis,

    H. Zen and A. Senior, “Deep mixture density networks for acous- tic modeling in statistical parametric speech synthesis,” in 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2014, pp. 3844–3848

    work page 2014

  7. [7]

    model, 3 incorporating a visual encoder and visual-text fu- sion, to synthesize speech. Two VisualSpeech models were 3Experiments are implemented based on the open-sourced reposi- tory: https://github.com/ming024/FastSpeech2 Table 4: MSE loss of three Models (FastSpeech2, VisualSpeech with Omnivore Features, and VisualSpeech with ResNet50 Fea- tures) on T...

    work page

  8. [8]

    Unidirectional long short-term memory re- current neural network with recurrent output layer for low-latency speech synthesis,

    H. Zen and H. Sak, “Unidirectional long short-term memory re- current neural network with recurrent output layer for low-latency speech synthesis,” in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2015, pp. 4470–4474

    work page 2015

Show all 36 references
  1. [9]

    NaturalSpeech: End-to-end text-to- speech synthesis with human-level quality,

    X. Tan, J. Chen, H. Liu, J. Cong, C. Zhang, Y . Liu, X. Wang, Y . Leng, Y . Yi, L. Heet al., “NaturalSpeech: End-to-end text-to- speech synthesis with human-level quality,” IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024

    2024

  2. [10]

    FastSpeech: Fast, robust and controllable text to speech,

    Y . Ren, Y . Ruan, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y . Liu, “FastSpeech: Fast, robust and controllable text to speech,” Advances in neural information processing systems, vol. 32, 2019

    2019

  3. [11]

    On granularity of prosodic representations in expressive text-to-speech,

    M. Babia ´nski, K. Pokora, R. Shah, R. Sienkiewicz, D. Korzekwa, and V . Klimkov, “On granularity of prosodic representations in expressive text-to-speech,” in 2022 IEEE Spoken Language Tech- nology Workshop (SLT). IEEE, 2023, pp. 892–899

    2022

  4. [12]

    FastSpeech 2: Fast and high-quality end-to-end text to speech,

    Y . Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y . Liu, “FastSpeech 2: Fast and high-quality end-to-end text to speech,” arXiv preprint arXiv:2006.04558, 2020

    2006 arXiv

  5. [13]

    Chive: Varying prosody in speech synthesis with a linguistically driven dynamic hierarchical conditional variational network,

    T. Kenter, V . Wan, C.-A. Chan, R. Clark, and J. Vit, “Chive: Varying prosody in speech synthesis with a linguistically driven dynamic hierarchical conditional variational network,” in Inter- national Conference on Machine Learning . PMLR, 2019, pp. 3331–3340

    2019

  6. [14]

    Mellotron: Mul- tispeaker expressive voice synthesis by conditioning on rhythm, pitch and global style tokens,

    R. Valle, J. Li, R. Prenger, and B. Catanzaro, “Mellotron: Mul- tispeaker expressive voice synthesis by conditioning on rhythm, pitch and global style tokens,” in ICASSP 2020-2020 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, ...

    2020

  7. [15]

    VIT-TTS: visual text-to-speech with scalable diffusion transformer,

    H. Liu, R. Huang, X. Lin, W. Xu, M. Zheng, H. Chen, J. He, and Z. Zhao, “VIT-TTS: visual text-to-speech with scalable diffusion transformer,” arXiv preprint arXiv:2305.12708, 2023

    2023 arXiv

  8. [16]

    The LJ Speech Dataset,

    K. Ito and L. Johnson, “The LJ Speech Dataset,” https://keithito. com/LJ-Speech-Dataset/, 2017

    2017

  9. [17]

    LibriTTS: A corpus derived from librispeech for text- to-speech,

    H. Zen, V . Dang, R. Clark, Y . Zhang, R. J. Weiss, Y . Jia, Z. Chen, and Y . Wu, “LibriTTS: A corpus derived from librispeech for text- to-speech,” arXiv preprint arXiv:1904.02882, 2019

    1904 arXiv

  10. [18]

    Ego4d: Around the world in 3,000 hours of egocentric video,

    K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liuet al., “Ego4d: Around the world in 3,000 hours of egocentric video,” in Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 1...

    2022

  11. [19]

    Condensed movies: Story based retrieval with contextual embeddings,

    M. Bain, A. Nagrani, A. Brown, and A. Zisserman, “Condensed movies: Story based retrieval with contextual embeddings,” in Proceedings of the Asian Conference on Computer Vision , 2020

    2020

  12. [20]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in International conference on Machine Learning . PMLR, 2023, pp. 28 492–28 518

    2023

  13. [21]

    CMD+: A D.I.Y . Audiovisual Dataset for Multi- Speaker TTS,

    J. Sanders, “CMD+: A D.I.Y . Audiovisual Dataset for Multi- Speaker TTS,” 2023, unpublished undergraduate thesis, Univer- sity of Sheffield

    2023

  14. [22]

    The sound demixing challenge 2023 music demixing track,

    G. Fabbro, S. Uhlich, C.-H. Lai, W. Choi, M. Mart ´ınez-Ram´ırez, W. Liao, I. Gadelha, G. Ramos, E. Hsu, H. Rodrigues et al., “The sound demixing challenge 2023 music demixing track,” arXiv preprint arXiv:2308.06979, 2023

    2023 arXiv

  15. [23]

    Resemblyzer,

    R. AI, “Resemblyzer,” https://github.com/resemble-ai/ Resemblyzer, 2024, accessed: 2024-08-22

    2024

  16. [24]

    Audiobox: Unified au- dio generation with natural language prompts,

    A. Vyas, B. Shi, M. Le, A. Tjandra, Y .-C. Wu, B. Guo, J. Zhang, X. Zhang, R. Adkins, W. Ngan et al. , “Audiobox: Unified au- dio generation with natural language prompts,” arXiv preprint arXiv:2312.15821, 2023

    2023 arXiv

  17. [25]

    A short- time objective intelligibility measure for time-frequency weighted noisy speech,

    C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, “A short- time objective intelligibility measure for time-frequency weighted noisy speech,” in 2010 IEEE International Conference on Acous- tics, Speech and Signal Processing. IEEE, 2010, pp. 4214–4217

    2010

  18. [26]

    Per- ceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs,

    A. W. Rix, J. G. Beerends, M. P. Hollier, and A. P. Hekstra, “Per- ceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs,” in 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proce...

    2001

  19. [27]

    SDR– half-baked or well done?

    J. Le Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, “SDR– half-baked or well done?” in ICASSP 2019-2019 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 626–630

    2019

  20. [28]

    Torchaudio-Squim: Reference-less speech quality and intelligibility measures in torchaudio,

    A. Kumar, K. Tan, Z. Ni, P. Manocha, X. Zhang, E. Henderson, and B. Xu, “Torchaudio-Squim: Reference-less speech quality and intelligibility measures in torchaudio,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 20...

    2023

  21. [29]

    Omnivore: A single model for many visual modalities,

    R. Girdhar, M. Singh, N. Ravi, L. Van Der Maaten, A. Joulin, and I. Misra, “Omnivore: A single model for many visual modalities,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 16 102–16 112

    2022

  22. [30]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778

    2016

  23. [31]

    Adam: A method for stochastic optimization,

    D. P. Kingma, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  24. [32]

    FastPitch: Parallel text-to-speech with pitch pre- diction,

    A. Ła ´ncucki, “FastPitch: Parallel text-to-speech with pitch pre- diction,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2021, pp. 6588–6592

    2021

  25. [33]

    Sonicvisionlm: Playing sound with vision language models,

    Z. Xie, S. Yu, Q. He, and M. Li, “Sonicvisionlm: Playing sound with vision language models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 26 866–26 875

    2024

  26. [34]

    End-to-end video-to-speech synthesis using gener- ative adversarial networks,

    R. Mira, K. V ougioukas, P. Ma, S. Petridis, B. W. Schuller, and M. Pantic, “End-to-end video-to-speech synthesis using gener- ative adversarial networks,” IEEE Transactions on Cybernetics , vol. 53, no. 6, pp. 3454–3466, 2022

    2022

  27. [35]

    Intelligible lip-to-speech synthe- sis with speech units,

    J. Choi, M. Kim, and Y . M. Ro, “Intelligible lip-to-speech synthe- sis with speech units,” arXiv preprint arXiv:2305.19603, 2023

    2023 arXiv

  28. [36]

    Camp: a two-stage approach to modelling prosody in context,

    Z. Hodari, A. Moinet, S. Karlapati, J. Lorenzo-Trueba, T. Mer- ritt, A. Joly, A. Abbas, P. Karanasou, and T. Drugman, “Camp: a two-stage approach to modelling prosody in context,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICA...

    2021

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.