Pith. sign in

REVIEW 4 major objections 7 minor 33 references

MRI2Speech: Speech Synthesis from Articulatory Movements Recorded by Real-time MRI

T0 review · 4 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Speech can be synthesized from real-time MRI video of the vocal tract by first decoding the MRI into text and then aligning that text in a clean voice, reaching a 15.18% WER on USC-TIMIT.

desk verdict The text-recognition result from rtMRI is a real advance; the 15.18% WER headline on synthesized speech is oversold, but the paper deserves a serious referee. read the letter →

arxiv 2412.18836 v2 pith:QHFSO7BZ submitted 2024-12-25 cs.SD cs.AIeess.AS

Neil Shah , Ayan Kashyap , Shirish Karande , Vineet Gandhi This is my paper · ORCID
classification cs.SDcs.AIeess.AS
keywords real-timeMRIspeechsynthesisarticulatory-to-acousticconversionsilentinterfaceself-supervisedaudio-visualmodeltextpredictionfromvideodurationvoice
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MRI2Speech claims that intelligible speech can be synthesized from real-time MRI video of the vocal tract by first predicting text rather than by directly mapping MRI frames to noisy acoustic features. The argument is that ground-truth audio recorded inside an MRI scanner contaminates every mel-spectrogram with scanner noise, so prior two-stage systems produce speech with poor intelligibility (WER up to 102.6%). The paper's system instead fine-tunes a multimodal self-supervised model (AV-HuBERT) to recognize text from silent rtMRI frames, uses a separate duration predictor trained on audio-text pairs to align phonemes to the video, and then feeds the text and durations into a normalizing-flow-based speech decoder. On USC-TIMIT, the predicted text reaches 10.95% CER/14.38% WER, and speech synthesized in a clean LJSpeech voice reaches 9.27% CER/15.18% WER, a large drop from the prior state of the art. If correct, this makes real-time MRI a usable signal for silent-speech interfaces and for studying how internal articulators encode speech.

What carries the argument

The load-bearing mechanism is a two-stage decomposition: a fine-tuned AV-HuBERT transformer (a self-supervised audio-visual speech model with its audio encoder removed, fine-tuned with CTC loss to emit text from silent rtMRI video), a VITS-style stochastic duration predictor (a conditional variational autoencoder with normalizing flow and monotonic alignment search, trained on noisy audio-text pairs, whose frozen weights predict per-phoneme durations for a given speaker), and a HiFi-GAN-based decoder trained on clean LJSpeech audio, into which the source speaker's durations are injected. The crucial move is that the duration predictor decouples what is said (text from MRI) from when it is said (duration from audio-text alignment) and who says it (the clean-voice decoder).

What would settle it

Rerun the synthesized-speech evaluation on the released samples with a publicly named automatic speech recognizer and with human listeners on the same USC-TIMIT test split; if either measurement moves far from the reported 15.18% WER, or fails to beat the prior system [1] by a wide margin, the central claim of intelligibility improvement would be overturned.

Watch

Extended reading notes

Core claim

The central discovery is that the correlation between rtMRI articulatory video and intelligible speech is best exploited through a text intermediate: a pre-trained audio-visual self-supervised model can be adapted to read the vocal tract from MRI sequences and output a CTC transcription, while a stochastic duration predictor learned from noisy audio and text supplies per-speaker timing. Because the acoustic decoder for the final voice is no longer trained on noisy MRI audio (the clean-voice decoder only borrows source-speaker durations), the scanner noise is bypassed rather than suppressed, and the synthesized speech is both more intelligible and voice-flexible. The paper reports this as 15.18% WER on USC-TIMIT for synthesized speech in a novel clean voice, versus 102.6% WER for the prior method, as well as credible zero-shot text prediction on unseen speakers of both USC-TIMIT and ASD1.

Load-bearing premise

The load-bearing premise is that the word error rates obtained with the paper's (unspecified) automatic speech recognizer faithfully measure the intelligibility of the synthesized speech; if that recognizer is biased toward the clean LJSpeech voice or the TIMIT vocabulary, the reported 15.18% WER may not hold for other voices or listeners.

Editorial extensions

If this is right

  • Real-time MRI video alone, without clean target audio, is sufficient to drive intelligible speech synthesis: 10.95% CER/14.38% WER on predicted text and 9.27% CER/15.18% WER on synthesized speech in a clean voice on USC-TIMIT.
  • The duration predictor makes synthesis voice-flexible: alignment learned from a noisy MRI audio source can be applied to a decoder trained on clean LJSpeech audio, producing a novel voice without re-recording the content.
  • The method generalizes to unseen speakers, with average 13.56% CER/17.06% WER on USC-TIMIT and 11.75% CER/21.07% WER on ASD1, across different languages and MRI hardware.
  • Masking the lip region lowers WER by 25.33% relative to lip-only input, indicating the model reads internal articulators (velum, tongue, glottis) rather than relying on lip movements alone.
  • The large gap from the prior state of the art (102.6% WER) implies that the earlier strategy of predicting mel-spectrograms from noisy MRI audio was the main bottleneck, not the MRI signal itself.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension the paper does not test is porting the same text-prediction stage to non-MRI articulatory sensors such as ultrasound tongue imaging or electromagnetic articulography; because the intermediate is text, only the video encoder would need re-training.
  • The masking study suggests a finer-grained articulatory atlas is possible: masking individual regions (velum, tongue dorsum, glottis) rather than the whole lip could reveal which articulators carry which phonemic cues.
  • Since the final voice is supplied by a clean text-to-speech decoder, end-to-end error is approximately text-recognition error plus alignment error, so the most direct path to further gains is improving the text recognizer or adding a language model, not replacing the decoder.
  • A fixed, named automatic speech recognizer and a small human-listening study would make the 15.18% WER comparable across future systems; the paper currently leaves the recognizer unnamed, so cross-paper comparisons are provisional.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes MRI2Speech, a two-stage approach for speech synthesis from real-time MRI (rtMRI) videos. In the first stage, a fine-tuned AV-HuBERT model predicts text from silent rtMRI frames using CTC loss. In the second stage, a VITS-based stochastic duration predictor is trained on noisy audio-text pairs to estimate speaker-specific phoneme durations. At inference, the predicted text and durations are fed to a speech decoder: for the speaker's own voice, a multi-speaker VITS is used, and for a novel clean voice, a single-speaker VITS trained on LJSpeech is used with durations from the source speaker. Experiments on USC-TIMIT and ASD1 report text-prediction CER/WER of 10.95%/14.38% and 11.14%/22.80% respectively, and a synthesized-speech WER of 15.18% on USC-TIMIT with the LJSpeech voice, which the authors claim is a 575.89% improvement over the previous SOTA (WER 102.6%). The paper also includes a masking analysis showing that internal articulators contribute more than lips to text prediction.

Significance. The text-recognition results are the strongest contribution of this paper: fine-tuning AV-HuBERT on rtMRI video yields single-digit-to-teen CERs that substantially improve over prior visual speech recognition from rtMRI, and the masking analysis provides an interesting finding about the model's reliance on non-lip articulators. The method is cleanly described and the demo page is a useful resource. However, the paper's central synthesis claim is not supported by the reported evaluation. The synthesized-speech WER is a cascade score of (a) MRI-to-text recognition, (b) duration prediction, and (c) a standard text-to-speech decoder, and the comparison to the direct acoustic-feature prediction of Otani et al. [1] is not apples-to-apples. If reframed as a silent speech interface with a TTS back-end, the contribution is meaningful; as presented, the claim of 'articulatory-to-speech synthesis' with a 575.89% improvement over SOTA overreaches the evidence.

major comments (4)
  1. [IV-B, Table IV] The WER reported in Table IV is computed on audio generated by a VITS decoder from predicted text and predicted durations; it is not a direct measure of articulatory-to-acoustic conversion. The ASR model used to compute the CER/WER is never specified. Without an oracle-text baseline (ground-truth text plus the same duration predictor and the same VITS decoder), the 15.18% WER cannot be attributed to the MRI-to-speech pipeline, and the comparison to Otani et al. [1] is not controlled because [1] uses a different synthesis paradigm and likely a different ASR. This leaves the central claim of a 575.89% improvement over SOTA unsupported.
  2. [II-C] The synthesis path feeds the phoneme sequence predicted by AV-HuBERT and the predicted durations into a VITS model trained on clean LJSpeech; the acoustic waveform is generated by a standard text-to-speech decoder. Consequently, the only MRI-dependent information entering the acoustics is the discrete phoneme identity and duration. This is a cascade of MRI-to-text followed by text-to-speech, not an articulatory-to-acoustic mapping. The paper should either reframe its contribution as a silent speech interface with a TTS back-end or provide an evaluation that isolates the articulatory contribution to the acoustic output.
  3. [IV-A, Table III] The comparison to Pandey and Arif [19] is confounded by mismatched test protocols. The authors report their own results averaged over leave-one-speaker-out folds, while the numbers for [19] are taken from the original paper and are based on only two randomly selected unseen speakers. The test splits are not the same, and no re-evaluation of [19] on the authors' splits is reported. This undermines the quantitative claim of a 65.58% CER / 59.47% WER reduction over [19].
  4. [General (Tables I, III, IV)] No error bars, confidence intervals, or significance tests are reported for any CER/WER result. Given the high variability across speakers (e.g., Table III CER ranges from 7.04% to 14.66% on ASD1), the improvement claims need variance estimates and, where appropriate, per-speaker paired significance tests (e.g., Wilcoxon signed-rank test across speakers).
minor comments (7)
  1. [Introduction] The text lists 'Electrophysiology [5], Electrophysiology [5]' with a duplicated reference; the second occurrence should be a different modality or removed.
  2. [Abstract and IV-B] The abstract states 'Our method achieves a 15.18% Word Error Rate (WER) on the USC-TIMIT MRI corpus' without specifying that this is the synthesized-speech WER (Table IV) while the text-prediction WER is 14.38% (Table I). Please clarify which result is being cited.
  3. [IV-A] The phrase 'vocal fold articulators' is used to refer to structures such as lips, tongue, velum, and glottis; these are vocal tract articulators, not the vocal folds. Please correct the terminology throughout the section and in the Table II caption.
  4. [Conclusion] The reported '575.89% improvement' uses the non-standard formula (old−new)/new; the standard relative reduction is (old−new)/old ≈ 85.2%. Please report both or clarify the definition.
  5. [II-A, Eq. (1)] Equation (1) is not precisely stated: the left-hand side is p(k|e_t), but the right-hand side sums over alignments π of the full sequence e^T. Please rewrite in standard CTC notation, e.g., p(k|x) = sum over π in β^{-1}(k) of product_t p(π_t | e_t).
  6. [IV-A] The word 'vellum' should be 'velum'.
  7. [II-C, Fig. 1] The masking experiment in Table II lacks procedural details, such as how the lip region is cropped or masked and how the mask is defined. Please specify the preprocessing steps to make the experiment reproducible.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the pipeline components are trained on external data and evaluated against ground-truth text or an external ASR, with no fitted parameter renamed as a prediction.

full rationale

The paper's derivation chain is: fine-tune AV-HuBERT on (rtMRI video, text) pairs with CTC loss; train a VITS-based stochastic duration predictor on (noisy audio, text) pairs; at inference, predict text from rtMRI, predict durations from text, and synthesize speech with a VITS decoder trained on clean LJSpeech. Each component is trained on external data (USC-TIMIT, ASD1, LJSpeech) and evaluated against ground-truth text or an external ASR benchmark. No fitted parameter is renamed as a prediction: the reported WER on synthesized speech is an end-to-end measurement of the cascade, not a quantity used to fit the model. The AV-HuBERT and VITS components are cited from external prior work (refs [23], [28]), not from the authors' own prior results, so there is no self-citation chain or imported uniqueness theorem. The reader's note about the unnamed ASR in Section IV-B and the fact that the LJSpeech decoder dominates the synthesized audio are validity concerns for the comparison to [1], but they do not make the derivation circular: the input rtMRI video still determines the predicted text and durations, and the speech output is not defined as the input. No equation in the paper equates a prediction with a training target by construction. A minor caveat is that the duration predictor is trained on the same noisy audio the paper criticizes, but that is a modeling choice, not circular reasoning. Hence no circular steps; score 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim relies on standard ML assumptions about domain transfer and evaluation metrics. No new physical entities or hand-fitted constants are introduced; the free parameters are training hyperparameters that could influence the reported WER, but they are not the kind of fitted constants that typically drive circularity concerns.

free parameters (3)
  • AV-HuBERT fine-tuning steps = 45K
    The model is fine-tuned for 45K steps, a hand-chosen stopping point; results may depend on this early stopping schedule.
  • AV-HuBERT peak learning rate = 0.001
    Peak learning rate used in the CTC fine-tuning schedule, chosen by the authors without a sweep report.
  • Frame rate conversion = 25 fps, 96x96
    rtMRI video is resized and upsampled to match AV-HuBERT input requirements; this choice could affect recognition accuracy.
assumptions (3)
  • domain assumption AV-HuBERT representations transfer to rtMRI domain
    The model pretrained on natural lip videos is fine-tuned on low-resolution, noisy rtMRI frames; the paper assumes the learned audio-visual features are general enough to transfer.
  • domain assumption VITS SDP trained on noisy audio-text pairs yields accurate phoneme durations
    Section II-B uses the SDP from a VITS model trained on noisy MRI audio; if the noise corrupts the alignment, the synthesized speech quality would degrade.
  • domain assumption ASR word error rate on synthesized speech measures intelligibility
    Section IV-B reports WER on synthesized speech without naming the ASR; the assumption is that the ASR is a fair proxy for human intelligibility.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MRI2Speech: Speech Synthesis from Articulatory Movements Recorded by Real-time MRI." pith.science (2026). https://pith.science/paper/QHFSO7BZ

@misc{pith2026241218836,
  author       = {Pith},
  title        = {Pith review of: MRI2Speech: Speech Synthesis from Articulatory Movements Recorded by Real-time MRI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QHFSO7BZ}},
  note         = {Machine review of arXiv:2412.18836}
}
abstract

Previous real-time MRI (rtMRI)-based speech synthesis models depend heavily on noisy ground-truth speech. Applying loss directly over ground truth mel-spectrograms entangles speech content with MRI noise, resulting in poor intelligibility. We introduce a novel approach that adapts the multi-modal self-supervised AV-HuBERT model for text prediction from rtMRI and incorporates a new flow-based duration predictor for speaker-specific alignment. The predicted text and durations are then used by a speech decoder to synthesize aligned speech in any novel voice. We conduct thorough experiments on two datasets and demonstrate our method's generalization ability to unseen speakers. We assess our framework's performance by masking parts of the rtMRI video to evaluate the impact of different articulators on text prediction. Our method achieves a $15.18\%$ Word Error Rate (WER) on the USC-TIMIT MRI corpus, marking a huge improvement over the current state-of-the-art. Speech samples are available at https://mri2speech.github.io/MRI2Speech/

Figures

Figures reproduced from arXiv: 2412.18836 by the authors.

Figure 1
Figure 1. System diagram of the inference process: AV-HuBERT predicts text [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Mel-spectrograms of (A) original speech, and synthesized speech [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 25 canonical work pages

  1. [1]

    Speech Synthesis from Articulatory Movements Recorded by Real-time MRI,

    Y . Otani, S. Sawada, H. Ohmura, and K. Katsurada, “Speech Synthesis from Articulatory Movements Recorded by Real-time MRI,” in Proc. INTERSPEECH 2023, 2023, pp. 127–131

    work page 2023

  2. [19]

    Silent speech and emotion recognition from vocal tract shape dynamics in real-time mri,

    L. Pandey and A. Sabbir Arif, “Silent speech and emotion recognition from vocal tract shape dynamics in real-time mri,” in ACM SIGGRAPH 2021 Posters, 2021, pp. 1–2

    work page 2021

  3. [2]

    Multi- task learning of speech recognition and speech synthesis parameters for ultrasound-based silent speech interfaces

    L. T ´oth, G. Gosztolya, T. Gr ´osz, A. Mark ´o, and T. G. Csap ´o, “Multi- task learning of speech recognition and speech synthesis parameters for ultrasound-based silent speech interfaces.” in INTERSPEECH, 2018, pp. 3172–3176

    work page 2018

  4. [3]

    Reliability of electromagnetic articulography recording during speaking sequences,

    H. Horn, G. G ¨oz, M. Bacher, M. M ¨ullauer, I. Kretschmer, and D. Axmann-Krcmar, “Reliability of electromagnetic articulography recording during speaking sequences,” European journal of orthodontics, vol. 19, no. 6, pp. 647–655, 1997. [Online]. Available: https://pubmed.ncbi.nlm.nih.gov/9458598/

    work page arXiv 1997

  5. [4]

    Direct speech reconstruction from articulatory sensor data by machine learning,

    J. A. Gonzalez, L. A. Cheah, A. M. Gomez, P. D. Green, J. M. Gilbert, S. R. Ell, R. K. Moore, and E. Holdsworth, “Direct speech reconstruction from articulatory sensor data by machine learning,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 12, pp. 2362–2374, 2017. [Online]. Available: https://ieeexplore.ieee.org/document/8114382

    work page arXiv 2017

  6. [5]

    Alterego: A personalized wearable silent speech interface,

    A. Kapur, S. Kapur, and P. Maes, “Alterego: A personalized wearable silent speech interface,” in Proceedings of the 23rd International Conference on Intelligent User Interfaces , ser. IUI ’18. New York, NY , USA: Association for Computing Machinery, 2018, p. 43–53. [Online]. Available: https://doi.org/10.1145/3172944.3172977

    arXiv 2018

  7. [6]

    Enhancement of electrolaryngeal speech by adaptive filtering,

    C. Y . Espy-Wilson, V . R. Chari, J. M. MacAuslan, C. B. Huang, and M. J. Walsh, “Enhancement of electrolaryngeal speech by adaptive filtering,” Journal of Speech, Language, and Hearing Research, vol. 41, no. 6, pp. 1253–1264, 1998. [Online]. Available: https://pubs.asha.org/doi/abs/10.1044/jslhr.4106.1253

    work page arXiv 1998

  8. [7]

    Development and evaluation of pitch adjustable electrolarynx,

    Y . Kikuchi and H. Kasuya, “Development and evaluation of pitch adjustable electrolarynx,” in Speech Prosody 2004, International Conference, 2004. [Online]. Available: https://www.isca-archive.org/ speechprosody 2004/kikuchi04 speechprosody.pdf

    work page 2004

Show all 33 references
  1. [8]

    Mobile, hands-free, silent speech texting using silentspeller,

    N. Kimura, T. Gemicioglu, J. Womack, R. Li, Y . Zhao, A. Bedri, A. Olwal, J. Rekimoto, and T. Starner, “Mobile, hands-free, silent speech texting using silentspeller,” in Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems , ser. CHI EA ’21. New...

    2021

  2. [9]

    An approach to real-time magnetic resonance imaging for speech production,

    S. Narayanan, K. Nayak, S. Lee, A. Sethy, and D. Byrd, “An approach to real-time magnetic resonance imaging for speech production,” JASA, vol. 115, no. 4, pp. 1771–1776, 2004

    2004

  3. [10]

    Seeing speech: Capturing vocal tract shaping using real-time magnetic reso- nance imaging [exploratory dsp],

    E. Bresch, Y .-C. Kim, K. Nayak, D. Byrd, and S. Narayanan, “Seeing speech: Capturing vocal tract shaping using real-time magnetic reso- nance imaging [exploratory dsp],” IEEE Signal Processing Magazine , vol. 25, no. 3, pp. 123–132, 2008

    2008

  4. [11]

    Magnetic resonance imaging in real time: advances using radial flash,

    S. Zhang, K. T. Block, and J. Frahm, “Magnetic resonance imaging in real time: advances using radial flash,” JMRI, vol. 31, no. 1, pp. 101– 109, 2010

    2010

  5. [12]

    Analysis of speech production real-time mri,

    V . Ramanarayanan, S. Tilsen, M. Proctor, J. T ¨oger, L. Goldstein, K. S. Nayak, and S. Narayanan, “Analysis of speech production real-time mri,” CSL, vol. 52, pp. 1–22, 2018

    2018

  6. [13]

    Advances in vocal tract imaging and analysis,

    A. Toutios, D. Byrd, L. Goldstein, and S. Narayanan, “Advances in vocal tract imaging and analysis,” in The Routledge handbook of phonetics . Routledge, 2019, pp. 34–50

    2019

  7. [14]

    An open-source toolbox for measuring vocal tract shape from real-time magnetic resonance images,

    M. Belyk, C. Carignan, and C. McGettigan, “An open-source toolbox for measuring vocal tract shape from real-time magnetic resonance images,” Behavior Research Methods , pp. 1–13, 2023

    2023

  8. [15]

    Speaker Dependent Articulatory-to-Acoustic Mapping Using Real-Time MRI of the V ocal Tract,

    T. G. Csap ´o, “Speaker Dependent Articulatory-to-Acoustic Mapping Using Real-Time MRI of the V ocal Tract,” in Proc. Interspeech 2020 , 2020, pp. 2722–2726

    2020

  9. [16]

    Using Transposed Convolution for Articulatory-to-Acoustic Conversion from Real-Time MRI Data,

    R. Tanji, H. Ohmura, and K. Katsurada, “Using Transposed Convolution for Articulatory-to-Acoustic Conversion from Real-Time MRI Data,” in Proc. Interspeech 2021 , 2021, pp. 3176–3180

    2021

  10. [17]

    Reconstructing speech from real- time articulatory mri using neural vocoders,

    Y . Yu, A. H. Shandiz, and L. T ´oth, “Reconstructing speech from real- time articulatory mri using neural vocoders,” in 2021 EUSIPCO. IEEE, 2021, pp. 945–949

    2021

  11. [18]

    Deep Speech Synthesis from MRI-Based Articulatory Representations,

    P. Wu, T. Li, Y . Lu, Y . Zhang, J. Lian, A. W. Black, L. Goldstein, S. Watanabe, and G. K. Anumanchipalli, “Deep Speech Synthesis from MRI-Based Articulatory Representations,” in Proc. INTERSPEECH 2023, 2023, pp. 5132–5136

    2023

  12. [20]

    Real- time magnetic resonance imaging and electromagnetic articulography database for speech production research (tc),

    S. Narayanan, A. Toutios, V . Ramanarayanan, A. Lammert, J. Kim, S. Lee, K. Nayak, Y .-C. Kim, Y . Zhu, L. Goldstein et al. , “Real- time magnetic resonance imaging and electromagnetic articulography database for speech production research (tc),” The Journal of the Acoustical ...

    2014

  13. [21]

    Multimodal dataset of real-time 2d and static 3d mri of healthy french speakers,

    K. Isaieva, Y . Laprie, J. Lecl `ere, I. K. Douros, J. Felblinger, and P.- A. Vuissoz, “Multimodal dataset of real-time 2d and static 3d mri of healthy french speakers,” Scientific Data, vol. 8, no. 1, p. 258, 2021

    2021

  14. [22]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Trans. Audio, Speech and Lang. Proc. , vol. 29, p. 3451–3460, oct 2021. [Online]. Availab...

    2021

  15. [23]

    Learning audio- visual speech representation by masked multimodal cluster prediction,

    B. Shi, W.-N. Hsu, K. Lakhotia, and A. Mohamed, “Learning audio- visual speech representation by masked multimodal cluster prediction,” arXiv preprint arXiv:2201.02184 , 2022

    2022 arXiv

  16. [24]

    Lrs3-ted: a large-scale dataset for visual speech recognition,

    T. Afouras, J. S. Chung, and A. Zisserman, “Lrs3-ted: a large-scale dataset for visual speech recognition,” arXiv preprint arXiv:1809.00496, 2018

    2018 arXiv

  17. [25]

    Connection- ist temporal classification: labelling unsegmented sequence data with recurrent neural networks,

    A. Graves, S. Fern ´andez, F. Gomez, and J. Schmidhuber, “Connection- ist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in Proceedings of the 23rd international conference on Machine learning , 2006, pp. 369–376

    2006

  18. [26]

    Cmu pronouncing dictionary,

    “Cmu pronouncing dictionary,” http://www.speech.cs.cmu.edu/cgi-bin/ cmudict, accessed: 06/12/2023

    2023

  19. [27]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 , 2014

    2014 arXiv

  20. [28]

    Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,

    J. Kim, J. Kong, and J. Son, “Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,” in ICML. PMLR, 2021, pp. 5530–5540

    2021

  21. [29]

    The lj speech dataset,

    K. Ito and L. Johnson, “The lj speech dataset,” https://keithito.com/ LJ-Speech-Dataset/, 2017

    2017

  22. [30]

    A multichannel articulatory speech database and its appli- cation for automatic speech recognition,

    A. Wrench, “A multichannel articulatory speech database and its appli- cation for automatic speech recognition,” in Proc. seminar on speech production: models and data , 2000

    2000

  23. [31]

    Synchronized and noise-robust audio recordings during realtime magnetic resonance imaging scans,

    E. Bresch, J. Nielsen, K. Nayak, and S. Narayanan, “Synchronized and noise-robust audio recordings during realtime magnetic resonance imaging scans,” JASA, vol. 120, no. 4, pp. 1791–1794, 2006

    2006

  24. [32]

    Automatic segmentation of vocal tract articulators in real- time magnetic resonance imaging,

    V . Ribeiro, K. Isaieva, J. Leclere, J. Felblinger, P.-A. Vuissoz, and Y . Laprie, “Automatic segmentation of vocal tract articulators in real- time magnetic resonance imaging,” Computer Methods and Programs in Biomedicine, p. 107907, 2023

    2023

  25. [2021]

    Available: https://doi.org/10.1145/3411763.3451552

    [Online]. Available: https://doi.org/10.1145/3411763.3451552

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.