{"id":"37cd99c4-68d8-4dc0-8dba-1d19fc7ffab2","arxiv_id":"2412.13387","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Multimodal pre-training across EMA, MRI, and EMG articulatory streams cuts MRI-to-speech word error rate from 69.5% to 33.4% in a single-speaker low-resource setting, with similar gains for EMG-to-speech.","lead":"This paper shows that pre-training a speech synthesizer on several articulatory signals (MRI, EMG, and EMA) at once, then fine-tuning on one signal, makes the resulting speech much more intelligible. In the MRI-to-speech test, the word error rate fell from 69.5% to 33.4% compared to the earlier model, a gain that could help build silent speech interfaces from minutes of data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The auxiliary EMA used for MRI and EMG pre-training is estimated from the same audio that produces the training target; without a no-estimated-EMA ablation, the claimed multimodal benefit may be an artifact of audio leakage or the large EMA dataset.","rationale":"The reader identifies the estimated-EMA proxy as the weakest assumption, and this is indeed the most load-bearing concern. The paper's within-architecture ablations (unimodal vs. bi-modal vs. tri-modal) give some support to the multimodal pre-training idea, but they do not separate the effect of the estimated EMA from the effect of adding the large real EMA dataset. The paper explicitly says in Section 4.2 that it studies whether including estimated EMA improves performance, yet Table 3 does not present the without-estimated-EMA condition, so the reader cannot determine whether the large WER drop comes from the estimated EMA on MRI datapoints or from pre-training on 7.9 hours of real EMA data. Additionally, the estimated EMA is not validated on the MRI or EMG audio domains; the reported correlation in Table 2 is on HPRC frames, so the fidelity of the auxiliary modality in the actual pre-training data is unknown. The proposed ablation test would settle whether the estimated-EMA channel is necessary. The baseline comparison to [4] is also confounded, but the core claim of multimodal pre-training is primarily supported by the within-architecture comparisons, so the EMA-proxy issue is the central one. Since the reader already recommends CONDITIONAL, my read does not change the verdict; the paper needs a revision that includes the missing ablation and domain validation of the estimated EMA.","tokens_in":9330,"tokens_out":8391,"duration_ms":74033,"concrete_test":"Re-run the MRI-to-speech bi-modal pre-training in three configurations: (1) current method with estimated EMA on MRI datapoints; (2) no estimated EMA, i.e., MRI datapoints have EMA=0 during pre-training while the model still trains on the real EMA dataset; and (3) estimated EMA replaced by frame-permuted or Gaussian noise on MRI datapoints, keeping all hyperparameters fixed. If configurations (2) or (3) retain the same WER improvement over the unimodal baseline (50.9%), the benefit is not due to the estimated-EMA channel; if the improvement disappears, the reported multimodal gain depends on audio-derived EMA and the articulatory interpretation fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that multimodal articulatory pre-training improves MRI- and EMG-to-speech depends on EMA being a genuine auxiliary articulatory modality. However, because no paired MRI-EMA or EMG-EMA data exist, the paper synthesizes EMA from the ground-truth waveforms using a linear WavLM-to-EMA regressor trained on 2000 HPRC frames (Section 4.1.2). This introduces a two-part problem. First, the estimated EMA is unvalidated in the exact domains where it is used: the correlation of 0.702 in Table 2 is computed on HPRC held-out frames, not on the noisy MRI audio or on the EMG audio, so the fidelity of the pseudo-EMA in the pre-training data is unknown. Second, because the estimated EMA is a deterministic function of the same waveform used to extract the HuBERT-Soft target, the pre-training objective L1(x, y) = |e(x) - y| can be minimized by copying audio-derived information through the EMA branch, without learning a genuine articulatory-to-acoustic mapping. The paper does not report an ablation that removes the estimated-EMA channel from MRI/EMG pre-training while still pre-training on the real EMA dataset, so the large WER reductions in Table 3 (50.9% unimodal to 39.2% bi-modal, and 33.4% with deep feature loss) cannot be attributed specifically to multimodal articulatory alignment; they could stem from the addition of a 7.9-hour EMA dataset as extra data or from leakage of target-speech information through the estimated EMA. Without this ablation, the paper's core interpretation is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multimodal pre-training framework for articulatory-to-speech synthesis. A shared encoder with per-modality convolutional encoders and a modality-invariant pooling layer is pre-trained to predict HuBERT-Soft features from EMA, MRI, and EMG inputs, then fine-tuned on a single low-resource articulatory modality; the decoder is a HiFi-CAR vocoder trained on VCTK. For the MRI and EMG tasks, the authors use a WavLM-based linear regressor to estimate EMA features from ground-truth waveforms, since no paired MRI-EMA or EMG-EMA data exist. The paper reports large WER reductions over a prior MRI-to-speech model and consistent improvements of multimodal over unimodal pre-training on multiple objective and subjective metrics.","tokens_in":9663,"tokens_out":10941,"duration_ms":99137,"significance":"The within-architecture unimodal-versus-multimodal ablation is a genuine experimental strength, and the missing-modality zeroing mechanism is a clean way to combine unimodal articulatory corpora. If the pre-training benefit is real, the recipe is practically useful for low-resource articulatory synthesis, transferring 7.9 hours of EMA data to 11-minute MRI and 3.9-hour EMG tasks. The paper also evaluates with three objective metrics plus MOS and is transparent about dataset splits. The unresolved issue is whether the audio-estimated EMA channel provides genuine articulatory alignment or leaks target-speech information; until that is controlled, the central claim is not fully established.","major_comments":[{"comment":"The auxiliary EMA channel used for MRI and EMG is not measured articulatory data; it is produced by a linear WavLM-to-EMA regressor trained on 2000 HPRC frames and applied to the ground-truth waveforms from which the HuBERT-Soft target y is also extracted. The L1 pre-training objective L1(x,y)=|e(x)-y| can therefore be minimized by copying audio-derived information through the estimated-EMA branch, rather than by learning a genuine articulatory-to-acoustic mapping. The within-architecture unimodal rows are the only conditions without this audio-derived channel, so the reported gains (MRI WER 50.9 to 33.4; EMG WER 24.4 to 19.6) conflate multimodal pre-training with target leakage and with the added EMA pre-training corpus. Section 4.2 says the effect of estimated EMA was studied, but Tables 3/4 do not report those variants. Please add a controlled ablation, for example zeroing the EMA branch for MRI/EMG datapoints while still pre-training on real HPRC EMA, or replacing the estimated EMA with a non-articulatory WavLM feature, and report it in the main results.","section":"Sections 4.1.2/4.1.3/4.2, Eq. (5), Tables 3/4"},{"comment":"The correlations in Table 2 do not validate the pseudo-EMA in the domains where it is used. The MRI and EMG datasets have no paired ground-truth EMA; the EMA targets for the MRI-to-EMA (0.577) and EMG-to-EMA (0.510) rows are the audio-estimated features described in Sections 4.1.2 and 4.1.3. A correlation between an articulatory feature and a regressor-derived target that itself comes from the waveform cannot support the statement that EMG and MRI share information with EMA. Please either obtain true paired targets or clearly state that these entries are consistency checks against pseudo-labels, and adjust the interpretation accordingly.","section":"Section 4.4 and Table 2"},{"comment":"The headline comparison of 33.4% WER versus 69.5% for the previous model [4] is not a controlled test of multimodal pre-training, because the baseline differs in input and output sampling rates, encoder architecture, audio enhancement, and decoder configuration. The within-architecture unimodal Encoder-Decoder row (50.9% WER) is the correct baseline for the pre-training claim and should be foregrounded. In addition, the abstract's '36% word error rate' should be stated as 36.1 absolute percentage points or as a relative WER reduction, to avoid ambiguity.","section":"Abstract, Section 4.2, Table 3"}],"minor_comments":[{"comment":"The phrase '36% word error rate' is ambiguous; Table 3 corresponds to 36.1 absolute WER percentage points (69.5% to 33.4%), which is a 51.9% relative reduction.","section":"Abstract"},{"comment":"The statement that the estimated-EMA inclusion was studied is not backed by a row or column in Tables 3 and 4; either add these variants or state explicitly that all multimodal rows use estimated EMA.","section":"Section 4.2"},{"comment":"The sentence 'Adding more modalities generally improves performance' is not supported by Table 3, where tri-modal with deep feature loss has WER 41.3 versus 33.4 for bi-modal with deep feature loss; please qualify the generalization and discuss the non-monotonicity.","section":"Section 4.5"},{"comment":"The EMG baseline labeled 'V ocoder [4]' is not literally the model from [4], since Section 4.2 specifies a HiFi-CAR with different upsample scales; rename it to something like 'HiFi-CAR EMG baseline' to avoid implying a published EMG comparison.","section":"Tables 3 and 4"}],"recommendation":"major_revision","confidential_remarks":"The core idea is promising and the within-architecture ablation is a real strength, but the estimated-EMA channel is the crux of the paper. If the authors can add a no-estimated-EMA control and reframe the [4] comparison, the paper could be suitable for publication; without those changes, the main claim is not distinguishable from target leakage or from simply adding more pre-training data. I saw no citation or novelty concerns, and the topic is within the journal's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi [Colleague],\n\nQuick take: this is a solid empirical workshop-style paper (Interspeech-length) on multimodal pre-training for articulatory-to-acoustic synthesis. The within-architecture ablations show real gains — unimodal MRI gives 50.9% WER, bi-modal gives 39.2%, tri-modal with deep feature loss gives 33.4% — and the EMG numbers improve similarly. That is a genuine result, and the simple modality-invariant fusion is clean.\n\nWhat I find new here is the recipe: pre-training an encoder on EMA+MRI+EMG, using audio-estimated EMA as a stand-in for missing paired data, then fine-tuning on the low-resource modality. The linear-correlation analysis in Table 2 is a nice sanity check, and the deep feature loss is a sensible addition.\n\nThe soft spots are real but addressable. The headline comparison to [4] (69.5% → 33.4%) changes architecture, sampling rate, feature representation, and test set all at once, so it’s not a fair comparison. The honest baseline is their own unimodal encoder-decoder, which still shows a large gain. More importantly, the estimated EMA used for MRI and EMG pre-training is a deterministic function of the same waveform that produces the HuBERT-Soft target. There is no ablation that keeps the EMA pre-training data but removes the estimated-EMA branch. Without that, the bi-modal gain could be from adding the 7.9-hour EMA dataset as extra data rather than from genuine multimodal alignment. The stress-test note lands.\n\nTwo smaller issues: the EMG feature-extractor split statement in 4.1.3 is worded backwards — if the test utterances were part of the feature-extractor training set, the EMG numbers are inflated, and that needs to be clarified. Also, the MRI test set is 12 utterances, so the WER point estimates carry variance. The \"36% WER\" phrasing should say absolute percentage points.\n\nThis is not a fatal paper. The core idea is reasonable, the in-architecture result is meaningful, and the missing ablations are easy to run. I’d send it to peer review with a request for the estimated-EMA drop control and a clearer EMG split description. A reader interested in silent-speech interfaces or low-resource articulatory synthesis will get value from it.","headline":"Useful multimodal pre-training recipe for articulatory synthesis; the in-architecture gains are real, but the headline comparison and the estimated-EMA leakage need tightening before the central claim is clean.","tokens_in":10255,"tokens_out":5056,"would_cite":true,"duration_ms":43868,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that pre-training on multiple articulatory modalities, then fine-tuning on a single scarce one, makes MRI- and EMG-to-speech synthesis substantially more intelligible.","keywords":["articulatory synthesis","multimodal pre-training","MRI-to-speech","EMG-to-speech","electromagnetic articulography","transfer learning","speech intelligibility","deep feature loss"],"falsifier":"Run the best MRI-to-speech recipe with the audio-estimated EMA features time-shuffled (or replaced by EMA features from a different utterance) and check whether the word error rate stays near 33.4% or rises toward the unimodal baseline; if it stays low, the gain is not about articulatory content. A paired dataset with concurrently recorded MRI and true EMA would settle it directly: if the benefit disappears when real EMA replaces audio-estimated EMA, the estimated features were leaking target information.","tokens_in":9101,"feed_emoji":"🔊","tokens_out":5598,"duration_ms":50490,"temperature":0.7,"pith_summary":"Articulatory-to-speech synthesis -- turning measurements of the vocal tract or articulator muscles into audible speech -- usually has far less training data than ordinary speech synthesis. The paper argues that this scarcity can be offset by pre-training a shared encoder on several articulatory modalities at once and then fine-tuning on whichever single modality is actually available. The headline result is a jump in intelligibility on the hard, low-resource MRI-to-speech task: test-set word error rate falls from 69.5% to 33.4% compared with the prior model, and similar gains appear for EMG-to-speech. If these results hold, the recipe gives practitioners a practical transfer-learning path for assistive and silent-speech technologies where recorded articulatory data is scarce. A notable part of the design is that the auxiliary EMA channel is estimated from the audio itself, so multimodal pre-training can run without paired multimodal recordings.","feed_headline":"MRI-to-speech word errors drop from 69.5% to 33.4%","feed_subtitle":"Pairing MRI or EMG with estimated articulatory data during pre-training makes synthesized speech far more intelligible.","key_machinery":"The load-bearing object is the multimodal encoder, built from unimodal per-modality encoders, a fusion layer that averages the non-zero modality encodings, and a shared transformer-convolution encoder. Because absent modalities are zeroed and the fusion only averages what is present, the same pretrained encoder can be fine-tuned with a single modality without architectural change. An optional deep feature loss pulls the unimodal encoders toward agreement on the same datapoint, and a GAN vocoder pretrained on ordinary speech turns encoder outputs into waveforms; the decoder is deliberately trained without articulatory labels so it can consume any speech corpus.","core_discovery":"The paper's central claim is that multimodal pre-training is an effective fix for the scarcity of articulatory data. The authors train a shared encoder jointly on several articulatory modalities -- electromagnetic articulography (EMA), MRI, and surface electromyography (EMG) -- then fine-tune the encoder on whichever single modality is actually available for the target speaker. On a single-speaker MRI task with under ten minutes of training speech, the best variant reaches a test-set word error rate of 33.4%, against 69.5% for the prior model, and the EMG-to-speech system also beats its unimodal baselines on intelligibility and on objective and subjective quality metrics. A distinctive part of the recipe is that the auxiliary EMA channel is not recorded for the MRI and EMG speakers at all: it is linearly estimated from the same audio, using a regression trained on a separate EMA corpus, which lets the authors run tri-modal pre-training without paired multimodal data.","pith_inferences":["Editorial inference: The same recipe should transfer to other low-resource articulatory inputs, such as ultrasound or silent EMG, whenever a linear map from that modality to EMA can be estimated from existing data.","Editorial inference: If the audio-estimated EMA channel is genuinely informative, the practical payoff is that silent-speech and assistive systems can be bootstrapped from large unlabeled speech corpora plus the smallest articulatory recordings.","Editorial inference: A direct control experiment -- replacing estimated EMA with time-shuffled or other-utterance EMA during pre-training -- would test whether the gains come from articulatory content or from the extra model capacity the multi-channel input provides.","Editorial inference: The reported linear correlations between modalities (EMA to MRI, EMG to EMA) suggest that the choice of which auxiliary modality to synthesize can be guided by how linearly predictable one articulatory representation is from another."],"forward_implications":["MRI-to-speech synthesizers trained on less than ten minutes of single-speaker data can reach a test-set word error rate of 33.4%, more than halving the 69.5% error of the prior model.","Adding more articulatory modalities during pre-training improves results across ASR word error rate, mel-cepstral distortion, SpeechBERTScore, and mean opinion score, with the largest jump from unimodal to bi-modal training.","EMG-to-speech, a task with no strong published baseline, also beats its unimodal counterpart on intelligibility and quality metrics under the same pre-training recipe.","Including audio-estimated EMA features and the deep feature loss generally improves synthesis quality, suggesting that modality alignment matters during pre-training.","The decoder can be trained on any large speech corpus, so the articulatory-data bottleneck is confined to encoder pre-training and fine-tuning."],"supporting_citations":[{"why":"Supplies the baseline MRI-to-speech model and the pretrained vocoder weights that the new model must beat.","marker":"[4]"},{"why":"Provides the linear-regression recipe for estimating EMA from self-supervised speech features, the source of the auxiliary EMA channel.","marker":"[42]"},{"why":"Supplies the intermediate-feature EMG-to-speech architecture, the EMG dataset, and the convolutional EMG feature extractor.","marker":"[8]"},{"why":"Introduces the deep feature loss for multimodal articulatory synthesis that this work adapts to align unimodal encoders.","marker":"[27]"},{"why":"Establishes the EMA-to-speech pipeline with soft speech-unit targets and the vocoder that the decoder builds on.","marker":"[3]"},{"why":"Provides the pretrained waveform-generation weights used to initialize the decoder.","marker":"[34]"},{"why":"Defines the soft speech-unit representation used as the decoder's target feature.","marker":"[40]"},{"why":"Supplies the EMA corpus used to train the audio-to-EMA regression and to pre-train the encoder with real articulatory data.","marker":"[15]"}],"fun_headline_variants":["Multimodal pre-training cuts MRI-to-speech word errors by 36%","Pre-training with estimated EMA data cuts MRI speech errors","MRI speech from 10 minutes of data improves WER to 33.4%","Multimodal pre-training turns sparse articulatory data into intelligible speech","Shared encoder pre-training halves MRI-to-speech word error rate"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that EMA features linearly estimated from the ground-truth audio are a faithful proxy for real articulator motion and do not leak the target speech; if they carry acoustic information about the utterance being synthesized, the pre-training gains could be an artifact rather than evidence about multimodal alignment.","fun_headline_variants_meta":{"raw":{"variants":["Multimodal pre-training cuts MRI-to-speech word errors by 36%","Pre-training with estimated EMA data cuts MRI speech errors","MRI speech from 10 minutes of data improves WER to 33.4%","Multimodal pre-training turns sparse articulatory data into intelligible speech","Shared encoder pre-training halves MRI-to-speech word error rate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000731,"raw_usage":{"total_tokens":3223,"prompt_tokens":845,"completion_tokens":2378,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":2283}},"tokens_in":461,"tokens_out":2378,"duration_ms":14753,"temperature":1.0,"reasoning_tokens":2283,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:11:33.301873+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the best MRI-to-speech recipe with the audio-estimated EMA features time-shuffled (or replaced by EMA features from a different utterance) and check whether the word error rate stays near 33.4% or rises toward the unimodal baseline; if it stays low, the gain is not about articulatory content. A paired dataset with concurrently recorded MRI and true EMA would settle it directly: if the benefit disappears when real EMA replaces audio-estimated EMA, the estimated features were leaking target information.","supporting_citations":[{"cited_title":"Parallel V ocalized Speech","cited_arxiv_id":null,"evidence_quote":"Supplies the baseline MRI-to-speech model and the pretrained vocoder weights that the new model must beat."},{"cited_title":"We train regression models for the fol- lowing speech features: EMA, MRI, EMG, and HuBERT-Soft [40]","cited_arxiv_id":null,"evidence_quote":"Provides the linear-regression recipe for estimating EMA from self-supervised speech features, the source of the auxiliary EMA channel."},{"cited_title":"Our MRI-to-speech synthesizer outperforms the test-set ASR WER of the previous model [4] by 36% WER, with our EMG- to-speech model similarly outperforming the baseline","cited_arxiv_id":null,"evidence_quote":"Supplies the intermediate-feature EMG-to-speech architecture, the EMG dataset, and the convolutional EMG feature extractor."},{"cited_title":"Ema2s: An end-to-end multimodal articulatory-to-speech system,","cited_arxiv_id":null,"evidence_quote":"Introduces the deep feature loss for multimodal articulatory synthesis that this work adapts to align unimodal encoders."},{"cited_title":"Like [8], the encoder and decoder are trained separately, since the former requires articulatory la- bels whereas the latter does not","cited_arxiv_id":null,"evidence_quote":"Establishes the EMA-to-speech pipeline with soft speech-unit targets and the vocoder that the decoder builds on."},{"cited_title":"Hifi-gan: High-fidelity denoising and dereverbera- tion based on speech deep features in adversarial networks,","cited_arxiv_id":null,"evidence_quote":"Provides the pretrained waveform-generation weights used to initialize the decoder."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the soft speech-unit representation used as the decoder's target feature."},{"cited_title":"Quantifying kinematic aspects of reduction in a contrasting rate production task,","cited_arxiv_id":null,"evidence_quote":"Supplies the EMA corpus used to train the audio-to-EMA regression and to pre-train the encoder with real articulatory data."}],"review_version":1}