{"id":"1a2ebf45-2c01-4868-8b90-ddf38bebb717","arxiv_id":"2412.08283","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"FastSpeech2 prosody embeddings, particularly energy embeddings extracted with the speech signal, improve automatic word and syllable prominence detection over heuristics and Wav2Vec2 in a preliminary study.","lead":"This paper tests whether prosody embeddings from a text-to-speech model (FastSpeech2) can detect stressed words and syllables in native and non-native English. It reports accuracy gains over heuristic features and Wav2Vec2 representations, especially when the embeddings are extracted from the actual speech signal.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The speech-plus-text gains likely stem from fine-tuning the variance adaptor on the same utterances used for classifier evaluation; the paper never reports a disjoint split.","rationale":"The reader's weakest_assumption identifies exactly the same concern: the speech-plus-text pipeline uses ground-truth prosodic values from the same audio and is fine-tuned without any stated separation from the classification split. This is the single most load-bearing issue because the paper's strongest claim is about the advantage of speech-plus-text embeddings, and both the 100% native accuracies and the non-native relative improvements depend on this protocol. If the adapter is fine-tuned on the test utterances, then the embeddings are not a fixed representation but a model fitted to the evaluation instances, making the comparison against frozen baselines (Wav2Vec2) and heuristic features invalid. The proposed concrete test directly settles this by forcing a disjoint split. I note a secondary issue: the abstract's relative improvements (13.7% and 5.9% for word and syllable vs heuristics; 16.2% and 6.9% vs Wav2Vec2) do not straightforwardly match Table I's values, but this is a reporting inconsistency, not a challenge to the core mechanism. The text-only results are unaffected by the leakage concern and provide partial support for the paper's general thesis, so the appropriate verdict remains conditional pending the split-check, matching the reader's CONDITIONAL verdict.","tokens_in":8800,"tokens_out":3055,"duration_ms":32271,"concrete_test":"Re-run the non-native and native speech-plus-text experiments with a strict split: (1) partition the utterance set into adapter-fine-tuning and classifier train/test subsets, with no overlap between adapter fine-tuning and classifier test; (2) fine-tune the variance adaptor only on the classifier training subset (or a separate validation set); (3) freeze the adapted model, extract embeddings for the classifier test subset, and re-evaluate DNN and K-Means accuracies. If the speech-plus-text gains over text-only and over the Wav2Vec2/heuristic baselines largely disappear or drop substantially, the reported results are due to leakage rather than to the embeddings' representational quality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim is that FastSpeech2 prosody embeddings, especially energy in the speech-plus-text mode, outperform heuristic features and Wav2Vec-2.0 for prominence detection. For that claim to hold as a feature-representation comparison, the TTS variance adaptor must be a fixed or properly validated feature extractor. Section IV-B states that in the speech-plus-text case the adaptor is run in training mode and fine-tuned for multiple epochs on the LJSpeech-initialized model, using the ground-truth duration, pitch, and energy of the input utterances. The paper never states whether the utterances used for this fine-tuning are disjoint from the utterances used to train and test the downstream DNN/K-Means classifiers. If the same utterances appear in both the adaptor fine-tuning and the classifier test set, the adaptor has been fit to the test inputs, so the embeddings are transductively optimized for those instances. This can inflate accuracy substantially; indeed the native speech-plus-text energy case reaches exactly 100% for both K-Means and DNN, which is suspiciously perfect and consistent with overfitting. The text-only results, where the adaptor is frozen in inference mode, are not subject to this leakage and provide a cleaner test of the TTS-embedding idea. But the central contribution in the abstract and conclusion emphasizes the speech-plus-text improvements, so this unresolved evaluation-protocol gap is the most load-bearing concern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using FastSpeech2's variance-adaptor prosody embeddings (duration, pitch, energy) for word- and syllable-level prominence detection. Embeddings are extracted in two conditions: text-only (inference mode) and speech-plus-text (training mode, where ground-truth duration, pitch, and energy from the audio are fed into the adaptor). Experiments on Tatoeba (native) and ISLE (non-native) use PCA, distance metrics, K-Means, and a DNN classifier, with heuristics-based features and Wav2Vec-2.0 representations as baselines. The abstract reports relative accuracy improvements up to 13.7% and 5.9% over heuristics and Wav2Vec-2.0, respectively, for non-native speech.","tokens_in":9036,"tokens_out":6246,"duration_ms":69668,"significance":"If the reported results are valid, the paper would be a useful exploration of TTS prosody embeddings for a CALL-relevant task, and it contributes new manual word-level prominence annotations for Tatoeba and a subset of ISLE. The text-only condition is a relatively clean test of the representation idea and shows gains over baselines. However, the headline claims rest on the speech-plus-text condition, whose evaluation protocol is not described with enough detail to rule out transductive fitting: the variance adaptor is fine-tuned on the input utterances, and the paper never states whether those utterances are disjoint from the classifier training/test sets. The 100% accuracy for native energy speech-plus-text is consistent with fitting rather than representation quality. The contribution is therefore currently conditional on an evaluation-protocol fix.","major_comments":[{"comment":"The speech-plus-text condition is confounded by a possible transductive fine-tuning protocol. Section IV-B states that the variance adaptor is run in training mode and fine-tuned for multiple epochs on the LJSpeech-initialized model using ground-truth duration, pitch, and energy from the input utterances, but the paper never states whether these utterances are disjoint from the utterances used to train and test the downstream DNN and K-Means classifiers. If the same utterances appear in both, the embeddings are fit to the evaluation instances, and the perfect 100% accuracy for native energy speech-plus-text in Table I becomes an expected artifact of fitting rather than evidence of representation quality. Please specify the exact data partition (e.g., adaptor fine-tuning nested inside the training folds only) and report all speech-plus-text results under a fully disjoint protocol.","section":"IV-B, VI-C, Table I"},{"comment":"The epoch-wise accuracy curves are not labeled as training or validation/test curves, and the text says accuracy 'shows a clear improvement' and 'surpasses the baseline' as training proceeds. If these curves are computed on the same utterances used for adaptor fine-tuning, the monotonic improvement reflects fitting to the evaluation data, not representation quality. Please report separate training and held-out test curves, state the early-stopping rule, and give the standard deviation across runs or cross-validation folds so the reader can judge whether the differences from the baselines are meaningful.","section":"VI-C, Figure 4"},{"comment":"The claimed highest relative improvements (13.7% & 5.9% and 16.2% & 6.9%) cannot be reproduced from Table I. For example, non-native energy embeddings with the DNN at word level give 89.5% accuracy versus 78.5% for heuristics (14.0% relative) and 82.0% for Wav2Vec-2.0 (9.1% relative); syllable-level numbers are similarly different from the abstract. Please specify exactly which embedding type, condition, and baseline produce each percentage, and reconcile the text with the table.","section":"Abstract, VII"},{"comment":"Even with a disjoint split, the speech-plus-text condition is not a standard feature-extraction protocol: at test time it requires ground-truth duration, pitch, and energy values from the test audio, and the adaptor is fine-tuned per corpus before the features are extracted. The paper should state clearly whether the speech-plus-text results are intended as an upper-bound analysis or as a deployable feature extractor. If the latter, the authors should explain how the ground-truth prosody values would be obtained for unseen speech in a CALL system, and how the fine-tuning is applied without per-test-utterance adaptation.","section":"IV-B, V"}],"minor_comments":[{"comment":"The introduction mentions 'a selected subset of 3000 audios' annotated for prominence, but Section III reports 7122 annotated words for Tatoeba and 2000 utterances for ISLE without stating the number of audio files; please clarify the dataset sizes consistently.","section":"III"},{"comment":"The figure caption contains the typo 'prosposed' and the model name is inconsistently capitalized as 'Fastspeech2' and 'FastSpeech2'; please standardize.","section":"I, Figure 1"},{"comment":"In the last paragraph of Section IV-A, 'whereas. in the \"speech plus text\" case' has a stray period after 'whereas'; also 'These errors would be more in the non-native speech' should be rephrased (e.g., 'these errors are more likely in non-native speech').","section":"IV-A"},{"comment":"The experimental setup does not report the number of speakers or utterances in each classification split, the class balance, or the K-Means initialization; please include these details so the accuracies can be interpreted.","section":"V, Table I"}],"recommendation":"major_revision","confidential_remarks":"The protocol gap in the speech-plus-text condition is the key issue; without a disjoint split the headline numbers cannot be trusted. The text-only results are cleaner and could support a more modest claim, but the paper currently frames the speech-plus-text improvements as the main contribution. The abstract's numerical claims also need verification against Table I. This is a preliminary analysis, and the required fixes (re-running with a proper split, reporting test curves, correcting the percentages) are within the scope of a major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is using FastSpeech2's variance-adapter embeddings (duration, pitch, energy) for word- and syllable-level prominence detection, and the explicit two-condition comparison of text-only versus speech-plus-text extraction. That is a sensible idea, and the text-only results do suggest these embeddings carry useful prosodic information for both native and non-native speech. The paper is also honest that the corpus is a preliminary, annotated subset and that syllable-level work is limited by data availability. The German-versus-Italian comparison is a nice touch, even if the explanation is speculative.\n\nThe load-bearing problem is the speech-plus-text condition. Section IV-B says the variance adaptor is run in training mode and fine-tuned for multiple epochs on the LJSpeech-initialized model, using ground-truth duration, pitch, and energy values from the input utterances. Nowhere does the paper state that the utterances used for this fine-tuning are disjoint from the utterances used to train and test the downstream classifiers. If they overlap, then the embeddings are effectively fit to the test instances, and the reported gains—including the suspiciously perfect 100% accuracy for native energy embeddings—are largely an artifact of transductive fitting rather than a demonstration of representation quality. The stress-test note is right that this is the central weakness, and it is not a minor omission: the abstract and conclusion emphasize the speech-plus-text improvements, so the main claim depends on resolving this protocol gap. The text-only condition is cleaner and largely supports the general idea, but the paper cannot present the speech-plus-text numbers as a fair feature comparison without a disjoint split.\n\nOther soft spots: no error bars or repeated runs; no majority-class baseline (given the class imbalance, especially in Tatoeba, this matters); no code or data release; and the reported relative improvements in the abstract do not obviously match the absolute numbers in Table I without a stated baseline reference point. These are fixable and should be addressed in revision.\n\nWho is this for? Researchers working on CALL prosody detection and anyone interested in repurposing TTS intermediate representations. It deserves a serious referee, because the application is new and the text-only results give a credible preliminary signal, but the revision needs to nail the evaluation protocol before the speech-plus-text findings can be trusted.","headline":"A plausible new application of TTS prosody embeddings to prominence detection, but the headline speech-plus-text results are likely inflated by fine-tuning on the same utterances used for evaluation.","tokens_in":9560,"tokens_out":566,"would_cite":false,"duration_ms":21673,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that prosody embeddings extracted from FastSpeech2's variance adapter, particularly the energy embeddings obtained when both speech and text are available, detect word- and syllable-level prominence in native and…","keywords":["FastSpeech2","prosody embeddings","prominence detection","text-to-speech","non-native speech","word stress","syllable stress"],"falsifier":"Re-run the prominence classification with a strict split in which the utterances used to fine-tune the variance adapter are held out from the training and test sets of the classifier, and also compare against a version that uses ground-truth prosody only for training the adapter, not at evaluation. If the speech-plus-text accuracy drops to the text-only or Wav2Vec-2.0 level, the claimed gains come from leakage rather than from the TTS embeddings.","tokens_in":8614,"feed_emoji":"🗣️","tokens_out":5539,"duration_ms":51519,"temperature":0.7,"pith_summary":"The paper asks whether the prosody embeddings that a modern text-to-speech system learns while predicting duration, pitch, and energy can double as features for detecting which words and syllables are prominent in speech. It compares these embeddings, extracted either from text alone or from speech plus text, against heuristic acoustic features and Wav2Vec-2.0 representations on both native English and non-native learner speech. The central finding is that the TTS embeddings separate stressed from unstressed units better, with energy embeddings from the speech-plus-text mode giving the largest gains: relative word- and syllable-level accuracy improvements of up to 16.2% and 6.9% over Wav2Vec-2.0, and 13.7% and 5.9% over heuristics. If this holds, TTS prosody embeddings are a ready-made feature source for computer-assisted language learning tools that give stress feedback to learners.","feed_headline":"TTS energy embeddings improve prominence detection by 16.2%","feed_subtitle":"Prosody embeddings from a text-to-speech model outperform heuristics and Wav2Vec2 on native and non-native English.","key_machinery":"The variance adapter of FastSpeech2, a non-autoregressive text-to-speech model, is a 2-layer 1D-convolutional network that predicts phoneme-level duration, pitch, and energy. In inference mode it produces embeddings from text alone; in training mode it takes the actual prosody values extracted from the speech, so the embeddings encode the realized prosody of the utterance. The paper averages these phoneme-level embeddings over syllables and words and feeds them to a DNN classifier and K-Means clustering for prominence detection, comparing against heuristic features and Wav2Vec-2.0 representations.","core_discovery":"The paper's claim is that the intermediate prosody embeddings of FastSpeech2, a variance-adapter text-to-speech model, carry usable prominence information that standard acoustic heuristics and self-supervised speech representations miss. When the variance adapter runs in training mode with the true duration, pitch, and energy values supplied from the audio (the speech-plus-text condition), the resulting word- and syllable-level embeddings separate stressed from unstressed categories better than the same embeddings produced from text alone, and better than the two baseline feature families. The energy embeddings are the strongest of the three, reaching perfect separation in the native speech-plus-text condition and the largest relative accuracy gains in the non-native condition. The paper also reports that German learners are detected more accurately than Italian learners, which it attributes to the closer phonological similarity of German to English.","pith_inferences":["If the fine-tuning and classification splits are not properly disjoint, the speech-plus-text gains may be inflated by the model memorizing acoustic properties of the exact utterances being classified; a clean split or a frozen variance adapter would test this.","The same variance-adapter embeddings could be probed for other prosodic labels such as focus, contrastive stress, or boundary tones, since the mechanism is not specific to binary prominence.","Because the text-only energy embeddings carry useful signal, the approach might extend to predicting prominence in languages where the TTS model has been trained but no speech is available."],"forward_implications":["Energy embeddings from the speech-plus-text mode can be used as a feature representation for automatic prominence detection in computer-assisted language learning systems, with accuracy gains over heuristic and Wav2Vec-2.0 baselines.","Because the text-only embeddings also outperform baselines for word prominence in the non-native condition, prominence detection could be run without access to the audio, relying on the text-to-speech model's predicted prosody.","The improvement of the speech-plus-text embeddings with additional fine-tuning epochs suggests that adapting the TTS model to the target speaker group can further boost prominence detection.","The German-versus-Italian gap implies that the pretrained TTS model transfers better to learner accents that are phonologically closer to the training language."],"supporting_citations":[{"why":"Supplies the variance-adapter model whose duration, pitch, and energy embeddings are the proposed features.","marker":"[18]"},{"why":"Defines the heuristic word-level prosodic features used as a baseline.","marker":"[5]"},{"why":"Defines the heuristic syllable-level acoustic and context features used as a baseline.","marker":"[8]"},{"why":"Provides the DNN classifier architecture for syllable stress that is adapted here.","marker":"[10]"},{"why":"Provides the self-supervised Wav2Vec-2.0 representation baseline for syllable stress.","marker":"[11]"},{"why":"Supplies the non-native learner speech corpus with manual transcriptions and stress annotations.","marker":"[21]"},{"why":"Supplies the native English pretrained TTS model used as the initialization for fine-tuning.","marker":"[25]"},{"why":"Provides the Wav2Vec-2.0 model whose representations are averaged as a baseline.","marker":"[26]"}],"fun_headline_variants":["TTS energy embeddings improve prominence detection by 16.2%","Prosody embeddings from TTS outperform self-supervised Wav2Vec2","Non-native prominence detection improved with TTS prosody embeddings","German learners more accurately detected than Italian learners in prominence task"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The speech-plus-text results assume that fine-tuning the variance adapter on the same recordings that are later used for classification does not leak the stress labels into the embeddings through the ground-truth duration, pitch, and energy inputs.","fun_headline_variants_meta":{"raw":{"variants":["TTS energy embeddings improve prominence detection by 16.2%","Prosody embeddings from TTS outperform self-supervised Wav2Vec2","Non-native prominence detection improved with TTS prosody embeddings","German learners more accurately detected than Italian learners in prominence task"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000497,"raw_usage":{"total_tokens":2455,"prompt_tokens":984,"completion_tokens":1471,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":1398}},"tokens_in":600,"tokens_out":1471,"duration_ms":14764,"temperature":1.0,"reasoning_tokens":1398,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:59:32.540998+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the prominence classification with a strict split in which the utterances used to fine-tune the variance adapter are held out from the training and test sets of the classifier, and also compare against a version that uses ground-truth prosody only for training the adapter, not at evaluation. If the speech-plus-text accuracy drops to the text-only or Wav2Vec-2.0 level, the claimed gains come from leakage rather than from the TTS embeddings.","supporting_citations":[{"cited_title":"Fastspeech 2: Fast and high-quality end-to-end text to speech,","cited_arxiv_id":null,"evidence_quote":"Supplies the variance-adapter model whose duration, pitch, and energy embeddings are the proposed features."},{"cited_title":"Word prominence detection using robust yet simple prosodic features,","cited_arxiv_id":null,"evidence_quote":"Defines the heuristic word-level prosodic features used as a baseline."},{"cited_title":"Comparison of automatic syllable stress detection quality with time-aligned boundaries and context dependencies,","cited_arxiv_id":null,"evidence_quote":"Defines the heuristic syllable-level acoustic and context features used as a baseline."},{"cited_title":"A compar- ison of learned representations with jointly optimized vae and dnn for syllable stress detection,","cited_arxiv_id":null,"evidence_quote":"Provides the DNN classifier architecture for syllable stress that is adapted here."},{"cited_title":"Exploring the use of self-supervised representations for automatic syllable stress detection,","cited_arxiv_id":null,"evidence_quote":"Provides the self-supervised Wav2Vec-2.0 representation baseline for syllable stress."},{"cited_title":"The isle corpus: Italian and german spoken learner’s english,","cited_arxiv_id":null,"evidence_quote":"Supplies the non-native learner speech corpus with manual transcriptions and stress annotations."},{"cited_title":"The lj speech dataset,","cited_arxiv_id":null,"evidence_quote":"Supplies the native English pretrained TTS model used as the initialization for fine-tuning."},{"cited_title":"wav2vec 2.0: a framework for self-supervised learning of speech representations,","cited_arxiv_id":null,"evidence_quote":"Provides the Wav2Vec-2.0 model whose representations are averaged as a baseline."}],"review_version":1}