{"id":"1e107dbe-9293-451c-9704-83fdafbfa831","arxiv_id":"2506.16580","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A streaming accent conversion model using an Emformer encoder and synthetic native TTS ground truth matches the quality of a non-streaming model at 0.8 seconds latency.","lead":"The paper presents a speech model that converts accented English into native-like pronunciation while it is still being spoken, with a fixed delay of about 0.8 seconds. It is claimed to be the first system that can do accent conversion in real-time streaming, with quality close to non-streaming systems.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ideal ground-truth premise is self-undermined: synthetic targets retain non-native prosody, and speaker identity score 0.83 falls below the paper's own 0.85 threshold.","rationale":"The reader's weakest_assumption correctly identifies the idealness of TTS-generated ground truth as the load-bearing premise. The paper's own numbers and prose support this concern: the synthetic ground truth has SECS 0.83 (below the stated same-speaker threshold), and the authors explicitly attribute residual non-native classifications to prosody preservation. This aligns with the ACC results and the modest WER improvement. The streaming architecture and the comparison to the non-streaming model are still plausible; the concern does not overturn the verdict, but it does justify the reader's CONDITIONAL stance. I agree with the reader's assessment and recommend no change to the verdict. The proposed test directly probes the premise by removing non-native prosody from the training target, isolating whether the leakage is the limiting factor.","tokens_in":8633,"tokens_out":5275,"duration_ms":54269,"concrete_test":"Retrain the streaming model using synthetic ground truth generated with native F0 contours (e.g., taken from temporally aligned native utterances from the same text) instead of the non-native F0 used in Equation 4, keeping everything else identical. Compare the resulting ACC and WER on the L2-ARCTIC test set with Table 2 values (ACC 16.3, WER 14.1). If ACC drops materially (toward the synthetic ground-truth level of 11.5) or WER improves, prosody leakage in the original ground truth is confirmed and the idealness premise is load-bearing; if results are statistically unchanged, the premise is not the limiting factor.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central training premise is that the native TTS generates 'ideal ground-truth' for every non-native utterance: native pronunciation with original duration, prosody, and speaker identity (Section 2.1.2, Equations 1–4). This premise fails on two counts that the paper itself supplies. First, the ground truth is generated by conditioning the prior and the HiFi-GAN decoder on F0 extracted from the non-native source audio (Equations 3–4). Non-native prosody is not a neutral carrier of speaker identity; it carries accent cues such as stress, intonation, and rhythm. The paper admits this: 'our model aims to retain the original audio’s prosody, which can sometimes reflect the speaker’s accent, it occasionally classifies the converted audio as non-native' (Section 3.4). Table 2 confirms the consequence: ACC for converted audio is 16.3%, and the synthetic ground truth itself is judged non-native 11.5% of the time. Second, the claim that speaker identity is preserved is contradicted by the synthetic ground truth's SECS of 0.83, below the paper's own 0.85 threshold for 'likely from the same speaker' (Section 3.2.3). Since the model is trained to reproduce these imperfect targets, it learns to retain accent-laden prosody and a somewhat shifted voice identity, which limits the achievable pronunciation improvement (WER only drops from 18.3 to 14.1) and weakens the headline claim of effective accent conversion. This is not a fatal flaw for the streaming architecture itself, but it means the 'ideal ground truth' assumption is not secure and the reported gains may understate what a genuinely accent-free target could provide.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a streaming accent conversion (AC) system that also aims at pronunciation improvement. The authors adapt a non-streaming AC architecture by replacing the Transformer-based ASR content encoder with an Emformer encoder, simplifying the conditional VAE into an autoencoder, and designing a chunk-based streaming inference with caching. They train a native TTS model on multi-speaker native data and use it to synthesize \"ideal ground-truth\" audio for non-native utterances, conditioning on F0 and speaker embeddings extracted from the original non-native audio. Experiments on L2-ARCTIC compare the streaming model with the non-streaming baseline and a causal streaming variant, reporting WER, accent classifier accuracy (ACC), speaker embedding cosine similarity (SECS), MOSNet, and subjective nativeness/similarity scores. The central claim is that the streaming model achieves performance comparable to the non-streaming model at about 0.8 s latency, constituting the first streaming AC system.","tokens_in":9142,"tokens_out":3882,"duration_ms":41444,"significance":"If the central claim holds, this is a useful contribution: it demonstrates a practical streaming formulation of a non-autoregressive accent converter, with an open-source streaming inference implementation and a Flex Attention-based Emformer optimization that yields substantial training speedups. The paper also gives concrete latency and real-time-factor numbers. However, the load-bearing evaluation has serious weaknesses: the test set is selected using the same ASR model that measures WER, objective results have no error bars or significance tests, subjective results come from only ten participants, and the synthetic ground truth itself shows signs of accent leakage and speaker-identity shift (ACC 11.5%, SECS 0.83). These issues prevent acceptance of the quantitative claims as stated, although the architectural contribution remains plausible and potentially valuable after revision.","major_comments":[{"comment":"The test set is selected using the authors' own competitive ASR model [37] by keeping utterances with average WER greater than 10, and the same ASR model is then used to measure WER improvement in Table 2. This creates a partial circularity and a regression-to-the-mean effect: utterances selected for high WER on this particular ASR are likely to show lower WER on re-test even without any true pronunciation improvement. The WER comparison between original (18.3) and converted (14.1) speech may therefore be inflated. Please report results on the full L2-ARCTIC test speakers/utterances, or at least on a split that was not used for selection, and additionally report WER with an independent ASR model. Confidence intervals or bootstrap estimates should accompany the Table 2 numbers.","section":"3.1 and 3.2.2"},{"comment":"The premise that the native TTS generates 'ideal ground-truth' audio is weakened by the paper's own design and results. The synthetic target is generated by conditioning the prior and HiFi-GAN decoder on F0 extracted from the non-native source audio, and the paper acknowledges that prosody can carry accent cues. Consistent with this, Table 2 shows the synthetic ground truth is judged non-native by the accent classifier 11.5% of the time, and its SECS is 0.83, below the paper's own 0.85 threshold for 'likely from the same speaker'. Because the AC model is trained to reproduce these imperfect targets, the claimed speaker-identity preservation and pronunciation-improvement ceiling are not established. Please provide direct validation of the synthetic ground truth's nativeness and speaker similarity (e.g., subjective ratings of synthetic targets against native references), or ablate the F0 conditioning to quantify how much accent prosody leaks into the target.","section":"2.1.2, Eqs. (3)-(4)"},{"comment":"The assertion that 'streaming inference is performed without any paddings, ensuring that the streaming output remains identical to full-segment processing' is not demonstrated. While the chunking and cache design for the Emformer is plausible, the WaveNet-based bottleneck and HiFi-GAN decoder also have receptive fields, and the algorithmic description in Algorithm 1 does not show how all components' caches are initialized and updated to guarantee bit-exact (or even near-exact) equivalence. Please provide either a formal argument covering all three components or an empirical comparison of streaming versus full-segment outputs (e.g., maximum absolute waveform difference or mel-spectrogram error) on a held-out set.","section":"2.2.3"},{"comment":"Table 2 reports objective metrics without error bars or significance testing, and Table 1's subjective confidence intervals overlap substantially (e.g., nativeness 3.78±0.18 for streaming vs. 3.87±0.20 for non-streaming). Given only ten participants and the small number of test speakers per accent, the claim that the streaming model 'performs comparably' to the non-streaming model is currently supported only by point estimates. Please report per-utterance or per-speaker standard deviations, confidence intervals, and, where appropriate, paired significance tests for WER, ACC, SECS, and MOSNet.","section":"Table 2"}],"minor_comments":[{"comment":"The pseudocode label reads 'Algorithm 0' but should be 'Algorithm 1'; the text in Section 2.2.3 also refers to 'Algorithm 0'.","section":"2.2.3"},{"comment":"Reference [26] cites 'EMF-former: An Efficient and Memory-Friendly Transformer', but the Emformer architecture used in the paper originates from Shi et al. (Interspeech 2021). Please cite the original Emformer work.","section":"References"},{"comment":"In Eq. (1), the MFA alignment A is computed but its role in Eq. (4) is implicit; clarify that A is used in Eq. (2) to construct cupsample, or rewrite the equations to make the dependency explicit.","section":"Eq. (1)"},{"comment":"The real-time factor is stated as 'achievable at around 0.25 on a GTX 1060 6GB' with no measurement methodology or variance; please specify how the RTF was computed and on which utterance lengths.","section":"3.3"},{"comment":"There are several typos and formatting errors: 'kikt' in the correspondence email, 'mprocess' in the Introduction, 'Y AAP T' for YAAPT, 'Machifi-ne' in reference [18], and 'T; European Commission' in the acknowledgment. Please proofread.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the streaming architecture is a plausible contribution. The main risk is not the architecture but the evaluation: the circular test-set selection and lack of error bars need to be addressed for the quantitative claims to be credible. I recommend major revision rather than rejection because the issues are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid systems paper that makes an existing non-autoregressive AC model streamable and shows no quality loss against its own non-streaming baseline. The new material is real but incremental: Emformer encoder replacing the global-attention ASR content encoder, Flex Attention to cut padding overhead, and chunked inference with cached Emformer/WaveNet/HiFi-GAN states. The causal-variant ablation makes a useful point: look-ahead matters for pronunciation. The 'first streaming AC' framing is plausible, though I'd want a prior-art check beyond StreamVC.\n\nWhere it gets soft: the 'ideal ground-truth' from native TTS is not as ideal as claimed. F0 is taken from the non-native source, and F0 carries stress, intonation, and rhythm accent cues. The paper itself admits converted audio is still classified non-native 16.3% of the time, and the synthetic ground truth itself is judged non-native 11.5% by their accent classifier. The SECS of the synthetic target is 0.83, below their own 0.85 same-speaker threshold. So the training targets retain some accent and a slightly shifted voice. That is a real limit on pronunciation improvement, not a fatal one: the streaming-vs-non-streaming comparison remains apples-to-apples because both models train on the same targets. The conclusion should soften 'ideal' to something like 'accent-reduced but not accent-free'.\n\nSecond soft spot: the test set was chosen by WER from the same ASR used to measure WER. That makes the absolute WER numbers hard to interpret and introduces selection bias; a held-out ASR should be used. Missing error bars on objective metrics is minor but annoying.\n\nThe paper is worth a serious referee. The engineering is nontrivial, the evaluation is reasonably multi-pronged, and the code for streaming inference is promised. Anyone building real-time dubbing or video-conferencing speech processing will get direct value from it. It belongs in a speech conference; I'd send it out. I wouldn't cite it myself unless I were building directly on streaming AC.","headline":"A credible streaming adaptation of the authors' own non-streaming accent conversion model with comparable quality, but the 'ideal ground truth' premise and ASR-selected test set keep it from being fully established.","tokens_in":9531,"tokens_out":3611,"would_cite":false,"duration_ms":34718,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Streaming accent conversion now matches offline quality at a stable 0.8-second latency, the paper claims, by processing speech in small chunks with no padding and training on native-TTS-generated ideal ground truth.","keywords":["streaming accent conversion","pronunciation improvement","Emformer","non-autoregressive speech synthesis","native TTS ground truth","chunked inference","latency","knowledge distillation"],"falsifier":"Take a set of non-native utterances, generate their synthetic ground-truth audio with the described native TTS, and run the accent classifier (ACC) on that synthetic audio: if the synthetic audio is labeled non-native at rates far above native speech, the 'ideal ground truth' premise fails. Separately, feed the same utterance to the model both chunk-by-chunk and as one full segment and compare the outputs sample-by-sample; any difference disproves the no-padding identical-output claim.","tokens_in":8465,"feed_emoji":"🗣️","tokens_out":3264,"duration_ms":33658,"temperature":0.7,"pith_summary":"This paper tries to establish that accent conversion and pronunciation improvement can be done in a streaming fashion without sacrificing the quality of the best offline systems. The authors modify a previous non-autoregressive accent conversion model by replacing its Transformer encoder with an Emformer, processing audio in 0.08-second chunks with a controlled look-ahead, and caching internal states so that streaming output is identical to full-segment output. They also use a native text-to-speech model to synthesize 'ideal' ground truth for non-native speech, preserving the original speaker's duration, prosody, and identity while delivering native pronunciation. If correct, this would be the first accent conversion system that works in live settings like video conferencing, with stable latency around 0.8 seconds and word error rates comparable to the non-streaming baseline.","feed_headline":"Streaming accent conversion matches offline quality at 0.8s","feed_subtitle":"A chunked Emformer model converts non-native speech in real time while preserving speaker identity and pronunciation gains.","key_machinery":"The key mechanism is the streaming inference protocol: the model consumes 0.08-second chunks, waits for ten chunks (0.8 seconds total delay) before emitting the first output chunk, and then processes one new chunk at a time while updating and reusing cached hidden states from the Emformer, WaveNet-based component, and HiFi-GAN. Because the receptive field extends 0.64 seconds into the future, the model needs the look-ahead, but the authors design the caching so that no padding is introduced, which they argue guarantees chunk-wise output equals full-segment output. The second load-bearing machinery is the native TTS ground-truth generator: a VITS model trained on a multi-speaker native dataset, conditioned on MFA upsampled phonemes, F0, and speaker embeddings, which synthesizes the training target for each non-native utterance.","core_discovery":"The central claim is that a streaming, non-autoregressive accent conversion model can perform comparably to the best non-streaming models while maintaining stable latency, making it the first AC system capable of streaming. The model achieves this by swapping a global-attention Transformer encoder for an Emformer with a finite right look-ahead of eight frames, and by running a chunk-based inference procedure where all components (Emformer, WaveNet-based bottleneck extractor, and HiFi-GAN decoder) update their caches after each chunk and never pad the input. The authors assert that because no padding is used, the streaming output is identical to what the same model would produce on the full utterance. Training relies on synthetic ground truth generated by a native TTS model (VITS) that uses Montreal Forced Aligner upsampled transcripts, F0, and speaker embeddings from the original non-native audio, so that the target retains the speaker's voice and prosody but has native pronunciation.","pith_inferences":["The identical-output claim is directly testable: feeding the same utterance chunk-by-chunk versus all-at-once should produce bit-identical audio if the caching is exact; this comparison would also catch any subtle state-handling bugs.","The same cache-based, no-padding chunking recipe could transfer to other streaming speech transformation tasks built on non-causal convolutional decoders, such as real-time voice conversion or speech enhancement.","The residual 16.3% non-native classification after conversion hints that prosody carried by the preserved F0 and speaker embedding may leak accent cues; validating the synthetic ground truth with the accent classifier would isolate whether the leak originates in the TTS target.","The dramatic failure of the causal variant (WER 33.5 versus 14.1 for the streaming model) suggests a measurable lower bound on look-ahead for pronunciation correction, which could guide a latency-quality tradeoff curve."],"forward_implications":["Live video conferencing and other real-time applications could get accent conversion and pronunciation correction with about 0.8 seconds of delay, instead of waiting for the whole utterance.","The padding-free chunking design means the same model can serve both offline and streaming deployments with identical outputs, simplifying evaluation and deployment.","Latency can be further reduced by shrinking the look-ahead receptive field, as the authors state, though the causal variant's poor performance suggests a minimum look-ahead is needed for pronunciation improvement.","The custom Flex Attention implementation cuts training time by threefold and allows an eightfold larger batch size, making the streaming model practical to train.","If the native TTS ground truth is truly accent-free, the approach can be extended to other non-native accents and languages without collecting parallel accented data."],"supporting_citations":[{"why":"The non-streaming accent conversion model this work adapts, providing the baseline architecture, loss, and training strategy.","marker":"[17]"},{"why":"VITS, the conditional variational autoencoder used as the native TTS that generates ideal ground truth.","marker":"[18]"},{"why":"Montreal Forced Aligner supplies the phoneme-level alignment used to upsample text to match audio length.","marker":"[19]"},{"why":"Emformer, the streaming-friendly Transformer variant that replaces the global-attention encoder.","marker":"[26]"},{"why":"HiFi-GAN generator and discriminator provide the waveform synthesis and the loss used in both training stages.","marker":"[21]"},{"why":"WaveNet-based bottleneck extractor with dilated convolutions, retained for streaming pronunciation correction.","marker":"[30]"},{"why":"FreeVC voice conversion generates the multi-speaker native-audio augmentation used to train the TTS and pre-train the AC model.","marker":"[25]"},{"why":"L2-ARCTIC corpus supplies the non-native English speech for fine-tuning and testing.","marker":"[36]"},{"why":"The competitive ASR model used to measure WER for pronunciation improvement and to select the test set.","marker":"[37]"}],"fun_headline_variants":["First streaming accent conversion model matches offline quality","No-padding chunked inference enables streaming accent conversion","Streaming accent conversion with stable latency matches offline quality","Streaming accent conversion: identical output to offline without padding","First streaming accent conversion uses synthetic native ground truth"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the native TTS generates ideal ground truth for every non-native utterance: native pronunciation with the original duration, prosody, and speaker identity, so the model learns a clean accent-to-native mapping rather than reproducing accent cues hidden in the synthetic target.","fun_headline_variants_meta":{"raw":{"variants":["First streaming accent conversion model matches offline quality","No-padding chunked inference enables streaming accent conversion","Streaming accent conversion with stable latency matches offline quality","Streaming accent conversion: identical output to offline without padding","First streaming accent conversion uses synthetic native ground truth"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000808,"raw_usage":{"total_tokens":3483,"prompt_tokens":816,"completion_tokens":2667,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":432,"completion_tokens_details":{"reasoning_tokens":2593}},"tokens_in":432,"tokens_out":2667,"duration_ms":20384,"temperature":1.0,"reasoning_tokens":2593,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:22:18.924145+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of non-native utterances, generate their synthetic ground-truth audio with the described native TTS, and run the accent classifier (ACC) on that synthetic audio: if the synthetic audio is labeled non-native at rates far above native speech, the 'ideal ground truth' premise fails. Separately, feed the same utterance to the model both chunk-by-chunk and as one full segment and compare the outputs sample-by-sample; any difference disproves the no-padding identical-output claim.","supporting_citations":[{"cited_title":"Vqmivc: Vector quantization and mutual information-based un- supervised speech representation disentanglement for one-shot voice conversion,","cited_arxiv_id":null,"evidence_quote":"WaveNet-based bottleneck extractor with dilated convolutions, retained for streaming pronunciation correction."},{"cited_title":"Experiments on cross-language acous- tic modeling","cited_arxiv_id":null,"evidence_quote":"The non-streaming accent conversion model this work adapts, providing the baseline architecture, loss, and training strategy."},{"cited_title":"Zero-Shot Accent Conversion using Pseudo Siamese Disentanglement Network,","cited_arxiv_id":null,"evidence_quote":"VITS, the conditional variational autoencoder used as the native TTS that generates ideal ground truth."},{"cited_title":"Our prior is conditioned on both the upsampled transcript and the F0 sequence","cited_arxiv_id":null,"evidence_quote":"Montreal Forced Aligner supplies the phoneme-level alignment used to upsample text to match audio length."},{"cited_title":"Wavlm: Large- scale self-supervised pre-training for full stack speech process- ing,","cited_arxiv_id":null,"evidence_quote":"HiFi-GAN generator and discriminator provide the waveform synthesis and the loss used in both training stages."},{"cited_title":"Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,","cited_arxiv_id":null,"evidence_quote":"FreeVC voice conversion generates the multi-speaker native-audio augmentation used to train the TTS and pre-train the AC model."},{"cited_title":"Dynamic transcription for low-latency speech translation","cited_arxiv_id":null,"evidence_quote":"L2-ARCTIC corpus supplies the non-native English speech for fine-tuning and testing."}],"review_version":2}