{"id":"3dff8ab6-d9a3-4f5d-94b2-23c3d4ddf2ad","arxiv_id":"2507.19374","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A fully automated pipeline generates audio-text pairs with grammar errors and disfluencies for spoken grammatical error correction, with four objective metrics to select the best generated data.","lead":"The paper presents an automated pipeline that turns clean text into spoken training data with grammar errors and disfluencies, using a reverse grammar-error model, disfluency insertion, and voice-cloning text-to-speech. It also introduces four metrics to judge the quality of such generated audio, and reports modest gains when using the augmented data to train spoken grammar error correction systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 4's best SGEC result confounds augmentation with fine-tuning on original data; without a no-augmentation fine-tuned control, the augmentation claim is not isolated.","rationale":"The reader's identified weakness (circular reverse-GEC sanity check) is real but secondary: even if generated errors are not perfectly representative of L2 learner errors, augmentation could still improve downstream SGEC through regularization or data diversity. The more epistemically prior concern is whether the headline end-to-end gain exists at all once the fine-tuning confound is removed. Table 4's row 4 differs from the baseline in two ways: generated audio data and fine-tuning on original data. Because no fine-tuned no-augmentation control is reported, the central claim is not isolated. The small magnitude of the raw gain and the absence of significance testing further weaken the inference, but the missing control is the concrete, fixable issue. The paper's text-only GEC results (Table 3) are better controlled because GEC vs GEC-Aug use the same pipeline, though those also lack significance testing. This concern does not invalidate the pipeline's plausibility; it means the paper should be conditional on adding the missing ablation and significance analysis. Since the reader already assigned CONDITIONAL, the verdict is unchanged.","tokens_in":8587,"tokens_out":3282,"duration_ms":34066,"concrete_test":"Retrain the end-to-end Whisper SGEC model exactly as the baseline (Train+Dev), then apply the same fine-tuning recipe used for row 4, but without any generated audio: i.e., Train+Dev -> fine-tune on Train+Dev. If this control reaches F0.5 approximately 41.93, the augmentation contribution is not evidenced. Additionally, run a paired bootstrap or per-utterance significance test (e.g., 1000 resamples) comparing row 4 vs this control and vs row 1. Also add the analogous ablation for the text-only GEC-Aug results if claiming augmentation benefit there.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that spoken augmented data improves end-to-end SGEC rests on Table 4. Row 4 (F0.5 41.93) is obtained by taking the filtered augmented-data model (row 3, 41.65) and fine-tuning it on the original Train+Dev set. The baseline row 1 (41.39) is trained only on Train+Dev and is not fine-tuned in the same way. Thus the +0.54 gain mixes two interventions: generated audio pretraining and fine-tuning on original data. Since fine-tuning on the target training distribution is itself a standard performance booster, the comparison is confounded. Row 2 shows that adding generated data without filtering and without fine-tuning actually slightly hurts (41.34 vs 41.39), so the only non-confounded augmentation comparison in the table does not support the claim. The reported improvement could be entirely due to the fine-tuning step. No significance testing or error bars are reported. This is a missing-control problem, not a theoretical flaw, and it is directly testable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a fully automated data augmentation pipeline for spoken grammatical error correction (SGEC): a reverse GEC model generates erroneous text from clean transcriptions, a disfluency insertion module adds hesitations, repetitions, false starts and incomplete sentences at original positions, and F5-TTS synthesizes audio with voice cloning. The authors introduce four objective metrics (speaker-verification cosine distance, ASR WER, BERT-based text scoring, and wav2vec2 audio scoring) to compare candidate generated datasets. They evaluate the augmented data on written GEC and on cascaded, semi-cascaded, and end-to-end SGEC systems using the Speak & Improve Corpus 2025. The reported results show consistent gains for text-only augmentation in cascaded and semi-cascaded pipelines, while the end-to-end gains in Table 4 are small and the best result confounds augmentation with fine-tuning on the original data.","tokens_in":8940,"tokens_out":4463,"duration_ms":43288,"significance":"If the result holds, the pipeline offers a valuable answer to the data scarcity problem in SGEC by reusing strong text GEC resources, TTS with voice cloning, and a public L2 speech corpus. The four objective metrics are a useful contribution in themselves, as they give a reusable, largely automated way to select between generated spoken datasets. The paper is also reasonably reproducible: the components (BART, BEA-2019, Whisper, F5-TTS, the S&I Corpus) are public, and the metric computations are described concretely. The main unresolved point is experimental control: the end-to-end SGEC claim rests on a comparison that mixes two interventions, and the only non-confounded comparison in Table 4 does not show a gain. The reverse-GEC quality check is also self-referential, so the representativeness of the generated errors remains an open risk.","major_comments":[{"comment":"The paper's central end-to-end SGEC claim is confounded. Row 4 (F0.5 41.93) is obtained by taking the model trained on Train+Dev+Gen(cos04) and fine-tuning it on Train+Dev, while the baseline row 1 (41.39) is trained on Train+Dev without an equivalent fine-tuning stage. The only non-confounded augmentation comparison, row 2 versus row 1, shows a slight degradation (41.34 vs 41.39). Therefore the sentence \"The results show that using augmented data improves performance\" in Section 4.2 is not supported by Table 4 for end-to-end models. Please add a no-augmentation control that is fine-tuned on Train+Dev with the same two-stage schedule, and report variance or significance across seeds; the current +0.54 difference is small and could be noise.","section":"Section 4.2, Table 4"},{"comment":"The sanity check that reverse GEC outputs can be corrected by the S&I 2025 GEC system is circular. Both the reverse GEC model and the GEC checker are trained on the same BEA-2019 data, so a high ERRANT F0.5 on the reverse-GEC outputs demonstrates self-consistency of the model family rather than naturalness of the generated errors. Because the usefulness of the augmented data for real L2 speech depends on the synthetic errors resembling genuine learner errors, please provide an external validation, for example human annotation of a sample of generated utterances or a comparison of error-type distributions against a held-out S&I eval subset that was not used to train the checker.","section":"Section 2.1, Table 1"}],"minor_comments":[{"comment":"The text says the metrics are computed on a subset of the S&I dev set of \"∼ 3300K utterances\"; 3.3 million utterances is implausible for a dev subset and this is likely a typo (perhaps 3.3K or 330K). Please clarify, since the size affects the reliability of the metric comparisons in Figures 3 and 4.","section":"Section 3"},{"comment":"\"We hypotheses that minimizing text edits may ease voice cloning\" should read \"We hypothesize that.\"","section":"Section 2.2"},{"comment":"Figure 4 is described as showing \"strong positive correlations,\" but no correlation coefficients are reported; please include numeric values such as Pearson r for both the BERT and wav2vec2 scatter plots.","section":"Section 3.3"},{"comment":"F5-TTS shows a much higher insertion rate (8.7) than the original (1.4), attributed to prompt words at the start of generated audio. Please clarify whether these prompt words are removed from the generated audio and text before the data are used for SGEC training, since otherwise they would be a systematic artifact in the augmented training set.","section":"Section 3.2, Table 2"},{"comment":"The comparison of TTS systems is described only as \"informally evaluated through subjective listening\"; please provide details on the number of listeners, the stimuli, and the criteria so that this comparison can be assessed and reproduced.","section":"Section 2.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and I see no citation or novelty disclosure problems. My recommendation is driven by the missing control in Table 4 and by the circularity of the reverse-GEC sanity check; both are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the reverse-GEC + disfluency insertion + F5-TTS pipeline is a sensible, genuinely useful recipe for a task that lacks training data, and the four objective metrics (speaker similarity AUC, ASR WER, BERT/wav2vec SLA score correlation) are a nice addition that goes beyond anecdotal listening tests. The paper is the first to apply this combination to the S&I Corpus, and it shows consistent improvements across cascaded, semi-cascaded, and end-to-end SGEC setups. Written GEC gains in Table 3 are more convincing than the spoken gains.\n\nThe real soft spot is Table 4. Row 4 (F0.5 41.93) mixes generated-audio pretraining with fine-tuning on the original training set, so it cannot isolate augmentation. The stress-test note is half right: row 3 (41.65) is a clean augmentation comparison and shows a small +0.26 gain over the 41.39 baseline, so the claim isn't empty. But that gain is modest, there are no error bars or significance tests across any of the runs, and the unfiltered condition (row 2) is a wash. The paper would be stronger with a fully crossed design: with and without augmentation, with and without fine-tuning, and repeated seeds.\n\nThe reverse-GEC sanity check in Table 1 is circular in the sense that the checker and the generator are trained on the same BEA-2019 data family; it shows internal consistency, not that generated errors match real learner errors. That's worth a sentence of caution but not fatal, because the downstream task evaluation is held-out. Also minor: Section 3 says the metrics are computed on a dev subset of \"~3300K utterances\" — that's presumably a typo (3.3M is far larger than the corpus), and it should be corrected.\n\nThe paper could be more open: no code, no data, no released filters. That limits reproducibility, especially for a pipeline that depends on many small choices (checkpoint selection, similarity threshold cos04, TTS prompt handling).\n\nOverall: a solid, incremental contribution to a genuinely under-resourced task. A serious referee should engage with it. My recommendation: send to peer review; expect revision with a proper ablation, significance tests, and a fix for the typo.","headline":"Useful automated augmentation pipeline for low-resource spoken GEC, worth peer review; the headline end-to-end gain is confounded by fine-tuning and the clean comparison is small.","tokens_in":9324,"tokens_out":2437,"would_cite":true,"duration_ms":22583,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fully automatic pipeline generates audio-text pairs with learner-like errors and disfluencies, and those synthetic pairs improve spoken grammatical error correction in cascaded, semi-cascaded, and end-to-end systems.","keywords":["spoken grammatical error correction","data augmentation","reverse grammatical error correction","text-to-speech","disfluency","speaker similarity filtering","objective evaluation metrics","second language assessment"],"falsifier":"Train a GEC system only on the augmented corpus and a matched system only on original annotated learner speech, then test both on a held-out set of real learner utterances containing error types not seen in the generation distribution; a clear gap would show the generated errors are not representative. A simpler check is a human Turing-style rating of whether generated errorful audio-text pairs come from real learners.","tokens_in":8427,"feed_emoji":"🗣️","tokens_out":8536,"duration_ms":77049,"temperature":0.7,"pith_summary":"Spoken grammatical error correction (SGEC) is under-resourced compared with text GEC, so this paper tries to manufacture the missing training data automatically. Its proposed pipeline first runs a text-to-error 'reverse GEC' model over clean transcriptions, then re-inserts disfluencies at positions taken from real learner speech, then synthesizes audio with a voice-cloned text-to-speech system. The paper also proposes four objective metrics to choose among candidate generated datasets without human listening. On the Speak & Improve Corpus 2025, the augmented text and audio improve cascaded, semi-cascaded, and end-to-end SGEC models, with the best end-to-end model reaching F0.5 41.93 versus 41.39 for the original-data baseline. If this holds, SGEC systems can be improved without costly manual annotation and without shifting assessment scores.","feed_headline":"41.93 F0.5: synthetic speech improves spoken grammar-error correction","feed_subtitle":"A fully automatic pipeline manufactures learner speech, lifting end-to-end grammar correction past the original-data baseline.","key_machinery":"The central mechanism is a reverse GEC model: a BART-large sequence-to-sequence model trained with input and output swapped so that clean text becomes errorful text with the same error-type distribution as the training corpus. Around this core, the pipeline inserts disfluencies at positions taken from the original aligned data, then uses F5-TTS, a flow-matching non-autoregressive TTS with zero-shot voice cloning, to synthesize audio that preserves the original speaker's voice and accent. The four proposed metrics—speaker-embedding cosine distance, ASR word error rate, a BERT-based text grader, and a wav2vec2-based audio grader—are the selection machinery that decides whether a candidate generated corpus is usable, and they identify F5-TTS as the best among the systems compared.","core_discovery":"The central claim, stated as a fair reader would state it: learner-like training data for SGEC can be synthesized rather than collected. Reversing a BART-large GEC model produces errorful text whose error-type distribution resembles the original corpus, copying disfluency positions preserves spoken disfluency patterns, and F5-TTS voice cloning preserves speaker identity; the resulting audio-text pairs are close enough to original data that they improve all three SGEC architectures tested. The decisive numbers are in Tables 3 and 4: written augmentation raises F0.5 on manual transcriptions from 52.01 to 54.94 and in every cascaded and semi-cascaded ASR condition, while spoken augmentation with cosine-distance filtering and fine-tuning pushes end-to-end F0.5 from 41.39 to 41.93. The objective metrics corroborate the choice of F5-TTS, which shows the smallest speaker-embedding distance and the closest WER to original audio among the TTS systems compared.","pith_inferences":["A stronger validation of the reverse-GEC errors is available than the paper's self-consistency check: have annotators or a GEC model trained on independent data judge whether the generated errors look like learner errors, since the current checker shares training data with the generator.","The speaker-similarity filter is used as a binary threshold; one testable extension is to weight generated samples by the continuous cosine distance instead, which may preserve more data and improve end-to-end training.","Because the pipeline separates error generation, disfluency placement, and synthesis, each module could be evaluated in isolation, for example by learning disfluency insertion from the corpus rather than copying positions, to see whether more varied disfluencies improve robustness."],"forward_implications":["Adding reverse-GEC text to the training set improves written GEC on the eval set, raising F0.5 from 52.01 to 54.94 on manual transcriptions.","In cascaded and semi-cascaded SGEC pipelines, augmented text helps across ASR configurations: Whisper-small with disfluency detection rises from F0.5 22.11 to 28.28, Whisper-large from 24.37 to 30.31, and the fluent-tuned Whisper from 35.89 to 38.72.","For end-to-end SGEC, generated F5-TTS audio alone roughly matches the original-data baseline, but filtering by speaker similarity lifts F0.5 to 41.65, and fine-tuning on original data after augmentation reaches 41.93.","The four objective metrics can rank candidate generated datasets without subjective listening, and the paper's chosen F5-TTS system is the one that scores best on them.","Because the augmented corpus preserves SLA grader scores, it can enrich learner speech data without altering the distribution of assessment outcomes."],"supporting_citations":[{"why":"Supplies the original L2 learner speech corpus with grammar-error and disfluency annotations, used for training, validation, and evaluation.","marker":"[16]"},{"why":"Provides the baseline GEC system used to sanity-check generated errors and the baseline SGEC results.","marker":"[18]"},{"why":"BART-large is the sequence-to-sequence model reversed to form the reverse GEC error generator.","marker":"[17]"},{"why":"F5-TTS is the voice-cloning text-to-speech system that synthesizes the augmented audio-text pairs.","marker":"[20]"},{"why":"ERRANT supplies the edit-based F0.5 scoring used to measure GEC quality on the eval set.","marker":"[19]"},{"why":"Defines the end-to-end Whisper SGEC model trained and fine-tuned in the spoken augmentation experiments.","marker":"[28]"},{"why":"Supplies the training recipe for the Whisper SGEC and cascaded systems used in evaluation.","marker":"[6]"},{"why":"Provides speaker embeddings for the cosine-distance similarity metric and for filtering generated audio by speaker match.","marker":"[27]"},{"why":"Supplies the text-based and audio-based assessment graders used as two of the four objective quality metrics.","marker":"[29]"}],"fun_headline_variants":["Synthetic learner speech beats original for spoken GEC","Voice-cloned errors lift grammar correction to 41.93 F0.5","Automated fake learner audio improves SGEC","Reverse BART, clone voice: SGEC data on demand","No human annotators needed: SGEC gets synthetic boost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The invented errors must be the same kind a real second-language learner would make; the paper checks this only with a corrector trained on the same data as the error generator, so the check could pass even if the generated errors are unnatural.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic learner speech beats original for spoken GEC","Voice-cloned errors lift grammar correction to 41.93 F0.5","Automated fake learner audio improves SGEC","Reverse BART, clone voice: SGEC data on demand","No human annotators needed: SGEC gets synthetic boost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000245,"raw_usage":{"total_tokens":1515,"prompt_tokens":904,"completion_tokens":611,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":528}},"tokens_in":520,"tokens_out":611,"duration_ms":6401,"temperature":1.0,"reasoning_tokens":528,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:52:34.297697+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a GEC system only on the augmented corpus and a matched system only on original annotated learner speech, then test both on a held-out set of real learner utterances containing error types not seen in the generation distribution; a clear gap would show the generated errors are not representative. A simpler check is a human Turing-style rating of whether generated errorful audio-text pairs come from real learners.","supporting_citations":[{"cited_title":"To Err Is Human, but Llamas Can Learn It Too,","cited_arxiv_id":null,"evidence_quote":"Supplies the original L2 learner speech corpus with grammar-error and disfluency annotations, used for training, validation, and evaluation."},{"cited_title":"Automatic Assessment of English as a Second Lan- guage,","cited_arxiv_id":null,"evidence_quote":"Provides the baseline GEC system used to sanity-check generated errors and the baseline SGEC results."},{"cited_title":"Artificial disfluency detection, uh no, disfluency generation for the masses,","cited_arxiv_id":null,"evidence_quote":"BART-large is the sequence-to-sequence model reversed to form the reverse GEC error generator."},{"cited_title":"SynthASR: Un- locking synthetic data for speech recognition,","cited_arxiv_id":null,"evidence_quote":"F5-TTS is the voice-cloning text-to-speech system that synthesizes the augmented audio-text pairs."},{"cited_title":"Grammatical Error Correction: A Survey of the State of the Art,","cited_arxiv_id":null,"evidence_quote":"Supplies the training recipe for the Whisper SGEC and cascaded systems used in evaluation."},{"cited_title":"V oiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild,","cited_arxiv_id":null,"evidence_quote":"Provides speaker embeddings for the cosine-distance similarity metric and for filtering generated audio by speaker match."},{"cited_title":"YourTTS: Towards zero-shot multi-speaker TTS and zero-shot voice conversion for everyone,","cited_arxiv_id":null,"evidence_quote":"Supplies the text-based and audio-based assessment graders used as two of the four objective quality metrics."}],"review_version":2}