{"id":"86ea42d8-5037-48b9-b08a-bf890a777a54","arxiv_id":"2606.25181","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"A wav2vec2 CTC recognizer with alignment-based error typing screens for sibilant substitutions in Polish children, reporting 88.7% sequence match and F1=0.67 on 559 utterances from 10 unseen children.","lead":"The paper describes a machine learning pipeline using wav2vec2 to screen Polish-speaking children for sibilant pronunciation substitutions via audio recordings and an explainable assistant. A generalist might read it to see how AI could extend basic speech screening to settings without easy access to specialists.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Alignment-based error typing may conflate recognizer mistakes with actual mispronunciations on the 10-child test set","rationale":"The identified concern matches the reader's weakest assumption exactly: reliability of alignment-based typing to distinguish true mispronunciations from recognizer errors on the small unseen set, absent validation details. This supports retaining UNVERDICTED pending explicit checks on label quality and error separation.","tokens_in":1763,"tokens_out":328,"duration_ms":16551,"concrete_test":"Obtain independent phonetic transcriptions of all 559 test utterances from a blinded Polish SLP; recompute precision, recall, and false-alarm rate treating expert substitution labels as ground truth. If F1 falls below 0.55 or false-alarm rate exceeds 5%, the proxy does not reliably separate model errors from clinical targets.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The screening claim (F1=0.67, 2.7% false-alarm rate) rests on the proxy that substitution-evidence bracketed tokens at target segments indicate true mispronunciations. With only 88.7% exact sequence match, the 11.3% CTC errors could be mis-typed as substitutions unless the alignment explicitly isolates model-specific failure modes from phonetic deviations. The held-out set comprises just 10 unseen children (559 utterances) with no reported details on selection criteria, inter-annotator agreement for ground-truth labels, or checks for Polish-specific acoustic biases. This leaves the distinction between true mispronunciations and recognizer errors unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript presents a screening pipeline for sibilant substitutions in Polish-speaking children that couples a wav2vec2-based CTC token recognizer with alignment-based error typing and a template-grounded explainable caregiver assistant. On a held-out test set of 559 utterances from 10 unseen children, the recognizer reports 88.7% exact sequence match; mismatches are flagged via substitution-evidence bracketed tokens at target segments to yield 72.9% precision, 61.4% recall, F1=0.67, and 2.7% false-alarm rate on target-correct items. The work emphasizes conservative screening (not diagnosis) and outlines a clinician-in-the-loop validation plan.","tokens_in":1873,"tokens_out":531,"duration_ms":22202,"significance":"If the alignment proxy reliably isolates true mispronunciations from CTC errors, the pipeline could address limited specialist access by enabling lightweight, explainable screening for Polish children. Strengths include evaluation on utterances from unseen children and explicit safety boundaries for the assistant. The conservative design (flagging only substitution-evidence tokens) supports its intended use case. The small test-set size, however, constrains claims about robustness across Polish-speaking populations.","major_comments":[{"comment":"Held-out test set (abstract and experimental results): The screening metrics (F1=0.67, 2.7% false-alarm rate) rest on ground-truth labels from only 10 children (559 utterances) with no reported selection criteria, demographic matching to training data, or inter-annotator agreement. This directly affects whether the 11.3% non-exact matches are correctly typed as substitutions rather than model errors.","section":"Experimental results / held-out test set"},{"comment":"Alignment-based error typing (abstract and methods): The proxy assumes substitution-evidence bracketed tokens at target segments indicate true mispronunciations, yet no error analysis or isolation procedure is described to show that the 11.3% CTC errors do not trigger false substitution flags on target-correct items. This is load-bearing for the reported false-alarm rate and F1 score.","section":"Error typing and proxy definition"}],"minor_comments":[{"comment":"The abstract would be strengthened by a one-sentence summary of training-set size, fine-tuning procedure, or Polish-specific acoustic adaptations to contextualize the 88.7% exact-match figure.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the thoughtful and constructive review. The comments correctly identify key limitations in the current presentation of the held-out evaluation and the error-typing proxy. We address both points below, agree that clarifications and additional analysis are warranted, and will revise the manuscript accordingly while preserving the conservative screening framing already present in the work.","responses":[{"response":"We agree that the small test-set size (10 unseen children) and lack of reported selection criteria, demographic details, and inter-annotator agreement constitute a genuine limitation that constrains generalizability claims. The 559 utterances were collected from children not seen in training to demonstrate speaker-independent behavior, but we did not include explicit selection criteria or matching information in the manuscript. The ground-truth labels were produced by a single experienced speech-language pathologist; no inter-annotator agreement was computed. We will revise the experimental-results section to (a) describe the data-collection protocol and any available demographic information, (b) explicitly state that labels come from a single annotator, and (c) reiterate the conservative screening (not diagnostic) intent together with the planned clinician-in-the-loop validation. These additions will not alter the reported numbers but will make the evidential basis clearer.","revision_made":"partial","referee_comment":"[Experimental results / held-out test set] Held-out test set (abstract and experimental results): The screening metrics (F1=0.67, 2.7% false-alarm rate) rest on ground-truth labels from only 10 children (559 utterances) with no reported selection criteria, demographic matching to training data, or inter-annotator agreement. This directly affects whether the 11.3% non-exact matches are correctly typed as substitutions rather than model errors."},{"response":"The alignment-based proxy was deliberately designed to be conservative: a flag is raised only when substitution-evidence bracketed tokens appear at the target segment, which produced the observed 2.7 % false-alarm rate on target-correct items. Nevertheless, the referee is correct that we provide no explicit error analysis separating CTC decoding mistakes from genuine mispronunciations within the 11.3 % non-exact matches. We will add a dedicated error-analysis subsection that manually inspects a sample of the mismatches, quantifies how often CTC errors alone produce substitution-evidence tokens on correct targets, and discusses the implications for the reported precision, recall, and F1. This analysis will be performed on the existing test set and reported in the revision.","revision_made":"yes","referee_comment":"[Error typing and proxy definition] Alignment-based error typing (abstract and methods): The proxy assumes substitution-evidence bracketed tokens at target segments indicate true mispronunciations, yet no error analysis or isolation procedure is described to show that the 11.3% CTC errors do not trigger false substitution flags on target-correct items. This is load-bearing for the reported false-alarm rate and F1 score."}],"tokens_in":1428,"tokens_out":633,"duration_ms":26059,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is that this paper takes an off-the-shelf wav2vec2 CTC model, applies it to screening sibilant substitutions in Polish-speaking children, and reports performance on a small held-out set. It adds an explainable assistant component for caregivers.\n\nWhat stands out as new is the language-specific application to Polish and the integration of alignment-based error typing with a template-grounded assistant. The work stays focused on screening rather than claiming diagnosis, which is appropriate.\n\nIt does a few things well. The held-out evaluation uses 559 utterances from 10 unseen children and gives clear numbers: 88.7 percent exact sequence match for the recognizer. For the screening proxy they achieve 72.9 percent precision, 61.4 percent recall, F1 of 0.67, and a low 2.7 percent false-alarm rate on correct items. They also describe safety boundaries for the assistant and sketch a clinician-in-the-loop plan.\n\nThe soft spots are in the validation details. The test set is limited to 10 children with no reported selection criteria or inter-annotator agreement. The central screening claim depends on the alignment correctly identifying true mispronunciations rather than the model's own errors, but the abstract provides little on how that separation is validated or whether Polish-specific acoustics introduce biases. No baseline systems or detailed error analysis are mentioned. If the full paper fills these gaps with reproducible methods, the results would be more convincing.\n\nThis paper is for researchers working on speech recognition applications in clinical or educational settings for under-resourced languages. A reader interested in practical screening tools might find the pipeline and metrics worth examining, but it is not aimed at broad theoretical advances.\n\nI recommend sending it for peer review. The concrete metrics and narrow scope make it suitable for referees to assess the technical claims and suggest improvements on the error-typing validation.","headline":"This applies an existing wav2vec2 CTC model to Polish sibilant screening and reports held-out metrics on a small set of 10 children, but the alignment-based error typing needs more checks to separate model mistakes from real mispronunciations.","tokens_in":2429,"tokens_out":479,"would_cite":false,"duration_ms":26389,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A wav2vec2 CTC recognizer screens sibilant substitutions in Polish children at 88.7 percent exact sequence match.","keywords":["mispronunciation screening","Polish speech","sibilant substitutions","wav2vec2","CTC recognizer","child speech disorders","explainable assistant","speech sound screening"],"falsifier":"Independent phonetic transcription by multiple speech therapists on a new set of at least 500 utterances from 20 additional Polish-speaking children, checking whether precision falls below 60 percent or false-alarm rate rises above 5 percent.","tokens_in":2651,"feed_emoji":"🗣️","tokens_out":735,"duration_ms":18058,"temperature":0.7,"pith_summary":"The paper develops a screening pipeline for speech sound errors in Polish-speaking children that targets sibilant substitutions and runs outside specialist clinics. It pairs a wav2vec2-based CTC token recognizer with alignment-based error typing and an explainable caregiver assistant that flags mismatches via bracketed substitution-evidence tokens. On 559 utterances from ten held-out children the recognizer reaches 88.7 percent exact sequence match. The conservative screening rule then delivers 72.9 percent precision, 61.4 percent recall, and a 2.7 percent false-alarm rate on correct targets. The work positions the system as a first-stage filter that defers final decisions to clinicians.","feed_headline":"Recognizer flags Polish children's sibilant errors at 72.9% precision","feed_subtitle":"Wav2vec2 CTC model reaches 88.7% sequence match on 559 utterances and keeps false alarms at 2.7% for conservative screening.","key_machinery":"wav2vec2-based CTC token recognizer coupled with alignment-based error typing that emits substitution-evidence bracketed tokens to flag mismatches at target segments.","core_discovery":"The recognizer achieves 88.7 percent exact sequence match on 559 utterances from ten unseen children; when a mismatch is declared only on emission of substitution-evidence bracketed tokens at the target segment, the resulting screen yields 72.9 percent precision, 61.4 percent recall, F1 of 0.67, and 2.7 percent false alarms on target-correct items, supporting use for conservative screening rather than diagnosis.","pith_inferences":["Extending the same token-bracketing method to other consonant classes or languages could test whether the low false-alarm property holds beyond sibilants.","Embedding the assistant in a mobile app would allow population-level data collection on prevalence of specific substitutions in Polish preschoolers.","Pairing the screen with short parent questionnaires could further reduce false alarms while preserving the conservative design."],"forward_implications":["The pipeline supplies a lightweight, non-diagnostic filter that can run on consumer devices for early identification of sibilant errors.","Safety boundaries built into the caregiver assistant limit over-interpretation and route uncertain cases to clinicians.","A clinician-in-the-loop validation plan is required before any deployment beyond the reported test set.","The approach is language-specific to Polish sibilant contrasts and is not claimed to generalize without retraining."],"fun_headline_variants":["Recognizer achieves 88.7% sequence match on 559 Polish utterances","72.9% precision for sibilant error screening in Polish children","61.4% recall with 2.7% false alarms in Polish speech screening","F1 of 0.67 for conservative Polish mispronunciation screening"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Alignment-based error typing and substitution-evidence tokens distinguish true mispronunciations from recognizer mistakes on unseen children without population-specific biases or post-hoc fixes in the ten-child test set.","fun_headline_variants_meta":{"raw":{"variants":["Recognizer achieves 88.7% sequence match on 559 Polish utterances","72.9% precision for sibilant error screening in Polish children","61.4% recall with 2.7% false alarms in Polish speech screening","F1 of 0.67 for conservative Polish mispronunciation screening"]},"model":"grok-4.3","cost_usd":0.006329,"raw_usage":{"total_tokens":2880,"prompt_tokens":643,"num_sources_used":0,"completion_tokens":80,"cost_in_usd_ticks":63290500,"prompt_tokens_details":{"text_tokens":643,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2157,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":643,"tokens_out":80,"duration_ms":20177,"temperature":1.0,"reasoning_tokens":2157,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-25T21:52:54.689383+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Independent phonetic transcription by multiple speech therapists on a new set of at least 500 utterances from 20 additional Polish-speaking children, checking whether precision falls below 60 percent or false-alarm rate rises above 5 percent.","supporting_citations":[],"review_version":1}