{"id":"311de282-ee51-44e1-931f-ac949994a053","arxiv_id":"2507.20884","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"On a 12-gloss German Sign Language dataset, mouth and full-face crops improve isolated sign recognition more than eyes do, with mouth matching full face when fused with body input.","lead":"The authors asked which facial region, eyes, mouth, or full face, matters most for automatic sign language recognition, and found that mouth and face streams add the most value on top of body-only inputs. A general reader should care because the answer guides how future sign language recognition systems fuse non-manual facial cues into end-to-end models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'body' baseline is not shown to exclude the face, so the manual-vs-manual+facial contrast and the mouth-over-face equivalence may be artifacts of duplicate facial pixels.","rationale":"The paper makes a genuine, well-controlled attempt to isolate facial regions, uses two architecturally different models, and reports consistent training settings; those are real strengths. However, the central claim depends on treating the body stream as the manual-feature baseline while adding facial streams on top. If the body crop already contains the face, then the baseline is contaminated and the observed improvements from body+mouth / body+face cannot be attributed to the introduction of facial features. Moreover, the inference that the mouth is the most important facial feature rests on the statistical equivalence of body+mouth and body+face; with the face already present in the body crop, that equivalence is exactly what one would expect, because the full face stream becomes redundant while the mouth stream provides a new high-resolution view. The paper does not quantify overlap or provide a face-masked control, and the text is ambiguous enough that the reader cannot tell whether the body crop includes the face. This is therefore the most load-bearing unresolved concern. If the authors can show that the body crop excludes facial pixels, or that masking the face in the body crop preserves the mouth advantage, the main conclusion survives; otherwise the headline claim is substantially weakened. The missing signer-independent split is also important for generalization, but it is secondary because it affects external validity rather than the internal comparison between ROIs. The abstract's claim that mouth features 'significantly improving accuracy' is additionally stronger than the MViT results support, since MViT's top-1 gains are not significant; this should be softened regardless of the outcome of the overlap check.","tokens_in":10415,"tokens_out":7905,"duration_ms":103964,"concrete_test":"Select a random sample (e.g., 100 clips) of body ROIs and facial ROIs; compute the fraction of body-crop pixels that fall inside the face bounding box (or the IoU between the body ROI and the face ROI). If the overlap is non-trivial (say IoU > 5%), retrain the body-only condition twice: (a) with the face region masked/pixelated in the body crop, and (b) with the original body crop, then compare body+mouth and body+face against each masked baseline. If body+face no longer matches body+mouth once the face is removed from the body stream, the mouth-is-most-important conclusion is an artifact of baseline contamination; if the mouth advantage persists, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central comparison is 'body' vs 'body + facial ROI', and the ranking mouth ≈ face > eyes is inferred from it. Section 3 defines the body ROI as the bounding box around MediaPipe upper-body and hand landmarks, but it does not state that facial landmarks or pixels are excluded. MediaPipe's upper-body landmark set commonly includes facial landmarks (nose, eyes, ears, mouth), and an axis-aligned box around those coordinates will contain the face. If so, the baseline is not a manual-feature stream: it already contains non-manual facial pixels. Consequently, (i) the accuracy gain from body+mouth / body+face over body is not evidence that adding facial features helps, only that adding a second, higher-resolution facial crop at a different scale helps; and (ii) the key equivalence body+mouth ≈ body+face, which is used to conclude that the mouth is the most important facial region, would be explained by the full face being redundant with the face already in the body crop, while the mouth crop supplies new information. This is not a minor labeling issue: it attacks the internal validity of the main ablation. The paper provides no overlap statistics and no control condition with the face masked or pixelated in the body stream. A secondary issue is the absence of a signer-independent split, but the body-crop content is more fundamental because it affects even the within-dataset comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates the contribution of facial regions (eyes, mouth, full face) to vision-based isolated sign language recognition. The authors construct a 12-class German Sign Language dataset from the Public DGS Corpus, crop videos into four regions of interest (eyes, mouth, face, body), train a CNN-based model (CSN) and a transformer-based model (MViT) on each ROI, and also evaluate late fusion of the body ROI with each facial ROI. They report top-1/top-3 accuracy and F1 scores, supplemented by SmoothGrad saliency maps. The central claim is that the mouth is the most important non-manual facial feature and that incorporating facial features, especially the mouth, improves recognition.","tokens_in":10640,"tokens_out":6964,"duration_ms":76724,"significance":"If the claims are valid, the paper offers a useful controlled comparison of facial-feature contributions across two architecturally distinct models, which is rare in the sign-language-recognition literature. Strengths include the use of a public corpus, a consistent late-fusion protocol applied identically to both models, the explicit framing of the study as a controlled analysis rather than a state-of-the-art contribution, and the inclusion of qualitative saliency evidence. The conclusion that mouth cues are the most informative facial region is plausible and consistent with linguistic and eye-tracking studies. However, the central quantitative claim depends on a body baseline whose construction is not shown to exclude facial pixels, and the paper's statistical and dataset-split choices weaken the stated conclusions.","major_comments":[{"comment":"The body ROI is defined as the bounding box around MediaPipe upper-body and hand landmarks, but the paper does not state that facial landmarks are excluded from this bounding box. MediaPipe's upper-body pose landmark set includes facial landmarks such as the nose, eyes, ears, and mouth corners, so an axis-aligned box around these landmarks will generally contain the face. If the body condition already contains facial pixels, then the central comparison between 'body' and 'body + facial ROI' is not a comparison of manual features versus manual plus facial features: the observed gains could reflect the addition of a second, higher-resolution facial crop rather than the addition of new facial information. Moreover, the equivalence between body+mouth and body+face, which is used to conclude that the mouth is the most important facial feature, could be an artifact of the full face being largely redundant with facial pixels already present in the body crop. The authors should recompute the body ROI after explicitly excluding facial landmarks, report overlap statistics between the body and face crops, or add a control condition with the face masked or pixelated in the body stream. Without such a control, the main claim is not cleanly supported.","section":"Section 3 (Dataset), body ROI definition"},{"comment":"The 8:1:1 split is described only as random while preserving class distribution; it is not stated whether clips from the same signer are kept in the same set. Since the DGS Corpus contains multiple clips per signer and the facial ROIs carry strong identity cues, a random clip split can allow the model to exploit signer appearance rather than sign identity. This could inflate the apparent contribution of face and mouth streams and affect the feature ranking. The authors should specify whether the split is signer-independent, report the degree of signer overlap between train and test, or re-run the experiments with a signer-exclusive split. If such a split is not feasible, the conclusions should be explicitly restricted to within-signer recognition rather than general ASLR.","section":"Section 3 (Dataset), split paragraph"},{"comment":"The paper uses overlapping 95% confidence intervals to conclude that body+eyes is 'not significantly different' from body alone and that body+mouth and body+face are 'statistically indistinguishable'. Overlap of confidence intervals is not an equivalence test, and a non-significant difference is not evidence of absence of difference. The paper also does not report how the confidence intervals were computed (number of training runs or seeds, bootstrap over clips). Because the mouth-versus-face equivalence is load-bearing for the conclusion that the mouth is the most important facial feature, the authors should either provide a paired statistical test or equivalence bounds, or weaken the conclusion to 'we found no significant difference in this setting' and explicitly treat it as a null result.","section":"Section 5.1, Table 1"},{"comment":"The abstract and conclusion state that the mouth 'significantly improving accuracy' and that incorporating facial features significantly improves recognition, but Table 1 shows that for MViT the top-1 gains of body+mouth (86.42±2.51) and body+face (86.98±2.47) over body (84.03±2.69) are not significant by the paper's own confidence-interval criterion, and Section 5.1 explicitly says so. The unqualified wording overstates the evidence. The claims should be qualified by model and metric, e.g., significant for CSN top-1 accuracy and MViT top-3 accuracy.","section":"Abstract and Section 5.1"}],"minor_comments":[{"comment":"The sentence 'the mouth area is the most important facial feature .' contains a stray space before the period; please correct the typographical error.","section":"Section 5.1"},{"comment":"The paper says the 12 glosses were 'randomly selected' but does not report the random seed or the full list of selected glosses; providing these details would improve reproducibility.","section":"Section 3 (Dataset)"},{"comment":"The experimental setup does not state the number of random seeds or training repetitions used to compute the confidence intervals in Table 1; please specify this for reproducibility.","section":"Section 4 (Experiments)"},{"comment":"The saliency analysis is described qualitatively; please clarify the normalization procedure for the attribution values and state how many videos/classes were inspected and whether any quantitative measure of mouth-region saliency was computed.","section":"Section 5.2 (Saliency Maps)"}],"recommendation":"major_revision","confidential_remarks":"The body-ROI contamination and the lack of a signer-independent split are fixable in principle by re-running the experiments; if the authors cannot re-run them, the conclusions should be substantially weakened. The paper would also benefit from releasing the crop-generation code and overlap statistics. The scope is modest but appropriate for a workshop/adjunct venue, and the core question is worth addressing once the methodological issues are resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take on arXiv:2507.20884. The paper compares eyes, mouth, and full face as auxiliary streams in isolated sign language recognition, using CSN and MViT on a 12-gloss subset of the Public DGS Corpus. The mouth and full-face streams both beat the body-alone baseline for CSN, and the two are statistically indistinguishable; eyes add nothing. That ranking is consistent across both model families, which is a point in the paper's favor.\n\nWhat's genuinely new: this is an end-to-end deep learning comparison on a class set that wasn't deliberately built from non-manual contrasts. The authors also deserve credit for reporting confidence intervals and for explicitly noticing that 6 of their 12 glosses share the same index-finger hand form. That kind of transparency helps the reader calibrate.\n\nThe soft spot is the baseline. The body ROI is defined as the outermost coordinates of the MediaPipe upper-body and hand landmarks. MediaPipe's upper-body landmarks include the nose, eyes, ears, and mouth corners, so an axis-aligned box around them almost certainly contains the face. The paper never says the face is excluded, and no overlap stats are given. If the face is already in the body stream, then 'body + mouth' isn't manual plus facial; it's low-res face plus high-res mouth. The improvement could come from resolution, not from adding a new modality. The equivalence of mouth and full face also becomes less surprising: the full-face crop is largely redundant with what's already in the body, while the mouth crop supplies new detail. This directly weakens the claim that facial features are necessary.\n\nSecond issue: the 8:1:1 split is described as random with class distribution preserved, with no statement that clips from the same signer are kept in one set. If signers appear in both train and test, the model can exploit signer appearance. That needs to be fixed.\n\nThe abstract overstates the significance: for MViT, the top-1 gains are not statistically significant, only top-3 for mouth. That should be qualified. No code or data is provided, so exact replication is also impossible.\n\nAll of these are addressable. The right fix is to report the actual pixel content of the body crop, or run a control with the face masked, and to do a signer-independent split. Without that, the headline conclusion isn't supported as written.\n\nI'd send it to review—the research question is legitimate and the experimental setup is informative even if the baseline is contaminated. A referee can require the clean baselines. But I wouldn't cite it as evidence for the mouth's importance until the redo is done.\n\nBest.","headline":"The mouth-over-eyes ranking is plausible, but the 'body' baseline likely contains the face, so the central claim needs a cleaner control before it can be trusted.","tokens_in":11193,"tokens_out":5788,"would_cite":false,"duration_ms":61020,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mouth, not eyes, is the facial feature sign recognition needs.","keywords":["sign language recognition","non-manual features","mouth actions","facial regions","isolated sign recognition","German Sign Language","saliency maps","video classification"],"falsifier":"Retrain the body-only and body+mouth models with a signer-disjoint split and with the face explicitly masked out of the body crop; if body+mouth no longer beats body alone, or the gap largely disappears, the mouth's central role is an artifact of leaked facial pixels or signer appearance.","tokens_in":10163,"feed_emoji":"👄","tokens_out":5842,"duration_ms":63221,"temperature":0.7,"pith_summary":"The paper asks which facial region in a video of a sign actually carries information for automatic sign recognition, and it argues that the mouth does the work. Using two deep video classifiers trained on isolated German Sign Language glosses, the authors compare crops of the eyes, mouth, full face, and upper body, and then fuse each facial crop with the body crop. The mouth alone reaches 48–51% top-1 accuracy, far above the eyes' 21–22%. Fusing the mouth with the body raises CSN top-1 accuracy from 80.53% to 88.24%, and this gain is statistically indistinguishable from fusing the full face, whereas adding the eyes gives no significant gain. Saliency maps also concentrate on the mouth, supporting the conclusion that mouthing and other mouth actions are the key non-manual facial signal.","feed_headline":"Mouth, not eyes, is the facial feature sign recognition needs","feed_subtitle":"Adding a mouth-only video stream to a hand/body stream lifts accuracy from 80.5% to 88.2% in isolated German Sign Language recognition.","key_machinery":"The experimental design is the mechanism: the same video clip is cropped into four regions of interest (eyes, mouth, full face, and upper body including hands), and each region is fed, alone or in a late-fusion two-stream configuration, into either a channel-separated convolutional network or a multiscale vision transformer. Because the only change between runs is which pixels are visible, any accuracy difference is attributed to that facial region. Gradient-based saliency maps (vanilla gradients plus SmoothGrad) provide the qualitative counterpart, showing where the trained models actually look.","core_discovery":"On its own terms, the paper establishes that in an end-to-end, RGB-video, gloss-level recognition setting, the mouth is the most informative facial region, and that the value of the full face comes largely from the mouth area. With 12 randomly selected glosses from a public German Sign Language corpus, a channel-separated convolutional network (CSN) and a multiscale vision transformer (MViT) both show the same pattern: mouth-only inputs outperform eye-only inputs by a wide margin; fusing mouth with the body stream improves accuracy over body alone (CSN: 80.53% to 88.24% top-1) while fusing eyes does not; and body+mouth and body+face are statistically indistinguishable across all metrics. The authors interpret this as evidence that mouth actions, which can distinguish signs with identical manual articulation, are the primary non-manual facial contribution, and they confirm it with gradient saliency maps that consistently highlight the mouth.","pith_inferences":["Editorial: the “body” ROI was cropped from upper-body and hand landmarks without explicitly removing the face, so facial pixels may leak into the manual-only baseline; a cleaner manual baseline would mask or crop out the face before testing how much the mouth adds.","Editorial: the 8:1:1 clip split is described only as random, not as signer-disjoint; if clips from the same signer appear in both training and test, signer appearance rather than mouthing could contribute. A signer-independent split would test this.","Editorial: a direct ablation that masks or blurs the mouth in full-face inputs, or swaps mouth regions between signers, would test whether mouthing itself, rather than head motion or identity, drives the gain.","Editorial: pretraining on lip-reading before fine-tuning on sign language is a natural next test, since the paper points to lip reading as a possible source of transferable features."],"forward_implications":["For isolated sign recognition, systems that already use manual features should include a mouth stream; an eye stream is unlikely to add accuracy.","Fusing body and mouth yields essentially the same accuracy as fusing body and full face, so a small mouth crop appears to retain most of the facial benefit at lower computational cost.","Facial information mainly helps the model decide between its top candidates, since top-3 accuracy shows the largest gains, consistent with mouthing disambiguating manually similar signs.","Because the pattern holds in both a convolutional and a transformer-based architecture, the result is not specific to one model family."],"supporting_citations":[{"why":"supplies the German Sign Language video data and gloss annotations from which the 12-class isolated-sign dataset is built.","marker":"[13]"},{"why":"provides the implementation used to crop the mouth region of interest.","marker":"[19]"},{"why":"provides the landmark detection used to crop the eyes, upper body, and hands.","marker":"[18]"},{"why":"defines the channel-separated convolutional network architecture, one of the two models tested.","marker":"[30]"},{"why":"defines the multiscale vision transformer architecture, the second model tested.","marker":"[5]"},{"why":"supplies the pretrained implementations of the two architectures under the same frame and size settings.","marker":"[4]"},{"why":"prior work showing mouth actions disambiguate manually similar signs, used to explain the mouth's benefit.","marker":"[25]"},{"why":"prior fine-grained comparison of manual features with eyebrows, mouth, and face that this study extends to end-to-end deep models.","marker":"[21]"},{"why":"introduces the vanilla gradient saliency method used for qualitative analysis.","marker":"[28]"},{"why":"introduces SmoothGrad, used to denoise the saliency maps.","marker":"[29]"}],"fun_headline_variants":["Mouth beats eyes for sign language recognition, study finds","Sign language AI: mouth is the key facial feature","Mouth, not eyes, drives sign recognition accuracy","Facial feature showdown: mouth wins in sign language AI","For sign recognition, mouth matters more than eyes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes the body crop is a manual-only baseline, yet the cropping procedure never explicitly removes facial pixels from it, so the “manual” stream may already contain mouth and eye information; the random clip split also assumes no signer appears in both training and test.","fun_headline_variants_meta":{"raw":{"variants":["Mouth beats eyes for sign language recognition, study finds","Sign language AI: mouth is the key facial feature","Mouth, not eyes, drives sign recognition accuracy","Facial feature showdown: mouth wins in sign language AI","For sign recognition, mouth matters more than eyes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000194,"raw_usage":{"total_tokens":1328,"prompt_tokens":897,"completion_tokens":431,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":364}},"tokens_in":513,"tokens_out":431,"duration_ms":5056,"temperature":1.0,"reasoning_tokens":364,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T13:10:30.998309+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the body-only and body+mouth models with a signer-disjoint split and with the face explicitly masked out of the body crop; if body+mouth no longer beats body alone, or the gap largely disappears, the mouth's central role is an artifact of leaked facial pixels or signer appearance.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the pretrained implementations of the two architectures under the same frame and size settings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"prior work showing mouth actions disambiguate manually similar signs, used to explain the mouth's benefit."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"prior fine-grained comparison of manual features with eyebrows, mouth, and face that this study extends to end-to-end deep models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"introduces the vanilla gradient saliency method used for qualitative analysis."}],"review_version":1}