Targeted Linguistic Analysis of Sign Language Models with Minimal Translation Pairs

Diane Brentari; Greg Shakhnarovich; Kanishka Misra; Karen Livescu; Serpil Karab\"ukl\"u; Shester Gueuwou

REVIEW 3 major objections 5 minor 6 references

Targeted Linguistic Analysis of Sign Language Models with Minimal Translation Pairs

T0 review · 3 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new minimal-pair benchmark reveals how a state-of-the-art sign-language translation model reads its video: hands carry most of the signal, while the eyebrow, mouth, and body cues that mark questions and other grammar in ASL are largely ig

desk verdict Valuable new benchmark and a careful case study; the missing text-only baseline is a real caveat but doesn't sink the main ablation findings. read the letter →

arxiv 2604.27232 v3 pith:SD55WV7D submitted 2026-04-29 cs.CL

Serpil Karabüklü , Kanishka Misra , Shester Gueuwou , Diane Brentari , Greg Shakhnarovich , Karen Livescu This is my paper · ORCID

classification cs.CL

keywords signlanguagetranslationminimalpairslinguisticphenomenanon-manualcuescueablationsurprisalanalysisAmericanbenchmarkdataset

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper establishes a method and a diagnosis. It introduces ASL-MTP, a dataset of 1,275 ASL video clips paired with matched and minimally mismatched English translations across nine linguistic phenomena, so a translation model can be tested phenomenon by phenomenon. In a case study, the paper shows that a current multi-channel ASL-to-English model scores above chance on 8 of 9 phenomena when all video cues are present, but masking the hands channel sharply lowers accuracy (numbers 0.87 to 0.61, fingerspelling 0.78 to 0.49), while masking face, eyes, or body leaves most accuracies nearly unchanged. Even polar questions, which in ASL are marked primarily by eyebrow raise, stay at 4–9% accuracy under every ablation, revealing a strong bias toward declarative sentences. The authors argue this matters because sign-language grammar is carried across multiple articulators, and standard translation metrics such as BLEURT cannot reveal this channel-level gap.

What carries the argument

The central mechanism is the minimal translation pair coupled with a sign-conditioned surprisal test. For each phenomenon, the dataset provides a matched English translation of an ASL video and a mismatched variant that differs only at the target site (e.g., 'The movie starts at 7' versus 'The movie starts at 8'; 'Are Jen and Joe married?' versus 'Jen and Joe are married.'). The model's per-token surprisal is computed for both sentences conditioned on the video, and the difference in surprisal is the sensitivity score; accuracy is the fraction of pairs for which the mismatched sentence is more surprising. To attribute sensitivity to channels, the paper masks detected regions (hands, face, ey

What would settle it

Run the surprisal comparison with the video replaced by neutral noise, or with a text-only model. If the same pattern persists — polar-question accuracy near 0.04 and hand-masking still drops numbers accuracy — then the reported sensitivities are artifacts of the English decoder, not of sign-language video. Conversely, native-signer judgments showing that some pairs, such as the classifier singular/plural example, are not minimal would invalidate those subsets.

Watch

Extended reading notes

Core claim

The central claim is that a sign-language translation model trained on multi-channel video does not use its channels the way sign-language linguistics would predict: it is strongly driven by hand cues and largely indifferent to non-manual cues (eye/brow, mouth, head/body), even when those cues are the only grammatical signal. Evidence comes from masking channels one at a time, at inference and during retraining: accuracy falls when hands are hidden and stays roughly flat when the face or eyes are hidden, and polar-question versus declarative accuracy remains at 4–9% in every ablation. The paper also claims ASL-MTP is the first minimal-pair benchmark for phenomenon-level sensitivity in sign-l

Load-bearing premise

The analysis assumes each matched/mismatched pair isolates the target phenomenon: that the two English sentences differ only in that structure, and that the measured surprisal difference reflects what the model saw in the video rather than the English decoder's own text priors or extra differences such as number agreement.

Editorial extensions

If this is right

If the finding holds, ASL-to-English models should be evaluated per phenomenon, not only with BLEU or BLEURT; minimal-pair accuracy exposes channel-specific blind spots that aggregate metrics hide.
Improving models' use of eyebrow, eye, and head/body cues should directly raise accuracy on questions, conditionals, and negation, where ablating those cues changes performance little or not at all.
The controlled-rearing experiments show the manual-heavy behavior is not merely an artifact of inference-time train/test mismatch, so fixing it will require changes in training or architecture, not just input presentation.
The model's declarative bias for polar questions is stable across every cue ablation, indicating a systematic prior that future models must correct to translate questions reliably.
ASL-MTP provides a reusable 9-phenomenon benchmark for comparing newly released sign-language translation models on the same linguistic structures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The strong declarative bias may come mostly from the English text decoder, not the video encoder: declarative sentences dominate training text, so a text-only baseline with the video replaced by noise would likely reproduce the 0.04–0.09 polar-question pattern. This is a testable extension the paper does not run.
Some pairs are not strictly minimal: the Classifier example also changes subject-verb agreement (singular vs. plural), so part of the low classifier accuracy may reflect agreement sensitivity rather than classifier handshape sensitivity. Native-signer validation of every pair would clean this up.
The same matched/mismatched surprisal recipe could be extended to other sign languages and to phenomena not in the dataset, such as topic marking, focus, or role shift, as long as minimal pairs can be constructed.
If future training adds explicit supervision on non-manual annotations (e.g., predicting brow or head markers), ASL-MTP could directly measure whether that supervision transfers to translation accuracy on polar questions and conditionals.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First, the punchline: this is a useful new resource and a careful case study, but the authors stopped one control short of proving their main headline. The benchmark, ASL-MTP, is the real contribution; the ablation findings are directionally sound but the absolute accuracy numbers should be treated with suspicion until a text-only baseline is run.

The new thing is the dataset: 1,275 ASL videos from ASLLRP with matched/mismatched English translation pairs covering nine phenomena across the manual and non-manual channels. That is genuinely novel. Previous minimal-pair work in sign language was single-channel or isolated signs; this spans multi-channel phenomena and is built from a corpus not used to train the model. The case study is also well executed: two evaluation metrics, Bonferroni-corrected exact binomial tests, and both inference-time and training-time ablations. The hands-ablation results are clear — removing hands pushes accuracy on Numbers, Fingerspelling, Classifiers, Wh-Questions, and Conditionals toward or below chance. That is a real video-driven effect.

The first soft spot is the missing text-only baseline. ByT5's English prior alone would explain the polar question result: 0.04 accuracy in every condition, even with hands only or face only. The paper already calls this 'declarative bias,' but the concern extends wider. For other phenomena, like Numbers, the absolute above-chance accuracy could be inflated by the decoder's preference for common sentences. The hand-ablation comparisons are still meaningful because they are relative, but the claim 'above chance on most phenomena' is not pinned to the video signal without a no-video control. That is the one experiment I would demand in a revision.

Second, some minimal pairs are not actually minimal. The Classifier example changes 'the friends/are' to 'the friend/is,' so it introduces a subject-number and agreement difference on top of the classifier. Third, the mismatched sentences are author-generated without native-signer validation. The matched sentences come from ASLLRP, so the risk is asymmetric, but it is worth stating.

These are fixable problems, not fatal ones. The dataset is worth having, the case study is reproducible, and the paper is honest about most of its limitations. I would send this to a serious referee; the revision should add the text-only control, tighten the minimal pairs, and hedge the absolute-accuracy claims. Reading group: yes. Cite: yes.

Referee Report

3 major / 5 minor

Summary. The paper introduces ASL-MTP, a benchmark of 1,275 ASL video/English-translation minimal pairs covering nine linguistic phenomena, and uses it to analyze SHuBERT+ByT5, a state-of-the-art ASL-to-English translation model. The evaluation metric is the per-token surprisal difference between a matched and a mismatched translation conditioned on the video (Eqs. 1–2); accuracy is the proportion of pairs where the matched sentence has lower surprisal. The case study applies eight inference-time cue ablations (hands, face, mouth, eyes/brows, body) and two training-time ablations. The main claims are that the model performs above chance on most of the phenomena, that it relies strongly on hand cues, and that it is often insensitive to non-manual cues. A secondary claim is that ASL-MTP is the first minimal-pair benchmark for phenomenon-level sensitivity in sign-language translation models.

Significance. If the conclusions hold, ASL-MTP is a valuable first benchmark for phenomenon-specific evaluation of sign-language translation models, and the case study provides a useful diagnostic: standard MT metrics such as BLEURT do not reveal the same distinctions as surprisal-based minimal-pair analysis. The paper has several strengths that should be acknowledged: the evaluation source (ASLLRP) is distinct from the model's training corpora; the statistical comparisons use Bonferroni-corrected exact binomial tests; the study includes both inference-time and training-time ablations; and the model, data, and code are publicly available. The benchmark itself is a genuine contribution. However, the central quantitative claims about video-encoded sensitivity and insensitivity depend on an untreated confound—the English text prior of ByT5—so the conclusions are conditional until that confound is controlled.

major comments (3)

[§3.2, Eq. (2); Table 2] The accuracy measure compares per-token surprisal of two English strings under p(x_t | x_<t, F_i). Since ByT5 is an English text decoder, a substantial part of ΔSurprisal may come from the unconditional English prior rather than from video-conditioned information. The polar-question rows are the clearest signature: 'Declaratives vs. Polar Questions' stays at 0.96–0.97 and 'Polar Questions vs. Declaratives' at 0.04–0.05 across every ablation, including NFB (hands only) and NHB (face only). An English-only model would produce exactly this asymmetry. Without a text-only or no-video control, the paper's load-bearing claim—above-chance performance on most phenomena and weak sensitivity to non-manuals—is not established as a fact about video encoding. Please add a no-video condition (e.g., ByT5 without SHuBERT features or with zeroed video inputs) and report the resulting accuracies/ΔSurprisal
[§3.1, Table 1 (Classifiers row)] The 'minimal' property is not met for all pairs. The Classifiers example changes not only the classifier but also number and subject–verb agreement ('Are the friends ... going out?' vs. 'Is the friend going out?'), so the model has additional surface cues beyond the target phenomenon. Other manipulation types (e.g., replacing numbers or fingerspelled names) also introduce lexical-frequency and plausibility differences that a text-only prior can exploit. The paper does not report native-signer validation or inter-annotator agreement for the author-generated mismatches. Because accuracy is defined as the model preferring the matched sentence, any non-minimal difference is an exploitable confound. Please provide validation by ASL signers/linguists, report agreement, or restrict the claims to subsets where strict minimality can be verified.
[§4.3, Table 2, rows 7b and 9b] The evidence for 'often missing crucial non-manual cues' relies heavily on null effects in small subsets, notably Conditionals (NM only) with N=50. A null ablation effect may reflect low statistical power rather than true insensitivity; the paper itself notes this possibility. Relatedly, the Polar Questions (NM only) row cannot separate 'no use of brow raise' from 'English prior overrides video', since the declarative-bias asymmetry persists in every condition. Please report effect sizes and confidence intervals for these subsets, and consider collecting additional NM-only stimuli or explicitly qualifying the non-manual-insensitivity conclusion as tentative pending the text-only control requested above.

minor comments (5)

[§3.1] The sentence 'yield the focused dataset ASL-MTP of 1,275 pairs across phenomena 1' appears to contain a stray '1.' after 'phenomena'.
[Abstract and §4.3] The abstract says 'above chance on most of the phenomena' (8 of 9), while the body reports 9 of 11 subsets after splitting Conditionals and Polar Questions. This is understandable, but the wording should be aligned.
[Table 2] The row for 'Conditionals (NM only)' changes from 0.70 in AC to 0.76 in NFB, which is directionally surprising; a brief comment in the text about possible noise in this small subset would help.
[§4.4] The BLEURT correlations are reported only as a range. Providing confidence intervals or at least the number of pairs used in each correlation would make the comparison with surprisal accuracy more interpretable.
[§4.1] Please clarify how the ByT5 decoder receives SHuBERT representations (e.g., cross-attention, prefix), since this determines whether a 'text-only' baseline can be implemented by simply dropping video features or requires a separate model.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical measurements on a publicly available model, with no fitted parameter renamed as a prediction and no equation reducing to its own inputs.

full rationale

The paper's central claims—above-chance accuracy on most ASL-MTP phenomena and relatively weak sensitivity to non-manual cues—are empirical measurements. The surprisal definition in Eqs. (1)–(2) is a standard evaluation statistic, not a derived quantity whose output is determined by its inputs: the model weights are fixed (publicly released SHuBERT+ByT5), and no parameter is fitted to ASL-MTP before evaluation. The cue ablations are input manipulations and the reported accuracies are observed frequencies, so they cannot reduce by construction to the dataset definition. The only self-referential elements are methodological: the case-study model comes from the same group (Gueuwou et al. 2025b) and the training recipe is followed from that citation. That is a legitimate use of a public, externally reviewed model with released weights; it does not function as an unverified premise for the paper's main conclusion. The phenomenon-to-channel expectations are sourced from external sign-language linguistics (Baker-Shenk 1983; Liddell 1980; Wilbur and Patschke 1999), not from the model being evaluated. The reviewer concern that a missing text-only baseline could mean ByT5's English prior drives some accuracies is a threat to the interpretation of the experiments, not a circularity: even if the no-video control were absent, the reported numbers would be contingent empirical facts rather than consequences of the equations. Likewise, the author-generated mismatched translations and the non-strict minimality of the Classifier example are data-quality issues, not circular reductions. There is no step in the paper where an output is defined in terms of the target quantity, where a fitted value is later called a prediction, or where a self-citation carries the load of a derivation. Overall, no significant circularity was found.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper contributes no fitted numbers and postulates no new theoretical entities; it measures behavior of an existing public model on an author-built benchmark. Free-parameter count is zero because the reported accuracies are measurements on pre-trained public weights, and dataset splits are corpus attributes, not numbers fit to make a derivation work. The ledger above records the uncharged premises: linguistic channel mappings, benchmark validity, masking fidelity, and the surprisal-metric interpretation — the last being the one the authors never state explicitly.

assumptions (4)

domain assumption The linguistic channel mappings used to design ASL-MTP are correct — e.g., Polar Questions are conveyed only by non-manuals (eyebrow raise), Wh-Questions by wh-sign plus brow lowering, Conditionals by IF plus brow raise.
Invoked in §2.3 and Table 1 to define phenomena and to interpret ablation results; based on the cited literature (Baker-Shenk 1983; Liddell 1986; Wilbur & Patschke 1999), but assumed, not tested in this corpus.
domain assumption ASLLRP's glosses and English translations are accurate, and the authors' generated mismatched sentence is unacceptable for the video in the intended way.
Basis of the matched/mismatched pairs in §3.1; no inter-annotator or native-signer validation is reported.
domain assumption Greying MediaPipe-detected regions removes the target channel information without destroying other channels.
Core of the cue ablation protocol in §4.2; footnote 4 concedes hand location survives in body-pose keypoints, and greying precision of small regions (eyebrows, mouth) is not quantitatively verified.
ad hoc to paper Per-token surprisal (Eqs. 1–2), averaged over the sentence, is dominated by video-conditioned phenomenon sensitivity rather than the English text prior of ByT5.
Unstated premise of the accuracy definition in §3.2; the invariant 0.97/0.04 declarative/question scores across all ablations suggest a text-level confound that is never controlled for.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Targeted Linguistic Analysis of Sign Language Models with Minimal Translation Pairs." pith.science (2026). https://pith.science/paper/SD55WV7D

@misc{pith2026260427232,
  author       = {Pith},
  title        = {Pith review of: Targeted Linguistic Analysis of Sign Language Models with Minimal Translation Pairs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SD55WV7D}},
  note         = {Machine review of arXiv:2604.27232}
}

read the original abstract

Models of sign language have historically lagged behind those for spoken language (text and speech). Recent work has greatly improved their performance on tasks like sign language translation and isolated sign recognition. However, it remains unclear to what extent existing models capture various linguistic phenomena of sign language, and how well they use cues from the multiple articulators used in sign language (hands, upper body, face). We introduce a new benchmark dataset for American Sign Language, ASL Minimal Translation Pairs (ASL-MTP), divided into multiple types of sign language phenomena and corresponding minimal pairs of translations, for performing such linguistic analyses. As a case study, we use ASL-MTP to analyze a state-of-the-art ASL-to-English translation model. We conduct a targeted analysis of the model by ablating various input cues during training and inference and evaluating on the phenomena in ASL-MTP. Our results show that, while the model performs above chance level on most of the phenomena, it relies strongly on manual cues while often missing crucial non-manual cues.

Figures

Figures reproduced from arXiv: 2604.27232 by the authors.

**Figure 1.** Our dataset construction and analysis ap view at source ↗

**Figure 2.** Left: A depiction of how SHUBERT (Gueuwou et al., 2025b) is combined with an off-the-shelf language model (here, ByT5) to perform ASL-to-English translation. Right: Examples of inputs provided to the model for the All Cues condition as well as the 8 Cue Ablations. 4 A Case Study Next we use ASL-MTP for a case study, in which we analyze an open, state-of-the-art ASLto-English translation model. Specifically, we use … view at source ↗

Figures from the paper (2 more)

**Figure 3.** Figure 3: Average difference in surprisal of mismatched and matched sentences across phenomena and across

**Figure 4.** Figure 4: Average difference in surprisal of mismatched and matched sentences across phenomena and across

Review history (3 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

6 extracted references · 3 linked inside Pith

[3]

InFindings of the As- sociation for Computational Linguistics: ACL- IJCNLP 2021, pages 4958–4969, Online

Language models use monotonicity to assess NPI licensing. InFindings of the As- sociation for Computational Linguistics: ACL- IJCNLP 2021, pages 4958–4969, Online. Asso- ciation for Computational Linguistics. Jaap Jumelet, Leonie Weissweiler, Joakim Nivre, and Arianna Bisazza. 2026. MultiBLiMP 1.0: A massively multilingual benchmark of linguistic minimal ...

arXiv 2021
[5]

InAdvances in Neural Information Processing Systems, volume 36, pages 29029– 29047

YouTube-ASL: A large-scale, open- domain American Sign Language-English paral- lel corpus. InAdvances in Neural Information Processing Systems, volume 36, pages 29029– 29047. Clayton Valli and Ceil Lucas. 2000.Linguistics of American Sign Language: An introduction. Gallaudet University Press. Silvana C Veinberg and Ronnie B. Wilbur. 1990. A linguistic ana...

arXiv
[6]

controlled rearing

Signed coreference resolution. InPro- ceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4950–4961, Online and Punta Cana, Dominican Republic. Association for Computational Lin- guistics. Kayo Yin, Chinmay Singh, Fyodor O Minakov, Vanessa Milan, Hal Daumé III, Cyril Zhang, Alex Xijie Lu, and Danielle Bragg. 2024. As...

2021
[2018]

InPro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, pages 7784–7793, Salt Lake City, Utah, USA

Neural sign language translation. InPro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, pages 7784–7793, Salt Lake City, Utah, USA. Necati Cihan Camgöz, Oscar Koller, Simon Had- field, and Richard Bowden. 2020a. Multi- channel transformers for multi-articulatory sign language translation. InComputer Vision–ECCV 2020...

2018
[2021]

InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

How2sign: A large-scale multimodal dataset for continuous american sign language. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Shester Gueuwou, Xiaodan Du, Greg Shakhnarovich, and Karen Livescu. 2025a. SignMusketeers: An efficient multi-stream ap- proach for sign language translation at scale. In Findings of the Ass...

2025
[2023]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu

Dinov2: Learning robust visual features without supervision.Transactions on Machine Learning Research. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for auto- matic evaluation of machine translation. InPro- ceedings of the 40th Annual Meeting of the As- sociation for Computational Linguistics, pages 311–318, Philadelphi...

arXiv 2002

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.

[1] [3]

InFindings of the As- sociation for Computational Linguistics: ACL- IJCNLP 2021, pages 4958–4969, Online

Language models use monotonicity to assess NPI licensing. InFindings of the As- sociation for Computational Linguistics: ACL- IJCNLP 2021, pages 4958–4969, Online. Asso- ciation for Computational Linguistics. Jaap Jumelet, Leonie Weissweiler, Joakim Nivre, and Arianna Bisazza. 2026. MultiBLiMP 1.0: A massively multilingual benchmark of linguistic minimal ...

arXiv 2021

[2] [5]

InAdvances in Neural Information Processing Systems, volume 36, pages 29029– 29047

YouTube-ASL: A large-scale, open- domain American Sign Language-English paral- lel corpus. InAdvances in Neural Information Processing Systems, volume 36, pages 29029– 29047. Clayton Valli and Ceil Lucas. 2000.Linguistics of American Sign Language: An introduction. Gallaudet University Press. Silvana C Veinberg and Ronnie B. Wilbur. 1990. A linguistic ana...

arXiv

[3] [6]

controlled rearing

Signed coreference resolution. InPro- ceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4950–4961, Online and Punta Cana, Dominican Republic. Association for Computational Lin- guistics. Kayo Yin, Chinmay Singh, Fyodor O Minakov, Vanessa Milan, Hal Daumé III, Cyril Zhang, Alex Xijie Lu, and Danielle Bragg. 2024. As...

2021

[4] [2018]

InPro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, pages 7784–7793, Salt Lake City, Utah, USA

Neural sign language translation. InPro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, pages 7784–7793, Salt Lake City, Utah, USA. Necati Cihan Camgöz, Oscar Koller, Simon Had- field, and Richard Bowden. 2020a. Multi- channel transformers for multi-articulatory sign language translation. InComputer Vision–ECCV 2020...

2018

[5] [2021]

InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

How2sign: A large-scale multimodal dataset for continuous american sign language. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Shester Gueuwou, Xiaodan Du, Greg Shakhnarovich, and Karen Livescu. 2025a. SignMusketeers: An efficient multi-stream ap- proach for sign language translation at scale. In Findings of the Ass...

2025

[6] [2023]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu

Dinov2: Learning robust visual features without supervision.Transactions on Machine Learning Research. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for auto- matic evaluation of machine translation. InPro- ceedings of the 40th Annual Meeting of the As- sociation for Computational Linguistics, pages 311–318, Philadelphi...

arXiv 2002