Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Automatically extracted context — background captions, previous-sentence translations, and pseudo-glosses — fed into a fine-tuned LLM substantially improves open-vocabulary sign language translation on BOBSL and transfers to How2Sign.

desk verdict A solid, useful SLT paper that never runs the one ablation—text-only, no video—that would separate the LLM's text priors from the video's contribution; the context-helps claim is likely true but the margin is modest. read the letter →

arxiv 2501.09754 v2 pith:J6D64BF6 submitted 2025-01-16 cs.CV

Youngjoon Jang , Haran Raajesh , Liliane Momeni , Gül Varol , Andrew Zisserman This is my paper · ORCID
classification cs.CV
keywords signlanguagetranslationcontextualcueslargemodelfine-tuningopen-vocabularyBOBSLBritishpseudo-glossesHow2Sign
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that sign language translation in the open-vocabulary setting improves when the translator is given the kind of context human interpreters use: what is happening in the background and what was just signed. It builds a single LLM-based decoder that receives visual signing features, automatically produced pseudo-glosses, keyword background descriptions, and the predicted translation of the previous sentence, all in one prompt. On the BOBSL British Sign Language benchmark, every cue adds a measurable gain and the full model outperforms previously reported results and two strong video-only baselines. The practical stake is that large interpreted TV archives, with their noisy subtitles, can support much better translation than video-only models achieve.

What carries the argument

The load-bearing mechanism is a single prompt assembled for a pre-trained decoder-only LLM, Llama3-8B, fine-tuned with LoRA. Visual features from a Video-Swin isolated-sign recogniser are projected through a trainable 2-layer MLP into the LLM embedding space, while pseudo-glosses, background keyword lists, and the previous predicted sentence are inserted as text with task prompts. This machinery lets one language model fuse weak visual and textual evidence, using pretrained English fluency for grammar while the visual stream and the contextual cues jointly disambiguate referents, homonyms, tense, and signs that are omitted or underspecified in the video.

What would settle it

Run the identical model with all text cues present but the visual tokens masked or replaced by random projections on BOBSL SENT-VAL; if BLEURT, IoU, and the LLM score do not drop materially below the full-cue model, the claim that the visual stream contributes beyond the text is refuted.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that open-vocabulary sign language translation should be treated as a context-dependent discourse task rather than an isolated video-to-text problem. The proposed model feeds a fine-tuned pre-trained LLM a prompt containing visual signing features, pseudo-glosses from an isolated-sign recogniser, a keyword summary of captions describing the background footage, and the predicted translation of the previous sentence. On BOBSL SENT-TEST the full-cue model reports BLEU-4 3.3, BLEURT 40.3, ROUGE-L 16.9, CIDEr 41.9, IoU 14.8, and an LLM-based score of 1.20, surpassing state-of-the-art baselines and the video-only version of the same model on every metric, with the ablation study showing that each cue adds a positive contribution. The same framework, without background cues, also achieves competitive results on the How2Sign American Sign Language dataset.

Load-bearing premise

The central assumption is that the fine-tuned LLM genuinely combines the textual cues with the visual signing features, rather than generating plausible English mostly from the fluent previous-sentence text and background keywords, since no text-only baseline is reported.

Editorial extensions

If this is right

  • Open-vocabulary translation on broad-domain interpreted TV is feasible without manual glosses, since pseudo-glosses are used only as an input cue rather than supervision.
  • Background descriptions help resolve pronoun reference, place names, pointing gestures, and object names that are not signed or are ambiguous in the video.
  • Previous-sentence context helps with topic-comment structure, tense, and coherence, and using the model's own predictions at inference avoids needing ground-truth context.
  • The drop-cue augmentation makes the final model robust when some cues are missing at test time, so the system does not collapse when background or previous-sentence information is unavailable.
  • An LLM-based evaluation metric adapted from CLAIR correlates more strongly with human judgements than BLEU-4 on the paper's small human study, giving a more interpretable measure of translation quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A text-only variant with the visual tokens masked or removed is not reported, so the marginal contribution of the visual stream — as opposed to fluent English inferable from the previous sentence and background keywords — remains an open question.
  • The auto-regressive use of previous predictions may accumulate errors over long stretches of signing, and the paper's own experiment with two or three previous sentences shows only marginal gains.
  • Because background descriptions are extracted generically from the whole frame, they can introduce noise or even distract the model; a more targeted extraction, such as focusing on the region a signer points to, is a natural testable extension.
  • The recipe should transfer to other sign languages that have a large weakly-aligned subtitle corpus plus an isolated-sign recogniser, but not to sentence-level datasets lacking discourse or background context.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes an LLM-based sign language translation (SLT) framework that combines visual signing features from a Video-Swin encoder with automatically extracted textual cues: pseudo-glosses, a background description from an image captioner, and the predicted translation of the previous sentence. All inputs are fed as a text-style prompt into a fine-tuned Llama3-8B decoder via LoRA and a learned mapping network. The authors evaluate on BOBSL and How2Sign, showing that each cue adds a positive contribution in ablations and that the full model outperforms GFSLT and Sign2GPT baselines on BOBSL, while achieving competitive results on How2Sign. They also introduce an LLM-based evaluation metric (GPT-4o-mini) that correlates strongly with human judgments.

Significance. If the central claim holds, the paper makes a strong empirical case that automatically extracted contextual cues, particularly previous-sentence translations and background scene descriptions, are valuable for open-vocabulary SLT from broadcast data. The work is also practically significant because it shows how to adapt a large pretrained LLM to sign language inputs without ground-truth glosses or spottings, and it provides a new evaluation metric that appears more aligned with human judgment than standard captioning metrics. The extensive ablations on BOBSL, the reproduction of prior baselines, and the code/project-page release are notable strengths. However, the missing no-video baseline and the backbone mismatch in the SOTA comparison prevent the current evidence from fully supporting the strong claims made in the paper.

major comments (3)
  1. [§4.3, Table 1; Appendix B.2, Table A.5] The ablations always include the visual features (Vid), and there is no text-only baseline, such as PrevPred+PG+BG without Vid or PrevPred+BG without Vid. Since the previous sentence is a fluent English sentence, the pseudo-glosses are English words, and the background is an English keyword list, a strong LM like Llama3-8B could plausibly generate a reasonable English translation from the text cues alone, with the visual signal contributing little to the reported gains. The claim that 'context has additional relevant information beyond the signing video' is therefore not cleanly supported. Please add a no-video ablation on SENT-VAL (and ideally on SENT-TEST) to quantify the marginal contribution of the visual features to the full-cue model.
  2. [§4.4, Table 3] The comparison to GFSLT and Sign2GPT is confounded by the visual backbone: the proposed model uses a Video-Swin ISLR encoder pre-trained on BOBSL with a vocabulary of 8,697 signs, while the baselines are reproduced with their own visual encoders (I3D/DINOv2-based). The row 'Ours (Vid)' already surpasses the baselines, but this could be driven by the stronger Video-Swin features rather than by the proposed LLM-based translation architecture. To make the SOTA claim fair, please either train the baselines with the same Video-Swin features or report a version of the proposed model with an I3D backbone matching [3] and [67], and state the backbone configuration explicitly in the table.
  3. [§3.2, Visual features] The sliding window stride s is a free parameter that controls the number of visual features (F) and the amount of temporal detail, yet the value used in the main experiments is not reported. The text only says 'when s=2, we have on average 56 features,' without stating that s=2 is the chosen value for the BOBSL experiments, and no ablation over s is provided. Please specify the exact stride for BOBSL and How2Sign, and if possible include a sensitivity analysis, since changing s directly changes the computational cost and the information available to the LLM.
minor comments (5)
  1. [§4.1, LLM Evaluation] The LLM-based evaluation metric is computed by GPT-4o-mini, and the translator itself is an LLM; although the correlation with human judgments (Fig. A.2) is reassuring, the metric may share systematic biases with LLM-generated translations. The paper could briefly note this potential bias and report the LLM scores with this caveat in mind.
  2. [§4.3, Table 1] The improvements from adding cues are reported without statistical significance tests. While +2.5 BLEURT and +0.27 LLM on a 1,973-sentence validation set are likely meaningful, providing error bars or a significance test (e.g., bootstrap) would strengthen the ablation claims.
  3. [Appendix A.3] The background descriptions overlap with ground-truth sentences by only 3.4% IoU, which the paper correctly interprets as not capturing the whole signal; however, the qualitative examples in Fig. 3 seem to show rather direct keyword matches (e.g., 'jellyfish', 'New Forest'). A quantitative analysis of how often background keywords appear in the final translations would help calibrate the reader's expectation.
  4. [§4.2, Baselines] The description of the Sincan [67] baseline is brief regarding the exact features and training setup; since the paper compares against it in the oracle setting, a sentence clarifying whether the same video features and decoder capacity are used would improve transparency.
  5. [§5, Conclusion] The conclusion lists challenges such as error accumulation from previous-sentence predictions and noise in background descriptions; these are good, but the paper could also mention that the missing no-video baseline is a limitation of the current analysis, since it prevents isolating the visual contribution.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the contextual-cue inputs are independently sourced and the claimed gains are measured on held-out test data with standard metrics; the missing text-only baseline is a confound, not a circular argument.

full rationale

The paper's derivation chain is: (i) extract visual features from a pretrained Video-Swin ISLR model; (ii) obtain pseudo-glosses from the same model's classifier; (iii) generate background descriptions with BLIP2 captions; (iv) use the model's own previous-sentence predictions autoregressively; (v) fine-tune Llama3-8B with LoRA on BOBSL; and (vi) evaluate on held-out SENT-VAL/SENT-TEST with BLEU, BLEURT, ROUGE-L, CIDEr, IoU, and an LLM-based score. None of these steps defines a cue in terms of the target translation, and no fitted parameter is relabeled as a prediction. The pseudo-glosses come from the same visual backbone as the video features, but they are intermediate classifier outputs, not the English translation target, so this is transfer learning rather than circularity. The LLM evaluation metric is calibrated with 12 human-annotated in-context examples and is shown to correlate with human judgments (0.84 in Fig. A.2), while the main conclusions also hold on standard non-LLM metrics such as BLEU-4 and ROUGE-L. The paper's self-citations provide datasets, alignments, spottings, and the ISLR backbone, but the central claim—that adding context to video improves open-vocabulary translation—is tested against independently reproduced baselines on held-out data and does not reduce to those citations. The absence of a text-only no-video baseline is a legitimate experimental limitation regarding how much of the gain is attributable to the visual signal, but it does not make the reported improvement equivalent to the inputs by construction. No circular step is therefore identified.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new physical entities or forces. The free parameters are mostly architectural choices and the custom LLM evaluation metric. The main axioms are the reliability of the dataset annotations and the pre-trained models.

free parameters (2)
  • sliding window stride s for visual features = s=2 for BOBSL, s=1 for How2Sign
    Chosen manually without systematic sweep; affects the number of visual features per sentence and thus the input sequence length.
  • LLM evaluation prompt and scoring rubric = GPT-4o-mini score 0-5 with 12 in-context examples
    The metric is designed by the authors, and the 12 examples are manually chosen, not derived from a formal benchmark.
assumptions (4)
  • domain assumption The BOBSL SENT-VAL and SENT-TEST splits provide temporally aligned English sentences that are reliable enough for training and evaluation.
    The paper relies on these splits as ground truth despite acknowledging the subtitles are noisy and weakly aligned.
  • domain assumption The Video-Swin ISLR model trained on BOBSL spottings produces visual features and pseudo-glosses that are informative for translation.
    The entire method depends on the quality of this pre-trained encoder, which is inherited from prior work without re-verification.
  • domain assumption The BLIP2 image captioner produces sufficiently accurate background descriptions.
    Background descriptions are extracted with an off-the-shelf captioner and summarized to keywords; errors in captioning directly affect the input.
  • domain assumption LoRA fine-tuning of Llama3-8B on 689k video-sentence pairs is sufficient to adapt the LLM to sign language translation.
    The paper assumes this training setup is adequate, though the LLM may overfit or not fully adapt.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues." pith.science (2026). https://pith.science/paper/J6D64BF6

@misc{pith2026250109754,
  author       = {Pith},
  title        = {Pith review of: Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J6D64BF6}},
  note         = {Machine review of arXiv:2501.09754}
}
read the original abstract

Our objective is to translate continuous sign language into spoken language text. Inspired by the way human interpreters rely on context for accurate translation, we incorporate additional contextual cues together with the signing video, into a new translation framework. Specifically, besides visual sign recognition features that encode the input video, we integrate complementary textual information from (i) captions describing the background show, (ii) translation of previous sentences, as well as (iii) pseudo-glosses transcribing the signing. These are automatically extracted and inputted along with the visual features to a pre-trained large language model (LLM), which we fine-tune to generate spoken language translations in text form. Through extensive ablation studies, we show the positive contribution of each input cue to the translation performance. We train and evaluate our approach on BOBSL -- the largest British Sign Language dataset currently available. We show that our contextual approach significantly enhances the quality of the translations compared to previously reported results on BOBSL, and also to state-of-the-art methods that we implement as baselines. Furthermore, we demonstrate the generality of our approach by applying it also to How2Sign, an American Sign Language dataset, and achieve competitive results.

Figures

Figures reproduced from arXiv: 2501.09754 by the authors.

Figure 1
Figure 1. Contextual cues in SLT: In addition to information extracted from the signing content (at the bottom right corner), we give the sign language translation model two contextual cues: the background de￾scription that identifies keywords describing the scene behind the signer, and the previous sentence translations. In this example, the ground truth (GT) translation has common words or semantics with the background cont… view at source ↗
Figure 2
Figure 2. Method overview: The input prompt combines contextual cues, the background descriptions and previous sentences, with the information from the current video sequence, specifically visual features and pseudo-glosses. Visual features corresponding to the signer {V } are extracted using a pre-trained Video-Swin model, which are projected to text space with a learnable mapping network. We obtain pseudo-glosses {P} by pas… view at source ↗
Figure 3
Figure 3. Qualitative analysis: We present visual examples to show how different cues affect the translation results. Starting with visual features, we incrementally add pseudo-glosses (PG), the predicted previous sentence (Prev), and the background description (BG). We observe that the previous sentence helps translation performance by providing further context (top left, bottom right). The background description also helps … view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation

    cs.CV 2025-05 conditional novelty 6.0 of 10

    LLM-generated pseudo glosses, reordered via weak video supervision, enable sign language translation that rivals gloss-supervised models while needing only 30 gloss examples.

Reference graph

Works this paper leans on

90 extracted references · 79 canonical work pages · cited by 1 Pith paper

  1. [3]

    BOBSL: BBC-Oxford British Sign Language Dataset.arXiv,

    Samuel Albanie, G¨ul V arol, Liliane Momeni, Hannah Bull, Triantafyllos Afouras, Himel Chowdhury, Neil Fox, Bencie Woll, Rob Cooper, Andrew McParland, and Andrew Zisserman. BOBSL: BBC-Oxford British Sign Language Dataset.arXiv,

  2. [67]

    Is context all you need? scaling neural sign language translation to large domains of discourse

    Ozge Mercanoglu Sincan, Necati Cihan Camgoz, and Richard Bowden. Is context all you need? scaling neural sign language translation to large domains of discourse. InProc. ICCV, 2023. 3, 5, 6, 7, 9

    work page 2023

  3. [1]

    Slowfast network for continuous sign language recognition

    Junseok Ahn, Y oungjoon Jang, and Joon Son Chung. Slowfast network for continuous sign language recognition. In Proc. ICASSP, 2024. 17

    2024

  4. [2]

    BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues

    Samuel Albanie, G ¨ul V arol, Liliane Momeni, Triantafyllos Afouras, Joon Son Chung, Neil Fox, and Andrew Zisserman. BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues. InProc. ECCV, 2020. 2, 16

    2020

  5. [4]

    The Notation of Signed Texts: Open Questions and Indications for Further Research

    Elena Antinoro Pizzuto and Paola Pietrandrea. The Notation of Signed Texts: Open Questions and Indications for Further Research. Journal of Sign Language and Linguistics, 2001. 1

    2001

  6. [5]

    Elisabeth engberg-pedersen, space in Danish Sign Language

    Brita Bergam. Elisabeth engberg-pedersen, space in Danish Sign Language. the semantics and morphosyntax of the use of space in a visual language. international studies on sign language research and communication of the deaf, vol. 19. hamburg: Signum verlag, 1993. 406 pp.Nordic Journal of Linguistics, 1995. 2

    1993

  7. [6]

    NLTK: The natural language toolkit

    Steven Bird and Edward Loper. NLTK: The natural language toolkit. In Proceedings of the ACL Interactive P oster and Demonstration Sessions. ACL, 2004. 4, 5

    2004

  8. [7]

    Sign language recognition, generation, and translation: An interdisciplinary perspective

    Danielle Bragg, Oscar Koller, Mary Bellard, Larwan Berke, Patrick Boudreault, Annelies Braffort, Naomi Caselli, Matt Huenerfauth, Hernisa Kacorri, Tessa V erhoef, Christian V ogler, and Meredith Ringel Morris. Sign language recognition, generation, and translation: An interdisciplinary perspective. In Proc. ACM SIGACCESSS, 2019. 2

    2019

Show all 90 references
  1. [8]

    Aligning subtitles in sign language videos

    Hannah Bull, Triantafyllos Afouras, G¨ul V arol, Samuel Albanie, Liliane Momeni, and Andrew Zisserman. Aligning subtitles in sign language videos. InProc. ICCV, 2021. 2, 5

    2021

  2. [9]

    Using convolutional 3D neural networks for user-independent continuous gesture recognition

    Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, and Richard Bowden. Using convolutional 3D neural networks for user-independent continuous gesture recognition. InIEEE International Conference of P attern Recognition, ChaLearn W orkshop, 2016. 2

    2016

  3. [10]

    Sumru¨Ozsoy, and Lale Akarun

    Necati Cihan Camg¨oz, Ahmet Alp Kındıroglu, Serpil Karab¨ukl¨u, Meltem Kelepir, A. Sumru¨Ozsoy, and Lale Akarun. Bospho- russign: A turkish sign language recognition corpus in health and finance domains. InInternational Conference on Language Resources and Evaluation, 2016. 2

    2016

  4. [11]

    SubUNets: End-to-end hand shape and continuous sign language recognition

    Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, and Richard Bowden. SubUNets: End-to-end hand shape and continuous sign language recognition. InProc. ICCV, 2017. 2

    2017

  5. [12]

    Neural sign language translation

    Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. Neural sign language translation. In Proc. CVPR, 2018. 2, 5, 17

    2018

  6. [13]

    Sign language transformers: Joint end-to-end sign language recognition and translation

    Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. Sign language transformers: Joint end-to-end sign language recognition and translation. InProc. CVPR, 2020. 2, 5

    2020

  7. [14]

    Quo vadis, action recognition? A new model and the Kinetics dataset

    Jo˜ao Carreira and Andrew Zisserman. Quo vadis, action recognition? A new model and the Kinetics dataset. In Proc. CVPR, 2017. 2

    2017

  8. [15]

    CLAIR: Evaluating image captions with large language models

    David Chan, Suzanne Petryk, Joseph Gonzalez, Trevor Darrell, and John Canny. CLAIR: Evaluating image captions with large language models. InProc. EMNLP, 2023. 5, 12

    2023

  9. [16]

    A simple multi-modality transfer learning baseline for sign language translation

    Y utong Chen, Fangyun Wei, Xiao Sun, Zhirong Wu, and Stephen Lin. A simple multi-modality transfer learning baseline for sign language translation. InProc. CVPR, 2022. 2

    2022

  10. [17]

    Two-stream network for sign language recognition and translation

    Y utong Chen, Ronglai Zuo, Fangyun Wei, Y u Wu, Shujie Liu, and Brian Mak. Two-stream network for sign language recognition and translation. InNeurIPS, 2022. 2

    2022

  11. [18]

    Factorized learning assisted with large language model for gloss-free sign language translation

    Zhigang Chen, Benjia Zhou, Jun Li, Jun Wan, Zhen Lei, Ning Jiang, Quan Lu, and Guoqing Zhao. Factorized learning assisted with large language model for gloss-free sign language translation. arXiv, 2024. 7

    2024

  12. [19]

    Fully convolutional networks for continuous sign language recognition

    Ka Leong Cheng, Zhaoyang Y ang, Qifeng Chen, and Y u-Wing Tai. Fully convolutional networks for continuous sign language recognition. InProc. ECCV, 2020. 2

    2020

  13. [20]

    Nonmanual structures in sign language

    Onno Crasborn. Nonmanual structures in sign language. Encyclopedia of Language and Linguistics, 2006. 1

    2006

  14. [21]

    A deep neural framework for continuous sign language recognition by iterative training

    Runpeng Cui, Hu Liu, and Changshui Zhang. A deep neural framework for continuous sign language recognition by iterative training. IEEE Transactions on Multimedia, 2019. 2

    2019

  15. [22]

    Flashattention-2: Faster attention with better parallelism and work partitioning.arXiv, 2023

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning.arXiv, 2023. 5

    2023

  16. [23]

    How2Sign: A large-scale multimodal dataset for continuous American Sign Language

    Amanda Duarte, Shruti Palaskar, Lucas V entura, Deepti Ghadi- yaram, Kenneth DeHaan, Florian Metze, Jordi Torres, and Xavier Giro-i Nieto. How2Sign: A large-scale multimodal dataset for continuous American Sign Language. InProc. CVPR, 2021. 4, 5

    2021

  17. [24]

    Sign language video retrieval with free-form textual queries

    Amanda Duarte, Samuel Albanie, Xavier Gir´o-i Nieto, and G¨ul V arol. Sign language video retrieval with free-form textual queries. InProc. CVPR, 2022. 4, 13, 16

    2022

  18. [25]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Y ang, Angela Fan, et al. The llama 3 herd of models. arXiv, 2024. 3, 4, 5

    2024

  19. [26]

    The confluence of space and language in signed languages

    Karen Emmorey. The confluence of space and language in signed languages. The MIT Press, 1996. 2

    1996

  20. [27]

    Elicitation and corpus of spontaneous sign language discourse representation diagrams

    Michael Filhol. Elicitation and corpus of spontaneous sign language discourse representation diagrams. In Proceedings of the LREC2020 9th W orkshop on the Representation and Processing of Sign Languages: Sign Language Resources in the Service of the Language Community, T echno...

    2020

  21. [28]

    Llms are good sign language translators

    Jia Gong, Lin Geng Foo, Yixuan He, Hossein Rahmani, and Jun Liu. Llms are good sign language translators. InProc. CVPR,

  22. [29]

    Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural ’networks

    Alex Graves, Santiago Fern´andez, Faustino Gomez, and J¨urgen Schmidhuber. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural ’networks. In Proc. ICML, 2006. 2

    2006

  23. [30]

    Accelerate: Training and inference at scale made simple, efficient and adaptable

    Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Marc Sun, and Benjamin Bossan. Accelerate: Training and inference at scale made simple, efficient and adaptable. https : //github.com/huggingface/accelerate, 2022. 17

    2022

  24. [31]

    Ksl-guide: A large-scale korean sign language dataset including interrogative sentences for guiding the deaf and hard-of-hearing

    Soomin Ham, Kibaek Park, Y eongJun Jang, Y oungtaek Oh, Seokmin Y un, Sukwon Y oon, Chang Jo Kim, Han-Mu Park, and In So Kweon. Ksl-guide: A large-scale korean sign language dataset including interrogative sentences for guiding the deaf and hard-of-hearing. InProc. FG, 2021. 2

    2021

  25. [32]

    Self-mutual distillation learning for continuous sign language recognition

    Aiming Hao, Y uecong Min, and Xilin Chen. Self-mutual distillation learning for continuous sign language recognition. In Proc. ICCV, 2021. 2

    2021

  26. [33]

    Gaussian error linear units (gelus)

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv, 2016. 4

    2016

  27. [34]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, Y elong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Y uanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InProc. ICLR,

  28. [35]

    Continuous sign language recognition with correlation network

    Lianyu Hu, Liqing Gao, Zekang Liu, and Wei Feng. Continuous sign language recognition with correlation network. InProc. CVPR, 2023. 14

    2023

  29. [36]

    Sign language recognition using 3D convolutional neural networks

    Jie Huang, Wengang Zhou, Houqiang Li, and Weiping Li. Sign language recognition using 3D convolutional neural networks. In Proc. International Conference on Multimedia and Expo (ICME),

  30. [37]

    CoSign: Exploring co-occurrence signals in skeleton-based continuous sign language recognition

    Peiqi Jiao, Y uecong Min, Y anan Li, Xiaotao Wang, Lei Lei, and Xilin Chen. CoSign: Exploring co-occurrence signals in skeleton-based continuous sign language recognition. InProc. ICCV, 2023. 2

    2023

  31. [38]

    Visual alignment pre- training for sign language translation

    Peiqi Jiao, Y uecong Min, and Xilin Chen. Visual alignment pre- training for sign language translation. InProc. ECCV, 2024. 3, 7

    2024

  32. [39]

    MS-ASL: A large-scale data set and benchmark for understanding american sign language

    Hamid Reza V aezi Joze and Oscar Koller. MS-ASL: A large-scale data set and benchmark for understanding american sign language. InBMVC, 2019. 2

    2019

  33. [40]

    Kingma and Jimmy Ba

    Diederik P . Kingma and Jimmy Ba. Adam: A method for stochastic optimization. InProc. ICLR, 2015. 5, 14

    2015

  34. [41]

    Continuous sign language recognition: Towards large vocabulary statistical recognition systems handling multiple signers.Computer V ision and Image Understanding, 2015

    Oscar Koller, Jens Forster, and Hermann Ney. Continuous sign language recognition: Towards large vocabulary statistical recognition systems handling multiple signers.Computer V ision and Image Understanding, 2015. 2, 5, 17

    2015

  35. [42]

    Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison

    Dongxu Li, Cristian Rodriguez Opazo, Xin Y u, and Hongdong Li. Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. InProc. WACV, 2019. 2

    2019

  36. [43]

    Transferring cross-domain knowledge for video sign language recognition

    Dongxu Li, Xin Y u, Chenchen Xu, Lars Petersson, and Hongdong Li. Transferring cross-domain knowledge for video sign language recognition. InProc. CVPR, 2020. 2

    2020

  37. [44]

    BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. InProc. ICML, 2023. 4, 16

    2023

  38. [45]

    Four functions of a locus: Reexamining the structure of space in ASL.Sign Language Research: Theoretical Issues, 1990

    Scott Liddell. Four functions of a locus: Reexamining the structure of space in ASL.Sign Language Research: Theoretical Issues, 1990. 1

    1990

  39. [46]

    Liddell.Grammar , Gesture, and Meaning in American Sign Language

    Scott K. Liddell.Grammar , Gesture, and Meaning in American Sign Language. Cambridge University Press, 2003. 2

    2003

  40. [47]

    ROUGE: A package for automatic evaluation of summaries

    Chin-Y ew Lin. ROUGE: A package for automatic evaluation of summaries. InProc. ACL, 2004. 5, 12

    2004

  41. [48]

    Gloss-free end-to-end sign language translation

    Kezhou Lin, Xiaohan Wang, Linchao Zhu, Ke Sun, Bang Zhang, and Yi Y ang. Gloss-free end-to-end sign language translation. In Proc. ACL, 2023. 3

    2023

  42. [49]

    Diab, V eselin Stoyanov, and Xian Li

    Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mon...

    2021

  43. [50]

    Video swin transformer

    Ze Liu, Jia Ning, Y ue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. InProc. CVPR, 2022. 2, 4

    2022

  44. [51]

    Meier, Kearsy Cormier, and David Quinto-Pozos, editors

    Richard P . Meier, Kearsy Cormier, and David Quinto-Pozos, editors. Modality and Structure in Signed and Spoken Languages. Cambridge University Press, 2002. 1

    2002

  45. [52]

    Autsl: A large scale multi-modal turkish sign language dataset and baseline methods

    Ozge Mercanoglu and Hacer Keles. Autsl: A large scale multi-modal turkish sign language dataset and baseline methods. IEEE Access, 8:181340–181355, 2020. 2

    2020

  46. [53]

    Visual alignment constraint for continuous sign language recognition

    Y uecong Min, Aiming Hao, Xiujuan Chai, and Xilin Chen. Visual alignment constraint for continuous sign language recognition. InProc. ICCV, 2021. 2

    2021

  47. [54]

    Automatic dense annotation of large-vocabulary sign language videosa

    Liliane Momeni, Hannah Bull, K R Prajwal, Samuel Albanie, G¨ul V arol, and Andrew Zisserman. Automatic dense annotation of large-vocabulary sign language videosa. InProc. ECCV, 2022. 4, 5, 6

    2022

  48. [55]

    Findings of the second WMT shared task on sign language translation (WMT-SL T23)

    Mathias M¨uller, Malihe Alikhani, Eleftherios Avramidis, Richard Bowden, Annelies Braffort, Necati Cihan Camg¨oz, Sarah Ebling, Cristina Espa˜na-Bonet, Anne G¨ohring, Roman Grundkiewicz, Mert Inan, Zifan Jiang, Oscar Koller, Amit Moryossef, Annette Rios, Dimitar Shterionov, Sa...

  49. [56]

    GPT-4 technical report.arXiv:2303.08774, 2024

    OpenAI. GPT-4 technical report.arXiv:2303.08774, 2024. 5, 12, 13, 18

    2024 arXiv

  50. [57]

    Maxime Oquab, Timoth´ee Darcet, Th´eo Moutakanni, Huy V . V o, Marc Szafraniec, V asil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Y ao Huang, Shang-Wen Li, Ishan Misra, Michael...

    2024

  51. [58]

    BLEU: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a method for automatic evaluation of machine translation. In Proc. ACL, 2002. 2, 5, 12

    2002

  52. [59]

    Weakly-supervised fingerspelling recognition in british sign language videos

    K R Prajwal, Hannah Bull, Liliane Momeni, Samuel Albanie, G¨ul V arol, and Andrew Zisserman. Weakly-supervised fingerspelling recognition in british sign language videos. In Proc. BMVC, 2022. 2, 4 10

    2022

  53. [60]

    Iterative alignment network for continuous sign language recognition

    Junfu Pu, Wen gang Zhou, and Houqiang Li. Iterative alignment network for continuous sign language recognition. In Proc. CVPR, 2019. 2

    2019

  54. [61]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InProc. ICML, 2021. 3

    2021

  55. [62]

    A tale of two languages: Large-vocabulary continuous sign language recognition from spoken language supervision.arXiv, 2024

    Charles Raude, K R Prajwal, Liliane Momeni, Hannah Bull, Samuel Albanie, Andrew Zisserman, and G¨ul V arol. A tale of two languages: Large-vocabulary continuous sign language recognition from spoken language supervision.arXiv, 2024. 2, 4, 5, 13, 16

    2024

  56. [63]

    Towards privacy-aware sign language translation at scale

    Phillip Rust, Bowen Shi, Skyler Wang, Necati Cihan Camgoz, and Jean Maillard. Towards privacy-aware sign language translation at scale. InProc. ACL, 2024. 3, 7

    2024

  57. [64]

    TTIC’s submission to WMT-SLT 23

    Marcelo Sandoval-Castaneda, Y anhong Li, Bowen Shi, Diane Brentari, Karen Livescu, and Gregory Shakhnarovich. TTIC’s submission to WMT-SLT 23. In Proceedings of the Eighth Conference on Machine Translation, 2023. 3

    2023

  58. [65]

    BLEURT: Learning robust metrics for text generation

    Thibault Sellam, Dipanjan Das, and Ankur P Parikh. BLEURT: Learning robust metrics for text generation. InProc. ACL, 2020. 5, 12

    2020

  59. [66]

    Open-domain sign language translation learned from online video

    Bowen Shi, Diane Brentari, Gregory Shakhnarovich, and Karen Livescu. Open-domain sign language translation learned from online video. InProc. EMNLP, 2022. 3

    2022

  60. [68]

    Using an LLM to turn sign spottings into spoken language sentences.arXiv, 2024

    Ozge Mercanoglu Sincan, Necati Cihan Camgoz, and Richard Bowden. Using an LLM to turn sign spottings into spoken language sentences.arXiv, 2024. 3

    2024

  61. [69]

    The linguistics of British Sign Language: An introduction

    Rachel Sutton-Spence and Bencie Woll. The linguistics of British Sign Language: An introduction. Cambridge University Press, 1999. 1, 2

    1999

  62. [70]

    Reconsidering Sentence-Level Sign Language Translation

    Garrett Tanzer, Maximus Shengelia, Ken Harrenstien, and David Uthus. Reconsidering Sentence-Level Sign Language Translation. arXiv, 2024. 1

    2024

  63. [71]

    G´allego, Amanda Duarte, Jordi Torres, and Xavier Gir´o i Nieto

    Laia Tarr´es, Gerard I. G´allego, Amanda Duarte, Jordi Torres, and Xavier Gir´o i Nieto. Sign language translation from instructional videos. In Proc. CVPRW, 2023. 3

    2023

  64. [72]

    Llama: Open and efficient foundation language models.arXiv, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv, 2023. 3

    2023

  65. [73]

    Y ouTube-ASL: A large-scale, open-domain american sign language-english parallel corpus

    David Uthus, Garrett Tanzer, and Manfred Georg. Y ouTube-ASL: A large-scale, open-domain american sign language-english parallel corpus. In Proc. NeurIPS Datasets and Benchmarks Track, 2023. 3, 7

    2023

  66. [74]

    Lawrence Zitnick, and Devi Parikh

    Ramakrishna V edantam, C. Lawrence Zitnick, and Devi Parikh. CIDEr: Consensus-based image description evaluation. InProc. CVPR, 2015. 5

    2015

  67. [75]

    BEVT: BERT pretraining of video transformers

    Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Y u-Gang Jiang, Luowei Zhou, and Lu Y uan. BEVT: BERT pretraining of video transformers. InProc. CVPR,

  68. [76]

    Improving continuous sign language recognition with cross-lingual signs

    Fangyun Wei and Y utong Chen. Improving continuous sign language recognition with cross-lingual signs. InProc. ICCV,

  69. [77]

    Phonological and prosodic layering of nonman- uals in american sign language.The signs of language revisited: An anthology to honor Ursula Bellugi and Edward Klima, 2000

    Ronnie Wilbur. Phonological and prosodic layering of nonman- uals in american sign language.The signs of language revisited: An anthology to honor Ursula Bellugi and Edward Klima, 2000. 1

    2000

  70. [78]

    Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation

    Ryan Wong, Necati Cihan Camgoz, and Richard Bowden. Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation. InProc. ICLR, 2024. 2, 3, 4, 5, 6, 7, 9, 15, 17

    2024

  71. [79]

    Knowcomp submis- sion for wmt23 sign language translation task

    Baixuan Xu, Haochen Shi, Tianshi Zheng, Qing Zong, Weiqi Wang, Zhaowei Wang, and Y angqiu Song. Knowcomp submis- sion for wmt23 sign language translation task. InProceedings of the Eighth Conference on Machine Translation (WMT), 2023. 3

    2023

  72. [80]

    Sign language translation with iterative prototype

    Huijie Y ao, Wengang Zhou, Hao Feng, Hezhen Hu, Hao Zhou, and Houqiang Li. Sign language translation with iterative prototype. InProc. ICCV, 2023. 2

    2023

  73. [81]

    Cross-modality data augmentation for end-to-end sign language translation

    Jinhui Y e, Wenxiang Jiao, Xing Wang, Zhaopeng Tu, and Hui Xiong. Cross-modality data augmentation for end-to-end sign language translation. InProc. EMNLP, 2023. 2

    2023

  74. [82]

    Improving gloss-free sign language translation by reducing representation density.arXiv, 2024

    Jinhui Y e, Xing Wang, Wenxiang Jiao, Junwei Liang, and Hui Xiong. Improving gloss-free sign language translation by reducing representation density.arXiv, 2024. 3

    2024

  75. [83]

    Better sign language translation with STMC-transformer

    Kayo Yin and Jesse Read. Better sign language translation with STMC-transformer. InProc. COLING, 2020. 2

    2020

  76. [84]

    SLTUNET: A simple unified model for sign language translation

    Biao Zhang, Mathias M¨uller, and Rico Sennrich. SLTUNET: A simple unified model for sign language translation. InProc. ICLR, 2023. 2

    2023

  77. [85]

    Scaling sign language translation.arXiv, 2024

    Biao Zhang, Garrett Tanzer, and Orhan Firat. Scaling sign language translation.arXiv, 2024. 3, 7

    2024

  78. [86]

    CVT-SLR: Contrastive visual-textual transformation for sign language recognition with variational alignment

    Jiangbin Zheng, Yile Wang, Cheng Tan, Siyuan Li, Ge Wang, Jun Xia, Yidong Chen, and Stan Z Li. CVT-SLR: Contrastive visual-textual transformation for sign language recognition with variational alignment. InProc. CVPR, 2023. 2

    2023

  79. [87]

    Gloss-free sign language translation: Improving from visual-language pretraining

    Benjia Zhou, Zhigang Chen, Albert Clap´es, Jun Wan, Y anyan Liang, Sergio Escalera, Zhen Lei, and Du Zhang. Gloss-free sign language translation: Improving from visual-language pretraining. InProc. ICCV, 2023. 2, 3, 5, 6, 7, 15, 17

    2023

  80. [88]

    Improving sign language translation with monolingual data by sign back-translation

    Hao Zhou, Wengang Zhou, Weizhen Qi, Junfu Pu, and Houqiang Li. Improving sign language translation with monolingual data by sign back-translation. InProc. CVPR, 2021. 2, 5

    2021

  81. [89]

    Spatial-temporal multi-cue network for sign language recognition and translation

    Hao Zhou, Wengang Zhou, Y un Zhou, and Houqiang Li. Spatial-temporal multi-cue network for sign language recognition and translation. IEEE Transactions on Multimedia, 2021. 2

    2021

  82. [90]

    man” was identified as the most common noun, “standing

    Ronglai Zuo and Brian Mak. C2SLR: Consistency-enhanced continuous sign language recognition. InProc. CVPR, 2022. 2 11 APPENDIX This appendix supplements the main paper by providing additional implementation details (Appendix A), experiments (Appendix B), and qualitative result...

    2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.